I am doing some work with scanning Chinese text with my computer and I have run into a problem that I do not have with other languages.

Because there are no spaces between characters, I cannot tell when one word ends and another one begins. Is there any easy way to do this? For example in English I am telling my computer to split things like this:

sentence = "My name is John"
the computer will return a list of words.
computer = ['my', 'name', 'is', 'John']
  • 1
    No easy way, no. Sometimes a character can belong to the character before it, be on its own, or belong to the character after it, and it will change the meaning ... – Ming Dec 2 '15 at 2:09
  • 1
    You'd probably need to match it against some sort of database. – user3306356 Dec 2 '15 at 2:31
  • 5
    Chinese word segmentation (中文分词) is a difficult problem in the field of natural language processing (NLP) that has been studying for a long time. There're a few solutions such as Stanford Word Segmenter, MMSEG, and Sphhinx. – Stan Dec 2 '15 at 6:53

Short of a lexicographic analysis, you can't. There is a little punctuation in Chinese, so you can split sentences into subgroups. But after that, there's no easy way to say where a word stops.

Even very common words like 的 (genitive marker, appears right after the possessor, eg 我的朋友: me, I; genitive; friend: my friend) cannot be relied on 100% to be always just that. 'Taxi' is written 的士, and in this case 的 would not be a single character delimiting two words, but the first character of a word.

And Chinese is not the only language that doesn't have spaces. You'll run into more problems like this.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.