Chinese Language Stack Exchange is a question and answer site for students, teachers, and linguists wanting to discuss the finer points of the Chinese language. It only takes a minute to sign up.
Sign up to join this community
I am doing some work with scanning Chinese text with my computer and I have run into a problem that I do not have with other languages.
Because there are no spaces between characters, I cannot tell when one word ends and another one begins. Is there any easy way to do this? For example in English I am telling my computer to split things like this:
sentence = "My name is John"
the computer will return a list of words.
computer = ['my', 'name', 'is', 'John']
Short of a lexicographic analysis, you can't. There is a little punctuation in Chinese, so you can split sentences into subgroups. But after that, there's no easy way to say where a word stops.
Even very common words like 的 (genitive marker, appears right after the possessor, eg 我的朋友: me, I; genitive; friend: my friend) cannot be relied on 100% to be always just that. 'Taxi' is written 的士, and in this case 的 would not be a single character delimiting two words, but the first character of a word.
And Chinese is not the only language that doesn't have spaces. You'll run into more problems like this.
