I am looking for a way to segment a Chinese sentence into the words that make the sentence up, using custom software, like this:

import pynlpir

s = '欢迎科研人员、技术工程师、企事业单位与个人参与NLPIR平台的建设工作。'

[('欢迎', 'verb'), ('科研', 'noun'), ('人员', 'noun'), ('、', 'punctuation mark'), ('技术', 'noun'), ('工程师', 'noun'), ('、', 'punctuation mark'), ('企事业', 'noun'), ('单位', 'noun'), ('与', 'conjunction'), ('个人', 'noun'), ('参与', 'verb'), ('NLPIR', 'noun'), ('平台', 'noun'), ('的', 'particle'), ('建设', 'verb'), ('工作', 'verb'), ('。', 'punctuation mark')]

Essentially that is taking the string of characters 欢迎科研人员、技术工程师、企事业单位与个人参与NLPIR平台的建设工作。 and dividing it into the words.

I am looking to segment Chinese texts automatically/programmatically with code.

This StackOverflow question/answers has a bunch of links to "Chinese text segmentation" open source software, like this one which references the MMSG segmentation algorithm, originally invented by Chih-Hao Tsai I guess. Here (I think) is some more detail on that.

My question is, how good/accurate are these sorts of "segmentation algorithms"? Do they get it right 100% of the time, 10% of the time, 50% of the time, sort of thing? Are they as good as a human at segmenting things into words, or much worse, or even better? And what are the best algorithms, if I may ask that additional related question?

  • 2
    A simple dictionary based solution which always try to match longest word in dictionary should have at least 80% precision. However, if you want better solutions, maybe you should read some NLP articles.
    – tsh
    Commented Apr 14, 2023 at 7:42
  • This is not a Chinese language question. “How good is algorithm X” is determined at a minimum by benchmark test cases and data sets.
    – dROOOze
    Commented Apr 14, 2023 at 7:54
  • I think we (yes, we) need to have a good plan and separate the whole pie into slices. We may have a couple of hundreds of questions to ask here.
    – PdotWang
    Commented Apr 14, 2023 at 17:17
  • This work may not only be related to words but also to grammar rules.
    – PdotWang
    Commented Apr 14, 2023 at 17:19

1 Answer 1


tsh has commented it already, but I am going to elaborate it as an answer since I beleieve it should apply:

Chinese does not contain spacing to separate vocabulary. For people it is no issue to read as they can use context to determine the correct places to split vocabulary. If you have any familiarity with ai and programs you know that we are still very far away from anything computerized understanding context. therefore it is actually impossible to have a program/algorythm accurately parse a language like chinese that is so heavily context based-- at least currently.

As tsh mentioned 99% of the time such algorythms will simply segment at the longest possible point, with the basic idea that five characters in a row making one term intentionally is way more statistically likely than those five characters being in a row by coincidence. Note, it is true that its statistically less likely to occur, but still realistically happens all the time. Two character vocab getting segmented out when it should have been each character individually is probably the biggest offender.

Once in awhile you may see an algorythm that attempts to combine this longest term parsing with more common term parsing, the way google translate attempts to choose the most common phrasing for the definitions it gives in sentences. This is uncommon, probably because it doesn't eliminate the inherent innacuracy of segmenting chinese and so doesn't really add much.

Another main issue with machine parsing of chinese is also context related, of the dropped context variety. Consider the fact that a single character in a phrase could actually be representing a longer vocab that was abbreviated due to the full term not being deemed necessary for clarity. Also consider that the dropped term could be the entire subject, object, tense, or predicate. When you consider all these factors of reading between the lines that are impossible to program, it becomes very clear why parsing chinese accurately will probably be a major benchmark of ai improving to human level comrehension haha.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.