How good are the "Chinese text segmentation algorithms" at isolating words from a string of Chinese characters?

Question

I am looking for a way to segment a Chinese sentence into the words that make the sentence up, using custom software, like this:

import pynlpir
pynlpir.open()

s = '欢迎科研人员、技术工程师、企事业单位与个人参与NLPIR平台的建设工作。'
pynlpir.segment(s)

[('欢迎', 'verb'), ('科研', 'noun'), ('人员', 'noun'), ('、', 'punctuation mark'), ('技术', 'noun'), ('工程师', 'noun'), ('、', 'punctuation mark'), ('企事业', 'noun'), ('单位', 'noun'), ('与', 'conjunction'), ('个人', 'noun'), ('参与', 'verb'), ('NLPIR', 'noun'), ('平台', 'noun'), ('的', 'particle'), ('建设', 'verb'), ('工作', 'verb'), ('。', 'punctuation mark')]

Essentially that is taking the string of characters 欢迎科研人员、技术工程师、企事业单位与个人参与NLPIR平台的建设工作。 and dividing it into the words.

I am looking to segment Chinese texts automatically/programmatically with code.

This StackOverflow question/answers has a bunch of links to "Chinese text segmentation" open source software, like this one which references the MMSG segmentation algorithm, originally invented by Chih-Hao Tsai I guess. Here (I think) is some more detail on that.

My question is, how good/accurate are these sorts of "segmentation algorithms"? Do they get it right 100% of the time, 10% of the time, 50% of the time, sort of thing? Are they as good as a human at segmenting things into words, or much worse, or even better? And what are the best algorithms, if I may ask that additional related question?

Answer 1

tsh has commented it already, but I am going to elaborate it as an answer since I beleieve it should apply:

Chinese does not contain spacing to separate vocabulary. For people it is no issue to read as they can use context to determine the correct places to split vocabulary. If you have any familiarity with ai and programs you know that we are still very far away from anything computerized understanding context. therefore it is actually impossible to have a program/algorythm accurately parse a language like chinese that is so heavily context based-- at least currently.

As tsh mentioned 99% of the time such algorythms will simply segment at the longest possible point, with the basic idea that five characters in a row making one term intentionally is way more statistically likely than those five characters being in a row by coincidence. Note, it is true that its statistically less likely to occur, but still realistically happens all the time. Two character vocab getting segmented out when it should have been each character individually is probably the biggest offender.

Once in awhile you may see an algorythm that attempts to combine this longest term parsing with more common term parsing, the way google translate attempts to choose the most common phrasing for the definitions it gives in sentences. This is uncommon, probably because it doesn't eliminate the inherent innacuracy of segmenting chinese and so doesn't really add much.

Another main issue with machine parsing of chinese is also context related, of the dropped context variety. Consider the fact that a single character in a phrase could actually be representing a longer vocab that was abbreviated due to the full term not being deemed necessary for clarity. Also consider that the dropped term could be the entire subject, object, tense, or predicate. When you consider all these factors of reading between the lines that are impossible to program, it becomes very clear why parsing chinese accurately will probably be a major benchmark of ai improving to human level comrehension haha.