I am looking for a way to segment a Chinese sentence into the words that make the sentence up, using custom software, like this:
import pynlpir pynlpir.open() s = '欢迎科研人员、技术工程师、企事业单位与个人参与NLPIR平台的建设工作。' pynlpir.segment(s) [('欢迎', 'verb'), ('科研', 'noun'), ('人员', 'noun'), ('、', 'punctuation mark'), ('技术', 'noun'), ('工程师', 'noun'), ('、', 'punctuation mark'), ('企事业', 'noun'), ('单位', 'noun'), ('与', 'conjunction'), ('个人', 'noun'), ('参与', 'verb'), ('NLPIR', 'noun'), ('平台', 'noun'), ('的', 'particle'), ('建设', 'verb'), ('工作', 'verb'), ('。', 'punctuation mark')]
Essentially that is taking the string of characters
欢迎科研人员、技术工程师、企事业单位与个人参与NLPIR平台的建设工作。 and dividing it into the words.
I am looking to segment Chinese texts automatically/programmatically with code.
This StackOverflow question/answers has a bunch of links to "Chinese text segmentation" open source software, like this one which references the MMSG segmentation algorithm, originally invented by Chih-Hao Tsai I guess. Here (I think) is some more detail on that.
My question is, how good/accurate are these sorts of "segmentation algorithms"? Do they get it right 100% of the time, 10% of the time, 50% of the time, sort of thing? Are they as good as a human at segmenting things into words, or much worse, or even better? And what are the best algorithms, if I may ask that additional related question?