I understand that automated Chinese word segmentation (breaking up text into constituent words) is a hard problem, as it's sometimes possible to segment the same sentence in different ways. As a result, most software packages that accomplish this task with high accuracy seem to use machine learning (e.g. https://pypi.org/project/pywordseg/ or https://nlp.stanford.edu/software/segmenter.shtml).

Naively, it seems like it should be possible to achieve modest accuracy with a much simpler method, especially if you're allowed to use a dictionary (like CEDict). For example, you could first match all four-character words, then three, then two.*

Would this be a sensible procedure, and would it produce reasonably accurate results? If not, how can one improve it without resorting to machine learning or something that takes a ton of manual effort?

(*ambiguities will occur, so you might choose at random. If you had a word frequency list, you could prioritize more commonly used words).

  • 1
    Some computer languages are interpreted languages. The interpreter analyses the code, line by line. What you want is an interpreter for human language. This is what AI is doing. Ever called a number and got an AI on the line? At least in China, keep saying 人工服务,eventually you will get to speak to a person, if you are lucky! If you are not deliberately obtuse, AI can deal with a lot of queries. The computer code to do that will however, be a lot more complex than say, a Python interpreter. Ask google for a glance at their AI code!
    – Pedroski
    Jan 27 at 3:06
  • 2
    Hei there. Here is a batter package pypi.org/project/jieba He seems to be doing better.
    – Chinohana
    Jan 28 at 12:11


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.