How does Wikipedia's simplified to traditional converter work?

Question

The relationship between simplified and traditional characters can be a headache for organizations using both orthographies.

"A number of surveys, such as [Xiandai 1986], have demonstrated that the 2000 most frequent SC characters account for approximately 97% of all characters occurring in contemporary SC corpora. Of these, 238 simplified forms, or almost 12%, are polygraphic; that is, they map to two or more traditional forms."

Nevertheless, it's a task that can one day be done relatively perfectly by a computer. Google Translate can do it for you more or less. But if you are converting a short story or a novel, the task is a little more difficult. That's why I actually prefer to use the Mediawiki converter.

Nevertheless, I don't really know how it works. I have a feeling the SCIM pinyin input tables are used somehow, and the tediously underdocumented non-bijective-mappings between the two orthographies coded for manually. Is that right?

Answer 1

The Mediawiki converter uses a combination of automatic information from the Unicode standard, SCIM tables, and other sources plus manual tweaks to build a set of translation tables. When going from Traditional to Simplified, some characters have been condensed into one. Translating back from Simplified to Traditional requires context that a computer is not likely to understand unless you implement a full translation engine, not just a text replacement engine. This is where the manual enhancements come into play.

Mediawiki has a build system which automatically combines the automatic and manual information into a PHP source file of translations between many different types of Chinese. It appears to cover dialectical differences for mainland Chinese, Hong Kong, Taiwan, and Singapore. The information is then changed out with PHP's strtr() function which replaces all occurrences of items in a given array with matches in the supplied string, longest matches first. This longest matching is the key when switching out characters because you have more context from the source to make a better replacement.

Source: Code which uses mediawiki's conversion functions: https://github.com/tszming/mediawiki-zhconverter/blob/master/mediawiki-zhconverter.inc.php

Build file to combine automatic conversion tables with manual tweaks: https://phabricator.wikimedia.org/diffusion/MW/browse/master/maintenance/language/zhtable/Makefile.py

Compiled translation tables (large file, 17k lines): https://phabricator.wikimedia.org/diffusion/MW/browse/master/includes/ZhConversion.php

Conversion tool to load tables for different conversions (used by first source link) https://phabricator.wikimedia.org/diffusion/MW/browse/master/languages/classes/LanguageZh.php

Answer 2

There is another converter at http://mandarintools.com/zhcode.html. I lack the capacity to judge its quality. Is that a suitable alternative? It may reveal how the conversion is done.

I may have overlooked some parts, but here is my understanding of it:

• It seems to have a large two-column list of words (hcutf8.txt). The first column contains simplified, the second column traditional. The second column occasionally contains two or more characters, in situations where one simplified character equals more than one traditional character.
• When converting from simplified to traditional, it replaces all occurrences of words found in the first column with words in the second column.
• When converting from traditional to simplified, it replaces all occurrences of words found in the second column with words in the first column.

Perhaps there is more in the code that I did not notice. Are additional steps needed for this conversion to be successful?