18

The relationship between simplified and traditional characters can be a headache for organizations using both orthographies.

"A number of surveys, such as [Xiandai 1986], have demonstrated that the 2000 most frequent SC characters account for approximately 97% of all characters occurring in contemporary SC corpora. Of these, 238 simplified forms, or almost 12%, are polygraphic; that is, they map to two or more traditional forms."

Nevertheless, it's a task that can one day be done relatively perfectly by a computer. Google Translate can do it for you more or less. But if you are converting a short story or a novel, the task is a little more difficult. That's why I actually prefer to use the Mediawiki converter.

Nevertheless, I don't really know how it works. I have a feeling the SCIM pinyin input tables are used somehow, and the tediously underdocumented non-bijective-mappings between the two orthographies coded for manually. Is that right?

7
  • 2
    Hello Magnetar. Thanks for posting but I don't think your question is on topic, I voted to close. It's more related to how some web-software works rather than the Chinese language. I think it's more fit in sites like Superuser or similar, I'm not sure which one, honestly... :D
    – Alenanno
    Dec 25, 2011 at 21:18
  • Do you have a link to all of the source code? Is this project considered completely finished?
    – Village
    Dec 26, 2011 at 6:16
  • Would you support this question, @Alenanno, if it didn't focus exclusively on the Mediawiki converter? Something like, "What techniques can software use to convert simplified characters to traditional characters when simplified characters can map to more than one? For example, Mediawiki seems to work well."
    – Don Kirkby
    Dec 26, 2011 at 7:17
  • 1
    @DonKirkby Uhm It still seems too related to software. Anyway, I wanted to be clear that this questions is not a bad one (and as you can see I didn't down vote), it's an interesting question to me as well, but I think that the knowledge required to answer is more on the computer side than the chinese side.
    – Alenanno
    Dec 26, 2011 at 11:17
  • @magnetar - Can you confirm if you want a technical or non-technical answer to your question? Are you looking for someone to explain how this can be done in code or just a general explanation to how it is done?
    – going
    Jan 12, 2012 at 1:23

2 Answers 2

7

The Mediawiki converter uses a combination of automatic information from the Unicode standard, SCIM tables, and other sources plus manual tweaks to build a set of translation tables. When going from Traditional to Simplified, some characters have been condensed into one. Translating back from Simplified to Traditional requires context that a computer is not likely to understand unless you implement a full translation engine, not just a text replacement engine. This is where the manual enhancements come into play.

Mediawiki has a build system which automatically combines the automatic and manual information into a PHP source file of translations between many different types of Chinese. It appears to cover dialectical differences for mainland Chinese, Hong Kong, Taiwan, and Singapore. The information is then changed out with PHP's strtr() function which replaces all occurrences of items in a given array with matches in the supplied string, longest matches first. This longest matching is the key when switching out characters because you have more context from the source to make a better replacement.

Source: Code which uses mediawiki's conversion functions: https://github.com/tszming/mediawiki-zhconverter/blob/master/mediawiki-zhconverter.inc.php

Build file to combine automatic conversion tables with manual tweaks: https://phabricator.wikimedia.org/diffusion/MW/browse/master/maintenance/language/zhtable/Makefile.py

Compiled translation tables (large file, 17k lines): https://phabricator.wikimedia.org/diffusion/MW/browse/master/includes/ZhConversion.php

Conversion tool to load tables for different conversions (used by first source link) https://phabricator.wikimedia.org/diffusion/MW/browse/master/languages/classes/LanguageZh.php

1
  • 1
    I've updated the links from the old SVN repository to the current one. A lot of work is ongoing for the Chinese language converter, see the recent huge patch: phabricator.wikimedia.org/…
    – Nemo
    Apr 5, 2015 at 8:59
1

There is another converter at http://mandarintools.com/zhcode.html. I lack the capacity to judge its quality. Is that a suitable alternative? It may reveal how the conversion is done.

I may have overlooked some parts, but here is my understanding of it:

  • It seems to have a large two-column list of words (hcutf8.txt). The first column contains simplified, the second column traditional. The second column occasionally contains two or more characters, in situations where one simplified character equals more than one traditional character.
  • When converting from simplified to traditional, it replaces all occurrences of words found in the first column with words in the second column.
  • When converting from traditional to simplified, it replaces all occurrences of words found in the second column with words in the first column.

Perhaps there is more in the code that I did not notice. Are additional steps needed for this conversion to be successful?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.