24

I'm trying to build some automated tools for processing Chinese text. I'd like to know the relative frequencies of different characters and character combinations. Are there any web sites with this data that I can download?

Improve this question

6 Answers 6

Reset to default
12

Chih-Hao Tsai has published a bunch of useful resources like this. I've used the character frequency and phrase frequency lists.

Improve this answer
8

Jun Da has published character frequency, bigram and phoneme frequency lists here.

I used the character and bigram frequency lists for my final degree project, creating an application to allow for input of handwritten Chinese using neural networks combined with other techniques. Very helpful they were too :)

If you are interested, bigrams are groups of two characters. For example,一个 has a high frequency of occurence in Jun Da's data, as 个 often follows 一. Along with the character frequencies, I loaded the bigram data into a database and used it as part of an algorithm that was used to suggest the next character as a user drew characters with a mouse or stylus.

Improve this answer
6

The Unihan standard contains the kFrequency field. This fields gives you a general idea of how frequent the character shows up. This data and other data gets aggregated from scholarly sources which the standard discloses in its notes. Note, Unihan will also contain the Korean and Japanese variants in use. As far as I know, it does not have data for character combinations. Some of the other answers point to resources that does have that data.

See: http://www.unicode.org/charts/unihan.html

and

http://www.unicode.org/reports/tr38/#AlphabeticalListing and search for field, kFrequency.

I have prepared Unihan 5.1.0 into a sqlite database a couple years ago, here: http://guwen.hiddenstorehouse.com/

The current version is Unihan 6.0 and I have not processed it.

Improve this answer
1
  • kFrequency isn't a very accurate frequency guage. For one thing, it only has about 5089 characters. Second, it only ranks characters on a scale of 1 to 5. Third, it's from usenet data... a bit old.
    – stevendaniels
    Commented Jun 4, 2012 at 10:03
5

Alan Hoenig's book, Chinese Characters: Learn & Remember 2,178 Characters and Their Meanings takes almost 2200 Chinese characters and ranks them from 1 to whatever by frequency of use, using "official" Chinese sources.

I was surprised to find out that 一 is only number two. The most frequent character is the possessive, 的.

Improve this answer
1
  • 1
    On every list I've seen 的 is number one.
    – stevendaniels
    Commented Jan 17, 2012 at 16:55
2

According to Zhonghua Zihai, the largest Chinese character dictionary, there are more than 85,000 Chinese characters! However, research (Huang 1994 and Da 2004) shows that the most frequently used 1200 Chinese Characters account for about 90% of the characters occurring in the real world. Here is the most used 1200 Chinese Characters by frequency.

http://www.chinesehulu.com/for-new-arrival/1200-most-frequently-used-chinese-characters-list-1

Improve this answer
2

I calculated the character frequencies for the Chinese Wikipedia. You have to check if this text corpus is representative for the texts you want to process. You can find the resulting frequencies here and the code being used here.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.