Character and word frequencies

Question

I'm trying to build some automated tools for processing Chinese text. I'd like to know the relative frequencies of different characters and character combinations. Are there any web sites with this data that I can download?

Answer 1

Chih-Hao Tsai has published a bunch of useful resources like this. I've used the character frequency and phrase frequency lists.

Answer 2

Jun Da has published character frequency, bigram and phoneme frequency lists here.

I used the character and bigram frequency lists for my final degree project, creating an application to allow for input of handwritten Chinese using neural networks combined with other techniques. Very helpful they were too :)

If you are interested, bigrams are groups of two characters. For example,一个 has a high frequency of occurence in Jun Da's data, as 个 often follows 一. Along with the character frequencies, I loaded the bigram data into a database and used it as part of an algorithm that was used to suggest the next character as a user drew characters with a mouse or stylus.

Answer 3

The Unihan standard contains the kFrequency field. This fields gives you a general idea of how frequent the character shows up. This data and other data gets aggregated from scholarly sources which the standard discloses in its notes. Note, Unihan will also contain the Korean and Japanese variants in use. As far as I know, it does not have data for character combinations. Some of the other answers point to resources that does have that data.

See: http://www.unicode.org/charts/unihan.html

and

http://www.unicode.org/reports/tr38/#AlphabeticalListing and search for field, kFrequency.

I have prepared Unihan 5.1.0 into a sqlite database a couple years ago, here: http://guwen.hiddenstorehouse.com/

The current version is Unihan 6.0 and I have not processed it.

Answer 4

Alan Hoenig's book, Chinese Characters: Learn & Remember 2,178 Characters and Their Meanings takes almost 2200 Chinese characters and ranks them from 1 to whatever by frequency of use, using "official" Chinese sources.

I was surprised to find out that 一 is only number two. The most frequent character is the possessive, 的.

Answer 5

According to Zhonghua Zihai, the largest Chinese character dictionary, there are more than 85,000 Chinese characters! However, research (Huang 1994 and Da 2004) shows that the most frequently used 1200 Chinese Characters account for about 90% of the characters occurring in the real world. Here is the most used 1200 Chinese Characters by frequency.

http://www.chinesehulu.com/for-new-arrival/1200-most-frequently-used-chinese-characters-list-1

Answer 6

I calculated the character frequencies for the Chinese Wikipedia. You have to check if this text corpus is representative for the texts you want to process. You can find the resulting frequencies here and the code being used here.