Character and word frequencies

Question

I'm trying to build some automated tools for processing Chinese text. I'd like to know the relative frequencies of different characters and character combinations. Are there any web sites with this data that I can download?

Answer 1

Chih-Hao Tsai has published a bunch of useful resources like this. I've used the character frequency and phrase frequency lists.

Answer 2

The Unihan standard contains the kFrequency field. This fields gives you a general idea of how frequent the character shows up. This data and other data gets aggregated from scholarly sources which the standard discloses in its notes. Note, Unihan will also contain the Korean and Japanese variants in use. As far as I know, it does not have data for character combinations. Some of the other answers point to resources that does have that data.

See: http://www.unicode.org/charts/unihan.html

and

http://www.unicode.org/reports/tr38/#AlphabeticalListing and search for field, kFrequency.

I have prepared Unihan 5.1.0 into a sqlite database a couple years ago, here: http://guwen.hiddenstorehouse.com/

The current version is Unihan 6.0 and I have not processed it.

Answer 3

Jun Da has published character frequency, bigram and phoneme frequency lists here.

I used the character and bigram frequency lists for my final degree project, creating an application to allow for input of handwritten Chinese using neural networks combined with other techniques. Very helpful they were too :)

If you are interested, bigrams are groups of two characters. For example,一个 has a high frequency of occurence in Jun Da's data, as 个 often follows 一. Along with the character frequencies, I loaded the bigram data into a database and used it as part of an algorithm that was used to suggest the next character as a user drew characters with a mouse or stylus.

Answer 4

Alan Hoenig's book, http://www.amazon.com/Chinese-Characters-Learn-Remember-Meanings/dp/0982232403 takes almost 2200 Chinese characters and ranks them from 1 to whatever by frequency of use, using "official" Chinese sources.

I was surprised to find out that 一 is only number two. The most frequent character is the possessive, 的.