Why do HSK lists differ from character frequency lists?

Question

I have been wondering how come the HSK list be so different from the list of the most used Chinese characters.

The only list I know about character frequency is Jun Da's list. On the first 2635 most common characters of the list (I picked 2635, because it's the total number of the HSK characters), there are exactly 305 characters that are not listed on any HSK.

On the rest of Jun Da's list (a total of 7000 characters) there are another 304 characters that are on the HSK list, but are way way farther than the most used characters (the last two rank at #4527 and #4771).

Questions:

1. What are the criteria for the character choice on HSK?
2. Why is the HSK list so different from the most used characters list?

Additional comments are very welcome.

Answer 1

To answer your second question:

From a mathematical perspective, it will depend on the texts that are sampled to compile any list and the variances associated with those samples. Languages evolve and there is no true character frequency, only an approximation. Spoken Chinese will have a different character frequency to written Chinese because the spoken language has a high entropy. It is likely that the HSK test had a panel of experts select characters based on what they deemed to be suitable, whereas the list you referenced was mass sampled by some kind of web crawler.

It is inconceivable that every single electronic Chinese character on the internet has been sampled, and if it was that would be very biased as that would ignore all of the written works not uploaded to the internet. Depending on the format of the inputs there will be a huge difference, if lots of text was taken from 微薄, then the character 哈 might rank as common, purely from comments that are like "hahahaha" for example. If classical Chinese was sampled heavily then characters like 曰 might feature much more frequently than 说.

Some of the HSK 5 and 6 words are very political and business oriented, but there are not many scientific words.

Answer 2

To give some partial information regarding your questions, Olle Linge at Hacking Chinese has a recent (July 2020) article What important words are missing from HSK?. One thing he mentions is that:

It should be clear that HSK is not meant to be a representation of the most commonly used Chinese words. This is very obvious in the lower levels, where words like “train station” and “bus” are part of HSK1, which has only 150 words in total. Those words are nowhere near the top 150 words in Chinese in general, but they are of course important for foreigners visiting and travelling in China, which probably is why they are included.

Overall, I think the lower levels of HSK match the needs of foreign students quite well.

Thus, one possibility is that the creators wanted to make sure that foreigners learned what they needed to get around in China when visiting/studying there. This would bias the most frequently used words away from more general frequency lists.

This being said, Olle also came up with a list of words from a frequency list gleaned from movie and TV subtitles (the SUBTLEX-CH corpus [Cai and Brysbaert, 2010]) that either do not appear in HSK or are delayed (ie. they are in HSK but appear in a higher HSK level than expected, based on their frequency). Based on this analysis, Olle found that these types of words seemed to be missing from the HSK lists:

• Many single-character words are missing
• Names of places and countries are missing
• Regional variants are missing
• Profanity is missing entirely
• Foreign things are mostly missing
• Particles in informal language are missing

Seeing what is missing from HSK could help give some insight into the decision making process behind making the lists. Olle goes into more detail about the types of words missing in his article, if you want to know more.

Also, Olle did the same analysis for TOCFL (though with a different frequency list): What important words are missing or delayed in TOCFL?

An important practical note: if you want to learn the words that are "missing" from the HSK (or TOCFL) lists, Olle has generated the word lists and placed them at the end of the article so that people can use them freely for their own study.

Frequency List Reference:

Cai, Q., & Brysbaert, M. (2010). SUBTLEX-CH: Chinese word and character frequencies based on film subtitles. PloS one, 5(6), e10729.