3

I am looking to do the (seemingly simple) task of creating a simple frequency table for traditional Chinese characters (not words). I'm encountering the following problems:

  1. Most resources are in simplified Chinese, such as all of these.
  2. Traditional Chinese resources are either too small (like this one which also has some weird non-identifiable characters such as #2959) or too old (like this one from 1993–1994 which also has 資 in 11th place which I think is very strange, even though it's very large (13000+ chars).
  3. I have also tried to find Traditional Chinese corpora in a usable format to calculate the frequency myself (which is obviously barely any effort), but all corpora I could find where either paid or not accessible/downloadable like this and this. I also downloaded the zhwiki-latest-pages-articles.xml.bz2 from here to use Wikipedia as a resource but it mixes simplified and traditional and I couldn't find any way of only getting the traditional Chinese Wikipedia articles.

I am surprised at how difficult it is to find any of the resources I'm looking for. If you know of any accessible Mandarin corpus with traditional characters (nothing fancy, don't need any parsing, tagging, tokenizing or other annotation, just pure text) or how to quickly build one, feel free to share! Thanks in advance.

2
  • The entire Chinese wikipedia is in Mandarin using traditional characters, perhaps you can crawl it? EDIT: Nevermind, apparently you say it is not. Look for Taiwanese resources would be a place where I would start.
    – Marko
    Commented Feb 4, 2020 at 3:13
  • Building the corpus on my own does sound fun (no sarcasm), but the effort is totally not worth it since it's for a small project only. If this is the only way, then I might as well just take the 1993–1994 one which is probably still the best out of the ones I mentioned in the question. Commented Feb 4, 2020 at 12:07

1 Answer 1

5

I made an effort to summarise available information about character and word frequencies for learners and teachers a while back and came up with the following which are relevant for traditional characters.

  1. Chih-hao Tsai has a list of well over 13 000 characters, described as follows: "The corpus consists of all the BIG-5 Chinese characters appeared on Usenet newsgroups during 1993-1994. It consists of 171,882,493 characters. This is perhaps by far the largest Chinese corpus in the world." More info here.
  2. The following list is also linked to on that page, referring to frequency data from Shih-Kun Huang. Maybe both lists are based on the same data; I'm not sure.
  3. I don't know about any downloadable corpora for traditional Chinese, but I have mainly relied on Academia Sinica Balanced Corpus of Modern Chinese, which is also tokenised, although I don't know if there's any frequency data based on that corpus.
  4. The Taiwan MoE offers a frequency lists that is available for free, based on mixed sources totalling around two million characters. You can get it here, but you have to download and run an executable file in Windows (but why?). It does work, though. Info about the samples used can be found here. The sources are not as new as one would wish, but they are more balanced than online Usenet newsgroups at least.

I summarised all the resources I found for both simplified and traditional Chinese, for both character components, characters and words here, in case anyone is interested: The most common Chinese words, characters and components for language learners and teachers

6
  • Hey Olle, thanks for your response and I really dig your website which I've read a lot of articles from in the course of studying Mandarin. However I must admit that the resources you named are both included in the ones I presented and don't work for me for the reasons described in my question. Commented Feb 5, 2020 at 10:43
  • Sorry about that, I somehow didin't realise that it was the same resource. There should be newer sources, I'll have a closer look and I'll update the answer if I find anything.
    – Olle Linge
    Commented Feb 5, 2020 at 10:52
  • Awesome, thanks a lot! I did spend a lot of time searching in English and Chinese and couldn't find anything, so I appreciate your time. Good luck! Commented Feb 6, 2020 at 6:23
  • 1
    @KonstantinH. I have updated my answer with the only additional resource I could find after spending quite a lot of time looking and asking a few people. I don't think there are any more sources freely available. I guess it would be possible to search the Academia Sinica corpus character for character (it gives frequencies at the top), but doing this manually is out of the question, of course.
    – Olle Linge
    Commented Feb 21, 2020 at 18:56
  • Thanks a lot of the effort! I had actually come across this one but didn't take a closer look because I don't have a Windows PC ... Commented Feb 24, 2020 at 6:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.