7

This post gives some great statistics on Chinese character frequency, but I'm curious about the total number of distinct characters that appear in a "typical" book. By "typical" I mean something like a popular modern novel: say, "The Three-Body Problem," or the translation of the first Harry Potter.

I'm guessing the number will be disturbingly high, much greater than 3000, which I think I've heard quoted as a good benchmark for reading fluently. If, as suggested in the first link, the top 3000 characters cover at most 99.4% of characters that appear in random text, that's depressingly low. It means if you know 3000 characters, then on every page (I just counted around 1000 characters/page in Harry Potter) there's a rough average of six characters you need to either look up or accept not knowing how to pronounce!

  • I'm voting to close this question as off-topic because it has nothing to do with Chinese studying. – Tang Ho Feb 4 at 5:54
  • 2
    I'd like to know how many characters I'd have to "study" in order to read a book comfortably without having to constantly look up characters or guess at pronunciations. – WillG Feb 4 at 5:59
  • Even if you knew every single characters in dictionary, that would be ten of thousands of them, you still can't read Chinese books without knowing compound words and Chinese grammar. What do you expect ? If you have to learn too many characters then you quit? You study one character word at a time, you study one compound word at a time, and you study one grammar rule after another until you reach the level that you can read typical Chinese text, How does knowing this trivial fact of "How many distinct characters a typical Chinese book contains" help you? – Tang Ho Feb 4 at 6:22
  • 1
    I won’t quit if I have to learn too many characters, but it’s helpful for me to have a concrete benchmark to strive toward (# of characters in a book) so that I can measure my progress with one extra metric. If someone were training to run a long race and asked how many miles it would be, would you say “who cares, you just need to keep running one mile at a time until you improve”? – WillG Feb 4 at 17:46
  • 1
    @WillG As a native who once roughly measured the amount of characters at my command and is confident of reading basically any modern material without the need of looking up dictionary, my answer is if you know 7000~8000 distinct characters, you will face no problem reading.(/≧ω\) – Toosky Hierot Feb 5 at 16:54
5

Counting distinct characters/words in chinese books is really easy, but what exactly does "typical" means?

You can solve it by python. Then every time you meet a "typical" book(txt,mobi,azw format etc), you can count it by yourself.

input

  • The txt file of The Three-Body Problem I:Remembrance of Earth's Past
  • The txt file of The Three-Body Problem II:The Dark Forest
  • The txt file of The Three-Body Problem III:Death's End

program

import sys
import string
from collections import defaultdict
file_loc="tb3.txt"
try:
    f = open(file_loc,"r",encoding='gbk')
    allstr=f.read()
except UnicodeDecodeError:
    f = open(file_loc,"r",encoding='utf-8')
    allstr=f.read()

#dict to count chars, default to zero
d=defaultdict(int) 
for i in allstr:
    d[i]+=1

#count how many dinstinct words
#import jieba
#for i in jieba.cut(allstr):
#    d[i]+=1

# one pass filter, you can just use this filter
for c in string.printable:
    d.pop(c,0)

# select chinese characters directly
unicode_ranges = [0x3007,0x3007],[0x3400,0x4DBF],[0x4E00,0x9FEF],[0x20000,0x2EBFF]
for i,key in list(enumerate(d)):
    if not any(map(lambda x:x[0]<=ord(key)<=x[1],unicode_ranges)):
        del d[key]
print(len(d))

# map word_count to list of chars
d2=defaultdict(list)
for key,val in d.items():
    d2[val].append(key)

#sort by val
for key,val in sorted(d2.items(),key=lambda x:x[0],reverse=True): 
    print(key,val)

chinese character reference

conclusion

  1. three body problem 1: 2854
  2. three body problem 2: 2986
  3. three body problem 3: 3034

(welcome to add your count results of other books here)

3000 characters cover 100%!,

You can use sum(d.values()) to count the total #chinese-characters in three body problem 1, you will find 2109 characters covers 99.393256% 2370 chars(delete char that occurs only once) covers 99.7%, In addition,

You said

there's a rough average of six characters you need to either look up or accept not knowing how to pronounce!

That's false in the 99.4% version, you can learn a meaning of character in this way. You are likely to meet the same char again in this book if you only know 2109 characters.

If you are insterested in details, see all the resultshere

  • Thanks, this is a handy piece of code with really fun stats it can generate. And I’m delighted, the situation is much more optimistic than I had hoped! – WillG Feb 18 at 6:58
  • Very nice! I wonder if there is any easy way to count the number of “words”? – Tom Gewecke Feb 18 at 17:38
  • @TomGewecke yes, good advice, I'll perfect it in my answer. – Voyager Feb 18 at 19:50
4

Actually, I think a more interesting question is how many distinct characters there are in a book that aren't considered "common". To that effect, I used the 2854 results from rambler's answer for the three body problem 1 and the list of the 3500 commonly used characters (一级字/常用字) as presented in 通用规范汉字表, which was published in 2013 by the government of China. Then I ran the following python script (I had to remove the data values because of space limitations here)--you can see it in action here:

# -*- coding: utf-8 -*-
unicode_ranges = [0x3007,0x3007],[0x3400,0x4DBF],[0x4E00,0x9FEF],[0x20000,0x2EBFF]

most_common = list("") #insert stuff here!

most_common = [hanzi for hanzi in most_common if any(map(lambda x:x[0]<=ord(hanzi)<=x[1],unicode_ranges))]

print(len(most_common))

threebody1result =  list("") #insert stuff here!

threebody1result = [hanzi for hanzi in threebody1result if any(map(lambda x:x[0]<=ord(hanzi)<=x[1],unicode_ranges))]
print(len(threebody1result))

not_common_words = [hanzi for hanzi in threebody1result if not hanzi in most_common]
print(not_common_words)
print(len(not_common_words))

I found 191 characters in the book that weren't in the list of characters considered "most common". A number of these are just onomatopoeia and characters that appear to be used primarily for names.

['淼', '霖', '纣', '陛', '伽', '阮', '穹', '曦', '朕', '雯', '楠', '眸', '篝', '眩', '兮', '炽', '粼', '阈', '忏', '咔', '嚓', '嘟', '姬', '嗯', '噼', '羲', '沌', '袅', '烬', '咯', '摞', '咚', '瞥', '踹', '曳', '噬', '嗤', '礴', '骸', '哒', '皙', '呗', '瘠', '阱', '惚', '舷', '瘙', '呓', '萦', '湍', '漉', '柯', '瑶', '咝', '觑', '炯', '诣', '锢', '褶', '涟', '漪', '瓮', '骷', '髅', '冕', '悻', '铠', '啷', '呸', '汐', '婷', '氦', '狙', '睑', '霏', '垩', '睽', '栩', '啾', '晗', '恫', '韦', '坍', '蜿', '癫', '柞', '栎', '瞟', '擀', '瘴', '狍', '潦', '渥', '囔', '盥', '咣', '攥', '壕', '侃', '遛', '裔', '撂', '涸', '诧', '霆', '涓', '晷', '幺', '揶', '晖', '髂', '戾', '溏', '噢', '杳', '惺', '忪', '霓', '靓', '颚', '寐', '诏', '纂', '羟', '黝', '鬈', '臆', '锵', '遁', '诠', '攘', '虔', '吋', '啐', '茨', '戟', '嬴', '楔', '俑', '鞘', '亘', '咦', '迸', '邃', '慑', '犄', '蹒', '跚', '昕', '鄢', '搐', '瞌', '剐', '嗔', '恻', '皲', '碴', '炙', '橇', '阂', '飒', '仟', '锨', '黏', '蚜', '咫', '谙', '燎', '悸', '袒', '皈', '湮', '啬', '铰', '梵', '斓', '煦', '阙', '邸', '甄', '狩', '坞', '镊', '磐', '潞', '怫', '汲', '熵', '霾', '湛', '眺']

4

OK, here is my test.

Dragonlance: Dragons of Autumn Twilight

total: 238,872 (only Chinese is counted),

individual: 2,882

I have never read this book, 2,882 is a friend number.There isn't a lot of 'magic' words in a fantasy novel.


Sherlock·Holmes: A Study in Scarlet

Total:72,217

Individual_sum: 2,546


聊斋志异

total:389,165,

individual:4,935

Classical is 2 times complicate than modern novels.(wrong)


射雕英雄传

total:781,529,

individual:4,088

A wuxia novel that full of traditional.


三国志

total:443,202

individual:4,433

Classical history book.


The Shawshank Redemption

total:69,530

individual: 2,439


The big four.

红楼梦 total:730,537, individual:4,251

水浒传 total":705,811, individual: 4,074

西游记 total":604,456, individual: 4,574

三国演义 total:486,106, individual:4,025


Finally, an Internet novel.

东北往事:黑道风云二十年(一): total:679,231, individual:3,625,

0

IMO, a direct answer should be obvious: It depends!

我觉得嘛,与其思考一本书里会有多少互不相同的汉字,不如关注一些更实际的事情。根据维基百科 (English):

Studies in China have shown that functional literacy in written Chinese requires a knowledge of between three and four thousand characters.

也就是说,如果你想要无障碍进行中文读写,需要认识 3 千到 4 千汉字。取决于你打算阅读的内容及来源1,你可能需要认识更多字。例如,一些古典名著(如《西游记》)会出现更多的现在已经不常用的字。

另一方面,常见的内容通常不会包含太多生僻字,所以据我自己估计,根据内容不同,一本书通常会包含 2 千到 5 千不同的字。一本《读者》杂志,每期64页,包含 1.5k 到 2k 不同汉字;而一本长篇小说(如《活着》,余华),则有 4 千以上。(Trivia: 小学一年级的语文书甚至都有 1 千以上的不同汉字)

如果你需要进行统计研究,我建议你获取电子版文档然后自己处理;如果你只是为了估计流畅阅读需要认识的字数,那么我觉得没必要在乎,遇到不认识的字查字典就行了。


Notes:

  1. 近几十年来中国大陆、Malaysia 和 Singapore 的出版物以简体中文为主,各自有一点用词等方面的偏好;而港澳台 (Hong Kong, Macau and Taiwan) 的出版物以繁体中文为主。如果你希望同时读懂这两种文字,你还需要认识更多字。
  • typing Chinese is easier for me ATM. I can give this answer in English if needed. – iBug Feb 7 at 12:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.