# How many distinct characters are there in a typical Chinese book?

This post gives some great statistics on Chinese character frequency, but I'm curious about the total number of distinct characters that appear in a "typical" book. By "typical" I mean something like a popular modern novel: say, "The Three-Body Problem," or the translation of the first Harry Potter.

I'm guessing the number will be disturbingly high, much greater than 3000, which I think I've heard quoted as a good benchmark for reading fluently. If, as suggested in the first link, the top 3000 characters cover at most 99.4% of characters that appear in random text, that's depressingly low. It means if you know 3000 characters, then on every page (I just counted around 1000 characters/page in Harry Potter) there's a rough average of six characters you need to either look up or accept not knowing how to pronounce!

• I'm voting to close this question as off-topic because it has nothing to do with Chinese studying.
– Tang Ho
Feb 4 '19 at 5:54
• I'd like to know how many characters I'd have to "study" in order to read a book comfortably without having to constantly look up characters or guess at pronunciations. Feb 4 '19 at 5:59
• Even if you knew every single characters in dictionary, that would be ten of thousands of them, you still can't read Chinese books without knowing compound words and Chinese grammar. What do you expect ? If you have to learn too many characters then you quit? You study one character word at a time, you study one compound word at a time, and you study one grammar rule after another until you reach the level that you can read typical Chinese text, How does knowing this trivial fact of "How many distinct characters a typical Chinese book contains" help you?
– Tang Ho
Feb 4 '19 at 6:22
• I won’t quit if I have to learn too many characters, but it’s helpful for me to have a concrete benchmark to strive toward (# of characters in a book) so that I can measure my progress with one extra metric. If someone were training to run a long race and asked how many miles it would be, would you say “who cares, you just need to keep running one mile at a time until you improve”? Feb 4 '19 at 17:46
• @WillG As a native who once roughly measured the amount of characters at my command and is confident of reading basically any modern material without the need of looking up dictionary, my answer is if you know 7000~8000 distinct characters, you will face no problem reading.(／≧ω＼) Feb 5 '19 at 16:54

Counting distinct characters/words in chinese books is really easy, but what exactly does "typical" means?

You can solve it by python. Then every time you meet a "typical" book(txt,mobi,azw format etc), you can count it by yourself.

## input

• The txt file of The Three-Body Problem I:Remembrance of Earth's Past
• The txt file of The Three-Body Problem II:The Dark Forest
• The txt file of The Three-Body Problem III:Death's End

## program

``````import sys
import string
from collections import defaultdict
file_loc="tb3.txt"
try:
f = open(file_loc,"r",encoding='gbk')
except UnicodeDecodeError:
f = open(file_loc,"r",encoding='utf-8')

#dict to count chars, default to zero
d=defaultdict(int)
for i in allstr:
d[i]+=1

#count how many dinstinct words
#import jieba
#for i in jieba.cut(allstr):
#    d[i]+=1

# one pass filter, you can just use this filter
for c in string.printable:
d.pop(c,0)

# select chinese characters directly
unicode_ranges = [0x3007,0x3007],[0x3400,0x4DBF],[0x4E00,0x9FEF],[0x20000,0x2EBFF]
for i,key in list(enumerate(d)):
if not any(map(lambda x:x[0]<=ord(key)<=x[1],unicode_ranges)):
del d[key]
print(len(d))

# map word_count to list of chars
d2=defaultdict(list)
for key,val in d.items():
d2[val].append(key)

#sort by val
for key,val in sorted(d2.items(),key=lambda x:x[0],reverse=True):
print(key,val)

``````

chinese character reference

## conclusion

1. three body problem 1: 2854
2. three body problem 2: 2986
3. three body problem 3: 3034

3000 characters cover 100%!,

You can use `sum(d.values())` to count the total #chinese-characters in three body problem 1, you will find 2109 characters covers 99.393256% 2370 chars(delete char that occurs only once) covers 99.7%, In addition,

You said

there's a rough average of six characters you need to either look up or accept not knowing how to pronounce!

That's false in the 99.4% version, you can learn a meaning of character in this way. You are likely to meet the same char again in this book if you only know 2109 characters.

If you are insterested in details, see all the resultshere

• Thanks, this is a handy piece of code with really fun stats it can generate. And I’m delighted, the situation is much more optimistic than I had hoped! Feb 18 '19 at 6:58
• Very nice! I wonder if there is any easy way to count the number of “words”? Feb 18 '19 at 17:38
• @TomGewecke yes, good advice, I'll perfect it in my answer. Feb 18 '19 at 19:50

Actually, I think a more interesting question is how many distinct characters there are in a book that aren't considered "common". To that effect, I used the 2854 results from rambler's answer for the three body problem 1 and the list of the 3500 commonly used characters (一级字/常用字) as presented in 通用规范汉字表, which was published in 2013 by the government of China. Then I ran the following python script (I had to remove the data values because of space limitations here)--you can see it in action here:

``````# -*- coding: utf-8 -*-
unicode_ranges = [0x3007,0x3007],[0x3400,0x4DBF],[0x4E00,0x9FEF],[0x20000,0x2EBFF]

most_common = list("") #insert stuff here!

most_common = [hanzi for hanzi in most_common if any(map(lambda x:x[0]<=ord(hanzi)<=x[1],unicode_ranges))]

print(len(most_common))

threebody1result =  list("") #insert stuff here!

threebody1result = [hanzi for hanzi in threebody1result if any(map(lambda x:x[0]<=ord(hanzi)<=x[1],unicode_ranges))]
print(len(threebody1result))

not_common_words = [hanzi for hanzi in threebody1result if not hanzi in most_common]
print(not_common_words)
print(len(not_common_words))
``````

I found 191 characters in the book that weren't in the list of characters considered "most common". A number of these are just onomatopoeia and characters that appear to be used primarily for names.

['淼', '霖', '纣', '陛', '伽', '阮', '穹', '曦', '朕', '雯', '楠', '眸', '篝', '眩', '兮', '炽', '粼', '阈', '忏', '咔', '嚓', '嘟', '姬', '嗯', '噼', '羲', '沌', '袅', '烬', '咯', '摞', '咚', '瞥', '踹', '曳', '噬', '嗤', '礴', '骸', '哒', '皙', '呗', '瘠', '阱', '惚', '舷', '瘙', '呓', '萦', '湍', '漉', '柯', '瑶', '咝', '觑', '炯', '诣', '锢', '褶', '涟', '漪', '瓮', '骷', '髅', '冕', '悻', '铠', '啷', '呸', '汐', '婷', '氦', '狙', '睑', '霏', '垩', '睽', '栩', '啾', '晗', '恫', '韦', '坍', '蜿', '癫', '柞', '栎', '瞟', '擀', '瘴', '狍', '潦', '渥', '囔', '盥', '咣', '攥', '壕', '侃', '遛', '裔', '撂', '涸', '诧', '霆', '涓', '晷', '幺', '揶', '晖', '髂', '戾', '溏', '噢', '杳', '惺', '忪', '霓', '靓', '颚', '寐', '诏', '纂', '羟', '黝', '鬈', '臆', '锵', '遁', '诠', '攘', '虔', '吋', '啐', '茨', '戟', '嬴', '楔', '俑', '鞘', '亘', '咦', '迸', '邃', '慑', '犄', '蹒', '跚', '昕', '鄢', '搐', '瞌', '剐', '嗔', '恻', '皲', '碴', '炙', '橇', '阂', '飒', '仟', '锨', '黏', '蚜', '咫', '谙', '燎', '悸', '袒', '皈', '湮', '啬', '铰', '梵', '斓', '煦', '阙', '邸', '甄', '狩', '坞', '镊', '磐', '潞', '怫', '汲', '熵', '霾', '湛', '眺']

OK, here is my test.

Dragonlance: Dragons of Autumn Twilight

total: 238,872 (only Chinese is counted),

individual: 2,882

I have never read this book, 2,882 is a friend number. There isn't a lot of 'magic' words in a fantasy novel.

Sherlock·Holmes: A Study in Scarlet

Total:72,217

Individual_sum: 2,546

total:389,165,

individual:4,935

Classical is 2 times complicate than modern novels. (wrong)

total:781,529,

individual:4,088

A wuxia novel that is full of traditional.

total:443,202

individual:4,433

A classical history book.

The Shawshank Redemption

total:69,530

individual: 2,439

The big four.

Finally, an Internet novel.

total:679,231, individual:3,625,

# Harry Potter

Referring to 哈利·波特与魔法石 (the version with the first sentence "家住女贞路4号的德思礼夫妇总是得意地说他们是非常规矩的人家。". There is another version floating around.)

• 132,016 total characters
• 2735 unique characters
• 2305 unique characters used more than once
• 87,584 total words (where a word is in CEDICT or a character name)
• 7780 unique words
• 4593 unique words used more than once

IMO, a direct answer should be obvious: It depends!

Studies in China have shown that functional literacy in written Chinese requires a knowledge of between three and four thousand characters.

Notes:

1. 近几十年来中国大陆、Malaysia 和 Singapore 的出版物以简体中文为主，各自有一点用词等方面的偏好；而港澳台 (Hong Kong, Macau and Taiwan) 的出版物以繁体中文为主。如果你希望同时读懂这两种文字，你还需要认识更多字。
• typing Chinese is easier for me ATM. I can give this answer in English if needed.
– iBug
Feb 7 '19 at 12:39