How many distinct characters are there in a typical Chinese book?

Question

This post gives some great statistics on Chinese character frequency, but I'm curious about the total number of distinct characters that appear in a "typical" book. By "typical" I mean something like a popular modern novel: say, "The Three-Body Problem," or the translation of the first Harry Potter.

I'm guessing the number will be disturbingly high, much greater than 3000, which I think I've heard quoted as a good benchmark for reading fluently. If, as suggested in the first link, the top 3000 characters cover at most 99.4% of characters that appear in random text, that's depressingly low. It means if you know 3000 characters, then on every page (I just counted around 1000 characters/page in Harry Potter) there's a rough average of six characters you need to either look up or accept not knowing how to pronounce!

Answer 1

Counting distinct characters/words in chinese books is really easy, but what exactly does "typical" means?

You can solve it by python. Then every time you meet a "typical" book(txt,mobi,azw format etc), you can count it by yourself.

input

• The txt file of The Three-Body Problem I:Remembrance of Earth's Past
• The txt file of The Three-Body Problem II:The Dark Forest
• The txt file of The Three-Body Problem III:Death's End

program

import sys
import string
from collections import defaultdict
file_loc="tb3.txt"
try:
    f = open(file_loc,"r",encoding='gbk')
    allstr=f.read()
except UnicodeDecodeError:
    f = open(file_loc,"r",encoding='utf-8')
    allstr=f.read()

#dict to count chars, default to zero
d=defaultdict(int) 
for i in allstr:
    d[i]+=1

#count how many dinstinct words
#import jieba
#for i in jieba.cut(allstr):
#    d[i]+=1

# one pass filter, you can just use this filter
for c in string.printable:
    d.pop(c,0)

# select chinese characters directly
unicode_ranges = [0x3007,0x3007],[0x3400,0x4DBF],[0x4E00,0x9FEF],[0x20000,0x2EBFF]
for i,key in list(enumerate(d)):
    if not any(map(lambda x:x[0]<=ord(key)<=x[1],unicode_ranges)):
        del d[key]
print(len(d))

# map word_count to list of chars
d2=defaultdict(list)
for key,val in d.items():
    d2[val].append(key)

#sort by val
for key,val in sorted(d2.items(),key=lambda x:x[0],reverse=True): 
    print(key,val)

chinese character reference

conclusion

1. three body problem 1: 2854
2. three body problem 2: 2986
3. three body problem 3: 3034

(welcome to add your count results of other books here)

3000 characters cover 100%!,

You can use sum(d.values()) to count the total #chinese-characters in three body problem 1, you will find 2109 characters covers 99.393256% 2370 chars(delete char that occurs only once) covers 99.7%, In addition,

You said

there's a rough average of six characters you need to either look up or accept not knowing how to pronounce!

That's false in the 99.4% version, you can learn a meaning of character in this way. You are likely to meet the same char again in this book if you only know 2109 characters.

If you are insterested in details, see all the resultshere

Answer 2

Actually, I think a more interesting question is how many distinct characters there are in a book that aren't considered "common". To that effect, I used the 2854 results from rambler's answer for the three body problem 1 and the list of the 3500 commonly used characters (一级字/常用字) as presented in 通用规范汉字表, which was published in 2013 by the government of China. Then I ran the following python script (I had to remove the data values because of space limitations here)--you can see it in action here:

# -*- coding: utf-8 -*-
unicode_ranges = [0x3007,0x3007],[0x3400,0x4DBF],[0x4E00,0x9FEF],[0x20000,0x2EBFF]

most_common = list("") #insert stuff here!

most_common = [hanzi for hanzi in most_common if any(map(lambda x:x[0]<=ord(hanzi)<=x[1],unicode_ranges))]

print(len(most_common))

threebody1result =  list("") #insert stuff here!

threebody1result = [hanzi for hanzi in threebody1result if any(map(lambda x:x[0]<=ord(hanzi)<=x[1],unicode_ranges))]
print(len(threebody1result))

not_common_words = [hanzi for hanzi in threebody1result if not hanzi in most_common]
print(not_common_words)
print(len(not_common_words))

I found 191 characters in the book that weren't in the list of characters considered "most common". A number of these are just onomatopoeia and characters that appear to be used primarily for names.

['淼', '霖', '纣', '陛', '伽', '阮', '穹', '曦', '朕', '雯', '楠', '眸', '篝', '眩', '兮', '炽', '粼', '阈', '忏', '咔', '嚓', '嘟', '姬', '嗯', '噼', '羲', '沌', '袅', '烬', '咯', '摞', '咚', '瞥', '踹', '曳', '噬', '嗤', '礴', '骸', '哒', '皙', '呗', '瘠', '阱', '惚', '舷', '瘙', '呓', '萦', '湍', '漉', '柯', '瑶', '咝', '觑', '炯', '诣', '锢', '褶', '涟', '漪', '瓮', '骷', '髅', '冕', '悻', '铠', '啷', '呸', '汐', '婷', '氦', '狙', '睑', '霏', '垩', '睽', '栩', '啾', '晗', '恫', '韦', '坍', '蜿', '癫', '柞', '栎', '瞟', '擀', '瘴', '狍', '潦', '渥', '囔', '盥', '咣', '攥', '壕', '侃', '遛', '裔', '撂', '涸', '诧', '霆', '涓', '晷', '幺', '揶', '晖', '髂', '戾', '溏', '噢', '杳', '惺', '忪', '霓', '靓', '颚', '寐', '诏', '纂', '羟', '黝', '鬈', '臆', '锵', '遁', '诠', '攘', '虔', '吋', '啐', '茨', '戟', '嬴', '楔', '俑', '鞘', '亘', '咦', '迸', '邃', '慑', '犄', '蹒', '跚', '昕', '鄢', '搐', '瞌', '剐', '嗔', '恻', '皲', '碴', '炙', '橇', '阂', '飒', '仟', '锨', '黏', '蚜', '咫', '谙', '燎', '悸', '袒', '皈', '湮', '啬', '铰', '梵', '斓', '煦', '阙', '邸', '甄', '狩', '坞', '镊', '磐', '潞', '怫', '汲', '熵', '霾', '湛', '眺']

Answer 3

OK, here is my test.

Dragonlance: Dragons of Autumn Twilight

total: 238,872 (only Chinese is counted),

individual: 2,882

I have never read this book, 2,882 is a friend number. There isn't a lot of 'magic' words in a fantasy novel.

---

Sherlock·Holmes: A Study in Scarlet

Total:72,217

Individual_sum: 2,546

---

聊斋志异

total:389,165,

individual:4,935

Classical is 2 times complicate than modern novels. (wrong)

---

射雕英雄传

total:781,529,

individual:4,088

A wuxia novel that is full of traditional.

---

三国志

total:443,202

individual:4,433

A classical history book.

---

The Shawshank Redemption

total:69,530

individual: 2,439

---

The big four.

红楼梦 total:730,537, individual:4,251

水浒传 total":705,811, individual: 4,074

西游记 total":604,456, individual: 4,574

三国演义 total:486,106, individual:4,025

---

Finally, an Internet novel.

东北往事:黑道风云二十年(一)

total:679,231, individual:3,625,

Answer 4

Harry Potter

Referring to 哈利·波特与魔法石 (the version with the first sentence "家住女贞路4号的德思礼夫妇总是得意地说他们是非常规矩的人家。". There is another version floating around.)

• 132,016 total characters
• 2735 unique characters
• 2305 unique characters used more than once
• 87,584 total words (where a word is in CEDICT or a character name)
• 7780 unique words
• 4593 unique words used more than once

Answer 5

IMO, a direct answer should be obvious: It depends!

我觉得嘛，与其思考一本书里会有多少互不相同的汉字，不如关注一些更实际的事情。根据维基百科 (English)：

Studies in China have shown that functional literacy in written Chinese requires a knowledge of between three and four thousand characters.

也就是说，如果你想要无障碍进行中文读写，需要认识 3 千到 4 千汉字。取决于你打算阅读的内容及来源¹，你可能需要认识更多字。例如，一些古典名著（如《西游记》）会出现更多的现在已经不常用的字。

另一方面，常见的内容通常不会包含太多生僻字，所以据我自己估计，根据内容不同，一本书通常会包含 2 千到 5 千不同的字。一本《读者》杂志，每期64页，包含 1.5k 到 2k 不同汉字；而一本长篇小说（如《活着》，余华），则有 4 千以上。（Trivia: 小学一年级的语文书甚至都有 1 千以上的不同汉字）

如果你需要进行统计研究，我建议你获取电子版文档然后自己处理；如果你只是为了估计流畅阅读需要认识的字数，那么我觉得没必要在乎，遇到不认识的字查字典就行了。

---

Notes:

1. 近几十年来中国大陆、Malaysia 和 Singapore 的出版物以简体中文为主，各自有一点用词等方面的偏好；而港澳台 (Hong Kong, Macau and Taiwan) 的出版物以繁体中文为主。如果你希望同时读懂这两种文字，你还需要认识更多字。