Is it true that the core essential characters(not vocabulary) used throughout China's history didn't change much?

Question

Recently I have made an interesting observation.

Although it is said that there are more than 80,000 characters in the most complete Chinese character dictionary, the majority are either variants of otherwise common characters or archaic and obsolete characters. According to the 通用规范汉字表 published by China's Ministry of Education, which divides 8105 characters into 3 levels, and according to its official description,

一级字表为常用字集，收字3500个，主要满足基础教育和文化普及的基本用字需要。二级字表收字3000个，使用度仅次于一级字。一、二级字表合计6500字，主要满足出版印刷、辞书编纂和信息处理等方面的一般用字需要。三级字表收字1605个，是姓氏人名、地名、科学技术术语和中小学语文教材文言文用字中未进入一、二级字表的较通用的字，主要满足信息化时代与大众生活密切相关的专门领域的用字需要。

the 6500 characters contained in level 1 and 2 should be more than enough to cover more than 99% of character usage for all media and publications found in modern day China. In fact, some sources would lower the requirement to read 99% to as few as 3000 characters. That said, most native speakers cannot even recognize all 6500.

Now, this should be an accurate representation of the usage of characters in modern Chinese. Looking at Middle Chinese, I find that the characters used in literatures from that era are almost the same as today, with obsolete character being the exception rather than the norm. What I mean is that the essential core set of characters used (most basic 3000 in modern Chinese compared to most basic 3000 in Middle Chinese) are largely the same, albeit some characters having shifted in meaning (走run -> walk、哭cry(shout) -> cry(shed tear),食verb+noun -> noun etc.)

Let's look at some statistics I have gathered (I calculated this myself with a script):

For the entire 唐诗三百首, there are:

Total characters: 23684
Unique characters: 2572
Level One characters: 2065
Level Two characters: 472
Others: 35
["晖","雊","馀","摵","飗","朓","閟","澒","鞿","迳","珓","曈","昽","掎","湓","於","哳","愬","隩","豗","衱","鞚","剺","鄹","鄜","橦","翃","矰","岧","畤","俜","飐","稹","晼","蟢"]

The most common 6500 in modern Chinese covered 98.63% of the 300 poems written in Tang Dynasty, with level one along covering 80.28%. Even among the 35 characters not in 6500 most common, I think 晖 and 曈 are still recognized by most Chinese people today (because of 余晖 and 千门万户曈曈日 which I see every single 新年)

Going back even further to Old Chinese, we have 诗经.

Total characters: 31770
Unique characters: 2747
Level One characters: 1678
Level Two characters: 655
Others: 414
["濩","绤","斁","罍","砠","瘏","痡","樛","藟","蕡","罝","椓","芣","袺","襭","惄","赪","喓","趯","惙","筥","紽","緎","塈","嘒","裯","麕","帨","尨","襛","驺","豝","訧","曀","睆","忮","鷕","卬","笱","洸","褎","泲","窭","泚","篨","浼","鄘","髧","冓","玼","鬒","髢","瑱","揥","晳","瑳","沬","彊","騋","蝃","旟","閟","蝱","芃","僴","咺","槃","薖","褧","蝤","螓","罛","濊","鳣","菼","朅","垝","咥","籊","滺","觿","韘","跂","殳","痗","佸","翿","蓷","暵","仳","歗","吪","罦","罿","漘","啍","襢","裼","狃","掤","鬯","麃","袪","蓵","魗","萚","藘","漙","瀼","儇","緌","蓺","忉","丱","偲","鞹","沵","儦","襋","藚","岵","貆","囷","慆","襮","匊","杕","湑","佽","睘","盬","蔹","驖","俴","楘","辀","靷","馵","觼","厹","韔","縢","涘","駮","檖","枌","鬷","荍","纻","晢","侜","甓","鹝","懰","慅","悁","慱","鬵","祋","咮","鸤","觱","馌","畯","斨","鵙","葽","豜","墐","薁","穋","绹","翛","蜎","蠋","烝","臝","蟏","畽","罭","疐","恌","啴","鵻","骃","韡","矧","酾","藇","羜","糇","饎","禴","骙","旐","痯","罶","鲿","衎","楰","耇","鞗","弨","櫜","炰","菑","涖","軧","玱","焞","麀","麌","晣","厎","絷","蓫","葍","喤","犉","濈","惔","瘥","膴","讻","酲","癙","瘉","惸","蹐","炤","佌","蔌","哿","崒","棸","楀","醘","氵","慭","噂","勚","暬","遹","潝","訿","蜾","蠃","鸴","踧","疢","漼","雊","掎","扡","幠","餤","毚","尰","饛","捄","氿","鞙","璲","桋","疧","湝","鼛","絜","祊","踖","戁","苾","畇","雰","霡","霂","骍","膋","耔","薿","螣","渰","穧","禋","韎","韐","鞸","珌","觩","怲","舝","鷮","殽","籥","嘏","僛","傞","羖","绋","膍","饇","瀌","暱","愒","襜","滮","煁","懆","鹙","蹢","亹","冔","伣","瓞","陾","棫","駾","槱","楛","栵","檿","訽","馘","祃","翯","牣","虡","瞍","淢","嫄","寘","穟","幪","唪","秬","秠","穈","羝","斝","醓","臄","咢","醽","鹥","潨","鞞","泂","餴","墍","昄","翙","菶","憯","惛","怓","瘅","荛","熇","懠","烋","螗","奰","逷","痻","僾","荓","甡","瘗","瘨","崧","粻","鞃","幭","姞","噳","虓","粺","庤","镈","铚","稌","鞉","柷","鲦","雝","鸧","嬛","拚","嗿","畟","挃","紑","鼒","嶞","坰","驈","伾","驒","騢","驔","駜","駽","茷","黮","侐","稙","楅","胾","綅","糦","娀","絿","厖","梴"]

Despite being written over like 3000 years ago(11th to 7th centuries BC), the 2747 unique characters it uses still overlaps 84.92% with the most common 6500 characters today, only 414 not in this list.

This is by no means a comprehensive study and the samples I used are of course VERY little, compared to the VAST sea of literature works produced in Chinese history. However, due to my own limitations, I chose these two as I thought that they're the most representative of the most common characters used in their respective era, and I draw my conclusion that the core set of characters, despite shift in meaning of individual characters, has largely remained unchanged over the centuries. Thus, those tens of thousands of 生僻字 in big dictionaries are either variants of other common characters, or those that were used in that one place for that one era in China, but never gained its popularity and never made it into the core set. Maybe some characters only appears a few times throughout history and that's all, but then the dictionary has to add them.

Am I right on this observation? I am no linguist or 汉学家 so I'd really like some expert opinion on this.

Answer 1

Is it true that the core essential characters (not vocabulary) used throughout China's history didn't change much?

1. Pre-Qín (that is, before ~200 BCE), which is when the pieces in the Shījīng in your question were authored, this is emphatically not true;

2. Post-Jìn (after ~400 CE), which is when modern regular script matured and became dominant, with little change up until now when compared with Modern Orthodox Chinese, this could arguably be true, although not to the extent suggested by your analysis, which, among other problems, uses non-Orthodox Simplified Chinese;

3. Between these periods, and even more so before the Qín era, the analysis does not have meaning if you don't account for vocabulary changes. This is because the further back in time we go, the more likely we see characters which we recognise as "surviving till the modern era", but used in a different or even completely unrelated manner.

Since Pre-Qín examples are the simplest to demonstrate that the claim is not true, the examples below should be convincing that the methodology employed in the question is seriously flawed.

---

Inscriptions in the pre-Qín era use significant amounts of vocabulary which don't survive today; many of the characters used to write down this vocabulary don't exist now either. Take the following example:

Shāng-era oracle bones, 《甲骨文合集》32420

Transliteration (numbered items indicate no modern equivalent):

囗戼鼎：「王𠀠爯【1】，尞三𫳅，戼......牢？」

Annotated transliteration, or transliterations with modern character equivalents as a first step to translation:

丁卯貞：「王其稱𤨟，燎三【2】，劉......牢？」

Translation:

Divination on the day of dīngmǎo, it is asked: "Shall the king offer up a ceremonial jade weapon, conduct a burning ritual with three penned sheep, and perform a slaughtering ritual......penned cattle?"

Right from this example, we can see multiple issues which complicate any naïve attempt at frequency data analysis:

• We no longer use several of the Shāng-era words which appear here, along with the characters used to represent these words, like
  • Ceremonial jade weapon, represented by a pictogram of a piece of jade 【1】;
  • A name for a Shāng-era burning ritual 「尞」;
  • A dedicated word for penned sheep used for sacrifice, 「𫳅」.
• Digital published texts, the ones available for data analysis, overwhelmingly use annotated transliterations, even if they claim to reproduce the "original text". However, annotated transliterations are not the most faithful representation of the original text, and the "most faithful" representation itself varies based on the transliterator and the available characters which are digitally encoded. What set of characters, then, do you account for in the frequency analysis, when the annotated transliteration is obviously not the most faithful, e.g. 鼎 > 貞, 尞 > 燎, or doesn't even have a 1-to-1 mapping, like 戼 > both 卯 and 劉?
• Pairs like 鼎 vs 貞, 卯 vs 劉 show that you must account for vocabulary when doing the analysis.

Even when the vocabulary does map to modern equivalents, pre-Qín scribes may simply write these words differently. Take the following example containing fragments overlapping Analects, which is around the same time as the earliest excavated texts overlapping with the Shījīng:

Chǔ bamboo slips,《上海博物館藏戰國楚竹書・䊷衣》簡11

Transliteration:

子曰：「大臣𡳿㔻【⿱目辛】𠃟，則𢘑敬㔻𠯁，而【⿱富貝】【⿱貞貝】𢎟【⿺辶化】」

Annotated transliteration:

子曰：「大臣之不親也，則忠敬不足，而富貴已過」

Translation:

The Master said: If the ministers are not close to their monarch, then [their loyalty to him] and [his respect for them] will be not sufficient, yet their wealth and rank will be excessive.

Answer 2

Am I right on this observation?

very doubtful lah 😿

first of all, using simplified chinese to “handle” classical / literary chinese text is, . . . unacceptable

Others: 35

[ . . ."於" . . .]

the character “於” categorised in “others” 🙀 omg

the poem 蜀道難 by 李白 has the verse:

蜀道之難難於上青天

clearly, a case that “於” & “于” is not distinguished, in simplified chinese 😾

other significant cases are:

“鬱” & “郁” -> “郁”

“干” & “乾” & “幹” -> “干”

secondly, how do you treat oracle bone script, seal script, regular script; & other not so common, but important scripts?

thirdly, the sampling is too, . . . tiny 😹

github has some similar researches, using 全唐詩、宋詞 + other texts

anyway, solve the problem of “simplified chinese” first, before further analysis, imo 😸

have fun :)

Answer 3

I, which am not a 汉学家 either, found a video which might help your study. “简化字”的近百年极简史. Assuming you are quite familiar with Chinese to understand. After accepting the knowledge of the video, I start to doubt the designing of your research study, especially about conducting the analysis of an ancient language based on a database with a modern printing form.

I also think looking for more professional help on Chinese forum is optional because your theme is akin to academical researches. Waiting for a valuable answer to your question with any non-Chinese context forum is a good choice to spread and receive various perspectives yet increases the diifculty.

Below are some of my critical thoughts, but you know I am trying to be nice :D.

---

Your statistics are obtained with restricted database/collection. I'm not just saying the size is small or the source is single. In my opinion, the collection of 唐诗三百首 has two drawbacks.

1. Data format. It uses html texts directly, maybe a utf-8 encoding or something else. This leads to the amount of text words is limited. All the words must match to one certain target in the collection, which might violate the consistency from the original version. If there is not an universally acknowledged version, a better choice for your study is using traditional Chinese encoding instead of the simplified one, considering the background when the poems are created. Either you can demonstrate the encoding format is irrelevant, or you may find your statistics significantly different through traditional Chinese encoding. To me there exists another ultimate choice of data, that is the collection of photograph prints from ancient books, which might be inaccessible and dramatically increase the requirement of analysing system :-P.
2. Data quality. I also find your collection not highly polished. I compare a random piece of poem with another collection of 唐詩三百首 ver.2.

The version what you choose (id:93) is

余禁所禁垣西，是法厅事也。有古槐数株焉，虽生意可知，同殷仲文之古树，而听\n讼斯在，即周召伯之甘棠。每至夕照低阴，秋蝉疏引，发声幽息，有切尝闻；岂人\n心异於曩时，将虫响悲於前听？嗟乎！声以动容，德以象贤，故洁其身也，禀君子\n达人之高行；蜕其皮也，有仙都羽化之灵姿。候时而来，顺阴阳之数；应节为变，\n审藏用之机。有目斯开，不以道昏而昧其视；有翼自薄，不以俗厚而易其真。吟乔\n树之微风，韵资天纵；饮高秋之坠露，清畏人知。仆失路艰虞，遭时徽徽，\n不哀伤而自怨，未摇落而先衰。闻蟪蛄之流声，悟平反之已奏；见螳螂之抱影，怯\n危机之未安。感而缀诗，贻诸知己。庶情沿物应，哀弱羽之飘零；道寄人知，悯馀\n声之寂寞。非谓文墨，取代幽忧云尔。\n西路蝉声唱，南冠客思侵。\n那堪玄鬓影，来对白头吟！\n露重飞难进，风多响易沉。\n无人信高洁，谁为表予心？

The version what I found is

西 陸 蟬 聲 唱 ， 　南 冠 客 思 深 。 不 堪 玄 鬢 影 ， 　來 對 白 頭 吟 。 露 重 飛 難 進 ， 　風 多 響 易 沉 。 無 人 信 高 潔 ， 　誰 為 表 予 心 ？

It seems that the version you use includes additional information rather than the poem itself. Also notes that two versions has different words of 那堪/不堪.

Answer 4

我不是语言学家或汉学家，我只是母语为中文的人。

你的观察基本上是正确的，我想说的是有些汉字的写法可能变了，但是意思没有变。写法有大的改变是因为中华人民共和国成立后，简化汉字的进程加速推进，可以参考 https://zh.wikipedia.org/zh-cn/%E7%AE%80%E5%8C%96%E5%AD%97 比如说门的写法，由繁体字变为简体字。

汉语中有几个字的说法我感觉值得说下，繁体字，简体字，常用汉字， 通用规范汉字，中华人民共和国成立后，实现了繁体字到简体字的转变，通用规范汉字就是普通话里用的汉字，常用汉字你需要重点关注，这个相当于英语中的牛津3000词汇表，意味着在英语中出现频率很高，然后英语中还有COCA Frequency 20000和COCA Frequency 60000词频表，对于一个英文学习者来说，应该先掌握牛津3000词汇表，再掌握COCA Frequency 20000，到这里一般可以阅读大多数英文文章和书籍了，当然有兴趣的话再掌握COCA Frequency 60000。这样的方法同样适用于中文学习者，中文学习者应该先掌握一级字表中3500个常用字，再掌握二级字表中3000个字，到这个程度基本就可以了，有兴趣的话掌握三级字表中的1605个字。

我个人感觉你统计诗词中字出现的频率什么的没有意义，首先你统计的诗词中有不少我不认识的生僻字，其次现在实际生活中，没有人相互对诗句，实际人们交谈中也没有人说什么乎，比如有朋自远方来，不亦乐乎的说法。诗词只存在于现在中国小学，初中，高中的中文教育和考试中，严重和实际生活脱节，我个人认为学诗词是浪费时间。清朝以前的科举制度选拔靠诗词，要背八股文。有时间看看中国的四大名著感觉要比学诗词好些。

我个人感觉学诗词还不如多学学物理，化学，数学，探索世界奥秘。