Thought Experiment: Twitter Character Limits

Question

This is hopefully a fun question that is a little bit off the standard path. It's meant to elicit a light conversation but nevertheless one that illuminates fundamental characteristics of Chinese as a written language.

Question: Twitter, which was designed with English speaking users in mind, has a limit of 140 characters per tweet, to enforce a certain brevity and effectively encourage the development of a continuous stream of new content. One advantage of Chinese is that, in Twitter, you can convey a LOT more information in 140 characters than you can in English! But just how much more? My question, and again of course this is purely in the realm of hypothetical, is: How many characters would Twitter have allow if it wanted to let English-language users convey as much information as Chinese speaking users now enjoy?

To go back to my earlier thread about the Hong Kong subway's warning about not standing in the way of a train's closing doors, in Chinese (Cantonese), it is: "請勿靠近車門" and in English is: "Please stand back from the doors." Almost magically, the two spoken languages, independently developed over the course of centuries on opposite sides of the planet, share precisely the same level of economy: Each announcement is just six syllables long. But there is a big disparity in the economy of the written languages. English consumes 27 written characters (and five spaces) to convey what Chinese says with just six characters.

So to extend this one random example into the world of Twitter character limits, where spaces between words do count against your limit, that's roughly five times more characters in English than in Chinese. Therefore, Twitter would have to raise its character limit to 700 characters for English language tweets to match a 140 character limit on Chinese language tweets. Or, looked at another way, Twitter would have to reduce the limit on Chinese language tweets to 28 characters to match the existing 140 character limit on English language tweets. Obviously this is not scientific or even remotely rigorous. Perhaps studies have been done on this question.

As a final disclaimer, I hope it is clear from this question that I am making NO normative judgment about whether one writing system is in some way "better" than another. All written languages have their inherent beauty, and all should be celebrated for that. I'm shedding those thoughts here for a focus purely on the mathematics of information-conveyed-per-character, and nothing more. Also, I'm not suggesting that Twitter ever would, should, or even could have separate limits for separate tweets based on the type of language being used, especially since any one tweet can draw from multiple languages. The thing about Twitter is just meant to serve as a proxy for the underlying question of how many characters Chinese generally uses in comparison to English.

Answer 1

Actually, if Twitter were to play fair, I think they would restrict Chinese users to 46 characters, since there are 3 bytes / UTF-8 encoded Chinese character, and only 1 byte / English letter. Alternately, in terms of number of bytes, the right comparison is probably between 140 characters of Chinese and 420 characters of English.

Consider the differences in the three:

"Please stand back from the doors" (English) (1)

請勿靠近車門 (Modern Chinese) (2)

勿近車門 (Classical Chinese) (3)

It's interesting to think about these things in terms of the Shannon Entropy, the basic concept in information theory. You might want to read up on the theory if you're interested in exploring the observation more deeply. If you take a "symbol" to refer to a byte (256 possible values or "meanings") of data and assume UTF-8 encoding for (2) and (3), then E(1) = 3.8 , E(2) = 3.5 , and E(3) = 3.3: so "information content" of a series of bytes-as-symbols actually drops as you move from English to Classical Chinese.

However, as you note, a character in Chinese maps to a meaning. So let's map each of the meanings to a series of symbols and calculate the entropies. Also rewrite (1) as "Please don't move near the car doors" to make it easier to compare the mappings across languages (byte-wise E for this rewrite is 3.7):

"Please don't move near the car doors." -> "A B C D E F G" (1) = 2.5

"請勿靠近車門" -> "ABCDFG" (2) = 2.6

"勿近車門" -> "BDFG" (3) = 2.0

So in this case the information content of (2) increases over (1), due to the repeated inclusion of spaces in (1). (3) remains lowest, but I think this would change using less formal MC: actual sentences, instead of terse "sign language". These trends would accelerate with longer texts and almost certainly the classical Chinese version would quickly become the most "entropic."

Why would it do so? Because I believe a statistical analysis of English v Modern Chinese v Classical Chinese would find that the first two incorporate far more repeated semantic symbols than Classical Chinese. An interesting thing to consider is the problem of error correction, which becomes a concern when concerning codings. You might want to think about how much repetition is present in English versus modern Chinese versus classical Chinese, and why that is: I would argue that Classical Chinese is minimally repetitive and maximally entropic, in the sense that Classical Chinese uses the bare minimum of symbols necessary to convey a thought. Contrast with English or even Modern Chinese, both of which have repetitive characters (spaces in English and words like "the"/"an", symbols like 的/个 in Modern Chinese , the far greater frequency of compound-character words in MC ex 绵羊／山羊 instead of 羊) that help preserve the meaning of a message in transmission. It would be interesting to consider any connection between this minimal repetitiveness might have with the thesis that comes up from time to time about how it's impossible to be "precise" (or "scientific") in Classical Chinese. Anyways, in human languages, preservation of the message entails the addition of extra words (just think how verbose English law is - there's a reason for that); in formal codings, preservation of the message entails the addition of extra bits.

These are just some random thoughts that might help you think of ways to begin quantifying the differences in "consumption of symbols" you note in Twitter feed.

Answer 2

Let's revisit why Twitter imposed the 140 character limit:

We like to keep it short and sweet! It also just so happens that 140 characters is the perfect length for sending status updates via text message. The standard text message length in most places is 160 characters per message. We reserve 20 characters for people's names, and the other 140 are all yours!

In SMS, characters in Chinese must be encoded using the 16-bit UCS-2 character encoding, which leads to the maximum individual short message size of 70 16-bit characters. To account for unusually long (sur)names, 10 characters should be reserved for people's names, so I argue that Chinese microblogging services should limit posts to 60 characters because Confucius said 「六十而耳顺」.

In reality, Chinese microblogging services like Sina Weibo all have the rather lax 140 character limit inherited from Twitter. If Twitter were to beef up its character limit to let English language users convey as much information as Chinese speaking users now enjoy, I would say 500 characters is a good choice. The answers in this post indicate that pro translators can express 1 English word in about 1.5 Chinese characters, and this website gives the average length of an English word to be 4.5 letters, so 140 Chinese characters translate to 140 / 1.5 × (4.5 + 1) = 513 English characters (letters plus space between words).

However, when it comes to internet languages, English has an advantage because it has contractions/abbreviations (e.g. srsly instead of seriously) and can usually do without some articles (e.g. a, the). It is worth noting that Chinese netizens often like to write in both Chinese and alphanumeric characters. Sometimes this reduces character count (OL vs 白领女性), but other times it has the opposite effect (1024 vs 顶/推). So I would wager that one can probably cram more English words in fewer characters if they take the shortcuts. Hence the 500 character approximation.

Answer 3

Short Answer

Undefined.

Long Answer

To fully support my opinion, There are a few points I would like to clarify:

• I would say 請勿靠近車門 is one of the outliers of the norm of microblogs (Twitter and Weibo are both microblogging services). More often, you would see colloquial sentences in Twitter and Weibo. There are two many variations in the colloquial context to convey the same meaning; Besides, some users are accustomed to certain usages that do not comply with standards of the written language – the number of characters used becomes indefinite thereby.

• The number of characters in Chinese and that in English of the same meaning are not proportionally related. In fact, Chinese does not necessary use less characters than English to convey the same meaning – particularly true for Chinese phrase and their English acronym equivalent. For instance,

  「順道一提」 vs "btw"

  「我的天啊」 vs "omg"

  Of course, Chinese generally uses less characters, but my point is that I do not think it is possible to model such relationship.

That said, I think the solution to your mathematical problem is too complicated, if not unsolvable, because of the numerous variables in the equation.

On a site note, I think most users post in less than 30 characters for Chinese, so technically that is probably a trivial concern for the purpose of encouraging brevity in a microblogging service.