For quite a long time I have wondered how many characters one needs to know to know 90%, 95% or even 99% of Chinese characters in an average text.
On the internet numbers of about 2000 get thrown around frequently, but rarely more concrete evidence is provided. In this post I would like to showcase the following short python script and graph:
Using the data from https://huggingface.co/datasets/zetavg/tw-sinica-corpus-word-freequency/raw/main/words.json and the script
```python
import numpy as np
import plotly.graph_objects as go
import json
def gen_normalization_table():
return str.maketrans({"麽": "麼", "爲": "為", "爲": "為", "説": "說", "裏": "裡"})
def normalize_chinese(txt, table=gen_normalization_table()):
return txt.translate(table)
def getFreqDict():
with open("words.json") as f:
freqList = json.load(f)
freqDict = {}
total = 0
for x in freqList:
freq = x["frequency"]
word = normalize_chinese(x["word"])
total += freq * len(word)
for c in word:
freqDict[c] = freqDict.get(c, 0) + freq
return {word: freq / total for word, freq in freqDict.items()}
d = getFreqDict()
X = np.arange(1, len(d) + 1)
vals = [val for _, val in sorted(d.items(), key=lambda item: -item[1])]
Y = np.cumsum(np.fromiter(vals, float))
fig = go.Figure()
fig.add_trace(go.Scatter(x=X, y=Y, mode="lines"))
fig.update_layout(
xaxis=dict(dtick=500, showgrid=True),
yaxis=dict(dtick=0.10, showgrid=True),
template="plotly_white",
)
fig.show()
Export as interactive HTML
fig.write_html("plot.html", full_html=True, include_plotlyjs=True)
```
I generated the figure you can see above. On [https://valentin-herrmann.de/p/how-many-chinese-characters-do-you-need-to-learn-to-know-99-out-of-100-in-an-average-text/](my blog) you can also play around with an interactive version of the graph.
From this graph we can read off the following table:
| % |
n |
| 50 |
170 |
| 60 |
260 |
| 70 |
395 |
| 80 |
610 |
| 90 |
1030 |
| 95 |
1510 |
| 99 |
2700 |
| 99.9 |
4350 |
So for knowledge of 99% of characters in an average text one needs to know at least 2700 characters.