How Chinese are Chinese Language Models? The Puzzling Lack of Language Policy in China’s LLMs

LLMs have grown increasingly multilingual over the last few years Conneau et al. (2020); Brown et al. (2020); Workshop et al. (2022), with significant efforts put into promoting the representation of low-resource languages Abadji et al. (2022); ImaniGooghari et al. (2023). Studies have shown that linguistic diversity in language technology promote access to information Lee (2020) and preservation of low-resource languages Bird and Chiang (2012).

Language policy has been an essential political tool for rulers in China to govern multiple ethnicities and cultures. These policies vacillated from being pluralist to assimilationist over the eras. In 221 B.C., after unifying the warring states, the Qin Emperor launched the first official program to standardize the Chinese script to consolidate central state power. Following the 1949 Chinese Communist Revolution, the Chinese Communist Party (CCP) continued this tradition by launching extensive linguistic campaigns to unify the minority population of over 106 million people who spoke 129 languages among them Mullaney (2011). Gradually this transitioned to a more monolingual, assimilationist policy (one nation, one language). In 1982, constitutional amendment Article 19 made Mandarin Chinese the official common spoken “super language” for all indigenous and ethnic people. In 2000, the PRC promulgated the Standard Spoken and Written Chinese Language law to promote standard spoken and written Mandarin in public including schools, publications, broadcasts, and product packaging. While Mandarin Chinese remains the standard in the public sphere, non-Mandarin languages exist in common usage but are not recognized as standard, official languages.

The PRC government has also been taking steps to regulate generative AI. In July 2023 the Cyberspace Administration of China issued the Administration of Generative AI Services (the “Interim AI Measures”), which requires that generative AI services with “public opinion attributes” or “social mobilization capabilities” must undergo rigorous security assessments and file detailed algorithm records with authorities. The Interim AI Measures require that AI-generated content complies with five principles such as upholding socialist values, preventing discriminatory content, and implementing transparency and reliability measures. As of March 2024, 117 generative AI services had completed the mandatory government filing process to comply with these requires. The Interim AI Measures define requisites for pretraining AI models in particular. When training AI models, providers must use lawfully sourced data and foundational models and avoid infringing on intellectual property rights. They must also employ measures to enhance training data quality, truthfulness, accuracy, objectivity, and diversity and comply with national laws ¹¹1Chinese national Cybersecurity Law, Data Security Law, and Personal Information Protection. While these requirements are vague and could include minority languages, there is no explicit mention of them.

Language models vary considerably in both size and capability. In order to provide the most fair comparison among models, we restrict experiments to models roughly at the 8-billion parameter scale. This scale is sufficient for at-or-near state of the art performance while limiting computational complexity. We access these models through the transformers Wolf et al. (2020) library.²²2We will release code.

We experiment on six open-source multilingual LLMs pretrained from scratch by Chinese entities: Qwen1.5-7B Bai et al. (2023), Yi-6B Young et al. (2024), DeepSeek-LLM-7B DeepSeek-AI (2024), InternLM2-7B Cai et al. (2024), XVERSE-7B XVERSE Technology (2023), and Baichuan2-7B-Base Yang et al. (2023). We also evaluate Llama3-8B AI@Meta (2024) and Mistral-7B-v0.3 Jiang et al. (2023), open LLMs developed by U.S. and French companies.

We also examine the models’ technical reports to investigate the pretraining data that shaped these LLMs. We find that Chinese AI companies’ focus on language models places a heavy emphasis on Mandarin and English but not on other languages. While the reports for Qwen, XVERSE and InternLM2 mention multilingual coverage, they do not explicate what languages they cover except English and Mandarin (See Table 3 in appendix for further details). Additionally, similar to the training of international models, sources of pretrained data rely heavily on web pages, with three reports specifically mentioning Common Crawl as their primary data source.

We conduct two experiments, measuring different aspects of language model capabilities on two datasets: the FLORES+ and Belebele benchmarks. Both datasets are developed based on the FLORES evaluation benchmark for machine translation Goyal et al. (2022). We evaluate 18 languages included in both datasets, spanning languages spoken in China, Northeast Asia, Southeast Asia, US/Europe, and by ethnic minorities in China. For Mandarin Chinese, we test two writing system variants — Simplified Chinese characters, used in the PRC, and Traditional Chinese characters, used in Taiwan. We were not able to evaluate Chinese Han dialects such as Cantonese or Shanghainese because the benchmarks did not include them. For the full list of languages, see Table 1 in appendix.

The models we evaluate are trained on a next-token prediction task. Our first experiment evaluates how well these language models predict the next token in sentences from the FLORES+ multilingual dataset.³³3https://github.com/openlanguagedata/flores 997 English sentences ⁴⁴4we use the dev split of FLORES+. are sampled from Wiki sources then translated into target languages by native speakers. There are 997 parallel sentences in all languages we evaluate. See example English sentences in Appendix A.

Modern language model tokenizers employ data-driven approaches like byte-pair encoding (BPE) to break words into sub-word units, often leading to varying segmentation rates across languages Rust et al. (2021); Ali et al. (2023). We also observe such discrepancy in segmentation rate with the language models we evaluate, see Appendix Figure 4.

Perplexity (PPL) is a common measure to evaluate the probability given to a sentence Gonen et al. (2023); Xiao et al. (2023). However, perplexity, as a per-token measure, is not directly comparable across languages tokenized at varying rates Mielke (2019). To account for the confounding influence of tokenization efficiency, we report unnormalized negative log likelihood, equivalent to the sum of negative log-likelihood given to each token in a sentence:

where $n_{tokens}$ refers to the total number of tokens in the sentence. Lower scores are better as they represent higher probability.

From Figure 1, we find that NLL is lowest for Mandarin and US/European languages, followed by Northeast Asian languages, Southeast Asian Languages, and with Chinese Ethnic Minorities languages having the worst. Language models show similar results among the top-performing languages, but the performance varies in lower-resource languages including Lao, Burmese, and Chinese minority languages. We find no clear distinctions between Chinese and international models across languages, with only slightly better performance in Mandarin from the Chinese models.

Perplexity is an intrinsic measure of how well a language model performs on the task they are trained to do. But it does not necessarily predict how well a model does in tasks that require text comprehension Holtzman et al. (2021); Wiegreffe et al. (2023). To account for this, in the second experiment we evaluate the language models’ performance on multiple-choice reading comprehension (MRC) using the Belebele benchmark dataset Bandarkar et al. (2023). The Belebele becnhmark is a parallel reading comprehension dataset covering 122 languages. For each language, there are 900 multiple-choice questions. Each question is related to a passage from the FLORES-200 dataset NLLB Team et al. (2022). Each question has four multiple choice answers with one correct answer. The questions are fully parallel across 122 languages and curated and verified by translators fluent in both English and the target language. We query models with zero-shot prompts and calculate zero-shot MRC accuracy of the models’ answers, higher is better. See example question and prompt in the appendix.

Figure 2 shows the result of the second experiment. Overall, the performance trend across languages is similar to Experiment 1. While zero-shot MRC accuracy is consistently lowest for Burmese, Lao, and Chinese minority languages, it is more varied for other languages, especially for higher-resource ones. Additionally, InternLM2 consistently outperforms other models in all languages, and XVERSE consistently underperforms, sometimes even lower than a random baseline. Similar to Experiment 1, there are no clear performance differences between Chinese and international models among the languages we test.

We find that Chinese national models show the same pattern of performance across languages that international models do. Both Chinese and international models, like Llama3 and Mistral, perform best in Mandarin and Western languages, then Japanese and Korean, and worst in Chinese minority languages. To test if the performance variance is a factor of data availability we use number of speakers and GDP as proxies Eberhard et al. (2024). We find that model performance on different languages highly correlates with the number of speakers of the language (Figure 3), and the trend is almost indistinguishable between Chinese and international models. On average, the logarithm of number of language speakers highly correlates with NLL, at $-0.845$ for Chinese LLMs and $-0.883$ for international LLMs. Correlations with zero-shot MRC accuracy are $0.837$ for Chinese LLMs and $0.884$ for international LLMs.⁵⁵5The population for Standard Malay (zsm) is not provided in Eberhard et al. (2024), so we use the population for Malay (msa) instead. We find the NLL and zero-shot MRC accuracy performances of national languages from Asian countries neighboring China (Japan, Korea, Indonesia, Malaysia, Vietnam, Thai, Myanmar, Laos) also highly correlate with the logarithm of national GDP (see Appendix Table 4 and Figure 5 for details).