Pretraining Data and Tokenizer for Indic LLM

In the ever-evolving realm of Natural Language Processing (NLP), the development of Large Language Models (LLMs) have seen a meteoric rise since the inception of transformers. The introduction of LLMs has ushered in a new era in NLP, where the boundaries of what machines could achieve with human language are constantly pushed to astonishing limits. OpenAI’s ChatGPT and Google’s BARD has been a pioneering force for redefining the landscape of language modeling performance and has also illuminated the vast societal implications inherent in these technological advancements. Alongside ChatGPT and BARD, various open source and proprietary LLMs have showcased remarkable natural language understanding and generation. There have been public releases by some key players in the realm of LLM including Llama 2 Touvron et al. (2023a), Mistral Jiang et al. (2023), Falcon, MPT, etc creating more affordable, efficient, and high-performing language models. The chat-models of these pretrained LLMs showcase comparable results on few benchmarks with proprietary LLM chat-models including ChatGPT, BARD and Claude. LLMs such as Llama and Falcon have outlined their data refinement and pre-training steps in their respective technical reports. We draw inspiration from the meticulous analysis of RefinedWeb Dataset by Falcon in our data filtering and deduplication approaches.

Indic culture boasts linguistic diversity, encompassing Indo-Aryan, Dravidian and Munda languages. Indo-Aryan and Dravidian languages constitute 96% of India’s spoken languages. Despite this richness, most open language models lack Indic language support, hindering innovation due to limited high-quality data and complex tokenization challenges AI (2023). Notable corpora like EMILLE/CIIL, Wikipedia for Indian languages, Samantar Corpus, and AI4Bharat-IndicNLP provide valuable resources Kunchukuttan et al. (2020). EMILLE/CIIL spans 14 languages with 92 million words, Wikipedia for Indian languages is limited, Samantar Corpus offers 49.7 million parallel sentences across 11 languages, and AI4Bharat-IndicNLP Corpus contains 2.7 billion words for 10 languages Kunchukuttan et al. (2020). IndicCorp with 8.8 billion tokens across 11 languages, supplements linguistic resources Kakwani et al. (2020). Despite these efforts, it is noteworthy that Hindi being the third most spoken language, does not rank among the top 20 languages in processed CommonCrawl documents, highlighting the scarcity of India-specific data for open large language model training Penedo et al. (2023).

In this study, we provide a technical report on clean indic dataset preparation and tokenizer training for Indic LLM : India’s own foundational model with indic-rich context. The study highlights a comprehensive analysis of available open source and proprietary datasets and data refinement steps. We also devise a state-of-art indic tokenizer through rigorous experimentations and validated the performance through a pretraining model.

Large language models (LLMs) owe their remarkable learning capabilities to massive model sizes and extensive training datasets. At present, there are numerous foundational models spanning from open source and proprietary LLMs. Some noteworthy open source foundational LLMs include Llama 2 Touvron et al. (2023a), Mistral 7B Jiang et al. (2023), Falcon Penedo et al. (2023), MPT Team (2023), Bloom Workshop et al. (2022), etc. whereas proprietary foundational LLMs comprise of GPT4 Achiam et al. (2023), LaMDA Thoppilan et al. (2022), etc. LLaMA 2 developed by Meta AI and Microsoft focuses on multilingual capabilities and is optimized for swift training and inference. MPT-7B by MosaicML and Mistral-7B by Mistral AI are 7-billion-parameter models that have demonstrated efficient open-source training code, promoting transparency and ease of use. These models have showcased superiority over other open-source models in the 7B-20B range. Falcon-40B developed by Technology Innovation Institute (TII) has 40-billion-parameters and is a causal decoder-only model trained on a causal language modeling task. It is trained on a large dataset and has demonstrated superior performance to GPT-3. BLOOM is the world’s largest open multilingual language model, with 176-billion-parameters. Generally, proprietary systems are more expensive and offer product solutions that can be tailored to fit very specific business needs. Open source models usually offer more affordable and customizable options but may lack the performance level and specialization of proprietary LLMs.

Despite the widespread availability of LLMs for public exploration, the lack of transparency regarding training datasets, especially for state-of-the-art models, hinders research on addressing relevant biases. Furthermore, LLMs are known to generate text lacking sufficient grounding to knowledge sources, thus posing risks of misinformation and hallucination Li et al. (2023). This challenge is exacerbated largely in multilingual learning scenarios where datasets are often inadequately collected. Researchers have been advancing in the development of LLMs tailored to specific regional languages Cui et al. (2023) Balachandran (2023) Kunchukuttan et al. (2020). For development of a robust multilingual LLM, two pivotal components are: presence of abundant multilingual data and a diverse vocabulary Yuan et al. (2023). The current state-of-art LLMs provide a rigid multilingual support, this is due to less multilingual data in the pre-training corpus. For example, Llama models Touvron et al. (2023a) Touvron et al. (2023b) have leveraged a vast pre-training corpus with over 1.6 trillion tokens, but less than 4.5% is multilingual data, over 20 different languages. This number is further enhanced in Llama 2 models where the proportion of multilingual data is increased to approximately 11% and the number of languages to around 26. CulturaX Nguyen et al. (2023) is another multilingual dataset with 6.3 trillion tokens across 167 languages. The dataset is created after meticulously cleaning and deduplication steps to facilitate advancements in multilingual LLMs. However, out of 167 languages only 14 languages amount to 90.38%, thus creating a very low pre-training corpus for developing an efficient and versatile LLM having contextual understanding of indic languages.

To cater this issue of procuring massive datasets for LLM development, researchers have been relying on open source datasets such as web-crawled Common Crawl, Wikipedia, Public domain books spanning from cultural and historical facets Gao et al. (2020), Stack Exchange and Github archives, Journal articles and educational resources Lewkowycz et al. (2022), News archives, Government and institutional legal repositories, Multimedia transcripts. In this study, we aim to exploit the aforementioned data sources for developing extensive indic data corpus.

Moreover, these massive corpora need meticulous rigorous data refinement and deduplication to ensure data quality while maintaining integrity. Recent LLMs have demonstrated a robust work on data preparation and filtering, The RefinedWeb Penedo et al. (2023) has been pivotal in providing an insightful technical background in this context, significantly enhancing our understanding of the nuances involved. RefinedWeb has executed filtering and deduplication techniques on Common Crawl corpus, these filters include incorporation of threshold-based language filtering, url-filtering, line-wise correction filters, document deduplication, etc. MassiveText Rae et al. (2021) has defined rules for reducing nuances from the text documents by implementing extensive quality filtering techniques. Upon rigorous filtering and deduplication steps, a large amount of noisy data is removed from the original corpora. It has been observed that the open source language datasets do hold a high number of boilerplate texts and similar-context text documents Penedo et al. (2023) Lee et al. (2021).

Consequently, deduplication process is a crucial step for producing high quality pre-training corpus. Various deduplication algorithms have been established in literature spanning from exact matching with suffix arrays Manber and Myers (1993), largest substring matching , minhash Broder (1997), simhash Charikar (2002), and other fuzzy techniques in order to minimize memory usage while maximizing efficiency. In this work, we have methodically incorporated the data filtering and deduplication techniques, drawing inspiration from the research findings presented in RefinedWeb and MassiveText and also proposed our own innovative filters for data preprocessing.

Conventional tokenization approaches often involve a complex preprocessing pipeline and are language-specific. A simplistic and multilingual tokenizer is required for diverse natural language processing (NLP) tasks. These tokenization techniques generally include BPE Sennrich et al. (2015), Wordpiece Wu et al. (2016) , Sentencepiece Kudo and Richardson (2018), IndicBERT Kakwani et al. (2020), Spacy AI (Accessed: 2024). Kunchukuttan et al. (2020) proposed an Indic NLP tokenizer Kunchukuttan (2020) which emerges as an effective tokenization tool for indic-languages (Assamese, Bengali, Gujarati, Hindi, Marathi, Odia, Punjabi, Kannada, Malayalam, Tamil, Telugu) including English. In addition, Stanford NLP Al-Rfou and Skiena (2013) has proposed Indic tokenizer Manning et al. (2014) which supports English, Indo-aryan and Dravidian languages along with preprocessing functionalities. SentencePiece Kudo and Richardson (2018) tokenizer enables a fully end-to-end language-independent system by directly training its subword models from raw input sentences. This approach eliminates the need for pre-tokenized word sequences. In this study, we experiment with the SentencePiece Kudo and Richardson (2018) tokenizer and fine-tune it based on our indic-rich corpora.

In this study, we meticulously compile data corpus from open source and proprietary datasets for a comprehensive selection of 12 indic languages including English. These languages span from Assamese, Bengali, English, Kannada, Gujarati, Hindi, Marathi, Malayali, Punjabi, Odia, Tamil, Telugu. Our contributions in paper are highlighted as :
1. Development of High-Quality Multilingual Indic Data
2. Novel Multilingual Tokenizer Training Strategy

Tokenization is a preprocessing step in natural language processing (NLP), facilitating machine comprehension and interpretation of human language by breaking down text into fundamental units such as words, characters and subwords. Among the various types of tokenization tools, one widely employed approach is SentencePiece, which sets itself apart by directly training subword models from raw sentence data. This method offers a fully end-to-end and language-agnostic system, eliminating the need for pre-tokenized word sequences.

In our study, we adopted Byte Pair Encoding (BPE) as the tokenization strategy. We assessed the quality of tokenization using two key metrics: the token-to-word ratio and the exact score, which measures the proportion of correctly tokenized units out of the total tokens.

In conclusion, our meticulous data acquisition and custom preprocessing pipeline have effectively curated a high-quality multilingual dataset, significantly enhancing the performance of our Indic large language models. Through rigorous filtering and deduplication processes, we ensured the elimination of redundant and low-quality content, optimizing the data for training. Our novel multilingual tokenizer training strategy demonstrated superior token-to-word ratios for Indic languages compared to the state-of-the-art OpenAI Tiktoken tokenizer. The experiments underscore the importance of tailored preprocessing and tokenizer design, paving the way for more accurate and efficient language models in Indic-rich multilingual contexts.