FuLG: 150B Romanian Corpus for Language Model Pretraining

In a world where Large Language Models (LLMs) have shown great potential for natural language tasks, the corpora used for pretraining play a central role in their overall capabilities. However, most state-of-the-art models are not accompanied by their training datasets and authors only sparsely discuss their composition.

In response to this, researchers and practitioners have produced numerous data corpora for widely spoken languages on the Internet. These are often derived from CommonCrawl ²²2https://commoncrawl.org, a public repository of crawled web pages, which has indexed more than 250 billion pages. Other datasets go further by including curated data such as books, social media discussions, and research papers. Trillion-token datasets such as Dolma (Soldaini et al., 2024), FineWeb (Penedo et al., 2024), or RedPajama (Computer, 2023) have enabled the development of billion-parameter models with loss-optimal training, but they cover only a small fraction of languages.

This limitation is evident in the language understanding capabilities of open models, as they are often trained on publicly available data. For instance, OLMo models (Groeneveld et al., 2024) support only six languages, while other initiatives such as LLM360 K2 (Liu et al., 2023) are English-centric. Even models developed by tech giants, such as Llama (Touvron et al., 2023) or Gemma (Team et al., 2024), have limited output capabilities in less commonly spoken languages. Only recently have we seen some developments in this area, with Llama 3 supporting 30 languages, albeit with lower performance levels compared to English and limited information about the training dataset.

To improve the standing of the Romanian language in future models, we delve into filtering Romanian content from CommonCrawl, documenting the process and releasing an open dataset. We built a pipeline that employs deduplication techniques, common signal filters, and FastText for language detection. We document the steps taken to obtain the final text corpus. As a result, we release FuLG, an openly available corpus that is three times larger than existing Romanian corpora.

In this work we make two main contributions. We release FuLG, a 156B-token (589GB tokenized), or 220B tokens with the Llama 3 tokenizer, corpus for LLM pretraining and fine-tuning in the Romanian language. Second, we document the process of filtering data from CommonCrawl, which may be employed for other underrepresented languages.

A crucial factor in model performance is the size, quality and diversity of the pretraining corpus. These datasets typically comprise content from various sources, primarily web pages, due to their sheer number and availability, followed by social media discussions, academic papers, books, code repositories, and encyclopedias, covering a wide range of topics. The main sources of data are either private crawls of the Internet or the publicly available CommonCrawl repository.

While information about closed models’ training data is scarce, even open models like Llama or Gemma lack transparency regarding their training datasets, as it is a central part of the competitive advantage. We are thus in a peculiar scenario, where the best open-source models aren’t truly open-source when it comes to pretraining data, which deeply affects any smaller initiative.

On the other hand, open datasets are essential components for truly open models and play a vital role in democratizing future LLMs. Currently, numerous open datasets exist, many derived from CommonCrawl. Notable examples include: Dolma (Soldaini et al., 2024) (3T tokens), C4 (Dodge et al., 2021) (175B tokens), The Pile (Gao et al., 2020) (387B tokens), ROOTS (Laurençon et al., 2022) (400B tokens), RefinedWeb (Penedo et al., 2023) (600B tokens), RedPajama v2 (Computer, 2023) (30T tokens), FineWeb (Penedo et al., 2024) (15T tokens), Zyda (Tokpanov et al., 2024) (1.3T tokens), LLM360 Amber (Liu et al., 2023) (1.2T tokens), MAP-Neo (Zhang et al., 2024) (4.5T tokens), OSCAR (Ortiz Suárez et al., 2019). While these datasets provide sufficient quality and size for English-based models, they often lack adequate representation of less commonly spoken languages. For example, after training a Romanian GPT-NeoX tokenizer on the Romanian part of OSCAR, we obtain a number of only 10B tokens on OSCAR, and 41B tokens on mC4 (Raffel et al., 2019) (slightly higher number of tokens if using an English tokenizer, similar to Llama or OLMo, due to worse compression ratio). However, considering the scaling laws of dataset size relative to model size, these quantities are insufficient for optimal LLM training. Beyond these, several smaller datasets exist for various Natural Language Processing (NLP) tasks in Romanian (Váradi et al., 2020; Mititelu et al., 2018; Ion et al., 2012; Midrigan-Ciochina et al., 2020; Schöch et al., 2021; Manolescu and Çöltekin, 2021) but they generally contain less than 500M tokens, which is far from adequate for LLM development.

FuLG is derived from CommonCrawl, a vast collection of web page crawls dating back to 2007. CommonCrawl releases snapshots of portions of the Internet several times annually. These snapshots exhibit low similarity rates between releases, enabling us to consistently extract new data from each snapshot. Our work encompasses snapshots from 2013 to May 2024.

We developed a pipeline based on existing software, which we elaborate on shortly. Given the petabytes of data requiring filtration, we utilized a distributed environment with multiple nodes for data acquisition. For deduplication and quality filtering, we employed a single large-memory node.

To evaluate FuLG we conducted ablation studies using a 1B decoder-only model based on OLMo (Groeneveld et al., 2024), with a sequence length of 2048 and the OLMo tokenizer trained on the OSCAR-Ro corpus. We used a global batch size of 256. We made several changes to the hyperparameters to adjust to the 1B model size: weight decay of 0.001 and a max learning rate of 4e-4 annealed by a cosine schedule, with an initial warmup of 1000 steps.

We trained three identical models on different pretraining datasets: Oscar, mC4 and FuLG. We leveraged the Hugging Face transformers (Wolf et al., 2020) library for training. We trained to completion on each dataset (290k steps for FuLG, 75k for mC4 and 20k for OSCAR). Since FuLG was much bigger than both OSCAR and mC4 (about 4 times bigger than mC4), we also included an earlier checkpoint from FuLG at 70k steps. We do note that it slightly underperforms, but we believe this naturally happens because of the cosine decay learning rate schedule.

A commonly employed method to assess the quality of a dataset is to fix a model, train it with different datasets, and measure perplexity against a curated evaluation set. While perplexity is not a definitive measure of dataset quality (Soldaini et al., 2024), we use it as a sanity check, with values similar to existing datasets confirming our approach. We constructed a perplexity dataset covering multiple domains, sourced from Wikipedia, news articles, textbooks, research papers, and books, totaling 74M tokens. Although we did not specifically perform a separate decontamination step, as this is only a preliminary evaluation, we do note that the collection methods of the evaluation dataset were manual, with care in picking high-quality sources. We do plan to apply a decontamination step in further experiments, to make sure our results are fair. Before calculating perplexity, the evaluation dataset was cleaned using the clean-text Python package ³³3https://github.com/jfilter/clean-text. The results are shown in Table 1. Both ablations of FuLG show perplexity values similar to those of existing datasets.

We further qualitatively evaluated the trained models. Inspired by the TinyStories (Eldan and Li, 2023) work, we used the models to generate stories from given prompts and asked GPT-4 to rate the creativity, grammar, and overall complexity of each story. We only considered responses that were coherent stories, discarding anomalous outputs. Both OSCAR and FuLG (70k) generated significantly more anomalies than the other ablations. In Table 2, we present our findings, which suggest hat FuLG may enable better performance for specific tasks.

The availability of high-quality datasets for less commonly spoken languages is crucial for the democratization of LLMs. While proprietary models often demonstrate proficiency across a wide range of languages, open models frequently underperform in this aspect. By developing large, high-quality open datasets for diverse languages, we can foster the creation of superior open-source models. These improved models could potentially be utilized by governments worldwide to meet their digitalization needs.

FuLG represents an initial effort to improve dataset size and quality for the Romanian language. We’re already underway to implementing a few other improvements which will further increase dataset quality:

• •

  Data processing: We currently work with Common Crawl snapshots in WET format. However, as previous research [citation] indicates, employing a more sophisticated HTML parser could significantly improve both the quality of extracted text and the overall volume of usable data.

• •

  Language-specific optimization: Romanian language particularities could be leveraged to adapt and refine quality filters. Currently, we use thresholds designed for the English language, but developing language-specific criteria could yield better results.

• •

  Novel quality filters: Identifying and implementing new quality filters tailored to the Romanian language could further enhance the dataset’s overall quality and relevance.

We acknowledge the EuroHPC Joint Undertaking for awarding this project access to the EuroHPC supercomputer LUMI, hosted by CSC (Finland) and the LUMI consortium through a EuroHPC Regular Access call. Some of the experiments were performed on the Luxembourg national supercomputer MeluXina. The authors gratefully acknowledge the LuxProvide teams for their expert support. We acknowledge the computational resources provided by the PRACE award granting access to Discoverer in SofiaTech, Bulgaria. The authors gratefully acknowledge the HPC RIVR consortium and EuroHPC JU for funding this research by providing computing resources of the HPC system Vega at the Institute of Information Science. We acknowledge VSB – Technical University of Ostrava, IT4Innovations National Supercomputing Center, Czech Republic, for awarding this project access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium through the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (grant ID: 90254). This work was funded by a VMWare gift.