FuLG: 150B Romanian Corpus for Language Model Pretraining
Authors:
Vlad-Andrei Bădoiu,
Mihai-Valentin Dumitru,
Alexandru M. Gherghescu,
Alexandru Agache,
Costin Raiciu
Abstract:
Research in the field of language models is rapidly evolving, with many open models being released to the public. Openly available pretraining corpora usually focus on only a handful of languages, with many others either missing completely or extremely underrepresented. In this report, we introduce FuLG, a hundred-fifty-billion-token Romanian corpus extracted from CommonCrawl. We present our metho…
▽ More
Research in the field of language models is rapidly evolving, with many open models being released to the public. Openly available pretraining corpora usually focus on only a handful of languages, with many others either missing completely or extremely underrepresented. In this report, we introduce FuLG, a hundred-fifty-billion-token Romanian corpus extracted from CommonCrawl. We present our methodology for filtering FuLG and compare it via ablation studies against existing Romanian corpora.
△ Less
Submitted 18 July, 2024;
originally announced July 2024.
A Look Into Training Large Language Models on Next Generation Datacenters
Authors:
Alexandru M. Gherghescu,
Vlad-Andrei Bădoiu,
Alexandru Agache,
Mihai-Valentin Dumitru,
Iuliu Vasilescu,
Radu Mantu,
Costin Raiciu
Abstract:
Is it still worth doing computer networking research? What are relevant problems in this space given the supremacy of hyperscalers in deployed large networks? We take an unconventional approach to finding relevant research directions, by starting from Microsoft's plans to build a $100 billion datacenter for ML. Our goal is to understand what models could be trained in such a datacenter, as well as…
▽ More
Is it still worth doing computer networking research? What are relevant problems in this space given the supremacy of hyperscalers in deployed large networks? We take an unconventional approach to finding relevant research directions, by starting from Microsoft's plans to build a $100 billion datacenter for ML. Our goal is to understand what models could be trained in such a datacenter, as well as the high-level challenges one may encounter in doing so.
We first examine the constraints imposed by cooling and power requirements for our target datacenter and find that it is infeasible to build in a single location. We use LLM scaling laws to determine that we could train models of 50T or 100T. Finally, we examine how distributed training might work for these models, and what the networking requirements are. We conclude that building the datacenter and training such models is technically possible, but this requires a novel NIC-based multipath transport along with a redesign of the entire training stack, outlining a research agenda for our community in the near future.
△ Less
Submitted 1 July, 2024;
originally announced July 2024.
