Tokenizer Choice For LLM Training: Negligible or Crucial?
Authors:
Mehdi Ali,
Michael Fromm,
Klaudia Thellmann,
Richard Rutmann,
Max Lübbering,
Johannes Leveling,
Katrin Klug,
Jan Ebert,
Niclas Doll,
Jasper Schulze Buschhoff,
Charvi Jain,
Alexander Arno Weber,
Lena Jurkschat,
Hammam Abdelwahab,
Chelsea John,
Pedro Ortiz Suarez,
Malte Ostendorff,
Samuel Weinbach,
Rafet Sifa,
Stefan Kesselheim,
Nicolas Flores-Herr
Abstract:
The recent success of Large Language Models (LLMs) has been predominantly driven by curating the training dataset composition, scaling of model architectures and dataset sizes and advancements in pretraining objectives, leaving tokenizer influence as a blind spot. Shedding light on this underexplored area, we conduct a comprehensive study on the influence of tokenizer choice on LLM downstream perf…
▽ More
The recent success of Large Language Models (LLMs) has been predominantly driven by curating the training dataset composition, scaling of model architectures and dataset sizes and advancements in pretraining objectives, leaving tokenizer influence as a blind spot. Shedding light on this underexplored area, we conduct a comprehensive study on the influence of tokenizer choice on LLM downstream performance by training 24 mono- and multilingual LLMs at a 2.6B parameter scale, ablating different tokenizer algorithms and parameterizations. Our studies highlight that the tokenizer choice can significantly impact the model's downstream performance and training costs. In particular, we find that the common tokenizer evaluation metrics fertility and parity are not always predictive of model downstream performance, rendering these metrics a questionable proxy for the model's downstream performance. Furthermore, we show that multilingual tokenizers trained on the five most frequent European languages require vocabulary size increases of factor three in comparison to English. While English-centric tokenizers have been applied to the training of multi-lingual LLMs in the past, we find that this approach results in a severe downstream performance degradation and additional training costs of up to 68%, due to an inefficient tokenization vocabulary.
△ Less
Submitted 17 March, 2024; v1 submitted 12 October, 2023;
originally announced October 2023.
Towards Supervised Extractive Text Summarization via RNN-based Sequence Classification
Authors:
Eduardo Brito,
Max Lübbering,
David Biesner,
Lars Patrick Hillebrand,
Christian Bauckhage
Abstract:
This article briefly explains our submitted approach to the DocEng'19 competition on extractive summarization. We implemented a recurrent neural network based model that learns to classify whether an article's sentence belongs to the corresponding extractive summary or not. We bypass the lack of large annotated news corpora for extractive summarization by generating extractive summaries from abstr…
▽ More
This article briefly explains our submitted approach to the DocEng'19 competition on extractive summarization. We implemented a recurrent neural network based model that learns to classify whether an article's sentence belongs to the corresponding extractive summary or not. We bypass the lack of large annotated news corpora for extractive summarization by generating extractive summaries from abstractive ones, which are available from the CNN corpus.
△ Less
Submitted 13 November, 2019;
originally announced November 2019.
