-
RecurrentGemma: Moving Past Transformers for Efficient Open Language Models
Authors:
Aleksandar Botev,
Soham De,
Samuel L Smith,
Anushan Fernando,
George-Cristian Muraru,
Ruba Haroun,
Leonard Berrada,
Razvan Pascanu,
Pier Giuseppe Sessa,
Robert Dadashi,
Léonard Hussenot,
Johan Ferret,
Sertan Girgin,
Olivier Bachem,
Alek Andreev,
Kathleen Kenealy,
Thomas Mesnard,
Cassidy Hardin,
Surya Bhupatiraju,
Shreya Pathak,
Laurent Sifre,
Morgane Rivière,
Mihir Sanjay Kale,
Juliette Love,
Pouya Tafti
, et al. (37 additional authors not shown)
Abstract:
We introduce RecurrentGemma, a family of open language models which uses Google's novel Griffin architecture. Griffin combines linear recurrences with local attention to achieve excellent performance on language. It has a fixed-sized state, which reduces memory use and enables efficient inference on long sequences. We provide two sizes of models, containing 2B and 9B parameters, and provide pre-tr…
▽ More
We introduce RecurrentGemma, a family of open language models which uses Google's novel Griffin architecture. Griffin combines linear recurrences with local attention to achieve excellent performance on language. It has a fixed-sized state, which reduces memory use and enables efficient inference on long sequences. We provide two sizes of models, containing 2B and 9B parameters, and provide pre-trained and instruction tuned variants for both. Our models achieve comparable performance to similarly-sized Gemma baselines despite being trained on fewer tokens.
△ Less
Submitted 28 August, 2024; v1 submitted 11 April, 2024;
originally announced April 2024.
-
StarCoder 2 and The Stack v2: The Next Generation
Authors:
Anton Lozhkov,
Raymond Li,
Loubna Ben Allal,
Federico Cassano,
Joel Lamy-Poirier,
Nouamane Tazi,
Ao Tang,
Dmytro Pykhtar,
Jiawei Liu,
Yuxiang Wei,
Tianyang Liu,
Max Tian,
Denis Kocetkov,
Arthur Zucker,
Younes Belkada,
Zijian Wang,
Qian Liu,
Dmitry Abulkhanov,
Indraneil Paul,
Zhuang Li,
Wen-Ding Li,
Megan Risdal,
Jia Li,
Jian Zhu,
Terry Yue Zhuo
, et al. (41 additional authors not shown)
Abstract:
The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership with Software Heritage (SWH), we build The Stack v2 on top of the digital commons of their source code archive. Alongside the SWH repositories spanning 619 programming languages, we carefully select other high-quality data…
▽ More
The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership with Software Heritage (SWH), we build The Stack v2 on top of the digital commons of their source code archive. Alongside the SWH repositories spanning 619 programming languages, we carefully select other high-quality data sources, such as GitHub pull requests, Kaggle notebooks, and code documentation. This results in a training set that is 4x larger than the first StarCoder dataset. We train StarCoder2 models with 3B, 7B, and 15B parameters on 3.3 to 4.3 trillion tokens and thoroughly evaluate them on a comprehensive set of Code LLM benchmarks. We find that our small model, StarCoder2-3B, outperforms other Code LLMs of similar size on most benchmarks, and also outperforms StarCoderBase-15B. Our large model, StarCoder2- 15B, significantly outperforms other models of comparable size. In addition, it matches or outperforms CodeLlama-34B, a model more than twice its size. Although DeepSeekCoder- 33B is the best-performing model at code completion for high-resource languages, we find that StarCoder2-15B outperforms it on math and code reasoning benchmarks, as well as several low-resource languages. We make the model weights available under an OpenRAIL license and ensure full transparency regarding the training data by releasing the SoftWare Heritage persistent IDentifiers (SWHIDs) of the source code data.
△ Less
Submitted 29 February, 2024;
originally announced February 2024.
-
Ramsey theorem for trees with successor operation
Authors:
Martin Balko,
David Chodounský,
Natasha Dobrinen,
Jan Hubička,
Matěj Konečný,
Jaroslav Nešetřil,
Andy Zucker
Abstract:
We prove a general Ramsey theorem for trees with a successor operation. This theorem is a common generalization of the Carlson-Simpson Theorem and the Milliken Tree Theorem for regularly branching trees.
Our theorem has a number of applications both in finite and infinite combinatorics. For example, we give a short proof of the unrestricted Nešetřil-Rödl theorem, and we recover the Graham-Rothsc…
▽ More
We prove a general Ramsey theorem for trees with a successor operation. This theorem is a common generalization of the Carlson-Simpson Theorem and the Milliken Tree Theorem for regularly branching trees.
Our theorem has a number of applications both in finite and infinite combinatorics. For example, we give a short proof of the unrestricted Nešetřil-Rödl theorem, and we recover the Graham-Rothschild theorem. Our original motivation came from the study of big Ramsey degrees - various trees used in the study can be viewed as trees with a successor operation. To illustrate this, we give a non-forcing proof of a theorem of Zucker on big Ramsey degrees.
△ Less
Submitted 12 November, 2023;
originally announced November 2023.
-
Type-respecting amalgamation and big Ramsey degrees
Authors:
Andrés Aranda,
Samuel Braunfeld,
David Chodounský,
Jan Hubička,
Matěj Konečný,
Jaroslav Nešetřil,
Andy Zucker
Abstract:
We give an infinitary extension of the Nešetřil-Rödl theorem for category of relational structures with special type-respecting embeddings.
We give an infinitary extension of the Nešetřil-Rödl theorem for category of relational structures with special type-respecting embeddings.
△ Less
Submitted 22 March, 2023;
originally announced March 2023.
-
Characterisation of the big Ramsey degrees of the generic partial order
Authors:
Martin Balko,
David Chodounský,
Natasha Dobrinen,
Jan Hubička,
Matěj Konečný,
Lluis Vena,
Andy Zucker
Abstract:
As a result of 33 intercontinental Zoom calls, we characterise big Ramsey degrees of the generic partial order. This is an infinitary extension of the well known fact that finite partial orders endowed with linear extensions form a Ramsey class (this result was announced by Nešetřil and Rödl in 1984 with first published proof by Paoli, Trotter and Walker in 1985). Towards this, we refine earlier u…
▽ More
As a result of 33 intercontinental Zoom calls, we characterise big Ramsey degrees of the generic partial order. This is an infinitary extension of the well known fact that finite partial orders endowed with linear extensions form a Ramsey class (this result was announced by Nešetřil and Rödl in 1984 with first published proof by Paoli, Trotter and Walker in 1985). Towards this, we refine earlier upper bounds obtained by Hubička based on a new connection of big Ramsey degrees to the Carlson-Simpson theorem and we also introduce a new technique of giving lower bounds using an iterated application of the upper-bound theorem.
△ Less
Submitted 17 June, 2024; v1 submitted 17 March, 2023;
originally announced March 2023.
-
Big Ramsey degrees of the generic partial order
Authors:
Martin Balko,
David Chodounský,
Natasha Dobrinen,
Jan Hubička,
Matěj Konečný,
Lluis Vena,
Andy Zucker
Abstract:
As a result of 33 intercontinental Zoom calls, we characterise big Ramsey degrees of the generic partial order in a similar way as Devlin characterised big Ramsey degrees of the generic linear order (the order of rationals).
As a result of 33 intercontinental Zoom calls, we characterise big Ramsey degrees of the generic partial order in a similar way as Devlin characterised big Ramsey degrees of the generic linear order (the order of rationals).
△ Less
Submitted 21 May, 2021;
originally announced May 2021.
