-
PaLI-X: On Scaling up a Multilingual Vision and Language Model
Authors:
Xi Chen,
Josip Djolonga,
Piotr Padlewski,
Basil Mustafa,
Soravit Changpinyo,
Jialin Wu,
Carlos Riquelme Ruiz,
Sebastian Goodman,
Xiao Wang,
Yi Tay,
Siamak Shakeri,
Mostafa Dehghani,
Daniel Salz,
Mario Lucic,
Michael Tschannen,
Arsha Nagrani,
Hexiang Hu,
Mandar Joshi,
Bo Pang,
Ceslee Montgomery,
Paulina Pietrzyk,
Marvin Ritter,
AJ Piergiovanni,
Matthias Minderer,
Filip Pavetic
, et al. (18 additional authors not shown)
Abstract:
We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture. Our model achieves new levels of performance on a wide-range of varied and complex tasks, including multiple image-based captioning and question-answering tasks, image-based document understanding and few-sh…
▽ More
We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture. Our model achieves new levels of performance on a wide-range of varied and complex tasks, including multiple image-based captioning and question-answering tasks, image-based document understanding and few-shot (in-context) learning, as well as object detection, video question answering, and video captioning. PaLI-X advances the state-of-the-art on most vision-and-language benchmarks considered (25+ of them). Finally, we observe emerging capabilities, such as complex counting and multilingual object detection, tasks that are not explicitly in the training mix.
△ Less
Submitted 29 May, 2023;
originally announced May 2023.
-
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints
Authors:
Aran Komatsuzaki,
Joan Puigcerver,
James Lee-Thorp,
Carlos Riquelme Ruiz,
Basil Mustafa,
Joshua Ainslie,
Yi Tay,
Mostafa Dehghani,
Neil Houlsby
Abstract:
Training large, deep neural networks to convergence can be prohibitively expensive. As a result, often only a small selection of popular, dense models are reused across different contexts and tasks. Increasingly, sparsely activated models, which seek to decouple model size from computation costs, are becoming an attractive alternative to dense models. Although more efficient in terms of quality an…
▽ More
Training large, deep neural networks to convergence can be prohibitively expensive. As a result, often only a small selection of popular, dense models are reused across different contexts and tasks. Increasingly, sparsely activated models, which seek to decouple model size from computation costs, are becoming an attractive alternative to dense models. Although more efficient in terms of quality and computation cost, sparse models remain data-hungry and costly to train from scratch in the large scale regime. In this work, we propose sparse upcycling -- a simple way to reuse sunk training costs by initializing a sparsely activated Mixture-of-Experts model from a dense checkpoint. We show that sparsely upcycled T5 Base, Large, and XL language models and Vision Transformer Base and Large models, respectively, significantly outperform their dense counterparts on SuperGLUE and ImageNet, using only ~50% of the initial dense pretraining sunk cost. The upcycled models also outperform sparse models trained from scratch on 100% of the initial dense pretraining computation budget.
△ Less
Submitted 17 February, 2023; v1 submitted 9 December, 2022;
originally announced December 2022.
-
Sparse MoEs meet Efficient Ensembles
Authors:
James Urquhart Allingham,
Florian Wenzel,
Zelda E Mariet,
Basil Mustafa,
Joan Puigcerver,
Neil Houlsby,
Ghassen Jerfel,
Vincent Fortuin,
Balaji Lakshminarayanan,
Jasper Snoek,
Dustin Tran,
Carlos Riquelme Ruiz,
Rodolphe Jenatton
Abstract:
Machine learning models based on the aggregated outputs of submodels, either at the activation or prediction levels, often exhibit strong performance compared to individual models. We study the interplay of two popular classes of such models: ensembles of neural networks and sparse mixture of experts (sparse MoEs). First, we show that the two approaches have complementary features whose combinatio…
▽ More
Machine learning models based on the aggregated outputs of submodels, either at the activation or prediction levels, often exhibit strong performance compared to individual models. We study the interplay of two popular classes of such models: ensembles of neural networks and sparse mixture of experts (sparse MoEs). First, we show that the two approaches have complementary features whose combination is beneficial. This includes a comprehensive evaluation of sparse MoEs in uncertainty related benchmarks. Then, we present Efficient Ensemble of Experts (E$^3$), a scalable and simple ensemble of sparse MoEs that takes the best of both classes of models, while using up to 45% fewer FLOPs than a deep ensemble. Extensive experiments demonstrate the accuracy, log-likelihood, few-shot learning, robustness, and uncertainty improvements of E$^3$ over several challenging vision Transformer-based baselines. E$^3$ not only preserves its efficiency while scaling to models with up to 2.7B parameters, but also provides better predictive performance and uncertainty estimates for larger models.
△ Less
Submitted 9 July, 2023; v1 submitted 7 October, 2021;
originally announced October 2021.
