-
ALLaM: Large Language Models for Arabic and English
Authors:
M Saiful Bari,
Yazeed Alnumay,
Norah A. Alzahrani,
Nouf M. Alotaibi,
Hisham A. Alyahya,
Sultan AlRashed,
Faisal A. Mirza,
Shaykhah Z. Alsubaie,
Hassan A. Alahmed,
Ghadah Alabduljabbar,
Raghad Alkhathran,
Yousef Almushayqih,
Raneem Alnajim,
Salman Alsubaihi,
Maryam Al Mansour,
Majed Alrubaian,
Ali Alammari,
Zaki Alawami,
Abdulmohsen Al-Thubaity,
Ahmed Abdelali,
Jeril Kuriakose,
Abdalghani Abujabal,
Nora Al-Twairesh,
Areeb Alowisheq,
Haidar Khan
Abstract:
We present ALLaM: Arabic Large Language Model, a series of large language models to support the ecosystem of Arabic Language Technologies (ALT). ALLaM is carefully trained considering the values of language alignment and knowledge transfer at scale. Our autoregressive decoder-only architecture models demonstrate how second-language acquisition via vocabulary expansion and pretraining on a mixture…
▽ More
We present ALLaM: Arabic Large Language Model, a series of large language models to support the ecosystem of Arabic Language Technologies (ALT). ALLaM is carefully trained considering the values of language alignment and knowledge transfer at scale. Our autoregressive decoder-only architecture models demonstrate how second-language acquisition via vocabulary expansion and pretraining on a mixture of Arabic and English text can steer a model towards a new language (Arabic) without any catastrophic forgetting in the original language (English). Furthermore, we highlight the effectiveness of using parallel/translated data to aid the process of knowledge alignment between languages. Finally, we show that extensive alignment with human preferences can significantly enhance the performance of a language model compared to models of a larger scale with lower quality alignment. ALLaM achieves state-of-the-art performance in various Arabic benchmarks, including MMLU Arabic, ACVA, and Arabic Exams. Our aligned models improve both in Arabic and English from their base aligned models.
△ Less
Submitted 22 July, 2024;
originally announced July 2024.
-
When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards
Authors:
Norah Alzahrani,
Hisham Abdullah Alyahya,
Yazeed Alnumay,
Sultan Alrashed,
Shaykhah Alsubaie,
Yusef Almushaykeh,
Faisal Mirza,
Nouf Alotaibi,
Nora Altwairesh,
Areeb Alowisheq,
M Saiful Bari,
Haidar Khan
Abstract:
Large Language Model (LLM) leaderboards based on benchmark rankings are regularly used to guide practitioners in model selection. Often, the published leaderboard rankings are taken at face value - we show this is a (potentially costly) mistake. Under existing leaderboards, the relative performance of LLMs is highly sensitive to (often minute) details. We show that for popular multiple-choice ques…
▽ More
Large Language Model (LLM) leaderboards based on benchmark rankings are regularly used to guide practitioners in model selection. Often, the published leaderboard rankings are taken at face value - we show this is a (potentially costly) mistake. Under existing leaderboards, the relative performance of LLMs is highly sensitive to (often minute) details. We show that for popular multiple-choice question benchmarks (e.g., MMLU), minor perturbations to the benchmark, such as changing the order of choices or the method of answer selection, result in changes in rankings up to 8 positions. We explain this phenomenon by conducting systematic experiments over three broad categories of benchmark perturbations and identifying the sources of this behavior. Our analysis results in several best-practice recommendations, including the advantage of a hybrid scoring method for answer selection. Our study highlights the dangers of relying on simple benchmark evaluations and charts the path for more robust evaluation schemes on the existing benchmarks. The code for this paper is available at https://github.com/National-Center-for-AI-Saudi-Arabia/lm-evaluation-harness.
△ Less
Submitted 3 July, 2024; v1 submitted 1 February, 2024;
originally announced February 2024.
-
M-Government, Wilaya and Women's Empowerment in the Kingdom of Saudi Arabia
Authors:
Norah Humus Alotaibi,
Salihu Dasuki,
Efpraxia Zamani
Abstract:
With the loosening of the male guardianship system (Wilaya) in the Kingdom of Saudi Arabia (KSA), this study examines the contribution of m-government to the empowerment of Saudi women. We employ the key concepts of Sen's Capability Approach (CA) to understand how m-government services in the KSA have provided opportunities for women to become empowered. The findings of the study indicate that m-g…
▽ More
With the loosening of the male guardianship system (Wilaya) in the Kingdom of Saudi Arabia (KSA), this study examines the contribution of m-government to the empowerment of Saudi women. We employ the key concepts of Sen's Capability Approach (CA) to understand how m-government services in the KSA have provided opportunities for women to become empowered. The findings of the study indicate that m-government contributes towards women's empowerment by providing opportunities to participate in social and economic activities. However, there are also key social and cultural factors that impede the use of m-government services for empowerment, and we found these to include religious beliefs, Saudi traditions and customs, and husbands' jealousy. The study makes some important contributions to theory and practice by being the first study to focus on the use that Saudi women make of the opportunities now available to them to access government services through m-government applications and to address the cultural barriers which may function to prevent their access.
△ Less
Submitted 22 August, 2021;
originally announced August 2021.
