-
Pretraining Data and Tokenizer for Indic LLM
Authors:
Rahul Kumar,
Shubham Kakde,
Divyansh Rajput,
Daud Ibrahim,
Rishabh Nahata,
Pidathala Sowjanya,
Deepak Kumar
Abstract:
We present a novel approach to data preparation for developing multilingual Indic large language model. Our meticulous data acquisition spans open-source and proprietary sources, including Common Crawl, Indic books, news articles, and Wikipedia, ensuring a diverse and rich linguistic representation. For each Indic language, we design a custom preprocessing pipeline to effectively eliminate redunda…
▽ More
We present a novel approach to data preparation for developing multilingual Indic large language model. Our meticulous data acquisition spans open-source and proprietary sources, including Common Crawl, Indic books, news articles, and Wikipedia, ensuring a diverse and rich linguistic representation. For each Indic language, we design a custom preprocessing pipeline to effectively eliminate redundant and low-quality text content. Additionally, we perform deduplication on Common Crawl data to address the redundancy present in 70% of the crawled web pages. This study focuses on developing high-quality data, optimizing tokenization for our multilingual dataset for Indic large language models with 3B and 7B parameters, engineered for superior performance in Indic languages. We introduce a novel multilingual tokenizer training strategy, demonstrating our custom-trained Indic tokenizer outperforms the state-of-the-art OpenAI Tiktoken tokenizer, achieving a superior token-to-word ratio for Indic languages.
△ Less
Submitted 17 July, 2024;
originally announced July 2024.
-
Brilla AI: AI Contestant for the National Science and Maths Quiz
Authors:
George Boateng,
Jonathan Abrefah Mensah,
Kevin Takyi Yeboah,
William Edor,
Andrew Kojo Mensah-Onumah,
Naafi Dasana Ibrahim,
Nana Sam Yeboah
Abstract:
The African continent lacks enough qualified teachers which hampers the provision of adequate learning support. An AI could potentially augment the efforts of the limited number of teachers, leading to better learning outcomes. Towards that end, this work describes and evaluates the first key output for the NSMQ AI Grand Challenge, which proposes a robust, real-world benchmark for such an AI: "Bui…
▽ More
The African continent lacks enough qualified teachers which hampers the provision of adequate learning support. An AI could potentially augment the efforts of the limited number of teachers, leading to better learning outcomes. Towards that end, this work describes and evaluates the first key output for the NSMQ AI Grand Challenge, which proposes a robust, real-world benchmark for such an AI: "Build an AI to compete live in Ghana's National Science and Maths Quiz (NSMQ) competition and win - performing better than the best contestants in all rounds and stages of the competition". The NSMQ is an annual live science and mathematics competition for senior secondary school students in Ghana in which 3 teams of 2 students compete by answering questions across biology, chemistry, physics, and math in 5 rounds over 5 progressive stages until a winning team is crowned for that year. In this work, we built Brilla AI, an AI contestant that we deployed to unofficially compete remotely and live in the Riddles round of the 2023 NSMQ Grand Finale, the first of its kind in the 30-year history of the competition. Brilla AI is currently available as a web app that livestreams the Riddles round of the contest, and runs 4 machine learning systems: (1) speech to text (2) question extraction (3) question answering and (4) text to speech that work together in real-time to quickly and accurately provide an answer, and then say it with a Ghanaian accent. In its debut, our AI answered one of the 4 riddles ahead of the 3 human contesting teams, unofficially placing second (tied). Improvements and extensions of this AI could potentially be deployed to offer science tutoring to students and eventually enable millions across Africa to have one-on-one learning interactions, democratizing science education.
△ Less
Submitted 30 April, 2024; v1 submitted 3 March, 2024;
originally announced March 2024.
-
Towards an AI to Win Ghana's National Science and Maths Quiz
Authors:
George Boateng,
Jonathan Abrefah Mensah,
Kevin Takyi Yeboah,
William Edor,
Andrew Kojo Mensah-Onumah,
Naafi Dasana Ibrahim,
Nana Sam Yeboah
Abstract:
Can an AI win Ghana's National Science and Maths Quiz (NSMQ)? That is the question we seek to answer in the NSMQ AI project, an open-source project that is building AI to compete live in the NSMQ and win. The NSMQ is an annual live science and mathematics competition for senior secondary school students in Ghana in which 3 teams of 2 students compete by answering questions across biology, chemistr…
▽ More
Can an AI win Ghana's National Science and Maths Quiz (NSMQ)? That is the question we seek to answer in the NSMQ AI project, an open-source project that is building AI to compete live in the NSMQ and win. The NSMQ is an annual live science and mathematics competition for senior secondary school students in Ghana in which 3 teams of 2 students compete by answering questions across biology, chemistry, physics, and math in 5 rounds over 5 progressive stages until a winning team is crowned for that year. The NSMQ is an exciting live quiz competition with interesting technical challenges across speech-to-text, text-to-speech, question-answering, and human-computer interaction. In this ongoing work that began in January 2023, we give an overview of the project, describe each of the teams, progress made thus far, and the next steps toward our planned launch and debut of the AI in October for NSMQ 2023. An AI that conquers this grand challenge can have real-world impact on education such as enabling millions of students across Africa to have one-on-one learning support from this AI.
△ Less
Submitted 8 August, 2023;
originally announced August 2023.
-
A Swarm of Simple Robots Constructing Planar Shapes
Authors:
Andrew Vardy,
Dalia S. Ibrahim
Abstract:
We present a new version of our previously proposed algorithm enabling a swarm of robots to construct a desired shape from objects in the plane. We also describe a hardware realization for this system which makes use of simple and readily sourced components. We refer to the task as planar construction which is the gathering of ambient objects into some desired shape. As an example application, a s…
▽ More
We present a new version of our previously proposed algorithm enabling a swarm of robots to construct a desired shape from objects in the plane. We also describe a hardware realization for this system which makes use of simple and readily sourced components. We refer to the task as planar construction which is the gathering of ambient objects into some desired shape. As an example application, a swarm of robots could use this algorithm to not only gather waste material into a pile, but shape that pile into a line for easy collection. The shape is specified by an image known as the scalar field. The scalar field serves an analogous role to the template pheromones that guide the construction of complex natural structures such as termite mounds. In addition to describing the algorithm and hardware platform, we develop some performance insights using a custom simulation environment and present experimental results on physical robots.
△ Less
Submitted 28 April, 2020;
originally announced April 2020.
-
Tackling Initial Centroid of K-Means with Distance Part (DP-KMeans)
Authors:
Ahmad Ilham,
Danny Ibrahim,
Luqman Assaffat,
Achmad Solichan
Abstract:
The initial centroid is a fairly challenging problem in the k-means method because it can affect the clustering results. In addition, choosing the starting centroid of the cluster is not always appropriate, especially, when the number of groups increases.
The initial centroid is a fairly challenging problem in the k-means method because it can affect the clustering results. In addition, choosing the starting centroid of the cluster is not always appropriate, especially, when the number of groups increases.
△ Less
Submitted 14 March, 2019;
originally announced March 2019.
