Harnessing the Power of Artificial Intelligence to Vitalize
Endangered Indigenous Languages: Technologies and Experiences

Doing research with Indigenous peoples is subjected to specific guidelines and legal constraints. Mihesuah (1993) is a good example of a comprehensive set of guidelines for research with US American Indigenous communities. Straits et al. (2012) also proposed a set of guidelines on how to engage in research with Native US American communities based on 11 principles, including native-centrism, co-learning and ownership, continual dialogue, transparency and accountability, integrity, and community relevance.

Most importantly, there is a lot of distrust from many Indigenous communities towards researchers and academic work, resulting from a history of exploitation, disregard, and knowledge extractivism: “… [the word research] is probably one of the dirtiest words in the Indigenous world’s vocabulary.” (Smith, 1999, p. 1). To address these issues, Smith (1999) proposes, as part of a decolonization perspective, that relational accountability Wilson (2008) should guide these engagements since it is inherent to Indigenous ways of doing. The key ideas behind relational accountability is that relationships with the communities are important in research and that all the parties are responsible for maintaining them.

Besides ethical considerations, there are legal and regulatory procedures that must be followed in different countries when working with specific Indigenous communities. Particularly relevant to our cases, of developing AI-based tools, are issues related to data sovereignty, consent, and intellectual property (IP) rights Harding et al. (2012) which should be considered in tribal research in the USA. Research, in such cases, should include special procedures for informed consent processes and the involvement of community members in defining exposure and risk to the community. Similarly, Sahota (2007) discusses the need for research regulation in American Indian and Alaska Native communities and the challenges to establish this regulation.

Llanes-Ortiz (2023) has proposed a research framework based on 7 key approaches for digital initiatives with Indigenous languages, based on: facilitating digital communication in Indigenous languages; multiplying Indigenous language content online; normalizing the use of Indigenous languages online; educating in and teaching Indigenous languages online; reclaiming and revitalizing Indigenous languages and knowledge digitally; imagining and creating new digital media in Indigenous languages; defending spaces for Indigenous languages and linguistic rights; and protecting Indigenous linguistic heritage and communities.

The research described in this paper aims to explore and develop technologies which are related to many of those issues and, in particular, to the need to normalize the usage of Indigenous languages in the online space. Similar issues are also discussed by Oliveira et al. (2024), with a focus on practical aspects such as fonts, keyboards, and Unicode representations.

When it comes to AI- and NLP-related research, gathering, controlling, and using linguistic data become essential parts of the research process and, therefore, of ethical concerns. The sovereignty of Indigenous data is an area that has attracted considerable discussion in recent years, particularly in the case of population and genetic data Kukutai and Taylor (2016); Walter et al. (2021); Kukutai (2023). Its importance for digital and AI research has attracted some recent attention Llanes-Ortiz (2023). There are also some efforts to create new agreements and practices, such as the Kaitiakitanga data license⁷⁷7https://github.com/TeHikuMedia/Kaitiakitanga-License proposed by members of the Maori language community (see also the various links provided by TeHiku⁸⁸8https://tehiku.nz/te-hiku-tech/te-hiku-dev-korero/25141/data-sovereignty-and-the-kaitiakitanga-license.).

A more specific guide for AI-related work, including tool and technology design methodologies, was proposed by the The Indigenous Protocol and Artificial Intelligence (A.I.) Working Group Lewis et al. (2020) as a result of two workshops with Indigenous leaderships, linguistic professionals, and computer researchers. Nevertheless, although large AI conferences have hosted workshops dedicated to Indigenous contexts, the discussion of ethical guidelines when working with Indigenous peoples is still limited within the AI community.

The engagement principles and ethical guidelines used in our research have been discussed in more detail in  Pinhanez et al. (2023). In particular, the paper describes a practice, which we have adopted in our work as described in the next sections, of putting in place damage containment procedures when dealing with Indigenous data.

Such damage containment procedures are based in four different damage mitigation actions Pinhanez et al. (2023). First, we ensure that everyone involved in handling language data is fully aware of the ethical issues and the dangers of releasing data without the proper authorizations. Second, as part of the containment process, we do not release the data or the created models publicly, not even among other AI researchers, which is a common practice in the field. Third, we make sure that prototypes of tools, especially in actual systems, are only deployed or tested with express authorization from the community, as they may generate inappropriate or offensive language. Fourth, the existence of the data and its associated models, including their shortcomings, is transparently disclosed to the Indigenous communities and other stakeholders, and protocols to control and use the data are established.

In the case of the works described in this paper, we started in 2022 to contact various Indigenous groups and organizations in Brazil. Following a series of meetings with the Tenondé Porã community on the outskirts of São Paulo city, in 2023 we were invited by the community to explore the use of writing assistants by Indigenous high school students and to conduct activities fostering community-lead linguistic documentation and analysis. This school is part of an Indigenous land, home to approximately 1,500 people, where the main language is Guarani Mbya.

The Guarani Mbya language is spoken by approximately 6,000 people in Brazil Morello and Seiffert (2011), mostly in the South-Southeast region. Although it is still actively spoken and well-studied, there are few sources of translated texts and digitized data. Guarani Mbya belongs to the Tupi linguistic family and it is related to the Guarani language spoken by millions of people in Paraguay and Bolivia. However, it is as different from Guarani as Portuguese is from Spanish.

The invitation from the Tenondé-Porã community led to weekly 2-hour workshops where various technologies and prototypes were introduced to and used by the high school students of the Gwyra Pepo Indigenous Guarani School. The students explored these technologies and discussed how to modify and improve them. As a result of this work, we developed an initial prototype of a writing assistant for the Guarani Mbya language, which included rudimentary electronic dictionaries, and word prediction, and basic translation.

The technical aspects of those tools are discussed in sections 4 and 5 but some of them could only be built because we applied state-of-the-art AI technology. For instance, the translator was developed by fine-tuning a German-English high-quality translator using data obtained from traditional dictionaries, schoolbooks, and folk tales, mostly from the collection of Prof. Robert Dooley Dooley (1985, 1988a, 1988b, 2016), an expert in Guarani Mbya.

We faced a major challenge when we began designing the workshops with the high school students. As discussed in more detail in Pinhanez (2023), the last 15 years have produced many scholarly works with cases of engagement with Indigenous communities for the design of digital artifacts Awori et al. (2015); Muntean et al. (2017); Taylor et al. (2017); Muashekele et al. (2019); Reitsma et al. (2019); Tzou et al. (2019); Moradi et al. (2020); Shedlock and Hudson (2022). Most of these works focus describing key principles that designers should follow when engaging with Indigenous communities.

However, while this literature offer extensive descriptions of what should be done in the design process, it often provides very limited on how the actual design process should happen. For instance, when considering the use of Respectful Design, proposed by Sheehan (2011), we could not find any literature on how it should be implemented in actual design processes. There are a few exceptions were details about how design workshops were conducted were actually provided Taylor et al. (2017); Muashekele et al. (2019); Tzou et al. (2019). However, even in those cases a systematic description of the methods and means was never provided nor good ideas and practices on how to actually conduct the design workshops.

Interestingly, some works discussing community research related to Indigenous education, schools, and connectivity Franchetto (2008a); Hermes et al. (2012); Arola (2017); Leal and Teles (2022) and Indigenous writing Johnson (1997); Franchetto (2008a) were more helpful. In particular, the discussion in Arola (2017) about being an Indigenous person vs. doing the Indigenous way was enlightening and helped us to outline some initial ideas for the workshop.

Based on this concept, we structured our workshops around three principles. First, we adopted the conversational structure observed in the three meetings we had with the leadership of the Tenondé-Porã community. In those meetings, everyone was welcomed to voice their opinions, whenever they felt comfortable, and the discussion progressed at its own pace toward a consensus. In our work at the high school, students could enter and leave the classroom, stop to smoke their pipes, small children were welcomed to observe, and humor and laughing were valued and welcomed, similar to what we had observed in the meetings with the community.

Second, we began the process by exploring the students’ current activities on the Internet, particularly on WhatsApp and in the game Free Fire. This served as a starting point to examined their use of written language and how a language aid could play a role in their current usage. For instance, students reported to us that sometimes they used their native language as code to talk among themselves with privacy in public forums of Free Fire.

Third, we tried to downplay our own eminence as scholars, linguists, designers, and computer wizards. On the second workshop, we presented a version of a Guarani Mbya to Portuguese translator which we knew had a poor performance. The goal was to showcase our incompetence in their language and also to present ourselves as people who can fail, sometimes miserably.

In total, 14 workshops were conducted over three months, during which different versions of the writing assistant prototype and its components were used and discussed within the context of various writing activities. One of the best workshops happened when we used, as a writing theme, images from a recent protest by the Guarani community in São Paulo for land rights and the subsequent confrontation with the local riot police. Possibly, since many of them knew people involved in the conflict, they felt more interested and engaged to write to document their struggles.

This first engagement had limited outcomes in terms of creating high-quality technology or deploying actual writing tools. Instead, its main result was to provide a context for us, for the students, and for the community to understand the opportunities and challenges involved in bringing AI to the realm of Indigenous languages.

For the researchers involved in the workshops, this engagement demonstrated the need for good writing tools and methods to support a generation of students who, despite being fluent in their native language, were still learning how to write in it. While these youngsters were actively involved in writing messages among themselves, reading social media, and sharing content, the presence of Guarani Mbya text in their virtual lives seemed to be almost nonexistent.

We concluded that there is a pressing need to develop tools to support writing among youngsters, who encounter difficulties translating concepts and ideas they can easily express verbally into text. Also, it became evident how diverse their writing abilities were and the challenge of creating supporting tools that could be used by students of different levels of literacy.

Conversations with the students during and after the workshops led us to believe that the workshops had some positive impacts on the participating students. The presence of a team of individuals, clearly unable to understand their language but interested in developing tools to enable them to write in it, was recognized by some as an indication of the value placed on their culture. We also observed some students expressing interesting on learning how to use computers to improve their writing skills, on how to employ them to gain a better understanding of their own language, and even on how to program them.

Finally, for the community where the school is located, our project seemed to have contributed to a larger discussion about how the community wants to interact with the digital world and the Internet. This is a community where high-speed Internet had arrived just one year before our engagement, through an optical cable provider, since before there was very poor cell phone coverage in the area. When we engaged with them, there were ongoing debates about which websites and apps the community would allow access to, and when.

We believe our project served as an opportunity for the community to explore potentially positive uses of these resources, in a controlled experiment. In fact, after the four months of the project, the Tenondé-Porã expressed concerns about exposing their language and culture to the Internet. They requested that our activities be suspended while the community deliberated on the proper continuation of the project. We assured them that no language data had been collected during the process, thanked them for the opportunity to collaborate with their students, and suspended further work on the Guarani Mbya writing assistant, accordingly.

Consider all the different ethical issues discussed in sections 3.1 and the lessons learned working with the Guarani Mbya students explored in 3.2, we started to question whether the traditional ways AI has been often developed would work for Indigenous communities.

Figure 4.a shows the traditional AI development cycle to text-based systems which starts with an initial cycle of data gathering, often from the Internet, identified as data crawling, followed by model training, and a repeating cycle of model usage, data collection from the users’ interaction with the model, and retraining of the model.

Is Internet crawling a feasible strategy for endangered Indigenous languages? Putting aside for a moment the issue of how to get approval and permission from Indigenous peoples to use data obtained from the Internet, an important technical challenge is to find public websites containing text in endangered Indigenous languages. As discussed before, data from disappearing languages is scarce Vasconcelos et al. (2024), even for the languages considered here, with the number of speakers ranging from 1,000 to 100,000.

To find this data, which is dispersed among vast amounts of content in more used languages, it is fundamental to employ automatic mechanisms for language identification. In Cavalin et al. (2023) we explored the development of language identifiers for 7 Brazilian Indigenous languages, using an SVM classifier with bag-of-words features trained on data from the Bibles dataset described in section 4. When tested with non-Bible data, recognition accuracy varied from $37.3\%$ to $91.7\%$ in test datasets with only Indigenous languages samples. Accuracy was higher when similar languages from the same linguistic family were considered. However, in a realistic evaluation where the data from more commonly used non-Indigenous languages were also present in high proportions, the performance was much worse Cavalin et al. (2023).

The first impulse of AI developers and scientists in such conditions would be to engage the community to create a data gathering mechanism based on the promise of future use of the system. In our experience, this is very difficult because of both the difficulties in communicating how AI is built and the common distrust of those communities towards research projects as discussed before. Instead, our experience with the Guarani Mbya people suggested that a different framework for AI development is needed, centralized on the use of the tools by the community.

Figure 4.b shows a diagram of the proposed AI development cycle. First, it is anchored on community engagement and community sovereignity, as central pillars to ethically and sustainably realize the focus of the development which is community usage. To enable the initial use by the community, there is an initialization step, with approval of the community, where a starter model is built based on linguistic data such as publicly available dictionaries, theses, grammars, books, and similar materials. The starter model is the seed for community usage, even if it has shortcomings such as the translator we used with the Tenondé Porã students.

In the AI development cycle we are proposing, there are two sub-cycles associated the community usage, short- and long-term. The short-term sub-cycle follows the path of traditional AI development with data collection and model training activities, but in this case with direct participation of the community in key decisions and under its strict supervision. Also, all data collected is considered private, belongs to the community, and is used only when authorized. The long-term sub-cycle refers to the establishment of structures and processes of data governance and to the training of people and the creation of mechanisms to allow model appropriation.

The centrality of community usage is key because it guarantees both that the tools being developed are useful and used and that the research and development efforts provide a clear and immediate return to the community. However, we still have to demonstrate that development of AI tools and systems based on this framework is feasible, useful, and efficient. Some initial works exploring the framework are presented in the next sections but first we describe our engagement efforts to find a community to explore those ideas with.

Following our engagement with the Guaranis, we decided to focus on another Indigenous language, Nheengatu, which is spoken by approximately 20,000 people across three different areas of the Amazon area and in the Northeast of Brazil. This language is used by various peoples and ethnicities, including cases where the language was adopted by groups after the loss of their original language, such as the Baré people Epps and Stenzel (2013).

The choice of Nheengatu was motivated the availability of public data and knowledge, its multi-ethnic characteristic, the participation of linguistic experts at the University of São Paulo, and ongoing Indigenous-led initiatives focused on translation in Nheengatu. Also, there have been some recent efforts to bring Nheengatu to the digital world, such as the self-learning tool Nheengatu app⁹⁹9Developed by Suellen Tobler, https://heengatu-app.web.app..

In our on-going engagement with the community, we have been working on two basic workstreams. First, we have recently started a partnership with a group of Indigenous students, mostly from the Baré ethnicity, from the State University of Campinas. This collaboration will involve a structured writing workshop using a new and improved version of the Nheengatu writing assistant, ultimately leading to the development of educational and cultural materials.

The second workstream aligns with the need for more comprehensive translation services and materials. The availability of translation services, especially from dominant to Indigenous languages, is important to increase the accessibility to legal, health, schooling, and other government services. At the same time, it is important that Indigenous individuals are able to communicate in their own language to denounce crimes, request services, and participate in the political and social debates, making the translation from their languages to a dominant language also essential to citizenship.

In both scenarios, the goal is not to replace the Indigenous translators but rather to make their work more productive and reliable, allowing an often small number of translators to support a larger volume of demands and tasks. We are currently establishing a collaboration with a team of professional translators and writers in the Nheengatu language, who recently undertook the translation of the Brazilian Constitution into Nheengatu¹⁰¹⁰10https://www.cnj.jus.br/wp-content/uploads/2023/07/constituicao-nheengatu-web.pdf.. The goal is to collaborate with them in developing a version of the writing assistant tailored for translation tasks and, possibly, focusing on legal and healthcare texts, incorporating as much as possible high-quality automatic translators as a way to enhance both productivity and quality.

Our plan is to work with those communities to build AI-based tools for Nheengatu as a proof point that it is possible to develop such tools for endangered Indigenous languages, in their typical contexts of ultra-low data resources. In the next two sections, we present the methods and technologies we have been exploring to create prototype tools together with Indigenous communities, following the proposed community-based AI development cycle depicted in figure 4.b.

To develop both writing assistants and translators, we have been exploring machine learning methods based on small amounts of data. The data used in our work can be divided into two basic types: data extracted from linguistic sources, such as dictionaries, lexicons, theses, and publicly available books; and data extracted from multiple versions of the Bible available on the Internet.

We have been working exclusively with Brazilian Indigenous languages (BILs). Brazil was home to about 270 Indigenous languages according to the 2010 Census IBGE (2010), although some linguistic experts believe the actual number is more likely to be close to 200 Franchetto (2020); Storto (2019). Those languages were spoken by approximately 800,000 people IBGE (2010), with half living in Indigenous lands, although the 2022 Census revised this number to about 1.7 million¹¹¹¹11https://biblioteca.ibge.gov.br/index.php/biblioteca-catalogo?view=detalhes&id=2102018.. Storto (2019) provides a good overview of the history, structure, and characteristics of a few BILs. Almost all of these languages are considered endangered Moseley (2010), remnants of the 1,000 languages estimated to be in use in Brazil before the arrival of Westerners 500 years ago Rodrigues (2019).

The first type of data comprises data extracted from publicly available dictionaries, lexicons, theses, and books. For the languages in the range of 1,000 to 100,000 speakers we are targeting, such sources are common, mostly created through linguistic research and educational efforts. We have mostly collected resources for two languages we have been working with, Guarani Mbya and Nheengatu. The data, often in the form of PDF files, are processed by scripts and manually, resulting in datasets with pairs of sentences, for the training of translators, or well-structured sentences, for the training of encoders of Indigenous languages models and to be used in development of the writing-support tools. Following the previous discussion, we do not release publicly or share this data without the permission of the associated Indigenous communities.

The second type of data was collected in the early stages of the project by researchers from IBM Research and comprises 39 Indigenous languages spoken in Brazil, for which we found translations of the New Testament of the Bible, a book with about 7,000 verses in its English versions. The Bible is often available in many of these languages due to translations by Christian churches Franchetto (2008b). Table 1 lists these languages, including 36 spoken primarily in Brazil and 3 Guarani-related languages used mostly in Paraguay and Bolivia, but also spoken in some areas in Brazil. We adopted the Indigenous language classification, nomenclature, and data from the 2010 Brazilian Census by IBGE (IBGE, 2010) and language acronyms according to ISO 639-3.

This dataset, referred as Bibles, was mainly obtained from the ebible website¹²¹²12https://ebible.org/download.php.. A few other languages were sourced from the YouVersion online platform¹³¹³13https://www.bible.com/en-GB/.. The Bibles dataset consists of 188,033 parallel verses from the New Testament in English and the 39 Indigenous languages listed in table 1. The parallelism among translations of the same verse, performed by some of the authors, is of reasonable quality, although we are aware that the sources of the translations come from different versions, languages, and narrative styles of the Bible.

To avoid cross-contamination in the decoder and to study memorization issues without data leakage between the training and test sets, we used the Matthew chapter from the New Testament as the source for the test set, and the remainder of the text as the training set. We are aware that some similarity among verses could happen between the book of Matthew and the other synoptic gospels, such as Mark and Luke. However, we consider the existence of some similarity to be positive, as in most practical multi-language training datasets the presence of similar sentences is common.

We understand that using the Bible as a source of data for Indigenous languages raises important ethical, historical, and cultural concerns. Many unfortunate aspects of past and present colonial history of Indigenous peoples, especially in the Americas, are connected to different forms of Christianity. At the same time, the Bible is one of the most commonly found documents translated to several of those languages, by Jesuits in the early days of colonization and often by Evangelical churches and initiatives over the last 100 years Franchetto (2008b). As such, the translations of the Bible are often associated with different forms of cultural abuse and violence and the establishment of orthographies of domination Franchetto (2008b).

However, such texts are one of the few available sources of parallel multilingual datasets for most Indigenous languages. We thus view the use of the Bible in this work as an “exceptional” first step, where it is treated as potentially “toxic” data that should not be used, in principle, for any actually deployed system unless with explicit agreement of the Indigenous community. Nevertheless, we believe Bible data can be used carefully for in-laboratory technical experiments in well-contained contexts, such as the studies we have performed. To mitigate some of these risks, we implemented the protocols suggested in Pinhanez et al. (2023), including the adoption of containment procedures.

To evaluate whether using multilingual MTs was a effective strategy for endangered Indigenous languages, as often suggested by the literature Lee et al. (2022); Chen and Abdul-Mageed (2022), we used the Bibles dataset, and fine-tuned two commonly-used LLMs using one bilingual and two multilingual fine-tuning strategies.

The first pre-trained model is mBART50 Tang et al. (2020), an extended version of mBART Liu et al. (2020), with 680M parameters, and pre-trained with masked language modeling on 203M sentences. The second LLM is WMT19 Ng et al. (2019), a 315M-parameter German-to-English machine translator pre-trained on about 28M pairs of translated sentences and over 500M back-translated sentences. For more details about the experimental methodology, refer to Cavalin et al. (2024).

Our evaluation considered three different fine-tuning strategies using data from the Bibles dataset, resulting in three different types of models, evaluated on two sets of test data from the same dataset. Firstly, we assessed bilingual (BL) models created by fine-tuning each LLMs exclusively on source-to-target pairs from the BILs listed in table 1, yielding 39 unique bilingual models. Secondly, we considered the extreme multilingual scenario by fine-tuning both LLMs with all languages (AL) at once. Lastly, we created in-between multilingual solutions, the Tupi-family (TF) models, where the training set comprised by 10 languages belonging to the Tupi-Guarani family.

With the goal of measuring the impact of the previously-mentioned models, we defined two distinct sets of experiments. The first one, BL39 vs. AL39, considered all 39 BILs in a single test set, so that we could compare the performance of each bilingual model on its own language test set against a multilingual one trained with all the languages for each language data set. The second set is BL10 vs TF10 vs AL10, where we compared the BL models not only against AL but also against TF, which is more targeted at Tupi-family languages. For this, the test dataset contained only the 10 languages used to train the TF models.

We used three metrics to evaluate the results. The first is the traditional BLEU score Papineni et al. (2002), using the average of sentence-level BLEU scores. This allowed us to compare all metrics with the same methodology, used by two neural-based metrics, BLEURT Sellam et al. (2020) and BERTScore Zhang et al. (2020).

A summary of the results is provided in figure 5. As it can be seen, mBART50 performed slightly better than WMT19 but with higher standard deviation, and the three metrics afforded similar results. The three models yielded similar results, but the bilingual ones presented smaller standard deviation in comparison to the AL models. The TF10 model performed slightly better than the bilingual models but with a higher standard deviation, although the differences were not statistically significant.

In this work, multilingual MTs did not seem to perform better than bilingual translators. Therefore, our conclusion was that in cases where there is just the need for a translator for one language or data is available for only one language, working with a bilingual translator seems to be the natural choice. However, it could be argued that, in cases where data from multiple languages is available, the converse, that is, working with multilingual translators, would be recommended. As described in Cavalin et al. (2024), this is not the case, where we showed that high scores of the multilingual translators were achieved not by improving the quality of the translation, but by learning a “wrong” strategy of memorization and retrieval, signaled by the higher standard deviations.

In fact, we found that high standard deviations were a symptom that the translation model had started to perform rogue memorization, that is, it had become a retriever of the contents of the training set instead of a translator Cavalin et al. (2024). This can be seen by looking at the distribution of BLEU scores computed with samples from the training set, shown in figure 6. From these distributions, we can clearly observe that the BL39 models (both for mBART50 and WMT19) usually result in right-skewed normal distributions, while for AL39 the shapes of the distributions resemble more binomial distributions.

The high number of training samples with perfect translations of those models on the right side of the histograms were obtained by the models adopting a erroneous strategy of storing internally some of the verses, as further demonstrated in Cavalin et al. (2024). The paper also shows that the cause of this pernicious learning behavior was connected to the many-to-one mapping of the multilingual training datasets which can be alleviated by the use of rephrasing.

While working with the Tenondé-Porã community and their commitment to enabling their high-school students to write in Guarani Mbya, we considered that having translators to and from Portuguese would be helpful. Following our experiences with fine-tuning MTs in the context of the Bibles dataset, we started developing a Guarani Mbya to English MT to be used in conjunction with a commercial English-Portuguese translator.

To train the MT, we created a dataset, referred to as the Dictionary dataset, with pairs of sentences from three different sources. The first source was a set of Guarani Mbya short stories with 1,022 sentences, available in Portuguese and English Dooley (1988a, b). The second source comprised 245 texts extracted from PDF files with a pedagogical character Dooley (1985). The third source was Robert A. Dooley’s Lexical Guarani Mbya dictionary Dooley (2016), a reference work for the language, from which we extracted 2,230 sentence pairs. In total, the Dictionary dataset had 3,155 training and 300 test sentence pairs.

We then considered using the Guarani Mbya subset of the Bibles’ dataset to increase the amount of training data. Keenly aware of the ethical issues of using texts from the Bible in a translator for Indigenous languages in Brazil, we conducted a series of experiments, detailed in Domingues et al. (2024), to determine whether the additional “toxic” data from the Bible was useful and, if so, how much its use would contaminate the outputs.

As a baseline for this study, we defined the zeroshot model, consisting of the original German-English WMT19 model Ng et al. (2019) without any fine-tuning. Using only the Bibles training set, we generated three different models based on directly fine-tuning WMT19: mbya, the WMT19 model fine-tuned with only the Guarani Mbya data from the Bibles training set; TGf, the WMT19 model fine-tuned with Bibles data from 10 languages of the Tupi-Guarani linguistic family; and all, the WMT19 model fine-tuned with data from all the 39 Indigenous languages of the Bibles training set.

Using the data from the Dictionary training set, we generated four additional models: dict, the WMT19 model fine-tuned only with Dictionary data; mbya>dict, the mbya model fine-tuned a second time with Dictionary data; TGf>dict: the TGf model fine-tuned a second time with Dictionary data; and all>dict: the all model fine-tuned a second time with Dictionary data. Finally, we trained mbya+dict, consisting of the WMT19 model fine-tuned with Guarani Mbya data from the Bibles training set and Dictionary data simultaneously.

We used two metrics to evaluate the results: the BLEU metric, computed with the SacreBLEU Python package Post (2018); and the chrF metric Popović (2015), in both cases using segment-level scores. For the two metrics, we computed the average and standard deviation over the score of each sentence in the two test sets created from the Dictionary and Bibles datasets.

Figure 7 shows the results of the 9 models, evaluated with the Dictionary test data. For the two metrics, the zeroshot model had an extremely low performance, as expected since it is basically a German-to-English translator. The performance of the three models fine-tuned with Bibles data was poor, as expected. This becomes clearer when comparing the dict model to them: the average accuracy was considerably improved. Although dict had a larger standard deviation, it was significantly better than the other models ($p<0.001$) for the 2 metrics, using standard one-tailed Student t-tests.

The three two-step models (marked with ->dict), had gains of about 16% to 36% in accuracy over dict. The t-tests confirm that each of these models was significantly better than dict. The best nominal performance was achieved with the both-at-once model, mbya+dict, across all metrics and test sets, although there was some statistically significant difference ($p<0.05$) compared to some of two-step models.

The results with the Dictionary test set indicated, with high confidence and across all metrics, that the best results were achieved by the fine-tuning of the WMT19 model with the two types of data, from linguistic resources and the Bible, at the same time. In other words, the quantity of data available for fine-tuning seems to matter, as expected, and the best training methodology involved using all the data together in a single fine-tuning process. A more detailed discussion and related results are available in Domingues et al. (2024).

Having established that adding the Bible data could improve the accuracy of the Guarani Mbya to English translator, even when tested only with non-Bible sentences, the work then focused on quantifying the extent of contamination in the output. Specifically, we aimed to determine how many of the translators’ outputs contained, either explicitly or implicitly, typical words or language from the Bible.

Table 2, from Domingues et al. (2024), contains a random selection of outputs for 15 sentences in Guarani Mbya, together with the expected output. Segments manually identified as associated with biblical texts and expressions are marked in red, for three of the models. All the 15 outputs of the mbya translator have contamination, as expected since it was trained only with Bible data, while the dict translator, which was not exposed to Bible data in the fine-tuning process, had none.

The mbya+dict outputs depicted in table 2 show only one possible case of contamination. This output, “and the one who drinks the spirit remains in it, though the spirit remains.” resembles the John 6:56 verse, “The one who eats my flesh and drinks my blood resides in me, and I in him.”.

The qualitative evaluation of all the 300 outputs of the mbya+dict for the Dictionary test set found that 14 (4.7%) of the 300 outputs had some level of contamination, including 2 obvious cases where the word “Jesus” appeared. These 14 outputs are shown in table 3, from Domingues et al. (2024). They include, as contaminated outputs, examples where words such as “prayer”, “dove”, “bagel” were produced; and expressions such as “washed his feet”, “blind eye to evil”, and “if I die, I die”, whose degree of connection to the Bible is questionable.

Based on those findings, as discussed in Domingues et al. (2024), we advise against the release of the mbya+dict translator in broader contexts and recommend its use only in tightly controlled situations where negative effects can be mitigated. Of course, the final decision shall be made by the relevant Indigenous communities Mihesuah (1993); Sahota (2007); Straits et al. (2012).

In our collaboration with the high school students of the Tenondé-Porã community, we incorporated the dict Guarani Mbya to English translator with a commercial, API-based English to Portuguese translator into a writing assistant, which will be discussed in detail in section 5. As expected, due to the low SacreBLEU and chrF average scores (see figure 7), the quality of the translations it generated were of limited usefulness to the students.

Improving the quality of this translator was important, so we began by seeking a better understanding of the errors it was producing and assessing the usefulness of different parts of the output. Drawing conclusions about human usefulness of a translator based only on values from automatic metrics such as SacreBLEU is challenging, since they rely on straightforward computations such as word comparison and $n$-grams, often overlooking semantic issues. Therefore, to determine the usefulness of the translators, we conducted a human evaluation on the texts generated from the test set inputs.

This evaluation, described in detail in Pinhanez et al. (2024) involved the ranking of each of the generated outputs in a seven-point scale according to how useful it would be for someone proficient in the language using the translator as a writing support: near-perfect, correct, mostly correct, usable, mostly incorrect, incorrect, and very wrong.

Figure 8 (left) shows the histogram of the distribution of the 300 outputs of the Guarani Mbya translator (dict), according to the usefulness scale. About 40% of all outputs were in the very wrong category, with 26% categorized as incorrect and mostly incorrect categories. Of the remaining 34%, about 28% were sentences requiring a significant level of human intervention to be used (usable and mostly correct categories), with only 7% suitable for an automatic translation scenario. As a reference, the distribution of the SacreBLEU scores of these 300 outputs is shown in figure 8 (right).

As we moved to work with the Nheengatu community, as discussed earlier, we decided to apply the same techniques to develop a Nheengatu to English translator. The Nheengatu dataset comprised sentences from five different sources with Portuguese translations. The first source was the Nheengatu lexicon Ávila (2021) with 6,846 sentences extracted from the lexicon examples. We processed the original file provided by the author. The second source was Corpus Lições Ávila (2021), containing 1,665 samples available in a spreadsheet format. The other sources, extracted from PDFs, included: Texto Anônimo Navarro (2011), with 427 samples; Brilhos na Floresta Ishikawa (2019), with 590 samples; and Curso LGA Navarro (2016), with a partial extract of 590 samples.

In total, the Nheengatu dataset contained 7,281 samples, with a random split of 6,804 samples for training and 233 samples (10% of the data from all sources except Nheengatu lexicon) for testing. We translated the Portuguese sentences to English, when necessary, using a Portuguese-to-English commercial translation service¹⁴¹⁴14IBM Watson Language Translation v9.0.0..

The top row of figure 9 shows the equivalent results of the Guarani Mbya for the Nheengatu translator. The numbers of the Nheengatu translator were slightly better, with 42% of the 233 outputs in the very wrong category but only 16% in the incorrect and mostly incorrect categories. Of the remaining 42%, 24% would need human correction to be usable, and 18% would be suitable for an automatic translation scenario. The distribution of the SacreBLEU scores of the outputs, shown on the right side, was also better than that of the Guarani Mbya translator, though not significantly better. The improved results of the Nheengatu translator can be attributed simply to the larger amount of training data.

A side effect of the manual evaluation of usefulness of the outputs was that it became clear to the team that there were many errors in the test set, included expected outcomes which were empty sentences and some residual lexical information in some entries. This was mainly caused by the semi-automatic conversion of the input materials, especially from the lexicon dictionary. This prompted a manual, exhaustive revision of the whole training set, removing incorrect pairs and fixing others, which resulted in the Nheengatu Clean dataset, with 6,848 pairs, about 6% smaller than the original Nheengatu dataset. This dataset was split into 6,621 pairs for training and 227 for testing.

The translator created using the Nheengatu Clean dataset was remarkably better than the one created with the original Nheengatu dataset, as shown in the bottom row of figure 9. About 48% of the 227 outputs were in the near perfect category, suitable to automatic translation; 17% were correct oder mostly correct; 8% were usable; and only 27% were unusable, summing up the very wrong, incorrect, and mostly incorrect categories. The average BLEU score jumped from $18.9\pm 16.8$ with the original dataset to $38.6\pm 47.1$, flipping the histograms of the SacreBLEU scores, as shown by the right graphs of figure 9.

Notice that all the above gains were obtained by removing only 6% of the entries and fixing basic errors in about 10% of them. Moreover, the manual inspection of the outputs of the original Nheengatu translator seems to indicate that the translator learned to output errors, that is, to generate empty outputs and to include incorrect elements. What we observed was that a fine-tuning process with very little data seems to see errors as valid outputs and learns to generate them. Fine-tuning LLMs with very small amounts of data seem to require that what is presented to the model to be heavily curated, because the LLMs learn to generate errors if they see even few examples of them.

These and the previous results seem to indicate three important guidelines when building MTs by fine-tuning large translators with very limited amounts of data. First, unlike what is recommended by most of the previous literature, there is almost no gain from using multilingual translators. Second, problematic data sources should not be used since contamination is likely to appear in the outputs. And third, the small datasets used for fine-tuning should be thoroughly inspected and cleaned from errors.

The development of prototypes of writing assistants started about one month after we begun to have weekly workshops at the Tenondé-Porã high school. We employed technology probes Hutchinson et al. (2003), a variation of the idea of cultural probes Gaver and Dunne (1999), a.k.a. design probes. The main idea was to insert some sort of technological artifact into the classroom which could elicit responses from the students in the context of actual writing tasks.

The first technology probe was a rudimentary version of the Guarani Mbya to Portuguese translator, accessible through a bare-bones Internet interface. It showed it had almost no use for the students, aggravated by the poor quality of the translations. But it triggered good conversations with the students about what kind of writing support would be useful for them. Based on this feedback, we started focusing on another component tools which seemed to be more useful, such as the tools to do word and sentence completion, and access to a dictionary.

For the next iteration we created a new version of the writing assistant, incorporating early versions of those tools. The basic layout is shown in figure 11, comprising an area for writing in Guarani Mbya, a button to generate translations to Portuguese, and an area where the translation was shown. It included a word completion tool, shown in action in figure 11, suggesting possible completions for the partial word “Avax” as a list in the bottom of the interface, followed by their meanings in Portuguese. The sentence completion tool worked in identical ways except that was triggered by the typing of spaces and punctuation marks.

We worked with the students in a couple of workshops with this prototype. Our anecdotal observations did not see much actual acceptance of the suggestions provided by the tools nor much use of the translator. However, the whole environment seemed to be conducive to explorations and to motivate the students to write, and by the third workshop we started to see longer and more complex sentences being produced as expected. This in spite of the tools still having a very limited performance and many students struggling with typing using the laptops we provided (they were used to type in cell phones). The main conclusions were that the idea of a writing assistant as a tool to foster writing was valid but that better component tools were needed and, eventually, a smartphone version.

As we moved to work with the Nheengatu language, we ported the writing assistant to Nheengatu. The main screen of this application (in its English version) is shown in figure 12, following basically the same design but with some improvements in the drop down mechanisms of the word and next-word completion. More importantly, for the Nheengatu language we developed a Portuguese to Nheengatu translator which we also made available in the interface (see figure 13).

We are in the process of introducing this version of the writing assistant to two groups of Nheengatu speakers. The first is composed of students of the State University of Campinas, mostly from the Baré ethnic group, who have very diverse knowledge and fluency levels in Nheengatu. A preliminary workshop indicated that the usage is quite different for individuals who know and do not know how to read and write. We could also see that the use of the translators was more common, possibly because of the better quality of our Nheengatu translators (see figure 13 for an example) compared to the Guarani Mbya one. We are currently organizing a writing workshop in the second half of 2024 to continue the co-design of this application.

The second group of Nheengatu speakers are Indigenous teachers and translators who have both a high command and a high need of writing in Nheengatu. This collaboration is still in its initial stages but one of our goals is to develop versions of the writing assistant which are more appropriate for writing and translation work, such as the ability to handle long texts, and investigating its incorporation into professional or open-source word editors.

From the interaction we are having with teachers and leaders of the Indigenous communities, it is fundamental that we make the writing assistants available in smartphones. First, the majority of teenagers and young adults do not have access to traditional screen-and-keyboard computers but only to cell phones. Second, as we saw in the work with the Guarani students, most of people in this group do not have experience nor proficiency with traditional keyboards, preferring the use of thumb-based typing common in mobile phones. And finally, in the hot and humid environment of many Indigenous groups in Brazil, smartphones have been much more successful in fending off damage caused by mold and humidity.

Considering those issues, we have been exploring, in parallel with the efforts described before, the deployment of writing assistants in more readily available platforms, such as smartphones and social media apps. Figure 14 shows an early prototype of the writing assistant as a chatbot in WhatsApp, where dictionary and translation services are made available through interactions with the chatbot¹⁵¹⁵15This work was developed in collaboration with undergraduate computer engineering students from Insper.. Currently we are starting to work on porting the writing assistant to an Android app, together with a system to re-configure the keyboard we have already developed¹⁶¹⁶16Also, in a collaboration with students from Insper..

In the development of the actual tools used in the prototypes of the writing assistants described in the previous paragraphs, we did not use a single ILM as the source of the component tools as advocated in the description of our technical strategy shown in figure 10. The main reason was that the process of co-design and development of the prototypes required to have versions of those tools working, to some extent, in the early steps of the initialization process, as part of the new development cycle proposed before (depicted in figure 4.b).

We now briefly describe the development process of the tools which were actually used and discuss how we will move to a single-source ILM framework. Notice that the existence of those tools will eventually simplify the training of the generic ILM because they can be easily employed to generate prompt-like synthetic training data for the generic ILM.

The word dictionary used in the Guarani Mbya and Nheengatu prototypes was implemented by looking up a database of words extracted from actual dictionaries available for the language. This database contained associated descriptions in Brazilian-Portuguese and English of each word. One shortcoming of that version was not listing valid variations of base words, such as the use of prefixes and suffixes to indicate gender, number and verbal tenses. Most dictionaries for major languages used in editing tools often incorporate the handling of variations by direct programming rules for valid modifications, a time- and effort-consuming process. We believe this can also be achieved by training the ILM with appropriate synthetic data, an idea yet to be verified in practice.

The word completion tool used in the Guarani Mbya and Nheengatu prototypes used a variation of the techniques used by the word dictionary, based on partial searches in the database of words. Partially entered words were matched against the list of valid words and the most likely words were listed in alphabetical order.

The next-word completion tool used in the Guarani Mbya and Nheengatu prototypes have been implement as a bag-of-words machine learning model Weiss et al. (2010), trained with sentences extracted from the bilingual datasets created for the training of the translators, detailed before. Next-word completion is a task extremely dependent on the context and good performance is often achieved through personalization, that is, on learning the most commonly used words by an individual. In this initial version, however, we focused on a general-purpose training set, where we decomposed the original sentences in sub-sentences with up to five tokens, and used the subsequent token as the label to train an SVM-based classifier. Consequently, when the user was typing in the writing assistant, the system sent the typed words to the classifier and the predicted class corresponded to the suggested next word.

We have not yet deployed a spell checker tool in the Guarani Mbya and Nheengatu prototypes, but one is being developed for Nheengatu using an LLM-based framework. The basic idea is to first generate a dataset with pairs of correct and incorrect sentences, where the incorrect versions are created synthetically by changing, removing, or adding letters, following common human patterns of producing typos. We first implemented this methodology with a dataset in Portuguese language, where we obtained an accuracy of about 60.8%. We are currently in the process of applying the same methodology to Nheengatu, using sentences extracted from the bilingual datasets created for the training of the translators, as we did with the next-word prediction tool. Besides, we are also investigating more effective ways to evaluate the results, such as applying a BLEU-like metric to compute the quality of the generate results.

The translator tools used in the Guarani Mbya and Nheengatu prototypes were different versions of the bilingual translators described previously in detail in section 4. The translators were deployed as API-based services and had an average response time of 1-2 seconds, which was found adequate for the task.

Notice that the translators we have used were based on fine-tuning large, high-resource, high-quality translators to Indigenous languages, generating translation-only ILMs and not generic, prompt-based LLMs. Given that the accuracy of translation-specific models is today often better than of generic LLMs, it is a good question whether having a translation-specific ILM in our technical strategy would not be a better option. We will explore this issue as we start developing the generic-task ILMs proposed in the strategy. An additional issue is that translators also include a second language to translate to and from, which may introduce noise into the generic ILM.

It is also important to highlight that the strategy of fine-tuning a single ILM to be a source of the component tools requires the development of methods and code which generate the appropriate synthetic training data, in the right format. This data can be complemented, eventually, by data logged from the writing assistants, obviously with the permission of the users and from the community, and with provisions to preserve privacy and remove personal information. We believe some of the synthetic data generators may also require language-specific knowledge which we will have to be encoded and tested by developers.

Those data generators are part of the set of tools depicted between the data sources and the fine-tuning process in figure 10, for which we are developing generic versions which will be eventually open-sourced, without language data. This layer is an important area of language-specific collaboration between Indigenous speakers, linguists, and developers of each language, and how to foster such collaborative environment and at the same preserve governance of the Indigenous language data is an open question.