Learning-From-Mistakes Prompting for Indigenous Language Translation

LLMs’ effectiveness on various task is primarily attributed to two main properties. Firstly, in-context learning Brown et al. (2020); Lester et al. (2021) allows the model to learn to solve specific problems by providing a small number of examples within the input context. The second one is the ability to follow the instruction Ouyang et al. (2022); Mishra et al. (2021); Wei et al. (2021), instruction-tuned LLMs can be guiding to solve new task based on text instruction just as the scenario they were trained.

Recently, some research has focused on enhancing these instruction following LLMs through in-context learning, Nguyen et al. (2023); Ahuja et al. (2023) explores the generation of unsupervised few-shot demonstrations to enhance translation effectiveness in low-resource scenarios. Additionally, Yao et al. (2023) utilized cultural awareness to optimize alignment in different languages, further augmenting the translation performance of LLMs.

It is noteworthy that, the mentioned works above have focused on low-resource data for LLMs in languages that have been encountered during the pre-training phase. In contrast, our emphasis lies in a scenario where the model has not been previously trained in this specific language. In contrast to conventional approaches, we refrain from training parameters on limited parallel corpora Gu et al. (2018); Lalrempuii and Soni (2023). Instead, our goal is to leverage the understanding and reasoning capabilities of LLMs, coupled with the provided data, to accomplish translation tasks for previously unseen languages.

In summary, to the best of our knowledge, no study has delved into the challenges and applications of utilizing LLMs for languages that have not been encountered before.

In the context of preserving and revitalizing indigenous languages, the work by Zheng et al. (2022) stands as a notable contribution. Zheng and colleagues introduce the Amis-Mandarin dataset, which includes a parallel corpus comprising 5,751 Amis and Mandarin sentences. This dataset is of particular relevance to our research on translating Chinese sentences into Taiwanese indigenous languages. The Amis-Mandarin dataset provides a valuable resource for studying indigenous language translation. It aligns with the objectives of our study, as it offers a substantial parallel corpus, a fundamental component for training and evaluating translation models. Our research similarly leverages parallel corpora, although we focus on the translation of Chinese into various indigenous languages, including but not limited to Amis. In this study, we conduct experiments on six different indigenous languages.

Furthermore, Zheng et al. (2022) compile a comprehensive dictionary containing 7,800 unique Amis words and phrases, each accompanied by its Mandarin definition. This lexical resource enhances the utility of their dataset for translation tasks. In our research, we assume the existence of a similar dictionary, emphasizing the importance of word-level translation between Chinese and Taiwanese indigenous languages.

Stap and Araabi (2023) evaluates the translation performance of different systems for Spanish to 11 indigenous languages from South America. The authors find that LLMs like ChatGPT are not yet good at translating into indigenous languages. This is likely due to a number of factors, including the lack of training data for indigenous languages, the complex grammar and sentence structure of indigenous languages, and the difficulty of capturing the nuances of indigenous culture in translation.

In recent research, Schick et al. (2023) discovered that LLMs exhibit the ability to discern how to employ tools provided by users, including external data. They adeptly combine this external information with their own knowledge to effectively address problem -solving tasks. These investigations delve into the mechanisms of CoT Inaba et al. (2023) and Self-instruction Yang et al. (2023) approaches, exploring how these methodologies assist LLMs in comprehending questions and utilizing the tools at their disposal. Additionally, there has been the development of question- answering datasets, such as ToolQA Zhuang et al. (2023); Inaba et al. (2023), which aimed at faithfully evaluating the ability of LLMs to use external tools for question-answering.

Inspired by these explorations into the understanding and application capabilities of LLMs, we take a similar approach in our method design. We offer KNN examples and word-by-word translation as tools for LLMs to improve their language translation abilities.

This research stands at the intersection of multiple areas, addressing the challenges of translating into low-resource indigenous languages using LLMs like ChatGPT. While prior works have explored low-resource translation and indigenous language preservation, our study distinguishes itself in two key aspects:

1. 1.

   Languages Unseen in Pre-training: Unlike previous research that has primarily focused on low-resource data for LLMs in languages encountered during pre-training, our work emphasizes the scenario where the model has not been previously trained in the specific target language. We tackle the challenge of translating into languages that lack representation in the model’s training data, making our approach more versatile and applicable to a broader range of indigenous languages.

2. 2.

   Few-Shot Prompting Techniques: Our research pioneers the application of few-shot prompting techniques to enhance translation capabilities for indigenous languages. We introduce innovative methods, including KNN-Prompting with RPC, CoT Prompting, and LFM Prompting, tailored to leverage LLMs’ inherent understanding and reasoning abilities. These techniques empower LLMs to effectively tackle low-resource language translation tasks, even when working with limited parallel corpora.

In summary, our paper bridges the gap between LLMs and low-resource indigenous language translation, offering practical and innovative solutions for preserving and revitalizing endangered languages. By exploring the potential of these models in an uncharted linguistic landscape, we provide a fresh perspective and a promising direction for future research in this domain. For clarity, we also compare the related work in Table 1.

We investigate the application of few-shot learning through the KNN-Prompting concept, as discussed in the works of Shi et al. Shi et al. (2022) and Xu et al. Xu et al. (2023). Our approach not only leverages contextually similar examples but also incorporates individual translations for each word in the source language. The methodology unfolds in the following manner:

• •

  When tasked with translating a sentence $s$, our method initiates by constructing a Retrieved Prompting Context (RPC) for $s$. This context includes:

  • -

    $k$ examples that are contextually analogous to $s$, selected based on their similarity.

  • -

    Translations for each word in $s$, sourced from a comprehensive dictionary.

• •

  For instances where direct word equivalents are unavailable, we employ the BERT-base-chinese model as an embedding tool. This model aids in computing similarities to identify the most appropriate substitute words.

• •

  The core principle of our method is to enable the LLM to assimilate the grammatical norms and sentence constructs of the target language. It achieves this through the analysis of the $k$ examples, thereby learning to organize the individually translated words into coherent and grammatically consistent sentences.

For a practical illustration of this process, please see Figure 2, which provides a concrete example of the RPC in action. We also show an example for prompting in Table 5.

In this methodology, we harness the CoT strategy, as delineated by Wei et al. Wei et al. (2022), to guide the LLM in effectively utilizing the RPC for translating a given sentence $s$. Specifically, this approach involves the following steps:

• •

  When presented with a sentence $s$ for translation, accompanied by KNN-RPC-prompting inputs (i.e., $k$ contextually similar examples and individual word translations), we further augment the LLM’s input with $q$ CoT demonstrations. These demonstrations are designed to illustrate how to use the provided RPC to formulate the final translated sentences.

• •

  An instance showcasing two CoT demonstrations is illustrated in Figure 3. It is important to note that these CoT demonstrations are integrated with the KNN-RPC-prompting inputs to serve as comprehensive prompting material for the LLM.

• •

  For detailed examples of this prompting structure, please refer to Table 6 in the Appendix.

The overarching aim of this methodology is to empower the LLM with an understanding of the grammatical rules and the proficiency to fully leverage the RPC, including both the retrieved sentences and individual word translations, for producing coherent and accurate translations.

LFM Prompting is a two-stage approach aimed at enhancing the quality of translations. This method leverages the result from the CoT KNN-Prompting and incorporates a feedback mechanism by conducting trial translation to refine the translation based on past translation errors. The method works in the following phases:

• •

  Phase 1: Trial Translation with CoT Prompting When given a sentence $s$ to translate, we start by retrieving $q$ contextually similar sentence pairs from the data store. Each pair ($s_{q_{i}},t_{q_{i}}$) consists of a Chinese sentence $s_{q_{i}}$ and its corresponding indigenous sentence $t_{q_{i}}$. For each $s_{q_{i}}$, we employ CoT KNN-Prompting (the method introduced in Section 3.2) to translate it, resulting in $\hat{t}_{q_{i}}$. At this stage, we have ($s_{q_{i}}$, $t_{q_{i}}$, $\hat{t}_{q_{i}}$). Our approach involves using these results as examples for the LLM to learn from its translation errors and make improvements.

• •

  Phase 2: Learning from Past Mistakes The second phase of LFM Prompting introduces a crucial element: the incorporation of past translation errors. Specifically, we treat ($s_{q_{i}}$, $t_{q_{i}}$, $\hat{t}_{q_{i}}$) from the Phase 1 as LFM examples. In this phase, we present the LLM with a set of such examples, alongside the translation $\hat{t}$ generated by using CoT KNN-Prompting to translate $s$. The language model is tasked with refining $\hat{t}$ by considering the error examples in translation. It uses the provided examples of mistranslations to correct and improve the initial translation $\hat{t}$, aligning it more closely with the correct target language structure and meaning.

Furthermore, Figure 4 provides a visual representation of the entire architecture’s workflow, illustrating the sequential processes outlined above. We also show a prompting example in Table 7.

UTF8bsmi

Utilizing the GPT-3.5-turbo-16k-0613 version with a temperature setting of 0, we employ Sentence BERT Reimers and Gurevych (2019) as the embedding model to retrieve $k$-nearest neighbor sentences. The similarity between sentences is computed using cosine similarity.

We use the learning materials for various indigenous languages from the ’Klokah’ website ¹¹1https://web.klokah.tw/ provided by the Foundation for the Research and Development of Indigenous Languages in Taiwan as our evaluation corpora. Each indigenous group consists of 450 sentences with corresponding Chinese translations and a dictionary of 1000 words (single word translation). For each language, we divide this dataset into two parts:

• •

  Test Data - A random selection of 100 sentences was used to evaluate the translation performance of various methods.

• •

  Reference Data - The remaining 350 sentences and all dictionaries were used as reference materials for the LLM translation.