Rule-Based, Neural and LLM Back-Translation:
Comparative Insights from a Variant of Ladin

In recent years, a variety of methods have been developed to apply neural machine translation (NMT) also in low-resource scenarios Shi et al. (2022); Haddow et al. (2022); Ranathunga et al. (2023). The back-translation technique has shown to be particularly effective in such settings Sennrich et al. (2016); Edunov et al. (2018), offering the potential for substantial improvements in translation quality.

This work investigates the influence of the back-translation model selection for a low-resource language. We do this, by comparing the results obtained by fine-tuning a pre-trained multilingual NMT model using synthesised translations generated by (i) a NMT system fine-tuned on the available parallel data, (ii) a rule-based machine translation (RBMT) system developed for the specific language pair, and (iii) a large language model (LLM) prompted to translate given texts, accompanied by 8 exemplary samples.

The quality of the synthesised data, which in turn is determined by the underlying models used to generate it, matters Burlot and Yvon (2018). In our case, the synthesised translations originate from three models based on a different paradigm. Thus, the synthesised data is characterised by the specific strengths and weaknesses of the respective paradigms.

Rule-based systems are robust and computationally lightweight, but may face challenges in dealing with ambiguity. Moreover, they lag behind at the grammatical level. Neural models show a high ability to adapt to provided texts, but perform less well when confronted with out-of-domain data Shen et al. (2021). In contrast, language model-based approaches (LLMs) are praised for their ability to produce fluent, coherent texts, but they are prone to hallucinations Rawte et al. (2023). It is therefore an interesting question to investigate how this affects the quality of the NMT models trained on this data. This comparative analysis sheds light on the nuanced contrasts inherent in the different MT methods.

Our results show that in this low-resource scenario the back-translation model does not have a significant impact, and the performance of the models converges to similar results in terms of BLEU/chrF++ points. This assertion is supported by an empirical analysis carried out on the Val Badia variant of Ladin. Our main contributions are:

• •

  we are the first to explore MT for Ladin in general, with a specific focus on the Val Badia variant,

• •

  we compare an RBMT-, an NMT- and an LLM-based back-translation, providing insights into the efficacy of the methods for Ladin,

• •

  we establish baseline results and make the test data, the RBMT system, as well as the best-performing models publicly available

In Section 2 we describe our data collection and corpus creation process. In Section 3 we present the three different methods used for the back-translation of monolingual Ladin data into Italian. Section 4 gives an overview of the conducted experiments and Section 5 presents the obtained results. Section 6 discusses related work and similar approaches. In Section 7, we summarize and discuss future work.

This section gives an overview of the linguistic resources available for the Ladin language and describes the method employed to collect data for the specific Val Badia variant of Ladin.

The so-called back-translation, first introduced in Sennrich et al. (2016), refers to the process of automatic translation of monolingual texts in the target language to the source language. This method of enriching additional training data in the source-to-target translation direction (where the target side remains authentic) has proven to be particularly effective and is particularly valuable in low-resource scenarios. In this section we present the three different back-translation strategies used in our research to translate monolingual Ladin texts into Italian.

We used the opus-mt-ine-ine¹⁶¹⁶16https://huggingface.co/Helsinki-NLP/opus-mt-ine-ine model as base model for the experiments. In the rest of the paper, we use BM to refer to this model. We fine-tuned BM with the various data sets using the Transformers library Wolf et al. (2020), specifically leveraging the Seq2SeqTrainer module. We always trained a single model for both directions using the corresponding prefixes.

We configured the training to process batches of 16 samples, and restricted the input and output sequences to a maximum of 128 tokens to ensure manageable computation loads. The models were evaluated each 16,000 steps. As a stopping criterion, we used three consecutive evaluations resulting in an improvement of less than $0.2$ chrF points on the validation set. For training, we utilised an NVIDIA TITAN RTX graphics card with 24 GB. In total, we have trained 15 models:

• •

  Model N1: BM fine-tuned with the available parallel data consisting of 18,139 sentences.

• •

  Models N2/R2/L2: BM fine-tuned with authentic data and Ladin monolingual data backtranslated (BT) to Italian using N1/R1/L1 respectively.

• •

  Models N3/R3/L3: This iteration extends the training base of N2/R2/L2 by integrating Italian monolingual data that has been translated into Ladin utilising N2/R2/L2 respectively.

• •

  Models N4/R4/L4: BM fine-tuned with same training data as N3/R3/L3 models, but with Ladin and Italian monolingual data backtranslated with N3/R3/L3 model.

• •

  Models N5/R5/L5: This iteration extends the training base of N4/R4/L4 by adding also the forward-translations (FT) as training data.

• •

  Models A1/A2: A1 was trained on the combined training data used to trained N4, R4 and L4. In A2 we additionally included the forward-translations into the training data.

We refer to the models N1, …, N5 that were trained with NMT backtranslated data as N-models. Analogously, we use the term R-models and L-models to refer to RBMT and LLM models, respectively.

Models N4/R4/L4 illustrate the gains achieved through iterative back-translation Hoang et al. (2018). Additionally, models N5/R5/L5 demonstrate potential improvements achievable with synthetically generated forward-translation data.

We evaluated these models on the 3 test sets presented in Section 2.5. The results are presented and analysed in the following section.

The results of the various experiments conducted are presented in Table 3 and Table 4, where the SacreBLEU Post (2018) and chrF++ Popović (2015) scores for different models and test sets are detailed. To facilitate comparison, the best scores for each approach have been underlined, and the overall best scores for each testset are highlighted in bold.

Additionally, as recommended in Edunov et al. (2020), in Table 2 we report the mean perplexity values for the Italian translations generated by the different models to complement BLEU’s emphasis on adequacy. Perplexity measures how well a language model can predict the next word in a sequence based on the preceding words. Lower perplexity means that the model is more confident and accurate in its predictions, indicating that it can better reproduce the structure and patterns of the language it generates. Therefore, we present the mean perplexity values obtained from GPT-2  Radford et al. (2019) , computed using the implementation available from Hugging Face¹⁷¹⁷17https://huggingface.co/spaces/evaluate-metric/perplexity.

Several findings can be deduced from these results, and will be discussed below. In general, there is evidence that augmenting the training data with monolingual data through back-translation is effective. N1 shows that fine-tuning the model with only authentic training data substantially improves the results in both directions (compared to BM) in terms of BLEU/chrF++ points. This shows on the one hand that the training is effective and on the other hand that the available data is adequate. However, it is also evident that the model generates less fluent text, as indicated by the perplexity scores which increase for this model.

The results reveal a progression in difficulty among the test sets, where Testset 3 emerges as the most challenging one. On this test set, all approaches achieve similar low scores, suggesting the presented approach may face limitations with more complex texts.

In the translation direction lvb $\rightarrow$ ita, the best results were achieved by combining the different back-translations, as model A2 results indicate. This emphasises the importance of a broad and diversified dataset. Remarkably, the A2 model is also competitive in the reverse translation direction (ita to lvb), although it does not achieve the best results.

A comparison of the models N1, R1 and L1 suggests that the LLM generates more fluent texts (low perplexity) but perhaps does not always accurately reproduce the meaning, as attested by the performance on Testset 3 (low perplexity but also low BLEU score). In this assessment, the RBMT system R1 also performs better than the fine-tuned NMT model N1. One of the reasons for the high perplexity values of N1 is that this model tends to hallucinate because it has been fine-tuned with a small data set. However, this does not seem to affect the performance as the models trained on this data do not perform considerably worse.

The performance of the LLM varies significantly, with pronounced differences between the three test sets in both directions of translation. The significant difference observed between Testset 1 and Testset 2 in the translation direction from lvb $\rightarrow$ ita cannot be seen in the R- and N-models. It remains unclear to what extent the LLM benefits from the given examples in the prompt. However, by providing an example, the propensity for errors was minimised, resulting in fewer mistakes during execution. Even though LLMs are not (yet) suitable for generating texts in low-resource languages out-of-the-box (see performance of L1 in Table 4), Ladin and low-resource languages in general could benefit from this technology. Our experiments show that models trained on back-translations from L1 performed best on Testset 1 and Testset 2 in the translation direction ita $\rightarrow$ lvb.

The inclusion of forward translations in the training data did not consistently improve the models, with the exception of the R-models for lvb $\rightarrow$ ita. This suggests that these synthesised texts introduce too much noise. However, model A2 was able to benefit from this data in the translation direction lvb $\rightarrow$ ita. Filtering this data could slightly improve the model.

As the models achieve similar scores on the test data, we also examined the quality of round-trip translations to gain additional insights. For this, we used 10k sentences from the monolingual Ladin and Italian data (which were also used for training, hence the high scores), translated them into the other language and then back-translated them. This concept of so-called round-trip translation is a suitable evaluation method  Zhuo et al. (2023). We used the R4/N4/L4 models for this purpose, applying one model $A$ for one direction and the same or a different model $B$ for the opposite direction. Table 5 shows the obtained results. It can be clearly seen that the results are worse when a different model is used for the reverse translation. This shows that although the models achieve similar results with the test data, they work differently. The R4 model proves to be the most stable here, as its translations can be back-translated well by all three models. For other combinations, a high variance can be observed.

The translation models N4¹⁸¹⁸18https://doi.org/10.57967/HF/2695, R4¹⁹¹⁹19https://doi.org/10.57967/HF/2693, and L4²⁰²⁰20https://doi.org/10.57967/HF/2694 have been released on Hugging Face, making them accessible for further research and application.

Data augmentation such as back-translation Sennrich et al. (2016); Hoang et al. (2018) and transfer learning Zoph et al. (2016) are established strategies to improve MT systems. These concepts are discussed in Haddow et al. (2022); Ranathunga et al. (2023), with a focus on low-resource scenarios. The fact that the synthesised data plays a critical role in the quality of the systems trained on it, as it also introduces a certain degree of noise, was discussed extensively in Edunov et al. (2018); Xu et al. (2022). It was shown that tagging synthetic data can be beneficial in the training process Caswell et al. (2019). In our work we do not apply advanced techniques to differentiate synthetic from real translations in training. The fact that RBMT systems can still be valuable for low-resource languages and can even help to achieve better results was also demonstrated for Northern Sámi Aulamo et al. (2021) . In our experiments, we could also observe this in the translation direction lvb $\rightarrow$ ita. This could be due to the ability of the RBMT system to provide general knowledge that is not available in the relatively limited parallel training datasets Aulamo et al. (2021). The use of LLMs for MT and different prompting techniques was investigated in Zhang et al. (2023) and their performance in the machine translation of low-resource languages has already been analysed in Moslem et al. (2023). Even if they struggle to generate texts in low-resource languages Robinson et al. (2023), it has already been claimed that they can contribute to advances in machine translation of such languages. Our work is an example of how LLMs can be used in machine translation of a low-resource language; however, further prompt engineering is needed to make better use of such models.

In this work, we conducted a detailed comparison of RBMT, NMT and LLMs for back-translation in a low-resource scenario. We have tested various back-translation approaches and evaluated them for a previously unexplored language in the field of machine translation.

Our current methodology involved the exclusion of numerous Ladin monolingual sentences. However, this filtering would be less important for the translation direction lvb $\rightarrow$ ita. This previously discarded data could be re-incorporated to improve the performance of the models in this particular translation direction.

The round-trip translation scores indicate that the initial back-translation with the RBMT system leads to more robust models. Improving the ambiguity resolution of this rule-based translation system could lead to even better results.

The simplicity of the prompts used to feed the LLMs provides a further starting point for investigations. In particular, the question arises as to whether the results can be improved by further prompt engineering, e.g., by including the meaning for the distinct words occurring in a text using the available dictionary. Investigating the effects of prompt optimisation could provide new insights into maximising the efficiency of LLMs in machine translation, especially in low-resource scenarios.

We plan to address these research questions in our future work.

This research was conducted in collaboration with the Ladin Cultural Institute Micurà de Rü and funded by the Regione Autonoma Trentino-Alto Adige/Südtirol.