Translate-and-Revise: Boosting Large Language Models
for Constrained Translation

Let $X=\{x_{1},x_{2},...,x_{n}\}$ be the source-language sentence with length $n$, and $Y=\{y_{1},y_{2},...,y_{m}\}$ be the target-language sentence with length $m$. The translation procedure can be written as:

where $\mathrm{Trans}(\cdot)$ symbolizes the translation model (either an NMT model or an LLM), and $f(\cdot)$ denotes a template by which we process $X$ to make it suitable as the input of $\mathrm{Trans}(\cdot)$.

Let the $\langle S,T\rangle=\{\langle s_{1},t_{1}\rangle,\langle s_{2},t_{2}\rangle,...,\langle s_{k},t_{k}\rangle\}$ represents the bilingual constraints with $k$ pairs in total. Constrained translation needs the system to accurately translate each source constraint $s_{i}$ to its corresponding target constraint $t_{i}$. This process can be represented by the following equation:

Since conventional translation instructions never impose constraints on LLMs, they frequently fall short of satisfying constraints. In this work, we propose to integrate these constraints directly into the prompts and employ an instruction based on natural language specifically tailored for constrained translation tasks. Our template $f(\cdot)$ is shown in Figure 2 which can effectively turn LLMs into constraint-aware translators.

However, the LLM-based translation cannot always cover all original constraints. We randomly sampled 20 incorrect translation results and observed that, in datasets like WMT21 Terminology Translation [2021], to $95\%(19/20)$ of the cases, the tokens generated by the model were similar to the expected constraints meaning and exhibited high confidence levels. The confidence level of LLMs in generating these tokens remained virtually unchanged, whether or not constraints were included in the instructions, revealing overconfidence in generation while overlooking the constraints.

We notice a strong connection between our real use case and ‘memo trap’ [2023] as unsatisfied constraints often pertain to non-mainstream translations resulting terms used with lower frequency and the incorrect translations usually refer to the mainstream translations. Compared ‘memo trap’, we show that the phenomenon extends to toy settings and is prominent even for real applications and for SOTA models like GPT-3.5.

To overcome these challenges, we initially employ a rule-based method to identify which constraints are not completed. Subsequently, these uncompleted constraints ${\langle S,T\rangle}^{un}$, along with the source language input $X$, flawed translations output $Y^{flawed}$, and all other given constraints $\langle S,T\rangle$, are passed to the LLM. At this juncture, the LLM assumes the role of a reviewer, tasked with revising flawed translation upon receipt of uncompleted constraints. The aforementioned process is defined by the following formula:

where $\mathrm{Revise}(\cdot)$ symbolizes the reviser. Furthermore, TAR can continuously iterate in the loop of detecting uncompleted constraints and making revisions until a stopping condition is met. This condition is either the iteration $i$ reaches a specified count or the translation satisfies all constraints. We represent this iterative process as follows:

where the translation $Y_{i}$ from the previous iteration, combined with its uncompleted constraints ${\langle S,T\rangle}^{un}_{i}$ and other inputs, is sent into the reviser to produce a more precise translation $Y_{i+1}$. By highlighting uncompleted translation constraints and comparing flawed translation results, human translators are able to satisfy these constraints and optimize translation output. Empirically, we find that LLMs can revise the translation results similarly to human translators, while being more efficient and cost-effective. Additionally, we discuss the time and financial costs of multiple iterations in Appendix A.

Datasets Detailed information about the datasets we used can be found in Table 1. Lexical constraints refer to sentences with predefined word or phrase constraints sourced from existing databases. Structural constraints, in contrast, encompass inline markup tag constraints like XML tags, for example, <ph> and </ph>. Here are more details about the datasets that we use in this work:

General Lexically Constrained Translation: This is based on a dataset⁰⁰0https://github.com/mtresearcher/terminology_dataset provided by [2019]. The dataset is derived from newstest2017 En $\to$ De. Lexical constraints are extracted with guidance from two general-domain term databases: IATE and Wiktionary¹¹1Available at https://iate.europa.eu/home and https://www.wiktionary.org/..

Terminology Translation: To benchmark against SoTA NMT systems, we employ the official test set from the terminology translation task in WMT21²²2https://www.statmt.org/wmt21/terminology-task.html(WMT21 TT) [2021].

Entity Translation: We also endeavor to evaluate our method using the extensive Entity Translation Corpus (ETC) [2023], which comprises six test sets from the WMT News Translation Task spanning 2015-2021. For alignment, we employ spaCy NER models³³3https://pypi.org/project/spacy/ to extract source entities and use awesome-align [2021] for their correspondence.

Structured Document Translation: Following recent works [2022a], we conduct experiments on the LXM dataset⁴⁴4https://github.com/salesforce/localization-xml-mt [2019], in which XML tags are hierarchically distributed throughout the source and target text.

Evaluation Metrics Consistent with previous studies [2019, 2022, 2022a, 2023], we employ BLEU [2002] and the constraints completion rate (CCR) to assess translation quality and constraint-based translation accuracy except for structured document translation. For entity translation, we also incorporate the COMET score⁵⁵5wmt22-comet-da [2020] for comparison with the work of Zeng [2023]. In structured document translation, we utilize sacreBLEU [2018] to compute the XML-based BLEU score⁶⁶6XML tags are treated as an integral part of the sentences during BLEU score calculation.. Additionally, we measure structured constraint-based translation accuracy using the structure accuracy rate (SAR) and structure match rate (SMR). Here, SAR evaluates the compatibility of translation results with XML parsers, while SMR ensures the translated XML structure aligns with the reference. Both metrics are assessed using lxml⁷⁷7https://lxml.de/.

Baselines To ensure thorough evaluation, apart from comparing TAR with the LLM baseline without revision, we also we compare TAR with representative methods across each of the four tasks. For general lexically constrained translation, our baselines include the vanilla Transformer [2017], Const.Dec. [2018], Code-switching [2019], Append [2019], and Robust Terminology Translation (RTT) [2023b]. For the terminology translation task, our baselines are derived from the top three submissions of WMT21. They include HW-TSC [2021b], Term-Mind-sys2 [2021a], ProMT.soft [2021], TildeMT [2021a], and Lingua Custodia [2021]. For entity translation, we utilize the vanilla Transformer, Code-switching, Placeholder [2019], and Extract and Attend [2023]. Structured document translation baselines consist of the vanilla Transformer, Split-Inject [1997], and Template [2022a] methods.

Model Configurations We initially investigate the potential of LLMs to act as both the translator and reviser within TAR. Our primary choice for LLMs is gpt-3.5-turbo-0613⁸⁸8https://openai.com/, chosen for its exceptional translation capabilities and proficiency in adhering to instructions. Additionally, we assess its revising process for NMT models in Section 4.1 and explore whether TAR provides consistent improvements across different LLMs in Section 4.2. The decoding parameters for these models remain at their default settings, except for the sampling temperature, which is set to 0. We employ natural language-based prompts in a one-shot manner, merging uncompleted constraints with the source language, flawed translation results, and original constraints to form the reviser’s input. These prompts are depicted in Figure 2.

Detection of Uncompleted Constraints To identify unmet constraints in translations, we employ a rule-based procedure that leverages scripts designed for calculating CCR. This procedure assesses how well the translation adheres to the constraints. We further explore the capacity of LLMs to autonomously verify constraint completion and offer detailed feedback to the reviser in Section 3.3.

Table 3, Table 3, Table 4, and Table 5 detail the performance of TAR on general lexically constrained translation, terminology translation, entity translation, and structured document translation, respectively. Here are our main results.

Comparison with base LLMs TAR consistently boosts the performance of LLMs in constrained translation. Two primary factors contribute to this improvement:

(1) Our natural language-based prompts, as opposed to the conventional few-shot translation prompts [2023], are more effective for constrained translation. Specifically, in terminology translation (refer to Table 3), our prompts lead to an average BLEU score increase of 0.3 and a CCR rise of 7.7%.

We noticed significant gains over base LLMs in entity translation. Across four language directions, there is a consistent uplift in BLEU scores, averaging an increase of 0.9. Notably, the CCR experiences increases of 19.4%, 8.9%, 24.3%, and 19.9% respectively. This marked improvement can primarily be attributed to the superior instruction-following capabilities of LLMs. By incorporating constraints in instructions, we can guide the model more effectively to address these constraints, alleviating the issue of LLMs struggling to correctly translate named entities.

(2) Revision effectively improves constraint-based translation accuracy across all datasets without sacrificing translation quality, addressing the tendency of LLM-based translations to overlook constraints.

As illustrated in Table 6, we observe that the LLM exhibits overconfidence in its prediction overriding the influence of constraints, especially when the constraint suggests an uncommon translation for a polysemous word. For instance, the LLM translates “COVID-19” to the more commonly used “{CJK*}UTF8gbsn新冠” instead of adhering to the target constraint “{CJK*}UTF8gbsn新型冠状病毒”. We speculate this overconfidence in LLMs stems from their greater exposure to “{CJK*}UTF8gbsn新冠” compared to “{CJK*}UTF8gbsn新型冠状病毒” during the pre-training phase, which may lead them to be overly loyal to certain patterns, thereby preventing them from meeting certain constraints.

However, the revision step can effectively cut down the possibility of missing constraints by explicitly prompting LLMs with which constraints are not satisfied. Experimental results indicate that our revision strategy led to average improvements in CCR by 2.2% for lexically constrained translation, 2.5% for terminology translation, and 1.5% for entity translation across various language directions. While the improvements in SCR and SMR for structured document translation might not seem prominent, it’s primarily because the initial translation is already at a 100% performance.

Comparison with supervised methods The SoTA methods for these four constrained translation datasets predominantly rely on pseudo-data augmentation. Through our experimental results, we observe that these methods nearly achieve perfection on the IATE, Wiktionary, and LXM datasets (as evidenced in Table 3 and Table 5). We contend that the test sets for these datasets might be relatively straightforward, and the constraints they encompass are frequently encountered in training sets. Therefore, they may not accurately reflect real-world applications where constraints might span multiple domains and are infrequently seen in the training data. However, when assessed on the ETC (as depicted in Table 4), which comprises test data spanning from 2015 to 2021, showcasing a rich diversity in constraint domains, the efficacy of traditional data augmentation methods seems to be poor. In contrast, TAR’s performance remains stable, demonstrating comparable constraint-based translation accuracy on ETC as with other datasets. This highlights TAR’s proficiency in handling the diverse constraint requirements found in real-world situations.

The reviser receives inputs including the source language sentence, translation results, given constraints, and the uncompleted constraints. To evaluate the significance of each component, we conducted experiments wherein we omitted specific elements from the input. The variations include: 1) Excluding uncompleted constraints; 2) Excluding original constraints; and 3) Only indicating to the model that the translation is flawed without specifying the uncompleted constraints. All other settings remain unchanged. The comparative outcomes on IATE and Wiktionary are presented in Table 7.

From our observations, the CCR scores of the variants show a decline compared to the default input of the TAR reviser on both datasets. Interestingly, the omission of the original constraints has a more pronounced impact on CCR. This could be attributed to the negative modification of completed constraints made by the reviser when it is inaccessible to the given constraints. Additionally, when both elements are excluded, there’s no noticeable difference in the CCR before and after the revision process.

Furthermore, as shown by ‘+ Detected by LLM’ in Table 7, using LLMs to detect uncompleted constraints and then feeding them back to the reviser may degrade translation performance. Further analysis of the results reveals challenges faced by LLMs inaccurately identifying unsatisfied constraints; they often mistakenly believe certain constraints have been met. Such inaccurate feedback not only fails to enhance the quality of the translation but might even deteriorate it. These insights emphasize the criticality of supplying the reviser with exact and thorough constraint information.

In our study, we initially depended on constraint-aware translators to produce preliminary translation results. However, in real-world scenarios, industry practitioners often possess powerful domain-agnostic NMT models. These models, due to their lack of training with specific constraints, frequently fall short in constrained translation tasks. In this section, we integrate TAR into these general-purpose NMT models. By iteratively optimizing the NMT translation results through TAR, we can significantly enhance the CCR of the translation while ensuring its quality.

Specifically, we first the WMT21 champion model [2021] to obtain a preliminary translation result. Since this model is not specifically trained for constraints, the initial translation often exhibits a suboptimal CCR. Building on this, we apply TAR to revise this outcome, iteratively optimizing to form the final translation result.

The experimental results on WMT21 TT and ETC datasets are presented in Table 8, we can see that TAR bolsters both BLEU and CCR scores of NMT models. On average, we observed an uplift of 0.7 and 0.8 in BLEU scores, coupled with impressive gains of 10.7% and 14.3% in CCR across the two datasets and language pairs. Although there remains a gap in constraint-based translation accuracy compared to the standard TAR, it generally exhibits superior translation quality. This insight demonstrates that when equipped with TAR, even domain-agnostic NMT models can adeptly tackle constrained translation. This eliminates the need for forced decoding algorithms or additional training, greatly enhancing their usability in constrained translation applications.

To evaluate the scalability and robustness of TAR across different models, we applied it to a variety of LLMs, including commercial models like GPT-3 and GPT-4, as well as the open-source Qwen⁹⁹9Qwen-14B-Chat [2023]. We maintained consistency in all other settings. Experiments were conducted in the ”En-Zh” and ”En-Ru” language directions for terminology translation. As shown in Figure 3, TAR consistently improves the performance of various LLMs in constrained translation tasks. To be specific, the average CCR for GPT-3, GPT-4, and Qwen increases by 20%, 11.7%, and 14.4%, respectively. Comparing the results of ChatGPT with those of GPT-3 and GPT-4, it’s evident that TAR enables more powerful models to fully harness their capabilities in constrained translation. Intriguingly, although the CCR score of GPT-3 in the initial translation substantially trails that of ChatGPT, it surpasses ChatGPT post-revision. While the performance of Qwen lags slightly behind ChatGPT, the improvement brought by TAR is still notable.

The reviser, in its function, takes the translation result and uncompleted constraints as input. Naturally, one might consider iteratively revising the output multiple times. The question arises: how many iterations strike the optimal balance? Here, we employed constraint-aware LLMs and NMT on the “En-Zh” and “En-Ru” directions of the terminology translation dataset. We assessed performance across different iterative rounds of the revision module, consistently using the same prompts for each iterative phase.

As presented in Figure 4, there is a significant leap in performance predominantly during the initial revision. Although performance does enhance with increasing iterations, the rate of improvement starts to taper off, indicating diminishing returns.

Furthermore, we also investigated whether the LLM reviser can benefit from varying prompts in multiple iterations. To experiment this, after designing several revision templates, we randomly select one for each iterative round. Thus, a complete multi-round revision process can utilize various templates. This design is in part similar to prompt ensemble methods [2023a, 2023], combining benefits of various prompts, akin to ensemble learning [2020]. Experimental results are shown in Figure 4. Compared to applying a single template in revision iterations, utilizing diverse templates achieves superior performance.

Due to space limitations, we provide a more detailed analysis of our method in the appendix, including the additional costs incurred by TAR, the impact on performance as the number of constraints increases, reasons for potential declines in BLEU scores during the revision phase for certain datasets, and the performance of traditional constrained translation data augmentation methods on LLMs.

Machine translation has made considerable progress in incorporating pre-specified constraint, which can be categorized into hard constrained translation and soft constrained translation.

Hard constrained translation: This line of research expanded the original search space via decoding algorithm modification to strictly incorporate constraints [2017, 2018, 2019]. However, while these methods achieve a high constraint-based translation accuracy, they tend to be computationally expensive and can sometimes compromise translation quality [2018, 2021].

Soft constrained translation: Here, research primarily centers on data augmentation strategies to train NMT models to integrate constraints. Several techniques have been proposed, including: replacing source language constraints with special token [2016, 2017, 2023b]; substituting source language constraints with target language constraints [2019, 2019]; using inline annotations to individually mark source and target language constraints [2022, 2021b]; and  [2022a] employing a template to transform the constrained translation into constraint reordering. There are also studies that modified the model architecture to better integrate vectorized constraints representation [2020, 2022b], alignment information [2021a]. These approaches heavily rely on data quality or necessitate structural modifications, limiting their practicality. Moreover, they often falter when addressing diverse real-world requirements. In contrast, TAR doesn’t require training on constraint-specific data and is adept at handling varied constraint scenarios.

Several studies have aimed to develop neural-based models for automatic post-editing (APE) in translation [2018, 2019, 2020]. Chatterjee[2019] investigated the application of deep learning techniques for APE and introduced novel architectures to improve the quality of post-edited translations. Góis[2020] examined the application of automated ordering methods to improve translations. Voita[2019] introduced a context-aware approach to APE, integrating source context information into the neural framework to produce improved post-edits. Chollampatt[2020] investigated the application of APE in enhancing the translation performance of NMT models. These methods primarily focus on enhancing the overall translation quality. However, it’s crucial to understand that not all words within a sentence carry equal importance. The precise translation of terminologies and entities significantly impacts user experience. Our proposed TAR specifically addresses the challenge of ensuring more accurate translations for these constraints.