Aligning Neural Machine Translation Models:
Human Feedback in Training and Inference

An NMT model has learnable parameters, $\theta$, to estimate the probability distribution, $p_{\theta}(y|x)$ over a set of hypotheses $\mathcal{Y}$, conditioned on a source sentence $x$. MLE is the training principle of estimating $\theta$, given parallel data, formalized as

NMT systems typically employ maximum a posteriori (MAP) decoding to generate translations,

where algorithms such as greedy decoding or beam search [Reddy (1977] approximate the most probable translation given the source. An alternative approach is to sample translations according to $p_{\theta}(y|x)$, using techniques such as top-$k$ or nucleus sampling [Fan et al. (2018, Holtzman et al. (2020].

In §3.3 of this paper, we also consider two distinct reranking approaches [Fernandes et al. (2022], namely $N$-best reranking and MBR decoding. While $N$-best reranking selects the candidate translation that maximizes a given (reference-free) metric, MBR decoding ranks candidates using reference-based metrics, maximizing the expected utility (or minimizing the risk).

Human evaluations are the most reliable way to assess the performance of MT systems, but they are time-consuming and costly. For that reason, the standard way to evaluate MT is through automatic evaluation metrics, which can be reference-based or quality estimation (QE) metrics.

Reference-based metrics compare the generated translation to human-written reference texts. Lexical reference-based metrics, such as the widely used BLEU [Papineni et al. (2002], rely on word overlap and n-gram matching, making them ineffective for translations that have the same meaning but are substantially different from the reference. On the other hand, neural metrics, such as COMET [Rei et al. (2022a], are a recent alternative that relies on neural networks trained on human-annotated data and that leverages contextual embeddings to address semantic similarity.

QE assesses translation quality without human references, being particularly useful in dynamic, data-intensive environments, where references are costly and time-consuming to obtain. This paper focuses on sentence-level QE as a reward model, providing a single quality assessment for each translation. COMET-QE [Rei et al. (2020] is a state-of-the-art reference-free quality estimation metric derived from COMET used to evaluate MT performance.

Neural reference-based and QE metrics are valuable preference models because they offer a more accurate and contextually-aware measure of translation quality, aligning better with human preferences and judgments [Freitag et al. (2022b].

In MT, approaches based on reinforcement learning (RL; ?)) cast the problem as a Markov decision process (MDP; ?)), where a source sentence $x=(x_{1},...,x_{n})$ is translated into a target sentence $y=(y_{1},...,y_{m})$. Under this perspective, the NMT system can be viewed as the agent with a conditional probability distribution based on its parameters, $p_{\theta}(y_{t}|x,y_{<t})$. The states of the MDP are defined by the target sentence that has already been decoded, $s_{t}=(y_{1},....,y_{t<m})$, and the action corresponds to the selection of the next word, $y_{t+1}$. Based on the states and actions, all transitions are deterministic and the reward function, $R$, is provided by the MT evaluation model which returns a quality score for the generated translation $\hat{y}$. The main purpose of using RL in NMT is to provide learning signals that go beyond a single reference translation, by providing reward signals for arbitrary translations. MLE provides less robust learning signals that are more susceptible to the shortcomings of noisy references. However, it is essential to note that if the reward model used relies on reference-based metrics, some vulnerability to noisy references may still persist. Accordingly, the goal of RL training is to maximize the expected reward, $L_{\mathrm{rl}}(\theta)=\mathbb{E}_{p_{\theta}(\hat{y}|x)}[R(\hat{y})].$ Commonly used RL training procedures include REINFORCE [Williams (1992], minimum risk training [Och (2003, Shen et al. (2016], and proximal policy optimization (PPO; ?)).

The success of fine-tuning NMT models with MLE is highly dependent on the quantity and quality of the training dataset [Wang et al. (2018, Koehn and Knowles (2017, Khayrallah and Koehn (2018]. This is because accurate references are crucial for computing meaningful learning signals that correctly guide the NMT model towards improved translations [Kong et al. (2018]. Despite its recent successes, RL-based training can be unstable, so using only high-quality data could help mitigate this instability. This can be addressed via data filtering, by seeking a good balance between the aggressiveness of filtering and the resulting dataset size: if the original dataset is already small, too much filtering can be detrimental to the performance of NMT systems [Zoph et al. (2016, Jiao et al. (2020]. Furthermore, when looking at the RL scenario, having a sufficiently large training dataset can help guarantee that the NMT model explores a wide range of scenarios for policy improvement.

We apply our data filtering method on the considerably large and noisy WMT datasets [Bojar et al. (2015, Bojar et al. (2016] since they have been reported to have less relevant and uncorrelated sentences that can lead to sub-optimal results when used during training [Koehn et al. (2020, Malli and Tambouratzis (2022]. We do not perform data filtering to the IWSLT2017 [Cettolo et al. (2012, Cettolo et al. (2017] dataset due to concerns about its limited amount of available data. Further dataset filtering could potentially result in a too-small training dataset, which is not be desirable for training MT systems.

As illustrated in Figure 1, to perform the training dataset filtering, we use a filter that reranks the sentence pairs according to quality scores that indicate the correlation and relevance of each sentence and its given reference. This approach allows us to filter out low-quality sentence pairs, thereby improving the overall quality of the data. In our approach, we use a robust preference model called COMET-QE [Rei et al. (2020] as the data filter, which combines the use of encoders and a regression model trained on human-annotated data to estimate the quality score of each sentence pair. This reference-less model is expected to be more accurate in quality score estimation and have a superior alignment with human judgments than just resorting to the currently used cross-lingual encoders which only take into account vector-space mapping similarity [Bane and Zaretskaya (2021]. Furthermore, COMET-QE seems particularly suitable as our preference model during data filtering, as it is a multilingual reference-free neural-based metric trained on human annotations of translation quality, and therefore can be used to filter by thresholding on predicted quality or on the number of sentences in the training set. After scoring all sentence pairs, we select the threshold based on the number of high-quality sentence pairs to use as the filtered dataset for RL training. For that, we apply different thresholds and sizes to the reranked sentences. We, then, MLE fine-tune our baseline on these subsets and select the subset that gives the overall best-performing model on the dev. set. These best-performing models serve as baselines for our RL-based training and reranking methods during decoding.

In conclusion, it is worth noting that our data filtering method is, as shown in Figure 1, one of three methods we cover for employing a preference model in the MT pipeline. This filtering method can significantly increase the performance of MT systems by introducing feedback in an earlier stage of the pipeline.

The use of RL-based training has the potential to bridge the gap between MLE training objectives, MT evaluation metrics and human-like translations. However, it faces challenges of instability and inefficiency, especially in gradient variance and reward computation. As illustrated in Figure 1, the RL training process is composed of an NMT model that generates translations that are evaluated by the reward model through rewards that represent the quality of the translation. This reward is used by the policy gradient algorithm to update the NMT model’s policy. To address the problem of gradient variance, we employ PPO [Schulman et al. (2017] as our policy gradient algorithm since it is a stable and efficient algorithm that updates the policy parameters in a controlled way with a predetermined proximity bound, avoiding sudden changes that might destabilize the learning.

Reward computation is the most crucial part of this entire process as it guides the NMT model during training. Previous work on RL-based NMT systems predominantly used BLEU as the reward function. However, BLEU has several limitations, as discussed in §2.2. To address these shortcomings, we leverage robust preference models during RL training, such as the reference-based COMET [Rei et al. (2022a] and the reference-free COMET-QE [Rei et al. (2020], as highlighted in Figure 1. Since learning these models is a complex task, we incorporate these pre-trained preference models, which have already been shown to correlate well with human judgments [Freitag et al. (2022b, Rei et al. (2022a, Rei et al. (2020], to ensure that RL systems can better capture the nuanced preferences of the user by receiving human-like feedback as rewards. These models assign numerical quality scores to each translation hypothesis based on their desirability, making them similar to utility functions. Our study aims to demonstrate that training with RL can generate higher-quality NMT models using neural metrics and investigate the competitiveness of COMET-QE as a reward model.

Another crucial decision was related to the exploitation vs. exploration problem of RL in the context of MT [Wu et al. (2018]. The beam search algorithm generates more accurate translations by exploiting the probability distribution/policy of the NMT model, while sampling aims to explore more diverse candidates. During generation, we observed that sampling techniques generally led to candidates of lower quality when compared to beam search, according to the preference models used. Therefore, all RL-based models used beam search during their training and inference.

Reranking methods [Ng et al. (2019, Bhattacharyya et al. (2021, Fernandes et al. (2022, Eikema and Aziz (2022] are an alternative to MAP-based decoding that relies on reranking techniques and presupposes access to $N$ candidate translations for each source sentence, generated by the NMT system through methods like beam search or sampling. The generated candidates are reranked according to their quality given an already determined metric/reward model.

We employ two reranking methods to select a final translation: $N$-best reranking [Ng et al. (2019, Bhattacharyya et al. (2021] and minimum Bayes risk decoding (MBR; ?)).

$N$-best reranking (3) employs a reference-free metric, $M_{\textsc{qe}}$, to reorder a set of $N$ candidate translations, denoted as $\bar{\mathcal{Y}}$, and selects the candidate with the highest estimated quality score as the final translation, $\hat{y}_{\textsc{rr}}$,

Considering the previous equation, and assuming $C_{M_{\mathrm{QE}}}$ as the computational cost of evaluating a candidate translation with QE metric, $M_{\mathrm{QE}}$, we obtain the final computational cost of finding the best translation from $N$ candidate translations as $O(N\times C_{M_{\mathrm{QE}}})$.

MBR decoding, in contrast, relies on a reference-based metric and chooses the candidate that has the highest quality when compared to other possible translations (in expectation). We define $u(y^{*},y)$ as the utility function, quantifying the similarity between a hypothesis $y\in\mathcal{Y}$ and a reference $y^{*}\in\bar{\mathcal{Y}}$. In our context, the utility function is represented by either BLEU or COMET. Therefore, MBR decoding can be mathematically expressed as

where in Eq. 4 the expectation is approximated as a Monte Carlo sum using model samples $y^{(1)},\ldots,y^{(N)}\sim p_{\theta}(y|x)$. These samples may be obtained through biased sampling (e.g., nucleus-p or top-k) or beam search. Knowing that the utility function is a reference-based metric $M_{\mathrm{REF}}$ with computational cost, $C_{M_{\mathrm{REF}}}$, and that to find the best translation we need to do pairwise comparisons between hypotheses, we obtain the final computational cost as $O(N^{2}\times C_{M_{\mathrm{REF}}})$. These reranking methods become particularly effective when $N$ is not excessively large, making the process computationally more manageable.

Preference models capture the preferences of human evaluators and can be used during the decoding stage to influence MT systems, as shown in Figure 1. By doing this, the MT system will prioritize translations that are more aligned with human judgments, therefore reducing the chances of generating severely incorrect translations. We believe that incorporating preference models during the decoding stage can lead to even better translation quality, even if the underlying model has already been RL-trained using the same or a different preference model. The benefits we expect to see include improved fluency, adequacy, and consistency compared to the respective baselines since our preference models have been trained on annotations that aim to optimize these linguistic aspects.

During the training phase, we investigate the advantages of RL training (with and without data filtering §3.1) for enhancing the performance of NMT systems. We employ a T5 model¹¹1We leverage the T5-Large model available in Huggingface’s Transformers framework [Wolf et al. (2020]., pre-trained on the C4 dataset [Raffel et al. (2019]. First, we fine-tune the models using MLE training with Adam [Kingma and Ba (2017] as the optimization algorithm, learning rate decay starting from $5\times 10^{-6}$ and early stopping. For RL training²²2Our RL implementation relies on the Transformer Reinforcement Learning X framework [Castricato et al. (2023, trlX]., we use PPO with learning rate set as $2\times 10^{-5}$, $\gamma$ set as $0.99$, trajectory limit set as $10,000$, beam search size set as $5$ and mini-batch updates were conducted using stochastic gradient descent with a batch size of $32$, gathered over $4$ PPO epochs. In the inference phase, our emphasis shifts towards reranking techniques and their impact on the performance of NMT systems. As for the candidate generation method used, early experiments, omitted for relevancy, show that the best configuration is to generate 100 candidates per source sentence and then use sampling with $p=0.6$ and $k=300$ to select the best translation. Consequently, the evaluation encompasses all the baseline and RL-trained models, both with and without $N$-best reranking and MBR decoding. These evaluations are conducted across the following datasets:

• •

  The small IWSLT2017 datasets [Cettolo et al. (2012, Cettolo et al. (2017] for English to German (EN → DE) and English to French (EN → FR), featuring 215k and 242k training examples, respectively.

• •

  The large WMT16 dataset [Bojar et al. (2016] for English to German (EN → DE) with 4.5M training examples.

• •

  The large WMT15 dataset [Bojar et al. (2015] for English to French (EN → FR) with over 40M training samples.

We assess the performance of each NMT system using well-established evaluation metrics, which include BLEU, chrF [Popović (2015], METEOR [Banerjee and Lavie (2005], COMET, COMET-QE, and BLEURT. Additionally, for certain experiments executed on a single NVIDIA RTX A6000 GPU, we provide wall clock time measurements to offer insights into computational efficiency.

In this section, we discuss our approach to quality-aware data filtering as a stabilizing strategy (§3.1), for the WMT datasets. Figure 2(a) summarizes our findings for the WMT16 EN→DE dataset [Bojar et al. (2016] on the influence of a high-quality subset on translation performance as we vary the subset size, based on various evaluation metrics and COMET-QE sentence filtering. Across all metrics, a consistent trend emerges: after reaching training sizes of $\numprint{500000}$, there is a notable decline in performance. Particularly, this decline is less prominent for lexical metrics, possibly due to their inherent limitations [Freitag et al. (2022b]. A similar analysis for WMT15 EN→FR that can be found in Figure 2(b) results in an optimal training size of $\numprint{300000}$ examples.

While the data filtering process has led to remarkable improvements in performance, it is important to note that the effectiveness of this process is dependent on the selected reranking metric. Using metrics that are not closely aligned with human judgments can result in poorly correlated and misaligned sentences, which can make the training process more unstable. Therefore, it is recommended to use robust QE models, such as COMET-QE. The more recent COMETKIWI [Rei et al. (2022b] model may offer even greater performance improvements.

After obtaining the best configuration for our data filtering process, we experiment with the use of the curated high-quality training subset from COMET-QE and assess its impact on the MLE and RL training performance. We compare our filtering method with no filtering by using the original full training dataset, random filtering and cross-lingual embedding similarity filtering using MUSE [Lample et al. (2017] and XLM-R [Conneau et al. (2019].

Table 1 provides a comprehensive overview of the experimental results using BLEU, COMET and COMET-QE as reward models. Both MT tasks demonstrate the same tendency when trained using MLE. COMET-QE and MUSE high-quality subsets have enough reduced noise to provide more stable training, as evidenced by the overall increase in performance across all metrics compared to the baseline training on the full original dataset. Moreover, a randomly selected subset fine-tuned with MLE performs worse or at most not significantly better than the baseline trained on the original dataset, as expected. Furthermore, in accordance with our expectations [Bane and Zaretskaya (2021], XLM-R filtering does not improve training and is actually the worst-performing model.

Regarding RL-based training on both MT tasks, we observe that most RL-trained models outperform their MLE-trained baseline counterparts across various metrics. Notably, the best-performing models are the ones that were MLE fine-tuned and then RL-trained on the COMET-QE high-quality subset using both COMET and COMET-QE as reward models. On top of that, we can see that in some cases RL training solely does not yield significant improvements, but when combined with high-quality training subsets, it results in substantial enhancements and a competitive edge over the normal, random and XLM-R baselines. Additionally, we see impressive BLEU scores with RL training with COMET(-QE) as reward model. This finding underscores that optimizing for COMET(-QE) yields superior BLEU scores compared to direct optimization for BLEU. This phenomenon is likely attributed to COMET(-QE) providing more effective reward signals during training, thus highlighting the limitations of BLEU.

The excellent performance gains with COMET-QE as a data filter and also as a reward model emphasize the potential of RL-based NMT models trained with a QE reward model (which does not require a corpus with references) to outperform other RL-trained models, offering promising opportunities for unsupervised NMT training with monolingual data, especially for low-resource languages, by eliminating the need for reference translations in evaluation and reward signal generation.

In conclusion, we highlight the importance of thoughtful data selection for achieving better translation quality, showing that COMET-QE can consistently outperform the remaining filtering methods. Furthermore, the top-performing models were RL-trained with neural metrics, showing once again that human-aligned preference models can constantly outperform simpler metrics, such as BLEU.