Fixed and Adaptive Simultaneous Machine Translation Strategies Using Adapters

Adapters Rebuffi et al. (2017); Houlsby et al. (2019) are typically small modules that are used in order to efficiently adapt a pre-trained model to a downstream task, where the pre-trained model can be either frozen Houlsby et al. (2019), or trained jointly with the adapters Stickland and Murray (2019).

Adapters have been used for efficient multi-task fine-tuning Stickland and Murray (2019), where each set of adapters is trained on a specific task. Pfeiffer et al. (2021) added AdapterFusion on top of the adapters as a way to compose the representations of different tasks. Pfeiffer et al. (2022) used adapters as language-specific parameters in order to address the curse of multilinguality in multilingual pre-training, where the adapter modules are introduced during pre-training instead of post-hoc.

For Neural Machine Translation (NMT), Bapna and Firat (2019) introduced a simple formulation of adapters to learn language-pair specific parameters, where they showed that it improves performance on high resource languages in Multilingual Translation. Chronopoulou et al. (2023) trained language-family adapters to address negative interference while allowing for parameter sharing between similar languages, which improved performance on low resource languages. Zhao and Calapodescu (2022) fine-tuned adapters on multimodal noise, then added a fusion layer in order to improve generalization to other types of noise. Adapters were also explored for other motivations like Zero-shot NMT and unsupervised domain adaptation Philip et al. (2020); Malik et al. (2023).

SiMT systems can be divided into fixed and adaptive policies. Fixed policies rely on predefined rules for READ/WRITE decisions. Ma et al. (2019) proposed the wait-$k$ policy, where the model starts by reading $k$ tokens then alternates between reading and writing one token. Elbayad et al. (2020) introduced multi-path training, where one model is trained to support multiple wait-$k$ values by sampling $k$ randomly during training. Zhang and Feng (2021) addressed interference in multi-path training by using Mixture-of-Experts. Zhang et al. (2021) used Knowledge Distillation from a Full-Sentence Transformer to embed future information into the SiMT model. For adaptive policies, Gu et al. (2017) trained a Reinforcement Learning agent to decide READ/WRITE actions, where the reward function is designed to consider both quality and latency. Zheng et al. (2019) generated supervised READ/WRITE actions then trained a classification model to predict the action based on encoder and decoder representations. Zheng et al. (2020) introduced a heuristic strategy to compose wait-$k$ models into an adaptive policy based on their uncertainty. Zhang and Zhang (2020) trained a sentence segmentation model to predict complete sentences and feed them through a full-sentence translation model. Arivazhagan et al. (2019) introduced MILK, where they modified the attention mechanism to learn a Bernoulli variable to decide READ/WRITE actions. Ma et al. (2020) adapted MILK to the transformer architecture. Zhang and Feng (2022) proposed ITST, which quantifies the transported information from source to target then generates a token when the quantity is deemed sufficient. Zhao et al. (2023) trained a supervised policy network based on automatically generated divergence between the predicted distribution of partial and full sentence input.

The majority of the techniques outlined require training multiple models to accommodate different latency levels. Our approach focuses on the efficient training of a single model that can support various latency levels at inference time.

Adapters are lightweight modules that can be inserted into a model for the purpose of task or domain adaptation Houlsby et al. (2019); Bapna and Firat (2019). They offer an efficient solution for fine-tuning the model and limiting catastrophic forgetting Houlsby et al. (2019).

Formally, for a set of $N$ tasks and a model $M$, the adapter parameters $A$ are introduced. We assume that for each task we have a dataset $D_{n}$. The model parameters can be frozen or jointly trained with the adapters. For a frozen model, the model $M$ is pre-trained and the objective function for task $n\in\{1,...,N\}$ can be defined as:

The parameters $A_{n}$ are randomly initialized for each task, then they are trained on the dataset $D_{n}$ in order to minimize the loss function $L_{n}$. This results in $N$ adapters that can specialize the model representations to each task $n$.

In the case of jointly training the model and the adapters, the model parameters $M$ can be randomly initialized or frozen. The objective function can be defined as:

where $M^{\prime}$ is both the parameters of the model $M$ and the adapters $A_{n}$ for $n\in\{1,...,N\}$. The parameters $A_{n}$ are activated during training depending on the task $n$.

The wait-$k$ policy Ma et al. (2019) trains a model to start translating after receiving $k$ source tokens. The model then alternates between writing and reading a new token. It is a fixed policy, where the $k$ value has to be chosen during training and inference. The model reads $g_{k}(t)$ number of source tokens from the source sentence $x=(x_{1},...,x_{m})$ when generating the target token $y_{t}$, where $g_{k}(t)$ is defined as:

Instead of training the model for a specific wait-$k$ value, Elbayad et al. (2020) introduced the multi-path training, which samples $k$ uniformly from $[1,...,|x|]$ for each batch during training. This enables the model to support multiple wait-$k$ values and allows for information sharing between different wait-$k$ paths. While it was shown that the multi-path training improves the results over the wait-k policy, it does not offer a solution to balance between parameter sharing and interference that we aim at solving by introducing adapters.

Multi-path training is highly advantageous as an efficient alternative to the wait-$k$ policy, where we need to train multiple models to support more than one latency at inference, but might introduce interference between wait-$k$ paths due to parameter sharing. In order to provide the ability to balance between parameter sharing and interference, we introduce adapters into each decoder layer and we activate adapters according to the wait-$k$ paths they are meant to support. Figure 1 shows an illustration of this. During training, the wait-$k$ value for each batch is sampled uniformly from $[1,...,|x|]$ following the multi-path training Elbayad et al. (2020) and based on that, the model decides which adapter will be activated. We set the adapter lagging $K_{A}$ as a list of equally spaced positive integers in increasing order, where each integer specifies the minimum wait-$k$ value supported by each adapter. We insert one adapter for each value in $K_{A}$. Since the train wait-$k$ is randomly sampled from $[1,…,|x|]$, we train each adapter on values starting from its minimum wait-$k$ up until the minimum wait-$k$ of the next adapter. For example, we can set $K_{A}=\{1,5,9,13\}$ and this will indicate adding 4 adapters, where each adapter will handle 4 wait-$k$ values (starting from each integer in $K_{A}$ until the next), except the fourth adapter ($k_{A}$ = 13), which will handle values starting from 13 up until the length of the input sequence $|x|$. We follow Bapna and Firat (2019) implementation and insert the residual adapter modules after the feed-forward layer. Algorithm 1 shows the pseudo-code for computing the decoder hidden states at decoding step $t$ using Adapters Wait-k, where $H^{0}$ is considered to be the input embeddings of the decoder, and $g_{k}(t)$ is computed based on equation 3.

We follow Zheng et al. (2020) to build an adaptive strategy by using adapters instead of different models for each wait-$k$ value, which can be computationally expensive and less efficient. At each decoding step, we activate one adapter based on the lagging behind the current generation step, which is calculated as $k=|x|-|y|$, where $|x|$ is the number of input tokens and $|y|$ is the number of generated tokens. At the beginning of generation, $|x|=1$ and $|y|=0$, which means $k$ starts from $1$. Then, we rely on the probability of the most likely token to decide whether to write or read a new token. If the probability is less than a threshold $\rho_{k}$, we read a new token, otherwise, we write. The possible values of $k$ are between $k_{min}$ and $k_{max}$ that we determine during inference. If $k$ is lower than $k_{min}$, we force the model to read, if it is higher or equal to $k_{max}$, we force the model to write, which means that the choice of $k_{min}$ and $k_{max}$ also impacts the trade-off between latency and quality (as we analyze in Section 6.1). When the whole input sequence is consumed (i.e. $x_{|x|}=\text{</s>}$), we set $k$ to $k_{max}$ and generate the rest of the target sequence. Algorithm 2 shows the pseudo-code of this method using adapters.

We evaluate our method on two public datasets: the En-Vi dataset for Transformer-Small and De-En for both Transformer-Base and Transformer-Big.

IWSLT15²²2nlp.stanford.edu/projects/nmt/ Englisch $\rightarrow$ Vietnamese (133K pairs) Cettolo et al. (2015). We follow the settings of Raffel et al. (2017) and Ma et al. (2020). We use TED tst2012 (1553 pairs) as the validation set and TED tst2013 (1268 pairs) as the test set. We replace tokens with frequency less than 5 with $<unk>$. The final vocabulary sizes are 17K and 7.7K for English and Vietnamese respectively.

WMT15³³3www.statmt.org/wmt15/ German $\rightarrow$ Englisch (4.5M pairs) We follow the settings of Ma et al. (2019). We use newstest2013 (3000 pairs) as the validation set and newstest2015 (2169 pairs) as the test set. We apply BPE Sennrich et al. (2016) with 32K merge operations jointly on the source and target to construct a shared vocabulary.

We conduct experiments on the following systems:

Full Sentence: Vaswani et al. (2017) Standard Transformer model that takes the full sentence as input before starting to translate.

Wait-$k$: Ma et al. (2019) A simple policy that waits for $k$ source tokens before starting to alternate between writing a target token and reading a source token.

Multi-path Wait-$k$: Elbayad et al. (2020) Trains a model to support multiple wait-$k$ policies by randomly sampling $k$ during training, then the $k$ value is fixed during inference.

Adaptive Wait-$k$: Zheng et al. (2020) It is a method for composing multiple wait-$k$ models during inference in order to build an adaptive strategy. The model is selected based on the lagging behind the generation step, and the decision to write or read is based on the output probabilities.

MoE Wait-$k$: Zhang and Feng (2021) Mixture-of-Experts Wait-$k$ is similar to Multipath Wait-$k$ but applies experts to learn different wait-$k$ policies to avoid interference.

MMA: Ma et al. (2020) Monotonic multi-head attention (MMA) jointly learns a Bernoulli variable that is used to decide READ/WRITE action.

Adapters Wait-$k$: Our method as described in Section 4.1.

Adaptive Adapters: Our method as described in Section 4.2.

All implementations are based on the original Transformer architecture Vaswani et al. (2017) and are using the Fairseq library Ott et al. (2019). We apply Transformer-Small (4 heads) for En-Vi and both Transformer-Base (8 heads) and Transformer-Big (16 heads) for De-En. The encoder is made unidirectional to avoid encoding the source input each time a new token is added.

The evaluation is performed using BLEU Papineni et al. (2002) for translation quality and Average Lagging (AL)⁴⁴4github.com/SimulTrans-demo/STACL Ma et al. (2019) for latency. AL measures by how many tokens the system is lagging behind an ideal policy (a wait-$k$ policy with $k=0$). Given $g(t)$, AL is computed as:

where $x$ and $y$ are the source and target sentences respectively, while $\tau_{g}(|x|)=\min\{t\,|\,g(t)=|x|\}$ is the decoding step where the source sentence finishes.

We set the adapter lagging to $K_{A}=\{1,3,5,7,9,11,13,15\}$ for our experiments, which means that 8 adapters are inserted into the model and we specify the adapter bottleneck size as 64. In Table 1, we report the number of parameters of each method and the number of models required to achieve the latency levels reported in the results section. Adapters Wait-$k$ policy introduces 79.94M parameters into Transformer-Big, but still has the advantage of using one model to support multiple latency levels. In Section 6.3, we experiment with other settings of $K_{A}$ in order to shed light on how much sharing is best between wait-$k$ values during the multi-path training.

The adaptive strategy requires three parameters to be specified at inference, namely, $k_{min},k_{max}$, and the probability threshold $\rho_{k}$. For En-Vi experiments, $k_{min}$ and $k_{max}$ are set to $1$ and $9$ respectively, while for De-En, we lower $k_{max}$ to $5$, which we have found to improve the results in low latency. We analyze this effect in Section 6.1. $\rho_{k}$ decreases as a function of the lagging $k$, since we want the model to be more aggressive when $k$ is low and more conservative when $k$ is high. We set $\rho_{k_{min}}$ and $\rho_{k_{max}}$ and compute the threshold as: $\rho_{k}=\rho_{k_{min}}-d.(k-1)$, where $k_{min}\leq k\leq k_{max}$ and $d=(\rho_{k_{min}}-\rho_{k_{max}})/(k_{max}-k_{min})$. In order to vary the latency, we test the following values of $\rho_{k_{min}}$ and $\rho_{k_{max}}$: $\rho_{k_{min}}\in\{0.2,0.4,0.6,0.8,1.\}$, $\rho_{k_{max}}$ = $0.$, and $\rho_{k_{min}}$ = $1.$, $\rho_{k_{max}}\in\{0.2,0.4,0.6,0.8\}$.

In Figure 2, we compare our methods to previous adaptive and fixed strategies on two language directions. We find that our method improves or competes with other strategies while using a single model. MMA, Wait-$k$, and Adaptive Wait-$k$ require the training of multiple models in order to support different latency levels (as seen in Table 1), while our method is more efficient in this regard. Adapters Wait-$k$ is competitive with other strong fixed strategies like MoE Wait-$k$ and Multi-path Wait-k and it brings further improvements to combine it with the adaptive strategy.

Our method does not support higher latency on De-En because we are using a $k_{max}$ value of $5$ (as seen in Figures 2b and 2c), which we have found to improve results for low latency. However, we show the results for higher $k_{max}$ and compare them with Adaptive Wait-k on De-En in Section 6.1.

Using adapters alone is competitive with other methods, especially on En-Vi (as seen as in Figure 2a). Compared to Multi-path Wait-k, our method achieves better results on most latency levels, which shows the importance of minimizing interference between different lagging values. Combining our method with an adaptive strategy further improves the results, especially in low latency. In comparison to Adaptive Wait-$k$, where wait-$k$ policy models are trained and composed during inference, we find that our method is better in all latency levels while being more efficient.

Compared to MoE Wait-$k$, which also aims at minimizing interference introduced by multi-path training Zhang and Feng (2021), we find that our method is better in all latency levels on En-Vi and De-En with Transformer-Big (as seen in Figures 2a and 2c), while achieving competitive results when using Transformer-Base (as seen in Figure 2b). Our method is more flexible in terms of balancing the trade-off between parameter sharing and interference, as we can choose the number of wait-$k$ values supported by each adapter and we can also manipulate the capacity of the adapters by adjusting the bottleneck size. This can bring further improvements but requires more experimentation to find the appropriate hyperparameters.

We found that lowering the value of $k_{max}$ for the adaptive strategy improves the results in low latency, which we believe is the priority in SiMT, but a lower $k_{max}$ value also limits the ability of supporting high latency. In Figure 3, we show that by increasing the value of $k_{max}$ we can support high latency and get better quality translations. We compare to Adaptive Wait-$k$ and show that we still achieve better results for all the values of $k_{max}$. A lower $k_{max}$ forces the model to be more aggressive, which in some cases can improve the results in lower latency. The fact that forcing the model to be more aggressive improves the performance signifies that the adaptive strategy decides to wait in cases where the model is able to make a correct prediction, which suggests that the adaptive strategy based on the probability threshold can still be improved by a better strategy.

Although our method has more parameters than the baseline Multi-path Wait-$k$ due to the additional adapters, the effect on the inference time is not proportional to the number of adapters because only one adapter is activated at a time. To illustrate this, we compare the wall-clock inference time (averaged over 5 runs) of Adapters Wait-$k$ and Multi-path Wait-$k$ in Figure 4. It seems that adapters are faster in low $k$ values which could be due to over generation by the Multi-path model (where the model generates longer sequences than it should), while starting from a $k$ value of $7$, Multi-path Wait-$k$ is better and the difference fluctuates between 0.29s and 0.66s.

The adapter lagging $K_{A}$ specifies the number of wait-$k$ values that one single adapter will support and also the number of adapters that we will use. We vary the adapter lagging window between $1$ and $5$, while maintaining the range between $1$ and $16$. The results are shown in Figure 5. The wait-$k$ values supported by an adapter controls the amount of sharing and interference between the values. For example, for $K_{A}=\{1,5,9,13\}$, adapter $A_{1}$ will be trained on $k\in\{1,2,3,4\}$. We note that although it has more parameters, a window of $1$ achieves the worst results, which signifies that parameter sharing between wait-$k$ values is crucial. Adapter lagging with window $4$ and $5$ are competitive especially in low latency, which indicates that lower wait-$k$ values benefit more from sharing. This is consistent with the fact that wait-$k$ models achieve better results when tested on lower wait-$k$ values Zhang and Feng (2021).

The adapter’s bottleneck size can be used to tune the representation capacity of the adapters and can be interesting to tune depending on the language pair and the adapter lagging. In Figure 6, we experiment with doubling the adapter’s bottleneck size from $8$ to $128$, which can be regarded as increasing the representation capacity of the adapter network. We found that the bottleneck size impacts the performance but not in a consistent way - as in larger size results in better performance - but it seems to interact with other hyperparameters (e.g. adapter lagging) to improve or hinder the performance, especially in high latency, where the gap in performance is larger.

We compute the L2-norm of the adapter representations in order to discover which adapter layers are involved in the representations Liu et al. (2020); Zhu et al. (2021). We measure the L2-norm during inference for $k_{min}=1$ and $k_{max}=9$ while varying the value of $\rho_{k_{min}}$ and $\rho_{k_{max}}$, as described in Section 5.2. As depicted in Figure 7, the norm for all layers except layer 6 decreases as we increase $\rho_{k_{min}}$ oder $\rho_{k_{max}}$, which correlates with making the adaptive strategy more conservative because the threshold for making a write action is higher. This shows that the adapters are more involved in the prediction when the model is forced to be more aggressive. Only layer 6 is stably invested in adapting the model representations at all the threshold values, which seems to indicate that only low threshold predictions are complex enough to recruit all the adapter layers. Based on this observation, we experiment with inserting adapters only in the last layer (i.e. layer 6). We show in Figure 8 the results of comparing between inserting adapters in all layers and inserting the adapters only in the last layer, where we see a drop in performance only in lower latency levels. This shows that we can make the model more efficient by removing lower layer adapters with a small drop in performance.