Simul-LLM: A Framework for Exploring High-Quality Simultaneous
Translation with Large Language Models

LLMs are capable of effectively zero-shot sentence-to-sentence neural machine translation (NMT) (Vilar et al., 2023), but their performance can still be improved via simple techniques. Prompt construction has been demonstrated to be critical to LLM performance, both before and after fine-tuning Zhang et al. (2023). One-shot or few-shot performance via In-Context Learning (ICL) can produce near competitive results with fine-tuned LLMs for translation and can even be employed to enhance fine-tuned model performance (Vilar et al., 2023; Xu et al., 2023).

One particularly interesting area of study related to LLMs applied towards NMT remains whether or not to fully fine-tune a given model or engage in Parameter-Efficient Fine-Tuning (PEFT) (Mangrulkar et al., 2022). Early work in this area demonstrated some potential for smaller models (Üstün and Cooper Stickland, 2022), and the accessibility that PEFT provides designers in terms of fine-tuning on low-to-mid performance hardware setups renders it desirable. One of the most popular forms of PEFT freezes a LLM’s weights and adds Low-Rank Adaptation (LoRA) (Hu et al., 2022) adapters between layers²²2Other forms of PEFT exist, but adapter-based PEFT is extremely common so we refer to adapter-based PEFT simply as PEFT hereafter.. While fully fine-tuned NMT LLMs tend to suffer from some level of catastrophic-forgetting (Kirkpatrick et al., 2017), intuitively, PEFT-based NMT LLMs should not suffer from any loss of off-task performance, as adapters can be loaded or detached depending on whether or not a given user is prompting for a translation. Given these factors, PEFT is an attractive option for NMT LLMs.

As a subset of typical NMT, simultaneous translation (SimulMT) focuses on engaging in translation (write decisions) while balancing the amount of available source context (read decisions) to reduce translation lagging behavior/latency. This necessarily increases the difficulty of translating a sequence in one language to a sequence in another language, especially when structural and/or syntactical differences exist in the language pair. As a brief example, we can consider translating from a subject-verb-object (SVO) language, such as English, to a subject-object-verb (SOV) language like German. Based on the available source context in English, we may have to guess at the translation in German without access to the necessary context to effectively make that prediction because of the aforementioned syntactical differences in the language pair.

There are two classical, high level approaches to scheduling the write and read decisions of SimulMT, those being static schedules like wait-k (Ma et al., 2019) or adaptive schedules which are flexible and learned, such as variants of monotonic multi-head attention, adaptive wait-k compositions, wait-if-worse, decision state assisted SMT, and others (Grissom II et al., 2014; Gu et al., 2017; Arivazhagan et al., 2019; Zheng et al., 2020). Wait-k remains a particularly popular baseline strategy given its ease of application during training and during inference. It functions by retaining a k-lagging factor between the source context (either in tokens or in words) x and the translation hypothesis y. We can model a typical wait-k schedule’s probability of generating a given output sequence, provided some source sequence, with Equation 1:

Under circumstances or in environments where additional computational latency is acceptable, variations of beam search have been applied in simultaneous scenarios. Speculative Beam Search (SBS) (Zheng et al., 2019) is one such example where, at a high level, each translation step attempts to speculatively translate future steps for some number of beams, eventually selecting a single token or word (the first one) from the most likely beam for some beam length. When applied to wait-k, single token or word SBS can be modeled via Equation 2, where $w$ is the length of the beam and $\hat{\textbf{y}}_{i+1:w}$ represents the speculative beam of maximum joint probability:

Chunk-wise variations are also possible, where for multiple consecutive translation steps, or write decisions, words or subwords from the last determined most-likely beam are employed to cut down on computational latency.

LLMs are clearly capable of state-of-the-art NMT via a variety of techniques. Extending them towards SimulMT is a natural step, mirroring the progression of classical SimulMT research (i.e. seeking SimulMT improvements from NMT novelties). In this case, smaller language models (LM) have been employed to augment SimulMT efforts before, although usually this augmentation is limited to rendering the translation scheduler more discerning (Indurthi et al., 2022). Instead of just augmenting translation schedulers with smaller, bi-directional LMs, we explore using generative LLMs as direct SimulMT models in this work.

Intuitively, this results in the LLM providing a deeper understanding of source and target languages (via their pretraining on massive, somewhat multilingual corpora) than is typical for most SimulMT translation models. This can matter when encountering context alignment obstacles, where necessary source context is missing for a given translation time-step. In such situations, SimulMT models must infer based on what context does exist and past experiences to determine what the most likely next translated word will be, a task LLMs are naturally suited for.

The fine-tuning wrapper of Simul-LLM is constructed with a focus on simplicity and extensibility. It is depicted in Figure 1 and supports the following set of user-friendly features.

Evaluation agents for Simul-LLM are similarly built for ease of use and extensibility in addition to customizability towards complex translation schedules and decoding strategies while seamlessly interfacing with the preeminent SimulMT evaluation framework, SimulEval (Ma et al., 2020). This is depicted in Figure 2 and includes the following features.

The first prompt structure follows the classical NMT prompts where the source sentence is included in the prompt and the target sentence is the output of the LLM. As illustrated in the first prompt structure in Figure 3, with the expanded examples, each example contains partial source along with some instruction in the prompt (starting with "<h>") and partial target that is k words behind the source in the model output (starting with "<a>"), where k is the intended inference wait-k value. While this split source-target prompt structure seems to be a natural and plausible way for model fine-tuning, it has exhibited difficulties when learning for simultaneous translation. The root of this stems from how fine-tuning with LLMs often works: when filtering the prompt from loss calculations during fine-tuning, a response template of some kind (e.g. "<a>:" or ">>ANSWER<<") is employed to ensure only the target word that is generated at the current step (i.e., $\textbf{y}_{i}$) is scored. Unfortunately, with the split source-target prompt, the template allows for all the target words that have been generated from previous translation steps (i.e., $\textbf{y}_{1}$ to $\textbf{y}_{i}$) to be scored. Without employing a more complex loss filtering, this leads to an inappropriate level of context.

This lack of loss filtering can be especially problematic near the end of a given sequence’s translation. As a simple example, suppose that a given source sequence is of length $|\textbf{x}|$ and $|\textbf{x}|>>k$. In the first prompt structure where up to $i$ source words have been supplied and where $|\textbf{x}|>i>>k$, the LLM is effectively being fine-tuned for varying wait-k values ranging from wait-i to wait-k in a single example (as $\textbf{y}_{i}$ is predicted from $\textbf{x}_{1}$ to $\textbf{x}_{i+k}$ which is wait-k, $\textbf{y}_{i-1}$ is predicted from $\textbf{x}_{1}$ to $\textbf{x}_{i+k}$ which is wait-$(k+1)$, and so on). If $i>>k$ to the point where it is close to normal NMT levels of context, where $i\>\>\tilde{=}\>\>|\textbf{x}|$, then the LLM is no longer being effectively fine-tuned with an appropriate amount of source context for a given translation (write) decision schedule.

The above problem can be entirely side-stepped by our proposed second approach, single output token prompt structure, that embeds only the current target translation hypothesis within the model output. As illustrated in the second prompt structure in Figure 3, instead of allowing target translation hypotheses from previous time-steps to be incorporated into the loss, the proposed prompt structure shifts those previous translation hypotheses into the prompt. Combined with the expanded examples that form a rigorous wait-k curriculum in terms of the fine-tuning dataset (rigorous meaning a complete curriculum as opposed to a random subset), inference behavior can be copied exactly for every fine-tuning example, completely closing the context mismatch between fine-tuning and inference.

Between the two prompt structures, it is clear that the single output word prompt structure more closely replicates the relationship observed in Equation 1 between the source and target sequence during fine-tuning. For both structures, a source-target sentence pair is expanded up to $max(\textbf{|x|}-(k-1),\textbf{|y|})$ examples.

No standardized dataset exists for SimulMT with LLMs. Due to its popularity in speech-to-text simultaneous translation (SimulST), we employ MuST-C for our experiments³³3This allows for future work that explores multi-modal simultaneous LLMs, engaging in SimulST via a cascaded model structure with a transcription model for the source speech or a joint speech/text-to-text framework.. For the purposes of adapting MuST-C for text-to-text usage, we preprocess the dataset and filter out certain acoustic indicators (e.g., floating "-" characters representing pauses). In some cases, this resulted in significant changes to some samples of the test set, such as the removal of (Applause) acoustic indicators.

We employ MuST-C across two language pairs, those being English-to-German (en-de) and English-to-Spanish (en-es). Some additional experiments are provided for the en-es language pair that validate fundamental SimulMT concepts and display BLEU scores, gathered via sacreBLEU (Post, 2018), with respect to samples observed during fine-tuning. The original dataset contains roughly 270K training set samples and approximately 2.5-3K test set samples (tst-COMMON split) per language pair. The expanded version of this dataset for single output word fine-tuning contains approximately 5M training set samples per language pair.

While classical encoder-decoder SimulMT systems usually engage in either word or token-based wait-k, they most typically engage with whichever is more suitable for their vocabulary (i.e. word versus sub-word vocabularies). In spite of the fact that LLMs function via sub-word vocabularies, we recommend, and employ for this work, word-based wait-k for SimulMT LLMs, as it more closely resembles the flow of engaging with a natural language interface. Moreover, supposing that the LLM is receiving a given sequence actively from a transcription system or something similar, it makes intuitive sense to wait for a word to be emitted from the system as opposed to a fragment.

In Table 1, we provide a breakdown of the performance of several different models, decoding strategies, and wait-k schedules. Regarding our exploration related to adapting NMT LLMs to SimulMT, we also include results related to our implementation of Speculative Beam Search (SBS) (Zheng et al., 2019). As demonstrated by these results, compared with classical models, LLMs fine-tuned for NMT are very capable of SimulMT upon being adapted during inference (even exceeding the score of the classical NMT transformer on en-de that performs non-simultaneous translation). It is worth reiterating that classical architectures often engage in subword-based wait-k whereas we employ word-based wait-k for LLMs, but the comparisons still serve as a useful reference.

SBS-based decoding strategies helped NMT LLMs in the en-de language pair, but lacked improvement for the en-es language pair. We noted that our implementation (and seemingly also the original implementation) was sensitive to both the window size and the number of committed chunks, with large values for either resulting in the speculative target translation getting too close to the size of the source context. In our tests, when reaching the same length as the source context (akin to context levels of wait-1), degenerate output began to appear that resulted in the output trailing off (e.g., final output of "que…" instead of correct output of "que"). Notably, too many committed chunks only explains performance gaps for chunk-wise SBS, not single SBS, which is normally a flat improvement upon greedy decoding (single SBS is still sensitive to window size). Future experiments can be conducted to utilize the proposed Simul-LLM framework to quantify these factors.

In Table 1, we also provide a breakdown of the performance of our exploration of SimulMT LLMs with our proposed prompt structure in Section 5.2 that carefully manages source context availability for target translation generation. Two models are employed for this exploration, one fine-tuned for wait-3 inference, and another fine-tuned for wait-7 inference that produces noticeably better quality translations than the first. In addition, we provide results for the fine-tuned wait-7 models after fine-tuning on 2M samples and 5M samples of the fine-tuning dataset, with the fine-tuned wait-3 model having fine-tuned for 2M samples. This separation is due to computational constraints on our side, as we wanted to provide further comparisons in Section 7.3 and Table 2 at various evaluation wait-k values but lacked the compute to finish fine-tuning the wait-3 model.

While SimulMT LLMs are a more promising approach in achieving higher translation quality than NMT LLMs due to more direct task-specific fine-tuning and better context alignment, our experimental results suggest that, for the time-being, the performance of SimulMT LLMs is not guaranteed to be advantageous compared to NMT LLMs adapted to SimulMT. As seen in Table 1, fully fine-tuned SimulMT LLMs outperformed their NMT LLM counterparts on en-es but failed to do so on en-de. This is not completely unexpected as NMT LLMs have been optimized heavily in recent years whereas the exploration of SimulMT LLMs has just started. We provide some analysis below that points out a few possible reasons for this observed performance gap and we call for additional community efforts to investigate further.

First, it is possible that the fine-tuning hyperparameters are ill-suited for this particular prompt. We consider this likely to be the most influential issue on our observed results, given the drastic differences between the original and expanded datasets (the fine-tuning hyperparameters were united for both fine-tuning tasks). Second, at least one other work related to NMT LLMs (Chen et al., 2023) has demonstrated that relative positional embeddings can cause issues via attention dilution that ends up being unhelpful, suggesting that distancing the source context, running target hypothesis, and the current translation step hypothesis can be unexpectedly problematic. We posit that our proposed Simul-LLM can be leveraged to verify the above reasons.

It is well documented that in typical SimulMT systems, training or fine-tuning with a slightly higher wait-k than intended during inference can boost translation quality and generalizability across slightly lower wait-k (Ma et al., 2019). While this likely applies to SimulMT LLMs, no existing work has validated that this behavior persists. We provide a brief comparison of two SimulMT LLMs fine-tuned via wait-3 and wait-7 context levels in Table 2. The results demonstrate that, generally, the expected behavior does hold, with all LLMs fine-tuned in wait-7 outperforming their corresponding wait-3 models for the same inference wait-k with up to a 2.6 BLEU improvement. We leave validating additional, previously understood SimulMT principles in SimulMT LLMs to future work.

We are aware of two other concurrent efforts at applying LLMs to simultaneous translation (Koshkin et al., 2024; Wang et al., 2023). The former focuses on high-resource settings (i.e. order-of-magnitude larger language models than we aim for in this work) and additionally touches on comparisons with GPT-4 and speech-to-text simultaneous translation efforts. Contrastingly, the latter work aligns with ours, approaching the problem of LLM simultaneous translation in a low-resource setting. Their work, however, is limited to a focused contribution: a variation of longest-common prefix (LCP) that they call relaxed agreement LCP (RALCP). Neither host open-source software to test their changes and both only test Llama-based models in addition to arguably shallower explorations of this subject matter. Nonetheless, we intend to add support for end-to-end speech-to-text translation soon and have already added support for RALCP, which we provide brief results for in our Appendix. Both works validate our choice to engage with MuST-C as a dataset, as both works also employ it.