BinaryAlign: Word Alignment as Binary Sequence Labeling

Word alignment refers to the task of uncovering word correspondences between translated text pairs. The automatic prediction of word alignments dates back to the earliest work in machine translation with the IBM models Brown et al. (1993) where they were used as hidden variables that permit the use of direct token to token translation probabilities. While state of the art machine translation techniques have largely abandoned the use of word alignment as an explicit task Li (2022) other use cases for alignments have emerged including lexical constraint incorporation Chen et al. (2021b), analysing and evaluating translation models Bau et al. (2018); Neubig et al. (2019), and cross-lingual language pre-training Chi et al. (2021b).

In many real-world applications word alignment must be performed across several languages, often including languages with manually annotated word alignment data and others lacking such annotations. We refer to those languages as high and low-resource languages respectively. While word alignment for high-resource languages can be learned in a few-shot or fully supervised setting depending on the amount of data, for low-resource languages zero-shot learning strategies must be employed due to data scarcity.

State-of-the-art supervised techniques formalize the task of word alignment as a collection of SQuAD-style span prediction problems Nagata et al. (2020); Wu et al. (2023) while in zero-shot settings the best performing methods induce word alignment from the contextualized word embeddings of mulitingual pre-trained language models (mPLMs) Jalili Sabet et al. (2020); Dou and Neubig (2021); Wang et al. (2022). From a practical perspective, this discrepancy in the preferred method adds complexity to the deployment of word alignment models in real-world applications where both high and low-resource languages must be supported.

We observe a deeper issue that both span prediction and contextualized word embeddings are sub-optimal as each induces a bias in word alignment models that limits their accuracy. Span prediction methods cannot robustly deal with discontinuous word alignments without relying on complex post-processing and hyper-parameter tuning. Contextualized word embeddings method cannot deal effectively with untranslated words and one-to-multiple alignments because they rely on a softmax function that normalizes predictions at a sentence-level while in word alignment; one token being aligned to token $T$ does not mean that another token is less likely to be aligned to $T$. This poses word alignment as a single-label classification problem, while in reality it is better viewed as a series of binary classifications applied to each possible pair of words. Figure 1 shows some cases of one-to-multiple alignments, non-contiguous spans and untranslated words.

In this paper, we present BinaryAlign, a novel word alignment solution that outperforms the state-of-the-art in zero-shot, few-shot and fully-supervised settings. In particular, we reformulate word alignment as a set of binary classification tasks in which an individual alignment prediction is made for each possible pair of words. This reformulation of the task outperforms all previous approaches over five different language pairs with varying levels of supervision.

Recently, mPLM based approaches have significantly outperformed bilingual statistical methods Och and Ney (2003); Dyer et al. (2013); Östling and Tiedemann (2016) and bilingual neural methods Garg et al. (2019); Zenkel et al. (2020); Chen et al. (2020, 2021a); Zhang and van Genabith (2021). Among those approaches, we distinguish methods that achieve good performance without relying on manually annotated word alignment datasets Jalili Sabet et al. (2020); Dou and Neubig (2021); Wang et al. (2022) from supervised methods that leverage existing word alignment datasets to train high performing word aligners Nagata et al. (2020); Wu et al. (2023).

The first type of method relies on the approach of SimAlign Jalili Sabet et al. (2020) which proposes to induce word alignment from the contextualized word embeddings of mPLMs pre-trained on non-parallel data. AwesomeAlign Dou and Neubig (2021) builds on top of this approach and proposes to fine-tune mPLMs on parallel text with different objectives to improve the quality of the contextualized word embeddings. More recently, AccAlign Wang et al. (2022) showed that models trained to learn language-agnostic sentence-level embeddings also learn strong language-agnostic word-level embeddings and set the state of the art in the zero-shot setting. Also, Wang et al. (2022) show that fine-tuning on existing word alignment datasets improves performance of AccAlign on language pairs unseen during word alignment fine-tuning. Our method is different from this body of work because our training and inference objective differ. We formalize word alignment as a binary sequence labeling task while we can see those methods as framing word alignment as a token retrieval task.

In terms of approaches trained and evaluated in a supervised setting SpanAlign Nagata et al. (2020) formalizes word alignment as a collection of SQuAD-style span prediction problems which differs from our binary classification objective. However, this method falls short in zero-shot and few-shot when word alignment training data is scarce. To remedy this problem, WSPAlign Wu et al. (2023) introduces a pre-training method based on weak supervision that significantly improves performance for all amounts of training data.

Given a sentence $\mathcal{X}$ with $n$ words and a translation into another language $\mathcal{Y}$ with $m$ words, the task of word alignment is to produce an $n$ by $m$ adjacency matrix for the bipartite graph with the words of $\mathcal{X}$ on one side and the words of $\mathcal{Y}$ on the other (refer to Figure 1 for an illustration). As our model will employ commonly used subword tokenization preprocessing we assume access to an invertible tokenizer, often implemented in practice with a list of subword units, a greedy algorithm for subword chunking, and leading symbols to denote word continuation in the vocab file Sennrich et al. (2016); Wu et al. (2016).

We present BinaryAlign, a novel word alignment approach using a binary sequence labeling model, shown in Figure 2. The inputs to this model are a subword tokenized source sentence $X=x_{1},x_{2},...,x_{|X|}$, a subword tokenized target sentence $Y=y_{1},y_{2},...,y_{|Y|}$, and a reference word $w_{X}=x[i:j]$ which is a subspan of X. We model the distribution of a binary alignment vector $A$ of size $|Y|$ in which each entry $a_{k}$ indicates if the word in $Y$ that contains $y_{k}$ is aligned to $w_{X}$.

We first preprocess $X$ by surrounding $w_{X}$ with unique separator tokens and then cross-encode the source and target sentences with an mPLM. For each token $y_{k}$ in the target sentence we pass its final encoded representation through a linear layer to produce a single logit $z_{k}$. We model $a_{k}$ with a logistic function using $z_{k}$ as its parameter:

A supervised signal for token level alignments $a_{k}$ is easily divined from word alignment data and so this form is sufficient to estimate the parameters of the model. However, our true inference-time goal of word to word alignment requires the use of additional heuristics. To motivate these heuristics we formalize $\mathcal{W}$ as the inverse of the surjective mapping between the token indices in $Y$ and its corresponding words; for any subspan $w_{Y}$ of $Y$, $\mathcal{W}(w_{Y})$ returns the token indices in $Y$ that compose $w_{Y}$.

Given an aggregation function $agg$, we define the probability of the event $a^{\prime}$ that there exists an alignment between word $w_{X}$ in $X$ and word $w_{Y}$ in $Y$ as

Preliminary experiments suggest that the maximum aggregation strategy yields slightly superior performance compared to the mean and minimum aggregation strategies; hence, we adopt it for all subsequent experiments.

Word alignment in its general form is a symmetric problem in that we would expect the same answer if the source and target were swapped. However, like most leading word alignment methods, our method is asymmetric; the source and target sentences are handled differently. This deficiency is empirically detrimental to performance with the common remedy being to perform alignment in both directions and then to merge the two predictions in some manner.

We use the following symmetrization technique: letting $p_{X\rightarrow Y}$ denote the use of $X$ as the source and $Y$ as the target sentence, we average $p_{X\rightarrow Y}(a^{\prime})$ and $p_{Y\rightarrow X}(a^{\prime})$ and apply a threshold decision rule to make our final inference prediction. While outside the scope of this study, we note that various other options exist and have been explored in previous work such as bidirectional average Nagata et al. (2020) or intersection, union and grow-diag-final: the default symmetrization heuristics supported in Moses Koehn et al. (2007).

We presented BinaryAlign, a novel word alignment training and inference procedure. In particular, we proposed to reformulate the word alignment problem as a binary token classification task. We showed that because of this reformulation BinaryAlign outperforms existing methods regardless of the degree of supervision. In addition we showed that it overcomes the inherent limitations of previous methods relying on span prediction and softmax. As a result, we made the word alignment task easier to tackle by using a single model for both high and low-resource languages.

In the future we plan to explore the use of larger decoder-only or encoder-decoder models such as mT5 Xue et al. (2021) to see how much alignment performance will increase. We also plan on investigating knowledge distillation techniques to improve the inference time of our method.

The inference cost is the main limitation of our method. When using symmetrization, it has to perform a forward pass for each word of both sentences, which can be slow with long sequences. However, this is a drawback that we share with previous state-of-the-art supervised approaches Nagata et al. (2020); Wu et al. (2023).

In addition, we did not experiment on extremely low-resource languages that the mPLM has not seen during pre-training Ebrahimi et al. (2023). While the benefits of our new formulation would likely apply to any language, it is unclear how our method will rapidly adapt the mPLM to new languages Garcia et al. (2021).

In real-world applications, translations are often partial and noisy. Unfortunately, we could not evaluate the robustness of our method to different translation pair quality because this type of word alignment dataset does not exist.