Improving Neural Biasing for Contextual Speech Recognition by Early Context Injection and Text Perturbation

Transducer model is first proposed in [36] to learn the transformation between sequences, e.g., from speech to text transcription. Formally, it learns the probability $p(\mathbf{W}|\mathbf{X})$ of word (or word piece) sequence $\mathbf{W}=(w_{1},w_{2},...,w_{U})$ of length $U$, given a speech feature sequence $\mathbf{X}=(\mathbf{x}_{1},\mathbf{x}_{2},...,\mathbf{x}_{T})$ of length $T$. Transducer model has been widely used in ASR due to its effectiveness and streaming nature. It has three components.

Encoder: The encoder serves the role of an acoustic model. It takes the feature sequence $\mathbf{X}$ as input, and transforms it to a sequence $\mathbf{H}^{enc}=(\mathbf{h}_{1}^{enc},\mathbf{h}_{2}^{enc},...,\mathbf{h}_{T}^{enc})$ of high-level representations of the input:

There can be many layers of the transformation. The output from the $i$-th layer $f^{enc}_{i}$ is denoted as $\mathbf{H}^{enc}_{i}$, and the final output is $\mathbf{H}^{enc}$. Note, there can be downsampling along the time axis, which is omitted here for simplicity. In practice, the encoder is implemented by a recurrent neural network (RNN) or more recently Conformer [37] architecture. Encoder can take up most parameters (e.g., more than $90\%$) in the transducer model.

Predictor. Given $\mathbf{H}^{enc}$ from the encoder, one may produce a frame-wise softmax distribution over the output word pieces vocabulary, which is the idea of a CTC model. The transducer model, on the other hand, explicitly imposes dependencies between the output word pieces in $\mathbf{W}$ by the predictor. It acts like a language model:

Joiner. The joiner takes the embeddings $\mathbf{h}^{enc}_{t}$ and $\mathbf{h}^{pred}_{u}$ from both the encoder and the predictor to produce a softmax probability distribution over the output word pieces vocabulary:

Note that the output from the joiner is of the shape $(T,U,V)$ for a single input sequence, where $V$ is the size of the word pieces vocabulary. From this output, the probability of the ground-truth transcription $\mathbf{\bar{W}}$ can be computed efficiently via dynamic programming, which will be maximized during training.

CLAS [4] was first proposed to bias the attention-based encoder-decoder (LAS) models with cross attention. Later, [8, 11, 12] applied cross attention to bias transducer models, which has become a popular solution recently. In general, contextual ASR models learn to predict the probability $p(\mathbf{W}|\mathbf{X},\mathbf{C})$ of word sequences given both acoustic feature sequence $\mathbf{X}$ and some context $\mathbf{C}$. In this work, we consider context for each utterance as a list of biasing words or phrases $\mathbf{C}=\{\mathbf{c}_{0},\mathbf{c}_{1},\mathbf{c}_{2},...,\mathbf{c}_{N}\}$. Note, $\mathbf{c}_{0}$ is a special entry corresponding to the no-bias option.

Intuitively, when we predict the word piece at the $t$-th frame, we expect the relevant entries in the biasing list get sufficient attention. To do this, one can first independently represent each word or phrase (of various number of characters) $\mathbf{c}_{j}\in\mathbf{C}$ by a fixed-dimensional embedding $\mathbf{c}_{j}^{e}\in\mathbf{C}^{e}$ of $D$ dimensions. An LSTM context encoder can be employed to accomplish this:

Note $\mathbf{c}_{0}^{e}$ is hard-wired to all zeros.

Then, for each frame $\mathbf{h}_{t}^{enc}$ from the encoder output — it can also be the output from the $i$-th encoder layer, we omit the subscript $i$ for simplicity — or for each frame $\mathbf{h}^{pred}_{u-1}$ from the predictor’s embedding, $\mathbf{h}_{t}^{enc}$ oder $\mathbf{h}^{pred}_{u-1}$ is used as the query to attend to the embedded biasing list, whose keys and values are both $\mathbf{c}_{j}^{e}\in\mathbf{C}^{e}$. This makes the cross-attention contextual biasing adapter:

where $\mathbf{A}^{enc}$, $\mathbf{H}^{enc}$, $\mathbf{C}^{e}$ are matrices of shape $T\times N$, $T\times D$ and $N\times D$ respectively (additional projection layers are omitted here). For the $t$-th frame, $\mathbf{a}_{t}^{enc}\in\mathbf{A}_{t}^{enc}$ defines an attention weights distribution over the $N$ biasing words. Their weighted sum is the attention output: $\mathbf{b}_{t}^{enc}={\mathbf{a}_{t}^{enc}}^{\top}\cdot\mathbf{C}^{e}$.

Finally, the attention output $\mathbf{b}_{t}^{enc}$ is used to “edit” the encoder’s (or decoder’s) embedding. This can be implemented via element-wise addition:

Thus, the internal representations of the transducer model become context-aware. Modified embeddings $\hat{\mathbf{h}}_{t}^{enc}$ and $\hat{\mathbf{h}}_{u-1}^{pred}$ are fed to the next encoder layers or the joiner to make context-aware predictions. This is illustrated in Figure 1.

The above context encoder and the cross-attention biasing adapter can be trained in an end-to-end fashion together with the transducer model. In [8, 11], all parameters are trained from scratch. In [12], the transducer’s parameters are pretrained and frozen. Only a small number of the rest of the parameters (yellow modules in Fig. 1) are updated. The benefit is that the neural biasing maintains the performance of the pretrained model when no context is provided. We follow [12] in this paper.

In [2, 4, 5, 8, 11, 12, 14, 17], cross-attention neural biasing architectures are used as described in section 2.2. All of them inject contexts only into the encoder’s final output. The drawback is that the contextualization of a frame $\hat{\mathbf{h}}_{t}^{enc}$ has no impact on the other frames, as they are edited essentially independently from one another. On the other hand, if contexts are injected to earlier encoder layers (e.g., at $\mathbf{h}_{i,t}^{enc}$ in Figure 1), it may have far-reaching impacts on the model’s internal states via its self-attention mechanism.

One might argue that cross-attention context lookup can introduce significant computation overhead, hence they perform it only once at the last encoder layer. However, as we will show later in Section 4, this overhead is negligible. In fact, context injection can be viewed as simply adding an extra layer to the encoder. For ASR, the input or intermediate sequences usually have several hundred frames. Thus, each layer of the encoder, e.g., a conformer [37], needs to do self-attention over several hundred items. This is a comparable computation to contexts lookup if we have several hundred biasing words to attend to.

The predictor of the transducer is usually implemented by a simple LSTM network or even a stateless $n$-gram feed-forward network [38]. Thus, we only bias the predictor’s final output.

When transcribing unfamiliar names of people, locations or products, humans tend to choose the most familiar or phonetically similar option. Given a reference guide, they will likely refer to it for guidance. Moreover, when the contents in the reference guide change, they adapt accordingly. Text perturbation follows this idea to train the model to optimally use the contextual information. While acoustic data augmentation (e.g., [39]) is a common practice for ASR, text perturbation has not been widely adopted yet.

On the other hand, we observe that ASR models can overfit the training data. The word error rates on the training data is so low that the end-to-end training of neural biasing modules may not have enough chances to learn to attend to the contexts. On LibriSpeech and SPGISpeech (Section 4.1), the training data distribution and word error rates are listed in Table 1. Even without contexts, there is only 29.38% and 4.42% utterances in the training data containing at least one mis-recognized rare word (defined in Section 4.1). This implies that only these utterances may potentially benefit from the contexts containing ground-truth rare words during training. After random text perturbation (details can be found in Section 4.1), the percentage of such mis-recognized utterances containing rare words increases to 90.13% and 36.37% of all utterances.

In our experiments, we apply hand-crafted linguistic rules²²2https://gist.github.com/huangruizhe/dd75cf44bde12751500b8c43c73f3f22 to obtain similar-sounding spelling alternatives. We replace the “maximal” matched pattern with its counterpart (e.g., “lee” $\rightarrow$ “li” although pattern “e”$\leftrightarrow$“a” is also a match). For some languages, e.g., Chinese, it is even easier to obtain spelling alternatives by pronunciation dictionary lookup for characters.

We use LibriSpeech [33] and SPGISpeech [34] to train contextual ASR models. The orthographically normalized version of SPGISpeech is used. Dataset statistics can be found in Table 1. We follow the same setup as in [10, 17] to generate artificial biasing lists during training. More specifically, we define rare words to be the words beyond top 5k/3k most frequent words for LibriSpeech/SPGISpeech. Due to the long tail nature, these words has poor ASR performance compared to the common words (Table 2 row 1). Thus, we provide the ASR model with such words, as well as some distractors, as the context, aiming to enhance the model’s ability to recognize these words. For each utterance, we include all rare words from its reference transcription into its context. We also add 100 distractors sampled from all the rare words in the training vocabulary. We observe no gains when adding more distractors.

For evaluation, we use the test sets in LibriSpeech and SPGISpeech. The LibriSpeech contexts are predefined in [10], and we define the SPGISpeech contexts similary. We also use ConEC [35], which consists of earnings calls in Earnings-21 [40] and their real-world supplementary materials including presentation slides, earnings news release, a list of meeting participants’ names and affiliations. We follow the evaluation metrics in [10, 17] to compute overall word error rate (WER) and the WER for common words and rare words (U-WER, B-WER). For ConEC, we also report the WERs for named entities.

For text perturbation, we randomly perturb the spellings of rare words with a given probability (0.2 for both datasets). For SPGISpeech, we also randomly (with probability of 0.8) discard utterances containing no rare words during training.

We use stateless transducer [38] with Zipformer [41] encoder. The model has 15 encoder layers and 71.5M parameters in total. The transducers are trained on LibriSpeech or SPGISpeech. Then, we freeze their parameters. We use a BiLSTM context encoder of two layers and 128-dim hidden states, which is shared across all contextual biasing modules. We do not notice improvements using separate context encoders or using BERT [42] as the encoder. Each biasing adapter is implemented by a 4-head, 128-dim multi-head dot product attention layer, as well as necessary projection layers. Overall, the contextual biasing modules account for 3.7%–6.7% of the parameters compared to the transducer model. Our main contextual ASR model injects contexts at both the 9th and 15th (the last) encoder layers. The model is optimized with ScaledAdam [41] for 30 epochs.

The WER results for LibriSpeech are reported in Table 2. When there is no context or biasing, the rare word WER can be as high as 21.83%. Shallow fusion with biasing words provides significant improvement on this dataset, which is nearly 50% relative WER reduction. Note that shallow fusion is insensitive to the size $N$ of the biasing words lists. The neural biasing results from two recently published papers [15, 17] and our vanilla neural biasing implementation do not outperform our shallow fusion baseline, even though their “no biasing” results (not shown here) are very close to ours. When contexts are injected to both the 9th and 15th layers, we see improvements over the vanilla neural biasing. When we further perturb the reference transcriptions, we achieve the best neural biasing results, with 60% and 25% B-WER reduction over no biasing and shallow fusion, without degrading U-WER. Also note that neural biasing is more sensitive to the biasing list size $N$ compared to shallow fusion.

On SPGISpeech (Table 3), with a test set of 100 hours long, our techniques again outperform the baselines. On ConEC, our methods surpass the baseline except for one entity class.

Next, we explore which layers are optimal for integrating the contexts (Table 4). Text perturbation is disabled here. It appears that the combination of layers 9 and 15 yields the best performance. We also measure the wall-clock time to decode LibriSpeech test-other (of 5.3 hours) with a beam size of 4 for beam search. It takes about 3.5 minutes on one NVIDIA Tesla V100 GPU, even when we inject contexts to 4 encoder layers. As a reference, it takes 2.6 minutes for a non-contextual transducer model to decode the same data.

Finally, we search for the best probability for perturbing each rare word (Table 5). When the probability takes 0.2, the contextual ASR model has a balanced WER performance for common and rare words.