Speculative Speech Recognition
by Audio-Prefixed Low-Rank Adaptation of Language Models

Our approach to solving speculative ASR is based on the hybrid approach from [11], with the LSTMs in the RNN-T and LM replaced by a pretrained Conformer-Transducer [15] ASR model and a pretrained Transformer language model [16]. The Transducer is used to transcribe the spoken prefix $\mathbf{\textbf{X}}^{p}$ into ${Y}^{p}=({y}_{1},\dots,{y}_{r})$, and the LM is used to predict the suffix $Y^{s}=(y_{r+1}\dots,y_{U})$ by conditioning on the transcription. This essentially parameterizes (2) thus:

where $\phi$ denotes the parameters of the Transducer model and $\theta$ denotes the parameters of the language model.

Simply stacking the Transducer and the LM as in Section 2.1 implicitly assumes that the transcription is an efficient enough summary of the input speech for the purposes of speculating the suffix, i.e., that $P(Y^{s}|X^{p},Y^{p})=P(Y^{s}|Y^{p})$. However, the speech signal itself contains information such as speaker, channel or domain clues which could be useful for better constraining the language model output. Furthermore, since the Transducer objective considers all possible alignments (in contrast to hybrid models which use forced alignment) , there is only a loose correspondence between the input frame and the output token ($j$ and $r$ respectively in Section 2.1), meaning that the ASR can choose to transcribe sounds later than they are uttered. We argue, therefore, that it would be useful to condition the suffix generation on a representation of the audio signal.

To this end, we add a fixed-length representation, $\mathbf{\textbf{A}}^{p}=(\mathbf{\textbf{a}}_{1},\dots,\mathbf{\textbf{a}}_{M})$, of the audio as a prefix to the language model. $\mathbf{\textbf{A}}^{p}$ is the output of a multihead attention layer whose keys and values are projections of the Transducer’s Conformer encoder output, and whose queries are projections a sequence of $M$ trainable vectors, $\mathbf{\textbf{Q}}=(\mathbf{\textbf{q}}_{1},\dots,\mathbf{\textbf{q}}_{M})$. Thus, when the Transformer speculates the $u$th overall output token, its input is:

where $\mathbf{\textbf{e}}[\cdot]$ denotes the LM’s input embedding lookup. Although it is possible in theory to use other speech representations, we choose the Conformer output since we get it for free as part of the ASR computation. Moreover, we know that it is rich in lexical content since it is an encoding that is used directly for ASR. We use a fixed-length summary of the encoding as prefix because doing so reduces (when $j>M$), and keeps fixed, the computational cost of the subsequent LM decoding compared to using the encoding directly. Ultimately, instead of (3), we have the following parameterization:

where $\zeta$ denotes the multihead attention parameters. Our training process will involve keeping the first term on the right hand side fixed, and learning parameters which maximize the second term—$P_{\phi_{e},\theta,Q,\zeta}(Y^{s}|Y^{p},\mathbf{\textbf{X}}^{p})\coloneqq\prod_{u=r+1}^{U}P_{\phi_{e},\theta,Q,\zeta}(y_{u}|y_{<u},\mathbf{\textbf{X}}^{p})$.

With the parameters of the Conformer-Transducer fixed, we add LoRA layers to each Transformer layer in the LM. The LoRA parameters are trained along with the LM’s tied embedding-softmax layer, the cross-attention parameters and query vectors to maximize the log-likelihood of the correct suffix.

We do the finetuning on a dataset of audio-text transcription $(\mathbf{\textbf{X}},Y)$ pairs. For each training sample, we first get $\mathbf{\textbf{X}}^{p}$ by truncating the last 1 second of audio. Then we pass the truncated audio to the Transducer-based ASR system to get a hypothesized transcript $\hat{Y}^{p}$. Next we feed this hypothesis into the language model—along with the prefix from the cross-attention on the encoder output—to predict $Y^{s}=(y_{r+1},\dots,y_{U})$—the portion of $Y$ corresponding to the truncated 1 second of audio.

Determining $r$ for training, however, is not as trivial as it may seem. Consider, for example, an utterance whose correct transcription is “i’d like to call my father”, but for which the ASR generates the prefix “i’d line to call ma-”. Because of the ASR errors, the correct suffix is not clearly defined. It is therefore necessary to first determine what part of the transcription has been covered—possibly erroneously—by the ASR and what part remains to be speculated.

We use stochastic gradient descent with Adam [18] to finetune. For each mini-batch $\mathcal{B}$, we minimize the cross-entropy:

with respect to the cross-attention parameters ($\zeta$), soft prompt query vectors (Q) and LoRA parameters ($\Delta\theta$).

SSR differs from conventional ASR in one crucial respect: it has no single correct answer; for a given prefix audio, it is impossible to determine the suffix with perfect certitude. This makes the word error rate (WER) metric unsuitable for out purposes. Moreover WER—or any other metric that is computed over the entire sentence—also encompasses a measure of the prefix transcription accuracy and thus can only be, at best, a diffuse proxy for speculation performance. Hence, we seek a metric which isolates the accuracy of the speculated suffix while also accounting for the uncertainty inherent in the task.

One solution would be to treat the task as a language modeling one, and to report the perplexity of the correct suffix. Perplexity however has the drawback that it is not an operational measure, i.e., we can use it to compare systems and say lower is better, but knowing the exact perplexity score says little about the conditions under which the system is usable. Furthermore, perplexity has to be computed with “teacher-forcing”, which does not reflect the practical usage of an ASR system.

Instead, we treat speculation as a quasi-retrieval problem and propose an analog of recall-at-k. The metric, which we term suffix oracle word error rate (SOWER) is computed by truncating $t$ seconds from the end of the audio, letting the system hypothesize $k$ suffixes $\hat{Y}^{s}(1),\dots,\hat{Y}^{s}(k)$, and returning the minimal WER between the hypotheses and the correct suffix $Y^{s}$:

This measures how well we expect the model to do if we return the top-$k$ hypotheses. By default, we set $t=1$ and $k=8$ in our experiments. Note that, to get the correct suffix for evaluation, we once again use the AWSED procedure described in Section 2.3 on the prefix transcription and the full reference.

We use Speechstew [19], an amalgamation of multiple public corpora totalling about 5000 hours, to pretrain the Conformer for transcription. We conduct two sets of experiments, varying the pretraining, finetuinnig and testing data:

The Transducer model has a 100 million parameter, 17-layer Conformer-L [15] encoder with 512-dimensional layers operating on 80-dimensional log-mel filterbank inputs, an LSTM prediction network with one 512-dimensional layer, and a 2-layer feedforward joint network with 512-dimensional intermediate layer and 1024-dimensional output softmax layer corresponding to 1024 Librispeech word-piece [26] targets.

The LM is a 100 million parameter transformer decoder [16] with eight 1024-dimensional layers, each split into 16 attention heads, and tied embedding-softmax with the same 1024 word piece units as the Transducer.

We use $M=64$ queries (equivalent in length to $2.56$ seconds of audio) of 1024 dimensions and a single 1024-dimensional cross-attention layer with four heads for computing the soft-prompt. We set the rank of the LoRA adapters to 10 for the Librispeech experiments and 50 for the multi-domain experiments, resulting in 12 million and 19 million trainable parameters respectively to be finetuned for speculation.

We report results for four model configurations:

First we conduct experiments of the Librispeech test sets, where we truncate the last second of audio from each utterance and speculate them with various systems. This 1 second of audio contains an average of around 2 words/utterance (1.98-2.2 depending on the test set) which the systems must speculate.

Table 1 shows SOWER—$S(1,8)$—on the Librispeech test sets. All the systems, including PM, yield SOWER below 100, i.e., on average, picking the best of top-8 speculated hypotheses is better than not speculating at all. HO significantly outperforms PM, highlighting the positive impact of making the LM ASR-error-aware. SP yields a further 6.7% relative improvement over HO. Finetuning jointly with unpaired text (ST) leads to further improvements, indicating that the SP—and HO—loses some capacity as a language model while fitting the ASR training set (which is much smaller than the unpaired data), and this capacity can be somewhat recovered by joint training. Finally, the table also shows that speculating only the next word is, as expected, easier than predicting the rest of the utterance—with ST-1w in particular averaging SOWER below 50% on single word prediction.

On inspecting the transcriptions and listening to the corresponding audio, we found several cases where the prefix audio ends with the beginnings of a sound, usually a stop consonant, which the ASR does not transcribe. Where the systems without audio prompt speculate semantically-appropriate completions, the ones with audio prompt speculate semantically-appropriate completions which also start with the correct phoneme.

Table 2 shows the Librispeech oracle WER computed over the entire utterances (not just the suffixes), and therefore show the impact of speculation on the whole WER. The speculation methods are compared with two purely Conformer-Transducer systems: a baseline, where each utterance is truncated and no speculation is done, and a topline which sees the entire audio without truncation or need to speculate. Overall, the oracle word error rates of the speculation methods follows the same trends as the SOWER, with the best speculation method (ST) recovering about 37.6% of the gap between the baseline and the topline.

Table 2 also shows the oracle WER of the best system, ST, at $k=1$. Unsurprisingly, the oracle WER (and SOWER although it is not shown) degrades as $k$ is decreased. More interestingly, we see that even at $k=1$, ST outperforms the baseline. In other words, even one-shot speculation with ST is slightly better than no speculation at all. In fact, we found that all the speculation systems improve upon the baseline with the exception of PM at $k=1$ (which has SOWER $>100$).

In addition to Librispeech, we report results on the AMI, Switchboard and TED-LIUM test sets in order to test generalization of the proposed speculation methods across domains.

Table 3 shows the SOWER across test sets. Here we find that the improvements from HO compared to PM are more subdued. In fact, for the Librispeech test sets, HO is slightly worse than PM. The largest improvements are on AMI, which is not represented in the LM pretraining data, and thus benefits the most from finetuning on the Speechstew training set–which does contain AMI data. SP outperforms HO by a larger margin across all datasets, underscoring the importance of conditioning the LM on audio. Finally, we observe that ST does not outperform SP in terms of SOWER, except on Librispeech. This is likely due to the fact that the Librispeech text is the most represented in the unpaired text used for joint training.

Qualitatively, we found that because the Speechstew data is an amalgamation of various datasets, some of which have transcripts with punctuation and extra tags like “[laughter]”, “[noise]” etc., HO learns to frequently speculate these tags even for datasets like Librispeech which are normalized, and thus increases number of errors, while SP does so less frequently. This explains some of the degradation of HO on Librispeech. It also hints at another advantage of having audio conditioning, namely it triggers domain-specific behavior in the LM.