Error Correction by Paying Attention to Both Acoustic and Confidence References for Automatic Speech Recognition

Attention-based encoder-decoder end-to-end ASR models directly convert audio to text. The encoder and decoder mainly act as acoustic and language models, respectively. The encoder takes the speech features as input, extracting acoustic features layer by layer, and outputs the acoustic hidden variables [28]. The decoder, acting as a language model, converts the input acoustic hidden variables into the output text sequence. There are information differences between the deep and shallow features of the encoder. Shallower layers contain richer acoustic information, such as speech style, tone, and timbre. In comparison, deeper layers encode more refined pronunciation details, which aligns with the information required for ASR tasks.

Confidence estimation is important in many fields, including ASR[29, 30, 31, 32]. In recent years, although the accuracy of end-to-end ASR has improved, the quality of confidence estimation has collapsed. In terms of confidence accuracy, the posterior probability given by the Softmax layer after the speech recognition decoder proves to be sharp and of poor quality[33]. In other words, when the ASR model gets the wrong decoding result, the model is still confident in its answer and has a high posterior probability pointing to the current answer.

The overall flowchart of the error correction model proposed in this work is shown in Figure 1. The error correction adopts both the acoustic and confidence references to correct the ASR hypothesis. Besides, the error correction adopts the 3-best candidates from ASR as inputs to obtain better error correction accuracy.

The error correction module adopts the non-autoregressive method for fast decoding speed. The error correction module is a three-layer cross-attention decoder structure. The fused word embedding is first calculated by self-attention, and then two cross-attention mechanisms are performed with acoustic feature and fused confidence embedding, respectively. In cross-attention calculations, Query is derived from the word embedding, and Key and Value are derived from acoustic feature or fused confidence embedding. The outputs of the three attention calculations are added and then reclassified to get the error correction result.

An attention-based CEM is trained to provide confidence scores for recognition results. The structure and training flow of the CEM are illustrated in Figure 2. The CEM consists of a three-layer decoder (the same as the Transformer) and a Sigmoid function that predicts the confidence score of the ASR hypothesis. The input of CEM is the embedding extracted from the ASR hypothesis sequence and the text feature output from the middle layer (5th out of 6) of the ASR decoder. The edit distance is calculated between each hypothesis sequence relative to the ground truth sequence. The correct token is specified as 1, while the replacement or inserted token is specified as 0, making the alignment of the edit distance calculation available as the target for CEM training.

For example, if the ground truth sequence is $<ABCD>$ and the ASR hypothesis is $<ABcD>$, then the binary target sequence is $c=<1,1,0,1>$. The forward process is in full sequence, not autoregressive mode. For each N-best hypothesis, the CEM is trained to minimize the binary cross-entropy loss as follows:

where $p$ is the estimated confidence score, $c$ is the target sequence, $L$ is the length of the hypothesis.

As shown in Figure 3, after obtaining the N-best hypotheses from ASR ($N=3$ in this experiment), all candidates were aligned according to the method proposed by FastCorrect2 [18]. After alignment, candidates of the same length are then passed over the Confidence Module respectively to obtain confidence embedding. According to the word embedding ($\text{emb}^{w}$) and confidence embedding ($\text{emb}^{c}$) of N-best candidates, a linear layer is used to learn the weight of each candidate (${w}_{i}$). Finally, the fusion word embedding and fusion confidence embedding are weighted summed by the learned weight of each candidate. Multiple candidates are fused before entering the error correction module, avoiding additional attention computing time and further reducing error rates. Adding N-best information for ASR also supports the recovery of deleted errors to a certain extent and achieves variable-length error correction. The fusion is as follows:

where $\text{emb}^{w}_{f}$ and $\text{emb}^{c}_{f}$ are the fused embedding of word embedding and confidence embedding.

We evaluate our system on the AISHELL-1 [34] Mandarin ASR dataset (150/10/5 hours for train/dev/test). The ASR model employs a Conformer architecture with SpecAugment, speed perturbation, and a joint decoding language model. The hyper-parameters of the ASR model follow the Wenet codebase [35] (12-encoder/6-decoder layers, 256d 4-head attention, 1024d feed-forward network). The vocabulary size was 18532.

This section compares four baselines of error correction, two autoregressive and two non-autoregressive methods. The autoregressive methods are both standard error correction methods based on Transformer, one using only the best candidate (B1) [36], and another using the aligned N-best candidates as input (B2) [15]. Two non-autoregressive error correction methods from Microsoft (FastCorrect (B3) [17], FastCorrect2 (B4) [18]) that calculate edit distance and utilize a length predictor. FastCorrect uses only the best candidates, and FastCorrect2 incorporates aligned N-best candidates. These baselines’ error correction experimental results are from [20]. All baseline error correction models were pre-trained with a 400M unpaired text dataset and then fine-tuned with the AISHELL-1 dataset.

Our proposed method requires acoustic features and confidence scores, necessitating a 1,000-hour speech dataset for pre-training. Due to these pre-training data differences, error correction comparisons should be considered for reference only. During training, the CEM is first pre-trained according to section 3.2. Then the CEM parameters are frozen and other modules of the error correction model are trained (same as ASR, first pre-trained with the 1,000-hour speech dataset then fine-tuned with AISHELL-1).

Both the confidence estimation and error correction modules utilize a 3-layer cross-attention decoder with a model dimension of 384, 768-dimensional feed-forward networks, and 6-head attention. The error correction module performs the cross-attention calculation twice for each layer.

Beyond standard classification metrics (accuracy, precision, recall, specificity, and f1-score), Normalized Cross-Entropy (NCE) was used to gauge the quality of confidence scores [37]. If confidence scores for all tokens are gathered as $p=[p_{1},...,p_{N}]$ where $p_{n}\in[0,1]$, and the corresponding target confidence is set to $c=[c_{1},...,c_{N}]$ where $c_{n}\in\{0,1\}$, the NCE is computed as follows:

where $H(c)$ is the entropy of the target sequence and $H(c,p)$ is the binary cross-entropy between the target and the estimated confidence scores. NCE metric measures how closely the estimated confidence score reflects the true probability of a recognized word being correct. When confidence estimation is systematically better than the word correct ratio of ASR, NCE is positive. The closer the NCE is to 1, the higher the quality of the confidence estimation.

The speech error correction method is evaluated using the following metrics: character error rate (CER) after correction, character error rate reduction (CERR) relative to the ASR hypothesis, and inference latency (Latency) measured on one NVIDIA V100 GPU.