Bridging the Gap: Integrating Pre-trained Speech Enhancement and Recognition Models for Robust Speech Recognition

Ensuring the noise robustness of ASR is critical for real-world applications. Previous studies have addressed this challenge by incorporating synthetic or real noisy speech during training to enhance the ASR robustness [24]. For instance, Whisper [24] was trained using a large amount of data collected on the Internet under weak supervision. Despite its inherent noise robustness, ASR can suffer from low signal-to-noise ratios (SNRs). To mitigate potential noise in input speech, previous studies employed SE as a preprocessing stage for ASR [13]. However, NN-based SE methods often introduce artifacts to enhanced signals and potentially deteriorate the ASR performance [3, 21]. Although the joint training of SE and ASR models has been proposed to improve consistency [22, 23], these methods may be impracticable when commercial ASR models are inaccessible or training costs are unaffordable. Alternatively, some studies have discovered that applying post-processing to SE models, such as OA techniques, can improve ASR performance without previous constraints [3, 21]. This study also focuses on developing post-processing techniques for robust ASR.

OA is a practical technique designed to alleviate speech distortion. In OA, the original and enhanced waveforms are linearly combined to form input signals for downstream tasks. Numerous studies have explored noise-robust speech applications based on OA, including ASR [3], speaker verification [6], and speech emotion recognition [25]. The kernel of the OA method lies in determining the coefficient of the waveform combination, which is often achieved through a data-driven process using an NN. Existing NN-based coefficient decision methods, including likelihood-based [3] and reinforcement-learning-based approaches [6], often leverage information from both the SE and downstream models during training. However, these methods tend to fit specific combinations of SE and downstream models, thereby exhibiting less generalizability. To address this limitation, we propose an OA technique that does not rely on ASR transcription.

In this study, we employed four well-known pre-trained SE models: CMGAN  [15], MP-SENet [16], DEMUCS [14], and SEMamba [17]. These SE models have diverse properties and can be used to investigate the effectiveness of the proposed method comprehensively. DEMUCS is a waveform-mapping-based SE model [26, 27]. In contrast, CMGAN, MP-SENet, and SEMamba are spectral-masking-based SE models that achieve state-of-the-art performance on the VoiceBank-DEMAND dataset. Moreover, DEMUCS is trained on the DNS-Challenge dataset [28]. CMGAN, MP-SENet, and SEMamba are trained on the VoiceBank-DEMAND dataset [29]. The parameters of these pre-trained SE models remained fixed when applied in this study.

Whisper [24] was applied as an ASR model in this study. Whisper is a powerful ASR model trained with a large amount of data under weak supervision and inherently exhibits noise robustness. Various Whisper versions with different model sizes exist. We selected the base and large-v3 versions and fixed their parameters as the SE models in this study.

Fig. 1 (b) illustrates the structure of the bridge module inspired by a previous study on noise-robust speech emotion recognition [25]. The bridge module first calculates the cosine similarity between the spectrogram of the input and enhanced signals with respect to the time dimension. A linear layer is then used to predict the SNR level [25] $S$. The final output $S^{\prime}$ used as the OA coefficient is constrained by a predefined clipping function, limiting its value within a specific range (e.g., 0.6 to 1). The design of the bridge module is based on the concept that ASR can perform well with the original speech at high SNR levels, where the enhanced and original spectrograms are similar. Moreover, the floor of the clipping function is specified, as we consider that Whisper ASR models are inherently noise-robust to some extent. In this study, the floor value was set to 0.6.

The key advantage of the bridge module lies in its generalization capability, as it can be applied to various SE and ASR models without fine-tuning. The training data for the bridge module contained pure clean speech, pure background noise, and enhanced signals, which were generated using CMGAN in this study. The labels for speech and noise signals were set to 1 and 0, respectively. Formulated as a regression problem, the bridge module can learn the distribution of input speech with different noise levels in an unsupervised manner.

The integration of the pre-trained SE, pre-trained ASR, and bridge modules for a robust ASR is presented in this subsection. The SE model transforms the noisy speech waveform $x$ into the enhanced speech waveform $\hat{x}$. The bridge module then predicts the coefficient for OA, $S’$, based on the spectrograms of the noisy speech $x$ and enhanced speech $\hat{x}$. The OA process is subsequently employed to produce the input waveform for ASR, $\tilde{x}$, using the coefficient $S’$. Finally, the ASR model generates the corresponding text. This process can be expressed by the following equations:

where SE and Bridge denote the SE model and bridge module, and $x$, $\hat{x}$, and $\tilde{x}$ denote the original, enhanced, and combined speech waveforms, respectively.

This study adopts clean speech from the LibriSpeech train-360 corpus [30] and 80% of the noise data from the DNS-Challenge [28] to train the bridge module. LibriSpeech train-360 contains 103,093 utterances from 921 speakers, and the DNS-Challenge comprises 65,303 foreground and background noise samples.

To evaluate the robustness of the ASR system, noisy utterances are sourced from three datasets: Librispeech with DNS-challenge, Aurora-4 [31], and the VoiceBank-DEMAND dataset [29]. For LibriSpeech, the test-clean corpus (2,580 clean utterances) is adopted, and each utterance is contaminated by five randomly selected noise types from the remaining 20% of DNS-Challenge at five SNR levels (0, ±6, and ±12 dB). Aurora-4 comprises 3960 noisy utterances, which are from 330 clean utterances distorted by six types of noise and two types of room impulse responses at an SNR in the range of 5-15 dB. The VoiceBank-DEMAND test set comprises 824 noisy utterances at four SNR levels (2.5, 7.5, 12.5, and 17.5 dB).

The sampling rate of all signals was set to 16 kHz. The STFT parameter setup comprised a Hanning window with a window length of 400 and a hop length of 100. To train the bridge module, we employed a pre-trained CMGAN [15] as the SE model. The batch size was 32, and the learning rate was 0.0001. An SGD optimizer with a momentum of 0.9 was adopted. The loss criterion was the mean square error (MSE).

The performance of ASR is evaluated by the word error rate (WER), where a lower WER denotes better ASR performance. The calculations of two commonly used SE evaluation criteria, perceptual evaluation of speech quality (PESQ) [32] and short-time objective intelligibility (STOI) [33], are also provided for reference. Furthermore, we implement a comparison method, SNR-level [25], following the settings in a previous study with the floor of the clipping function set to zero [25].

Fig. 2 presents the predictions of the bridge module before clipping on the test dataset of LibriSpeech with DNS-Challenge. We calculate the averages and standard deviations of the predicted values for different SNRs. The predicted values positively correlate with the SNR levels, indicating that the bridge module can effectively model SNR variations in the input signals.

Table I summarizes the ASR performance of the different methods with CMGAN, MP-SENet, DEMUCS, and SEMamba on the test set of LibriSpeech with the DNS-challenge. It can be observed that the proposed technique outperforms the other methods under most conditions, proving that SE artifacts can be effectively mitigated by the bridge module. Compared to noisy signals, the proposed method can reduce the overall WERs for the Whisper base by 6.9% (from 32.1% to 29.9%) with CMGAN, 2.5% (from 32.1% to 31.3%) with MP-SENet, 9.3% (from 32.1% to 29.1%) with DEMUCS, and 7.5% (from 32.1% to 29.7%) with SEMamba. The overall WER reductions for Whisper large-v3 are 2.6% (from 15.4% to 15.0%) with CMGAN, 6.5% (from 15.4% to 14.4%) with DEMUCS, and 2.6% (from 15.4% to 15.0%) with SEMamba.

The WER reductions are particularly notable when compared to the enhanced signals. Specifically, CMGAN achieved WER reductions of 21.1% (from 37.9% to 29.9%) for Whisper base and 44.9% (from 27.2% to 15.0%) for Whisper large-v3; MP-SENet achieved reductions of 38.5% (from 50.9% to 31.3%) for Whisper base and 61.2% (from 39.9% to 15.5%) for Whisper large-v3; DEMUCS achieved reductions of 17.8% (from 35.4% to 29.1%) for Whisper base and 36.9% (from 22.8% to 14.4%) for Whisper large-v3; SEMamba achieved reductions of 20.8% (from 37.5% to 29.7%) for Whisper base and 37.8% (from 24.1% to 15.0%) for Whisper large-v3. Furthermore, we observe that the bridge module is more robust than the SNR-level method, confirming the importance of considering the inherent noise robustness of Whisper ASR models.

Note that speech signals enhanced by CMGAN, SEMamba, MP-SENet, and DEMUCS all obtain higher PESQ and STOI scores than their SNR-level and bridge module counterparts. These results suggest that acquiring the highest PESQ and STOI may not guarantee better recognition results when using Whisper ASR models.

Subsequently, experiments were conducted using two unseen datasets to evaluate the proposed bridge module. Table II lists the results of the Aurora-4 test set in which the proposed method performed best under all conditions. Compared to the use of noisy signals, the bridge module achieved WER reductions for Whisper base of 0.5% (from 22.1% to 22.0%) with CMGAN, 3.2% (from 22.1% to 21.4%) with MP-SENet, 1.8% (from 22.1% to 21.7%) with DEMUCS, and 3.2% (from 22.1% to 21.4%) with SEMamba. For Whisper large-v3, the WER reductions were 0.6% (from 17.3% to 17.2%) with CMGAN, 0.6% (from 17.3% to 17.2%) with MP-SENet, 1.2% (from 17.3% to 17.1%) with DEMUCS, and 0.6% (from 17.3% to 17.2%) with SEMamba. Similar to the results from the Librispeech+DNS Challenge, higher PESQ and STOI scores did not yield better recognition results when using Whisper ASR models.

Table III summarizes the results of the VoiceBank-DEMAND test set. Compared to noisy signals, the proposed method yielded WER reductions of 13.3% (from 9.0% to 7.8%) with CMGAN, 8.9% (from 9.0% to 8.2%) with MP-SENet, 5.6% (from 9.0% to 8.5%) with DEMUCS, and 15.6% (from 9.0% to 7.6%) with SEMamba. For Whisper large-v3, the WER reductions were 12.9% (from 3.1% to 2.7%) with CMGAN, 16.1% (from 3.1% to 2.6%) with MP-SENet, 6.5% (from 3.1% to 2.9%) with DEMUCS, and 12.9% (from 3.1% to 2.7%) with SEMamba. Enhanced speech with higher PESQ and STOI scores cannot guarantee better recognition results.