Improving Self-supervised Pre-training using Accent-Specific Codebooks

Self-supervised learning (SSL) has been established as a powerful technique to learn representations for speech and language processing [layerwise_analysis]. With pretrained SSL models as a starting point, even small amounts of labeled data are sufficient to achieve commercially acceptable results in various downstream speech tasks [baevski2020wav2vec, zhao2022improving]. This has led to the wide adoption of the current defacto training regime of pretraining a model with an SSL objective, followed by fine-tuning on downstream tasks such as automatic speech recognition (ASR) and spoken language understanding [yang21c_interspeech, tsai-etal-2022-superb]. A key limitation of SSL-based models is that performance on the downstream task suffers if there is a domain shift from the pretraining data [hsu2021robust]. In this work, we focus on speech accents as our domain of interest. The problem statement can be outlined as - How can we make SSL-trained models more robust to varying speech accents, both seen and unseen during training?

While prior work has extensively explored improving accented ASR in the fine-tuning stage [asr_residual, jain2018improved, winata2020learning, accentgrapheme, layer_accnt, e2emultitask], limited work has gone into improving the robustness of SSL pretraining to varying speech accents. Prior work on accent adaptation of SSL models includes introducing an additional classifier into SSL pretraining [deng21b_interspeech], employing accent-specific adapters [Bhatia2023], and normalizing pseudo-targets to a single accent [Poncelet2023UnsupervisedAA].

In this work, we adapt the technique introduced in Prabhu et al. [Prabhu2023AccentedSR] for SSL pretraining. We introduce accent information during self-supervised pre-training via a set of accent-specific codebooks. These codebooks are integrated into the model using a cross-attention module. For each accent seen in the training data, we introduce a learnable codebook consisting of vectors that capture accent-specific information during the pre-training process. While [Prabhu2023AccentedSR] used this technique exclusively during ASR finetuning, we show that the benefits increase when used during the SSL stage. To evaluate the effectiveness of our method, we show ASR experiments on the multi-accented Mozilla Common Voice corpus and show significant word error rate (WER) reductions compared to multiple variants of HuBERT and other accent adaptation techniques including an accent-agnostic approach (Domain Adversarial Training (DAT) [bobw]) and an accent-aware approach (MultiTask Learning (MTL) [asr_clf]).

While many self-supervised models exist for speech-related tasks [baevski2020wav2vec, chen2022wavlm, chung2021w2v, discretebert, decoar2, vqwav2vec, hubert], in this work we use HuBERT — a state-of-the-art architecture that uses a masked language modeling (MLM) objective and aims to reconstruct a sequence of discrete pseudo-targets derived via a $k$-means clustering step. The clustering step in HuBERT is used to generate pseudo-targets for pretraining. Prior work has aimed to improve the quality of these generated pseudo-targets either through teacher-forcing [hubert_teacher] or by designing task-specific pseudo-targets [ssldeepcluster]. Other efforts have focused on improving the HuBERT architecture itself [fan2023ctcbert, jointssldecoder]. To the best of our knowledge, adapting HuBERT pretraining to be more robust across accents has not been attempted before.

In summary, our main contributions are as follows:

• •

  We introduce codebook-based accent adaptation for self-supervised pre-training that aims at using a collection of accent-specific codebooks and cross-attention to improve SSL representations in the presence of accents.¹¹1Code is available at https://github.com/csalt-research/accented-codebooks-asr/tree/accented-pretraining. Our model outperforms baselines and all other previous approaches and achieves a significant improvement in performance with up to 9% relative WER reduction on the MCV dataset.

• •

  In a zero-shot setting on the L2-Arctic dataset, we show that our codebook-based pretrained model significantly outperforms baselines, thus demonstrating the generalization capability of our proposed system.

Figure 1 illustrates the main workflow of our proposed technique. We adopt the standard two-stage training pipeline used in self-supervised architectures:

1. 1.

   A self-supervised pretraining stage that involves training an encoder using a self-supervised learning objective [mohamed2022self].

2. 2.

   A supervised ASR fine-tuning stage that uses the pretrained encoder from the previous stage and further finetunes it using an ASR-specific supervised loss.

In our proposed framework, both these stages benefit from accent-specific codebooks elaborated in Sections $\S$2.1 and $\S$2.2, respectively.

We conduct all our pre-training and fine-tuning experiments using Fairseq [fairseq] and ESPnet [espnet] toolkits, respectively, on NVIDIA RTX A6000 GPUs.

Table 1 compares the performance (WER %) of our proposed system against four alternative approaches: 1. Conformer baseline [conformer] 2. Replacing the encoder in the Conformer with a pre-trained HuBERT model [hubert], pretrained with the SSL objective on data Mcv-Accent-600 3. Jointly training HuBERT baseline with an accent classifier (MTL) [asr_clf] 4. Domain Adversarial Training (DAT) of the HuBERT baseline with an accent classifier on the $12^{\text{th}}$ encoder layer.

Similar to previous studies [baevski2020wav2vec, hubert], we find that replacing Conformer’s encoder with a pre-trained HuBERT model leads to significant improvement in performance; we note here that the HuBERT-based encoder is pretrained from scratch on Mcv-Accent-600. Furthermore, we observe that, during self-supervised pre-training, using an existing checkpoint (i.e., the HUBERT-Base Librispeech checkpoint from Fairseq [fairseq]) leads to additional performance gains. Given the limited amount of supervised finetuning data in Mcv-Accent-100, to combat overfitting, we freeze the feature extractor and the first $3$ layers of the HuBERT encoder; this yields further benefits. (In all subsequent experiments, we will use this best setup of selective ASR finetuning.) We also compare our system against the MTL and DAT approaches. Overall, our proposed approach performs significantly better on nearly all accents, particularly the unseen accents.

In this work, we propose an accent-aware ASR adaptation technique where accent-specific codebooks are incorporated within the Transformer layers of a HuBERT model via cross-attention. This integration happens right from the SSL-based pretraining stage. The pretrained codebooks and encoder layers are further finetuned using supervised ASR fine-tuning. Compared to existing accent adaptation techniques, we observe that this yields significant WER reductions on English utterances in both seen and unseen accents in the Mozilla Common Voice (MCV) corpus. The accent-aware models trained on MCV also generalize well to out-of-domain accented English samples (from a different corpus, L2Arctic) when evaluated in a zero-shot setting. In future work, we aim to use self-training with unlabeled data (with accent labels) to further refine the accent codebooks.

We acknowledge the financial support from a SERB Core Research Grant, Department of Science and Technology, Government of India on accented speech processing.