Leveraging ASR Pretrained Conformers for
Speaker Verification through
Transfer Learning and Knowledge Distillation

Several studies have explored the application of self-supervised pretrained Transformers for speaker verification tasks. Fan et al. [25], and Vaessen et al. [26] adopted a direct fine-tuning approach on the pretrained model by incorporating an additional pooling layer on top of the model’s output. However, this method did not surpass the performance of CNN- or TDNN-based speaker verification models, which typically have fewer parameters than the pretrained Transformer. Novoselov et al. [27] fine-tuned wav2vec 2.0 by integrating two simple TDNN layers and a statistic pooling layer. Their findings suggested that utilizing the entire deep pretrained encoder architecture was unnecessary, as earlier layers potentially provided more speaker information.

Another prevalent method replaces the handcrafted feature with the pretrained frame-level feature to train TDNN- or CNN-based speaker embedding networks [28, 29]. This approach, employing a layer-wise weighted average to aggregate features from different Transformer layers, has improved performance over models using handcrafted spectral features. However, this comes at the cost of using a large number of pretrained parameters alongside a full TDNN- or CNN-based speaker embedding network. Expanding on the concept of layer-wise weighted average as a feature aggregation method, Peng et al. [30] proposed multi-head factorized attentive pooling, which can be viewed as a fusion of layer-wise weighted average and multi-head attentive pooling.

In this paper, instead of self-supervised pretrained Transformers, an ASR-pretrained Conformer is used as the network backbone for the speaker embedding network since there are already many large-scale publicly open ASR datasets available. We directly apply to fine-tune the pretrained Conformer with a multi-scale feature aggregation module, eliminating the need for an additional TDNN- or CNN-based speaker network. This transfer learning strategy allows the knowledge learned from ASR to be effectively transferred to speaker verification tasks.

ASR or phonetic information plays an essential role in speaker verification. In the statistical i-vector framework, substituting a Gaussian mixture model (GMM) with an ASR-trained DNN to gather sufficient statistics for i-vector extraction results in significant performance improvement [23, 31]. Alternatively, some researchers utilize a tandem feature that merges spectral and ASR-derived features for GMM modeling [22, 32].

In the realm of deep learning, the integration of ASR and phoneme information into speaker verification is gaining increasing attention. Three main strategies have been investigated for such integration, each with merits and challenges.

The first strategy involves applying frame-level phonetic features from an ASR to a speaker verification model. In this context, Rahman et al. used bottleneck phonetic features from an ASR acoustic model to replace spectral features in speaker network training, indicating the potential of phonetic features to carry speaker-specific information [33]. In similar efforts, researchers have also incorporated phonetic features alongside spectral features for speaker modeling. Zheng et al. used separate network stems to model these two types of features [34], while Zhou et al. processed these features jointly by concatenating them [21]. Depending on the modeling stage, phonetic features can be incorporated at the input of the speaker network [34, 21] or before the pooling layer [21, 35]. These research indicate that incorporating auxiliary phoneme information benefits speaker modeling. Besides, Chen et al. proposed to model speaker characteristics in phoneme units, termed as phoneme-unit-specific network [36]. This method can be considered as modeling speaker characteristics using multi-phonetic-head attention, which has the attention weight of phoneme posterior probability.

The second strategy employs a multi-task learning approach, leveraging phoneme recognition as an auxiliary task alongside the primary task of speaker recognition. Studies have shown that frame-level phoneme modeling enhances speaker verification performance [35, 37, 38].

The last strategy involves employing phonetic information as a guided signal to be removed from speaker modeling. A study by Wang et al. suggested that adversarial training to remove phonetic information at the segment level can boost speaker verification performance [38]. In contrast, Tawara et al. found that removing phonetic information at the frame level is beneficial for extremely short utterances of less than 1.4 seconds [39]. Hong et al. introduced a self-constraint learning and reconstruction strategy that eliminates phonetic information in lower layers, thereby allowing subsequent layers to capture speaker-specific features more efficiently [40].

In our study, we extend the benefits of the second approach through knowledge distillation from the ASR Conformer to the speaker verification model. This method aligns with the recognized advantages of employing phoneme recognition as an auxiliary task, thus aiming to improve speaker verification performance.

The concept of adaptors stems from the idea of fine-tuning large pre-trained models using lightweight neural modules, which can be considered a parameter-efficient transfer learning technique [41]. This approach incorporates trainable lightweight neural modules into a large pre-trained model while keeping the pre-trained parameters frozen during fine-tuning. This technique has seen successful applications across various domains, including computer vision [42], natural language processing [41, 43], and machine translation [44].

While adaptors have been successful in different domains, their integration into speech-processing tasks presents multiple applications. For example, adaptors are applied to self-supervised pre-trained models for speech recognition [45]. In the context of multilingual ASR, language-specific adaptors have been used to adapt a pre-trained ASR model to various languages [46, 47]. In speech translation, adaptors enable a pre-trained model to specialize in specific language pairs [48]. Additionally, adaptors have been employed to connect an ASR encoder with a multilingual denoising auto-encoder for multilingual speech translation [48]. Other applications of adaptors include speaker verification [49, 50] and other speech processing tasks [50].

Most existing applications of adaptors focus on self-supervised pre-trained models for specific downstream tasks [41, 43, 49, 50]. Moreover, adaptors have been employed to perform domain adaptation for the same task, as seen in multilingual ASR [46, 47] and multilingual speech translation [48]. These methods usually incorporate adaptor modules within the network architecture, altering the output of the pre-trained model.

In contrast, our study motivated from the application of the adaptor mechanism. We apply a similar idea to transfer knowledge across different tasks: from ASR to speaker verification. We uniquely position an adaptation module on top of the original model, ensuring that the output of the ASR Conformer remains unchanged. This design enables the simultaneous execution of ASR and speaker verification tasks within a single Conformer model.

Developed primarily for ASR tasks, the Conformer encoder is adept at modeling both local and global dependencies within speech signals [17]. It improves upon the Transformer encoder [16] by incorporating a CNN to capture local spectral feature information. The Conformer consists of a convolutional subsampling layer, which reduces the length of input sequences, and a series of Conformer blocks that transform the input signal into higher-level representations. Fig. 1 presents the Conformer encoder structure.

A Conformer block consists of two feed-forward networks (FFNs) flanked by a multi-head self-attention (MHSA) module and a convolution (Conv) module. In the Conformer, the MHSA employs relative sinusoidal positional encoding [51], allowing for efficient sequence handling at unseen lengths. The convolutional module features a point-wise convolution followed by a gated linear unit, succeeded by a one-dimensional depthwise convolution. Batch normalization and Swish activation are subsequently applied. The feed-forward network contains two linear layers separated by a nonlinear activation, with dropout applied after each linear transformation. As illustrated in Fig. 1, residual connections are used between the modules, while half-step residual connections are utilized within feed-forward modules, akin to a Macaron-Net [52]. Layer normalization is applied prior to the output. Mathematically, for a given input $\mathbf{h}_{i-1}\in\mathbb{R}^{d\times T}$, the output $\mathbf{h}_{i}\in\mathbb{R}^{d\times T}$ of the $i$-th Conformer block is represented as follows:

where $d$ denotes the dimension of the input and the output sequences, and $T$ represents the length of the time sequence.

Multi-scale feature aggregation (MFA) is a technique that concatenates output feature maps from all frame-level modeling modules in a speaker embedding network before utterance-level pooling. This approach has been shown to improve performance for TDNN-based networks, suggesting that lower-level features can contribute useful speaker information [6].

To apply the Conformer encoder in the speaker verification task, MFA-Conformer proposed to integrate an MFA module into the Conformer encoder [18]. Specifically, this MFA module concatenates the frame-level outputs from all Conformer blocks prior to the pooling layer:

where $L$ is the number of Conformer blocks in the Conformer encoder, and $\mathbf{H},\mathbf{H}^{\prime}\in\mathbb{R}^{D\times T}$ with $D=L\times d$.

With this concatenated frame-level feature map $\mathbf{H}$, attentive statistics pooling is applied to produce an utterance-level representation [53]. Finally, the speaker embedding is extracted by applying batch normalization and a fully-connected layer to this utterance-level representation. During training, an additional fully-connected layer is applied to classify speakers in the training set from speaker embeddings.

While deeper Transformers are known to yield superior results as more training data become available [29, 54], training these models from scratch often requires large datasets [55]. Further, research indicates that increasing the number of layers in Conformer architectures can result in a performance drop in speaker verification tasks, suggesting potential issues of overfitting [18].

To mitigate the risks of overfitting, we employ an ASR pretrained Conformer to initialize the MFA-Conformer-based speaker embedding network. The pretraining on ASR tasks affords several advantages, such as faster convergence and enhanced generalization capabilities in the speaker verification domain.

In our approach, the parameters of the ASR pretrained Conformer encoder are used to initialize the MFA-Conformer speaker embedding network. During the early training phases, we keep these encoder parameters frozen and allow only the pooling and subsequent linear layers to be updated for a few epochs. In later stages, we proceed to fine-tune the parameters across the entire MFA-Conformer architecture to better align it with the specific needs of speaker verification. By limiting updates to the pooling and linear layers initially, these layers are tailored to adapt the frame-level feature maps derived from the ASR model to the speaker verification objective. This structured training approach ensures that the pretrained Conformer transitions smoothly to the speaker verification objective without being significantly disrupted by the random initialization of these layers.

Knowledge distillation involves training a “student” model to reproduce the behavior of a more complex “teacher” model [24]. In our setting, an ASR pretrained Conformer acts as the teacher model, guiding the learning process of the MFA-Conformer-based speaker verification model, which serves as the student.

Given a speaker recognition dataset $\mathcal{D}$, the objective of a speaker verification model is to minimize the difference between its predictions and the ground-truth speaker labels. The loss function $L_{\mathrm{spk}}$ can be expressed as:

Here, $f(\cdot)$ is the MFA-Conformer speaker verification model, $f({\mathbf{x}})$ is the Conformer’s prediction for the input spectral sequence ${\mathbf{x}}$, and $y$ is the speaker label. The speaker classification loss $\ell_{\mathrm{spk}}$ commonly adopts a cross-entropy format or an angular-softmax variant [7].

For distillation, the speaker MFA-Conformer student is trained to align its outputs with the ASR teacher model, as described in the loss $L_{\mathrm{distill}}$:

In this setting, $f_{\mathrm{student}}(\cdot)$ refers to the MFA-Conformer coupled with an ASR decoder, while $f_{\mathrm{teacher}}(\cdot)$ is the ASR model. In the distillation process, the loss function $L_{\mathrm{distill}}$ is formulated based on the Kullback-Leibler (KL) divergence, which quantify the divergence between the student and teacher frame-level logits outputs.

The ultimate training objective combines both the speaker classification and the distillation losses:

where $\alpha$ is a hyperparameter determining the strength of the distillation effect. Fig. 2 illustrates the knowledge distillation process from ASR to speaker verification.

Our approach harnesses the strengths of both knowledge distillation and multi-task learning, offering advantages for speaker verification. Firstly, it enables the speaker verification model to utilize robust feature representations from an ASR-pretrained model, enhancing performance without extensive ASR data. This method, diverging from traditional knowledge distillation, incorporates the ASR model’s outputs as an auxiliary objective, enriching phonetic feature learning in a multi-task framework. Secondly, this synergy improves speaker discrimination by leveraging nuanced phonetic information. Lastly, our method with knowledge distillation offers more architectural flexibility, allowing for optimized designs that can cater to the specific requirements of both ASR and speaker verification tasks.

To leverage the versatility of Conformer encoders across multiple tasks, this section explores the possibility of crafting a unified model that serves both ASR and speaker verification objectives.

The experiments are conducted on VoxCeleb [56, 57]. For model training, we opted to employ the development set from VoxCeleb 2. This training dataset encompasses 1,092,009 audio recordings from a diverse set of 5,994 distinct speakers.

For the evaluation phase, we use both the development and test sets from VoxCeleb 1. We present the speaker verification performances based on three predefined trial lists as described in [57]:

• •

  VoxCeleb 1-O: This represents the original trial list associated with VoxCeleb 1, encompassing 37,720 trials derived from 40 speakers.

• •

  VoxCeleb 1-E: An expanded trial list that comprises 581,480 trials sourced from 1,251 speakers.

• •

  VoxCeleb 1-H: A more challenging trial list with 552,536 trials from 1,190 speakers. All test pairings within this list share the same linguistic background and gender.

To enhance the robustness and versatility of our model, we integrated various data augmentation methodologies. First, we apply speed perturbation to the audio samples by accelerating or decelerating the content by factors of 1.1 and 0.9, respectively [58, 59]. As a result, this approach produced two supplementary replicas of each original audio, expanding the entire training dataset to include 17,982 distinct speakers and 3,276,027 unique utterances.

For the enlarged training dataset, two primary strategies were utilized:

• •

  Additive noise augmentation: The MUSAN dataset [60] served as our noise source, enabling us to add ambient noise, musical sounds, and babble noise onto our audio files. The babble noise was generated by merging between three to eight separate speech files in the MUSAN dataset. The signal-to-noise ratios (SNR) range from 0 to 20 dB.

• •

  Convolutional reverberation noise augmentation: We employed the collection of 40,000 simulated room impulse responses (RIR) from the study in [61]. Only simulated RIRs originating from small to medium-sized rooms are used.

To maintain variability during training epochs, we integrated on-the-fly data augmentation, applying the aforementioned noise augmentations with a likelihood of 0.6 for each training speech.

We utilize pretrained ASR models from the NEMO toolkit [62]. The choice of the Conformer model from NEMO was driven by its performance and generalization capabilities, as demonstrated in various benchmarks. This ASR Conformer adopts the same encoder architecture as illustrated in [17] but uses a linear decoder and the connectionist temporal classification (CTC) for decoding.

In our experiments, we use three sizes of the NEMO ASR Conformer: small, medium, and large. Despite variations in size, each of these models shares a convolution subsampling rate of $\frac{1}{4}$, along with a consistent kernel size of 31 for their convolution modules. Table I shows the differences in Conformer layer numbers, encoder dimensions, attention heads, and linear hidden units across the three Conformer encoders.

According to the NEMO toolkit documentation, each Conformer-CTC model is trained on English corpora collated from 10 distinct datasets.⁴⁴4These datasets include Librispeech, Fisher Corpus, Switchboard-1, WSJ-0 and WSJ-1, National Speech Corpus (Part 1, Part 6), VCTK, VoxPopuli (EN), Europarl-ASR (EN), Multilingual Librispeech (MLS EN 2,000 hours subset), and Mozilla Common Voice (v7.0). In total, this collection spans approximately 10,000 hours of speech data.⁵⁵5This estimate is derived from the training data descriptions provided at the mentioned link in the previous footnote.

Speech utterances are cropped to 2 seconds for training the speaker embedding network. We use a logarithmic Mel-spectrogram with 80 frequency bins as the acoustic feature, computed over Hamming windows of 20ms with a 10ms shift.

During training, the Additive angular margin (AAM) loss [7] is employed with a re-scaling factor of 32 and an angular margin of 0.2 to learn discriminative representations. The speaker embedding dimension is set to 256. We utilize the AdamW optimizer, beginning with a learning rate of 0.001. Additionally, we implement a cosine annealing learning rate scheduler, incorporating a warm-up phase spanning one training epoch. Our chosen batch size is 512, with a weight decay of $10^{-7}$.

After convergence, we employ large margin fine-tuning (LMFT) [11]. Speech segments are expanded to 6 seconds, and the angular margin in the AAM loss is increased to 0.5. We turn off speed perturbation data augmentation, reverting the training data to its original set.

To generate speaker verification scores, we apply the adapted score normalization [63] after cosine similarity on two given speaker embeddings. In adapted score normalization, we utilize an imposter cohort randomly chosen from 30,000 training utterances, with an adapted cohort size of 700.

Although our standard procedure involves only this score normalization, we further calibrate the verification scores using the Quality Measure Function (QMF) [64, 11] for specific systems as per their requirements. The calibration model is trained on 30,000 trials generated from the VoxCeleb 2 development set. This model incorporates several quality metrics including the duration and SNR of the enrollment and testing utterances, the magnitudes of the embeddings, and the verification score itself.

We evaluate speaker verification performance using two metrics: (1) Equal Error Rate (EER): This denotes the error rate at the point where the false acceptance rate equals the false rejection rate. (2) Minimum Detection Cost (minDCF): This represents the minimal value of a detection cost function. The function is a weighted sum of false-reject and false-alarm error rates for a given decision threshold [65]. The parameters for this function are set as follows: $C_{\mathrm{Miss}}=1$, $C_{\mathrm{FA}}=1$, and $P_{\mathrm{Target}}=0.01$.

In this subsection, we present speaker verification results using our first proposed method. Specifically, we explore the efficacy of initializing the MFA-Conformer speaker verification model with a pretrained ASR Conformer. The performance of various MFA-Conformer speaker embedding networks, both with and without ASR pretraining, are detailed in Table II.

This section presents the results of our second proposal, which explores the application of knowledge distillation from ASR to speaker verification. For these experiments, we used the NEMO Large ASR-CTC model in Table I, as the teacher model in the knowledge distillation process. We set the hyperparameter $\alpha$ in equation 5 to 1. The speaker verification performance of various MFA-Conformer models, considering different training methodologies and model sizes, are shown in Table IV.

In this section, we delve into the effectiveness of our proposed speaker adaptation approach in bridging the gap between ASR and speaker verification tasks.

We employ three pretrained ASR Conformer encoders — small, medium, and large, as referenced in Table I. These encoders are integrated with our speaker adaptation technique. For each encoder size, we assess three distinct configurations of the speaker adaptation module, based on the one depicted in Fig. 4:

• •

  V1: This version extracts directly from the first $L$ layers of the ASR Conformer without the intervention of layer adaptors.

• •

  V2: In alignment with Fig. 4a, this version integrates layer adaptors to refine the outputs from the first $L$ ASR Conformer layers. Subsequently, $K$ lightweight Conformer layers process the frame-level outputs derived from the $L$-th ASR Conformer layer.

• •

  V3: As illustrated in Fig. 4b, this configuration feeds the $K$ lightweight Conformer layers with a concatenated output from the first $L$ Conformer layers of the pretrained ASR model. An auxiliary linear layer ensures the alignment of concatenated feature dimensions.

All configurations use a lightweight Conformer layer architecture consistent across the ASR encoders. These lightweight Conformer layers have 174 dimensions, 704 hidden units, and 4 attention heads, the same as the Conformer layer configuration in the NEMO Small ASR-CTC model. Additionally, the layer adaptor always maps the frame-level outputs from ASR Conformer layers to a 128-dimensional feature space. A detailed architecture configuration can be found in Table V. Notably, when the model lacks trainable Conformer layers (i.e., $K=0$), the V2 and V3 configurations converge to become identical. The specific EERs for distinct model configurations, considering variations in both $L$ and $K$, are outlined in Tables 8, VIII, and IX, each corresponding to a unique pretrained ASR model.