Leave No Knowledge Behind During Knowledge Distillation:
Towards Practical and Effective Knowledge Distillation for Code-Switching ASR Using Realistic Data

Code-switching ASR (CS-ASR) is a challenging sub-problem in automatic speech recognition. Unlike monolingual utterances, a code-switched utterance may contain transitions between languages at one or multiple positions. The problem is particularly pronounced in regions with more than one official language. Conventional methods address this issue by incorporating language identification (LID) [11, 12, 13, 14, 15], using monolingual data [16, 17, 18, 19, 20], or leveraging data augmentation techniques [21, 22]. Recent approaches focus on utilizing large language models (LLMs) to generate CS data or investigating large speech foundation models CS-ASR with different techniques [23, 24, 25]. Our work differs from those mentioned above by optimally utilizing large-scale unlabeled CS data collected from real-world scenarios.

Knowledge distillation (KD) [26] has been a common and effective method for model compression. In KD, a student model learns to mimic the behavior of a stronger teacher model to achieve similar performance. With the advent of powerful large foundation models, KD has been employed across multiple fields to obtain smaller and more efficient models [27, 28, 29, 30], further strengthening its effectiveness. Typically, the targets of KD can be categorized into two types: the teacher’s output and hidden representations. Distilling the teacher’s hidden representation is common when the task is not specific [27, 28, 29]. However, these methods are not tailored for CS-ASR and do not leverage large-scale unbalanced data effectively. In this work, we address these limitations by focusing specifically on CS-ASR in real-world scenarios and employing the learning objectives of Distil-Whisper [10], which utilize the teacher model’s outputs during knowledge distillation. Our approach optimally uses large-scale unlabeled CS data, enhancing the efficiency and performance of the distilled models for CS-ASR.

Inspired by the success of Whisper [7], Distil-Whisper [10] aims to provide a more efficient model through knowledge distillation using large-scale pseudo-labels. Distil-Whisper utilizes 21,170 hours of audio data from 9 publicly available datasets for KD. However, the datasets are monolingual and exclusively in English, which motivates us to develop our own distilled model for more effective generations in a code-switching context. Unlike Distil-Whisper, which utilizes data in labeled monolingual corpora for training, the data we collected is code-switched and has no true labels. Therefore, we cannot filter data based on the word error rate (WER) between the real and the pseudo-labeled transcriptions, as suggested in Distil-Whisper. To address the issue, we aim to provide a simple and effective method based on a small auxiliary model to perform cross-model validation for data filtering without any labeled data.

Semi-supervised learning and self-training focus on developing techniques to exploit the unlabeled data. In semi-supervised learning, the data can be separated into labeled and unlabeled ones. Utilizing the labeled data for supervised learning is trivial. Thus, most efforts are dedicated to leveraging the unlabeled data, which is often large-scale and noisy. One of the most common solutions is to use the supervised-learned model to generate pseudo-labels and then use the pseudo-labels for self-training [31]. Due to the model’s imperfections, directly leveraging all pseudo-labels may lead to slight improvement. To address the issue, previous works may use additional LM to improve pseudo-labels quality [4] or conduct data filtering [32, 33, 34, 35]. Moreover, recent works propose techniques like model ensembling, iterative pseudo-labeling, or continuous pseudo-labeling for self-training [4, 6, 36, 37]. These methods are more effective but often involve complicated algorithms during training, resulting in even higher computation resources.

Given a long-form realistic speech $X$ and the teacher model ${\rm M}_{\rm teacher}$, we first generate the corresponding transcriptions with timestamps $Y={\rm M}_{\rm teacher}(X)$. Specifically, we use Whisper [7] as the teacher model, which produces timestamps along with the transcriptions when performing sequential generation on the long-form audio. With an audio-transcription pair $(X,Y)$, we then generate $M$ segments based on the timestamps in $Y$. The segments can be formulated as $[(X^{\prime}_{1},Y^{\prime}_{1},d^{\prime}_{1}),(X^{\prime}_{2},Y^{\prime}_{2},d^{\prime}_{2}),\ldots,(X^{\prime}_{M},Y^{\prime}_{M},d^{\prime}_{M})]$, where each segment tuple $(X^{\prime}_{i},Y^{\prime}_{i},d^{\prime}_{i}),\forall i\in[1,M]$ represents the audio, transcription, and the duration of the $i$-th segment.

Next, we concatenate the adjacent segments into chunks with a maximum duration of 30 seconds based on $[d^{\prime}_{1},\ldots,d^{\prime}_{M}]$. We iterate through each segment in order, add the segment to the current chunk, and update the chunk’s duration, provided that doing so does not make the current chunk exceed 30 seconds. If adding a segment increases the duration of the chunk to more than 30 seconds, we finalize the current chunk and start a new chunk with the current segment. This process continues until all segments have been processed, ensuring that each chunk has a duration $\leq$ 30 seconds. We describe the chunked audio-transcription pairs as $[(\hat{X}_{1},\hat{Y}_{1}),\ldots,(\hat{X}_{N},\hat{Y}_{N})],N\leq M$.

Finally, let $f$ be a function that generates the set of chunked audio-transcription pairs given $X$; we can construct the realistic pseudo-labeled dataset based on the original dataset $\mathcal{X}$:

$$\mathcal{D}_{\rm RPL}=\bigcup_{\forall X\in\mathcal{X}}f(X),\quad f(X)=\{(\hat{X}_{i},\hat{Y}_{i})\mid i\in[1,N]\}.$$

We keep all the timestamp tokens in the transcription $\hat{Y}$ to facilitate segmentation behavior learning during knowledge distillation.

Due to the imperfection of the teacher model, the pseudo-labeled audio-transcription pairs $(\hat{X},\hat{Y})\in\mathcal{D}_{\rm RPL}$ may inevitably contain different levels of errors or hallucinations. As mentioned in the previous works [4, 10], one of the most common types of hallucination is looping or repetitive generation. The hallucination can be detected trivially by counting the n-gram according to [4]. Specifically, we may consider a pseudo-label $\hat{Y}$ as hallucinated if it contains a $n$-gram pattern for over $c$ times. We refer this detection as trivial oder $n$-gram method, denoted as follows:

$$h^{n}_{c}(\hat{Y})=\begin{cases}1,\quad&\text{if }\hat{Y}\text{ contains a $n$-gram pattern over $c$ times},\\ 0,\quad&\text{otherwise}.\end{cases}$$

By filtering out the pseudo-labels with the trivial hallucinations, we can form a new dataset:

$\displaystyle\mathcal{D}_{\rm trivial}=\{(\hat{X},\hat{Y})\in\mathcal{D}_{\rm RPL}\mid h^{n}_{c}(\hat{Y})=0\}.$

(1)

While being suitable for detecting the looping errors, the $n$-gram method might fall short for the other types of hallucinations.

In this work, we aim to provide a simple and effective method to validate the pseudo-labels by introducing a small auxiliary model ${\rm M}_{\rm validator}$ for validation. Specifically, we perform data filtering by validating across the outputs from the teacher model ${\rm M}_{\rm teacher}$ and the small auxiliary model ${\rm M}_{\rm validator}$. Given a pseudo-labeled audio-transcription pair $(\hat{X},\hat{Y})$, we first generate the validation-oriented transcription $\hat{V}={\rm M}_{\rm validator}(\hat{X})$ with the small model, and then perform cross-model validation between $\hat{Y}$ and $\hat{V}$. The cross-model validation is accomplished by calculating the distance metric $\delta$, accompanied by an additional threshold $\alpha\in[0.0,1.0)$ for the final determination. We formulate the filtered dataset after cross-model validation as follows:

$\displaystyle\mathcal{D}_{\rm validated}=\{(\hat{X},\hat{Y})\in\mathcal{D}_{\rm RPL}\mid\delta(\hat{Y},\hat{V})\leq\alpha\}.$

(2)

As presented, the distance metric $\delta$ is essential during filtering. We introduce the three different kinds of distance metrics we experiment with. First, we introduce the two direct measurements. We define the distance metric by directly calculating the mixed error rate (MER) between the teacher’s transcription $\hat{Y}$ and the validator’s transcription $\hat{V}$, or the phoneme error rate (PER) between the phonemicized transcriptions $\hat{Y}_{\rm p}$ and $\hat{V}_{\rm p}$:

Last but not least, we introduce the composite metric, which explicitly considers the trivial hallucinations of $\hat{Y}$ and $\hat{V}$:

$\displaystyle\delta_{\rm comp}=\max(h^{n}_{c}(\hat{Y}),\min(1-h^{n}_{c}(\hat{V}),\delta_{\rm PER}(\hat{Y}_{\rm p},\hat{V}_{\rm p}))).$

(5)

When $\hat{Y}$ is trivially hallucinated, we should directly discard the pseudo-labels; on the other hand, we should keep the pseudo-labels when $\hat{V}$ is trivially hallucinated.

In K²D, we conduct knowledge distillation on the cross-validated realistic dataset $\mathcal{D}_{\rm validated}$ (Eq. 2). Otherwise, the distillation process mainly follows Distil-Whisper [10]. First, we construct the student model ${\rm M}_{\rm student}$. The student model is an encoder-decoder model, in which we initialized parameters from the teacher model ${\rm M}_{\rm teacher}$. Given a pseudo-labeled audio-transcription pair $(\hat{X},\hat{Y})\in\mathcal{D}_{\rm validated}$, we can generate student’s prediction $\hat{S}=[\hat{s}_{1},\hat{s}_{2},\ldots,\hat{s}_{k}]$ based on ${\rm M}_{\rm student}$ and teacher forcing with $\hat{Y}$; and calculate the cross-entropy loss as follows:

$$\mathcal{L}_{\rm CE}=-\sum_{i=1}^{k}\log p(\hat{s}_{i}=\hat{y}_{i}\mid\hat{y}_{<i},\hat{X}),$$

where $p$ indicates the probability function. Moreover, we generate the prediction distribution $\hat{Q}={\rm M}_{\rm teacher}(\hat{X},\hat{Y})$. The distribution $\hat{Q}=\hat{q}_{1:k}$ then served as the soft-label during knowledge distillation. We use the soft-label $\hat{Q}$ from the teacher model and the probability distribution $\hat{P}=p(\hat{S})=\hat{p}_{1:k}$ to calculate the Kullback-Leibler (KL) divergence loss between $\hat{Q}$ and $\hat{P}$:

$$\mathcal{L}_{\rm KL}=\sum_{i=1}^{k}{\rm KL}(\hat{q}_{i},\hat{p}_{i}).$$

Finally, we combine the cross-entropy loss and the KL divergence loss. The overall objective is defined as follows:

$\displaystyle\mathcal{L}_{\rm KD}=\beta\mathcal{L}_{\rm CE}+\gamma\mathcal{L}_{\rm KL},$

(6)

where $\beta$ and $\gamma$ are the weighted coefficients.

As mentioned in Section 1, we use self-collected data for training. The data is collected from the course recordings of National Taiwan University, resulting in about 60,000 hours of audio. The courses can be categorized into different subjects, and we present the time distribution of the top six subjects with the most extended durations in Table 1. As for the testing sets, we introduce them as follows:

COOLTEST represents our own in-domain and in-house testing set. The testing set contains 5 hours of speech from different subjects. We self-annotated the testing data by revising the transcriptions generated by Whisper Large-v2.

NTUML2021 is a speech dataset from National Taiwan University’s 2021 "Machine Learning" course. Since the dataset is derived from lecture videos, we use the dataset as a publicly available in-domain Mandarin-English CS-ASR corpus. We use the 9-hour testing split to perform the evaluation.

CommonVoice is a publicly available, multilingual dataset of voice recordings collected by Mozilla for ASR [38]. It includes contributions from volunteers worldwide, providing a diverse range of accents, languages, and demographic variations. The diversity of the corpora makes it suitable for serving as the out-of-domain evaluation set. We evaluate our method on the $16$-th version of CommonVoice with the zh-TW split flag, indicating the speech collected from Taiwan.

ASCEND [39] is a publicly available, spontaneous, Mandarin-English CS dataset collected from conversational recordings. Furthermore, it is collected in Hong Kong, which differs from the chosen CommonVoice testing split mentioned earlier. The dataset, which is code-switched and realistic, is suitable for serving as out-of-domain evaluation of our method.

In K²D, we have three kinds of models: the teacher model ${\rm M}_{\rm teacher}$, the small validation model ${\rm M}_{\rm validator}$, and the student model ${\rm M}_{\rm student}$. Practically, we use Whisper Large-v2 as ${\rm M}_{\rm teacher}$, and Whisper Base as ${\rm M}_{\rm validator}$. The validator model is over 20 times smaller than the teacher model, facilitating the fast validation-oriented transcription generation on large-scale realistic data. On the other hand, the student model comprises 32 encoder layers and two decoder layers and is two times smaller than the teacher model. The parameter initialization follows Distil-Whisper, where the encoder is identical to the teacher model’s; and the 2-layer decoder is initialized from the first and the last layer of the teacher’s decoder. We perform knowledge distillation on 4 NVIDIA H100. We use batch size 256 and update for 120,000 steps, which takes about 42 hours of training. The encoder is frozen during training. We set the threshold $\alpha=0.4$ in Eq. 2 for cross-model validation; while the weighted coefficients $\beta$ and $\gamma$ of $\mathcal{L}_{\rm KD}$ in Eq. 6 are set to $0.8$ and $1.0$. We use g2p [40] for English and pinyin for Mandarin phonemicization.

We use the mixed error rate (MER) to evaluate the quality of code-switching ASR. The metric gathers the two common evaluation metrics in the two languages: the character error rate (CER) in Mandarin and the word error rate (WER) in English. We perform the long-form evaluation if not specified to simulate the realistic scenario. Moreover, we use the real-time factor (RTF) under the same computing resource to measure the speed-up brought by the student model.

We present the MER of each method on all the in-domain and out-of-domain testing sets in Table 2. As the table indicates, K²D provides clear performance improvements on all the testing sets. Our method with the composite distance metric $\delta_{\rm comp}$ (Eq. 5) produces improvements for over 17% reduction rate on both of the in-domain testing sets. Furthermore, it demonstrates strong generalizability to different domains, especially to ASCEND, a publicly available code-switching realistic dataset. On the other hand, using the direct distance metrics, ${\rm MER}$ and ${\rm PER}$, give performance improvements as well. Note that the data usage of the two direct metrics is around $60\%$, which is much less than the baseline methods. The result further strengthens the effectiveness of our method. In K²D, we can perform simple and efficient data pre-filtering through cross-model validation, leveraging the knowledge from both the large and small models. Last but not least, our method achieves a five times faster generation speed than the teacher model.

Next, we discuss the performance improvements our method brings to the two languages separately. We show the detailed evaluations of the CER for the Mandarin parts and the WER for the English parts on the code-switching datasets, COOLTEST and ASCEND, in Table 3. We find out that using the direct MER as the distance metric produces the most improvements over English on both testing sets. In comparison, the composite metric yields the best performance in Mandarin. Applying K²D can improve ASR performance for both languages across both the in-domain and out-of-domain testing sets.

As previously mentioned, repetitive hallucinations can be effectively detected using the $n$-gram method. In Table 4, we present the evaluation results and the counts of hallucinations identified by this method. We observe a significant reduction in repetitive hallucinations when data filtering is applied. Even with direct methods, the number of repetitive hallucinations drops substantially compared to using the entire dataset. The composite metric yields the fewest hallucinations, likely due to its combination of the $n$-gram and distance metrics, which leverages the strengths of both approaches during filtering. The deletion rate is also lowest with the composite metric, probably due to the reduced number of repetitive hallucinations.

As we have mentioned in Section 4.2 We use Whisper-Base as the validator to provide additional knowledge during cross-model validation. In this analysis, we wish to communicate that utilizing a smaller model for validation may provide efficiency, effectiveness, and robustness when the data is large-scale and realistic. First, we show that our method is efficient by comparing the time cost between using different variants of Whisper models as the ${\rm M}_{\rm validator}$. As shown in the table, Whisper-Base takes only 9 hours on 4 NVIDIA H100 GPUs, while Whisper-Medium takes over 30 times longer.

Next, we show that our proposed filtering method is effective in filtering out pseudo-labels with high error rates. We first define that a pseudo-label has a high error rate if its $\text{MER}>0.4$. We can then calculate the filter-out rate of the high-MER pseudo-labels for each cross-model validation method, which is composed of a ${\rm M}_{\rm validator}$ and a distance metric $\delta$. The measurement of the filter-out rate is denoted as the recall in Table 5. The Max Recall indicates the maximum filter-out rate of a valid $\alpha$ that yields over 50% of the data remaining in each cross-model validation method. Our results show that even with Whisper-Base, the max recall is similar to using the larger ones. This illustrates that we can achieve effective filtering on high-MER pseudo-labels under a reasonable filtering rate.

Finally, we highlight that using Whisper-Base as the validation model is robust over the selection of $\alpha$. By calculating the average recall (Avg. Recall) across different $\alpha\in[0.1,0.2,\ldots,0.9]$, we find out that using Whisper-Base yields the highest average recall. This shows that the cross-model validation with Whisper-Base is less sensitive to the selection of $\alpha$, enhancing the robustness of our method.