Empowering Whisper as a Joint Multi-Talker and Target-Talker
Speech Recognition System

Whisper is a speech recognition model featuring an attention-based encoder-decoder structure, which has been trained on massive amounts of web-scale labeled speech data [1]. Nowadays, it is increasingly being utilized as a speech foundation model even beyond speech recognition [24, 30, 31]. In this study, we are inspired to extend its capability to handling joint multi-talker and target-talker ASR tasks.

Whisper takes log-Mel spectrogram as input, followed by transformer encoder and decoder modules to decode the output tokens in an auto-regressive manner. Different from other ASR models, Whisper adopts several special tokens as the prefix of input sequences for decoder to specify tasks and condition information. By default, the prefix tokens are "<|PREV|>, text prompt, <|SOT|>, <|LANGUAGE|>, <|TRANSCRIBE|>, <|NO_TIMESTAMP|>", where <|PREV|> and text prompt are optional.

Recently, the Sidecar separator (SS) has been introduced as a parameter-efficient module to convert a well-trained single-talker ASR model into a multi-talker one [20]. In this work, we incorporated the Sidecar separator with Whisper to harness its capability acquired from extensive training data.

The Sidecar separator is a temporal convolutional network inserted between the early layers of the ASR encoders. It consists of stacked 1-D dilated convolutional blocks inspired by Conv-TasNet [19]. As the shallower layers of the ASR encoder are believed to encode more acoustic information rather than the linguistic ones[17, 20], the Sidecar separator is able to separate the mixed representation with talker-related masks, producing disentangled representation of different speakers.

As depicted in Figure 1, the Sidecar separator accompanied by two 1-D convolutional layers is positioned after the second encoder block. Talker-dependent masks are generated, which are element-wisely multiplied with the mixed embedding, yielding separated embeddings of each talker. The subsequent encoder blocks and decoder process these branches, ultimately transcribing the corresponding text for each talker.

We introduce the Target Talker Identifier (TTI) module, which equips the system with the capability for target-talker ASR.

During the forward process, as illustrated in Figure 1 (b), the encoder-output embeddings corresponding to different talkers will be segmented into two distinct segments: a prefix segment aligning with the length of three-second enrollment speech, and a main segment that corresponds to the duration of the multi-talker speech. The prefix segments are then fed into the TTI module, which determines the branch associated with the target talker, while only the main embedding segments are sent to the Whisper decoder for transcription.

Specifically, the prefix segments hold a tensor shape of $(B\times S,\textit{150},C)$, where $B$ denotes batch size, $S$ denotes the number of talkers, $C$ denotes the number of channels, and 150 denotes the number of time frames. Given that each time frame spans a duration of 20 ms, 150 coincides with the three-second duration of the enrollment speech. As shown in Figure 1, within the TTI module, the prefix segments traverse a linear layer followed by the ReLU activation function, yielding a tensor with a shape of $(B\times S,\textit{150},\textit{1})$. Upon squeezing and reshaping, the tensor proceeds through another linear layer and the softmax function to produce the probability $(B,S)$ of each branch being the target talker.

Consequently, the target talker branch is efficiently determined on the fly, introducing minimal computational overhead. Underpinned by the superior performance of the separation module, TTI can be considered as performing target-talker activity detection, which is a more economical task compared with methods necessitating speaker embedding extraction.

The original Whisper model allows for the inclusion of text prompt tokens as prefix to the decoder’s input sequences, conditioned on which the model yields improved ASR performance on ambiguous audios [1]. In this context, by exploiting this inherent characteristic of Whisper combined with soft prompt tuning technique [32], we aim to adapt the model more efficiently to multi-talker and target-talker ASR tasks.

Specifically, as shown in Figure 1 (a), we insert a learnable embedding as soft prompt between <|PREV|> and <|SOT|> tokens where hard prompt tokens were originally specified. Note that we mask the position of the soft prompt when calculating the training loss, since the model does not require learning to generate them. The soft prompt embedding will be updated as the model learns to transcribe the multi-talker speech.

Considering that the original Whisper was not trained using CTC loss, we refrain from employing an additional CTC loss for early permutation assignment as done in [5, 6].

The experiments are conducted on three multi-talker public datasets, namely LibriMix [34] and LibriSpeechMix [13] in English, and Aishell1Mix²²2https://github.com/huangzj421/Aishell1Mix in Mandarin. Audio exceeding Whisper’s maximum handling duration of 30 seconds are time-stretched to conform to this limit. For target-talker ASR on LibriMix and LibriSpeechMix, we randomly trim three-second clips from LibriSpeech as enrollment speech for each talker.

Throughout this study, we employ Whisper-small, -medium, and -large-v3 as the foundation models, respectively. We freeze these models and only train the Sidecar separator, Target Talker Identifier, and soft prompt embedding. The number of trainable parameters for systems using different foundation models and for various numbers of talkers are listed in Table 1.

The Conv-TasNet-like Sidecar separator [20] comprises a series of K temporal convolutional blocks with dilation rates ranging from 1 to $2^{\textit{K}-1}$, with each block repeats up to R times. Consistent with the protocol in [20, 21], we use K = 8 and R = 3 and plug it between the second and third encoder blocks. The length for soft prompt embeddings are investigated through ablation experiments (Section 4.4). As a result, we establish a length of 4, which gives the best performance.

For systems with the TTI module, at each training step, there’s an 80% probability of undertaking multi-talker ASR training, while a 20% probability for joint multi-talker and target-talker ASR training. We set the coefficient of TTI loss $\lambda$ to 0.01. The systems are trained and evaluated on two- and three-talker subsets of LibriMix and LibriSpeechMix, respectively. Each training session lasts for a maximum of 200k steps on 8 NVIDIA V100 GPUs with a total batch size of 16, employing AdamW optimizer with an initial learning rate of 2e-4 that decreases linearly to 1e-4.

Permutations with minimum errors are used to compute word error rate (WER) or character error rate (CER) for multi-talker ASR as in prior studies [14, 20]. For target-talker ASR, we use standard WER for evaluation. Both the model’s predictions and the references are normalized following [1].

We compared the performance of various systems for multi-talker ASR on the two- and three-talker LibriMix and LibriSpeechMix test sets, as shown in Table 2. Empowered by Sidecar separator (SS), TTI, and soft prompt, our systems (f)-(k) consistently illustrated improved performance across both the two- and three-speaker subsets. Even with Whisper-small-SS-TTI (g), owing to Whisper’s extensive pre-training, our method has already surpassed the original Sidecar scheme[21]. As the size of the Whisper model increases, we observed a steady improvement in performance, which aligns with our expectations. Ultimately, our systems outperformed previous approaches across all datasets except for LibriSpeechMix-3spk, demonstrating the superiority of the proposed approach.

Interestingly, we find that systems with the TTI module (g) (i) (k) outperform their counterparts without TTI (f) (h) (j) in multi-talker ASR task, even though the TTI is specifically designed to support target-talker ASR. This suggests that the training objective of learning to distinguish the target talker can benefit the Sidecar’s ability to separate embeddings, thereby facilitating the task of multi-talker ASR.

We evaluate target-talker ASR performance on two- and three-speaker subsets of LibriMix and LbriSpeechMix, as illustrated in Table 3.³³3We did not include results reported in [26], which delivers better performance but undergoes about ten times training efforts as ours. Our systems outperform previous state-of-the-art method on LibriMix dataset by a large margin, and we are the first to perform target-talker ASR task on Libri3Mix. To guarantee a fair comparison, we further trained three additional systems with limited training data to ensure consistency with the data used in [9, 24]. These systems are denoted as ”-limited”. The results demonstrate that our method still outperforms [9, 24] though under this restriction.

For LibriSpeechMix, the speech signals are partially overlapped, which means the target talker’s speech can incur a considerable delay before it commences, resulting in a substantial time interval away between it and the enrollment speech. Nevertheless, despite the existence of delays, our systems still demonstrate good performance on the LibriSpeechMix dataset, validating the effectiveness of the proposed method.

We investigated whether the multi-lingual characteristics of Whisper are retained after fine-tuning it on an English multi-talker dataset. Specifically, we conducted evaluations for systems (g) (i) (k) listed in Table 1 using the two-speaker AishellMix Mandarin dataset, which is the first time to be used on multi-talker ASR task. The evaluations are performed using two schemes: zero-shot and one-batch-tuning. Zero-shot refers to directly evaluating the system on the AishellMix, while one-batch-tuning implies conducting an additional training epoch on the AishellMix training set prior to the evaluation.

As Table 5 illustrates, the medium and large models demonstrated acceptable CER performance even under zero-shot conditions. With just one batch tuning, these models exhibited satisfactory results. This suggests that our method largely maintains the inherent multilingual capabilities of Whisper.

We investigated the optimal prompt length by examining the multi-talker ASR performance on the Libri2Mix dataset with Whisper-medium and Whisper-large models. As shown in Table 5, a soft prompt of length 4 yields the best performance. However, as the soft prompt length increases to 16, the systems see a decline in performance. This may be due to overly long sequence sequences, making the model difficult to optimize, given the original Whisper model is frozen.

This study has several limitations. Firstly, our method relies on PIT which requires pre-defining of the maximum number of speakers. Future efforts will integrate SOT [13] or HEAT [11] to address this issue and reduce training costs. Secondly, when the target talker’s speech undergoes excessive delay, there could be potential degradation in the target-talker ASR’s performance. We anticipate future work to enable the TTI module synthesize information across the entire utterance duration rather than only the three-second enrollment speech.