Controlling Whisper: Universal Acoustic Adversarial Attacks to Control Speech Foundation Models

Speech enabled foundation models are trained on a large quantity of data to process input speech signals and perform various different speech processing tasks. The development of these models have taken two main forms. First, several approaches have attempted to extend powerful textual decoder-only Large Language Models (LLMs) to support direct speech inputs with a trained connection module to provide the embedded audio as a soft prompt [2, 5, 6, 7, 8, 9]. Alternatively, generalist speech enabled foundation models have emerged in the form of flexible Automatic Speech Recognition (ASR) models [1] using an encoder-decoder architecture [10]. ASR models are traditionally designed to perform a single task: transcribe the input speech signal. However, recent powerful ASR models have been augmented with further speech processing abilities. Some of the most powerful flexible ASR models include OpenAI’s Whisper model [1] and NVIDIA’s Canary model [11], where these models can perform both speech transcription and speech translation tasks. These models adopt an encoder-decoder architecture, and a textual task tag is input to the decoder to indicate the speech task to be performed (transcribe or translate). Given its popularity, this work uses Whisper to illustrate the risk of control-attacks on such multi-task ASR models.

Traditional audio adversarial attacks on ASR models have one of two objectives: corrupt the output transcription or deceive the ASR model into transcribing a desired target phrase. Early research on audio attacks explored gradient-based methods to perturb input audio for end-to-end ASR systems like WaveCNN and HMM-DNN, aiming to increase word error rates (WER) in transcriptions [3, 4]. Later studies introduced targeted attacks on ASR systems such as DeepSpeech, HMM-DNN, and LSTM-based neural networks to generate specific transcriptions [12, 13, 14, 15], while other research focused on making audio adversarial attacks more imperceptible [16, 17]. Practical advancements include generating universal adversarial perturbations for various speech signals, although initial methods required synchronizing with the entire speech signal [18]. This was addressed by creating perturbations that do not need to be synchronized with the source speech signal [19], and extending these attacks to newer end-to-end ASR systems like LAS, CTC, and RNN-T [20]. Additional techniques involve transferability from substitute models [21, 22, 23], evolutionary attacks [24, 25, 26, 27, 28], utterance-based attacks [29], and featurization attacks [30, 31, 32]. With the establishment of the Whisper model, new vulnerabilities to audio adversarial attacks have been identified, showing for example that adversarial attacks can lead to incorrect transcriptions [33] or even force entirely muted outputs for any speech input [34].

However, with the emergence of the multi-task speech enabled foundation models, in this work, we demonstrate that there is the threat of a new kind of adversarial attack: model-control attack. With the flexibility to perform multiple tasks, an adversary can override the operational setting of a deployed ASR model and force it to perform a different target task - i.e., take control of the model. With Whisper, we illustrate that despite being deployed in ‘transcribe’ mode, a simple universal adversarial attack can always force Whisper to operate in ‘translate’ model.

When Whisper is deployed for a specific speech processing task, an adversary does not have access to the internal structure of the model and cannot simply change the special tokens input to the decoder to alter the task setting. Realistically, an adversary can only modify the source audio to achieve their goal of overriding Whisper’s task setting. Thus, we assume the adversary can only make changes in the acoustic space. Moreover, our threat model requires the adversarial attack to be easy to apply, as complex manipulations of the audio are impractical for live speech processing. To address this, we use a prepend adversarial attack form, where the attack requires only a short acoustic adversarial segment to be prepended to the input speech. Learning a different adversarial manipulation for each new speech signal is impractical, especially for live streamed audio. Therefore, our threat model aims to learn a universal audio manipulation—a single short acoustic adversarial segment that can be prepended to any speech signal to achieve the desired model control. While we allow whitebox access (gradient access to the model) for learning the universal attack segment, it must be universally applicable to unseen speech signals.

To perturb a speech signal $\mathbf{x}=x_{1:N}$, we prepend a short adversarial audio segment of $T$ frames, $\tilde{\mathbf{x}}=\tilde{x}_{1:T}$. The perturbed speech signal becomes $\tilde{\mathbf{x}}\oplus\mathbf{x}$, where $\oplus$ represents concatenation in the raw audio space. To override Whisper’s ‘transcribe’ setting with ‘translate,’ we need to find the optimal adversarial audio segment, $\hat{\tilde{\mathbf{x}}}$, that maximizes the likelihood of generating the translated sequence, despite Whisper running in transcribe mode:

$$\hat{\tilde{\mathbf{x}}}=\operatorname*{arg\,max}_{\tilde{\mathbf{x}}}P(\mathbf{y}^{(\texttt{tl})}|\tilde{\mathbf{x}}\oplus\mathbf{x},\mathcal{T}=\texttt{tc}),$$

(2)

where

$$\mathbf{y}^{(\texttt{tl})}=\operatorname*{arg\,max}_{\mathbf{y}}P(\mathbf{y}|\mathbf{x},\mathcal{T}=\texttt{tl}).$$

(3)

To learn a universal acoustic attack, we adapt Equation 2 to maximize the likelihood of generating the translated sequence over a training set of samples, $\{\mathbf{x}_{j}\}_{j=1}^{J}$,

$$\hat{\tilde{\mathbf{x}}}=\operatorname*{arg\,max}_{\tilde{\mathbf{x}}}\prod_{j}P(\mathbf{y_{j}}^{(\texttt{tl})}|\tilde{\mathbf{x}}\oplus\mathbf{x}_{j},\mathcal{T}=\texttt{tc}).$$

(4)

To maximize the likelihood of Equation 4, standard gradient descent based approaches can be used to update $\tilde{\mathbf{x}}$. It is important for the adversarial audio segment generated to be somewhat imperceptible such that it is not flagged as suspicious when prepended to natural speech signals. We achieve this imperceptibility in two ways: by ensuring the adversarial audio segment is short in duration and by limiting its ‘power’ or amplitude relative to natural speech. To limit the power, we introduce a constraint in the optimization objective of Equation 4 that limits the amplitude of the adversarial audio,

$$||\hat{\tilde{x}}_{1:T}||_{\infty}\leq\epsilon,$$

(5)

where $||\cdot||_{\infty}$ represents the l-infinity norm. During the gradient-based learning of the adversarial audio segment $\hat{\tilde{\mathbf{x}}}$, this l-infinity norm constraint is incorporated by clamping the values at $\epsilon$ [35]. In our experiments (Section 5) we explore the impact of varying these two imperceptibility constraints on the efficacy of the control-attack.

The model-control universal acoustic segments are trained such that they can be prepended to any input speech signal and cause Whisper to perform speech translation despite being set to perform speech transcription. Using French-English (fr-en) as the primary language pair for evaluating the model-control attack, Table 1 presents the impact of the model-control attacks of each strength. For reference, the performances with respect to the English transcriptions are given when Whisper is run with no attack in transcription mode (expected to be low performance as Whisper transcribes in the source audio language French) and when Whisper is run with no attack in translation mode (the upperbound of performance for the attack). As the attack strength is increased from weak to strong, the attack performance approach the translation mode upperbound performance. This demonstrates that the model-control attack can be successful in overriding the transcription mode to cause Whisper to perform speech translation instead.

Although the model-control attack is highly effective in forcing Whisper to perform speech translation, it is unable to perfectly match the upperbound performance with Whisper running freely in translation mode (Table 1). This could be due to either the attack failing to make Whisper enter translation mode for specific input speech signals or due to the attack resulting in Whisper generating lower quality translations than when Whisper is run directly in translation mode. We next investigate the cause of the discrepancy. For each attack strength, we compute BLEU-recall curves for samples on which the universal model-control attack is ‘successful’ and ‘unsuccessful’ (Figure 2) ¹¹1Identical trends were observed with COMET-recall curves.. A sample is classified as successfully attacked if $P(\text{en})>\tau$. Conversely, a sample is classified as a ‘failed’ attack if $P(\text{en})<\tau$. We sweep the arbitrary threshold $\tau$ from 0 to 100% and compute BLEU scores for both the successfully and unsuccessfully attacked samples recalled at each value of $\tau$. These curves help explain the quality of the translations generated by the model when attacked, separately for when the attack succeeds or fails to make Whisper generate English tokens (measured by P(en)).

From 2, in general the greater the probability of English in the generated translation, the higher the quality of the translation as measured by BLEU. However, the curves also reveal a more interesting behaviour of the attacks: the success recall curves (2a), have an observable discontinuity in the BLEU scores when 43%, 62% and 98% of the samples are ‘successfully’ attacked for the weak, mid and strong attacks respectively, and conversely in the fail recall curves (2b), there is a discontinuity when approximately 57%, 38% and 2% samples ‘fail’ to be attacked. These percentages almost perfectly align with the average probability of English given by each attack in Table 1. This suggests that the attacks result in a highly bi-modal response, where the attack on a specific input audio sample is either perfectly successful in causing Whisper to generate English text or perfectly unsuccessful, with near 0% probability of English. This bi-modal split is further verified in the distribution of P(en) given for each attack in Figure 3. When Whisper is running in transcribe (tc) mode, it is entirely non-English (left-most bar), whilst when Whisper is running in translate (tl) mode, it generates entirely English text (right-most bar). However for all of the attacks in Figure 3, we observe the above-mentioned bi-modal pattern, where the model-control attacks do not result in a gradual distributional shift from the French transcriptions to English translations, but instead display binary success. The stronger the attack the greater the proportion of the samples that are ‘successfully’ attacked (greater fraction of samples on the right side) resulting in almost complete English text. On the whole, the universal model-control attack, when successful, causes Whisper to behave as though as if it is in translation mode, but when the attack is unsuccessful it results in Whisper continuing to behave as set in its default transcription mode - there are no partial modes. It is the strictness around the imperceptibility constraints (strength of the attack) that dictates the fraction of samples the universal model-control attack is successful upon.

In this section we assess the portability of the model-control attack method to other language pairs beyond French-English, where the aim as before is to learn a universal acoustic segment that can be universally prepended to any input speech signal and cause Whisper to carry out speech translation (to English), despite being deployed in speech transcription mode. We consider the strong attack setting in this section.

Table 2 presents the impact of the universal model-control attack for the new languages: German-English (de-en), Russian-English (ru-en) and Korean-English (ko-ru). On the whole, the attack appears to fairly successful for all language pairs in forcing Whisper to generate English translations, as indicated by the average probability of English (P(en)) going from 0.0% to above 95% for all language pairs. However, when considering the BLEU and COMET scores, it appears the quality of translations with the attack (relative to the upperbound performance given by no attack in tl mode) for de/ru/ko is not quite as high as for fr, where the strong attack closely approached the upperbound performance (Table 1). The gap between the attack (strong) performance and the upperbound performance for de/ru/ko can be explained by the decomposition of the WER between the translations generated. Table 3 presents the WER decomposition between translations generated by the attack (on Whisper in transcribe model) and the no attack translations. It is interesting to note that a significant contributor to the increase in WER for de/ru/ko relative to fr, comes from a large increase in the insertion rate. This suggests that the attack causes Whisper to hallucinate. As an example, we find that for Russian, the attack causes 167 samples to have the phrase however, it is clear that inserted at the beginning of the translation, whereas only 1 sample has this phrase in its translation when there is no attack, i.e., it seems that the attack can sometimes can learn an acoustic realization for the prepended acoustic audio, i.e, for some languages the translations generated at inference (free-running) can contain a prepended text sequence before the actual translation, resulting in lower translation quality.

Next, to verify that the attacks on de/ru/ko display the same bi-modal behaviour as the attack on fr-en, we consider the BLEU-recall curves for each language pair in Figure 4. Most easily observable in Figure 4b the discontinuity in BLEU performance occurs as before with fr-en at the percentage of samples corresponding to approximately the average probability of not English (in Table 2) - between 2% and 5% for the different languages. This verifies that for all languages the same trend holds: the model-control attack is highly binary in either successfully overriding Whisper to translate the source sample to complete English, or failing completely, such that the source audio is not translated at all.