Learning Video Temporal Dynamics with Cross-Modal Attention
for Robust Audio-Visual Speech Recognition

Recent AVSR works have focused on creating better audio-visual multi-modal representations via sophisticated training schemes or scaling up to larger datasets. AV-HuBERT [5] learns to predict the cluster assignments of audio features for the masked prediction training. Modality dropout is introduced for audio-visual fusion to prevent models from excessively relying on the audio modality. Several approaches [27, 28, 29] employ a teacher-student framework, where the teacher model weights are updated via an exponential moving average of the student model weights, to predict contextualized target representations for the masked frames. Auto-AVSR [6] incorporates the pretrained automatic speech recognition (ASR) model for creating pseudo-labels of the unlabeled video dataset. The most recent finding [30] illustrates that the linear projection is sufficient as a visual front-end with large-scale datasets.

Since speech recognition is often susceptible to background noise or ambient speech, addressing noise robustness in the AVSR system is a practical and important problem. To this end, the follow-up study of AV-HuBERT [26] suggests to leverage noise-augmented audio for pretraining AV-HuBERT [5]. UniVPM [11] proposes a viseme-phoneme mapping to restore clean audio from lip movements under noisy environments. Reinforcement learning is also utilized for robust AVSR by encouraging an agent to explore optimal strategies for WER [10]. GILA [12] fuses audio and video representations with consecutive cross-modal attention blocks and implements contrastive loss to model the temporal consistency between audio-visual frames. Our work departs from prior works in the aspect of enhancing video features through temporal dynamics learning. We propose integrating a cross-modal attention module into the existing AVSR system to enhance its robustness against various types of noise.

Temporal dynamics refers to the information about changing patterns over multiple consecutive temporal frames, which has proven to help understanding videos, not necessarily with sound, in action recognition tasks [18, 22]. The correlation between adjacent video frames can be boosted by learning temporal self-supervision tasks, such as predicting the direction of token’s temporal flow [18] or whether certain frames are skipped [22]. Recent work has extended temporal dynamics into a multi-modal scope, proposing an inter-modal contrastive loss that learns longer-term dynamics through context ordering between video and audio data [21]. Our work aims to enhance video features for noise-robust AVSR by training temporal dynamics with simple binary classification tasks in a self-supervised manner. This approach differs from contrastive learning [21], which involves a complicated process of sampling positive and negative pairs and challenging optimization.

Our approach (Figure 1 and 2) is designed to reinforce the features of each modality, achieved by the temporal dynamics loss ($\mathcal{L}_{\text{temp}}$) and the refinement loss ($\mathcal{L}_{\text{ref}}$). We insert a cross-modal attention structure between the front-end feature extractors and the pretrained AVSR encoder and train this attention structure with the two aforementioned losses. In the video streamline, audio information is injected by the audio-to-video (A2V) cross-modal attention as a key and a value to train the temporal dynamics of video features. Vice-versa, the video-to-audio (V2A) cross-modal attention utilizes clean video information as a key and a value to refine the noisy audio features and reduce the impact of noise on the subsequent A2V module. We block the gradient flow from one side to the other for a stable training, ensuring that each loss can only train its corresponding streamline’s cross-modal attention. Consequently, the reinforced video and audio features are input to the pretrained AVSR encoder.

Let us denote the outputs of the video and audio front-ends as $\mathbf{f}_{\text{v}},\,\mathbf{f}_{\text{a}}\in\mathbb{R}^{T\times D}$, respectively. They share the same sequence length $T$ and the channel dimension $D$. To inject audio information into the video features, we implement a stacked attention²²2Multi-head attention [33] mechanism is employed for the SA and CA module, but we omit its notation for brevity. module; self-attention (SA) and then cross-modal attention (CA). The SA module in the beginning is for preparing the features with referencing the other modality information. $\mathbf{f}_{\text{v}}$ is first transformed by the query, key, and value matrices, $\mathbf{W}_{q},\mathbf{W}_{k},\mathbf{W}_{v}\in\mathbb{R}^{D\times D}$, followed by a SA mechanism. Given a single FC layer $\mathbf{W}_{\text{fc}}\in\mathbb{R}^{D\times D}$,

	$\displaystyle\text{SA}(\mathbf{f}_{\text{v}})$	$\displaystyle=\text{Attention}(\mathbf{f}_{\text{v}};\,\mathbf{f}_{\text{v}};\,\mathbf{f}_{\text{v}})$
		$\displaystyle=\text{softmax}\left(\frac{(\mathbf{f}_{\text{v}}\mathbf{W}_{q})(\mathbf{f}_{\text{v}}\mathbf{W}_{k})^{\top}}{\sqrt{D}}\right)(\mathbf{f}_{\text{v}}\mathbf{W}_{v}),$		(1)
	$\displaystyle\mathbf{f}^{\prime}_{\text{v}}$	$\displaystyle=\text{SA}(\mathbf{f}_{\text{v}})\cdot\mathbf{W}_{\text{fc}_{1}}.$		(2)

The resulting video features are then processed by the A2V cross-modal attention module, employing $\mathbf{f}^{\prime}_{\text{v}}\in\mathbb{R}^{T\times D}$ as the query and the audio features $\tilde{\mathbf{f}}_{\text{a}}$ as the key and value. Here, $\tilde{\mathbf{f}}_{\text{a}}$ is refined by another cross-modal attention in order to facilitate the accurate learning of temporal dynamics for the video features (refer to Section 3.3 for details on the audio feature refinement process).

	$\displaystyle\text{CA}(\mathbf{f}^{\prime}_{\text{v}};\,\tilde{\mathbf{f}}_{\text{a}})$	$\displaystyle=\text{Attention}(\mathbf{f}^{\prime}_{\text{v}};\,\tilde{\mathbf{f}}_{\text{a}};\,\tilde{\mathbf{f}}_{\text{a}}),$		(3)
	$\displaystyle\tilde{\mathbf{f}}_{\text{v}}=\mathbf{f}_{\text{v}}$	$\displaystyle+\text{CA}(\mathbf{f}^{\prime}_{\text{v}};\,\tilde{\mathbf{f}}_{\text{a}})\cdot\mathbf{W}_{\text{fc}_{2}}.$		(4)

Our final audio-incorporated video features $\tilde{\mathbf{f}}_{\text{v}}\in\mathbb{R}^{T\times D}$ are residual summation of the original features and the FC layer’s output of the A2V cross-modal attention.

To synchronize the temporal positions of the context in video and audio, we involve the temporal order loss in a cross-modal way. The order loss learns to predict the context order of two randomly selected frames, one from the video sequence and another from the audio sequence (refer to Figure 1(a)). Let us denote $\tilde{\mathbf{f}}_{\text{v}}=(\tilde{v}_{1},\cdots,\tilde{v}_{T})^{\top}$ and $\tilde{\mathbf{f}}_{\text{a}}=(\tilde{a}_{1},\cdots,\tilde{a}_{T})^{\top}$ for the enhanced video and audio features, which are the outputs from each cross-modal attention module. The context order loss $\mathcal{L}_{\text{order}}$ is defined as

$$\mathcal{L}_{\text{order}}=\sum_{i,j,i\neq j}\text{BCE}(g(\tilde{v}_{i}\lVert\tilde{a}_{j}),y),$$

(5)

where $\lVert$ is a channel-wise concatenation. The frame order labels are $y=1$ for $i<j$ and $y=0$ for $i>j$. BCE refers to a binary cross-entropy loss function, and $g(\cdot)$ is a binary predictor network, composed of 1D temporal convolution [34] followed by an FC layer. The purpose of temporal convolution is to fuse temporally adjacent features, avoiding ambiguity where a single characteristic (e.g., phoneme) may appear in multiple places within a sequence.

Furthermore, we propose the direction loss and speed loss to train temporal dynamics that are revealed in a short time duration, particularly learning the local temporal dynamics. The direction loss predicts whether consecutive temporal length $t$ video frames are playing forward ($y=1$) or backward ($y=0$) (refer to Figure 1(b)).

	$\displaystyle\mathcal{L}_{\text{direction}}=\sum_{i}\,\,$	$\displaystyle\mathrm{BCE}(g(\tilde{v}_{i}\lVert\cdots\lVert\tilde{v}_{i+t-1}),1)$
		$\displaystyle+\mathrm{BCE}(g(\tilde{v}_{i+t-1}\lVert\cdots\lVert\tilde{v}_{i}),0).$		(6)

Additionally, the speed loss predicts whether a given video sequence is playing in a regular speed ($y=1$) or skipping the frames in the speed of $k>1$ ($y=0$) (refer to Figure 1(c)).

	$\displaystyle\mathcal{L}_{\text{speed}}=\sum_{i}\,\,$	$\displaystyle\mathrm{BCE}(g(\tilde{v}_{i}\lVert\tilde{v}_{i+1}\lVert\cdots\lVert\tilde{v}_{i+t-1}),1)$
		$\displaystyle+\mathrm{BCE}(g(\tilde{v}_{i}\lVert\tilde{v}_{i+k}\lVert\cdots\lVert\tilde{v}_{i+(t-1)k}),0).$		(7)

These two loss functions help the video features learn how lip shape moves naturally over a short duration of time. We also mark that the predictors are not shared across the different temporal dynamics losses and discarded for inference.

We have suggested the audio information be injected into the video features by the A2V cross-modal attention for training the video temporal dynamics. However, perturbed audio information with noise may lead misguidance in learning the temporal dynamics. Similar to previous AVSR works [7, 8, 11] that have performed audio enhancement with accompanying clean video information, we aim to refine the audio inputs through a V2A cross-modal attention.

Analogous to the video streamline (Eq. 3.2–4), audio features are modified by the stacked SA-CA module, where the V2A cross-modal attention is performed to refine the noisy audio features with the help of video features.

	$\displaystyle\mathbf{f}^{\prime}_{\text{a}}$	$\displaystyle=\text{SA}(\mathbf{f}_{\text{a}})\cdot\mathbf{W}_{\text{fc}_{3}},$		(8)
	$\displaystyle\tilde{\mathbf{f}}_{\text{a}}$	$\displaystyle=\mathbf{f}_{\text{a}}+\text{CA}(\mathbf{f}^{\prime}_{\text{a}};\mathbf{f}_{\text{v}})\cdot\mathbf{W}_{\text{fc}_{4}}.$		(9)

We remind this audio refinement process in Eq. 9 is preceding the video feature enhancement in Eq. 4. The modified audio features $\tilde{\mathbf{f}}_{\text{a}}$ reference the clean audio features $\mathbf{f}_{\text{a, clean}}$, with loss calculated by the mean-squared error (MSE). The clean audio features are not processed further through our audio streamline.

$$\mathcal{L}_{\text{ref}}=\lVert\tilde{\mathbf{f}}_{\text{a}}-\mathbf{f}_{\text{a, clean}}\rVert^{2}.$$

(10)

Datasets. We perform our experiments on LRS2 [23] and LRS3 [24], datasets comprising around 224 and 433 hours of audio-visual speech data, respectively, from over 5,000 speakers. Most of our experimental configurations follow [11] and [26], including the noise augmentation and evaluation protocol. We extract noise from MUSAN [25] (babble, music, and natural) and LRS3 (speech) datasets, and partition them into train, validation, and test sets. For training, we sample noise at 0 dB SNR and always add it to the clean speech signal. For evaluation, we use noise from the MUSAN test set, as done in [10, 11, 36], as well as synthesizing the babble noise by randomly mixing 30 audio clips from LRS3, following [5, 7]. We report WER evaluated on noise-perturbed test set with 5 different SNR levels: $\{-10,-5,0,5,10\}$. The evaluation metric is N-WER [26] (AVG), the average WER across 4 noise categories and 5 SNRs. We also report noise-dominant N-WER (N $\geq$ S), which only considers 3 non-positive SNR levels: $\{-10,-5,0\}$.

In Table 1, we present the AVSR performance of our proposed method, evaluated on the LRS3 [24] benchmark. Our model consistently surpasses AV-HuBERT [26] across all four noise types, as indicated by N-WER of 5.7% and 4.9%, depending on whether the encoder is pretrained with noise-augmented audio (i.e., PT Type). Also, it outperforms MIR-GAN [36] and UniVPM [11] by 5.4%/4.6% N-WER for clean/noisy PT Type, respectively, attaining a new state-of-the-art performance. Our methods especially excelling in babble and speech noise while offering comparable results in music and natural noises. This highlights the importance of learning video temporal dynamics with audio information, rendering the AVSR model to accurately distinguish the target speech signal in multi-speaker scenarios by attending lip movements in the video data. Meanwhile, it is crucial to acknowledge a trade-off in noise robustness. Our method, catered to noise-corrupted conditions, leads to exceptional performance gain in such scenarios but a slight degradation in the clean speech setting, which has been similarly observed in other noise-robust ASR works [39, 40, 41].

For the noise-dominant N-WER (N $\geq$ S), our results exhibit great effectiveness in certain scenarios, highlighting its robustness and real-world applicability. The comparisons with recent works, including MSRL [10] and UniVPM [11], substantiate that our method bolsters the robustness of our AVSR system with achieving 7.8% and 6.4% noise-dominant N-WER regard to PT Type. Our method also outperforms AV-HuBERT [26] for both PT Type with 8.3% and 6.9%. In contrast to previous approaches that rely on the general methods for reducing the modality gap, such as contrastive learning [12] or adversarial learning [11, 36], our method incorporates the inherent characteristics of video features, thereby resulting in superior performance for the noise-dominant setting. Importantly, refined audio information is injected into video features at this stage, making our cross-modal attention design well-suited for the noise-robust AVSR task.

The trend of the aforementioned results continues in the LRS2 [23] benchmark (Table 2). Our method surpasses the recent noise-robust AVSR works, MIR-GAN [36] and UniVPM [11], with 5.9% N-WER and 7.7% noise-dominant N-WER. Comparing with AV-HuBERT [26], around 11% of relative performance gain is achieved in both average and noise-dominant N-WER. This confirms that our method is still effective, regardless of the difference between the pretraining and fine-tuning datasets.

Training losses. Table 3 shows our investigation on the independent effects of the video temporal learning and audio refinement losses, by systematically adding each loss component. The plain ASR loss ($\mathcal{L}_{\text{ASR}}$) matches the ASR fine-tuning loss of AV-HuBERT [5] baseline. Starting from this, we find that utilizing the video temporal losses ($\mathcal{L}_{\text{ASR}}$ + $\mathcal{L}_{\text{temp}}$) plays a crucial role in improving overall performance, verifying the importance of strengthening the video features for robust AVSR. Employing the audio refinement loss ($\mathcal{L}_{\text{ASR}}$ + $\mathcal{L}_{\text{ref}}$) also shows the performance improvement but not as much as video temporal losses. Combining all the proposed losses ($\mathcal{L}_{\text{ASR}}$ + $\mathcal{L}_{\text{temp}}$ + $\mathcal{L}_{\text{ref}}$), we achieve the best AVSR performance which shows refining noise-perturbed audio is particularly crucial for correctly guiding the temporal dynamics to video features.

We further examine the performance of visual speech recognition (VSR) to demonstrate the enhancement of visual features without any audio information. Our method, which is accompanied with the video temporal learning, produces a lower 32.5% WER compared to the baseline (33.7%), indicating its better representations of lip movements. However, when video temporal learning is missing or has been misguided by noisy audio, the VSR performance is adversely affected, resulting in 33.2% and 33.3% WER, respectively.