MMSD-Net: Towards Multi-modal Stuttering Detection

Current Multimodal Language Models are mainly geared towards understanding video and text-based content. MMSD-Net incorporates specialized modality encoders to process not just video and text, but also auditory data. These encoders are designed to extract the most relevant features from each modality. This upgrade significantly enhances MMSD-Net’s ability to process and interpret information across various modalities efficiently. Unlike traditional models that might rely on Convolutional Neural Networks (CNNs), MMSD-Net leverages three transformer encoders for video, audio and text data respectively. Transformers excel at capturing long-range dependencies within sequences, making them well-suited for stuttering detection. By capturing these nuanced details, MMSD-Net gains a richer understanding of the input data corresponding to different modalities, allowing for more comprehensive reasoning. The specialized encoders process the video (v), audio (a) and textual (t) inputs as follows:

where $\mathbf{h}_{v}$, $\mathbf{h}_{a}$, and $\mathbf{h}_{t}$ represent the extracted features for video, and audio, respectively. $d_{v}$, $d_{a}$, and $d_{t}$ denote the dimensionality of the features for each modality.

To reduce computational costs and minimize the number of tokens in the prefix, we employ a 1-D convolutional layer to compress the length of the multi-modal features to a smaller value. Subsequently, a linear layer is employed to adjust the hidden size of the features, fusing it with the size of the MLMs’ embeddings as follows:

where $\mathbf{h}^{\prime}_{v},\mathbf{h}^{\prime}_{a},\mathbf{h}^{\prime}_{t}$ are the transformed features with a fixed length of $L^{\prime}$ and an embedding dimension of $d_{e}$. The value of $L^{\prime}$ is significantly smaller than $L_{v}$, $L_{a}$, and $L_{t}$, while $d_{e}$ corresponds to the dimensionality of the embedding matrix $\mathbf{E}\in\mathbb{R}^{v\times d_{e}}$ associated with the MMSD-Net.

Modality encoders are often trained independently, which can result in differences in the representations they produce. Therefore, it is essential to merge these distinct representations into a unified space. We design our fusion strategy based on the Q-Former [24], and modify it to efficiently integrate video and audio features with textual features, thereby facilitating more rapid adaptation. In this context, we designate the video features derived from our visual modality encoder by $h_{v}\in\mathbb{R}^{L_{v}\times d_{v}}$, and the audio features extracted from the audio modality encoder are denoted as $h_{a}\in\mathbb{R}^{L_{a}\times d_{a}}$.

In order to fuse the distinct representations learnt separately by the individual modality encoders, we consider the transformed visual and audio modality representations obtained in Equation 4, 5 and 6 as the soft tokens of our MLM module. The visual and audio representations are fused with the textual embedding space using the attention mechanism from Equation 1, as follows:

where $\mathbf{h}^{\prime}$ is the modality representation obtained in Equations 4, 5, and 6, E is the embedding matrix as defined in Section 3.1, and $\mathbf{h}^{a}$ is the corresponding fused representation, specifically $\mathbf{h}^{a}_{v}$, $\mathbf{h}^{a}_{a}$, and $\mathbf{h}^{a}_{t}$. After this fusion operation facilitated by the attention mechanism, the MMSD can seamlessly process the representations from various modalities.

In order to integrate the fused modality representations with the instruction information, we employ the concatenation operation. Given the fused modality representations, we define the integration as follows

where $[:]$ represents the concatenation operation, $x$ represents the multi-modal instruction, $x_{t}$ represents the sequence of tokens in the textual instruction, and $Embed$ ($x_{t}$) represents the sequence of embeddings of $x_{t}$.

Multimodal Language Models (MLM) [25] have demonstrated exceptional aptitude in comprehending and executing human directives. In particular, cross-modal approaches have demonstrated improved performance for audio synchronization tasks [26], [27]. In MMSD-Net, we utilize pretrained MLMs as the core modules, establishing the basis of MMSD-Net’s functionality. The pretrained MLM network processes three primary inputs: the query vector $Q\in\mathbb{R}^{n_{q}\times d_{q}}$, the key vector $K\in\mathbb{R}^{n_{k}\times d_{k}}$, and the value vector $V\in\mathbb{R}^{n_{v}\times d_{v}}$. Through the scaled dot-product attention, the model evaluates attention scores by comparing each query in $Q$ with all keys in $K$. These scores are then utilized to refine the query representations by generating a weighted sum of the values in $V$. The operation is mathematically represented as:

Here, $d_{k}$ denotes the size of the key and query vectors, while $n_{q}$ and $n_{k}$ represent the counts of queries and keys, respectively.

We use stacked multi-head attention as well as positional encodings to model the complete sequential information encoded by the inputs. The decoder employs masked multi-head attention followed by softmax normalization for binary classification. The positional encodings are added to the input as well as output embeddings, enabling the model to capture the sequentiality of the input sentence without having recurrence. The encodings are computed from the position ($pos$) and the dimension ($i$) as follows:

where $PE$ stands for positional encodings and $d_{model}$ is the dimensionality of the vectors resulting from the embeddings learned from the input and output tokens.

This study uses four publicly available datasets (1 audio-modality and 3 audio-visual modalities) for the experiments. The details of modalities present in each dataset, along with the amount of data used for training and testing in each of our experiments, are described in Table 1.

For training efficiency, we leverage LoRA [30] for optimization on 4 Nvidia A100 GPUs. Each GPU handles a batch size of 4, with gradients accumulated for 5 steps before updating the model. The training process lasts for 10 epochs, employing a cosine learning rate scheduler with an initial learning rate of $5\times 10^{-10}$ and a warmup ratio of 0.02. To promote efficiency, FP16 precision is used for both training and inference. The maximum sequence length is capped at 512 tokens.

Table 2 gives the results (in terms of precision, recall and F1-score) of the proposed MMSD-Net method along with the four baseline methods. The results are presented for the dataset obtained by combining the FluencyBank [23], Sept28K [18] and Adults Who Stutter [23] datasets as described in Table 1. In addition to model performance, the comparison of modalities used by each method is also presented in the Table 2. Among the four baseline methods, StutterNet [17] achieved better results compared to other three methods (FluentSpeech [28], ResNet+BiLSTM [9] and ConvLSTM [18]), and ResNet+BiLSTM [9] achieved the lowest scores among the baseline methods. While both StutterNet [17] and ResNet+BiLSTM [9] are specifically trained on audio data, StutterNet achieves superior performance using MFCC features extracted from the audio samples. This suggests that MFCC features are more suitable for stutter detection compared to the features employed by ResNet+BiLSTM, i.e. the spectrograms.

Our proposed MMSD-Net outperforms all other methods in terms of precision, recall, and F1-score, achieving the highest scores across all metrics, which demonstrates its superiority in stuttering detection. Our findings demonstrate that the fusion module can effectively combine information from three different modalities. Quantitatively, the proposed MMSD-Net gave an absolute improvement of 2% in the F1-score (and 3% in the Precision) over the best baseline method (StutterNet [17]) and 16% in the F1-score (and 17% in the Precision) over ResNet+BiLSTM [9]. These results validate our hypothesis that facial expressions serve as an important cues to detect stuttered speech, and employing the visual signal as part of a multi-modal framework improves the performance of automatic stuttering detection.