GSIFN: A Graph-Structured and Interlaced-Masked Multimodal Transformer Based Fusion Network for Multimodal Sentiment Analysis

Multimodal sentiment analysis (MSA) is an increasingly popular research field. Its data includes more than two modals. Text, vision, and audio are the most widely used modals. Earlier models focus on modal fusion. Zadeh et al. were among the first promoters in this field. They proposed TFN (Zadeh et al., 2017), built a power set form of modal combination, and realized complete modal fusion by using Cartesian product. TFN did not consider the temporal information of non-verbal modals, thus, MFN(Zadeh et al., 2018) is designed. MFN uses an LSTM system to extract the timing information of the three modes through explicit modal alignment (CTC, padding etc.) and uses attention mechanism and gated memory to realize the efficient fusion of multimodal temporal information.

With the rise of Transformer, MulT (Tsai et al., 2019) proposes a cross-modal attention mechanism (CMA) from the perspective of modal translation. CMA can effectively integrate multimodal data while realizing implicit modal alignment. Based on MulT and CMA, models such as TETFN (Wang et al., 2023a) and ALMT (Zhang et al., 2023) focus on text data to enhance non-verbal modal data, for text data contains stronger emotional information. Thus, they achieve superior representation performance and modal fusion. MAG-BERT (Rahman et al., 2020) uses a Multimodal Adaptation Gate (MAG) to fine-tune BERT using multi-modal data. CENet (Wang et al., 2023b) constructs non-verbal modal vocabularies, realizes non-verbal modal representation enhancement, and realizes MSA capability enhancement of fine-tuned BERT.

To improve the robustness of the model and the representation ability of non-verbal modals, and thus improve the overall multimodal sentiment analysis ability of the model, representation learning-based models such as Self-MM (Yu et al., 2021), ConFEDE (Yang et al., 2023), and MTMD (Lin and Hu, 2024) were proposed. They use self-supervised learning, contrast learning or knowledge distillation to achieve robust representation of modal information consistency and difference.

TETFN, MMML (Wu et al., 2024) and AcFormer (Zong et al., 2023) combine multimodal Transformer with representation learning to effectively improve model performance, and verify the feasibility of combining the two to learn from other strengths.

Due to the excessive use of traditional multimodal Transformer architecture in these methods, they often have a high number of parameters in the core fusion module. Additionally, different fusion combinations are decoupled to multiple independent Transformers (Vaswani et al., 2017), the interaction of modal information is insufficient, and there are problems of insufficient weight regularity. In the concrete implementation, we refer to the idea of graph attention networks (Velickovic et al., 2018; Brody et al., 2022) and construct a graph-structured multimodal Transformer with modal subgraph units.

In the field of natural language processing (NLP), reducing the computational cost of Transformers while maintaining performance has become a popular research topic. RWKV (Peng et al., 2023a), RetNet (Sun et al., 2023), Mamba (Gu and Dao, 2023), Mamba-2 (Dao and Gu, 2024) are representatives among them. xLSTM (Pöppel et al., 2024), as an extension of LSTM, introduces exponential gating to solve the limitations of memory capacity and parallelization, especially when dealing with long sequences.

At the same time, recent works in the field of MSA have begun to use more advanced feature extractors to enhance non-verbal modal features, taking into account the weak representation capability of non-verbal modals. For instance, TETFN and AcFormer (Zong et al., 2023) use Vision Transformer (ViT) (Dosovitskiy et al., 2021) to extract vision features, AcFormer uses Wav2Vec (Schneider et al., 2019) to extract features, and MMML (Wu et al., 2024) uses raw audio data to fine-tune Data2Vec (Baevski et al., 2022). However, these methods often result in excessive growth in the number of parameters, with obscure improvement over traditional features. To reduce model parameters and ensure model performance at the same time, the self-supervised learning method is used to strengthen the capture and representation of sentiment information. In GSIFN, mLSTM module in xLSTM is used to enhance the non-verbal input feature to unimodal label generation, it significantly reduces the computation overhead and ensures model performance.

The objective of multimodal sentiment analysis (MSA) is to evaluate sentiment polarity using multimodal data. Existing MSA datasets generally contain three modals: $t,v,a$ represent text, vision, and audio, respectively. Specially, $m$ denotes multimodal. The input of MSA task is $S_{u}\in\mathbb{R}^{T^{s}_{u}\times d^{s}_{u}}$, where $u\in\{t,v,a\}$, $T^{s}_{u}$ denotes the raw sequence length and $d^{s}_{u}$ denotes the raw representation dimension of modal $u$. In this paper, we define multiple outputs $\hat{y}_{u}\in R$, where $u\in\{t,v,a,m\}$, $\hat{y}_{\{t,v,a\}}$ denote unimodal outputs, obtained for unimodal label generation. $\hat{y}_{m}$ denotes the fusion output, obtained for the final prediction. Other symbols are defined as follows, fusion module inputs are $\{X_{t},X_{v},X_{a}\}$. ULGM inputs are $\{\mathcal{X}_{t},\mathcal{X}_{v},\mathcal{X}_{a}\}$. The predictor input is $\mathcal{X}_{m}$. In particular, in the interpretation of GsiT $\{X_{t},X_{v}X_{a}\}$ are abstracted to sequences of vertices $\{\mathcal{V}_{t},\mathcal{V}_{v},\mathcal{V}_{a}\}$. Labels for $y_{u}\in R$, where $u\in\{t,v,a,m\}$, $y_{\{t,v,a\}}$ are unimodal label generated by ULGM, $y_{m}$ is the ground truth label for fusion output.

The overview of our model is shown in Figure 1 which consists of three major parts: (1) Modal Encoding utilizes tokenizer (for text modality), feature extractors and temporal enhancers (for non-verbal modals vision and audio) to convert raw multimodal data into numerical feature sequences. Enhanced non-verbal modal features are utilized for unimodal label generation. (2) Graph-Structured Multimodal Fusion takes the processed text, vision, and audio embedding as input. The module graph-structured and interlaced-masked multimodal Transformer utilizes interlaced masks to construct multimodal graph embedding. It employs weight-sharing to facilitate comprehensive multimodal information interaction and eliminate redundant data, thereby enhancing fusion efficiency and enabling model pruning. (3) Self-Supervised Learning Framework generates final representations and defines positive and negative centers by projecting text features, enhanced vision audio features, and fusion output to hidden states. Unimodal labels are separately generated using text, vision, and audio hidden states.

For text modal, we use the pretrained transformer BERT as the text encoder. Input text token sequence is constructed by the raw sentence $S_{t}$ = $\{w_{1},w_{2},\dots,w_{n}\}$ concatenated with two special tokens ([CLS] at the head and [SEP] at the end) which form $S_{t}^{{}^{\prime}}$ = $\{\text{[CLS]},w_{1},w_{2},\dots,w_{n},\text{[SEP]}\}$. Then input $S_{t}^{{}^{\prime}}$ into BERT to construct $\mathcal{X}_{t}$, which is used to generate text modal labels.

Following previous works (Tsai et al., 2019), input sequences $X_{\{t,v,a\}}$ are handled by one dimensional convolution layer from $\mathcal{X}_{t}$ and raw vision, audio sequences $S_{\{v,a\}}$.

After that, we employ an extended Long Short Term Memory which is fully parallelizable with a matrix memory and a covariance update rule (mLSTM) as the temporal enhancer of vision and audio modal. mLSTM can improve model representation capability. Meanwhile, using it can avoid the overhead of large models. The detailed definition of mLSTM is in Appendix E.2.

We use mLSTM to enhance the temporal features of vision and audio.

mLSTM can enhance non-verbal modal features utilized for unimodal label generation.

Following previous works (Tsai et al., 2019; Wang et al., 2023a), we only use the low-level temporal feature sequences $\{X_{t},X_{v},X_{a}\}$ as input of multimodal fusion. Then $\{X_{t},X_{v},X_{a}\}$ are regarded as graph vertex sequences $\{\mathcal{V}_{t},\mathcal{V}_{v},\mathcal{V}_{a}\}$. Then, concatenate vertices into a single sequence $\mathcal{V}_{m}$ = $[\mathcal{V}_{t};\mathcal{V}_{v};\mathcal{V}_{a}]^{\top}$. $\mathcal{V}_{m}$ is treated as the multimodal graph embedding (MGE). The architecture of Graph-Structured and Interlaced-Masked Multimodal Transformer Architecture (GsiT) is shown in Figure 2.

Graph Structure Construction To start with, we utilize the self-attention mechanism as the basic theory to construct a naive fully connected graph. The attention weight matrix is regarded as the adjacency matrix $\mathcal{A}$ with dynamic weights. In $\mathcal{A}$, $\mathcal{E}^{i,j}\in\mathbb{R}^{T_{i}\times T_{j}}$, $\{i,j\}\in\{t,v,a\}$ is the adjacency matrix of the subgraph constructed by $\mathcal{V}_{i}$ and $\mathcal{V}_{j}$.

The derivation process of detailed graph structure construction (from vertex to subgraph) is in Appendix D.1.

Interlaced Mask Mechanism Interlaced Mask (IM) is a modal-wise mask mechanism, thus all of the elements in the mask matrix are subgraph adjacency matrices. The mask matrix is represented as a block matrix. Then the construction procedure of IM is described in detail. The computation procedure with IM is shown in Figure 2.

To start with, to avoid the influence of intra-modal subgraph $\mathcal{E}^{i,i}_{i\in\{t,v,a\}}$, we apply modal-wise intra mask as shown in Equation 6. We define $\mathcal{O}^{i,j}\in\mathbb{R}^{T_{i}\times T_{j}}$ as all zero matrix, $\mathcal{J}^{i,j}\in\mathbb{R}^{T_{i}\times T_{j}}$ as all negative infinity matrix.

$\mathcal{M}_{inter}$ can already make cross-modal fusion not be affected by intra-modal subgraphs. However, in the fusion procedure, different modal sequences should not be recognized as the same sequence. Therefore, we extend $\mathcal{M}_{inter}$ to the following two mask matrices, which is called as Interlaced-Inter-Fusion Mask (IFM). The explanation of aforementioned information disorder is in Appendix D.2

Based on the two matrices, two opposite uni-directional ring graphs can be constructed to achieve a complete fusion procedure. We define softmax operation as $\mathcal{S}$, dropout operation as $\mathcal{D}$, and function composition operator as $\circ$.

By now, $\mathcal{G}_{inter}^{forward}$ and $\mathcal{G}_{inter}^{backward}$ truly make MGE $\mathcal{V}_{m}$ graph-structured. Both of the two matrices manage to aggregate the information of the trimodal without temporal disorder and intra-modal information influence.

After aggregation, the fusion process is performed.

Where $\mathcal{W}_{v}$ denotes the value projection weight of $\mathcal{V}_{m}$.

As shown in Figure 2, two MGEs are constructed by IFM in two separated Transformers, they are two opposite unidirectional rings. Due to their special structure, a complete fusion process is achieved.

After fusion, intra-modal subgraphs need to be enhanced accordingly. Therefore, the Intelaced-Intra-Enhancement Mask (IEM) is constructed.

Where $\mathcal{J}$ denotes a negative infinity matrix at the same size of $\mathcal{M}_{inter}$.

$\mathcal{M}_{intra}$ leaves only intra-modal subgraphs visible to enhance the fused MGEs.

After IEM construction, concatenate two opposite unidirectional ring MGEs on feature dimension into one bidirectional MGE. We define $\parallel$ as the concatenation operation on the feature dimension.

Utilizing the bidirectional MGE $\overline{\mathcal{V}}_{m}^{bidirection}$ and $\mathcal{M}_{intra}$, the intra-modal enhancement graph could be constructed. We define $\overline{\mathcal{V}}_{m}^{b}$ = $\overline{\mathcal{V}}_{m}^{bidirection}$, $\mathcal{W}_{q}^{b}$, $\mathcal{W}_{k}^{b}$ as the query, key projection weight of $\mathcal{V}_{m}^{b}$.

Then, we construct the final feature sequence.

Where $\mathcal{W}_{v}^{b}$ denotes the value projection weight of $\mathcal{V}_{m}^{b}$.

Finally, the sequence is decomposed according to the length of the original feature sequence. Then, the final hidden states of different modals are concatenated on the feature dimension to construct the fusion feature $\mathcal{X}_{m}$.

The detailed generation algorithm of IM is described in Appendix E.1

A unimodal label generation module (ULGM) is integrated into our approach to capture unimodal-specific information. As shown in figure 1, we use input features $\mathcal{X}_{\{t,v,a\}}$ to generate unimodal final hidden states $\hat{y}_{\{t,v,a\}}$. During the prediction process, ULGM uses $h_{\{t,v,a\}}$ and ground truth multimodal labels to define positive and negative centers, which are determined based on the predicted unimodal labels and multimodal fusion representations. Next, we calculate the relative distance of each modal representation from the positive and negative centers. Then, we generate new unimodal labels $y_{\{t,v,a\}}^{i}$ from the unimodal labels to the ground truth multimodal label, where $i$ represents the $i$ training iteration. In this way, sentiment analysis can be more conducive to obtaining the distinguishing information of different modals, while maintaining the consistency of each modal.

Using the predicted results $\hat{y}_{\{m,t,v,a\}}$ and the ground truth multimodal label $y_{m}$ and the generated labels $y_{\{t,v,a\}}$, we implement a weighted loss to optimize our model.

The weighted loss is defined by Equation 15 whereas the unimodal loss for each modality is defined as Equation LABEL:u_loss

Where $\mathcal{B}$ denotes the appointed batch size.

The performance comparison of all methods on MOSI, MOSEI and CH-SIMS are summarized in Table 1 and Table 2.

For all metrics, the best results are highlighted in bold, the second-best results are double-underlined, and the third-best results are single-underlined. ^† denotes that the model is sourced from the GitHub page^4.1 and the scores are reproduced, ^∗ denotes the result is obtained directly from the original paper.

GSIFN is trained end-to-end without any pre-training. Thus, all replicated results ensure the consistency and fairness in the experimental environment. Except for MulT, the reproducible results differ greatly from the original results.

In Table 1, for a fair comparison in CMU-MOSI and CMU-MOSEI, we split models into two categories based on data state: Unaligned and Aligned. For Acc-2 and F1, the left of the "/" corresponds to "negative/non-negative" and the right corresponds to "negative/positive".

As shown in Table 1 and 2, GSIFN outperforms all of the previous SOTAs in most of the metrics in all of the datasets. The comparable metrics in CMU-MOSEI (Acc-7, MAE, Corr) and CMU-MOSI (MAE) also reach at least the third-best performance. GSIFN achieves all-modal-in-one fusion and enhanced self-supervised learning, which ensures its superior performance over previous SOTA.

In this session, we will discuss our ablation study on modules in Table 3. Further ablation study results are in Appendix A.2

There are three main modules in our model, including Graph-Structured Interlaced-Masked Multimodal Transformer (GsiT) for multimodal fusion, extended LSTM with matrix memory (mLSTM) for vision, audio temporal enhancement, Unimodal Label Generation Module (ULGM) for self-supervision. In Table 3, w/o denotes the absence of the corresponding module in the model.

The results in Table 3 indicate all the modules are necessary for achieving SOTA performance. GsiT module realizes all-modal-in-one Transformer-based fusion, without module GsiT, the performance of the whole model has a substantial decrease in all metrics. GsiT is the core module of GSIFN, and it is especially important in fine-grained tasks. Without module mLSTM, the performance weakens mainly on fine-grained metrics, it is a necessary module of GSIFN. Without module ULGM, the performance weakens on almost all the metrics. ULGM is significant to GSIFN in coarse-grained tasks.

We discuss the GsiT comparison of performance and efficiency with MulT and mLSTM efficiency in this section. In particular, Params denotes the number of parameters, FLOPS denotes floating-point operations per second.

GsiT and MulT MulT mainly uses CMA to realize effective modal fusion. Like the core module of GSIFN, which is GsiT, MulT realizes complete fusion and post-fusion enhancement separately in 9 Transformers (Vaswani et al., 2017). However, GsiT uses IM to realize the graph structure construction in MGEs. Each of the MGE contains trimodal information altogether. GsiT reduces the number of Transformers from 9 to 3. Through weight-sharing without information disorder, each of the Transformers in GsiT can completely fuse trimodal sentiment information all in one, achieving better weight regularization and fusion performance at the same time.

For a fair comparison, we trained MulT and GsiT with the same hyperparameters. The experiments are shown in the Table 4. GsiT outperforms MulT in all metrics. The Params and FLOPS of GsiT are much lower than MulT.

Vision/Audio Encoder Efficiency As shown in Table 4, Params and FLOPS of widely used non-verbal modal feature extractors. Vision Transformer (ViT)(Dosovitskiy et al., 2021), Wav2Vec(Schneider et al., 2019) and Whisper(Radford et al., 2023) are employed to extract high-quality features. We employ mLSTM to enhance low-quality features extracted by COVAREP(Degottex et al., 2014) (for audio), OpenFace(Baltrusaitis et al., 2016) (for vision). The Params and FLOPS of mLSTMs is way lower than ViT, Wav2Vec and Whisper.

To analyze the efficiency and effectiveness of mLSTM, GSIFN is compared with two models using large model extractors: TETFN, AcFormer(Zong et al., 2023). As shown in Table 6, GSIFN performs close even better than TETFN and AcFormer in most of the metrics.