Cross-Modal Augmentation for Few-Shot Multimodal Fake News Detection

Unimodal fake news detection aims to extract significant semantics from either news texts or images. Given the precision of semantics in text, previous approaches have concentrated on the task of text-based unimodal fake news detection. Early works focused on analyzing statistical characteristics of text (e.g., length, punctuation, exclamation marks) [23] and metadata (e.g., likes, shares) [24, 25] for manual fake news detection. However, these manual feature engineering approaches are time-consuming and struggle with processing large-scale, real-time data [26, 27].

The advent of deep learning has significantly advanced automated fake news detection. These methods primarily utilize deep learning models like BiLSTM [28, 29], GNNs [30], and pre-trained models (e.g., BERT, GPT) [31, 32, 33] to analyze text features, extracting various attributes such as emotional [34], stance-based [35], and stylistic elements [36]. However, the recent proliferation of multimodal information (text, images, videos) in social networks has shifted the propagation of fake news from solely text-based to multimodal formats.

Multimodal methods employing cross-modal discriminative patterns have been introduced, aiming to enhance performance in fake news detection. For example, MCAN [36] employs multiple co-attention layers to more effectively integrate textual and visual features in detecting fake news. CAFE [5] quantifies cross-modal ambiguity through the assessment of the Kullback-Leibler (KL) divergence among the distributions of unimodal features. LIIMR [37] determines the modality that exhibits greater confidence in the context of fake news detection. COOLANT [7] focuses on improving the alignment between image and text representations, utilizing contrastive learning for finer semantic alignment and cross-modal fusion to learn inter-modality correlations. However, these approaches are limited by the need for extensive annotated data in the context of emerging topics.

Few-shot learning is designed to master new tasks using a limited number of labeled examples [38]. Current few-shot learning methodologies, such as prototypical networks, acquire class-specific features in metric spaces for swift adaptation to novel tasks [39, 40]. Within computer vision, the concept of few-shot domain adaptation is explored in image classification for transferring knowledge to novel target domains [41, 42]. In natural language processing, meta-learning is suggested as a means to enhance few-shot learning performance in tasks like language modeling [43, 44] and misinformation detection [45, 46]. To our knowledge, the application of few-shot multimodal fake news detection through cross-modal augmentation remains unexplored in existing literature.

Meanwhile, previous multimodal learning approaches have sought to enhance unimodal tasks by leveraging data from various modalities [47, 48]. With multimodal pre-trained models achieving notable success in classic vision tasks [22, 49], there is a growing interest in formulating more efficient cross-modal augmentation techniques.

However, the prevailing techniques are based on successful strategies originally designed for multimodal foundational models. For example, CLIP utilizes linear probing [50, 51] and comprehensive fine-tuning [52] in its application to downstream tasks. CLIP-Adapter [53] and Tip-Adapter [54] draw inspiration from parameter-efficient finetuning approaches [55] that focus on optimizing lightweight MLPs while maintaining a fixed encoder. However, all the aforementioned methods, including WiSE-FT [56], employ an alternative modality, such as textual labels, as classifier weights, and continue to compute a unimodal Softmax loss on few-shot tasks. In contrast, this paper demonstrates the enhanced effectiveness of incorporating additional modalities as training samples.

Initially, unimodal few-shot FND learns from a labeled dataset of $(x,y)\in X$, where $x$ is either the text or image passing to a pre-trained feature encoder $\phi(\cdot)$. The ultimate goal is to allocate a binary classification label of $y\in\{0,1\}$, in which 0 denotes real news and 1 denotes fake news. We assume only an n-shot subset $(x_{i},y_{i})$ from $X$ is provided for training, where $i\in[1,n]$ (i.e., $n$ samples per class); the rest of $X$ is used as the test set.

Therefore, the standard unimodal FND can be denoted as minimizing the cross-entropy loss $L$:

where $y_{i}^{\prime}$ is the model inference from the linear classifier $MLP$ after softmax.

where $f$ is the feature representation from an MLP layer after the unimodal feature encoder, and $w_{y}$ and $w_{y^{\prime}}$ are the weights of the ground truth label and the predicted label, respectively.

To extend to multimodal FND, we assume that for each training sample, $f$ is a combination of five feature representations: 1) a text-only feature $f_{t}$; 2) an image-only feature $f_{m}$; 3) concatenation of L2 normalized $f_{c}=[f_{t}\oplus f_{m}]$, where $\oplus$ is the concatenation operation; 4) an image-text cross-attended feature $f_{mt}$; 5) a text-image cross-attended feature $f_{tm}$. The cross-attention mechanism, which swaps the text query $Q_{t}$ with the image query $Q_{m}$, to obtain the cross-attended feature $f_{mt}$ is denoted as follows:

In contrast, by swapping the image query $Q_{m}$ with the text query $Q_{t}$, the cross-attended feature $f_{tm}$ can be obtained:

where $K_{t}$ and $K_{m}$ represent the key vectors for text and image features respectively, $V_{t}$ and $V_{m}$ denote the corresponding value vectors, and $d$ refers to the dimensionality of the model.

For the sake of simplification, we assume that the number of $z$ different types of features are considered as distinct modalities. Therefore, each modality can be processed through the linear classifier MLP in the unimodal learning approach, as discussed above, to obtain five inferred probabilities.

Inspired by the Representer Theorem [57], which indicates that optimally trained classifiers can be depicted as linear combinations of their training samples, we concatenate the five inferred probabilities as a new input to a meta-linear MLP classifier for making the final prediction:

Instead of optimizing modality-specific weights independently, linear classification through the proposed CMA simultaneously determines all weights to minimize the training loss. Consequently, we convert the standard n-shot classification to an $(n\times z)$-shot problem. The training details for CMA are presented in Algorithm 1.

Three publicly available datasets are utilized for evaluation.

PolitiFact [13] comprises a dataset of political news categorized as either fake or real by expert evaluators and is part of the benchmark FakeNewsNet project. Using the provided data crawling scripts, news with no images or invalid image URLs are removed, resulting in 198 multimodal news articles.

GossipCop [13] features entertainment stories rated on a scale from 0 to 10, with stories scoring less than five classified as fake news by the author of FakeNewsNet. Using the same retrieval strategies as PolitiFact, 6,805 multimodal news articles are collected.

Weibo [58], a dataset sourced from Chinese social media platforms, comprises a multimodal fake news collection featuring both text and images. Authentic news items were crawled from a reputable source (Xinhua News), and fake news was obtained from Weibopiyao, an official rumor refutation platform of Weibo, that aggregates content either through crowdsourcing or official rumor refutation efforts. The same pre-processing methods as in previous work [7] are followed, resulting in 7,853 Chinese news articles.

Notably, a news article might be accompanied by multiple images. To find the most relevant image, the cosine similarity between each image and its corresponding text is calculated, and the image-text pair with the highest similarity, as determined by the pre-trained CLIP, is retained. The resulting dataset statistics are presented in Table 1.

The pre-trained OpenAI CLIP (ViT-B-32) [22] and Chinese CLIP (ViT-B-16) [59] models are utilized to respectively extract text and image features for different languages. The hidden size for the cross-attention projection layer is 512, which is the same as the output dimension of CLIP encoders. The AdamW optimizer is employed with a learning rate of $1\mathrm{e}{-3}$ and a decay parameter of $1\mathrm{e}{-2}$. The model is trained for 20 epochs, with the optimal checkpoint being determined by peak validation performance. Early stopping is utilized with a patience of three epochs.

In the few-shot context, the model is trained using a restricted set of samples, selected from the dataset to form an n-shot scenario. Here, $n\in[2,8,16,32]$ represents the number of samples for each class, while the remainder of the samples are reserved for testing purposes. Given that the data quality of the sampled training set might significantly impact the model’s performance, data sampling is repeated 10 times with random seeds, and the average score is reported after excluding the highest and lowest scores.

The proposed CMA is benchmarked against 11 representative models. Specifically, we extensively compare the proposed method with unimodal approaches (1)-(3), multimodal approaches (4)-(6), and the few-shot approaches (7)-(11).

(1) dEFEND [60] utilizes the hierarchical attention network for FND. In this study, we remove the user comments from the original model.

(2) LDA-HAN [33] integrates pre-calculated topic distributions from Latent Dirichlet Allocation into a hierarchical attention network for text classification.

(3) FT-RoBERTa is a standard, fine-tuned version of the pre-trained language model RoBERTa; we use Huggingface Trainer to conduct the fine-tuning experiment.

(4) SpotFake [61] employs the pre-trained VGG and BERT for extracting image and text features, respectively, and then concatenating them for final classification.

(5) SAFE [62] transforms images into textual descriptions and utilizes the correlation between text and visual information for FND.

(6) CAFE [5] employs an ambiguity-aware multimodal strategy to adaptively aggregate unimodal features and their correlations.

(7) KPL [19] employs prompt learning in RoBERTa by enhancing it with external knowledge representations.

(8) M-SAMPLE [20] incorporates prompt learning with multimodal FND. It also applies a similarity-aware fusing to adaptively combine the intensity of multimodal representation for FND.

(9) PET [63] employs PLMs with task descriptions for supervised training, employing task-related cloze questions and verbalizers.

(10) KPT [64] enhances the label word space by incorporating class-related tokens that exhibit diverse granularities and perspectives.

(11) P&A [16] combines prompt-based learning with social alignment techniques and addresses label scarcity by using task-specific prompts in PLMs to elicit relevant knowledge.

Table 2 demonstrates the FND accuracy comparison between the proposed CMA and all the baselines at various few-shot settings over the three datasets.

Comparing with unimodal baselines. First, we assess the accuracy of both unimodal approaches and the proposed CMA to evaluate their performances. Overall, CMA outperforms the best unimodal approach, FT-RoBERTa, achieving a 15.6% enhancement in average accuracy across all datasets, demonstrating its superiority in few-shot scenarios.

Surprisingly, FT-RoBERTa emerges as the most accurate model among both unimodal and multimodal approaches, suggesting that conventional fine-tuning methods can reach competitive levels of performance solely through the analysis of textual information from fake news. However, this method necessitates increased epoch time due to the adjustment of numerous parameters in the pre-trained language model (as shown in Table LABEL:tab:params), making it impractical for real-world few-shot FND applications.

LDA-HAN yields the second best in accuracy among unimodal models, with dEFEND coming in next. This could be attributed to two factors: firstly, the vanilla LDA model struggles to effectively generate topics from short texts, a characteristic of the datasets from GossipCop and Weibo (as detailed in Table 1) used in LDA-HAN; secondly, the employment of GloVe embeddings for initializing LDA-HAN and dEFEND may not perform as effectively as the contextualized embeddings generated by the BERT family.

Comparing with multimodal baselines. We evaluate the performance of CMA in comparison with multimodal approaches. CMA outperforms the best multimodal baseline, CAFE, with a 27.4% improvement in average accuracy across all datasets. The reason might be that the complex architecture of multimodal approaches inherently comes with a large number of trainable parameters, which might easily lead to overfitting in few-shot scenarios.

Excluding FT-RoBERTa, all multimodal baselines outperform unimodal models on average, showing that the inclusion of the image modality can significantly affect model accuracy. While these multimodal approaches excel in scenarios with abundant data, their effectiveness heavily relies on the availability of high-quality annotated training samples, which may not be readily accessible during the initial stages of FND. Moreover, all multimodal approaches utilize pre-trained unimodal models, such as VGG, ResNet, and BERT, to independently extract features from images and text. Yet, since these unimodal models are trained separately, merging their extracted features during the multimodal fusion process could potentially introduce noise[20].

Comparing with few-shot baselines. The effectiveness of the proposed CMA is evaluated in comparison with the latest prompt-based few-shot models. CMA outperforms the best few-shot baseline, P&A, with a 3.3% improvement in average accuracy, showing that using unimodal features to assist multimodal probing without prompting the pre-trained language model could also benefit the FND task.

While P&A demonstrates performance on par with CMA, it requires the pre-calculation of a news proximity graph. However, such social context data may not always be accessible, particularly in datasets not sourced from Twitter, like Weibo. After analyzing PET and KPT, it’s evident that these methods yield comparable outcomes, likely due to variations introduced by the manually crafted verbalizers used in prompting. This underscores the significance of hand-designed discrete templates in prompt-based learning. Concurrently, M-SAMPLE, a multimodal adaptation of KPL, demonstrates superior performance, suggesting that incorporating image modality can significantly enhance FND effectiveness.

We investigate the impact of key components in CMA by assessing the framework’s performance in a range of complete and partial configurations. In each experiment, CMA is selectively utilized by removing different components, followed by training the framework from scratch. The results are averaged over five random seeds in each shot, and indicate the performance decay of CMA in the absence of each component in most configurations, underscoring the significance of each key module within CMA, as shown in Table 3.

Specifically, removing the cross-attention from the CMA (i.e., -cross) results in a slight decrease in accuracy, showing that the cross-attended features from text and image capture semantic correlations and contribute to improved performance. Further removal of the meta-linear layer from the CMA (i.e., -meta) transforms the model into a standard n-shot classification, where it simply classifies concatenated multimodal features. This leads to a significant decrease in accuracy, emphasizing the importance of jointly updating all modality-specific weights in a meta-linear classifier for cross-modal adaptation and accuracy improvement. The meta-linear layer integrates modality-specific features, resembling an ensemble that transforms n-shot classification into a more robust $(n\times z)$-shot problem, enhancing cross-modal adaptation in few-shot classification.

Additionally, experiments are performed by excluding either the image features (-img) or the text features (-txt), relying solely on the remaining modality for classification. Such setups led to additional reductions in accuracy, underscoring the comparative importance of text over image features in FND. This highlights the complexities in multimodal FND tasks, where the spatial discrepancies between visual and textual semantics tend to be more subtle than in broader multimodal datasets.

Given the selection of few-shot examples can significantly affect the model performance, we assess the stability of the CMA and other prompt-based baselines by measuring the standard deviation of accuracies in the few-shot settings, as shown in Figure 3.

Overall, the standard deviation for all models decreases in tandem with an increase in the number of n-shot settings, underscoring the importance of augmenting training examples in few-shot scenarios. This augmentation can be further observed that the standard deviation of the CMA tends to be the most stable among the few-shot approaches, indicating that the ensemble of unimodal features in the meta-linear layer can enhance the robustness of multimodal fusion in classification. Additionally, the GossipCop dataset exhibits greater instability compared to the PolitiFact dataset. This instability may be attributed to the semantic complexity in GossipCop, which is responsible for the lower accuracy across all models.

Given the CMA achieves the best performance with a surprisingly simple augmentation, we further explore its efficiency in comparison to other baseline models. Table LABEL:tab:params showcases a comparison of the accuracies and epoch times between baselines and the CMA. The average accuracy of each model is determined in a 16-shot setting as shown in Table 2, along with the recording of average epoch times for each model. All experiments are tested with batch size 32 on a single RTX 4090 GPU in the GossipCop dataset for a fair comparison.

Among unimodal models, dEFEND and LDA-HAN exhibit comparable accuracy and epoch times, attributed to their analogous hierarchical architectural design. While FT-RoBERTa exceeds the performance of various unimodal (e.g., 18% higher than dEFEND) and multimodal methods (e.g., 6.9% higher than CAFE), it requires modifying a significant number of trainable parameters, thus extending epoch durations (on average, four minutes per epoch) relative to other unimodal baselines.

In the multimodal models, SAFE yields the lengthiest epoch durations owing to its prerequisite for independently pre-generating image descriptions. Although Spotfake achieves the fastest epoch duration due to its simple concatenation of the image and text features from the BERT and VGG respectively, it achieves the worst performance compared with other models. CAFE achieves the best multimodal FND outcomes by integrating a degree of ambiguity in the similarity across text and image features, albeit at the cost of marginally increased model complexity and consequently, slightly extended epoch durations.

All few-shot baselines demonstrate significant improvements over both unimodal and multimodal counterparts, indicating the suboptimality of traditional methods in contexts with limited annotated data. Specifically, the integration of external knowledge into the prompt-tuning phase by both KPL and KPT results in comparable epoch durations. However, KPL’s design of an FND-specific prompt may underlie its superior performance over KPT. PET records the lengthiest epoch duration among the few-shot baselines, potentially due to the repeated fine-tuning of the PLM for reconfiguring input examples with the task description. P%A not only achieves the second-best performance but also the second-shortest epoch durations, benefiting from the integration of user engagements. However, it incorporates an external alignment module to correlate user engagement with the PLM’s predictions, consequently increasing epoch times relative to CMA. Finally, CMA is more efficient and precise as it avoids the need for extensive parameter fine-tuning and does not depend on intensive image augmentation processes. Additionally, the inclusion of linear probing layers atop the image and text features presents a more streamlined approach than extensive fine-tuning and precise-crafted complex model designs.

Real-world fake news demonstrates significant distribution discrepancies, which is also referred to as domain shift [zhu2022generalizing, zhu2022memory]. Consequently, automatic FND methods are required to rapidly adapt to emerging topics by using limited resources.

To address this, we investigate the cross-domain capability of the proposed CMA against three strong few-shot FND baselines (i.e., P&A, PET and KPT). Considering Politifact’s focus on political news using formal language and Gossipcop’s emphasis on entertainment and celebrity narratives in a more casual tone, we first utilize Politifact for training and Gossipcop for testing, later inverting this arrangement.

The outcomes following domain shift are presented in Table 5. Notably,while the CMA model records the highest average accuracy among the few-shot baselines, the performance of each model markedly differs from that observed in the comparison experiments (as shown in Table 2). For example, KPT exhibits the strongest performance in both 2- and 8-shot scenarios in Goss$\rightarrow$Poli. PET and P%A also achieve the highest performance in Goss$\rightarrow$Poli and Poli$\rightarrow$Goss respectively, highlighting the disparity between present few-shot FND methodologies and their adaptability to domain adaptation.

At last, we present a visual comparison of the features extracted by M-SAMPLE and CMA, both of which are multimodal few-shot approaches. This involves the visualization of multimodal features alongside an assessment of their semantic correlations. For each dataset, a specific sample is chosen, with the corresponding multimodal features depicted in Figure 4.

Observations indicate that: 1) CMA can capture more consistent features from the image-text pair of fake news than those of M-SAMPLE. For example, although both M-SAMPLE and CMA successfully correlate the flag in the image with the word “Chinese” in the text, CMA can also identify the semantic meaning of “moon landing” between the text and image in the PolitiFact example; 2) The proposed CMA is more accurate in capturing important features from the image than M-SAMPLE. For example, although both models can identify the person “Nicole Kidman” and “black tarantula” in both the text and the image in the GossipCop example, the image region of the tarantula slightly overlaps with that of Nicole Kidman provided by M-SAMPLE. This is even more obvious in the Weibo example, as CMA successfully captures the “blue” color bar in the toothpaste, but M-SAMPLE fails to do so.