Similarity over Factuality: Are we making progress on multimodal out-of-context misinformation detection?

We define the task of evidence-based out-of-context detection as follows: given dataset $(I^{v}_{i},T^{v}_{i},I^{e}_{i},T^{e}_{i},y_{i})_{i=1}^{N}$ where $I^{v}_{i},T^{v}_{i}$ represent the image-text pair under verification, $I^{e}_{i},T^{e}_{i}$ image and textual external information, or evidence, retrieved for the pair and $y_{i}\in\{0,1\}$ is the pair’s ground-truth label, being either truthful (0) or out-of-context (1), the objective is to train classifier $f:(\mathcal{I}^{v},\mathcal{T}^{v},\mathcal{I}^{e},\mathcal{T}^{e})\rightarrow\hat{y}^{v}$.

As shown in Fig.2(a), given feature extractor $F(\cdot)$ and extracted features $F_{I^{v}}$, $F_{T^{v}}$, $F_{I^{e}}$, $F_{T^{e}}$, we use cosine similarity $s$ to calculate the Multimodal Similarities (MUSE) vector $S^{v/e}$ between $s(F_{I^{v}},F_{T^{v}})$ image text pairs, $s(F_{I^{v}},F_{I^{e}})$ image to image evidence, $s(F_{T^{v}},F_{I^{e}})$ text to image evidence, $s(F_{I^{v}},F_{T^{e}})$ image to text evidence, $s(F_{T^{v}},F_{T^{e}})$ text to text evidence and $s(F_{I^{e}},F_{T^{e}})$ image evidence to text evidence.

Afterwards, the $S^{v/e}$ vectors are used to train a machine learning classification such as Decision Tree (DT), Random Forest (RF) and Multi-layer Perceptron (MLP), denoted as MUSE-DT/RF/MLP, respectively, or are integrated within the “Attentive Intermediate Transformer Representations” (AITR) network.

Attentive Intermediate Transformer Representations (AITR) attempts to model how human fact-checkers may iterate multiple times over the claim and collected evidence during verification, drawing various inferences and interpretations at each pass, exploring both general and fine-grained aspects and finally reassessing the entire process while assigning different weights to different aspects at each stage of analysis.

As shown in Fig.2(b), AITR utilizes a stack of $n$ Transformer encoder layers $E(\cdot)=[E_{1},E_{2},\cdots,E_{n}]$ with $h=[h_{1},h_{2},\cdots,h_{n}]$ number of multi-head attention enabling both stable attention (e.g., $h=[8,8,8,8]$) and granular attention, ranging from general to fine-grained (e.g., $h=[1,2,4,8]$) or from fine-grained to general (e.g., $h=[8,4,2,1]$). Given initial input:

$$x_{0}=[C_{0};F^{v};F^{e};S^{v/e}]$$

(1)

where $C_{0}$ is a learnable classification token, $F^{v}$ represents element-wise modality fusion [24] defined as $F^{v}=[F_{I^{v}};F_{T^{v}};F_{I^{v}}+F_{T^{v}};F_{I^{v}}-F_{T^{v}};F_{I^{v}}*F_{T^{v}}]$, $F_{e}=[F_{I^{e}};F_{T^{e}}]$ and “;” denoting concatenation, intermediate Transformer outputs are given by:

$$x_{i}=E_{i}(x_{i-1})\quad\text{for}\quad i\in\{1,2,\ldots,n\}$$

(2)

From each intermediate output, we extract the processed classification tokens $\mathcal{C}=[C_{1},C_{2},\ldots,C_{n}]$ and apply the scaled-dot product self-attention mechanism:

$$\mathcal{C}_{a}=\text{softmax}\left(\frac{Q\cdot K^{T}}{\sqrt{d}}\right)\cdot V$$

(3)

with fully connected layers $Q=\textbf{W}_{q}\cdot\mathcal{C}$, $K=\textbf{W}_{k}\cdot\mathcal{C}$, $V=\textbf{W}_{v}\cdot\mathcal{C}$ and $\textbf{W}_{q},\textbf{W}_{k},\textbf{W}_{v}\in\mathbb{R}^{d\times d}$. Afterwards, we use average pooling to calculate $\mathcal{C}_{p}=\frac{1}{n}\sum_{i=1}^{n}\mathcal{C}_{a}[:,i,:]$ and a final classification layer to predict $\hat{y}^{v}=\textbf{W}_{1}\cdot\text{GELU}(\textbf{W}_{0}\cdot\mathcal{C}_{p})$ with $\textbf{W}_{0}\in\mathbb{R}^{1\times d}$ and $\textbf{W}_{1}\in\mathbb{R}^{d\times 1}$.

We utilize the NewsCLIPpings Merged/Balanced dataset, comprising 85,360 samples in total [14], 42,680 “Pristine” or truthful $\mathcal{I}^{v},\mathcal{T}^{v}$ pairs sourced from credible news sources -as provided by the VisualNews dataset [34]- and 42,680 algorithmically created OOC pairs. Specifically, OOC pairs are generated by mismatching the initial image or text with another, utilizing semantic similarities, either CLIP text-to-image or text-to-text similarities, SBERT-WK for text-to-text person mismatching, and ResNet Place for scene mismatching. Furthermore, we utilize the VERITE evaluation benchmark [26] comprising 1,000 annotated samples, 338 truthful pairs, 338 miscaptioned images and 324 out-of-context pairs.

For NewsCLIPpings, we use the external evidence $\mathcal{I}^{e},\mathcal{T}^{e}$ as provided by [21] comprising up to 19 text evidence and up to 10 image evidence for each $I^{v}_{i},T^{v}_{i}$ pair, collected via Google API; totaling to 146,032 and 736,731 textual and image evidence, respectively. Specifically, the authors employ cross-modal retrieval, namely the text $T^{v}_{i}$ is used to retrieve potentially relevant image evidence $I^{e}_{i}$ and image $I^{v}_{i}$ to retrieve potentially relevant textual evidence $T^{e}_{i}$. We use the same Training, Validation and Testing sets as prior works to ensure comparability. For VERITE, we employ the external evidence as provided by [24].

Instead of utilizing all provided evidence as in [21], we follow [24], in re-ranking the external evidence based on CLIP [25] intra-modal similarities (image-to-image evidence, text-to-text evidence). We only select the top-1 items, as leveraging additional items was shown to degrade performance by introducing less relevant and noisy information into the detection model.

Nach [21, 24] we use the pre-trained CLIP ViT B/32 and ViT L/14 [25] as the backbone encoders in order to extract visual $F_{I^{v}},F_{I^{e}}\in{R}^{d\times 1}$ and textual features $F_{T^{v}},F_{T^{e}}\in{R}^{d\times 1}$ with dimensionality $d=512$ oder $d=768$ for CLIP ViT B/32 and L/14, respectively. Unless stated otherwise, we employ L/14 while using B/32 only for comparability purposes with some older works. We use the “openai” version of the models as provided by OpenCLIP²²2https://github.com/mlfoundations/open_clip.

We train each model on the NewsCLIPpings train set, tune the models’ hyper-parameters on the validation set and report the best version’s accuracy on the NewsCLIPpings test set and unless stated otherwise, as in Table 5, we report the “True vs OOC” accuracy for VERITE.

To ensure comparability with [24] on VERITE, we report the mean “out-of-distribution cross-validation" (OOD-CV) accuracy for VERITE in Table 2. Specifically, we validate and checkpoint a model on a single VERITE fold (k=3) while evaluating its performance on the other folds. We then retrieve the model version (hyper-parameter combination) that achieved the highest mean validation score and report its mean performance of the testing folds.

We train AITR for a maximum of 50 epochs, with early stopping and check-pointing set at 10 epochs to prevent overfitting. The AdamW optimizer is utilized with $\epsilon=1e-8$ and $\text{weight decay}=0.01$. We employ a batch size of 512 and a transformer dropout rate of 0.1 During hyperparameter tuning, we explore learning rates $lr\in\{1e-4,5e-5\}$, transformer feed-forward layer dimension $z\in\{256,1024,2048\}$ and for $h$ we try the following values $[4,4,4,4],[8,8,8,8]$, $[1,2,4,8]$, $[8,4,2,1]$. In the ablation experiments that do not leverage intermediate transformer representations, we exclude the $h=[1,2,4,8]$ and $[8,4,2,1]$ configurations. To ensure reproducibility of our experiments, we use a constant random seed of 0 for PyTorch, Python Random, and NumPy.

Table 1 presents the ablation study results for AITR which consistently achieves the highest performance among all ablation configurations, underscoring the importance of each component. Specifically, substituting the attention mechanism with max pooling or weighted pooling leads to a notable reduction in performance across both datasets. Similarly, using the default transformer encoder (Pooling = None) without leveraging intermediate representations lowers performance. Notably, the most critical component of AITR appears to be MUSE, as removing it significantly deteriorates the model’s performance, especially on VERITE, in both AITR and the default Transformer encoder. The best AITR performance was achieved with $h=[1,2,4,8]$, $z=2048$ and learning rate $5e-5$.

In comparison with the current SotA, as shown in Table 2, MUSE-MLP competes and even outperforms much more complex architectures on NewsCLIPpings. Specifically, MUSE-MLP (90%) performs similar to RED-DOT (90.3%) while surpassing SNIFFER (88.4%), ECENet (87.7%), SEN (87.1%) and CCN (84.7%). Notably, MUSE-MLP also significantly outperforms RED-DOT on VERITE, with +4.8% relative improvement. Furthermore, integrating MUSE within AITR, significantly outperforms the SotA on NewsCLIPpings by +3.3% and VERITE by +7.5%.

While this study primarily focuses on evidence-based approaches, we may also note that MUSE-MLP with $s(F_{I^{v}},F_{T^{v}})$ and no external evidence, achieves 80.7% on NewsCLIPpings, as seen in Table 4, and thus outperforms complex and resource-intensive architectures such as the Self-Supervised Distilled Learning [18] (71%) that uses a fully fine-tuned CLIP ResNet50 backbone on NewsCLIPpings, the Detector Transformer [15] (77.1%) and even competes against RED-DOT without evidence (81.7%) [24].

Furthermore, we examine the contribution of each similarity measure within $S^{v/e}$. Table 3 illustrates the performance and feature importance by the Decision Tree and Random Forest classifiers. We observe that both classifier put the highest emphasis on the image-text pair similarity $s(F_{T^{v}},F_{I^{e}})$ followed by image to image evidence $s(F_{I^{v}},F_{I^{e}})$ and text to text evidence $s(F_{T^{v}},F_{T^{e}})$.

Table 4 demonstrates an ablation of MUSE-MLP classifier on NewsCLIPpings (N) and VERITE (V), while excluding certain similarities. We observe that employing $S^{v/e}$ with all 6 similarity measures consistently achieves the highest overall accuracy (N=89.86, V=80.54) on both datasets. Therefore, each similarity measure contributes to some extend to the overall performance. Nevertheless, among single similarity experiments, we observe that $s(F_{T^{v}},F_{I^{e}})$ and $s(F_{I^{v}},F_{T^{e}})$ yield near-random performance while image-text pair similarities $s(F_{I^{v}},F_{T^{v}})$ yield the highest performance (N=80.69, V=70.89), followed by image to image evidence $s(F_{I^{v}},F_{I^{e}})$ (N=79.86, V=68.02) and then text to text evidence $s(F_{T^{v}},F_{T^{e}})$ (N=71.83, V=52.19), where performance, especially on VERITE, drops significantly. Similarly, removing the image-text pair $s(F_{I^{v}},F_{T^{v}})$ results in a notable drop performance with N=85.64% and V=69.68%. Again, similar to the Random Forest and Decision Tree classifiers, it is $s(F_{I^{v}},F_{T^{v}})$ that has the highest contribution and is followed by $s(F_{I^{v}},F_{I^{e}})$ and $s(F_{T^{v}},F_{T^{e}})$.

As shown in Fig. 3, MUSE-MLP maintains high performance on both dataset with only using 25% of the NewsCLIPpings training set. Notably, MUSE-RF maintains high performance even when trained with 1% of the training set, which translates to only 710 samples. Surprisingly, even with 0.1% and 0.05% of the dataset, or 71 and 36 samples, respectively, the performance of MUSE-RF does not completely deteriorate. This means that the patterns that MUSE-RF relies on are simple enough that can be learned from even from a few tens or hundreds of samples.

By examining Fig.4, illustrating the distributions of the 6 similarity measures in NewsCLIPpings, we observe clear differences between Truthful and OOC distributions, primarily on $s(F_{I^{v}},F_{T^{v}})$, $s(F_{I^{v}},F_{I^{e}})$, $s(F_{T^{v}},F_{T^{e}})$ and $s(F_{I^{e}},F_{T^{e}})$. Indicatively, the median values of $s(F_{I^{v}},F_{T^{v}})$, $s(F_{I^{v}},F_{I^{e}})$, $s(F_{T^{v}},F_{T^{e}})$ are 0.27, 0.91, 0.63 for Truthful pairs and 0.19, 0.69, 0.32 for OOC pairs. In contrast, $s(F_{T^{v}},F_{I^{e}})$ and $s(F_{I^{v}},F_{T^{e}})$ demonstrate mostly overlapping distributions between True and OOC classes which explains why they result in near-random performance in single-similarity experiments of Table 4.

In Fig. 5 we observe that the similarity distributions of VERITE exhibits relatively similar “True vs OOC” distributions with NewsCLIPpings in terms of $s(F_{I^{v}},F_{T^{v}})$ and $s(F_{I^{v}},F_{I^{e}})$ but not $s(F_{T^{v}},F_{T^{e}})$ that have mostly overlapping distributions. Indicatively, the median values of $s(F_{I^{v}},F_{T^{v}})$, $s(F_{I^{v}},F_{I^{e}})$, $s(F_{T^{v}},F_{T^{e}})$ are 0.31, 0.83, 0.32 for Truthful pairs and 0.24, 0.69, 0.28 for OOC pairs.

Importantly, we also observe that “True vs Miscaptioned” distributions are overlapping on $s(F_{I^{v}},F_{I^{e}})$ and that $s(F_{T^{v}},F_{T^{e}})$ similarities of the “Miscaptioned” class is skewed towards higher similarity, with median values of 0.29, 0.82 and 0.46 for $s(F_{I^{v}},F_{T^{v}})$, $s(F_{I^{v}},F_{I^{e}})$ and $s(F_{T^{v}},F_{T^{e}})$, respectively, thus inverting the pattern found in NewsCLIPpings. As a result, as seen in Table 5, while MUSE and AITR exhibit high performance on VERITE in terms of “True vs OOC”, their performance completely degrades on the “True vs Miscaptioned” evaluation.