M$\mathbf{5}$ – A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks

We chose the LMMs included in this benchmark for the following reasons: Firstly, we focussed on publicly available models released on Hugging Face except for GPT-4 Vision and Gemini Pro. Secondly, we included LMMs well-performing on popular multimodal English-only benchmark s such as MMMU (Yue et al., 2023) and MME (Fu et al., 2023). Thirdly, we aimed to cover a mixture of different model families and a broad model size spectrum, including small models with $3$B to $9$B, medium models with $10$B to $19$B, and large models with $20$B to $40$B parameters. For an overview of all models, including their number of parameters and other architectural details, see Table 8.

This section briefly introduces the existing datasets included in our benchmark. In addition to these, we crafted two novel datasets described in Section 4. For details about the languages covered by the datasets, please refer to Table 6.

Inspired by MaRVL, the goal of the M5-VGR dataset is to provide a visually grounded reasoning (VGR) evaluation dataset that covers a wide range of topologically different languages and, at the same time, visually represents a diverse set of cultures in which the respective languages are spoken. However, since the MaRVL dataset contains only five languages, we chose $11$ additional topologically diverse languages for our dataset. To guarantee visual and linguistic diversity and high data quality in our dataset, we hired professional native-speaker annotators of the respective languages to annotate the data. Moreover, we performed several rounds of data quality assessment in close collaboration with the annotators.

A task sample $s$ in M5-VGR contains two images $i_{a}$ and $i_{b}$, a textual visually grounded hypothesis $h$, and a binary label $c$ which is either true or false concerning the two visual premises (see Figure 3). More specifically, for each language $l\in\mathbb{L}$, we created $120$ tasks $s_{l}\in\mathbb{S}_{l}$ as follows: In the first step, we sampled $120$ unique images $a_{l}^{t}\in\mathbb{B}$ from our image basis so that each topic $t\in\mathbb{T}$ occurs at least once across all $12$ languages. Then, for each of the $120$ images, we randomly selected another image $b_{l_{2}}^{t}\in\mathbb{B}$ associated with another language $l_{2}\neq l\in\mathbb{L}$ that shares the topic $t$. In the third step, we asked the native-speaker annotators of the language $l$ to manually create a hypothesis $h$ and a label $c$ which is either true or false concerning the image premises $\left(a_{l}^{t},b_{l_{2}}^{t}\right)$. Further, the annotators were instructed to generate a hypothesis semantically related to the topic $t$ if possible.

With the M5-VLOD dataset, we introduce a novel multimodal task: Visio-Linguistic Outlier Detection. The objective of the task is to detect an outlier image from a set of images considering a textual statement. An example of the task is shown in Figure 4, where five images related to the topic “soap for hands and body” are shown. The machine-translated English statement is: “All the images show soap applied to the hands and body without anyone.”. Because only the first image shows a person, the statement is incorrect for the first image and, therefore, is considered the outlier image.

The dataset was collected similarly to M5-VGR, as described in the previous section. The major difference is that instead of sampling only one image in the second step, we sample four images so that a sample $s_{l_{0}}\in\mathbb{S}_{l_{0}}$ for language $l_{0}\in\mathbb{L}$ comprises of five images: $\{a_{l_{0}}^{t},b_{l_{1}}^{t},c_{l_{2}}^{t},d_{l_{3}}^{t},e_{l_{4}}^{t},\}$ associated with five different languages $\{l_{0},\dots,l_{4}\in\mathbb{L}\}$ that share one topic $t\in\mathbb{T}$. In the third step, we asked the native-speaker annotators of the language $l$ to manually create a textual statement $h$, valid for all but one of the images labeled as the outlier image.

Table 1 shows a clear pattern: Generally, LMMs perform significantly worse in non-English languages across all tasks. More specifically, the average performance across all models and datasets in English is $0.63$ versus $0.47$ in non-English languages. Most models have an average performance difference from English to non-English larger or equal to $0.12$. However, for GPT 4V and despite their much smaller size also for mBlip BloomZ, and mBlip T0, the difference is smaller than $0.1$. For the two mBLIP models, the authors explicitly stated in their paper the language distribution in the training data, which covers $96$ languages. Hence, it can be assumed that this is the reason for this slight absolute performance difference, and, further, this might indicate that GPT 4V was also trained in a multilingual fashion. Due to the difference in size and the architecture²²2While the architecture of GPT 4V is not known, it is likely different from the mBlip models’ architecture, which employs Q-Formers, rarely used in state-of-the-art LMMs. of the mBlip models and GPT 4V, applying this multilingual training strategy for LMMs would generally lead to more robust multilingual performance.

The average performance difference of the models is most significant on the MaXM, XM3600, and xFlickrCo datasets, for which the models are required to generate non-English text.

Interestingly, for the M5-VLOD dataset, the models that performed worse than the random baseline of $0.2$ in English performed better in non-English languages. An explanation for this could be false assumptions drawn from the English text. This finding also explains why the average English versus non-English performance disparity across all models is equal for the dataset and lies around the random baseline, indicating the challenge introduced by our dataset.

Figure 5 shows the average performances aggregated by language³³3We do not show all $36$ languages of XM3600 for better readability. or language taxonomy classes Joshi et al. (2020). These taxonomy classes indicated how well a respective language is represented and considered within the research field of NLP based on papers published at ⁢CL conferences. High-resource languages such as English or German are in Class 5, whereas low-resource languages such as Berber are in Class 0. For details about the languages and their taxonomy classes, please refer to Table 12.

As can be observed from Figure 5(a) and Figure 5(b), the models perform best in English, followed by other European languages across all datasets. Our newly presented M5-VLOD dataset is an exception, where the average performance for all languages is around the random baseline, indicating the challenge it implies. As expected, the models consistently perform worse on low-resource languages than on high-resource languages on all datasets. This is also displayed in Figure 5(c), where it can be observed that the average performance decreases with the language taxonomy class. Note that this is not precisely true for xFlickrCO and XVNLI because the average on Class-5 languages is lowered by outliers, as indicated by the large error bars. In contrast, the models performed comparably well in only one Class 3 or 4 language, respectively.

In Figure 6, we plot the English and non-English average performance on the employed datasets versus the models’ sizes in multiple regression plots. Note that, on the x-axes, we indicated the unknown sizes of GPT 4V and Gemini Pro V by “???”, which are estimated to be of magnitudes larger than all other models evaluated in this benchmark hence should be much further right. However, we did not do so to improve the readability of the plots.

In the figures, we can make several observations: Firstly, the average English performance is higher than the non-English performance for all models on all datasets. Secondly, the markers, which represent the average performance of a specific model on a dataset, show that the largest model does not always perform best and that the difference between smaller and larger models is often neglectable. The same finding is shown by the relatively flat slope of the regression lines. However, for the M5-VLOD and VGR datasets, the regression line for the average English scores is steeper, meaning that larger models perform considerably better than the smaller models. Since this work introduces the datasets and M5-VLOD even introduces a novel task, it can be concluded that larger models can better generalize to unseen data.

Inspired by Holtermann et al. (2024), we report the results of a language fidelity analysis, which assesses how often a model responds in the requested language on average. For this, we used GlotLIDv3 Kargaran et al. (2023) to predict the language based on the output text of the respective models. Since it is hard to predict the language of a word or a multi-word expression due to ambiguity, we selected the xFlickrCO dataset, where the expected response of a model is an image caption, i.e., a sentence, in one of eight languages. As it can be observed from Table 2, all models achieve (almost) perfect fidelity in English where, whereas for Japanese, Russian, and Turkish, the average fidelity drops to two-thirds. Interestingly, the small-sized mBLIP models have almost perfect fidelity in all languages, (slightly) surpassing larger models like InternVL V1.2+ and GPT 4V.

While the language fidelity of a model focuses on the generated text, we argue that the fidelity is also an indicator of the model’s general language capabilities. To prove this hypothesis, we computed Pearson correlation coefficients between the reported fidelity and the models’ performance on the datasets for the xFlickrCO languages. As shown in Table 17, there is a positive moderate or high correlation between the average fidelity and the average score for most datasets. However, for xGQA and M5-VLOD, there is only a minor positive average correlation.

Our benchmark and current research generally lack robust metrics for evaluating multilingual image captioning, especially for non-Latin script languages. The issue, which is the same for machine translation tasks, arises because of the nature of most metrics, such as chrF Popović (2017), CIDEr Vedantam et al. (2015), ROUGE Lin (2004), BLUE Papineni et al. (2002), or METEOR Banerjee and Lavie (2005), which are based on comparing word or character n-grams between the source and target sequence. For non-Latin scripts, tokenization or segmentation can be challenging because it might not contain spaces or punctuation, or the characters are logographic. Hence, their usability or effectiveness is doubtful in such scenarios because the metrics rely on tokenization.

Other metrics, such as BERTScore Zhang et al. (2020), CLIPScore Hessel et al. (2021), or COMET Rei et al. (2020), do not rely on the captions’ surface forms but on their token or sentence embeddings. However, they suffer from other issues: They require strong multilingual or cross-lingual encoder models capable of computing embeddings for many languages, which itself is a challenging task. Further, the scores computed with these metrics are often not calibrated across languages and thus not directly comparable between different languages.

A promising currently popular solution might be the use of robust multilingual state-of-the-art LLMs such as GPT 4o⁴⁴4https://openai.com/index/hello-gpt-4o/, Claude 3 Opus⁵⁵5https://www.anthropic.com/news/claude-3-family, or Gemini 1.5 Ultra⁶⁶6https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/ as a judge Zheng et al. (2024). However, this would require more computational and financial resources and, most importantly, more investigation.

The problem when employing and evaluating generative language models on question-answering tasks is that the models can generally output arbitrary token sequences. However, the gold label answers are limited and often comprise only a short phrase, a single word, or even a binary label. Hence, mapping the predicted answers to their gold labels is not straightforward, and the difficulty drastically increases in multilingual scenarios. The relaxed accuracy metric employed in this study (see Section A.1) has been found to occasionally incorrectly classify correct answers, leading to false negatives, especially in open vocabulary visual question answering (VQA). One way to address this issue is to leverage strong state-of-the-art LLMs as judges, as described above, to enhance the accuracy of the evaluations.

Another limitation of this and most, if not all, other current studies is grounded in the model prompting. Since different models might react differently to specific prompting styles, and we only employ a single prompt per dataset for all models⁷⁷7We do apply the model-specific prompt or chat templates, though. (see Figure 7), the results might not be optimal. This issue has been partially addressed by Ahuja et al. (2023a) but is out of the scope of this work.

Since the pace of current research in NLP, CV, and multimodal machine learning is swift, the models employed in our benchmarking exercise might be considered slightly outdated. Note that we considered models released until March 2024. Since then, numerous improved LMMs based on state-of-the-art LLMs, such as Llama3⁸⁸8https://ai.meta.com/blog/meta-llama-3 and novel image encoders techniques such as NaVIT Dehghani et al. (2024), have been publicly released. Because this was foreseeable, we designed our benchmark to be easily extendable with newer models, which we will include in future work.

This work introduced two datasets, M5-VGR and M5-VLOD, which comprise about $115$ samples for each of the $12$ languages. Compared to other datasets, they can be considered small. We will increase their sizes in future work to obtain more robust and generalizable results.

Currently, the M5 Benchmark comprises $5$ text-image tasks, i.e., VQA, VGR, VNLI, and image captioning, thus missing other suitable tasks like multimodal and multilingual summarization. Further, other multimodal multilingual VQA and VGR datasets have emerged while writing this paper. We will include both new tasks and new datasets in future versions of the M5.