With the success of LLMs, there is a trend to extend LLMs to handle multimodal information, called MLLMs. MLLMs, such as BLIP [12], KOSMOS [6], LLaMA-Adapter [4], and LLaVA [13, 10, 11], show promising progress in multimodal information understanding and reasoning. To achieve this, a typical MLLM is composed of an LLM, a modality encoder, and a projector to connect them. The modality encoder projects raw multimodal inputs into vectors to connect with LLMs [29].

One efficient method [4, 13] is to directly use a pretrained LLM and a pretrained modality encoder, such as CLIP [22]. To achieve this, LLaVA uses two training stages. The first stage aligns the text and image with image-text pairs by only training the projector between LLM and modality encoder, and the second stage fine-tunes the model on a visual instruction dataset, which ensure it can follow complex instructions like LLM, such as represent the multimodal inputs in our work.

While the impressive performance of MLLMs in understanding multimodal information and instruction following, the representation of multimodal information using MLLMs remains largely unexplored. Although recent studies [28, 7] have shown advancements in embedding texts with LLMs and scaling up has demonstrated improved performance in text representation, a significant challenge persists: the batch sizes requires to train multimodal embeddings models such as CLIP is signficantly larger than text embeddings models. For example, CLIP requires a batch size of 32k samples with contrastive learning, while the text embedding models such as E5 [28] only requires a batch size of 2k samples. Limited by the size of MLLMs, it can be very challenging to use the similar batch size like CLIP to train robust multimodal embeddings models.

CLIP [22], as a pioneering work on multimodal embeddings, has been widely used in subsequent works. CLIP uses separate encoders for image and text by aligning them with contrastive learning on large-scale image-text pairs. Despite its strong performance in text-image retrieval, CLIP has several limitations due to its internal framework. The text encoder of CLIP has a low capacity for understanding complicated text because it is pretrained on short image captions, which also limits CLIP’s performance on long text retrieval [32]. Additionally, due to the use of separate encoders, CLIP struggles to represent interleaved visual and language inputs, such as in composed image retrieval [16, 27].

To achieve universal multimodal embeddings, there are several works, such as and UNIIR [26], fine-tune CLIP on interlevaed dataset with fusion modal to fuse the visual and language information. There are also some works like VISTA [31] or UniVL-DR [17] feed text encoder with CLIP outputs to input visual information. However, it can harm the original text-image retrieval performance of CLIP, and is hard to make text encoder understand the visual information with only contrastive learning, which show poor zero-shot performance on composed image retrieval tasks.

To achieve universal multimodal embeddings, several works, such as UNIIR [26], fine-tune CLIP with a fusion model to integrate visual and language information. Other works, like VISTA [31] or UniVL-DR [17], feed the text embedding models with CLIP outputs to incorporate visual information. However, this approach can harm the original text-image retrieval performance of CLIP and makes it difficult for the text embedding models to understand visual information using only contrastive learning. As a result, these methods show poor zero-shot performance on composed image retrieval tasks. Moreover, these methods require large interleaved training data to achieve universal multimodal embeddings. Collecting such high-quality interleaved pairs for performing contrastive learning is more challenging than gathering image-text pairs or text pairs. This process can require complex annotation and sometimes even synthesizing data from GPT-4 [31].

Previous works [18] have demonstrated the existence of a modality gap between text and image embeddings in multimodal models like CLIP [22], which can negatively impact the performance of multimodal embeddings. Similarly, we observe this phenomenon when using MLLMs to represent multimodal inputs.

We visualize the distribution of multimodal embeddings from MLLM in Figure 3(a) following  [18]. For implementation, we use the last token embeddings of LLaVA-NeXT-8B [19] to represent the images and captions of COCO. The embeddings are obtained directly from MLLM without fine-tuning and visualized with PCA. Compared to CLIP, although MLLM represents the image and text with the same encoder, the multimodal embeddings from MLLM show a clear modality gap between text and image embeddings.

To unify multimodal embeddings, we propose a prompt-based representation method with MLLMs inspired by previous text embedding work [7]. The key idea is to explicitly instruct MLLMs to represent the multimodal inputs into words. We can use prompts like <text> \n Summary of the above sentence in one word: to represent the text and <image> \n Summary above image in one word: to represent the image. We notice these prompts directly remove the modality gap between text and image embeddings, as shown in Figure 3(b). For the design of the prompts, it has two parts: the first part is about extracting the meaning of the multimodal inputs, and the second part is about compressing the meaning into the next token embeddings and unifying the multimodal embeddings by using in one word:. Specifically, the embeddings of image and caption about a plane in Figure 1 will have a close distance to the token embeddings of “Plane”, “Air”, “Flying” and “Above”, which represent multimodal inputs based on the corresponding meaning instead of their modality. By removing modality gap, it also allows MLLMs to represent interleaved inputs for tasks like composed image retrieval. We show our method significantly improves the performance of MLLM on multimodal retrieval tasks in Table 6.

By unifying multimodal embeddings, we propose single modality training for multimodal embeddings, as shown in Figure 2. Since there is no longer a modality gap in the embeddings, we can transfer the single modality representation capabilities to multimodal embeddings by training on text pairs only. In this way, our method is trained without any visual or interleaved inputs and no longer relies on multimodal training data, which can be difficult to collect.

To achieve it, E5-V trains MLLMs with contrastive learning on text pairs. Since there are no visual inputs during training, we remove the modality encoder and projector and only remain the LLM of MLLM. For the training data, we simply use sentence pairs from NLI datasets following [5], which have no relation to the multimodal tasks. Each sentence pair $(x_{i},x_{i}^{+},x_{i}^{-})$ has a positive sentence $x_{i}^{+}$ and a negative sentence $x_{i}^{-}$ for the input sentence $x_{i}$. We use the prompt <text> \n Summary above sentence in one word: to embed the sentence pairs into $(\mathbf{h}_{i},\mathbf{h}_{i}^{+},\mathbf{h}_{i}^{-})$. The training objective is folowing:

where $\tau$ is the temperature hyperparameter and $N$ is the batch size and in contrastive learning. Compared to multimodal training, we find single modality training achieve better performance on multimodal retrieval tasks, while significantly reducing the training cost in Table 7.

We first benchmark E5-V on text-image retrieval with Flickr30K [30] and COCO [15] to evaluate zero-shot image retrieval and zero-shot text retrieval performance. For the baselines, we select the following text-image retrieval models: CLIP with ViT-B and ViT-L[22], BLIP with ViT-L[14], and the large CLIP model EVA-02-CLIP with 5B parameters [23]. All baselines are trained with contrastive learning on large-scale image-text pairs using separate visual and language encoders, while cannot represent interleaved visual and language inputs. For the prompt used in text-image retrieval tasks, we use the following prompts to represent image and text inputs, respectively:

We report Recall@K (R@K) for K=1, 5, 10 with image retrieval and text retrieval in Table 1. Compared to strong baselines, E5-V, as a universal multimodal embeddings model, achieves competitive performance on both the Flickr30K and COCO datasets.

Compared to EVA-02-CLIP, which uses a 4.4B visual encoder with contrastive learning on large-scale image-text pairs [23], E5-V shows better ability on zero-shot image retrieval, while it is only trained on text pairs with contrastive learning. It is worth noting that E5-V uses the same visual encoder as CLIP ViT-L and keeps it frozen during training. Although E5-V shares the same visual encoder with CLIP, referring to the same way to encode visual inputs, E5-V demonstrates significantly better performance than CLIP on both the Flickr30K and COCO datasets for image retrieval and text retrieval tasks. Specifically, in image retrieval tasks, E5-V outperforms CLIP ViT-L by 12.2% on Flickr30K and 15.0% on COCO with Recall@1.

E5-V shows a strong ability to transfer single modality representation capabilities to multimodal embeddings by following task-specific prompts that were not included in the training data. It also seamlessly integrates visual and language information into the same embedding space with prompts. For unseen prompt in training, E5-V can successfully follow it like “Summary the above image in one word:” to represent the image according to its semantics.

To understand the effectiveness of E5-V in representing interleaved visual and language inputs, we evaluate it on composed image retrieval tasks with two popular datasets: FashionIQ [27] and CIRR [16]. This task focuses on retrieving images based on interleaved inputs, which requires the model to retrieve target images based on modified reference images, where the modification is described in the text. For FashionIQ, it contains three subtypes of fashion products: Dress, Shirt and Toptee. Given a picture of a fashion product and a modification corresponding to the style, the model needs to retrieve the target image that matches the modification. For CIRR, it extend FashionIQ on real-life images, which has more diverse images and modifications.

We compare E5-V with several zero-shot image-composed baselines: Pic2Word [24], Context-I2W [25], LinCIR [3], the LLM-based method CIReVL [9], and the current state-of-the-art method iSEARLE-XL [1]. For a fair comparison, we report the results of all baseline models using the large visual encoder CLIP ViT-L, as in E5-V. Note that the E5-V also freezes visual encoder same as other baselines. These baselines are designed exclusively for zero-shot composed image retrieval tasks and can not apply to other tasks. Most of the baselines are not end-to-end embedding interleaved inputs, which introduce complex pipelines like textual inversion. For example, CIReVL requires captioning an image first, generating the target image caption based on LLMs, and then retrieving the target image based on the caption. However, E5-V can directly represent the interleaved visual and language inputs with prompts without any textual inversion.

To represent the interleaved inputs for E5-V, we use the following prompts for FashionIQ and CIRR. For FashionIQ, which requires the model to mainly represent the style of the fashion product, we can directly let E5-V represent the style of the corresponding fashion products. Since the evaluation of FashionIQ is split into three subtypes, including Dress, Shirt, and Toptee, we can also provide the subtype information in the prompts. For CIRR, we can directly let E5-V modify the image based on the modification described in the text and then represent the modified image in one word. Although these prompts are unseen during training and have a complex format, E5-V can still correctly represent the interleaved inputs, even in specific domains like fashion products.

We report the composed image retrieval performance of CIRR and FashionIQ on Table 2 and 3. All methods use CLIP ViT-L as the visual encoder. The results of other baselines are directly from their original papers. Following previous works, we report Recall@K for K=1, 5, 10, and 50 on CIRR test set with their test evaluation server, and report Recall@K for K=10, 50 on three subsets of FashionIQ. For the settings of E5-V, we use original E5-V without additional fine-tuning on specific datasets and tricks like textual inversion. E5-V directly represents the interleaved inputs and image inputs with above prompts and uses the last token embeddings to represent the multimodal embeddings.

Compared to zero-shot composed image retrieval baselines, E5-V achieves significant improvements on both the CIRR and FashionIQ datasets without using techniques like textual inversion or annotation. Specifically, E5-V outperforms the current state-of-the-art method iSEARLE-XL by 8.50% on Recall@1 and 10.07% on Recall@5 on CIRR. For FashionIQ, E5-V outperforms by 2.56% on Recall@10 and 4.24% on Recall@50 compared to iSEARLE-XL, which demonstrates the great ability of E5-V understanding the interleaved visual and language inputs and representing them correctly.

By unifying multimodal representations into the same embedding space with prompts, E5-V demonstrates a strong ability to understand text through visual input and represent it accurately. To validate this, we designed an image-image retrieval task based on Flickr30K and COCO, referred to as I2I-Flickr30K and I2I-COCO. We rendered all textual captions in the datasets as images and used the embeddings of these images as the caption embeddings. The detailed implementation of text rendering can be found in Appendix A. For the prompts of E5-V, we simply used the image prompt in text-image retrieval tasks to represent images.

We report the results of CLIP, BLIP, EVA-02-CLIP, and E5-V in Table 4. Compared to text-image retrieval tasks, we notice that the performance of baselines drops significantly on image-image retrieval tasks, which indicates the difficulty of understanding text through visual input and representing it accurately. Due to separate visual and language encoders, these models struggle to understand the textual information via images by using their visual encoders. However, E5-V correctly represents text through visual input and shows outstanding results on these two datasets.

Since E5-V is trained on text pairs, it also shows strong performance in representing textual inputs. We evaluate E5-V on the sentence embedding tasks using 7 STS tasks. Compared to other sentence embedding methods, including SimCSE-RoBERTa [5], PromptRoBERTa [8], and LLM-based methods such as SGPT [20], ST5-Enc [21], and PromptEOL [7], E5-V, as a universal multimodal model, achieves the best performance on the STS tasks in Table 5, demonstrating its strong ability to represent textual inputs according to their semantics.

To validate the effectiveness of our prompt representation method, we compare it with two other methods: 1) Last: using the last token embeddings of the input as the multimodal embeddings, and 2) Prompt: using the same prompt as our methods, but removing in one word: in prompt. We report the performance of these methods with and without fine-tuning in Table 6. For the fine-tuning, we fine-tune each method with corresponding prompts on sentence pairs with contrastive learning following the same training settings as E5-V.

Our method shows significant improvements on all tasks compared to the Last and Prompt. For the setting without fine-tuning, we observe that our method can directly leverage the MLLM to represent the multimodal embeddings. However, other methods cannot represent the multimodal inputs properly. We also find that these methods have a large modality gap between image and text embeddings, as shown in Appendix B. For the setting with fine-tuning, we also observe the performance gap between our method and other methods. Although Prompt uses same template with our method and just removes in one word: in it, it still shows significant performance drop compared to our method especially on tasks with visual inputs. One possible reason may be the modality gap limit it to transfer the single modality representation capabilities learned on text inputs to multimodal embeddings.

We also compare single modality training with multimodal training. For multimodal training, we train the MLLM on 558K text-image pairs from CC3M using the same training settings and prompts as single modality training. We report the performance of single modality training and multimodal training on different tasks in Table 7. We find that MLLM achieves better multimodal embeddings with single modality training. Even on the image-text retrieval tasks, where multimodal training uses similar training data, single modality training still shows better performance. For other tasks, we notice that multimodal training cannot represent the interleaved inputs in FashionIQ and CIRR, or text inputs in STS well, which leads to a performance drop compared to single modality training. Moreover, single modality training is more efficient by removing the visual encoder and only uses 32 max tokens for text inputs, significantly reducing the training time compared to multimodal training. Single modality training only takes 1.5 hours on 32 V100 GPUs, while multimodal training takes 34.9 hours under same environment.

We find an interesting ability of E5-V to represent inputs based on fully zero-shot instructions. Although E5-V is trained on text inputs with the static prompt, it can correctly represent visual and interleaved inputs based on unseen prompts. These prompts can be more detailed and specific based on the tasks. For example, in FashionIQ, a specific domain dataset about fashion products, we can design specific prompts to let E5-V embed the image based on their styles. Moreover, the interaction between visual and language inputs in E5-V can also be more detailed, such as change the style of this shirt to. Compared to other methods, which simply fuse the visual and language inputs, E5-V provides a more nuanced and specific approach.