Visualizing Dialogues: Enhancing Image Selection through
Dialogue Understanding with Large Language Models

Considering that visual descriptors can significantly enhance the understanding of visual content Menon and Vondrick (2022), we focus on generating dialogue-associated visual descriptors to improve image retrieval capabilities. To create high-quality visual descriptors that can connect with visual elements in the photo, we define a set of visually-focused queries, denoted as $Q=\{q_{i}\}$. These queries encompass various visual attributes related to an image, such as main subject and background scene, which are instrumental in linking the target photo to the dialogue.

Drawing from prior work Kuznetsova et al. (2020); Zang et al. (2021) and our common experiences, we assume that photos shared in online messaging typically contain components such as main subjects, prominent foreground objects, background scenes, events, and materials and attributes, as detailed in Table 2. Note that we do not expect all answers to these queries to be perfectly extracted from the dialogue context or found in the ground-truth image. Instead, our goal is to leverage automatically inferred visual descriptors to bridge the gap between the image and the given dialogue context.

Leveraging the powerful reasoning capabilities of large language models (LLMs) Touvron et al. (2023), we construct a prompt comprising the dialogue $D$ and the set of queries $Q$ and input it into the LLM. This process yields a set of dialogue-associated visual descriptors in a zero-shot manner:

For instance, a generated visual descriptor regarding the main subject might read, “The main subject of the photo is a group of friends.” The used prompts can be found in Appendix A.

To measure the relevance of each image candidate in the context of a given dialogue $D$, we calculate two retrieval scores based on their generated visual descriptors desc: $S_{\text{scene}}(o_{j},\text{desc})$ and $S_{\text{vision}}(v_{j},\text{desc})$. The former score assesses if the objects in the photo candidate align with the inferred visual descriptors in their text-only forms, referred to as the scene-aligned score. The latter score evaluates if the photo candidate matches the visual descriptions through multimodal methods, termed the vision-aligned score.

Our task involves retrieving the target image from a pre-constructed photo set, and it can be approached in two settings: 1) zero-shot and 2) training with contrastive leanrning.

Our approach is evaluated on a benchmark multimodal dialogue dataset called PhotoChat. PhotoChat dataset Zang et al. (2021) is characterized by open-domain, high-quality multimodal dialogues and comprises 10,917 images paired with 12,286 dialogues. Specifically, the dataset is divided into 10,286 instances for training, 1,000 for validation, and another 1,000 for testing. Each image in the dataset is accompanied by an associated object list presented in textual form. In each data instance, one photo is shared within the context of the conversation.

Given that this task can be formulated as an image retrieval task, we employed $Recall@k$ ($R@k$) as our evaluation metric. During the training phase, we select the final model based on the highest $avg(R@1,R@5,R@10)$ score on the validation set. In the testing phase, for each dialogue instance, the trained models retrieved images from the candidate photos in the testing set.

We compare our approach against several established baselines:

• •

  VSE++: Faghri et al. (2018) incorporated hard negatives in the ranking loss to learn visual-semantic embeddings for text-image retrieval.

• •

  SCAN: Lee et al. (2018) utilized stacked cross attention to align image regions and words in a sentence and calculate image-text similarity.

• •

  Dual Encoder (DE): Previous work Parekh et al. (2021); Zang et al. (2021) employed a dual encoder architecture, where one encoder processes the image and its object list using CLIP ViT-B/32 for images and FFNN for object features. For the dialogue encoder, two different text encoders were experimented with: CLIP ViT-B/32 Text and BERT Devlin et al. (2019) with an additional projection to ensure consistent dimensions. The retrieval similarity between the image and dialogue encodings is measured using dot product.

In addition to the query-based descriptors, we conduct experiments using the following descriptor variants for in-depth analysis:

• •

  Desc - Diag (whole dialogue as descriptors): All utterances are concatenated to form the descriptors, allowing the image retriever to utilize complete cues within the dialogue.

• •

  Desc - Caption (caption as descriptors): Inspired by Li et al. (2023), we performed zero-shot image captioning on images in the training set using BLIP-2. We then trained a text generator to create image captions as descriptors based on a given dialogue.

• •

  Desc - Summary (summary as descriptors): Descriptors are generated by LLMs based on a dialogue summary, offering a more concise representation of the conversation.

• •

  Desc - Guessing (visually-focused guessing as descriptors): LLMs are allowed to speculate about the features of the upcoming shared photo from the dialogue without being constrained by a specific query.

• •

  Desc - Queries (visually-focused query descriptors): Utilizing our designed visually-focused attributes as descriptors.

Table 3 provides a comprehensive overview of the results for both zero-shot and fully-trained settings on PhotoChat dataset. In zero-shot scenarios, Desc - Guessing emerges as the top-performing method among all results. Notably, Desc - Queries outperforms Desc - Summary, indicating that visually-focused queries and guessing contribute valuable information for linking the desired images. We hypothesize that the poor performance of Desc - Summary is because the LLM summarize all content of the dialogue, rather than focusing solely on the visual information. Consequently, Desc - Summary often includes sentences that do not aid in text-to-image retrieval. As for why Desc - Queries did not perform better than Desc - Guessing, we hypothesize that it is because we construct our descriptors using the templates from 3.1. These sentences tend to be less fluent and are not the types of sentences CLIP was familiar with during pre-training stage. In contrast, Desc - Guessing generates more fluent descriptors without any constraints, making them closer to the sentences seen by CLIP during pre-training stage. This gives Desc - Guessing an advantage in zero-shot scenarios.

In the fully-trained setting, the descriptor-based results (Desc - Summary, Desc - Guessing, Desc - Queries) with LLaMA-2-13B-Chat exhibit similar performance, with Desc - Queries achieving the highest average performance. These results validate the effectiveness of our proposed approach, demonstrating that the generated visual descriptors successfully facilitate the connection between associated images through the LLM’s understanding of dialogue. Additionally, it is evident that LLaMA-2-13B-Chat outperforms LLaMA-2-7B-Chat due to its stronger reasoning abilities for understanding dialogues. When compared to the fully-trained baselines, our proposed descriptor-based methods achieve superior performance even in zero-shot settings, establishing a new state-of-the-art performance achieved by a single model.

We further conduct experiments on ensemble learning using all descriptor-based results based on the validation set. The results in Table 4 demonstrate that ensemble learning consistently improves performance. Even in cases where the caption model performs poorly in a fully-trained setting, ensemble learning benefits other models. These findings highlight the efficacy of combining various types of descriptors, leading to the best overall performance and establishing a new state-of-the-art for PhotoChat. This suggests that the generated descriptors focus on diverse patterns that can complement each other and enhance scores.

Our proposed method incorporates two scores: scene-aligned (text-only) and vision-aligned (multimodal) scores. We conduct an ablation study to assess the impact of each score. Table 5 presents the experimental results. The results show that models trained solely on the scene-aligned score (text-only) perform better in terms of $R@1$, whereas models trained on the vision-aligned score (multimodal) perform better for $R@5$ and $R@10$.

In addition, we conducted experiments with different weighting parameter $\lambda$ in (2). Figure 2 illustrates the results obtained for various $\lambda$ values in the zero-shot scenarios. It can be observed that the best performance is achieved when $\lambda$ approaches 1. This indicates that excessive bias towards either the scene-aligned score or the vision-aligned score leads to a decrease in performance. Interestingly, our findings reveal that the optimal zero-shot performance occurs at $\lambda$ = 1.2.

Given our experimental assumption of having available object lists for the images, a degradation in system performance may occur when object detection is poor. Consequently, we explored the sensitivity of error propagation in our proposed method by simulating various error rates in detected objects. Specifically, we randomly removed objects from the gold object list or substituted them with others from the training set to simulate scenarios of “missing” or “incorrect” objects. As shown in Table 6, it is evident that with increasing error rates, regardless of the error type, performance tends to degrade. This suggests that our method is still influenced by error propagation. Moreover, the impact of incorrect objects is particularly severe since they may mislead the model in image selection. Nevertheless, even with a 35% error rate, our results outperform relying solely on the vision-aligned score, indicating a relative robustness to missing objects. Therefore, precision in object detection during object list extraction is more important for our proposed approach.

To assess the influence of different visually-focused queries on our results, we conducted experiments by systematically removing individual queries from the original query set (see Table 7). Remarkably, the query concerning the main subject emerges as the most pivotal feature for bridging the dialogue context and the target image, as its removal leads to a significant decrease in scores. Following in descending order of impact are the queries related to foreground objects, events, background scene, and materials and attributes.

Additionally, we augmented the original query set with two common photo features: atmosphere or mood and lighting.

• •

  Atmosphere or mood: Research by Sun et al. (2022) suggests that photographs convey not only object details but also emotions and ambiance, evoking feelings like happiness, boredom, and coziness.

• •

  Lighting: As highlighted by Hunter et al. (2021), the presence and quality of light are fundamental in photographic composition. Our investigation explores whether an LLM can accurately predict lighting conditions within a photo solely based on dialogue context and whether this predictive information enhances text-image retrieval capabilities.

Results indicate that atmosphere or mood improves performance at $R@5$, while lighting enhances results at both $R@1$ and $R@5$ compared to the original set. This suggests that these abstract and challenging-to-predict queries have varying impacts on performance.

To explore the generalizability of our approach, we conducted experiments on other benchmark datasets, VisDial Das et al. (2017) and MMDialog Feng et al. (2023), with identical settings.

• •

  VisDial Das et al. (2017): Each instance in VisDial comprises a single image from the COCO dataset Lin et al. (2014) and a dialogue context consisting of question-answer pairs. During testing, the model’s task is to retrieve the correct image from a pool of 2,064 images in the testing set.

• •

  MMDialog Feng et al. (2023): This dataset comprises multimodal open-domain dialogues gathered from a global social media platform. Instances in the testing set are linked with 999 corresponding negative images. MMDialog uniquely allows dialogue context to contain both text and images. In our experiments, we employed BLIP2 Li et al. (2023) for image captioning within the dialogue context. These resulting captions replaced the images, enabling textual dialogue context input into the LLM. Our experiments were conducted on a random sample of 1,000 instances from the testing set.

The average performance of LLaMA-2-13B-Chat in zero-shot scenarios across different datasets is summarized in Table 8, with detailed scores available in Appendix B. Our findings indicate that our proposed approach consistently outperforms Desc - Diag baseline and even recent work Koh et al. (2023), showcasing its robust generalization capability across diverse datasets.

To further explore the quality of our generated descriptors, we conducted a qualitative analysis across various models and descriptors. We present an example of a testing dialogue using different LLMs and descriptor generation methods in Table 9. A comparison between LLaMA-2-7B-Chat and LLaMA-2-13B-Chat in the Desc - Queries domain revealed that LLaMA-2-7B-Chat often delivers inaccurate or unrelated answers (highlighted in red). In contrast, LLaMA-2-13B-Chat tends to produce more precise answers and demonstrates an ability to infer information not explicitly present in the dialogue context (e.g., medieval architecture). This discrepancy is likely attributable to differences in reasoning capabilities between the two models.

Both Desc - Summary and Desc - Guessing can accurately describe the features of the photos. However, Desc - Summary occasionally incorporates additional details not directly related to the photos, such as “A is willing to share these pictures with B,” likely because we instructed the LLM to summarize information about the users sharing the photo rather than solely focusing on the photo-related information. Our analysis suggests that our visually-focused approaches are particularly effective for this task, especially when coupled with the powerful reasoning capabilities of LLMs.