ViCor: Bridging Visual Understanding and Commonsense Reasoning with Large Language Models

We first illustrate our categorization of VCR problems to distinguish the capabilities of VLMs and LLMs.

Both categories of visual commonsense reasoning tasks share a common formulation. In visual commonsense reasoning, the input consists of two parts: an image denoted as $I$ and a multiple-choice question input represented as ${q,{c_{i}}}$, where $q$ corresponds to the question, and $c_{i}$ stands for the $i$-th answer choice. The model needs to choose the choice $c_{i}$ that is most likely to be true based on the image $I$.

Evaluating answer choices in VCR requires drawing new conclusions based on commonsense knowledge, which LLMs excels at (Anil et al., 2023). On the other hand, pre-trained vision-and-language models have exhibited a capability for visual understanding, such as image captioning and image-text alignment, with a demonstrated ability to generalize across various datasets (Li et al., 2023). Therefore, we decided to harness the strengths of vision-and-language models for visual understanding and the capabilities of large language models for evaluating answer candidates in the context of visual commonsense reasoning.

Captioning serves as a fundamental unsupervised pre-training task and the most generalized capabilities of pre-trained VLMs, which capture the most salient information from an image. Therefore, we first prompt the LLMs to take the caption of the image $C_{I}$ as the initial information and perform chain-of-thought reasoning on the question:

$$r_{1}=LLM(\{c_{i}\},q,C_{I}).$$

(3)

The reasoning result $r_{1}$ includes both intermediate reasoning steps and the final answer. However, it’s important to note that the image caption may not encompass all the relevant information within the image, potentially omitting critical contextual details essential for answering the question. In such cases, it becomes necessary to gather additional relevant visual observations from the image. Before this, we must first judge whether there is a lack of supportive visual evidence that would allow us to make a confident decision. As in Figure 2, we let the LLM take the initial reasoning $r_{1}$ and the history prompt as input to judge if current visual information adequately supports the decision. If it does, the model will directly output the result. Conversely, if there is a lack of sufficient evidence, the model will progress to the second stage, where it will actively seek additional visual evidence.

As mentioned before, we found the capabilities of VLMs are suitable for solving VCU problems, and LLMs are more capable of solving VCI problems. Therefore, we propose to leverage VLMs in distinct manners when facing different problem types. To this end, we first prompt the LLM to classify the problem into two categories. To achieve this, we provide the definitions of these two categories in the prompt. Additionally, we include a set of manually annotated in-context examples to aid in problem classification, where the questions of in-context examples are selected from the training set. Figure 3 illustrates the prompt.

The pre-training dataset of vision-and-language models contains millions to billions of image-text pairs. Therefore, VLMs have learned the mapping between visual features and the high-level commonsense concept well. In light of this, for visual commonsense understanding (VCU) problems, we leverage pre-trained VLM in a zero-shot manner. Specifically, for each choice $c_{i}$, we first instruct the LLM to transfer it and the question to a declarative sentence with instruction and in-context examples:

$$s_{i}=LLM(q,c_{i})$$

(4)

For instance, for the question What will the people face? and the choice earthquake, we will transform them to The people will face earthquake. Then, we feed $s_{i}$ and the image $I$ to the pre-trained VLM to calculate the image-text alignment score. Following  (Li et al., 2023), we use the sum of ITM and ITC scores to compare choices:

$$S_{i}=ITM(I,s_{i})+ITC(I,s_{i})$$

(5)

We will directly take the choice with the highest score as the final output.

For the visual commonsense inference (VCI) problems, the model needs to acquire related visual observations and use relevant commonsense knowledge to reason about the answer. Some crucial visual observations are often neglected in the descriptions of the image. Therefore, as in Figures 2 and 3, we first prompt the LLMs to think about some visual factors $f_{j}$ that influence the answer to the question, like ‘the action of the person’, ‘the interaction between people’, etc. Then, we could acquire the visual observation of the visual factor in the image with a visual question-answering model by asking a question about the visual factor:

$$o_{j}=VQA(I,f_{j})$$

(6)

where $o_{j}$ is the answer to the question which we call visual clue. However, the answer of VQA does not consider the context of the main question and therefore may lack the most related information. To better leverage the contextualized reasoning capabilities of LLMs, we further propose to prompt the LLM to reason the potential instantiations of the visual factors that can support the choices as in Figure 3:

$$o_{ij}=LLM(f_{j},c_{i},q)$$

(7)

For instance, when $f_{j}$ is “category of the plant," the potential values for $o_{ij}$ may include specific plant names like “cactus." Then, we could leverage the image-text matching (ITM) and image-text contrastive (ITC) functions of pre-trained VLMs to select the observation that most align with the image among the observations for each choice $i$:

	$\displaystyle o_{j}$	$\displaystyle=o_{kj}$		(8)
	$\displaystyle\text{ where }k$	$\displaystyle=argmax_{i}\{ITM(o_{ij},I)+ITC(o_{ij},I)\}$		(9)

Finally, we append the visual clues $\{o_{j}\}$ after the caption as extra information for LLM to perform final reasoning:

$$r_{2}=LLM(\{c_{i}\},q,C_{I},\{o_{j}\})$$

(10)

We mainly evaluate our approach on two datasets focused on visual commonsense reasoning: VCR (Zellers et al., 2019) and AOKVQA (Schwenk et al., 2022).¹¹1We provide the result on OKVQA dataset in Appendix. A. Both datasets formulate visual commonsense reasoning as 4-choice QA problems about an image, containing various visual commonsense understanding and inference problems. VCR dataset focuses on human-centric visual commonsense reasoning problems. In contrast, A-OKVQA dataset requires various commonsense knowledge about common objects and events in daily life. For A-OKVQA, we use the validation set with 1145 examples. For VCR dataset, we randomly sample 3000 / 26534 examples from the validation set for the ablation study, and sample 500 examples to compare with other methods due to the GPT4 API cost. We divide the image from left to right into three bins and name the person depending on which bin they are located in when feeding text to VLMs and LLMs, similar to (You et al., 2023). The performance of both datasets is evaluated by accuracy.

In our experiments, we use GPT-3.5-turbo-0613 and GPT-4-0613 as the LLMs for reasoning. To ensure reproducibility, we set the temperature of the LLMs to 0. For image captioning, we employ LLAVA-7B-v1.1. Furthermore, we use the pre-trained BLIP2 model for image-text alignment and BLIP2-FlanT5 XL for visual question answering. The number of in-context examples used in the prompts shown in Figure 3 is 6, 1, and 3, respectively. All the questions in the in-context examples are from the training set.

We implement the following training-free baselines for comparison: (1) BLIP2-Pretrain (Li et al., 2023): We use the pre-trained BLIP-2 model directly to perform image-text alignment on both datasets. On both datasets, we utilize GPT-3.5-turbo-0613 to transform the questions and choices into declarative sentences and feed them to the BLIP-2 model to calculate the image-text alignment score. We select the choice with the highest alignment score as the answer. (2) IdealGPT (You et al., 2023): It prompts LLMs to break down the question and iteratively query a VQA model to answer sub-questions for visual reasoning. In our experiments, we employ the original source code of IdealGPT while utilizing the same version of LLM and VLMs for caption, VQA, and reasoning as our method.

We conduct ablation studies about VLMs and LLMs collaboration on VCR and AOKVQA datasets. Results are shown in Table 1.

From Table 1, we can first find that visual factors and visual clues are less helpful in VCU problems. On VCU problems, besides directly taking the concept being asked by the original question as the visual factor. The model will also consider low-level visual features as visual factors for the question. For example, for the question What is the event in the image, and the choice dinner, the visual factor could be objects in the image, and the reasoned visual clues could be plates with food on the table.

On BLIP2-Pretrain, using clues for image-text alignment is not better than using directly transferred declarative sentences. This validates that BLIP2 can already align visual features with different concepts well. However, introducing visual factors and observations as extra context improves performance on LLM reasoning, especially when the LLM is not confident about its initial judgment with only caption as context. In this case, the performance of LLM reasoning (‘Cap + Clue’ in Table 1) is comparable with pre-trained BLIP2.

For VCI problems, visual factors and visual clue generations help both reasoning paradigms. First, the improvement in the BLIP2-Pretrain paradigm validates that (1) pre-trained BLIP2 cannot well-align statements that go beyond literal visual content, requiring commonsense inference; (2) LLM can reason about the visual factors that may contribute to supporting candidate commonsense inferences, and guide the VLM to focus on relevant factors accordingly. Second, the improvement in the LLM reasoning paradigm shows that LLM clues successfully provide subtle details of the scene that are crucial for solving the problem. Third, visual clues reasoned by LLM are better than VQA as the visual information provider. There are mainly two reasons. First, the pre-trained VLM sometimes could not understand or correctly answer the question due to the lack of language alignment. Second, the VQA model lacks the main question as the context and may not get the intention of the visual factor. Therefore, it may produce irrelevant answers. We provide examples to further illustrate these in Section 6.3.

In Fig. 4, we demonstrate several qualitative examples. The left example shows a case where the problem is classified as VCU, and the BLIP2-Pretrain selects the correct answer. The middle example presents a case where the initial evaluation is incorrect, and both the VQA and clue reasoning methods give the correct observation for the visual factor ‘weather’, based on which the LLM selects the correct answer. The BLIP2-Pretrain here selects ‘block sun’ due to the lighting condition of the image. The example on the right demonstrates a case when the LLM reasoned answer is better than the answer generated by the VQA model. Here, the VQA does not understand the intention of the visual factor without the context of the main question. The LLM reasoned answer, however, can provide the most relevant information to the question and help the final reasoning. The BLIP2-Pretrain fails here due to the textual similarity between ‘wind’ and ‘still wind’.