Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generation

The development of LLMs like ChatGPT [23], GPT-4 [24], and LLaMA [31] has paved the way for research into large-scale multi-modal models that extend beyond text-based modalities. This section highlights studies particularly relevant to our research, focusing on models integrating vision and language modalities.

BLIP-2 [11] and InstructBLIP [5] utilize pre-trained image encoders and text decoders, employing a Transformer model known as Q-Former as an adapter. The training of these models is typically divided into two stages. In the first stage, Q-Former learns the alignment between image and text using large-scale image-text paired datasets. The second stage involves training the Q-Former alongside the text decoder to enable the decoder to produce outputs according to given instructions. Generally, while the first training stage prioritizes the scale of the dataset, the second stage emphasizes the quality of smaller, high-quality instruction and answer datasets. This two-stage learning approach is widely adopted in the training of VLMs, as seen in models like BLIVA [9], MiniGPT (v1&2) [39, 2], and LLaVA (v1&1.5) [18, 17].

Additionally, there’s growing interest in VLMs that not only produce outputs aligned with overall image instructions but also focus on specific regions within images. This research direction is evident in models such as KOSMOS-2 [26], Shikra [3], MiniGPT v2 [2], and LLaVA v1.5 [17]. These models transform bounding box coordinates, representing input regions, into special tokens for textual representation. Another noteworthy model is GPT4RoI [38], which applies an RoIAlign based on the target region to the features outputted by the image encoder, thus obtaining features focused on the specified region.

In the field of natural language processing tasks, LLMs have shown the capability to perform explicit reasoning in a format known as Chain-of-Thought [4], thanks to their advanced language generation abilities. However, in vision-and-language (V&L) tasks, the adoption of explicit reasoning using large-scale models is still in its developmental phase compared to language-only tasks. The closest research in this area involves using LLMs to predict necessary reasoning steps in a programmatic format and subsequently calling APIs specialized for V&L tasks, such as in Visual Programming [7] and ViperGPT [30]. A significant limitation of these approaches is that the LLM predicting the reasoning steps does not have direct access to the images, leading to a lack of assurance that the reasoning is genuinely based on the image content. Our model addresses this gap by allowing the V&L model to directly predict reasoning steps, ensuring that the reasoning is firmly grounded in the image content.

A related field to the concept of acquiring information through question generation is Visual Question Generation (VQG). In the early stages of VQG, the primary focus was on generating questions related to images, without any specific goal of acquiring knowledge [22, 12, 16, 35]. K-VQG [33, 34] does involve generating questions for knowledge acquisition; however, there has been no emphasis on utilizing the acquired knowledge in further reasoning steps.

Learning by Asking (LBA) is another relevant area. LBA is an approach where question generation is used to acquire data for learning purposes. For instance, LBA has been employed in tasks like VQA [21, 32, 34], caption generation [29], and scene graph completion [37]. However, our research significantly diverges from LBA, as we focus on generating questions as part of explicit reasoning during the inference process. The acquired answers are then immediately used to update and refine the reasoning quality, differentiating our approach from traditional LBA work.

In pursuit of realizing the CoR, we developed a novel dataset. This dataset was created using a combination of the specially designed prompt, manually curated examples and the text generation capabilities of LLM. We show an overview of the dataset construction process in Figure 2 and an example of the dataset in Figure 3.

The data structure of our dataset consists of the image, an instruction text of the task, a question (optional, if the task is in QA format), the answer, and the reasoning process required to derive the answer. We used different approaches to build the dataset depending on the types of annotations provided with the base dataset.

(1) If the dataset had rich annotations, including captions and object detection annotations, such as labels and bounding boxes, we followed the LLaVA approach [18]. In this case, we used an LLM (GPT-4 [25]) that does not process images to build the dataset. Coupling these inputs with thoughtfully designed prompts (Listing LABEL:lst:prompt) enables the LLM to generate the appropriate reasoning steps. It’s crucial that the prompts ensure that the LLM simulates the act of observing the image directly, without mentioning any references to captions, descriptions, or bounding boxes. This approach ensures the consistency of the reasoning process when training VLMs, which do view the image directly. To achieve this, in the prompt given to the LLM, we clearly state like you must not use the word “caption”, “description”, or “bounding box” in the reasoning steps. Especially, do not mention the coordinates of each object in the image. If you want to mention the position of the object, please refer to it by relative position, such as “at the left of the image” or “in front of xx”. Furthermore, in the prompt, we instruct the LLM to suppose that the image is of size 1000$\times$1000 pixels. This ensures accurate position estimations based on given textual coordinates of the bounding boxes. The prompts also emphasize succinct reasoning steps without redundancy, and the need to add uncertainty scores at the end of each reasoning step.

(2) If the dataset did not have such rich annotations, we used a model that can input images (GPT-4V) to build the dataset. We used the similar prompts as previous case, excluding the parts that required GPT to imagine image input from text information.

We devised three variants of reasoning step data: without QA, with QA, and with GT. In “without QA” setting, the reasoning process to the answer is conducted without generating any questions midway. This setting aims to mimic the process a VLM undergoes when attempting to answer a question directly, without generating any questions. In this scenario, the LLM is instructed to generate reasoning steps without seeing the correct answer to the given task. We instruct the model to generate “Uncertainty score” along with each reasoning step, indicating the model’s confidence in its reasoning. This plays a crucial role in “with QA” setting described below.

In contrast, the “with QA” setting incorporates the generation of questions as part of the reasoning steps. This setup is crucial for training VLMs to perform explicit reasoning while simultaneously generating questions. As the model generates the reasoning steps, it also generates an uncertainty score for each step. If the score significantly rises at any point in the “without QA” setting, it triggers the generation of a question just before the step. This mechanism helps the model to identify moments of uncertainty and seek external information, thereby simulating a more interactive and dynamic reasoning process.

Lastly, the “with GT” setting involves the LLM generating reasoning steps while having access to the correct answers to the tasks (i.e., the ground-truth answers for the given questions or captions for the given images). This approach differs from the previous two as the reasoning steps invariably lead to the correct answer, serving as a ground truth for correct reasoning without question generation. This data is primarily used to train VLMs that do not involve question generation in their reasoning process.

Each question step is designed to consist of three key elements: Imagined Knowledge Needed, the Question, and the Answer. Firstly, “Imagined Knowledge Needed” component represents the knowledge the model predicts it needs to acquire during the reasoning process. Let us consider the example shown in Figure 2. Here, the model need to answer the question “What holiday do we use this for?”, according to the given image of Christmas decorations. The model might need more specific information about different holidays and their associated symbols or decorations.

Secondly, we include the “Question” that would facilitate the acquisition of this knowledge. For instance, the model might generate a question to acquire imagined knowledge needed, such as “Which holiday is generally associated with a decorated tree placed in a living room?” This step is crucial as it reflects the model’s ability to formulate relevant questions based on the required knowledge.

The third component is the “Answer”, which is essentially the answer to the generated question. For example, the answer to the question above would be “Christmas”, and then the model is able to successfully acquire the knowledge it needs to answer the original question.

We provide a comprehensive overview of our dataset statistics in Table 1. In our research, we based our dataset generation on seven datasets: COCO Caption [13], Visual Genome Caption [10], VQA v2 [6], OK-VQA [19], A-OKVQA [28], OVEN [8], and Encyclopedic-VQA [20]. We can categorize these datasets into three groups:
(1) Visual understanding tasks.
This group includes COCO Caption, Visual Genome Caption, and VQA v2, which are captioning tasks and VQA tasks that require the model to understand or describe the content of the image. Usually, these tasks do not require the model to have expert knowledge that cannot be directly observed from the image.
(2) Vision + common-sense understanding tasks.
This group includes OK-VQA and A-OKVQA, which require the model to have common-sense knowledge to answer the questions. Here, common-sense knowledge refers to general knowledge about topics such as geography, brands, and vehicles. [19]
(3) Vision + encyclopedic knowledge tasks.
This group includes Encyclopedic-VQA and OVEN, which require the model to have encyclopedic knowledge to answer the questions. Encyclopedic knowledge refers to more in-depth and specific knowledge compared to common-sense knowledge. This includes knowledge such as the names of animal and plant species, or details about when and by whom a building in an image was constructed [20].

We assume that the difficulty of the reasoning process increases as we move from visual understanding tasks to vision + encyclopedic knowledge tasks, as the latter requires detailed and expert knowledge to answer the questions. For each of these datasets, we generated approximately 6,000 data samples, randomly selecting from their training sets. We show examples of each category in Figure 4.

From Table 1, we can see that when generating reasoning data without questions and answers (without QA) and when not seeing ground-truth answers (with GT), the average number of reasoning steps is shorter compared to the scenario where both questions and answers are included (with QA). This is attributed to the fact that the QA generation process inherently requires more reasoning steps.

The “with GT” setting shows slightly fewer reasoning steps than the without QA setting. This can be attributed to providing the correct answers to GPT-4, which likely allows for concise reasoning steps leading to the correct answer.

The COCO Caption, Encyclopedic-VQA, and OVEN dataset tend to have longer reasoning steps compared to the others. For COCO Caption, this is likely due to the nature of the task, which involves generating captions based on the entire content of the image, requiring a more comprehensive reasoning process. The Encyclopedic-VQA and OVEN datasets often contains questions that require very detailed observations or expert knowledge to answer, leading to longer reasoning steps. In contrast, the other tasks focus on specific questions or target regions within the images, thereby requiring less information to be incorporated into the reasoning steps.

Our model fundamentally builds upon the LLaVA architecture [18], consisting of an image encoder, a text decoder, and an adapter component based on two-layer MLP blocks. We show an overview of our model in Figure 5. The input image is first resized to 336$\times$336 pixels and passed through a pre-trained image encoder (e.g., CLIP-ViT [27]). The image encoder extracts the image features, which are then fed into the Adapter MLPs to transform the image features into a format that can be processed by the LLM. The instruction text is also fed into the LLM, and the model generates a text response based on the image features and instruction text.

We utilizes our dataset to make the model learn to generate reasoning steps and questions. In the training process, we utilized a pre-trained VLM, which is already instruction-tuned, and fine-tuned it on our dataset. In the fine-tuning process, we froze the parameters of the image encoder, and updated the parameters of the text decoder and the adapter MLP. Along with the image and original question, we provided a specially designed prompts, such as “Analyze the image and outline your reasoning process step by step before providing your final answer,” to let the model generate reasoning steps and questions.

When the model is applied to test data, it performs a two-stage reasoning process to reach the final answer. The first stage encompasses the reasoning that leads up to the generation of a question. The second stage occurs after an external response to the question is obtained; the model is given the original inputs, the intermediate reasoning steps, and the external response. Then, the model generates the remaining reasoning steps based on this new information.

In this study, we leverage GPT-4o as the external answerer. It is noteworthy that the questions generated by the model typically demand an understanding of the image content as well as common-sense and encyclopedic knowledge. Given its extensive training data and large model parameters, GPT-4 is expected to adeptly handle the knowledge required to answer the questions.

Our experimental setup incorporates the following models: for image processing, we use CLIP-ViT-Large¹¹1https://huggingface.co/openai/clip-vit-large-patch14-336, and for text generation, we employ vicuna-7b-1.5²²2https://huggingface.co/lmsys/vicuna-7b-v1.5. The entire model is trained from a publicly available pre-trained checkpoint ³³3https://huggingface.co/llava-hf/llava-1.5-7b-hf.

In fine-tuning stage, we utilized 8 $\times$ NVIDIA H100 (80GB) GPU for about 1 hour. We trained the model for one epoch with a total batch size of 128. The learning rate was set to a maximum of 2e-5, linearly increasing until 3% of the training steps, followed by a cosine decay schedule. Training was conducted using bfloat16 precision.

In our experiments, we investigated how the presence or absence of CoR affects the behavior of the model when solving tasks. We also evaluated several ablations related to the CoR method. These included listing the necessary knowledge before question generation and generating an uncertainty score along with the reasoning steps. We used validation set of seven datasets for this evaluation: MS COCO, Visual Genome, VQAv2, OK-VQA, A-OKVQA, OVEN, and Encyclopedic VQA. Using entire validation set is computationally expensive, so we randomly sampled 1000 instances from each dataset for evaluation. Our experiments compared the model trained with CoR to a baseline model trained without question generation, specifically using the “with GT” setting from the dataset (referred to as “w/o Question”).

Currently, there are no established metrics for evaluating results that include long reasoning steps. Therefore, in this study, we used GPT-4 for scoring the answers. We instructed GPT-4 to extract the final answer from the reasoning steps and then evaluate the answer with a score from 1 to 4 (1: incorrect, 2: partially correct, 3: mostly correct, 4: correct).

The results of our experiments are shown in Table 2. Also, we show an example of the predicted reasoning steps in Figure 6.

The evaluation results show that “Ours CoR” performs the best on average across various datasets. In contrast, the CoR model without questions (CoR w/o question) has the lowest performance. This might be because the current model cannot consistently provide coherent outputs over long reasoning processes. The challenge of developing VLMs that can produce coherent and consistent long reasoning steps in line with the given tasks remains an important area for future research.

Comparing “Ours CoR” with the original LLaVA, the latter performs relatively well on VQA v2 and COCO Caption, which are part of LLaVA’s training data.However, for datasets not included in LLaVA’s training data, like OVEN and Encyclopedic VQA, which require specialized knowledge, models that ask questions (such as “Ours CoR”) show significantly better performance. This indicates that asking questions helps the model acquire necessary specialized knowledge that it does not inherently possess.

Furthermore, outputting uncertainty did not significantly improve performance (CoR w/ uncertainty vs. Ours CoR). This is likely because generating uncertainty outputs makes the model’s responses longer and less consistent, leading to performance degradation. However, if the model can handle uncertainty effectively, it could ask questions during the most uncertain reasoning steps, potentially improving performance.

Comparing models with and without prior knowledge output (CoR w/o knowledge vs. Ours CoR), it’s evident that pre-outputting necessary knowledge improves performance. This prevents the model from asking irrelevant questions and ensures that it focuses on helpful inquiries, leading to better results.

Here, we provide the full example of the prompt given to GPT-4 for answer generation and evaluation. The prompt for answer generation is shown in Listing LABEL:lst:prompt_answer, and the prompt for answer evaluation is shown in Listing LABEL:lst:prompt_eval. In both prompts, we provide an instruction and several examples to guide GPT-4 to generate and evaluate the answer, respectively. We instructed GPT-4 to return the answer in JSON format, which enables us to parse the answer and evaluate it automatically.

In Figure 7 and 8, we show more examples of the dataset we crafted based on different datasets: OK-VQA, A-OKVQA, MS COCO caption, Visual Genome caption, OVEN, and Encyclopedic-VQA. Each example consists of an image, a question, and a reasoning chain that explains how to answer the question or generate a caption for the image. We provide the reasoning chain of three types: (1) reasoning steps without answer and question, in which the data is generated without access to the ground-truth answer and there is no question generation step; (2) reasoning steps with question, in which the data is generated with access to the ground-truth answer and there is a question generation step; and (3) reasoning steps with answer and without question, same as the second type but without the question generation step.

We show more qualitative results of generated reasoning steps by our model for VQA, A-OKVQA, COCO Caption, Encyclopedic VQA, OVEN, OK-VQA, and Visual Genome in Figure 10,  11,  12, and  13.