Vript: A Video Is Worth Thousands of Words

If videos are densely annotated, a possible way to increase the amount of text for alignment is to concatenate captions of multiple successive clips. Though clips can be easily concatenated to create a longer video, captions are annotated separately so that the concatenated caption may not have coherence in the semantics. Inspired by video scripts, we reformulate the successive captions into scenes of the video script. In the right panel of Figure 2, a script in Vript with multiple scenes is coherent in the semantics despite they are annotated separately because: 1) each scene caption is very detailed and has similar descriptions for the shared background or context and 2) title of each scene acts as a separator rather than concatenating them directly. In Vript, We can easily sample several successive clips to create a "sub-script", e.g., 10 successive clips with corresponding "sub-script" containing about 1.5K words, which is nearly 100x longer than short captions.

We add voice-over transcription as the additional speech modality. As the Vript is annotated with joint input of voice-overs and video frames, the captions contain information that comes from the voice-over as shown in Figure 2.

Commonly video LLMs [7, 32] implement a certain sampling strategy to extract multiple frames as the video input. Some models [7, 32] that use a strategy of sampling a fixed number of frames, the models treat all videos as videos with the same duration. These models are weak in time awareness as they only know the order of frames but do not know how long the frames last. We find that timestamps are crucial for the video-script alignment of multiple scenes. As shown in Figure 2, we add two kinds of timestamps in the text format: voice-over timestamps in the input and video timestamps in the output caption. Predicting the timestamps of the video helps the model to know the start and the end of each scene.

As shown in Figure 2, Vriptor supports two types of instructions: describe the whole video and scene by scene. For the whole-video instruction, Vriptor gives a general description of 100~150 words. For the scene-by-scene instruction, Vriptor gives a dense description of the video with each scene of 100~150 words. In Table 4.2, compared to the whole-video description, Vriptor gives more details of the video in the scene-by-scene description with an increasing recall in the Vript-HAL and the MSR-VTT as the number of output scenes increases. However, as the captions get longer and more detailed (more scenes), models are easier to generate hallucinations with a drop in precision. In Figure 4.2, we showcase the ability of Vriptor to caption long videos with longer texts. Models like VideoChat2 [7] only give a relatively fixed length of captions for videos of different lengths. Vriptor-S (scene-by-scene) can scale up the caption length as the video gets longer, just like writing a longer video script.

In the last two rows in Table 4.2, we showcase the increments in both precision and recall that the model can give more detailed and accurate descriptions with the help of voice-over. We also observe a 14% increment in the proportion of proper nouns of all nouns in the captions. This suggests that the model is capable of inferring the names of objects rather than only their appearance by analyzing the voice-over.

To verify the effectiveness of adding timestamps, we also train another model without adding timestamps. Comparing these two models, we find the improvement is minor in whole-video description but significant in scene-by-scene description. The model with timestamps is less likely to generate duplicated descriptions from previous scenes because it can understand the start and end of each scene and identify which scene corresponds to which period. Besides, the model with timestamps gives more detailed captions with a 12% higher recall on Vript-HAL while the model without timestamps is more likely to forget to describe some parts of the videos.

Previous researchers [38, 39, 40] have explored methods to detect and evaluate hallucinations of powerful image LLMs. Similar to image LLMs, current video LLMs have a deeper understanding of videos and a stronger ability to generate more detailed captions for videos but also suffer from severe hallucinations. If we ask the video LLMs to describe a video, they may misread the objects and actions and generate a description with hallucinations. Captioning benchmarks, e.g., MSR-VTT [16] and MSVD [24], consist of short captions of no more than 10 words, giving superficial video descriptions without details. Thus we can not use them to evaluate hallucinations if many objects and actions are not included in the ground truth. To fill this gap, we construct Vript-HAL, a benchmark to evaluate object and action hallucinations in the video captions. Each video in Vript-HAL is annotated with two captions separately, approximately 250 words each, which are 25x longer than those in MSR-VTT. By building such strong ground truth captions, we can check if the video LLMs generate hallucinations in the captions.

Traditional metrics, such as BLEU [41], ROUGE [42], and CIDEr [43], focus on word-for-word precision by measuring the token similarity between the predicted and ground truth texts, which are not suitable for evaluating if the objects and actions are correctly described. Following previous works [44, 45], we evaluate whether the nouns (objects) and verbs (actions) are correctly described in the captions by using the precision score. In addition to evaluating accuracy through precision, it is noted that various models give descriptions varying in length and detail. We observe that shorter captions typically include fewer details thus tending to contain fewer hallucinations. To balance this, we introduce the recall score, which measures how many objects and actions in the ground truth are correctly described. We calculate the F1 score as the comprehensive score of hallucination evaluation as follows:

$$\mathcal{P}(\mathbf{p},\mathbf{g})=\frac{\#\{\mathbf{p}\cap\mathbf{g}\}}{\#\{\mathbf{p}\}},\quad\mathcal{R}(\mathbf{p},\mathbf{g})=\frac{\#\{\mathbf{p}\cap\mathbf{g}\}}{\#\{\mathbf{g}\}},\quad F_{1}=2\cdot\frac{\mathcal{P}\cdot\mathcal{R}}{\mathcal{P}+\mathcal{R}},$$

(1)

where $\#{\{\mathbf{p}\}}$ and $\#{\{\mathbf{g}\}}$ represent the number of objects and actions described in the prediction and ground truth caption respectively. We leverage the SpaCy ²²2https://spacy.io/. We use the largest model en_core_web_lg. to extract the nouns, proper nouns, and verbs as the objects and actions. $\#\{\mathbf{p}\cap\mathbf{g}\}$ represents the number of objects and actions that are correctly described in the prediction. We then encode the objects and actions into word embeddings using the sentence-transformers ³³3https://www.sbert.net. We use the top-performing embedding model all-mpnet-base-v2.. Instead of using the exact match, for each object or action, we consider it to be correctly described if the cosine similarity between the prediction and the ground truth is greater than 0.5. It is noted that using similarity may result in many-to-one matching because objects or actions with similar meanings in the prediction are all matched by one object or action in the ground truth, potentially yielding a score greater than 1 if the prediction is much longer than the ground truth, e.g., the recall score in MSR-VTT in Table 4.2.

We evaluate a large range of models on Vript-HAL, including image LLMs supporting multiple image inputs and video LLMs. From Figure 4, we observe some models, e.g., BLIP2 and VideoChat 2 have fewer hallucinations only because they give shorter captions containing fewer details. Vriptor-W (whole-video) giving general descriptions has a higher precision while Vript-S (scene-by-scene) giving dense descriptions describes more details in the videos with a higher recall. Both models have performance on par with the GPT-4V in video captioning.

If we ask about details in the long video, we may encounter ambiguity in the questions that: 1) there are multiple answers that match the question in the different timestamps; 2) the answer changes as time goes on. The ambiguity issue can be commonly seen in the long video understanding benchmarks, e.g., EgoShecma [18]. We propose Vript-RR (Retrieval then Reasoning), a long video reasoning benchmark that has no such worries. Different from these benchmarks [19, 7, 18] that only provide questions, we first give a hint for the model to locate the scene in the video that the question refers to. The hint is a detailed description of the relevant scene. We then ask the question based on the scene, which eliminates the ambiguity. In practice, as shown in Figure 5, we input the hint and the question along with the entire video together, and the models directly output the answer, which is an end-to-end process. We carefully craft the hints to ensure the model can not find short paths through hints. We design various questions for Vript-RR to evaluate the different capabilities of video LLMs, where each question requires at least one reasoning step or additional processing, e.g., text reading, and meticulous inspection of details.

Vript-RR consists of two subtasks differing in the video inputs: one is inputting the whole videos and another is directly inputting the related scenes. Vript-RR provides questions both in multiple-choice and open-ended formats. For the open-ended outputs, we leverage the GPT-4 turbo [15] as the judge [46] to evaluate if the answer is correct by comparing the prediction with the ground truth. As shown in Table 4, the "Scene" columns represent using the related scene as input, which is an easier task because the models do not need to retrieve across the entire video to find the related scene. The results of the "Scene" columns mainly showcase the models’ video reasoning ability. For "Whole" columns using the whole video as input, we require models to first find the relevant scenes using the hint, requiring the additional long video understanding ability. The closed-source models like GPT-4V and Claude 3 have better performance than open-source video LLMs.

For each video in Vript-RR, we design the questions for scenes extracted from four various timestamps, corresponding to 15%, 40%, 60%, and 85% of the video respectively. We want to explore whether the temporal positions of scenes in the long video will influence the results of Vript-RR. We describe it as finding a "needle" in the "timestack", whose name is derived from the "needle-in-a-haystack" task [47] for testing the long-context ability of LLMs. We require models to go through visual tokens instead of text tokens to find the "needles" (related scenes). In the "needle-in-a-haystack" task, there is a phenomenon that the model performance drops significantly when the "needle" falls between 15% and 85% of the long context, particularly when the text length exceeds at least 16K tokens. As shown in Figure 6 (a), though the number of visual tokens is significantly smaller than 16K, performance drops are also observed for most of the models if the scenes fall in the middle of the visual tokens (40% and 60% of the video).

There have been some benchmarks [5, 19, 7] that test the temporal understanding ability of the models. Unfortunately, they focus on asking questions about the temporal order of the actions happening in a short clip but few explore the temporal understanding of events in the long videos. To fill the gap, we propose the Vript-ERO (Event Re-ordering) task. We sample three distinct scenes (lasting 10s on average) in different timestamps from a long video (varying from 2min to 2h) and shuffle their chronological order. As shown in Figure 7, given the long video and the detailed descriptions of shuffled three scenes, the model is required to give the correct temporal order of the scenes based on the understanding of the entire video.

In Table 4, "-" means these models fail to give answers. Different from previous tasks that only have questions, Vript-ERO also contains long descriptions of scenes, which indicates these models are weak in processing long instructions. For models having scores, they only give the correct orders of all three scenes (@3) in about 20% of questions. In Figure 6 (b), we collect answers to the questions that are answered incorrectly and analyze the reasons. We observe that the models can be easily misled by the provided descriptions. For example, environment descriptions like sunlight may imply the morning or evening, however, these events may come from different days in the video rather than sequentially happening in one day. In 31.4% of cases, some events are absent in the input frames due to the limitation of the number of input images for models like GPT-4V. Besides, in 25.1% of cases, the models do not recognize which scene to be sequenced based on the descriptions.