MM-Forecast: A Multimodal Approach to Temporal Event Forecasting with Large Language Models

In-context learning leverages both intrinsic and extrinsic factors to construct historical events. Specifically, the intrinsic factors of an event are related to its inherent elements, particularly the subject. In contrast, the extrinsic factors are driven by the contextual environment surrounding the event. Therefore, whether the data is structured or unstructured, we construct the historical events based on the subject and the complex event, separately. The details are as follows:

• •

  Structured Data. For structured data, the method takes the discrete event graph as the input. To capture the intrinsic factors, we use the subject of the current event as a guiding clue to construct the historical event graph $\mathbf{G}_{<t}^{s}=\{G_{0}^{s},G_{1}^{s},...,G_{t-1}^{s}\}$, where $G_{t}^{s}$ represents historical events graph at timestamp $t$ with the same subject as the current event. To account for the extrinsic factors, we construct the historical event graph from the complex event, i.e. $\mathbf{G}_{<t}^{c}=\{G_{0}^{c},G_{1}^{c},...,G_{t-1}^{c}\}$, where $G_{t}^{c}$ represents historical events graph at timestamp $t$ with the same complex event as the current event. Finally, with the highlighting and complementary functions of the images, the input historical event graph is $\mathbf{G}_{input}=[\textbf{G}_{k},\textbf{G}_{r},\textbf{G}_{c}]$, where $\mathbf{G}_{input}\in\mathbf{G}_{<t}^{s}\bigcup\mathbf{G}_{<t}^{c}$ and $\mathbf{G}_{k}$ denotes the key events, $\mathbf{G}_{r}$ represents the remaining events, and $\mathbf{G}_{c}$ corresponds to the complementary events, respectively.

• •

  Unstructured Data. For unstructured data, the method takes the textual sub-events as input. Firstly, we identify the events by the historical events graph from the subject and complex event and find the corresponding textual sub-events set $\mathbf{A}_{<t}^{s}=\{A_{0}^{s},A_{1}^{s},...,A_{t-1}^{s}\}$ and $\mathbf{A}_{<t}^{c}=\{A_{0}^{c},A_{1}^{c},...,A_{t-1}^{c}\}$ through the relationships between textual sub-events and graph sub-events. Then, with the highlighting and complementary functions of the images, the input historical textual sub-events are similarly $\mathbf{A}_{input}=[\textbf{A}_{k},\textbf{A}_{r},\textbf{A}_{c}]$, where $\mathbf{A}_{input}\in\mathbf{A}_{<t}^{s}\bigcup\mathbf{A}_{<t}^{c}$ and $\mathbf{A}_{k}$ denotes the key events, $\mathbf{A}_{r}$ represents the remaining events, and $\mathbf{A}_{c}$ corresponds to the complementary events, respectively.

Despite the rich information provided by in-context learning methods, the inherent nature of the temporal event means that the existing historical event still contains substantial noise. Inspired by the recent research of RAG (Lewis et al., 2020), we also adopt the retrieve-then-generate paradigm to find the most relevant historical events to mitigate the problem of noise. Similar to ICL methods, we utilize two forms of data representation, structured data and unstructured data:

• •

  Structured Data. Due to the structured nature of the data representation, the event graphs adhered to a unified quintuple format. Therefore, we first retrieve the entities that have interacted with the subject of the query event. Once we have obtained the related entity set, we can construct the history with the historical events where the subject or object is within this set. Similarly, through the function of images, the retrieval process also contains key events and complementary events.

• •

  Unstructured Data. Unlike structured data, we can use the embedding techniques to directly retrieve relevant news events from a set of historical news articles for the unstructured data. Following this, we filter historical news events based on timestamps, eliminating outdated and irrelevant events. We also select the key events and complement information based on the images, which will be input according to the prompts described in Section 3.2, and finally obtain the prediction results.

We build our dataset based on MidEast-TE-mini (Chang et al., 2024), which includes structured atomic events and news articles. We aim to add images that correspond to the events in the dataset, hence we will have the data in visual modality. An intuitive way is to download the web page according to the URL provided by the original dataset. However, the original web page always contains a lot of irrelevant images, such as advertisement images, that are cumbersome and difficult to be accurately filtered out. Instead of directly solving this problem, we propose an alternative solution that we use Google Image Search⁴⁴4https://images.google.com/ to search the images using the news article title as the query. Among the returned images, we select the top-ranked ones as the associated images of the news article. In order to further filter out irrelevant images, we instruct the Gemini-1.0-Pro-Vision model to determine the relevance of images to news articles. We give three options: highlighting, complementary and irrelevant. Highlighting means that the images and the content of the news are highly matched, and complementary means that the image has supplementary meaning to the content of the news. Images beyond these two are regarded as irrelevant. We further remove images that are classified as irrelevant. Finally, we name our dataset as MidEast-TE-multimodal, short as MidEast-TE-mm.

The compared methods are categorised into non-LLM-based methods and LLM-based methods. For non-LLM-based methods, only text or graph modalities are involved, since these methods architecture are fixed. We train the models on the training set, selecting the best-performing model based on the validation set results, and obtain the final results of the testing set. For LLM-based methods, we use the proprietary LLMs due to their superior performance compared to open-source LLMs. Therefore, testing is generally done in a zero-shot manner, i.e., directly test them on the testing set. The specific methods are shown below:

• •

  ConvTransE (Shang et al., 2019): This method employs a convolutional neural network (CNN) and a translational operation to capture the relational patterns within triplet data.

• •

  RGCN (Schlichtkrull et al., 2018): RGCN leverages a graph convolutional neural network (GCN) to capture the diverse relations between entities.

• •

  RE-GCN (Li et al., 2021): RE-GCN utilizes a combination of GCN and recurrent neural network (RNN) to capture both the relational patterns and temporal dynamics.

• •

  LoGo (Ma et al., 2023b): This method models relationships within and between complex events from both local and global perspectives.

• •

  GPT-3.5-Turbo⁷⁷7https://platform.openai.com/docs/models/gpt-3-5-turbo: The GPT-3.5-turbo model is the prevalent iteration of the GPT (Generative Pre-trained Transformer) language model developed by OpenAI⁵⁵5https://openai.com/.

• •

  Gemini-1.0⁶⁶6https://ai.google.dev/models/gemini: Gemini-1.0 is a cutting-edge family of multimodal models developed by the Gemini Team at Google.

To ensure the reproducibility, we fixed the temperature parameter of the proprietary LLMs used to 0 and set the seed parameter to a constant value. When making forecasting, we limit the maximum output token length to 256 to prevent invalid responses. To ensure fairness across the experiments, the history that can be retrieved is set to 30 days. Notably, the retrieval models that we employ include: BM25 (Robertson et al., 2009), Contriever (Izacard et al., 2021), and LlamaIndex (Liu, 2022). Additionally, considering the limitation of the context window, we further restrict the maximum number of sub-events in the historical context to 50. Following previous methods (Chang et al., 2024), we employ the Accuracy (Acc) as the evaluation metric.

The overall performance comparison is presented in the Table 1. To comprehensively explore and evaluate methods, we conduct experiments across multiple dimensions, including the format of data representation (Text or Graph), the construction of historical information (RAG-based or ICL-based), and the prediction objective (Object or Relation). Clearly, we have the following observations.

First, enhancing LLM-based methods with visual information consistently improves their accuracy across all experimental settings. This demonstrates that our proposed MM-Forecast makes effective use of visual information, leading to a better contextual understanding of historical information. Hence, our method strengthens the inference ability of LLM and makes more accurate event forecasting performance.

Second, even though the performances of all LLM-based methods have been improved, they still under-perform the traditional Non-LLM based methods. The reason is that LLM-based methods are tested in zero-shot manner, while the Non-LLM methods, which follow supervised learning, are still competitive. Notably, by using our MM-Forecast method, LLM-based methods can achieve close or even better performance than Non-LLM methods for the object entity prediction task.

Third, the relation prediction task exhibits higher accuracy compared to the object entity prediction task. This suggests that the forecasting of entities is more challenging than relations. There are a few potential reasons for this. First, the set of entities (5909) is much larger than the set of relation types (267), so predicting specific entities is inherently more difficult given the larger candidate pool. Second, we deem that the information implied in entities is more explicit. Thus when two entities are given for a relation prediction, it is easier than when the subject and relation are given for an object prediction.

To illustrate the limitations of existing MLLMs in the task of temporal event forecasting, we also conduct experiments using the Gemini-1.0-Pro-Vsion model (Team et al., 2023) and directly consuming the images in the sub-events. Specifically, this approach leverages the inherent image understanding capabilities of the Gemini-1.0-Pro-Vision model, which embeds image patches as features and seamlessly concatenates thes image features with textual features. From Table 1, we can observe that the accuracy of using images directly is not only lower than our MM-Forecast, but also even worse than the method using only textual data (Uni-modal methods). This illustrates that existing proprietary MLLMs still struggle to make effective event forecasting with multiple images, and reflects the superiority of our MM-Forecast.

The choice of retrieval model may have a significant impact on forecasting. The experiments here involve only unstructured event forecasting, since the structured approach employs retrieval based on keyword search techniques. To explore the effect of retrieval model, we adopt three different retrieval models, i.e., BM25 (Robertson et al., 2009), Contriver (Izacard et al., 2021), and LLamaIndex (Liu, 2022), then equip them into our forecasting framework, and obtain the forecasting results. From the results in Table 2, we can observe that the performance progressively improves by using stronger retrieval models, with LLamaIndex performing the best, followed by Contriver, and then BM25. These results verify that stronger retrieval capabilities lead to better forecasting performance, suggesting that retrieval-oriented method design is a promising direction for future research. This phenomenon is consistent with the observation concluded from recent works (Chang et al., 2024).

We conduct ablation experiments for the highlighting and complementary function of images. The results are shown in Figure 3. First, the model that leverages both the highlighting of key events and the complementary information performs the best across the experimental settings. In addition, the performance of the model with only key events highlighted is sub-optimal. This illustrates the effectiveness of the highlighting function of images, and it elicit the fact that highlighting and complementary reinforce each other to achieve even better prediction results. Second, we can observe that in some settings (Text-ICL, Text-RAG), the performances of the model with only complementary information are even worse than the baseline model. The possible reason for this is that the offering of complementary information also introduces more noise and therefore leads the degradation of performance. Third, the performance of RAG-based method is obviously worse than the ICL-based method in the relation prediction task, meanwhile, such performance gap does not exist in the entity prediction task. This is may because that relation prediction is easier than object entity prediction, as mentioned in section 4.2.1. As a result, ICL-based historical events may already contain enough information to make accurate relation prediction, whereas the retrieval model may not retrieve relevant information instead.

In addition to overall forecasting performance analysis, we conduct in-depth study to directly assess the efficacy of highlighting and complementary function. Specifically, we design additional experiments at the data level and prompt level to further verify the function of images. At the data level, we randomly sample 100 images of two categories respectively, and then judge the correctness of the classification by the powerful MLLM GPT-4-Vision ⁸⁸8https://platform.openai.com/docs/models/gpt-4-turbo-and-gpt-4. As shown in Table 3, both classification of highlighting and complementary functions show high accuracy. Furthermore, we can observe that the accuracy of highlighting is lower than that of complementary on all settings, which should be due to its more strict definition. The high accuracy of image functions in both LLM and human identification indicate that the images we used can indeed play the highlighting and complementary functions. In addition to direct assessment of the quality of image function identification, we conduct another ablation study by replacing our identified functions with randomly selected sub-events. Looking into the forecasting results in Table 4, random selection of sub-events leads to a decrease in forecasting accuracy, indicating that correct image function identification is crucial to the forecasting.

Finally, on top of quantitative evaluation, we conduct qualitative analysis and demonstrate two examples in Figure 4. The first image emphasizes the event of Makhdoom Shah Mahmood Qureshi’s visit to Abdel Fattah Al-Sisi, highlighting their efforts to strengthen and diversify bilateral relations. This highlighting function leads to a correct prediction of the event type. The second image provides supplementary information about the meeting between the two politicians, enabling an accurate prediction of the question.