Retrieve, Summarize, Plan: Advancing Multi-hop Question Answering with an Iterative Approach

As mentioned earlier, our goal is to address issues of context overload and redundant planning. To tackle context overload, a straightforward approach is to employ summarization to compress information. However, even with summarization, the model still lacks an explicit record of the planning path, which does not resolve the issue of redundant planning. During the iterative process, over-planning could arise if summaries overlook information crucial for directly addressing the overarching question, or repetitive planning might occur if the information difference between different rounds of summaries is not significant. Therefore, a more sophisticated design of the summarizer is necessary to distinguish the functions of various pieces of information.

Drawing on the idea of query-focused summarization, we have designed a dual-function summarizer. Confronted with retrieved documents, this summarizer concurrently executes two tasks: producing summaries of supporting information pertinent to the overarching question and generating responses for the current sub-question, while managing two distinct memory queues–the global evidence memory and the local pathway memory. Summaries related to the overarching question are deposited into the global evidence memory, serving to explicitly signal the model to cease iterations when information is enough, thus mitigating the risk of over-planning. Concurrently, responses for the current sub-question, alongside the sub-question itself, are stored in the local pathway memory. This explicitly guides the model’s recognition of the progress in the question planning path as well as the sub-questions that have been historically retrieved, preventing repetitive planning.

Here we delineate the refined iterative RAG workflow that incorporates the dual-function summarizer.

Given a query Q and a document corpus D, we initially deploy a retriever to identify the K documents from D that are most relative to Q. These documents are then directed into the summarizer for summary creation and memory queue updates (note that during the first iteration, the sub-question is essentially the overarching question, so there is no generation of response for the current sub-question). Subsequently, the contents of the two memory queues are concatenated to provide context input for the reasoner, which is responsible for determining whether the current information is sufficient to address the overarching question. Should it be adequate, the iterative process is halted, and the memory queues are utilized as context for the generator to produce the final answer. If the information is insufficient, the reasoner generates a subsequent sub-question Q* that is distinct from previously retrieved sub-questions based on the current context, prompting the next iteration round.

We conduct experiments on two multi-hop question-answering benchmark datasets: HotpotQA (Yang et al., 2018) and 2WikiMultihopQA (Ho et al., 2020). Following the open-sourced RAG toolkit FlashRAG (Jin et al., 2024), we employ its preprocessed dataset format. For each dataset, we utilize the first 1,000 entries from the original development set for testing. We report the token-level F1 score of answer strings to evaluate the quality of the generation.

In our main experiment, we employ the Llama3-8B-instruct (AI@Meta, 2024) as the base model and the E5-base-v2 (Wang et al., 2022) as the retriever, while utilizing Wikipedia data from December 2018 as the retrieval corpus. For the model and data links, please refer to the FlashRAG open-source repository ¹¹1https://github.com/RUC-NLPIR/FlashRAG.

The model’s maximum input length is set to 12,000, and the maximum output length is set to 200. For each query, we retrieve the top-5 documents from the corpus based on vector similarity as the result. The maximum number of iterations is set to 3. If the retrieval process is still in iteration after 3 attempts, the model will directly proceed to the final response generation. All experiments are conducted on 4 NVIDIA A100 GPUs.

We select representative methods from single-round RAG and iterative RAG as baselines for comparison.

Our results on HotpotQA and 2WikiMultihopQA are displayed in Table 1. First, we notice that iterative RAG, especially IRCoT, demonstrates significant performance improvements compared to single-round RAG. This suggests that conducting multiple rounds of retrieval can indeed capture information more comprehensively and produce more accurate responses. Second, within single-round RAG, RECOMP, which incorporates summarization, exhibits superior performance, indicating that summarization is an effective method of information compression even within single-round RAG. These findings validate the rationale behind our approach, which combines multi-round retrieval with summarization.

Our method, ReSP, achieves significant improvements on both datasets, outperforming the SOTA by 4.1 F1 score on HotpotQA and 5.9 F1 score on 2WikiMultihopQA, surpassing a range of existing iterative RAG methods. This demonstrates the effectiveness of the approach we propose.

ReSP features a modular design, wherein each module works independently, allowing for the use of different base models to collaborate. To provide empirical guidance for model selection in practical applications, we test how the size of the model affects the performance of each module.

We substitute Llama3-70B-Instruct for Llama3-8B-Instruct and use this larger model as the base for the Reasoner, Summarizer, and Generator modules respectively, comparing the effect changes with the original results. Table 2 presents our experimental results.

Firstly, regarding the reasoner module, the changes are inconsistent across the two datasets, with an improvement on HotpotQA but a decline on 2WikiMultihopQA. The reason for this inconsistency is that 2WikiMultihopQA has questions with more logical hops compared to HotpotQA. A larger model is likely to give more detailed planning steps, leading to a failure to obtain all the necessary information to answer the question within the set maximum number of iterations, hence causing a drop in performance.

Secondly, for the summarizer module, we observe that using a larger model size does not result in performance improvements; in fact, there is a significant decline on 2WikiMultihopQA. Upon reviewing the summarization logs, we find that Llama3-70B-Instruct is more lenient in discerning relevance. It tends to extract information that seems related but is actually irrelevant to the question, which can disrupt the planning and ultimately the generation of responses.

Lastly, regarding the generator module, we observe consistent improvements when using Llama3-70B-Instruct, which suggests that even when provided with clear evidence, a model with stronger semantic comprehension capabilities still aids in generating more accurate responses.

In summary, in real-world applications, it is advisable to allocate a larger base model to the generator, as long as the available resources allow for it. However, for the reasoner module, if the allowable number of iterations is low, there is no need to use a larger base model. The summarizer also does not require a larger base model.

To determine whether ReSP can address the issue of context overload, we adjust the number of documents retrieved per iteration and observe the changes in performance. Our comparison involves using standard RAG and IRCoT as control groups. These experiments are carried out on the HotpotQA dataset. Results are shown in Figure 2.

Firstly, when the number of documents retrieved per iteration k is set to 3, all three methods experience a notable decrease in performance. This indicates that the adequacy of information retrieved in a single round significantly affects both single-round and iterative RAG methods. Therefore, it is necessary to extend the context length during application appropriately.

When k is greater than 5, the standard RAG and IRCoT exhibit a performance degradation trend. Particularly, IRCoT, which utilizes iterative retrieval, suffers from a more significant performance drop due to the accumulation of retrieved information. This demonstrates that context overload has a pervasive impact on existing RAG methods.

Our method demonstrates exceptional robustness to context length, delivering consistent performance regardless of whether k is set to 15 or 5. This is because we extract key information in each iteration, effectively maintaining a stable and concise context for the generator. As a result, the generator remains unaffected by changes in the length of retrieved documents during the process.

We present evidence of ReSP addressing over-planning and repetitive planning by comparing cases of IRCoT and ReSP on two questions, as shown in Table 3.

In the first case, the retrieved documents are consistent since the initial retrieval question is the same. However, the two models make different decisions about what to do next. IRCoT, which integrates information processing and planning in one step, has a higher level of task complexity. It mistakenly misses information related to "Twisted Fortune", which leads the model to decide that further retrieval is needed, resulting in over-planning. On the other hand, ReSP accurately and comprehensively acquires the supporting facts related to the overarching question through the summarizer. Consequently, the reasoner determines that the question can be answered, and the generator produces the correct response, thereby halting the iteration after the first round of retrieval.

In the second case, at the end of the first round of retrieval, both models make similar decisions due to the absence of information related to the main subject, "Rachelle Amy Beinart" in the documents: they both query for documents related to "Rachelle Amy Beinart". However, due to limitations in document coverage or retriever capability, no relevant documents on "Rachelle Amy Beinart" are found. At this point, the two models propose different sub-questions. Lacking a recorded retrieval trajectory, IRCoT can only make judgments based on the current information, thus continuing to query "Rachelle Amy Beinart", leading to repetitive planning. In contrast, ReSP, which avoids outputting previously retrieved sub-questions, adjusts the retrieval subject and produces a sub-question about "film that follows a group of rebels on a mission", thereby avoiding repetitive planning.