GenSco: Can Question Decomposition based Passage Alignment improve Question Answering?

Researchers have experimented with in-domain few-shot examples with LLMs to generate queries Dai et al. (2022); Bonifacio et al. (2022); Boytsov et al. (2023)

Thereafter, neural retrieval models are fine-tuned over this enhanced dataset.

While LLMs have been previously used to score paragraphs based on their relevance to a certain query Sachan et al. (2022), directly applying them to determine pertinent paragraphs based on a complex query requiring intricate reasoning results in sub-optimal performance. Our novel approach of breaking down a complex query into a set of simpler elemental queries allows our method to leverage generative ability of LLMs for accurate retrieval and better answering performance. Zhang et al. (2023) introduce beam retrieval for multihop QA optimizing learnable parameters across all hops. In contrast, our suggested method exclusively employs an inference approach, eliminating the necessity for training or training examples.

One of the well known techniques for MHQA is to decompose the complex query into simpler sub-questions, answer them and then combine the results to get the overall answer Fu et al. (2021). Recently researchers have proposed deconstruct a complex problem into a series of simpler sub-problems before feeding to LLMs Zhou et al. (2022); Press et al. (2022).

Despite significant advancements in LLMs that have enhanced their reasoning capabilities and reduced the disparity between machine and human intelligence, using them directly to answer sub-questions might result in inaccurate responses due to the lack of appropriate knowledge. Thus, instead of directly solving the sub-problems via prompting, we leverage LLMs in finding the right context for each of the sub-problems. This is inline with the conclusions drawn by  Zheng et al. (2023) - providing fine-grained external knowledge can boost the truthfulness of LLMs in answering a question.

For decomposition, the multi-hop Question $Q$ is provided to G along with the previously generated subquestions and corresponding passages selected for each subquestion. The prompt also includes an instruction to flag the end of decomposition when no more subquestions can be created for $Q$ by generating a specified keyword. Apart from relying on G to flag the end of decomposition or detecting a repeated subquestion, we propose a stopping criterion based on the log-likelihood scores from S: Scorer model S is instructed to generate a multihop question given the decomposition and we evaluate the following expression where $nll_{S}$ refers to negative log-likelihood from Scorer S:

When the likelihood given the decomposition including the generated subquestion $q_{i}$ falls below the likelihood while excluding $q_{i}$ from the prompt, we stop exploring more passages. Note that the prompt to S also includes $C$ which represents the concatenation of passages selected for $q_{1},...,q_{i-1}$ represented as $:P^{\prime}_{1},...,P^{\prime}_{i-1}$. This is included to provide the model enough context to synthesize(indirectly score) the composite question $Q$. The prompt is designed only to extract likelihood scores from the scorer S in order to get a proxy for the significance of the new subquestion $q_{i}$ in the decomposition of $Q$, therefore, take note that we ignore the question generated here by S.

Once the subquestion $q_{m}$ has been formulated for level $m$, we proceed to expand the most favorable node from the previous level $(m-1)$ into k child nodes. Each of these child nodes, denoted as $n_{m,i}$, represents the concatenated sequence of passages chosen along the greedy path from the root node to the parent of $n_{m,i}$, followed by the passage $p_{i}$. For each child node at level $m$, a score is computed relative to the subquestion $q_{m}$ using the scorer LLM S. The LLM S is instructed to produce a question based on the passage sequence represented by each node at level $m$, yielding $m$ distinct scores for subquestion $q_{m}$ being posed as the output text sequence.

The score for node $n_{m,i}$ is computed by taking the log likelihood of the token sequence $q_{m}$ in the output distribution of S given the already selected passages($P^{\prime}[:m-1]$) plus passage $p_{i}$ to the scorer S.

After the passage selection is done, the generator is called to which we provide the multi-hop question and selected passage sequence in a few-shot setting to generate the answer.

Here $P^{\prime}$ denotes the sequence of passages chosen following a greedy approach, with the final level being either the maximum permitted levels or determined by stopping conditions, whichever is reached first.

Note that in our log-likelihood expressions, we compute the probability of the question conditioned on the passages rather than the other way around which may be less intuitive since we should be scoring the passages(and not the question). As has been discussed in prior work Sachan et al. (2023), either of the two forms could be used theoretically (using the Baye’s rule and taking passage priors as uniform which is a reasonable assumption for reranking). This form is more practical(and faster) since passages tend to be much longer than the question and the likelihood of longer sequences could approach zero.

We have conducted experiments on three different datasets: (i) 2WikiMultiHop Ho et al. (2020), (ii) Adversarial HotPot Yang et al. (2018); Ye and Durrett (2022); (iii) MuSiQue Trivedi et al. (2022). More details of the datasets can be found in Section A in Appendix.

We start with retrieval methods, BM25 Robertson and Walker (1994) and GTR Ni et al. (2021), by extracting the most relevant passages, and fed to the LLM for answer generation. We compare against Verify-and-Edit Zhao et al. (2023a) (CoT-SC + VE), that focuses on post-editing ‘Chain of Thought’-style reasoning, ‘Iter-Retgen’ Shao et al. (2023), ReAct Yao et al. (2023b) and SelfAsk Press et al. (2023). Both ReAct  Yao et al. (2023b) and Self-Ask Press et al. (2023) are approaches that involve the decomposition of complex/multihop questions using Large Language Model (LLM) prompting techniques.Also, we compare with a cross-encoder³³3https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-6-v2 trained for MS Marco Passage Ranking. Our apporach of GenSco is an inference method, hence not directly comparable to the full finetuning setup. We draw inspiration from different approaches researchers have tried to make zero-shot/few-shot settings effective for the MHQA Gao et al. (2024). The detailed explanation can be found in Apendix.

For 2WikiMultiHop, as the dataset has atmost 5 supporting passages per instance, we select the top-5 passages for each of the implemented baselines (top three rows against each dataset in Table 1). The passages are then provided as context in the prompt to the LLM for QA. Also, as part of the prompt we provide two instances from the train set for our few-shot prompting of the Generator LLM . For Adversarial HotPot, we provide top-2 passages using each of the three implemented baseline methods, prompting the Generator LLM in the same way but with 4-shots. For MuSiQue, we augment the prompt with the top-4 passages for each of the three implemented baselines and include 3 shots of train instances in the prompt.

For 2WikiMultiHop, GenSco explores the search tree to a limit of 5 levels, considering that each instance has a maximum of 5 supporting passages. In terms of prompting the Generator LLM for GenSco , we include two instances from the training set as for the baseline retrieval methods. For the Adversarial HotPot dataset and MuSiQue similarly, we utilize 4 shots and 3 shots of training instances respectively as for the implemented baselines. Verify-and-Edit Zhao et al. (2023a) however, uses six shots for both its datasets.
We leverage GPT3.5⁴⁴4text-davinci-003 model as the generator LLM for all techniques as has been used for the Verify-and-Edit baseline. For GenSco, 3B version of Open-llama⁵⁵5Open LLAMA on Hugging Face is taken as the scorer LLM. We implement two main variations of our GenSco approach. The first variant, named GenSco-stop, includes the scorer log-likelihood based condition in inequality (1) as an additional stopping criteria. The second variant, called GenSco-max, either detects the specified end-of-decomposition keyword in the model output or identifies redundant subquestions (already generated for the instance) as the stopping conditions. Note that GenSco-stop incorporates the likelihood based stopping criterion as a third alternative beyond the two criteria in GenSco-max to stop the greedy search.

The scores of CoT-SC+VE Zhao et al. (2023a), ReAct Yao et al. (2023b), Self-Ask Press et al. (2023) and Iter-Retgen Shao et al. (2023) for all three datasets have been directly taken from prior work Zhao et al. (2023a); Shao et al. (2023) while all other methods listed in the tables are locally evaluated.

In this section we will discuss the performance of our methodology on the three datasets and will compare with the baseline systems. Refer to the Table 1 for the results.

To gauge the role of question decomposition in GenSco, we setup an alternate implementation with the following two modifications to GenSco-max algorithm:

1. 1.

   Instead of using the subquestion at the level to compute the log-likehood expression for passage selection in equation 2, we use the original multi-hop question to compute the scores:

   $$\begin{multlined}score(n_{m,i})=nll_{S}(p(Q/P^{\prime}[1:m-1],P[i])),\\ i\epsilon[1,k]\end{multlined}score(n_{m,i})=nll_{S}(p(Q/P^{\prime}[1:m-1],P[i])),\\ i\epsilon[1,k]$$

   (4)

2. 2.

   For stopping criterion, we check if the best scoring paragraph at any level has already been selected along the greedy path. i.e. if $P^{\prime}[m]\quad\epsilon\quad P^{\prime}[1:m-1]$.

This variant is called GenSco-no-QD in Table 2 where we compare the correctness scores with two of the baseline methods and variants of GenSco on a subset of 250 instances from 2WikiMultiHop. The numbers severely drop down wrt to both GenSco-stop and GenSco-max but are still better than BM25.

We also evaluated the model responses in comparison to baseline retrieval methods over faithfulness. K-precision Adlakha et al. (2023) serves to quantify how well a model’s response is grounded within the provided passages. Consequently, if the generated answer shares more words with the passages, the k-precision value will be higher, regardless of its correctness. While GenSco methods are more accurate with respect to the ground truth answers, the GTR method achieves higher K-precision value (Table 3) for 2WikiMultiHop. Figure 4 shows a scatter plot over a set of 960 instance responses for 2WikiMultihop from GenSco-stop and the GTR based method to observe the correlation between K-precision (indicating faithfulness) and F1 (which is a correctness metric). It’s surprising that a higher K-precision score doesn’t necessarily indicate increased correctness, as the data points are almost uniformly dispersed across the F1 axis for high k-precision values. The Pearson correlation coefficients for the two metrics for both GTR and GenSco-stop are hence low: $0.138$ and $0.238$, respectively. Typical RAG methods such as with GTR lack flexibility by relying on a fixed number of passages (k) for each instance typically setting k to be the estimated maximum number of hops required across the instances in the dataset which may result in higher value of k-precision. In contrast, our GenSco methods effectively tailor the number of passages for each multi-hop query(based on the decomposition), potentially leading to lower k-precision. However, the correctness measures, which are our primary concern, are consistently higher for our method indicating the importance of supplying only relevant passages to the generator LLM.

Since GenSco retrieves the passages in alignment with the implicit reasoning implied by the multi-hop question, the order in which these passages should be input to the Generator LLM is derived as part of our algorithm. To assess if the order of passages plays any role in the MHQA task, we performed a simple experiment where the retrieved passage sequences for GenSco-stop and GenSco-max are randomly shuffled before feeding into the Generator LLM. Table  5 shows how the correctness scores significantly degrade on randomizing the order. This experiment confirms the intuition based on which GenSco has been proposed as not just as a passage selection technique but also as a technique that identifies the passage sequence. The retrieval baselines are not equipped to determine the order since the passage relevance is computed with respect to the entire question as opposed to its specific parts/subquestions.

To check whether the performance remains consistent over different sizes of data subsets, we plot each of our four correctness metrics averaged over the instances in figure 3. The aggregate values look fairly stable for GenSco-stop across various splits of 2WikiMultiHop dataset.