-
GenSco: Can Question Decomposition based Passage Alignment improve Question Answering?
Authors:
Barah Fazili,
Koustava Goswami,
Natwar Modani,
Inderjeet Nair
Abstract:
Retrieval augmented generation (RAG) with large language models (LLMs) for Question Answering (QA) entails furnishing relevant context within the prompt to facilitate the LLM in answer generation. During the generation, inaccuracies or hallucinations frequently occur due to two primary factors: inadequate or distracting context in the prompts, and the inability of LLMs to effectively reason throug…
▽ More
Retrieval augmented generation (RAG) with large language models (LLMs) for Question Answering (QA) entails furnishing relevant context within the prompt to facilitate the LLM in answer generation. During the generation, inaccuracies or hallucinations frequently occur due to two primary factors: inadequate or distracting context in the prompts, and the inability of LLMs to effectively reason through the facts. In this paper, we investigate whether providing aligned context via a carefully selected passage sequence leads to better answer generation by the LLM for multi-hop QA. We introduce, "GenSco", a novel approach of selecting passages based on the predicted decomposition of the multi-hop questions}. The framework consists of two distinct LLMs: (i) Generator LLM, which is used for question decomposition and final answer generation; (ii) an auxiliary open-sourced LLM, used as the scorer, to semantically guide the Generator for passage selection. The generator is invoked only once for the answer generation, resulting in a cost-effective and efficient approach. We evaluate on three broadly established multi-hop question answering datasets: 2WikiMultiHop, Adversarial HotPotQA and MuSiQue and achieve an absolute gain of $15.1$ and $5.9$ points in Exact Match score with respect to the best performing baselines over MuSiQue and 2WikiMultiHop respectively.
△ Less
Submitted 14 July, 2024;
originally announced July 2024.
-
ReAct: A Review Comment Dataset for Actionability (and more)
Authors:
Gautam Choudhary,
Natwar Modani,
Nitish Maurya
Abstract:
Review comments play an important role in the evolution of documents. For a large document, the number of review comments may become large, making it difficult for the authors to quickly grasp what the comments are about. It is important to identify the nature of the comments to identify which comments require some action on the part of document authors, along with identifying the types of these c…
▽ More
Review comments play an important role in the evolution of documents. For a large document, the number of review comments may become large, making it difficult for the authors to quickly grasp what the comments are about. It is important to identify the nature of the comments to identify which comments require some action on the part of document authors, along with identifying the types of these comments. In this paper, we introduce an annotated review comment dataset ReAct. The review comments are sourced from OpenReview site. We crowd-source annotations for these reviews for actionability and type of comments. We analyze the properties of the dataset and validate the quality of annotations. We release the dataset (https://github.com/gtmdotme/ReAct) to the research community as a major contribution. We also benchmark our data with standard baselines for classification tasks and analyze their performance.
△ Less
Submitted 2 October, 2022;
originally announced October 2022.
-
Density Functions subject to a Co-Matroid Constraint
Authors:
Venkatesan T. Chakaravarthy,
Natwar Modani,
Sivaramakrishnan R. Natarajan,
Sambuddha Roy,
Yogish Sabharwal
Abstract:
In this paper we consider the problem of finding the {\em densest} subset subject to {\em co-matroid constraints}. We are given a {\em monotone supermodular} set function $f$ defined over a universe $U$, and the density of a subset $S$ is defined to be $f(S)/\crd{S}$. This generalizes the concept of graph density. Co-matroid constraints are the following: given matroid $\calM$ a set $S$ is feasibl…
▽ More
In this paper we consider the problem of finding the {\em densest} subset subject to {\em co-matroid constraints}. We are given a {\em monotone supermodular} set function $f$ defined over a universe $U$, and the density of a subset $S$ is defined to be $f(S)/\crd{S}$. This generalizes the concept of graph density. Co-matroid constraints are the following: given matroid $\calM$ a set $S$ is feasible, iff the complement of $S$ is {\em independent} in the matroid. Under such constraints, the problem becomes $\np$-hard. The specific case of graph density has been considered in literature under specific co-matroid constraints, for example, the cardinality matroid and the partition matroid. We show a 2-approximation for finding the densest subset subject to co-matroid constraints. Thus, for instance, we improve the approximation guarantees for the result for partition matroids in the literature.
△ Less
Submitted 30 July, 2012; v1 submitted 22 July, 2012;
originally announced July 2012.
