RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems

RAGBench comprises five distinct domains: bio-medical research (PubmedQA, CovidQA), general knowledge (HotpotQA, MS Marco, HAGRID, ExperQA), legal contracts (CuAD), customer support (DelucionQA, EManual, TechQA), and finance (FinBench, TAT-QA). We select these specific domains based on availability of data, and applicability to real-world RAG applications across different industry verticals. For detailed descriptions of each component data source, refer to Appendix 9.2.

Context token length in RAGBench ranges from 100 to 11k tokens, which we report in Table 1. Notably, CUAD documents feature long contexts of up to 11k tokens each, compared to the relatively short context in PubMedQA.

We curate RAGBench to inlcude a variety of difficult RAG task types. Customer support datasets simulate a common application of RAG in industry settings. FinQA and TAT-QA require numerical reasoning over hybrid tabular and text data. HotpotQA, CovidQA, and PubMedQA necessitate retrieval and reasoning over multiple context docs. The CUAD dataset is a challenging addition to RAGBench for several reasons: (i) it represents a difficult and highly-specialized real-world domain in which of-the-shelf pre-trained LLM models struggle to perform well [25], and (ii) it is equally challenging in RAG context due to very long context lengths of legal contract documents.

All component datasets include domain-specific questions that represent real-world user queries about various topics. Questions for DelucionQA, HotpotQA, and EManual are crowd-sourced; questions for CovidQA, CUAD, HAGRID, ExpertQA, and FinQA are composed by domain experts; MS Marco is sourced from real-world user web search queries; likewise, TechQA questions are user queries posted on IBM technical forums; PubMedQA is the only dataset with automatically-generated questions from research article titles.

For each component dataset we generate responses with LLMs. Exceptions to this are HAGRID and ExpertQA datasets, which contain LLM-generated responses in the original data. To introduce variability into the dataset, we generate two responses per input with different modes: GPT-3.5 (gpt-3.5-0125) and Claude 3 Haiku. Both are proprietary models that are offered at a reasonable price point²²2https://openai.com/api/pricing/, https://www.anthropic.com/api, which we believe make them suitable candidates for generating real-world RAG responses. For CUAD we only generate responses with Claude 3 Haiku due to prohibitively long context lengths that exceed the GPT-3.5 16k token limit. To encourage a diverse distribution of labels in RAGBench, we use a basic prompt (Appendix 9.3) that does not explicitly require the model to stick to the provided context when generating the response. We set the temperature to 1.0 for generation.

We split each component dataset into train, validation, and test sets, ensuring there is no overlap in queries across splits from the same data source. RAGBench totals 100k samples, split across train, validation, and test sets. Component dataset statistics are reported in Table 1.

RAGBench component datasets contain between 1% - 20% hallucinations. ExpertQA, CovidQA, and MS Marco contain the highest fraction of hallucinated responses (12%, 16%, and 13%, respectively), while Cuad, FinQA, and TAT-QA contain the least (about 1% for each). We visualize distributions of relevance, utilization, and completeness scores in Figure 2.

Let $D$ be a set of context documents $\{d_{1}...d_{n}\}$ retrieved for a RAG input query. We define a set of relevant tokens in $d_{i}$ as $R_{i}=\{t_{1},...t_{r}\}$. $R_{i}$ encodes information in context document $d_{i}$ that is useful for answering the query. Similarly, we define $U_{i}=\{t_{1},...t_{u}\}$ as the set of utilized tokens in document $d_{i}$, which reflect information that the generation model is using to produce a response. Refer to Figure 3 for a visual representation of relevant and utilized spans. $Len(x)$ measures the length of strings in $x$, which can be interpreted as character length, token length, or sentence length. For calculating ground-truth metrics, we employ sentence-length, since it aligns best with our annotation schema (Section 3.3). However, token or character length may also be suitable for other use cases.

Context Relevance is defined in [9, 33] as the fraction of the retrieved context that is relevant to the input query. Low relevance points to an inefficient retriever that supplies excess information to the generation model. Long context inputs into the generator may accrue unnecessary costs, as well as compromise the quality of the generated output. We measure relevance of context document $d_{i}$ as:

$$\text{document relevance}=\frac{{Len(R_{i})}}{Len(d_{i})}$$

(1)

Example-level relevance can be aggregated over all context documents in the example as:

$$\text{example relevance}=\frac{\sum_{i=1}^{|D|}{Len(R_{i})}}{\sum_{i=1}^{|D|}Len(d_{i})}$$

(2)

Context Utilization is a new metric introduced in TRACe. We aim to measure the the fraction of the retrieved context that is used by the generator to produce the response. Low Utilization in combination with low Relevance points to a greedy retriever, while low Utilization alone points to a weak generator that fails to leverage the provided context efficiently. Document-level and example-level Utilization are defined as:

$$\text{document utilization}=\frac{{Len(U_{i})}}{Len(d_{i})}\quad\text{example utilization}=\frac{\sum_{i=1}^{|D|}{Len(U_{i})}}{\sum_{i=1}^{|D|}Len(d_{i})}$$

(3)

Completeness is another new metrics we introduce to measure how well the response incorporates all the relevant information in the context. Note that this is different from Utilization; it is possible to have high Relevance and high Utilization, but low Completeness when the generator utilizes irrelevant information in the context to produce a low quality response. Completeness for document $d_{i}$ is calculated as the fraction of utilized substrings among all relevant substrings:

$$\text{completeness}=\frac{Len(R_{i}\cap U_{i})}{Len(R_{i})}$$

(4)

And can be extended to example-level by considering all relevant and utilized substrings across all context documents.

Adherence is designed to detect hallucinations in RAG responses. Our definition of Adherence is synonymous with answer faithfullness [9, 33], groundednes [37], and attribution [32]. For alignment with existing hallucination detection approaches, we define example-level adherence as a boolean indicating whether or not all parts of the response are grounded in the context. However, in our annotation schema (Section 3.3) we also define $A_{i}=\{t_{1},...t_{a}\}$ as the set of response tokens that are supported by the context to enable granular Adherence evaluation.

PubMedQA is a collection of PubMed research abstracts with corresponding yes/no/maybe questions paired with each abstract. The original dataset comprises 3 subsets: PQA-L, PQA-U, and PQA-A, with 1k, 60k, and 210k abstracts, respectively. For all subsets, the question is derived from the title of the PubMed article using rule-based heuristics. Long answers are automatically derived from the last sentence of the abstract for PQA-L and PQA-U, and QA-L answers are further reviewed by expert annotators and annotated as yes/no/maybe. PQA-A comprises exclusively automatically generated questions and short answers.

For RAGBench we utilize the PQA-U subset and re-frame it from QA into a RAG task. To simulate RAG, we leverage already segmented PQA-U abstracts context chunks and we encode them into a vector DB with OpenAI embeddings. The size of the resulting DB is 200k. We retrieve 4 chunks for each PQA-U question using FAISS with eucledian distance as the similarity function. We ignore the responses and labels in the original dataset and generate new responses with an LLM.

CovidQA-RAG is a combination of 2k expert-annotated questions sourced from COVID-QA [27] and a vector database of 250,000 100-word passages built by Siriwardhana et al. [35]. Both questions and answers are sourced from CORD-19 [39] collection of research articles about COVID-19.

We embed the questions and database passages with OpenAI embeddings and retrieve up to N passages for each COVID-QA question from the vector database using FAISS with eucledian distance as the similarity function and max_distance=0.25. We generate responses for each resulting RAG (context, question) instance with an LLM.

HotpotQA comprises 113K crowd-sourced question-answer pairs sourced from Wikipedia. Each pair is associated with a set of related context passages from one or multiple Wikipedia pages. The dataset is constructed in a way that requires multi-hop reasoning over multiple context documents to arrive at the answer, which renders it a valuable candidate for our benchmark. We sample data from the dev-distractor split, which contains up to 8 distractor context documents per sample. We downsample the context documents to 4 per example, making sure to include the document containing the response. We treat the context passages in HotpotQA as RAG context documents, and generate responses for each (context, question) instance with an LLM.

MS Marco is an open-domain question answering dataset sourced from Bing search engine user query logs. Each question is associated with 10 context passages retrieved via Bing web search. Human annotators compose a response based on the provided context documents, and label the documents utilized in the response as relevant. We sample data from the original version of the dataset, comprising 80k train, 10k validation, and 10k test samples. As with other datasets, we ignore the human annotated answers and generate responses with an LLM in RAG setting.

CUAD is a collection of commercial legal contracts with expert annotated questions and responses. The contracts are sourced from a public legal contract library(EDGAR) and range from 1-100 pages in length. Experts in the legal domain compose multiple questions per contract and label the relevant parts of the contract that are useful for answering the questions. There are 21k questions pertaining to 510 documents in total. The questions are very specific to each contract, thus we don’t perform additional retrieval over the contract corpus, and form RAG examples with 1 context contract each for our benchmark. Due to high anntoation costs associated with long-context RAG, we sample 5 question per doc. As with other datasets, we generate responses with an LLM in RAG setting.

DelucionQA is a domain-specific RAG dataset leveraging Jeep’s 2023 Gladiator model manual as the source of knowledge. The questions and answers are automatically generated by large language models. RAG context passages are retrieved from the Jeep car manual via both sparse and dense retrieval methods to add variance in the sample distribution. Further, MTurk workers annotate whether or not responses are supported by the context.

Upon closer inspection, we found only 1 relevant passage associated with each question in the DelucionQA dataset. To make the dataset more challenging for RAGBench, we build a vector database from the 1,046 context passages in DelucionQA and and retrieve up to 3 context documents per question from it. We use text-embedding-ada-002 embeddings from OpenAI to build the database. There are 913 unique questions in DelucionQA. For each resulting (context, question) sample, we generate responses with an LLM.

EManual is a question answer dataset comprising consumer electronic device manuals and realistic questions about them composed by human annotators. The subset made available at the time of writing amounts to 659 unique questions about the Samsung Smart TV/remote and the accompanying user manual, segmented into 261 chunks. To form a RAG dataset, we embed the manual segments into a vector database with OpenAI embedding and retrieve up to 3 context documents per question from it. For each resulting (context, question) sample, we generate responses with an LLM.

TechQA is a collection of real-world user questions posted on IBMDeveloper and DeveloperWorks forums, along with 50 technical support documents relating to each question. The documents are sourced from database of 800k technical documents that support accepted answers on the tech forums. The authors release 1.4k questions, split between train, validation, and test sets. The data are curated such that fractions on the each split unanswerable given the information in the linked documents, which makes it a good candidate for RAGBench. To reduce annotation costs, we sub-sample the data down to 10 documents per question, making sure to include the document containing the answer, when applicable. We use the provided splits with (context document, question) examples and generate responses for each with an LLM.

FinQA is a QA dataset of financial report passages and associated questions. Questions are curated such that numerical reasoning over multiple unstructured and tabular inputs is required to arrive at the answer. FinQA totals 8,281 financial QA pairs, split between train, validation, and test splits. We retain the original splits and generate 2 LLM responses per each context-query example in FinQA.

TAT-QA is another financial QA dataset that requires numerical reasoning over tables and text. The data are sourced from 500 financial reports released on https://www.annualreports.com/. Expert annotators with background in finance annotate question-answer pairs based on the available documents. We leverage the full dataset (13k train, 1.6k validation and test) but generate new responses with LLMs for RAGBench.

HAGRID is a QA dataset built on top of MIRACL [45], a multi-lingual information-retrieval dataset. HAGRID passes questions and relevant context documents from MIRACLE through an LLM to produce a response for each example in the dataset. Annotors then rate the response on informativeness and attribution dimensions. The original context documents are sourced from Wikipedia and associated questions are generated by expert annotators. Since HAGRID already contains LLM-generated responses, we directly use them and don’t generate additional responses for RAGBench.

ExpertQA is a collection of curated questions from domain-experts in various fields of sicence, arts, and law. The dataset also contains expert curated passsages relevant to each question, alongside LLM-generated responses. As with HAGRID, we leverage the LLM-generated responses in ExpertQA directly for our RAG dataset.