Evaluating Quality of Answers for Retrieval-Augmented Generation:
A Strong LLM Is All You Need

Since the launch of ChatGPT in November 2022, Large Language Models (LLMs) have become increasingly popular integrated components for organizations seeking to enhance productivity and enrich their product portfolio offerings. However, it is well known that GPT-4 Turbo’s training corpora cut-off date is April 2023, rendering the model lacking in current events knowledge. Furthermore, as LLMs are pre-trained on public domain text and do not possess proprietary information, their capabilities are limited when it comes to our company’s knowledge-intensive applications.

Fine-tuning (Radford et al., 2018; Dodge et al., 2020) is a technique that can be used to inject new knowledge into pre-trained LLMs by adjusting their gradient parameters through further, specialized training. However, OpenAI’s fine-tuning API is currently only available through an experimental access program, and GPT-4 fine-tuning requires more effort to achieve significant improvements, as noted by OpenAI (2023).

Retrieval-augmented generation (RAG) is proposed initially by Lewis et al. (2021). This method involves storing extra knowledge in a non-parametric dense vector index and using a pre-trained neural retriever to search relevant context, followed by generating content with a pre-trained sequence-to-sequence (seq2seq) model. Ovadia et al. (2024) argue that RAG consistently outperforms unsupervised fine-tuning on a wide range of knowledge-intensive tasks.

A RAG application leveraging LLMs enhanced with the company’s proprietary knowledge has become one of the pivotal factors advancing the adoption of GPT-4-based applications at our enterprise, a global payment technology company. This phenomenon highlights the need to apply viable approaches to evaluate RAG applications, as lacking performance metrics poses risks to the enterprise’ business and may have negative consequences.

One approach to evaluate the quality of RAGs focuses on their unique characteristic: that they consist of a Retriever model and a Generator model. Therefore, studies have used metrics such as context relevance, and answer relevance to evaluate the two components separately then assess the answer faithfulness between them (Saad-Falcon et al., 2024; Es et al., 2023).

The other paradigm for evaluating RAG applications treats them as traditional question answering systems. Researchers have proposed metrics such as SAS, a semantic answer similarity estimation (Risch et al., 2021), and F1/EM scores (Wang et al., 2024) to evaluate these applications on public question answering (QA) datasets. Zheng et al. (2023) propose using strong LLMs such as GPT-4 to evaluate other LLMs. They argue that traditional benchmarks cannot effectively align quality measurement with human preferences in open-ended tasks.

In some companies, RAG applications are mostly developed for a closed-domain, closed-ended setting, where employees seek factual answers based on proprietary knowledge. Meanwhile, they often develop external-facing RAG applications that allow clients to search for authoritative information that is not necessarily available on the Internet. Therefore, it is crucial to establish definitive answer quality evaluation practices that are suitable for these businesses’ needs.

To study this, we introduce a benchmark dataset called VND-Bench, comprising 155 high-quality questions across 14 subject areas in a proprietary payment network data domain. We also collect their corresponding answers from a communication channel on an internal employee collaboration platform to serve as ground truth labels. We build a vanilla RAG application, tRAG, as a test bed and ask it to answer the 155 questions. We then request five human experts, GPT-4, and Llama 3 (Meta, 2024) to assess tRAG’s answer quality based on the labels and vRAG-Eval, a set of grading instructions and a rubric. Our results show that GPT-4’s evaluation agrees with human evaluators’ scores at a rate of 83% based on a binary accept/reject category.

We summarize our contributions as follows:

• •

  We design a RAG evaluation mechanism, vRAG-Eval, which includes a set of grading instructions and a rubric that measures answer quality in three aspects: correctness, completeness, and honesty, resulting in one single score. It can be readily converted to a binary accept/reject decision that suits business settings.

• •

  We build a high-quality benchmark dataset, VND-Bench, with 155 questions/answers pairs that cover 14 subject areas in data domain.

• •

  We conduct experiments using LLMs as a RAG’s quality evaluators in a closed-domain closed-ended single-turn QA setting and find that GPT-4’s grading agrees with human experts’ opinions at a rate of 83% in terms of answer accepted or rejected decisions.

The rest of the paper is organized as follows: in section 2, we introduce related work. And in section 3, we explain the motivation, problem setting, and our method of experimentation. In section 4, we provide our experimental results. We suggest future research directions and make concluding remarks in sections 5&6.

Embedding and semantic similarity Zhang et al. (2020) propose the BERTScore metric as an automatic evaluation method for text generation tasks. Unlike traditional lexical approaches that rely on word matching, BERTScore sums the cosine similarities between the token embeddings of both two sentences: the reference answer and the QA system-generated answer. Risch et al. (2021) introduce SAS, a metric designed to estimate the semantic similarity between ground-truth annotations and answers generated by a QA model. It is found that semantic similarity metrics, especially those based on contemporary transformer models (Reimers and Gurevych, 2019), align more accurately with human assessments compared to the conventional lexical similarity metrics.

LLM as a judge Two projects closely related to our research are LLM-as-a-Judge (Zheng et al., 2023) and A Case Study on the Databricks Documentation Bot (Leng et al., 2023). To evaluate LLM-based chat assistants, Zheng et al. (2023) examine the usage and limitations of LLMs as judges for open-ended questions. This setting may not be suitable for most RAG applications in enterprises, where users typically seek definitive answers. Leng et al. (2023) employ LLMs to generate grades as a weighted composite score of Correctness, Comprehensiveness, and Readability. While they report an alignment rate of 80% with human scores on individual factors, their grading system lacks a crucial element: one single metric that provides a decision-making-ready quality score. In business-centric settings, this limitation becomes particularly noticeable.

Reference-free evaluation In scenarios where human annotations are not readily available, Es et al. (2023) present RAGAS, a framework for automating the assessment of RAG systems. This framework considers RAG’s 2-module composition and proposes three key metrics: Context Relevance to evaluate the Retriever, Answer Relevance to assess the Generator, and Faithfulness to measure the coherence between the two modules. Similarly, ARES (Saad-Falcon et al., 2024) evaluates along the same 3 dimensions to assess the quality of RAG components. While not entirely reference-free, ARES leverages a few hundred human annotations during evaluation and employs a lightweight fine-tuned LLM as a judge, rather than relying on a frozen LLM.

Prior research has shown that LLMs are powerful tools and become increasingly popular for evaluating RAG answer qualities. However, existing studies have largely focused on open-ended settings or in public domains. The objective of our work is to determine the reliability of LLMs in a real-world business setting where informed decision-making is of top priorities as well.

We ask GPT-4, Llama 3 8B, and human experts at our company to evaluate the quality of the aforementioned tRAG’s answers respectively and independently. Their assessment is guided by the vRAG-Eval instructions and grading rubric.

While LLMs continue to grow in size and intelligence, utilizing powerful LLMs as an automated alternative to human expert graders is a promising and prominent area of research. This approach complements traditional lexical and semantic measurements and demonstrates exceptional explainability and human preference alignment capabilities. We identify several areas for potential future research and practice directions:

Enterprise’s Values In designing the vRAG-Eval grading system, we incorporate honesty into the grading rubric to reflect that human evaluators prefer answers that admit uncertainty over hallucination. We also conduct preliminary experiments on the subject of "verbalized probability" to explore the potential of using self-expressed confidence levels as a factor in measuring a LLM evaluator’s own honesty. Additionally, we consider integrating more ethical aspects, such as inclusiveness and fairness, into the grading metrics to ensure that the system is both effective and responsible.

More Language Support The VND-Bench dataset currently only includes Q&A pairs in English, yet our enterprise is a global company that supports businesses worldwide. We consider extending the dataset to include other languages, reflecting the company’s value of inclusiveness and diversity.

Domain Adaptation We curate the VND-Bench dataset, focusing on transactional data domain knowledge. We consider building benchmark datasets in other areas and study vRAG-Eval’s applicability to explore its potential in those business priorities.

Few-shot Evaluation. Our experiments demonstrate that the GPT-4 evaluator exhibits high agreement rates in accept/reject decisions. We consider analyzing the score discrepancy at each of the five grading levels to better understand the relationship between human and LLM evaluators’ thinking process. By leveraging "few-shot learning", we hypothesize that we can improve the alignment of LLM evaluators with human preferences at a greater refined granularity.

More Question Categories. The VND-Bench dataset currently focuses on factual knowledge questions in the transactional data domain. We consider expanding to include math and logical reasoning questions, which are essential for evaluating business-centric RAG applications. By developing high-order grading capabilities, we can better support the company’s growth and innovation.

Llama 3 (70B). We contemplate the quantization of the 70B model into 4-bit integers and evaluate its grading agreement with human experts. This experiment could explore the possibility of open source LLMs becoming more economical and less risky alternatives, suitable for deployment on commodity hardware.

In this paper, we thoroughly study evaluating Retrieval-Augmented Generation (RAG) applications in a high-stakes business environment where answer quality is paramount.

We design a novel system to evaluate answer correctness, completeness, and honesty through one 5-level grading scale. Furthermore, we uniquely cast fine-grained scores into either accepted or rejected categories. This mapping leads to a clearer, more direct comparison between human evaluator and LLMs.

We observe a remarkably high agreement rate of 82.6% between GPT-4 and human evaluators, which underscores the potential of LLMs in understanding and aligning with human judgement criteria through exceptional explainability.

Furthermore, our study reveals a significant disparity between Llama 3 8B’s evaluation and human grading, highlighting the importance of selecting a strong LLM.

We thank Ranjan Dutta, Toni Wang, and Ajit Patil for their thoughtful discussions and assistance with data annotation. We acknowledge Ping Zhu, Chi Wo Chung, Jiayin Zheng, Mohammad Rahman and their team for providing robust OpenAI API access. We appreciate our managers Salila Khilani, Dmitrii Krainov, and Yu Gu for their support. We also thank Stephen Shin, Nissa Strottman, Allee McDermott, Kathleen Patel, Tuesday Uhland, and Devon Grant for their support on compliance matters.