Evaluation of RAG Metrics for Question Answering in the Telecom Domain

The Research Questions (RQs) considered in this work are:

• •

  RQ1: How do LLM-based evaluation metrics, specifically RAGAS, go through the step-by-step evaluation procedure specified by the prompts?

• •

  RQ2: Are RAGAS metrics appropriate for evaluation of telecom QA task using RAG?

• •

  RQ3: Are RAGAS metrics affected by retriever performance, domain adapted embeddings and instruction tuned LLM.

The contributions of our work are as follows:

1. 1.

   We have enhanced the RAGAS public repository code by capturing the intermediate outputs of all the prompts used to compute RAGAS metrics. This provides better visibility of inner workings of the metrics and possibility to modify the prompts.

2. 2.

   We manually evaluate, for RQ1, the intermediate outputs of the considered metrics with respect to the context and ground truth. We critically analyse them for their appropriateness for RAG using telecom domain data.

3. 3.

   We establish, for RQ2, that two of the metrics - Factual Correctness and Faithfulness, are good indicators of correctness of the RAG response with respect to expert evaluation. We demonstrate that use of these two metrics together is better at identifying correctness of response; this improves further on using domain adapted LLMs.

4. 4.

   We establish, for RQ3, that Factual Correctness metric improves with instruction fine tuning of generator LLMs, irrespective of retrieved context. We observe lower Faithfulness metric for RAG answers which are identified to be correct but from wrong retrieved context. This indicates that the generator (LLM) has answered from out of context information. The ability to answer from out-of-context information is more pronounced for domain adapted generator. Thus, the metrics are able to reflect the expected negative correlation between faithfulness and factual correctness for wrong retrieval - although ideally the RAG system should have not provided an answer for a wrong retrieved context.

All experiments in this work are based on subset of TeleQuAD (Holm, 2021), a telecom domain QA dataset derived from 3GPP Release 15 documents (3GPP, 2019). Our experimental setup is shown in Figure 1. The input to our pipeline is the QA dataset, which has contexts (from the 3GPP documents) along with associated questions and (ground truth) answers. These questions have been prepared by Subject Matter Experts (SMEs). The training and test data considered comprises of 5,167 and 715 QA, respectively, derived from 452 contexts (sections) from 14 3GPP documents. Appendix B shows a sample set of QAs along with the contexts.

The RAG pipeline is comprised of a retriever followed by generator module. The retriever module is comprised of the following steps. Data from reference documents are chunked. An encoder-based language model computes embeddings for query and sentences from the reference documents. For every question embedding, retriever outputs top-$k$ most similar sentence/context embeddings. Cosine similarity is used for selecting top-$k$ sentences/contexts.

We evaluate multiple models in our experiments. From the BAAI family of embedding models, we consider bge-large-en (Xiao et al., 2023) and llm-embedder (Zhang et al., 2023), both with an embedding dimension of $1024$. These models have been trained on publicly available datasets; hence, the embeddings may not be optimal for telecom domain. To address this, we also evaluate pre-trained and fine-tuned variants of these models using telecom data. We use sentences from the corpus of technical documents from telecom domain to pre-train (Li et al., 2020) the base model; we refer to this process as PT in subsequent sections. For fine-tuning (Mosbach et al., 2020), we prepare triplets of the form $<q,p,n>$ where $q$ corresponds to the user query, $p$ represents the correct (positive) answer and $n$ is a list of incorrect (negative) answers. The base model is fine-tuned using these triplets; this process is referred to as FT in subsequent experiments. It may be noted here that the fine-tuning may be performed independently on the base model (without pre-training) or post pre-training; in the latter case, we refer to it as PT-FT to report results.

The output of the retriever forms the input context to the generator. For evaluation, we only use $k=1$ retrieved context, to study the behaviour of the generator when presented with correct and wrong contexts. Once the relevant context has been retrieved for a question from the retriever module, the query and context are passed to the LLM for generating the response, indicated as “RAG Response” in Figure 1. We have considered Mistral-7b (Jiang et al., 2023) and GPT3.5 as the LLMs for our experiments. We also report results on pre-trained (PT) and instruction fine-tuned (PT-IFT) variants of Mistral-7b using mistral-finetune (MistralAI, 2024).

We discuss our findings about the four RAGAS metrics.

• •

  Faithfulness ($FaiFul$) - intends to provide reliability scores with respect to human evaluation. Simple statements might be paraphrased into multiple sentences, while complex statements may not be fully broken down. These factors can introduce variation in the faithfulness score. Despite these challenges, we found that the $FaiFul$ metric is generally concordant to manual evaluation.

• •

  Context Relevance ($ConRel$) - is mainly indicative and dependent on the context length. Typically, chunks such as sections form the retrieved context in a RAG pipeline; hence, context can vary in length, which affects the denominator component of $ConRel$. Typical metrics are either “higher is better” (e.g., accuracy) or “lower is better” (e.g., mean squared error). Also, all statements are assigned equal weight, regardless of the length or quality of the sentence. Therefore, we infer $ConRel$ cannot be appropriately clubbed into either of these types and the final metric is hard to interpret or even have an intuition about.

• •

  Answer Relevance ($AnsRel$) - The generated questions from the LLM in this metric may not be the best way to measure answer relevance. We have observed some cases where the generated questions are either trivial paraphrasing or incorrect. However, the major problem with this metric is that it tries to use cosine similarity as an absolute component of this metric. This makes the metric dependent on the choice of LLM. In addition, various studies on cosine similarity have pointed out that it may not be indicative of similarity (Steck et al., 2024), known to give artificially low values (Connor, 2016), is difficult to provide thresholds on (Zhu et al., 2010) and is subject to the isotropy of embeddings (Timkey & Van Schijndel, 2021). For many embeddings, similarity can be very high even between random words/sentences (Ethayarajh, 2019). All this points to the fact that using cosine similarity as a metric, like is done for $AnsRel$, is not very interpretable.

• •

  Answer Correctness ($AnsCor$) - The FC component of this score is dependent on the LLM correctly identifying the True Positives (TP), False Positives (FP), and False Negatives (FN). Our analysis shows that this process can sometimes result in incorrect mapping of sentences to these groups. Occasionally, irrelevant statements might be included, or relevant sentences might not be classified into any of the groups. Additionally, the semantic correctness component of this score being a cosine similarity is subject to the concerns raised on the $AnsRel$ metric. Despite this, our manual analysis shows that the answer correctness score is relatively well aligned with a human expert evaluation.

In summary, our results indicate that of these metrics, $FaiFul$ and $AnsCor$ are perhaps best aligned with human expert judgment; scores for $AnsSim$, $AnsRel$ and $ConRel$ are subject to inherent variations and are relatively unreliable for interpretation. Hence, we report results in more detail for only $FaiFul$ and $FacCor$ (i.e., $AnsCor=FacCor$ with weight for $AnsSim$ set to $0$) in Table 3. For correct retrieval, both $FaiFul$ and $FacCor$ are as good or better ($p<0.05$) with domain adaptation as expected. For wrong retrieval, an improvement in $FaiFul$ should necessarily lead to a reduction in $FacCor$ and vice-versa. We observe that the generator LLM is answering questions from it’s enriched domain adapted knowledge, leading to a lower $FaiFul$ and higher $FacCor$. Although not desirable from the expected generator response, the metrics correctly captures this.

Further, the RAG responses are evaluated for correctness by SMEs, all responses from each of Mistral-7b, Mistral-7b-IFT and Mistral-7b-PT-IFT. Considering this evaluation as ground truth for correctness, we evaluate the probability of the answer being correct based on the RAGAS metrics.

For each of $FaiFul$ and $FacCor$ metrics, we compute the probability of correct generated answer considering both the metrics $(m1,m2)\in\{FaiFul,FacCor\}$ being above a certain threshold, using Equations (1) - (2).

It is possible to use the Bayesian formulae with only one metric considered instead of the joint conditional distribution. Table 4 shows the results for $\theta_{11}=\theta_{12}=0.7$ and $\theta_{21}=\theta_{22}=0.3$ considering each metric independently and both jointly. We observe better concordance of RAGAS metrics with that of SME evaluation, if both the metrics are considered together. We also observe that domain adaptation of the LLM via IFT and PT-IFT improves the scores significantly ($p<0.05$). However, there is little difference ($p>0.05$) between IFT alone and PT-IFT. We conclude that these two metrics are well aligned with human judgement and can be used in an end-to-end pipeline reliably.

Sample outputs of RAGAS metrics for some questions and the corresponding metrics are shown in Appendix B.