Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models

RelQA consists of nine sub-datasets: SQuAD (Rajpurkar et al., 2016), DuReader (He et al., 2017), HotpotQA (Yang et al., 2018), MSMARCO (Nguyen et al., 2016), NewsQA (Trischler et al., 2016), QuAC (Choi et al., 2018), CoQA (Reddy et al., 2019), TriviaQA-Web (Joshi et al., 2017), and TriviaQA-Wikipedia (Joshi et al., 2017). The detailed collecting steps are as follows:

Step 1 (Dataset Selection): These datasets are selected due to their unique characteristics, diverse sources, and the enrichment they bring to the overall collection. They cover extractive reading comprehension (ERC), multiple-choice (MC), and multi-turn dialogues (MTD) categories. They originate from sources such as Wikipedia, Baidu Zhidao, Bing search, and other platforms, while encompassing domains such as student education, news, web articles, and general knowledge.

Step 2 (Formatting and Integration): To ensure compatibility and remove dataset boundaries, we perform formatting and integration for all selected datasets based on the aforementioned categories. Each dataset follows a specific standardized format, as illustrated in Table 1 (the second column). We represent the datasets of all categories as $\{L_{i},D_{i}\}$, where $L_{i}$ denotes a specific dataset and $D_{i}$ denotes its standardized format.

Step 3 (Preprocessing): To facilitate effective processing and generation of answers, we employ preprocessing techniques on the dataset. This involves two primary aspects: personalized prompt instruction design and addressing the limitations associated with long texts. For personalized prompt instruction design, we create question-adaptive prompt instructions for each question based on the question type, as shown in Table 1 (the third column). These prompt instructions guide LLMs in generating better answers that align with different types of questions. To address the challenge of long texts, we implement a sliding window approach (Koay et al., 2021), segmenting the texts into smaller windows, each containing 4,000 tokens. This ensures that LLMs receive clear prompt instructions and can effectively handle texts of varying lengths, resulting in more accurate and contextually appropriate answers.

Step 4 (Answer Generation): We employ several powerful LLMs, including LLaMA (Touvron et al., 2023), BLOOM (Scao et al., 2022), GPT-J (Wang and Komatsuzaki, 2021), GPT-3 (Brown et al., 2020), and GPT-3.5 ¹¹1https://chat.openai.com/, to generate answers for evaluation. In the case of longer texts, we slide the window over the text and generate outputs for each window. The generated outputs for each window are stored to facilitate subsequent filtering and selection of the optimal answers. To maintain answer stability, we ask an LLM to generate the answer three times for each question and select the majority answer as the final answer. Furthermore, to ensure the overall quality and reliability of the generated answers, we conduct quality assurance procedures, including automated checks to identify and re-generate incomplete sentences by detecting missing sentence-ending punctuation, among others.

To evaluate the reliability of LLMs’ generated answers, it is crucial to select appropriate metrics that capture different aspects of answer quality. We employ four types of metrics, including LLM-assessment metric, human metric, machine metric, and composite metric, to comprehensively evaluate the generated answers.

LLM-assessment metric is inspired by the concept of LLMs’ self-evaluation, where LLMs occasionally demonstrate the ability to assess their own output correctly without human intervention (Chiang and Lee, 2023; Yan and Xu, 2023). This metric comprises two specific indicators: the goodness of a generated answer and the similarity between the generated answer and the ground-truth answer. By obtaining the goodness score and similarity score of a generated answer, we can evaluate its quality and how closely it aligns with the ground-truth answer. Higher scores indicate better quality and semantic alignment. The LLM-assessment metric provides valuable insights into the LLMs’ ability to evaluate the quality of generated answers.

Human metric plays a significant role in evaluating the LLM’s performance from a human perspective. It includes a human score, which is a binary label assigned to each answer based on the degree of match between the LLM’s generated answer and the ground-truth answer, along with the assigned goodness score. The human metric labeling is as follows: i) When the LLM’s generated answer is the same as the ground-truth answer and receives a goodness score of 4 or 5, the human metric is labeled as 1. This indicates that the LLM has successfully generated a correct and high-quality answer that aligns with the expected answer. ii) When the LLM’s generated answer is different from the ground-truth answer and receives a goodness score of 1, 2, or 3, the human metric is labeled as 2. This suggests that the LLM’s generated answer is incorrect or of lower quality compared to the ground-truth answer. iii) For cases where the LLM’s generated answer neither matches the ground-truth answer nor falls within the aforementioned goodness score ranges, the human metric is labeled as 0. This label represents a neutral or ambiguous classification, indicating that the answer may require further examination or subjective judgment. The human metric captures the human perception of the LLM’s performance.

Machine metric draws inspiration from question-answering and dialogue systems, which rely on objective metrics to assess the quality of generated answers. It encompasses various categories, including accuracy metrics, overlap metrics, similarity metrics, and diversity metrics. Examples of machine metrics include F1 score, Recall, BLEU (Papineni et al., 2002), BERT score (Zhang et al., 2019), ROUGE (ROUGE-1, ROUGE-2, ROUGE-L) (Lin, 2004), Distinct-N (Distinct-1, Distinct-2) (Li et al., 2015), Greedy matching, and Embedding scores (average, extreme) (Liu et al., 2016). Specifically, accuracy metrics assess the correctness of generated answers compared to the ground truth, including F1 score. Overlap metrics measure the overlap between generated answers and the ground truth, including BLEU, Recall, ROUGE. Similarity metrics capture the semantic similarity between generated answers and the ground truth, including BERT score, Greedy matching and Embedding scores (average, extreme). Diversity metrics measure the diversity of the generated answers, including Distinct-N. These metrics objectively evaluate the semantic alignment, relevance, diversity, and quality of generated answers, enabling a comprehensive assessment of LLMs’ answers.

Composite metric is designed to provide a comprehensive evaluation of a model’s performance by combining multiple aspects. It includes a final score and a final tag to summarize the evaluation. Each of the metrics mentioned above contributes to the final score, with specific emphasis given to certain metrics. For instance, Recall and ROUGE (ROUGE-1, ROUGE-2, ROUGE-L) may be assigned higher weights (e.g., twice the weight) to highlight the importance of maintaining information (Lyu et al., 2022, 2023). The weights of different metrics can be dynamically optimized to better assess their importance in real-world scenarios as demonstrated in Experiment 7. The final tag is a binary label assigned based on the average score. If the average score is greater than 0.5, it is labeled as 1; otherwise, it is labeled as 0. The final tag simplifies the evaluation outcome, indicating whether the LLMs’ generated answer is considered reliable or not. In summary, these metrics collectively evaluate the quality of answers generated by LLMs compared to the ground-truth answers.

In this section, we conduct a data exploratory analysis of the constructed RelQA dataset, which comprises a total of 1,372,130 samples, including generated answers by five selected LLMs. Among these, 743,910 samples are assigned as reliable and 628,220 samples as unreliable based on the final tag metric. We divide the possible ranges of all metrics into three equal parts, representing low, medium, and high levels. Fig 2 illustrates the distribution of each dataset at the high level for each metric. We also present the distributions of different datasets among various metrics as shown in Table 2, Table 3, Table 4 and Table 5.

First, we analyze the differences in the LLM-assessment metric across different datasets. Regarding the “goodness” metric, the QUAC dataset performs poorly in terms of answer quality, with a high score percentage of 82.72%, while the SQuAD dataset excels in generating high-quality answers, with a high score percentage of 99.47%. Other datasets generally achieve high score percentages above 90%. Regarding the “similarity” metric, the MSMARCO dataset demonstrates the highest similarity to the reference answers, with a high similarity percentage of 74.89%. Conversely, the QUAC dataset also performs poorly in terms of similarity, with a low similarity percentage of 60.28%.

Next, we analyze the differences in the human metric across different datasets. The proportions of reliable evaluations vary significantly in the “human score” metric. The lowest proportion is 0.42% for DuReader-master, while the highest is 32.79% for SQuAD. Similarly, the proportions of unreliable evaluations differ, with the lowest being 0.49% for SQuAD and the highest being 17.16% for QUAC. Additionally, the proportion of ambiguous evaluations is highest for newsQA at 96.38% and lowest for QUAC at 66.71%.

Afterwards, we analyze the differences in the machine metric across different datasets. In terms of “accuracy metrics”, the QUAC dataset performs the worst, with a high score percentage of only 4.54%. The high score percentages for other datasets range between 4.54% and 30.8%, with a median around 20%. In terms of “overlap metrics”, the QUAC dataset also performs poorly in terms of low overlap, with a low score percentage of 87.52%. The low score percentages for other datasets range from 32.47% to 75.28%, with no significant high scores observed overall. Regarding “similarity metrics”, DuReader, SQuAD, and MSMARCO perform well in terms of high similarity scores, with the highest scores being 95.89%, 94.71%, and 93.41% respectively. In contrast, newsQA and QUAC exhibit lower similarity scores, with the highest scores being 66.6% and 64.13% respectively. Notably, there are consistencies between the similarity scores in machine metrics and the similarity scores in LLM-assessment metrics. In “diversity metrics”, QUAC, newsQA, and MSMARCO perform well in terms of high diversity scores, with the highest scores being 97.77%, 96.83%, and 94.97% respectively. This is likely due to the higher question diversity in these datasets, allowing models to exhibit more creativity and diversity in generating answers. Other datasets also maintain high diversity scores, all above 80%.

Finally, we analyze the differences in composite evaluation metrics across different datasets. In terms of the “final score” metric, the QUAC dataset performs the worst, with a high composite score percentage of 6.88%. Conversely, the SQuAD dataset achieves the highest composite score, with a high percentage of 52.43%. It is evident that none of the datasets achieve particularly high composite scores. In terms of the “final tag” metric, the SQuAD dataset exhibits the highest proportion indicating answer reliablity, at 78.57%, while the QUAC dataset has the lowest proportion at 16.44%. This aligns with the human metric, as the SQuAD dataset primarily consists of simple extractive reading comprehension, making it easier for models to generate reliable answers. On the other hand, QUAC involves open-domain dialogue with more complex semantic understanding, posing challenges for models to generate reliable answers.

Initially, we employ a regression approach to train the discriminator RelD in order to fit the final score and align with human evaluation. However, our experiments reveal that the regression approach performs poorly, possibly due to the use of the mean square error loss function. Consequently, we convert the regression task into a classification task to improve the fitting. Specifically, In this process, we normalize the final score into different numbers of classes, such as four, six, eight, and ten, for multi-class classification. For instance, we assign the first category in a four-category classification to final scores ranging from 0 to 0.25. After experiments as shown in Sec. 6, we ultimately choose a ten-class classification approach. The theoretical foundation of this method mainly lies in information theory and the cross-entropy loss function. Cross-entropy is a common information theory measure used to quantify the distance between two probability distributions. In the case of multi-classification problems, the cross-entropy loss function is defined as follows:

where $y_{i}$ represents the true label of the $i$-th category, and $p_{i}$ represents the predicted probability of the $i$-th category by the discriminator RelD. Our objective is to minimize this loss function during the training of RelD. In practice, we employ the softmax function to convert the original output of RelD into a probability distribution.

One potential advantage of this method is that the classification task, which focuses on distinguishing different categories, may facilitate capturing subtle differences among the final scores. Furthermore, the cross-entropy loss function exhibits greater stability compared to the mean square error loss function when dealing with imbalanced datasets. However, it is important to note that in certain situations, multi-class tasks may introduce overly complex information, leading to a notable disparity between the concepts learned by the discriminator and human intuitive perception. For example, dividing a problem into five categories, such as “not reliable”, “weakly reliable”, “moderately reliable”, “strongly reliable” and “highly reliable”, may surpass most people’s intuitive understanding of the fundamental categories of “reliable” and “unreliable”.

Based on the aforementioned analysis, we further convert the multi-class task into a binary classification task, which may better align with human intuitive perception. Here, we present three possible approaches for this conversion, each with its theoretical support and definition:

Normalization. This method is based on threshold decision theory. It involves converting all class information into binary labels by directly normalizing the final score to 0 and 1, which serves as the final probability value for classification. However, this approach may result in some information loss as continuous scores are transformed into discrete classes.

Discrete Values. This method is grounded in maximum likelihood estimation, a commonly used parameter estimation technique in statistics. Here, we consider the highest predicted probability from the discriminator as the final probability value for classification. For example, in a four-class classification scenario, if the probabilities corresponding to the classes are 0.1, 0.1, 0.1, and 0.7, respectively, we would use 0.7 as the final probability value. The advantage of this method lies in its simplicity, although the drawback is that we do not know which class the maximum probability value corresponds to.

Weighted Average Probability. The theoretical basis for this method stems from decision theory, particularly the concept of expected utility, which involves taking a weighted average of all possible outcomes and their corresponding utilities (in this case, predicted probabilities). The goal of this approach is to determine a weighted average value that best represents the predicted probabilities for each class from the discriminator. In this method, we multiply the probability of each class predicted by the discriminator with its corresponding weight, summing them up to obtain a final probability value. This value can then be used for binary classification tasks. The formula for this method is as follows:

where $p_{i}$ represents the probability output of the discriminator for class $i$, $w_{i}$ denotes the weight for class $i$, and $w_{\text{min}}$ and $w_{\text{max}}$ are the minimum and maximum weights, respectively. We set the threshold to 0.5 and use the cross-entropy loss function for approximation. It allows for a more refined fitting of regression tasks and has demonstrated better performance compared to the previous two methods, as indicated by Sec. 4.3.

We utilize a Pre-trained Language Model (PLM), such as ELECTRA (Clark et al., 2020), as the backbone of the discriminator RelD. Through our experiments, we have demonstrated that ELECTRA outperforms other PLMs, including BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), and DeBERTa (He et al., 2020), as indicated in Section 8. RelD takes questions along with contexts and LLMs’ generated answers as input, generating a classification label to determine the reliability of a generated answer. It uses the weighted average probability approach to fit the ground-truth answers.

The experiments are conducted using TESLA A100 GPUs for answer generation and GTX 3090 GPUs for training RelD with PyTorch in Python. During the training of RelD, we set the batch size to 32 and the sequence length to 128. Hyperparameters such as weight decay (0.01), $\beta_{1}$ (0.9), and $\beta_{2}$ (0.999) are maintained. The learning rate is set to 2e-05. We train RelD for 20 epochs.

Baselines and metrics. We validate the effectiveness of the proposed RelD on well-known LLMs, including LLaMA (LLaMA-7B)(Touvron et al., 2023), BLOOM (BLOOM-7B)(Scao et al., 2022), GPT-J (GPT-J-6B)(Wang and Komatsuzaki, 2021), GPT-3(Brown et al., 2020), and GPT-3.5 ${}^{\ref{GPT-3.5}}$. To evaluate the performance of RelD, we use accuracy (ACC) as the automatic metrics and ROC curve analysis with the area under the ROC curve (AUC) as the human-in-the-loop metrics. The automatic evaluation process utilizes the final tag as the ground-truth label, while the human-in-the-loop evaluation involves human ratings as the ground-truth labels. Specifically, we randomly select 9,000 QA pairs, with 1,000 from each dataset in RelQA, for human ratings. We enroll nine volunteers and divide them into three groups to ensure evaluation stability. Each group provides scores of 0 or 1 for the randomly selected 3,000 QA pairs. Inter-rater agreement is calculated using Krippendorff’s Alpha (IRA) to ensure the confidence of the human ratings. For controversial ratings with low agreement ($<$0.7), we discard the corresponding QA pair and replace it with another.

We conduct experiments to evaluate the effectiveness of the proposed RelD as follows:

Experiment 1: RelD’s Performance across Different LLMs. We conduct ten-fold cross-validation and report the average performance on the validation dataset. Based on the results presented in Table 6, it’s observed that both the automatic and human-in-the-loop evaluations consistently exceed 0.8 for all LLMs, with minimal variation between different models (p<0.01). The strong correlation between the automatic and human-in-the-loop evaluations (p<0.01) suggests that the automatic scoring of the RelQA dataset could largely replace human scoring. It also indicates the robustness of RelD in detecting the reliability of different LLMs.

Experiment 2: RelD’s Performance on IID and OOD Datasets We evaluate the performance of RelD on both In-distribution (IID) and Out-of-distribution (OOD) datasets. We randomly assign nine datasets from RelQA to the IID and OOD sets in various ratios, such as 1:8, 2:7, 3:6, and 4:5, and vice versa. For example, we train on 8 datasets and validate on 1 dataset. To ensure a balanced quantity of data in both the IID and OOD sets, we perform downsampling by randomly selecting 3,000 samples from each dataset. Considering that different datasets acting as IID or OOD may yield different results, we conduct five experiments for each ratio group and provide average values along with the range of error. This approach allows us to accurately assess the generalization ability of RelD. To evaluate the performance on the IID dataset, we use 30% of the IID data as a validation dataset. For the OOD evaluation, we directly test RelD on the entire OOD dataset. The results are presented in Table 7 and Fig. 5. We observe that when the IID ratio is set to 5 or higher, RelD consistently achieves automatic and human-in-the-loop evaluations above 0.7 on both the IID and OOD datasets. This indicates that RelD exhibits a strong generalization capability in handling OOD data as well as alignment with human evaluation predictions.

After that, we conduct several experiments to evaluate the effectiveness of different modules in the proposed RelD. All results are performed on the validation dataset using ten-fold cross-validation.

Experiment 3: Effectiveness of Weighted Average Probability. We compare the performance of using normalization, discrete values, and weighted average probability in the conversion from multi-class to binary-class classification in both automatic and human-in-the-loop metrics. The results are presented in Fig. 6. We observe that while using weighted average probability slightly underperforms normalization in terms of automatic metrics, it significantly outperforms normalization and discrete values in human-in-the-loop metrics across all LLMs. Therefore, we adopt weighted average probability as it offers a more intuitive and aligned approach from a human perspective.

Experiment 4: Optimal Number of Categories. We investigate the impact of the number of categories when converting regression into multi-class classification. We test four categories, six categories, eight categories, and ten categories. The results are shown in Fig. 7. It is evident that a higher number of categories leads to improved performance in human-in-the-loop metrics. This suggests that a larger number of categories brings the classification task closer to regression and enhances alignment with human cognition. Consequently, we ultimately convert the regression task into a ten-category classification task and then discern it as a binary classification using weighted average probability.

Experiment 5: Optimizing Weights of Each Metric. Relying solely on prior knowledge to determine the weights of each metric may not achieve the best performance. Therefore, we explore the optimal weights for each metric. To achieve this, we calculate the optimal weight for each metric as the weighted average of two values: the AUC when each metric is treated as the ground-truth compared to human evaluation, and the Pearson coefficient between each metric and human evaluation. In our experiment, we set the ratio for the former as 0.9 and for the latter as 0.1, as it yields the best performance. The optimal weights of each metric are depicted in Fig. 8(a). Subsequently, we evaluate whether the optimal weights can enhance the performance of RelD in detecting hallucination of LLMs’ generated answers as shown in Fig. 8(b)(c). Remarkably, we observe improvements in both automatic (b) and human-in-the-loop metrics (c) after optimizing the weights of each metric.

Experiment 6: Backbone Selection for RelD. We experiment with different PLMs, including BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), DeBERTa (He et al., 2020), and ELECTRA (Clark et al., 2020), for RelD in order to choose the most effective backbone, as shown in Table 8. Through this comparison, we observe that ELECTRA achieves the best performance in both automatic and human-in-the-loop metrics. Consequently, we select ELECTRA as the preferred backbone for RelD.

We classify the predictions generated by RelD into four categories, as presented in Table 9. To gain insights into the characteristics of these categories and understand the functioning of RelD, we conduct an exploratory analysis.

Analysis 1: Distribution Analysis To analyze the distributions within each category, we utilize boxplots (Fig.9(a)) to illustrate key statistics such as median, quartiles, and outliers of samples. Additionally, we employ density plots (Fig.9(b)) to visualize the probability distribution of samples within each category. In the first category, the boxplot exhibits a wide range and the density plot shows a concentrated distribution with multiple peaks. This suggests that RelD may have some uncertainties in its predictions for this category. For the second and third categories, the boxplot widths fall between those of the first and fourth categories and the density plots display more dispersed probability distributions. This indicates that RelD is more hesitant in its predictions or has lower proficiency in learning for these types of questions. In contrast, the fourth category exhibits a narrower boxplot and the density plot shows a concentrated probability distribution. It indicates that RelD is more confident in its predictions for this category.

Analysis 2: Clustering Analysis. By applying clustering algorithms to the text data, we investigate whether each category exhibits distinct cluster centers, as illustrated in Fig. 10. For the first category, the data distribution appears clustered and relatively uniform, indicating consistent and accurate performance by RelD within this category. The second category contains an extremely small number of samples, suggesting that RelD rarely misclassifies the correct answers generated by the LLMs. In the third category, the clustering results reveal significant variability, indicating that errors can occur in various aspects when RelD misclassifies the incorrect answer as correct, such as grammar or comprehension errors. Similarly, the fourth category displays a wide and dispersed clustering distribution, indicating diverse performance by RelD within this category. This suggests the presence of different types of errors that make it challenging for RelD to detect. From the clustering graph, we observe that RelD performs best in the first category. However, for the second, third, and fourth categories, the performance of RelD may be influenced by the complexity and ambiguity of the input contexts or questions.

Analysis 3: Vocabulary Distribution. We can compare the vocabulary distribution between correctly predicted samples and incorrectly predicted samples by RelD, as depicted in Fig. 11. There is a noticeable distinction between the left side (RelD predicts correctly) and the right side (RelD predicts incorrectly). It appears that content related to “story” is relatively easy for RelD to classify correctly, while content related to “country” poses more difficulty for RelD in accurate classification. However, it is important to note that vocabulary alone may not be the sole determining factor for RelD’s recognition accuracy. The critical factors might involve underlying semantic relationships, which would necessitate further research and investigation.