Leveraging Graph Structures to Detect Hallucinations in Large Language Models

Consider a model $\phi(x)=e$, where $e\in\mathbb{R}^{768}$, which maps $x_{i}$ to its sentence-level embedding representation $e_{i}$. We construct a graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ as follows:

• •

  $\mathcal{V}$ is the set of nodes. Each node $v\in\mathcal{V}$ corresponds to a single data point $x_{i}$ from dataset $D$. More precisely, the features of a node consist of the sentence-level embedding $\phi(x_{i})$. We employ BERT Devlin et al. (2019) as the model $\phi$ to transform textual representations into embeddings.

• •

  Two utterances $\{u,v\}\in\mathcal{V}$ are connected with an edge $(u,v)\in\mathcal{E}$ if and only if the semantic similarity between nodes $u$ and $v$ exceeds a threshold $\tau$. The semantic similarity between two utterances is calculated using cosine similarity.

The choice of $\tau$ must ensure a balance in graph connectivity. Ideally, the node degree distribution should be relatively uniform, with a limited number of both highly connected and disconnected nodes. Additionally, we aim to avoid spikes in node degrees, as they may indicate the formation of hubs. The value of $\tau$ is dependent on the $D$.

We employ a GAT model Veličković et al. (2018) on our semantically-driven graph structure $\mathcal{G}$. Computing attention scores over sentence embeddings involves significant computational costs due to their high-dimensional representation. Consequently, we first reduce the dimensionality of node features by training a basic MLP. Then, we apply GAT on the reduced node features. The model will learn to map the sentence embeddings to a label indicating its degree of hallucination. We choose to model this problem with graph structures for two primary reasons: 1) to aggregate information via message passing, driven by our intuition that sentences that exhibit similar degrees of factuality share common structural components, and 2) to leverage edge weights, such that the level of similarity influences the information shared through message passing is expected to influence the message passing mechanism. We formalize the problem as an ordinal regression task as follows:

• •

  Label Encoding: If a data point $x_{i}$ has associated label $L$, then it is classified into all lower labels. Let $L$ be an ordinal label and $\text{encode}(L)$ be the corresponding encoding. Then, we can define $\text{encode}(L)$ as:

  $$\text{encode}(L)_{i}=\begin{cases}1&\text{if }i\leq L\\ 0&\text{otherwise}\end{cases}$$

  (1)

  where $\text{encode}(L)_{i}$ represents the $i$th element of the encoding vector.

By employing a graph-based model, we aim to validate our assumptions that hallucinations exhibit shared characteristics within the embedding space. We add connections between sentences that show a high degree of similarity and, during training, we exchange information between nodes and their local neighborhood.

Assume a new evaluation dataset $D^{\prime}$. We first append $D^{\prime}$ to the existing graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$, and then perform one forward pass to the trained GAT using the new graph $\mathcal{G^{\prime}}=(\mathcal{V}\cup\mathcal{V^{\prime}},\mathcal{E}\cup\mathcal{E^{\prime}}\cup\mathcal{A})$ to solve a label recovery task. $\mathcal{A}$ represents the edges formed between the nodes $V$ and $V^{\prime}$. We define the label recovery task as follows:

• •

  Label recovery: Consider a graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where $\mathcal{V}$ is the set of vertices and $\mathcal{E}$ the set of edges. The label recovery task involves inferring missing labels for a subset of nodes $\mathcal{U}\subseteq\mathcal{V}$ based on available information of the known labels of nodes $\mathcal{V}\setminus\mathcal{U}$.

We first apply our method to our own hallucination-generated dataset. Creating our own dataset allows for more control over the modeling choices and requirements of our methodology, such as degree of connectivity, homophily, and encoding techniques, ensuring a more targeted evaluation. However, we recognize the importance of generalizability, and therefore we also validate our approach on existing datasets to assess its performance across diverse contexts and benchmarks.

We used data from MSMARCO-QA Nguyen et al. (2016), selecting 2000 questions within the biomedical domain Xu et al. (2020) with answers consisting of a minimum of five words, to avoid sentences with explicit meaning only when paired with their corresponding question (examples are: "Yes", "No", "Non-surgical", or "Virus infection"). As mentioned in Section 3.4, each entry in the MSMARCO-QA dataset consists of the query, answer, and context. We use the queries and context to generate true w/ context, true w/o context, and hallucinated statements by prompting Meta’s instruction-tuned Llama2 Touvron et al. (2023) $13B$ model²²2https://huggingface.co/OpenAssistant/llama2-13b-orca-8k-3319. The dataset is randomly divided into three segments: training, validation, and test, with partitions of 70%, 15%, and 15%, respectively. The random division is done on the sentence level, therefore same query generations are not necessarily in the same data partition.

We use the English uncased version of BERT Devlin et al. (2019)³³3https://huggingface.co/bert-base-uncased for sentence embeddings. The graph is constructed over the entire dataset, with designated masks for the training, validation, and test sets. Edges are formed between nodes with cosine similarity above a threshold $\tau=0.85$, which was selected empirically to strike a balance, ensuring a reasonable level of graph connectivity. The resulting undirected graph has approximately $26M$ out of a potential of $240M$ edges (fully-connected). Additionally, we use the cosine similarity values as edge attributes.

The embeddings are reduced to a dimensionality of 32 features using a trained single-layer MLP. The GAT model incorporates a single graph attention convolutional layer, transitioning from 32 to 3 dimensions, with 2 attention heads. The GAT model explores and aggregates information from the immediate neighborhood based on the attention mechanism.

We employ a three-layer MLP with ReLU activations, using the same BERT model for sentence embeddings, but with concatenated query-answer as inputs. As a second baseline, we employ a larger pre-trained language model DeBERTa He et al. (2021)⁴⁴4https://huggingface.co/MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli, processing query-answer pairs in a unified encoder for improved contextual understanding and predictions. The MLP offers a simple yet powerful method for analyzing relationships between the generations and respective queries and has been shown to be efficient in sentence-level analysis Ramdhani et al. (2022); Akhtar et al. (2017). Meanwhile, DeBERTa is a complex transformer model that facilitates deeper comparisons of attention effects and structural differences between transformers and lightly connected graphs. Unlike our graph-based model, which relies solely on answer embeddings, the baselines utilize embeddings from both queries and answers, enhancing semantic expressiveness for better differentiation among classes.

The model selection process is based on optimal macro-recall performance on the validation split, offering a comprehensive evaluation of the model’s ability to identify instances across all classes, crucial in the context of highly imbalanced data. Training spans over $500$ epochs utilizing the Adam optimizer Kingma and Ba (2017) with fixed learning rate $1\times 10^{-3}$. The model is trained to solve a categorical regression task using Binary Cross Entropy (BCE) loss Good (1952). This choice reflects the need for nuanced penalization of misclassifications, where the model applies a lower penalty for misclassifying adjacent, compared to further-apart labels. Efforts are made to prevent data leakage between the data partitions. During training, weights corresponding to edges that connect nodes with either validation or test nodes are nullified, ensuring no information exchange (Equation 2). Backpropagation happens exclusively over the training nodes. Similarly, during validation, we modify edges connecting to test nodes (Equation 3). Recall is calculated exclusively on validation nodes. The full graph is used to assess performance on the test set (Equation 4).

We calculate three metrics to evaluate the performance of our approach. Macro-recall assesses the model’s accuracy in identifying individual classes, while macro-precision evaluates prediction accuracy per class. AUC-PR calculates the area under the precision-recall curve, providing a measure for binary classification performance. Additionally, these metrics are robust against class imbalance, making them suitable for evaluating our model on our imbalanced dataset.

In our study, we utilize two human-generated and annotated benchmarking datasets, namely FEVER Thorne et al. (2018a) and SelfCheckGPT Manakul et al. (2023). Our method can be generalized across other domains. To apply our method to another dataset, we re-construct the graph with the respective data partitions and redefine the labeling accordingly. These datasets are used for evaluating the performance of our models under conditions that mimic real-world scenarios. The specific methodologies employed for their use are discussed in Section 2.

The FEVER dataset Thorne et al. (2018a) contains $185445$ claims which are divided into three types of claims, namely Supports, Refutes, and Not Enough Info, each paired with evidence sentences. To apply our method to the FEVER dataset, we re-define the label as $y_{i}\in[0,1,2]$ and generate the graph based on the train/val/test partitions of FEVER. The participating models Thorne et al. (2018a) formulate careful data processing approaches and make use of external sources to verify the factuality of the claims. If search-based evidence is found for a claim, it is classified as Supports or Refutes. Similarly, if no evidence is found, it is labeled as Not Enough Info.

The SelfCheckGPT dataset Manakul et al. (2023) consists of 1908 sentences categorized as Accurate, Minor inaccurate, and Major inaccurate. To apply our method to SelfCheckGPT, we again redefine the labels as $y_{i}\in[0,1,2]$ and generate the graph by randomly splitting the data into train/val/test sets.

To address our first research question, more specifically 1. Do LLM-generated hallucinations share characteristics?, we analyze if our framework identifies an underlying structure of the embedding space. As shown in Table 1, GAT exhibits better performance compared to DeBERTa-QA and MLP-QA on all metrics. GAT has approximately 17% higher recall than both baseline models on the validation set. This suggests its superior ability to identify positive instances, reduce false positives, and effectively differentiate between true and hallucinated statements.

Initial experiments revealed that BERT embeddings are not discriminative enough for our task. This is intuitively to be expected: we hypothesize that hallucinations share features in the latent space. However, this does not imply that these features are inherently discriminative within BERT embeddings, as BERT is trained to capture contextual, syntactic, and semantic information, rather than “validity” or “truthfulness”.

To acquire enriched embeddings, we train a Contrastive Learning (CL) Khosla et al. (2020) MLP on the train set. This choice aims to strengthen the model’s ability to differentiate between classes. In CL, larger batch sizes often enhance performance by allowing more comparisons with negative examples, smoothing loss gradients. We found that a batch size of 256 suffices for good results. Extended training periods notably benefit CL. We train for 1000 epochs using a decoupled weight decay optimizer Loshchilov and Hutter (2019). Parameter group learning rates are set with a cosine annealing schedule Loshchilov and Hutter (2017). Our contrastive learned MLP (CL + MLP) consists of two linear layers: input size 768, sequentially transitioning to 768, and then to 128 with ReLU activation. After contrastive learning, the 32-dimensionality reduction MLP is applied.

To assess the impact of incorporating CL, we compare the metrics of GAT with and without CL, alongside the MLP baseline. We train the MLP with CL to differentiate between answers only, leading to a new baseline MLP-A. This MLP is a two-layer model with hidden sizes 64 and 32, and ReLU activation. This comparison is excluded for the DeBERTa model, as MLP-A is trained solely on answers, and DeBERTa uses different embeddings, potentially leading to a distribution shift.

To further address our first research question, we evaluate the model’s performance both with and without contrastive learning (CL). Table 2 reveals significant improvements in GAT’s performance with CL, particularly a remarkable 32% increase in recall on the train set. While the validation set also shows overall improvements, there is a slight 3% dip in precision, countered by an approximate 3% increase in recall. Without CL, MLP’s performance appears random. After using CL, there is an apparent improvement across all metrics. In particular, there is a 20% improvement in both precision and recall for the train set. Validation recall sees an approximate 10% increase, while precision increases by around 20%.

Despite CL significantly improving the results, the ablation study reveals it is not the only factor in improving performance. To answer our next research question 2. Can we leverage graph structures to identify and learn these characteristics?, we employ $k$-Nearest Neighbour (kNN) with CL-learned embeddings. We assess the independent expressiveness of these embeddings, anticipating that sufficiently robust embeddings would enable a reliable majority-voting mechanism. However, with $k=5$, kNN shows consistent underperformance (detailed in Table 2). Further exploration involved training the same MLP as introduced in Section 5.2, which showed improved performance compared to MLP without CL-learned embeddings. However, the MLP still trailed the performance of GAT, with approximately a 20% decrease in validation recall, highlighting the significance of the graph structure. The attention mechanism of GAT is crucial in accurately identifying important neighbors. This refined approach which is solely reliant on spatial similarity outperformed the kNN method, highlighting the advantages of graph structures for efficient information propagation. Furthermore, edge masking ensures robustness by preventing information exchange between training and validation/test nodes during training. This method acts as a regularizer, enhancing the model’s generalization capabilities Rong et al. (2020).

Finally, to address the research question 3. If learned, can we use this knowledge to identify hallucinations among new incoming LLM generations through label recovery?, we analyze the best-performing models by validation recall. The models that showcase the highest performance are GAT with and without contrastive learning. To ensure a fair comparison, we also consider the performance of the third-best model on the test set. The results are shown in Table 3. Performance on the test set reveals that GAT with CL outperforms the other models on every metric except precision. The GAT structure proves crucial for higher recall.

We assess the generalizability of our method on two real-world datasets, namely FEVER Thorne et al. (2018a) and SelfCheckGPT Manakul et al. (2023). Section 2 discusses their original applications, while Section 4.7 details how we modify the labels for our model.

To benchmark our model’s performance, we compare the results against the best performance of the first FEVER Shared Task challenge Thorne et al. (2018b), shown in Table 4 as UNC-NLP. Our model outperforms UNC-NLP in precision, with accuracy being $4\%$ lower. However, it is important to stress that, in comparison, we solve a closed-book problem, avoiding the computational overload and necessity of any external data or search-based model.

Following the methodology in SelfCheckGPT Manakul et al. (2023), we use DeBERTa-large for sentence embeddings. Our model falls short of the pairwise consistency metrics computed using DeBERTa-large embeddings (with BERTScore), as demonstrated in Table 5. A plausible explanation is the dataset’s small size. The method with BERTScore needs multiple LLM-generated statements, while our method, trained on a small set, requires more examples for effective learning.