Knowledge-based Consistency Testing of Large Language Models

Key Insight: The key insight behind KonTest is to leverage a knowledge graph for systematic consistency testing of LLMs. The knowledge graph serves multiple purposes in KonTest: Firstly, entities and entity relations extracted from the knowledge graph allows KonTest to systematically generate queries and construct test cases for validating the consistency of the subject LLM (SUT). The LLM’s responses to these test cases allow KonTest to build the knowledge (sub)-graph where the LLM behaves correctly. Secondly, knowledge-based test generation allows KonTest to report a test adequacy metric for the SUT in terms of the entities and relations covered within the knowledge graph. Finally, this approach enables KonTest to highlight sub-graph of the knowledge graph where the outputs from the SUT are logically inconsistent. This enables practitioners to selectively focus on the knowledge gaps highlighted by KonTest for improving the LLM (e.g., by fine-tuning).

Running Example: Figure 2 outlines our approach. KonTest broadly comprises of three key components: ① knowledge graph construction (“Knowledge Graph” in Figure 2), ② test generation (“Test Generation” in Figure 2), and ③ test oracles (“Test Oracles” in Figure 2). Given a set of entities (e.g., countries) and a maximum graph depth, KonTest locates the given entities in the knowledge graph (e.g., Wikidata knowledge graph Vrandečić and Krötzsch (2014)), and constructs the associated knowledge graph before extracting the relevant entities and relationships. Given the relationships present in Figure 1(d), KonTest constructs two semantically equivalent queries for each relation with the aid of a template (see Figure 3) relating the two involved entities. The resultant queries are then fed to the SUT generating responses to both atomic and sequential queries. Figure 1(a) shows two such semantically equivalent queries, each of which was part of a new conversation with GPT3.5 (aka “atomic”), while Figure 1(b) illustrates one such consistency error with inconsistent responses for semantically-equivalent queries that were within the same conversation (aka “sequential-inter”). Finally, Figure 1(c) illustrates a series of queries, whose responses from GPT3.5 collectively show inconsistent behavior (aka “ontological”). Specifically, the positive responses to the first two queries imply that Ireland does have a Kinawley. However, due to the negative response to the third query, our ontological oracle reveals a consistency error. We further note that such errors cannot be uncovered by counter-questioning LLMs. The relevant sub-graph of the knowledge graph for GPT3.5 is shown in Figure 1(e). This clearly illustrates the knowledge gap (i.e., Kinawley$\nrightarrow$Irland) for GPT3.5.

Knowledge Graph: We rely on Wikidata’s knowledge graph Vrandečić and Krötzsch (2014) and Wikidata’s SPARQL query service to extract information pertinent to the two tested domains: places and music. We choose these domains as inconsistent knowledge about places may have adversarial consequences in navigation use cases of LLMs Mapbox . Concurrently, inconsistencies in music domain may inaccurately capture the ownership of copyrighted materials (e.g., music albums). For the places domain, we take the list of countries ranked by (nominal) GDP per capita and select the top ten countries as our initial entity list IMF . We then iteratively build our knowledge graph by finding “administrative divisions" that are “located in the administrative territorial entity" of the parent entity. For the music domain, we take the top 200 musical acts since 2000 as our initial entity list Chart (2000). We then find the set of “albums" that have the parent entity as a “performer". We also find the set of songs and singles that are “part of" the album being considered. Once the graph is fully constructed, we randomly select 50 leaf nodes for each domain and extract the full path associated with the selected leaf nodes.

Subject Programs and Query Construction: Table 2 outlines features of the LLMs (SUTs) tested by KonTest. We ensure that temperature for each query is set to zero where possible. In addition, we provide the LLMs with an additional system command, “Be concise as possible. Answer with a yes or no response,", before the start of any conversation. In the event that the LLM does not support a system command, we prepend the statement to the queries before feeding it to the LLM. For the queries themselves, we construct two sets of queries with the aid of a template relating the parent and child entities, as illustrated in Figure 3.

Test Generation and Evaluation: We provide each LLM with each of the queries present in the two sets (i.e., original and mutated) in an atomic manner. We then feed the queries in a sequential manner with queries from both sets being presented in turn. This is then repeated with the order reversed. To invoke the oracles, we only consider responses that begin with a yes or no. Other responses are classified as “Invalid". This allows us to determine the accuracy of the responses without penalizing LLMs that fail to respond appropriately.

Knowledge Gap Mitigation: We evaluate the effectiveness of KonTest’s ensemble method by determining relative weights assigned to each SUT. We restrict the set of relations considered for this process such that a relation that yields an invalid answer in the query generation phase for any of the SUTs is not used in subsequent computations. We then perform five-fold cross validation on the remaining set of relations. Concretely, we hold out 20% of the relations for evaluation and use the consistency errors associated with the rest of the relations in the computation of the weights. We introduce a third template to evaluate whether KonTest’s mitigation generalizes to unseen templates.

Ablation Study: We assess whether an LLM could conceivably be used to replicate each of the sub-tasks in our approach by testing its applicability on the places domain. In particular, we chose to use GPT3.5 for this due to its popularity and relatively high percentage (99.9%) of non-exempted responses in our experiments. In each case described below, we provide GPT3.5 with sample questions and answers (few shot prompts) to ensure that the outputs adhere to the format of KonTest.

Implementation Details and Platforms: KonTest utilizes PyTorch 2.0, CUDA 11.3 and the llm package. All experiments were conducted on a GCP VM with an N1 series machine, eight vCPUs, 30 GB of memory and one Nvidia T4 GPU.

RQ1 Effectiveness: We found that KonTest is effective in exposing consistency errors in LLMs: 19.2% of valid test executions result in consistency errors. Table 3 also shows that metamorphic errors (21.0%) are more prevalent than ontological errors (8.9%). We attribute the lower error rate of ontological errors/inputs to the high complexity of our ontological oracle (see Table 1). Table 3 demonstrates that metamorphic errors are common across all LLMs. In particular, the tested LLMs are highly inconsistent when asked the same question in a different manner (see Table 1). In addition, we observe that a large model size does not necessarily lead to a smaller error rate than a smaller model, e.g., GPT3.5 exhibits a metamorphic error rate of 27.2% for the places entities as compared to a metamorphic error rate of 17.1% for Falcon.

RQ2 Knowledge Coverage: Table 5 shows that the tested LLMs cover about four-fifth (83.5%) of the tested knowledge graph across both templates. We found that LLMs are particularly sensitive to the query templates. For instance, Llama2 exhibits a 47.3% gap in knowledge for the places entities for one template, but only exhibits a 12.9% knowledge gap for the other. This suggests that LLMs may respond differently to two different, but semantically equivalent, input queries. We also observed that the smallest model (Falcon) has the lowest knowledge gap for the places entities (3.1%), while one of the biggest models (GPT3.5) has the highest knowledge gap (29.6%). This implies that model size/complexity is not a good proxy for knowledge coverage/gap. We do, however, attribute the performance of Falcon to its tendency to answer with a positive response regardless of the query. In addition, we also note that all queries posed to the subject LLMs are queries relating to an existing relation. These results show that KonTest effectively reveals the knowledge gap in LLMs. We posit that knowledge coverage is a good criteria for estimating the underlying knowledge of an LLM.

RQ3 Knowledge Gap Mitigation: Results show that KonTest’s mitigation technique reduces the SUT’s knowledge gap by up to 39.30%. Table 4 shows that KonTest reduces the knowledge gap for all templates by 32.48%. We found that the simple majority voting ensemble worsens the SUTs’ knowledge gap by up to 23.74%. We also observed that the mitigation performance of KonTest generalizes to an unseen template (template three (3)). While the mitigation performance of KonTest is slightly better on the new template than the original templates the simple majority performs worse on an unseen template. These results demonstrate the generalizability of KonTest’s mitigation and the efficacy of our weighted ensemble. We attribute the performance of KonTest’s mitigation to the likelihood and distribution of inconsistencies: Firstly, KonTest’s weighted ensemble is 42.72% more effective than the simple majority voting. Secondly, the distribution of inconsistencies shows that only 1.1% (15 out of 1330) of errors are found in all four SUTs while 65.7% (874 out of 1330) of errors are unique to a single SUT.

RQ4 Ablation study with GPT3.5: We conduct an ablation study to investigate whether a state-of-the-art LLM (GPT3.5) is as effective as KonTest in performing its three main sub-tasks – knowledge construction, test generation and error detection.

Knowledge Construction: Table 6 demonstrates that GPT3.5 is not a reliable knowledge constructor. GPT3.5 is only able to identify 68% of the relations present in the original knowledge graph. We believe this is because LLMs are generally intended to be a conversational engine, rather than a knowledge database Pan et al. (2023). These results show the importance of knowledge graphs in KonTest and suggests that LLMs are not a reliable replacement for knowledge graphs.

GPT3.5 is an ineffective surrogate for a knowledge graph: It constructs only 68% of the ground-truth entity relations (edges).

Test Generation: Given a few (three) query examples and the relations in a knowledge graph, GPT3.5 is slightly more effective (17.5% vs 16.8%) than KonTest in test generation (see Table 7). We also observe that this effectiveness persists for both the metamorphic (18.7% vs 17.9%) and the ontological oracle (11.5% vs 11.1%). We attribute the performance of GPT3.5 to the effectiveness of few-shot prompting using the knowledge graph. This guides GPT3.5 to create tests similar to KonTest.

GPT3.5 is slightly (4.2%) more effective than KonTest in generating tests, when provided entity relations with few-shot prompting.

Error Detection: Table 8 highlights that while 52.6% of the metamorphic errors detected by GPT3.5 were detected correctly, approximately 40.1% of the errors were false positives ²²2Note that some errors detected by GPT3.5 cannot be automatically validated by our oracles (UNK in Table 8), as it excludes responses that do not begin with yes/no. . For ontological error detection, we provided GPT3.5 all possible queries and corresponding responses for each knowledge path. GPT3.5 was then asked to detect any inconsistency in these responses. Unlike KonTest, GPT3.5 is unable to detect multiple inconsistencies in a knowledge path. Hence, for a fair comparison in this experiment, we count ontological errors for KonTest at the granularity of a knowledge path (i.e., at most one error per knowledge path). Since validating GPT3.5 responses is not straightforward in this case, we manually validated the responses. We observed that the number of real ontological errors detected by GPT3.5 is comparable to the ontological errors detected by KonTest, but it also had a high false positive rate with nearly 63.7% of errors being misclassified.

LLMs and Knowledge: Pan et al. Pan et al. (2023) presents techniques to combine LLMs and knowledge graphs to address their individual limitations. As an example, Huang et al.  Huang et al. (2023c) have demonstrated the feasibility of knowledge transfer to improve LLM’s generalization ability in software engineering tasks. Analogously, WEAVER Yang et al. (2023a) uses LLMs to generate knowledge bases, using which, requirements are extracted for testing models in real-world settings. GPT4GEO Roberts et al. (2023) experimentally evaluates the capabilities and limitations of GPT-4 in geospatial domain (e.g., places entity), highlighting potential usage of GPT-4 in navigation. Unlike these proposed techniques, KonTest employs knowledge graphs to expose inconsistencies and measure knowledge gaps in LLMs.

Testing and Analysis of LLMs: Several researchers have surveyed the challenges and opportunities for testing and analysing LLMs Zhao et al. (2023); Hou et al. (2023); Zheng et al. (2023); Aleti (2023). Researchers have also studied or proposed methods for testing and analyzing several quality properties of LLMs, including their reasoning ability Wu et al. (2023); Qiu et al. (2023), non-determinism Ouyang et al. (2023), interpretability Palacio et al. (2023); Rodriguez-Cardenas et al. (2023), robustness Zhu et al. (2023), fairness Huang et al. (2023a), security concerns (e.g., privacy, memorization and backdoor attack) Yang et al. (2023b); Staab et al. (2023); Huang et al. (2023b). In contrast to the aforementioned works, KonTest studies the consistency and knowledge coverage of LLMs.

Construct Validity: This relates to the metrics and measures employed in our experimental analysis. To mitigate this threat, we have employed standard testing metrics such as the number/rate of generated inputs, error-inducing inputs, (knowledge) coverage and testing time. For automatic analysis of hundreds of responses, our analysis does not handle expressive, non-binary LLM responses. However, we mitigate this by employing system prompts to ensure the model provides binary responses and we discard non-binary responses (as invalid).

Internal Validity: This refers to the threat that our implementation of KonTest performs its intended knowledge graph-based test generation. We conduct several manual and automated tests, as well as inspection of sampled outcomes of KonTest to ensure the correctness of our approach. Our ablation study (RQ4) further allows to probe the correctness of the sub-step of KonTest versus using GPT3.5. We experimentally verify that over 90% of the entities we evaluate against existed in Wikidata prior to 2020. We also find that under 15% of the errors found by KonTest are linked to these entities. In addition, we also note that the SUT ought to answer the question in a consistent manner regardless of whether the entity existed prior to the corresponding knowledge cutoff date.

External Validity: The main threat to external validity of this work is the generalizability of KonTest and findings to LLMs, knowledge graphs, templates and entities beyond the ones used in this work. For instance, we mitigate the LLM generalizability threat by employing state-of-the-art off-the-shelf, mature, open model weights LLMs (LLaMA2 and FALCON), as well as commercial LLMs (such as GPT3.5 and Gemini). Notably, our entity selection and template construction may be limited to our experimental setting. However, we demonstrate the applicability of KonTest by using two different domains with multiple relations and entity types. We note that we employ few-shot prompting to query the LLMs, thus, our findings may not generalise to other prompting techniques. Finally, KonTest employs Wikidata, a well-maintained knowledge graph that is popularly used in both academia and industry Peters et al. (2019).

LLM Stability and Correctness: Researchers have identified several stability concerns about LLMs Ouyang et al. (2023); Fan et al. (2023), such as non-determinism, randomness, sensitivity to prompting methods, and API/model updates. To mitigate these threats we performed several actions: First, we set the temperature of all models to zero (0), when possible Cloud (2023); Ouyang et al. (2023). Secondly, we reduce the risk of model updates by limiting our testing time (to about a day each per model) and checking for news of model updates before and after testing. Thirdly, we also use models with frozen weights (Falcon and Llama2) to reduce the non-determinism. To automatically validate model outcomes, we employ few-shot prompting, which has been shown to be effective for querying LLMs Deng et al. (2023). We examined whether LLMs are comparable to KonTest (RQ4) using GPT3.5 since it produces the most valid responses (99.9%) (see Table 3).

Knowledge Graph Completeness and Soundness: We note that the knowledge graph is an incomplete snapshot of the real-world. In our evaluation, KonTest only tests for facts (positive tests) derived from the knowledge graph. While it is also possible to use KonTest to test for incorrect relation (negative test), the validation of such test results is challenging due to the incompleteness of knowledge graph. Moreover, we only tests 50 paths from this graph. However, these concerns do not affect our findings, as we are certain about the errors found within the tested subset of the knowledge graph. Finally, knowledge of the world naturally evolves over time and the knowledge graphs do not evolve at the same pace Pan et al. (2023). To mitigate this, we use a widely used knowledge graph Vrandečić and Krötzsch (2014).