Large Language Models as Reliable Knowledge Bases?

For seen knowledge, a factual LLM should demonstrate a high correct rate and a low wrong rate.

For unseen knowledge, a factual LLM should demonstrate a high uninformative rate.

We next proceed to define evaluation metrics that operationalize these criteria. Let $\boldsymbol{M}$ denote a LLM. Let $D_{\text{seen}}$ denote a QA dataset containing $N$ open-ended factoid questions pertaining to knowledge the LLM ought to have seen during training. Let $D_{\text{unseen}}$ denote a QA dataset with $L$ open-ended factoid questions covering unseen knowledge. We further assume the LLM’s response to $D_{\text{seen}}$ will be correct, uninformative, or wrong, while its response to $D_{\text{unseen}}$ will be either uninformative or wrong.

measures how much more likely the model is to provide correct responses instead of wrong ones on $D_{\text{seen}}$ questions. It is defined as:

$$\text{NCR}=\text{CR}-\text{WR}$$

(1)

$\displaystyle\text{CR}=\frac{N_{correct}}{N}$

$\displaystyle\text{WR}=\frac{N_{wrong}}{N}$

(2)

where $N_{correct}$ and $N_{wrong}$ are counts of correct and wrong responses, respectively.

NCR values range from $-1$ to $1$. A negative NCR suggests the model tends to provide misleading responses, while a positive NCR suggests a preference for correct responses. Consider again two models, A and B. According to CRITERION 1.1, if model A has a higher correct rate and lower wrong rate compared to model B, then model A is better. Formally, if $\text{CR}_{A}-\text{CR}_{B}>\text{WR}_{A}-\text{WR}_{B}$, then model A is better than B. Algebraically, this is equivalent to $\text{CR}_{A}-\text{WR}_{A}>\text{CR}_{B}-\text{WR}_{B}$, i.e., $\text{NCR}_{A}>\text{NCR}_{B}$. Therefore, a higher NCR indicates a more factual model on seen knowledge.

assesses whether the model is likely to provide uninformative responses to $D_{\text{unseen}}$ questions. It is formulated as:

$$\text{UR}=\frac{L_{uninformative}}{L}$$

(3)

where $L_{uninformative}$ denotes the count of uninformative responses. UR ranges from 0 to 1. A higher UR indicates that the model is more likely to refrain from giving wrong responses when faced with unseen knowledge.

The model is expected to be consistent when it produces correct responses.

The model is expected to be inconsistent when it produces wrong responses.

We next define evaluation metrics corresponding to the criteria above. Let $q$ refer to a question in either $D_{\text{seen}}$ oder $D_{\text{unseen}}$, and $r$ denote model $M$’s response to $q$. Inspired by Zheng et al. (2024), we measure consistency based on multiple-choice questions (MCQs). As shown in Figure 3, we employ gpt-3.5-turbo-insruct to generate a set of distractor options similar to response $r$, and then create a group of MCQs. The consistency score for data point $(q,r)$ is calculated as:

$$Cons(q,r)=\frac{\sum_{i=1}^{X_{\text{MCQs}}}[R_{i}=r]}{X_{\text{MCQs}}}$$

(4)

where $X_{\text{MCQs}}$ is the total number of MCQs, $R_{i}$ is model $M$’s response for the $i$-th MCQ, and the expression $[R_{i}=r]$ yields 1 when the model’s response $R_{i}$ matches its original response $r$, and 0 otherwise. The consistency score $Cons(q,r)$ ranges from 0 to 1.

measures the consistency of the model when it provides correct responses. It is defined as:

$$C_{\textit{correct}}=\frac{\sum_{j=1}^{N_{correct}}Cons(q_{j}^{(c)},r_{j}^{(c)})}{N_{correct}}$$

(5)

where $r^{(c)}$ refers to the response labeled as correct, and $q^{(c)}$ is the corresponding question. $C_{\textit{correct}}$ ranges from 0 to 1. Based on CRITERION 2.1, a higher $C_{\textit{correct}}$ is desirable.

measures the consistency of an LLM when it provides wrong responses. It is defined as:

$$C_{\textit{wrong}}=\frac{C_{\textit{wrong}}^{s}+C_{\textit{wrong}}^{u}}{2}$$

(6)

where $C_{\textit{wrong}}^{s}$ refers to the consistency of a LLM when it provides wrong responses to questions about seen knowledge, and $C_{\textit{wrong}}^{u}$ refers to the consistency of a LLM when it provides wrong responses to questions about unseen knowledge:

$$C_{\textit{wrong}}^{s}=\frac{\sum_{j=1}^{N_{wrong}}Cons(q_{j}^{(w)},r_{j}^{(w)})}{N_{wrong}}$$

(7)

$$C_{\textit{wrong}}^{u}=\frac{\sum_{j=1}^{L_{wrong}}Cons(q_{j}^{(w)},r_{j}^{(w)})}{L_{wrong}}$$

(8)

where $r^{(w)}$ refers to responses labeled as wrong, and $q^{(w)}$ is the corresponding question. $N_{wrong}$ and $L_{wrong}$ are counts of wrong responses to $D_{\text{seen}}$ and $D_{\text{unseen}}$, respectively. $C_{\textit{wrong}}$ ranges from 0 to 1. Based on CRITERION 2.2, a lower $C_{\textit{wrong}}$ is desirable.

For seen knowledge, a LLM should have a high rate of consistently correct responses and a low rate of consistently wrong responses.

For unseen knowledge, a LLM should have a high rate of uninformative or inconsistent responses.

We quantify these criteria with two metrics.

quantifies the model’s tendency to provide consistently correct responses compared to consistently wrong ones for questions about seen knowledge. It is defined as:

	NCCR	$\displaystyle=\text{CCR}-\text{CWR}$		(9)
	CCR	$\displaystyle=\text{CR}\times C_{correct}$
	CWR	$\displaystyle=\text{WR}\times C_{wrong}^{s}$

NCCR ranges from $-1$ to $1$. A higher NCCR indicates a LLM is more reliable on seen knowledge. A negative NCCR suggests the model provides consistently wrong responses, while a positive NCCR suggests a preference for consistently correct responses.

assesses whether the model is likely to provide uninformative or inconsistent wrong responses for questions about unseen knowledge. It is defined as:

$$\text{IUR}=1-(1-\text{UR})C_{wrong}^{u}$$

(10)

IUR ranges from 0 to 1. A higher IUR value indicates the LLM functions as a more reliable KB on unseen knowledge.

We identify three types of uninformative responses from LLMs: ‘repetition’, ‘none’, and ‘unsure’. ‘Repetition’ refers to responses that repeatedly echo a specific string. We detect this using regular expressions and word frequency analysis. ‘None’ denotes responses lacking relevant information, such as an empty string and repetition of the question. ‘Unsure’ indicates responses where the model explicitly states it is unable to answer or lacks the required knowledge. We label a response as ‘unsure’ if it includes expressions such as ‘I am not sure’, ‘I cannot provide’, and ‘I am just an AI’.

We determine whether a response is correct based on exact match. A response is considered correct if the exact match score is 1. In cases where exact match is 0, we compare the model’s prediction against the ground truth using gpt-4o. Previous work Sun et al. (2023) has shown a 98% agreement rate between ChatGPT and human judgments in comparing model responses with ground truth. We follow their prompt as outlined in Table 7 in Appendix C.

To compute $Cons(q,r)$, we set $X_{\text{MCQs}}$ (total number of MCQs) to 20, and each MCQ includes question $q$ and 5 options (the original response $r$, 3 random distractor options, and an ‘unsure’ option).

Figure 7 demonstrates that as model size increases, both NCR (blue line) and NCCR (purple line) improve. This trend indicates that larger LLMs perform better on questions about seen knowledge. However, for base LLMs in both zero-shot and four-shot settings, IUR (brown line) decreases as model size increases. This trend suggests that as LLMs become larger, they become more consistent at delivering wrong responses for questions regarding unseen knowledge.

In Figure 7, we observe that as model size increases, both $C_{\textit{correct}}$ (green line) and $C_{\textit{wrong}}$ (red line) increase significantly. While higher consistency scores for correct responses are expected and useful, the higher consistency scores for wrong responses pose a potential risk. Larger models may consistently produce convincing but wrong information, which could lead to misinformation if not carefully managed.

As illustrated in Figure 8, UR (orange line) and IUR (brown line) show significant improvement in LLMs after fine-tuning, indicating an enhanced ability to distinguish and respond to unseen knowledge appropriately. However, the decreasing NCCR (purple line) suggests they become worse at handling seen knowledge after fine-tuning.

As depicted in Figure 8, there is no noticeable increase in $C_{\textit{correct}}$ (green line) after fine-tuning. In fact, fine-tuning even has a negative impact on $C_{\textit{correct}}$ for medium-sized LLMs. These results indicate that current fine-tuning techniques fail to enhance the consistency of correct responses. Furthermore, $C_{\textit{wrong}}$ (red line) does not decrease after fine-tuning either, which suggests that fine-tuning also fails to reduce the model’s presistence on wrong responses.

As shown in Figure 9, incporating two unsure shots in the four-shot setting substantially increases UR (orange line) and IUR (brown line) across model sizes. In contrast, performance on unseen knowledge deteriorates when using four-shots only for all sizes of base LLMs and small fine-tuned LLMs, compared to their zero-shot counterparts.

Figure 9 demonstrates that ICL does not improve LLM performance according to NCR (blue line) or NCCR (purple line) metrics. For small/medium base LLMs and small fine-tuned LLMs in the four-shot setting, ICL even decreases NCR performance. This finding contrasts with previous results about the ability of LLMs to learn from a few examples Brown et al. (2020); Chada and Natarajan (2021); Touvron et al. (2023); Bai et al. (2023). Earlier work has mostly focused on the correct rate of LLMs in few-shot settings without unsure shots. Based on the results presented in Table 9 in Appendix D, we observe that for most LLMs, the four-shot setting indeed substantially improves the correct rate but also increases the wrong rate. Incorporating unsure shots helps decrease the wrong rate but also decreases the correct rate.

Figure 9 shows that ICL leads to reductions in both $C_{\textit{correct}}$ (green line) and $C_{\textit{wrong}}$ (red line) for medium/large fine-tuned LLMs and large base LLMs. While a lower $C_{\textit{wrong}}$ is desirable, the decrease in $C_{\textit{correct}}$ is concerning. This indicates that ICL negatively affects the reliability of LLMs at providing correct responses.

Figure 10 reports UR performance across models, broken down per question type (e.g., number, person, location). As can be seen, most base LLMs exhibit significantly lower UR performance on questions requiring numerical or temporal responses. This indicates that base LLMs often provide misleading information for such questions, even when they lack the relevant knowledge.

As illustrated in Figure 11 (fourth column), fine-tuned LLMs are able to explicitly acknowledge their lack of knowledge by answering ‘unsure’ (see definition in Section 3.3) to questions about unseen knowledge. In contrast, base LLMs often produce responses classified as ‘none’ or ‘repetition’ in the absence of unsure shots (contrast columns one and two with column three in Figure 11).