Establishing Knowledge Preference in Language Models

To evaluate LLMs’ preference for instruction knowledge, we focus on the case where counterfactual assumptions are introduced by the instruction, which is a typical scenario calling for the preference for instruction knowledge and it’s more likely to introduce explicit and direct knowledge conflicts between the instruction knowledge and other types of knowledge.

Among existing works, IfQA Yu et al. (2023) is a human annotated counterfactual QA benchmark where the question introduces hypothetical conditions. We adopt the test set of its full split which has 700 instances in total for evaluating the priority of instruction knowledge in retrieval-augmented setting. We utilize two setups for retrieval augmented setting: (i) w/ Gold Passages where the oracle context following the question is given, and (ii) w/ Mixed Passages where the top-3 retrieved passages from Wikipedia dump along with the oracle contexts and the question is given to be more realistic. The F₁ and Exact Matching (EM) scores are reported.

However, the knowledge conflicts introduced by IfQA may not be explicit and significant enough. For example, in the question If sea levels had risen significantly over the past decade, which country would have been the first to be submerged?, the instruction knowledge sea levels had risen significantly over the past decade does not directly conflict with the oracle context passage which is about the world’s lowest-lying country.

Therefore, we further extend a knowledge editing benchmark MQuAKE-CF-3k Zhong et al. (2023) to be InstructMH-3k for evaluating testee LLMs’ preference between instruction knowledge and the context knowledge. MQuAKE-CF-3k contains multi-hop QA instances based on human-filtered relations, entities, and crafted templates for verbalizing relation triples, but without context passages. Each relation triple is guaranteed to be recallable by GPT-J Wang and Komatsuzaki (2021). Each multi-hop QA instance is associated with a fact chain (sequentially linked relation triples), and knowledge edits. So we integrate the knowledge edits with the original question to obtain a counterfactual multi-hop question (see the question in Fig. 7 for an example). For each factual relation triple needed to get to both the original answer before fact chain editing and the new answer after fact chain editing, we adopt GPT-3.5 to synthesize one supporting context passage which will be given along with the question to the testee LLMs. We evaluate the F₁ and EM scores according to both the original answer and the new answer. If testee LLMs well prioritize the instruction knowledge and generally prefer context knowledge than parametric knowledge, they should follow the counterfactual instruction assumptions, focus on the suitable passages in the context, and reach the new answer instead of the original answer, leading to a higher evaluation scores with new answers than with original answers.

To evaluate LLMs’ preference for context knowledge, we adopt the test set of MRQA Fisch et al. (2019), covering BioASQ Tsatsaronis et al. (2015), DROP Dua et al. (2019), DuoRC Saha et al. (2018), RACE Lai et al. (2017), RelationExtraction Levy et al. (2017), and TextbookQA Kembhavi et al. (2017) across various domains. We divide the evaluation into two parts. The first part is the evaluation on the open-book QA on the whole test set, denoted as MRQA. This quantifies the generally capability of testee LLM to comprehend and prioritize the context knowledge regardless of whether the context knowledge conflicts with their parametric knowledge or not. Here F₁ and EM are reported.

The second part of the evaluation (denoted as CounterMemoryMRQA) is conducted on the subset of the test set where LLMs’ parametric knowledge is conflicted with the context knowledge. So we first probe the parametric knowledge of each testee LLM with 3-shot exemplars (taken from MRQA dev set) to obtain the target test subset. Then, we measure the proportion of test instances for which the model correctly updates its answer (denoted as $\mathbb{P}(\mathbf{U_{c}})$) and the proportion of test instances for which model incorrectly updates its answer (denoted as $\mathbb{P}(\mathbf{U_{i}})$).²²2We decide whether an answer is the same as the gold-standard answers or not, we use F₁ to tolerate minor deviates and set F₁ higher than 0.8 as the same and F₁ lower than 0.2 as different. If a testee LLM well prioritizes the context knowledge, $\mathbb{P}(\mathbf{U_{c}})$ should be significantly higher than $\mathbb{P}(\mathbf{U_{i}})$.

In terms of the instance contents, in contrast to synthesis-based approaches which rely on LLMs to synthesize the entire input and output of each instance, our goal is to provide maximal control on the synthesized contents while ensuring the expected quality. In terms of the data format, we mainly focus on the single-hop and the multi-hop question answering data given reference passages which is related to broad downstream applications of LLMs, especially in the retrieval-augmented setting.

First, we gather a corpus of Wikipedia passage chunks as oracle contexts for subsequent single-hop QA data synthesis. To enhance the efficiency of the corpus to serve for fact-related QA data synthesis, we trace back to the Wikipedia passages that contain evidence for verifiable instances from the FEVER dataset Thorne et al. (2018). A heuristic rule is further applied to filter passages whose number of distinct named entities are fewer than 5. This step results in a corpus of high-quality Wikipedia passages denoted as $\mathcal{C}$.

Second, we traverse the Wikidata to extract a set of fact chains ranging from 2 hops to 4 hops⁴⁴4We assume the questions with the number of hops exceeding 4 are relatively rare in reality. for multi-hop QA data synthesis. The underlying traversal algorithm is based on breadth-first search (BFS) on the knowledge graph. Our fact chain mining algorithms targets at mining both a fact chain $l_{i}$ and its counterfactually edited derivative $l_{i}^{\prime}$. Suppose each fact chain $l_{i}$ with $m_{i}$ hops acquired from BFS is $\displaystyle[e^{i}_{0},r^{i}_{0},e^{i}_{1},r^{i}_{1},\dots,r^{i}_{m_{i}-1},e^{i}_{m_{i}}]$ which consists of triples $\displaystyle(e^{i}_{0},r^{i}_{0},e^{i}_{1}),\dots,(e^{i}_{m_{i}-1},r^{i}_{m_{i}-1},e^{i}_{m_{i}})$ in order. We will randomly choose the number of edits applied on $l_{i}$ as $K_{i}$ and recursively conduct the edit one by one. Each edit is conducted over the previously edited fact chain. At each edit, we will first randomly choose one relation triple from the fact chain (while allowing enough subsequent relation triples for remaining edits) and replace the tail entity with an counterfactual entity of the same type, similar to the misinformation training data generation approach proposed by Fung et al. (2021). Then all the relation triples after this edited relation triple will update their entities factually following this newly changed tail entity without changing any relation. This completes one edit on the fact chain, resulting in a different fact chain. Completing all the $K_{i}$ edits eventually leads to $l_{i}^{\prime}$ as the counterfactually edited derivative of $l_{i}$.

Please refer to Appendix B.1 for more details including the heuristic rules applied for the diversity and quality of the mined fact chains. The set of candidate original and edited fact chains extracted in this step is denoted as $\mathcal{F}$.

For data synthesis in Sec. 4.2 and Sec. 4.3, we randomly sample a set of Wikipedia passages $\displaystyle\{d_{i}\}_{i=1}^{n}\subset\mathcal{C}$ and a set of original and edited Wikidata fact chains $\displaystyle\{(l_{i},l_{i}^{\prime})\}_{i=1}^{m}\subset\mathcal{F}$ respectively for each step.

To synthesize instruction tuning data which grants the highest preference priority for instruction knowledge, we resort to counterfactual question answering. The counterfactual assumptions or hypotheses set up the instruction knowledge which will directly conflict with the parts of the factual “retrieved passages” and likely deviates from the LLMs’ parametric knowledge. Such synthesized data can guide LLMs to prioritize the instruction knowledge, overriding conflicted parts of the context knowledge and potentially the parametric knowledge, to reach the correct answer.

Specifically, for each randomly sampled passage $d_{i}$, we prompt GPT-4o based on $d_{i}$ to synthesize an single-hop QA instance containing:

• •

  The counterfactual question which introduces counterfactual and hypothetical conditions or incidents.

• •

  The precise, concise, no-trivial, and uniquely-derivable answer through counterfactual reasoning based on $d_{i}$, the hypothetical question, and common sense⁵⁵5As counterfactual reasoning might inherently use some common sense knowledge beyond the context and the question, and it’s hard to elaborate them one by one, we do not prevent GPT-4o from using them..

• •

  Extra information as an additional passage to make sure the answer is uniquely derivable.

• •

  The step-by-step answer derivation explanation.

Please refer to Appendix A.4 for prompt templates used to obtain these components. Through prompting GPT-4o for instance synthesis, we expect that GPT-4o can bring more diversity and non-trivial difficulty through leveraging its reasoning power and external knowledge beyond the provided Wikipedia passage $d_{i}$. Human annotators could provide higher quality for this kind of data as they can be better at recalling related external knowledge and capturing their underlying associations through complex reasoning. However, the disadvantages of relying human annotators include the expense and the potentially limited counterfactual reasoning patterns that human annotators can think of. To encourage diversity, we adopt no in-context demonstrations for synthesis.

The synthesis for multi-hop QA instance is similar except that the counterfactual assumption is predefined by the counterfactual fact chain edits and the target answer is just the tail entity of the edited fact chain. We mainly prompt GPT-4o for synthesizing based on $(l_{i},l_{i}^{\prime})$:

• •

  The multi-hop question that starts from and includes only the head entity of the edited fact chain $l_{i}^{\prime}$, incorporates all the relations, and has the tail entity of $l_{i}^{\prime}$ as the final answer. Later we will apply a template to integrate the counterfactual edits as the assumptions with the generated multi-hop question.

• •

  A list of passages for all factual relation triples from $l_{i}$ and $l_{i}^{\prime}$ so that each factual relation triple can be uniquely derived given all of the passages simultaneously.

• •

  The step-by-step answer derivation explanation.

Please refer to Appendix A.4 for prompt templates used to obtain these components. Since we can only mine relation triples from Wikidata, we adopt GPT-4o for synthesis relying on its power to understand and verbalize the relation triples into fluent and coherent natural language. To ensure the quality of synthesized multi-hop questions, we took a fixed set of 5 exemplars demonstrating the synthesis of multi-hop question from a given fact chain.

The goal of modeling the preference for context knowledge is to teach LLMs to prefer the “retrieved contexts” over their own parametric knowledge. Sticking to the data format of single-hop and multi-hop QA with reference passages, we achieve this goal by synthesizing factual QA instances with answers supported by the passages but opposed by the LLMs’ parametric knowledge.

For single-hop QA instances, we prompt GPT-4o with passage $d_{i}$ to synthesize the factual question, the corresponding answer, the step-by-step answer derivation, and an additional passage to further make sure the answer is uniquely derivable from the contexts. For multi-hop QA instances, we leverage the unedited fact chain $l_{i}$ and prompt GPT-4o to synthesize the multi-hop question, a list of passages verbalized from relation triples of $l_{i}$ to ensure the tail entity of $l_{i}$ is uniquely derivable, and the step-by-step answer derivation. One special design is that, we will first probe a list of base LLMs with the synthesized question to filter questions that can be correctly answered by the base LLMs’ parametric knowledge. This step is done before further synthesizing the remaining components of the new instance for efficiency. Please refer to Appendix A.4 for prompt templates used here.

Without tuning the LLMs, we also experimented with different prompts to see whether they can enhance or establish the hierarchical knowledge preference. In this work, we mainly apply three prompting templates (see Appendix A.3): (i) Alpaca Taori et al. (2023)’s prompt template as baseline. (ii) Assumption-in-Instruction based on (i) which puts instruction knowledge in the instruction and the instruction explicitly requires LLMs to follow the hierarchical knowledge preference, (iii) Assumption-in-Question based on (i) which puts instruction knowledge along with the question in the input and the instruction explicitly requires LLMs to follow the hierarchical knowledge preference. We denote (i) as Normal Prompt and denote (ii) and (iii) as Explicit Prompt.

Our comparison mainly focuses on the base LLM trained with Alpaca’s 52K instruction tuning data (denoted as w/ Alpaca) and the same base LLM trained with the same 52K data plus our HierPref data (denoted as w/ HierPref). We select Mistral-v0.3-7B released in 05/22/2024 as the base LLM. In addition to this, we also include LLMs include Llama-2 Touvron et al. (2023), Llama-3 AI@Meta (2024), Qwen-2 Bai et al. (2023), GPT-3.5 OpenAI (2023), and GPT-4o OpenAI (2024) with both the base model and instruction-tuned model for reference.

Fig. 5 shows the test results of LLM trained with IfQA train set, our synthesized single-hop counterfactual QA data, and with a combination of them. The test performance of the LLM tuned on the train set of the IfQA saturates, which shows that the human annotations lead to limited patterns. Furthermore, our synthesized data together with the train set of IfQA further improve the test set performance. We can also see that simply tuning the LLM with our synthesized data which is generated through zero-shot prompting still cannot match the in-domain human annotated IfQA train set.

We provide the zero-shot results of HierPref with different training strategies on IfQA and MRQA, which are both human annotated, to justify our final choice: (i) add randomly sampled noise context passages, (ii) do not add the step-by-step answer derivation in training, and (iii) randomly shuffle the oracle passages and assumptions (if possible). Table 6 justifies our design choice. Table 21 and Table 22 further shows that shuffling the assumptions and oracle contexts can avoid LLMs to take shortcuts for multi-hop QA.

Previous related studies have focused on the preference of language models between external context knowledge and the internal parametric knowledge Longpre et al. (2021); Xie et al. (2024); Kortukov et al. (2024); Zhang et al. (2024b). Xie et al. finds that LLMs generally prefer evidence consistent with their parametric knowledge over the conflicting evidence (2024). Another finding is that LLMs demonstrate strong confirmation bias when external evidence contains consistent information with parametric knowledge which is also supported by a more recent study Kortukov et al. (2024). On the other hand, external evidences that are coherent, convincing, though conflicting with parametric knowledge can still make LLMs highly receptive to them (Xie et al., 2024; Kortukov et al., 2024). Different from these works, we further refine the knowledge conflicts into instruction knowledge, context knowledge, and parametric knowledge for study and we resort to regularizing LLMs’ behaviors under different knowledge conflicts.

Existing works have investigated how to regularize the behaviors of LLMs in conflicts. One typical scenario is to edit new knowledge into LLM artifacts to inject external knowledge to override the parametric knowledge. Corresponding methods include revising the LLM weights, applying adaptor networks, and integrating explicit memories Meng et al. (2022a, b); De Cao et al. (2021); Mitchell et al. (2022); Zhong et al. (2023). Our work introduces the instruction knowledge to integrate the goal of this research direction with the more complex scenario where external contexts cause additional knowledge conflicts. Furthermore our work resort to instruction tuning to enable such knowledge injections against knowledge conflicts inherently in inference time.

Recent works have also explored improving the safety of LLMs against jailbreak attacks inside instructions. OpenAI has introduced the instruction hierarchy Wallace et al. (2024) to teach LLMs to ignore jailbreak instructions for this direction. In contrast, our work focuses more on the knowledge conflicts and building the preference hierarchy between the instruction as a whole, the context passages, and LLMs’ parameters.

Some LLM related works have focused on aligning LLMs with human preference to direct LLMs towards being more helpful, less toxic, and more powerful Korbak et al. (2023); Dai et al. (2023); Tian et al. (2023); Havrilla et al. (2024). Reinforcement learning from human feedback (RLHF) is the most widely adopted technique for such alignment, with its classical pipeline comprising supervised fine-tuning, reward model training, and reinforcement learning via policy optimization Ouyang et al. (2022). Various works have been developed for improved efficiency, less bias, and enhanced alignment Lu et al. (2022); Dong et al. (2023); Wang et al. (2024); Sun et al. (2024); Rafailov et al. (2024); Meng et al. (2024). Our work has similar goals with these works to build LLMs that are more aligned to human preference. However, different from this line of research, our work focuses more on the knowledge utilization aspect of LLMs, namely the way LLMs leverage external knowledge and parametric knowledge while adhering to human preference (instruction knowledge).