Cross-Lingual Multi-Hop Knowledge Editing – Benchmarks, Analysis and a Simple Contrastive Learning based Approach

Large language models (LLMs) are seeing an increasing adoption across users having different cultural and linguistic background, and need to be up to date about the ever-changing knowledge in the world for maintaining their utility and reliability in various applications. Due to the ever increasing compute and data requirements to train these models, there has been a surge in the development of knowledge editing techniques to modify the language models in an efficient way, such that they adhere to the world dynamics.

Prior work on knowledge editing has largely focused on editing LLMs in a monolingual setting (Zhong et al., 2023; Gu et al., 2024), where both user queries and edited facts are expressed in the form of English. These works can be grouped into two categories: parameter-update and parameter-preserving methods. The former directly updates the parameters within LLMs for updating knowledge about the edited facts through meta-learning, fine-tuning, or knowledge locating (De Cao et al., 2021; Dai et al., 2022; Mitchell et al., 2022a; Meng et al., 2022a, b). The later approach freezes the parameters and explicitly stores the edited facts in an external memory and retrieves them for answering user queries (Zhong et al., 2023; Gu et al., 2024; Mitchell et al., 2022c; Hartvigsen et al., 2023). Existing monolingual knowledge editing techniques aren’t broadly applicable since new knowledge can emerge in different languages. Some works have made progress in this direction (Beniwal et al., 2024; Xu et al., 2023a; Si et al., 2024), but they have considered a simplistic setting of assuming the edited facts as independent without any multi-hop rippling consequences on entailed reasoning process, and are primarily focused on parameter-modifying based editing methods.

There has only been a limited focus on the realistic case of cross-lingual multi-hop knowledge editing (see Fig 1), where the edited knowledge can come in through users who communicate in different languages. Further, much of edited knowledge often has a rippling effect on other facts of the world. For example, the club change of Messi affects deduction process of question “indicating a superficial word matching rather than a contextual grasp of the entities involved." This knowledge editing setting, which we argue is important to study, is challenging since the model needs to transfer knowledge about fact edits between different languages, while also reasoning about the facts which are modified as a consequence to the given edit. Poor knowledge transfer between languages can lead to error propagation across reasoning steps which can increase failure cases of model editing.

In this work, we formulate the notion of cross-lingual multi-hop knowledge-editing and analyze existing approaches for their editing ability in different languages, following which a simple yet highly effective approach is designed. Specifically,
① We create one of the first benchmark datasets for measuring cross-lingual multi-hop knowledge editing capabilities of knowledge editing methods. Besides parameter-update based approaches, we contribute strong retrieval-based baselines for knowledge editing and provide a comprehensive analysis.
② We provide a detailed analysis and find significant gaps in the performance of methods for cross-lingual knowledge editing. The gaps are mainly due to challenges in accurately recalling fact edits made in language other than input query.
③ To bridge such gap, we design a competitive method, termed as Contrastive Language-aware Verification for Cross-lingual Knowledge Editing (CLeVer-CKE), for improving performance of cross-lingual multi-hop knowledge editing. Our approach is based on decomposing a multi-hop question in a particular language into sub-questions and retrieving fact edits (if any) from memory using a cross-lingual retriever, which is integrated for answering sub-questions. In particular, the cross-lingual retriever is regularized by novel language-guided and hard-negative based contrastive losses, which leads to improved language and fine-grained sentence understanding of the edits, leading to high quality cross-lingual retrievals. CLeVer-CKE improves over previous SoTA by up-to 30% increase in knowledge editing accuracy when tested on multiple LLMs, datasets and languages.

Following prior work Zhong et al. (2023), a fact is defined as a triplet $(s,r,o)$, where $s$ is the subject, $o$ is the object, and $r$ is the relation (e.g., Shakespeare, author of, Hamlet). Given that a parametric LLM can become outdated or incorrect, knowledge editing is required to be performed on it. An edited fact stores information about updated knowledge of an existing fact and is denoted as $e=(s,r,o^{*})$, where the object is replaced with a new one $o*$.

We consider a collection of $n$ fact edits in the diverse languages: $\mathcal{E}=\{e_{1}^{L_{1}},e_{2}^{L_{2}},e_{3}^{L_{2}},...,e_{n}^{L_{i}}\}$, where $L_{1},L_{2},...,L_{i}$ are different languages for e.g., German, Hindi, Swahili, etc. A language model $f$ is said to be edited with new knowledge facts if the model generations adheres to all the edits present in $\mathcal{E}$. The model is required to seamlessly transfer knowledge about an edit in one language to answer queries in other languages.

For evaluating models in a cross-lingual multi-hop setting, we make use of multi-hop questions such as $Q_{L_{n}}$, in language $L_{n}$ which is different from $L_{1\dots k}$. The question asks about the head entity $s_{1}^{L_{1}}$ for which the answer is $o_{n}^{L_{k}}$ before editing. After editing, the fact chain changes to $\mathcal{P^{*}}=\langle(s_{1}^{L_{1}},r_{1}^{L_{1}},o_{1}^{{L_{1}}*})$ $,(s_{2}^{L_{2}},r_{2}^{L_{2}},o_{2}^{{L_{2}}*}),\dots,(s_{n}^{L_{k}},r_{n}^{L_{k}},o_{n}^{{L_{k}}*})\rangle$ since edits in the first fact can effect the subsequent facts it’s linked to. For answering $Q_{L_{n}}$ after editing, the model has to account for this rippling effect, and provide the final answer as $o_{n}^{{L_{k}}*}$. For this, model has to transfer knowledge of the edited fact and the answer, between languages $L_{1\dots k}$ and $L_{n}$, while correctly reasoning about fact edits via $\mathcal{P^{*}}$.

We develop one of the first parallel cross-lingual for measuring the knowledge editing capabilities of the existing approaches. A parallel benchmark has the same test examples across all the languages, enabling a direct comparison between them. For this, we use existing datasets measuring the multi-hop model editing in English: MQuAKE-CF and MQuAKE-T released by Zhong et al. (2023), which have counterfactual edits and real-world temporal edits respectively. We translate one fact edit per example in these datasets using Google Translate (Google, ) into 7 languages with diverse writing scripts across medium to high resourcedness - German, Spanish, Chinese, Rissian, Hindi, Bengali, Swahili. This results in the benchmark: Cross-Lingual Multi-Hop QnA for Knowledge Editing (CroLin-MQuAKE). It has two datasets, CroLin-MQuAKE-CF and CroLin-MQuAKE-T, each having 8 languages, and 3k and 1.8k parallel examples (same examples in all languages) per language, respectively. The translations are verified by human experts proficient in particular languages and evaluation of BLEU score Papineni et al. (2002) using backtranslation. We find that the translation is highly accurate, since we study medium to high resource languages. See Section A.2 for more details.

Concurrently, Wei et al. (2024) created a multilingual knowledge editing dataset using Wikipedia, offering translocalized knowledge but lacking parallel multilingual examples like ours. CroLin-MQuAKE enables comparing the knowledge editing performance difference across languages directly without being affected by the variation of test sets between different languages.

❶ MeLLo-CL: When edits are made in English, 63.3% of the samples are correct, 29.3% have the contradiction error, 3.6% have Incorrect retrieval, and 3.6% have the incorrect LLM response. For edits made in Hindi, 33.3% of the samples are correct, 60% exhibit an error combination of incorrect retrieval and subsequent contradiction error, where the model first makes an incorrect retrieval and then fails in the contradiction step and 6.6% of erroneous samples are due to the incorrect LLM response. In the CroLin-MQuAKE-CF case when the multilingual edited fact memory containing edits in English and Hindi, MeLLo-CL’s retriever rarely retrieves edits in Hindi, indicating a limitation in its multilingual capabilities. The limitation of MeLLo-CL lies in its retriever-then-contradict mechanism which is up to the LLM.

❷ PokeMQA-CL: When edits are made in English, 53.3% of the samples are correct and 46.3% have the incorrect retrieval error. When edits are made in Hindi, 43.3% are correct, 51% have errors due to the incorrect retrieval and 5.6% are due to the incorrect LLM response. The limitation of PokeMQA-CL lies in its reliance on a bag-of-words model for retrieval. For instance, when presented with the sub-question “Who is the head of state of the USA?", it retrieves the fact “The head of state of Mongolia is Khürelsükh Ukhnaa." This example underscores that PokeMQA-CL prioritizes facts with the highest word overlap, specifically “head of state" indicating a superficial word matching rather than a contextual grasp of the entities involved.

❸ When trained in a cross-lingual setting, PokeMQA-CL exacerbates the issue of bag-of-words retrieval. For example, for the sub-question “Where was Bob Dylan born?", it correctly retrieves “Bob Dylan was born in the city of Nankoku" in English. However, if the same edit is made in German, it retrieves “Bob Dylan spricht die Sprache von Malayalam" (Bob Dylan speaks the language of Malayalam). This issue is a likely a consequence of high word overlap in retriever’s internal translation process and is a limitation of current systems.

Section 4 hints signficant gapS between English-only and cross-lingual case, and that proper knowledge retrieval technique is critical to the performance of cross-lingual knowledge editing.

For overcoming limitations in cross-lingual multi-hop knowledge editing, we design CLeVer-CKE, a cross-lingual and light-weight model editor that seamlessly integrates into any backbone LLM, without changing its parameters. CLeVer-CKE is inspired by memory-based and retrieval-augmented knowledge editing methods (Zhong et al., 2023; Gu et al., 2024; Mitchell et al., 2022b) for mutlihop question answering. CLeVer-CKE follows the following procedure: Given an input query, it a) decomposes the multi-hop question into multiple sub-questions for getting to the final answer, and for answering each sub-question b) retrieves a relevant fact from the edit memory, c) disambiguates whether the retrieved new knowledge is relevant to answering the sub-question, and d) continues the model generation process based on that. In this work, we primarily aim at showing the importance of having a high-quality retriever for the retrieve-and-verify steps at b) and c) described as follows. See Fig. 3 for an overview.

In this paper, we contributed a benchmark having parallel multilingual examples for evaluating cross-lingual multi-hop knowledge editing. We provide new baselines and a detailed analysis of SoTA knowledge editing methods and find various gaps in existing methods, particularly in the cross-lingual setting. Motivated by this, we propose a generic, simple and highly effective method, CLeVer-CKE, for improving the knowledge editing capabilities of parameter-preserving, retrieval augmented editing methods. CLeVer-CKE improves cross-lingual and fine-grained retrieval in the case of knowledge editing, by introducing language aware and hard-negative mining based contrastive losses to train retrievers. Improved retrieval leads to precise knowledge retrieval and reduced error propagation in the multi-hop reasoning setting. CLeVer-CKE is parameter-preserving in terms of the LLM weights, and uses a lightweight retriever with low latency as compared to methods like Zhong et al. (2023).

Our analysis and methods has some limitations. Firstly, although CroLin-MQuAKE is a parallel cross-lingual benchmark, it predominantly contains fact edits related to English-speaking knowledge changes, while the edits could be localized to any part of the world in practice. This reliance on translation rather than trans-localization may lead to gaps in accurately understanding regional and local fact edits. However, having parallel data in all languages is advantageous to accurately measure per-language performance without confounding factors. Secondly, our method is primarily focused on the retriever component and does not address the inherent inaccuracies of the LLMs. This includes issues such as understanding and generation capabilities of LLMs in different languages, correctly breaking down multi-hop questions into sub-questions, accurately extracting the final answer in the desired language. Lastly, our analysis is currently limited to a broad range of medium to high-resource languages. Extending this analysis to low-resource languages presents a significant challenge due to the inaccuracies in translation, which can hinder the proper representation and understanding of facts in low resource languages. Improving translation accuracy and extending our work to low-resource languages is part of our future work.