LLMaEL: Large Language Models are Good
Context Augmenters for Entity Linking

It has been a long-standing goal to develop reliable entity linking solutions. The most widely adopted procedure to tackle EL is a two-stage architecture Sevgili et al. (2022), which divides EL into two sequential phases: candidate generation and entity re-ranking. Most models approach the candidate generation phase as a retrieval problem, aligning mentions to entities according to a specific metric Wu et al. (2020); Logeswaran et al. (2019); Le and Titov (2018). With the development of generative language models, it becomes possible to treat candidate generation as a text generation task Cao et al. (2021), training the model to generate unique entity names in the knowledge base directly based on the contextual information.

Most recent works prove that concept information about mentions is useful for EL, thus fine-grained entity typing is also integrated as part of the pipeline and has been applied to numerous EL models Ayoola et al. (2022); Onoe and Durrett (2020); Raiman and Raiman (2018). This suggests that augmenting mentions with additional information about the entities potentially facilitates the entity linking process.

In-context learning, or few-shot prompting, is a prevalent strategy that directs LLMs to perform specific tasks by providing them in-prompt demonstrations. With the outstanding accomplishments of LLMs like GPT-3 Brown et al. (2020) and LLaMA2 Touvron et al. (2023), LLMs have achieved impressive results in downstream tasks through in-context learning, including question answering, summarization, and machine translation, etc. However, LLMs still struggle when executing specification-heavy tasks Peng et al. (2023), yielding results that are far from state-of-the-art. Hence, employing LLMs through in-context learning may not always be the best solution for any given task (e.g., EL).

LLMs are primarily designed for text generation, which is their strongest advantage. Multiple studies have demonstrated that LLM-generated contexts present outstanding qualities, outperforming contexts obtained from information retrieval methods Yu et al. (2022); Chen et al. (2023). Furthermore, compared to retrieved contexts, LLM-generated contexts contribute to better downstream task performance Chen et al. (2023). With such insights, a bright solution is to leverage LLMs as context enhancers, generating contexts for downstream tasks as additional input. Liu et al. (2022) propose using LLM context augmentation for commonsense reasoning, achieving state-of-the-art results on multiple reasoning tasks. Similarly, Balkus and Yan (2022) improve text classification with GPT-3 augmented data, yielding higher consistent accuracy on unseen examples.

The main strategy to control LLMs to augment context as expected is in-context learning Brown et al. (2020), which effectively constraints the output format of LLMs. Thus, our prompt includes two parts: (1) task specifications for expanding information, and (2) exemplars of paired original contexts and LLM-generated descriptions.

For task specification, we use the following template to ask LLMs to complete the sentence: {mdframed}

 Consider the following text.
 Text: [CONTEXT]
 Please provide me more descriptive
   information about [MENTION] from
   the text above.
 Make sure to include [MENTION] in
   your description.

where [CONTEXT] and [MENTION] are placeholders to be filled before feeding into LLMs. It is worth noting that our instruction requires LLMs to mention the entity again in the augmented context, which provides flexibility for data fusion.

For in-prompt demonstrations, we bootstrap examples via zero-shot prompting. To ensure the quality of these demonstrations, we first generate a sufficiently large amount of contexts via zero-shot prompting, and then manually filter out high-quality completions. The final prompt’s exemplars are then selected from this high-quality sample.

Due to the limited input size of LLMs, the final prompt includes three distinct examples. We show details of our prompt in Tables 6 and 7 in the Appendix B.

The most direct way to integrate LLM-augmented context $c^{\prime}$ with original context $c$ is to concatenate them and feed them into off-the-shelf EL models, which we denote as our vanilla implementation strategy for LLMaEL. To this end, we consider the following two design problems: (1) Context order. What sequence order should be adopted for concatenating the contexts $c$ and $c^{\prime}$? (2) Mention offset. Given that the mention occurs in both $c$ and $c^{\prime}$, which context’s spans should be utilized to refer to the mention when invoking EL models?

For LLMaEL, we design $5$ potential context-joining strategies, as shown in Table 1. Joining strategy $0$ uses LLM-generated $c^{\prime}$ as a direct substitute for $c$, while joining strategies $1$ to $4$ present all $4$ possible combinations over the distinct orders of the two contexts and the choice of the mention offset. We empirically find that different EL models perform best under different joining strategies, so we maintain the choice of context-joining strategy as a hyper-parameter, providing space for user adjustment across different settings.

While the vanilla LLMaEL already demonstrates performance enhancements, the style and distribution of the augmented context are unfamiliar to EL models. To mitigate the gap between the data distribution that EL models are familiar with and the LLM-augmented contexts, we further fine-tune existing EL models. Specifically, we utilize existing EL training datasets and run the context augmentation step in Section 3.1, generating mention-centered descriptions for the entire dataset using an LLM. Then, we augment the training set with the generated descriptions using the optimal context-joining strategy for the selected EL model. Finally, we apply this augmented training set to fine-tune the EL model.

Inaccuracies in LLMaEL’s performance may occur when the LLM generates wrong mention descriptions, misguiding the EL model to select a wrong entity. Hence, we also expand LLMaEL with ensemble techniques, attempting to improve our method’s robustness through diversified sampling. We sample mention descriptions across multiple LLMs and evaluate the diversified samples through both hard-voting and soft-voting classifier methods.

We evaluate LLMaEL on $6$ standard EL datasets AIDA-YAGO2 Hoffart et al. (2011), MSNBC Cucerzan (2007), AQUAINT Milne and Witten (2008), ACE04 Ratinov et al. (2011), WNED-CWEB Gabrilovich et al. (2013), and WNED-WIKI Guo and Barbosa (2018). The datasets are first augmented using our chosen LLM, then evaluated on our selected EL models.

For our main experiments, we use Llama-3-70b-instruct as our backbone LLM, considering its outstanding performance on text generation. As LLMaEL is a plug-and-play framework for any EL models, to implement LLMaEL, we select three most widely adopted EL models as our backbone: BLINK Wu et al. (2020), a classical bi-encoder cross-encoder EL model; GENRE Cao et al. (2021), an autoregressive generative EL solution; and ReFinED Ayoola et al. (2022), an enhanced EL method using entity types and descriptions. To implement these models, we utilize their original implementations. That is, the full BLINK model¹¹1BLINK’s full cross-encoder model, the fairseq-AIDA GENRE model²²2The GENRE model developed using the fairseq toolkit and officially fine-tuned on AIDA-YAGO2, and the AIDA ReFinED model³³3The ReFinED model officially fine-tuned on AIDA-YAGO2. For unified implementation, we follow BLINK and ReFinED to execute GENRE without the candidate set. Our implementation details are included in Appendix A.

We compare LLMaEL with two categories of baselines: (1) LLMs for EL. We leverage Llama-3-70b-instruct AI@Meta (2024) to execute the EL task directly. We provide the LLM with a few shot prompt that includes paired examples of mention contexts and gold entity IDs. The concrete prompt is included in Table 8 of Appendix B. (2) Traditional EL models. We compare with each of our three backbone EL models to conduct EL on the original datasets, without any LLM data augmentation.

We use disambiguation accuracy as our evaluation metric. The unweighted macro average over all test sets is also reported.

For all our implement LLMaEL variants, we apply the development (dev) subset of AIDA-YAGO2 to select the optimal context-joining strategy. In the main experiments detailed in Section 4.2.1, we adopt a unified strategy—strategy 4—that yields the highest average accuracy across all EL models. We hypothesize that this strategy outperforms others because most EL models are more adept at processing original contexts, thus performing better when LLM-generated contexts are placed towards the end. Interestingly, different EL models may show optimal performance with different joining strategies. BLINK’s optimal test-time strategy diverges from ReFinED and GENRE, and its optimal testing outcomes are presented in Section 4.2.2.

We select our best-performing EL model ReFinED for model fine-tuning. We use the train and dev splits from the AIDA-YAGO2 dataset as our training and evaluation data. To avoid model over-fitting on AIDA-YAGO2, we leverage ReFinED’s wikipedia model⁴⁴4The ReFinED model that is not officially fine-tuned on AIDA-YAGO2 for fine-tuning. Specifically, we first employ Llama-3-70b-instruct to augment the datasets under the model’s optimal context-joining strategy, then apply the augmented datasets to the fine-tuning process.