DIM: Dynamic Integration of Multimodal Entity Linking with Large Language Model

In our investigation and study, we analyze the existing Entity Linking datasets and their methods of entity representation:

• •

  Wikimel and Richpedia (Zhou, \APACyear2021) employ concise attributes from Wikidata. This representation lacks representativeness for entities, as many entities share similar attributes. It is easy to link accurate cognition to the wrong entity incorrectly.

• •

  Wikiperson (Sun, \APACyear2022), similar to Wikimel, uses attributes of individuals as representatives, but in a more simplified manner. In this example, the attribute “American President” is inadequate as a representative of Joe Biden, given that there have been multiple American Presidents.

• •

  Weibo (Zhang, \APACyear2021) utilizes individuals from Weibo as entities, using user-authored personal bios as entity representations. These bios, relying on user-generated content, may contain biases or errors and do not accurately reflect the broader public’s understanding of the entity.

• •

  Wikidiverse (Wang, \APACyear2022) uses images collected from Wikipedia as entity representation. However, images can deviate from a person’s true appearance due to factors like angles and time, lacking real-time accuracy.

In summary, existing multimodal entity linking methods suffer from the limitation that entity representation fails to effectively represent entities. More importantly, these representations are manually collected from Wikipedia or other knowledge bases and can only represent the entity’s state at a specific time. Human understanding of entities changes over time and events. For instance, Donald Trump is no longer the President of the United States in 2023. In such cases, rigid and less adaptable entity representations can lead to errors. Additionally, when a mentioned entity is not in the dataset, there is no corresponding entity representation, causing potential issues in entity linking.

Table 1 shows the diverse representation methods for entities in existing research. (It is worth noting that Weibo dataset (Zhang, \APACyear2021) is primarily designed for Chinese celebrities and does not involve international figures. For the sake of convenient presentation and visual comparison, following the definition of the Weibo dataset, we manually gathered personal profiles of relevant entities from Twitter. ) Therefore, the paper proposes a dynamic method for constructing entity representations, leveraging the capabilities of ChatGPT to enhance real-time and scalable entity representations based on the evolving understanding of the world.

ChatGPT (OpenAI, \APACyear2023) is a powerful large model trained on massive amounts of web data and is continuously updated over time. We plan to utilize the interface provided by ChatGPT to inquire about entities, so as to subsequently link with the entities in the implicit knowledge base of ChatGPT. Candidate entities are input into ChatGPT for inquiries using the prompt: “You are a helpful assistant designed to give a comprehensive introduction about people. Who is this one?” The generated response from ChatGPT is shown in Table 1. For experimental convenience and dataset quality enhancement, we collect the response of ChatGPT to enhance the dataset. In detail, we construct entity representations for 17391, 17804, and 57007 entities from Wikimel, Richpedia, Wikidiverse, respectively. The newly built entity representations better reflect the general public’s understanding of entities, align closely with their inherent semantics, and facilitate a unified approach to cognition and knowledge base.

Taking the Wikimel (Zhou, \APACyear2021) as an example, out of the 17474 entities collected from the dataset, 131 entities did not return any results, and 220 inquiries returned “Sorry, I cannot provide an introduction to this entity.” Besides, ChatGPT provides speculative information based on cultural, regional, or other contextual cues for 462 entities. For example, “John Abbott is a common English given name and surname,” but did not provide specific representation. Additionally, 2997 entities require additional information for verification. For instance, “It is possible that Edward J. Livernash is a private individual without any notable achievements.” Furthermore, 599 entities are speculated to be fictional names, such as “John McDuffie is a fictional name, so there is no information.” The specific reasons and their proportions are as shown in Figure 2. In summary, out of 17391 entities, 5517 entities were unable to be enhanced through ChatGPT. For these entities, we continue to use the original entity representations.

Given a sentence $x_{t}$, mention $x_{m}$ and Wikipedia description of entity $x_{e}$, we follow CLIP (Radford \BOthers., \APACyear2021) to tokenize it into a sequence of word embeddings. Then the special tokens startoftext and endoftext are added at the beginning and end positions of word embeddings. As a result, with $N$ sentences and $N_{e}$ candidate entities, we feed sentence representation $t\in\mathbb{R}^{N\times d}$, mention representation $m\in\mathbb{R}^{N\times d}$ and entity representation $e\in\mathbb{R}^{N_{e}\times d}$ into model. Similarly, we feed image into visual encoder of CLIP to get visual feature $v\in\mathbb{R}^{N\times d}$. $d$ is the hidden size of textual and visual features.

We employ BLIP-2 as the expert for extracting information from images, employing various approaches to process BLIP-2 images: 1) Image Captioning: We extract corresponding captions $c_{1}$ for images, such as “A man and a woman on the red carpet.” 2) Prompt-based Inquiry: We utilize prompts to ask the detailed information $c_{2}$ about the images, with specific prompt designs such as “Question: Who are the characters in the picture? Answer: ”

We concatenate the image-related information obtained through these two methods to obtain expert information $c=\text{[CLS]}c1\text{[SEP]}c2$, where ’[CLS]’ and ’[SEP]’ are special characters used to indicate the beginning and separation of text, respectively. $c$ is fed into the text encoder of CLIP to get expert feature $f_{c}$.

We combine the extracted image and text features with the supplementary information provided by the expert, concatenating them to form the final feature representation. To manage the expression of features, we employ multi-head attention for selection. Through multi-head attention, $c$ will interact separately with text feature $t$ and image feature $v$ to extract useful information and control noise.

where $W^{Q}\in\mathbb{R}^{d\times d_{q}},W^{K}\in\mathbb{R}^{d\times d_{k}},W^{V}\in\mathbb{R}^{d\times d_{v}}$ are randomly initialized projection matrices. We set $d_{q}=d_{k}=d_{v}=d/h$. $h$ is the number of heads of attention layer.

On one hand, text information, image information, and expert features are fused to form the fused feature $g$.

On the other hand, the textual representations of $N_{e}$ candidate entities in the knowledge base also encoded by CLIP’s text encoder to obtain $e$. During this process, the limit of text length is imposed to prevent excessively long entity representations. The fused feature $g$ is then matched with the textual features of the $N_{e}$ entities using cosine similarity, calculating the Match Score. If $g$ is most similar to the correct entity among the $N_{e}$ entities, it is considered a successful T@1 prediction. Similarly, if the similarity ranks within the top 5, it constitutes a successful T@5 prediction.

We utilize Npairloss (Sohn, \APACyear2016) as our optimization training objective to enhance the learning process. NpairLoss is applied to networks with multiple outputs, where each output corresponds to a specific task. Its purpose is to optimize the network by maximizing the similarity of sample pairs within the same category, thereby enhancing the effectiveness of multi-task learning. Specifically, for each sample, NpairLoss defines the loss by comparing the similarity between positive sample pairs (belonging to the same category) and negative sample pairs (belonging to different categories). It encourages the network to make sample pairs within the same category more similar while ensuring that sample pairs from different categories are more dissimilar.

The mathematical formulation of NpairLoss $\mathcal{L}_{N}$ is typically expressed as the sum of losses over each sample pair. The similarity scores of positive sample pairs are maximized, while those of negative sample pairs are minimized. This helps effectively balance the trade-offs between different tasks in multi-task learning, thereby improving the generalization performance of network.

where $g_{i}$ is the fused feature of the $i$-th sample. $p_{i}$ is the representation of the positive sample corresponding to the $i$-th sample pair. $n_{j}$ is the representation of the $j$-th negative sample corresponding to the $i$-th sample pair. $sim(a,b)$ denotes the similarity measure between two representations, typically cosine similarity.

Our experimental dataset comprises authoritative datasets in the entity linking domain, including Richpedia, WikiMEL, Wikidiverse. We conduct a comprehensive analysis of the augmented dataset, and the statistics are presented in Table 2.

We select several representative methods from the current research community as our baseline: (1) BLINK (L. Wu \BOthers., \APACyear2019) is a two-step entity linking model based on the BERT model. (2) BERT (Devlin \BOthers., \APACyear2019) is a deep learning model based on the attention mechanism and Transformer architecture (3) ARNN (Eshel \BOthers., \APACyear2017) utilizes the attention-RNN structure to establish the relationship link between entities and input information. (4) DZMNED (Moon \BOthers., \APACyear2018) focuses on utilizing a multimodal attention mechanism to analyze information related to mentions in both images and text. (5) JMEL (Adjali \BOthers., \APACyear2020) leverage fully connected layers to project multimodal features into a shared latent space facilitating the representation of features. (6) MEL-HI (Zhang \BOthers., \APACyear2021) employs multiple attention mechanisms to focus on different aspects of multimodal information and decrease the effects of noisy images. (7) HieCoAtt (Lu \BOthers., \APACyear2016) is a multimodal fusion mechanism, using alternating co-attention and three textual levels (tokens, phrases, and sentences) to calculate relationship. (8) GHMFC (P. Wang, \APACyear2022) takes the gated multimodal fusion and novel attention mechanism to link entities in knowledge base. (9) MMEL (Yang \BOthers., \APACyear2023) is a joint feature extraction module to learn the representations of context and entity candidates, from both the visual and textual perspectives. (10) CLIP-text (Radford \BOthers., \APACyear2021) only uses textual information and focuses on the ability to build textual relationships between text and entity. (11) CLIP (Radford \BOthers., \APACyear2021) take both textual and visual features into consideration. The model concatenates multimodal features and calculates the similarity between fused features and entities.

For metric evaluation, we adopted the T@1, 5, 10, and 20 metrics as employed in GHMFC (P. Wang, \APACyear2022). These metrics represent the ranking of the similarity scores for candidate entities, where the linked entity’s similarity score is within the top 1, 5, 10, and 20 positions, respectively. Following the previous approach (P. Wang, \APACyear2022; Yang \BOthers., \APACyear2023), calculations are conducted among 100 candidate entities.

To be specific, following the definition in DWE (Song \BOthers., \APACyear2023), the formula is as follows:

where $N$ represents the total number of samples, and $\eta$ is the indicator function. When the receiving condition is satisfied, $\eta$ is set to 1, and 0 otherwise. $gt$ is ground truth entity while $C_{e}$ is a set of candidate entities. $cos$ means cosine similarity function. $I$ is a function to calculate the rank of similarity between joint feature $g$ and ground truth $gt$ among a set of candidate entities $C_{e}$.

Following previous work(Zhou, \APACyear2021; Wang, \APACyear2022; Song \BOthers., \APACyear2023), we select 100 potential entities as candidates. In the Wiki+, Rich+, and Diverse+ datasets, we utilize fuzzy matching ¹¹1https://github.com/seatgeek/fuzzywuzzy technology to recognize candidate entities that resemble the particular mentions.

Our experiments are conducted on RTX 3090 using PyTorch 2.0. ChatGPT is based on the GPT-3.5-turbo version. The version of CLIP employed is Vit-base-patch16-224-in21. The training consisted of 300 epochs, with both image and text hidden layer dimensions set to 512, and the output layer dimension set to 512. We utilized the AdamW optimizer with a learning rate of 5e-5, and the batch size was set to 64.

We conduct experiments with the DIM method on the datasets before dynamic enhancement to validate its effectiveness. As shown in Table 3, the performance of the DIM method surpasses the majority of existing models on three public datasets. DIM demonstrates T@1 performance improvements of 5.7%, 28.5%, and 10.3% on the Richpedia, Wikimel, and Wikidiverse datasets, respectively. This indicates that the DIM method is more effective in capturing information related to entity identity in image data.

In Table 2, to illustrate the distinctions between the enhanced datasets and the original datasets, we conducted feature statistics on the datasets and a series of experiments. As shown in Table 4, not only did we experiment with the DIM method on the enhanced datasets, but for comparison, we also replicated several classic baselines, including BERT, CLIP, and GHMFC. Our approach outperforms most existing models on the original dataset without dynamic enhancement, showcasing the effectiveness of our proposed DIM method. Furthermore, on our enhanced dataset(Rich+, Wiki+, and Diverse+), our method continues to demonstrate robust performance, validating the effectiveness of our enhancement approach. The improved entity representation by ChatGPT aligns more coherently with entities in the knowledge base, achieving better semantic consistency.