Struct-X: Enhancing Large Language Models Reasoning with Structured Data
Abstract
Structured data, rich in logical and relational information, has the potential to enhance the reasoning abilities of large language models (LLMs). Still, its integration poses a challenge due to the risk of overwhelming LLMs with excessive tokens and irrelevant context information. To address this, we propose Struct-X, a novel framework that operates through five key phases: “read-model-fill-reflect-reason” efficiently enabling LLMs to utilize structured data. It begins by encoding structured data into a topological space using graph embeddings, followed by filling in missing entity information with knowledge retrieval modules, and filtering out irrelevant tokens via a self-supervised module. The final phase involves constructing a topological network with selected tokens to further reduce the total token length for more effective LLM inference. Additionally, Struct-X includes an Auxiliary Module trained to generate prompts, aiding LLMs in analyzing structured data. Extensive experiments on benchmarks, including the knowledge graph question-answer task and the long document reading comprehension task, show that Struct-X notably improves LLM reasoning, demonstrating the effectiveness of structured data augmentation in improving LLM inference with complex input context. The code has been open-sourced and can be found in Appendix I.
Struct-X: Enhancing Large Language Models Reasoning with Structured Data
1 Introduction
In recent years, significant advancements have been made in the field of large language models (LLMs), particularly in natural language understanding Fan et al. (2023). This progress has been largely driven by extensive pre-training on vast text corpora Gao et al. (2023), which has enhanced their generation capabilities. These advancements are often viewed as critical steps towards the development of artificial general intelligence (AGI) Pei et al. (2019). During the deployment of LLMs as general-purpose assistants for a variety of real-world applications, it becomes necessary for LLMs to process multimodal inputs. Among these inputs, structured data, like structured knowledge graphs (KGs), is particularly important Ryen et al. (2022). These graphs, with their rich repository of entity relationships and hierarchical knowledge, have the potential to significantly enhance the reasoning capabilities of LLMs, leading to more precise and reliable inferences. However, in real-world applications, the effective utilization of structured knowledge in LLMs presents a significant challenge Pan et al. (2024). A common approach is to flatten the structured information into a lengthy text sequence before inputting it into LLMs Li et al. (2023). However, this method often introduces an excessive amount of task-irrelevant context. Excess information can overwhelm the models, thereby impairing inference efficiency and accuracy Han et al. (2024). Additionally, it hinders the ability of LLMs to accurately comprehend and represent the complex knowledge embedded within structured data Zhou (2023).
To address this issue, various approaches have been explored. Some studies have focused on converting knowledge graph triples into textual statements Zhang (2023), while others have emphasized incorporating knowledge graph embeddings Chen (2023). Additionally, efforts are underway to embed knowledge graph entities and relations directly into the encoder layers of LLMs Jiang (2023). More previous work is summarized in Appendix B. However, these methods primarily concentrate on converting the structural data of knowledge graphs into different formats. They tend to overlook the need to reduce the information density of this structural data, which often includes task-irrelevant information. Moreover, these approaches face challenges in preserving the global topological structure of knowledge graphs, a critical aspect that warrants further attention.
In addition to the issues of redundant information and the lack of a global topological structure in knowledge graphs, another significant challenge is the high sparsity of these graphs Lazaridou et al. (2022), characterized by missing semantic connections between entities. This sparsity presents a challenge for leveraging structural data in LLMs Hadi et al. (2023). LLMs tend to prioritize explicit semantic connections presented in the context while overlooking implicit connections, which are crucial for enhancing inference performance. Although current research, such as Lv et al. (2022) and Chai (2023), has been directed towards automatic knowledge completion and data augmentation to boost overall performance, these approaches tend to overlook the aforementioned challenges of redundancy and topological structure representation in utilizing structural data.
To overcome the existing bottlenecks discussed above, we introduce Struct-X, a novel framework designed to utilize Structured data to enhance the interaction and compleX reasoning capabilities of LLMs. This framework is centered around a workflow of “read-model-fill-reflect-reason”. It employs the transformation of structured data into a topological space, achieved through the application of graph embeddings. This is followed by the augmentation of incomplete entity information utilizing knowledge retrieval modules. Subsequently, a self-retrieved generation module called Self-Reg is employed to eliminate irrelevant tokens. The final stage encompasses the development of a topological network incorporating the chosen tokens, which serves to diminish the overall token length, thereby enhancing the efficacy of LLM inference. Furthermore, an Auxiliary Module is also designed in Struct-X, which adjusts prompts based on the loss, guiding the LLM generation. Extensive evaluation of knowledge graph QA and reading comprehension benchmarks have proven Struct-X’s superior reasoning abilities. These tests confirm that augmenting LLMs with structured data can significantly improve their inference skills in complex context environments. We refer interested readers to Appendix A for more information about Struct-X’s interaction examples. The code of Struct-X has also been open-sourced and can be found in Appendix I. The main contributions of this paper include:
-
1.
We propose a novelty framework Struct-X that implements a process of “read-model-fill-reflect-reason” on structured data, enabling LLMs to perform effective complex reasoning over structured data.
-
2.
We design a knowledge learning and filtering process to dynamically fill in structured knowledge gaps, coupled with a self-retrieved generation module called Self-Reg to filter and verify the relevance of retrieved knowledge, retaining valuable token information to alleviate learning burdens on LLMs.
-
3.
We construct specialized graph network encoders to fully learn the potential features of associated tokens and enable efficient cross-layer message passing in Transformers. We also devise an original Auxiliary Module for generating coherent prompts and improving answer responses.
2 Preliminaries
The task of text generation in LLMs involves creating a sequence of output , where represents the total number of tokens Tang et al. (2023), based on a given input prompt . This process is often modeled in an autoregressively manner, which estimates the likelihood of each token, where represents the tokens that come before the current sequence Zhang (2022). Enhancements to this process can be made by incorporating relevant information from external documents into the input, thereby refining the model’s predictions Hofstätter (2023).
![Refer to caption](extracted/5737542/1.png)
Moreover, we can develop a novel decoding strategy that produces critique tokens alongside the main text output. These tokens are generated at each step and are designed to enable the LLMs to self-evaluate aspects such as relevance, factuality, and completeness of the generated content in Table 1.
(1) |
where the critique token depends on all preceding text and critiques. We define four types of critique tokens: IFReT - predicts if retrieval is needed, IFReL - assesses passage relevance, IFSuP - checks output is supported and IFUsE - decides whether it is useful.
These critique tokens enable better control of the decoding process through re-ranking or constraints Asai (2023). For instance, the probability of a desirable IFReL token can upweight certain outputs. As an example, the attention distillation critique token is computed between the input and response . This critiques the attention alignment between and . By generating such reflective signals, IFReT can adapt its decoding strategy over time based on its critiques. The IFReT approach allows customization of model behavior through constraints on desired critique tokens. The detailed algorithm implementation process can be found in Appendix E.
, which predicts whether passage retrieval is needed () given the input using a scoring function parameterized by . The relevance scoring between a passage and the input is , where produces a relevance score modulated by the IFReL gate value. The factual consistency between a response and passage is evaluated by , where is element-wise production and is the sigmoid activation function. The overall utility is decided using .
3 Methods
3.1 Topological Knowledge Injection
We first implement “read-model-fill” process and we start by processing input KGs using a graph attention encoder (GAE) that consists of layers Xu et al. (2021). The initial node features, denoted as , are set up using information obtained from the KG completion module Fei et al. (2021). After processing through layers, we obtain the final node embeddings, , which effectively represent both the semantic and structural information of the KGs. These encoded graph embeddings, , are then partially masked at a specific rate, , to assist in learning about missing knowledge. This masking process can be mathematically represented as , where symbolizes the masking operation. The masked nodes, denoted as , are then fed into the knowledge retrieval module, , which is explained in the following section. This module plays a crucial role in supplementing the missing information, thereby facilitating the generation of complete graph embeddings Reda et al. (2022).
![Refer to caption](x1.png)
To address the gaps in entity information within the structured knowledge graph, we have developed a knowledge learning module, denoted as . This module is designed to retrieve pertinent facts from the knowledge base to enhance the masked node embeddings, Yasunaga et al. (2022). More specifically, for each masked node, we calculate a similarity score between its embedding and all tail entities that are part of the set . This is achieved using the scoring function , which can be represented as:
(2) |
This process enables us to efficiently fill in the missing information in the knowledge graph.
Typ | Inputs | Outputs | Descriptions |
---|---|---|---|
IFReT |
Decides when extra facts can assist reasoning | ||
\hdashline
IFReL |
Whether evidence provides useful clues to solve query | ||
\hdashline
IFSuP |
Alignment between statements in response and evidence | ||
\hdashline
IFUsE |
Usefulness score of response in answering query |
The scoring function evaluates both feature and topological similarities within the graph in Figure 2. It selects the top entities based on the highest scores and retrieves the related facts from the knowledge base. To incorporate these facts into the node embeddings, a relation-aware aggregation function is used. This function accumulates relevant knowledge for each node, using a score threshold to filter out irrelevant facts Yu et al. (2022). The aggregation function adeptly manages various relations in structured knowledge by considering information from retrieved triples in a relation-aware manner. Additionally, before being input into the Transformer encoder, one linear layer concatenates and processes embeddings from all GAT layers.
(3) | ||||
The merges inputs and reduces dimensionality. This process retains rich multi-scale structural and semantic features at various depths. The output is flattened and prepared into sequences to replace token embeddings for the Transformer encoder input, as suggested by Wang et al. (2021). The refined node embeddings , enriched with retrieved entity information, supply additional knowledge for reasoning in the downstream LLMs.
3.2 Knowledge and Information Retrieval
Here we perform the “reflect” process. The module retrieves relevant knowledge absent in masked graph node inputs . Related entities can be dynamically discovered by matching tail entities to each masked node using a similarity scoring function considering both feature and topological similarity Lewis (2020). Related facts are recalled to fill gaps. After concatenating retrieved knowledge sequences for all nodes ordered by similarity scores, we employ a pruning algorithm leveraging multi-head self-attention distillation and thresholds to filter out lower-weighted tokens. The remaining dense sequence provides supplemental external knowledge to complete masked graph node inputs.
To filter and verify the relevance of retrieved knowledge, we design a self-retrieved generation module parameterized by that takes as input the retrieved knowledge sequences and outputs a filtered subset containing only the most valuable tokens Shuster (2021). Specifically, first encodes the knowledge sequence using a Transformer encoder to obtain representations . Next, we compute an importance score for each token , where is a scoring network and is a sigmoid activation function. To train the scoring network in a self-supervised manner, we create corrupted knowledge sequences by randomly masking or shuffling some tokens. A contrastive loss is implemented to assign higher scores to tokens from the original versus corrupted :
(4) |
where is a margin hyperparameter. This drives the model to identify the most valuable knowledge. Finally, we filter the sequence by discarding tokens scoring below a threshold of to retain only the most relevant phrases, significantly reducing the learning burden when provided as supplements to the LLMs.
(5) |
The filtered relevant knowledge provides targeted assistance to improve reasoning without overwhelming the LLMs with extraneous and irrelevant information.
3.3 Graph Topology Encoder
Here we perform the “reason” phase. To capture semantic and structural interactions between entities within the KGs, we use a specialized graph encoder in Figure 3, denoted as , which is parameterized by . This KG is represented as , where is the set of node entities and is the set of relation edges Li (2022). For each entity node in , we first derive its initial feature representation , which is a vector in a high-dimensional space.
The graph encoder works through a series of layers, each layer enhancing the node representations through message passing. This process can be described as:
(6) |
where refers to the neighboring nodes of , and is a function that aggregates information from these neighbors to update the node’s embedding.
To focus on the most relevant semantic connections, we use a graph self-attention layer within . This layer calculates attention weights as follows , where and are derived from the embeddings of the nodes. This method allows the model to selectively emphasize the most informative signals from neighboring nodes Cui et al. (2020).
After processing through layers, we obtain refined node embeddings , which encapsulate both semantic and structural information of the graph. To make these embeddings more manageable for downstream tasks, we compress them through a trainable down-projection layer:
(7) |
This step reduces the dimensionality of the embeddings to , which is smaller than the original . The resulting condensed embeddings still retain crucial token-level interactions but are more concise, making them better suited for training models for specific tasks. This approach ensures that while the size of the input sequence is significantly reduced, the essential semantic and structural features of the knowledge graph are preserved for subsequent reasoning.
3.4 Auxiliary Module
To further guide the LLMs in effectively reasoning over the structured input with a knowledge graph, we have developed an Auxiliary Module. This module is designed to create dynamic prompts that enhance the coherence of answers generated by LLMs. It functions by analyzing the LLM’s predicted answer, denoted as , along with the current loss, . Based on these inputs, it generates a refined prompt, , which is then used for a new round of inference. We use the pre-trained Bert model (i.e., bert-base-NER) Devlin et al. (2018) to construct this Auxiliary Module, symbolized as and parameterized by . This generator crafts the prompt text, taking into account the input values . The generator is trained jointly with the overall system using policy gradient methods to maximize the expected reward of producing coherent answers: , .
This reward function is designed to encourage the LLMs to generate responses that are not only fluent but also logically consistent, particularly when using the updated prompt. This feature enables the module to dynamically adjust prompts based on the current performance, thereby offering new approaches to improve the quality of answers. Throughout the training process, the module progressively learns to produce more effective prompts, leading to enhanced accuracy and coherence in the LLM’s reasoning. For more detailed experimental testing and analysis, please refer to Appendix G.
![Refer to caption](extracted/5737542/TL.png)
4 Experiment
4.1 Datasets and Tasks
We assess the performance of our proposed Struct-X framework on four open-source benchmark datasets designed for knowledge graph reasoning and multi-hop reasoning abilities on graphs.
Task1:WebQSP contains 4,737 QA pairs where the questions require logical reasoning over a knowledge graph derived from Wikipedia to infer the correct answer. The knowledge graph consists of 5,719 entities and 2,150 relations.
Task2:MetaQA comprises a set of more complex compositional questions constructed from an underlying knowledge graph with a vocabulary of 300 entities and 100 relations. It has a total of 1,200 unique questions that test the multi-hop, logical, and comparative reasoning abilities of models.
Task3:Family Tree Age Consider a family tree , where each individual in is associated with a description specifying their age. The objective of this task is to identify the triplet comprising an individual, one of their grandparents, and a grand-uncle/grand-aunt by marriage that collectively has the highest cumulative age.
Task4:Travel Route Optimization Let be a graph representing connected cities, where each city in has a description with the travel toll or tax. The LLM must plan the route from a source to a destination city that minimizes the total toll paid.
For all datasets, we incorporate the encoded graph representations into Llama2, which has been pre-trained on BookCorpus and English Wikipedia. Appendix C is a case analysis of the experimental results on the datasets.
Model/Datasets | WebQSP | MetaQA | Family Tree | Travel Route | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Acc. | Prec. | Rec. | F1 | Acc. | Prec. | Rec. | F1 | Acc. | Prec. | Rec. | F1 | Acc. | Prec. | Rec. | F1 | |
Embedding-based Model | ||||||||||||||||
TransE | 67.32 | 68.41 | 65.47 | 66.91 | 74.63 | 74.44 | 75.01 | 74.72 | 73.45 | 71.83 | 66.92 | 63.84 | 62.27 | 60.45 | 65.28 | 62.56 |
DistMult | 69.21 | 69.91 | 69.08 | 69.49 | 69.34 | 72.32 | 61.93 | 62.83 | 60.93 | 61.47 | 60.08 | 60.62 | 61.74 | 60.36 | 64.19 | 62.05 |
EmbedKGQA | 63.26 | 72.38 | 74.33 | 73.35 | 70.17 | 70.38 | 69.57 | 69.97 | 65.62 | 64.31 | 68.94 | 65.92 | 64.95 | 63.01 | 68.64 | 65.37 |
ComplEx | 67.36 | 68.21 | 66.27 | 65.64 | 72.86 | 71.64 | 71.18 | 69.53 | 59.83 | 58.45 | 62.74 | 60.27 | 57.45 | 55.92 | 60.93 | 57.84 |
RotatE | 74.55 | 72.68 | 76.93 | 74.77 | 78.19 | 78.16 | 78.22 | 77.19 | 67.63 | 66.28 | 65.79 | 64.37 | 62.64 | 63.01 | 61.83 | 62.27 |
Open-source LLM | ||||||||||||||||
Llama27B | 27.13 | 29.17 | 25.23 | 27.36 | 26.45 | 25.73 | 28.82 | 26.94 | 32.81 | 34.73 | 30.16 | 32.26 | 25.92 | 27.84 | 23.68 | 25.47 |
Alpaca7B | 33.56 | 31.83 | 36.29 | 33.72 | 22.74 | 25.93 | 19.82 | 21.45 | 28.36 | 26.92 | 31.74 | 28.91 | 39.45 | 41.27 | 37.63 | 39.18 |
Llama213B | 29.83 | 28.73 | 31.91 | 39.95 | 35.92 | 36.19 | 36.84 | 35.91 | 31.45 | 33.82 | 39.27 | 31.28 | 36.37 | 38.62 | 34.93 | 36.45 |
Alpaca13B | 38.37 | 40.92 | 36.74 | 38.45 | 42.83 | 33.62 | 31.74 | 32.56 | 40.83 | 32.76 | 38.84 | 40.37 | 37.92 | 35.74 | 40.83 | 37.94 |
ChatGPT | 41.27 | 49.84 | 44.92 | 41.72 | 38.74 | 37.56 | 40.83 | 38.94 | 35.92 | 33.84 | 38.62 | 35.83 | 34.83 | 36.73 | 42.15 | 44.71 |
LLM-based Fine-tuning | ||||||||||||||||
KG-LLaMA | 42.45 | 43.28 | 40.39 | 41.37 | 46.28 | 45.94 | 47.36 | 46.47 | 49.74 | 48.36 | 42.64 | 40.27 | 48.56 | 50.28 | 45.74 | 47.83 |
KG-Alpaca | 48.92 | 47.74 | 51.83 | 49.45 | 43.56 | 42.41 | 46.28 | 43.94 | 45.92 | 46.84 | 44.56 | 45.62 | 42.91 | 50.74 | 47.28 | 43.45 |
KG-BERT | 56.28 | 55.94 | 57.36 | 56.47 | 70.92 | 69.74 | 63.82 | 61.45 | 63.27 | 61.19 | 68.29 | 64.23 | 74.32 | 75.61 | 74.45 | 72.37 |
PKGC | 64.56 | 67.44 | 62.64 | 64.10 | 77.79 | 76.92 | 74.27 | 73.49 | 79.32 | 78.41 | 75.36 | 74.91 | 76.35 | 75.42 | 74.37 | 76.91 |
Vanilla IT | 65.62 | 69.86 | 66.29 | 65.54 | 78.28 | 74.31 | 72.98 | 73.62 | 71.57 | 70.58 | 69.63 | 70.05 | 65.94 | 67.32 | 63.69 | 64.96 |
KoPA | 72.48 | 72.82 | 71.64 | 71.52 | 75.82 | 74.69 | 73.95 | 71.58 | 73.56 | 72.18 | 72.08 | 71.53 | 73.13 | 75.05 | 71.12 | 71.24 |
Ours(Struct-X) | 75.13 | 73.40 | 77.25 | 75.29 | 79.63 | 78.27 | 77.53 | 76.61 | 82.68 | 82.95 | 79.34 | 78.92 | 81.69 | 81.53 | 78.62 | 78.04 |
Methods | WQSP |
|
|
|
||||||
---|---|---|---|---|---|---|---|---|---|---|
KV-Mem | 48.3 | 84.3 | 74.5 | 46.2 | ||||||
GraftNet | 53.1 | 84.1 | 72.3 | 62.3 | ||||||
EmbedKGQA | 63.3 | 85.3 | 74.1 | 79.8 | ||||||
NSM | 69.6 | 84.6 | 88.7 | |||||||
UniKGQA | 86.8 | 82.8 | 92.3 | |||||||
StructGPT | 64.8 | 87.1 | ||||||||
Ours | ||||||||||
SI (struct inference) | 54.2 | 85.6 | 85.3 | 86.5 | ||||||
KGP (KG perform better) | 56.4 | 86.9 | 87.5 | 89.8 | ||||||
model (graph study) | 61.6 | 87.4 | 89.2 | 85.7 | ||||||
encoder (location) | 69.3 | 88.1 | 76.6 | 79.3 | ||||||
self-reg (correction) | 70.4 | 85.3 | 87.4 | 86.8 |
4.2 Implementation Details
Baseline
- •
- •
-
•
LLM-based Fine-tuning: To assess the performance of LM fine-tuning approaches, we include as baselines KG-LlaMA and KG-Alpaca Yao et al. (2023b), KG-BERT Yao et al. (2019), PKGC Lv et al. (2022) and vanilla IT Zhang (2023) which incorporate techniques to enhance LMs using annotated KG datasets or self-supervision.
Our implementation is in PyTorch and we run experiments on NVIDIA A100 GPUs. More details of training can be found in Appendix D.
4.3 Main Results
The results presented in Table 2 indicate that Struct-X consistently outperforms existing baseline methods across various datasets. Specifically, in the WebQSP benchmark, Struct-X achieves an accuracy of 75.13%, which is 2.65% higher than the previously best-performing method, KoPA. Additionally, Struct-X shows modest improvements in precision and recall, with increases of 9.51% and 10.96%, respectively, compared to Vanilla IT. In the more challenging MetaQA dataset, Struct-X’s performance is notably better, surpassing the state-of-the-art accuracy scores by 1.84% and achieving a 1.68% higher precision. Furthermore, Struct-X demonstrates significant advancements in specialized tasks such as Family Tree and Travel Route, where it exceeds the top baseline results by 3.36% and 5.34% in accuracy, respectively.
Compared to embedding models such as TransE, DistMult, and EmbedKGQA, Struct-X also shows promising improvements in reasoning abilities by integrating both semantic and topological structures of knowledge graphs. For instance, against RotatE’s accuracy of 74.55% on the WebQSP dataset, Struct-X achieves higher performance with a 75.13% accuracy, an increase of 0.58%. The difference is slightly more pronounced on the MetaQA dataset, where Struct-X exceeds RotatE’s score of 78.19% by 1.44% in accuracy. In scenarios requiring complex reasoning inferences, Struct-X demonstrates enhanced capabilities, outperforming peak embedding model accuracy by a notable margin of 16.74% in Task4.
The results also show that Struct-X can enhance the capabilities of the Llama2, which itself achieves a 27.13% accuracy on the WebQSP benchmark. This enhancement is achieved through masking graph embeddings and using topology matching to retrieve relevant facts, thus addressing the gaps in factual knowledge that Llama2 requires. By overcoming these deficiencies in LLMs, Struct-X significantly improves performance, increasing accuracy by 47.4%. This indicates the effectiveness of structured augmentation, which is not present in the Llama2. Further, Struct-X filters out less important tokens using the Self-Reg module, ensuring focus on the most relevant information. In comparison to previous methods like KG-BERT fine-tuning, StructX offers essential enhancements, particularly in complex reasoning tasks, as evidenced by increases of up to 10.24% in accuracy and 5.61% in recall.
Based on the experimental results, the “reflect” process also plays a crucial role in enhancing reasoning capabilities. This process involves the IFReT , which selectively gathers evidence as needed, and the IFReL , which filters less relevant passages using relevance scores from Eq.3 to improve context for LLMs. Additionally, the IFSuP and IFUsE ensure passage-response consistency and assess overall utility, contributing to higher quality results. For further case studies and experiments on this topic, readers are directed to Appendix H.
4.4 Ablation Study
4.4.1 Q1: Different functional modules
Table 3 shows that each component of Struct-X plays a crucial role in enhancing various reasoning capabilities. For 1-hop single fact questions, while all versions of Struct-X are effective, the complete model excels with a 91.3% accuracy due to its ability to perform combinatorial reasoning using multi-head attention. This is key for interpreting semantic connections. In 2-hop and 3-hop multi-step reasoning, the absence of knowledge retrieval and injection modules results in a significant performance drop, with decreases of 7.4% and 7.3% respectively. However, the full Struct-X model, utilizing these modules, reaches 92.7% and 93.8% accuracy by effectively traversing distant nodes. The graph topology encoder also proves vital; its omission leads to a 5.8% decline in location-based reasoning, highlighting its importance in connecting nodes and facilitating spatial/hierarchical reasoning through message passing. Furthermore, the lower accuracy without the Auxiliary Module underlines its utility in guiding coherent inference across multiple steps.
Model Variant | WebQSP | MetaQA | ||||
---|---|---|---|---|---|---|
Acc. (%) | Rec. (%) | F1 (%) | Acc. (%) | Rec. (%) | F1 (%) | |
No Filtering | ||||||
Random (20% cut) | ||||||
Random (40% cut) | ||||||
Random (60% cut) | ||||||
Self-Reg (20% cut) | ||||||
Self-Reg (40% cut) | ||||||
Self-Reg (60% cut) |
4.4.2 Q2:Filtering and reflection mechanism
Table 4 compares reasoning performance with the following variants: : Directly injects all retrieved knowledge without filtering. : Randomly removes of retrieved tokens. : Uses the proposed Self-Reg module to score and filter tokens.
Across WebQSP and MetaQA datasets, incorporating filtering mechanisms leads to consistent gains over no filtering baselines. Randomly removing tokens brings minor improvements, showing that some knowledge reduction is beneficial. However, learned filtering with Self-Reg leads to more substantial gains. Comparing different Self-Reg cutting ratios, 40% filtering seems to achieve the optimal trade-off, maximizing accuracy and recall. More aggressive 60% cutting starts to degrade performance likely due to removing pertinent facts. On the other hand, light 20% filtering retains more distracting information. By balancing knowledge breadth and depth, 40% Self-Reg filtering enhances language model inference without overwhelming models. By scoring and removing extraneous tokens based on contextual representations, Self-Reg retains the essence to augment language models without diverting attention.
4.4.3 Q3: Learning by Auxiliary Module
The results in Table 5 demonstrate that incorporating the Auxiliary Module leads to significant performance gains over the base Struct-X model without this component. We observe absolute improvements of 3.9% in accuracy, 2.58% in precision, and 5.72% in recall after implementing the Auxiliary Module. This validates its efficacy in providing adaptive prompts that elicit more accurate and logically coherent reasoning from the LLM when inference is made over structured knowledge graphs. The gains over the previous best model, PKGC are also substantial, at 10.57% higher accuracy. Hence, the auxiliary module proves important for multi-hop reasoning and steering deductions in the right direction over complex topological structures. The consistent benefits confirm that modeling explicit prompt-answering mechanisms customized for structured reasoning tasks is an effective approach.
Model | Accuracy (%) | Precision (%) | Recall (%) |
---|---|---|---|
KG-BERT | |||
PKGC | |||
Struct-X w/o Auxiliary Module | |||
Struct-X |
4.4.4 Q4:Knowledge injection variants
To validate the contributions of different components of our knowledge injection mechanism, we conduct an ablation study with the following variants: : The base LLM (i.e., Llama2) without any graph representation injection. : Encoded graph embeddings are directly injected without any masking or knowledge retrieval. : Graph embeddings are masked but missing facts are not filled via retrieval. : Masked embeddings are completed with the knowledge retrieval module but without graph encoding. We compare reasoning performance on WebQSP and MetaQA benchmarks against these reduced injection variants. The results in Table 6 demonstrate clear improvements from collectively incorporating all knowledge injection components compared to ablated variants. The full Struct-X model with topological encoding, masking, and retrieval achieves 1.68% and 1.31% higher accuracy over the best partial variant on WebQSP and MetaQA respectively. This confirms that each mechanism provides unique benefits - topological encoding better retain intricate connections, masking identifies missing facts, and retrieval fills knowledge gaps. The experiment proves that dynamic masking and retrieval to address inherent incompleteness in structured data are most impactful. Variants without these processes show worse performance as they fail to overcome language models’ factual deficiencies.
Model Variant | WebQSP Accuracy (%) | MetaQA Accuracy (%) |
---|---|---|
No Injection | ||
Embeddings Only | ||
Masking Only | ||
Retrieval Only | ||
Struct-X |
5 Conclusion
In this paper, we introduce Struct-X, a groundbreaking framework designed to enhance LLMs in complex reasoning tasks. Struct-X applies an efficient “read-model-fill-reflect-reason” methodology to structured data. It is adept at learning graph embeddings that are sensitive to geometric contexts, capturing the content of entities as well as their topological relationships. This enables Struct-X to effectively infer missing facts about entities by matching similar topological features. Furthermore, it enhances the LLMs by distributing multi-scale features, which bolsters the representation of underlying connections that are not explicitly apparent. Struct-X excels in tasks such as knowledge graph-based QA tasks and reading comprehension, especially in scenarios that require multi-hop logical reasoning.
6 Limitiation
The knowledge graph encoding may not fully capture complex relationships beyond structural topology, the auxiliary module’s prompting could be overly biased by the current loss landscape. Exploring more expressive graph representations and smarter prompting strategies could potentially address these limitations.
References
- Asai (2023) Akari Asai. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511.
- Banerjee et al. (2020) Chaity Banerjee, Tathagata Mukherjee, and Eduardo Pasiliao. 2020. Feature representations using the reflected rectified linear unit (rrelu) activation. Big Data Mining and Analytics, 3(2):102–120.
- Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In NIPS, pages 2787–2795.
- Brody et al. (2021) Samuel Brody, Uri Alon, and Eran Yahav. 2021. How attentive are graph attention networks? arXiv preprint arXiv:2105.14491.
- Chai (2023) Ziwei Chai. 2023. Graphllm: Boosting graph reasoning ability of large language model. arXiv preprint arXiv:2310.05845.
- Chen (2023) Zhikai Chen. 2023. Exploring the potential of large language models (llms) in learning on graphs. arXiv preprint arXiv:2307.03393.
- Cui et al. (2020) G. Cui, J. Zhou, C. Yang, and Z. Liu. 2020. Adaptive graph encoder for attributed graph embedding. In SIGKDD, pages 976–985.
- Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR.
- Fan et al. (2023) A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang. 2023. Large language models for software engineering: Survey and open problems. arXiv preprint arXiv:2310.03533.
- Fei et al. (2021) H. Fei, Y. Ren, Y. Zhang, D. Ji, and X. Liang. 2021. Enriching contextualized language model from knowledge graph for biomedical information extraction. Briefings in bioinformatics, 22(3):bbaa110.
- Gao et al. (2023) L. Gao, Z. Dai, and et al. 2023. Rarr: Researching and revising what language models say, using language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16477–16508.
- Hadi et al. (2023) M. U. Hadi, R. Qureshi, A. Shah, M. Irfan, A. Zafar, M. B. Shaikh, and S. Mirjalili. 2023. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints.
- Han et al. (2024) J. Han, J. Kang Lu, Y. Xu, J. You, and B. Wu. 2024. Intelligent practices of large language models in digital government services. IEEE Access.
- Harnoune (2021) Ayou Harnoune. 2021. Bert based clinical knowledge extraction for biomedical knowledge graph construction and analysis. Computer Methods and Programs in Biomedicine Update, (1):100042.
- Hofstätter (2023) Sebastian Hofstätter. 2023. Fid-light: Efficient and effective retrieval-augmented text generation. In ACM SIGIR.
- Honovich (2022) Or Honovich. 2022. True: Re-evaluating factual consistency evaluation. arXiv preprint arXiv:2204.04991.
- Jiang (2023) Jinhao Jiang. 2023. Structgpt: A general framework for large language model to reason over structured data. arXiv preprint arXiv:2305.09645.
- Khan (2022) Salman Khan. 2022. Transformers in vision: A survey. ACM Computing Surveys (CSUR), 54(10s):1–41.
- Lazaridou et al. (2022) A. Lazaridou, E. Gribovskaya, W. Stokowiec, and N. Grigorev. 2022. Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115.
- Lewis (2020) Patrick Lewis. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474.
- Li (2021a) Ce Li. 2021a. Memory attention networks for skeleton-based action recognition. IEEE Transactions on Neural Networks and Learning Systems, 33(9):4800–4814.
- Li et al. (2023) Lei Li, Yongfeng Zhang, and Li Chen. 2023. Prompt distillation for efficient llm-based recommendation. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management.
- Li (2022) Yang Li. 2022. Dual encoder-based dynamic-channel graph convolutional network with edge enhancement for retinal vessel segmentation. IEEE Transactions on Medical Imaging, 41(8):1975–1989.
- Li (2021b) Yawei Li. 2021b. Localvit: Bringing locality to vision transformers. arXiv preprint arXiv:2104.05707.
- Liu (2020) Weijie Liu. 2020. K-bert: Enabling language representation with knowledge graph. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34.
- Liu (2022) Xiao Liu. 2022. Selfkg: Self-supervised entity alignment in knowledge graphs. In Proceedings of the ACM Web Conference 2022.
- Lv et al. (2022) Xin Lv, Yankai Lin, Yixin Cao, and et al. 2022. Do pre-trained models benefit knowledge graph completion? a reliable evaluation and a reasonable approach. In ACL (Findings), pages 3570–3581. Association for Computational Linguistics.
- Pan et al. (2024) S. Pan, L. Luo, Y. Wang, C. Chen, J. Wang, and X. Wu. 2024. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering.
- Pan (2024) Shirui Pan. 2024. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering.
- Pei et al. (2019) Jian Pei, Lei Deng, Shuai Song, and Ming Zhao. 2019. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 572(7767):106–111.
- Reda et al. (2022) Haftu Tasew Reda, Adnan Anwar, and Abdun Mahmood. 2022. Comprehensive survey and taxonomies of false data injection attacks in smart grids: attack models, targets, and impacts. Renewable and Sustainable Energy Reviews, 163:112423.
- Roux (2022) Claude Roux. 2022. The sydney declaration–revisiting the essence of forensic science through its fundamental principles. Forensic Science International, 332:111182.
- Ryen et al. (2022) Vincent Ryen, Ahmet Soylu, and Dumitru Roman. 2022. Building semantic knowledge graphs from (semi-) structured data: a review. Future Internet, 14(5):129.
- Saxena et al. (2020) Apoorv Saxena, Aditay Tripathi, and Partha P. Talukdar. 2020. Improving multi-hop question answering over knowledge graphs using knowledge base embeddings. In ACL 2020, Online, July 5-10, 2020, pages 4498–4507.
- Shah (2020) Koonal K. Shah. 2020. An exploration of methods for obtaining 0= dead anchors for latent scale eq-5d-y values. The European Journal of Health Economics, 21:1091–1103.
- Shuster (2021) Kurt Shuster. 2021. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567.
- Sun et al. (2019) Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019. Rotate: Knowledge graph embedding by relational rotation in complex space. In ICLR (Poster).
- Tang et al. (2023) Ruize Tang, Xianpei Han, Xia Jiang, and Xiaohua Hu. 2023. Does synthetic data generation of llms help clinical text mining? arXiv preprint arXiv:2303.04360.
- Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Pierre Albert, Amr Almahairi, Yannis Babaei, …, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
- Trouillon et al. (2016) Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In ICML, pages 2071–2080. JMLR.org.
- Tu (2021) Ke Tu. 2021. Conditional graph attention networks for distilling and refining knowledge graphs in recommendation. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management.
- Wang et al. (2021) Changgang Wang, Jun An, and Gang Mu. 2021. Power system network topology identification based on knowledge graph and graph neural network. Frontiers in Energy Research, 8:613331.
- Wu (2020) Zonghan Wu. 2020. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 32(1):4–24.
- Xu et al. (2021) H. Xu, W. Xia, Q. Gao, J. Han, and X. Gao. 2021. Graph embedding clustering: Graph attention auto-encoder with cluster-specificity distribution. Neural Networks, 142:221–230.
- Xue (2020) Xiaoming Xue. 2020. Affine transformation-enhanced multifactorial optimization for heterogeneous problems. IEEE Transactions on Cybernetics, 52(7):6217–6231.
- Yang et al. (2015) Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In ICLR (Poster).
- Yang (2020) Liang Yang. 2020. Graph attention topic modeling network. In Proceedings of the Web Conference.
- Yao et al. (2019) Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Kg-bert: Bert for knowledge graph completion. CoRR, abs/1909.03193.
- Yao et al. (2023a) Liang Yao, Jiazhen Peng, Chengsheng Mao, and Yuan Luo. 2023a. Exploring large language models for knowledge graph completion. CoRR, abs/2308.13916.
- Yao et al. (2023b) Liang Yao, Jiazhen Peng, Chengsheng Mao, and Yuan Luo. 2023b. Exploring large language models for knowledge graph completion. CoRR, abs/2308.13916.
- Yasunaga et al. (2022) M. Yasunaga, A. Bosselut, and H. et al. Ren. 2022. Deep bidirectional language-knowledge graph pretraining. In Advances in Neural Information Processing Systems, volume 35, pages 37309–37323.
- Yu et al. (2022) D. Yu, C. Zhu, Y. Yang, and M. Zeng. 2022. Jaket: Joint pre-training of knowledge graph and language understanding. In AAAI, volume 36, pages 11630–11638.
- Zhang (2023) Yichi Zhang. 2023. Making large language models perform better in knowledge graph completion. arXiv preprint arXiv:2310.06671.
- Zhang (2022) Yizhe Zhang. 2022. Retgen: A joint framework for retrieval and grounded text generation modeling. In AAAI, volume 36.
- Zhang et al. (2020) Zhiqing Zhang, Fuzhen Zhuang, Hengshu Zhu, Zhiping Shi, Hui Xiong, and Qing He. 2020. Relational graph neural network with hierarchical attention for knowledge graph completion. In AAAI, volume 34, pages 9612–9619.
- Zhou (2023) Bin Zhou. 2023. Semantic-aware event link reasoning over industrial knowledge graph embedding time series data. International Journal of Production Research, 61(12):4117–4134.
Appendix A StructX Interaction Examples
Appendix B Related Work
Prior efforts have explored various techniques to enhance language models with structured knowledge. Pan et al. Pan (2024) directly converted knowledge graph triples into textual statements as inputs. Liu et al. Liu (2020) embedded knowledge graphs into hidden layers of BERT models. Harnoune et al.Harnoune (2021) utilized memory modules to provide facts for multi-hop reasoning. Wu et al. Wu (2020) designed graph recurrent networks to propagate information over knowledge graphs. Roux et al.Roux (2022) employed reinforcement learning for path-finding in knowledge-augmented environments. Tu et al. Tu (2021) used graph convolutional networks to distill subgraph knowledge. Liu et al. Liu (2022) pruned knowledge graph inputs via self-supervised learning objectives. Honovich et al.Honovich (2022) focused on improving factual consistency in conditional text generation. Shah et al. Shah (2020) iteratively expanded anchor sets for more comprehensive grounding. Xue et al. Xue (2020) optimized path selection on heterogeneous knowledge sources. While providing valuable techniques, existing methods lack consideration for efficiently filtering and condensing structured knowledge supplements tailored for language model integration. Our work aims to address these limitations through specialized contextual encoding, retrieval, and distillation components in the proposed Struct-X framework.
Appendix C Case Study
In this family tree example, Struct-X first encodes the hierarchical relations between members like “John is Nikhil’s paternal grandfather" and “Meera is John’s daughter" as structured knowledge. The connections allow tracing ancestry and descendent paths. Age details like “John was born in 1950" are also injected. With encoded topology and dates, Struct-X can traverse the tree to identify the oldest member, Shekhar and the youngest, Nikhil, before subtracting their years to find the 54-year difference. This showcases interpreting familial links and performing numerical reasoning over injected facts. By attending over neighborhood nodes, Struct-X focuses on the most relevant semantic connections. The topology learning distinguishes close and distant relatives to assess ages. Retrieved date details fill knowledge gaps for calculation. Overall, this case validates Struct-X’s capabilities in encoding complex structure topology and performing multi-step inference by combining connection reasoning and data-driven deduction.
The example proves Struct-X can encode intricate hierarchical structures and use encoded topology to trace relationships and inject valuable factual knowledge. By learning contextual representations and connections in structured data, Struct-X successfully interprets semantic links between entities and integrates supplementary date details for numerical reasoning over multiple inference steps. This supports complex reasoning across topological dimensions.
Appendix D Experimental Parameter Settings
The Variable Description Details in Table 7 and hyperparameters in Table 9 provide concrete configuration details for Struct-X when evaluated on the four benchmark datasets. We can observe some key modeling choices - all models use a 4-layer graph encoder to learn topological representations, apply 30-40% node masking for knowledge gap simulation, and dedicate 256 dimensions to the Auxiliary Module for steering prompt/answer generation. Training hyperparameters are also shown, including batch sizes of 16-32, learning rates around 1e-4, and 10-20 training epochs. The number of tunable parameters indicates comparable model complexity across datasets.
Variables | Description |
---|---|
Knowledge graph | |
Node/entity set | |
Edge/relation set | |
Head, relation, tail | |
Node feature at layer | |
Neighbor nodes | |
Aggregates neighbor info | |
Activation function | |
Similarity score | |
Masked node embedding | |
Aggregates retrieved facts | |
Scores token relevance | |
Contrastive loss | |
Text generation distribution | |
, | Relevance and consistency scores |
Appendix E Self-Reg Module
IFReT Module This module decides if passage retrieval is needed using a scoring function:
(8) |
Where is the input text, and outputs a binary decision on whether to activate retrieval given , parameterized by . For example, if the input is : "Tell me more about Van Gogh’s paintings", the module may predict , indicating that retrieval would be useful to supplement details about Van Gogh’s works.
IFReL Module This module scores the relevance of a retrieved passage using:
(9) |
Where produces a relevance score between input text and passage , modulated by the IFReL gate value passed through a sigmoid . For instance, if a retrieved passage discusses Surrealism instead of Van Gogh, the model can set a lower IFReL score to downweight it.
IFSuP Module This evaluates the factual consistency between response and passage :
(10) |
Where is element-wise production, calculates consistency between and , controlled via IFSuP . This helps verify if details in like dates or places align with the evidence in .
IFUsE Module This directly outputs a usefulness score between input and response :
(11) |
For example, may be lower if fails to answer the query in about Van Gogh’s paintings. The modules apply self-supervision for relevance, coherence, and consistency.
![Refer to caption](extracted/5737542/a1.png)
The IFReT () module for selective passage retrieval plays a key role in improving accuracy by retrieving evidence only when needed, avoiding unnecessary information. For instance, in closed-domain QA, Struct-X achieves higher recall by learning a tight retrieval threshold via IFReT (), while open-ended generation benefits from more selective retrieval. Furthermore, the IFReL (,) module filters out lower-quality passages that are less relevant, as quantified by the relevance score in Eq.3 and calibrated by IFReL gates. This enhances the contextual signals passed to the language model. The IFSuP (,) and IFUsE (,) critiques help further verify passage-response consistency and overall utility, ensuring higher quality outputs.
Figure 4 shows the performance of the model before and after applying different levels of knowledge filtering in question-answering comprehension tasks. The filtering ratio varies between 0 and 60%, and the improvement in fit between the response of the large language model and the real-world answer is used as the criterion for determining the effectiveness. Firstly, when there is no filtering (0%), the fitting degree R2 is around 0.7. This is the original level when injecting all knowledge. Subsequently, we observed that as the filtering ratio increased, the fitting degree R2 showed a trend of first increasing and then decreasing. When filtering out about 40% of low correlation knowledge, the model accuracy reaches a peak of around 0.82. This indicates that through algorithms such as Self Reg, the model has learned to recognize the most critical knowledge for the current question and answer. Overfiltering knowledge actually makes the model unable to learn comprehensively. However, continuing to increase the filtration ratio to 40-60% will result in a reversal and decline in the fit. The model has lost some useful knowledge, and the contextual information is insufficient for the model to make accurate inferences. Therefore, we validated and demonstrated that appropriate knowledge filtering can improve the effectiveness of question answering, but a balance needs to be found between denoising and preserving information. The Self Reg class module demonstrates a satisfactory fit, suggesting optimal model use at approximately 40% of the filtering points. The Retrieve module decides when passage retrieval is needed. The Relevant module filters out irrelevant passages. The Coherent module verifies whether the generated response is coherent with the input.
![Refer to caption](extracted/5737542/ab20.png)
![Refer to caption](extracted/5737542/ab21.png)
![Refer to caption](extracted/5737542/ab22.png)
![Refer to caption](extracted/5737542/ab23.png)
Appendix F Preliminaries
We first introduce the primary knowledge of knowledge graphs, text generation in LLMs, and information retrieval.
F.1 Knowledge Graphs and Graph Networks
A knowledge graph (KG) is defined as , with “head” and “tail” entities from and relation type from . Each triplet represents unique knowledge and such knowledge representation in KG can enhance LLMs reasoning Liu (2020). Graph neural networks (GNNs) process graphs with nodes and edges , learning node representations by message passing, combining node features and graph topology Wu (2020). GNNs encode KGs’ topology and structure. Node ’s feature vector at layer is , with neighboring nodes and edge feature , where the function aggregates neighboring node information, and is an activation function:
(12) |
Graph attention networks (GAT), a subclass of graph networks, leverage self-attention mechanisms Brody et al. (2021), similar to the Transformer architecture, to enhance node representations Li (2021a). Each layer projects node features into queries , keys , and values , with attention coefficients calculated between connected nodes, following the Transformer model Yang (2020); Khan (2022). These coefficients are used to aggregate neighboring value vectors, updating the node feature representation Li (2021b):
(13) |
(14) |
Through learning to focus on the most relevant semantic connections, our networks can refine node embeddings efficiently. The model constructs contextual node representations in KGs using graph attention layers Zhang et al. (2020), and subsequently integrates this structural knowledge into language models. This process improves the overall understanding and reasoning capabilities in tasks like semantic analysis and knowledge inference Banerjee et al. (2020).
F.2 Implementation Details
Training details We optimize model parameters using Adam optimizer with a learning rate of 1e-4, batch size of 32, and train for a maximum of 20 epochs. For testing, model accuracy is evaluated by the exact match of the predicted response with ground truth answers in the datasets. We report average accuracy over 5 runs with different random seeds and report the average value.
Model | WebQSP | MetaQA | ||||
---|---|---|---|---|---|---|
Accuracy (%) | Precision (%) | Recall (%) | Accuracy (%) | Precision (%) | Recall (%) | |
bert-base-NER111https://huggingface.co/dslim/bert-base-NER | 75.13 | 73.40 | 77.25 | 79.63 | 78.27 | 77.53 |
bert-multilingual-sentiment222https://huggingface.co/nlptown/bert-base-multilingual-uncased-sentiment | 73.24 | 72.29 | 74.86 | 76.45 | 75.36 | 74.62 |
BERT-large-whole-word333https://huggingface.co/bert-large-cased-whole-word-masking | 68.92 | 67.53 | 69.45 | 72.74 | 71.45 | 70.36 |
Appendix G Performance of Core Components of Auxiliary Modules
The results in Table 8 clearly demonstrate Bert-base-NER’s superiority as the Struct-X Auxiliary Module, with over 6-7% performance gains in accuracy, precision, and recall compared to alternatives. In contrast, the whole-word masked BERT-large model gives even poorer results than no Auxiliary Module, while the multilingual sentiment BERT model remains insufficient.
The likely explanation lies in the Named Entity Recognition pre-training of Bert-base-NER, which equips the model with a finer-grained understanding of named entities and relational reasoning - highly valuable for multi-hop questions over knowledge graphs. By steering prompt/answering iterations towards logically consistent outputs, it provides vital signals previously lacking. Meanwhile, whole-word masking seems to hinder BERT-large from learning compositional word structures crucial for precisely interpreting relations. Although also a BERT model, the sentiment classification tuning causes multilingual BERT to underperform on topological tasks. The significant gaps quantified via controlled ablation experiments validate that selective BERT-tuning surpasses superior architectural variants when specifically matched to complex reasoning tasks involving entities and relations.
Hyperparameter | WebQSP | MetaQA | Family Tree | Travel Route |
Graph Encoder Layers | 4 | 4 | 4 | 4 |
Graph Encoder Dimensions | 512 | 512 | 512 | 512 |
Encoder Heads | 8 | 8 | 8 | 8 |
Node Masking Rate | 0.4 | 0.4 | 0.3 | 0.3 |
Auxiliary Dimensions | 512 | 512 | 256 | 256 |
Prefix | 5 | 5 | 5 | 5 |
Generator Layers | 2 | 2 | 4 | 4 |
MLM Probability | 0.2 | 0.3 | 0.15 | 0.2 |
Tunable parameters | B | B | B | B |
Batch Size | 32 | 32 | 16 | 16 |
Dropout | 0 | 0 | 0 | 0 |
Prefix | 5 | 5 | 5 | 5 |
Batch Size | 32 | 32 | 16 | 16 |
Learning Rate | 1e-4 | 1e-4 | ||
Training Epochs | 15 | 20 | 15 | 10 |
Warmup epochs | 1 | 1 | 1 | 1 |
Weight decay | 0.01 | 0.01 | 0.01 | 0.01 |
Appendix H Experimental Results Illustration
Figure 5 illustrates performance on four reasoning tasks under variants of the Struct-X framework with different components disabled. For multi-hop QA (Fig 5(a)), removing the knowledge retrieval module causes an evident drop of 8.2% in accuracy. This validates its role in traversing distant nodes by filling gaps. Omitting the encoding also degrades topological reasoning, seen in the 4.3% lower location QA (Fig 5(b)) accuracy without the graph encoder. For complex inferences (Fig 5(c)), the Auxiliary Module contributes to steering coherent deductions, reflected by 6.5% weaker decomposition scores when absent. Finally, inconsistent responses increase by 4.9% (Fig 5(d)) after disabling relevance filtering, confirming the module’s benefits for factual verification.
Across diverse reasoning dimensions, targeted ablation of key components consistently reveals measurable impacts validating their individual contributions. The knowledge retrieval module facilitates connecting facts between entities, the graph encoder grounds topological representations, the Auxiliary Module directs logical inferences, and relevance filtering retains consistency. As a novel framework, Struct-X combines complementary mechanisms for enhancing language models’ structured reasoning capacities. The ablation analyses quantify these modular effects through performance differentials on specialized probing tasks.
Appendix I Code
The code can be accessed at the following URL:https://github.com/AnonymousBoy123/anonymous_structx.
Input Tell me more about the painter Van Gogh
Output [ IFReT =Yes]
Reference [35]
[ IFReL = Relevant] Van Gogh was a famous Dutch painter during the post-impressionist period. [ IFSuP = Partially Supported] He was known for paintings like the Starry Night and Sunflowers.
[ IFReT = Continue] However, he struggled with mental illness and remained poor and virtually unknown throughout his life. [ IFSeP = Fully Supported] Most of his best works were produced in the last two years before his death. [ IFUsE =4]
Reference [35] Vincent Willem van Gogh was a Dutch post-impressionist painter who is among the most famous and influential figures in the history of Western art. In a decade, he created about 2,100 artworks, including around 860 oil paintings, most of which date from the last two years of his life. They include landscapes, still lifes, portraits and self-portraits, and are characterised by bold colours and dramatic, impulsive and expressive brushwork that contributed to the foundations of modern art. He was little known during his lifetime, and many of his masterpieces were not appreciated until the 20th century. By the late 1920s, he had become one of the most celebrated artists in history. Nonetheless, mental illness plagued him throughout his life, and after he shot himself in the chest with a revolver at age 37, he succumbed to his injuries two days later.
Correctness of IFReL and IFSuP IFSuP is incorrect in claiming full support about his lack of fame during life, when the reference clearly states he only gained appreciation after death. So IFSuP should be partially supported. IFReL is appropriately marked as relevant overall.