Spinach: SPARQL-Based Information Navigation
for Challenging Real-World Questions

Over the past decade, numerous Knowledge Base Question Answering (KBQA) benchmarks with logical forms have been introduced and can be classified into two categories:

Datasets with natural questions originally collected through search engines or crowd-sourcing. This includes the popular dataset WebQuestionSP (Yih et al., 2016), along with the QALD datasets Usbeck et al. (2017, 2018, 2023); Perevalov et al. (2022), RuBQ (Korablinov and Braslavski, 2020), and SimpleQuestions (Bordes et al., 2015), among others;

Datasets with synthetically generated logical forms and questions, where some datasets paraphrase synthetically generated questions via crowdsourcing. This category includes ComplexWebQuestions (Talmor and Berant, 2018), GrailQA (Gu et al., 2021), KQA Pro (Cao et al., 2022a), CFQ (Keysers et al., 2020), and LC-QuAD-2 Dubey et al. (2019), among others.

Some of the aforementioned datasets were originally based on Freebase (Bollacker et al., 2008) or DBPedia Lehmann et al. (2015). Recently, however, the KBQA community has been shifting toward using Wikidata as the underlying knowledge base for KBQA datasets, due to its larger size and continuous updates from community contributions.³³3For more on why Wikidata is better suited for KBQA benchmarks, refer to Section 1.1 of Usbeck et al. (2023). Several datasets have been converted to use Wikidata: WebQuestionSP has been converted to WikiWebQuestions (Xu et al., 2023); SimpleQuestions has a Wikidata version (Diefenbach et al., 2017b); and CFQ has been converted to MCWQ (Cui et al., 2022).

In Table 1, we present quantitative statistics on these datasets. Existing datasets with natural questions typically involve relatively simple queries (e.g., 2.63 clauses per query for WikiWebQuestions). In contrast, synthetically generated datasets generally contain more complex queries by design. However, due to the limited natural language variety between training and evaluation data, models can achieve artificially high accuracy. For instance, a simple semantic parser based on the BART Lewis et al. (2020) model can achieve an accuracy of over 90% on KQA-Pro even without an entity linking module (Cao et al., 2022a). Furthermore, systems that excel at synthetic datasets do not necessarily perform well in natural, complex tasks (Oren et al., 2021; Campagna et al., 2022). This indicates a need for a high-quality dataset with natural, complex queries and logical forms in the KBQA community.

Current KBQA systems can be classified mainly into three categories: (1) subgraph retrieval approaches (Sun et al., 2018, 2019; Sen et al., 2021; Verga et al., 2021; Mavromatis and Karypis, 2022), (2) semantic parsing systems (Yih et al., 2015, 2016; Luo et al., 2018; Lan and Jiang, 2020; Das et al., 2021; Ye et al., 2022; Cao et al., 2022b; Gu and Su, 2022; Xu et al., 2023), and more recently, (3) LLM-based graph exploration agents (Sun et al., 2024a; Xiong et al., 2024). Some works also use a combination of these techniques (Yu et al., 2023; Luo et al., 2024a, b).

Due to the sheer size of Wikidata, none of the previous subgraph retrieval approaches have tackled the problem of embedding the entire graph, instead operating only on a small fixed subset of Wikidata. This limitation renders them inapplicable in real-world scenarios where access to the entire graph is needed. Among semantic parsing systems, Xu et al. (2023) fine-tune LLaMA Touvron et al. (2023) with a modified SPARQL syntax and achieve state-of-the-art results on 2 KBQA benchmarks. Among LLM-based approaches, Sun et al. (2024a) instruct an LLM to dynamically explore the graph to fetch answers, achieving SOTA on 6 KBQA datasets. In Section 5, we benchmark the performance of these two systems on our new Spinach dataset.

Our goal in building this data set is to have examples focus more on end-users simply accessing the Wikidata content, as opposed to performing obscure optimizations or formatting. As such, we manually remove the following SPARQL clauses:

Wikimedia presentation queries: We filter out clauses for analyzing or improving Wikimedia itself, such as asking which articles exist in one language in Wikipedia and not the other.

Overly complicated SPARQL. We exclude certain conversations where users ask for help debugging their complex SPARQL queries.

Queries obscured by optimizations: Because Wikidata restricts query runtime to 60 seconds (MediaWiki, 2024a), users may include clauses to optimize the performance, which are often not directly relevant to answering the question. We remove these clauses.

Formatting clauses: We remove clauses that only format the results.

Appendix B.2 and B.3 show specific examples of these modifications and exclusions.

For each conversation, the experts annotate a self-contained, decontextualized natural language question that accurately captures the meaning of the user-written SPARQL. We follow these steps:

Disambiguate entities and properties: Entities and properties are important parts of a query. For ambiguous entities, the verbalization includes additional information to resolve the ambiguity. As for the properties, the verbalization should disintinguish between similar properties. For example, instead of asking “where a movie takes place”, we distinguish between the “narrative location” and the “filming location”; instead of asking “where a person comes from”, we distinguish between the “courtry of citizenship” vs. “country of birth”.

Natural verbalizations: Whenever possible, to the extent that no ambiguities are introduced, the question should refrain from directly using entity and property names, instead using a more natural way to express the meaning. For instance, instead of asking “what is the point of time of the goal?”, a more natural question with the same level of accuracy like “when does the goal take place?” should be used.

Accurately capturing OPTIONAL clauses and projections: In SPARQL, OPTIONAL clauses are used to include additional data that may or may not exist for queried items without excluding results lacking this optional data.⁵⁵5https://www.wikidata.org/wiki/Wikidata:SPARQL_tutorial#OPTIONAL The verbalization should specify which fields are optional with clauses such as “if available”. Similarly, the verbalization should accurately capture what is requested in the projections, using phrases like: “For each result, return their name and location” if necessary.

As illustrated in Table 1, the Spinach dataset is the first dataset with both natural and complex logical forms, which represents the real-world KBQA needs of Wikidata users.

The state at time step $n$ is $\mathcal{S}_{n}=\{t_{1},a_{1},o_{1},...,t_{n-1},a_{n-1},o_{n-1}\}$, that is, the list of all thoughts, actions, and observations up to and including the previous time step. $\mathcal{S}_{1}$ is set to $\emptyset$.

Complex KBQA tasks require high reasoning ability. One promising approach to improve the reasoning capabilities of LLMs is ReAct prompting Yao et al. (2023), in which the LLM is instructed to output its reasoning trace in natural language first, before selecting an action. We require the agent to output a thought at the beginning of each time step.

At time step $n$, the Spinach agent takes $a_{n}$, one of the following five actions, and receives the result of the execution as observation $o_{n}$. These actions are selected for their similarity to what a human expert can do.

Importantly, the results of the execution of each action are put in a human-readable format to make it easier for the LLM to process. To limit the amount of information that the agent has to process, we limit the output of search results to at most 8 entities and 4 properties, and limit large results of SPARQL queries to the first and last 5 rows. The outputs of get_wikidata_entry() are pruned using a separate few-shot prompt (Prompt  5) to only keep entries that are relevant to the question.

The policy of Spinach is implemented using a zero-shot prompt that only includes high-level instructions such as “start by constructing very simple queries and gradually build towards the complete query” and “confirm all your assumptions about the structure of Wikidata before proceeding” (Prompt 4). The decision of selecting the action at each time step is left to the LLM. This decision is made after experimenting with more restrictive policies, but we empirically observe that accounting for all various edge cases the agent might confuse the agent and hurt the performance.

The agent loop constructs the state and uses this policy prompt to ask LLM to output a thought and an action. It then executes the action and adds the observation to the state. The agent loop continues until either the stop() action is selected or it reaches the predefined maximum number of 30 actions.

In practice, the LLM can occasionally fall into a cycle of repeating the same action and argument over and over again, or deviate from its instructions and call stop() even though the last SPARQL execution did not return any results. To remedy this, the agent loop checks for these two situations and resets the state to before the first time the repeated action was performed, or the beginning, respectively, and continues from there. Note that the policy LLM is run using sampling with high temperature of 1.0 to encourage exploration, so rerunning from a previous state does not repeat the exact same sequence of actions.

In the example in Figure 2, Spinach goes through the following four phases, just by following the high-level instruction we provided: it (1) looks up relevant PIDs and QIDs, (2) attempts to write the SPARQL in one go, which fails due to a hallucinated QID that the agent did not previously check, (3) starts the debugging process by executing simpler SPARQL queries, and double-checking its assumptions. It realizes its mistake after looking at the result of a get_wikidata_entry, and finally (4) moves on to write the correct SPARQL query.

Prior works mostly use two metrics: Exact Match (EM) and F1 (Yih et al., 2016; Usbeck et al., 2023). As shown in Table 1, the Spinach dataset contains 2.50 projections on average for each query, whereas prior datasets all only contain only one field in projection. This means that naively applying EM and F1 is not possible; therefore, we propose a generalization of these metrics.

Concretely, during evaluation, for each data point $x$, a gold result $y$ and a predicted result $y^{\prime}$ are given, where:

each $y_{i}$ being either an entity (e.g. ‘Q123’) or a literal (e.g. ‘123’). For each data point, EM is calculated to be 1 if $y=y^{\prime}$ and $0$ otherwise. If both results are lists, F1 is calculated as the harmonic mean of precision and recall :

where tp denotes the number of true positives ($|y\cap y^{\prime}|$), fp denotes the number of false positives ($|y^{\prime}-y|$), and fn denotes the number of false negatives ($|y-y^{\prime}|$). If one of $y,y^{\prime}$ is a boolean, then $F_{1}(y,y^{\prime})$ is set to $1$ if $y=y^{\prime}$ and $0$ otherwise (Usbeck et al., 2023).

In prior datasets, each $y_{1},y_{2},\cdots$ is only an entity or literal, but this is not necessarily the case in real-life SPARQLs. For instance, in 1Figure 1(b), the user is interested in a tuple of the doctoral advisee and advisor for each result, where each $y_{i}$ is a list of entities and literals. In other words, the results are in the general case a matrix, not a list. Every row in the matrix is one result, and each column corresponds to one projection in the query.

From the user’s perspective, a good system could opt to return additional information when it helps clarify or enrich the results. For instance, for the query “what is the county with most people in South Dakota?”, a LLM-based system can choose to return the top county along with its population, even though the gold SPARQL might only include the county in projection. Penalizing the system for the additional projection columns is not meaningful in this case. To reflect this, we introduce a row-major generalization of EM and F1 to handle matrix-wise comparisons in real-life KBQA tasks, where each row is handled similarly to the traditional F1 score, and extra columns are not penalized.

Concretely, given two matrices of the form:

we calculate the recall between a gold row $y_{i}=(y_{i1},\cdots y_{i,m})$ and predicted row $y^{\prime}_{i^{\prime}}=(y_{i^{\prime}1},\cdots y_{i^{\prime},m^{\prime}})$ as

An assignment algorithm (SciPy, 2024) is then run between the two matrices to find the row-by-row matching with the highest cumulative recall score (without matching rows with $0$ recall). Each matched row $(y_{i},y^{\prime}_{i^{\prime}})$ adds $\text{recall}(y_{i},y^{\prime}_{i^{\prime}})$ to the true positives and $1-\text{recall}(y_{i},y^{\prime}_{i^{\prime}})$ to the false negatives. Each unmatched gold row counts as $1$ false negative, and each unmatched prediction row counts as $1$ false positive. The final $F_{1}$ is then calculated according to Equation 1, and EM is defined to be $1$ if the row-major $F_{1}$ score is $1$ and $0$ otherwise. Note that with this definition, EM and $F_{1}$ are exactly equal to EM and $F_{1}$ in the traditional sense when there is only one projection.

We evaluate our approach on four previous Wikidata datasets. We experiment with three QALD challenges that have annotated Wikidata SPARQLs: QALD-7 (task 4 for Wikidata) (Usbeck et al., 2017), QALD-9 Plus Perevalov et al. (2022), and the English subset of QALD-10 (Usbeck et al., 2023). In addition, we also evaluate with WikiWebQuestions (Xu et al., 2023), the Wikidata version of the popular WebQuestionSP dataset.

As shown in Table 2, the Spinach agent achieves new SOTA on the three QALD datasets and comes within 1.6% F1 of the fine-tuned SOTA (WikiSP) on WikiWebQuestion. In particular, our 0-shot agent outperforms WikiSP by 24.2% EM and 31.0% F1 on Qald-7 (Task 4), showing its impressive cross-dataset generalization capability compared to a fine-tuned model. Our approach also outperforms the GPT-4-based ToG Sun et al. (2024a) by 10.0% EM on the same subset of QALD-10 used for evaluation in its paper.

We benchmark the performance of four baseline systems on the Spinach dataset. These baselines are: (1) directly asking GPT-4o to answer the question without writing SPARQL, (2) directly asking GPT-4o to write a SPARQL query, (3) WikiSP Xu et al. (2023) which is a fine-tuned LLaMA (7B-parameter) model and (4) the GPT-4-based ToG agent (Sun et al., 2024a).

WikiSP expects a Named Entity Disambiguation (NED) module to predict the relevant entities given a question. The original NED module was based on a fine-tuned version of ReFinED (Ayoola et al., 2022). To make the task easier for WikiSP, we directly use the gold entities that appear in the annotated SPARQL as inputs. Similarly, the ToG (Sun et al., 2024a) pipeline uses LLM to extract “topic entities” from the input question as the starting point of its knowledge graph exploration. We observed that using the gold entities as topic entities, the model was unable to return any exploration paths for the first 80 questions in the validation set and simply defaulted to GPT-4. To make the task even easier for ToG, we sample up to 20 entities from the results of the gold SPARQL queries as the topic entities. Additional details on our setup and comparison of baseline systems can be found in Appendix C.

As shown in Table 3, the Spinach agent drastically outperforms all baselines. The median number of actions Spinach takes is 8. In particular, we highlight that the low accuracy of the GPT-4o QA system shows that this dataset mainly consists of long-tail knowledge unfamiliar to LLMs. For reference, GPT-4 was able to achieve 90.5% EM on WebQuestionSP (Tan et al., 2023). Although Spinach was able to achieve the SOTA on this dataset, the relatively lower performance compared to those achieved on previous dataset indicates that there is a lot of room for improving KBQA systems in future work.

We randomly sample 20 cases on the Spinach dataset where the Spinach agent achieves less than 0.05 F1 and conduct an error analysis. We observe that:

• •

  Property-related problems: 40% of errors are due to problems with properties. This includes cases when the Spinach agent fails to fetch the correct property or incorrectly uses a property (e.g. using it as a wdt: relation as opposed to the correct ps: or pq: qualifier).

• •

  Complicated SPARQL: 30% of errors are due to failure of the Spinach agent to write complex SPARQL to fetch results, e.g., a complicated filter for finding people born in a specific month.⁷⁷7An interesting observation is that in 2 cases, GPT-4o ended the reasoning with “I will fetch all related records and proceed to filter the results in Python” after observing its previous SPARQL returned no results, suggesting there may be benefits to leveraging programming languages that are more familar to LLMs.

• •

  Not enough exploration: 15% of errors are due to insufficient exploration performed by the LLM after reaching the maximum allowed number of actions. Here, the LLM has determined the infeasibility of its prior approaches and is actively exploring new methods.

• •

  Inaccurate semantic parsing: 10% of errors are due to the LLM injecting a clause that should not exist. For instance, when asked “what items were published in …”, the LLM assumes that the results have to be an instance of the domain entity “book”, but there are many other types of results returned by the gold query (e.g. “printed matter” and “legal act”).

• •

  Formatting issues: 5% of errors are due to format errors. This one case is due to LLM returning the date in full (“June 23 2021”) even though the question only asked for the year (“2021”).