-
A Benchmark for Generalizable and Interpretable Temporal Question Answering over Knowledge Bases
Authors:
Sumit Neelam,
Udit Sharma,
Hima Karanam,
Shajith Ikbal,
Pavan Kapanipathi,
Ibrahim Abdelaziz,
Nandana Mihindukulasooriya,
Young-Suk Lee,
Santosh Srivastava,
Cezar Pendus,
Saswati Dana,
Dinesh Garg,
Achille Fokoue,
G P Shrivatsa Bhargav,
Dinesh Khandelwal,
Srinivas Ravishankar,
Sairam Gurajada,
Maria Chang,
Rosario Uceda-Sosa,
Salim Roukos,
Alexander Gray,
Guilherme Lima,
Ryan Riegel,
Francois Luus,
L Venkata Subramaniam
Abstract:
Knowledge Base Question Answering (KBQA) tasks that involve complex reasoning are emerging as an important research direction. However, most existing KBQA datasets focus primarily on generic multi-hop reasoning over explicit facts, largely ignoring other reasoning types such as temporal, spatial, and taxonomic reasoning. In this paper, we present a benchmark dataset for temporal reasoning, TempQA-…
▽ More
Knowledge Base Question Answering (KBQA) tasks that involve complex reasoning are emerging as an important research direction. However, most existing KBQA datasets focus primarily on generic multi-hop reasoning over explicit facts, largely ignoring other reasoning types such as temporal, spatial, and taxonomic reasoning. In this paper, we present a benchmark dataset for temporal reasoning, TempQA-WD, to encourage research in extending the present approaches to target a more challenging set of complex reasoning tasks. Specifically, our benchmark is a temporal question answering dataset with the following advantages: (a) it is based on Wikidata, which is the most frequently curated, openly available knowledge base, (b) it includes intermediate sparql queries to facilitate the evaluation of semantic parsing based approaches for KBQA, and (c) it generalizes to multiple knowledge bases: Freebase and Wikidata. The TempQA-WD dataset is available at https://github.com/IBM/tempqa-wd.
△ Less
Submitted 15 January, 2022;
originally announced January 2022.
-
SYGMA: System for Generalizable Modular Question Answering OverKnowledge Bases
Authors:
Sumit Neelam,
Udit Sharma,
Hima Karanam,
Shajith Ikbal,
Pavan Kapanipathi,
Ibrahim Abdelaziz,
Nandana Mihindukulasooriya,
Young-Suk Lee,
Santosh Srivastava,
Cezar Pendus,
Saswati Dana,
Dinesh Garg,
Achille Fokoue,
G P Shrivatsa Bhargav,
Dinesh Khandelwal,
Srinivas Ravishankar,
Sairam Gurajada,
Maria Chang,
Rosario Uceda-Sosa,
Salim Roukos,
Alexander Gray,
Guilherme LimaRyan Riegel,
Francois Luus,
L Venkata Subramaniam
Abstract:
Knowledge Base Question Answering (KBQA) tasks that in-volve complex reasoning are emerging as an important re-search direction. However, most KBQA systems struggle withgeneralizability, particularly on two dimensions: (a) acrossmultiple reasoning types where both datasets and systems haveprimarily focused on multi-hop reasoning, and (b) across mul-tiple knowledge bases, where KBQA approaches are…
▽ More
Knowledge Base Question Answering (KBQA) tasks that in-volve complex reasoning are emerging as an important re-search direction. However, most KBQA systems struggle withgeneralizability, particularly on two dimensions: (a) acrossmultiple reasoning types where both datasets and systems haveprimarily focused on multi-hop reasoning, and (b) across mul-tiple knowledge bases, where KBQA approaches are specif-ically tuned to a single knowledge base. In this paper, wepresent SYGMA, a modular approach facilitating general-izability across multiple knowledge bases and multiple rea-soning types. Specifically, SYGMA contains three high levelmodules: 1) KB-agnostic question understanding module thatis common across KBs 2) Rules to support additional reason-ing types and 3) KB-specific question mapping and answeringmodule to address the KB-specific aspects of the answer ex-traction. We demonstrate effectiveness of our system by evalu-ating on datasets belonging to two distinct knowledge bases,DBpedia and Wikidata. In addition, to demonstrate extensi-bility to additional reasoning types we evaluate on multi-hopreasoning datasets and a new Temporal KBQA benchmarkdataset on Wikidata, namedTempQA-WD1, introduced in thispaper. We show that our generalizable approach has bettercompetetive performance on multiple datasets on DBpediaand Wikidata that requires both multi-hop and temporal rea-soning
△ Less
Submitted 27 September, 2021;
originally announced September 2021.
-
The TechQA Dataset
Authors:
Vittorio Castelli,
Rishav Chakravarti,
Saswati Dana,
Anthony Ferritto,
Radu Florian,
Martin Franz,
Dinesh Garg,
Dinesh Khandelwal,
Scott McCarley,
Mike McCawley,
Mohamed Nasr,
Lin Pan,
Cezar Pendus,
John Pitrelli,
Saurabh Pujar,
Salim Roukos,
Andrzej Sakrajda,
Avirup Sil,
Rosario Uceda-Sosa,
Todd Ward,
Rong Zhang
Abstract:
We introduce TechQA, a domain-adaptation question answering dataset for the technical support domain. The TechQA corpus highlights two real-world issues from the automated customer support domain. First, it contains actual questions posed by users on a technical forum, rather than questions generated specifically for a competition or a task. Second, it has a real-world size -- 600 training, 310 de…
▽ More
We introduce TechQA, a domain-adaptation question answering dataset for the technical support domain. The TechQA corpus highlights two real-world issues from the automated customer support domain. First, it contains actual questions posed by users on a technical forum, rather than questions generated specifically for a competition or a task. Second, it has a real-world size -- 600 training, 310 dev, and 490 evaluation question/answer pairs -- thus reflecting the cost of creating large labeled datasets with actual data. Consequently, TechQA is meant to stimulate research in domain adaptation rather than being a resource to build QA systems from scratch. The dataset was obtained by crawling the IBM Developer and IBM DeveloperWorks forums for questions with accepted answers that appear in a published IBM Technote---a technical document that addresses a specific technical issue. We also release a collection of the 801,998 publicly available Technotes as of April 4, 2019 as a companion resource that might be used for pretraining, to learn representations of the IT domain language.
△ Less
Submitted 7 November, 2019;
originally announced November 2019.
-
CFO: A Framework for Building Production NLP Systems
Authors:
Rishav Chakravarti,
Cezar Pendus,
Andrzej Sakrajda,
Anthony Ferritto,
Lin Pan,
Michael Glass,
Vittorio Castelli,
J. William Murdock,
Radu Florian,
Salim Roukos,
Avirup Sil
Abstract:
This paper introduces a novel orchestration framework, called CFO (COMPUTATION FLOW ORCHESTRATOR), for building, experimenting with, and deploying interactive NLP (Natural Language Processing) and IR (Information Retrieval) systems to production environments. We then demonstrate a question answering system built using this framework which incorporates state-of-the-art BERT based MRC (Machine Readi…
▽ More
This paper introduces a novel orchestration framework, called CFO (COMPUTATION FLOW ORCHESTRATOR), for building, experimenting with, and deploying interactive NLP (Natural Language Processing) and IR (Information Retrieval) systems to production environments. We then demonstrate a question answering system built using this framework which incorporates state-of-the-art BERT based MRC (Machine Reading Comprehension) with IR components to enable end-to-end answer retrieval. Results from the demo system are shown to be high quality in both academic and industry domain specific settings. Finally, we discuss best practices when (pre-)training BERT based MRC models for production systems.
△ Less
Submitted 19 June, 2020; v1 submitted 16 August, 2019;
originally announced August 2019.
