The TechQA Dataset
Authors:
Vittorio Castelli,
Rishav Chakravarti,
Saswati Dana,
Anthony Ferritto,
Radu Florian,
Martin Franz,
Dinesh Garg,
Dinesh Khandelwal,
Scott McCarley,
Mike McCawley,
Mohamed Nasr,
Lin Pan,
Cezar Pendus,
John Pitrelli,
Saurabh Pujar,
Salim Roukos,
Andrzej Sakrajda,
Avirup Sil,
Rosario Uceda-Sosa,
Todd Ward,
Rong Zhang
Abstract:
We introduce TechQA, a domain-adaptation question answering dataset for the technical support domain. The TechQA corpus highlights two real-world issues from the automated customer support domain. First, it contains actual questions posed by users on a technical forum, rather than questions generated specifically for a competition or a task. Second, it has a real-world size -- 600 training, 310 de…
▽ More
We introduce TechQA, a domain-adaptation question answering dataset for the technical support domain. The TechQA corpus highlights two real-world issues from the automated customer support domain. First, it contains actual questions posed by users on a technical forum, rather than questions generated specifically for a competition or a task. Second, it has a real-world size -- 600 training, 310 dev, and 490 evaluation question/answer pairs -- thus reflecting the cost of creating large labeled datasets with actual data. Consequently, TechQA is meant to stimulate research in domain adaptation rather than being a resource to build QA systems from scratch. The dataset was obtained by crawling the IBM Developer and IBM DeveloperWorks forums for questions with accepted answers that appear in a published IBM Technote---a technical document that addresses a specific technical issue. We also release a collection of the 801,998 publicly available Technotes as of April 4, 2019 as a companion resource that might be used for pretraining, to learn representations of the IT domain language.
△ Less
Submitted 7 November, 2019;
originally announced November 2019.
CFO: A Framework for Building Production NLP Systems
Authors:
Rishav Chakravarti,
Cezar Pendus,
Andrzej Sakrajda,
Anthony Ferritto,
Lin Pan,
Michael Glass,
Vittorio Castelli,
J. William Murdock,
Radu Florian,
Salim Roukos,
Avirup Sil
Abstract:
This paper introduces a novel orchestration framework, called CFO (COMPUTATION FLOW ORCHESTRATOR), for building, experimenting with, and deploying interactive NLP (Natural Language Processing) and IR (Information Retrieval) systems to production environments. We then demonstrate a question answering system built using this framework which incorporates state-of-the-art BERT based MRC (Machine Readi…
▽ More
This paper introduces a novel orchestration framework, called CFO (COMPUTATION FLOW ORCHESTRATOR), for building, experimenting with, and deploying interactive NLP (Natural Language Processing) and IR (Information Retrieval) systems to production environments. We then demonstrate a question answering system built using this framework which incorporates state-of-the-art BERT based MRC (Machine Reading Comprehension) with IR components to enable end-to-end answer retrieval. Results from the demo system are shown to be high quality in both academic and industry domain specific settings. Finally, we discuss best practices when (pre-)training BERT based MRC models for production systems.
△ Less
Submitted 19 June, 2020; v1 submitted 16 August, 2019;
originally announced August 2019.
