-
An Approach to Build Zero-Shot Slot-Filling System for Industry-Grade Conversational Assistants
Authors:
G P Shrivatsa Bhargav,
Sumit Neelam,
Udit Sharma,
Shajith Ikbal,
Dheeraj Sreedhar,
Hima Karanam,
Sachindra Joshi,
Pankaj Dhoolia,
Dinesh Garg,
Kyle Croutwater,
Haode Qi,
Eric Wayne,
J William Murdock
Abstract:
We present an approach to build Large Language Model (LLM) based slot-filling system to perform Dialogue State Tracking in conversational assistants serving across a wide variety of industry-grade applications. Key requirements of this system include: 1) usage of smaller-sized models to meet low latency requirements and to enable convenient and cost-effective cloud and customer premise deployments…
▽ More
We present an approach to build Large Language Model (LLM) based slot-filling system to perform Dialogue State Tracking in conversational assistants serving across a wide variety of industry-grade applications. Key requirements of this system include: 1) usage of smaller-sized models to meet low latency requirements and to enable convenient and cost-effective cloud and customer premise deployments, and 2) zero-shot capabilities to serve across a wide variety of domains, slot types and conversational scenarios. We adopt a fine-tuning approach where a pre-trained LLM is fine-tuned into a slot-filling model using task specific data. The fine-tuning data is prepared carefully to cover a wide variety of slot-filling task scenarios that the model is expected to face across various domains. We give details of the data preparation and model building process. We also give a detailed analysis of the results of our experimental evaluations. Results show that our prescribed approach for slot-filling model building has resulted in 6.9% relative improvement of F1 metric over the best baseline on a realistic benchmark, while at the same time reducing the latency by 57%. More over, the data we prepared has helped improve F1 on an average by 4.2% relative across various slot-types.
△ Less
Submitted 13 June, 2024;
originally announced June 2024.
-
Fast and Light-Weight Answer Text Retrieval in Dialogue Systems
Authors:
Hui Wan,
Siva Sankalp Patel,
J. William Murdock,
Saloni Potdar,
Sachindra Joshi
Abstract:
Dialogue systems can benefit from being able to search through a corpus of text to find information relevant to user requests, especially when encountering a request for which no manually curated response is available. The state-of-the-art technology for neural dense retrieval or re-ranking involves deep learning models with hundreds of millions of parameters. However, it is difficult and expensiv…
▽ More
Dialogue systems can benefit from being able to search through a corpus of text to find information relevant to user requests, especially when encountering a request for which no manually curated response is available. The state-of-the-art technology for neural dense retrieval or re-ranking involves deep learning models with hundreds of millions of parameters. However, it is difficult and expensive to get such models to operate at an industrial scale, especially for cloud services that often need to support a big number of individually customized dialogue systems, each with its own text corpus. We report our work on enabling advanced neural dense retrieval systems to operate effectively at scale on relatively inexpensive hardware. We compare with leading alternative industrial solutions and show that we can provide a solution that is effective, fast, and cost-efficient.
△ Less
Submitted 31 May, 2022; v1 submitted 27 May, 2022;
originally announced May 2022.
-
Ensembling Strategies for Answering Natural Questions
Authors:
Anthony Ferritto,
Lin Pan,
Rishav Chakravarti,
Salim Roukos,
Radu Florian,
J. William Murdock,
Avirup Sil
Abstract:
Many of the top question answering systems today utilize ensembling to improve their performance on tasks such as the Stanford Question Answering Dataset (SQuAD) and Natural Questions (NQ) challenges. Unfortunately most of these systems do not publish their ensembling strategies used in their leaderboard submissions. In this work, we investigate a number of ensembling techniques and demonstrate a…
▽ More
Many of the top question answering systems today utilize ensembling to improve their performance on tasks such as the Stanford Question Answering Dataset (SQuAD) and Natural Questions (NQ) challenges. Unfortunately most of these systems do not publish their ensembling strategies used in their leaderboard submissions. In this work, we investigate a number of ensembling techniques and demonstrate a strategy which improves our F1 score for short answers on the dev set for NQ by 2.3 F1 points over our single model (which outperforms the previous SOTA by 1.9 F1 points).
△ Less
Submitted 6 November, 2019; v1 submitted 30 October, 2019;
originally announced November 2019.
-
CFO: A Framework for Building Production NLP Systems
Authors:
Rishav Chakravarti,
Cezar Pendus,
Andrzej Sakrajda,
Anthony Ferritto,
Lin Pan,
Michael Glass,
Vittorio Castelli,
J. William Murdock,
Radu Florian,
Salim Roukos,
Avirup Sil
Abstract:
This paper introduces a novel orchestration framework, called CFO (COMPUTATION FLOW ORCHESTRATOR), for building, experimenting with, and deploying interactive NLP (Natural Language Processing) and IR (Information Retrieval) systems to production environments. We then demonstrate a question answering system built using this framework which incorporates state-of-the-art BERT based MRC (Machine Readi…
▽ More
This paper introduces a novel orchestration framework, called CFO (COMPUTATION FLOW ORCHESTRATOR), for building, experimenting with, and deploying interactive NLP (Natural Language Processing) and IR (Information Retrieval) systems to production environments. We then demonstrate a question answering system built using this framework which incorporates state-of-the-art BERT based MRC (Machine Reading Comprehension) with IR components to enable end-to-end answer retrieval. Results from the demo system are shown to be high quality in both academic and industry domain specific settings. Finally, we discuss best practices when (pre-)training BERT based MRC models for production systems.
△ Less
Submitted 19 June, 2020; v1 submitted 16 August, 2019;
originally announced August 2019.
-
SHOP2: An HTN Planning System
Authors:
T. C. Au,
O. Ilghami,
U. Kuter,
J. W. Murdock,
D. S. Nau,
D. Wu,
F. Yaman
Abstract:
The SHOP2 planning system received one of the awards for distinguished performance in the 2002 International Planning Competition. This paper describes the features of SHOP2 which enabled it to excel in the competition, especially those aspects of SHOP2 that deal with temporal and metric planning domains.
The SHOP2 planning system received one of the awards for distinguished performance in the 2002 International Planning Competition. This paper describes the features of SHOP2 which enabled it to excel in the competition, especially those aspects of SHOP2 that deal with temporal and metric planning domains.
△ Less
Submitted 23 June, 2011;
originally announced June 2011.
