Measuring Conversational Fluidity in Automated Dialogue Agents
Authors:
Keith Vella,
Massimo Poesio,
Michael Sigamani,
Cihan Dogan,
Aimore Dutra,
Dimitrios Dimakopoulos,
Alfredo Gemma,
Ella Walters
Abstract:
We present an automated evaluation method to measure fluidity in conversational dialogue systems. The method combines various state of the art Natural Language tools into a classifier, and human ratings on these dialogues to train an automated judgment model. Our experiments show that the results are an improvement on existing metrics for measuring fluidity.
We present an automated evaluation method to measure fluidity in conversational dialogue systems. The method combines various state of the art Natural Language tools into a classifier, and human ratings on these dialogues to train an automated judgment model. Our experiments show that the results are an improvement on existing metrics for measuring fluidity.
△ Less
Submitted 25 October, 2019;
originally announced October 2019.
Fine-Grained Named Entity Recognition using ELMo and Wikidata
Authors:
Cihan Dogan,
Aimore Dutra,
Adam Gara,
Alfredo Gemma,
Lei Shi,
Michael Sigamani,
Ella Walters
Abstract:
Fine-grained Named Entity Recognition is a task whereby we detect and classify entity mentions to a large set of types. These types can span diverse domains such as finance, healthcare, and politics. We observe that when the type set spans several domains the accuracy of the entity detection becomes a limitation for supervised learning models. The primary reason being the lack of datasets where en…
▽ More
Fine-grained Named Entity Recognition is a task whereby we detect and classify entity mentions to a large set of types. These types can span diverse domains such as finance, healthcare, and politics. We observe that when the type set spans several domains the accuracy of the entity detection becomes a limitation for supervised learning models. The primary reason being the lack of datasets where entity boundaries are properly annotated, whilst covering a large spectrum of entity types. Furthermore, many named entity systems suffer when considering the categorization of fine grained entity types. Our work attempts to address these issues, in part, by combining state-of-the-art deep learning models (ELMo) with an expansive knowledge base (Wikidata). Using our framework, we cross-validate our model on the 112 fine-grained entity types based on the hierarchy given from the Wiki(gold) dataset.
△ Less
Submitted 23 April, 2019;
originally announced April 2019.
