What's the Meaning of Superhuman Performance in Today's NLU?
Authors:
Simone Tedeschi,
Johan Bos,
Thierry Declerck,
Jan Hajic,
Daniel Hershcovich,
Eduard H. Hovy,
Alexander Koller,
Simon Krek,
Steven Schockaert,
Rico Sennrich,
Ekaterina Shutova,
Roberto Navigli
Abstract:
In the last five years, there has been a significant focus in Natural Language Processing (NLP) on developing larger Pretrained Language Models (PLMs) and introducing benchmarks such as SuperGLUE and SQuAD to measure their abilities in language understanding, reasoning, and reading comprehension. These PLMs have achieved impressive results on these benchmarks, even surpassing human performance in…
▽ More
In the last five years, there has been a significant focus in Natural Language Processing (NLP) on developing larger Pretrained Language Models (PLMs) and introducing benchmarks such as SuperGLUE and SQuAD to measure their abilities in language understanding, reasoning, and reading comprehension. These PLMs have achieved impressive results on these benchmarks, even surpassing human performance in some cases. This has led to claims of superhuman capabilities and the provocative idea that certain tasks have been solved. In this position paper, we take a critical look at these claims and ask whether PLMs truly have superhuman abilities and what the current benchmarks are really evaluating. We show that these benchmarks have serious limitations affecting the comparison between humans and PLMs and provide recommendations for fairer and more transparent benchmarks.
△ Less
Submitted 15 May, 2023;
originally announced May 2023.
Natural Language Dialogue Service for Appointment Scheduling Agents
Authors:
Stephan Busemann,
Thierry Declerck,
Abdel Kader Diagne,
Luca Dini,
Judith Klein,
Sven Schmeier
Abstract:
Appointment scheduling is a problem faced daily by many individuals and organizations. Cooperating agent systems have been developed to partially automate this task. In order to extend the circle of participants as far as possible we advocate the use of natural language transmitted by e-mail. We describe COSMA, a fully implemented German language server for existing appointment scheduling agent…
▽ More
Appointment scheduling is a problem faced daily by many individuals and organizations. Cooperating agent systems have been developed to partially automate this task. In order to extend the circle of participants as far as possible we advocate the use of natural language transmitted by e-mail. We describe COSMA, a fully implemented German language server for existing appointment scheduling agent systems. COSMA can cope with multiple dialogues in parallel, and accounts for differences in dialogue behaviour between human and machine agents. NL coverage of the sublanguage is achieved through both corpus-based grammar development and the use of message extraction techniques.
△ Less
Submitted 11 February, 1997;
originally announced February 1997.
