What's the Meaning of Superhuman Performance in Today's NLU?
Authors:
Simone Tedeschi,
Johan Bos,
Thierry Declerck,
Jan Hajic,
Daniel Hershcovich,
Eduard H. Hovy,
Alexander Koller,
Simon Krek,
Steven Schockaert,
Rico Sennrich,
Ekaterina Shutova,
Roberto Navigli
Abstract:
In the last five years, there has been a significant focus in Natural Language Processing (NLP) on developing larger Pretrained Language Models (PLMs) and introducing benchmarks such as SuperGLUE and SQuAD to measure their abilities in language understanding, reasoning, and reading comprehension. These PLMs have achieved impressive results on these benchmarks, even surpassing human performance in…
▽ More
In the last five years, there has been a significant focus in Natural Language Processing (NLP) on developing larger Pretrained Language Models (PLMs) and introducing benchmarks such as SuperGLUE and SQuAD to measure their abilities in language understanding, reasoning, and reading comprehension. These PLMs have achieved impressive results on these benchmarks, even surpassing human performance in some cases. This has led to claims of superhuman capabilities and the provocative idea that certain tasks have been solved. In this position paper, we take a critical look at these claims and ask whether PLMs truly have superhuman abilities and what the current benchmarks are really evaluating. We show that these benchmarks have serious limitations affecting the comparison between humans and PLMs and provide recommendations for fairer and more transparent benchmarks.
△ Less
Submitted 15 May, 2023;
originally announced May 2023.
The European Language Technology Landscape in 2020: Language-Centric and Human-Centric AI for Cross-Cultural Communication in Multilingual Europe
Authors:
Georg Rehm,
Katrin Marheinecke,
Stefanie Hegele,
Stelios Piperidis,
Kalina Bontcheva,
Jan Hajič,
Khalid Choukri,
Andrejs Vasiļjevs,
Gerhard Backfried,
Christoph Prinz,
José Manuel Gómez Pérez,
Luc Meertens,
Paul Lukowicz,
Josef van Genabith,
Andrea Lösch,
Philipp Slusallek,
Morten Irgens,
Patrick Gatellier,
Joachim Köhler,
Laure Le Bars,
Dimitra Anastasiou,
Albina Auksoriūtė,
Núria Bel,
António Branco,
Gerhard Budin
, et al. (22 additional authors not shown)
Abstract:
Multilingualism is a cultural cornerstone of Europe and firmly anchored in the European treaties including full language equality. However, language barriers impacting business, cross-lingual and cross-cultural communication are still omnipresent. Language Technologies (LTs) are a powerful means to break down these barriers. While the last decade has seen various initiatives that created a multitu…
▽ More
Multilingualism is a cultural cornerstone of Europe and firmly anchored in the European treaties including full language equality. However, language barriers impacting business, cross-lingual and cross-cultural communication are still omnipresent. Language Technologies (LTs) are a powerful means to break down these barriers. While the last decade has seen various initiatives that created a multitude of approaches and technologies tailored to Europe's specific needs, there is still an immense level of fragmentation. At the same time, AI has become an increasingly important concept in the European Information and Communication Technology area. For a few years now, AI, including many opportunities, synergies but also misconceptions, has been overshadowing every other topic. We present an overview of the European LT landscape, describing funding programmes, activities, actions and challenges in the different countries with regard to LT, including the current state of play in industry and the LT market. We present a brief overview of the main LT-related activities on the EU level in the last ten years and develop strategic guidance with regard to four key dimensions.
△ Less
Submitted 30 March, 2020;
originally announced March 2020.
