Showing 1–1 of 1 results for author: Grosch, S
-
Predictions from language models for multiple-choice tasks are not robust under variation of scoring methods
Authors:
Polina Tsvilodub,
Hening Wang,
Sharon Grosch,
Michael Franke
Abstract:
This paper systematically compares different methods of deriving item-level predictions of language models for multiple-choice tasks. It compares scoring methods for answer options based on free generation of responses, various probability-based scores, a Likert-scale style rating method, and embedding similarity. In a case study on pragmatic language interpretation, we find that LLM predictions a…
▽ More
This paper systematically compares different methods of deriving item-level predictions of language models for multiple-choice tasks. It compares scoring methods for answer options based on free generation of responses, various probability-based scores, a Likert-scale style rating method, and embedding similarity. In a case study on pragmatic language interpretation, we find that LLM predictions are not robust under variation of method choice, both within a single LLM and across different LLMs. As this variability entails pronounced researcher degrees of freedom in reporting results, knowledge of the variability is crucial to secure robustness of results and research integrity.
△ Less
Submitted 1 March, 2024;
originally announced March 2024.
