-
Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning
Authors:
Bahare Fatemi,
Mehran Kazemi,
Anton Tsitsulin,
Karishma Malkan,
Jinyeong Yim,
John Palowitch,
Sungyong Seo,
Jonathan Halcrow,
Bryan Perozzi
Abstract:
Large language models (LLMs) have showcased remarkable reasoning capabilities, yet they remain susceptible to errors, particularly in temporal reasoning tasks involving complex temporal logic. Existing research has explored LLM performance on temporal reasoning using diverse datasets and benchmarks. However, these studies often rely on real-world data that LLMs may have encountered during pre-trai…
▽ More
Large language models (LLMs) have showcased remarkable reasoning capabilities, yet they remain susceptible to errors, particularly in temporal reasoning tasks involving complex temporal logic. Existing research has explored LLM performance on temporal reasoning using diverse datasets and benchmarks. However, these studies often rely on real-world data that LLMs may have encountered during pre-training or employ anonymization techniques that can inadvertently introduce factual inconsistencies. In this work, we address these limitations by introducing novel synthetic datasets specifically designed to assess LLM temporal reasoning abilities in various scenarios. The diversity of question types across these datasets enables systematic investigation into the impact of the problem structure, size, question type, fact order, and other factors on LLM performance. Our findings provide valuable insights into the strengths and weaknesses of current LLMs in temporal reasoning tasks. To foster further research in this area, we are open-sourcing the datasets and evaluation framework used in our experiments: https://huggingface.co/datasets/baharef/ToT.
△ Less
Submitted 13 June, 2024;
originally announced June 2024.
-
Do Transformer Modifications Transfer Across Implementations and Applications?
Authors:
Sharan Narang,
Hyung Won Chung,
Yi Tay,
William Fedus,
Thibault Fevry,
Michael Matena,
Karishma Malkan,
Noah Fiedel,
Noam Shazeer,
Zhenzhong Lan,
Yanqi Zhou,
Wei Li,
Nan Ding,
Jake Marcus,
Adam Roberts,
Colin Raffel
Abstract:
The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which have seen widespread adoption. In this paper, we comprehensively evaluate many of these modifications in a shared experimental setting that covers most of the common uses of the Transformer in natural language processing. Surprisingly, we f…
▽ More
The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which have seen widespread adoption. In this paper, we comprehensively evaluate many of these modifications in a shared experimental setting that covers most of the common uses of the Transformer in natural language processing. Surprisingly, we find that most modifications do not meaningfully improve performance. Furthermore, most of the Transformer variants we found beneficial were either developed in the same codebase that we used or are relatively minor changes. We conjecture that performance improvements may strongly depend on implementation details and correspondingly make some recommendations for improving the generality of experimental results.
△ Less
Submitted 10 September, 2021; v1 submitted 23 February, 2021;
originally announced February 2021.
-
WT5?! Training Text-to-Text Models to Explain their Predictions
Authors:
Sharan Narang,
Colin Raffel,
Katherine Lee,
Adam Roberts,
Noah Fiedel,
Karishma Malkan
Abstract:
Neural networks have recently achieved human-level performance on various challenging natural language processing (NLP) tasks, but it is notoriously difficult to understand why a neural network produced a particular prediction. In this paper, we leverage the text-to-text framework proposed by Raffel et al.(2019) to train language models to output a natural text explanation alongside their predicti…
▽ More
Neural networks have recently achieved human-level performance on various challenging natural language processing (NLP) tasks, but it is notoriously difficult to understand why a neural network produced a particular prediction. In this paper, we leverage the text-to-text framework proposed by Raffel et al.(2019) to train language models to output a natural text explanation alongside their prediction. Crucially, this requires no modifications to the loss function or training and decoding procedures -- we simply train the model to output the explanation after generating the (natural text) prediction. We show that this approach not only obtains state-of-the-art results on explainability benchmarks, but also permits learning from a limited set of labeled explanations and transferring rationalization abilities across datasets. To facilitate reproducibility and future work, we release our code use to train the models.
△ Less
Submitted 29 April, 2020;
originally announced April 2020.
