-
ThoughtSource: A central hub for large language model reasoning data
Authors:
Simon Ott,
Konstantin Hebenstreit,
Valentin Liévin,
Christoffer Egeberg Hother,
Milad Moradi,
Maximilian Mayrhauser,
Robert Praas,
Ole Winther,
Matthias Samwald
Abstract:
Large language models (LLMs) such as GPT-4 have recently demonstrated impressive results across a wide range of tasks. LLMs are still limited, however, in that they frequently fail at complex reasoning, their reasoning processes are opaque, they are prone to 'hallucinate' facts, and there are concerns about their underlying biases. Letting models verbalize reasoning steps as natural language, a te…
▽ More
Large language models (LLMs) such as GPT-4 have recently demonstrated impressive results across a wide range of tasks. LLMs are still limited, however, in that they frequently fail at complex reasoning, their reasoning processes are opaque, they are prone to 'hallucinate' facts, and there are concerns about their underlying biases. Letting models verbalize reasoning steps as natural language, a technique known as chain-of-thought prompting, has recently been proposed as a way to address some of these issues. Here we present ThoughtSource, a meta-dataset and software library for chain-of-thought (CoT) reasoning. The goal of ThoughtSource is to improve future artificial intelligence systems by facilitating qualitative understanding of CoTs, enabling empirical evaluations, and providing training data. This first release of ThoughtSource integrates seven scientific/medical, three general-domain and five math word question answering datasets.
△ Less
Submitted 27 July, 2023; v1 submitted 27 January, 2023;
originally announced January 2023.
-
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Authors:
BigScience Workshop,
:,
Teven Le Scao,
Angela Fan,
Christopher Akiki,
Ellie Pavlick,
Suzana Ilić,
Daniel Hesslow,
Roman Castagné,
Alexandra Sasha Luccioni,
François Yvon,
Matthias Gallé,
Jonathan Tow,
Alexander M. Rush,
Stella Biderman,
Albert Webson,
Pawan Sasanka Ammanamanchi,
Thomas Wang,
Benoît Sagot,
Niklas Muennighoff,
Albert Villanova del Moral,
Olatunji Ruwase,
Rachel Bawden,
Stas Bekman,
Angelina McMillan-Major
, et al. (369 additional authors not shown)
Abstract:
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access…
▽ More
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages (59 in total). We find that BLOOM achieves competitive performance on a wide variety of benchmarks, with stronger results after undergoing multitask prompted finetuning. To facilitate future research and applications using LLMs, we publicly release our models and code under the Responsible AI License.
△ Less
Submitted 27 June, 2023; v1 submitted 9 November, 2022;
originally announced November 2022.
-
BigBIO: A Framework for Data-Centric Biomedical Natural Language Processing
Authors:
Jason Alan Fries,
Leon Weber,
Natasha Seelam,
Gabriel Altay,
Debajyoti Datta,
Samuele Garda,
Myungsun Kang,
Ruisi Su,
Wojciech Kusa,
Samuel Cahyawijaya,
Fabio Barth,
Simon Ott,
Matthias Samwald,
Stephen Bach,
Stella Biderman,
Mario Sänger,
Bo Wang,
Alison Callahan,
Daniel León Periñán,
Théo Gigant,
Patrick Haller,
Jenny Chim,
Jose David Posada,
John Michael Giorgi,
Karthik Rangasai Sivaraman
, et al. (18 additional authors not shown)
Abstract:
Training and evaluating language models increasingly requires the construction of meta-datasets --diverse collections of curated data with clear provenance. Natural language prompting has recently lead to improved zero-shot generalization by transforming existing, supervised datasets into a diversity of novel pretraining tasks, highlighting the benefits of meta-dataset curation. While successful i…
▽ More
Training and evaluating language models increasingly requires the construction of meta-datasets --diverse collections of curated data with clear provenance. Natural language prompting has recently lead to improved zero-shot generalization by transforming existing, supervised datasets into a diversity of novel pretraining tasks, highlighting the benefits of meta-dataset curation. While successful in general-domain text, translating these data-centric approaches to biomedical language modeling remains challenging, as labeled biomedical datasets are significantly underrepresented in popular data hubs. To address this challenge, we introduce BigBIO a community library of 126+ biomedical NLP datasets, currently covering 12 task categories and 10+ languages. BigBIO facilitates reproducible meta-dataset curation via programmatic access to datasets and their metadata, and is compatible with current platforms for prompt engineering and end-to-end few/zero shot language model evaluation. We discuss our process for task schema harmonization, data auditing, contribution guidelines, and outline two illustrative use cases: zero-shot evaluation of biomedical prompts and large-scale, multi-task learning. BigBIO is an ongoing community effort and is available at https://github.com/bigscience-workshop/biomedical
△ Less
Submitted 30 June, 2022;
originally announced June 2022.
-
A global analysis of metrics used for measuring performance in natural language processing
Authors:
Kathrin Blagec,
Georg Dorffner,
Milad Moradi,
Simon Ott,
Matthias Samwald
Abstract:
Measuring the performance of natural language processing models is challenging. Traditionally used metrics, such as BLEU and ROUGE, originally devised for machine translation and summarization, have been shown to suffer from low correlation with human judgment and a lack of transferability to other tasks and languages. In the past 15 years, a wide range of alternative metrics have been proposed. H…
▽ More
Measuring the performance of natural language processing models is challenging. Traditionally used metrics, such as BLEU and ROUGE, originally devised for machine translation and summarization, have been shown to suffer from low correlation with human judgment and a lack of transferability to other tasks and languages. In the past 15 years, a wide range of alternative metrics have been proposed. However, it is unclear to what extent this has had an impact on NLP benchmarking efforts. Here we provide the first large-scale cross-sectional analysis of metrics used for measuring performance in natural language processing. We curated, mapped and systematized more than 3500 machine learning model performance results from the open repository 'Papers with Code' to enable a global and comprehensive analysis. Our results suggest that the large majority of natural language processing metrics currently used have properties that may result in an inadequate reflection of a models' performance. Furthermore, we found that ambiguities and inconsistencies in the reporting of metrics may lead to difficulties in interpreting and comparing model performances, impairing transparency and reproducibility in NLP research.
△ Less
Submitted 25 April, 2022;
originally announced April 2022.
-
Mapping global dynamics of benchmark creation and saturation in artificial intelligence
Authors:
Simon Ott,
Adriano Barbosa-Silva,
Kathrin Blagec,
Jan Brauner,
Matthias Samwald
Abstract:
Benchmarks are crucial to measuring and steering progress in artificial intelligence (AI). However, recent studies raised concerns over the state of AI benchmarking, reporting issues such as benchmark overfitting, benchmark saturation and increasing centralization of benchmark dataset creation. To facilitate monitoring of the health of the AI benchmarking ecosystem, we introduce methodologies for…
▽ More
Benchmarks are crucial to measuring and steering progress in artificial intelligence (AI). However, recent studies raised concerns over the state of AI benchmarking, reporting issues such as benchmark overfitting, benchmark saturation and increasing centralization of benchmark dataset creation. To facilitate monitoring of the health of the AI benchmarking ecosystem, we introduce methodologies for creating condensed maps of the global dynamics of benchmark creation and saturation. We curated data for 3765 benchmarks covering the entire domains of computer vision and natural language processing, and show that a large fraction of benchmarks quickly trended towards near-saturation, that many benchmarks fail to find widespread utilization, and that benchmark performance gains for different AI tasks were prone to unforeseen bursts. We analyze attributes associated with benchmark popularity, and conclude that future benchmarks should emphasize versatility, breadth and real-world utility.
△ Less
Submitted 7 October, 2022; v1 submitted 9 March, 2022;
originally announced March 2022.
-
A curated, ontology-based, large-scale knowledge graph of artificial intelligence tasks and benchmarks
Authors:
Kathrin Blagec,
Adriano Barbosa-Silva,
Simon Ott,
Matthias Samwald
Abstract:
Research in artificial intelligence (AI) is addressing a growing number of tasks through a rapidly growing number of models and methodologies. This makes it difficult to keep track of where novel AI methods are successfully -- or still unsuccessfully -- applied, how progress is measured, how different advances might synergize with each other, and how future research should be prioritized.
To hel…
▽ More
Research in artificial intelligence (AI) is addressing a growing number of tasks through a rapidly growing number of models and methodologies. This makes it difficult to keep track of where novel AI methods are successfully -- or still unsuccessfully -- applied, how progress is measured, how different advances might synergize with each other, and how future research should be prioritized.
To help address these issues, we created the Intelligence Task Ontology and Knowledge Graph (ITO), a comprehensive, richly structured and manually curated resource on artificial intelligence tasks, benchmark results and performance metrics. The current version of ITO contain 685,560 edges, 1,100 classes representing AI processes and 1,995 properties representing performance metrics.
The goal of ITO is to enable precise and network-based analyses of the global landscape of AI tasks and capabilities. ITO is based on technologies that allow for easy integration and enrichment with external data, automated inference and continuous, collaborative expert curation of underlying ontological models. We make the ITO dataset and a collection of Jupyter notebooks utilising ITO openly available.
△ Less
Submitted 6 October, 2021; v1 submitted 4 October, 2021;
originally announced October 2021.
-
SAFRAN: An interpretable, rule-based link prediction method outperforming embedding models
Authors:
Simon Ott,
Christian Meilicke,
Matthias Samwald
Abstract:
Neural embedding-based machine learning models have shown promise for predicting novel links in knowledge graphs. Unfortunately, their practical utility is diminished by their lack of interpretability. Recently, the fully interpretable, rule-based algorithm AnyBURL yielded highly competitive results on many general-purpose link prediction benchmarks. However, current approaches for aggregating pre…
▽ More
Neural embedding-based machine learning models have shown promise for predicting novel links in knowledge graphs. Unfortunately, their practical utility is diminished by their lack of interpretability. Recently, the fully interpretable, rule-based algorithm AnyBURL yielded highly competitive results on many general-purpose link prediction benchmarks. However, current approaches for aggregating predictions made by multiple rules are affected by redundancies. We improve upon AnyBURL by introducing the SAFRAN rule application framework, which uses a novel aggregation approach called Non-redundant Noisy-OR that detects and clusters redundant rules prior to aggregation. SAFRAN yields new state-of-the-art results for fully interpretable link prediction on the established general-purpose benchmarks FB15K-237, WN18RR and YAGO3-10. Furthermore, it exceeds the results of multiple established embedding-based algorithms on FB15K-237 and WN18RR and narrows the gap between rule-based and embedding-based algorithms on YAGO3-10.
△ Less
Submitted 16 September, 2021;
originally announced September 2021.
-
Scalable and interpretable rule-based link prediction for large heterogeneous knowledge graphs
Authors:
Simon Ott,
Laura Graf,
Asan Agibetov,
Christian Meilicke,
Matthias Samwald
Abstract:
Neural embedding-based machine learning models have shown promise for predicting novel links in biomedical knowledge graphs. Unfortunately, their practical utility is diminished by their lack of interpretability. Recently, the fully interpretable, rule-based algorithm AnyBURL yielded highly competitive results on many general-purpose link prediction benchmarks. However, its applicability to large-…
▽ More
Neural embedding-based machine learning models have shown promise for predicting novel links in biomedical knowledge graphs. Unfortunately, their practical utility is diminished by their lack of interpretability. Recently, the fully interpretable, rule-based algorithm AnyBURL yielded highly competitive results on many general-purpose link prediction benchmarks. However, its applicability to large-scale prediction tasks on complex biomedical knowledge bases is limited by long inference times and difficulties with aggregating predictions made by multiple rules. We improve upon AnyBURL by introducing the SAFRAN rule application framework which aggregates rules through a scalable clustering algorithm. SAFRAN yields new state-of-the-art results for fully interpretable link prediction on the established general-purpose benchmark FB15K-237 and the large-scale biomedical benchmark OpenBioLink. Furthermore, it exceeds the results of multiple established embedding-based algorithms on FB15K-237 and narrows the gap between rule-based and embedding-based algorithms on OpenBioLink. We also show that SAFRAN increases inference speeds by up to two orders of magnitude.
△ Less
Submitted 10 December, 2020;
originally announced December 2020.
-
The Lazarus Effect: Healing Compromised Devices in the Internet of Small Things
Authors:
Manuel Huber,
Stefan Hristozov,
Simon Ott,
Vasil Sarafov,
Marcus Peinado
Abstract:
We live in a time when billions of IoT devices are being deployed and increasingly relied upon. This makes ensuring their availability and recoverability in case of a compromise a paramount goal. The large and rapidly growing number of deployed IoT devices make manual recovery impractical, especially if the devices are dispersed over a large area. Thus, there is a need for a reliable and scalable…
▽ More
We live in a time when billions of IoT devices are being deployed and increasingly relied upon. This makes ensuring their availability and recoverability in case of a compromise a paramount goal. The large and rapidly growing number of deployed IoT devices make manual recovery impractical, especially if the devices are dispersed over a large area. Thus, there is a need for a reliable and scalable remote recovery mechanism that works even after attackers have taken full control over devices, possibly misusing them or trying to render them useless.
To tackle this problem, we present Lazarus, a system that enables the remote recovery of compromised IoT devices. With Lazarus, an IoT administrator can remotely control the code running on IoT devices unconditionally and within a guaranteed time bound. This makes recovery possible even in case of severe corruption of the devices' software stack. We impose only minimal hardware requirements, making Lazarus applicable even for low-end constrained off-the-shelf IoT devices. We isolate Lazarus's minimal recovery trusted computing base from untrusted software both in time and by using a trusted execution environment. The temporal isolation prevents secrets from being leaked through side-channels to untrusted software. Inside the trusted execution environment, we place minimal functionality that constrains untrusted software at runtime.
We implement Lazarus on an ARM Cortex-M33-based microcontroller in a full setup with an IoT hub, device provisioning and secure update functionality. Our prototype can recover compromised embedded OSs and bare-metal applications and prevents attackers from bricking devices, for example, through flash wear out. We show this at the example of FreeRTOS, which requires no modifications but only a single additional task. Our evaluation shows negligible runtime performance impact and moderate memory requirements.
△ Less
Submitted 19 May, 2020;
originally announced May 2020.
-
OpenBioLink: A benchmarking framework for large-scale biomedical link prediction
Authors:
Anna Breit,
Simon Ott,
Asan Agibetov,
Matthias Samwald
Abstract:
SUMMARY: Recently, novel machine-learning algorithms have shown potential for predicting undiscovered links in biomedical knowledge networks. However, dedicated benchmarks for measuring algorithmic progress have not yet emerged. With OpenBioLink, we introduce a large-scale, high-quality and highly challenging biomedical link prediction benchmark to transparently and reproducibly evaluate such algo…
▽ More
SUMMARY: Recently, novel machine-learning algorithms have shown potential for predicting undiscovered links in biomedical knowledge networks. However, dedicated benchmarks for measuring algorithmic progress have not yet emerged. With OpenBioLink, we introduce a large-scale, high-quality and highly challenging biomedical link prediction benchmark to transparently and reproducibly evaluate such algorithms. Furthermore, we present preliminary baseline evaluation results. AVAILABILITY AND IMPLEMENTATION: Source code, data and supplementary files are openly available at https://github.com/OpenBioLink/OpenBioLink CONTACT: matthias.samwald ((at)) meduniwien.ac.at
△ Less
Submitted 19 February, 2020; v1 submitted 10 December, 2019;
originally announced December 2019.
-
A Combinatorial Approximation Algorithm for Graph Balancing with Light Hyper Edges
Authors:
Chien-Chung Huang,
Sebastian Ott
Abstract:
Makespan minimization in restricted assignment $(R|p_{ij}\in \{p_j, \infty\}|C_{\max})$ is a classical problem in the field of machine scheduling. In a landmark paper in 1990 [8], Lenstra, Shmoys, and Tardos gave a 2-approximation algorithm and proved that the problem cannot be approximated within 1.5 unless P=NP. The upper and lower bounds of the problem have been essentially unimproved in the in…
▽ More
Makespan minimization in restricted assignment $(R|p_{ij}\in \{p_j, \infty\}|C_{\max})$ is a classical problem in the field of machine scheduling. In a landmark paper in 1990 [8], Lenstra, Shmoys, and Tardos gave a 2-approximation algorithm and proved that the problem cannot be approximated within 1.5 unless P=NP. The upper and lower bounds of the problem have been essentially unimproved in the intervening 25 years, despite several remarkable successful attempts in some special cases of the problem [2,4,12] recently.
In this paper, we consider a special case called graph-balancing with light hyper edges, where heavy jobs can be assigned to at most two machines while light jobs can be assigned to any number of machines. For this case, we present algorithms with approximation ratios strictly better than 2. Specifically,
Two job sizes: Suppose that light jobs have weight $w$ and heavy jobs have weight $W$, and $w < W$. We give a $1.5$-approximation algorithm (note that the current 1.5 lower bound is established in an even more restrictive setting [1,3]). Indeed, depending on the specific values of $w$ and $W$, sometimes our algorithm guarantees sub-1.5 approximation ratios.
Arbitrary job sizes: Suppose that $W$ is the largest given weight, heavy jobs have weights in the range of $(βW, W]$, where $4/7\leq β< 1$, and light jobs have weights in the range of $(0,βW]$. We present a $(5/3+β/3)$-approximation algorithm.
Our algorithms are purely combinatorial, without the need of solving a linear program as required in most other known approaches.
△ Less
Submitted 2 October, 2015; v1 submitted 27 July, 2015;
originally announced July 2015.
-
A Fully Polynomial-Time Approximation Scheme for Speed Scaling with Sleep State
Authors:
Antonios Antoniadis,
Chien-Chung Huang,
Sebastian Ott
Abstract:
We study classical deadline-based preemptive scheduling of tasks in a computing environment equipped with both dynamic speed scaling and sleep state capabilities: Each task is specified by a release time, a deadline and a processing volume, and has to be scheduled on a single, speed-scalable processor that is supplied with a sleep state. In the sleep state, the processor consumes no energy, but a…
▽ More
We study classical deadline-based preemptive scheduling of tasks in a computing environment equipped with both dynamic speed scaling and sleep state capabilities: Each task is specified by a release time, a deadline and a processing volume, and has to be scheduled on a single, speed-scalable processor that is supplied with a sleep state. In the sleep state, the processor consumes no energy, but a constant wake-up cost is required to transition back to the active state. In contrast to speed scaling alone, the addition of a sleep state makes it sometimes beneficial to accelerate the processing of tasks in order to transition the processor to the sleep state for longer amounts of time and incur further energy savings. The goal is to output a feasible schedule that minimizes the energy consumption. Since the introduction of the problem by Irani et al. [16], its exact computational complexity has been repeatedly posed as an open question (see e.g. [2,8,15]). The currently best known upper and lower bounds are a 4/3-approximation algorithm and NP-hardness due to [2] and [2,17], respectively. We close the aforementioned gap between the upper and lower bound on the computational complexity of speed scaling with sleep state by presenting a fully polynomial-time approximation scheme for the problem. The scheme is based on a transformation to a non-preemptive variant of the problem, and a discretization that exploits a carefully defined lexicographical ordering among schedules.
△ Less
Submitted 3 July, 2014;
originally announced July 2014.
