-
SciCode: A Research Coding Benchmark Curated by Scientists
Authors:
Minyang Tian,
Luyu Gao,
Shizhuo Dylan Zhang,
Xinan Chen,
Cunwei Fan,
Xuefei Guo,
Roland Haas,
Pan Ji,
Kittithat Krongchon,
Yao Li,
Shengyan Liu,
Di Luo,
Yutao Ma,
Hao Tong,
Kha Trinh,
Chenyu Tian,
Zihan Wang,
Bohao Wu,
Yanyu Xiong,
Shengzhu Yin,
Minhui Zhu,
Kilian Lieret,
Yanxin Lu,
Genglin Liu,
Yufeng Du
, et al. (5 additional authors not shown)
Abstract:
Since language models (LMs) now outperform average humans on many challenging tasks, it has become increasingly difficult to develop challenging, high-quality, and realistic evaluations. We address this issue by examining LMs' capabilities to generate code for solving real scientific research problems. Incorporating input from scientists and AI researchers in 16 diverse natural science sub-fields,…
▽ More
Since language models (LMs) now outperform average humans on many challenging tasks, it has become increasingly difficult to develop challenging, high-quality, and realistic evaluations. We address this issue by examining LMs' capabilities to generate code for solving real scientific research problems. Incorporating input from scientists and AI researchers in 16 diverse natural science sub-fields, including mathematics, physics, chemistry, biology, and materials science, we created a scientist-curated coding benchmark, SciCode. The problems in SciCode naturally factorize into multiple subproblems, each involving knowledge recall, reasoning, and code synthesis. In total, SciCode contains 338 subproblems decomposed from 80 challenging main problems. It offers optional descriptions specifying useful scientific background information and scientist-annotated gold-standard solutions and test cases for evaluation. Claude3.5-Sonnet, the best-performing model among those tested, can solve only 4.6% of the problems in the most realistic setting. We believe that SciCode demonstrates both contemporary LMs' progress towards becoming helpful scientific assistants and sheds light on the development and evaluation of scientific AI in the future.
△ Less
Submitted 18 July, 2024;
originally announced July 2024.
-
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Authors:
John Yang,
Carlos E. Jimenez,
Alexander Wettig,
Kilian Lieret,
Shunyu Yao,
Karthik Narasimhan,
Ofir Press
Abstract:
Language model (LM) agents are increasingly being used to automate complicated tasks in digital environments. Just as humans benefit from powerful software applications, such as integrated development environments, for complex tasks like software engineering, we posit that LM agents represent a new category of end users with their own needs and abilities, and would benefit from specially-built int…
▽ More
Language model (LM) agents are increasingly being used to automate complicated tasks in digital environments. Just as humans benefit from powerful software applications, such as integrated development environments, for complex tasks like software engineering, we posit that LM agents represent a new category of end users with their own needs and abilities, and would benefit from specially-built interfaces to the software they use. We investigate how interface design affects the performance of language model agents. As a result of this exploration, we introduce SWE-agent: a system that facilitates LM agents to autonomously use computers to solve software engineering tasks. SWE-agent's custom agent-computer interface (ACI) significantly enhances an agent's ability to create and edit code files, navigate entire repositories, and execute tests and other programs. We evaluate SWE-agent on SWE-bench and HumanEvalFix, achieving state-of-the-art performance on both with a pass@1 rate of 12.5% and 87.7%, respectively, far exceeding the previous state-of-the-art achieved with non-interactive LMs. Finally, we provide insight on how the design of the ACI can impact agents' behavior and performance.
△ Less
Submitted 30 May, 2024; v1 submitted 6 May, 2024;
originally announced May 2024.
-
High Pileup Particle Tracking with Object Condensation
Authors:
Kilian Lieret,
Gage DeZoort,
Devdoot Chatterjee,
Jian Park,
Siqi Miao,
Pan Li
Abstract:
Recent work has demonstrated that graph neural networks (GNNs) can match the performance of traditional algorithms for charged particle tracking while improving scalability to meet the computing challenges posed by the HL-LHC. Most GNN tracking algorithms are based on edge classification and identify tracks as connected components from an initial graph containing spurious connections. In this talk…
▽ More
Recent work has demonstrated that graph neural networks (GNNs) can match the performance of traditional algorithms for charged particle tracking while improving scalability to meet the computing challenges posed by the HL-LHC. Most GNN tracking algorithms are based on edge classification and identify tracks as connected components from an initial graph containing spurious connections. In this talk, we consider an alternative based on object condensation (OC), a multi-objective learning framework designed to cluster points (hits) belonging to an arbitrary number of objects (tracks) and regress the properties of each object. Building on our previous results, we present a streamlined model and show progress toward a one-shot OC tracking algorithm in a high-pileup environment.
△ Less
Submitted 6 December, 2023;
originally announced December 2023.
-
Software Sustainability & High Energy Physics
Authors:
Daniel S. Katz,
Sudhir Malik,
Mark S. Neubauer,
Graeme A. Stewart,
Kétévi A. Assamagan,
Erin A. Becker,
Neil P. Chue Hong,
Ian A. Cosden,
Samuel Meehan,
Edward J. W. Moyse,
Adrian M. Price-Whelan,
Elizabeth Sexton-Kennedy,
Meirin Oan Evans,
Matthew Feickert,
Clemens Lange,
Kilian Lieret,
Rob Quick,
Arturo Sánchez Pineda,
Christopher Tunnell
Abstract:
New facilities of the 2020s, such as the High Luminosity Large Hadron Collider (HL-LHC), will be relevant through at least the 2030s. This means that their software efforts and those that are used to analyze their data need to consider sustainability to enable their adaptability to new challenges, longevity, and efficiency, over at least this period. This will help ensure that this software will b…
▽ More
New facilities of the 2020s, such as the High Luminosity Large Hadron Collider (HL-LHC), will be relevant through at least the 2030s. This means that their software efforts and those that are used to analyze their data need to consider sustainability to enable their adaptability to new challenges, longevity, and efficiency, over at least this period. This will help ensure that this software will be easier to develop and maintain, that it remains available in the future on new platforms, that it meets new needs, and that it is as reusable as possible. This report discusses a virtual half-day workshop on "Software Sustainability and High Energy Physics" that aimed 1) to bring together experts from HEP as well as those from outside to share their experiences and practices, and 2) to articulate a vision that helps the Institute for Research and Innovation in Software for High Energy Physics (IRIS-HEP) to create a work plan to implement elements of software sustainability. Software sustainability practices could lead to new collaborations, including elements of HEP software being directly used outside the field, and, as has happened more frequently in recent years, to HEP developers contributing to software developed outside the field rather than reinventing it. A focus on and skills related to sustainable software will give HEP software developers an important skill that is essential to careers in the realm of software, inside or outside HEP. The report closes with recommendations to improve software sustainability in HEP, aimed at the HEP community via IRIS-HEP and the HEP Software Foundation (HSF).
△ Less
Submitted 16 October, 2020; v1 submitted 10 October, 2020;
originally announced October 2020.
