AstroMLab 1: Who Wins Astronomy Jeopardy!?
Authors:
Yuan-Sen Ting,
Tuan Dung Nguyen,
Tirthankar Ghosal,
Rui Pan,
Hardik Arora,
Zechang Sun,
Tijmen de Haan,
Nesar Ramachandra,
Azton Wells,
Sandeep Madireddy,
Alberto Accomazzi
Abstract:
We present a comprehensive evaluation of proprietary and open-weights large language models using the first astronomy-specific benchmarking dataset. This dataset comprises 4,425 multiple-choice questions curated from the Annual Review of Astronomy and Astrophysics, covering a broad range of astrophysical topics. Our analysis examines model performance across various astronomical subfields and asse…
▽ More
We present a comprehensive evaluation of proprietary and open-weights large language models using the first astronomy-specific benchmarking dataset. This dataset comprises 4,425 multiple-choice questions curated from the Annual Review of Astronomy and Astrophysics, covering a broad range of astrophysical topics. Our analysis examines model performance across various astronomical subfields and assesses response calibration, crucial for potential deployment in research environments. Claude-3.5-Sonnet outperforms competitors by up to 4.6 percentage points, achieving 85.0% accuracy. For proprietary models, we observed a universal reduction in cost every 3-to-12 months to achieve similar score in this particular astronomy benchmark. Open-source models have rapidly improved, with LLaMA-3-70b (80.6%) and Qwen-2-72b (77.7%) now competing with some of the best proprietary models. We identify performance variations across topics, with non-English-focused models generally struggling more in exoplanet-related fields, stellar astrophysics, and instrumentation related questions. These challenges likely stem from less abundant training data, limited historical context, and rapid recent developments in these areas. This pattern is observed across both open-weights and proprietary models, with regional dependencies evident, highlighting the impact of training data diversity on model performance in specialized scientific domains. Top-performing models demonstrate well-calibrated confidence, with correlations above 0.9 between confidence and correctness, though they tend to be slightly underconfident. The development for fast, low-cost inference of open-weights models presents new opportunities for affordable deployment in astronomy. The rapid progress observed suggests that LLM-driven research in astronomy may become feasible in the near future.
△ Less
Submitted 15 July, 2024;
originally announced July 2024.
Deep Forest: Neural Network reconstruction of the Lyman-alpha forest
Authors:
Lawrence Huang,
Rupert A. C. Croft,
Hitesh Arora
Abstract:
We explore the use of Deep Learning to infer physical quantities from the observable transmitted flux in the Lyman-alpha forest. We train a Neural Network using redshift z=3 outputs from cosmological hydrodynamic simulations and mock datasets constructed from them. We evaluate how well the trained network is able to reconstruct the optical depth for Lyman-alpha forest absorption from noisy and oft…
▽ More
We explore the use of Deep Learning to infer physical quantities from the observable transmitted flux in the Lyman-alpha forest. We train a Neural Network using redshift z=3 outputs from cosmological hydrodynamic simulations and mock datasets constructed from them. We evaluate how well the trained network is able to reconstruct the optical depth for Lyman-alpha forest absorption from noisy and often saturated transmitted flux data. The Neural Network outperforms an alternative reconstruction method involving log inversion and spline interpolation by approximately a factor of 2 in the optical depth root mean square error. We find no significant dependence in the improvement on input data signal to noise, although the gain is greatest in high optical depth regions. The Lyman-alpha forest optical depth studied here serves as a simple, one dimensional, example but the use of Deep Learning and simulations to approach the inverse problem in cosmology could be extended to other physical quantities and higher dimensional data.
△ Less
Submitted 5 September, 2021; v1 submitted 22 September, 2020;
originally announced September 2020.
