-
A Solvable Model of Neural Scaling Laws
Authors:
Alexander Maloney,
Daniel A. Roberts,
James Sully
Abstract:
Large language models with a huge number of parameters, when trained on near internet-sized number of tokens, have been empirically shown to obey neural scaling laws: specifically, their performance behaves predictably as a power law in either parameters or dataset size until bottlenecked by the other resource. To understand this better, we first identify the necessary properties allowing such sca…
▽ More
Large language models with a huge number of parameters, when trained on near internet-sized number of tokens, have been empirically shown to obey neural scaling laws: specifically, their performance behaves predictably as a power law in either parameters or dataset size until bottlenecked by the other resource. To understand this better, we first identify the necessary properties allowing such scaling laws to arise and then propose a statistical model -- a joint generative data model and random feature model -- that captures this neural scaling phenomenology. By solving this model in the dual limit of large training set size and large number of parameters, we gain insight into (i) the statistical structure of datasets and tasks that lead to scaling laws, (ii) the way nonlinear feature maps, such as those provided by neural networks, enable scaling laws when trained on these datasets, (iii) the optimality of the equiparameterization scaling of training sets and parameters, and (iv) whether such scaling laws can break down and how they behave when they do. Key findings are the manner in which the power laws that occur in the statistics of natural datasets are extended by nonlinear random feature maps and then translated into power-law scalings of the test loss and how the finite extent of the data's spectral power law causes the model's performance to plateau.
△ Less
Submitted 30 October, 2022;
originally announced October 2022.
-
Machine Learning Applications to Kronian Magnetospheric Reconnection Classification
Authors:
Tadhg M. Garton,
Caitriona M. Jackman,
Andy W. Smith,
Kiley L. Yeakel,
Shane A. Maloney,
Jon Vandegriff
Abstract:
The products of magnetic reconnection in Saturn's magnetotail are identified in magnetometer observations primarily through characteristic deviations in the north-south component of the magnetic field. These magnetic deflections are caused by travelling plasma structures created during reconnection rapidly passing over the observing spacecraft. Identification of these signatures have long been per…
▽ More
The products of magnetic reconnection in Saturn's magnetotail are identified in magnetometer observations primarily through characteristic deviations in the north-south component of the magnetic field. These magnetic deflections are caused by travelling plasma structures created during reconnection rapidly passing over the observing spacecraft. Identification of these signatures have long been performed by eye, and more recently through semi-automated methods, however these methods are often limited through a required human verification step. Here, we present a fully automated, supervised learning, feed forward neural network model to identify evidence of reconnection in the Kronian magnetosphere with the three magnetic field components observed by the Cassini spacecraft in Kronocentric radial-theta-phi (KRTP) coordinates as input. This model is constructed from a catalogue of reconnection events which covers three years of observations with a total of 2093 classified events, categorized into plasmoids, travelling compression regions and dipolarizations. This neural network model is capable of rapidly identifying reconnection events in large time-span Cassini datasets, tested against the full year 2010 with a high level of accuracy (87%), true skill score (0.76), and Heidke skill score (0.73). From this model, a full cataloguing and examination of magnetic reconnection events in the Kronian magnetosphere across Cassini's near Saturn lifetime is now possible.
△ Less
Submitted 1 April, 2021;
originally announced April 2021.
-
Single-Frame Super-Resolution of Solar Magnetograms: Investigating Physics-Based Metrics \& Losses
Authors:
Anna Jungbluth,
Xavier Gitiaux,
Shane A. Maloney,
Carl Shneider,
Paul J. Wright,
Alfredo Kalaitzis,
Michel Deudon,
Atılım Güneş Baydin,
Yarin Gal,
Andrés Muñoz-Jaramillo
Abstract:
Breakthroughs in our understanding of physical phenomena have traditionally followed improvements in instrumentation. Studies of the magnetic field of the Sun, and its influence on the solar dynamo and space weather events, have benefited from improvements in resolution and measurement frequency of new instruments. However, in order to fully understand the solar cycle, high-quality data across tim…
▽ More
Breakthroughs in our understanding of physical phenomena have traditionally followed improvements in instrumentation. Studies of the magnetic field of the Sun, and its influence on the solar dynamo and space weather events, have benefited from improvements in resolution and measurement frequency of new instruments. However, in order to fully understand the solar cycle, high-quality data across time-scales longer than the typical lifespan of a solar instrument are required. At the moment, discrepancies between measurement surveys prevent the combined use of all available data. In this work, we show that machine learning can help bridge the gap between measurement surveys by learning to \textbf{super-resolve} low-resolution magnetic field images and \textbf{translate} between characteristics of contemporary instruments in orbit. We also introduce the notion of physics-based metrics and losses for super-resolution to preserve underlying physics and constrain the solution space of possible super-resolution outputs.
△ Less
Submitted 4 November, 2019;
originally announced November 2019.
-
Probabilistic Super-Resolution of Solar Magnetograms: Generating Many Explanations and Measuring Uncertainties
Authors:
Xavier Gitiaux,
Shane A. Maloney,
Anna Jungbluth,
Carl Shneider,
Paul J. Wright,
Atılım Güneş Baydin,
Michel Deudon,
Yarin Gal,
Alfredo Kalaitzis,
Andrés Muñoz-Jaramillo
Abstract:
Machine learning techniques have been successfully applied to super-resolution tasks on natural images where visually pleasing results are sufficient. However in many scientific domains this is not adequate and estimations of errors and uncertainties are crucial. To address this issue we propose a Bayesian framework that decomposes uncertainties into epistemic and aleatoric uncertainties. We test…
▽ More
Machine learning techniques have been successfully applied to super-resolution tasks on natural images where visually pleasing results are sufficient. However in many scientific domains this is not adequate and estimations of errors and uncertainties are crucial. To address this issue we propose a Bayesian framework that decomposes uncertainties into epistemic and aleatoric uncertainties. We test the validity of our approach by super-resolving images of the Sun's magnetic field and by generating maps measuring the range of possible high resolution explanations compatible with a given low resolution magnetogram.
△ Less
Submitted 4 November, 2019;
originally announced November 2019.
