Showing 1–8 of 8 results for author: Pettit, C

Exploring hierarchical framework of nonlinear sparse Bayesian learning algorithm through numerical investigations

Authors: Nastaran Dabiran, Brandon Robinson, Rimple Sandhu, Mohammad Khalil, Chris L. Pettit, Dominique Poirel, Abhijit Sarkar

Abstract: Sparse Bayesian learning (SBL) has been extensively utilized in data-driven modeling to combat the issue of overfitting. While SBL excels in linear-in-parameter models, its direct applicability is limited in models where observations possess nonlinear relationships with unknown parameters. Recently, a semi-analytical Bayesian framework known as nonlinear sparse Bayesian learning (NSBL) was introdu… ▽ More Sparse Bayesian learning (SBL) has been extensively utilized in data-driven modeling to combat the issue of overfitting. While SBL excels in linear-in-parameter models, its direct applicability is limited in models where observations possess nonlinear relationships with unknown parameters. Recently, a semi-analytical Bayesian framework known as nonlinear sparse Bayesian learning (NSBL) was introduced by the authors to induce sparsity among model parameters during the Bayesian inversion of nonlinear-in-parameter models. NSBL relies on optimally selecting the hyperparameters of sparsity-inducing Gaussian priors. It is inherently an approximate method since the uncertainty in the hyperparameter posterior is disregarded as we instead seek the maximum a posteriori (MAP) estimate of the hyperparameters (type-II MAP estimate). This paper aims to investigate the hierarchical structure that forms the basis of NSBL and validate its accuracy through a comparison with a one-level hierarchical Bayesian inference as a benchmark in the context of three numerical experiments: (i) a benchmark linear regression example with Gaussian prior and Gaussian likelihood, (ii) the same regression problem with a highly non-Gaussian prior, and (iii) an example of a dynamical system with a non-Gaussian prior and a highly non-Gaussian likelihood function, to explore the performance of the algorithm in these new settings. Through these numerical examples, it can be shown that NSBL is well-suited for physics-based models as it can be readily applied to models with non-Gaussian prior distributions and non-Gaussian likelihood functions. Moreover, we illustrate the accuracy of the NSBL algorithm as an approximation to the one-level hierarchical Bayesian inference and its ability to reduce the computational cost while adequately exploring the parameter posteriors. △ Less

Submitted 23 October, 2023; originally announced October 2023.

Discovering Language Model Behaviors with Model-Written Evaluations

Authors: Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion , et al. (38 additional authors not shown)

Abstract: As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data sources (which are not always available). Here, we automatically generate evaluations with LMs. We explore approaches with varying amounts of human effort, from inst… ▽ More As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data sources (which are not always available). Here, we automatically generate evaluations with LMs. We explore approaches with varying amounts of human effort, from instructing LMs to write yes/no questions to making complex Winogender schemas with multiple stages of LM-based generation and filtering. Crowdworkers rate the examples as highly relevant and agree with 90-100% of labels, sometimes more so than corresponding human-written datasets. We generate 154 datasets and discover new cases of inverse scaling where LMs get worse with size. Larger LMs repeat back a dialog user's preferred answer ("sycophancy") and express greater desire to pursue concerning goals like resource acquisition and goal preservation. We also find some of the first examples of inverse scaling in RL from Human Feedback (RLHF), where more RLHF makes LMs worse. For example, RLHF makes LMs express stronger political views (on gun rights and immigration) and a greater desire to avoid shut down. Overall, LM-written evaluations are high-quality and let us quickly discover many novel LM behaviors. △ Less

Submitted 19 December, 2022; originally announced December 2022.

Comments: for associated data visualizations, see https://www.evals.anthropic.com/model-written/ for full datasets, see https://github.com/anthropics/evals

Measuring Progress on Scalable Oversight for Large Language Models

Authors: Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilė Lukošiūtė, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Jackson Kernion, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse , et al. (21 additional authors not shown)

Abstract: Developing safe and useful general-purpose AI systems will require us to make progress on scalable oversight: the problem of supervising systems that potentially outperform us on most skills relevant to the task at hand. Empirical work on this problem is not straightforward, since we do not yet have systems that broadly exceed our abilities. This paper discusses one of the major ways we think abou… ▽ More Developing safe and useful general-purpose AI systems will require us to make progress on scalable oversight: the problem of supervising systems that potentially outperform us on most skills relevant to the task at hand. Empirical work on this problem is not straightforward, since we do not yet have systems that broadly exceed our abilities. This paper discusses one of the major ways we think about this problem, with a focus on ways it can be studied empirically. We first present an experimental design centered on tasks for which human specialists succeed but unaided humans and current general AI systems fail. We then present a proof-of-concept experiment meant to demonstrate a key feature of this experimental design and show its viability with two question-answering tasks: MMLU and time-limited QuALITY. On these tasks, we find that human participants who interact with an unreliable large-language-model dialog assistant through chat -- a trivial baseline strategy for scalable oversight -- substantially outperform both the model alone and their own unaided performance. These results are an encouraging sign that scalable oversight will be tractable to study with present models and bolster recent findings that large language models can productively assist humans with difficult tasks. △ Less

Submitted 11 November, 2022; v1 submitted 4 November, 2022; originally announced November 2022.

Comments: v2 fixes a few typos from v1

Probabilistic Prediction of Coalescence Flutter Using Measurements: Application to the Flutter Margin Method

Authors: Sandip Chajjed, Mohammad Khalil, Dominique Poirel, Chris Pettit, Abhijit Sarkar

Abstract: Zimmerman and Weissenburger's flutter margin method is widely used to estimate the aeroelastic coalescence flutter speed. In contrast to aeroelastic decay rates, the flutter margin exhibits monotonic decay with respect to airspeed redering it effective in extrapolating the flutter speed using flight test data conducted at pre-flutter airspeeds. This paper reports the generalization of the Bayesian… ▽ More Zimmerman and Weissenburger's flutter margin method is widely used to estimate the aeroelastic coalescence flutter speed. In contrast to aeroelastic decay rates, the flutter margin exhibits monotonic decay with respect to airspeed redering it effective in extrapolating the flutter speed using flight test data conducted at pre-flutter airspeeds. This paper reports the generalization of the Bayesian formulation of the flutter margin method by Khalil et al. developed to tackle measurement and modeling uncertainties. This paper improves the predictive performance of the previous algorithm by incorporating the joint prior of aeroelastic modal frequencies and decay rates among airspeeds in order to better estimate the joint posterior of modal parameters using observational data. The modal parameter prior is constructed using the classical two-degree-of-freedom pitch-plunge aeroelastic model whose system matrices (e.g. structural stiffness and damping matrices) vary randomly. Such joint modal parameter prior enforces statistical dependence among posteriors of modal parameters and the associated flutter margins across airspeeds. Numerical studies demonstrate a considerable reduction of uncertainties on the predicted flutter speed obtained from the generalized Bayesian flutter margin method. This improved algorithm can cut cost by reducing the number of flight tests and better assess the uncertainty against aeroelastic flutter. △ Less

Submitted 24 October, 2022; originally announced October 2022.

Encoding nonlinear and unsteady aerodynamics of limit cycle oscillations using nonlinear sparse Bayesian learning

Authors: Rimple Sandhu, Brandon Robinson, Mohammad Khalil, Chris L. Pettit, Dominique Poirel, Abhijit Sarkar

Abstract: This paper investigates the applicability of a recently-proposed nonlinear sparse Bayesian learning (NSBL) algorithm to identify and estimate the complex aerodynamics of limit cycle oscillations. NSBL provides a semi-analytical framework for determining the data-optimal sparse model nested within a (potentially) over-parameterized model. This is particularly relevant to nonlinear dynamical systems… ▽ More This paper investigates the applicability of a recently-proposed nonlinear sparse Bayesian learning (NSBL) algorithm to identify and estimate the complex aerodynamics of limit cycle oscillations. NSBL provides a semi-analytical framework for determining the data-optimal sparse model nested within a (potentially) over-parameterized model. This is particularly relevant to nonlinear dynamical systems where modelling approaches involve the use of physics-based and data-driven components. In such cases, the data-driven components, where analytical descriptions of the physical processes are not readily available, are often prone to overfitting, meaning that the empirical aspects of these models will often involve the calibration of an unnecessarily large number of parameters. While it may be possible to fit the data well, this can become an issue when using these models for predictions in regimes that are different from those where the data was recorded. In view of this, it is desirable to not only calibrate the model parameters, but also to identify the optimal compromise between data-fit and model complexity. In this paper, this is achieved for an aeroelastic system where the structural dynamics are well-known and described by a differential equation model, coupled with a semi-empirical aerodynamic model for laminar separation flutter resulting in low-amplitude limit cycle oscillations. For the purpose of illustrating the benefit of the algorithm, in this paper, we use synthetic data to demonstrate the ability of the algorithm to correctly identify the optimal model and model parameters, given a known data-generating model. The synthetic data are generated from a forward simulation of a known differential equation model with parameters selected so as to mimic the dynamics observed in wind-tunnel experiments. △ Less

Submitted 17 October, 2022; originally announced October 2022.

Robust Bayesian state and parameter estimation framework for stochastic dynamical systems with combined time-varying and time-invariant parameters

Authors: Philippe Bisaillon, Brandon Robinson, Mohammad Khalil, Chris L. Pettit, Dominique Poirel, Abhijit Sarkar

Abstract: We consider state and parameter estimation for a dynamical system having both time-varying and time-invariant parameters. It has been shown that the robustness of the Markov Chain Monte Carlo (MCMC) algorithm for estimating time-invariant parameters alongside nonlinear filters for state estimation provided more reliable estimates than the estimates obtained solely using nonlinear filters for combi… ▽ More We consider state and parameter estimation for a dynamical system having both time-varying and time-invariant parameters. It has been shown that the robustness of the Markov Chain Monte Carlo (MCMC) algorithm for estimating time-invariant parameters alongside nonlinear filters for state estimation provided more reliable estimates than the estimates obtained solely using nonlinear filters for combined state and parameter estimation. In a similar fashion, we adopt the extended Kalman filter (EKF) for state estimation and the estimation of the time-varying system parameters, but reserve the task of estimating time-invariant parameters to the MCMC algorithm. In a standard method, we augment the state vector to include the original states of the system and the subset of the parameters that are time-varying. Each time-varying parameter is perturbed by a white noise process, and we treat the strength of this artificial noise as an additional time-invariant parameter to be estimated by MCMC, circumventing the need for manual tuning. Conventionally, both time-varying and time-invariant parameters are appended in the state vector, and thus for the purpose of estimation, both are free to vary in time. However, allowing time-invariant system parameters to vary in time introduces artificial dynamics into the system, which we avoid by treating these time-invariant parameters as static and estimating them using MCMC. Furthermore, by estimating the time-invariant parameters by MCMC, the augmented state is smaller and the nonlinearity in the ensuing state space model will tend to be weaker than in the conventional approach. We illustrate the above-described approach for a simple dynamical system in which some model parameters are time-varying, while the remaining parameters are time-invariant. △ Less

Submitted 16 October, 2022; originally announced October 2022.

Domain Decomposition of Stochastic PDEs: Development of Probabilistic Wirebasket-based Two-level Preconditioners

Authors: Ajit Desai, Mohammad Khalil, Chris L. Pettit, Dominique Poirel, Abhijit Sarkar

Abstract: Realistic physical phenomena exhibit random fluctuations across many scales in the input and output processes. Models of these phenomena require stochastic PDEs. For three-dimensional coupled (vector-valued) stochastic PDEs (SPDEs), for instance, arising in linear elasticity, the existing two-level domain decomposition solvers with the vertex-based coarse grid show poor numerical and parallel scal… ▽ More Realistic physical phenomena exhibit random fluctuations across many scales in the input and output processes. Models of these phenomena require stochastic PDEs. For three-dimensional coupled (vector-valued) stochastic PDEs (SPDEs), for instance, arising in linear elasticity, the existing two-level domain decomposition solvers with the vertex-based coarse grid show poor numerical and parallel scalabilities. Therefore, new algorithms with a better resolved coarse grid are needed. The probabilistic wirebasket-based coarse grid for a two-level solver is devised in three dimensions. This enriched coarse grid provides an efficient mechanism for global error propagation and thus improves the convergence. This development enhances the scalability of the two-level solver in handling stochastic PDEs in three dimensions. Numerical and parallel scalabilities of this algorithm are studied using MPI and PETSc libraries on high-performance computing (HPC) systems. Implementational challenges of the intrusive spectral stochastic finite element methods (SSFEM) are addressed by coupling domain decomposition solvers with FEniCS general purpose finite element package. This work generalizes the applications of intrusive SSFEM to tackle a variety of stochastic PDEs and emphasize the usefulness of the domain decomposition-based solvers and HPC for uncertainty quantification. △ Less

Submitted 22 August, 2022; originally announced August 2022.

Geospatial Big Data Handling Theory and Methods: A Review and Research Challenges

Authors: S. Li, S. Dragicevic, F. Anton, M. Sester, S. Winter, A. Coltekin, C. Pettit, B. Jiang, J. Haworth, A. Stein, T. Cheng

Abstract: Big data has now become a strong focus of global interest that is increasingly attracting the attention of academia, industry, government and other organizations. Big data can be situated in the disciplinary area of traditional geospatial data handling theory and methods. The increasing volume and varying format of collected geospatial big data presents challenges in storing, managing, processing,… ▽ More Big data has now become a strong focus of global interest that is increasingly attracting the attention of academia, industry, government and other organizations. Big data can be situated in the disciplinary area of traditional geospatial data handling theory and methods. The increasing volume and varying format of collected geospatial big data presents challenges in storing, managing, processing, analyzing, visualizing and verifying the quality of data. This has implications for the quality of decisions made with big data. Consequently, this position paper of the International Society for Photogrammetry and Remote Sensing (ISPRS) Technical Commission II (TC II) revisits the existing geospatial data handling methods and theories to determine if they are still capable of handling emerging geospatial big data. Further, the paper synthesises problems, major issues and challenges with current developments as well as recommending what needs to be developed further in the near future. Keywords: Big data, Geospatial, Data handling, Analytics, Spatial Modeling, Review △ Less

Submitted 10 November, 2015; originally announced November 2015.

Comments: 25 pages, 3 figures

Journal ref: ISPRS International Journal of Geo-Information, 5(5), 55, 2016