Showing 1–50 of 59 results for author: Nott, D J

Model-Free Local Recalibration of Neural Networks

Authors: R. Torres, D. J. Nott, S. A. Sisson, T. Rodrigues, J. G. Reis, G. S. Rodrigues

Abstract: Artificial neural networks (ANNs) are highly flexible predictive models. However, reliably quantifying uncertainty for their predictions is a continuing challenge. There has been much recent work on "recalibration" of predictive distributions for ANNs, so that forecast probabilities for events of interest are consistent with certain frequency evaluations of them. Uncalibrated probabilistic forecas… ▽ More Artificial neural networks (ANNs) are highly flexible predictive models. However, reliably quantifying uncertainty for their predictions is a continuing challenge. There has been much recent work on "recalibration" of predictive distributions for ANNs, so that forecast probabilities for events of interest are consistent with certain frequency evaluations of them. Uncalibrated probabilistic forecasts are of limited use for many important decision-making tasks. To address this issue, we propose a localized recalibration of ANN predictive distributions using the dimension-reduced representation of the input provided by the ANN hidden layers. Our novel method draws inspiration from recalibration techniques used in the literature on approximate Bayesian computation and likelihood-free inference methods. Most existing calibration methods for ANNs can be thought of as calibrating either on the input layer, which is difficult when the input is high-dimensional, or the output layer, which may not be sufficiently flexible. Through a simulation study, we demonstrate that our method has good performance compared to alternative approaches, and explore the benefits that can be achieved by localizing the calibration based on different layers of the network. Finally, we apply our proposed method to a diamond price prediction problem, demonstrating the potential of our approach to improve prediction and uncertainty quantification in real-world applications. △ Less

Submitted 8 March, 2024; originally announced March 2024.

Comments: 25 pages, 5 figures

MSC Class: 62G07 (Primary); 68T07; 68T37 (Secondary); 68Q10 ACM Class: G.3; I.5.1; I.6.4

Deep mixture of linear mixed models for complex longitudinal data

Authors: Lucas Kock, Nadja Klein, David J. Nott

Abstract: Mixtures of linear mixed models are widely used for modelling longitudinal data for which observation times differ between subjects. In typical applications, temporal trends are described using a basis expansion, with basis coefficients treated as random effects varying by subject. Additional random effects can describe variation between mixture components, or other known sources of variation in c… ▽ More Mixtures of linear mixed models are widely used for modelling longitudinal data for which observation times differ between subjects. In typical applications, temporal trends are described using a basis expansion, with basis coefficients treated as random effects varying by subject. Additional random effects can describe variation between mixture components, or other known sources of variation in complex experimental designs. A key advantage of these models is that they provide a natural mechanism for clustering, which can be helpful for interpretation in many applications. Current versions of mixtures of linear mixed models are not specifically designed for the case where there are many observations per subject and a complex temporal trend, which requires a large number of basis functions to capture. In this case, the subject-specific basis coefficients are a high-dimensional random effects vector, for which the covariance matrix is hard to specify and estimate, especially if it varies between mixture components. To address this issue, we consider the use of recently-developed deep mixture of factor analyzers models as the prior for the random effects. The resulting deep mixture of linear mixed models is well-suited to high-dimensional settings, and we describe an efficient variational inference approach to posterior computation. The efficacy of the method is demonstrated on both real and simulated data. △ Less

Submitted 13 November, 2023; originally announced November 2023.

Cutting Feedback in Misspecified Copula Models

Authors: Michael Stanley Smith, Weichang Yu, David J. Nott, David Frazier

Abstract: In copula models the marginal distributions and copula function are specified separately. We treat these as two modules in a modular Bayesian inference framework, and propose conducting modified Bayesian inference by "cutting feedback". Cutting feedback limits the influence of potentially misspecified modules in posterior inference. We consider two types of cuts. The first limits the influence of… ▽ More In copula models the marginal distributions and copula function are specified separately. We treat these as two modules in a modular Bayesian inference framework, and propose conducting modified Bayesian inference by "cutting feedback". Cutting feedback limits the influence of potentially misspecified modules in posterior inference. We consider two types of cuts. The first limits the influence of a misspecified copula on inference for the marginals, which is a Bayesian analogue of the popular Inference for Margins (IFM) estimator. The second limits the influence of misspecified marginals on inference for the copula parameters by using a pseudo likelihood of the ranks to define the cut model. We establish that if only one of the modules is misspecified, then the appropriate cut posterior gives accurate uncertainty quantification asymptotically for the parameters in the other module. Computation of the cut posteriors is difficult, and new variational inference methods to do so are proposed. The efficacy of the new methodology is demonstrated using both simulated data and a substantive multivariate time series application from macroeconomic forecasting. In the latter, cutting feedback from misspecified marginals to a 1096 dimension copula improves posterior inference and predictive accuracy greatly, compared to conventional Bayesian inference. △ Less

Submitted 27 June, 2024; v1 submitted 5 October, 2023; originally announced October 2023.

Bayesian inference for misspecified generative models

Authors: David J. Nott, Christopher Drovandi, David T. Frazier

Abstract: Bayesian inference is a powerful tool for combining information in complex settings, a task of increasing importance in modern applications. However, Bayesian inference with a flawed model can produce unreliable conclusions. This review discusses approaches to performing Bayesian inference when the model is misspecified, where by misspecified we mean that the analyst is unwilling to act as if the… ▽ More Bayesian inference is a powerful tool for combining information in complex settings, a task of increasing importance in modern applications. However, Bayesian inference with a flawed model can produce unreliable conclusions. This review discusses approaches to performing Bayesian inference when the model is misspecified, where by misspecified we mean that the analyst is unwilling to act as if the model is correct. Much has been written about this topic, and in most cases we do not believe that a conventional Bayesian analysis is meaningful when there is serious model misspecification. Nevertheless, in some cases it is possible to use a well-specified model to give meaning to a Bayesian analysis of a misspecified model and we will focus on such cases. Three main classes of methods are discussed - restricted likelihood methods, which use a model based on a non-sufficient summary of the original data; modular inference methods which use a model constructed from coupled submodels and some of the submodels are correctly specified; and the use of a reference model to construct a projected posterior or predictive distribution for a simplified model considered to be useful for prediction or interpretation. △ Less

Submitted 18 May, 2023; v1 submitted 15 May, 2023; originally announced May 2023.

Comments: Review paper under submission for Annual Review of Statistics and its Application

Dropout Regularization in Extended Generalized Linear Models based on Double Exponential Families

Authors: Benedikt Lütke Schwienhorst, Lucas Kock, Nadja Klein, David J. Nott

Abstract: Even though dropout is a popular regularization technique, its theoretical properties are not fully understood. In this paper we study dropout regularization in extended generalized linear models based on double exponential families, for which the dispersion parameter can vary with the features. A theoretical analysis shows that dropout regularization prefers rare but important features in both th… ▽ More Even though dropout is a popular regularization technique, its theoretical properties are not fully understood. In this paper we study dropout regularization in extended generalized linear models based on double exponential families, for which the dispersion parameter can vary with the features. A theoretical analysis shows that dropout regularization prefers rare but important features in both the mean and dispersion, generalizing an earlier result for conventional generalized linear models. To illustrate, we apply dropout to adaptive smoothing with B-splines, where both the mean and dispersion parameters are modeled flexibly. The important B-spline basis functions can be thought of as rare features, and we confirm in experiments that dropout is an effective form of regularization for mean and dispersion parameters that improves on a penalized maximum likelihood approach with an explicit smoothness penalty. An application to traffic detection data from Berlin further illustrates the benefits of our method. △ Less

Submitted 28 July, 2024; v1 submitted 11 May, 2023; originally announced May 2023.

Comments: Added a real-world data application and comments on approximations in the appendix

Bayesian Synthetic Likelihood

Authors: David T. Frazier, Christopher Drovandi, David J. Nott

Abstract: Bayesian statistics is concerned with conducting posterior inference for the unknown quantities in a given statistical model. Conventional Bayesian inference requires the specification of a probabilistic model for the observed data, and the construction of the resulting likelihood function. However, sometimes the model is so complicated that evaluation of the likelihood is infeasible, which render… ▽ More Bayesian statistics is concerned with conducting posterior inference for the unknown quantities in a given statistical model. Conventional Bayesian inference requires the specification of a probabilistic model for the observed data, and the construction of the resulting likelihood function. However, sometimes the model is so complicated that evaluation of the likelihood is infeasible, which renders exact Bayesian inference impossible. Bayesian synthetic likelihood (BSL) is a posterior approximation procedure that can be used to conduct inference in situations where the likelihood is intractable, but where simulation from the model is straightforward. In this entry, we give a high-level presentation of BSL, and its extensions aimed at delivering scalable and robust posterior inferences. △ Less

Submitted 10 May, 2023; v1 submitted 8 May, 2023; originally announced May 2023.

Comments: This manuscript will eventually appear in Wiley StatsRef-Statistics Reference Online, and should not be confused with the original article on Bayesian Synthetic Likelihood by Price, Drovandi, Nott and Lee (2018)

Structured variational approximations with skew normal decomposable graphical models

Authors: Robert Salomone, Xuejun Yu, David J. Nott, Robert Kohn

Abstract: Although there is much recent work developing flexible variational methods for Bayesian computation, Gaussian approximations with structured covariance matrices are often preferred computationally in high-dimensional settings. This paper considers approximate inference methods for complex latent variable models where the posterior is close to Gaussian, but with some skewness in the posterior margi… ▽ More Although there is much recent work developing flexible variational methods for Bayesian computation, Gaussian approximations with structured covariance matrices are often preferred computationally in high-dimensional settings. This paper considers approximate inference methods for complex latent variable models where the posterior is close to Gaussian, but with some skewness in the posterior marginals. We consider skew decomposable graphical models (SDGMs), which are based on the closed skew normal family of distributions, as variational approximations. These approximations can reflect the true posterior conditional independence structure and capture posterior skewness. Different parametrizations are explored for this variational family, and the speed of convergence and quality of the approximation can depend on the parametrization used. To increase flexibility, implicit copula SDGM approximations are also developed, where elementwise transformations of an approximately standardized SDGM random vector are considered. Our parametrization of the implicit copula approximation is novel, even in the special case of a Gaussian approximation. Performance of the methods is examined in a number of real examples involving generalized linear mixed models and state space models, and we conclude that our copula approaches are most accurate, but that the SDGM methods are often nearly as good and have lower computational demands. △ Less

Submitted 7 February, 2023; originally announced February 2023.

Misspecification-robust Sequential Neural Likelihood for Simulation-based Inference

Authors: Ryan P. Kelly, David J. Nott, David T. Frazier, David J. Warne, Chris Drovandi

Abstract: Simulation-based inference techniques are indispensable for parameter estimation of mechanistic and simulable models with intractable likelihoods. While traditional statistical approaches like approximate Bayesian computation and Bayesian synthetic likelihood have been studied under well-specified and misspecified settings, they often suffer from inefficiencies due to wasted model simulations. Neu… ▽ More Simulation-based inference techniques are indispensable for parameter estimation of mechanistic and simulable models with intractable likelihoods. While traditional statistical approaches like approximate Bayesian computation and Bayesian synthetic likelihood have been studied under well-specified and misspecified settings, they often suffer from inefficiencies due to wasted model simulations. Neural approaches, such as sequential neural likelihood (SNL) avoid this wastage by utilising all model simulations to train a neural surrogate for the likelihood function. However, the performance of SNL under model misspecification is unreliable and can result in overconfident posteriors centred around an inaccurate parameter estimate. In this paper, we propose a novel SNL method, which through the incorporation of additional adjustment parameters, is robust to model misspecification and capable of identifying features of the data that the model is not able to recover. We demonstrate the efficacy of our approach through several illustrative examples, where our method gives more accurate point estimates and uncertainty quantification than SNL. △ Less

Submitted 7 March, 2024; v1 submitted 30 January, 2023; originally announced January 2023.

Accurate semi-modular posterior inference with a user-defined loss function

Authors: David T. Frazier, David J. Nott

Abstract: Bayesian inference has widely acknowledged advantages in many problems, but it can also be unreliable if the model is misspecified. Bayesian modular inference is concerned with inference in complex models which have been specified through a collection of coupled sub-models. The sub-models are called modules in the literature, and they often arise from modeling different data sources, or from combi… ▽ More Bayesian inference has widely acknowledged advantages in many problems, but it can also be unreliable if the model is misspecified. Bayesian modular inference is concerned with inference in complex models which have been specified through a collection of coupled sub-models. The sub-models are called modules in the literature, and they often arise from modeling different data sources, or from combining domain knowledge from different disciplines. When some modules are misspecified, cutting feedback is a widely used Bayesian modular inference method which ensures that information from suspect model components is not used in making inferences about parameters in correctly specified modules. However, in general settings it is difficult to decide when this ``cut posterior'' is preferable to the exact posterior. When misspecification is not severe, cutting feedback may increase the uncertainty in Bayesian posterior inference greatly without reducing estimation bias substantially. This motivates semi-modular inference methods, which avoid the binary cut of cutting feedback approaches. In this work, using a local model misspecification framework, we provide the first precise formulation of the the bias-variance trade-off that has motivated the literature on semi-modular inference. We then implement a mixture-based semi-modular inference approach, demonstrating theoretically that it delivers inferences that are more accurate, in terms of a user-defined loss function, than if either the cut or full posterior were used by themselves. The new method is demonstrated in a number of applications. △ Less

Submitted 28 July, 2023; v1 submitted 25 January, 2023; originally announced January 2023.

Better Together: pooling information in likelihood-free inference

Authors: David T. Frazier, Christopher Drovandi, David J. Nott

Abstract: Likelihood-free inference (LFI) methods, such as Approximate Bayesian computation (ABC), are now routinely applied to conduct inference in complex models. While the application of LFI is now commonplace, the choice of which summary statistics to use in the construction of the posterior remains an open question that is fraught with both practical and theoretical challenges. Instead of choosing a si… ▽ More Likelihood-free inference (LFI) methods, such as Approximate Bayesian computation (ABC), are now routinely applied to conduct inference in complex models. While the application of LFI is now commonplace, the choice of which summary statistics to use in the construction of the posterior remains an open question that is fraught with both practical and theoretical challenges. Instead of choosing a single vector of summaries on which to base inference, we suggest a new pooled posterior and show how to optimally combine inferences from different LFI posteriors. This pooled approach to inference obviates the need to choose a single vector of summaries, or even a single LFI algorithm, and delivers guaranteed inferential accuracy without requiring the computational resources associated with sampling LFI posteriors in high-dimensions. We illustrate this approach through a series of benchmark examples considered in the LFI literature. △ Less

Submitted 5 December, 2022; originally announced December 2022.

Bayesian score calibration for approximate models

Authors: Joshua J Bon, David J Warne, David J Nott, Christopher Drovandi

Abstract: Scientists continue to develop increasingly complex mechanistic models to reflect their knowledge more realistically. Statistical inference using these models can be challenging since the corresponding likelihood function is often intractable and model simulation may be computationally burdensome. Fortunately, in many of these situations, it is possible to adopt a surrogate model or approximate li… ▽ More Scientists continue to develop increasingly complex mechanistic models to reflect their knowledge more realistically. Statistical inference using these models can be challenging since the corresponding likelihood function is often intractable and model simulation may be computationally burdensome. Fortunately, in many of these situations, it is possible to adopt a surrogate model or approximate likelihood function. It may be convenient to conduct Bayesian inference directly with the surrogate, but this can result in bias and poor uncertainty quantification. In this paper we propose a new method for adjusting approximate posterior samples to reduce bias and produce more accurate uncertainty quantification. We do this by optimizing a transform of the approximate posterior that maximizes a scoring rule. Our approach requires only a (fixed) small number of complex model simulations and is numerically stable. We demonstrate good performance of the new method on several examples of increasing complexity. △ Less

Submitted 27 October, 2023; v1 submitted 10 November, 2022; originally announced November 2022.

Comments: 27 pages, 8 figures, 5 tables

Improving the Accuracy of Marginal Approximations in Likelihood-Free Inference via Localisation

Authors: Christopher Drovandi, David J Nott, David T Frazier

Abstract: Likelihood-free methods are an essential tool for performing inference for implicit models which can be simulated from, but for which the corresponding likelihood is intractable. However, common likelihood-free methods do not scale well to a large number of model parameters. A promising approach to high-dimensional likelihood-free inference involves estimating low-dimensional marginal posteriors b… ▽ More Likelihood-free methods are an essential tool for performing inference for implicit models which can be simulated from, but for which the corresponding likelihood is intractable. However, common likelihood-free methods do not scale well to a large number of model parameters. A promising approach to high-dimensional likelihood-free inference involves estimating low-dimensional marginal posteriors by conditioning only on summary statistics believed to be informative for the low-dimensional component, and then combining the low-dimensional approximations in some way. In this paper, we demonstrate that such low-dimensional approximations can be surprisingly poor in practice for seemingly intuitive summary statistic choices. We describe an idealized low-dimensional summary statistic that is, in principle, suitable for marginal estimation. However, a direct approximation of the idealized choice is difficult in practice. We thus suggest an alternative approach to marginal estimation which is easier to implement and automate. Given an initial choice of low-dimensional summary statistic that might only be informative about a marginal posterior location, the new method improves performance by first crudely localising the posterior approximation using all the summary statistics to ensure global identifiability, followed by a second step that hones in on an accurate low-dimensional approximation using the low-dimensional summary statistic. We show that the posterior this approach targets can be represented as a logarithmic pool of posterior distributions based on the low-dimensional and full summary statistics, respectively. The good performance of our method is illustrated in several examples. △ Less

Submitted 14 July, 2022; originally announced July 2022.

Comments: 30 pages, 9 figures

Robust Bayesian inference in complex models with possibility theory

Authors: Jeremie Houssineau, David J. Nott

Abstract: We propose a general solution to the problem of robust Bayesian inference in complex settings where outliers may be present. In practice, the automation of robust Bayesian analyses is important in the many applications involving large and complex datasets. The proposed solution relies on a reformulation of Bayesian inference based on possibility theory, and leverages the observation that, in this… ▽ More We propose a general solution to the problem of robust Bayesian inference in complex settings where outliers may be present. In practice, the automation of robust Bayesian analyses is important in the many applications involving large and complex datasets. The proposed solution relies on a reformulation of Bayesian inference based on possibility theory, and leverages the observation that, in this context, the marginal likelihood of the data assesses the consistency between prior and likelihood rather than model fitness. Our approach does not require additional parameters in its simplest form and has a limited impact on the computational complexity when compared to non-robust solutions. The generality of our solution is demonstrated via applications on simulated and real data including matrix estimation and change-point detection. △ Less

Submitted 14 April, 2022; originally announced April 2022.

Modularized Bayesian analyses and cutting feedback in likelihood-free inference

Authors: Atlanta Chakraborty, David J. Nott, Christopher Drovandi, David T. Frazier, Scott A. Sisson

Abstract: There has been much recent interest in modifying Bayesian inference for misspecified models so that it is useful for specific purposes. One popular modified Bayesian inference method is "cutting feedback" which can be used when the model consists of a number of coupled modules, with only some of the modules being misspecified. Cutting feedback methods represent the full posterior distribution in t… ▽ More There has been much recent interest in modifying Bayesian inference for misspecified models so that it is useful for specific purposes. One popular modified Bayesian inference method is "cutting feedback" which can be used when the model consists of a number of coupled modules, with only some of the modules being misspecified. Cutting feedback methods represent the full posterior distribution in terms of conditional and sequential components, and then modify some terms in such a representation based on the modular structure for specification or computation of a modified posterior distribution. The main goal of this is to avoid contamination of inferences for parameters of interest by misspecified modules. Computation for cut posterior distributions is challenging, and here we consider cutting feedback for likelihood-free inference based on Gaussian mixture approximations to the joint distribution of parameters and data summary statistics. We exploit the fact that marginal and conditional distributions of a Gaussian mixture are Gaussian mixtures to give explicit approximations to marginal or conditional posterior distributions so that we can easily approximate cut posterior analyses. The mixture approach allows repeated approximation of posterior distributions for different data based on a single mixture fit, which is important for model checks which aid in the decision of whether to "cut". A semi-modular approach to likelihood-free inference where feedback is partially cut is also developed. The benefits of the method are illustrated in two challenging examples, a collective cell spreading model and a continuous time model for asset returns with jumps. △ Less

Submitted 18 March, 2022; originally announced March 2022.

Weakly informative priors and prior-data conflict checking for likelihood-free inference

Authors: Atlanta Chakraborty, David J. Nott, Michael Evans

Abstract: Bayesian likelihood-free inference, which is used to perform Bayesian inference when the likelihood is intractable, enjoys an increasing number of important scientific applications. However, many aspects of a Bayesian analysis become more challenging in the likelihood-free setting. One example of this is prior-data conflict checking, where the goal is to assess whether the information in the data… ▽ More Bayesian likelihood-free inference, which is used to perform Bayesian inference when the likelihood is intractable, enjoys an increasing number of important scientific applications. However, many aspects of a Bayesian analysis become more challenging in the likelihood-free setting. One example of this is prior-data conflict checking, where the goal is to assess whether the information in the data and the prior are inconsistent. Conflicts of this kind are important to detect, since they may reveal problems in an investigator's understanding of what are relevant values of the parameters, and can result in sensitivity of Bayesian inferences to the prior. Here we consider methods for prior-data conflict checking which are applicable regardless of whether the likelihood is tractable or not. In constructing our checks, we consider checking statistics based on prior-to-posterior Kullback-Leibler divergences. The checks are implemented using mixture approximations to the posterior distribution and closed-form approximations to Kullback-Leibler divergences for mixtures, which make Monte Carlo approximation of reference distributions for calibration computationally feasible. When prior-data conflicts occur, it is useful to consider weakly informative prior specifications in alternative analyses as part of a sensitivity analysis. As a main application of our methodology, we develop a technique for searching for weakly informative priors in likelihood-free inference, where the notion of a weakly informative prior is formalized using prior-data conflict checks. The methods are demonstrated in three examples. △ Less

Submitted 21 February, 2022; originally announced February 2022.

Cutting feedback and modularized analyses in generalized Bayesian inference

Authors: David T. Frazier, David J. Nott

Abstract: This work considers Bayesian inference under misspecification for complex statistical models comprised of simpler submodels, referred to as modules, that are coupled together. Such ``multi-modular" models often arise when combining information from different data sources, where there is a module for each data source. When some of the modules are misspecified, the challenges of Bayesian inference u… ▽ More This work considers Bayesian inference under misspecification for complex statistical models comprised of simpler submodels, referred to as modules, that are coupled together. Such ``multi-modular" models often arise when combining information from different data sources, where there is a module for each data source. When some of the modules are misspecified, the challenges of Bayesian inference under misspecification can sometimes be addressed by using ``cutting feedback" methods, which modify conventional Bayesian inference by limiting the influence of unreliable modules. Here we investigate cutting feedback methods in the context of generalized posterior distributions, which are built from arbitrary loss functions, and present novel findings on their behaviour. We make three main contributions. First, we describe how cutting feedback methods can be defined in the generalized Bayes setting, and discuss the appropriate scaling of the loss functions for different modules to each other and the prior. Second, we derive a novel result about the large sample behaviour of the posterior for a given module's parameters conditional on the parameters of other modules. This formally justifies the use of conditional Laplace approximations, which provide better approximations of conditional posterior distributions compared to conditional distributions from a Laplace approximation of the joint posterior. Our final contribution leverages the large sample approximations of our second contribution to provide convenient diagnostics for understanding the sensitivity of inference to the coupling of the modules, and to implement a new semi-modular posterior approach for conducting robust Bayesian modular inference. The usefulness of the methodology is illustrated in several benchmark examples from the literature on cut model inference. △ Less

Submitted 1 August, 2023; v1 submitted 20 February, 2022; originally announced February 2022.

Variational inference for cutting feedback in misspecified models

Authors: Xuejun Yu, David J. Nott, Michael Stanley Smith

Abstract: Bayesian analyses combine information represented by different terms in a joint Bayesian model. When one or more of the terms is misspecified, it can be helpful to restrict the use of information from suspect model components to modify posterior inference. This is called "cutting feedback", and both the specification and computation of the posterior for such "cut models" is challenging. In this pa… ▽ More Bayesian analyses combine information represented by different terms in a joint Bayesian model. When one or more of the terms is misspecified, it can be helpful to restrict the use of information from suspect model components to modify posterior inference. This is called "cutting feedback", and both the specification and computation of the posterior for such "cut models" is challenging. In this paper, we define cut posterior distributions as solutions to constrained optimization problems, and propose optimization-based variational methods for their computation. These methods are faster than existing Markov chain Monte Carlo (MCMC) approaches for computing cut posterior distributions by an order of magnitude. It is also shown that variational methods allow for the evaluation of computationally intensive conflict checks that can be used to decide whether or not feedback should be cut. Our methods are illustrated in a number of simulated and real examples, including an application where recent methodological advances that combine variational inference and MCMC within the variational optimization are used. △ Less

Submitted 24 June, 2022; v1 submitted 25 August, 2021; originally announced August 2021.

Bayesian clustering using random effects models and predictive projections

Authors: Yinan Mao, David J. Nott

Abstract: Linear mixed models are widely used for analyzing hierarchically structured data involving missingness and unbalanced study designs. We consider a Bayesian clustering method that combines linear mixed models and predictive projections. For each observation, we consider a predictive replicate in which only a subset of the random effects is shared between the observation and its replicate, with the… ▽ More Linear mixed models are widely used for analyzing hierarchically structured data involving missingness and unbalanced study designs. We consider a Bayesian clustering method that combines linear mixed models and predictive projections. For each observation, we consider a predictive replicate in which only a subset of the random effects is shared between the observation and its replicate, with the remainder being integrated out using the conditional prior. Predictive projections are then defined in which the number of distinct values taken by the shared random effects is finite, in order to obtain different clusters. Integrating out some of the random effects acts as a noise filter, allowing the clustering to be focused on only certain chosen features of the data. The method is inspired by methods for Bayesian model checking, in which simulated data replicates from a fitted model are used for model criticism by examining their similarity to the observed data in relevant ways. Here the predictive replicates are used to define similarity between observations in relevant ways for clustering. To illustrate the way our method reveals aspects of the data at different scales, we consider fitting temporal trends in longitudinal data using Fourier cosine bases with a random effect for each basis function, and different clusterings defined by shared random effects for replicates of low or high frequency terms. The method is demonstrated in a series of real examples. △ Less

Submitted 6 July, 2021; v1 submitted 30 June, 2021; originally announced June 2021.

Comments: 27 pages, 16 figures, 2 tables

Variational Inference and Sparsity in High-Dimensional Deep Gaussian Mixture Models

Authors: Lucas Kock, Nadja Klein, David J. Nott

Abstract: Gaussian mixture models are a popular tool for model-based clustering, and mixtures of factor analyzers are Gaussian mixture models having parsimonious factor covariance structure for mixture components. There are several recent extensions of mixture of factor analyzers to deep mixtures, where the Gaussian model for the latent factors is replaced by a mixture of factor analyzers. This construction… ▽ More Gaussian mixture models are a popular tool for model-based clustering, and mixtures of factor analyzers are Gaussian mixture models having parsimonious factor covariance structure for mixture components. There are several recent extensions of mixture of factor analyzers to deep mixtures, where the Gaussian model for the latent factors is replaced by a mixture of factor analyzers. This construction can be iterated to obtain a model with many layers. These deep models are challenging to fit, and we consider Bayesian inference using sparsity priors to further regularize the estimation. A scalable natural gradient variational inference algorithm is developed for fitting the model, and we suggest computationally efficient approaches to the architecture choice using overfitted mixtures where unnecessary components drop out in the estimation. In a number of simulated and two real examples, we demonstrate the versatility of our approach for high-dimensional problems, and demonstrate that the use of sparsity inducing priors can be helpful for obtaining improved clustering results. △ Less

Submitted 1 September, 2021; v1 submitted 4 May, 2021; originally announced May 2021.

Journal ref: Stat Comput 32, 70 (2022)

Synthetic Likelihood in Misspecified Models: Consequences and Corrections

Authors: David T. Frazier, Christopher Drovandi, David J. Nott

Abstract: We analyse the behaviour of the synthetic likelihood (SL) method when the model generating the simulated data differs from the actual data generating process. One of the most common methods to obtain SL-based inferences is via the Bayesian posterior distribution, with this method often referred to as Bayesian synthetic likelihood (BSL). We demonstrate that when the model is misspecified, the BSL p… ▽ More We analyse the behaviour of the synthetic likelihood (SL) method when the model generating the simulated data differs from the actual data generating process. One of the most common methods to obtain SL-based inferences is via the Bayesian posterior distribution, with this method often referred to as Bayesian synthetic likelihood (BSL). We demonstrate that when the model is misspecified, the BSL posterior can be poorly behaved, placing significant posterior mass on values of the model parameters that do not represent the true features observed in the data. Theoretical results demonstrate that in misspecified models the BSL posterior can display a wide range of behaviours depending on the level of model misspecification, including being asymptotically non-Gaussian. Our results suggest that a recently proposed robust BSL approach can ameliorate this behavior and deliver reliable posterior inference under model misspecification. We document all theoretical results using a simple running example. △ Less

Submitted 7 April, 2021; originally announced April 2021.

On a Variational Approximation based Empirical Likelihood ABC Method

Authors: Sanjay Chaudhuri, Subhroshekhar Ghosh, David J. Nott, Kim Cuc Pham

Abstract: Many scientifically well-motivated statistical models in natural, engineering, and environmental sciences are specified through a generative process. However, in some cases, it may not be possible to write down the likelihood for these models analytically. Approximate Bayesian computation (ABC) methods allow Bayesian inference in such situations. The procedures are nonetheless typically computatio… ▽ More Many scientifically well-motivated statistical models in natural, engineering, and environmental sciences are specified through a generative process. However, in some cases, it may not be possible to write down the likelihood for these models analytically. Approximate Bayesian computation (ABC) methods allow Bayesian inference in such situations. The procedures are nonetheless typically computationally intensive. Recently, computationally attractive empirical likelihood-based ABC methods have been suggested in the literature. All of these methods rely on the availability of several suitable analytically tractable estimating equations, and this is sometimes problematic. We propose an easy-to-use empirical likelihood ABC method in this article. First, by using a variational approximation argument as a motivation, we show that the target log-posterior can be approximated as a sum of an expected joint log-likelihood and the differential entropy of the data generating density. The expected log-likelihood is then estimated by an empirical likelihood where the only inputs required are a choice of summary statistic, it's observed value, and the ability to simulate the chosen summary statistics for any parameter value under the model. The differential entropy is estimated from the simulated summaries using traditional methods. Posterior consistency is established for the method, and we discuss the bounds for the required number of simulated summaries in detail. The performance of the proposed method is explored in various examples. △ Less

Submitted 12 November, 2020; originally announced November 2020.

Comments: arXiv admin note: substantial text overlap with arXiv:1810.01675

MSC Class: 62F15; 62F12; 62G07

Detecting conflicting summary statistics in likelihood-free inference

Authors: Yinan Mao, Xueou Wang, David J. Nott, Michael Evans

Abstract: Bayesian likelihood-free methods implement Bayesian inference using simulation of data from the model to substitute for intractable likelihood evaluations. Most likelihood-free inference methods replace the full data set with a summary statistic before performing Bayesian inference, and the choice of this statistic is often difficult. The summary statistic should be low-dimensional for computation… ▽ More Bayesian likelihood-free methods implement Bayesian inference using simulation of data from the model to substitute for intractable likelihood evaluations. Most likelihood-free inference methods replace the full data set with a summary statistic before performing Bayesian inference, and the choice of this statistic is often difficult. The summary statistic should be low-dimensional for computational reasons, while retaining as much information as possible about the parameter. Using a recent idea from the interpretable machine learning literature, we develop some regression-based diagnostic methods which are useful for detecting when different parts of a summary statistic vector contain conflicting information about the model parameters. Conflicts of this kind complicate summary statistic choice, and detecting them can be insightful about model deficiencies and guide model improvement. The diagnostic methods developed are based on regression approaches to likelihood-free inference, in which the regression model estimates the posterior density using summary statistics as features. Deletion and imputation of part of the summary statistic vector within the regression model can remove conflicts and approximate posterior distributions for summary statistic subsets. A larger than expected change in the estimated posterior density following deletion and imputation can indicate a conflict in which inferences of interest are affected. The usefulness of the new methods is demonstrated in a number of real examples. △ Less

Submitted 14 October, 2020; originally announced October 2020.

Deep Distributional Time Series Models and the Probabilistic Forecasting of Intraday Electricity Prices

Authors: Nadja Klein, Michael Stanley Smith, David J. Nott

Abstract: Recurrent neural networks (RNNs) with rich feature vectors of past values can provide accurate point forecasts for series that exhibit complex serial dependence. We propose two approaches to constructing deep time series probabilistic models based on a variant of RNN called an echo state network (ESN). The first is where the output layer of the ESN has stochastic disturbances and a shrinkage prior… ▽ More Recurrent neural networks (RNNs) with rich feature vectors of past values can provide accurate point forecasts for series that exhibit complex serial dependence. We propose two approaches to constructing deep time series probabilistic models based on a variant of RNN called an echo state network (ESN). The first is where the output layer of the ESN has stochastic disturbances and a shrinkage prior for additional regularization. The second approach employs the implicit copula of an ESN with Gaussian disturbances, which is a deep copula process on the feature space. Combining this copula with a non-parametrically estimated marginal distribution produces a deep distributional time series model. The resulting probabilistic forecasts are deep functions of the feature vector and also marginally calibrated. In both approaches, Bayesian Markov chain Monte Carlo methods are used to estimate the models and compute forecasts. The proposed models are suitable for the complex task of forecasting intraday electricity prices. Using data from the Australian National Electricity Market, we show that our deep time series models provide accurate short term probabilistic price forecasts, with the copula model dominating. Moreover, the models provide a flexible framework for incorporating probabilistic forecasts of electricity demand as additional features, which increases upper tail forecast accuracy from the copula model significantly. △ Less

Submitted 27 May, 2021; v1 submitted 5 October, 2020; originally announced October 2020.

Journal ref: Journal of Applied Econometrics (2023), 38( 4), 493-511

Fast and Accurate Variational Inference for Models with Many Latent Variables

Authors: Rubén Loaiza-Maya, Michael Stanley Smith, David J. Nott, Peter J. Danaher

Abstract: Models with a large number of latent variables are often used to fully utilize the information in big or complex data. However, they can be difficult to estimate using standard approaches, and variational inference methods are a popular alternative. Key to the success of these is the selection of an approximation to the target density that is accurate, tractable and fast to calibrate using optimiz… ▽ More Models with a large number of latent variables are often used to fully utilize the information in big or complex data. However, they can be difficult to estimate using standard approaches, and variational inference methods are a popular alternative. Key to the success of these is the selection of an approximation to the target density that is accurate, tractable and fast to calibrate using optimization methods. Most existing choices can be inaccurate or slow to calibrate when there are many latent variables. Here, we propose a family of tractable variational approximations that are more accurate and faster to calibrate for this case. It combines a parsimonious parametric approximation for the parameter posterior, with the exact conditional posterior of the latent variables. We derive a simplified expression for the re-parameterization gradient of the variational lower bound, which is the main ingredient of efficient optimization algorithms used to implement variational estimation. To do so only requires the ability to generate exactly or approximately from the conditional posterior of the latent variables, rather than to compute its density. We illustrate using two complex contemporary econometric examples. The first is a nonlinear multivariate state space model for U.S. macroeconomic variables. The second is a random coefficients tobit model applied to two million sales by 20,000 individuals in a large consumer panel from a marketing study. In both cases, we show that our approximating family is considerably more accurate than mean field or structured Gaussian approximations, and faster than Markov chain Monte Carlo. Last, we show how to implement data sub-sampling in variational inference for our approximation, which can lead to a further reduction in computation time. MATLAB code implementing the method for our examples is included in supplementary material. △ Less

Submitted 18 April, 2021; v1 submitted 15 May, 2020; originally announced May 2020.

Comments: Macroeconomic example was replaced by the bigger and more challenging time varying parameter vector autoregression model with stochastic volatility. Microeconomic example was extended to 20,000 individuals and variational subsampling is also implemented for this example. Small microeconomics example now uses 1000 individuals

MSC Class: 62P20 ACM Class: G.3

Assessment and adjustment of approximate inference algorithms using the law of total variance

Authors: Xuejun Yu, David J. Nott, Minh-Ngoc Tran, Nadja Klein

Abstract: A common method for assessing validity of Bayesian sampling or approximate inference methods makes use of simulated data replicates for parameters drawn from the prior. Under continuity assumptions, quantiles of functions of the simulated parameter values within corresponding posterior distributions are uniformly distributed. Checking for uniformity when a posterior density is approximated numeric… ▽ More A common method for assessing validity of Bayesian sampling or approximate inference methods makes use of simulated data replicates for parameters drawn from the prior. Under continuity assumptions, quantiles of functions of the simulated parameter values within corresponding posterior distributions are uniformly distributed. Checking for uniformity when a posterior density is approximated numerically provides a diagnostic for algorithm validity. Furthermore, adjustments to achieve uniformity can improve the quality of approximate inference methods. A weakness of this general approach is that it seems difficult to extend beyond scalar functions of interest. The present article develops an alternative to quantile-based checking and adjustment methods which is inherently multivariate. The new approach is based on use of the tower property of conditional expectation and the law of total variance for relating prior and posterior expectations and covariances. For adjustment, approximate inferences are modified so that the correct prior to posterior relationships hold. We illustrate the method in three examples. The first uses an auxiliary model in a likelihood-free inference problem. The second considers corrections for variational Bayes approximations in a deep neural network generalized linear mixed model. Our final application considers a deep neural network surrogate for approximating Gaussian process regression predictive inference. △ Less

Submitted 20 November, 2019; originally announced November 2019.

Marginally-calibrated deep distributional regression

Authors: Nadja Klein, David J. Nott, Michael Stanley Smith

Abstract: Deep neural network (DNN) regression models are widely used in applications requiring state-of-the-art predictive accuracy. However, until recently there has been little work on accurate uncertainty quantification for predictions from such models. We add to this literature by outlining an approach to constructing predictive distributions that are `marginally calibrated'. This is where the long run… ▽ More Deep neural network (DNN) regression models are widely used in applications requiring state-of-the-art predictive accuracy. However, until recently there has been little work on accurate uncertainty quantification for predictions from such models. We add to this literature by outlining an approach to constructing predictive distributions that are `marginally calibrated'. This is where the long run average of the predictive distributions of the response variable matches the observed empirical margin. Our approach considers a DNN regression with a conditionally Gaussian prior for the final layer weights, from which an implicit copula process on the feature space is extracted. This copula process is combined with a non-parametrically estimated marginal distribution for the response. The end result is a scalable distributional DNN regression method with marginally calibrated predictions, and our work complements existing methods for probability calibration. The approach is first illustrated using two applications of dense layer feed-forward neural networks. However, our main motivating applications are in likelihood-free inference, where distributional deep regression is used to estimate marginal posterior distributions. In two complex ecological time series examples we employ the implicit copulas of convolutional networks, and show that marginal calibration results in improved uncertainty quantification. Our approach also avoids the need for manual specification of summary statistics, a requirement that is burdensome for users and typical of competing likelihood-free inference methods. △ Less

Submitted 3 September, 2020; v1 submitted 26 August, 2019; originally announced August 2019.

Journal ref: Journal of Computational and Graphical Statistics (2020)

Likelihood-free approximate Gibbs sampling

Authors: G. S. Rodrigues, D. J. Nott, S. A. Sisson

Abstract: Likelihood-free methods such as approximate Bayesian computation (ABC) have extended the reach of statistical inference to problems with computationally intractable likelihoods. Such approaches perform well for small-to-moderate dimensional problems, but suffer a curse of dimensionality in the number of model parameters. We introduce a likelihood-free approximate Gibbs sampler that naturally circu… ▽ More Likelihood-free methods such as approximate Bayesian computation (ABC) have extended the reach of statistical inference to problems with computationally intractable likelihoods. Such approaches perform well for small-to-moderate dimensional problems, but suffer a curse of dimensionality in the number of model parameters. We introduce a likelihood-free approximate Gibbs sampler that naturally circumvents the dimensionality issue by focusing on lower-dimensional conditional distributions. These distributions are estimated by flexible regression models either before the sampler is run, or adaptively during sampler implementation. As a result, and in comparison to Metropolis-Hastings based approaches, we are able to fit substantially more challenging statistical models than would otherwise be possible. We demonstrate the sampler's performance via two simulated examples, and a real analysis of Airbnb rental prices using a intractable high-dimensional multivariate non-linear state space model containing 13,140 parameters, which presents a real challenge to standard ABC techniques. △ Less

Submitted 10 June, 2019; originally announced June 2019.

Conditionally structured variational Gaussian approximation with importance weights

Authors: Linda S. L. Tan, Aishwarya Bhaskaran, David J. Nott

Abstract: We develop flexible methods of deriving variational inference for models with complex latent variable structure. By splitting the variables in these models into "global" parameters and "local" latent variables, we define a class of variational approximations that exploit this partitioning and go beyond Gaussian variational approximation. This approximation is motivated by the fact that in many hie… ▽ More We develop flexible methods of deriving variational inference for models with complex latent variable structure. By splitting the variables in these models into "global" parameters and "local" latent variables, we define a class of variational approximations that exploit this partitioning and go beyond Gaussian variational approximation. This approximation is motivated by the fact that in many hierarchical models, there are global variance parameters which determine the scale of local latent variables in their posterior conditional on the global parameters. We also consider parsimonious parametrizations by using conditional independence structure, and improved estimation of the log marginal likelihood and variational density using importance weights. These methods are shown to improve significantly on Gaussian variational approximation methods for a similar computational cost. Application of the methodology is illustrated using generalized linear mixed models and state space models. △ Less

Submitted 21 April, 2019; originally announced April 2019.

Comments: 18 pages, 7 figures

High-dimensional copula variational approximation through transformation

Authors: Michael Stanley Smith, Ruben Loaiza-Maya, David J. Nott

Abstract: Variational methods are attractive for computing Bayesian inference for highly parametrized models and large datasets where exact inference is impractical. They approximate a target distribution - either the posterior or an augmented posterior - using a simpler distribution that is selected to balance accuracy with computational feasibility. Here we approximate an element-wise parametric transform… ▽ More Variational methods are attractive for computing Bayesian inference for highly parametrized models and large datasets where exact inference is impractical. They approximate a target distribution - either the posterior or an augmented posterior - using a simpler distribution that is selected to balance accuracy with computational feasibility. Here we approximate an element-wise parametric transformation of the target distribution as multivariate Gaussian or skew-normal. Approximations of this kind are implicit copula models for the original parameters, with a Gaussian or skew-normal copula function and flexible parametric margins. A key observation is that their adoption can improve the accuracy of variational inference in high dimensions at limited or no additional computational cost. We consider the Yeo-Johnson and G&H transformations, along with sparse factor structures for the scale matrix of the Gaussian or skew-normal. We also show how to implement efficient reparametrization gradient methods for these copula-based approximations. The efficacy of the approach is illustrated by computing posterior inference for three different models using six real datasets. In each case, we show that our proposed copula model distributions are more accurate variational approximations than Gaussian or skew-normal distributions, but at only a minor or no increase in computational cost. △ Less

Submitted 20 November, 2019; v1 submitted 16 April, 2019; originally announced April 2019.

Using prior expansions for prior-data conflict checking

Authors: David J. Nott, Max Seah, Luai Al-Labadi, Michael Evans, Hui Khoon Ng, Berthold-Georg Englert

Abstract: Any Bayesian analysis involves combining information represented through different model components, and when different sources of information are in conflict it is important to detect this. Here we consider checking for prior-data conflict in Bayesian models by expanding the prior used for the analysis into a larger family of priors, and considering a marginal likelihood score statistic for the e… ▽ More Any Bayesian analysis involves combining information represented through different model components, and when different sources of information are in conflict it is important to detect this. Here we consider checking for prior-data conflict in Bayesian models by expanding the prior used for the analysis into a larger family of priors, and considering a marginal likelihood score statistic for the expansion parameter. Consideration of different expansions can be informative about the nature of any conflict, and extensions to hierarchically specified priors and connections with other approaches to prior-data conflict checking are discussed. Implementation in complex situations is illustrated with two applications. The first concerns testing for the appropriateness of a LASSO penalty in shrinkage estimation of coefficients in linear regression. Our method is compared with a recent suggestion in the literature designed to be powerful against alternatives in the exponential power family, and we use this family as the prior expansion for constructing our check. A second application concerns a problem in quantum state estimation, where a multinomial model is considered with physical constraints on the model parameters. In this example, the usefulness of different prior expansions is demonstrated for obtaining checks which are sensitive to different aspects of the prior. △ Less

Submitted 12 March, 2020; v1 submitted 27 February, 2019; originally announced February 2019.

Comments: Accepted version, to appear in Bayesian Analysis

Bayesian inference using synthetic likelihood: asymptotics and adjustments

Authors: David T. Frazier, David J. Nott, Christopher Drovandi, Robert Kohn

Abstract: Implementing Bayesian inference is often computationally challenging in applications involving complex models, and sometimes calculating the likelihood itself is difficult. Synthetic likelihood is one approach for carrying out inference when the likelihood is intractable, but it is straightforward to simulate from the model. The method constructs an approximate likelihood by taking a vector summar… ▽ More Implementing Bayesian inference is often computationally challenging in applications involving complex models, and sometimes calculating the likelihood itself is difficult. Synthetic likelihood is one approach for carrying out inference when the likelihood is intractable, but it is straightforward to simulate from the model. The method constructs an approximate likelihood by taking a vector summary statistic as being multivariate normal, with the unknown mean and covariance matrix estimated by simulation for any given parameter value. Our article makes three contributions. The first shows that if the summary statistic satisfies a central limit theorem, then the synthetic likelihood posterior is asymptotically normal and yields credible sets with the correct level of frequentist coverage. This result is similar to that obtained by approximate Bayesian computation. The second contribution compares the computational efficiency of Bayesian synthetic likelihood and approximate Bayesian computation using the acceptance probability for rejection and importance sampling algorithms with a "good" proposal distribution. We show that Bayesian synthetic likelihood is computationally more efficient than approximate Bayesian computation, and behaves similarly to regression-adjusted approximate Bayesian computation. Based on the asymptotic results, the third contribution proposes using adjusted inference methods when a possibly misspecified form is assumed for the covariance matrix of the synthetic likelihood, such as diagonal or a factor model, to speed up the computation. The methodology is illustrated with some simulated and real examples. △ Less

Submitted 12 March, 2021; v1 submitted 13 February, 2019; originally announced February 2019.

An easy-to-use empirical likelihood ABC method

Authors: Sanjay Chaudhuri, Subhro Ghosh, David J. Nott, Kim Cuc Pham

Abstract: Many scientifically well-motivated statistical models in natural, engineering and environmental sciences are specified through a generative process, but in some cases it may not be possible to write down a likelihood for these models analytically. Approximate Bayesian computation (ABC) methods, which allow Bayesian inference in these situations, are typically computationally intensive. Recently, c… ▽ More Many scientifically well-motivated statistical models in natural, engineering and environmental sciences are specified through a generative process, but in some cases it may not be possible to write down a likelihood for these models analytically. Approximate Bayesian computation (ABC) methods, which allow Bayesian inference in these situations, are typically computationally intensive. Recently, computationally attractive empirical likelihood based ABC methods have been suggested in the literature. These methods heavily rely on the availability of a set of suitable analytically tractable estimating equations. We propose an easy-to-use empirical likelihood ABC method, where the only inputs required are a choice of summary statistic, it's observed value, and the ability to simulate summary statistics for any parameter value under the model. It is shown that the posterior obtained using the proposed method is consistent, and its performance is explored using various examples. △ Less

Submitted 8 October, 2018; v1 submitted 3 October, 2018; originally announced October 2018.

Robust Bayesian Synthetic Likelihood via a Semi-Parametric Approach

Authors: Ziwen An, David J. Nott, Christopher Drovandi

Abstract: Bayesian synthetic likelihood (BSL) is now a well established method for performing approximate Bayesian parameter estimation for simulation-based models that do not possess a tractable likelihood function. BSL approximates an intractable likelihood function of a carefully chosen summary statistic at a parameter value with a multivariate normal distribution. The mean and covariance matrix of this… ▽ More Bayesian synthetic likelihood (BSL) is now a well established method for performing approximate Bayesian parameter estimation for simulation-based models that do not possess a tractable likelihood function. BSL approximates an intractable likelihood function of a carefully chosen summary statistic at a parameter value with a multivariate normal distribution. The mean and covariance matrix of this normal distribution are estimated from independent simulations of the model. Due to the parametric assumption implicit in BSL, it can be preferred to its non-parametric competitor, approximate Bayesian computation, in certain applications where a high-dimensional summary statistic is of interest. However, despite several successful applications of BSL, its widespread use in scientific fields may be hindered by the strong normality assumption. In this paper, we develop a semi-parametric approach to relax this assumption to an extent and maintain the computational advantages of BSL without any additional tuning. We test our new method, semiBSL, on several challenging examples involving simulated and real data and demonstrate that semiBSL can be significantly more robust than BSL and another approach in the literature. △ Less

Submitted 3 October, 2019; v1 submitted 15 September, 2018; originally announced September 2018.

Comments: 37 pages Latex; the paper has been re-organised, moved section 4 and 5 to appendices, moved less important example figures to appendices, added "sensitivity to n" section to appendices, added a shrinkage example to appendices, typos and references corrected

High-dimensional ABC

Authors: D. J. Nott, V. M. -H. Ong, Y. Fan, S. A. Sisson

Abstract: This Chapter, "High-dimensional ABC", is to appear in the forthcoming Handbook of Approximate Bayesian Computation (2018). It details the main ideas and concepts behind extending ABC methods to higher dimensions, with supporting examples and illustrations. This Chapter, "High-dimensional ABC", is to appear in the forthcoming Handbook of Approximate Bayesian Computation (2018). It details the main ideas and concepts behind extending ABC methods to higher dimensions, with supporting examples and illustrations. △ Less

Submitted 27 February, 2018; originally announced February 2018.

Gaussian variational approximation for high-dimensional state space models

Authors: Matias Quiroz, David J. Nott, Robert Kohn

Abstract: Our article considers a Gaussian variational approximation of the posterior density in a high-dimensional state space model. The variational parameters to be optimized are the mean vector and the covariance matrix of the approximation. The number of parameters in the covariance matrix grows as the square of the number of model parameters, so it is necessary to find simple yet effective parameteriz… ▽ More Our article considers a Gaussian variational approximation of the posterior density in a high-dimensional state space model. The variational parameters to be optimized are the mean vector and the covariance matrix of the approximation. The number of parameters in the covariance matrix grows as the square of the number of model parameters, so it is necessary to find simple yet effective parameterizations of the covariance structure when the number of model parameters is large. We approximate the joint posterior distribution over the high-dimensional state vectors by a dynamic factor model, having Markovian time dependence and a factor covariance structure for the states. This gives a reduced description of the dependence structure for the states, as well as a temporal conditional independence structure similar to that in the true posterior. The usefulness of the approach is illustrated for prediction in two high-dimensional applications that are challenging for Markov chain Monte Carlo sampling. The first is a spatio-temporal model for the spread of the Eurasian Collared-Dove across North America; the second is a Wishart-based multivariate stochastic volatility model for financial returns. △ Less

Submitted 18 February, 2020; v1 submitted 24 January, 2018; originally announced January 2018.

Comments: Significantly revised, especially the multivariate stochastic volatility model example

A Semi-Automatic Method for History Matching using Sequential Monte Carlo

Authors: Christopher C Drovandi, David J Nott, Daniel E Pagendam

Abstract: The aim of the history matching method is to locate non-implausible regions of the parameter space of complex deterministic or stochastic models by matching model outputs with data. It does this via a series of waves where at each wave an emulator is fitted to a small number of training samples. An implausibility measure is defined which takes into account the closeness of simulated and observed o… ▽ More The aim of the history matching method is to locate non-implausible regions of the parameter space of complex deterministic or stochastic models by matching model outputs with data. It does this via a series of waves where at each wave an emulator is fitted to a small number of training samples. An implausibility measure is defined which takes into account the closeness of simulated and observed outputs as well as emulator uncertainty. As the waves progress, the emulator becomes more accurate so that training samples are more concentrated on promising regions of the space and poorer parts of the space are rejected with more confidence. Whilst history matching has proved to be useful, existing implementations are not fully automated and some ad-hoc choices are made during the process, which involves user intervention and is time consuming. This occurs especially when the non-implausible region becomes small and it is difficult to sample this space uniformly to generate new training points. In this article we develop a sequential Monte Carlo (SMC) algorithm for implementing history matching that is semi-automated. Our novel SMC approach reveals that the history matching method yields a non-implausible region that can be multi-modal, highly irregular and very difficult to sample uniformly. Our SMC approach offers a much more reliable sampling of the non-implausible space, which requires additional computation compared to other approaches used in the literature. △ Less

Submitted 8 May, 2021; v1 submitted 9 October, 2017; originally announced October 2017.

Comments: Accepted by SIAM/ASA Journal on Uncertainty Quantification

Gaussian variational approximation with a factor covariance structure

Authors: Victor M. -H. Ong, David J. Nott, Michael S. Smith

Abstract: Variational approximation methods have proven to be useful for scaling Bayesian computations to large data sets and highly parametrized models. Applying variational methods involves solving an optimization problem, and recent research in this area has focused on stochastic gradient ascent methods as a general approach to implementation. Here variational approximation is considered for a posterior… ▽ More Variational approximation methods have proven to be useful for scaling Bayesian computations to large data sets and highly parametrized models. Applying variational methods involves solving an optimization problem, and recent research in this area has focused on stochastic gradient ascent methods as a general approach to implementation. Here variational approximation is considered for a posterior distribution in high dimensions using a Gaussian approximating family. Gaussian variational approximation with an unrestricted covariance matrix can be computationally burdensome in many problems because the number of elements in the covariance matrix increases quadratically with the dimension of the model parameter. To circumvent this problem, low-dimensional factor covariance structures are considered. General stochastic gradient approaches to efficiently perform the optimization are described, with gradient estimates obtained using the so-called "reparametrization trick". The end result is a flexible and efficient approach to high-dimensional Gaussian variational approximation, which we illustrate using eight real datasets. △ Less

Submitted 11 January, 2017; originally announced January 2017.

Random samples of quantum states: Online resources

Authors: Jiangwei Shang, Yi-Lin Seah, Boyu Wang, Hui Khoon Ng, David John Nott, Berthold-Georg Englert

Abstract: This is the documentation for generating random samples from the quantum state space in accordance with a specified distribution, associated with this webpage: http://tinyurl.com/QSampling . Ready-made samples (each with at least a million points) from various distributions are available for download, or one can generate one's own samples from a chosen distribution using the provided source codes.… ▽ More This is the documentation for generating random samples from the quantum state space in accordance with a specified distribution, associated with this webpage: http://tinyurl.com/QSampling . Ready-made samples (each with at least a million points) from various distributions are available for download, or one can generate one's own samples from a chosen distribution using the provided source codes. The sampling relies on the Hamiltonian Monte Carlo algorithm as described in New J. Phys. 17, 043018 (2015). The random samples are reposited in the hope that they would be useful for a variety of tasks in quantum information and quantum computation. Constructing credible regions for tomographic data, optimizing a function over the quantum state space with a complicated landscape, testing the typicality of entanglement among states from a multipartite quantum system, or computing the average of some quantity of interest over a subset of quantum states are but some exemplary applications among many. △ Less

Submitted 19 December, 2016; v1 submitted 15 December, 2016; originally announced December 2016.

Comments: v2: link in the abstract fixed. Documentation associated with the QSampling project on the webpage http://www.quantumlah.org/publications/software/QSampling/, where all the ready-made samples as well as source codes can be downloaded. Please send your comments or suggestions to: [email protected]. 10 pages, 2 tables, and 6 references

Checking for prior-data conflict using prior to posterior divergences

Authors: David J. Nott, Xueou Wang, Michael Evans, Berthold-Georg Englert

Abstract: When using complex Bayesian models to combine information, the checking for consistency of the information being combined is good statistical practice. Here a new method is developed for detecting prior-data conflicts in Bayesian models based on comparing the observed value of a prior to posterior divergence to its distribution under the prior predictive distribution for the data. The divergence m… ▽ More When using complex Bayesian models to combine information, the checking for consistency of the information being combined is good statistical practice. Here a new method is developed for detecting prior-data conflicts in Bayesian models based on comparing the observed value of a prior to posterior divergence to its distribution under the prior predictive distribution for the data. The divergence measure used in our model check is a measure of how much beliefs have changed from prior to posterior, and can be thought of as a measure of the overall size of a relative belief function. It is shown that the proposed method is intuitive, has desirable properties, can be extended to hierarchical settings, and is related asymptotically to Jeffreys' and reference prior distributions. In the case where calculations are difficult, the use of variational approximations as a way of relieving the computational burden is suggested. The methods are compared in a number of examples with an alternative but closely related approach in the literature based on the prior predictive distribution of a minimal sufficient statistic. △ Less

Submitted 28 November, 2016; v1 submitted 31 October, 2016; originally announced November 2016.

Comments: Corrected typos in Example 3.2 and Section 4

Variational Bayes with Synthetic Likelihood

Authors: Victor M-H. Ong, David J. Nott, Minh-Ngoc Tran, Scott A. Sisson, Christopher C. Drovandi

Abstract: Synthetic likelihood is an attractive approach to likelihood-free inference when an approximately Gaussian summary statistic for the data, informative for inference about the parameters, is available. The synthetic likelihood method derives an approximate likelihood function from a plug-in normal density estimate for the summary statistic, with plug-in mean and covariance matrix obtained by Monte… ▽ More Synthetic likelihood is an attractive approach to likelihood-free inference when an approximately Gaussian summary statistic for the data, informative for inference about the parameters, is available. The synthetic likelihood method derives an approximate likelihood function from a plug-in normal density estimate for the summary statistic, with plug-in mean and covariance matrix obtained by Monte Carlo simulation from the model. In this article, we develop alternatives to Markov chain Monte Carlo implementations of Bayesian synthetic likelihoods with reduced computational overheads. Our approach uses stochastic gradient variational inference methods for posterior approximation in the synthetic likelihood context, employing unbiased estimates of the log likelihood. We compare the new method with a related likelihood free variational inference technique in the literature, while at the same time improving the implementation of that approach in a number of ways. These new algorithms are feasible to implement in situations which are challenging for conventional approximate Bayesian computation (ABC) methods, in terms of the dimensionality of the parameter and summary statistic. △ Less

Submitted 10 August, 2016; originally announced August 2016.

Comments: 30 pages; 7 figures

Journal ref: Stat.Comput. 28 (2018) 971-988

Using history matching for prior choice

Authors: Xueou Wang, David J. Nott, C. C. Drovandi, Kerrie Mengersen, Michael Evans

Abstract: It can be important in Bayesian analyses of complex models to construct informative prior distributions which reflect knowledge external to the data at hand. Nevertheless, how much prior information an analyst can elicit from an expert will be limited due to constraints of time, cost and other factors. This paper develops effective numerical methods for exploring reasonable choices of a prior dist… ▽ More It can be important in Bayesian analyses of complex models to construct informative prior distributions which reflect knowledge external to the data at hand. Nevertheless, how much prior information an analyst can elicit from an expert will be limited due to constraints of time, cost and other factors. This paper develops effective numerical methods for exploring reasonable choices of a prior distribution from a parametric class, when prior information is specified in the form of some limited constraints on prior predictive distributions, and where these prior predictive distributions are analytically intractable. The methods developed may be thought of as a novel application of the ideas of history matching, a technique developed in the literature on assessment of computer models. We illustrate the approach in the context of logistic regression and sparse signal shrinkage prior distributions for high-dimensional linear models. △ Less

Submitted 9 November, 2017; v1 submitted 28 May, 2016; originally announced May 2016.

Gaussian variational approximation with sparse precision matrices

Authors: Linda S. L. Tan, David J. Nott

Abstract: We consider the problem of learning a Gaussian variational approximation to the posterior distribution for a high-dimensional parameter, where we impose sparsity in the precision matrix to reflect appropriate conditional independence structure in the model. Incorporating sparsity in the precision matrix allows the Gaussian variational distribution to be both flexible and parsimonious, and the spar… ▽ More We consider the problem of learning a Gaussian variational approximation to the posterior distribution for a high-dimensional parameter, where we impose sparsity in the precision matrix to reflect appropriate conditional independence structure in the model. Incorporating sparsity in the precision matrix allows the Gaussian variational distribution to be both flexible and parsimonious, and the sparsity is achieved through parameterization in terms of the Cholesky factor. Efficient stochastic gradient methods which make appropriate use of gradient information for the target distribution are developed for the optimization. We consider alternative estimators of the stochastic gradients which have lower variation and are more stable. Our approach is illustrated using generalized linear mixed models and state space models for time series. △ Less

Submitted 12 April, 2017; v1 submitted 18 May, 2016; originally announced May 2016.

Comments: 18 pages, 9 figures

Journal ref: Statistics and Computing 28 (2018) 259-275

Flexible online multivariate regression with variational Bayes and the matrix-variate Dirichlet process

Authors: Meng Hwee Victor Ong, David J. Nott, Ajay Jasra

Abstract: Flexible regression methods where interest centres on the way that the whole distribution of a response vector changes with covariates are very useful in some applications. A recently developed technique in this regard uses the matrix-variate Dirichlet process as a prior for a mixing distribution on a coefficient in a multivariate linear regression model. The method is attractive, particularly in… ▽ More Flexible regression methods where interest centres on the way that the whole distribution of a response vector changes with covariates are very useful in some applications. A recently developed technique in this regard uses the matrix-variate Dirichlet process as a prior for a mixing distribution on a coefficient in a multivariate linear regression model. The method is attractive, particularly in the multivariate setting, for the convenient way that it allows for borrowing strength across different component regressions and for its computational simplicity and tractability. The purpose of the present article is to develop fast online variational Bayes approaches to fitting this model and to investigate how they perform compared to MCMC and batch variational methods in a number of scenarios. △ Less

Submitted 29 February, 2016; originally announced February 2016.

Extending approximate Bayesian computation methods to high dimensions via a Gaussian copula model

Authors: Jingjing Li, David J. Nott, Yanan Fan, Scott A. Sisson

Abstract: Approximate Bayesian computation (ABC) refers to a family of inference methods used in the Bayesian analysis of complex models where evaluation of the likelihood is difficult. Conventional ABC methods often suffer from the curse of dimensionality, and a marginal adjustment strategy was recently introduced in the literature to improve the performance of ABC algorithms in high-dimensional problems.… ▽ More Approximate Bayesian computation (ABC) refers to a family of inference methods used in the Bayesian analysis of complex models where evaluation of the likelihood is difficult. Conventional ABC methods often suffer from the curse of dimensionality, and a marginal adjustment strategy was recently introduced in the literature to improve the performance of ABC algorithms in high-dimensional problems. The marginal adjustment approach is extended using a Gaussian copula approximation. The method first estimates the bivariate posterior for each pair of parameters separately using a 2-dimensional Gaussian copula, and then combines these estimates together to estimate the joint posterior. The approximation works well in large sample settings when the posterior is approximately normal, but also works well in many cases which are far from that situation due to the nonparametric estimation of the marginal posterior distributions. If each bivariate posterior distribution can be well estimated with a low-dimensional ABC analysis then this Gaussian copula method can extend ABC methods to problems of high dimension. The method also results in an analytic expression for the approximate posterior which is useful for many purposes such as approximation of the likelihood itself. This method is illustrated with several examples. △ Less

Submitted 7 July, 2016; v1 submitted 15 April, 2015; originally announced April 2015.

Comments: TO appear in CSDA

Variational Bayes with Intractable Likelihood

Authors: Minh-Ngoc Tran, David J. Nott, Robert Kohn

Abstract: Variational Bayes (VB) is rapidly becoming a popular tool for Bayesian inference in statistical modeling. However, the existing VB algorithms are restricted to cases where the likelihood is tractable, which precludes the use of VB in many interesting situations such as in state space models and in approximate Bayesian computation (ABC), where application of VB methods was previously impossible. Th… ▽ More Variational Bayes (VB) is rapidly becoming a popular tool for Bayesian inference in statistical modeling. However, the existing VB algorithms are restricted to cases where the likelihood is tractable, which precludes the use of VB in many interesting situations such as in state space models and in approximate Bayesian computation (ABC), where application of VB methods was previously impossible. This paper extends the scope of application of VB to cases where the likelihood is intractable, but can be estimated unbiasedly. The proposed VB method therefore makes it possible to carry out Bayesian inference in many statistical applications, including state space models and ABC. The method is generic in the sense that it can be applied to almost all statistical models without requiring too much model-based derivation, which is a drawback of many existing VB algorithms. We also show how the proposed method can be used to obtain highly accurate VB approximations of marginal posterior distributions. △ Less

Submitted 4 August, 2016; v1 submitted 30 March, 2015; originally announced March 2015.

Comments: 40 pages, 6 figures

A Simple Approach to Constructing Quasi-Sudoku-based Sliced Space-Filling Designs

Authors: Diane Donovan, Benjamin Haaland, David J. Nott

Abstract: Sliced Sudoku-based space-filling designs and, more generally, quasi-sliced orthogonal array-based space-filling designs are useful experimental designs in several contexts, including computer experiments with categorical in addition to quantitative inputs and cross-validation. Here, we provide a straightforward construction of doubly orthogonal quasi-Sudoku Latin squares which can be used to gene… ▽ More Sliced Sudoku-based space-filling designs and, more generally, quasi-sliced orthogonal array-based space-filling designs are useful experimental designs in several contexts, including computer experiments with categorical in addition to quantitative inputs and cross-validation. Here, we provide a straightforward construction of doubly orthogonal quasi-Sudoku Latin squares which can be used to generate sliced space-filling designs which achieve uniformity in one and two-dimensional projections for both the full design and each slice. A construction of quasi-sliced orthogonal arrays based on these constructed doubly orthogonal quasi-Sudoku Latin squares is also provided and can, in turn, be used to generate sliced space-filling designs which achieve uniformity in one and two-dimensional projections for the full design and and uniformity in two-dimensional projections for each slice. These constructions are very practical to implement and yield a spectrum of design sizes and numbers of factors not currently broadly available. △ Less

Submitted 19 February, 2015; originally announced February 2015.

Comments: 15 pages, 9 figures

MSC Class: 05B15

Functional regression approximate Bayesian computation for Gaussian process density estimation

Authors: G. S. Rodrigues, David J. Nott, S. A. Sisson

Abstract: We propose a novel Bayesian nonparametric method for hierarchical modelling on a set of related density functions, where grouped data in the form of samples from each density function are available. Borrowing strength across the groups is a major challenge in this context. To address this problem, we introduce a hierarchically structured prior, defined over a set of univariate density functions, u… ▽ More We propose a novel Bayesian nonparametric method for hierarchical modelling on a set of related density functions, where grouped data in the form of samples from each density function are available. Borrowing strength across the groups is a major challenge in this context. To address this problem, we introduce a hierarchically structured prior, defined over a set of univariate density functions, using convenient transformations of Gaussian processes. Inference is performed through approximate Bayesian computation (ABC), via a novel functional regression adjustment. The performance of the proposed method is illustrated via a simulation study and an analysis of rural high school exam performance in Brazil. △ Less

Submitted 30 October, 2014; originally announced October 2014.

MSC Class: 62G07

Monte Carlo sampling from the quantum state space. II

Authors: Yi-Lin Seah, Jiangwei Shang, Hui Khoon Ng, David John Nott, Berthold-Georg Englert

Abstract: High-quality random samples of quantum states are needed for a variety of tasks in quantum information and quantum computation. Searching the high-dimensional quantum state space for a global maximum of an objective function with many local maxima or evaluating an integral over a region in the quantum state space are but two exemplary applications of many. These tasks can only be performed reliabl… ▽ More High-quality random samples of quantum states are needed for a variety of tasks in quantum information and quantum computation. Searching the high-dimensional quantum state space for a global maximum of an objective function with many local maxima or evaluating an integral over a region in the quantum state space are but two exemplary applications of many. These tasks can only be performed reliably and efficiently with Monte Carlo methods, which involve good samplings of the parameter space in accordance with the relevant target distribution. We show how the Markov-chain Monte Carlo method known as Hamiltonian Monte Carlo, or hybrid Monte Carlo, can be adapted to this context. It is applicable when an efficient parameterization of the state space is available. The resulting random walk is entirely inside the physical parameter space, and the Hamiltonian dynamics enable us to take big steps, thereby avoiding strong correlations between successive sample points while enjoying a high acceptance rate. We use examples of single and double qubit measurements for illustration. △ Less

Submitted 27 April, 2015; v1 submitted 29 July, 2014; originally announced July 2014.

Comments: 11 pages, 4 figures, 12 references

Journal ref: Yi-Lin Seah et al 2015 New J. Phys. 17 043018

Monte Carlo sampling from the quantum state space. I

Authors: Jiangwei Shang, Yi-Lin Seah, Hui Khoon Ng, David John Nott, Berthold-Georg Englert

Abstract: High-quality random samples of quantum states are needed for a variety of tasks in quantum information and quantum computation. Searching the high-dimensional quantum state space for a global maximum of an objective function with many local maxima or evaluating an integral over a region in the quantum state space are but two exemplary applications of many. These tasks can only be performed reliabl… ▽ More High-quality random samples of quantum states are needed for a variety of tasks in quantum information and quantum computation. Searching the high-dimensional quantum state space for a global maximum of an objective function with many local maxima or evaluating an integral over a region in the quantum state space are but two exemplary applications of many. These tasks can only be performed reliably and efficiently with Monte Carlo methods, which involve good samplings of the parameter space in accordance with the relevant target distribution. We show how the standard strategies of rejection sampling, importance sampling, and Markov-chain sampling can be adapted to this context, where the samples must obey the constraints imposed by the positivity of the statistical operator. For a comparison of these sampling methods, we generate sample points in the probability space for two-qubit states probed with a tomographically incomplete measurement, and then use the sample for the calculation of the size and credibility of the recently-introduced optimal error regions [see New J. Phys. 15 (2013) 123026]. Another illustration is the computation of the fractional volume of separable two-qubit states. △ Less

Submitted 27 April, 2015; v1 submitted 29 July, 2014; originally announced July 2014.

Comments: 13 pages, 5 figures, 1 table, 26 references

Journal ref: Jiangwei Shang et al 2015 New J. Phys. 17 043017

Efficient variational inference for generalized linear mixed models with large datasets

Authors: David J Nott, Minh-Ngoc Tran, Anthony Y. C. Kuk, Robert Kohn

Abstract: The article develops a hybrid Variational Bayes algorithm that combines the mean-field and fixed-form Variational Bayes methods. The new estimation algorithm can be used to approximate any posterior without relying on conjugate priors. We propose a divide and recombine strategy for the analysis of large datasets, which partitions a large dataset into smaller pieces and then combines the variationa… ▽ More The article develops a hybrid Variational Bayes algorithm that combines the mean-field and fixed-form Variational Bayes methods. The new estimation algorithm can be used to approximate any posterior without relying on conjugate priors. We propose a divide and recombine strategy for the analysis of large datasets, which partitions a large dataset into smaller pieces and then combines the variational distributions that have been learnt in parallel on each separate piece using the hybrid Variational Bayes algorithm. The proposed method is applied to fitting generalized linear mixed models. The computational efficiency of the parallel and hybrid Variational Bayes algorithm is demonstrated on several simulated and real datasets. △ Less

Submitted 8 August, 2013; v1 submitted 30 July, 2013; originally announced July 2013.

Comments: 23 pages, 4 figures