-
FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearch
Authors:
Virginia Aglietti,
Ira Ktena,
Jessica Schrouff,
Eleni Sgouritsa,
Francisco J. R. Ruiz,
Alan Malek,
Alexis Bellot,
Silvia Chiappa
Abstract:
The sample efficiency of Bayesian optimization algorithms depends on carefully crafted acquisition functions (AFs) guiding the sequential collection of function evaluations. The best-performing AF can vary significantly across optimization problems, often requiring ad-hoc and problem-specific choices. This work tackles the challenge of designing novel AFs that perform well across a variety of expe…
▽ More
The sample efficiency of Bayesian optimization algorithms depends on carefully crafted acquisition functions (AFs) guiding the sequential collection of function evaluations. The best-performing AF can vary significantly across optimization problems, often requiring ad-hoc and problem-specific choices. This work tackles the challenge of designing novel AFs that perform well across a variety of experimental settings. Based on FunSearch, a recent work using Large Language Models (LLMs) for discovery in mathematical sciences, we propose FunBO, an LLM-based method that can be used to learn new AFs written in computer code by leveraging access to a limited number of evaluations for a set of objective functions. We provide the analytic expression of all discovered AFs and evaluate them on various global optimization benchmarks and hyperparameter optimization tasks. We show how FunBO identifies AFs that generalize well in and out of the training distribution of functions, thus outperforming established general-purpose AFs and achieving competitive performance against AFs that are customized to specific function types and are learned via transfer-learning algorithms.
△ Less
Submitted 1 July, 2024; v1 submitted 7 June, 2024;
originally announced June 2024.
-
Consistency of Causal Inference under the Additive Noise Model
Authors:
Samory Kpotufe,
Eleni Sgouritsa,
Dominik Janzing,
Bernhard Schölkopf
Abstract:
We analyze a family of methods for statistical causal inference from sample under the so-called Additive Noise Model. While most work on the subject has concentrated on establishing the soundness of the Additive Noise Model, the statistical consistency of the resulting inference methods has received little attention. We derive general conditions under which the given family of inference methods co…
▽ More
We analyze a family of methods for statistical causal inference from sample under the so-called Additive Noise Model. While most work on the subject has concentrated on establishing the soundness of the Additive Noise Model, the statistical consistency of the resulting inference methods has received little attention. We derive general conditions under which the given family of inference methods consistently infers the causal direction in a nonparametric setting.
△ Less
Submitted 4 February, 2014; v1 submitted 19 December, 2013;
originally announced December 2013.
-
Identifying Finite Mixtures of Nonparametric Product Distributions and Causal Inference of Confounders
Authors:
Eleni Sgouritsa,
Dominik Janzing,
Jonas Peters,
Bernhard Schoelkopf
Abstract:
We propose a kernel method to identify finite mixtures of nonparametric product distributions. It is based on a Hilbert space embedding of the joint distribution. The rank of the constructed tensor is equal to the number of mixture components. We present an algorithm to recover the components by partitioning the data points into clusters such that the variables are jointly conditionally independen…
▽ More
We propose a kernel method to identify finite mixtures of nonparametric product distributions. It is based on a Hilbert space embedding of the joint distribution. The rank of the constructed tensor is equal to the number of mixture components. We present an algorithm to recover the components by partitioning the data points into clusters such that the variables are jointly conditionally independent given the cluster. This method can be used to identify finite confounders.
△ Less
Submitted 26 September, 2013;
originally announced September 2013.
-
On Causal and Anticausal Learning
Authors:
Bernhard Schoelkopf,
Dominik Janzing,
Jonas Peters,
Eleni Sgouritsa,
Kun Zhang,
Joris Mooij
Abstract:
We consider the problem of function estimation in the case where an underlying causal model can be inferred. This has implications for popular scenarios such as covariate shift, concept drift, transfer learning and semi-supervised learning. We argue that causal knowledge may facilitate some approaches for a given problem, and rule out others. In particular, we formulate a hypothesis for when semi-…
▽ More
We consider the problem of function estimation in the case where an underlying causal model can be inferred. This has implications for popular scenarios such as covariate shift, concept drift, transfer learning and semi-supervised learning. We argue that causal knowledge may facilitate some approaches for a given problem, and rule out others. In particular, we formulate a hypothesis for when semi-supervised learning can help, and corroborate it with empirical results.
△ Less
Submitted 27 June, 2012;
originally announced June 2012.
-
Detecting low-complexity unobserved causes
Authors:
Dominik Janzing,
Eleni Sgouritsa,
Oliver Stegle,
Jonas Peters,
Bernhard Schoelkopf
Abstract:
We describe a method that infers whether statistical dependences between two observed variables X and Y are due to a "direct" causal link or only due to a connecting causal path that contains an unobserved variable of low complexity, e.g., a binary variable. This problem is motivated by statistical genetics. Given a genetic marker that is correlated with a phenotype of interest, we want to detect…
▽ More
We describe a method that infers whether statistical dependences between two observed variables X and Y are due to a "direct" causal link or only due to a connecting causal path that contains an unobserved variable of low complexity, e.g., a binary variable. This problem is motivated by statistical genetics. Given a genetic marker that is correlated with a phenotype of interest, we want to detect whether this marker is causal or it only correlates with a causal one. Our method is based on the analysis of the location of the conditional distributions P(Y|x) in the simplex of all distributions of Y. We report encouraging results on semi-empirical data.
△ Less
Submitted 14 February, 2012;
originally announced February 2012.
