-
Development and validation of an interpretable machine learning-based calculator for predicting 5-year weight trajectories after bariatric surgery: a multinational retrospective cohort SOPHIA study
Authors:
Patrick Saux,
Pierre Bauvin,
Violeta Raverdy,
Julien Teigny,
Hélène Verkindt,
Tomy Soumphonphakdy,
Maxence Debert,
Anne Jacobs,
Daan Jacobs,
Valerie Monpellier,
Phong Ching Lee,
Chin Hong Lim,
Johanna C Andersson-Assarsson,
Lena Carlsson,
Per-Arne Svensson,
Florence Galtier,
Guelareh Dezfoulian,
Mihaela Moldovanu,
Severine Andrieux,
Julien Couster,
Marie Lepage,
Erminia Lembo,
Ornella Verrastro,
Maud Robert,
Paulina Salminen
, et al. (9 additional authors not shown)
Abstract:
Background Weight loss trajectories after bariatric surgery vary widely between individuals, and predicting weight loss before the operation remains challenging. We aimed to develop a model using machine learning to provide individual preoperative prediction of 5-year weight loss trajectories after surgery. Methods In this multinational retrospective observational study we enrolled adult participa…
▽ More
Background Weight loss trajectories after bariatric surgery vary widely between individuals, and predicting weight loss before the operation remains challenging. We aimed to develop a model using machine learning to provide individual preoperative prediction of 5-year weight loss trajectories after surgery. Methods In this multinational retrospective observational study we enrolled adult participants (aged $\ge$18 years) from ten prospective cohorts (including ABOS [NCT01129297], BAREVAL [NCT02310178], the Swedish Obese Subjects study, and a large cohort from the Dutch Obesity Clinic [Nederlandse Obesitas Kliniek]) and two randomised trials (SleevePass [NCT00793143] and SM-BOSS [NCT00356213]) in Europe, the Americas, and Asia, with a 5 year followup after Roux-en-Y gastric bypass, sleeve gastrectomy, or gastric band. Patients with a previous history of bariatric surgery or large delays between scheduled and actual visits were excluded. The training cohort comprised patients from two centres in France (ABOS and BAREVAL). The primary outcome was BMI at 5 years. A model was developed using least absolute shrinkage and selection operator to select variables and the classification and regression trees algorithm to build interpretable regression trees. The performances of the model were assessed through the median absolute deviation (MAD) and root mean squared error (RMSE) of BMI. Findings10 231 patients from 12 centres in ten countries were included in the analysis, corresponding to 30 602 patient-years. Among participants in all 12 cohorts, 7701 (75$\bullet$3%) were female, 2530 (24$\bullet$7%) were male. Among 434 baseline attributes available in the training cohort, seven variables were selected: height, weight, intervention type, age, diabetes status, diabetes duration, and smoking status. At 5 years, across external testing cohorts the overall mean MAD BMI was 2$\bullet$8 kg/m${}^2$ (95% CI 2$\bullet$6-3$\bullet$0) and mean RMSE BMI was 4$\bullet$7 kg/m${}^2$ (4$\bullet$4-5$\bullet$0), and the mean difference between predicted and observed BMI was-0$\bullet$3 kg/m${}^2$ (SD 4$\bullet$7). This model is incorporated in an easy to use and interpretable web-based prediction tool to help inform clinical decision before surgery. InterpretationWe developed a machine learning-based model, which is internationally validated, for predicting individual 5-year weight loss trajectories after three common bariatric interventions.
△ Less
Submitted 31 August, 2023;
originally announced August 2023.
-
Biased Hypothesis Formation From Projection Pursuit
Authors:
John Patterson,
Chris Avery,
Tyler Grear,
Donald J. Jacobs
Abstract:
The effect of bias on hypothesis formation is characterized for an automated data-driven projection pursuit neural network to extract and select features for binary classification of data streams. This intelligent exploratory process partitions a complete vector state space into disjoint subspaces to create working hypotheses quantified by similarities and differences observed between two groups o…
▽ More
The effect of bias on hypothesis formation is characterized for an automated data-driven projection pursuit neural network to extract and select features for binary classification of data streams. This intelligent exploratory process partitions a complete vector state space into disjoint subspaces to create working hypotheses quantified by similarities and differences observed between two groups of labeled data streams. Data streams are typically time sequenced, and may exhibit complex spatio-temporal patterns. For example, given atomic trajectories from molecular dynamics simulation, the machine's task is to quantify dynamical mechanisms that promote function by comparing protein mutants, some known to function while others are nonfunctional. Utilizing synthetic two-dimensional molecules that mimic the dynamics of functional and nonfunctional proteins, biases are identified and controlled in both the machine learning model and selected training data under different contexts. The refinement of a working hypothesis converges to a statistically robust multivariate perception of the data based on a context-dependent perspective. Including diverse perspectives during data exploration enhances interpretability of the multivariate characterization of similarities and differences.
△ Less
Submitted 3 January, 2022;
originally announced January 2022.
-
On the Similarity between the Laplace and Neural Tangent Kernels
Authors:
Amnon Geifman,
Abhay Yadav,
Yoni Kasten,
Meirav Galun,
David Jacobs,
Ronen Basri
Abstract:
Recent theoretical work has shown that massively overparameterized neural networks are equivalent to kernel regressors that use Neural Tangent Kernels(NTK). Experiments show that these kernel methods perform similarly to real neural networks. Here we show that NTK for fully connected networks is closely related to the standard Laplace kernel. We show theoretically that for normalized data on the h…
▽ More
Recent theoretical work has shown that massively overparameterized neural networks are equivalent to kernel regressors that use Neural Tangent Kernels(NTK). Experiments show that these kernel methods perform similarly to real neural networks. Here we show that NTK for fully connected networks is closely related to the standard Laplace kernel. We show theoretically that for normalized data on the hypersphere both kernels have the same eigenfunctions and their eigenvalues decay polynomially at the same rate, implying that their Reproducing Kernel Hilbert Spaces (RKHS) include the same sets of functions. This means that both kernels give rise to classes of functions with the same smoothness properties. The two kernels differ for data off the hypersphere, but experiments indicate that when data is properly normalized these differences are not significant. Finally, we provide experiments on real data comparing NTK and the Laplace kernel, along with a larger class ofγ-exponential kernels. We show that these perform almost identically. Our results suggest that much insight about neural networks can be obtained from analysis of the well-known Laplace kernel, which has a simple closed-form.
△ Less
Submitted 14 November, 2020; v1 submitted 3 July, 2020;
originally announced July 2020.
-
Frequency Bias in Neural Networks for Input of Non-Uniform Density
Authors:
Ronen Basri,
Meirav Galun,
Amnon Geifman,
David Jacobs,
Yoni Kasten,
Shira Kritchman
Abstract:
Recent works have partly attributed the generalization ability of over-parameterized neural networks to frequency bias -- networks trained with gradient descent on data drawn from a uniform distribution find a low frequency fit before high frequency ones. As realistic training sets are not drawn from a uniform distribution, we here use the Neural Tangent Kernel (NTK) model to explore the effect of…
▽ More
Recent works have partly attributed the generalization ability of over-parameterized neural networks to frequency bias -- networks trained with gradient descent on data drawn from a uniform distribution find a low frequency fit before high frequency ones. As realistic training sets are not drawn from a uniform distribution, we here use the Neural Tangent Kernel (NTK) model to explore the effect of variable density on training dynamics. Our results, which combine analytic and empirical observations, show that when learning a pure harmonic function of frequency $κ$, convergence at a point $\x \in \Sphere^{d-1}$ occurs in time $O(κ^d/p(\x))$ where $p(\x)$ denotes the local density at $\x$. Specifically, for data in $\Sphere^1$ we analytically derive the eigenfunctions of the kernel associated with the NTK for two-layer networks. We further prove convergence results for deep, fully connected networks with respect to the spectral decomposition of the NTK. Our empirical study highlights similarities and differences between deep and shallow networks in this model.
△ Less
Submitted 10 March, 2020;
originally announced March 2020.
-
The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
Authors:
Ronen Basri,
David Jacobs,
Yoni Kasten,
Shira Kritchman
Abstract:
We study the relationship between the frequency of a function and the speed at which a neural network learns it. We build on recent results that show that the dynamics of overparameterized neural networks trained with gradient descent can be well approximated by a linear system. When normalized training data is uniformly distributed on a hypersphere, the eigenfunctions of this linear system are sp…
▽ More
We study the relationship between the frequency of a function and the speed at which a neural network learns it. We build on recent results that show that the dynamics of overparameterized neural networks trained with gradient descent can be well approximated by a linear system. When normalized training data is uniformly distributed on a hypersphere, the eigenfunctions of this linear system are spherical harmonic functions. We derive the corresponding eigenvalues for each frequency after introducing a bias term in the model. This bias term had been omitted from the linear network model without significantly affecting previous theoretical results. However, we show theoretically and experimentally that a shallow neural network without bias cannot represent or learn simple, low frequency functions with odd frequencies. Our results lead to specific predictions of the time it will take a network to learn functions of varying frequency. These predictions match the empirical behavior of both shallow and deep networks.
△ Less
Submitted 2 December, 2019; v1 submitted 2 June, 2019;
originally announced June 2019.
-
Adversarially robust transfer learning
Authors:
Ali Shafahi,
Parsa Saadatpanah,
Chen Zhu,
Amin Ghiasi,
Christoph Studer,
David Jacobs,
Tom Goldstein
Abstract:
Transfer learning, in which a network is trained on one task and re-purposed on another, is often used to produce neural network classifiers when data is scarce or full-scale training is too costly. When the goal is to produce a model that is not only accurate but also adversarially robust, data scarcity and computational limitations become even more cumbersome. We consider robust transfer learnin…
▽ More
Transfer learning, in which a network is trained on one task and re-purposed on another, is often used to produce neural network classifiers when data is scarce or full-scale training is too costly. When the goal is to produce a model that is not only accurate but also adversarially robust, data scarcity and computational limitations become even more cumbersome. We consider robust transfer learning, in which we transfer not only performance but also robustness from a source model to a target domain. We start by observing that robust networks contain robust feature extractors. By training classifiers on top of these feature extractors, we produce new models that inherit the robustness of their parent networks. We then consider the case of fine tuning a network by re-training end-to-end in the target domain. When using lifelong learning strategies, this process preserves the robustness of the source network while achieving high accuracy. By using such strategies, it is possible to produce accurate and robust models with little data, and without the cost of adversarial training. Additionally, we can improve the generalization of adversarially trained models, while maintaining their robustness.
△ Less
Submitted 21 February, 2020; v1 submitted 20 May, 2019;
originally announced May 2019.
-
Understanding the (un)interpretability of natural image distributions using generative models
Authors:
Ryen Krusinga,
Sohil Shah,
Matthias Zwicker,
Tom Goldstein,
David Jacobs
Abstract:
Probability density estimation is a classical and well studied problem, but standard density estimation methods have historically lacked the power to model complex and high-dimensional image distributions. More recent generative models leverage the power of neural networks to implicitly learn and represent probability models over complex images. We describe methods to extract explicit probability…
▽ More
Probability density estimation is a classical and well studied problem, but standard density estimation methods have historically lacked the power to model complex and high-dimensional image distributions. More recent generative models leverage the power of neural networks to implicitly learn and represent probability models over complex images. We describe methods to extract explicit probability density estimates from GANs, and explore the properties of these image density functions. We perform sanity check experiments to provide evidence that these probabilities are reasonable. However, we also show that density functions of natural images are difficult to interpret and thus limited in use. We study reasons for this lack of interpretability, and show that we can get interpretability back by doing density estimation on latent representations of images.
△ Less
Submitted 25 February, 2019; v1 submitted 5 January, 2019;
originally announced January 2019.
-
Big Batch SGD: Automated Inference using Adaptive Batch Sizes
Authors:
Soham De,
Abhay Yadav,
David Jacobs,
Tom Goldstein
Abstract:
Classical stochastic gradient methods for optimization rely on noisy gradient approximations that become progressively less accurate as iterates approach a solution. The large noise and small signal in the resulting gradients makes it difficult to use them for adaptive stepsize selection and automatic stopping. We propose alternative "big batch" SGD schemes that adaptively grow the batch size over…
▽ More
Classical stochastic gradient methods for optimization rely on noisy gradient approximations that become progressively less accurate as iterates approach a solution. The large noise and small signal in the resulting gradients makes it difficult to use them for adaptive stepsize selection and automatic stopping. We propose alternative "big batch" SGD schemes that adaptively grow the batch size over time to maintain a nearly constant signal-to-noise ratio in the gradient approximation. The resulting methods have similar convergence rates to classical SGD, and do not require convexity of the objective. The high fidelity gradients enable automated learning rate selection and do not require stepsize decay. Big batch methods are thus easily automated and can run with little or no oversight.
△ Less
Submitted 6 April, 2017; v1 submitted 18 October, 2016;
originally announced October 2016.
-
Nonparametric Maximum Entropy Probability Density Estimation
Authors:
Jenny Farmer,
Donald J. Jacobs
Abstract:
Given a sample of independent and identically distributed random variables, a novel nonparametric maximum entropy method is presented to estimate the underlying continuous univariate probability density function (pdf). Estimates are found by maximizing a log-likelihood function based on single order statistics after transforming through a sequence of trial cumulative distribution functions that it…
▽ More
Given a sample of independent and identically distributed random variables, a novel nonparametric maximum entropy method is presented to estimate the underlying continuous univariate probability density function (pdf). Estimates are found by maximizing a log-likelihood function based on single order statistics after transforming through a sequence of trial cumulative distribution functions that iteratively improve using a Monte Carlo random search method. Improvement is quantified by assessing the random variables against the statistical properties of sampled uniform random data. Quality is determined using an empirically derived scoring function that is scaled to be sample size invariant. The scoring function identifies atypical fluctuations, for which threshold values are set to define objective criteria that prevent under-fitting as trial iterations continue to improve the model pdf, and, stopping the iteration cycle before over-fitting occurs. No prior knowledge about the data is required. An ensemble of pdf models is used to reflect uncertainties due to statistical fluctuations in random samples, and the quality of the estimates is visualized using scaled residual quantile plots that show deviations from size-invariant statistics. These considerations result in a tractable method that holistically employs key principles of random variables and their statistical properties combined with employing orthogonal basis functions and data-driven adaptive algorithms. Benchmark tests show that the pdf estimates readily converge to the true pdf as sample size increases. Robust results are demonstrated on several test probability densities that include cases with discontinuities, multi-resolution scales, heavy tails and singularities in the pdf, suggesting a generally applicable approach for statistical inference.
△ Less
Submitted 28 June, 2016;
originally announced June 2016.
-
Efficient Representation of Low-Dimensional Manifolds using Deep Networks
Authors:
Ronen Basri,
David Jacobs
Abstract:
We consider the ability of deep neural networks to represent data that lies near a low-dimensional manifold in a high-dimensional space. We show that deep networks can efficiently extract the intrinsic, low-dimensional coordinates of such data. We first show that the first two layers of a deep network can exactly embed points lying on a monotonic chain, a special type of piecewise linear manifold,…
▽ More
We consider the ability of deep neural networks to represent data that lies near a low-dimensional manifold in a high-dimensional space. We show that deep networks can efficiently extract the intrinsic, low-dimensional coordinates of such data. We first show that the first two layers of a deep network can exactly embed points lying on a monotonic chain, a special type of piecewise linear manifold, mapping them to a low-dimensional Euclidean space. Remarkably, the network can do this using an almost optimal number of parameters. We also show that this network projects nearby points onto the manifold and then embeds them with little error. We then extend these results to more general manifolds.
△ Less
Submitted 15 February, 2016;
originally announced February 2016.
