-
Label-wise Aleatoric and Epistemic Uncertainty Quantification
Authors:
Yusuf Sale,
Paul Hofman,
Timo Löhr,
Lisa Wimmer,
Thomas Nagler,
Eyke Hüllermeier
Abstract:
We present a novel approach to uncertainty quantification in classification tasks based on label-wise decomposition of uncertainty measures. This label-wise perspective allows uncertainty to be quantified at the individual class level, thereby improving cost-sensitive decision-making and helping understand the sources of uncertainty. Furthermore, it allows to define total, aleatoric, and epistemic…
▽ More
We present a novel approach to uncertainty quantification in classification tasks based on label-wise decomposition of uncertainty measures. This label-wise perspective allows uncertainty to be quantified at the individual class level, thereby improving cost-sensitive decision-making and helping understand the sources of uncertainty. Furthermore, it allows to define total, aleatoric, and epistemic uncertainty on the basis of non-categorical measures such as variance, going beyond common entropy-based measures. In particular, variance-based measures address some of the limitations associated with established methods that have recently been discussed in the literature. We show that our proposed measures adhere to a number of desirable properties. Through empirical evaluation on a variety of benchmark data sets -- including applications in the medical domain where accurate uncertainty quantification is crucial -- we establish the effectiveness of label-wise uncertainty quantification.
△ Less
Submitted 4 June, 2024;
originally announced June 2024.
-
Connecting the Dots: Is Mode-Connectedness the Key to Feasible Sample-Based Inference in Bayesian Neural Networks?
Authors:
Emanuel Sommer,
Lisa Wimmer,
Theodore Papamarkou,
Ludwig Bothmann,
Bernd Bischl,
David Rügamer
Abstract:
A major challenge in sample-based inference (SBI) for Bayesian neural networks is the size and structure of the networks' parameter space. Our work shows that successful SBI is possible by embracing the characteristic relationship between weight and function space, uncovering a systematic link between overparameterization and the difficulty of the sampling problem. Through extensive experiments, w…
▽ More
A major challenge in sample-based inference (SBI) for Bayesian neural networks is the size and structure of the networks' parameter space. Our work shows that successful SBI is possible by embracing the characteristic relationship between weight and function space, uncovering a systematic link between overparameterization and the difficulty of the sampling problem. Through extensive experiments, we establish practical guidelines for sampling and convergence diagnosis. As a result, we present a deep ensemble initialized approach as an effective solution with competitive performance and uncertainty quantification.
△ Less
Submitted 27 May, 2024; v1 submitted 2 February, 2024;
originally announced February 2024.
-
Second-Order Uncertainty Quantification: Variance-Based Measures
Authors:
Yusuf Sale,
Paul Hofman,
Lisa Wimmer,
Eyke Hüllermeier,
Thomas Nagler
Abstract:
Uncertainty quantification is a critical aspect of machine learning models, providing important insights into the reliability of predictions and aiding the decision-making process in real-world applications. This paper proposes a novel way to use variance-based measures to quantify uncertainty on the basis of second-order distributions in classification problems. A distinctive feature of the measu…
▽ More
Uncertainty quantification is a critical aspect of machine learning models, providing important insights into the reliability of predictions and aiding the decision-making process in real-world applications. This paper proposes a novel way to use variance-based measures to quantify uncertainty on the basis of second-order distributions in classification problems. A distinctive feature of the measures is the ability to reason about uncertainties on a class-based level, which is useful in situations where nuanced decision-making is required. Recalling some properties from the literature, we highlight that the variance-based measures satisfy important (axiomatic) properties. In addition to this axiomatic approach, we present empirical results showing the measures to be effective and competitive to commonly used entropy-based measures.
△ Less
Submitted 30 December, 2023;
originally announced January 2024.
-
Probabilistic Self-supervised Learning via Scoring Rules Minimization
Authors:
Amirhossein Vahidi,
Simon Schoßer,
Lisa Wimmer,
Yawei Li,
Bernd Bischl,
Eyke Hüllermeier,
Mina Rezaei
Abstract:
In this paper, we propose a novel probabilistic self-supervised learning via Scoring Rule Minimization (ProSMIN), which leverages the power of probabilistic models to enhance representation quality and mitigate collapsing representations. Our proposed approach involves two neural networks; the online network and the target network, which collaborate and learn the diverse distribution of representa…
▽ More
In this paper, we propose a novel probabilistic self-supervised learning via Scoring Rule Minimization (ProSMIN), which leverages the power of probabilistic models to enhance representation quality and mitigate collapsing representations. Our proposed approach involves two neural networks; the online network and the target network, which collaborate and learn the diverse distribution of representations from each other through knowledge distillation. By presenting the input samples in two augmented formats, the online network is trained to predict the target network representation of the same sample under a different augmented view. The two networks are trained via our new loss function based on proper scoring rules. We provide a theoretical justification for ProSMIN's convergence, demonstrating the strict propriety of its modified scoring rule. This insight validates the method's optimization process and contributes to its robustness and effectiveness in improving representation quality. We evaluate our probabilistic model on various downstream tasks, such as in-distribution generalization, out-of-distribution detection, dataset corruption, low-shot learning, and transfer learning. Our method achieves superior accuracy and calibration, surpassing the self-supervised baseline in a wide range of experiments on large-scale datasets like ImageNet-O and ImageNet-C, ProSMIN demonstrates its scalability and real-world applicability.
△ Less
Submitted 5 September, 2023;
originally announced September 2023.
-
Diversified Ensemble of Independent Sub-Networks for Robust Self-Supervised Representation Learning
Authors:
Amirhossein Vahidi,
Lisa Wimmer,
Hüseyin Anil Gündüz,
Bernd Bischl,
Eyke Hüllermeier,
Mina Rezaei
Abstract:
Ensembling a neural network is a widely recognized approach to enhance model performance, estimate uncertainty, and improve robustness in deep supervised learning. However, deep ensembles often come with high computational costs and memory demands. In addition, the efficiency of a deep ensemble is related to diversity among the ensemble members which is challenging for large, over-parameterized de…
▽ More
Ensembling a neural network is a widely recognized approach to enhance model performance, estimate uncertainty, and improve robustness in deep supervised learning. However, deep ensembles often come with high computational costs and memory demands. In addition, the efficiency of a deep ensemble is related to diversity among the ensemble members which is challenging for large, over-parameterized deep neural networks. Moreover, ensemble learning has not yet seen such widespread adoption, and it remains a challenging endeavor for self-supervised or unsupervised representation learning. Motivated by these challenges, we present a novel self-supervised training regime that leverages an ensemble of independent sub-networks, complemented by a new loss function designed to encourage diversity. Our method efficiently builds a sub-model ensemble with high diversity, leading to well-calibrated estimates of model uncertainty, all achieved with minimal computational overhead compared to traditional deep self-supervised ensembles. To evaluate the effectiveness of our approach, we conducted extensive experiments across various tasks, including in-distribution generalization, out-of-distribution detection, dataset corruption, and semi-supervised settings. The results demonstrate that our method significantly improves prediction reliability. Our approach not only achieves excellent accuracy but also enhances calibration, surpassing baseline performance across a wide range of self-supervised architectures in computer vision, natural language processing, and genomics data.
△ Less
Submitted 1 September, 2023; v1 submitted 28 August, 2023;
originally announced August 2023.
-
Towards Efficient MCMC Sampling in Bayesian Neural Networks by Exploiting Symmetry
Authors:
Jonas Gregor Wiese,
Lisa Wimmer,
Theodore Papamarkou,
Bernd Bischl,
Stephan Günnemann,
David Rügamer
Abstract:
Bayesian inference in deep neural networks is challenging due to the high-dimensional, strongly multi-modal parameter posterior density landscape. Markov chain Monte Carlo approaches asymptotically recover the true posterior but are considered prohibitively expensive for large modern architectures. Local methods, which have emerged as a popular alternative, focus on specific parameter regions that…
▽ More
Bayesian inference in deep neural networks is challenging due to the high-dimensional, strongly multi-modal parameter posterior density landscape. Markov chain Monte Carlo approaches asymptotically recover the true posterior but are considered prohibitively expensive for large modern architectures. Local methods, which have emerged as a popular alternative, focus on specific parameter regions that can be approximated by functions with tractable integrals. While these often yield satisfactory empirical results, they fail, by definition, to account for the multi-modality of the parameter posterior. In this work, we argue that the dilemma between exact-but-unaffordable and cheap-but-inexact approaches can be mitigated by exploiting symmetries in the posterior landscape. Such symmetries, induced by neuron interchangeability and certain activation functions, manifest in different parameter values leading to the same functional output value. We show theoretically that the posterior predictive density in Bayesian neural networks can be restricted to a symmetry-free parameter reference set. By further deriving an upper bound on the number of Monte Carlo chains required to capture the functional diversity, we propose a straightforward approach for feasible Bayesian inference. Our experiments suggest that efficient sampling is indeed possible, opening up a promising path to accurate uncertainty quantification in deep learning.
△ Less
Submitted 6 April, 2023;
originally announced April 2023.
-
Automated wildlife image classification: An active learning tool for ecological applications
Authors:
Ludwig Bothmann,
Lisa Wimmer,
Omid Charrakh,
Tobias Weber,
Hendrik Edelhoff,
Wibke Peters,
Hien Nguyen,
Caryl Benjamin,
Annette Menzel
Abstract:
Wildlife camera trap images are being used extensively to investigate animal abundance, habitat associations, and behavior, which is complicated by the fact that experts must first classify the images manually. Artificial intelligence systems can take over this task but usually need a large number of already-labeled training images to achieve sufficient performance. This requirement necessitates h…
▽ More
Wildlife camera trap images are being used extensively to investigate animal abundance, habitat associations, and behavior, which is complicated by the fact that experts must first classify the images manually. Artificial intelligence systems can take over this task but usually need a large number of already-labeled training images to achieve sufficient performance. This requirement necessitates human expert labor and poses a particular challenge for projects with few cameras or short durations. We propose a label-efficient learning strategy that enables researchers with small or medium-sized image databases to leverage the potential of modern machine learning, thus freeing crucial resources for subsequent analyses.
Our methodological proposal is two-fold: (1) We improve current strategies of combining object detection and image classification by tuning the hyperparameters of both models. (2) We provide an active learning (AL) system that allows training deep learning models very efficiently in terms of required human-labeled training images. We supply a software package that enables researchers to use these methods directly and thereby ensure the broad applicability of the proposed framework in ecological practice.
We show that our tuning strategy improves predictive performance. We demonstrate how the AL pipeline reduces the amount of pre-labeled data needed to achieve a specific predictive performance and that it is especially valuable for improving out-of-sample predictive performance.
We conclude that the combination of tuning and AL increases predictive performance substantially. Furthermore, we argue that our work can broadly impact the community through the ready-to-use software package provided. Finally, the publication of our models tailored to European wildlife data enriches existing model bases mostly trained on data from Africa and North America.
△ Less
Submitted 2 August, 2023; v1 submitted 28 March, 2023;
originally announced March 2023.
-
Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?
Authors:
Lisa Wimmer,
Yusuf Sale,
Paul Hofman,
Bern Bischl,
Eyke Hüllermeier
Abstract:
The quantification of aleatoric and epistemic uncertainty in terms of conditional entropy and mutual information, respectively, has recently become quite common in machine learning. While the properties of these measures, which are rooted in information theory, seem appealing at first glance, we identify various incoherencies that call their appropriateness into question. In addition to the measur…
▽ More
The quantification of aleatoric and epistemic uncertainty in terms of conditional entropy and mutual information, respectively, has recently become quite common in machine learning. While the properties of these measures, which are rooted in information theory, seem appealing at first glance, we identify various incoherencies that call their appropriateness into question. In addition to the measures themselves, we critically discuss the idea of an additive decomposition of total uncertainty into its aleatoric and epistemic constituents. Experiments across different computer vision tasks support our theoretical findings and raise concerns about current practice in uncertainty quantification.
△ Less
Submitted 25 June, 2023; v1 submitted 7 September, 2022;
originally announced September 2022.
-
MONROE-Nettest: A Configurable Tool for Dissecting Speed Measurements in Mobile Broadband Networks
Authors:
Cise Midoglu,
Leonhard Wimmer,
Andra Lutu,
Ozgu Alay,
Carsten Griwodz
Abstract:
As the demand for mobile connectivity continues to grow, there is a strong need to evaluate the performance of Mobile Broadband (MBB) networks. In the last years, mobile "speed", quantified most commonly by data rate, gained popularity as the widely accepted metric to describe their performance. However, there is a lack of consensus on how mobile speed should be measured. In this paper, we design…
▽ More
As the demand for mobile connectivity continues to grow, there is a strong need to evaluate the performance of Mobile Broadband (MBB) networks. In the last years, mobile "speed", quantified most commonly by data rate, gained popularity as the widely accepted metric to describe their performance. However, there is a lack of consensus on how mobile speed should be measured. In this paper, we design and implement MONROE-Nettest to dissect mobile speed measurements, and investigate the effect of different factors on speed measurements in the complex mobile ecosystem. MONROE-Nettest is built as an Experiment as a Service (EaaS) on top of the MONROE platform, an open dedicated platform for experimentation in operational MBB networks. Using MONROE-Nettest, we conduct a large scale measurement campaign and quantify the effects of measurement duration, number of TCP flows, and server location on measured downlink data rate in 6 operational MBB networks in Europe. Our results indicate that differences in parameter configuration can significantly affect the measurement results. We provide the complete MONROE-Nettest toolset as open source and our measurements as open data.
△ Less
Submitted 30 January, 2018; v1 submitted 21 October, 2017;
originally announced October 2017.
