-
Mitigating Biases with Diverse Ensembles and Diffusion Models
Authors:
Luca Scimeca,
Alexander Rubinstein,
Damien Teney,
Seong Joon Oh,
Armand Mihai Nicolicioiu,
Yoshua Bengio
Abstract:
Spurious correlations in the data, where multiple cues are predictive of the target labels, often lead to a phenomenon known as shortcut learning, where a model relies on erroneous, easy-to-learn cues while ignoring reliable ones. In this work, we propose an ensemble diversification framework exploiting Diffusion Probabilistic Models (DPMs) to mitigate this form of bias. We show that at particular…
▽ More
Spurious correlations in the data, where multiple cues are predictive of the target labels, often lead to a phenomenon known as shortcut learning, where a model relies on erroneous, easy-to-learn cues while ignoring reliable ones. In this work, we propose an ensemble diversification framework exploiting Diffusion Probabilistic Models (DPMs) to mitigate this form of bias. We show that at particular training intervals, DPMs can generate images with novel feature combinations, even when trained on samples displaying correlated input features. We leverage this crucial property to generate synthetic counterfactuals to increase model diversity via ensemble disagreement. We show that DPM-guided diversification is sufficient to remove dependence on primary shortcut cues, without a need for additional supervised signals. We further empirically quantify its efficacy on several diversification objectives, and finally show improved generalization and diversification performance on par with prior work that relies on auxiliary data collection.
△ Less
Submitted 6 March, 2024; v1 submitted 23 November, 2023;
originally announced November 2023.
-
Leveraging Diffusion Disentangled Representations to Mitigate Shortcuts in Underspecified Visual Tasks
Authors:
Luca Scimeca,
Alexander Rubinstein,
Armand Mihai Nicolicioiu,
Damien Teney,
Yoshua Bengio
Abstract:
Spurious correlations in the data, where multiple cues are predictive of the target labels, often lead to shortcut learning phenomena, where a model may rely on erroneous, easy-to-learn, cues while ignoring reliable ones. In this work, we propose an ensemble diversification framework exploiting the generation of synthetic counterfactuals using Diffusion Probabilistic Models (DPMs). We discover tha…
▽ More
Spurious correlations in the data, where multiple cues are predictive of the target labels, often lead to shortcut learning phenomena, where a model may rely on erroneous, easy-to-learn, cues while ignoring reliable ones. In this work, we propose an ensemble diversification framework exploiting the generation of synthetic counterfactuals using Diffusion Probabilistic Models (DPMs). We discover that DPMs have the inherent capability to represent multiple visual cues independently, even when they are largely correlated in the training data. We leverage this characteristic to encourage model diversity and empirically show the efficacy of the approach with respect to several diversification objectives. We show that diffusion-guided diversification can lead models to avert attention from shortcut cues, achieving ensemble diversity performance comparable to previous methods requiring additional data collection.
△ Less
Submitted 18 November, 2023; v1 submitted 3 October, 2023;
originally announced October 2023.
-
Learning Diverse Features in Vision Transformers for Improved Generalization
Authors:
Armand Mihai Nicolicioiu,
Andrei Liviu Nicolicioiu,
Bogdan Alexe,
Damien Teney
Abstract:
Deep learning models often rely only on a small set of features even when there is a rich set of predictive signals in the training data. This makes models brittle and sensitive to distribution shifts. In this work, we first examine vision transformers (ViTs) and find that they tend to extract robust and spurious features with distinct attention heads. As a result of this modularity, their perform…
▽ More
Deep learning models often rely only on a small set of features even when there is a rich set of predictive signals in the training data. This makes models brittle and sensitive to distribution shifts. In this work, we first examine vision transformers (ViTs) and find that they tend to extract robust and spurious features with distinct attention heads. As a result of this modularity, their performance under distribution shifts can be significantly improved at test time by pruning heads corresponding to spurious features, which we demonstrate using an "oracle selection" on validation data. Second, we propose a method to further enhance the diversity and complementarity of the learned features by encouraging orthogonality of the attention heads' input gradients. We observe improved out-of-distribution performance on diagnostic benchmarks (MNIST-CIFAR, Waterbirds) as a consequence of the enhanced diversity of features and the pruning of undesirable heads.
△ Less
Submitted 30 August, 2023;
originally announced August 2023.
-
ICLR 2021 Challenge for Computational Geometry & Topology: Design and Results
Authors:
Nina Miolane,
Matteo Caorsi,
Umberto Lupo,
Marius Guerard,
Nicolas Guigui,
Johan Mathe,
Yann Cabanes,
Wojciech Reise,
Thomas Davies,
António Leitão,
Somesh Mohapatra,
Saiteja Utpala,
Shailja Shailja,
Gabriele Corso,
Guoxi Liu,
Federico Iuricich,
Andrei Manolache,
Mihaela Nistor,
Matei Bejan,
Armand Mihai Nicolicioiu,
Bogdan-Alexandru Luchian,
Mihai-Sorin Stupariu,
Florent Michel,
Khanh Dao Duc,
Bilal Abdulrahman
, et al. (8 additional authors not shown)
Abstract:
This paper presents the computational challenge on differential geometry and topology that happened within the ICLR 2021 workshop "Geometric and Topological Representation Learning". The competition asked participants to provide creative contributions to the fields of computational geometry and topology through the open-source repositories Geomstats and Giotto-TDA. The challenge attracted 16 teams…
▽ More
This paper presents the computational challenge on differential geometry and topology that happened within the ICLR 2021 workshop "Geometric and Topological Representation Learning". The competition asked participants to provide creative contributions to the fields of computational geometry and topology through the open-source repositories Geomstats and Giotto-TDA. The challenge attracted 16 teams in its two month duration. This paper describes the design of the challenge and summarizes its main findings.
△ Less
Submitted 25 August, 2021; v1 submitted 22 August, 2021;
originally announced August 2021.
