-
Dimensionality Reduction and Nearest Neighbors for Improving Out-of-Distribution Detection in Medical Image Segmentation
Authors:
McKell Woodland,
Nihil Patel,
Austin Castelo,
Mais Al Taie,
Mohamed Eltaher,
Joshua P. Yung,
Tucker J. Netherton,
Tiffany L. Calderone,
Jessica I. Sanchez,
Darrel W. Cleere,
Ahmed Elsaiey,
Nakul Gupta,
David Victor,
Laura Beretta,
Ankit B. Patel Kristy K. Brock
Abstract:
Clinically deployed deep learning-based segmentation models are known to fail on data outside of their training distributions. While clinicians review the segmentations, these models tend to perform well in most instances, which could exacerbate automation bias. Therefore, detecting out-of-distribution images at inference is critical to warn the clinicians that the model likely failed. This work a…
▽ More
Clinically deployed deep learning-based segmentation models are known to fail on data outside of their training distributions. While clinicians review the segmentations, these models tend to perform well in most instances, which could exacerbate automation bias. Therefore, detecting out-of-distribution images at inference is critical to warn the clinicians that the model likely failed. This work applied the Mahalanobis distance (MD) post hoc to the bottleneck features of four Swin UNETR and nnU-net models that segmented the liver on T1-weighted magnetic resonance imaging and computed tomography. By reducing the dimensions of the bottleneck features with either principal component analysis or uniform manifold approximation and projection, images the models failed on were detected with high performance and minimal computational load. In addition, this work explored a non-parametric alternative to the MD, a k-th nearest neighbors distance (KNN). KNN drastically improved scalability and performance over MD when both were applied to raw and average-pooled bottleneck features.
△ Less
Submitted 5 August, 2024;
originally announced August 2024.
-
Feature Extraction for Generative Medical Imaging Evaluation: New Evidence Against an Evolving Trend
Authors:
McKell Woodland,
Austin Castelo,
Mais Al Taie,
Jessica Albuquerque Marques Silva,
Mohamed Eltaher,
Frank Mohn,
Alexander Shieh,
Suprateek Kundu,
Joshua P. Yung,
Ankit B. Patel,
Kristy K. Brock
Abstract:
Fréchet Inception Distance (FID) is a widely used metric for assessing synthetic image quality. It relies on an ImageNet-based feature extractor, making its applicability to medical imaging unclear. A recent trend is to adapt FID to medical imaging through feature extractors trained on medical images. Our study challenges this practice by demonstrating that ImageNet-based extractors are more consi…
▽ More
Fréchet Inception Distance (FID) is a widely used metric for assessing synthetic image quality. It relies on an ImageNet-based feature extractor, making its applicability to medical imaging unclear. A recent trend is to adapt FID to medical imaging through feature extractors trained on medical images. Our study challenges this practice by demonstrating that ImageNet-based extractors are more consistent and aligned with human judgment than their RadImageNet counterparts. We evaluated sixteen StyleGAN2 networks across four medical imaging modalities and four data augmentation techniques with Fréchet distances (FDs) computed using eleven ImageNet or RadImageNet-trained feature extractors. Comparison with human judgment via visual Turing tests revealed that ImageNet-based extractors produced rankings consistent with human judgment, with the FD derived from the ImageNet-trained SwAV extractor significantly correlating with expert evaluations. In contrast, RadImageNet-based rankings were volatile and inconsistent with human judgment. Our findings challenge prevailing assumptions, providing novel evidence that medical image-trained feature extractors do not inherently improve FDs and can even compromise their reliability. Our code is available at https://github.com/mckellwoodland/fid-med-eval.
△ Less
Submitted 7 August, 2024; v1 submitted 22 November, 2023;
originally announced November 2023.
-
Dimensionality Reduction for Improving Out-of-Distribution Detection in Medical Image Segmentation
Authors:
McKell Woodland,
Nihil Patel,
Mais Al Taie,
Joshua P. Yung,
Tucker J. Netherton,
Ankit B. Patel,
Kristy K. Brock
Abstract:
Clinically deployed segmentation models are known to fail on data outside of their training distribution. As these models perform well on most cases, it is imperative to detect out-of-distribution (OOD) images at inference to protect against automation bias. This work applies the Mahalanobis distance post hoc to the bottleneck features of a Swin UNETR model that segments the liver on T1-weighted m…
▽ More
Clinically deployed segmentation models are known to fail on data outside of their training distribution. As these models perform well on most cases, it is imperative to detect out-of-distribution (OOD) images at inference to protect against automation bias. This work applies the Mahalanobis distance post hoc to the bottleneck features of a Swin UNETR model that segments the liver on T1-weighted magnetic resonance imaging. By reducing the dimensions of the bottleneck features with principal component analysis, OOD images were detected with high performance and minimal computational load.
△ Less
Submitted 19 October, 2023; v1 submitted 7 August, 2023;
originally announced August 2023.
-
Correlation between image quality metrics of magnetic resonance images and the neural network segmentation accuracy
Authors:
Rajarajeswari Muthusivarajan,
Adrian Celaya,
Joshua P. Yung,
Satish Viswanath,
Daniel S. Marcus,
Caroline Chung,
David Fuentes
Abstract:
Deep neural networks with multilevel connections process input data in complex ways to learn the information.A networks learning efficiency depends not only on the complex neural network architecture but also on the input training images.Medical image segmentation with deep neural networks for skull stripping or tumor segmentation from magnetic resonance images enables learning both global and loc…
▽ More
Deep neural networks with multilevel connections process input data in complex ways to learn the information.A networks learning efficiency depends not only on the complex neural network architecture but also on the input training images.Medical image segmentation with deep neural networks for skull stripping or tumor segmentation from magnetic resonance images enables learning both global and local features of the images.Though medical images are collected in a controlled environment,there may be artifacts or equipment based variance that cause inherent bias in the input set.In this study, we investigated the correlation between the image quality metrics of MR images with the neural network segmentation accuracy.For that we have used the 3D DenseNet architecture and let the network trained on the same input but applying different methodologies to select the training data set based on the IQM values.The difference in the segmentation accuracy between models based on the random training inputs with IQM based training inputs shed light on the role of image quality metrics on segmentation accuracy.By running the image quality metrics to choose the training inputs,further we may tune the learning efficiency of the network and the segmentation accuracy.
△ Less
Submitted 1 November, 2021;
originally announced November 2021.
-
Augmenting the User-Item Graph with Textual Similarity Models
Authors:
Federico López,
Martin Scholz,
Jessica Yung,
Marie Pellat,
Michael Strube,
Lucas Dixon
Abstract:
This paper introduces a simple and effective form of data augmentation for recommender systems. A paraphrase similarity model is applied to widely available textual data, such as reviews and product descriptions, yielding new semantic relations that are added to the user-item graph. This increases the density of the graph without needing further labeled data. The data augmentation is evaluated on…
▽ More
This paper introduces a simple and effective form of data augmentation for recommender systems. A paraphrase similarity model is applied to widely available textual data, such as reviews and product descriptions, yielding new semantic relations that are added to the user-item graph. This increases the density of the graph without needing further labeled data. The data augmentation is evaluated on a variety of recommendation algorithms, using Euclidean, hyperbolic, and complex spaces, and over three categories of Amazon product reviews with differing characteristics. Results show that the data augmentation technique provides significant improvements to all types of models, with the most pronounced gains for knowledge graph-based recommenders, particularly in cold-start settings, leading to state-of-the-art performance.
△ Less
Submitted 20 September, 2021;
originally announced September 2021.
-
MLP-Mixer: An all-MLP Architecture for Vision
Authors:
Ilya Tolstikhin,
Neil Houlsby,
Alexander Kolesnikov,
Lucas Beyer,
Xiaohua Zhai,
Thomas Unterthiner,
Jessica Yung,
Andreas Steiner,
Daniel Keysers,
Jakob Uszkoreit,
Mario Lucic,
Alexey Dosovitskiy
Abstract:
Convolutional Neural Networks (CNNs) are the go-to model for computer vision. Recently, attention-based networks, such as the Vision Transformer, have also become popular. In this paper we show that while convolutions and attention are both sufficient for good performance, neither of them are necessary. We present MLP-Mixer, an architecture based exclusively on multi-layer perceptrons (MLPs). MLP-…
▽ More
Convolutional Neural Networks (CNNs) are the go-to model for computer vision. Recently, attention-based networks, such as the Vision Transformer, have also become popular. In this paper we show that while convolutions and attention are both sufficient for good performance, neither of them are necessary. We present MLP-Mixer, an architecture based exclusively on multi-layer perceptrons (MLPs). MLP-Mixer contains two types of layers: one with MLPs applied independently to image patches (i.e. "mixing" the per-location features), and one with MLPs applied across patches (i.e. "mixing" spatial information). When trained on large datasets, or with modern regularization schemes, MLP-Mixer attains competitive scores on image classification benchmarks, with pre-training and inference cost comparable to state-of-the-art models. We hope that these results spark further research beyond the realms of well established CNNs and Transformers.
△ Less
Submitted 11 June, 2021; v1 submitted 4 May, 2021;
originally announced May 2021.
-
SI-Score: An image dataset for fine-grained analysis of robustness to object location, rotation and size
Authors:
Jessica Yung,
Rob Romijnders,
Alexander Kolesnikov,
Lucas Beyer,
Josip Djolonga,
Neil Houlsby,
Sylvain Gelly,
Mario Lucic,
Xiaohua Zhai
Abstract:
Before deploying machine learning models it is critical to assess their robustness. In the context of deep neural networks for image understanding, changing the object location, rotation and size may affect the predictions in non-trivial ways. In this work we perform a fine-grained analysis of robustness with respect to these factors of variation using SI-Score, a synthetic dataset. In particular,…
▽ More
Before deploying machine learning models it is critical to assess their robustness. In the context of deep neural networks for image understanding, changing the object location, rotation and size may affect the predictions in non-trivial ways. In this work we perform a fine-grained analysis of robustness with respect to these factors of variation using SI-Score, a synthetic dataset. In particular, we investigate ResNets, Vision Transformers and CLIP, and identify interesting qualitative differences between these.
△ Less
Submitted 9 April, 2021;
originally announced April 2021.
-
On Robustness and Transferability of Convolutional Neural Networks
Authors:
Josip Djolonga,
Jessica Yung,
Michael Tschannen,
Rob Romijnders,
Lucas Beyer,
Alexander Kolesnikov,
Joan Puigcerver,
Matthias Minderer,
Alexander D'Amour,
Dan Moldovan,
Sylvain Gelly,
Neil Houlsby,
Xiaohua Zhai,
Mario Lucic
Abstract:
Modern deep convolutional networks (CNNs) are often criticized for not generalizing under distributional shifts. However, several recent breakthroughs in transfer learning suggest that these networks can cope with severe distribution shifts and successfully adapt to new tasks from a few training examples. In this work we study the interplay between out-of-distribution and transfer performance of m…
▽ More
Modern deep convolutional networks (CNNs) are often criticized for not generalizing under distributional shifts. However, several recent breakthroughs in transfer learning suggest that these networks can cope with severe distribution shifts and successfully adapt to new tasks from a few training examples. In this work we study the interplay between out-of-distribution and transfer performance of modern image classification CNNs for the first time and investigate the impact of the pre-training data size, the model scale, and the data preprocessing pipeline. We find that increasing both the training set and model sizes significantly improve the distributional shift robustness. Furthermore, we show that, perhaps surprisingly, simple changes in the preprocessing such as modifying the image resolution can significantly mitigate robustness issues in some cases. Finally, we outline the shortcomings of existing robustness evaluation datasets and introduce a synthetic dataset SI-Score we use for a systematic analysis across factors of variation common in visual data such as object size and position.
△ Less
Submitted 23 March, 2021; v1 submitted 16 July, 2020;
originally announced July 2020.
-
Big Transfer (BiT): General Visual Representation Learning
Authors:
Alexander Kolesnikov,
Lucas Beyer,
Xiaohua Zhai,
Joan Puigcerver,
Jessica Yung,
Sylvain Gelly,
Neil Houlsby
Abstract:
Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components,…
▽ More
Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. BiT performs well across a surprisingly wide range of data regimes -- from 1 example per class to 1M total examples. BiT achieves 87.5% top-1 accuracy on ILSVRC-2012, 99.4% on CIFAR-10, and 76.3% on the 19 task Visual Task Adaptation Benchmark (VTAB). On small datasets, BiT attains 76.8% on ILSVRC-2012 with 10 examples per class, and 97.0% on CIFAR-10 with 10 examples per class. We conduct detailed analysis of the main components that lead to high transfer performance.
△ Less
Submitted 5 May, 2020; v1 submitted 24 December, 2019;
originally announced December 2019.