-
Barlow Twins: Self-Supervised Learning via Redundancy Reduction
Authors:
Jure Zbontar,
Li Jing,
Ishan Misra,
Yann LeCun,
Stéphane Deny
Abstract:
Self-supervised learning (SSL) is rapidly closing the gap with supervised methods on large computer vision benchmarks. A successful approach to SSL is to learn embeddings which are invariant to distortions of the input sample. However, a recurring issue with this approach is the existence of trivial constant solutions. Most current methods avoid such solutions by careful implementation details. We…
▽ More
Self-supervised learning (SSL) is rapidly closing the gap with supervised methods on large computer vision benchmarks. A successful approach to SSL is to learn embeddings which are invariant to distortions of the input sample. However, a recurring issue with this approach is the existence of trivial constant solutions. Most current methods avoid such solutions by careful implementation details. We propose an objective function that naturally avoids collapse by measuring the cross-correlation matrix between the outputs of two identical networks fed with distorted versions of a sample, and making it as close to the identity matrix as possible. This causes the embedding vectors of distorted versions of a sample to be similar, while minimizing the redundancy between the components of these vectors. The method is called Barlow Twins, owing to neuroscientist H. Barlow's redundancy-reduction principle applied to a pair of identical networks. Barlow Twins does not require large batches nor asymmetry between the network twins such as a predictor network, gradient stopping, or a moving average on the weight updates. Intriguingly it benefits from very high-dimensional output vectors. Barlow Twins outperforms previous methods on ImageNet for semi-supervised classification in the low-data regime, and is on par with current state of the art for ImageNet classification with a linear classifier head, and for transfer tasks of classification and object detection.
△ Less
Submitted 14 June, 2021; v1 submitted 4 March, 2021;
originally announced March 2021.
-
Implicit Rank-Minimizing Autoencoder
Authors:
Li Jing,
Jure Zbontar,
Yann LeCun
Abstract:
An important component of autoencoders is the method by which the information capacity of the latent representation is minimized or limited. In this work, the rank of the covariance matrix of the codes is implicitly minimized by relying on the fact that gradient descent learning in multi-layer linear networks leads to minimum-rank solutions. By inserting a number of extra linear layers between the…
▽ More
An important component of autoencoders is the method by which the information capacity of the latent representation is minimized or limited. In this work, the rank of the covariance matrix of the codes is implicitly minimized by relying on the fact that gradient descent learning in multi-layer linear networks leads to minimum-rank solutions. By inserting a number of extra linear layers between the encoder and the decoder, the system spontaneously learns representations with a low effective dimension. The model, dubbed Implicit Rank-Minimizing Autoencoder (IRMAE), is simple, deterministic, and learns compact latent spaces. We demonstrate the validity of the method on several image generation and representation learning tasks.
△ Less
Submitted 14 October, 2020; v1 submitted 1 October, 2020;
originally announced October 2020.
-
End-to-End Variational Networks for Accelerated MRI Reconstruction
Authors:
Anuroop Sriram,
Jure Zbontar,
Tullie Murrell,
Aaron Defazio,
C. Lawrence Zitnick,
Nafissa Yakubova,
Florian Knoll,
Patricia Johnson
Abstract:
The slow acquisition speed of magnetic resonance imaging (MRI) has led to the development of two complementary methods: acquiring multiple views of the anatomy simultaneously (parallel imaging) and acquiring fewer samples than necessary for traditional signal processing methods (compressed sensing). While the combination of these methods has the potential to allow much faster scan times, reconstru…
▽ More
The slow acquisition speed of magnetic resonance imaging (MRI) has led to the development of two complementary methods: acquiring multiple views of the anatomy simultaneously (parallel imaging) and acquiring fewer samples than necessary for traditional signal processing methods (compressed sensing). While the combination of these methods has the potential to allow much faster scan times, reconstruction from such undersampled multi-coil data has remained an open problem. In this paper, we present a new approach to this problem that extends previously proposed variational methods by learning fully end-to-end. Our method obtains new state-of-the-art results on the fastMRI dataset for both brain and knee MRIs.
△ Less
Submitted 15 April, 2020; v1 submitted 14 April, 2020;
originally announced April 2020.
-
Advancing machine learning for MR image reconstruction with an open competition: Overview of the 2019 fastMRI challenge
Authors:
Florian Knoll,
Tullie Murrell,
Anuroop Sriram,
Nafissa Yakubova,
Jure Zbontar,
Michael Rabbat,
Aaron Defazio,
Matthew J. Muckley,
Daniel K. Sodickson,
C. Lawrence Zitnick,
Michael P. Recht
Abstract:
Purpose: To advance research in the field of machine learning for MR image reconstruction with an open challenge. Methods: We provided participants with a dataset of raw k-space data from 1,594 consecutive clinical exams of the knee. The goal of the challenge was to reconstruct images from these data. In order to strike a balance between realistic data and a shallow learning curve for those not al…
▽ More
Purpose: To advance research in the field of machine learning for MR image reconstruction with an open challenge. Methods: We provided participants with a dataset of raw k-space data from 1,594 consecutive clinical exams of the knee. The goal of the challenge was to reconstruct images from these data. In order to strike a balance between realistic data and a shallow learning curve for those not already familiar with MR image reconstruction, we ran multiple tracks for multi-coil and single-coil data. We performed a two-stage evaluation based on quantitative image metrics followed by evaluation by a panel of radiologists. The challenge ran from June to December of 2019. Results: We received a total of 33 challenge submissions. All participants chose to submit results from supervised machine learning approaches. Conclusion: The challenge led to new developments in machine learning for image reconstruction, provided insight into the current state of the art in the field, and highlighted remaining hurdles for clinical adoption.
△ Less
Submitted 6 January, 2020;
originally announced January 2020.
-
GrappaNet: Combining Parallel Imaging with Deep Learning for Multi-Coil MRI Reconstruction
Authors:
Anuroop Sriram,
Jure Zbontar,
Tullie Murrell,
C. Lawrence Zitnick,
Aaron Defazio,
Daniel K. Sodickson
Abstract:
Magnetic Resonance Image (MRI) acquisition is an inherently slow process which has spurred the development of two different acceleration methods: acquiring multiple correlated samples simultaneously (parallel imaging) and acquiring fewer samples than necessary for traditional signal processing methods (compressed sensing). Both methods provide complementary approaches to accelerating the speed of…
▽ More
Magnetic Resonance Image (MRI) acquisition is an inherently slow process which has spurred the development of two different acceleration methods: acquiring multiple correlated samples simultaneously (parallel imaging) and acquiring fewer samples than necessary for traditional signal processing methods (compressed sensing). Both methods provide complementary approaches to accelerating the speed of MRI acquisition. In this paper, we present a novel method to integrate traditional parallel imaging methods into deep neural networks that is able to generate high quality reconstructions even for high acceleration factors. The proposed method, called GrappaNet, performs progressive reconstruction by first mapping the reconstruction problem to a simpler one that can be solved by a traditional parallel imaging methods using a neural network, followed by an application of a parallel imaging method, and finally fine-tuning the output with another neural network. The entire network can be trained end-to-end. We present experimental results on the recently released fastMRI dataset and show that GrappaNet can generate higher quality reconstructions than competing methods for both $4\times$ and $8\times$ acceleration.
△ Less
Submitted 30 March, 2020; v1 submitted 27 October, 2019;
originally announced October 2019.
-
fastMRI: An Open Dataset and Benchmarks for Accelerated MRI
Authors:
Jure Zbontar,
Florian Knoll,
Anuroop Sriram,
Tullie Murrell,
Zhengnan Huang,
Matthew J. Muckley,
Aaron Defazio,
Ruben Stern,
Patricia Johnson,
Mary Bruno,
Marc Parente,
Krzysztof J. Geras,
Joe Katsnelson,
Hersh Chandarana,
Zizhao Zhang,
Michal Drozdzal,
Adriana Romero,
Michael Rabbat,
Pascal Vincent,
Nafissa Yakubova,
James Pinkerton,
Duo Wang,
Erich Owens,
C. Lawrence Zitnick,
Michael P. Recht
, et al. (2 additional authors not shown)
Abstract:
Accelerating Magnetic Resonance Imaging (MRI) by taking fewer measurements has the potential to reduce medical costs, minimize stress to patients and make MRI possible in applications where it is currently prohibitively slow or expensive. We introduce the fastMRI dataset, a large-scale collection of both raw MR measurements and clinical MR images, that can be used for training and evaluation of ma…
▽ More
Accelerating Magnetic Resonance Imaging (MRI) by taking fewer measurements has the potential to reduce medical costs, minimize stress to patients and make MRI possible in applications where it is currently prohibitively slow or expensive. We introduce the fastMRI dataset, a large-scale collection of both raw MR measurements and clinical MR images, that can be used for training and evaluation of machine-learning approaches to MR image reconstruction. By introducing standardized evaluation criteria and a freely-accessible dataset, our goal is to help the community make rapid advances in the state of the art for MR image reconstruction. We also provide a self-contained introduction to MRI for machine learning researchers with no medical imaging background.
△ Less
Submitted 11 December, 2019; v1 submitted 21 November, 2018;
originally announced November 2018.
-
Simulating single-coil MRI from the responses of multiple coils
Authors:
Mark Tygert,
Jure Zbontar
Abstract:
We convert the information-rich measurements of parallel and phased-array MRI into noisier data that a corresponding single-coil scanner could have taken. Specifically, we replace the responses from multiple receivers with a linear combination that emulates the response from only a single, aggregate receiver, replete with the low signal-to-noise ratio and phase problems of any single one of the or…
▽ More
We convert the information-rich measurements of parallel and phased-array MRI into noisier data that a corresponding single-coil scanner could have taken. Specifically, we replace the responses from multiple receivers with a linear combination that emulates the response from only a single, aggregate receiver, replete with the low signal-to-noise ratio and phase problems of any single one of the original receivers (combining several receivers is necessary, however, since the original receivers usually have limited spatial sensitivity). This enables experimentation in the simpler context of a single-coil scanner prior to development of algorithms for the full complexity of multiple receiver coils.
△ Less
Submitted 27 May, 2019; v1 submitted 19 November, 2018;
originally announced November 2018.
-
Fast Incremental Learning for Off-Road Robot Navigation
Authors:
Artem Provodin,
Liila Torabi,
Beat Flepp,
Yann LeCun,
Michael Sergio,
L. D. Jackel,
Urs Muller,
Jure Zbontar
Abstract:
A promising approach to autonomous driving is machine learning. In such systems, training datasets are created that capture the sensory input to a vehicle as well as the desired response. A disadvantage of using a learned navigation system is that the learning process itself may require a huge number of training examples and a large amount of computing. To avoid the need to collect a large trainin…
▽ More
A promising approach to autonomous driving is machine learning. In such systems, training datasets are created that capture the sensory input to a vehicle as well as the desired response. A disadvantage of using a learned navigation system is that the learning process itself may require a huge number of training examples and a large amount of computing. To avoid the need to collect a large training set of driving examples, we describe a system that takes advantage of the huge number of training examples provided by ImageNet, but is able to adapt quickly using a small training set for the specific driving environment.
△ Less
Submitted 26 June, 2016;
originally announced June 2016.
-
Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches
Authors:
Jure Žbontar,
Yann LeCun
Abstract:
We present a method for extracting depth information from a rectified image pair. Our approach focuses on the first stage of many stereo algorithms: the matching cost computation. We approach the problem by learning a similarity measure on small image patches using a convolutional neural network. Training is carried out in a supervised manner by constructing a binary classification data set with e…
▽ More
We present a method for extracting depth information from a rectified image pair. Our approach focuses on the first stage of many stereo algorithms: the matching cost computation. We approach the problem by learning a similarity measure on small image patches using a convolutional neural network. Training is carried out in a supervised manner by constructing a binary classification data set with examples of similar and dissimilar pairs of patches. We examine two network architectures for this task: one tuned for speed, the other for accuracy. The output of the convolutional neural network is used to initialize the stereo matching cost. A series of post-processing steps follow: cross-based cost aggregation, semiglobal matching, a left-right consistency check, subpixel enhancement, a median filter, and a bilateral filter. We evaluate our method on the KITTI 2012, KITTI 2015, and Middlebury stereo data sets and show that it outperforms other approaches on all three data sets.
△ Less
Submitted 18 May, 2016; v1 submitted 20 October, 2015;
originally announced October 2015.
-
Computing the Stereo Matching Cost with a Convolutional Neural Network
Authors:
Jure Žbontar,
Yann LeCun
Abstract:
We present a method for extracting depth information from a rectified image pair. We train a convolutional neural network to predict how well two image patches match and use it to compute the stereo matching cost. The cost is refined by cross-based cost aggregation and semiglobal matching, followed by a left-right consistency check to eliminate errors in the occluded regions. Our stereo method ach…
▽ More
We present a method for extracting depth information from a rectified image pair. We train a convolutional neural network to predict how well two image patches match and use it to compute the stereo matching cost. The cost is refined by cross-based cost aggregation and semiglobal matching, followed by a left-right consistency check to eliminate errors in the occluded regions. Our stereo method achieves an error rate of 2.61 % on the KITTI stereo dataset and is currently (August 2014) the top performing method on this dataset.
△ Less
Submitted 20 October, 2015; v1 submitted 15 September, 2014;
originally announced September 2014.