-
Self-Supervised MultiModal Versatile Networks
Authors:
Jean-Baptiste Alayrac,
Adrià Recasens,
Rosalia Schneider,
Relja Arandjelović,
Jason Ramapuram,
Jeffrey De Fauw,
Lucas Smaira,
Sander Dieleman,
Andrew Zisserman
Abstract:
Videos are a rich source of multi-modal supervision. In this work, we learn representations using self-supervision by leveraging three modalities naturally present in videos: visual, audio and language streams. To this end, we introduce the notion of a multimodal versatile network -- a network that can ingest multiple modalities and whose representations enable downstream tasks in multiple modalit…
▽ More
Videos are a rich source of multi-modal supervision. In this work, we learn representations using self-supervision by leveraging three modalities naturally present in videos: visual, audio and language streams. To this end, we introduce the notion of a multimodal versatile network -- a network that can ingest multiple modalities and whose representations enable downstream tasks in multiple modalities. In particular, we explore how best to combine the modalities, such that fine-grained representations of the visual and audio modalities can be maintained, whilst also integrating text into a common embedding. Driven by versatility, we also introduce a novel process of deflation, so that the networks can be effortlessly applied to the visual data in the form of video or a static image. We demonstrate how such networks trained on large collections of unlabelled video data can be applied on video, video-text, image and audio tasks. Equipped with these representations, we obtain state-of-the-art performance on multiple challenging benchmarks including UCF101, HMDB51, Kinetics600, AudioSet and ESC-50 when compared to previous self-supervised work. Our models are publicly available.
△ Less
Submitted 30 October, 2020; v1 submitted 29 June, 2020;
originally announced June 2020.
-
Data-Efficient Image Recognition with Contrastive Predictive Coding
Authors:
Olivier J. Hénaff,
Aravind Srinivas,
Jeffrey De Fauw,
Ali Razavi,
Carl Doersch,
S. M. Ali Eslami,
Aaron van den Oord
Abstract:
Human observers can learn to recognize new categories of images from a handful of examples, yet doing so with artificial ones remains an open challenge. We hypothesize that data-efficient recognition is enabled by representations which make the variability in natural signals more predictable. We therefore revisit and improve Contrastive Predictive Coding, an unsupervised objective for learning suc…
▽ More
Human observers can learn to recognize new categories of images from a handful of examples, yet doing so with artificial ones remains an open challenge. We hypothesize that data-efficient recognition is enabled by representations which make the variability in natural signals more predictable. We therefore revisit and improve Contrastive Predictive Coding, an unsupervised objective for learning such representations. This new implementation produces features which support state-of-the-art linear classification accuracy on the ImageNet dataset. When used as input for non-linear classification with deep neural networks, this representation allows us to use 2-5x less labels than classifiers trained directly on image pixels. Finally, this unsupervised representation substantially improves transfer learning to object detection on the PASCAL VOC dataset, surpassing fully supervised pre-trained ImageNet classifiers.
△ Less
Submitted 1 July, 2020; v1 submitted 22 May, 2019;
originally announced May 2019.
-
Hierarchical Autoregressive Image Models with Auxiliary Decoders
Authors:
Jeffrey De Fauw,
Sander Dieleman,
Karen Simonyan
Abstract:
Autoregressive generative models of images tend to be biased towards capturing local structure, and as a result they often produce samples which are lacking in terms of large-scale coherence. To address this, we propose two methods to learn discrete representations of images which abstract away local detail. We show that autoregressive models conditioned on these representations can produce high-f…
▽ More
Autoregressive generative models of images tend to be biased towards capturing local structure, and as a result they often produce samples which are lacking in terms of large-scale coherence. To address this, we propose two methods to learn discrete representations of images which abstract away local detail. We show that autoregressive models conditioned on these representations can produce high-fidelity reconstructions of images, and that we can train autoregressive priors on these representations that produce samples with large-scale coherence. We can recursively apply the learning procedure, yielding a hierarchy of progressively more abstract image representations. We train hierarchical class-conditional autoregressive models on the ImageNet dataset and demonstrate that they are able to generate realistic images at resolutions of 128$\times$128 and 256$\times$256 pixels. We also perform a human evaluation study comparing our models with both adversarial and likelihood-based state-of-the-art generative models.
△ Less
Submitted 8 October, 2019; v1 submitted 6 March, 2019;
originally announced March 2019.
-
Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapy
Authors:
Stanislav Nikolov,
Sam Blackwell,
Alexei Zverovitch,
Ruheena Mendes,
Michelle Livne,
Jeffrey De Fauw,
Yojan Patel,
Clemens Meyer,
Harry Askham,
Bernardino Romera-Paredes,
Christopher Kelly,
Alan Karthikesalingam,
Carlton Chu,
Dawn Carnell,
Cheng Boon,
Derek D'Souza,
Syed Ali Moinuddin,
Bethany Garie,
Yasmin McQuinlan,
Sarah Ireland,
Kiarna Hampton,
Krystle Fuller,
Hugh Montgomery,
Geraint Rees,
Mustafa Suleyman
, et al. (4 additional authors not shown)
Abstract:
Over half a million individuals are diagnosed with head and neck cancer each year worldwide. Radiotherapy is an important curative treatment for this disease, but it requires manual time consuming delineation of radio-sensitive organs at risk (OARs). This planning process can delay treatment, while also introducing inter-operator variability with resulting downstream radiation dose differences. Wh…
▽ More
Over half a million individuals are diagnosed with head and neck cancer each year worldwide. Radiotherapy is an important curative treatment for this disease, but it requires manual time consuming delineation of radio-sensitive organs at risk (OARs). This planning process can delay treatment, while also introducing inter-operator variability with resulting downstream radiation dose differences. While auto-segmentation algorithms offer a potentially time-saving solution, the challenges in defining, quantifying and achieving expert performance remain. Adopting a deep learning approach, we demonstrate a 3D U-Net architecture that achieves expert-level performance in delineating 21 distinct head and neck OARs commonly segmented in clinical practice. The model was trained on a dataset of 663 deidentified computed tomography (CT) scans acquired in routine clinical practice and with both segmentations taken from clinical practice and segmentations created by experienced radiographers as part of this research, all in accordance with consensus OAR definitions. We demonstrate the model's clinical applicability by assessing its performance on a test set of 21 CT scans from clinical practice, each with the 21 OARs segmented by two independent experts. We also introduce surface Dice similarity coefficient (surface DSC), a new metric for the comparison of organ delineation, to quantify deviation between OAR surface contours rather than volumes, better reflecting the clinical task of correcting errors in the automated organ segmentations. The model's generalisability is then demonstrated on two distinct open source datasets, reflecting different centres and countries to model training. With appropriate validation studies and regulatory approvals, this system could improve the efficiency, consistency, and safety of radiotherapy pathways.
△ Less
Submitted 13 January, 2021; v1 submitted 12 September, 2018;
originally announced September 2018.
-
A Probabilistic U-Net for Segmentation of Ambiguous Images
Authors:
Simon A. A. Kohl,
Bernardino Romera-Paredes,
Clemens Meyer,
Jeffrey De Fauw,
Joseph R. Ledsam,
Klaus H. Maier-Hein,
S. M. Ali Eslami,
Danilo Jimenez Rezende,
Olaf Ronneberger
Abstract:
Many real-world vision problems suffer from inherent ambiguities. In clinical applications for example, it might not be clear from a CT scan alone which particular region is cancer tissue. Therefore a group of graders typically produces a set of diverse but plausible segmentations. We consider the task of learning a distribution over segmentations given an input. To this end we propose a generativ…
▽ More
Many real-world vision problems suffer from inherent ambiguities. In clinical applications for example, it might not be clear from a CT scan alone which particular region is cancer tissue. Therefore a group of graders typically produces a set of diverse but plausible segmentations. We consider the task of learning a distribution over segmentations given an input. To this end we propose a generative segmentation model based on a combination of a U-Net with a conditional variational autoencoder that is capable of efficiently producing an unlimited number of plausible hypotheses. We show on a lung abnormalities segmentation task and on a Cityscapes segmentation task that our model reproduces the possible segmentation variants as well as the frequencies with which they occur, doing so significantly better than published approaches. These models could have a high impact in real-world applications, such as being used as clinical decision-making algorithms accounting for multiple plausible semantic segmentation hypotheses to provide possible diagnoses and recommend further actions to resolve the present ambiguities.
△ Less
Submitted 29 January, 2019; v1 submitted 13 June, 2018;
originally announced June 2018.
-
Exploiting Cyclic Symmetry in Convolutional Neural Networks
Authors:
Sander Dieleman,
Jeffrey De Fauw,
Koray Kavukcuoglu
Abstract:
Many classes of images exhibit rotational symmetry. Convolutional neural networks are sometimes trained using data augmentation to exploit this, but they are still required to learn the rotation equivariance properties from the data. Encoding these properties into the network architecture, as we are already used to doing for translation equivariance by using convolutional layers, could result in a…
▽ More
Many classes of images exhibit rotational symmetry. Convolutional neural networks are sometimes trained using data augmentation to exploit this, but they are still required to learn the rotation equivariance properties from the data. Encoding these properties into the network architecture, as we are already used to doing for translation equivariance by using convolutional layers, could result in a more efficient use of the parameter budget by relieving the model from learning them. We introduce four operations which can be inserted into neural network models as layers, and which can be combined to make these models partially equivariant to rotations. They also enable parameter sharing across different orientations. We evaluate the effect of these architectural modifications on three datasets which exhibit rotational symmetry and demonstrate improved performance with smaller models.
△ Less
Submitted 26 May, 2016; v1 submitted 8 February, 2016;
originally announced February 2016.
