We introduce PixelPlayer, a system that, by leveraging large amounts of unlabeled videos,
learns to locate image regions which produce sounds and separate the input sounds into a …

Multi-modal learning from video data has seen increased attention recently as it allows
training of semantically meaningful embeddings without human annotation, enabling tasks like …

Current methods for learning visually grounded language from videos often rely on text
annotation, such as human generated captions or machine generated automatic speech …

In this paper, we first extend the recent Masked Auto-Encoder (MAE) model from a single
modality to audio-visual multi-modalities. Subsequently, we propose the Contrastive Audio-…

Segmenting objects in images and separating sound sources in audio are challenging tasks,
in part because traditional approaches require large amounts of labeled data. In this paper …

Multimodal self-supervised learning is getting more and more attention as it allows not only
to train large networks without human supervision but also to search and retrieve data across …

Audio classification is an active research area with a wide range of applications. Over the
past decade, convolutional neural networks (CNNs) have been the de-facto standard building …

Recent advances in representation learning have demonstrated an ability to represent
information from different modalities such as video, text, and audio in a single high-level …

Recent models such as XLS-R and Whisper have made multilingual speech technologies
more accessible by pre-training on audio from around 100 spoken languages each. However, …

Audio-visual speech contains synchronized audio and visual information that provides cross-modal
supervision to learn representations for both automatic speech recognition (ASR) …