Recent advances in representation learning have demonstrated an ability to represent
information from different modalities such as video, text, and audio in a single high-level
embedding vector. In this work we present a self-supervised learning framework that is able
to learn a representation that captures finer levels of granularity across different modalities
such as concepts or events represented by visual objects or spoken words. Our framework
relies on a discretized embedding space created via vector quantization that is shared …

In contrast to recent advances focusing on highlevel representation learning across
modalities, in this work we present a self-supervised learning framework that is able to learn
a representation that captures finer levels of granularity across different modalities such as
concepts or events represented by visual objects or spoken words. Our framework relies on
a discretized embedding space created via vector quantization that is shared across
different modalities. Beyond the shared embedding space, we propose a Cross-Modal Code …