-
Continuous diffusion for categorical data
Authors:
Sander Dieleman,
Laurent Sartran,
Arman Roshannai,
Nikolay Savinov,
Yaroslav Ganin,
Pierre H. Richemond,
Arnaud Doucet,
Robin Strudel,
Chris Dyer,
Conor Durkan,
Curtis Hawthorne,
Rémi Leblond,
Will Grathwohl,
Jonas Adler
Abstract:
Diffusion models have quickly become the go-to paradigm for generative modelling of perceptual signals (such as images and sound) through iterative refinement. Their success hinges on the fact that the underlying physical phenomena are continuous. For inherently discrete and categorical data such as language, various diffusion-inspired alternatives have been proposed. However, the continuous natur…
▽ More
Diffusion models have quickly become the go-to paradigm for generative modelling of perceptual signals (such as images and sound) through iterative refinement. Their success hinges on the fact that the underlying physical phenomena are continuous. For inherently discrete and categorical data such as language, various diffusion-inspired alternatives have been proposed. However, the continuous nature of diffusion models conveys many benefits, and in this work we endeavour to preserve it. We propose CDCD, a framework for modelling categorical data with diffusion models that are continuous both in time and input space. We demonstrate its efficacy on several language modelling tasks.
△ Less
Submitted 15 December, 2022; v1 submitted 28 November, 2022;
originally announced November 2022.
-
Multi-Task Dynamical Systems
Authors:
Alex Bird,
Christopher K. I. Williams,
Christopher Hawthorne
Abstract:
Time series datasets are often composed of a variety of sequences from the same domain, but from different entities, such as individuals, products, or organizations. We are interested in how time series models can be specialized to individual sequences (capturing the specific characteristics) while still retaining statistical power by sharing commonalities across the sequences. This paper describe…
▽ More
Time series datasets are often composed of a variety of sequences from the same domain, but from different entities, such as individuals, products, or organizations. We are interested in how time series models can be specialized to individual sequences (capturing the specific characteristics) while still retaining statistical power by sharing commonalities across the sequences. This paper describes the multi-task dynamical system (MTDS); a general methodology for extending multi-task learning (MTL) to time series models. Our approach endows dynamical systems with a set of hierarchical latent variables which can modulate all model parameters. To our knowledge, this is a novel development of MTL, and applies to time series both with and without control inputs. We apply the MTDS to motion-capture data of people walking in various styles using a multi-task recurrent neural network (RNN), and to patient drug-response data using a multi-task pharmacodynamic model.
△ Less
Submitted 8 October, 2022;
originally announced October 2022.
-
The Chamber Ensemble Generator: Limitless High-Quality MIR Data via Generative Modeling
Authors:
Yusong Wu,
Josh Gardner,
Ethan Manilow,
Ian Simon,
Curtis Hawthorne,
Jesse Engel
Abstract:
Data is the lifeblood of modern machine learning systems, including for those in Music Information Retrieval (MIR). However, MIR has long been mired by small datasets and unreliable labels. In this work, we propose to break this bottleneck using generative modeling. By pipelining a generative model of notes (Coconet trained on Bach Chorales) with a structured synthesis model of chamber ensembles (…
▽ More
Data is the lifeblood of modern machine learning systems, including for those in Music Information Retrieval (MIR). However, MIR has long been mired by small datasets and unreliable labels. In this work, we propose to break this bottleneck using generative modeling. By pipelining a generative model of notes (Coconet trained on Bach Chorales) with a structured synthesis model of chamber ensembles (MIDI-DDSP trained on URMP), we demonstrate a system capable of producing unlimited amounts of realistic chorale music with rich annotations including mixes, stems, MIDI, note-level performance attributes (staccato, vibrato, etc.), and even fine-grained synthesis parameters (pitch, amplitude, etc.). We call this system the Chamber Ensemble Generator (CEG), and use it to generate a large dataset of chorales from four different chamber ensembles (CocoChorales). We demonstrate that data generated using our approach improves state-of-the-art models for music transcription and source separation, and we release both the system and the dataset as an open-source foundation for future work in the MIR community.
△ Less
Submitted 28 September, 2022;
originally announced September 2022.
-
Multi-instrument Music Synthesis with Spectrogram Diffusion
Authors:
Curtis Hawthorne,
Ian Simon,
Adam Roberts,
Neil Zeghidour,
Josh Gardner,
Ethan Manilow,
Jesse Engel
Abstract:
An ideal music synthesizer should be both interactive and expressive, generating high-fidelity audio in realtime for arbitrary combinations of instruments and notes. Recent neural synthesizers have exhibited a tradeoff between domain-specific models that offer detailed control of only specific instruments, or raw waveform models that can train on any music but with minimal control and slow generat…
▽ More
An ideal music synthesizer should be both interactive and expressive, generating high-fidelity audio in realtime for arbitrary combinations of instruments and notes. Recent neural synthesizers have exhibited a tradeoff between domain-specific models that offer detailed control of only specific instruments, or raw waveform models that can train on any music but with minimal control and slow generation. In this work, we focus on a middle ground of neural synthesizers that can generate audio from MIDI sequences with arbitrary combinations of instruments in realtime. This enables training on a wide range of transcription datasets with a single model, which in turn offers note-level control of composition and instrumentation across a wide range of instruments. We use a simple two-stage process: MIDI to spectrograms with an encoder-decoder Transformer, then spectrograms to audio with a generative adversarial network (GAN) spectrogram inverter. We compare training the decoder as an autoregressive model and as a Denoising Diffusion Probabilistic Model (DDPM) and find that the DDPM approach is superior both qualitatively and as measured by audio reconstruction and Fréchet distance metrics. Given the interactivity and generality of this approach, we find this to be a promising first step towards interactive and expressive neural synthesis for arbitrary combinations of instruments and notes.
△ Less
Submitted 12 December, 2022; v1 submitted 10 June, 2022;
originally announced June 2022.
-
Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$
Authors:
Adam Roberts,
Hyung Won Chung,
Anselm Levskaya,
Gaurav Mishra,
James Bradbury,
Daniel Andor,
Sharan Narang,
Brian Lester,
Colin Gaffney,
Afroz Mohiuddin,
Curtis Hawthorne,
Aitor Lewkowycz,
Alex Salcianu,
Marc van Zee,
Jacob Austin,
Sebastian Goodman,
Livio Baldini Soares,
Haitang Hu,
Sasha Tsvyashchenko,
Aakanksha Chowdhery,
Jasmijn Bastings,
Jannis Bulian,
Xavier Garcia,
Jianmo Ni,
Andrew Chen
, et al. (18 additional authors not shown)
Abstract:
Recent neural network-based language models have benefited greatly from scaling up the size of training datasets and the number of parameters in the models themselves. Scaling can be complicated due to various factors including the need to distribute computation on supercomputer clusters (e.g., TPUs), prevent bottlenecks when infeeding data, and ensure reproducible results. In this work, we presen…
▽ More
Recent neural network-based language models have benefited greatly from scaling up the size of training datasets and the number of parameters in the models themselves. Scaling can be complicated due to various factors including the need to distribute computation on supercomputer clusters (e.g., TPUs), prevent bottlenecks when infeeding data, and ensure reproducible results. In this work, we present two software libraries that ease these issues: $\texttt{t5x}$ simplifies the process of building and training large language models at scale while maintaining ease of use, and $\texttt{seqio}$ provides a task-based API for simple creation of fast and reproducible training data and evaluation pipelines. These open-source libraries have been used to train models with hundreds of billions of parameters on datasets with multiple terabytes of training data.
Along with the libraries, we release configurations and instructions for T5-like encoder-decoder models as well as GPT-like decoder-only architectures.
$\texttt{t5x}$ and $\texttt{seqio}$ are open source and available at https://github.com/google-research/t5x and https://github.com/google/seqio, respectively.
△ Less
Submitted 31 March, 2022;
originally announced March 2022.
-
Improving Source Separation by Explicitly Modeling Dependencies Between Sources
Authors:
Ethan Manilow,
Curtis Hawthorne,
Cheng-Zhi Anna Huang,
Bryan Pardo,
Jesse Engel
Abstract:
We propose a new method for training a supervised source separation system that aims to learn the interdependent relationships between all combinations of sources in a mixture. Rather than independently estimating each source from a mix, we reframe the source separation problem as an Orderless Neural Autoregressive Density Estimator (NADE), and estimate each source from both the mix and a random s…
▽ More
We propose a new method for training a supervised source separation system that aims to learn the interdependent relationships between all combinations of sources in a mixture. Rather than independently estimating each source from a mix, we reframe the source separation problem as an Orderless Neural Autoregressive Density Estimator (NADE), and estimate each source from both the mix and a random subset of the other sources. We adapt a standard source separation architecture, Demucs, with additional inputs for each individual source, in addition to the input mixture. We randomly mask these input sources during training so that the network learns the conditional dependencies between the sources. By pairing this training method with a block Gibbs sampling procedure at inference time, we demonstrate that the network can iteratively improve its separation performance by conditioning a source estimate on its earlier source estimates. Experiments on two source separation datasets show that training a Demucs model with an Orderless NADE approach and using Gibbs sampling (up to 512 steps) at inference time strongly outperforms a Demucs baseline that uses a standard regression loss and direct (one step) estimation of sources.
△ Less
Submitted 28 March, 2022;
originally announced March 2022.
-
General-purpose, long-context autoregressive modeling with Perceiver AR
Authors:
Curtis Hawthorne,
Andrew Jaegle,
Cătălina Cangea,
Sebastian Borgeaud,
Charlie Nash,
Mateusz Malinowski,
Sander Dieleman,
Oriol Vinyals,
Matthew Botvinick,
Ian Simon,
Hannah Sheahan,
Neil Zeghidour,
Jean-Baptiste Alayrac,
João Carreira,
Jesse Engel
Abstract:
Real-world data is high-dimensional: a book, image, or musical performance can easily contain hundreds of thousands of elements even after compression. However, the most commonly used autoregressive models, Transformers, are prohibitively expensive to scale to the number of inputs and layers needed to capture this long-range structure. We develop Perceiver AR, an autoregressive, modality-agnostic…
▽ More
Real-world data is high-dimensional: a book, image, or musical performance can easily contain hundreds of thousands of elements even after compression. However, the most commonly used autoregressive models, Transformers, are prohibitively expensive to scale to the number of inputs and layers needed to capture this long-range structure. We develop Perceiver AR, an autoregressive, modality-agnostic architecture which uses cross-attention to map long-range inputs to a small number of latents while also maintaining end-to-end causal masking. Perceiver AR can directly attend to over a hundred thousand tokens, enabling practical long-context density estimation without the need for hand-crafted sparsity patterns or memory mechanisms. When trained on images or music, Perceiver AR generates outputs with clear long-term coherence and structure. Our architecture also obtains state-of-the-art likelihood on long-sequence benchmarks, including 64 x 64 ImageNet images and PG-19 books.
△ Less
Submitted 14 June, 2022; v1 submitted 15 February, 2022;
originally announced February 2022.
-
MT3: Multi-Task Multitrack Music Transcription
Authors:
Josh Gardner,
Ian Simon,
Ethan Manilow,
Curtis Hawthorne,
Jesse Engel
Abstract:
Automatic Music Transcription (AMT), inferring musical notes from raw audio, is a challenging task at the core of music understanding. Unlike Automatic Speech Recognition (ASR), which typically focuses on the words of a single speaker, AMT often requires transcribing multiple instruments simultaneously, all while preserving fine-scale pitch and timing information. Further, many AMT datasets are "l…
▽ More
Automatic Music Transcription (AMT), inferring musical notes from raw audio, is a challenging task at the core of music understanding. Unlike Automatic Speech Recognition (ASR), which typically focuses on the words of a single speaker, AMT often requires transcribing multiple instruments simultaneously, all while preserving fine-scale pitch and timing information. Further, many AMT datasets are "low-resource", as even expert musicians find music transcription difficult and time-consuming. Thus, prior work has focused on task-specific architectures, tailored to the individual instruments of each task. In this work, motivated by the promising results of sequence-to-sequence transfer learning for low-resource Natural Language Processing (NLP), we demonstrate that a general-purpose Transformer model can perform multi-task AMT, jointly transcribing arbitrary combinations of musical instruments across several transcription datasets. We show this unified training framework achieves high-quality transcription results across a range of datasets, dramatically improving performance for low-resource instruments (such as guitar), while preserving strong performance for abundant instruments (such as piano). Finally, by expanding the scope of AMT, we expose the need for more consistent evaluation metrics and better dataset alignment, and provide a strong baseline for this new direction of multi-task AMT.
△ Less
Submitted 15 March, 2022; v1 submitted 4 November, 2021;
originally announced November 2021.
-
Sequence-to-Sequence Piano Transcription with Transformers
Authors:
Curtis Hawthorne,
Ian Simon,
Rigel Swavely,
Ethan Manilow,
Jesse Engel
Abstract:
Automatic Music Transcription has seen significant progress in recent years by training custom deep neural networks on large datasets. However, these models have required extensive domain-specific design of network architectures, input/output representations, and complex decoding schemes. In this work, we show that equivalent performance can be achieved using a generic encoder-decoder Transformer…
▽ More
Automatic Music Transcription has seen significant progress in recent years by training custom deep neural networks on large datasets. However, these models have required extensive domain-specific design of network architectures, input/output representations, and complex decoding schemes. In this work, we show that equivalent performance can be achieved using a generic encoder-decoder Transformer with standard decoding methods. We demonstrate that the model can learn to translate spectrogram inputs directly to MIDI-like output events for several transcription tasks. This sequence-to-sequence approach simplifies transcription by jointly modeling audio features and language-like output dependencies, thus removing the need for task-specific architectures. These results point toward possibilities for creating new Music Information Retrieval models by focusing on dataset creation and labeling rather than custom model design.
△ Less
Submitted 19 July, 2021;
originally announced July 2021.
-
Symbolic Music Generation with Diffusion Models
Authors:
Gautam Mittal,
Jesse Engel,
Curtis Hawthorne,
Ian Simon
Abstract:
Score-based generative models and diffusion probabilistic models have been successful at generating high-quality samples in continuous domains such as images and audio. However, due to their Langevin-inspired sampling mechanisms, their application to discrete and sequential data has been limited. In this work, we present a technique for training diffusion models on sequential data by parameterizin…
▽ More
Score-based generative models and diffusion probabilistic models have been successful at generating high-quality samples in continuous domains such as images and audio. However, due to their Langevin-inspired sampling mechanisms, their application to discrete and sequential data has been limited. In this work, we present a technique for training diffusion models on sequential data by parameterizing the discrete domain in the continuous latent space of a pre-trained variational autoencoder. Our method is non-autoregressive and learns to generate sequences of latent embeddings through the reverse process and offers parallel generation with a constant number of iterative refinement steps. We apply this technique to modeling symbolic music and show strong unconditional generation and post-hoc conditional infilling results compared to autoregressive language models operating over the same continuous embeddings.
△ Less
Submitted 25 November, 2021; v1 submitted 30 March, 2021;
originally announced March 2021.
-
Contributions to the theory of $F$-automatic sets
Authors:
Christopher Hawthorne
Abstract:
Fix an abelian group $Γ$ and an injective endomorphism $F \colon Γ\to Γ$. Improving on the results of Bell and Moosa, new characterizations are here obtained for the existence of spanning sets, $F$-automaticity, and $F$-sparsity. The model theoretic status of these sets is also investigated, culminating with a combinatorial description of the $F$-sparse sets that are stable in $(Γ, +)$, and a proo…
▽ More
Fix an abelian group $Γ$ and an injective endomorphism $F \colon Γ\to Γ$. Improving on the results of Bell and Moosa, new characterizations are here obtained for the existence of spanning sets, $F$-automaticity, and $F$-sparsity. The model theoretic status of these sets is also investigated, culminating with a combinatorial description of the $F$-sparse sets that are stable in $(Γ, +)$, and a proof that the expansion of $(Γ, +)$ by any $F$-sparse set is NIP. These methods are also used to show for prime $p\ge 7$ that the expansion of $(\mathbb{F}_p[t], +)$ by multiplication restricted to $t^\mathbb{N}$ is NIP.
△ Less
Submitted 24 November, 2020;
originally announced November 2020.
-
Automata and tame expansions of $(\mathbb{Z},+)$
Authors:
Christopher D. C. Hawthorne
Abstract:
The problem of characterizing which automatic sets of integers are stable is here solved. Given a positive integer $d$ and a subset $A\subseteq \mathbb{Z}$ whose set of representations base $d$ is recognized by a finite automaton, a necessary condition is found for $x+y\in A$ to be a stable formula in $\operatorname{Th}(\mathbb{Z},+,A)$. Combined with a theorem of Moosa and Scanlon this gives a co…
▽ More
The problem of characterizing which automatic sets of integers are stable is here solved. Given a positive integer $d$ and a subset $A\subseteq \mathbb{Z}$ whose set of representations base $d$ is recognized by a finite automaton, a necessary condition is found for $x+y\in A$ to be a stable formula in $\operatorname{Th}(\mathbb{Z},+,A)$. Combined with a theorem of Moosa and Scanlon this gives a combinatorial characterization of the $d$-automatic $A\subseteq \mathbb{Z}$ such that $(\mathbb{Z},+,A)$ is stable. This characterization is in terms of what were called "$F$-sets" by Moosa and Scanlon and "elementary $p$-nested sets" by Derksen. Automata-theoretic methods are also used to produce some NIP expansions of $(\mathbb{Z},+)$, in particular the expansion by the monoid $(d^\mathbb{N},\times )$.
△ Less
Submitted 7 October, 2020; v1 submitted 30 June, 2020;
originally announced July 2020.
-
Improving Perceptual Quality of Drum Transcription with the Expanded Groove MIDI Dataset
Authors:
Lee Callender,
Curtis Hawthorne,
Jesse Engel
Abstract:
We introduce the Expanded Groove MIDI dataset (E-GMD), an automatic drum transcription (ADT) dataset that contains 444 hours of audio from 43 drum kits, making it an order of magnitude larger than similar datasets, and the first with human-performed velocity annotations. We use E-GMD to optimize classifiers for use in downstream generation by predicting expressive dynamics (velocity) and show with…
▽ More
We introduce the Expanded Groove MIDI dataset (E-GMD), an automatic drum transcription (ADT) dataset that contains 444 hours of audio from 43 drum kits, making it an order of magnitude larger than similar datasets, and the first with human-performed velocity annotations. We use E-GMD to optimize classifiers for use in downstream generation by predicting expressive dynamics (velocity) and show with listening tests that they produce outputs with improved perceptual quality, despite similar results on classification metrics. Via the listening tests, we argue that standard classifier metrics, such as accuracy and F-measure score, are insufficient proxies of performance in downstream tasks because they do not fully align with the perceptual quality of generated outputs.
△ Less
Submitted 1 December, 2020; v1 submitted 31 March, 2020;
originally announced April 2020.
-
Encoding Musical Style with Transformer Autoencoders
Authors:
Kristy Choi,
Curtis Hawthorne,
Ian Simon,
Monica Dinculescu,
Jesse Engel
Abstract:
We consider the problem of learning high-level controls over the global structure of generated sequences, particularly in the context of symbolic music generation with complex language models. In this work, we present the Transformer autoencoder, which aggregates encodings of the input data across time to obtain a global representation of style from a given performance. We show it is possible to c…
▽ More
We consider the problem of learning high-level controls over the global structure of generated sequences, particularly in the context of symbolic music generation with complex language models. In this work, we present the Transformer autoencoder, which aggregates encodings of the input data across time to obtain a global representation of style from a given performance. We show it is possible to combine this global representation with other temporally distributed embeddings, enabling improved control over the separate aspects of performance style and melody. Empirically, we demonstrate the effectiveness of our method on various music generation tasks on the MAESTRO dataset and a YouTube dataset with 10,000+ hours of piano performances, where we achieve improvements in terms of log-likelihood and mean listening scores as compared to baselines.
△ Less
Submitted 30 June, 2020; v1 submitted 10 December, 2019;
originally announced December 2019.
-
The Bach Doodle: Approachable music composition with machine learning at scale
Authors:
Cheng-Zhi Anna Huang,
Curtis Hawthorne,
Adam Roberts,
Monica Dinculescu,
James Wexler,
Leon Hong,
Jacob Howcroft
Abstract:
To make music composition more approachable, we designed the first AI-powered Google Doodle, the Bach Doodle, where users can create their own melody and have it harmonized by a machine learning model Coconet (Huang et al., 2017) in the style of Bach. For users to input melodies, we designed a simplified sheet-music based interface. To support an interactive experience at scale, we re-implemented…
▽ More
To make music composition more approachable, we designed the first AI-powered Google Doodle, the Bach Doodle, where users can create their own melody and have it harmonized by a machine learning model Coconet (Huang et al., 2017) in the style of Bach. For users to input melodies, we designed a simplified sheet-music based interface. To support an interactive experience at scale, we re-implemented Coconet in TensorFlow.js (Smilkov et al., 2019) to run in the browser and reduced its runtime from 40s to 2s by adopting dilated depth-wise separable convolutions and fusing operations. We also reduced the model download size to approximately 400KB through post-training weight quantization. We calibrated a speed test based on partial model evaluation time to determine if the harmonization request should be performed locally or sent to remote TPU servers. In three days, people spent 350 years worth of time playing with the Bach Doodle, and Coconet received more than 55 million queries. Users could choose to rate their compositions and contribute them to a public dataset, which we are releasing with this paper. We hope that the community finds this dataset useful for applications ranging from ethnomusicological studies, to music education, to improving machine learning models.
△ Less
Submitted 14 July, 2019;
originally announced July 2019.
-
Multi-Task Time Series Analysis applied to Drug Response Modelling
Authors:
Alex Bird,
Christopher K. I. Williams,
Christopher Hawthorne
Abstract:
Time series models such as dynamical systems are frequently fitted to a cohort of data, ignoring variation between individual entities such as patients. In this paper we show how these models can be personalised to an individual level while retaining statistical power, via use of multi-task learning (MTL). To our knowledge this is a novel development of MTL which applies to time series both with a…
▽ More
Time series models such as dynamical systems are frequently fitted to a cohort of data, ignoring variation between individual entities such as patients. In this paper we show how these models can be personalised to an individual level while retaining statistical power, via use of multi-task learning (MTL). To our knowledge this is a novel development of MTL which applies to time series both with and without control inputs. The modelling framework is demonstrated on a physiological drug response problem which results in improved predictive accuracy and uncertainty estimation over existing state-of-the-art models.
△ Less
Submitted 21 March, 2019;
originally announced March 2019.
-
Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset
Authors:
Curtis Hawthorne,
Andriy Stasyuk,
Adam Roberts,
Ian Simon,
Cheng-Zhi Anna Huang,
Sander Dieleman,
Erich Elsen,
Jesse Engel,
Douglas Eck
Abstract:
Generating musical audio directly with neural networks is notoriously difficult because it requires coherently modeling structure at many different timescales. Fortunately, most music is also highly structured and can be represented as discrete note events played on musical instruments. Herein, we show that by using notes as an intermediate representation, we can train a suite of models capable of…
▽ More
Generating musical audio directly with neural networks is notoriously difficult because it requires coherently modeling structure at many different timescales. Fortunately, most music is also highly structured and can be represented as discrete note events played on musical instruments. Herein, we show that by using notes as an intermediate representation, we can train a suite of models capable of transcribing, composing, and synthesizing audio waveforms with coherent musical structure on timescales spanning six orders of magnitude (~0.1 ms to ~100 s), a process we call Wave2Midi2Wave. This large advance in the state of the art is enabled by our release of the new MAESTRO (MIDI and Audio Edited for Synchronous TRacks and Organization) dataset, composed of over 172 hours of virtuosic piano performances captured with fine alignment (~3 ms) between note labels and audio waveforms. The networks and the dataset together present a promising approach toward creating new expressive and interpretable neural models of music.
△ Less
Submitted 17 January, 2019; v1 submitted 29 October, 2018;
originally announced October 2018.
-
Music Transformer
Authors:
Cheng-Zhi Anna Huang,
Ashish Vaswani,
Jakob Uszkoreit,
Noam Shazeer,
Ian Simon,
Curtis Hawthorne,
Andrew M. Dai,
Matthew D. Hoffman,
Monica Dinculescu,
Douglas Eck
Abstract:
Music relies heavily on repetition to build structure and meaning. Self-reference occurs on multiple timescales, from motifs to phrases to reusing of entire sections of music, such as in pieces with ABA structure. The Transformer (Vaswani et al., 2017), a sequence model based on self-attention, has achieved compelling results in many generation tasks that require maintaining long-range coherence.…
▽ More
Music relies heavily on repetition to build structure and meaning. Self-reference occurs on multiple timescales, from motifs to phrases to reusing of entire sections of music, such as in pieces with ABA structure. The Transformer (Vaswani et al., 2017), a sequence model based on self-attention, has achieved compelling results in many generation tasks that require maintaining long-range coherence. This suggests that self-attention might also be well-suited to modeling music. In musical composition and performance, however, relative timing is critically important. Existing approaches for representing relative positional information in the Transformer modulate attention based on pairwise distance (Shaw et al., 2018). This is impractical for long sequences such as musical compositions since their memory complexity for intermediate relative information is quadratic in the sequence length. We propose an algorithm that reduces their intermediate memory requirement to linear in the sequence length. This enables us to demonstrate that a Transformer with our modified relative attention mechanism can generate minute-long compositions (thousands of steps, four times the length modeled in Oore et al., 2018) with compelling structure, generate continuations that coherently elaborate on a given motif, and in a seq2seq setup generate accompaniments conditioned on melodies. We evaluate the Transformer with our relative attention mechanism on two datasets, JSB Chorales and Piano-e-Competition, and obtain state-of-the-art results on the latter.
△ Less
Submitted 12 December, 2018; v1 submitted 12 September, 2018;
originally announced September 2018.
-
Learning a Latent Space of Multitrack Measures
Authors:
Ian Simon,
Adam Roberts,
Colin Raffel,
Jesse Engel,
Curtis Hawthorne,
Douglas Eck
Abstract:
Discovering and exploring the underlying structure of multi-instrumental music using learning-based approaches remains an open problem. We extend the recent MusicVAE model to represent multitrack polyphonic measures as vectors in a latent space. Our approach enables several useful operations such as generating plausible measures from scratch, interpolating between measures in a musically meaningfu…
▽ More
Discovering and exploring the underlying structure of multi-instrumental music using learning-based approaches remains an open problem. We extend the recent MusicVAE model to represent multitrack polyphonic measures as vectors in a latent space. Our approach enables several useful operations such as generating plausible measures from scratch, interpolating between measures in a musically meaningful way, and manipulating specific musical attributes. We also introduce chord conditioning, which allows all of these operations to be performed while keeping harmony fixed, and allows chords to be changed while maintaining musical "style". By generating a sequence of measures over a predefined chord progression, our model can produce music with convincing long-term structure. We demonstrate that our latent space model makes it possible to intuitively control and generate musical sequences with rich instrumentation (see https://goo.gl/s2N7dV for generated audio).
△ Less
Submitted 1 June, 2018;
originally announced June 2018.
-
Remarks on recognizable subsets and local rank
Authors:
Christopher D. C. Hawthorne
Abstract:
Given a monoid $(M,\varepsilon,\cdot )$ it is shown that a subset $A\subseteq M$ is recognizable in the sense of automata theory if and only if the $\varphi $-rank of $x=x$ is zero in the first-order theory $\operatorname{Th}(M,\varepsilon ,\cdot ,A)$, where $\varphi (x;u)$ is the formula $xu\in A$. In the case where $M$ is a finitely generated free monoid on a finite alphabet $Σ$, this gives a mo…
▽ More
Given a monoid $(M,\varepsilon,\cdot )$ it is shown that a subset $A\subseteq M$ is recognizable in the sense of automata theory if and only if the $\varphi $-rank of $x=x$ is zero in the first-order theory $\operatorname{Th}(M,\varepsilon ,\cdot ,A)$, where $\varphi (x;u)$ is the formula $xu\in A$. In the case where $M$ is a finitely generated free monoid on a finite alphabet $Σ$, this gives a model-theoretic characterization of the regular languages over $Σ$. If $A$ is a regular language over $Σ$ then the $\varphi $-multiplicity of $x=x$ is the state complexity of $A$. Similar results holds for $\varphi' (x;u,v)$ given by $uxv\in A$, with the $\varphi' $-multiplicity now equal to the size of the syntactic monoid of $A$.
△ Less
Submitted 7 June, 2019; v1 submitted 19 March, 2018;
originally announced March 2018.
-
A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
Authors:
Adam Roberts,
Jesse Engel,
Colin Raffel,
Curtis Hawthorne,
Douglas Eck
Abstract:
The Variational Autoencoder (VAE) has proven to be an effective model for producing semantically meaningful latent representations for natural data. However, it has thus far seen limited application to sequential data, and, as we demonstrate, existing recurrent VAE models have difficulty modeling sequences with long-term structure. To address this issue, we propose the use of a hierarchical decode…
▽ More
The Variational Autoencoder (VAE) has proven to be an effective model for producing semantically meaningful latent representations for natural data. However, it has thus far seen limited application to sequential data, and, as we demonstrate, existing recurrent VAE models have difficulty modeling sequences with long-term structure. To address this issue, we propose the use of a hierarchical decoder, which first outputs embeddings for subsequences of the input and then uses these embeddings to generate each subsequence independently. This structure encourages the model to utilize its latent code, thereby avoiding the "posterior collapse" problem, which remains an issue for recurrent VAEs. We apply this architecture to modeling sequences of musical notes and find that it exhibits dramatically better sampling, interpolation, and reconstruction performance than a "flat" baseline model. An implementation of our "MusicVAE" is available online at http://g.co/magenta/musicvae-code.
△ Less
Submitted 11 November, 2019; v1 submitted 13 March, 2018;
originally announced March 2018.
-
Onsets and Frames: Dual-Objective Piano Transcription
Authors:
Curtis Hawthorne,
Erich Elsen,
Jialin Song,
Adam Roberts,
Ian Simon,
Colin Raffel,
Jesse Engel,
Sageev Oore,
Douglas Eck
Abstract:
We advance the state of the art in polyphonic piano music transcription by using a deep convolutional and recurrent neural network which is trained to jointly predict onsets and frames. Our model predicts pitch onset events and then uses those predictions to condition framewise pitch predictions. During inference, we restrict the predictions from the framewise detector by not allowing a new note t…
▽ More
We advance the state of the art in polyphonic piano music transcription by using a deep convolutional and recurrent neural network which is trained to jointly predict onsets and frames. Our model predicts pitch onset events and then uses those predictions to condition framewise pitch predictions. During inference, we restrict the predictions from the framewise detector by not allowing a new note to start unless the onset detector also agrees that an onset for that pitch is present in the frame. We focus on improving onsets and offsets together instead of either in isolation as we believe this correlates better with human musical perception. Our approach results in over a 100% relative improvement in note F1 score (with offsets) on the MAPS dataset. Furthermore, we extend the model to predict relative velocities of normalized audio which results in more natural-sounding transcriptions.
△ Less
Submitted 5 June, 2018; v1 submitted 30 October, 2017;
originally announced October 2017.
-
Input-Output Non-Linear Dynamical Systems applied to Physiological Condition Monitoring
Authors:
Konstantinos Georgatzis,
Christopher K. I. Williams,
Christopher Hawthorne
Abstract:
We present a non-linear dynamical system for modelling the effect of drug infusions on the vital signs of patients admitted in Intensive Care Units (ICUs). More specifically we are interested in modelling the effect of a widely used anaesthetic drug (Propofol) on a patient's monitored depth of anaesthesia and haemodynamics. We compare our approach with one from the Pharmacokinetics/Pharmacodynamic…
▽ More
We present a non-linear dynamical system for modelling the effect of drug infusions on the vital signs of patients admitted in Intensive Care Units (ICUs). More specifically we are interested in modelling the effect of a widely used anaesthetic drug (Propofol) on a patient's monitored depth of anaesthesia and haemodynamics. We compare our approach with one from the Pharmacokinetics/Pharmacodynamics (PK/PD) literature and show that we can provide significant improvements in performance without requiring the incorporation of expert physiological knowledge in our system.
△ Less
Submitted 8 October, 2016; v1 submitted 31 July, 2016;
originally announced August 2016.
-
Efficient analysis and representation of geophysical processes using localized spherical basis functions
Authors:
Frederik J. Simons,
Jessica C. Hawthorne,
Ciaran D. Beggan
Abstract:
While many geological and geophysical processes such as the melting of icecaps, the magnetic expression of bodies emplaced in the Earth's crust, or the surface displacement remaining after large earthquakes are spatially localized, many of these naturally admit spectral representations, or they may need to be extracted from data collected globally, e.g. by satellites that circumnavigate the Eart…
▽ More
While many geological and geophysical processes such as the melting of icecaps, the magnetic expression of bodies emplaced in the Earth's crust, or the surface displacement remaining after large earthquakes are spatially localized, many of these naturally admit spectral representations, or they may need to be extracted from data collected globally, e.g. by satellites that circumnavigate the Earth. Wavelets are often used to study such nonstationary processes. On the sphere, however, many of the known constructions are somewhat limited. And in particular, the notion of `dilation' is hard to reconcile with the concept of a geological region with fixed boundaries being responsible for generating the signals to be analyzed. Here, we build on our previous work on localized spherical analysis using an approach that is firmly rooted in spherical harmonics. We construct, by quadratic optimization, a set of bandlimited functions that have the majority of their energy concentrated in an arbitrary subdomain of the unit sphere. The `spherical Slepian basis' that results provides a convenient way for the analysis and representation of geophysical signals, as we show by example. We highlight the connections to sparsity by showing that many geophysical processes are sparse in the Slepian basis.
△ Less
Submitted 29 September, 2009;
originally announced September 2009.