text: Discovering and exploring the underlying structure of multi-instrumental music using learning-based approaches remains an open problem. We extend the recent MusicVAE model to represent multitrack polyphonic measures as vectors in a latent space. Our approach enables several useful operations such as generating plausible measures from scratch, interpolating between measures in a musically meaningful way, and manipulating specific musical attributes. We also introduce chord conditioning, which allows all of these operations to be performed while keeping harmony fixed, and allows chords to be changed while maintaining musical "style". By generating a sequence of measures over a predefined chord progression, our model can produce music with convincing long-term structure. We demonstrate that our latent space model makes it possible to intuitively control and generate musical sequences with rich instrumentation (see this https URL for generated audio).
  @cite_39: The Variational Autoencoder (VAE) has proven to be an effective model for producing semantically meaningful latent representations for natural data. However, it has thus far seen limited application to sequential data, and, as we demonstrate, existing recurrent VAE models have difficulty modeling sequences with long-term structure. To address this issue, we propose the use of a hierarchical decoder, which first outputs embeddings for subsequences of the input and then uses these embeddings to generate each subsequence independently. This structure encourages the model to utilize its latent code, thereby avoiding the "posterior collapse" problem which remains an issue for recurrent VAEs. We apply this architecture to modeling sequences of musical notes and find that it exhibits dramatically better sampling, interpolation, and reconstruction performance than a "flat" baseline model. An implementation of our "MusicVAE" is available online at this http URL
summary: The MusicVAE architecture introduced by @cite_39 is able to learn a latent space of musical sequences using a novel hierarchical decoder that allows it to model long-term structure and multi-instrument sequences. However, this work applies strict constraints to the sequences to reach its goals. In order to guarantee a constant number of events per measure, non-drum tracks are limited to monophonic sequences, and all tracks are represented with a single velocity and quantized at the level of 16 th notes. This reduces the challenge of modeling longer-term structure, but at the expense of expressiveness. Furthermore, while the trio'' MusicVAE is capable of modeling three broad instrument classes--melody, bass, and drums--it is limited to exactly three instruments, arbitrarily excluding potentially pivotal voices and disregarding the specific identity of each instrument (e.g. eletric guitar and piano are both considered melody'' instruments). Further, do not consider modeling fine-grained timing and velocity, nor do they develop a method for chord conditioning. Nevertheless, the MusicVAE architecture and its implementation provide a powerful basis for exploring a more expressive and complete multitrack latent space, and thus we position our work as an extension of it.