Musical Duet Generation with Tree-Structured Variational Autoencoder

Recent successful latent space models based on Variational AutoEncoder (VAE) can generate polyphonic music for solo instrument. Polyphony with multiple tracks is more challenging in many aspects. Towards this goal, we propose the intermediate task of musical duet generation. In this setting, it is common to have to deal with two different instruments and note overlaps among tracks occur frequently. Unfortunately, these meaningful overlaps are discarded by current musical models. This limitation hinders their ability to generate multi-track music. We thus propose two data structures, MergedTree and HierarchicalTree, to overcome this limitation and three models, MergedTree VAE, SharedHierarchicalTree VAE and HierarchicalTree VAE, which leverage these data representations to reconstruct musical duets3. We evaluate our models on Lakh MIDI dataset and compare them to a PianoTree VAE baseline. Our MergedTree VAE model outperforms the baseline model on reconstruction scores. In addition, all proposed models are able to incorporate a statistically relevant ratio of overlaps among tracks.