  Numerous studies in the field of music generation have demonstrated
impressive performance, yet virtually no models are able to directly generate
music to match accompanying videos. In this work, we develop a generative music
AI framework, Video2Music, that can match a provided video. We first curated a
unique collection of music videos. Then, we analysed the music videos to obtain
semantic, scene offset, motion, and emotion features. These distinct features
are then employed as guiding input to our music generation model. We transcribe
the audio files into MIDI and chords, and extract features such as note density
and loudness. This results in a rich multimodal dataset, called MuVi-Sync, on
which we train a novel Affective Multimodal Transformer (AMT) model to generate
music given a video. This model includes a novel mechanism to enforce affective
similarity between video and music. Finally, post-processing is performed based
on a biGRU-based regression model to estimate note density and loudness based
on the video features. This ensures a dynamic rendering of the generated chords
with varying rhythm and volume. In a thorough experiment, we show that our
proposed framework can generate music that matches the video content in terms
of emotion. The musical quality, along with the quality of music-video matching
is confirmed in a user study. The proposed AMT model, along with the new
MuVi-Sync dataset, presents a promising step for the new task of music
generation for videos.
