Video2Music: Suitable Music Generation from Videos using an Affective Multimodal Transformer model

Demo | Website and Examples | Paper | Dataset (MuVi-Sync)

This repository contains the code and dataset accompanying the paper "Video2Music: Suitable Music Generation from Videos using an Affective Multimodal Transformer model" by Dr. Jaeyong Kang, Prof. Soujanya Poria, and Prof. Dorien Herremans.

🔥 Live demo available on HuggingFace and Replicate.

Introduction

We propose a novel AI-powered multimodal music generation framework called Video2Music. This framework uniquely uses video features as conditioning input to generate matching music using a Transformer architecture. By employing cutting-edge technology, our system aims to provide video creators with a seamless and efficient solution for generating tailor-made background music.

Change Log

• 2023-11-28: add new input method (YouTube URL) on HuggingFace

Quickstart Guide

Generate music from video:

import IPython
from video2music import Video2music

input_video = "input.mp4"

input_primer = "C Am F G"
input_key = "C major"

video2music = Video2music()
output_filename = video2music.generate(input_video, input_primer, input_key)

IPython.display.Video(output_filename)

Installation

This repo is developed using python version 3.8

apt-get update
apt-get install ffmpeg
apt-get install fluidsynth
git clone https://github.com/AMAAI-Lab/Video2Music
cd Video2Music
pip install -r requirements.txt

• Download the processed training data AMT.zip from HERE and extract the zip file and put the extracted two files directly under this folder (saved_models/AMT/)

• Download the soundfont file default_sound_font.sf2 from HERE and put the file directly under this folder (soundfonts/)

• Our code is built on pytorch version 1.12.1 (torch==1.12.1 in the requirements.txt). But you might need to choose the correct version of torch based on your CUDA version

Dataset

• Obtain the dataset:

  • MuVi-Sync (Link)

• Put all directories started with vevo in the dataset under this folder (dataset/)

Directory Structure

• saved_models/: saved model files
• utilities/
  • run_model_vevo.py: code for running model (AMT)
  • run_model_regression.py: code for running model (bi-GRU)
• model/
  • video_music_transformer.py: Affective Multimodal Transformer (AMT) model
  • video_regression.py: Bi-GRU regression model used for predicting note density/loudness
  • positional_encoding.py: code for Positional encoding
  • rpr.py: code for RPR (Relative Positional Representation)
• dataset/
  • vevo_dataset.py: Dataset loader
• script/ : code for extracting video/music features (sementic, motion, emotion, scene offset, loudness, and note density)
• train.py: training script (AMT)
• train_regression.py: training script (bi-GRU)
• evaluate.py: evaluation script
• generate.py: inference script
• video2music.py: Video2Music module that outputs video with generated background music from input video
• demo.ipynb: Jupyter notebook for Quickstart Guide

Training

python train.py

Inference

python generate.py

Subjective Evaluation by Listeners

Model	Overall Music Quality ↑	Music-Video Correspondence ↑	Harmonic Matching ↑	Rhythmic Matching ↑	Loudness Matching ↑
Music Transformer	3.4905	2.7476	2.6333	2.8476	3.1286
Video2Music	4.2095	3.6667	3.4143	3.8714	3.8143

TODO

• Add other instruments (e.g., drum) for live demo

Citation

If you find this resource useful, please cite the original work:

@article{KANG2024123640,
  title = {Video2Music: Suitable music generation from videos using an Affective Multimodal Transformer model},
  author = {Jaeyong Kang and Soujanya Poria and Dorien Herremans},
  journal = {Expert Systems with Applications},
  pages = {123640},
  year = {2024},
  issn = {0957-4174},
  doi = {https://doi.org/10.1016/j.eswa.2024.123640},
}

Kang, J., Poria, S. & Herremans, D. (2024). Video2Music: Suitable Music Generation from Videos using an Affective Multimodal Transformer model, Expert Systems with Applications (in press).

Acknowledgements

Our code is based on Music Transformer.