project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## play music of data\n",
    "\n",
    "\n",
    "////////////////  TODO  ////////////////\n",
    "\n",
    "- create specialist\n",
    "- how to decide binary key-press? sample with probability og pressing X(beeing avg. keys pressed at any time?)\n",
    "    - https://github.com/calclavia/DeepJ/blob/icsc/generate.py \n",
    "    - apply_temperature from code above makes it more likely to dont press buttons when there has been a pause\n",
    "    - regularizing predicting the same note. by small negative\n",
    "    - normalize predictionvectors before applying threshold\n",
    "    - would be nice with teacher forcing, think it would be better with shortterm memory and rely less on \"beat\"\n",
    "    - chunks probably work better to not make it so relyant on specific song and training on remembering.\n",
    "    - could use scheduler to decrease learningrate after 150 epochs\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(\"helpers/\")\n",
    "\n",
    "#Libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "import pickle\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "print \"torchversion: \", torch.__version__\n",
    "\n",
    "#Custom libraries\n",
    "import dataset\n",
    "import datapreparation\n",
    "from datapreparation import plot_loss\n",
    "from train import train\n",
    "from models import Generalist, Specialist\n",
    "\n",
    "#PATHS\n",
    "DATASET_PATH = 'datasets/training/'\n",
    "FS1 = 'piano_roll_fs1/'\n",
    "FS2 = 'piano_roll_fs2/'\n",
    "FS5 = 'piano_roll_fs5/'\n",
    "FULL_PATH = DATASET_PATH + FS5\n",
    "\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inspect the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "#LOAD\n",
    "dataset_names = datapreparation.load_all_dataset_names(FULL_PATH)\n",
    "print(dataset_names)\n",
    "datasets = datapreparation.load_all_dataset(FULL_PATH)\n",
    "#datasets_df = pd.DataFrame(datasets)\n",
    "\n",
    "#CONSTANTS\n",
    "max_length = datapreparation.get_max_length(datasets)\n",
    "num_keys = datapreparation.get_numkeys(datasets)[0]\n",
    "print(\"max_length: \", max_length)\n",
    "print(\"num_keys: \", num_keys)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Split datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# shuffles the dataset on creation\n",
    "Dataset = dataset.pianoroll_dataset_batch(FULL_PATH)\n",
    "# splits the datasets\n",
    "training_data, validation_data, testing_data = Dataset.split_datasets(split=[0.7,0.85, 1.0])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sample from dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#VISUALIZE\n",
    "print('Bach')\n",
    "print('timesteps in song:', len(datasets[0][0]))\n",
    "datapreparation.visualize_piano_roll(datasets[0])\n",
    "\n",
    "datapreparation.embed_play_v1(datasets[0])\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "# Generalist composer\n",
    "The aim is to train a neural network that can create music in general. It shall be kickstarted by real music. Meaning that it will get some seconds of real music and then it wil continue playing music."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set parameters for the model\n",
    "input_size = num_keys\n",
    "hidden_size = 64\n",
    "num_tags = len(pd.unique(dataset_names))\n",
    "num_layers = 1\n",
    "\n",
    "# create model\n",
    "generalist = Generalist(input_size, hidden_size, num_layers)\n",
    "\n",
    "# set parameters for the training process\n",
    "name = \"generalist_BCE_1e-2\"\n",
    "epochs = 1000\n",
    "batch_size = 4 # does not effect training. need to implement in def train\n",
    "lr = 1e-2\n",
    "criterion = None # it will create default criterion\n",
    "optimizer = None # it will create default optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TRAIN GENERALIST\n",
    "\n",
    "trained_generalist, gen_train_loss, gen_val_loss, gen_train_acc, gen_test_acc  = train(generalist, training_data, validation_data, name, criterion, epochs, lr, optimizer, batch_size)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load loss\n",
    "loaded_generalist_metrics = pickle.load(open( 'saved_losses/generalist_BCE_1e-2_999.pickle', \"rb\" ))\n",
    "plot_loss(loaded_generalist_metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# generate song from generalist\n",
    "\n",
    "# pick song\n",
    "#song = training_data[10]\n",
    "song = testing_data[1]\n",
    "# load best model\n",
    "model = Generalist(input_size, hidden_size, num_layers) # need to know input_size, hidden_size excetera.. :/\n",
    "model.load_state_dict(torch.load('saved_models/generalist_BCE_1e-2_208.pth')) #best so far on song test:1, train:10 with init=10 not good now\n",
    "#model.load_state_dict(torch.load('saved_models/generalist_BCE_1e-2_175.pth'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datapreparation.gen_music_seconds(model,init=song[0],composer=song[1],fs=5,gen_seconds=60,init_seconds=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "# Specialist composer\n",
    "The aim is to train a neural network that can create music similar to a specific composer. It shall be kickstarted by a song from that composer. Meaning that it will get some seconds of the song and then it should be able to compose music that is typically similar to that specific composer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set parameters for the model\n",
    "input_size = num_keys\n",
    "hidden_size = 64\n",
    "num_tags = len(pd.unique(dataset_names))\n",
    "num_layers = 1\n",
    "\n",
    "# create model\n",
    "specialist = Specialist(input_size, hidden_size, num_layers)\n",
    "\n",
    "# set parameters for the training process\n",
    "name = \"Specialist_BCE_1e-2\"\n",
    "epochs = 1000\n",
    "batch_size = 4 # does not effect training. need to implement in def train\n",
    "lr = 1e-2\n",
    "criterion = None # it will create default criterion\n",
    "optimizer = None # it will create default optimizer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TRAIN SPECIALIST\n",
    "\n",
    "trained_specialist, spe_train_loss, spe_val_loss, spe_train_acc, spe_test_acc  = train(specialist, training_data, validation_data, name, criterion, epochs, lr, optimizer, batch_size)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load loss\n",
    "loaded_specialist_metrics = pickle.load(open( 'saved_losses/specialist_BCE_1e-2_999.pickle', \"rb\" ))\n",
    "plot_loss(loaded_specialist_metrics)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# generate song from specialist\n",
    "\n",
    "# pick song\n",
    "#song = training_data[10]\n",
    "song = testing_data[1]\n",
    "\n",
    "# load best model\n",
    "model = Specialist(input_size, hidden_size, num_layers) # need to know input_size, hidden_size excetera.. :/\n",
    "model.load_state_dict(torch.load('saved_models/specialist_BCE_1e-2_208.pth')) \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# generate\n",
    "datapreparation.gen_music_seconds(model,init=song[0],composer=song[1],fs=5,gen_seconds=60,init_seconds=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comments\n",
    "- specializes on one song because of small sample size, but huge samples\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Further works\n",
    "issue 1: specializes very quickly\n",
    "- bigger dataset which there is here: http://www.piano-midi.de/\n",
    "- chunks should maybe remove some of that effect\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}