examples/Colab_TrainNetwork_VideoAnalysis.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of latest_Colab_TrainNetwork_VideoAnalysis.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/AlexEMG/DeepLabCut/blob/master/examples/Colab_TrainNetwork_VideoAnalysis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RK255E7YoEIt",
        "colab_type": "text"
      },
      "source": [
        "# DeepLabCut Toolbox - Colab\n",
        "https://github.com/AlexEMG/DeepLabCut\n",
        "\n",
        "Nath\\*, Mathis\\* et al.: Using DeepLabCut for markerless pose estimation during behavior across species\n",
        "https://www.biorxiv.org/content/biorxiv/early/2018/11/24/476531.full.pdf\n",
        "\n",
        "This notebook demonstrates the necessary steps to use DeepLabCut for your own project.\n",
        "\n",
        "This shows the most simple code to do so, but many of the functions have additional features, so please check out the overview & the protocol paper!\n",
        "\n",
        "This notebook illustrates how to use the cloud to:\n",
        "- create a training set\n",
        "- train a network\n",
        "- evaluate a network\n",
        "- analyze a novel video\n",
        "\n",
        "###This notebook assumes you already have a project folder with labeled data! \n",
        "\n",
        "Note, if you used WINDOWS to create the project and label, you must run \"create_training_dataset\" on Colaboratory. Now (in DeepLabCut 2.0.4+) our code will automatically detect if you used Windows for project creation, and convert the files appropriately for using on Linux (aka Colab)! \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "txoddlM8hLKm",
        "colab_type": "text"
      },
      "source": [
        "## Let's look at info about the Colab Environment (and decide, GPU or TPU):\n",
        "\n",
        "First, go to \"Runtime\" ->\"change runtime type\"->select \"Python3\", and then decide, \"GPU or TPU!\"\"\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4C5WRoS9g5Od",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!nvcc --version"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_kwKDIgdySNQ",
        "colab_type": "text"
      },
      "source": [
        "If you want to use a GPU:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HxVNyimFp-PJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "tf.__version__"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Pm_PC1Q8lRrH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#let's make sure we see a GPU:\n",
        "tf.test.gpu_device_name()\n",
        "\n",
        "#from tensorflow.python.client import device_lib\n",
        "#device_lib.list_local_devices()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "StY_kCtQyUno",
        "colab_type": "text"
      },
      "source": [
        "If you selected to use a TPU: "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oKTs9zVGyedR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#let's make sure we see a TPU:\n",
        "import os\n",
        "import pprint\n",
        "import tensorflow as tf\n",
        "\n",
        "if 'COLAB_TPU_ADDR' not in os.environ:\n",
        "  print('ERROR: Not connected to a TPU runtime; please see the first cell in this notebook for instructions!')\n",
        "else:\n",
        "  tpu_address = 'grpc://' + os.environ['COLAB_TPU_ADDR']\n",
        "  print ('TPU address is', tpu_address)\n",
        "\n",
        "  with tf.Session(tpu_address) as session:\n",
        "    devices = session.list_devices()\n",
        "    \n",
        "  print('TPU devices:')\n",
        "  pprint.pprint(devices)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cQ-nlTkri4HZ",
        "colab_type": "text"
      },
      "source": [
        "## Link your Google Drive (with your labeled data, or the demo data):\n",
        "\n",
        "### First, place your porject folder into you google drive! \"i.e. move the folder named \"Project-YourName-TheDate\" into google drive."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KS4Q4UkR9rgG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Now, let's link to your GoogleDrive. Run this cell and follow the authorization instructions:\n",
        "#(We recommend putting a copy of the github repo in your google drive if you are using the demo \"examples\")\n",
        "\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q23BzhA6CXxu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#(this will take a few minutes to install all the dependences!)\n",
        "\n",
        "!pip install deeplabcut"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "flnaqXzcBZ_X",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#These are some Colab specific work-arounds, but they work! (typically not required, as they are installed with \"pip install deeplabcut\")\n",
        "\n",
        "!pip install Pillow==4.0.0\n",
        "\n",
        "from PIL import Image\n",
        "def register_extension(id, extension): Image.EXTENSION[extension.lower()] = id.upper()\n",
        "Image.register_extension = register_extension\n",
        "def register_extensions(id, extensions): \n",
        "  for extension in extensions: register_extension(id, extension)\n",
        "Image.register_extensions = register_extensions\n",
        "\n",
        "!pip install ruamel.yaml==0.15\n",
        "!pip install pandas==0.21.0\n",
        "!pip install matplotlib==3.0.3"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "25wSj6TlVclR",
        "colab_type": "text"
      },
      "source": [
        "**(Be sure to click \"RESTART RUNTIME\" above before moving on !)**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sXufoX6INe6w",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#GUIs don't work on the cloud, so label your data locally on your computer! \n",
        "import os\n",
        "os.environ[\"DLClight\"]=\"True\"\n",
        "os.environ[\"Colab\"]=\"True\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3K9Ndy1beyfG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import deeplabcut\n",
        "\n",
        "#sometimes the backend kernel is defaulting to something other than what we want\n",
        "#if you see the warnings, you can re-run this cell and they go way, as we re-set the backend! :) \n",
        "\n",
        "#possible warning--> UserWarning: This call to matplotlib.use() has no effect because the backend has already\n",
        "#been chosen; matplotlib.use() must be called *before* pylab, matplotlib.pyplot, or matplotlib.backends is imported for the first time."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o4orkg9QTHKK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "deeplabcut.__version__"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uJsZyt6Agmzp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Be sure you have a copy of your project folder in your Google Drive. Then, let's go to your deeplabcut folder!\n",
        "%cd /content/drive/My Drive/DeepLabCut/examples"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Frnj1RVDyEqs",
        "colab_type": "text"
      },
      "source": [
        "YOU WILL NEED TO EDIT THE PROJECT PATH **in the config.yaml file** TO BE SET TO YOUR GOOGLE DRIVE LINK!\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z7ZlDr3wV4D1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#create a path variable that links to your google drive copy:\n",
        "# EDIT to be your project config.yaml:\n",
        "from pathlib import Path\n",
        "path_config_file = os.path.join(os.getcwd(),'openfield-Pranav-2018-10-30/config.yaml')\n",
        "path_config_file"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xNi9s1dboEJN",
        "colab_type": "text"
      },
      "source": [
        "## Create a training dataset:\n",
        "### You must do this step inside of Colab:\n",
        "This function generates the training data information for DeepCut (which requires a mat file) based on the pandas dataframes that hold label information. The user can set the fraction of the training set size (from all labeled image in the hd5 file) in the config.yaml file. While creating the dataset, the user can create multiple shuffles. \n",
        "\n",
        "After running this script the training dataset is created and saved in the project directory under the subdirectory **'training-datasets'**\n",
        "\n",
        "This function also creates new subdirectories under **dlc-models** and appends the project config.yaml file with the correct path to the training and testing pose configuration file. These files hold the parameters for training the network. Such an example file is provided with the toolbox and named as **pose_cfg.yaml**.\n",
        "\n",
        "Now it is the time to start training the network!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "eMeUwgxPoEJP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# If you are using the demo data (i.e. examples/Reaching-Mackenzie-2018-08-30/), first delete the folder called dlc-models! \n",
        "#Then, run this cell. \n",
        "deeplabcut.create_training_dataset(path_config_file)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c4FczXGDoEJU",
        "colab_type": "text"
      },
      "source": [
        "## Start training:\n",
        "This function trains the network for a specific shuffle of the training dataset. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_pOvDq_2oEJW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#let's also change the display and save_iters just in case Colab takes away the GPU... \n",
        "#if that happens, you can reload from a saved point. Typically, you want to train to 200,000 + iterations.\n",
        "#more info and there are more things you can set: https://github.com/AlexEMG/DeepLabCut/blob/master/docs/functionDetails.md#g-train-the-network\n",
        "\n",
        "deeplabcut.train_network(path_config_file, shuffle=1, displayiters=10,saveiters=500)\n",
        "\n",
        "#this will run until you stop it (CTRL+C), or hit \"STOP\" icon, or when it hits the end (default, 1.03M iterations). \n",
        "#Whichever you chose, you will see what looks like an error message, but it's not an error - don't worry...."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RiDwIVf5-3H_",
        "colab_type": "text"
      },
      "source": [
        "**When you hit \"STOP\" you will get a KeyInterrupt \"error\"! No worries! :)**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xZygsb2DoEJc",
        "colab_type": "text"
      },
      "source": [
        "## Start evaluating:\n",
        "This funtion evaluates a trained model for a specific shuffle/shuffles at a particular state or all the states on the data set (images)\n",
        "and stores the results as .csv file in a subdirectory under **evaluation-results**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nv4zlbrnoEJg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%matplotlib notebook\n",
        "deeplabcut.evaluate_network(path_config_file,plotting=True)\n",
        "\n",
        "# Here you want to see a low pixel error! Of course, it can only be as good as the labeler, so be sure your labels are good!"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BaLBl3TQtrfB",
        "colab_type": "text"
      },
      "source": [
        "## There is an optional refinement step you can do outside of Colab:\n",
        "- if your pixel errors are not low enough, please check out the protocol guide on how to refine your network!\n",
        "- You will need to adjust the labels **outside of Colab!** We recommend coming back to train and analyze videos... \n",
        "- pplease see the repo and protocol instructions on how to refine your data!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OVFLSKKfoEJk",
        "colab_type": "text"
      },
      "source": [
        "## Start Analyzing videos: \n",
        "This function analyzes the new video. The user can choose the best model from the evaluation results and specify the correct snapshot index for the variable **snapshotindex** in the **config.yaml** file. Otherwise, by default the most recent snapshot is used to analyse the video.\n",
        "\n",
        "The results are stored in hd5 file in the same directory where the video resides. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y_LZiS_0oEJl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "videofile_path = ['/content/drive/My Drive/DeepLabCut/examples/Reaching-Mackenzie-2018-08-30/videos/MovieS2_Perturbation_noLaser_compressed.avi'] #Enter the list of videos to analyze.\n",
        "deeplabcut.analyze_videos(path_config_file,videofile_path)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pCrUvQIvoEKD",
        "colab_type": "text"
      },
      "source": [
        "## Create labeled video:\n",
        "This funtion is for visualiztion purpose and can be used to create a video in .mp4 format with labels predicted by the network. This video is saved in the same directory where the original video resides. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6aDF7Q7KoEKE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "deeplabcut.create_labeled_video(path_config_file,videofile_path)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8GTiuJESoEKH",
        "colab_type": "text"
      },
      "source": [
        "## Plot the trajectories of the analyzed videos:\n",
        "This function plots the trajectories of all the body parts across the entire video. Each body part is identified by a unique color."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gX21zZbXoEKJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "deeplabcut.plot_trajectories(path_config_file,videofile_path)"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}