2_training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: Training a pretrained model with TF-Object-Detection-API repo\n",
    "\n",
    "In this notebook, we will train an [SSD mobilnet v2](https://arxiv.org/pdf/1801.04381.pdf) model (pre-trained on the COCO object detection task) to spot Syd in the Olympics Opening Ceremony and stadiums.\n",
    "\n",
    "![wanted Syd](https://raw.githubusercontent.com/floydhub/object-detection-template/master/images/wanted-syd.jpg)\n",
    "*Are you able to catch Syd in this image?*\n",
    "\n",
    "For more info about Tensorflow Object Detection, please refer to [the official docs](https://github.com/tensorflow/models/tree/master/research/object_detection/g3doc).\n",
    "\n",
    "**IMPORTANT**\n",
    "\n",
    "Train on a GPU instance otherwise you will get an *Out-Of-Memory Error*.\n",
    "\n",
    "### Initial Setup\n",
    "Let's start by importing and installing the packages, setting the training variables and the path where to save the models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install extra-dependency\n",
    "! pip -q install pycocotools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The current configuration is set to run on a single machine and without overwriting the `pipeline.config` file of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use the retry module or similar alternatives.\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "sys.path.append(\"slim\")\n",
    "\n",
    "import functools\n",
    "import json\n",
    "import os\n",
    "import tensorflow as tf\n",
    "\n",
    "from object_detection import trainer\n",
    "from object_detection.builders import dataset_builder\n",
    "from object_detection.builders import graph_rewriter_builder\n",
    "from object_detection.builders import model_builder\n",
    "from object_detection.utils import config_util\n",
    "from object_detection.utils import dataset_util\n",
    "\n",
    "from support import get_train_config, get_eval_config\n",
    "\n",
    "# For reproducibility\n",
    "from tensorflow import set_random_seed\n",
    "from numpy.random import seed\n",
    "seed(2018)\n",
    "set_random_seed(2018)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train\n",
    "\n",
    "The configuration is defined in the `PIPELINE_CONFING` file, edit this file or the `train_config` variable to modify the training settings. Training for 3700 steps will take about 30-35 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train configuration: \n",
      " ------------------------------ \n",
      " batch_size: 8\n",
      "data_augmentation_options {\n",
      "  random_horizontal_flip {\n",
      "  }\n",
      "}\n",
      "data_augmentation_options {\n",
      "  ssd_random_crop {\n",
      "  }\n",
      "}\n",
      "optimizer {\n",
      "  rms_prop_optimizer {\n",
      "    learning_rate {\n",
      "      exponential_decay_learning_rate {\n",
      "        initial_learning_rate: 0.004000000189989805\n",
      "        decay_steps: 10000\n",
      "        decay_factor: 0.949999988079071\n",
      "      }\n",
      "    }\n",
      "    momentum_optimizer_value: 0.8999999761581421\n",
      "    decay: 0.8999999761581421\n",
      "    epsilon: 1.0\n",
      "  }\n",
      "}\n",
      "fine_tune_checkpoint: \"/floyd/code/models/ssdlite_mobilenet_v2_coco_2018_05_09/model.ckpt\"\n",
      "num_steps: 3700\n",
      "fine_tune_checkpoint_type: \"detection\"\n",
      "\n"
     ]
    }
   ],
   "source": [
    "##################################################\n",
    "# CONFIGURATION (Local, Multi-GPUs, Distributed) #\n",
    "##################################################\n",
    "\n",
    "MASTER = ''  # Name of the TensorFlow master to use\n",
    "TASK = 0  # task id\n",
    "NUM_CLONES = 1  # Number of clones to deploy per worker.\n",
    "\n",
    "# Force clones to be deployed on CPU.  Note that even if \n",
    "# set to False (allowing ops to run on gpu), some ops may \n",
    "# still be run on the CPU if they have no GPU kernel.\n",
    "CLONE_ON_CPU = False\n",
    "\n",
    "WORKER_REPLICAS = 1  # Number of worker+trainer replicas.\n",
    "PS_TASKS = 0  # Number of parameter server tasks. If None, does not use a parameter server.\n",
    "TRAIN_DIR = 'trained_models/ssdlite_mobilenet_v2_coco_2018_05_09'  # Directory to save the checkpoints and training summaries.\n",
    "\n",
    "# Path to a pipeline_pb2.TrainEvalPipelineConfig config\n",
    "# file. If provided, other configs are ignored\n",
    "PIPELINE_CONFING_PATH = 'models/ssdlite_mobilenet_v2_coco_2018_05_09/pipeline.config'\n",
    "TRAIN_CONFIG_PATH = ''  # Path to a train_pb2.TrainConfig config file.\n",
    "INPUT_CONFIG_PATH = ''  # Path to an input_reader_pb2.InputReader config file.\n",
    "MODEL_CONFIG_PATH = ''  # Path to a model_pb2.DetectionModel config file.\n",
    "\n",
    "# Get configuration for training\n",
    "(create_input_dict_fn,\n",
    "model_fn,\n",
    "train_config,\n",
    "master,\n",
    "task,\n",
    "worker_job_name,\n",
    "ps_tasks,\n",
    "worker_replicas,\n",
    "is_chief,\n",
    "graph_rewriter_fn) = get_train_config(TASK,\n",
    "                                     PS_TASKS,\n",
    "                                     TRAIN_DIR,\n",
    "                                     PIPELINE_CONFING_PATH,\n",
    "                                     TRAIN_CONFIG_PATH,\n",
    "                                     MODEL_CONFIG_PATH,\n",
    "                                     INPUT_CONFIG_PATH,\n",
    "                                     WORKER_REPLICAS,\n",
    "                                     MASTER)\n",
    "\n",
    "\n",
    "######################################\n",
    "# HOW TO EDIT TRAIN CONFIG FROM CODE #\n",
    "######################################\n",
    "\n",
    "# You can change the setting in this way,\n",
    "# e.g. for batch size and num_steps\n",
    "# train_config.batch_size = 4\n",
    "# train_config.num_steps = 4000\n",
    "\n",
    "print(\"Train configuration: \\n\", '-'*30, '\\n', train_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training\n",
    "trainer.train(\n",
    "  create_input_dict_fn,\n",
    "  model_fn,\n",
    "  train_config,\n",
    "  master,\n",
    "  task,\n",
    "  NUM_CLONES,\n",
    "  worker_replicas,\n",
    "  CLONE_ON_CPU,\n",
    "  ps_tasks,\n",
    "  worker_job_name,\n",
    "  is_chief,\n",
    "  TRAIN_DIR,\n",
    "  graph_hook_fn=graph_rewriter_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate\n",
    "\n",
    "Now it's time to evaluate our trained model.\n",
    "\n",
    "**Note**\n",
    "\n",
    "Unfortunately a bug prevent the correct output of [IOU(Intersection Over Union) metric](https://en.wikipedia.org/wiki/Jaccard_index). We will fix it as soon as possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Eval configuration: \n",
      " ------------------------------ \n",
      " num_examples: 1\n",
      "max_evals: 1\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from object_detection import evaluator\n",
    "from object_detection.utils import label_map_util\n",
    "\n",
    "tf.reset_default_graph()\n",
    "\n",
    "##############################\n",
    "# CONFIGURATION (Evaluation) #\n",
    "##############################\n",
    "\n",
    "# Directory to write eval summaries to.\n",
    "EVAL_DIR = 'trained_models/ssdlite_mobilenet_v2_coco_2018_05_09/eval' \n",
    "\n",
    "# If training data should be evaluated for this job.\n",
    "EVAL_TRAINING_DATA = False\n",
    "\n",
    "# Option to only run a single pass of evaluation. \n",
    "# Overrides the `max_evals` parameter in the provided config\n",
    "RUN_ONCE = True\n",
    "\n",
    "# Directory containing checkpoints to evaluate, typically\n",
    "# set to `train_dir` used in the training job.\n",
    "CHECKPOINT_DIR = TRAIN_DIR\n",
    "\n",
    "EVAL_CONFIG_PATH = ''  # Path to a eval_pb2.TrainConfig config file.\n",
    "EVAL_INPUT_CONFIG_PATH = ''  # Path to an input_reader_pb2.InputReader config file.\n",
    "MODEL_CONFIG_PATH = ''  # Path to a model_pb2.DetectionModel config file.\n",
    "\n",
    "# Get configuration for Evaluation\n",
    "(create_input_dict_fn,\n",
    "model_fn,\n",
    "eval_config,\n",
    "categories,\n",
    "graph_rewriter_fn) = get_eval_config(EVAL_DIR,\n",
    "                                    PIPELINE_CONFING_PATH,\n",
    "                                    EVAL_CONFIG_PATH,\n",
    "                                    MODEL_CONFIG_PATH,\n",
    "                                    EVAL_INPUT_CONFIG_PATH,\n",
    "                                    EVAL_TRAINING_DATA,\n",
    "                                    RUN_ONCE)\n",
    "\n",
    "######################################\n",
    "# HOW TO EDIT EVAL CONFIG FROM CODE #\n",
    "######################################\n",
    "\n",
    "# You can change the setting in this way,\n",
    "# e.g. for evaluate only once\n",
    "# eval_config.max_evals = 1\n",
    "\n",
    "print(\"Eval configuration: \\n\", '-'*30, '\\n', eval_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:depth of additional conv before box predictor: 0\n",
      "INFO:tensorflow:depth of additional conv before box predictor: 0\n",
      "INFO:tensorflow:depth of additional conv before box predictor: 0\n",
      "INFO:tensorflow:depth of additional conv before box predictor: 0\n",
      "INFO:tensorflow:depth of additional conv before box predictor: 0\n",
      "INFO:tensorflow:depth of additional conv before box predictor: 0\n",
      "INFO:tensorflow:Restoring parameters from trained_models/ssdlite_mobilenet_v2_coco_2018_05_09/model.ckpt-3700\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from trained_models/ssdlite_mobilenet_v2_coco_2018_05_09/model.ckpt-3700\n",
      "WARNING:root:image 0 does not have groundtruth difficult flag specified\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'PascalBoxes_Precision/mAP@0.5IOU': 0.0,\n",
       " 'PascalBoxes_PerformanceByCategory/AP@0.5IOU/syd': 0.0,\n",
       " 'Losses/Loss/localization_loss': 15.152876,\n",
       " 'Losses/Loss/classification_loss': 24.296291}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Evaluate\n",
    "evaluator.evaluate(\n",
    "    create_input_dict_fn,\n",
    "    model_fn,\n",
    "    eval_config,\n",
    "    categories,\n",
    "    CHECKPOINT_DIR,\n",
    "    EVAL_DIR,\n",
    "    graph_hook_fn=graph_rewriter_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exporting the model\n",
    "\n",
    "In the last section of the notebook, we will export the model to run prediction on images converted into tensors (matrix representation for [height, width, color_channels])."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:depth of additional conv before box predictor: 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:depth of additional conv before box predictor: 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:depth of additional conv before box predictor: 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:depth of additional conv before box predictor: 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:depth of additional conv before box predictor: 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:depth of additional conv before box predictor: 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:depth of additional conv before box predictor: 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:depth of additional conv before box predictor: 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:depth of additional conv before box predictor: 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:depth of additional conv before box predictor: 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:depth of additional conv before box predictor: 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:depth of additional conv before box predictor: 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /.floyd_code/object_detection/exporter.py:358: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please switch to tf.train.get_or_create_global_step\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /.floyd_code/object_detection/exporter.py:358: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please switch to tf.train.get_or_create_global_step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from trained_models/ssdlite_mobilenet_v2_coco_2018_05_09/model.ckpt-3700\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from trained_models/ssdlite_mobilenet_v2_coco_2018_05_09/model.ckpt-3700\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from trained_models/ssdlite_mobilenet_v2_coco_2018_05_09/model.ckpt-3700\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from trained_models/ssdlite_mobilenet_v2_coco_2018_05_09/model.ckpt-3700\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Froze 404 variables.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Froze 404 variables.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Converted 404 variables to const ops.\n",
      "INFO:tensorflow:No assets to save.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:No assets to save.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:No assets to write.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:No assets to write.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:SavedModel written to: b'trained_models/ssdlite_mobilenet_v2_coco_2018_05_09_exported/saved_model/saved_model.pb'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:SavedModel written to: b'trained_models/ssdlite_mobilenet_v2_coco_2018_05_09_exported/saved_model/saved_model.pb'\n"
     ]
    }
   ],
   "source": [
    "# Exporting the model for Evaluation\n",
    "from google.protobuf import text_format\n",
    "from object_detection import exporter\n",
    "from object_detection.protos import pipeline_pb2\n",
    "\n",
    "slim = tf.contrib.slim\n",
    "tf.reset_default_graph()\n",
    "\n",
    "config_override=''\n",
    "input_shape=None\n",
    "trained_checkpoint_prefix = os.path.join(TRAIN_DIR, 'model.ckpt-3700') # EDIT: model.ckpt-<iteration>\n",
    "output_directory='trained_models/ssdlite_mobilenet_v2_coco_2018_05_09_exported'\n",
    "\n",
    "# Type of input node. Can be one of :\n",
    "# [`image_tensor`, `encoded_image_string_tensor`,`tf_example`]\n",
    "input_type='image_tensor' \n",
    "\n",
    "pipeline_config = pipeline_pb2.TrainEvalPipelineConfig()\n",
    "with tf.gfile.GFile(PIPELINE_CONFING_PATH, 'r') as f:\n",
    "    text_format.Merge(f.read(), pipeline_config)\n",
    "text_format.Merge(config_override, pipeline_config)\n",
    "if input_shape:\n",
    "    input_shape = [\n",
    "        int(dim) if dim != '-1' else None\n",
    "        for dim in input_shape.split(',')\n",
    "    ]\n",
    "else:\n",
    "    input_shape = None\n",
    "exporter.export_inference_graph(input_type, pipeline_config,\n",
    "                                  trained_checkpoint_prefix,\n",
    "                                  output_directory, input_shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! Will we be able to find Syd in never seen images? Let's jump to the  `3_prediction` notebook to find the answer."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}