Policy-Network.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simple Reinforcement Learning in Tensorflow Part 2: Policy Gradient Method\n",
    "This tutorial contains a simple example of how to build a policy-gradient based agent that can solve the CartPole problem. For more information, see this [Medium post](https://medium.com/@awjuliani/super-simple-reinforcement-learning-tutorial-part-2-ded33892c724#.mtwpvfi8b).\n",
    "\n",
    "For more Reinforcement Learning algorithms, including DQN and Model-based learning in Tensorflow, see my Github repo, [DeepRL-Agents](https://github.com/awjuliani/DeepRL-Agents). \n",
    "\n",
    "Parts of this tutorial are based on code by [Andrej Karpathy](https://gist.github.com/karpathy/a4166c7fe253700972fcbc77e4ea32c5) and [korymath](https://gym.openai.com/evaluations/eval_a0aVJrGSyW892vBM04HQA)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from __future__ import division\n",
    "\n",
    "import numpy as np\n",
    "try:\n",
    "    import cPickle as pickle\n",
    "except:\n",
    "    import pickle\n",
    "import tensorflow as tf\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import math\n",
    "\n",
    "try:\n",
    "    xrange = xrange\n",
    "except:\n",
    "    xrange = range"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the CartPole Environment\n",
    "If you don't already have the OpenAI gym installed, use  `pip install gym` to grab it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-03-09 18:51:17,880] Making new env: CartPole-v0\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "env = gym.make('CartPole-v0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What happens if we try running the environment with random actions? How well do we do? (Hint: not so well.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Reward for this episode was:', 17.0)\n",
      "('Reward for this episode was:', 65.0)\n",
      "('Reward for this episode was:', 12.0)\n",
      "('Reward for this episode was:', 20.0)\n",
      "('Reward for this episode was:', 22.0)\n",
      "('Reward for this episode was:', 11.0)\n",
      "('Reward for this episode was:', 13.0)\n",
      "('Reward for this episode was:', 33.0)\n",
      "('Reward for this episode was:', 12.0)\n",
      "('Reward for this episode was:', 19.0)\n"
     ]
    }
   ],
   "source": [
    "env.reset()\n",
    "random_episodes = 0\n",
    "reward_sum = 0\n",
    "while random_episodes < 10:\n",
    "    env.render()\n",
    "    observation, reward, done, _ = env.step(np.random.randint(0,2))\n",
    "    reward_sum += reward\n",
    "    if done:\n",
    "        random_episodes += 1\n",
    "        print(\"Reward for this episode was:\",reward_sum)\n",
    "        reward_sum = 0\n",
    "        env.reset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of the task is to achieve a reward of 200 per episode. For every step the agent keeps the pole in the air, the agent recieves a +1 reward. By randomly choosing actions, our reward for each episode is only a couple dozen. Let's make that better with RL!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting up our Neural Network agent\n",
    "This time we will be using a Policy neural network that takes observations, passes them through a single hidden layer, and then produces a probability of choosing a left/right movement. To learn more about this network, see [Andrej Karpathy's blog on Policy Gradient networks](http://karpathy.github.io/2016/05/31/rl/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# hyperparameters\n",
    "H = 10 # number of hidden layer neurons\n",
    "batch_size = 5 # every how many episodes to do a param update?\n",
    "learning_rate = 1e-2 # feel free to play with this to train faster or more stably.\n",
    "gamma = 0.99 # discount factor for reward\n",
    "\n",
    "D = 4 # input dimensionality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "#This defines the network as it goes from taking an observation of the environment to \n",
    "#giving a probability of chosing to the action of moving left or right.\n",
    "observations = tf.placeholder(tf.float32, [None,D] , name=\"input_x\")\n",
    "W1 = tf.get_variable(\"W1\", shape=[D, H],\n",
    "           initializer=tf.contrib.layers.xavier_initializer())\n",
    "layer1 = tf.nn.relu(tf.matmul(observations,W1))\n",
    "W2 = tf.get_variable(\"W2\", shape=[H, 1],\n",
    "           initializer=tf.contrib.layers.xavier_initializer())\n",
    "score = tf.matmul(layer1,W2)\n",
    "probability = tf.nn.sigmoid(score)\n",
    "\n",
    "#From here we define the parts of the network needed for learning a good policy.\n",
    "tvars = tf.trainable_variables()\n",
    "input_y = tf.placeholder(tf.float32,[None,1], name=\"input_y\")\n",
    "advantages = tf.placeholder(tf.float32,name=\"reward_signal\")\n",
    "\n",
    "# The loss function. This sends the weights in the direction of making actions \n",
    "# that gave good advantage (reward over time) more likely, and actions that didn't less likely.\n",
    "loglik = tf.log(input_y*(input_y - probability) + (1 - input_y)*(input_y + probability))\n",
    "loss = -tf.reduce_mean(loglik * advantages) \n",
    "newGrads = tf.gradients(loss,tvars)\n",
    "\n",
    "# Once we have collected a series of gradients from multiple episodes, we apply them.\n",
    "# We don't just apply gradeients after every episode in order to account for noise in the reward signal.\n",
    "adam = tf.train.AdamOptimizer(learning_rate=learning_rate) # Our optimizer\n",
    "W1Grad = tf.placeholder(tf.float32,name=\"batch_grad1\") # Placeholders to send the final gradients through when we update.\n",
    "W2Grad = tf.placeholder(tf.float32,name=\"batch_grad2\")\n",
    "batchGrad = [W1Grad,W2Grad]\n",
    "updateGrads = adam.apply_gradients(zip(batchGrad,tvars))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Advantage function\n",
    "This function allows us to weigh the rewards our agent recieves. In the context of the Cart-Pole task, we want actions that kept the pole in the air a long time to have a large reward, and actions that contributed to the pole falling to have a decreased or negative reward. We do this by weighing the rewards from the end of the episode, with actions at the end being seen as negative, since they likely contributed to the pole falling, and the episode ending. Likewise, early actions are seen as more positive, since they weren't responsible for the pole falling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def discount_rewards(r):\n",
    "    \"\"\" take 1D float array of rewards and compute discounted reward \"\"\"\n",
    "    discounted_r = np.zeros_like(r)\n",
    "    running_add = 0\n",
    "    for t in reversed(xrange(0, r.size)):\n",
    "        running_add = running_add * gamma + r[t]\n",
    "        discounted_r[t] = running_add\n",
    "    return discounted_r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running the Agent and Environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we run the neural network agent, and have it act in the CartPole environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average reward for episode 20.000000.  Total average reward 20.000000.\n",
      "Average reward for episode 22.000000.  Total average reward 20.000000.\n",
      "Average reward for episode 26.000000.  Total average reward 20.000000.\n",
      "Average reward for episode 25.000000.  Total average reward 20.000000.\n",
      "Average reward for episode 22.000000.  Total average reward 20.000000.\n",
      "Average reward for episode 48.000000.  Total average reward 21.000000.\n",
      "Average reward for episode 25.000000.  Total average reward 21.000000.\n",
      "Average reward for episode 29.000000.  Total average reward 21.000000.\n",
      "Average reward for episode 36.000000.  Total average reward 21.000000.\n",
      "Average reward for episode 30.000000.  Total average reward 21.000000.\n",
      "Average reward for episode 26.000000.  Total average reward 21.000000.\n",
      "Average reward for episode 31.000000.  Total average reward 21.000000.\n",
      "Average reward for episode 62.000000.  Total average reward 22.000000.\n",
      "Average reward for episode 28.000000.  Total average reward 22.000000.\n",
      "Average reward for episode 28.000000.  Total average reward 22.000000.\n",
      "Average reward for episode 37.000000.  Total average reward 22.000000.\n",
      "Average reward for episode 32.000000.  Total average reward 22.000000.\n",
      "Average reward for episode 33.000000.  Total average reward 22.000000.\n",
      "Average reward for episode 30.000000.  Total average reward 22.000000.\n",
      "Average reward for episode 38.000000.  Total average reward 22.000000.\n",
      "Average reward for episode 45.000000.  Total average reward 23.000000.\n",
      "Average reward for episode 38.000000.  Total average reward 23.000000.\n",
      "Average reward for episode 25.000000.  Total average reward 23.000000.\n",
      "Average reward for episode 36.000000.  Total average reward 23.000000.\n",
      "Average reward for episode 27.000000.  Total average reward 23.000000.\n",
      "Average reward for episode 45.000000.  Total average reward 23.000000.\n",
      "Average reward for episode 36.000000.  Total average reward 23.000000.\n",
      "Average reward for episode 34.000000.  Total average reward 23.000000.\n",
      "Average reward for episode 44.000000.  Total average reward 24.000000.\n",
      "Average reward for episode 40.000000.  Total average reward 24.000000.\n",
      "Average reward for episode 31.000000.  Total average reward 24.000000.\n",
      "Average reward for episode 38.000000.  Total average reward 24.000000.\n",
      "Average reward for episode 45.000000.  Total average reward 24.000000.\n",
      "Average reward for episode 43.000000.  Total average reward 24.000000.\n",
      "Average reward for episode 41.000000.  Total average reward 24.000000.\n",
      "Average reward for episode 36.000000.  Total average reward 25.000000.\n",
      "Average reward for episode 40.000000.  Total average reward 25.000000.\n",
      "Average reward for episode 44.000000.  Total average reward 25.000000.\n",
      "Average reward for episode 50.000000.  Total average reward 25.000000.\n",
      "Average reward for episode 46.000000.  Total average reward 25.000000.\n",
      "Average reward for episode 41.000000.  Total average reward 26.000000.\n",
      "Average reward for episode 43.000000.  Total average reward 26.000000.\n",
      "Average reward for episode 49.000000.  Total average reward 26.000000.\n",
      "Average reward for episode 44.000000.  Total average reward 26.000000.\n",
      "Average reward for episode 43.000000.  Total average reward 26.000000.\n",
      "Average reward for episode 32.000000.  Total average reward 26.000000.\n",
      "Average reward for episode 40.000000.  Total average reward 27.000000.\n",
      "Average reward for episode 40.000000.  Total average reward 27.000000.\n",
      "Average reward for episode 35.000000.  Total average reward 27.000000.\n",
      "Average reward for episode 61.000000.  Total average reward 27.000000.\n",
      "Average reward for episode 43.000000.  Total average reward 27.000000.\n",
      "Average reward for episode 65.000000.  Total average reward 28.000000.\n",
      "Average reward for episode 46.000000.  Total average reward 28.000000.\n",
      "Average reward for episode 37.000000.  Total average reward 28.000000.\n",
      "Average reward for episode 44.000000.  Total average reward 28.000000.\n",
      "Average reward for episode 47.000000.  Total average reward 28.000000.\n",
      "Average reward for episode 56.000000.  Total average reward 28.000000.\n",
      "Average reward for episode 48.000000.  Total average reward 29.000000.\n",
      "Average reward for episode 74.000000.  Total average reward 29.000000.\n",
      "Average reward for episode 50.000000.  Total average reward 29.000000.\n",
      "Average reward for episode 47.000000.  Total average reward 30.000000.\n",
      "Average reward for episode 54.000000.  Total average reward 30.000000.\n",
      "Average reward for episode 51.000000.  Total average reward 30.000000.\n",
      "Average reward for episode 52.000000.  Total average reward 30.000000.\n",
      "Average reward for episode 67.000000.  Total average reward 31.000000.\n",
      "Average reward for episode 76.000000.  Total average reward 31.000000.\n",
      "Average reward for episode 49.000000.  Total average reward 31.000000.\n",
      "Average reward for episode 62.000000.  Total average reward 32.000000.\n",
      "Average reward for episode 50.000000.  Total average reward 32.000000.\n",
      "Average reward for episode 34.000000.  Total average reward 32.000000.\n",
      "Average reward for episode 36.000000.  Total average reward 32.000000.\n",
      "Average reward for episode 69.000000.  Total average reward 32.000000.\n",
      "Average reward for episode 43.000000.  Total average reward 32.000000.\n",
      "Average reward for episode 80.000000.  Total average reward 33.000000.\n",
      "Average reward for episode 59.000000.  Total average reward 33.000000.\n",
      "Average reward for episode 49.000000.  Total average reward 33.000000.\n",
      "Average reward for episode 71.000000.  Total average reward 34.000000.\n",
      "Average reward for episode 37.000000.  Total average reward 34.000000.\n",
      "Average reward for episode 78.000000.  Total average reward 34.000000.\n",
      "Average reward for episode 75.000000.  Total average reward 34.000000.\n",
      "Average reward for episode 56.000000.  Total average reward 35.000000.\n",
      "Average reward for episode 45.000000.  Total average reward 35.000000.\n",
      "Average reward for episode 50.000000.  Total average reward 35.000000.\n",
      "Average reward for episode 61.000000.  Total average reward 35.000000.\n",
      "Average reward for episode 63.000000.  Total average reward 35.000000.\n",
      "Average reward for episode 48.000000.  Total average reward 36.000000.\n",
      "Average reward for episode 59.000000.  Total average reward 36.000000.\n",
      "Average reward for episode 69.000000.  Total average reward 36.000000.\n",
      "Average reward for episode 69.000000.  Total average reward 36.000000.\n",
      "Average reward for episode 54.000000.  Total average reward 37.000000.\n",
      "Average reward for episode 63.000000.  Total average reward 37.000000.\n",
      "Average reward for episode 40.000000.  Total average reward 37.000000.\n",
      "Average reward for episode 59.000000.  Total average reward 37.000000.\n",
      "Average reward for episode 80.000000.  Total average reward 38.000000.\n",
      "Average reward for episode 43.000000.  Total average reward 38.000000.\n",
      "Average reward for episode 71.000000.  Total average reward 38.000000.\n",
      "Average reward for episode 74.000000.  Total average reward 38.000000.\n",
      "Average reward for episode 89.000000.  Total average reward 39.000000.\n",
      "Average reward for episode 82.000000.  Total average reward 39.000000.\n",
      "Average reward for episode 65.000000.  Total average reward 39.000000.\n",
      "Average reward for episode 92.000000.  Total average reward 40.000000.\n",
      "Average reward for episode 47.000000.  Total average reward 40.000000.\n",
      "Average reward for episode 55.000000.  Total average reward 40.000000.\n",
      "Average reward for episode 61.000000.  Total average reward 40.000000.\n",
      "Average reward for episode 89.000000.  Total average reward 41.000000.\n",
      "Average reward for episode 67.000000.  Total average reward 41.000000.\n",
      "Average reward for episode 88.000000.  Total average reward 42.000000.\n",
      "Average reward for episode 42.000000.  Total average reward 42.000000.\n",
      "Average reward for episode 75.000000.  Total average reward 42.000000.\n",
      "Average reward for episode 97.000000.  Total average reward 43.000000.\n",
      "Average reward for episode 116.000000.  Total average reward 43.000000.\n",
      "Average reward for episode 56.000000.  Total average reward 43.000000.\n",
      "Average reward for episode 69.000000.  Total average reward 44.000000.\n",
      "Average reward for episode 89.000000.  Total average reward 44.000000.\n",
      "Average reward for episode 83.000000.  Total average reward 44.000000.\n",
      "Average reward for episode 69.000000.  Total average reward 45.000000.\n",
      "Average reward for episode 80.000000.  Total average reward 45.000000.\n",
      "Average reward for episode 79.000000.  Total average reward 45.000000.\n",
      "Average reward for episode 88.000000.  Total average reward 46.000000.\n",
      "Average reward for episode 106.000000.  Total average reward 46.000000.\n",
      "Average reward for episode 79.000000.  Total average reward 47.000000.\n",
      "Average reward for episode 97.000000.  Total average reward 47.000000.\n",
      "Average reward for episode 81.000000.  Total average reward 48.000000.\n",
      "Average reward for episode 78.000000.  Total average reward 48.000000.\n",
      "Average reward for episode 101.000000.  Total average reward 48.000000.\n",
      "Average reward for episode 105.000000.  Total average reward 49.000000.\n",
      "Average reward for episode 94.000000.  Total average reward 49.000000.\n",
      "Average reward for episode 78.000000.  Total average reward 50.000000.\n",
      "Average reward for episode 136.000000.  Total average reward 51.000000.\n",
      "Average reward for episode 103.000000.  Total average reward 51.000000.\n",
      "Average reward for episode 75.000000.  Total average reward 51.000000.\n",
      "Average reward for episode 98.000000.  Total average reward 52.000000.\n",
      "Average reward for episode 117.000000.  Total average reward 53.000000.\n",
      "Average reward for episode 129.000000.  Total average reward 53.000000.\n",
      "Average reward for episode 122.000000.  Total average reward 54.000000.\n",
      "Average reward for episode 134.000000.  Total average reward 55.000000.\n",
      "Average reward for episode 82.000000.  Total average reward 55.000000.\n",
      "Average reward for episode 106.000000.  Total average reward 56.000000.\n",
      "Average reward for episode 137.000000.  Total average reward 56.000000.\n",
      "Average reward for episode 116.000000.  Total average reward 57.000000.\n",
      "Average reward for episode 152.000000.  Total average reward 58.000000.\n",
      "Average reward for episode 126.000000.  Total average reward 59.000000.\n",
      "Average reward for episode 149.000000.  Total average reward 59.000000.\n",
      "Average reward for episode 108.000000.  Total average reward 60.000000.\n",
      "Average reward for episode 133.000000.  Total average reward 61.000000.\n",
      "Average reward for episode 122.000000.  Total average reward 61.000000.\n",
      "Average reward for episode 95.000000.  Total average reward 62.000000.\n",
      "Average reward for episode 168.000000.  Total average reward 63.000000.\n",
      "Average reward for episode 123.000000.  Total average reward 63.000000.\n",
      "Average reward for episode 101.000000.  Total average reward 64.000000.\n",
      "Average reward for episode 145.000000.  Total average reward 64.000000.\n",
      "Average reward for episode 159.000000.  Total average reward 65.000000.\n",
      "Average reward for episode 140.000000.  Total average reward 66.000000.\n"
     ]
    }
   ],
   "source": [
    "xs,hs,dlogps,drs,ys,tfps = [],[],[],[],[],[]\n",
    "running_reward = None\n",
    "reward_sum = 0\n",
    "episode_number = 1\n",
    "total_episodes = 10000\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "# Launch the graph\n",
    "with tf.Session() as sess:\n",
    "    rendering = False\n",
    "    sess.run(init)\n",
    "    observation = env.reset() # Obtain an initial observation of the environment\n",
    "\n",
    "    # Reset the gradient placeholder. We will collect gradients in \n",
    "    # gradBuffer until we are ready to update our policy network. \n",
    "    gradBuffer = sess.run(tvars)\n",
    "    for ix,grad in enumerate(gradBuffer):\n",
    "        gradBuffer[ix] = grad * 0\n",
    "    \n",
    "    while episode_number <= total_episodes:\n",
    "        \n",
    "        # Rendering the environment slows things down, \n",
    "        # so let's only look at it once our agent is doing a good job.\n",
    "        if reward_sum/batch_size > 100 or rendering == True : \n",
    "            env.render()\n",
    "            rendering = True\n",
    "            \n",
    "        # Make sure the observation is in a shape the network can handle.\n",
    "        x = np.reshape(observation,[1,D])\n",
    "        \n",
    "        # Run the policy network and get an action to take. \n",
    "        tfprob = sess.run(probability,feed_dict={observations: x})\n",
    "        action = 1 if np.random.uniform() < tfprob else 0\n",
    "        \n",
    "        xs.append(x) # observation\n",
    "        y = 1 if action == 0 else 0 # a \"fake label\"\n",
    "        ys.append(y)\n",
    "\n",
    "        # step the environment and get new measurements\n",
    "        observation, reward, done, info = env.step(action)\n",
    "        reward_sum += reward\n",
    "\n",
    "        drs.append(reward) # record reward (has to be done after we call step() to get reward for previous action)\n",
    "\n",
    "        if done: \n",
    "            episode_number += 1\n",
    "            # stack together all inputs, hidden states, action gradients, and rewards for this episode\n",
    "            epx = np.vstack(xs)\n",
    "            epy = np.vstack(ys)\n",
    "            epr = np.vstack(drs)\n",
    "            tfp = tfps\n",
    "            xs,hs,dlogps,drs,ys,tfps = [],[],[],[],[],[] # reset array memory\n",
    "\n",
    "            # compute the discounted reward backwards through time\n",
    "            discounted_epr = discount_rewards(epr)\n",
    "            # size the rewards to be unit normal (helps control the gradient estimator variance)\n",
    "            discounted_epr -= np.mean(discounted_epr)\n",
    "            discounted_epr //= np.std(discounted_epr)\n",
    "            \n",
    "            # Get the gradient for this episode, and save it in the gradBuffer\n",
    "            tGrad = sess.run(newGrads,feed_dict={observations: epx, input_y: epy, advantages: discounted_epr})\n",
    "            for ix,grad in enumerate(tGrad):\n",
    "                gradBuffer[ix] += grad\n",
    "                \n",
    "            # If we have completed enough episodes, then update the policy network with our gradients.\n",
    "            if episode_number % batch_size == 0: \n",
    "                sess.run(updateGrads,feed_dict={W1Grad: gradBuffer[0],W2Grad:gradBuffer[1]})\n",
    "                for ix,grad in enumerate(gradBuffer):\n",
    "                    gradBuffer[ix] = grad * 0\n",
    "                \n",
    "                # Give a summary of how well our network is doing for each batch of episodes.\n",
    "                running_reward = reward_sum if running_reward is None else running_reward * 0.99 + reward_sum * 0.01\n",
    "                print('Average reward for episode %f.  Total average reward %f.' % (reward_sum//batch_size, running_reward//batch_size))\n",
    "                \n",
    "                if reward_sum//batch_size > 200: \n",
    "                    print(\"Task solved in\",episode_number,'episodes!')\n",
    "                    break\n",
    "                    \n",
    "                reward_sum = 0\n",
    "            \n",
    "            observation = env.reset()\n",
    "        \n",
    "print(episode_number,'Episodes completed.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the network not only does much better than random actions, but achieves the goal of 200 points per episode, thus solving the task!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:py2]",
   "language": "python",
   "name": "conda-env-py2-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}