pratical-rl/week6_outro/practice_mcts.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Seminar: Monte-carlo tree search\n",
    "\n",
    "In this seminar, we'll implement a vanilla MCTS planning and use it to solve some Gym envs.\n",
    "\n",
    "But before we do that, we first need to modify gym env to allow saving and loading game states to facilitate backtracking."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gym.core import Wrapper\n",
    "from pickle import dumps,loads\n",
    "from collections import namedtuple\n",
    "\n",
    "#a container for get_result function below. Works just like tuple, but prettier\n",
    "ActionResult = namedtuple(\"action_result\",(\"snapshot\",\"observation\",\"reward\",\"is_done\",\"info\"))\n",
    "\n",
    "\n",
    "class WithSnapshots(Wrapper):\n",
    "    \"\"\"\n",
    "    Creates a wrapper that supports saving and loading environemnt states.\n",
    "    Required for planning algorithms.\n",
    "\n",
    "    This class will have access to the core environment as self.env, e.g.:\n",
    "    - self.env.reset()           #reset original env\n",
    "    - self.env.ale.cloneState()  #make snapshot for atari. load with .restoreState()\n",
    "    - ...\n",
    "\n",
    "    You can also use reset, step and render directly for convenience.\n",
    "    - s, r, done, _ = self.step(action)   #step, same as self.env.step(action)\n",
    "    - self.render(close=True)             #close window, same as self.env.render(close=True)\n",
    "    \"\"\"\n",
    "\n",
    "        \n",
    "    def get_snapshot(self):\n",
    "        \"\"\"\n",
    "        :returns: environment state that can be loaded with load_snapshot \n",
    "        Snapshots guarantee same env behaviour each time they are loaded.\n",
    "        \n",
    "        Warning! Snapshots can be arbitrary things (strings, integers, json, tuples)\n",
    "        Don't count on them being pickle strings when implementing MCTS.\n",
    "        \n",
    "        Developer Note: Make sure the object you return will not be affected by \n",
    "        anything that happens to the environment after it's saved.\n",
    "        You shouldn't, for example, return self.env. \n",
    "        In case of doubt, use pickle.dumps or deepcopy.\n",
    "        \n",
    "        \"\"\"\n",
    "        self.render() #close popup windows since we can't pickle them\n",
    "        if self.unwrapped.viewer is not None:\n",
    "            self.unwrapped.viewer.close()\n",
    "            self.unwrapped.viewer = None\n",
    "        return dumps(self.env)\n",
    "    \n",
    "    def load_snapshot(self,snapshot):\n",
    "        \"\"\"\n",
    "        Loads snapshot as current env state.\n",
    "        Should not change snapshot inplace (in case of doubt, deepcopy).\n",
    "        \"\"\"\n",
    "        \n",
    "        assert not hasattr(self,\"_monitor\") or hasattr(self.env,\"_monitor\"), \"can't backtrack while recording\"\n",
    "        \n",
    "        self.render()\n",
    "#         self.render(close=True) #close popup windows since we can't load into them\n",
    "        self.env = loads(snapshot)\n",
    "    \n",
    "    def get_result(self,snapshot,action):\n",
    "        \"\"\"\n",
    "        A convenience function that \n",
    "        - loads snapshot, \n",
    "        - commits action via self.step,\n",
    "        - and takes snapshot again :)\n",
    "        \n",
    "        :returns: next snapshot, next_observation, reward, is_done, info\n",
    "        \n",
    "        Basically it returns next snapshot and everything that env.step would have returned.\n",
    "        \"\"\"\n",
    "        \n",
    "#         <your code here load,commit,take snapshot>\n",
    "        self.load_snapshot(snapshot)\n",
    "        s, r, done, info = self.step(action)\n",
    "        next_snapshot = self.get_snapshot()\n",
    "        \n",
    "        return ActionResult(next_snapshot,    #fill in the variables\n",
    "                            s, \n",
    "                            r, done, info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### try out snapshots:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#make env\n",
    "env = WithSnapshots(gym.make(\"CartPole-v0\"))\n",
    "env.reset()\n",
    "\n",
    "n_actions = env.action_space.n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "initial_state:\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAD8CAYAAACW/ATfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAEztJREFUeJzt3XGsnfV93/H3ZziQlHbY0DvLsy1BFCsITSo4V5lRqimLlxRYFDMpRaBouMzTjTa2JWNSa5Y/pkr7I2xTaZAmghXSORUloTQZFmJNmSGa9kdoLgklBEK5UKhtGXyhQLawdmX97o/zMxxuDfcc+1yO7/29X9LR+T2/5/ec5/fTc/3xc3/3eZ6TqkKStLb9jWl3QJK08gx7SeqAYS9JHTDsJakDhr0kdcCwl6QOrEjYJ7ksyZNJFpLsXYl9SJJGl0lfZ5/kDOCPgY8Dh4HvAddU1eMT3ZEkaWQrcWb/YWChqp6pqv8LfB3YtQL7kSSNaN0KfOZm4NDQ8mHg7y5tlGQOmAM4++yzP3ThhReuQFckaXV69tlnefHFFzOpz1uJsB9JVe0D9gHMzs7W/Pz8tLoiSaed2dnZiX7eSkzjHAG2Di1vaXWSpClZibD/HrAtyQVJzgSuBg6swH4kSSOa+DROVb2e5F8A3wbOAL5aVT+a9H4kSaNbkTn7qroPuG8lPluSND7voJWkDhj2ktQBw16SOmDYS1IHDHtJ6oBhL0kdMOwlqQOGvSR1wLCXpA4Y9pLUAcNekjpg2EtSBwx7SeqAYS9JHTDsJakDhr0kdcCwl6QOGPaS1IFlwz7JV5McS/LYUN25Se5P8lR739Dqk+SWJAtJHk2yfSU7L0kazShn9v8FuGxJ3V7gYFVtAw62ZYDLgW3tNQfcOpluSpJOxbJhX1X/A/izJdW7gP2tvB+4cqj+azXwXWB9kk2T6qwk6eSc7Jz9xqo62srPAxtbeTNwaKjd4VYnSZqiU/4DbVUVUONul2QuyXyS+cXFxVPthiTpHZxs2L9wfHqmvR9r9UeArUPttrS6v6aq9lXVbFXNzszMnGQ3JEmjONmwPwDsbuXdwD1D9de2q3J2AK8OTfdIkqZk3XINktwJfBT4+SSHgX8HfBG4K8ke4Dngqtb8PuAKYAF4DbhuBfosSRrTsmFfVde8zaqdJ2hbwPWn2ilJ0mR5B60kdcCwl6QOGPaS1AHDXpI6YNhLUgcMe0nqgGEvSR0w7CWpA4a9JHXAsJekDhj2ktQBw16SOmDYS1IHDHtJ6oBhL0kdMOwlqQOGvSR1wLCXpA4Y9pLUgWXDPsnWJA8meTzJj5J8rtWfm+T+JE+19w2tPkluSbKQ5NEk21d6EJKkdzbKmf3rwL+pqouAHcD1SS4C9gIHq2obcLAtA1wObGuvOeDWifdakjSWZcO+qo5W1fdb+X8BTwCbgV3A/tZsP3BlK+8CvlYD3wXWJ9k08Z5LkkY21px9kvOBS4CHgI1VdbSteh7Y2MqbgUNDmx1udUs/ay7JfJL5xcXFMbstSRrHyGGf5GeB3wM+X1U/GV5XVQXUODuuqn1VNVtVszMzM+NsKkka00hhn+Q9DIL+jqr6Zqt+4fj0THs/1uqPAFuHNt/S6iRJUzLK1TgBbgeeqKrfGFp1ANjdyruBe4bqr21X5ewAXh2a7pEkTcG6Edp8BPjHwA+TPNLq/i3wReCuJHuA54Cr2rr7gCuABeA14LqJ9liSNLZlw76q/ieQt1m98wTtC7j+FPslSZog76CVpA4Y9pLUAcNekjpg2EtSBwx7SeqAYS9JHTDsJakDhr0kdcCwl6QOGPaS1AHDXpI6YNhLUgcMe0nqgGEvSR0w7CWpA4a9JHXAsJekDhj2ktSBUb5w/L1J/jDJHyX5UZJfb/UXJHkoyUKSbyQ5s9Wf1ZYX2vrzV3YIkqTljHJm/xfAx6rqF4CLgcuS7ABuAm6uqg8ALwN7Wvs9wMut/ubWTpI0RcuGfQ3877b4nvYq4GPA3a1+P3BlK+9qy7T1O5O83ReWS5LeBSPN2Sc5I8kjwDHgfuBp4JWqer01OQxsbuXNwCGAtv5V4LwTfOZckvkk84uLi6c2CknSOxop7Kvq/1XVxcAW4MPAhae646raV1WzVTU7MzNzqh8nSXoHY12NU1WvAA8ClwLrk6xrq7YAR1r5CLAVoK0/B3hpIr2VJJ2UUa7GmUmyvpXfB3wceIJB6H+6NdsN3NPKB9oybf0DVVWT7LQkaTzrlm/CJmB/kjMY/OdwV1Xdm+Rx4OtJ/j3wA+D21v524LeTLAB/Bly9Av2WJI1h2bCvqkeBS05Q/wyD+ful9X8O/PJEeidJmgjvoJWkDhj2ktQBw16SOmDYS1IHDHtJ6oBhL0kdGOU6e2lNenjfZ/9a3YfmbptCT6SV55m9JHXAsJekDhj2ktQBw16SOmDYS1IHDHtJ6oBhL0kdMOylISe69l5aCwx7SeqAYS9JHTDsJakDI4d9kjOS/CDJvW35giQPJVlI8o0kZ7b6s9ryQlt//sp0XZI0qnHO7D8HPDG0fBNwc1V9AHgZ2NPq9wAvt/qbWztJ0hSNFPZJtgD/EPhKWw7wMeDu1mQ/cGUr72rLtPU7W3vptOITLtWTUc/sfxP4VeCv2vJ5wCtV9XpbPgxsbuXNwCGAtv7V1v4tkswlmU8yv7i4eJLdlySNYtmwT/JJ4FhVPTzJHVfVvqqararZmZmZSX60JGmJUb685CPAp5JcAbwX+JvAl4D1Sda1s/ctwJHW/giwFTicZB1wDvDSxHsuSRrZsmf2VXVjVW2pqvOBq4EHquozwIPAp1uz3cA9rXygLdPWP1BVNdFeS5LGcirX2f8acEOSBQZz8re3+tuB81r9DcDeU+uiJOlUjfUdtFX1HeA7rfwM8OETtPlz4Jcn0DdJ0oR4B60kdcCwl6QOGPaS1AHDXpI6YNhLUgcMe2kJv61Ka5FhL0kdMOwlqQOGvSR1wLCXpA4Y9pLUAcNekjpg2EtSBwx7SeqAYS9JHTDsJakDhr0kdcCwV9c+NHfbtLsgvStGCvskzyb5YZJHksy3unOT3J/kqfa+odUnyS1JFpI8mmT7Sg5AkrS8cc7s/35VXVxVs215L3CwqrYBB3nzi8UvB7a11xxw66Q6K0k6OacyjbML2N/K+4Erh+q/VgPfBdYn2XQK+5EknaJRw76AP0jycJK5Vrexqo628vPAxlbeDBwa2vZwq3uLJHNJ5pPMLy4unkTXJUmjWjdiu1+sqiNJ/hZwf5IfD6+sqkpS4+y4qvYB+wBmZ2fH2laSNJ6Rzuyr6kh7PwZ8C/gw8MLx6Zn2fqw1PwJsHdp8S6uTJE3JsmGf5OwkP3e8DHwCeAw4AOxuzXYD97TyAeDadlXODuDVoekeSdIUjDKNsxH4VpLj7X+nqn4/yfeAu5LsAZ4Drmrt7wOuABaA14DrJt5rSdJYlg37qnoG+IUT1L8E7DxBfQHXT6R3kqSJ8A5aSeqAYS9JHTDsJakDhr10Ag/v++y0uyBNlGEvSR0w7CWpA4a9JHXAsJekDhj2ktQBw17d86sJ1QPDXpI6YNhLUgcMe0nqgGGvNSvJyK+V2F46nRj2ktSBUb+DVlrz7j0690b5k5v2TbEn0uR5Zi/x1qA/0bK02hn2ktSBkcI+yfokdyf5cZInklya5Nwk9yd5qr1vaG2T5JYkC0keTbJ9ZYcgSVrOqGf2XwJ+v6ouZPB9tE8Ae4GDVbUNONiWAS4HtrXXHHDrRHssrYClc/TO2WutyeD7wd+hQXIO8Ajw/hpqnORJ4KNVdTTJJuA7VfXBJLe18p1L273dPmZnZ2t+fn4Cw5He9G5eErncvyNpXLOzs8zPz0/sh3iUM/sLgEXgt5L8IMlXkpwNbBwK8OeBja28GTg0tP3hVidJmpJRwn4dsB24taouAX7Km1M2ALQz/rFObZLMJZlPMr+4uDjOppKkMY0S9oeBw1X1UFu+m0H4v9Cmb2jvx9r6I8DWoe23tLq3qKp9VTVbVbMzMzMn239J0giWDfuqeh44lOSDrWon8DhwANjd6nYD97TyAeDadlXODuDVd5qvlyStvFHvoP2XwB1JzgSeAa5j8B/FXUn2AM8BV7W29wFXAAvAa62tJGmKRgr7qnoEmD3Bqp0naFvA9afYL0nSBHkHrSR1wLCXpA4Y9pLUAR9xrDXLu1qlN3lmL0kdMOwlqQOGvSR1wLCXpA4Y9pLUAcNekjpg2EtSBwx7SeqAYS9JHTDsJakDhr0kdcCwl6QOGPaS1AHDXpI6sGzYJ/lgkkeGXj9J8vkk5ya5P8lT7X1Da58ktyRZSPJoku0rPwxJ0jtZNuyr6smquriqLgY+xOBLxL8F7AUOVtU24GBbBrgc2NZec8CtK9FxSdLoxp3G2Qk8XVXPAbuA/a1+P3BlK+8CvlYD3wXWJ9k0kd5Kkk7KuGF/NXBnK2+sqqOt/DywsZU3A4eGtjnc6iRJUzJy2Cc5E/gU8LtL19Xg+9/G+g64JHNJ5pPMLy4ujrOpJGlM45zZXw58v6peaMsvHJ+eae/HWv0RYOvQdlta3VtU1b6qmq2q2ZmZmfF7Lkka2Thhfw1vTuEAHAB2t/Ju4J6h+mvbVTk7gFeHpnskSVOwbpRGSc4GPg58dqj6i8BdSfYAzwFXtfr7gCuABQZX7lw3sd5Kkk7KSGFfVT8FzltS9xKDq3OWti3g+on0TpI0Ed5BK0kdMOwlqQOGvSR1wLCXpA4Y9pLUAcNekjpg2EtSBwx7SeqAYS9JHTDsJakDhr0kdcCwl6QOGPaS1AHDXpI6YNhLUgcMe0nqgGEvSR0w7CWpA4a9JHVgpLBP8q+T/CjJY0nuTPLeJBckeSjJQpJvJDmztT2rLS+09eev5AAkSctbNuyTbAb+FTBbVX8HOAO4GrgJuLmqPgC8DOxpm+wBXm71N7d2kqQpGnUaZx3wviTrgJ8BjgIfA+5u6/cDV7byrrZMW78zSSbTXUnSyVi3XIOqOpLkPwF/Cvwf4A+Ah4FXqur11uwwsLmVNwOH2ravJ3kVOA94cfhzk8wBc23xL5I8dopjOV39PEvGvkY4rtVnrY5trY7rg5P8sGXDPskGBmfrFwCvAL8LXHaqO66qfcC+to/5qpo91c88Ha3VsTmu1Wetjm0tj2uSnzfKNM4/AP6kqhar6i+BbwIfAda3aR2ALcCRVj4CbG2dXQecA7w0yU5LksYzStj/KbAjyc+0ufedwOPAg8CnW5vdwD2tfKAt09Y/UFU1uS5Lksa1bNhX1UMM/tD6feCHbZt9wK8BNyRZYDAnf3vb5HbgvFZ/A7B3hH7sG7/rq8ZaHZvjWn3W6tgc1wjiSbckrX3eQStJHTDsJakDUw/7JJclebI9XmGU+f3TRpKtSR5M8nh7nMTnWv25Se5P8lR739Dqk+SWNtZHk2yf7gjeWZIzkvwgyb1teU08IiPJ+iR3J/lxkieSXLoWjtlaeqxJkq8mOTZ8/83JHKMku1v7p5LsPtG+3k1vM67/2H4WH03yrSTrh9bd2Mb1ZJJfGqofPzeramovBo9eeBp4P3Am8EfARdPs05j93wRsb+WfA/4YuAj4D8DeVr8XuKmVrwD+GxBgB/DQtMewzPhuAH4HuLct3wVc3cpfBv5ZK/9z4MutfDXwjWn3fZlx7Qf+aSufCaxf7ceMwc2MfwK8b+hY/cpqPWbA3wO2A48N1Y11jIBzgWfa+4ZW3nAajusTwLpWvmloXBe1TDyLwX1OT7fMPKncnPYBvRT49tDyjcCN0/5BO4Xx3AN8HHgS2NTqNgFPtvJtwDVD7d9od7q9GNw7cZDBYzHubf+QXhz6oXzj2AHfBi5t5XWtXaY9hrcZ1zktFLOkflUfM968c/3cdgzuBX5pNR8z4PwloTjWMQKuAW4bqn9Lu9NlXEvW/SPgjlZ+Sx4eP2Ynm5vTnsZ549EKzfBjF1aV9mvwJcBDwMaqOtpWPQ9sbOXVNN7fBH4V+Ku2fB4jPiIDOP6IjNPRBcAi8FttiuorSc5mlR+zqjoCHH+syVEGx2Dkx5pweh+z48Y9Rqvi2C3xTxj8lgITHte0w35NSPKzwO8Bn6+qnwyvq8F/vavq+tYknwSOVdXD0+7LCljH4NfoW6vqEuCnLLkXZJUes+HHmvxt4Gwm8FiT09VqPEbLSfIF4HXgjpX4/GmH/RuPVmiGH7uwKiR5D4Ogv6OqvtmqX0iyqa3fBBxr9atlvB8BPpXkWeDrDKZyvsTaeETGYeBwDW4WhMENg9tZ/cesh8eajHuMVsuxI8mvAJ8EPtP+I4MJj2vaYf89YFu7YuBMBn8oOjDlPo0sSRjcMfxEVf3G0KrhR0YsfZTEte3qgR3Aq0O/lp42qurGqtpSVeczOCYPVNVnWAOPyKiq54FDSY4/UfD44z9W9TGjj8eajHuMvg18IsmG9pvPJ1rdaSXJZQymTD9VVa8NrToAXN2unLoA2Ab8ISebm6fBHyuuYHAVy9PAF6bdnzH7/osMfpV8FHikva5gMPd5EHgK+O/Aua19gP/cxvpDBl8IM/VxLDPGj/Lm1Tjvbz9sCwyefnpWq39vW15o698/7X4vM6aLgfl23P4rgys1Vv0xA34d+DHwGPDbDK7iWJXHDLiTwd8e/pLBb2N7TuYYMZgDX2iv607TcS0wmIM/niFfHmr/hTauJ4HLh+rHzk0flyBJHZj2NI4k6V1g2EtSBwx7SeqAYS9JHTDsJakDhr0kdcCwl6QO/H+kh6imSxaxygAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"initial_state:\")\n",
    "\n",
    "plt.imshow(env.render('rgb_array'))\n",
    "\n",
    "#create first snapshot\n",
    "snap0 = env.get_snapshot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Whoops! We died!\n",
      "final state:\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAD8CAYAAACW/ATfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAE3FJREFUeJzt3X+s3fV93/HnaziQlHbYUM/ybEsQxQpCkwrOVWaUqsriJQUWxUxKESgaDvN0o41tyZjUmOWPpdL+CNtUGqSJxArpTEWTUJrMFmJNmSGa9gc0l4QSAqFcKNS2DL5QIFtYm7K+98f5GA6u4Z7je26O7/08H9LR+Xw/38855/PR137d7/l8f5xUFZKk1e1vTbsDkqTlZ9hLUgcMe0nqgGEvSR0w7CWpA4a9JHVgWcI+yWVJnkgyn2TPcnyGJGl0mfR59knOAP4E+DBwGPgucE1VPTbRD5IkjWw59uzfD8xX1dNV9VPg68DOZfgcSdKI1izDe24CDg0tHwb+/omNkswCswBnn332+y688MJl6IokrUzPPPMML7zwQib1fssR9iOpqr3AXoCZmZmam5ubVlck6bQzMzMz0fdbjmmcI8CWoeXNrU6SNCXLEfbfBbYmuSDJmcDVwIFl+BxJ0ogmPo1TVa8l+ZfAt4EzgK9W1Q8n/TmSpNEty5x9Vd0D3LMc7y1JGp9X0EpSBwx7SeqAYS9JHTDsJakDhr0kdcCwl6QOGPaS1AHDXpI6YNhLUgcMe0nqgGEvSR0w7CWpA4a9JHXAsJekDhj2ktQBw16SOmDYS1IHDHtJ6sCiYZ/kq0mOJXl0qO7cJPcmebI9r2v1SXJLkvkkjyTZtpydlySNZpQ9+/8KXHZC3R7gYFVtBQ62ZYDLga3tMQvcOpluSpKWYtGwr6r/Cfz5CdU7gX2tvA+4cqj+9hp4AFibZOOkOitJOjWnOme/oaqOtvJzwIZW3gQcGmp3uNVJkqZoyQdoq6qAGvd1SWaTzCWZW1hYWGo3JElv41TD/vnj0zPt+VirPwJsGWq3udX9DVW1t6pmqmpm/fr1p9gNSdIoTjXsDwC7WnkXsH+o/tp2Vs524JWh6R5J0pSsWaxBkq8BHwR+Mclh4N8DXwDuTLIbeBa4qjW/B7gCmAdeBa5bhj5Lksa0aNhX1TVvsWrHSdoWcP1SOyVJmiyvoJWkDhj2ktQBw16SOmDYS1IHDHtJ6oBhL0kdMOwlqQOGvSR1wLCXpA4Y9pLUAcNekjpg2EtSBwx7SeqAYS9JHTDsJakDhr0kdcCwl6QOGPaS1AHDXpI6sGjYJ9mS5P4kjyX5YZJPt/pzk9yb5Mn2vK7VJ8ktSeaTPJJk23IPQpL09kbZs38N+LdVdRGwHbg+yUXAHuBgVW0FDrZlgMuBre0xC9w68V5LksayaNhX1dGq+l4r/2/gcWATsBPY15rtA65s5Z3A7TXwALA2ycaJ91ySNLKx5uyTnA9cAjwIbKiqo23Vc8CGVt4EHBp62eFWd+J7zSaZSzK3sLAwZrclSeMYOeyT/Dzw+8BnqurHw+uqqoAa54Oram9VzVTVzPr168d5qSRpTCOFfZJ3MAj6O6rqm636+ePTM+35WKs/AmwZevnmVidJmpJRzsYJcBvweFX95tCqA8CuVt4F7B+qv7adlbMdeGVoukeSNAVrRmjzAeCfAD9I8nCr+3fAF4A7k+wGngWuauvuAa4A5oFXgesm2mNJ0tgWDfuq+l9A3mL1jpO0L+D6JfZLkjRBXkErSR0w7CWpA4a9JHXAsJekDhj2ktQBw16SOmDYS1IHDHtJ6oBhL0kdMOwlqQOGvSR1wLCXpA4Y9pLUAcNekjpg2EtSBwx7SeqAYS9JHTDsJakDo/zg+DuT/FGSP07ywyS/0eovSPJgkvkk30hyZqs/qy3Pt/XnL+8QJEmLGWXP/i+BD1XVLwEXA5cl2Q7cBNxcVe8BXgJ2t/a7gZda/c2tnSRpihYN+xr4P23xHe1RwIeAu1r9PuDKVt7ZlmnrdyR5qx8slyT9DIw0Z5/kjCQPA8eAe4GngJer6rXW5DCwqZU3AYcA2vpXgPNO8p6zSeaSzC0sLCxtFJKktzVS2FfV/6uqi4HNwPuBC5f6wVW1t6pmqmpm/fr1S307SdLbGOtsnKp6GbgfuBRYm2RNW7UZONLKR4AtAG39OcCLE+mtJOmUjHI2zvoka1v5XcCHgccZhP7HW7NdwP5WPtCWaevvq6qaZKclSeNZs3gTNgL7kpzB4I/DnVV1d5LHgK8n+Q/A94HbWvvbgN9JMg/8OXD1MvRbkjSGRcO+qh4BLjlJ/dMM5u9PrP8L4Ncm0jtJ0kR4Ba0kdcCwl6QOGPaS1AHDXpI6YNhLUgcMe0nqgGEvSR0Y5aIqqSsP7f3U6+X3zX55ij2RJsewl96Gwa/VwmkcaQQGvVY6w16SOmDYS0OGp22k1cSwl6QOGPaS1AHDXlqEB2e1Ghj2ktQBw15qPDir1cywl6QOjBz2Sc5I8v0kd7flC5I8mGQ+yTeSnNnqz2rL8239+cvTdUnSqMbZs/808PjQ8k3AzVX1HuAlYHer3w281Opvbu2kFcmDs1otRgr7JJuBfwR8pS0H+BBwV2uyD7iylXe2Zdr6Ha29dNpyvl6r3ah79r8F/Drw1235PODlqnqtLR8GNrXyJuAQQFv/Smv/Jklmk8wlmVtYWDjF7kuSRrFo2Cf5KHCsqh6a5AdX1d6qmqmqmfXr10/yraWJcApHq8kotzj+APCxJFcA7wT+NvBFYG2SNW3vfTNwpLU/AmwBDidZA5wDvDjxnkuSRrbonn1V3VhVm6vqfOBq4L6q+gRwP/Dx1mwXsL+VD7Rl2vr7qqom2mtJ0liWcp79Z4EbkswzmJO/rdXfBpzX6m8A9iyti9Ly8uCsejDWL1VV1XeA77Ty08D7T9LmL4Bfm0DfJEkT4hW00kl4cFarjWEvSR0w7NU15+vVC8Nekjpg2EtSBwx76QQenNVqZNhLUgcMe3XLg7PqiWEvSR0w7KUhztdrtTLsJakDhr0kdcCwV5c8OKveGPaS1AHDXmo8OKvVzLCXpA4Y9uqO8/XqkWEvSR0YKeyTPJPkB0keTjLX6s5Ncm+SJ9vzulafJLckmU/ySJJtyzkASdLixtmz/wdVdXFVzbTlPcDBqtoKHOSNHxa/HNjaHrPArZPqrLRcPDir1W4p0zg7gX2tvA+4cqj+9hp4AFibZOMSPkeStESjhn0Bf5jkoSSzrW5DVR1t5eeADa28CTg09NrDre5NkswmmUsyt7CwcApdl8bnwVn1as2I7X65qo4k+TvAvUl+NLyyqipJjfPBVbUX2AswMzMz1mslSeMZac++qo6052PAt4D3A88fn55pz8da8yPAlqGXb251kqQpWTTsk5yd5BeOl4GPAI8CB4BdrdkuYH8rHwCubWflbAdeGZrukU47HpxVD0aZxtkAfCvJ8fa/W1V/kOS7wJ1JdgPPAle19vcAVwDzwKvAdRPvtSRpLIuGfVU9DfzSSepfBHacpL6A6yfSO2mCPDirnnkFrbrmFI56YdhLUgcMe3XBKRz1zrCXpA4Y9pLUAcNe3fLgrHpi2EtSBwx7rXoenJUMe0nqgmEvSR0w7NUlD86qN4a9JHXAsNeq5sFZacCw14qSZKzHUt5HWk0Me3Vn5lN7p90F6Wdu1N+glVaku4/Ovmn5oxsNevXJPXutWp///NzfqDsx/KVeGPaS1IGRwj7J2iR3JflRkseTXJrk3CT3JnmyPa9rbZPkliTzSR5Jsm15hyBJWsyoe/ZfBP6gqi5k8Hu0jwN7gINVtRU42JYBLge2tscscOtEeyyN6GTz85///MwUeiJN36IHaJOcA/wK8EmAqvop8NMkO4EPtmb7gO8AnwV2Are3Hx5/oH0r2FhVRyfee+ltDM668YCsBKPt2V8ALAC/neT7Sb6S5Gxgw1CAPwdsaOVNwKGh1x9udZKkKRkl7NcA24Bbq+oS4Ce8MWUDQNuLr3E+OMlskrkkcwsLC+O8VJI0plHC/jBwuKoebMt3MQj/55NsBGjPx9r6I8CWoddvbnVvUlV7q2qmqmbWr19/qv2XJI1g0bCvqueAQ0ne26p2AI8BB4BdrW4XsL+VDwDXtrNytgOvOF8vSdM16hW0/wq4I8mZwNPAdQz+UNyZZDfwLHBVa3sPcAUwD7za2kqSpmiksK+qh4GTnbO24yRtC7h+if2SJE2QV9BKUgcMe0nqgGEvSR3wFsdaUQaHhCSNyz17SeqAYS9JHTDsJakDhr0kdcCwl6QOGPaS1AHDXpI6YNhLUgcMe0nqgGEvSR0w7CWpA4a9JHXAsJekDhj2ktSBRcM+yXuTPDz0+HGSzyQ5N8m9SZ5sz+ta+yS5Jcl8kkeSbFv+YUiS3s6iYV9VT1TVxVV1MfA+Bj8i/i1gD3CwqrYCB9sywOXA1vaYBW5djo5LkkY37jTODuCpqnoW2Ansa/X7gCtbeSdwew08AKxNsnEivZUknZJxw/5q4GutvKGqjrbyc8CGVt4EHBp6zeFWJ0makpHDPsmZwMeA3ztxXQ1+K26s34tLMptkLsncwsLCOC+VJI1pnD37y4HvVdXzbfn549Mz7flYqz8CbBl63eZW9yZVtbeqZqpqZv369eP3XJI0snHC/hremMIBOADsauVdwP6h+mvbWTnbgVeGpnskSVOwZpRGSc4GPgx8aqj6C8CdSXYDzwJXtfp7gCuAeQZn7lw3sd5Kkk7JSGFfVT8Bzjuh7kUGZ+ec2LaA6yfSO0nSRHgFrSR1wLCXpA4Y9pLUAcNekjpg2EtSBwx7SeqAYS9JHTDsJakDhr0kdcCwl6QOGPaS1AHDXpI6YNhLUgcMe0nqgGEvSR0w7CWpA4a9JHXAsJekDhj2ktSBkcI+yb9J8sMkjyb5WpJ3JrkgyYNJ5pN8I8mZre1ZbXm+rT9/OQcgSVrcomGfZBPwr4GZqvp7wBnA1cBNwM1V9R7gJWB3e8lu4KVWf3NrJ0maolGncdYA70qyBvg54CjwIeCutn4fcGUr72zLtPU7kmQy3ZUknYo1izWoqiNJ/jPwZ8D/Bf4QeAh4uapea80OA5taeRNwqL32tSSvAOcBLwy/b5JZYLYt/mWSR5c4ltPVL3LC2FcJx7XyrNaxrdZxvXeSb7Zo2CdZx2Bv/QLgZeD3gMuW+sFVtRfY2z5jrqpmlvqep6PVOjbHtfKs1rGt5nFN8v1Gmcb5h8CfVtVCVf0V8E3gA8DaNq0DsBk40spHgC2ts2uAc4AXJ9lpSdJ4Rgn7PwO2J/m5Nve+A3gMuB/4eGuzC9jfygfaMm39fVVVk+uyJGlci4Z9VT3I4EDr94AftNfsBT4L3JBknsGc/G3tJbcB57X6G4A9I/Rj7/hdXzFW69gc18qzWsfmuEYQd7olafXzClpJ6oBhL0kdmHrYJ7ksyRPt9gqjzO+fNpJsSXJ/ksfa7SQ+3erPTXJvkifb87pWnyS3tLE+kmTbdEfw9pKckeT7Se5uy6viFhlJ1ia5K8mPkjye5NLVsM1W021Nknw1ybHh629OZRsl2dXaP5lk18k+62fpLcb1n9q/xUeSfCvJ2qF1N7ZxPZHkV4fqx8/Nqprag8GtF54C3g2cCfwxcNE0+zRm/zcC21r5F4A/AS4C/iOwp9XvAW5q5SuA/w4E2A48OO0xLDK+G4DfBe5uy3cCV7fyl4B/3sr/AvhSK18NfGPafV9kXPuAf9bKZwJrV/o2Y3Ax458C7xraVp9cqdsM+BVgG/DoUN1Y2wg4F3i6Pa9r5XWn4bg+Aqxp5ZuGxnVRy8SzGFzn9FTLzFPKzWlv0EuBbw8t3wjcOO1/aEsYz37gw8ATwMZWtxF4opW/DFwz1P71dqfbg8G1EwcZ3Bbj7vYf6YWhf5Svbzvg28Clrbymtcu0x/AW4zqnhWJOqF/R24w3rlw/t22Du4FfXcnbDDj/hFAcaxsB1wBfHqp/U7vTZVwnrPvHwB2t/KY8PL7NTjU3pz2N8/qtFZrh2y6sKO1r8CXAg8CGqjraVj0HbGjllTTe3wJ+HfjrtnweI94iAzh+i4zT0QXAAvDbbYrqK0nOZoVvs6o6Ahy/rclRBttg5NuacHpvs+PG3UYrYtud4J8y+JYCEx7XtMN+VUjy88DvA5+pqh8Pr6vBn94VdX5rko8Cx6rqoWn3ZRmsYfA1+taqugT4CSdcC7JCt9nwbU3+LnA2E7ityelqJW6jxST5HPAacMdyvP+0w/71Wys0w7ddWBGSvINB0N9RVd9s1c8n2djWbwSOtfqVMt4PAB9L8gzwdQZTOV9kddwi4zBwuAYXC8LggsFtrPxt1sNtTcbdRitl25Hkk8BHgU+0P2Qw4XFNO+y/C2xtZwycyeBA0YEp92lkScLgiuHHq+o3h1YN3zLixFtJXNvOHtgOvDL0tfS0UVU3VtXmqjqfwTa5r6o+wSq4RUZVPQccSnL8joLHb/+xorcZfdzWZNxt9G3gI0nWtW8+H2l1p5UklzGYMv1YVb06tOoAcHU7c+oCYCvwR5xqbp4GByuuYHAWy1PA56bdnzH7/ssMvko+AjzcHlcwmPs8CDwJ/A/g3NY+wH9pY/0Bgx+Emfo4FhnjB3njbJx3t39s8wzufnpWq39nW55v69897X4vMqaLgbm23f4bgzM1Vvw2A34D+BHwKPA7DM7iWJHbDPgag2MPf8Xg29juU9lGDObA59vjutN0XPMM5uCPZ8iXhtp/ro3rCeDyofqxc9PbJUhSB6Y9jSNJ+hkw7CWpA4a9JHXAsJekDhj2ktQBw16SOmDYS1IH/j8eGrmC1/3y6wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#play without making snapshots (faster)\n",
    "while True:\n",
    "    is_done = env.step(env.action_space.sample())[2]\n",
    "    if is_done: \n",
    "        print(\"Whoops! We died!\")\n",
    "        break\n",
    "        \n",
    "print(\"final state:\")\n",
    "plt.imshow(env.render('rgb_array'))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "After loading snapshot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAD8CAYAAACW/ATfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAEztJREFUeJzt3XGsnfV93/H3ZziQlHbY0DvLsy1BFCsITSo4V5lRqimLlxRYFDMpRaBouMzTjTa2JWNSa5Y/pkr7I2xTaZAmghXSORUloTQZFmJNmSGa9kdoLgklBEK5UKhtGXyhQLawdmX97o/zMxxuDfcc+1yO7/29X9LR+T2/5/ec5/fTc/3xc3/3eZ6TqkKStLb9jWl3QJK08gx7SeqAYS9JHTDsJakDhr0kdcCwl6QOrEjYJ7ksyZNJFpLsXYl9SJJGl0lfZ5/kDOCPgY8Dh4HvAddU1eMT3ZEkaWQrcWb/YWChqp6pqv8LfB3YtQL7kSSNaN0KfOZm4NDQ8mHg7y5tlGQOmAM4++yzP3ThhReuQFckaXV69tlnefHFFzOpz1uJsB9JVe0D9gHMzs7W/Pz8tLoiSaed2dnZiX7eSkzjHAG2Di1vaXWSpClZibD/HrAtyQVJzgSuBg6swH4kSSOa+DROVb2e5F8A3wbOAL5aVT+a9H4kSaNbkTn7qroPuG8lPluSND7voJWkDhj2ktQBw16SOmDYS1IHDHtJ6oBhL0kdMOwlqQOGvSR1wLCXpA4Y9pLUAcNekjpg2EtSBwx7SeqAYS9JHTDsJakDhr0kdcCwl6QOGPaS1IFlwz7JV5McS/LYUN25Se5P8lR739Dqk+SWJAtJHk2yfSU7L0kazShn9v8FuGxJ3V7gYFVtAw62ZYDLgW3tNQfcOpluSpJOxbJhX1X/A/izJdW7gP2tvB+4cqj+azXwXWB9kk2T6qwk6eSc7Jz9xqo62srPAxtbeTNwaKjd4VYnSZqiU/4DbVUVUONul2QuyXyS+cXFxVPthiTpHZxs2L9wfHqmvR9r9UeArUPttrS6v6aq9lXVbFXNzszMnGQ3JEmjONmwPwDsbuXdwD1D9de2q3J2AK8OTfdIkqZk3XINktwJfBT4+SSHgX8HfBG4K8ke4Dngqtb8PuAKYAF4DbhuBfosSRrTsmFfVde8zaqdJ2hbwPWn2ilJ0mR5B60kdcCwl6QOGPaS1AHDXpI6YNhLUgcMe0nqgGEvSR0w7CWpA4a9JHXAsJekDhj2ktQBw16SOmDYS1IHDHtJ6oBhL0kdMOwlqQOGvSR1wLCXpA4Y9pLUgWXDPsnWJA8meTzJj5J8rtWfm+T+JE+19w2tPkluSbKQ5NEk21d6EJKkdzbKmf3rwL+pqouAHcD1SS4C9gIHq2obcLAtA1wObGuvOeDWifdakjSWZcO+qo5W1fdb+X8BTwCbgV3A/tZsP3BlK+8CvlYD3wXWJ9k08Z5LkkY21px9kvOBS4CHgI1VdbSteh7Y2MqbgUNDmx1udUs/ay7JfJL5xcXFMbstSRrHyGGf5GeB3wM+X1U/GV5XVQXUODuuqn1VNVtVszMzM+NsKkka00hhn+Q9DIL+jqr6Zqt+4fj0THs/1uqPAFuHNt/S6iRJUzLK1TgBbgeeqKrfGFp1ANjdyruBe4bqr21X5ewAXh2a7pEkTcG6Edp8BPjHwA+TPNLq/i3wReCuJHuA54Cr2rr7gCuABeA14LqJ9liSNLZlw76q/ieQt1m98wTtC7j+FPslSZog76CVpA4Y9pLUAcNekjpg2EtSBwx7SeqAYS9JHTDsJakDhr0kdcCwl6QOGPaS1AHDXpI6YNhLUgcMe0nqgGEvSR0w7CWpA4a9JHXAsJekDhj2ktSBUb5w/L1J/jDJHyX5UZJfb/UXJHkoyUKSbyQ5s9Wf1ZYX2vrzV3YIkqTljHJm/xfAx6rqF4CLgcuS7ABuAm6uqg8ALwN7Wvs9wMut/ubWTpI0RcuGfQ3877b4nvYq4GPA3a1+P3BlK+9qy7T1O5O83ReWS5LeBSPN2Sc5I8kjwDHgfuBp4JWqer01OQxsbuXNwCGAtv5V4LwTfOZckvkk84uLi6c2CknSOxop7Kvq/1XVxcAW4MPAhae646raV1WzVTU7MzNzqh8nSXoHY12NU1WvAA8ClwLrk6xrq7YAR1r5CLAVoK0/B3hpIr2VJJ2UUa7GmUmyvpXfB3wceIJB6H+6NdsN3NPKB9oybf0DVVWT7LQkaTzrlm/CJmB/kjMY/OdwV1Xdm+Rx4OtJ/j3wA+D21v524LeTLAB/Bly9Av2WJI1h2bCvqkeBS05Q/wyD+ful9X8O/PJEeidJmgjvoJWkDhj2ktQBw16SOmDYS1IHDHtJ6oBhL0kdGOU6e2lNenjfZ/9a3YfmbptCT6SV55m9JHXAsJekDhj2ktQBw16SOmDYS1IHDHtJ6oBhL0kdMOylISe69l5aCwx7SeqAYS9JHTDsJakDI4d9kjOS/CDJvW35giQPJVlI8o0kZ7b6s9ryQlt//sp0XZI0qnHO7D8HPDG0fBNwc1V9AHgZ2NPq9wAvt/qbWztJ0hSNFPZJtgD/EPhKWw7wMeDu1mQ/cGUr72rLtPU7W3vptOITLtWTUc/sfxP4VeCv2vJ5wCtV9XpbPgxsbuXNwCGAtv7V1v4tkswlmU8yv7i4eJLdlySNYtmwT/JJ4FhVPTzJHVfVvqqararZmZmZSX60JGmJUb685CPAp5JcAbwX+JvAl4D1Sda1s/ctwJHW/giwFTicZB1wDvDSxHsuSRrZsmf2VXVjVW2pqvOBq4EHquozwIPAp1uz3cA9rXygLdPWP1BVNdFeS5LGcirX2f8acEOSBQZz8re3+tuB81r9DcDeU+uiJOlUjfUdtFX1HeA7rfwM8OETtPlz4Jcn0DdJ0oR4B60kdcCwl6QOGPaS1AHDXpI6YNhLUgcMe2kJv61Ka5FhL0kdMOwlqQOGvSR1wLCXpA4Y9pLUAcNekjpg2EtSBwx7SeqAYS9JHTDsJakDhr0kdcCwV9c+NHfbtLsgvStGCvskzyb5YZJHksy3unOT3J/kqfa+odUnyS1JFpI8mmT7Sg5AkrS8cc7s/35VXVxVs215L3CwqrYBB3nzi8UvB7a11xxw66Q6K0k6OacyjbML2N/K+4Erh+q/VgPfBdYn2XQK+5EknaJRw76AP0jycJK5Vrexqo628vPAxlbeDBwa2vZwq3uLJHNJ5pPMLy4unkTXJUmjWjdiu1+sqiNJ/hZwf5IfD6+sqkpS4+y4qvYB+wBmZ2fH2laSNJ6Rzuyr6kh7PwZ8C/gw8MLx6Zn2fqw1PwJsHdp8S6uTJE3JsmGf5OwkP3e8DHwCeAw4AOxuzXYD97TyAeDadlXODuDVoekeSdIUjDKNsxH4VpLj7X+nqn4/yfeAu5LsAZ4Drmrt7wOuABaA14DrJt5rSdJYlg37qnoG+IUT1L8E7DxBfQHXT6R3kqSJ8A5aSeqAYS9JHTDsJakDhr10Ag/v++y0uyBNlGEvSR0w7CWpA4a9JHXAsJekDhj2ktQBw17d86sJ1QPDXpI6YNhLUgcMe0nqgGGvNSvJyK+V2F46nRj2ktSBUb+DVlrz7j0690b5k5v2TbEn0uR5Zi/x1qA/0bK02hn2ktSBkcI+yfokdyf5cZInklya5Nwk9yd5qr1vaG2T5JYkC0keTbJ9ZYcgSVrOqGf2XwJ+v6ouZPB9tE8Ae4GDVbUNONiWAS4HtrXXHHDrRHssrYClc/TO2WutyeD7wd+hQXIO8Ajw/hpqnORJ4KNVdTTJJuA7VfXBJLe18p1L273dPmZnZ2t+fn4Cw5He9G5eErncvyNpXLOzs8zPz0/sh3iUM/sLgEXgt5L8IMlXkpwNbBwK8OeBja28GTg0tP3hVidJmpJRwn4dsB24taouAX7Km1M2ALQz/rFObZLMJZlPMr+4uDjOppKkMY0S9oeBw1X1UFu+m0H4v9Cmb2jvx9r6I8DWoe23tLq3qKp9VTVbVbMzMzMn239J0giWDfuqeh44lOSDrWon8DhwANjd6nYD97TyAeDadlXODuDVd5qvlyStvFHvoP2XwB1JzgSeAa5j8B/FXUn2AM8BV7W29wFXAAvAa62tJGmKRgr7qnoEmD3Bqp0naFvA9afYL0nSBHkHrSR1wLCXpA4Y9pLUAR9xrDXLu1qlN3lmL0kdMOwlqQOGvSR1wLCXpA4Y9pLUAcNekjpg2EtSBwx7SeqAYS9JHTDsJakDhr0kdcCwl6QOGPaS1AHDXpI6sGzYJ/lgkkeGXj9J8vkk5ya5P8lT7X1Da58ktyRZSPJoku0rPwxJ0jtZNuyr6smquriqLgY+xOBLxL8F7AUOVtU24GBbBrgc2NZec8CtK9FxSdLoxp3G2Qk8XVXPAbuA/a1+P3BlK+8CvlYD3wXWJ9k0kd5Kkk7KuGF/NXBnK2+sqqOt/DywsZU3A4eGtjnc6iRJUzJy2Cc5E/gU8LtL19Xg+9/G+g64JHNJ5pPMLy4ujrOpJGlM45zZXw58v6peaMsvHJ+eae/HWv0RYOvQdlta3VtU1b6qmq2q2ZmZmfF7Lkka2Thhfw1vTuEAHAB2t/Ju4J6h+mvbVTk7gFeHpnskSVOwbpRGSc4GPg58dqj6i8BdSfYAzwFXtfr7gCuABQZX7lw3sd5Kkk7KSGFfVT8FzltS9xKDq3OWti3g+on0TpI0Ed5BK0kdMOwlqQOGvSR1wLCXpA4Y9pLUAcNekjpg2EtSBwx7SeqAYS9JHTDsJakDhr0kdcCwl6QOGPaS1AHDXpI6YNhLUgcMe0nqgGEvSR0w7CWpA4a9JHVgpLBP8q+T/CjJY0nuTPLeJBckeSjJQpJvJDmztT2rLS+09eev5AAkSctbNuyTbAb+FTBbVX8HOAO4GrgJuLmqPgC8DOxpm+wBXm71N7d2kqQpGnUaZx3wviTrgJ8BjgIfA+5u6/cDV7byrrZMW78zSSbTXUnSyVi3XIOqOpLkPwF/Cvwf4A+Ah4FXqur11uwwsLmVNwOH2ravJ3kVOA94cfhzk8wBc23xL5I8dopjOV39PEvGvkY4rtVnrY5trY7rg5P8sGXDPskGBmfrFwCvAL8LXHaqO66qfcC+to/5qpo91c88Ha3VsTmu1Wetjm0tj2uSnzfKNM4/AP6kqhar6i+BbwIfAda3aR2ALcCRVj4CbG2dXQecA7w0yU5LksYzStj/KbAjyc+0ufedwOPAg8CnW5vdwD2tfKAt09Y/UFU1uS5Lksa1bNhX1UMM/tD6feCHbZt9wK8BNyRZYDAnf3vb5HbgvFZ/A7B3hH7sG7/rq8ZaHZvjWn3W6tgc1wjiSbckrX3eQStJHTDsJakDUw/7JJclebI9XmGU+f3TRpKtSR5M8nh7nMTnWv25Se5P8lR739Dqk+SWNtZHk2yf7gjeWZIzkvwgyb1teU08IiPJ+iR3J/lxkieSXLoWjtlaeqxJkq8mOTZ8/83JHKMku1v7p5LsPtG+3k1vM67/2H4WH03yrSTrh9bd2Mb1ZJJfGqofPzeramovBo9eeBp4P3Am8EfARdPs05j93wRsb+WfA/4YuAj4D8DeVr8XuKmVrwD+GxBgB/DQtMewzPhuAH4HuLct3wVc3cpfBv5ZK/9z4MutfDXwjWn3fZlx7Qf+aSufCaxf7ceMwc2MfwK8b+hY/cpqPWbA3wO2A48N1Y11jIBzgWfa+4ZW3nAajusTwLpWvmloXBe1TDyLwX1OT7fMPKncnPYBvRT49tDyjcCN0/5BO4Xx3AN8HHgS2NTqNgFPtvJtwDVD7d9od7q9GNw7cZDBYzHubf+QXhz6oXzj2AHfBi5t5XWtXaY9hrcZ1zktFLOkflUfM968c/3cdgzuBX5pNR8z4PwloTjWMQKuAW4bqn9Lu9NlXEvW/SPgjlZ+Sx4eP2Ynm5vTnsZ549EKzfBjF1aV9mvwJcBDwMaqOtpWPQ9sbOXVNN7fBH4V+Ku2fB4jPiIDOP6IjNPRBcAi8FttiuorSc5mlR+zqjoCHH+syVEGx2Dkx5pweh+z48Y9Rqvi2C3xTxj8lgITHte0w35NSPKzwO8Bn6+qnwyvq8F/vavq+tYknwSOVdXD0+7LCljH4NfoW6vqEuCnLLkXZJUes+HHmvxt4Gwm8FiT09VqPEbLSfIF4HXgjpX4/GmH/RuPVmiGH7uwKiR5D4Ogv6OqvtmqX0iyqa3fBBxr9atlvB8BPpXkWeDrDKZyvsTaeETGYeBwDW4WhMENg9tZ/cesh8eajHuMVsuxI8mvAJ8EPtP+I4MJj2vaYf89YFu7YuBMBn8oOjDlPo0sSRjcMfxEVf3G0KrhR0YsfZTEte3qgR3Aq0O/lp42qurGqtpSVeczOCYPVNVnWAOPyKiq54FDSY4/UfD44z9W9TGjj8eajHuMvg18IsmG9pvPJ1rdaSXJZQymTD9VVa8NrToAXN2unLoA2Ab8ISebm6fBHyuuYHAVy9PAF6bdnzH7/osMfpV8FHikva5gMPd5EHgK+O/Aua19gP/cxvpDBl8IM/VxLDPGj/Lm1Tjvbz9sCwyefnpWq39vW15o698/7X4vM6aLgfl23P4rgys1Vv0xA34d+DHwGPDbDK7iWJXHDLiTwd8e/pLBb2N7TuYYMZgDX2iv607TcS0wmIM/niFfHmr/hTauJ4HLh+rHzk0flyBJHZj2NI4k6V1g2EtSBwx7SeqAYS9JHTDsJakDhr0kdcCwl6QO/H+kh6imSxaxygAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#reload initial state\n",
    "env.load_snapshot(snap0)\n",
    "\n",
    "print(\"\\n\\nAfter loading snapshot\")\n",
    "plt.imshow(env.render('rgb_array'))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#get outcome (snapshot, observation, reward, is_done, info)\n",
    "res = env.get_result(snap0,env.action_space.sample())\n",
    "\n",
    "snap1, observation, reward = res[:3]\n",
    "\n",
    "#second step\n",
    "res2 = env.get_result(snap1,env.action_space.sample())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MCTS: Monte-Carlo tree search\n",
    "\n",
    "In this section, we'll implement the vanilla MCTS algorithm with UCB1-based node selection.\n",
    "\n",
    "We will start by implementing the `Node` class - a simple class that acts like MCTS node and supports some of the MCTS algorithm steps.\n",
    "\n",
    "This MCTS implementation makes some assumptions about the environment, you can find those _in the notes section at the end of the notebook_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert isinstance(env,WithSnapshots)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Node:\n",
    "    \"\"\" a tree node for MCTS \"\"\"\n",
    "    \n",
    "    #metadata:\n",
    "    parent = None          #parent Node\n",
    "    value_sum = 0.         #sum of state values from all visits (numerator)\n",
    "    times_visited = 0      #counter of visits (denominator)\n",
    "\n",
    "    \n",
    "    def __init__(self,parent,action,):\n",
    "        \"\"\"\n",
    "        Creates and empty node with no children.\n",
    "        Does so by commiting an action and recording outcome.\n",
    "        \n",
    "        :param parent: parent Node\n",
    "        :param action: action to commit from parent Node\n",
    "        \n",
    "        \"\"\"\n",
    "        \n",
    "        self.parent = parent\n",
    "        self.action = action        \n",
    "        self.children = set()       #set of child nodes\n",
    "\n",
    "        #get action outcome and save it\n",
    "        res = env.get_result(parent.snapshot,action)\n",
    "        self.snapshot,self.observation,self.immediate_reward,self.is_done,_ = res\n",
    "        \n",
    "        \n",
    "    def is_leaf(self):\n",
    "        return len(self.children)==0\n",
    "    \n",
    "    def is_root(self):\n",
    "        return self.parent is None\n",
    "    \n",
    "    def get_mean_value(self):\n",
    "        return self.value_sum / self.times_visited if self.times_visited !=0 else 0\n",
    "    \n",
    "    def ucb_score(self,scale=10,max_value=1e100):\n",
    "        \"\"\"\n",
    "        Computes ucb1 upper bound using current value and visit counts for node and it's parent.\n",
    "        \n",
    "        :param scale: Multiplies upper bound by that. From hoeffding inequality, assumes reward range to be [0,scale].\n",
    "        :param max_value: a value that represents infinity (for unvisited nodes)\n",
    "        \n",
    "        \"\"\"\n",
    "        \n",
    "        if self.times_visited == 0:\n",
    "            return max_value\n",
    "        \n",
    "        #compute ucb-1 additive component (to be added to mean value)\n",
    "        #hint: you can use self.parent.times_visited for N times node was considered,\n",
    "        # and self.times_visited for n times it was visited\n",
    "        \n",
    "#         U = <your code here>\n",
    "        U = np.sqrt(2 * np.log(self.parent.times_visited) / self.times_visited)\n",
    "        \n",
    "        return self.get_mean_value() + scale*U\n",
    "    \n",
    "    \n",
    "    #MCTS steps\n",
    "    \n",
    "    def select_best_leaf(self):\n",
    "        \"\"\"\n",
    "        Picks the leaf with highest priority to expand\n",
    "        Does so by recursively picking nodes with best UCB-1 score until it reaches the leaf.\n",
    "        \n",
    "        \"\"\"\n",
    "        if self.is_leaf():\n",
    "            return self\n",
    "        \n",
    "        children = self.children\n",
    "        \n",
    "#         best_child = <select best child node in terms of node.ucb_score()>\n",
    "        best_child = max( [(child.ucb_score(), child) for child in children], key=lambda x: x[0] )[1]\n",
    "        \n",
    "        return best_child.select_best_leaf()\n",
    "    \n",
    "    def expand(self):\n",
    "        \"\"\"\n",
    "        Expands the current node by creating all possible child nodes.\n",
    "        Then returns one of those children.\n",
    "        \"\"\"\n",
    "        \n",
    "        assert not self.is_done, \"can't expand from terminal state\"\n",
    "\n",
    "        for action in range(n_actions):\n",
    "            self.children.add(Node(self,action))\n",
    "        \n",
    "        return self.select_best_leaf()\n",
    "    \n",
    "    def rollout(self,t_max=10**4):\n",
    "        \"\"\"\n",
    "        Play the game from this state to the end (done) or for t_max steps.\n",
    "        \n",
    "        On each step, pick action at random (hint: env.action_space.sample()).\n",
    "        \n",
    "        Compute sum of rewards from current state till \n",
    "        Note 1: use env.action_space.sample() for random action\n",
    "        Note 2: if node is terminal (self.is_done is True), just return 0\n",
    "        \n",
    "        \"\"\"\n",
    "            \n",
    "        #set env into the appropriate state\n",
    "        env.load_snapshot(self.snapshot)\n",
    "        obs = self.observation\n",
    "        is_done = self.is_done\n",
    "        \n",
    "#         <your code here - rollout and compute reward>\n",
    "        rollout_reward = 0\n",
    "        while not is_done and t_max > 0:\n",
    "            t_max-=1\n",
    "            _, r, is_done, _ = env.step(env.action_space.sample())\n",
    "            rollout_reward += r\n",
    "\n",
    "        return rollout_reward\n",
    "    \n",
    "    def propagate(self,child_value):\n",
    "        \"\"\"\n",
    "        Uses child value (sum of rewards) to update parents recursively.\n",
    "        \"\"\"\n",
    "        #compute node value\n",
    "        my_value = self.immediate_reward + child_value\n",
    "        \n",
    "        #update value_sum and times_visited\n",
    "        self.value_sum+=my_value\n",
    "        self.times_visited+=1\n",
    "        \n",
    "        #propagate upwards\n",
    "        if not self.is_root():\n",
    "            self.parent.propagate(my_value)\n",
    "        \n",
    "    def safe_delete(self):\n",
    "        \"\"\"safe delete to prevent memory leak in some python versions\"\"\"\n",
    "        del self.parent\n",
    "        for child in self.children:\n",
    "            child.safe_delete()\n",
    "            del child"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Root(Node):\n",
    "    def __init__(self,snapshot,observation):\n",
    "        \"\"\"\n",
    "        creates special node that acts like tree root\n",
    "        :snapshot: snapshot (from env.get_snapshot) to start planning from\n",
    "        :observation: last environment observation\n",
    "        \"\"\"\n",
    "        \n",
    "        self.parent = self.action = None\n",
    "        self.children = set()       #set of child nodes\n",
    "        \n",
    "        #root: load snapshot and observation\n",
    "        self.snapshot = snapshot\n",
    "        self.observation = observation\n",
    "        self.immediate_reward = 0\n",
    "        self.is_done=False\n",
    "    \n",
    "    @staticmethod\n",
    "    def from_node(node):\n",
    "        \"\"\"initializes node as root\"\"\"\n",
    "        root = Root(node.snapshot,node.observation)\n",
    "        #copy data\n",
    "        copied_fields = [\"value_sum\",\"times_visited\",\"children\",\"is_done\"]\n",
    "        for field in copied_fields:\n",
    "            setattr(root,field,getattr(node,field))\n",
    "        return root"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Main MCTS loop\n",
    "\n",
    "With all we implemented, MCTS boils down to a trivial piece of code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plan_mcts(root,n_iters=10):\n",
    "    \"\"\"\n",
    "    builds tree with monte-carlo tree search for n_iters iterations\n",
    "    :param root: tree node to plan from\n",
    "    :param n_iters: how many select-expand-simulate-propagete loops to make\n",
    "    \"\"\"\n",
    "    for _ in range(n_iters):\n",
    "\n",
    "#         node = <select best leaf>\n",
    "        node = root.select_best_leaf()\n",
    "\n",
    "        if node.is_done:\n",
    "            node.propagate(0)\n",
    "\n",
    "        else: #node is not terminal\n",
    "#             <expand-simulate-propagate loop>\n",
    "            node_child = node.expand()\n",
    "            child_reward = node_child.rollout()\n",
    "            node.propagate(child_reward)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plan and execute\n",
    "In this section, we use the MCTS implementation to find optimal policy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "root_observation = env.reset()\n",
    "root_snapshot = env.get_snapshot()\n",
    "root = Root(root_snapshot,root_observation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'NoneType' object has no attribute 'setView_'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-19-0f867ad35a95>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m#plan from root:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mplan_mcts\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mroot\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mn_iters\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1000\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-17-9d271bb82a1c>\u001b[0m in \u001b[0;36mplan_mcts\u001b[0;34m(root, n_iters)\u001b[0m\n\u001b[1;32m     15\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m#node is not terminal\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     16\u001b[0m \u001b[0;31m#             <expand-simulate-propagate loop>\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 17\u001b[0;31m             \u001b[0mnode_child\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnode\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexpand\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     18\u001b[0m             \u001b[0mchild_reward\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnode_child\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrollout\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     19\u001b[0m             \u001b[0mnode\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpropagate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mchild_reward\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-15-f66f0ec9168c>\u001b[0m in \u001b[0;36mexpand\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     85\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     86\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0maction\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn_actions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 87\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchildren\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mNode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0maction\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     88\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     89\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mselect_best_leaf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-15-f66f0ec9168c>\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, parent, action)\u001b[0m\n\u001b[1;32m     23\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     24\u001b[0m         \u001b[0;31m#get action outcome and save it\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 25\u001b[0;31m         \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_result\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparent\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msnapshot\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0maction\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     26\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msnapshot\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mobservation\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mimmediate_reward\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_done\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0m_\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mres\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     27\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-7-2fd7aafb887f>\u001b[0m in \u001b[0;36mget_result\u001b[0;34m(self, snapshot, action)\u001b[0m\n\u001b[1;32m     68\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     69\u001b[0m \u001b[0;31m#         <your code here load,commit,take snapshot>\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 70\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload_snapshot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msnapshot\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     71\u001b[0m         \u001b[0ms\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minfo\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maction\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     72\u001b[0m         \u001b[0mnext_snapshot\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_snapshot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-7-2fd7aafb887f>\u001b[0m in \u001b[0;36mload_snapshot\u001b[0;34m(self, snapshot)\u001b[0m\n\u001b[1;32m     51\u001b[0m         \u001b[0;32massert\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\"_monitor\"\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\"_monitor\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"can't backtrack while recording\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     52\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 53\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     54\u001b[0m \u001b[0;31m#         self.render(close=True) #close popup windows since we can't load into them\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     55\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mloads\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msnapshot\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/gym/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, mode, **kwargs)\u001b[0m\n\u001b[1;32m    273\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    274\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'human'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 275\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    276\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    277\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/gym/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, mode, **kwargs)\u001b[0m\n\u001b[1;32m    273\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    274\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'human'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 275\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    276\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    277\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/gym/gym/envs/classic_control/cartpole.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, mode)\u001b[0m\n\u001b[1;32m    149\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mviewer\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    150\u001b[0m             \u001b[0;32mfrom\u001b[0m \u001b[0mgym\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menvs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclassic_control\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mrendering\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 151\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mviewer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrendering\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mViewer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mscreen_width\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mscreen_height\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    152\u001b[0m             \u001b[0ml\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mr\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m-\u001b[0m\u001b[0mcartwidth\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcartwidth\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcartheight\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m-\u001b[0m\u001b[0mcartheight\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    153\u001b[0m             \u001b[0maxleoffset\u001b[0m \u001b[0;34m=\u001b[0m\u001b[0mcartheight\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0;36m4.0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/gym/gym/envs/classic_control/rendering.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, width, height, display)\u001b[0m\n\u001b[1;32m     49\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwidth\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mwidth\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     50\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mheight\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mheight\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 51\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwindow\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpyglet\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwindow\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mWindow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwidth\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mwidth\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mheight\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mheight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdisplay\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdisplay\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     52\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwindow\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_close\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwindow_closed_by_user\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     53\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0misopen\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/rl/lib/python3.6/site-packages/pyglet/window/__init__.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, width, height, caption, resizable, style, fullscreen, visible, vsync, display, screen, config, context, mode)\u001b[0m\n\u001b[1;32m    569\u001b[0m         \u001b[0;32mfrom\u001b[0m \u001b[0mpyglet\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mapp\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    570\u001b[0m         \u001b[0mapp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwindows\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 571\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_create\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    572\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    573\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mswitch_to\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/rl/lib/python3.6/site-packages/pyglet/window/cocoa/__init__.py\u001b[0m in \u001b[0;36m_create\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    196\u001b[0m         \u001b[0;31m# Create a canvas with the view as its drawable and attach context to it.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    197\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcanvas\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mCocoaCanvas\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdisplay\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscreen\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_nsview\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 198\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontext\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mattach\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcanvas\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    199\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    200\u001b[0m         \u001b[0;31m# Configure the window.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/rl/lib/python3.6/site-packages/pyglet/gl/cocoa.py\u001b[0m in \u001b[0;36mattach\u001b[0;34m(self, canvas)\u001b[0m\n\u001b[1;32m    261\u001b[0m         \u001b[0;31m# The NSView instance should be attached to a nondeferred window before calling\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    262\u001b[0m         \u001b[0;31m# setView, otherwise you get an \"invalid drawable\" message.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 263\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_nscontext\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msetView_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcanvas\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnsview\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    264\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_nscontext\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mview\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msetWantsBestResolutionOpenGLSurface_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    265\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_current\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'NoneType' object has no attribute 'setView_'"
     ]
    }
   ],
   "source": [
    "#plan from root:\n",
    "plan_mcts(root,n_iters=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import clear_output\n",
    "from itertools import count\n",
    "from gym.wrappers import Monitor\n",
    "\n",
    "total_reward = 0                #sum of rewards\n",
    "test_env = loads(root_snapshot) #env used to show progress\n",
    "\n",
    "for i in count():\n",
    "    \n",
    "    #get best child\n",
    "    best_child = <select child with highest mean reward>\n",
    "    \n",
    "    #take action\n",
    "    s,r,done,_ = test_env.step(best_child.action)\n",
    "    \n",
    "    #show image\n",
    "    clear_output(True)\n",
    "    plt.title(\"step %i\"%i)\n",
    "    plt.imshow(test_env.render('rgb_array'))\n",
    "    plt.show()\n",
    "\n",
    "    total_reward += r\n",
    "    if done:\n",
    "        print(\"Finished with reward = \",total_reward)\n",
    "        break\n",
    "    \n",
    "    #discard unrealized part of the tree [because not every child matters :(]\n",
    "    for child in root.children:\n",
    "        if child != best_child:\n",
    "            child.safe_delete()\n",
    "\n",
    "    #declare best child a new root\n",
    "    root = Root.from_node(best_child)\n",
    "    \n",
    "    assert not root.is_leaf(), \"We ran out of tree! Need more planning! Try growing tree right inside the loop.\"\n",
    "    \n",
    "    #you may want to expand tree here\n",
    "    #<your code here>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Submit to Coursera"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from submit import submit_mcts\n",
    "\n",
    "submit_mcts(total_reward, <EMAIL>, <TOKEN>)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## More stuff\n",
    "\n",
    "There's a few things you might want to try if you want to dig deeper:\n",
    "\n",
    "### Node selection and expansion\n",
    "\n",
    "\"Analyze this\" assignment\n",
    "\n",
    "UCB-1 is a weak bound as it relies on a very general bounds (Hoeffding Inequality, to be exact). \n",
    "* Try playing with alpha. The theoretically optimal alpha for CartPole is 200 (max reward). \n",
    "* Use using a different exploration strategy (bayesian UCB, for example)\n",
    "* Expand not all but several random actions per `expand` call. See __the notes below__ for details.\n",
    "\n",
    "The goal is to find out what gives the optimal performance for `CartPole-v0` for different time budgets (i.e. different n_iter in plan_mcts.\n",
    "\n",
    "Evaluate your results on `AcroBot-v1` - do the results change and if so, how can you explain it?\n",
    "\n",
    "\n",
    "### Atari-RAM\n",
    "\n",
    "\"Build this\" assignment\n",
    "\n",
    "Apply MCTS to play atari games. In particular, let's start with ```gym.make(\"MsPacman-ramDeterministic-v0\")```.\n",
    "\n",
    "This requires two things:\n",
    "* Slightly modify WithSnapshots wrapper to work with atari.\n",
    "\n",
    " * Atari has a special interface for snapshots:\n",
    "   ```   \n",
    "   snapshot = self.env.ale.cloneState()\n",
    "   ...\n",
    "   self.env.ale.restoreState(snapshot)\n",
    "   ```\n",
    " * Try it on the env above to make sure it does what you told it to.\n",
    " \n",
    "* Run MCTS on the game above. \n",
    " * Start with small tree size to speed-up computations\n",
    " * You will probably want to rollout for 10-100 steps (t_max) for starters\n",
    " * Consider using discounted rewards (see __notes at the end__)\n",
    " * Try a better rollout policy\n",
    " \n",
    " \n",
    "### Integrate learning into planning\n",
    "\n",
    "Planning on each iteration is a costly thing to do. You can speed things up drastically if you train a classifier to predict which action will turn out to be best according to MCTS.\n",
    "\n",
    "To do so, just record which action did the MCTS agent take on each step and fit something to [state, mcts_optimal_action]\n",
    "* You can also use optimal actions from discarded states to get more (dirty) samples. Just don't forget to fine-tune without them.\n",
    "* It's also worth a try to use P(best_action|state) from your model to select best nodes in addition to UCB\n",
    "* If your model is lightweight enough, try using it as a rollout policy.\n",
    "\n",
    "While CartPole is glorious enough, try expanding this to ```gym.make(\"MsPacmanDeterministic-v0\")```\n",
    "* See previous section on how to wrap atari\n",
    "\n",
    "* Also consider what [AlphaGo Zero](https://deepmind.com/blog/alphago-zero-learning-scratch/) did in this area.\n",
    "\n",
    "### Integrate planning into learning \n",
    "_(this will likely take long time, better consider this as side project when all other deadlines are met)_\n",
    "\n",
    "Incorporate planning into the agent architecture. \n",
    "\n",
    "The goal is to implement [Value Iteration Networks](https://arxiv.org/abs/1602.02867)\n",
    "\n",
    "For starters, remember [week7 assignment](https://github.com/yandexdataschool/Practical_RL/blob/master/week7/7.2_seminar_kung_fu.ipynb)? If not, use [this](http://bit.ly/2oZ34Ap) instead.\n",
    "\n",
    "You will need to switch it into a maze-like game, consider MsPacman or the games from week7 [Bonus: Neural Maps from here](https://github.com/yandexdataschool/Practical_RL/blob/master/week7/7.3_homework.ipynb).\n",
    "\n",
    "You will need to implement a special layer that performs value iteration-like update to a recurrent memory. This can be implemented the same way you did attention from week7 or week8."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notes\n",
    "\n",
    "\n",
    "#### Assumptions\n",
    "\n",
    "The full list of assumptions is\n",
    "* __Finite actions__ - we enumerate all actions in `expand`\n",
    "* __Episodic (finite) MDP__ - while technically it works for infinite mdp, we rollout for $ 10^4$ steps. If you are knowingly infinite, please adjust `t_max` to something more reasonable.\n",
    "* __No discounted rewards__ - we assume $\\gamma=1$. If that isn't the case, you only need to change a two lines in `rollout` and use `my_R = r + gamma*child_R` for `propagate`\n",
    "* __pickleable env__ - won't work if e.g. your env is connected to a web-browser surfing the internet. For custom envs, you may need to modify get_snapshot/load_snapshot from `WithSnapshots`.\n",
    "\n",
    "#### On `get_best_leaf` and `expand` functions\n",
    "\n",
    "This MCTS implementation only selects leaf nodes for expansion.\n",
    "This doesn't break things down because `expand` adds all possible actions. Hence, all non-leaf nodes are by design fully expanded and shouldn't be selected.\n",
    "\n",
    "If you want to only add a few random action on each expand, you will also have to modify `get_best_leaf` to consider returning non-leafs.\n",
    "\n",
    "#### Rollout policy\n",
    "\n",
    "We use a simple uniform policy for rollouts. This introduces a negative bias to good situations that can be messed up completely with random bad action. As a simple example, if you tend to rollout with uniform policy, you better don't use sharp knives and walk near cliffs.\n",
    "\n",
    "You can improve that by integrating a reinforcement _learning_ algorithm with a computationally light agent. You can even train this agent on optimal policy found by the tree search.\n",
    "\n",
    "#### Contributions\n",
    "* Reusing some code from 5vision [solution for deephack.RL](https://github.com/5vision/uct_atari), code by Mikhail Pavlov\n",
    "* Using some code from [this gist](https://gist.github.com/blole/dfebbec182e6b72ec16b66cc7e331110)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}