Classic_Control_Introduction.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Urmh5ntuuEaL"
      },
      "source": [
        "# Classic Control: Control theory problems from the classic RL literature\n",
        "\n",
        "<br><br>\n",
        "\n",
        "In this notebook we will present some classic environments in Reinforcement Learning research. These environments have continuous states spaces (i.e., infinite possible states) and therefore tabular methods cannot solve them. To tackle these environments (and more complex ones) we will have two tools:\n",
        "\n",
        "- Extend the tabular methods with the techniques of discretization and tile coding\n",
        "- Use function approximators (Neural Networks)\n",
        "\n",
        "<br>"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# @title Setup code (not important) - Run this cell by pressing \"Shift + Enter\"\n",
        "\n",
        "\n",
        "!pip install -qq gym==0.23.0\n",
        "\n",
        "\n",
        "import matplotlib\n",
        "from matplotlib import animation\n",
        "from IPython.display import HTML\n",
        "\n",
        "\n",
        "def display_video(frames):\n",
        "    # Copied from: https://colab.research.google.com/github/deepmind/dm_control/blob/master/tutorial.ipynb\n",
        "    orig_backend = matplotlib.get_backend()\n",
        "    matplotlib.use('Agg')\n",
        "    fig, ax = plt.subplots(1, 1, figsize=(5, 5))\n",
        "    matplotlib.use(orig_backend)\n",
        "    ax.set_axis_off()\n",
        "    ax.set_aspect('equal')\n",
        "    ax.set_position([0, 0, 1, 1])\n",
        "    im = ax.imshow(frames[0])\n",
        "    def update(frame):\n",
        "        im.set_data(frame)\n",
        "        return [im]\n",
        "    anim = animation.FuncAnimation(fig=fig, func=update, frames=frames,\n",
        "                                    interval=50, blit=True, repeat=False)\n",
        "    return HTML(anim.to_html5_video())\n",
        "\n",
        "\n",
        "def test_env(environment, episodes=10):\n",
        "    frames = []\n",
        "    for episode in range(episodes):\n",
        "        state = environment.reset()\n",
        "        done = False\n",
        "        frames.append(environment.render(mode=\"rgb_array\"))\n",
        "\n",
        "        while not done:\n",
        "            action = environment.action_space.sample()\n",
        "            next_state, reward, done, extra_info = environment.step(action)\n",
        "            img = environment.render(mode=\"rgb_array\")\n",
        "            frames.append(img)\n",
        "            state = next_state\n",
        "\n",
        "    return display_video(frames)\n",
        "\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "cellView": "form",
        "id": "FX6nq6g7wAys",
        "outputId": "07f12cbc-90a9-4028-dd20-ce89c8f35b26"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/624.4 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K     \u001b[91m━━━━━━━━━━━━━━━\u001b[0m\u001b[90m╺\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m235.5/624.4 kB\u001b[0m \u001b[31m6.7 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K     \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[90m╺\u001b[0m \u001b[32m614.4/624.4 kB\u001b[0m \u001b[31m9.9 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m624.4/624.4 kB\u001b[0m \u001b[31m8.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for gym (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "m_fXRjU9uEaO"
      },
      "outputs": [],
      "source": [
        "import gym\n",
        "import numpy as np\n",
        "from IPython import display\n",
        "from matplotlib import pyplot as plt\n",
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M9vnZWVouEaP"
      },
      "source": [
        "## CartPole: Keep the tip of the pole straight."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "rEwvRFsauEaP"
      },
      "outputs": [],
      "source": [
        "env = gym.make('CartPole-v1')\n",
        "test_env(env, 1)\n",
        "env.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JS5l98EauEaQ"
      },
      "source": [
        "##### The state\n",
        "\n",
        "The states of the cartpole task will be represented by a vector of four real numbers:\n",
        "\n",
        "        Num     Observation               Min                     Max\n",
        "        0       Cart Position             -4.8                    4.8\n",
        "        1       Cart Velocity             -Inf                    Inf\n",
        "        2       Pole Angle                -0.418 rad (-24 deg)    0.418 rad (24 deg)\n",
        "        3       Pole Angular Velocity     -Inf                    Inf\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "kC_5K14quEaQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "417e9ffb-50d7-4824-cdc0-664590b41d3d"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Box([-4.8000002e+00 -3.4028235e+38 -4.1887903e-01 -3.4028235e+38], [4.8000002e+00 3.4028235e+38 4.1887903e-01 3.4028235e+38], (4,), float32)"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ],
      "source": [
        "env.observation_space"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fwSR1g5GuEaQ"
      },
      "source": [
        "##### The actions available\n",
        "\n",
        "We can perform two actions in this environment:\n",
        "\n",
        "        0     Push cart to the left.\n",
        "        1     Push cart to the right.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "hphxRLb1uEaQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "09f47fb7-3a68-49c6-d637-0ee76b5b7286"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Discrete(2)"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ],
      "source": [
        "env.action_space"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9NPvm9fxuEaQ"
      },
      "source": [
        "## Acrobot: Swing the bar up to a certain height."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "kUp2Pj45uEaQ"
      },
      "outputs": [],
      "source": [
        "env = gym.make('Acrobot-v1')\n",
        "test_env(env, 1)\n",
        "env.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QILrkNtAuEaQ"
      },
      "source": [
        "##### The state\n",
        "\n",
        "The states of the cartpole task will be represented by a vector of six real numbers. The first two are the cosine and sine of the first joint. The next two are the cosine and sine of the other joint. The last two are the angular velocities of each joint.\n",
        "    \n",
        "$\\cos(\\theta_1), \\sin(\\theta_1), \\cos(\\theta_2), \\sin(\\theta_2), \\dot\\theta_1, \\dot\\theta_2$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "cx-K5Z_8uEaR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dcd649d1-4bd6-47e1-dd60-0ca43582b51f"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Box([ -1.        -1.        -1.        -1.       -12.566371 -28.274334], [ 1.        1.        1.        1.       12.566371 28.274334], (6,), float32)"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ],
      "source": [
        "env.observation_space"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SfVu41CEuEaR"
      },
      "source": [
        "##### The actions available\n",
        "\n",
        "We can perform two actions in this environment:\n",
        "\n",
        "    0    Apply +1 torque on the joint between the links.\n",
        "    1    Apply -1 torque on the joint between the links."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "UJOCoO9QuEaR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d3fed225-c8ca-459d-b14b-4b7dcbab3952"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Discrete(3)"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ],
      "source": [
        "env.action_space"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MPNceYXHuEaR"
      },
      "source": [
        "## MountainCar: Reach the goal from the bottom of the valley."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "UI00Fw4LuEaR"
      },
      "outputs": [],
      "source": [
        "env = gym.make('MountainCar-v0')\n",
        "test_env(env, 1)\n",
        "env.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vY6aE8TduEaR"
      },
      "source": [
        "##### The state\n",
        "\n",
        "The observation space consists of the car position $\\in [-1.2, 0.6]$ and car velocity $\\in [-0.07, 0.07]$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "7vOD0LvbuEaR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a6325253-06e1-496d-d9ee-d3790ed7b12a"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Box([-1.2  -0.07], [0.6  0.07], (2,), float32)"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ],
      "source": [
        "env.observation_space"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fs5q1ZYZuEaR"
      },
      "source": [
        "##### The actions available\n",
        "\n",
        "\n",
        "The actions available three:\n",
        "\n",
        "    0    Accelerate to the left.\n",
        "    1    Don't accelerate.\n",
        "    2    Accelerate to the right."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "XrF5V1SIuEaR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "62124d99-e05e-4edf-e7e0-c6a1352900ca"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Discrete(3)"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ],
      "source": [
        "env.action_space"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vPmwqfEjuEaS"
      },
      "source": [
        "## Pendulum: swing it and keep it upright"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "Q7csEunxuEaS"
      },
      "outputs": [],
      "source": [
        "env = gym.make('Pendulum-v1')\n",
        "test_env(env, 1)\n",
        "env.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UusKijA0uEaS"
      },
      "source": [
        "##### The state\n",
        "\n",
        "The state is represented by a vector of three values representing $\\cos(\\theta), \\sin(\\theta)$ and speed ($\\theta$ is the angle of the pendulum)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "zIQaIFAsuEaS",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0ff28f6e-03fb-46a1-cab6-a617d9afe252"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Box([-1. -1. -8.], [1. 1. 8.], (3,), float32)"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ],
      "source": [
        "env.observation_space"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QeYAfguwuEaS"
      },
      "source": [
        "##### The actions available\n",
        "\n",
        "The action is a real number in the interval $[-2, 2]$ that represents the torque applied on the pendulum."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "01d2orXsuEaS",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dce5aced-a685-4739-9afc-1e9710afa9e8"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Box(-2.0, 2.0, (1,), float32)"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ],
      "source": [
        "env.action_space"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q0kA6q7auEaS"
      },
      "source": [
        "## Resources"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7PnV4kmDuEaS"
      },
      "source": [
        "[[1] OpenAI gym: classic control environments](https://gym.openai.com/envs/#classic_control)"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "-fnF-sXIwwsn"
      },
      "execution_count": 14,
      "outputs": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.0"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}