rl-baselines-zoo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/rl-baselines-zoo.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XJy9QoDC7XA7"
   },
   "source": [
    "# RL Baselines3 Zoo: Training in Colab\n",
    "\n",
    "\n",
    "\n",
    "Github Repo: [https://github.com/DLR-RM/rl-baselines3-zoo](https://github.com/DLR-RM/rl-baselines3-zoo)\n",
    "\n",
    "Stable-Baselines3 Repo: [https://github.com/DLR-RM/rl-baselines3-zoo](https://github.com/DLR-RM/stable-baselines3)\n",
    "\n",
    "\n",
    "# Install Dependencies\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "                <script type=\"application/javascript\" id=\"jupyter_black\">\n",
       "                (function() {\n",
       "                    if (window.IPython === undefined) {\n",
       "                        return\n",
       "                    }\n",
       "                    var msg = \"WARNING: it looks like you might have loaded \" +\n",
       "                        \"jupyter_black in a non-lab notebook with \" +\n",
       "                        \"`is_lab=True`. Please double check, and if \" +\n",
       "                        \"loading with `%load_ext` please review the README!\"\n",
       "                    console.log(msg)\n",
       "                    alert(msg)\n",
       "                })()\n",
       "                </script>\n",
       "                "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# for autoformatting\n",
    "# %load_ext jupyter_black"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "AXVDDlTn02M9"
   },
   "outputs": [],
   "source": [
    "!apt-get update && apt-get install swig cmake ffmpeg freeglut3-dev xvfb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kDjF3qRg7oGH"
   },
   "source": [
    "## Clone RL Baselines3 Zoo Repo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SCjGikdT1DFy"
   },
   "outputs": [],
   "source": [
    "!git clone https://github.com/DLR-RM/rl-baselines3-zoo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "REMQlh-ezyVt"
   },
   "outputs": [],
   "source": [
    "%cd /content/rl-baselines3-zoo/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5tmD_QTBqTMb"
   },
   "source": [
    "### Install pip dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OWIDzgJTqShY"
   },
   "outputs": [],
   "source": [
    "!pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6gJ-pAbF7zRZ"
   },
   "source": [
    "## Train an RL Agent\n",
    "\n",
    "\n",
    "The train agent can be found in the `logs/` folder.\n",
    "\n",
    "Here we will train A2C on CartPole-v1 environment for 100 000 steps. \n",
    "\n",
    "\n",
    "To train it on Pong (Atari), you just have to pass `--env PongNoFrameskip-v4`\n",
    "\n",
    "Note: You need to update `hyperparams/algo.yml` to support new environments. You can access it in the side panel of Google Colab. (see https://stackoverflow.com/questions/46986398/import-data-into-google-colaboratory)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9bIR_N7R11XI"
   },
   "outputs": [],
   "source": [
    "!python -m rl_zoo3.train --algo a2c --env CartPole-v1 --n-timesteps 100000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-fHBq73665yD"
   },
   "source": [
    "#### Evaluate trained agent\n",
    "\n",
    "\n",
    "You can remove the `--folder logs/` to evaluate pretrained agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Bw8YuEgU6bT3"
   },
   "outputs": [],
   "source": [
    "!python -m rl_zoo3.enjoy --algo a2c --env CartPole-v1 --no-render --n-timesteps 5000 --folder logs/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "w5Il2J0VHPLC"
   },
   "source": [
    "#### Tune Hyperparameters\n",
    "\n",
    "We use [Optuna](https://optuna.org/) for optimizing the hyperparameters.\n",
    "\n",
    "Tune the hyperparameters for PPO, using a tpe sampler and median pruner, 2 parallels jobs,\n",
    "with a budget of 1000 trials and a maximum of 50000 steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "w2sC22eGHTH-"
   },
   "outputs": [],
   "source": [
    "!python -m rl_zoo3.train --algo ppo --env MountainCar-v0 -n 50000 -optimize --n-trials 1000 --n-jobs 2 --sampler tpe --pruner median"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xVm9QPNVwKXN"
   },
   "source": [
    "### Record  a Video"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MPyfQxD5z26J"
   },
   "outputs": [],
   "source": [
    "# Set up display; otherwise rendering will fail\n",
    "import os\n",
    "os.system(\"Xvfb :1 -screen 0 1024x768x24 &\")\n",
    "os.environ['DISPLAY'] = ':1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ip3AauLzwNGP"
   },
   "outputs": [],
   "source": [
    "!python -m rl_zoo3.record_video --algo a2c --env CartPole-v1 --exp-id 0 -f logs/ -n 1000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qBuUfnzI8DN6"
   },
   "source": [
    "### Display the video"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZC3OTfpf8CXu"
   },
   "outputs": [],
   "source": [
    "import base64\n",
    "from pathlib import Path\n",
    "\n",
    "from IPython import display as ipythondisplay\n",
    "\n",
    "\n",
    "def show_videos(video_path=\"\", prefix=\"\"):\n",
    "    \"\"\"\n",
    "    Taken from https://github.com/eleurent/highway-env\n",
    "\n",
    "    :param video_path: (str) Path to the folder containing videos\n",
    "    :param prefix: (str) Filter the video, showing only the only starting with this prefix\n",
    "    \"\"\"\n",
    "    html = []\n",
    "    for mp4 in Path(video_path).glob(\"{}*.mp4\".format(prefix)):\n",
    "        video_b64 = base64.b64encode(mp4.read_bytes())\n",
    "        html.append(\n",
    "            \"\"\"<video alt=\"{}\" autoplay \n",
    "                    loop controls style=\"height: 400px;\">\n",
    "                    <source src=\"data:video/mp4;base64,{}\" type=\"video/mp4\" />\n",
    "                </video>\"\"\".format(\n",
    "                mp4, video_b64.decode(\"ascii\")\n",
    "            )\n",
    "        )\n",
    "    ipythondisplay.display(ipythondisplay.HTML(data=\"<br>\".join(html)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oKOjFuwK9HI0"
   },
   "outputs": [],
   "source": [
    "show_videos(video_path='logs/a2c/CartPole-v1_1/videos/', prefix='')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RjdpP0HE8D2p"
   },
   "source": [
    "### Continue Training\n",
    "\n",
    "Here, we will continue training of the previous model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zgMZQJJF6u1C"
   },
   "outputs": [],
   "source": [
    "!python -m rl_zoo3.train --algo a2c --env CartPole-v1 --n-timesteps 50000 -i logs/a2c/CartPole-v1_1/CartPole-v1.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GSaoyiAE8cVj"
   },
   "outputs": [],
   "source": [
    "!python -m rl_zoo3.enjoy --algo a2c --env CartPole-v1 --no-render --n-timesteps 1000 --folder logs/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jL9u4I1H-48O"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "rl-baselines-zoo.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "767d51c1340bd893661ea55ea3124f6de3c7a262a8b4abca0554b478b1e2ff90"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}