examples/notebooks/rllib_single_agent.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Single Agent training using Malmo\n",
    "This guide goes through the usage of Malmo and gives an example on how to train a Reinforcement Learning agent from [Rllib](https://docs.ray.io/en/master/) in Malmo.\n",
    "\n",
    "This notebook requires the ```ray``` python package to be installed. It can easily be installed using pip:\n",
    "```pip install ray ray[rllib] ray[tune]```"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "The first steps are the same as for the Random Agent example."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# imports\n",
    "from pathlib import Path\n",
    "import os\n",
    "\n",
    "# malmoenv imports\n",
    "import malmoenv\n",
    "from malmoenv.utils.launcher import launch_minecraft\n",
    "from malmoenv.utils.wrappers import DownsampleObs\n",
    "\n",
    "import ray\n",
    "from ray.tune import register_env"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to define some constants.\n",
    "\n",
    "The ```MISSION_XML``` is the file defining the current mission. Using RLlib can change the current working directory, so we use its absolute path. This example has been setup to work correctly with both 1 and multiple workers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "ENV_NAME = \"malmo\"\n",
    "MISSION_XML = os.path.realpath('../../MalmoEnv/missions/mobchase_single_agent.xml')\n",
    "COMMAND_PORT = 8999 # first port's number\n",
    "xml = Path(MISSION_XML).read_text()\n",
    "\n",
    "CHECKPOINT_FREQ = 100     # in terms of number of algorithm iterations\n",
    "LOG_DIR = \"results/\"       # creates a new directory and puts results there\n",
    "\n",
    "NUM_WORKERS = 1\n",
    "NUM_GPUS = 0\n",
    "TOTAL_STEPS = int(1e6)\n",
    "launch_script = \"./launchClient_quiet.sh\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Next we want to create a function that defines how the environment is generated in RLlib. This is going to be the python client connecting to the malmo instances, so make sure that these PORT numbers match the ports used later to create the Minecraft instances.\n",
    "When using RLlib each worker has an index accessible by calling ```config.worker_index```, using this variable we can easily set the correct ports for each env.\n",
    "If we would like to use wrappers the ```create_env``` function is a good place to add them, see the ```DownsampleObs``` wrapper added in this example.\n",
    "We downsample the observations from the default ```(800, 600, 3)``` to ```(84, 84, 3)``` as the default vision models in RLlib only support a few dimensions, this being one of them. RLlib can work with any vector based observation and uses convolutional networks for input sizes of (84, 84) and (42, 42) by default. If you want to work with different input sizes check out the [RLlib documentation](https://docs.ray.io/en/master/rllib-models.html).\n",
    "\n",
    "Finally we have to register the env generator function to make it visible to RLlib."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def create_env(config):\n",
    "    env = malmoenv.make()\n",
    "    env.init(xml, COMMAND_PORT + config.worker_index, reshape=True)\n",
    "    env.reward_range = (-float('inf'), float('inf'))\n",
    "\n",
    "    env = DownsampleObs(env, shape=tuple((84, 84)))\n",
    "    return env\n",
    "\n",
    "register_env(ENV_NAME, create_env)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "The next step is to start up the Minecraft instances. Note that this step might take a few minutes.\n",
    "In the background each Malmo instance get copied to the ```/tmp/malmo_<hash>/malmo``` directory, where it gets executed (Each Minecraft instance requires its own directory).\n",
    "After copying the instances are started using a the provided ```launch_script```, this is where we can define if we want to run it without rendering a window for example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "GAME_INSTANCE_PORTS = [COMMAND_PORT + 1 + i for i in range(NUM_WORKERS)]\n",
    "instances = launch_minecraft(GAME_INSTANCE_PORTS, launch_script=launch_script)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "After the Malmo instances are setup and running the next step is to get an agent training.\n",
    "In this example we use ray's tune API to run the training. The algorithm in this example is ```PPO```, but RLlib provides a large collection of RL algorithms and to use a different one you can just replace the first line with the desired algorithm, i.e: ```DQN```.\n",
    "\n",
    "Then we define the ```config```, it includes the environment and the resources we would like ray to use for training. Note that to use a custom environment with ray it has to be registered first and then it can be referred to by its name.\n",
    "The remaining arguments to ```tune.run``` are optional, but are useful in this example. We set the stop condition to be based on the number of agent-env interactions and to make checkpoints every ```CHECKPOINT_FREQ``` algorithm iterations and to save the log files to a custom location (default would be ```~/ray_results/```).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "ray.tune.run(\n",
    "    \"PPO\",\n",
    "    config={\n",
    "        \"env\": ENV_NAME,\n",
    "        \"num_workers\": NUM_WORKERS,\n",
    "        \"num_gpus\": NUM_GPUS,\n",
    "    },\n",
    "    stop={\"timesteps_total\": TOTAL_STEPS},\n",
    "    checkpoint_at_end=True,\n",
    "    checkpoint_freq=CHECKPOINT_FREQ,\n",
    "    local_dir=LOG_DIR\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "To change the algorithm or the arguments check out these links:\n",
    "- [Available algorithms](https://docs.ray.io/en/latest/rllib-algorithms.html)\n",
    "- [Common arguments](https://docs.ray.io/en/master/rllib-training.html#common-parameters)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}