getting_started/2_Observation_Agents.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# This Notebook will develop how to build an Agent and assess its performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is recommended to have a look at the [0_basic_functionalities](0_basic_functionalities.ipynb) notebook before getting into this one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Objective**\n",
    "\n",
    "This notebook will cover the basics of how to \"code\" an Agent that takes actions on the powergrid. Examples of \"expert agents\" that can take actions based on some fixed rules, will be given. More generic types of *Agents*, relying for example on machine learning / deep learning will be covered in the notebook [3_TrainingAnAgent](3_TrainingAnAgent.ipynb).\n",
    "\n",
    "This notebook will also cover the description of the *Observation* class, which is useful to take some actions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import grid2op"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Impossible to automatically add a menu / table of content to this notebook.\n",
      "You can download \"jyquickhelper\" package with: \n",
      "\"pip install jyquickhelper\"\n"
     ]
    }
   ],
   "source": [
    "res = None\n",
    "try:\n",
    "    from jyquickhelper import add_notebook_menu\n",
    "    res = add_notebook_menu()\n",
    "except ModuleNotFoundError:\n",
    "    print(\"Impossible to automatically add a menu / table of content to this notebook.\\nYou can download \\\"jyquickhelper\\\" package with: \\n\\\"pip install jyquickhelper\\\"\")\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## I) Description of the observations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this paragraph we will cover the observation class. For more information about it, we recommend to have a look at the official documentation, or [here](https://grid2op.readthedocs.io/en/latest/observation.html) or in the [Observations.py](grid2op/Observation/Observation.py) files for more information. Only basic concepts are detailed in this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### I.A) Getting an observation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An observation can be accessed by calling `env.step()`. The next cell is dedicated to creating an environment and getting an observation instance. We use the default `rte_case14_realistic` environment from Grid2Op framework."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/benjamin/Documents/grid2op_dev/getting_started/grid2op/MakeEnv/Make.py:240: UserWarning: You are using a development environment. This environment is not intended for training agents.\n",
      "  warnings.warn(_MAKE_DEV_ENV_WARN)\n"
     ]
    }
   ],
   "source": [
    "env = grid2op.make(test=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To perform a step, as stated on the short description above, we need an action. More information about actions is given in the [2_ActionRepresentation](2_ActionRepresentation.ipynb) notebook. Here we use a *DoNothingAgent*, that does nothing. *obs* is the observation of the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "do_nothing_act = env.helper_action_player({})\n",
    "obs, reward, done, info = env.step(do_nothing_act)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### I.B) Information contained in an Observation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we will detail only the \"CompleteObservation\". `Grid2Op` allows to model different kinds of observations. For example, some observations could have incomplete data, or noisy data, etc. `CompleteObservation` gives the full state of the powergrid, without any noise. It's the default type of observation used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### a) Some of its attributes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An observation has calendar data (eg the time stamp of the observation):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2019, 1, 6, 0, 5, 6)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obs.year, obs.month, obs.day, obs.hour_of_day, obs.minute_of_hour, obs.day_of_week"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It has some powergrid generic information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of generators of the powergrid: 5\n",
      "Number of loads of the powergrid: 11\n",
      "Number of powerline of the powergrid: 20\n",
      "Number of elements connected to each substations in the powergrid: [3 6 4 6 5 6 3 2 5 3 3 3 4 3]\n",
      "Total number of elements: 56\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of generators of the powergrid: {}\".format(obs.n_gen))\n",
    "print(\"Number of loads of the powergrid: {}\".format(obs.n_load))\n",
    "print(\"Number of powerline of the powergrid: {}\".format(obs.n_line))\n",
    "print(\"Number of elements connected to each substations in the powergrid: {}\".format(obs.sub_info))\n",
    "print(\"Total number of elements: {}\".format(obs.dim_topo))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It has some information about the generators (each generator can be viewed as a point in a 3-dimensional space)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generators active production: [81.6    81.1    12.9     0.     77.7201]\n",
      "Generators reactive production: [ 21.790668  70.214264  48.05804   24.508774 -16.541656]\n",
      "Generators voltage setpoint : [142.1      142.1       22.        13.200001 142.1     ]\n"
     ]
    }
   ],
   "source": [
    "print(\"Generators active production: {}\".format(obs.prod_p))\n",
    "print(\"Generators reactive production: {}\".format(obs.prod_q))\n",
    "print(\"Generators voltage setpoint : {}\".format(obs.prod_v))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It has some information about the loads (each load is a point in a 3-dimensional space, too)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loads active consumption: [25.4 84.8 45.   6.8 12.7 28.8  9.5  3.4  5.6 11.9 15.4]\n",
      "Loads reactive consumption: [ 21.790668  70.214264  48.05804   24.508774 -16.541656]\n",
      "Loads voltage (voltage magnitude of the bus to which it is connected) : [142.1      142.1      138.70158  139.39479   22.        21.092138\n",
      "  21.08566   21.453533  21.569204  21.430758  20.69996 ]\n"
     ]
    }
   ],
   "source": [
    "print(\"Loads active consumption: {}\".format(obs.load_p))\n",
    "print(\"Loads reactive consumption: {}\".format(obs.prod_q))\n",
    "print(\"Loads voltage (voltage magnitude of the bus to which it is connected) : {}\".format(obs.load_v))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this setting, a powerline can be viewed as a point in an 8-dimensional space:\n",
    "   * active flow\n",
    "   * reactive flow\n",
    "   * voltage magnitude\n",
    "   * current flow\n",
    "   \n",
    "for both its origin and its extremity.\n",
    "\n",
    "For example, suppose the powerline `line1` is connecting two node `A` and `B`. There are two separate values for the active flow on `line1` : the active flow from `A` to `B` (origin) and the active flow from `B` to `A` (extremity).\n",
    "\n",
    "These powerline features can be accessed with :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Origin active flow: [ 3.9922398e+01  3.7797703e+01  2.1780769e+01  4.0161697e+01\n",
      "  3.3859901e+01  1.7860813e+01 -2.8052214e+01  9.7739801e+00\n",
      "  7.7287230e+00  1.8051615e+01  3.3593693e+00  7.7858996e+00\n",
      " -6.1440678e+00  2.0364611e+00  7.8760457e+00  2.5437183e+01\n",
      "  1.4508085e+01  3.5354317e+01  1.5543122e-14 -2.5437183e+01]\n",
      "Origin reactive flow: [-15.334058    -1.2075976   -7.0024953    0.663526    -0.383117\n",
      "   7.329237    -2.9436882   10.462834     5.576318    14.927625\n",
      "  -0.85521334   4.0310593   -7.464343     1.4842943    7.410275\n",
      " -15.625636    -2.7032974   -5.641245   -23.634314    -5.573329  ]\n",
      "Origin current flow: [173.75766  153.64987   92.956    163.19867  137.5811    78.4405\n",
      " 117.409485 375.74683  250.10808  614.7267    94.88822  239.99174\n",
      " 264.71527   67.45319  291.3341   124.26482   61.42983  148.28416\n",
      " 918.84076  712.8031  ]\n",
      "Origin voltage (voltage magnitude to the bus to which the origin end is connected): [142.1      142.1      142.1      142.1      142.1      142.1\n",
      " 138.70158   22.        22.        22.        21.092138  21.092138\n",
      "  21.08566   21.569204  21.430758 138.70158  138.70158  139.39479\n",
      "  14.850535  21.092138]\n",
      "Extremity active flow: [-3.9602367e+01 -3.7068695e+01 -2.1560818e+01 -3.9274380e+01\n",
      " -3.3242977e+01 -1.7618677e+01  2.8157352e+01 -9.6130629e+00\n",
      " -7.6364613e+00 -1.7751646e+01 -3.3559325e+00 -7.6980476e+00\n",
      "  6.2130628e+00 -2.0243990e+00 -7.7019520e+00 -2.5437183e+01\n",
      " -1.4508085e+01 -3.5354317e+01 -1.5987212e-14  2.5437183e+01]\n",
      "Extremity reactive flow: [ 10.712756    -0.90132874   3.285029    -1.4910296   -1.3327575\n",
      "  -8.036348     3.2753313  -10.125853    -5.3842945  -14.336893\n",
      "   0.86434317  -3.8441856    7.625853    -1.473381    -7.0558143\n",
      "  17.390247     3.8292       8.391265    24.508774     6.244066  ]\n",
      "Extremity current flow: [ 166.68697   153.57782    88.61224   163.59877   137.79755    80.60723\n",
      "  117.409485  375.74683   250.10808   614.7267     94.88822   239.99174\n",
      "  264.71527    67.45319   291.3341   1197.9484    410.72595   953.58575\n",
      " 1071.9808   1018.29016 ]\n",
      "Extremity voltage (voltage magnitude to the bus to which the origin end is connected): [142.1      139.39479  142.1      138.70158  139.39479  138.70158\n",
      " 139.39479   21.453533  21.569204  21.430758  21.08566   20.69996\n",
      "  21.453533  21.430758  20.69996   14.850535  21.092138  22.\n",
      "  13.200001  14.850535]\n"
     ]
    }
   ],
   "source": [
    "print(\"Origin active flow: {}\".format(obs.p_or))\n",
    "print(\"Origin reactive flow: {}\".format(obs.q_or))\n",
    "print(\"Origin current flow: {}\".format(obs.a_or))\n",
    "print(\"Origin voltage (voltage magnitude to the bus to which the origin end is connected): {}\".format(obs.v_or))\n",
    "print(\"Extremity active flow: {}\".format(obs.p_ex))\n",
    "print(\"Extremity reactive flow: {}\".format(obs.q_ex))\n",
    "print(\"Extremity current flow: {}\".format(obs.a_ex))\n",
    "print(\"Extremity voltage (voltage magnitude to the bus to which the origin end is connected): {}\".format(obs.v_ex))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another powerline feature is the $\\rho$ ratio, *ie.* for each powerline, the ratio between the current flow in the powerline and its thermal limit. It can be accessed with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.45143566, 0.3991941 , 0.24462105, 0.42947018, 0.87631273,\n",
       "       0.20642236, 0.30897233, 0.34864962, 0.54149985, 0.7985534 ,\n",
       "       0.3521812 , 0.62351686, 0.34830958, 0.1775084 , 0.38333434,\n",
       "       0.32284945, 0.26599896, 0.86817706, 0.27006915, 0.20950978],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obs.rho"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The observation (*obs*) also stores information on the topology and the state of the powerline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=int32)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obs.timestep_overflow # the number of timestep each of the powerline is in overflow (1 powerline per component)\n",
    "obs.line_status # the status of each powerline: True connected, False disconnected\n",
    "obs.topo_vect  # the topology vector the each element (generator, load, each end of a powerline) to which the object\n",
    "# is connected: 1 = bus 1, 2 = bus 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In `grid2op`, all objects (end of a powerline, load or generator) can be either disconnected, connected to the first bus of its substation, or connected to the second bus of its substation.\n",
    "\n",
    "`topo_vect` is the vector containing the connection information, it is part of the observation.\n",
    "If an object is disconnected, then its corresponding component in `topo_vect` will be `-1`. If it's connected to the first bus of its substation, its component will be `1` and if it's connected to the second bus, its component will be `2`.\n",
    "\n",
    " More information about this topology vector is given in the documentation [here](https://grid2op.readthedocs.io/en/latest/observation.html).\n",
    " \n",
    " More information about this topology vector will be given in the notebook dedicated to vizualisation. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### b) Some of its methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The observation can be converted to / from a flat numpy array. This function is useful for interacting with machine learning libraries or to store it, but it probably makes it less readable for a human. The function proceeds by stacking all the features mentionned above in a single `numpy.float64` vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 2.01900000e+03,  1.00000000e+00,  6.00000000e+00,  0.00000000e+00,\n",
       "        5.00000000e+00,  6.00000000e+00,  8.15999985e+01,  8.10999985e+01,\n",
       "        1.28999996e+01,  0.00000000e+00,  7.77201004e+01,  2.17906685e+01,\n",
       "        7.02142639e+01,  4.80580406e+01,  2.45087738e+01, -1.65416565e+01,\n",
       "        1.42100006e+02,  1.42100006e+02,  2.20000000e+01,  1.32000008e+01,\n",
       "        1.42100006e+02,  2.53999996e+01,  8.48000031e+01,  4.50000000e+01,\n",
       "        6.80000019e+00,  1.26999998e+01,  2.87999992e+01,  9.50000000e+00,\n",
       "        3.40000010e+00,  5.59999990e+00,  1.18999996e+01,  1.53999996e+01,\n",
       "        1.77999992e+01,  5.95999985e+01,  3.07999992e+01,  4.59999990e+00,\n",
       "        8.69999981e+00,  1.97000008e+01,  6.59999990e+00,  2.50000000e+00,\n",
       "        3.90000010e+00,  8.39999962e+00,  1.08999996e+01,  1.42100006e+02,\n",
       "        1.42100006e+02,  1.38701584e+02,  1.39394791e+02,  2.20000000e+01,\n",
       "        2.10921383e+01,  2.10856609e+01,  2.14535332e+01,  2.15692043e+01,\n",
       "        2.14307575e+01,  2.06999607e+01,  3.99223976e+01,  3.77977028e+01,\n",
       "        2.17807693e+01,  4.01616974e+01,  3.38599014e+01,  1.78608131e+01,\n",
       "       -2.80522137e+01,  9.77398014e+00,  7.72872305e+00,  1.80516148e+01,\n",
       "        3.35936928e+00,  7.78589964e+00, -6.14406776e+00,  2.03646111e+00,\n",
       "        7.87604570e+00,  2.54371834e+01,  1.45080853e+01,  3.53543167e+01,\n",
       "        1.55431223e-14, -2.54371834e+01, -1.53340578e+01, -1.20759761e+00,\n",
       "       -7.00249529e+00,  6.63525999e-01, -3.83116990e-01,  7.32923698e+00,\n",
       "       -2.94368815e+00,  1.04628344e+01,  5.57631779e+00,  1.49276247e+01,\n",
       "       -8.55213344e-01,  4.03105927e+00, -7.46434307e+00,  1.48429430e+00,\n",
       "        7.41027498e+00, -1.56256361e+01, -2.70329738e+00, -5.64124489e+00,\n",
       "       -2.36343136e+01, -5.57332897e+00,  1.42100006e+02,  1.42100006e+02,\n",
       "        1.42100006e+02,  1.42100006e+02,  1.42100006e+02,  1.42100006e+02,\n",
       "        1.38701584e+02,  2.20000000e+01,  2.20000000e+01,  2.20000000e+01,\n",
       "        2.10921383e+01,  2.10921383e+01,  2.10856609e+01,  2.15692043e+01,\n",
       "        2.14307575e+01,  1.38701584e+02,  1.38701584e+02,  1.39394791e+02,\n",
       "        1.48505354e+01,  2.10921383e+01,  1.73757660e+02,  1.53649872e+02,\n",
       "        9.29560013e+01,  1.63198669e+02,  1.37581100e+02,  7.84404984e+01,\n",
       "        1.17409485e+02,  3.75746826e+02,  2.50108078e+02,  6.14726685e+02,\n",
       "        9.48882217e+01,  2.39991745e+02,  2.64715271e+02,  6.74531937e+01,\n",
       "        2.91334106e+02,  1.24264816e+02,  6.14298286e+01,  1.48284164e+02,\n",
       "        9.18840759e+02,  7.12803101e+02, -3.96023674e+01, -3.70686951e+01,\n",
       "       -2.15608177e+01, -3.92743797e+01, -3.32429771e+01, -1.76186771e+01,\n",
       "        2.81573524e+01, -9.61306286e+00, -7.63646126e+00, -1.77516460e+01,\n",
       "       -3.35593247e+00, -7.69804764e+00,  6.21306276e+00, -2.02439904e+00,\n",
       "       -7.70195198e+00, -2.54371834e+01, -1.45080853e+01, -3.53543167e+01,\n",
       "       -1.59872116e-14,  2.54371834e+01,  1.07127562e+01, -9.01328743e-01,\n",
       "        3.28502893e+00, -1.49102962e+00, -1.33275747e+00, -8.03634834e+00,\n",
       "        3.27533126e+00, -1.01258526e+01, -5.38429451e+00, -1.43368931e+01,\n",
       "        8.64343166e-01, -3.84418559e+00,  7.62585306e+00, -1.47338104e+00,\n",
       "       -7.05581427e+00,  1.73902473e+01,  3.82920003e+00,  8.39126492e+00,\n",
       "        2.45087738e+01,  6.24406624e+00,  1.42100006e+02,  1.39394791e+02,\n",
       "        1.42100006e+02,  1.38701584e+02,  1.39394791e+02,  1.38701584e+02,\n",
       "        1.39394791e+02,  2.14535332e+01,  2.15692043e+01,  2.14307575e+01,\n",
       "        2.10856609e+01,  2.06999607e+01,  2.14535332e+01,  2.14307575e+01,\n",
       "        2.06999607e+01,  1.48505354e+01,  2.10921383e+01,  2.20000000e+01,\n",
       "        1.32000008e+01,  1.48505354e+01,  1.66686966e+02,  1.53577820e+02,\n",
       "        8.86122437e+01,  1.63598770e+02,  1.37797546e+02,  8.06072311e+01,\n",
       "        1.17409485e+02,  3.75746826e+02,  2.50108078e+02,  6.14726685e+02,\n",
       "        9.48882217e+01,  2.39991745e+02,  2.64715271e+02,  6.74531937e+01,\n",
       "        2.91334106e+02,  1.19794836e+03,  4.10725952e+02,  9.53585754e+02,\n",
       "        1.07198083e+03,  1.01829016e+03,  4.51435655e-01,  3.99194092e-01,\n",
       "        2.44621053e-01,  4.29470181e-01,  8.76312733e-01,  2.06422359e-01,\n",
       "        3.08972329e-01,  3.48649621e-01,  5.41499853e-01,  7.98553407e-01,\n",
       "        3.52181196e-01,  6.23516858e-01,  3.48309577e-01,  1.77508399e-01,\n",
       "        3.83334339e-01,  3.22849452e-01,  2.65998960e-01,  8.68177056e-01,\n",
       "        2.70069152e-01,  2.09509775e-01,  1.00000000e+00,  1.00000000e+00,\n",
       "        1.00000000e+00,  1.00000000e+00,  1.00000000e+00,  1.00000000e+00,\n",
       "        1.00000000e+00,  1.00000000e+00,  1.00000000e+00,  1.00000000e+00,\n",
       "        1.00000000e+00,  1.00000000e+00,  1.00000000e+00,  1.00000000e+00,\n",
       "        1.00000000e+00,  1.00000000e+00,  1.00000000e+00,  1.00000000e+00,\n",
       "        1.00000000e+00,  1.00000000e+00,  0.00000000e+00,  0.00000000e+00,\n",
       "        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,\n",
       "        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,\n",
       "        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,\n",
       "        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,\n",
       "        0.00000000e+00,  0.00000000e+00,  1.00000000e+00,  1.00000000e+00,\n",
       "        1.00000000e+00,  1.00000000e+00,  1.00000000e+00,  1.00000000e+00,\n",
       "        1.00000000e+00,  1.00000000e+00,  1.00000000e+00,  1.00000000e+00,\n",
       "        1.00000000e+00,  1.00000000e+00,  1.00000000e+00,  1.00000000e+00,\n",
       "        1.00000000e+00,  1.00000000e+00,  1.00000000e+00,  1.00000000e+00,\n",
       "        1.00000000e+00,  1.00000000e+00,  1.00000000e+00,  1.00000000e+00,\n",
       "        1.00000000e+00,  1.00000000e+00,  1.00000000e+00,  1.00000000e+00,\n",
       "        1.00000000e+00,  1.00000000e+00,  1.00000000e+00,  1.00000000e+00,\n",
       "        1.00000000e+00,  1.00000000e+00,  1.00000000e+00,  1.00000000e+00,\n",
       "        1.00000000e+00,  1.00000000e+00,  1.00000000e+00,  1.00000000e+00,\n",
       "        1.00000000e+00,  1.00000000e+00,  1.00000000e+00,  1.00000000e+00,\n",
       "        1.00000000e+00,  1.00000000e+00,  1.00000000e+00,  1.00000000e+00,\n",
       "        1.00000000e+00,  1.00000000e+00,  1.00000000e+00,  1.00000000e+00,\n",
       "        1.00000000e+00,  1.00000000e+00,  1.00000000e+00,  1.00000000e+00,\n",
       "        1.00000000e+00,  1.00000000e+00,  0.00000000e+00,  0.00000000e+00,\n",
       "        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,\n",
       "        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,\n",
       "        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,\n",
       "        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,\n",
       "        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,\n",
       "        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,\n",
       "        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,\n",
       "        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,\n",
       "       -1.00000000e+00, -1.00000000e+00, -1.00000000e+00, -1.00000000e+00,\n",
       "       -1.00000000e+00, -1.00000000e+00, -1.00000000e+00, -1.00000000e+00,\n",
       "       -1.00000000e+00, -1.00000000e+00, -1.00000000e+00, -1.00000000e+00,\n",
       "       -1.00000000e+00, -1.00000000e+00, -1.00000000e+00, -1.00000000e+00,\n",
       "       -1.00000000e+00, -1.00000000e+00, -1.00000000e+00, -1.00000000e+00,\n",
       "        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,\n",
       "        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,\n",
       "        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,\n",
       "        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,\n",
       "        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,\n",
       "        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,\n",
       "        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,\n",
       "        0.00000000e+00,  0.00000000e+00], dtype=float32)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vector_representation_of_observation = obs.to_vect()\n",
    "vector_representation_of_observation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An observation can be copied, of course:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "obs2 = obs.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or reset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nan nan nan nan nan]\n"
     ]
    }
   ],
   "source": [
    "obs2.reset()\n",
    "print(obs2.prod_p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or loaded from a vector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([81.6   , 81.1   , 12.9   ,  0.    , 77.7201], dtype=float32)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obs2.from_vect(vector_representation_of_observation)\n",
    "obs2.prod_p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also possible to assess whether two observations are equal or not:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obs == obs2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this type of observation, it is also possible to retrieve the topology as a matrix. The topology matrix can be obtained in two different formats."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Format 1*: the `connectivity matrix` which has as many rows / columns as the number of elements in the powergrid (remember that an element is either an end of a powerline, or a generator or a load) and that tells if 2 elements are connected to one another or not:\n",
    "\n",
    "$$\n",
    "\\left\\{\n",
    "\\begin{aligned}\n",
    "\\text{conn mat}[i,j] = 0 & ~\\text{element i and j are NOT connected to the same bus}\\\\\n",
    "\\text{conn mat}[i,j] = 1 & ~\\text{element i and j are connected to the same bus, or i and j are both ends of the same powerline}\\\\\n",
    "\\end{aligned}\n",
    "\\right.\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 1., 1., ..., 0., 0., 0.],\n",
       "       [1., 0., 1., ..., 0., 0., 0.],\n",
       "       [1., 1., 0., ..., 0., 0., 0.],\n",
       "       ...,\n",
       "       [0., 0., 0., ..., 0., 1., 1.],\n",
       "       [0., 0., 0., ..., 1., 0., 1.],\n",
       "       [0., 0., 0., ..., 1., 1., 0.]], dtype=float32)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obs.connectivity_matrix()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This representation has the advantage to always have the same dimension, regardless of the topology of the powergrid."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Format 2*: the `bus connectivity matrix` has as many rows / columns as the number of active buses of the powergrid. It should be understood as follows:\n",
    "\n",
    "$$\n",
    "\\left\\{\n",
    "\\begin{aligned}\n",
    "\\text{bus conn mat}[i,j] = 0 & ~\\text{if no powerline connects bus i to bus j}\\\\\n",
    "\\text{bus conn mat}[i,j] = 1 & ~\\text{if at least one powerline connects bus i to bus j (or i == j)}\\\\\n",
    "\\end{aligned}\n",
    "\\right.\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 1., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       [1., 1., 1., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       [0., 1., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       [0., 1., 1., 1., 1., 0., 1., 0., 1., 0., 0., 0., 0., 0.],\n",
       "       [1., 1., 0., 1., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       [0., 0., 0., 0., 1., 1., 0., 0., 0., 0., 1., 1., 1., 0.],\n",
       "       [0., 0., 0., 1., 0., 0., 1., 1., 1., 0., 0., 0., 0., 0.],\n",
       "       [0., 0., 0., 0., 0., 0., 1., 1., 0., 0., 0., 0., 0., 0.],\n",
       "       [0., 0., 0., 1., 0., 0., 1., 0., 1., 1., 0., 0., 0., 1.],\n",
       "       [0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 1., 0., 0., 0.],\n",
       "       [0., 0., 0., 0., 0., 1., 0., 0., 0., 1., 1., 0., 0., 0.],\n",
       "       [0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 1., 1., 0.],\n",
       "       [0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 1., 1., 1.],\n",
       "       [0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 1., 1.]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obs.bus_connectivity_matrix()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### c) Simulate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As opposed to most RL problems, in this framework we add the possibility to \"simulate\" the impact of a possible action on the power grid. This helps calculating roll-outs in the RL setting, and can be close to \"model-based\" reinforcement learning approaches (except that nothing more has to be learned).\n",
    "\n",
    "This \"simulate\" method uses the available forecast data (forecasts are made available by the same way we loaded the data here, with the class `GridStateFromFileWithForecasts`. For this class, only forecasts for 1 time step are provided, but this might be adapted in the future).\n",
    "\n",
    "Note that this `simulate` function can use a different simulator than the one used by the Environment. Fore more information, we encourage you to read the official documentation, or if it has been built locally (recommended), to consult [this page](https://grid2op.readthedocs.io/en/latest/observation.html#grid2op.Observation.Observation.simulate).\n",
    "\n",
    "This function will:\n",
    "\n",
    "1. apply the forecasted injection on the powergrid\n",
    "2. run a powerflow with the decidated `simulate` powerflow simulator\n",
    "3. return:\n",
    "    1. the anticipated observation (after the action has been taken)\n",
    "    2. the anticipated reward (of this simulated action)\n",
    "    3. whether or not there has been an error\n",
    "    4. some more informations\n",
    "    \n",
    "From a user point of view, this is the main difference with the previous [pypownet](https://github.com/MarvinLer/pypownet) framework. In pypownet, this \"simulation\" used to be performed directly by the environment, thus giving direct access of the environment's future data to the agent, which could break the RL framework since the agent is only supposed to know about the current state of the environment (it was not the case in the first edition of the Learning to Run A Power Network as the Environment was fully observable). In grid2op, the simulation is now performed from the current state of the environment and it is imperfect since it does not have access to future information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is an example of some features of the observation, in the current state and in the simulated next state :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "do_nothing_act = env.helper_action_player({})\n",
    "obs_sim, reward_sim, is_done_sim, info_sim = obs.simulate(do_nothing_act)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([81.6   , 81.1   , 12.9   ,  0.    , 77.7201], dtype=float32)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obs.prod_p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([81.5   , 79.7   , 12.9   ,  0.    , 79.5781], dtype=float32)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obs_sim.prod_p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## II) Taking actions based on the observation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we will make our first *Agent* that will act based on these observations.\n",
    "\n",
    "All *Agents* must derive from the grid2op.Agent class. The main function to implement for the Agents is the \"act\" function (more information can be found on the official documentation or [here](https://grid2op.readthedocs.io/en/latest/agent.html) ). \n",
    "\n",
    "Basically, the Agent receives a reward and an observation, and chooses a new action. Some different *Agents* are pre-defined in the `grid2op` package. We won't talk about them here (for more information, see the documentation or the [Agent.py](grid2op/Agent/Agent.py) file), but rather we will make a custom Agent.\n",
    "\n",
    "This *Agent* will select among:\n",
    "\n",
    "- doing nothing \n",
    "- disconnecting the powerline having the higher relative flows\n",
    "- reconnecting a powerline disconnected\n",
    "- disconnecting the powerline having the lower relative flows\n",
    "\n",
    "by using `simulate` on the corresponding actions, and choosing the one that has the highest predicted reward.\n",
    "\n",
    "Note that this kind of Agent is not particularly smart and is given only as an example.\n",
    "\n",
    "More information about the creation / manipulation of *Action* will be given in the notebook [2_Action_GridManipulation](2_Action_GridManipulation.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from grid2op.Agent import BaseAgent\n",
    "import numpy as np\n",
    "import pdb\n",
    "\n",
    "\n",
    "class MyAgent(BaseAgent):\n",
    "    def __init__(self, action_space):\n",
    "        # python required method to code\n",
    "        BaseAgent.__init__(self, action_space)\n",
    "        self.do_nothing = self.action_space({})\n",
    "        self.print_next = False\n",
    "        \n",
    "    def act(self, observation, reward, done=False):\n",
    "        i_max = np.argmax(observation.rho)\n",
    "        new_status_max = np.zeros(observation.rho.shape)\n",
    "        new_status_max[i_max] = -1\n",
    "        act_max = self.action_space({\"set_line_status\": new_status_max})\n",
    "        \n",
    "        i_min = np.argmin(observation.rho)\n",
    "        new_status_min = np.zeros(observation.rho.shape)\n",
    "        if observation.rho[i_min] > 0:\n",
    "            # all powerlines are connected, i try to disconnect this one\n",
    "            new_status_min[i_min] = -1\n",
    "            act_min = self.action_space({\"set_line_status\": new_status_min})\n",
    "        else:\n",
    "            # at least one powerline is disconnected, i try to reconnect it\n",
    "            new_status_min[i_min] = 1\n",
    "#             act_min = self.action_space({\"set_status\": new_status_min})\n",
    "            act_min = self.action_space({\"set_line_status\": new_status_min,\n",
    "                                         \"set_bus\": {\"lines_or_id\": [(i_min, 1)], \"lines_ex_id\": [(i_min, 1)]}})\n",
    "    \n",
    "        _, reward_sim_dn, *_ = observation.simulate(self.do_nothing)\n",
    "        _, reward_sim_max, *_ = observation.simulate(act_max)\n",
    "        _, reward_sim_min, *_ = observation.simulate(act_min)\n",
    "            \n",
    "        if reward_sim_dn >= reward_sim_max and reward_sim_dn >= reward_sim_min:\n",
    "            self.print_next = False\n",
    "            res = self.do_nothing\n",
    "        elif reward_sim_max >= reward_sim_min:\n",
    "            self.print_next = True\n",
    "            res = act_max\n",
    "            print(res)\n",
    "        else:\n",
    "            self.print_next = True\n",
    "            res = act_min\n",
    "            print(res)\n",
    "        return res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We compare this Agent with the `Donothing` agent (already coded) on the 3 episodes made available with this package. To make this comparison more interesting, it's better to use the L2RPN rewards (`L2RPNReward`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The results for DoNothing agent are:\n",
      "\tFor chronics with id 000\n",
      "\t\t - cumulative reward: 11076.768555\n",
      "\t\t - number of time steps completed: 10 / 10\n"
     ]
    }
   ],
   "source": [
    "from grid2op.Runner import Runner\n",
    "from grid2op.Agent import DoNothingAgent\n",
    "from grid2op.Reward import L2RPNReward\n",
    "from grid2op.Chronics import GridStateFromFileWithForecasts\n",
    "\n",
    "max_iter = 10  # to make computation much faster we will only consider 50 time steps instead of 287\n",
    "runner = Runner(**env.get_params_for_runner(),\n",
    "                agentClass=DoNothingAgent\n",
    "               )\n",
    "res = runner.run(nb_episode=1, max_iter=max_iter)\n",
    "\n",
    "print(\"The results for DoNothing agent are:\")\n",
    "for _, chron_name, cum_reward, nb_time_step, max_ts in res:\n",
    "    msg_tmp = \"\\tFor chronics with id {}\\n\".format(chron_name)\n",
    "    msg_tmp += \"\\t\\t - cumulative reward: {:.6f}\\n\".format(cum_reward)\n",
    "    msg_tmp += \"\\t\\t - number of time steps completed: {:.0f} / {:.0f}\".format(nb_time_step, max_ts)\n",
    "    print(msg_tmp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The results for the custom agent are:\n",
      "\tFor chronics with id 000\n",
      "\t\t - cumulative reward: 11076.768555\n",
      "\t\t - number of time steps completed: 10 / 10\n"
     ]
    }
   ],
   "source": [
    "runner = Runner(**env.get_params_for_runner(),\n",
    "                agentClass=MyAgent\n",
    "               )\n",
    "res = runner.run(nb_episode=1, max_iter=max_iter)\n",
    "print(\"The results for the custom agent are:\")\n",
    "for _, chron_name, cum_reward, nb_time_step, max_ts in res:\n",
    "    msg_tmp = \"\\tFor chronics with id {}\\n\".format(chron_name)\n",
    "    msg_tmp += \"\\t\\t - cumulative reward: {:.6f}\\n\".format(cum_reward)\n",
    "    msg_tmp += \"\\t\\t - number of time steps completed: {:.0f} / {:.0f}\".format(nb_time_step, max_ts)\n",
    "    print(msg_tmp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, both agents obtain the same score here, but there would be a difference if we didn't limit the episode length to 10 time steps.\n",
    "\n",
    "**NB** Disabling the time limit for the episode can be done by setting `max_iter=-1` in the previous cells. Here, setting `max_iter=10` is only done so that this notebook can run faster, but increasing or disabling the time limit would allow us to spot differences in the agents' performances."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same can be done for the `PowerLineSwitch` agent :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The results for the PowerLineSwitch agent are:\n",
      "\tFor chronics with ID 000\n",
      "\t\t - cumulative reward: 2164.011719\n",
      "\t\t - number of time steps completed: 3 / 10\n"
     ]
    }
   ],
   "source": [
    "from grid2op.Agent import PowerLineSwitch\n",
    "runner = Runner(**env.get_params_for_runner(),\n",
    "                agentClass=PowerLineSwitch\n",
    "               )\n",
    "res = runner.run(nb_episode=1, max_iter=max_iter)\n",
    "print(\"The results for the PowerLineSwitch agent are:\")\n",
    "for _, chron_name, cum_reward, nb_time_step, max_ts in res:\n",
    "    msg_tmp = \"\\tFor chronics with ID {}\\n\".format(chron_name)\n",
    "    msg_tmp += \"\\t\\t - cumulative reward: {:.6f}\\n\".format(cum_reward)\n",
    "    msg_tmp += \"\\t\\t - number of time steps completed: {:.0f} / {:.0f}\".format(nb_time_step, max_ts)\n",
    "    print(msg_tmp)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}