Deep_Reinforcement_Learning_Part_6_.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Deep_Reinforcement_Learning_Part_6_.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [ 
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qN-g6xM8GZd1",
        "colab_type": "text"
      },
      "source": [
        "This is the Part-6 of the Deep Reinforcement Learning Notebook series. In this Notebook I have introduced introduces the first Combined method known as Advantage Actor-Critic (A2C).\n",
        "The Notebook series is about Deep RL algorithms so it excludes all other techniques that can be used to learn functions in reinforcement learning and also the Notebook Series is not exhaustive i.e. it contains the most widely used Deep RL algorithms only."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dTS51o0jG_NR",
        "colab_type": "text"
      },
      "source": [
        "# Actor-Critic Algorithms\n",
        "Actor-Critic algorithms elegantly combine the policy gradient and a learned value function. In these algorithms, a policy is reinforced with a learned reinforcing signal generated using a learned value function. This contrasts with REINFORCE which uses a high-variance Monte Carlo estimate of the return to reinforce the policy.\n",
        "\n",
        "All Actor - Critic algorithms have two components which are learned jointly — an actor which learns a parameterized policy, and a critic which learns a value function to evaluate state-action pairs. The critic provides a reinforcing signal to the actor.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ohbo4rmA3OAb",
        "colab_type": "text"
      },
      "source": [
        "#Concept of Advantage function \n",
        "In Actor-Critic algorithm learning the policy depends on the feedback given by value function estimations which are being learned parallelly.So the problem arises in initial stages where value function is not generating reasonable signals for the policy, learning how to select good actions.\n",
        "\n",
        "It is common to learn the advantage function $A^π$(s, a) = $Q^π(s, a) – V^π(s)$ as the reinforcing signals in these methods. The key idea is that it is better to select an action based on how it performs relative to the other actions available in a particular state instead of using the absolute value of that action as measured by the Q-function. The advantage quantifies how much better or worse an action is than the average available action. Actor-Critic algorithms that learn the advantage function are known as Advantage Actor-Critic (A2C) algorithms.\n",
        "\n",
        "You might think now we have to construct two neural networks for both the Q value and the V value (in addition to the policy network). But we know that would be very inefficient and also care needs to be taken to ensure the two estimates are consistent. We calculate V and not Q and the reason behind this is $Q^π$ is a more complex function and may require more samples and\n",
        "to learn a good estimate and also estimating V(s) from Q(s, a) requires computing the values for all possible actions in state s, then taking the action-probability weighted average to obtain $V^π$(s) which can be computationally expensive. Don't worry we don't do that here. Instead, we use the relationship between the Q and the V from the Bellman optimality equation(Equation 6.1):\n",
        "![alt text](https://github.com/Machine-Learning-rc/Unimportant/blob/master/Screenshot%202020-07-21%20at%204.01.06%20PM.png?raw=true)\n",
        "\n",
        "Equation 6.1\n",
        "\n",
        "So, we the advantage can be written as(Equation 6.2):\n",
        "![alt text](https://github.com/Machine-Learning-rc/Unimportant/blob/master/Screenshot%202020-07-21%20at%204.01.12%20PM.png?raw=true)\n",
        "\n",
        "Equation 6.2\n",
        "\n",
        "We would see 2 methods of estimating of Advantage => n-step returns and Generalized Advantage Estimation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "enAVDBAs48R5",
        "colab_type": "text"
      },
      "source": [
        "# Actor\n",
        "An Actor is one that controls how our agent behaves (policy-based). Actors learn parametrized policies $π_θ$ using the policy gradient similar to reinforce but instead of return we use advantage.\n",
        "\n",
        "Actor update equation can be written as(Equation 6.3):\n",
        "\n",
        "![alt text](https://github.com/Machine-Learning-rc/Unimportant/blob/master/Screenshot%202020-07-21%20at%204.01.20%20PM.png?raw=true)\n",
        "\n",
        "Equation 6.3"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EW9wBj429hT6",
        "colab_type": "text"
      },
      "source": [
        "# CRITIC\n",
        "A Critic is the one that measures how good the action taken by the actor is (value-based). Critics are responsible for learning how to evaluate (s, a) pairs and using this to generating $A^π$.\n",
        "\n",
        "Critic Update is similar to value-update we have seen before(in SARSA,DQN) but the only difference their we learn state-action value function ($Q^π(s,a)$) and here we learn state value function ($V^π$(s))."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u19SJ9uUCZSB",
        "colab_type": "text"
      },
      "source": [
        "# Estimating Advantage: n-step Returns\n",
        "If we assume for a moment that we have a perfect estimate of $V^π(s)$, then the Q-function can be rewritten as a mix of the expected rewards for n time steps, followed by $V^π(s_{n+1})$ as shown in Equation 6.4. To make this tractable to estimate, we use a single trajectory of rewards (r1, . . . , rn) in place of the expectation, and substitute in value-function learned by the critic. Shown in Equation 6.5, this is known as n-step forward returns.\n",
        "\n",
        "![alt text](https://github.com/Machine-Learning-rc/Unimportant/blob/master/Screenshot%202020-07-21%20at%204.59.16%20PM.png?raw=true)\n",
        "\n",
        "Equation 6.5 makes the trade-off between bias and variance of the estimator explicit. The n-steps of actual rewards are unbiased but have high variance since they come from only a single trajectory. value-function learned by the critic has lower variance since it reflects an\n",
        "expectation over all of the trajectories seen so far, but is biased because it is calculated using a function approximator. The intuition between mixing these two types of estimates is that the variance of the actual rewards typically increases the more steps away from t you take. Close to t, the benefits of using an unbiased estimate may outweigh the variance introduced. As n increases, the variance in the estimates will likely start to become problematic, and switching to a lower variance but the biased estimate is better. The number of steps of actual rewards, n, controls the trade-off between the two.\n",
        "\n",
        "The formula for estimating the advantage function combining the n-step estimate for Q with V is(Equation 6.6)\n",
        "\n",
        "![alt text](https://github.com/Machine-Learning-rc/Unimportant/blob/master/Screenshot%202020-07-21%20at%205.11.32%20PM.png?raw=true)\n",
        "\n",
        "Equation 6.6\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qWZwlNNaMoJa",
        "colab_type": "text"
      },
      "source": [
        "# Estimating Advantage: Generalized Advantage Estimation (GAE)\n",
        "\n",
        "Generalized Advantage Estimation (GAE) was proposed by Schulman et al. as an improvement over the n-step return estimate for the advantage function. It addresses the problem of having to explicitly choose the number of steps of returns, n. The main idea behind GAE is that instead of picking one value of n, we mix multiple values of n. That is, calculate the advantage using a weighted average of individual advantages calculated with n = 1, 2, 3, . . . , k. The purpose of GAE is to significantly reduce the variance of the estimator while keeping the bias introduced as low as possible.\n",
        "\n",
        "Intuitively, GAE is taking a weighted average of several advantage estimators with different bias and variance. GAE weights the high bias, low variance 1-step advantage the most, but also includes contributions from lower bias, higher variance estimators using 2, 3, . . . , n steps. The contribution decays at an exponential rate as the number of steps increases. The decay rate is controlled by the coefficient λ. Therefore, the larger λ the higher the variance.\n",
        "\n",
        "![alt text](https://github.com/Machine-Learning-rc/Unimportant/blob/master/Screenshot%202020-07-22%20at%2012.05.47%20PM.png?raw=true)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "21384JnCbYHe",
        "colab_type": "text"
      },
      "source": [
        "# Ways to calculate $V^π_{tar}$\n",
        "For learning the advantage function we need an estimate for $V^π$.\n",
        "\n",
        "We do this by learning $V^π$ using TD learning in the same way that it is used to learn $Q^π$ for DQN. Parametrize $V^π$ with θ, generate $V^π_{tar}$ for each of the experiences an agent gathers and minimize the difference between $V^π$ and $V^π_{tar}$ using a regression loss such as MSE. Repeat this process for many steps.\n",
        "\n",
        "Now $V^π_{tar}$ can be generated a few ways.\n",
        "\n",
        "1) Simple Method: $V^π_{tar}(s)$ = r + $V^π(s';θ)$\n",
        "\n",
        "2) Monte Carlo estimate : $V^π_{tar}(s)$=$\\sum_{t'=t}^Tγ^{t'-t}r_{t'}$\n",
        "\n",
        "3) $V^π_{tar}(s_t)$ = $A^π_{GAE}(s_t,a_t)$ + $V^π(s_t)$\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zZZ5O5rarW-K",
        "colab_type": "text"
      },
      "source": [
        "# Advantage Actor-Critic Algorithm\n",
        "![alt text](https://github.com/Machine-Learning-rc/Unimportant/blob/master/Screenshot%202020-07-22%20at%202.17.47%20PM.png?raw=true)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JxklUmCzuHRe",
        "colab_type": "text"
      },
      "source": [
        "# IMPLEMENTING Advantage Actor-Critic Algorithm"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pkNmUFD4vNNp",
        "colab_type": "text"
      },
      "source": [
        "Below code setups the environment required to run and record the game and also loads the required library."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TfQvkenkvOD4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!apt-get update > /dev/null 2>&1\n",
        "!apt-get install cmake > /dev/null 2>&1\n",
        "!pip install --upgrade setuptools 2>&1\n",
        "!pip install ez_setup > /dev/null 2>&1\n",
        "!pip install gym[atari] > /dev/null 2>&1\n",
        "!pip install gym pyvirtualdisplay > /dev/null 2>&1\n",
        "!apt-get install -y xvfb python-opengl ffmpeg > /dev/null 2>&1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vvz1iHI5vkCV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow.keras.layers import Dense,Dropout,Conv2D, Flatten,MaxPooling2D ,Activation,Input\n",
        "from tensorflow.keras.models import Sequential,load_model,Model\n",
        "import gym\n",
        "import numpy as np\n",
        "import random\n",
        "from collections import deque\n",
        "from tensorflow.keras.utils import normalize as normal_values\n",
        "import cv2\n",
        "from gym import logger as gymlogger\n",
        "from gym.wrappers import Monitor\n",
        "gymlogger.set_level(40) #error only\n",
        "import matplotlib\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "import math\n",
        "import glob\n",
        "import io\n",
        "import base64\n",
        "from IPython.display import HTML\n",
        "from IPython.display import clear_output\n",
        "from IPython import display as ipythondisplay "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WQSBX54PvrOx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from pyvirtualdisplay import Display\n",
        "display = Display(visible=0, size=(1400, 900))\n",
        "display.start()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ePc4f7P3vuPh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\"\"\"\n",
        "Utility functions to enable video recording of gym environment and displaying it\n",
        "To enable video, just do \"env = wrap_env(env)\"\"\n",
        "\"\"\"\n",
        "\n",
        "def show_video():\n",
        "  mp4list = glob.glob('video/*.mp4')\n",
        "  if len(mp4list) > 0:\n",
        "    mp4 = mp4list[0]\n",
        "    video = io.open(mp4, 'r+b').read()\n",
        "    encoded = base64.b64encode(video)\n",
        "    ipythondisplay.display(HTML(data='''<video alt=\"test\" autoplay \n",
        "                loop controls style=\"height: 400px;\">\n",
        "                <source src=\"data:video/mp4;base64,{0}\" type=\"video/mp4\" />\n",
        "             </video>'''.format(encoded.decode('ascii'))))\n",
        "  else: \n",
        "    print(\"Could not find video\")\n",
        "    \n",
        "\n",
        "def wrap_env(env):\n",
        "  env = Monitor(env, './video', force=True)\n",
        "  return env"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XmnPHFL7vytj",
        "colab_type": "text"
      },
      "source": [
        "This part ensures the reproducibility of the code below by using a random seed and setups the environment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "szHgWQ07vwnl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "RANDOM_SEED=1\n",
        "N_EPISODES=500\n",
        "\n",
        "# random seed (reproduciblity)\n",
        "np.random.seed(RANDOM_SEED)\n",
        "tf.random.set_seed(RANDOM_SEED)\n",
        "\n",
        "# set the env\n",
        "env = (gym.make(\"KungFuMaster-v0\")) # env to import\n",
        "env.seed(RANDOM_SEED)\n",
        "env.reset() # reset to env "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3ki3yQwawQyH",
        "colab_type": "text"
      },
      "source": [
        "Defining the A2C Class.You can see that I have commented out few things like temperature_parameter .You can uncomment them if you can want to use Boltzmann policy.In this epsilon greedy works good. So I have used that."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lD4MFuwSwR4l",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class A2C:\n",
        "\n",
        "  def __init__(self, env,path_1=None,path_2=None):\n",
        "    self.env=env #import env\n",
        "    self.state_shape=70, 160, 4 # the state space\n",
        "    self.action_shape=env.action_space.n # the action space\n",
        "    self.gamma=[0.99] # decay rate of past observations\n",
        "    self.learning_rate= 1e-5 # learning rate in deep learning\n",
        "    self.lambda_=0.90       #λ is a hyperparameter for GAE(Generalized Advantage Estimation)\n",
        "    self.alpha=1e-4\n",
        "    if not path_1:\n",
        "      self.Actor_model=self._create_model('Actor')    #Target Model is model used to calculate target values\n",
        "      self.Critic_model=self._create_model('Critic')  #Training Model is model to predict q-values to be used.\n",
        "    else:\n",
        "      self.Actor_model=load_model(path_1) #import model\n",
        "      self.Critic_model=load_model(path_2) #import model\n",
        "    \n",
        "        # record observations\n",
        "    self.states=[]\n",
        "    self.gradients=[]\n",
        "    self.rewards=[]\n",
        "    self.probs=[]\n",
        "    self.next_states=[]\n",
        "    self.actions=[]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "84BlR5ZPwhic",
        "colab_type": "text"
      },
      "source": [
        "Creating a Neural Network Model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XBODWL0kwiQS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "  def _create_model(self,model_type):\n",
        "    ''' builds the model using keras'''\n",
        "    inputs=Input(shape=(self.state_shape))\n",
        "    layer_1=(Conv2D(256, (8, 8), padding='same',strides=(4, 4)))(inputs)  \n",
        "    layer_2=(MaxPooling2D(pool_size=(2,2),padding='same'))(layer_1)\n",
        "    layer_3=(Activation('relu'))(layer_2)\n",
        "    layer_4=(Conv2D(64, (4, 4),strides=(2, 2),  padding='same'))(layer_3)\n",
        "    layer_5=(MaxPooling2D(pool_size=(2,2),padding='same'))(layer_4)\n",
        "    layer_6=(Activation('relu'))(layer_5)\n",
        "    layer_7=(Conv2D(32, (3, 3),strides=(1, 1),  padding='same'))(layer_6)\n",
        "    layer_8=(MaxPooling2D(pool_size=(2,2),padding='same'))(layer_7)\n",
        "    layer_9=(Activation('relu'))(layer_8)\n",
        "    layer_10=(Flatten())(layer_9)\n",
        "    layer_11=(Dense(512))(layer_10)\n",
        "    layer_12=(Activation('relu'))(layer_11)\n",
        "    layer_13=(Dense(self.action_shape))(layer_12)\n",
        "    layer_14=(Activation('softmax'))(layer_13)\n",
        "    layer_15=(Dense(1))(layer_12)\n",
        "\n",
        "    if model_type=='Actor':\n",
        "      model=Model(inputs,layer_14)\n",
        "    else:\n",
        "       model=Model(inputs,layer_15)\n",
        "    model.compile(loss='mse',optimizer=tf.keras.optimizers.Adam(learning_rate=self.learning_rate))\n",
        "    return model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vOoBN2750AJG",
        "colab_type": "text"
      },
      "source": [
        "This is the preprocessing we do to the image we obtained by interacting with the environment. This is the preprocessing we do to the image we obtained by interacting with the environment. Here I have done grayscaling and also cropped the image to remove game scores and area which I found was not necessary to train the agent. This speeds up the training process."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4th1mSqI0Aig",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "  def get_crop_and_grayscale_frame(self,frame):\n",
        "    frame=frame[95:-45,:]\n",
        "    frame=cv2.cvtColor(frame, cv2.COLOR_RGB2GRAY)\n",
        "    frame=(frame-frame.mean())/frame.std()\n",
        "    return frame"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1kOV4ehgxaGQ",
        "colab_type": "text"
      },
      "source": [
        "Action Selection \n",
        "\n",
        "The get_action method guides its action choice. It uses the neural network to generate a normalized probability distribution for a given state. Then, it samples its next action from this distribution.The hot_encode_action method encodes the actions into a one-hot-encoder format."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k-5J5TKfxwUK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "  def get_action(self, state):\n",
        "    '''samples the next action based on the policy probabilty distribution \n",
        "      of the actions'''\n",
        "    action_probability_distribution=self.Actor_model.predict(state).flatten()\n",
        "    # norm action probability distribution\n",
        "    action_probability_distribution/=np.sum(action_probability_distribution)\n",
        "    \n",
        "    # sample action   \n",
        "    action=np.random.choice(self.action_shape,1,p=action_probability_distribution)[0]\n",
        "\n",
        "    return action, action_probability_distribution\n",
        "    \n",
        "  def hot_encode_action(self, action):\n",
        "  '''encoding the actions into a binary list'''\n",
        "\n",
        "  action_encoded=np.zeros(self.action_shape, np.float32)\n",
        "  action_encoded[action]=1\n",
        "\n",
        "  return action_encoded"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TZRXkIwXyUCc",
        "colab_type": "text"
      },
      "source": [
        "The remember method records the observations of each step."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hAzMkXREyS12",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "  def remember(self, state, next_state, action_prob, reward,action):\n",
        "    '''stores observations'''\n",
        "    encoded_action=self.hot_encode_action(action)\n",
        "    self.gradients.append(encoded_action-action_prob)\n",
        "    self.states.append(state)\n",
        "    self.rewards.append(reward)\n",
        "    self.actions.append(action)\n",
        "    self.probs.append(action_prob)\n",
        "    self.next_states.append(next_state)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SEcd889Zyg--",
        "colab_type": "text"
      },
      "source": [
        "The get_GAEs method calculates Generalized advantage estimation (GAE) which later is used to calculate Advantage."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HX_QD8pQyhST",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "  def get_GAEs(self,v_preds):\n",
        "    '''\n",
        "    Advantage Estimation with GAE\n",
        "    '''\n",
        "    gaes = np.zeros((len(self.rewards),1))\n",
        "    future_gae = 0.0\n",
        "    for t in reversed(range(len(self.rewards))):\n",
        "      delta = self.rewards[t] + np.asarray(self.gamma) * v_preds[t + 1] - v_preds[t]\n",
        "      gaes[t] = future_gae = delta + np.asarray(self.gamma) * np.asarray(self.lambda_) *np.asarray(future_gae)\n",
        "    return gaes"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RDBe5ucRz7Oz",
        "colab_type": "text"
      },
      "source": [
        "Updating the models\n",
        "The update_models method updates the Actor and Critic Models."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oJAavZyk0Gly",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "  def update_models(self):\n",
        "    '''\n",
        "    Updates the network.\n",
        "    '''\n",
        "  #get V_preds and V_tar from critic model\n",
        "    states=(np.array(self.states))[:,0,:,:,:]\n",
        "    next_states=np.array(self.next_states)[:,0,:,:,:]\n",
        "    V_s=self.Critic_model.predict(states)\n",
        "    V_next_s=self.Critic_model.predict(next_states)\n",
        "    V_last_state=np.reshape(np.array(V_next_s[-1]),(1,1))\n",
        "    v_all=np.concatenate((V_s,V_last_state),axis=0)\n",
        "\n",
        "    Advatanges=self.get_GAEs(v_all) #Calculating the Advantage\n",
        "  \n",
        "    critic_targets = Advatanges +  V_s\n",
        "    \n",
        "    self.Critic_model.fit(states, critic_targets,epochs=3) #Training the Critic Model\n",
        "\n",
        "    gradients=self.gradients\n",
        "    gradients*=Advatanges\n",
        "    actor_targets=np.asarray(self.alpha)*(gradients)+self.probs\n",
        "    \n",
        "    self.Actor_model.fit(states, actor_targets,epochs=3)  #Training the Actor Model\n",
        "\n",
        "        '''\n",
        "    #Use this if you want to use regularisation for Actor Model training\n",
        "    \tself.beta=0.01\n",
        "    \toptimizer = tf.keras.optimizers.Adam(learning_rate=self.learning_rate)\n",
        "    \t\n",
        "    \tdef train_step(states):\n",
        "      \t\twith tf.GradientTape() as tape:\n",
        "        \t\tprobs= (self.Actor_model)(states,training=True) \n",
        "        \t\tEntropy = (self.probs)*(np.log(self.probs))              \n",
        "        \t\tloss=-(np.asarray(self.alpha)*(gradients) + (self.beta)*Entropy)/(len(states))\n",
        "      \t\tgrads = tape.gradient(loss,self.Actor_model.trainable_variables)\n",
        "      \t\toptimizer.apply_gradients(zip(grads, self.Actor_model.trainable_variables))\n",
        "      \t\t\n",
        "   \t \ttrain_step(states)\n",
        "\n",
        "    '''\n",
        "\n",
        "    self.states=[];self.gradients=[];self.probs=[];self.rewards=[];self.next_states=[];self.actions=[];self.deltas=[];"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4BfouqkW0m-1",
        "colab_type": "text"
      },
      "source": [
        "Training the model\n",
        "This method creates a training environment for the model. Iterating through a set number of episodes, it uses the model to sample actions and play them. When such a timestep ends, the model is using the observations to update the policy.\n",
        "We know that in a dynamic game we cannot predict action based on 1 observation(which is 1 frame of the game in this case) so we will use a stack of 4 frames to predict the output.\n",
        "We can also clip the rewards to help model learn faster."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7NCMYEOW0sDH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "  def train(self,episodes):\n",
        "    env=self.env\n",
        "    for episode in range(episodes):\n",
        "      # each episode is a new game env\n",
        "      state=env.reset()\n",
        "      done=False\n",
        "      state= self.get_crop_and_grayscale_frame(state)\n",
        "      stacked_frames = np.stack((state,state,state,state),axis=2)\n",
        "      stacked_frames = stacked_frames.reshape(1,stacked_frames.shape[0],stacked_frames.shape[1],stacked_frames.shape[2]) \n",
        "      state_=stacked_frames\n",
        "      episode_reward=0 #record episode reward\n",
        "      print(\"Episode Started\")\n",
        "      while not done:\n",
        "        # play an action and record the game state & reward per episode\n",
        "        action,action_prob=self.get_action(state_)\n",
        "        print(\"Episode Going On.\"+\"\\n\"+\"Action taken:\",action)\n",
        "        next_state, reward, done, _=env.step(action)\n",
        "        next_state=self.get_crop_and_grayscale_frame(next_state)\n",
        "        next_state_ = next_state.reshape(1,next_state.shape[0],next_state.shape[1],1)\n",
        "        stacked_frames_1 = np.append(next_state_, stacked_frames[:, :, :, :3], axis=3)\n",
        "        next_state_=stacked_frames_1\n",
        "        self.remember(state_, next_state_, action_prob, reward,action)\n",
        "        state_=next_state_\n",
        "        episode_reward+=reward\n",
        "      print(\"Episode:{}  reward:{}\".format(episode,episode_reward))\n",
        "      self.update_models()\n",
        "      if episode%200==0 and episode!=0:\n",
        "        self.Actor_model.save('Actor_{}.h5'.format(episode))\n",
        "        self.Critic_model.save('Critic_{}.h5'.format(episode))\n",
        "      print(\"Episode Ended\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zp6eLLWk1jpw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "no_of_episodes=10000\n",
        "\n",
        "Agent=A2C(env)\n",
        "Agent.train(no_of_episodes)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "91ZMHg4z1gyx",
        "colab_type": "text"
      },
      "source": [
        "With the help of below code we run our algorithm and see the success of it.With the help of below code we run our algorithm and see the success of it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tp6Kk0-x1hgP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class tester:\n",
        "\n",
        "  def __init__(self,model):\n",
        "      self.Actor_model= load_model(model)     #import model\n",
        "\n",
        "  \n",
        "  def get_action(self, state):\n",
        "    '''samples the next action based on the policy probabilty distribution \n",
        "      of the actions'''\n",
        "    action_probability_distribution=(self.model.predict(state))[0]\n",
        "    action=np.argmax(action_probability_distribution)\n",
        "    return action\n",
        "    \n",
        "\n",
        "  def get_frame(self,frame):\n",
        "    frame=frame[95:-45,:]\n",
        "    frame=cv2.cvtColor(frame, cv2.COLOR_RGB2GRAY)\n",
        "    frame=(frame-frame.mean())/frame.std()\n",
        "    return frame"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m9pA3E6W16K-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "env=(wrap_env(gym.make(\"KungFuMaster-v0\")))\n",
        "state=env.reset()\n",
        "done=False\n",
        "test=tester(\"Actor_Model.h5\")\n",
        "state=test.get_frame(state)\n",
        "stacked_frames = np.stack((state,state,state,state),axis=2)\n",
        "stacked_frames = stacked_frames.reshape(1,stacked_frames.shape[0],stacked_frames.shape[1],stacked_frames.shape[2]) \n",
        "state_=stacked_frames\n",
        "while True:\n",
        "  env.render('ipython')\n",
        "  action = test.get_action(state_)\n",
        "  next_state, reward, done, _=env.step(action)\n",
        "  print(action,reward)\n",
        "  next_state=test.get_frame(next_state)\n",
        "  next_state_ = next_state.reshape(1,next_state.shape[0],next_state.shape[1],1)\n",
        "  stacked_frames_1 = np.append(next_state_, stacked_frames[:, :, :, :3], axis=3)\n",
        "  next_state_=stacked_frames_1\n",
        "  state_=next_state_\n",
        "  if done:\n",
        "    break\n",
        "env.close()\n",
        "show_video()"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}