cartPole/cartPole2WithExperienceReplaySaveBestWeights.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **CartPole-v1 (maximum 500 steps)**\n",
    "#### Single Neural Network With Experience Replay\n",
    "#### The input to the network consists of 4 float values (Cart Position, Cart Velocity, Pole Angle and Pole Velocity at the Tip).\n",
    "#### The outputs of the network don't go through any activation function; instead of probabilities, the outputs are directly state-action values.\n",
    "##### (The optimal policy involves choosing the Argmax of the output vector of the Neural Network as the action for the next step).\n",
    "##### **This environment is considered to be solved when getting an average of 475 over 100 consecutive episodes.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 340
    },
    "colab_type": "code",
    "id": "oqXU0xmddNLk",
    "outputId": "0e418af3-12fc-462d-9ac2-d7fba6734855"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 32)                160       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 32)                1056      \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 8)                 264       \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 8)                 72        \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 2)                 18        \n",
      "=================================================================\n",
      "Total params: 1,570\n",
      "Trainable params: 1,570\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# Imports the libraries and creates the model\n",
    "# Hyperparameter tuning was carried out for the model (learning_rate, optimizers, batch_size, etc.) and for the learning loop (number of replay steps, discount factor, epsilon_decay, etc.)\n",
    "\n",
    "import random\n",
    "import numpy as np\n",
    "from tensorflow import keras\n",
    "import gym\n",
    "env = gym.make('CartPole-v1')\n",
    "model = keras.Sequential()\n",
    "model.add (keras.layers.Dense(units = 32, input_shape=(4,), activation = \"relu\"))\n",
    "model.add (keras.layers.Dense(units = 32, activation = \"relu\"))\n",
    "model.add (keras.layers.Dense(units = 8, activation = \"relu\"))\n",
    "model.add (keras.layers.Dense(units = 8, activation = \"relu\"))\n",
    "model.add (keras.layers.Dense(units = 2))\n",
    "model.compile(optimizer=keras.optimizers.Adam(lr=0.001),loss =\"mse\")\n",
    "print (model.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_9ZHkM86M-NI"
   },
   "outputs": [],
   "source": [
    "# Memory Replay is carried out in this function.\n",
    "# 32 random previous experiences per step are used to fit the model without overfitting most recent states\n",
    "# This function has been vectorised with numpy \"Put_along_axis\" function (see comment down below) to avoid a slow for loop.\n",
    "\n",
    "def executeMemoryReplayStep(memoryPreviousStates, memoryStates, memoryARD, model, itemsInMemory):\n",
    "    randomList = np.random.permutation(itemsInMemory) # After that position, everything is 0 in Memory\n",
    "    \n",
    "    memoryPreviousStates = memoryPreviousStates[randomList[:numberOfLearningFromReplaySteps]] # Shape: (32, 1)\n",
    "    memoryStates = memoryStates[randomList[:numberOfLearningFromReplaySteps]] # Shape: (32, 1)\n",
    "    memoryARD = memoryARD[randomList[:numberOfLearningFromReplaySteps]] # Shape: (32, 3) Action, Reward, Done\n",
    "\n",
    "    # These variables are vectorised and contain the whole batch:\n",
    "    \n",
    "    previousState = memoryPreviousStates \n",
    "    \n",
    "    state = memoryStates\n",
    "    \n",
    "    \n",
    "    action, reward, done = np.split(memoryARD,3,axis=1)\n",
    "    \n",
    "    action = action.astype(int)\n",
    "    \n",
    "    target = model.predict(previousState)\n",
    "    \n",
    "    # This function changes the value for the action selected in \"action\" into the \"target\" array.\n",
    "    # I.e., for each row, one of the two columns are modified, these indexes (0 or 1) for each row appear in the \"action\" array\n",
    "    # The individual updates are those of the Q-learning algorithm: only the reward obtained if done=False=0 and the reward plus the discounted value of the next state if done=True=1\n",
    "    \n",
    "    np.put_along_axis(target, action, (reward + (1-done)*discount*np.expand_dims(np.amax(model.predict(state), axis=1),axis=1)), axis =1)\n",
    "    \n",
    "    \n",
    "    targetArray = np.array(target)\n",
    "    \n",
    "    model.fit(memoryPreviousStates, targetArray, epochs=1, verbose = 0, batch_size =1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def checkIfTheseWeightsAreOptimal (model):\n",
    "\n",
    "# This function checks if our weights are optimal.\n",
    "# The environment is considered to be solved if its average over 100 consecutive episodes is over 475.0. Doing sanity checks at the power-of-2 episodes avoid completing the whole testing loop if the weights are not good:\n",
    "\n",
    "    rewards = []\n",
    "    for episode in range (100):\n",
    "        \n",
    "        done = False\n",
    "        prevObs = env.reset()\n",
    "\n",
    "        episodeReward = 0\n",
    "        for step in range (600):\n",
    "            if done:\n",
    "                break\n",
    "            env.render()\n",
    "\n",
    "            target = model.predict(np.expand_dims(prevObs, axis = 0))[0]\n",
    "            action = np.argmax(target)\n",
    "\n",
    "            obs, reward, done, _ = env.step (action)\n",
    "\n",
    "\n",
    "\n",
    "            prevObs = obs\n",
    "            episodeReward += reward\n",
    "\n",
    "        rewards.append(episodeReward)\n",
    "        \n",
    "        if (episode in [0,1,3,7,15,31,63]):\n",
    "            if (np.average(rewards)<450):\n",
    "                env.close()\n",
    "                return False\n",
    "            else:\n",
    "                print (\"Up until the episode number \", (episode+1), \". The average reward looks promising...\")\n",
    "        \n",
    "    if (np.average(rewards)>=475):\n",
    "        import matplotlib.pyplot as plt\n",
    "        plt.plot (list(range(len(rewards))), rewards)\n",
    "        plt.title(\"Rewards obtained along the testing process\")\n",
    "        plt.xlabel(\"Episode number\")\n",
    "        plt.ylabel(\"Reward obtained\")\n",
    "        plt.show()\n",
    "        return True\n",
    "    else:\n",
    "        return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "y48ruy6h9tcg",
    "outputId": "bd58d048-f905-4754-9918-f7ac6d242016"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0  Episode Reward:  31.0 .Epsilon: 0.995\n",
      "Weights are not optimal\n",
      "1  Episode Reward:  12.0 .Epsilon: 0.990025\n",
      "2  Episode Reward:  13.0 .Epsilon: 0.985074875\n",
      "3  Episode Reward:  31.0 .Epsilon: 0.9801495006250001\n",
      "4  Episode Reward:  12.0 .Epsilon: 0.9752487531218751\n",
      "5  Episode Reward:  15.0 .Epsilon: 0.9703725093562657\n",
      "6  Episode Reward:  14.0 .Epsilon: 0.9655206468094844\n",
      "7  Episode Reward:  74.0 .Epsilon: 0.960693043575437\n",
      "8  Episode Reward:  17.0 .Epsilon: 0.9558895783575597\n",
      "9  Episode Reward:  23.0 .Epsilon: 0.9511101304657719\n",
      "10  Episode Reward:  16.0 .Epsilon: 0.946354579813443\n",
      "Weights are not optimal\n",
      "11  Episode Reward:  16.0 .Epsilon: 0.9416228069143757\n",
      "12  Episode Reward:  15.0 .Epsilon: 0.9369146928798039\n",
      "13  Episode Reward:  26.0 .Epsilon: 0.9322301194154049\n",
      "14  Episode Reward:  13.0 .Epsilon: 0.9275689688183278\n",
      "15  Episode Reward:  15.0 .Epsilon: 0.9229311239742362\n",
      "16  Episode Reward:  18.0 .Epsilon: 0.918316468354365\n",
      "17  Episode Reward:  28.0 .Epsilon: 0.9137248860125932\n",
      "18  Episode Reward:  13.0 .Epsilon: 0.9091562615825302\n",
      "19  Episode Reward:  28.0 .Epsilon: 0.9046104802746175\n",
      "20  Episode Reward:  18.0 .Epsilon: 0.9000874278732445\n",
      "Weights are not optimal\n",
      "21  Episode Reward:  9.0 .Epsilon: 0.8955869907338783\n",
      "22  Episode Reward:  12.0 .Epsilon: 0.8911090557802088\n",
      "23  Episode Reward:  115.0 .Epsilon: 0.8866535105013078\n",
      "24  Episode Reward:  24.0 .Epsilon: 0.8822202429488013\n",
      "25  Episode Reward:  19.0 .Epsilon: 0.8778091417340573\n",
      "26  Episode Reward:  31.0 .Epsilon: 0.8734200960253871\n",
      "27  Episode Reward:  83.0 .Epsilon: 0.8690529955452602\n",
      "28  Episode Reward:  29.0 .Epsilon: 0.8647077305675338\n",
      "29  Episode Reward:  32.0 .Epsilon: 0.8603841919146962\n",
      "30  Episode Reward:  49.0 .Epsilon: 0.8560822709551227\n",
      "Weights are not optimal\n",
      "31  Episode Reward:  13.0 .Epsilon: 0.851801859600347\n",
      "32  Episode Reward:  35.0 .Epsilon: 0.8475428503023453\n",
      "33  Episode Reward:  16.0 .Epsilon: 0.8433051360508336\n",
      "34  Episode Reward:  17.0 .Epsilon: 0.8390886103705794\n",
      "35  Episode Reward:  45.0 .Epsilon: 0.8348931673187264\n",
      "36  Episode Reward:  71.0 .Epsilon: 0.8307187014821328\n",
      "37  Episode Reward:  47.0 .Epsilon: 0.8265651079747222\n",
      "38  Episode Reward:  39.0 .Epsilon: 0.8224322824348486\n",
      "39  Episode Reward:  18.0 .Epsilon: 0.8183201210226743\n",
      "40  Episode Reward:  35.0 .Epsilon: 0.8142285204175609\n",
      "Weights are not optimal\n",
      "41  Episode Reward:  22.0 .Epsilon: 0.810157377815473\n",
      "42  Episode Reward:  82.0 .Epsilon: 0.8061065909263957\n",
      "43  Episode Reward:  78.0 .Epsilon: 0.8020760579717637\n",
      "44  Episode Reward:  66.0 .Epsilon: 0.798065677681905\n",
      "45  Episode Reward:  42.0 .Epsilon: 0.7940753492934954\n",
      "46  Episode Reward:  18.0 .Epsilon: 0.7901049725470279\n",
      "47  Episode Reward:  22.0 .Epsilon: 0.7861544476842928\n",
      "48  Episode Reward:  16.0 .Epsilon: 0.7822236754458713\n",
      "49  Episode Reward:  26.0 .Epsilon: 0.778312557068642\n",
      "50  Episode Reward:  65.0 .Epsilon: 0.7744209942832988\n",
      "Weights are not optimal\n",
      "51  Episode Reward:  62.0 .Epsilon: 0.7705488893118823\n",
      "52  Episode Reward:  9.0 .Epsilon: 0.7666961448653229\n",
      "53  Episode Reward:  92.0 .Epsilon: 0.7628626641409962\n",
      "54  Episode Reward:  104.0 .Epsilon: 0.7590483508202912\n",
      "55  Episode Reward:  54.0 .Epsilon: 0.7552531090661897\n",
      "56  Episode Reward:  15.0 .Epsilon: 0.7514768435208588\n",
      "57  Episode Reward:  33.0 .Epsilon: 0.7477194593032545\n",
      "58  Episode Reward:  18.0 .Epsilon: 0.7439808620067382\n",
      "59  Episode Reward:  65.0 .Epsilon: 0.7402609576967045\n",
      "60  Episode Reward:  78.0 .Epsilon: 0.736559652908221\n",
      "Weights are not optimal\n",
      "61  Episode Reward:  25.0 .Epsilon: 0.7328768546436799\n",
      "62  Episode Reward:  34.0 .Epsilon: 0.7292124703704616\n",
      "63  Episode Reward:  13.0 .Epsilon: 0.7255664080186093\n",
      "64  Episode Reward:  8.0 .Epsilon: 0.7219385759785162\n",
      "65  Episode Reward:  32.0 .Epsilon: 0.7183288830986236\n",
      "66  Episode Reward:  209.0 .Epsilon: 0.7147372386831305\n",
      "67  Episode Reward:  88.0 .Epsilon: 0.7111635524897149\n",
      "68  Episode Reward:  91.0 .Epsilon: 0.7076077347272662\n",
      "69  Episode Reward:  55.0 .Epsilon: 0.7040696960536299\n",
      "70  Episode Reward:  20.0 .Epsilon: 0.7005493475733617\n",
      "Up until the episode number  1 . The average reward looks promising...\n",
      "Weights are not optimal\n",
      "71  Episode Reward:  104.0 .Epsilon: 0.697046600835495\n",
      "72  Episode Reward:  58.0 .Epsilon: 0.6935613678313175\n",
      "73  Episode Reward:  107.0 .Epsilon: 0.6900935609921609\n",
      "74  Episode Reward:  43.0 .Epsilon: 0.6866430931872001\n",
      "75  Episode Reward:  38.0 .Epsilon: 0.6832098777212641\n",
      "76  Episode Reward:  91.0 .Epsilon: 0.6797938283326578\n",
      "77  Episode Reward:  41.0 .Epsilon: 0.6763948591909945\n",
      "78  Episode Reward:  47.0 .Epsilon: 0.6730128848950395\n",
      "79  Episode Reward:  46.0 .Epsilon: 0.6696478204705644\n",
      "80  Episode Reward:  33.0 .Epsilon: 0.6662995813682115\n",
      "Up until the episode number  1 . The average reward looks promising...\n",
      "Weights are not optimal\n",
      "81  Episode Reward:  35.0 .Epsilon: 0.6629680834613705\n",
      "82  Episode Reward:  47.0 .Epsilon: 0.6596532430440636\n",
      "83  Episode Reward:  26.0 .Epsilon: 0.6563549768288433\n",
      "84  Episode Reward:  36.0 .Epsilon: 0.653073201944699\n",
      "85  Episode Reward:  45.0 .Epsilon: 0.6498078359349755\n",
      "86  Episode Reward:  92.0 .Epsilon: 0.6465587967553006\n",
      "87  Episode Reward:  149.0 .Epsilon: 0.6433260027715241\n",
      "88  Episode Reward:  29.0 .Epsilon: 0.6401093727576664\n",
      "89  Episode Reward:  133.0 .Epsilon: 0.6369088258938781\n",
      "90  Episode Reward:  144.0 .Epsilon: 0.6337242817644086\n",
      "Up until the episode number  1 . The average reward looks promising...\n",
      "Up until the episode number  2 . The average reward looks promising...\n",
      "Up until the episode number  4 . The average reward looks promising...\n",
      "Up until the episode number  8 . The average reward looks promising...\n",
      "Up until the episode number  16 . The average reward looks promising...\n",
      "Up until the episode number  32 . The average reward looks promising...\n",
      "Up until the episode number  64 . The average reward looks promising...\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO2deZxcdZXov6eWrk66EwIhICTBsERRGRaNgIIbuAAyog6uoI7KoPN0dNRxwTfOqE+f4xvfgNuouIKoIw83hlFHFFFRFhNAQFGILBJAkkCWTtJdXct5f9zf79atqlvVtdzqWvp8P5/+9K271e/W7957fmf5nSOqimEYhmEApPrdAMMwDGNwMKFgGIZhhJhQMAzDMEJMKBiGYRghJhQMwzCMEBMKhmEYRogJhQWAiHxFRD6U8DmvFpFzEjzfb0XkmUmdL3LeZ4rIpg6Pfb+IXJJ0mzpBRFREDkvgPGvcuTJJtKvN736aiPxhvr/XaA8TCl0gIveIyLSI7BKRP7uX72S/29VvOnnxqOoTVPXqHjZraEha4A4KqvoLVX1sv9thNMeEQvf8papOAkcDxwDn9ash/Rj9GQYM1703TG3tByYUEkJV/wz8N4FwAEBEciLyMRH5k4g8JCKfFZFFbtvPROSv3PKJbmR9mvv8bBG52S0fKiJXicjDIrJVRL4mIssi33GPiLxbRG4BdotIRkSOEZEbRWRKRL4JjEf231dErhCR7SLyiIj8QkRi7wMReaqI/FpEdrj/T63Z5VARucFt/56I7OPW/9z93+60qKe0eB3PdsvvF5FLReRidw2/FZF1kX0PFJFvicgWEblbRN4S2bbIaWzbROR3wJOb9ZuIfFxE7hORnSKyQUSe1mTfF7i2bHej+cfVtP8fROQW93t8U0Siv/u7RORBEXlARM5pZA4SkQ8DTwM+5X67T0U2P1tE7nTX9mkRkchxrxOR2922/xaRRze77shxe4nIF13b7heRD4lI2m3r5N5r+DtIjSkvqd/M7Xu1iHwk7n6Uiub6ehH5E3BVC/25WkS+7e6xh6P90Oi3loDzRWSza8MtInKE23aaiPzO3c/3i8g/tNI/fUFV7a/DP+Ae4NlueRVwK/DxyPYLgMuBfYAlwH8CH3HbPgh80i2/F/gj8NHIto+75cOA5wA5YAXBC/eCmjbcDKwGFgFjwL3A24AscCZQAD7k9v8I8Fm3LUvwApKYa9sH2Aa8CsgAr3Cfl7vtVwP3A0cAE8C3gEvctjWAApnI+Vq5Dv9bvh+YAU4D0q7N17ltKWAD8E/uWg8B7gKe57b/C/AL1/7VwG3ApiZ9eDaw3F3jO4A/A+ORdvhregyw211DFngXsBEYi7T/BuBA9923A290205x530CsBj4qvt9DmvQpquBc2rWKXAFsAw4CNgCnOK2vdC15XHuOv4R+FWDc1f1DfBd4HOuD/dz1/CGTu69Fn6HZ0b7oge/2Vz348Vu26Jm/Ulwz/0GON/tPw6cONdvDTyP4N5cBojb5wC37UHgaW55b+CJ/X5/NXwm+t2AYf5zN/UuYMrddD8Blrlt4m66QyP7PwW42y2fDNziln8InEPlxfcz4MUNvvOFwE01bXhd5PPTgQeIvOiBX1ERCh8Evtfo4Yoc8yrghpp11wJ/7ZavBv4lsu3xwKx7oPxDmGly/rjriAqFH9ece9otHwf8qeZc5wFfdst34V6W7vO5NBEKMe3aBhwVaYd/sbwPuDSyX4rgJfTMSPvPjmz/P8Bn3fKXcIMB9/kwOhMKJ0Y+Xwq8xy3/AHh9Tdv2AI+OOXfYN8D+QB73QnfbXwH8tJN7r4Xf4ZnUC4Ukf7O57sdDItsb9ifBc7qFmPu32W8NnATcARwPpGqO+xPwBmBpq/div/7MfNQ9L1TVJQQ30+HAvm79CoIRzgannm4nePmvcNuvBR4jIvsTmJwuBlaLyL7AsTgTjIjsJyL/4VTOncAlke/w3BdZPhC4X92d6Lg3svyvBCOdH4nIXSLyngbXdWDNcf48Kxt8770EI67attHGdUT5c2R5DzAugS340cCB/jd1v+t7CV5wvt217WqIiLzDmQJ2uHPt1aBdVb+Hqpbd90R/j9o2+6CD2jZFl9uh0fkfDXw88ns8QjAoWUlzHk3QZw9Gjv0cgcbQyb03VzvbuaZOfrO57sfa56RRf64G7lXVYsx3NPytVfUq4FPAp4GHRORCEVnqjvsrAs33XglMx09p4Xr6ggmFhFDVnwFfAT7mVm0FpoEnqOoy97eXBk5pVHUPgar5VuA2VZ0lGNG/Hfijqm515/kIwSjnSFVdSmDuCG3J/usjyw8CK6P2ZgJzg2/nlKq+Q1UPAf4SeLuInBxzSQ8QPABRDiIYTXlW12wruOuOS73bynW0wn0E2tayyN8SVT3NbX8wpl2xSOA/eDfwUmBvVV0G7GjQrqrfw/2+q6n+PRrxIIF50bO60Y6OdlMX30dg8on+JotU9VctHJcH9o0ct1RVn+C2t3vvJUm7v1ntPtH70RNta7P+vA84SOId0k1/a1X9hKo+icDs9RjgnW79r1X1DAKB+10CTW8gMaGQLBcAzxGRo93I4/PA+SLiR14rReR5kf1/BrzZ/YdABY5+hsAXsYvAabsSd5M14VqgCLzFOf5eTKB54Npwuogc5h6CnUDJ/dXyfQJN5pXuPC8jUMmviOxztog8XkQWE5ilLlPVEoHqXSaw93d6HY24AdjpHJyLRCQtIkeIiHcoXwqcJyJ7i8gq4O+anGsJwW+1BciIyD8BSxvseynwfBE5WUSyBP6HPIEgn4tLgdeKyOPcb/VPc+z/ENW/3Vx8luCanwCh8/glcx2kqg8CPwL+r4gsFZGUBM7lZ7hdkuqzTmj3N4PG92Oj8zfqzxsIhNK/iMiEiIyLyAnuuIa/tYg8WUSOc+fbTeAXK4nImIicJSJ7qWqBynM3kJhQSBBV3UJgBnqfW/VuAlPNdU79/jEQjdP+GcGD9/MGnwE+ADyRYAT7X8C352jDLPBi4K8J7OMvqzlmrWvHLgIB8u8aMz9AVR8GTid4WB4mcMSdHtFgIHD+fQXnnAXe4o7dA3wY+KVTs49v9zqaXF+JQMM5GribYCT4BQKzD+577nXbfuTa2Ij/JrAR3+GOmaGBmUJV/0AwUv6k+86/JAhHnm2hzT8APgH8lOB+uNZtyjc45OPAmS665RMtnP87wEeB/3D32W3AqXMd53g1gXP1dwT3y2XAAW5bIn3WCR38ZtDgfmxw/ob9GbnHDiPwBWwieI7m+q2XEgwEtxHcTw9TsRy8CrjHHfNG990DiVSbng3D6DUShD7eBuQa2K2NGub6zUTkaoKggC/Md9tGDdMUDGMeEJEXOTPC3gQjzf80gdAc+836gwkFw5gf3kDgu/gjgT35b/vbnKHAfrM+YOYjwzAMI8Q0BcMwDCNkqBND7bvvvrpmzZp+N8MwDGOo2LBhw1ZVXRG3baiFwpo1a1i/fn2/m2EYhjFUiEjDmf5mPjIMwzBCTCgYhmEYISYUDMMwjBATCoZhGEaICQXDMAwjpKdCQYJye7eKyM0ist6t20dErpSgrOCVbgq7L2X3CRHZKEEZuyf2sm2GYRhGPfOhKTxLVY9WVV9j9z3AT1R1LUGlMl/k5VSCDJ5rCaplfWYe2mYYhmFE6Mc8hTMIqpQBXERQQ+Ddbv3FrmLYdSKyTEQOcDnfE+XX9zzCL+7YkvRpO+KYg/bmWYfvV7Xuge3TXLr+Psrl1lOQjI+lec1T1jCRa61Lf/TbP3Pb/TvaausRK/fiuU94VNW6zTtn+MYN91Eql9s6lyeTTvHK4w5i38lc1fqf3bGFg5dPcNDyxVXrr7vrYX61sZK9+/hDl/PUQ6uLgd2zdTffvul+GIAULieuXcGxB+/TcPt9j+zhsg2bfMlG9ls6zlnHHUS0RtLufJGv/Ooe8oXkU/A/8/D9eOJBezfcvmnbHu58aFfdPfrI7lm+dt29FEpBvy9dlOV1JxxMKhVfN0lVueT6P7Fl50zzBonwgqMO4LD9llStvvm+7Vx1+0MtXNHC4eTH7c9Rq5clft5eCwUlKPuowOdU9UJgf/+iV9UHfQEagjJ40Vz2m9y6KqEgIucSaBIcdFDDolpNufHebXzypxs7OjZJVOHRyxfXPXD/b/0mLvjxnUiLdcn8u2/tfkt4zuP3b74zwQP6jkt/w1S+2NZ37LckVycULv/NA5z/4zsAWj5X9JwAk7kMrzvx4Kptb/76jbz4mJV84IwjqtZ/5Ae/5zf3bUckOP6qP2zmir97WtU+F117D1/+5T1ttydpVOFXf3yYy/72qQ33ueS6e/ncz+8Krwdg9T6LecZjKpNNP/uzP/LJqzYmfj2qcOOftnPJOcc13OeiX93DJdf9idv/1ylV679/64P83yvvqFr3tLUreOyjql/mnod25nnfd28Dmt8nqvCtDZu48u1PZ/FY8HravHOGV33xeqZmWr9fFwL7LR0fSqFwgqo+4F78V4rI75vsG9fddUM9J1guBFi3bl1HQ8E3PONQ3vCMQzs5NFHO+/at/Dhm9LOnUCSXSfGHD7VWJ+WOh6Z47vk/J19sbST5wI4ZpvJFPvTCIzj7+NqKm/H88/du43u/eaBufb4YjBT/8KFTyGXSLZ3LUy4rh/7P77NjulC1vlgqMzVTZOdMfZbkqekCpx95AJ965RP5+/+4iZvu2163z87pIiuXLeKX7zmprfYkzRu+up57H97TdJ/NU3lW7b2Ia959ErPFMs/62NVc8OM7ePrafRERtu+Z5cu/vIfn/8UBfPqsZN1sZ3/heqbn0D52z5aYLpQolspk0hVr85Trm9s/eArXbNzK31y8PtQa4s8T7P/xlx/NGUc3Lh3963se4SWfvZYLfnwn7z3tcQB88IrfkS+Uueodz+CQFc3KPRtJ0FOfgqo+4P5vBr5DUBbyIRE5AMD93+x230R1jdVVBHVUR5ZcJhVrEsgXyuQyrXdN1j2sxVJrMvKOh6YAeMz+8aO6ONKpVOz5/Ysgk2r/VkqlhMlchp0z1UJhV75Y9b9226QzkU2OZ9gVIzh25QvhPv1kYiwTvgwbsWUqz4olgelsLJPiTc86jJv+tJ2f3xmYyL54zd3syhd5y8lrE29fLpNiZg6h4LfvzlfvtztfJCUwnk2RcSajZkJhejY4fjzbfODw5DX78IpjV/PFa+7mtw/s4Kd/2MwVtzzIm551mAmEeaJnQsHVNl3il4HnElROuhx4jdvtNcD33PLlwKtdFNLxwI5e+BMGiVw2FY60o+SLZXJzPDxR/EM52+ShjLLxoV0ArN2v9Ycsm5bYh75YUkQg3cCWPBdLx7PsnK5+cXrNYXeMUNidL4Z+k8lclqlGgmO8/0JhcS7Nnnzzl+6WqTwrIv6UM5+0ipXLFnHBj+9g2+6KltDILNMNje6/KPlCsL1WuO2eLTIxlkFEwr4vNfGBeS12LqEA8O5TDmfvxVnO+/atvO+7t3Hoigne+Mx2SlYb3dBLTWF/4BoR+Q1BIez/UtUfAv9CUNz+TuA57jMEheLvIqjH+nngf/SwbQPBeCZNvlimtqZFvlBiPNt614w5raLZSC3KHQ9Nse9kjr0nxlr+jkxaYh/6YlnJdqAleJYuytZpCl5I1AqFclnZPVuKCIU0s8UyszUvtl35UssO917SiqawdVdFU4BqbeENl2zomZYA/v5rLrT89tq+2J0vsjgXvOAz6UAoFJsIhenZoI8WtSAUli0e432nP55bNu1g07Zp/veL/qJt06TROT17clT1LuComPUPAyfHrFfgTb1qzyCScy/+fLFcNYLKF8ttPQReU2jZfLR5F4/Zvz1VPJNKUSwrqloVGRPYmjv3/i0dz7CzxqfghUSt+WiPM2VMupeRNxHtzhcZy1QE3K6ZAquWLeq4TUmxeCzDTKFcZ4/3FEplHtkzWxd5deaTVvHpn27khrsf6ZmWAMH9N1NoPpCYCTWFGvNRRDh702Gz+8+boVod7LzgqAP51caH2W9pjuMOWd7SMUYy2IzmPjLuXvz5mgdzpk1NIduGpqCqbHxoqi3TEQTmI6gfDRbLGgqlTgg0heqX/87QfFRvxwbCl5H/Xys8on6HfjLhhNeeBnb7R3bPokqVpgCBtvD3z17LWDrVMy0BIJdJzxnmWvEp1GsKEy46yJuPik3Ckr1DuxVNAUBE+OiZR/KO5z62pf2N5Oj/k7OAqWgKJSAbrm9XUxhLe6Ewt6bwwI4Zds+WWNuGkxkCRzMEo8Hoc11oMApulcCnsLNqndcUal9E/uXvX/hLxuOFwu4BMR/5kMo9+RJLx7N127dM5YF6oQDwknWrOeWIR7Ek5rikaMWnMNPAfLQnXwqFXjhgaElTMDPQoGOaQh/xmkKtCt+uptBK9Ienk8gjqDz4hZrRYLHUraZQH33kfQq7ZotV/pZQUxirOJqhWiiUyzowjmb/0mzkV9iyq7FQAHoqEMBpCjE+rSgzDRzNUW2soimYUBgFTCj0kWpNoUK7mkI6JYgE9v25uNMJhXbNR438FoVyOQyJ7YSl41mmZopVTmwvJFRhT8SWvavOfBT8RtGw1Fq/Qz+JagpxhJrCZLxQ6DU+7LmZtuDvzV21przZYnh9vv+bRR+F5qOx/veL0RwTCn0k10BTyBdLbc1TEBGyqRSzLZiP7nxoV9uRR0BoIqq1G5fK2p2jeZEb7Ude7FHHc9Rs4fdpZj6q7NPbUXYrTIzNoSk4oVDraJ4vWhEK/t7c08RE14pPwZ9nvI372ugP1kN9ZLyBpjBTKLetZjeaR1BLJ5FH/vxQryl0bT5yL/aoCSnqeI6+8P3L1ZuG4sxHu/KFqn36yWL30tzTQChs3ZVnSS7Tt9Gzv8eaOZubO5pdSGoL0W/ThRLZtHTlfzLmB+uhPuI1hdqRWruaAgQj+bnMRz7yqF1/AjQOOyyUujQfOU0hmuqiWlOImo+CZW82ijMf+X0GwXzk21AbReWJzmbuBy2Zj2JCUktlZboQCUltoEVGmSmUQh+aMdiYUOgjXlOoTTXQmaYwt/no/u3TLvKofU0h08jRXNaOZzMDYVROtaZQCM9ZpSnURB95h/Ogmo9Cn0IT81G/TEdAOGu+0QS2clnDWfJRTcFfjxfKmRYdzePmTxgKTCj0kSQ1hbG0zKkp3LnZp7dIVlPoKiR1kTMfTUd9CkX2dyPo3TVCISWVWPdUSpgYS8ebjwYgJNULrYaawq7+agrevt9oAlv0voxqCt75X5m8Nneai2CgY6+bYcB6qY/EaQqqykyhvdxHEKjwc/kU7gzDUTvXFOIczdkeaAoHuBnJUSftrnwl346nNilexXzUf6HgfQVxOZwAtvbbfDSHphC9L6sc/jUamx8wNJsnMz1bannimtFfTCj0kThNwavr7WoK2bRQmKMozx0P7WLFkhzLFrcXeeTPDw0czQlEH+2s8SkcsNc4UG8+qp2UNpnLsCsqOGYGx9E8lkkxlk7VpYiA4IW7c6Y4GD6FBprCTDFeKPgQW28eS6e9ptB8RrMJheHAhEIfidMUvIDoxKdQmGN26p0PTXWkJUDEfFTz4Hc7T2FJLoNIJeKoWCqze7bEgV5TqBIKlVm0nslctaawe7baGd1vFufSsT6Frbt8OGr7AjopwuijBvdN1KxUq7FB+z6FdrVfoz+YUOgjcZqCFxDtawpzm4/u2rKbQzvMSV+ZNZ1sSGpYU8FpCr54y/5LnaZQE55aaxaaHM9UaRNTM0XG0qmByao5MZaJ9Sk0S3ExX+Qy8YEOHm9Wyqal6hpqZ5a3EpI6Y5rC0GBCoY/EPZRelW8/JFWajtQgmO0bl4OntfM3djSnu0idDS7/kTP7+P/LFmWdE7n6ZRRnPtpd42geBNORZ/FYI01hFoAVk+Pz3aSQuUJSvaawz8RYtcYWRh+1k+bCHM3DgvVSH0mlhLF0dVKydoqRRMmmU3V1BaKoKqUuwkebhaRmu/ApgMuU6qKP/P+li7JM1L3w64XCRC4TahcQmJgGwcnsWZzLxPoUBkFTGG/R0bzPRK7OjAcV85GIkEk1j34zn8LwYEKhzwQlOaPmo840hbF0qulIzW/r1NTjC+mUajSFIM1Ft5pCpk5TWDqeqXMi756tNx8tyVUXspmaqRcc/WRiLF2XIgIqQmF5H30KuTlCUr1QWD4xxp5CibK7h/bUaAoQaAvNQ1JLlgxvSOi5UBCRtIjcJCJXuM8ni8iNInKziFwjIoe59TkR+aaIbBSR60VkTa/bNgjksumqKI9ONYXMHGkuvNmn0xd4o5DUQqncVUgqeE3BCQX3P05TiNMCfEiqz/S5K19gyQAJhcVjDTSFXTPsvTjblZO+W+YKSfUa7D4TY6hWktrtqvEpQKCpNq28ZkJhaJiPO/KtwO2Rz58BzlLVo4GvA//o1r8e2KaqhwHnAx+dh7b1nVpNoVOfQuBobqYpBOftWFNIN3E0d2s+cplSIaIpLMrW+wtitICJXIZiWcMX2O58aaB8CpONoo+mZvtqOoLK5LWGIaleU3DajNfI9swGqd2jpsj0HOajfKFsGVKHhJ4KBRFZBTwf+EJktQJL3fJewANu+QzgIrd8GXCyRGcpjSjj2VSNptBpSGpzTcGr9p2+wNMNQlKL5QQczYsyEU3B+RTGM0zkMqGjebZYZrZUrstp5LUCP3qN8zv0k8W5BtFHfZ7NDIHWmE5J1f0XxQuL5S6jrr+OuCiwbJNAh2Ip6DvLfTQc9PrpuQB4FxDNq3AO8H0RmQZ2Ase79SuB+wBUtSgiO4DlwNboCUXkXOBcgIMOOqinjZ8PgpKIMSGpbUZqzBWS6kf4nWoKjUJSC6UEHM3jWabyQU2FnTMFUhKYJiZz6VBTqC3F6QlLcs4U2Xcyx9TMYJTi9EyMpWNnNG+ZynPMQcv60KJqajXVKF5YLJ+sTjmyO1+ppeAJNIV4oTDjBjqLxsyFOQz0rJdE5HRgs6puqNn0NuA0VV0FfBn4N39IzGnq7jJVvVBV16nquhUrViTa5n4QlESs1xTajbPPpFJN48S9ptDpqD7bICS1WCqHE9s6JVpTYed0gSXj2SCvUcR8VFtgxzNZoynszhfDOguDwOKxDNOFUp0TdstUvm/FdaKMZ9NNQlJ99JHXFPxvXF/uNJNq7FOwqmvDRS+fnhOAF4jIacA4sFRE/gs4XFWvd/t8E/ihW94ErAY2iUiGwLT0SA/bNxCMN9AU2o3pHstImCIjDq9FdGo+yjRIZZBISGqkpsLOmSJ7OSExmatMTAtrKcQ4miEQCsVSOUjpPDY4QsGHbU4XKk7y3fki04VS381HEGgKDSev1ZqPZiuCd6LGP5BJS8M0F9OzJhSGiZ5pCqp6nqquUtU1wMuBqwj8BnuJyGPcbs+h4oS+HHiNWz4TuEqbFY8dEXINfArtagrZOeoplBIKSa1zNHdZeQ2qayrsmC6EmVMnchnyxTKFUrmh+WgyYj7yUT6D5GiulOSsmJD6XXEtSi6TaqwpFEukUxIKae9T2DNb77dJpxrn3uo0os7oD/P69Dhfwd8A3xKRMrANeJ3b/EXgqyKykUBDePl8tq1fJKUpZFJzRR91F5KajglJ9RPiujYfRTKl7pwuhJ8nIiPrRsVzouajSvbOwXn5eE0hGpa6ZVf/J655AvNRo8lrZcYzqap+gOC3Xrn3oqp9s6lU3RwWz/Ss8ymYUBgK5kUoqOrVwNVu+TvAd2L2mQFeMh/tGSQS0xQyc8xT6DIkNc7R3K3z2hOtqbBzpsAh+wb5mfzLfVe+2FhTiJiPBqnAjmfxWPULFQZjNrMnMB819imMZ9OVuhCzXlOoN9GlU9Kw8pq/v00oDAcWDtBn4jSFbFraTkeRTTWPPvIO4k7TXMQ5mkNB0+UELG+eCDSFYpX5CAKzRdyEKWigKQyQ+WgirL5WEfxbB0hTyGUaawr5YplcJsXisKxo47DfZrm3Kj4Fe90MA9ZLfSYu+qiTDJ/ZdIqyNq5+5R/YTp3C6ZQgUu1o7vacnmhNhZ0z9eajqKZQ62helE2TksCnMIjmo/CFOlutKaQE9u6grkXSBPdfc00hm04xlkmxezaYOb5ntj6FeaZJmotpiz4aKkwo9Jla9b2TUpwQSVjXQFvwL/NuJpplU6kqZ2IxIfPR5FhQU+GR3bPsmS2FQiIardPIfCQibpLbYJqPQk0hMoFty1Se5ZO5rmpbJ0Uuk25iPqpUAPSzy/PFMqWyxoakNrr3LCR1uDCh0Ge8o88HWgUphtt/eMbSPjoo/sH0L/Bu8hTVpjIolpIxH6VSwpJchvu3TwOVENWJsWpH81gmGLHWssQJhd2DaD6K0RQ2T+XZbwBMR1CvqUbJF0uhyWfxWLqpGa9ZQjwvFCzNxXBgQqHP5DKB2cc7bb0dt10alcv0FMvd+RTAJ92LOJoTMh9BYELatG06XAbCSWj+hd9oprJPijflhcIgzVNo4GgeBH8C1Pu0oswUSmFqCq8p7AnTZrfuU/CayHgH97Ux/1gv9ZnanPadli3MzKUpdJn7CHwmzHpNodvcRxCEpW7atidchtqQ1GLDEpsTLn12xcQ0OCNS71OIOpo3T80MmKYQf8/ki+Uw3Yr/jSuaQr1PodGAZNo0haHChEKfqc1p36mmEJqPGozWSmFIauddXvvge60hGU0hw0M7827ZC4VKjP+ufLHhTOVJV2hnV77Iomy6a3NWkoylU2RSEgqsUlnZumuW/Zb0r+JalCD3UeMiO15T8OajuFoKEAxK5kxzYQnxhoLBeXoWKLU57YOIjy4czQ1GfYUuQ1KhPmd+ZZZ0MppCuOxCUnOZNNm0MDXT3Hy0xNVpHrQCOxA4woOSnEH/btszS6msg2M+apr7qFJC05uPGuWgalZ5bboQ+INSA+BYN+bGhEKfidcUOgtJhWbRR1q1Xydk0tUPfrf5lKJ47QCqBYRPihdXnzncZ6yyzyAlw/NEE/ttdtrQwJiPMilmS+WwqlqUaLW0xWMZ9syWQuFWa6Jr6mietVKcw4QJhT5T61PId6gpNCqC4ymE9v/uoo+qQlKTdDRXaQoRoTBWGaHO5Whutk8/iWoKm6dmANhv6aAIBX//1Q8moqbMyVy6aoJgrSmvWXfbezcAACAASURBVOW1qMZhDD7WU33GP3T+oey1ptDNnIJsKhUbkpqIo9mZjIJaCpXrnwzDTesnTHmWuFrOUzOFgXIye7yTFiIpLiYHw6fgX9ZxYalVmkJEY4N681GzymvTBdMUhgkTCn3GCwDvjMsXOpu8FqahaJB/Jonoo8B8FONoTsBW7DWFpYuyRAvuTeTSYWRRQ/NRLoNq8MIdpIlrnsVj6TCUc7MTCoOuKaiqiz6qhKQWy8q2PUGFvFrh26zy2ozVZx4qTCj0mcpILXgoZ4qdTV7zL/vZYqOSiN07hWsjTColPpPQFJxQGK9+qU/kMi4tdnPzEcCDO2YG06cwVq0pLBnPDMxLsuLTqtYUKmVhXUiq0962TM2QSUkY7eYJEuI1DkkdlOs15saEQp8JR2pdagpzzWiupLnoxnxUnQmzUE7Q0exe5t6M5JnMZdi6a5ay1pssovtA8CIbRPPR4lwm9CkM0sQ1iPq0qu+b2jDSxe433rwzz0QuQ2359EyqcT2PvPkUhgrrqT6TnKbQ3HyUxJyCdKp6RnMldUZvNQXvnG2oKUTWD6L5aGIsHTpoB2niGkR8WjWzmsMU7pGQVAjMX3H9MFdCPPMpDA89FwoikhaRm0TkCvdZROTDInKHiNwuIm+JrP+EiGwUkVtE5Im9btsg4G22M4USxVKQbKybNBeNzEelBNJc1FZ3KyYZkjoeLxQmc5lQELUiFAbSfJTLhJXXtkzlB2biGlRe+jM1juY6TcGZjzZPzYTLUdLpxpXXpgslm808RMzHE/RWgpKbS93nvyaoxXy4qpZFZD+3/lRgrfs7DviM+z/SjEeijyp23M4T4s3laO56nkLkwS8kENHk8WajOPORp5mjOVwewJfPxFiaPYUS5bKyeVDNR4Va81H1vej7YeuuWQ7Yq7rqGrjKa80czTabeWjoqaYgIquA5wNfiKz+W+CDqloGUNXNbv0ZwMUacB2wTEQO6GX7BoGopuBHZ7mOZjTPlSW1e59CbcnPpLKkQnPzUWW5QUhqRDuYHB8889FiFx21dXeePbOlwTQfNdIUwiypwW8cpM2O0RSc+SiurPpMocT4AAprI55em48uAN4FRN9UhwIvE5H1IvIDEVnr1q8E7ovst8mtq0JEznXHrt+yZUuv2j1vRG26lVKcyU9eKyYxTyEtsUV2ktAUJscyPGrpOGv3n6xeH3kBteZTGLyXj9de7n04SPg3KOGoEA2JbuBTyFRrClA/cQ0q90BcBFJQ63nw+sWIp2dCQUROBzar6oaaTTlgRlXXAZ8HvuQPiTlN3R2mqheq6jpVXbdixYpE29wPsukU6ZSQL5a7KkYy1+S1Yrnsqqd1M08hVV2Os9S9ScqTSgm/es9JvHTd6qr1E22ajwbR0exH2Xdv3Q0MzsQ1aENTiAjbuH7w2mKcCSnwKVhMy7DQS5/CCcALROQ0YBxYKiKXEGgA33L7fAf4slveROBr8KwCHuhh+waGoPpaqUtNwQmFBsnNimXtutJXJiVhGGpwzuQczUBswrTqF3787ZrLpMi6Wg+DVGDH480t9zihMEiawpwhqdkYTSFGG/OaQqFUHT1XcMETpikMDz0T36p6nqquUtU1wMuBq1T1bOC7wElut2cAd7jly4FXuyik44Edqvpgr9o3SPhMlRWfQueT1xpNICqVtOuZxw1TZycQktqIVhzNviRnsP/gvXy8puDNRysmB0coNJq8NlMzeS2XSYWDinhNIdhWqylYLYXhox/Dqn8BviYibwN2Aee49d8HTgM2AnuA1/ahbX0hCU3BRx/NNimy07WmkI53NKcT0hTiiL6AFjcRlpO5DNv3FAbSfORH1ndv3c1YOsWyxYPTxlzNPBlPOEBxI3yfAnxqJr6uRSOfgtVnHj7mRSio6tXA1W55O0FEUu0+CrxpPtozaHhNoZuQ1DD3UUNHc7nrKKFeOpob4Uf+E2Pppvn4vUYxiOYjrync8/BuVizJdeXXSZrKjPrmk9egUswoTlPwSRFr77+Z2c7vaaM/mPdnAPCaQmV01n63BE7k5llSu315B6kMeuNoboR/Ac1VPMcLhWbaRL/wI+s9syX2HaBwVAjum2xa6iav5WNG+H7SWtxckIr5svr+C81HA9gvRjwmFAaAXAKaAgQv58b1FBIQCul6R7NId3Mf5qLiK5hDKIxnmMxlBrK6VzRyZ5DmKHhymXTM5LX6EpqTTQR0aD6q1RRqopiMwcd6agBIQlOAIGFdU02hyxF9nKO5l05mqIyy5zILTeYyA5kMD6oF2iAKhfFsKiYktUxKqnNleTNYo8lrUO9TME1h+Bg8A+wCJJdJMTVTrJsw1C7ZTKqhUCiUygloCkHqbFVFJCiq0kstAYKXzeKxdKxzM8rLn3wQxx2yvKdt6ZRcJkVKoKwMVIoLTy5TX6c5XyyRy6Rrals4oRDTF9kG8xS6iagz+kPDJ01EpoiZPOZR1aWNthntMZ5Ns3XXbMSO29nouzYNRZRSAtFH2cho0BdVSWqOQjMmcpk5fQonrt2XE9fu2/O2dIKIMDGWYSpfHKhkeB6vqUaJK6EZOv1jHc2VeQrV5zFNYdho+KSp6hIAEfkg8GfgqwSzjs8ClsxL6xYIuUyKfFVIamcP0Fi6sfmomIT5KDIazKYDn0IvncyeNcsX8+jli3v+Pb1kcS7thMIAagrZek0hrlra4iY+hWyDeQo+fYbNUxgeWjEfPU9Vo9lKPyMi1wP/p0dtWnCEIamFUp0dtx2ymcaFTooJmI8q+ZWCWavFBJzXrfDV1x/XczNVrwlMLoOVIdWTy6TqhUJMXY+KoznOpxCfpXfaHM1DRys9VRKRs1xdhJSInAXUV/k2OiZ0NBfLdXbcdsjUFMGJkoSpJ10TYVIo6bxoCuPZ9Lx8Ty/xEUiDlOLCE28+qq8A6H0JTSev1dx/07NmPho2WtEUXgl83P0p8Eu3zkiIqKbQzYgqCEltpCkkEZLq8iu50WAwIW64R/DzhY/c2XeAUlx4xrNptk8Xqtbli+U65/CLn7iSfSayTUNS68xHRZvRPGzMKRRU9R6CWgdGj6iEpJY79idAc6GQqKPZjQaLpe7PuVCYGEuzz8TYQGo83qcVJSiMU93W1fss5lVPWRN7Dj84qK2+NjPbXZi1Mf/M2VMi8hgR+YmI3OY+Hyki/9j7pi0ccpk0xbKye7bYUYEdj88UGkcSTuHa9MjFcrnn8xRGhcc+ailHr17W72bEEudozsc4mpuRSfl7I843kRqo1B5Gc1oxH30eeCfwOQBVvUVEvg58qJcNW0h4k9HOmWJXKYYzzcxHSWgK6eqww2JpfkJSR4H3nHp4v5vQkPFYTaE+JLUZlZDUep+C+ROGi1Z6fbGq3lCzrtiLxixUvGq9Y7rQlaYw1mOfQu2s1UICYa5G/8ll66OP/OS1VmmUOjsutNUYbFp5oreKyKG4iWwiciawIOoczBf+odk5XehKU/ATyuIolsuhit8p/viKplDuukaD0X9ymXRLk9eakQlDUuvTXJimMFy0Yj56E3AhcLiI3A/cDZzd01YtMLx2sGO6wKq9F3V8nkw6xWyzymtdmnq8+cgczaPFeIymMFNs16fg7425J8EZg00r0Ud3Ac8WkQkgpapTvW/WwsJrBzunC11FH4253ERxlMoJVF5LV09QKpTLc+YkMgYfH+hQLFVqbrT7Mm9U+a9djcPoP3M+0SKSA/4KWANkfBSBqn6wlS8QkTSwHrhfVU+PrP8k8FpVnYx8z8XAk4CHgZe5cNiRx2sKxbJ25VPINEtzUdJw1mmn1IakluYp95HRW7xPK18MhIKqBi/zNsJIMw2K7EwXSpbiYshopde/RzBPoQjsjvy1yluB26MrRGQdUBuf93pgm6oeBpwPfLSN7xhqotpBN/Hc2XSKQkPzUTJZUoNzVWY0d+unMPqP1wi8CcmXdG0ns2k6nLwWYz7qQvs15p9WdP9VqnpKJycXkVUEpTc/DLzdrUsD/0owK/pFkd3PAN7vli8DPiUi4sp0jjRR9bob+2s2LXWThzxJhI/WZsIslsod52kyBoeKphA4m30Su3YGKNkG5qPpQolx0xSGilZ6/Vci8hcdnv8C4F1AdPjwZuByVa2NYFoJ3AegqkVgB1CXIF9EzhWR9SKyfsuWLR02a7BIVFNoliU1oXkKoaPZQlJHAm+y9MIgrhTnXNTmxfLkC2XTFIaMVp7oE4ENIvIHEblFRG4VkVvmOkhETgc2q+qGyLoDgZcAn4w7JGZd3bBXVS9U1XWqum7FihUtNH/wSU5TSNU9lJ5kKq/VOJoTyLxq9B//0q7VFNq5F7PpJiGpYzZwGCZaMR+d2uG5TwBeICKnAePAUuC3QB7Y6BzWi0Vko/MjbAJWA5tEJAPsBTzS4XcPFUlpCpm0hPbgWpJ4gdeaCEoJaB9G//Gagq/T7JPYtXMvNvIp2Izm4aNhr4uIr6w21eCvKap6nqquUtU1wMuBq1R1b1V9lKqucev3OIEAcDnwGrd8ptt/5P0JQFXEUTeawli6cT2FJBLihY7mSOpsMx8NP35Q4iew5TvQFDIxaS5Ute35Dkb/aaYpfB04HdhAYMaJvlEUOCThtnwR+KqIbCTQEF6e8PkHlsQ0hVSKstYLAFVNpvJaraO5bI7mUcCbL330USXddev3i4iQTklVmot8sYyqpc0eNpqV4zzd/T+42y9R1auBq2PWT0aWZwj8DQuOxHwKmcpLO52qnMc/qN2HpFabj4oWkjoS5DLVIakzHTiaITAhFSLmo040DqP/tDQdVUT2BtYS+AYAUNWf96pRC42xyAi+q+ijSG6i6IPoX+LdhqRWJihVHM2mKQw/jUJS240ayqSEUsR85Etxmk9huGhlRvM5BBPQVgE3A8cD1wIn9bZpCwcRCevkdldkJz59cTEhTaH2/Emk4zb6jx9AhCGp3tHc5uz6TKo6IeOM1WceSlrprbcCTwbuVdVnAccAozFBYIDwD2ZX5Tgz1SN5jx+9dZvmIlpkR1UTCXM1+k9imkI6FYYrQ0VTMPPRcNHKEz3j7P2ISE5Vfw88trfNWnj4B7MrTcG99GvDUv2D2q2pJ3Q0l8vhiNBSZw8/oU+hUOtTaF9TqHU0d3Ieo7+04lPYJCLLgO8CV4rINuCB3jZr4ZGMphA/q9S/wLsOSY3MWvXfYZrC8BPOaA41BW8+at+nEDVd+pnR3Qx0jPmnldTZPj/R+0XkpwSTyn7Q01YtQJLQFGqL4Hgqo/ruXuDpSM78QkLah9F/QvNR6FNoP/cRQDptmsIoMGdvichX/bKq/kxVLwe+1NNWLUAS0RTSDcxH7nO3moKIhEn3iqVktA+j/4gIY5lKoZ18oYRI+0Ihm6qu51ERLqYpDBOt9PoToh9cltMn9aY5C5dEfArp5uajJGofZFIpSmUN/RRmPhoNcplUaDaaKZbJZVL42imtkk5JVZBDvoN0GUb/aZbm4jwRmQKOFJGdIjLlPm8mqLFgJIi363ZTZMdrCrXmo8rkte4fTl/IxwseczSPBuPZdNXktU4ihjI1lf/yBdMUhpGGbwlV/YiqLgH+VVWXquoS97dcVc+bxzYuCHz4XzdphitCoVpTKCRkPgIXi26O5pEjmCcTjOy37yl0dB9majSFmQ7nOxj9pZXoo/eKyIsJUmgr8AtV/W5vm7XwSEZTqM5N5EkqzQVUYtHN0Txa5DIp8oUyv//zTr5/64Oc+aRVbZ8jXTN5Ld9BsR6j/7TSW58G3gjcCtwGvFFEPt3TVi1A/Mis2yI7QNUEIqhoDkn4FLIu7DDUFCz30Ugwnk0zXSjx3m/fytJFWd59yuFtnyPbIPrIzEfDRSuawjOAI3waaxG5iEBAGAmSy6YY68C5F8W/9GeL1eajZH0KQXruJE1SRv/JZVJcc+dWZktl/u2lR7H3xFjb50g706LHm6PGTFMYKlrprT8AB0U+rwbmrLxmtMfyiRzLO3gQo4w1cDRXIoWSMB8FJgIvaMx8NBrkMmlmS2WeeuhyXnTMyo7Oka1Jc5EvBgkTbeAwXDTUFETkPwl8CHsBt4vIDW7TscCv5qFtC4o3POMQXvbk1V2dI9PAfFQx9SToaLaQ1JFi8ViasUyKD7/oLzrWVuvqKVh95qGkmfnoY0l8gZvXsB64X1VPF5GvAeuAAnAD8AZVLUhwJ34cOA3YA/y1qt6YRBuGgSXjWZaMZ7s6R+hobmA+Sib6yDmaLSR1pPi7k9fy6qeu4eB9Jzo+R12ai2LJIo+GkGZFdn7ml0Vkf4JMqQA3qOrmNr7jrcDtBDWaAb4GnO2Wvw6cA3yGoBb0Wvd3nFt3XBvfs+AJzUd1jmYfKdT9A5pN1ziaTVMYCY5evazrc/iJjZ5uU8Eb/aGVNBcvJRjRvwR4KXC9iJzZyslFZBXwfOALfp2qfl8d7rw+9u0M4GK36TpgmYgc0NbVLHD8C7pQjA9JTURTqAlJTcJPYYwG6XRN5TU3M9oYLlqJPvqfwJO9diAiK4AfA5e1cOwFwLuAJbUbRCQLvIpAkwBYCdwX2WWTW/dgzXHnAucCHHRQ1P9tzFVkJwmnsPcplBL0UxijQbbGpzBTKFnk0RDSSo+lasxFD7dynIicDmxW1Q0Ndvl34Oeq+gt/SMw+WrdC9UJVXaeq61asWDFXMxYU2QbmI+8U7rbIjv+OYjT3kc1TMBzpVKomJLXcdvpto/+0oin8UET+G/iG+/wy4PstHHcC8AIROY2gtvNSEblEVc8WkX8GVgBviOy/iSDc1bMKq9vQFqFQqHE0Jxl95JOehY5mMx8ZjqAcZ8R8VCiZ+WgImbPHVPWdwOeAI4GjgAtV9d0tHHeeqq5S1TXAy4GrnEA4B3ge8ApVjQ5pLwdeLQHHAztU9cH6MxuNSKeElNSHpJYSzJIaOpotJNWoIRMzo9mEwvDRiqaAqn4b+HZC3/lZ4F7gWhcP/W1V/SCB9nEasJEgJPW1CX3fgiKTTtXVUyj0MCTVfAqGpz4ktcy+Fn00dLQkFLpFVa8GrnbLsd/popHeNB/tGWXG0qm6egolH5KaUOrs6iypJhSMgHRdSGrJqq4NIdZjI4avdxAlrNGciPko5dJcmKPZqCabrvUp2DyFYcSe6BEjm041FArmaDZ6SX1CvLLNaB5CmuU+upWYkFCPqh7ZkxYZXZGtsetCsllSwxrN5mg2avCV11QVEQnSXJijeeho5lM43f33dv6vuv9nETiCjQEkm6nXFPznZBLipao0BXM0Gx5/L5TKSiYtZj4aUprlProXQEROUNUTIpveIyK/BD7Y68YZ7ZONczSXlZRAKpE0F9WO5iTyKRmjgY9uK5aVlCizJQtJHUZa6bEJETnRfxCRpwKdp1I0ekomJfUhqSVNzCFc62g2RcHweP9SqazhPWg+heGjlZDU1wFfFpG9CHwMO9w6YwAZy6SqiqcDlMrlxAqd+FmrhbKSTUtXleKM0cKnUYlqkmY+Gj6aCgURSQGHqepRIrIUEFXdMT9NMzqhdgIRBOp8UvMJ/PmLpbKFoxpVeE2hWC6HwQ1mPho+mvaYS0PxZre80wTC4JONmdFcLGliDmEfbZQvlm3imlFFOuJozrv07eOWEG/oaEWMXyki/yAiq0VkH//X85YZHRE4muvnKSQVOuoFwfRsyZzMRhV+4FEoK/liCTBNYRhp1acA1SkoFDgk+eYY3eIT1kUJTD3JjOp9qozpQsnCUY0qvDmxVFJmCs7RbEJh6JhTKKjqwfPRECMZ4mY0l8qanKPZaQozheQEjTEa+HujUC6H5iOrpzB8tJQQT0SOAB5PUBcBAFW9uFeNMjqnUZqLpEw9XhDkiyWbzWxUEWoKZSVfMPPRsDKnUHAFcZ5JIBS+D5wKXAOYUBhAgqRktdFHCYakOkEwPVsyR7NRRTh5rVRxNJtQGD5a6bEzgZOBP6vqawkK7eR62iqjYzLpFIViD6OP3HmmC6VEUnEbo0MmVQlJrTiazXw0bLTyVE+70NSim6uwGXMyDyxBSGp9moukRvXeDDVdME3BqCaTrqS5qPgUbOAwbLTSY+tFZBnweWADcCNwQ6tfICJpEblJRK5wnw8WketF5E4R+aaIjLn1Ofd5o9u+pu2rMepy2kMQIphOaFQfOppnLfrIqCYTmdGcL9g8hWGllRrN/0NVt6vqZ4HnAK9xZqRWeStwe+TzR4HzVXUtsA14vVv/emCbqh4GnO/2M9okG2M+KpXLZBMzHwW3zEyxbI5mo4pMOs58ZPfIsDFnj4nIxSLyNyJyuKreo6q3tHpyEVkFPB/4gvsswEnAZW6Xi4AXuuUz3Gfc9pPFEuu0TTadCmsyewqlBENSUz4k1TQFo5pMzIxmEwrDRys99hXgAOCTIvJHEfmWiLy1xfNfALwL8EPX5cB2VS26z5uAlW55JXAfgNu+w+1fhYicKyLrRWT9li1bWmzGwiEbU44zSZ9COKO5YDOajWrio4/MfDRstGI+ugr4MPA+ghH/OuBv5zpORE4HNqvqhujquK9oYVu0PReq6jpVXbdixYq5mrHgyKZTqFJVQL1YTjZ1NoAq5mg2qvD3RrGszBRKiFi51mGklXkKPyGon3At8Avgyaq6uYVznwC8QEROI5j0tpRAc1gmIhmnDawCHnD7bwJWA5tEJAPsBTzS5vUseMJZpaUy6VQwSksyzUX0PJYl1YhS0RSCGc25TMpSqw8hrTzVtwCzwBHAkcARIrJoroNU9TxVXaWqa4CXA1ep6lnATwnmPgC8BvieW77cfcZtv0pVG9aINuIZc6O1aKbUZM1HlVvGfApGlEyk8lq+UDLT0ZDSivnobar6dOBFwMPAl4HtXXznu4G3i8hGAp/BF936LwLL3fq3A+/p4jsWLJmIXddTSLD2QZWmYKYBI4IfMHhHszmZh5NWzEdvBp4GPAm4F/gSgRmpZVT1auBqt3wXcGzMPjPAS9o5r1FP1j2IhRpNIemEeGD1mY1qwtTZ3nxkE9eGklYS4i0C/g3YEIkaMgYU/6KOCoUkK69lzXxkNCCTjoaklhg389FQ0or56F+BLPAqABFZISKWTntAyYaO5kj0UQ9yHwE2ec2oIl3lUzBNYVhpZfLaPxP4Ac5zq7LAJb1slNE5YVhgnaaQbEhqsGyaglGhkubCRx+ZpjCMtPKmeBHwAmA3gKo+ACzpZaOMzvEP5myVUEgwJDUiCJLyUxijQXVCvJI5moeUVnpt1oWGKoCITPS2SUY3jGXqzUelBNNcRM9jjmYjSjQkdaZg0UfDSiu9dqmIfI5g0tnfAD/G5TIyBo+oCu9JsvJatIaCOZqNKFWV14o2T2FYaaVG88dE5DnATuCxwD+p6pU9b5nREdmYyWvJVl4zR7MRT6Ym95E5moeTlmo0OyFwJYT1Ec5S1a/1tGVGR3jzUXTyWrGsiaXOrnI0m6ZgREilBBGXOtvMR0NLw14TkaUicp6IfEpEnisBbwbuAl46f0002sGr8H6eQqmsqJJckZ2IIEhb9JFRQzaVijiazXw0jDTTFL5KUATnWuAc4J3AGHCGqt48D20zOqAyeS3QFHwVtqQmr1U5mi0hnlFDOiVhSOq4mY+GkmZC4RBV/QsAEfkCsBU4SFWn5qVlRkdkI1lSoZJCOymnsIiQSUmis6SN0SGTlrBGs2kKw0kzUV7wC6paAu42gTD41Ka58BpDknMKvDAwR7NRSyYl5ItlSmU1n8KQ0kxTOEpEdrplARa5zwKoqi7teeuMthlzD+JssTeaAgRmoxmSq/tsjA7pVIrd+SBFmkUfDScNhYKqmu43hEzkgi7d5R7Mik8huQfUNAWjEdm0VISCmY+GEnuqR4zJWqFQSl5T8MLAJq8ZtaRTEt57Zj4aTnrWayIyLiI3iMhvROS3IvIBt/5kEblRRG4WkWtE5DC3Pici3xSRjSJyvYis6VXbRpl0Slg8lmbXTPBghuajJDWFlNcUTCgY1WTTKfbMlgAzHw0rvey1PHCSqh4FHA2cIiLHA58BzlLVo4GvA//o9n89sE1VDwPOBz7aw7aNNJO5TDha8w7nZDUFJxQsJNWoIZ0y89Gw07OnWgN2uY9Z9+cT63kn9V7AA275DOAit3wZcLJY1e+OmByvCAWvKSQZfeTnJ1jqbKOWTErYnXeagpmPhpKW0lx0ioikgQ3AYcCnVfV6ETkH+L6ITBPkUzre7b4SuA9AVYsisoOghvPWXrZxFIlqCkUnFJJ8gZuj2WhEJi3sng3uvfGsaQrDSE+falUtOTPRKuBYETkCeBtwmqquAr5MUOoTglDXulPUrhCRc0VkvYis37JlS6+aPtRM5jKhT6EYzlNI0qfgNAVzNBs1VIWkmqYwlMxLr6nqduBq4FTgKFW93m36JvBUt7wJWA0gIhkC09IjMee6UFXXqeq6FStW9LrpQ0m1ppBsmovouazIjlFLNiU45dR8CkNKL6OPVojIMre8CHg2cDuwl4g8xu32HLcO4HLgNW75TOAqV9zHaJPJ8QxTM9Xmo0QdzSkzHxnxRAcKFn00nPTSp3AAcJHzK6SAS1X1Cleo51siUiZIuPc6t/8Xga+KyEYCDeHlPWzbSLMkqin0JM2FOZqNeKIaqZmPhpOeCQVVvQU4Jmb9d4DvxKyfAV7Sq/YsJHz0kaqG0UdJls7MWkiq0YDoPWHmo+HEnuoRZDKXpeTq5BacTyFRTcFCUo0GRM2UpikMJ9ZrI8jkeKAATuULlJz5KMnaB/7BN0ezUYv5FIYf67URZInPfzRTDKOPepE6O0mTlDEaRO8JMx8NJ/ZUjyDRpHhh9FGiIampxM9pjAZ+8JFNi2mSQ4oJhRHEm492zRR7VE/BHM1GPH6gYFrC8GJP9QjiNYWpfDGsvJbkC9xCUo1G+MGHOZmHF+u5EWRJlaaQ/IzmrOU+Mhrg06mYUBheepoQz+gPXlPYPVsM7bpJmo96cU5jNPADhpwlwxtaTJyPIGFIqOEbFQAADL5JREFU6kyxNzOaU1Z5zYgnbeajocd6bgTJZdJk01ITfZT8jGaLLjFq8SGpJhSGF+u5EcWnzy72oPLa3hNj7L04i9VAMmqpaApmPhpWzKcwovj8R8UeVF577VMP5i+PPDCx8xmjQxh9ZLOZhxbruRFlMpdlKjJPIcnZx4vG0qzeZ3Fi5zNGh0wYfWSawrBiQmFECdJnF0LzkZn/jfkgnLxmmsLQYj03okTNR9m0mP3fmBds8trwYz03ooSO5rJalJAxb5ijefjpZTnOcRG5QUR+IyK/FZEPuPUiIh8WkTtE5HYReUtk/SdEZKOI3CIiT+xV2xYCoaZQUstRZMwbpikMP72MPsoDJ6nqLhHJAteIyA+AxwGrgcNVtSwi+7n9TwXWur/jgM+4/0YHLMllnKO5bNlMjXnDz4cxn8Lw0stynArsch+z7k+BvwVeqaplt99mt88ZwMXuuOtEZJmIHKCqD/aqjaPMZC5DvlhmulCymcfGvJEx89HQ01NxLiJpEbkZ2AxcqarXA4cCLxOR9SLyAxFZ63ZfCdwXOXyTW1d7znPdseu3bNnSy+YPNT7VxfY9BTMfGfNGxmY0Dz097TlVLanq0cAq4FgROQLIATOqug74PPAlt3vccFZjznmhqq5T1XUrVqzoVdOHHp8Ub/t0wRzNxrzhNYVxS4g3tMyLOFfV7cDVwCkEGsC33KbvAEe65U0EvgbPKuCB+WjfKOLTZ+/YUzCfgjFvWEK84aeX0UcrRGSZW14EPBv4PfBd4CS32zOAO9zy5cCrXRTS8cAO8yd0zmQuC8D26VnzKRjzRpg624TC0NLL6KMDgItEJE0gfC5V1StE5BrgayLyNgJH9Dlu/+8DpwEbgT3Aa3vYtpHH+xS27SmwbNFYn1tjLBTCIjtmPhpaehl9dAtwTMz67cDzY9Yr8KZetWehMZkLHsrZooWkGvOHzVMYfqznRhRvPgIrhmPMHxkzHw091nMjijcfgRXDMeaPfSYCU+WKJbk+t8ToFKunMKIszqYRAdVkq64ZRjOecOBe/OJdz7LU6kOMvS1GlFRKmBwLZL6Zj4z5xATCcGNCYYTxJiTTFAzDaBV7W4wwflazaQqGYbSKCYURxmsK5mg2DKNVTCiMMF5TyNo8BcMwWsSEwgizJNQUrJsNw2gNe1uMMOZTMAyjXUwojDB+VrMJBcMwWsWEwghTCUk1oWAYRmuYUBhhloTmI+tmwzBaw94WI4yFpBqG0S4mFEaYCXM0G4bRJiYURpjQfGRpLgzDaJFeluMcF5EbROQ3IvJbEflAzfZPisiuyOeciHxTRDaKyPUisqZXbVsohI5m0xQMw2iRXg4h88BJqnoUcDRwiqu9jIisA5bV7P96YJuqHgacD3y0h21bEITzFCz6yDCMFumZUNAArwlk3Z+6ms3/Cryr5pAzgIvc8mXAySJib7MusMlrhmG0S0+NzSKSFpGbgc3Alap6PfBm4HJVfbBm95XAfQCqWgR2AMtjznmuiKwXkfVbtmzpZfOHHktzYRhGu/T0baGqJVU9GlgFHCsiTwdeAnwyZve44azGnPNCVV2nqutWrFiRbINHjL0WZXnn8x7LqUc8qt9NMQxjSJiXcpyqul1ErgaeBRwGbHSWocUistH5ETYBq4FNIpIB9gIemY/2jSoiwpuedVi/m2EYxhDRy+ijFSKyzC0vAp4NbFDVR6nqGlVdA+xxAgHgcuA1bvlM4CpVrdMUDMMwjN7RS03hAOAi51hOAZeq6hVN9v8i8FUR2UigIby8h20zDMMwYuiZUFDVW4Bj5thnMrI8Q+BvMAzDMPqEhaUYhmEYISYUDMMwjBATCoZhGEaICQXDMAwjxISCYRiGESLDPBVARLYA93Z4+L7A1gSbMywsxOteiNcMC/O6F+I1Q/vX/WhVjU0JMdRCoRtEZL2qrut3O+abhXjdC/GaYWFe90K8Zkj2us18ZBiGYYSYUDAMwzBCFrJQuLDfDegTC/G6F+I1w8K87oV4zZDgdS9Yn4JhGIZRz0LWFAzDMIwaTCgYhmEYIQtSKIjIKSLyBxHZKCLv6Xd7eoGIrBaRn4rI7SLyWxF5q1u/j4hcKSJ3uv9797utSePKwN4kIle4zweLyPXumr8pImP9bmPSiMgyEblMRH7v+vwpC6Sv3+bu79tE5BsiMj5q/S0iXxKRzSJyW2RdbN9KwCfcu+0WEXliu9+34ISCq+/waeBU4PHAK0Tk8f1tVU8oAu9Q1ccBxwNvctf5HuAnqroW+In7PGq8Fbg98vmjwPnumrcBr+9Lq3rLx4EfqurhwFEE1z/SfS0iK4G3AOtU9QggTVCHZdT6+yvAKTXrGvXtqcBa93cu8Jl2v2zBCQXgWGCjqt6lqrPAfwBn9LlNiaOqD6rqjW55iuAlsZLgWi9yu10EvLA/LewNIrIKeD7wBfdZgJOAy9wuo3jNS4GnExSqQlVnVXU7I97XjgywyJXwXQw8yIj1t6r+nPrSxI369gzgYg24DlgmIge0830LUSisBO6LfN7k1o0sIrKGoODR9cD+qvogBIID2K9/LesJFwDvAsru83Jgu6oW3edR7O9DgC3Al53Z7AsiMsGI97Wq3g98DPgTgTDYAWxg9PsbGvdt1++3hSgUJGbdyMblisgk8C3g71V1Z7/b00tE5HRgs6puiK6O2XXU+jsDPBH4jKoeA+xmxExFcTg7+hnAwcCBwASB+aSWUevvZnR9vy9EobAJWB35vAp4oE9t6SkikiUQCF9T1W+71Q95ddL939yv9vWAE4AXiMg9BGbBkwg0h2XOvACj2d+bgE2qer37fBmBkBjlvgZ4NnC3qm5R1QLwbeCpjH5/Q+O+7fr9thCFwq+BtS5CYYzAMXV5n9uUOM6W/kXgdlX9t8imy4HXuOXXAN+b77b1ClU9T1VXqeoagn69SlXPAn4KnOl2G6lrBlDVPwP3ichj3aqTgd8xwn3t+BNwvIgsdve7v+6R7m9Ho769HHi1i0I6HtjhzUytsiBnNIvIaQQjyDTwJVX9cJ+blDgiciLwC+BWKvb19xL4FS4FDiJ4qF6iqrVOrKFHRJ4J/IOqni4ihxBoDvsANwFnq2q+n+1LGhE5msC5PgbcBbyWYNA30n0tIh8AXkYQbXcTcA6BDX1k+ltEvgE8kyA99kPAPwPfJaZvnXD8FEG00h7gtaq6vq3vW4hCwTAMw4hnIZqPDMMwjAaYUDAMwzBCTCgYhmEYISYUDMMwjBATCoZhGEaICQVjaBGRkojcHPlrOotXRN4oIq9O4HvvEZF9uz1PAu14v4j8Q7/bYYwWmbl3MYyBZVpVj251Z1X9bC8bM0y4eHZR1fKcOxsLCtMUjJHDjeQ/KiI3uL/D3PpwZC0ibxGR37mc8//h1u0jIt91664TkSPd+uUi8iOXbO5zRPLLiMjZ7jtuFpHPudTsce35gIjcKCK3isjhte1xn28TkTXu7/cusd1tIvI1EXm2iPzS5c8/NnL6o0TkKrf+byLneqeI/NpdywfcujUS1Fr4d+BGqtMhGAZgQsEYbhbVmI9eFtm2U1WPJZjdeUHMse8BjlHVI4E3unUfAG5y694LXOzW/zNwjUs2dznBLFJE5HEEs2lPcBpLCTirQVu3quoTCfLbt2LyOYygRsKRwOHAK4ET3bHvjex3JEGq8KcA/yQiB4rIcwny6R8LHA08SUSe7vZ/LEFq5WNU9d4W2mEsMMx8ZAwzzcxH34j8Pz9m+y3A10TkuwQpAyB46f4VgKpe5TSEvQhqFbzYrf8vEdnm9j8ZeBLw68AawyIaJ53zCQk3+HPNwd2qeiuAiPyWoKCKisitwJrIft9T1WlgWkR+SiAITgSeS5DiAWCSQEj8CbjX5dk3jFhMKBijijZY9jyf4GX/AuB9IvIEmqcdjjuHABep6nkttMfn3ilRee6KVGvr4zH7Q5C7Kh9Zjj63te1S166PqOrnqhob1NXY3UJbjQWMmY+MUeVlkf/XRjeISApYrao/JSjIs4xgNP1znPnHJdTb6mpQRNefCvhaxz8BzhSR/dy2fUTk0W208R6CFNdIUEv34LauMOAMCeoSLydImvZr4L+B10lQSwMRWenbaBhzYZqCMcwsEpGbI59/qKo+LDUnItcTDHxeUXNcGrjEmYaEoJ7vdhF5P0H1slsIMkz61MQfAL4hIjcCPyMww6CqvxORfwR+5ARNAXgT0Kqt/lsEaY5vJniZ39HqhUe4AfgvAj/H/1LVB4AHnL/jWmfW2gWcTaClGEZTLEuqMXJIUGRnnapu7XdbDGPYMPORYRiGEWKagmEYhhFimoJhGIYRYkLBMAzDCDGhYBiGYYSYUDAMwzBCTCgYhmEYIf8fYykqIPy2Gu0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Weights are optimal\n"
     ]
    }
   ],
   "source": [
    "# Main Training Loop\n",
    "\n",
    "numberEpisodes = 500\n",
    "\n",
    "discount = 0.99\n",
    "counter = 0\n",
    "epsilon = 1\n",
    "epsilonDecay = 0.995\n",
    "rewards = []\n",
    "\n",
    "memorySize = 2000\n",
    "memoryPreviousStates = np.zeros((memorySize,4))\n",
    "memoryStates = np.zeros((memorySize,4))\n",
    "memoryARD = np.zeros((memorySize,3))\n",
    "numberOfLearningFromReplaySteps = 32\n",
    "itemsInMemory = 0\n",
    "initialiseReplayAfter = 300 \n",
    "optimalWeights = False\n",
    "for episode in range (numberEpisodes):\n",
    "    if (optimalWeights): # Stop the training if we already have \"perfect\" weights\n",
    "        break\n",
    "        \n",
    "    epsilon = max (epsilon*epsilonDecay, 0.1)\n",
    "  \n",
    "    done = False\n",
    "    prevObs = env.reset()\n",
    "    \n",
    "    episodeReward = 0\n",
    "    for step in range (600):\n",
    "        if done:\n",
    "            break\n",
    "        #env.render()\n",
    "        \n",
    "       \n",
    "      \n",
    "        target = model.predict(np.expand_dims(prevObs, axis = 0))[0]\n",
    "        if (random.random()<epsilon):\n",
    "            action = random.choice([0,1])\n",
    "        else:\n",
    "            action = np.argmax(target)\n",
    "        \n",
    "        obs, reward, done, _ = env.step (action)\n",
    "\n",
    "       \n",
    "\n",
    "        memoryPreviousStates[counter] = prevObs\n",
    "        memoryStates[counter] = obs\n",
    "        if (episodeReward <= 498):\n",
    "            memoryARD[counter] = [float(action), reward, float (done)]\n",
    "        else: # Is not the fault of the agent's action that the environment restarts after 500 epochs. So we set done=False for the memory replay\n",
    "            memoryARD[counter] = [float(action), reward, float (0)]\n",
    "\n",
    "\n",
    "        if (itemsInMemory < (counter+1)):\n",
    "            itemsInMemory = min ((itemsInMemory+1), memorySize)\n",
    "        if (itemsInMemory >= initialiseReplayAfter): # After a few episodes, Memory Replay is activated\n",
    "            executeMemoryReplayStep(memoryPreviousStates, memoryStates, memoryARD, model, itemsInMemory)\n",
    "      \n",
    "\n",
    "        prevObs = obs\n",
    "        episodeReward += reward\n",
    "        counter = (counter + 1) % memorySize\n",
    "    print (episode, \" Episode Reward: \", episodeReward, \".Epsilon:\", epsilon)\n",
    "    rewards.append(episodeReward)\n",
    "    if (episode % 10 ==0):\n",
    "        if (checkIfTheseWeightsAreOptimal(model)):\n",
    "            model.save_weights(\"BestWeights2.h5\")\n",
    "            print (\"Weights are optimal\")\n",
    "            optimalWeights = True\n",
    "        else:\n",
    "            print (\"Weights are not optimal\")\n",
    "\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qg__BmZpNRtA"
   },
   "source": [
    "### Even though the scores obtained before getting the optimal weights are rather low (around 150.0) compared to the maximum score (500.0), it should be considered that, at that point, epsilon was above 0.6. That means than more than 60% of the times the action was chosen randomly and the agent managed to score almost 150.0 nevertheless. If we don't check for optimal weights and keep the training loop going, we obtain a graph that is more stable and needs fewer episodes than the single network without memory replay.\n",
    "### This is the first algorithm with which we have obtained weights that can solve the environment (when stopping the training, the algorithm gets an average over 475.0 for 100 consecutive episodes)\n",
    "### Further research with a Double Neural Network improved the stability of the algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "openAIGym.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}