Tennis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Collaboration and Competition\n",
    "\n",
    "---\n",
    "\n",
    "You are welcome to use this coding environment to train your agent for the project.  Follow the instructions below to get started!\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "Run the next code cell to install a few packages.  This line will take a few minutes to run!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31mtensorflow 1.7.1 has requirement numpy>=1.13.3, but you'll have numpy 1.12.1 which is incompatible.\u001b[0m\r\n",
      "\u001b[31mipython 6.5.0 has requirement prompt-toolkit<2.0.0,>=1.0.15, but you'll have prompt-toolkit 3.0.5 which is incompatible.\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!pip -q install ./python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The environment is already saved in the Workspace and can be accessed at the file path provided below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: TennisBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 8\n",
      "        Number of stacked Vector Observation: 3\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 2\n",
      "        Vector Action descriptions: , \n"
     ]
    }
   ],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np\n",
    "\n",
    "env = UnityEnvironment(file_name=\"/data/Tennis_Linux_NoVis/Tennis\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 2\n",
      "Size of each action: 2\n",
      "There are 2 agents. Each observes a state with length: 24\n",
      "The state for the first agent looks like: [ 0.          0.          0.          0.          0.          0.          0.\n",
      "  0.          0.          0.          0.          0.          0.          0.\n",
      "  0.          0.         -6.65278625 -1.5        -0.          0.\n",
      "  6.83172083  6.         -0.          0.        ]\n",
      "The state for the second agent looks like: [ 0.          0.          0.          0.          0.          0.          0.\n",
      "  0.          0.          0.          0.          0.          0.          0.\n",
      "  0.          0.         -6.4669857  -1.5         0.          0.\n",
      " -6.83172083  6.          0.          0.        ]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents \n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])\n",
    "print('The state for the second agent looks like:', states[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.\n",
    "\n",
    "Note that **in this coding environment, you will not be able to watch the agents while they are training**, and you should set `train_mode=True` to restart the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total score (averaged over agents) this episode: 0.09500000160187483\n",
      "Total score (averaged over agents) this episode: -0.004999999888241291\n",
      "Total score (averaged over agents) this episode: -0.004999999888241291\n",
      "Total score (averaged over agents) this episode: 0.04500000085681677\n",
      "Total score (averaged over agents) this episode: 0.04500000085681677\n"
     ]
    }
   ],
   "source": [
    "for i in range(5):                                         # play game for 5 episodes\n",
    "    env_info = env.reset(train_mode=False)[brain_name]     # reset the environment    \n",
    "    states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "    scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "    while True:\n",
    "        actions = np.random.randn(num_agents, action_size) # select an action (for each agent)\n",
    "        actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1\n",
    "        env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "        next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "        rewards = env_info.rewards                         # get reward (for each agent)\n",
    "        dones = env_info.local_done                        # see if episode finished\n",
    "        scores += env_info.rewards                         # update the score (for each agent)\n",
    "        states = next_states                               # roll over states to next time step\n",
    "        if np.any(dones):                                  # exit loop if episode finished\n",
    "            break\n",
    "    print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ddpg(episodes=1000, max_t=1000, window=100):\n",
    "    agent = Agent(state_size=state_size, action_size=action_size, random_seed=1)\n",
    "    \n",
    "    scores_window = deque(maxlen = window)\n",
    "    scores_all = []\n",
    "    actor_losses = []\n",
    "    critic_losses = []\n",
    "    \n",
    "    for i in range(1, episodes+1):\n",
    "        env_info = env.reset(train_mode=True)[brain_name]\n",
    "        states = env_info.vector_observations\n",
    "        scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "        agent.reset()\n",
    "        for t in range(max_t):\n",
    "            actions = agent.act(states)\n",
    "            actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1\n",
    "            env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "            next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "            rewards = env_info.rewards                         # get reward (for each agent)\n",
    "            dones = env_info.local_done                        # see if episode finished\n",
    "            scores += env_info.rewards                         # update the score (for each agent)\n",
    "            for state,action,reward,next_state,done in zip(states,actions,rewards,next_states,dones):\n",
    "                agent.step(state,action,reward,next_state,done,t)\n",
    "            states = next_states                               # roll over states to next time step\n",
    "            if np.any(dones):                                  # exit loop if episode finished\n",
    "                break\n",
    "        scores_window.append(np.max(scores))\n",
    "        scores_all.append(np.max(scores))\n",
    "        print('\\rEpisode: {} \\tEpisode max: {:.2f} \\tScore Average: {:.2f}'.format(i+1, np.max(scores), np.mean(scores_window)),end=\"\")\n",
    "        if i%10==0:                                        # print the mean score over the last 100 episodes\n",
    "            print('\\rEpisode: {} \\tEpisode max: {:.2f} \\tAverage score: {:.2f}'.format(i+1, np.max(scores), np.mean(scores_window)))\n",
    "        if np.mean(scores_window) >= 0.8:\n",
    "            print('\\rSolved! Episode: {}\\tAverage score:{:.2f}'.format(i+1,np.mean(scores_window)))\n",
    "            torch.save(agent.actor_local.state_dict(), \"09May_actor.pt\")\n",
    "            torch.save(agent.critic_local.state_dict(), \"09May_critic.pt\")\n",
    "            break\n",
    "    return scores_all, scores_window"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 11 \tEpisode max: 0.00 \tAverage score: 0.00\n",
      "Episode: 21 \tEpisode max: 0.00 \tAverage score: 0.00\n",
      "Episode: 31 \tEpisode max: 0.00 \tAverage score: 0.00\n",
      "Episode: 41 \tEpisode max: 0.00 \tAverage score: 0.00\n",
      "Episode: 51 \tEpisode max: 0.09 \tAverage score: 0.00\n",
      "Episode: 61 \tEpisode max: 0.00 \tAverage score: 0.01\n",
      "Episode: 71 \tEpisode max: 0.00 \tAverage score: 0.01\n",
      "Episode: 81 \tEpisode max: 0.09 \tAverage score: 0.01\n",
      "Episode: 91 \tEpisode max: 0.09 \tAverage score: 0.02\n",
      "Episode: 101 \tEpisode max: 0.10 \tAverage score: 0.03\n",
      "Episode: 111 \tEpisode max: 0.10 \tAverage score: 0.04\n",
      "Episode: 121 \tEpisode max: 0.10 \tAverage score: 0.05\n",
      "Episode: 131 \tEpisode max: 0.10 \tAverage score: 0.06\n",
      "Episode: 141 \tEpisode max: 0.10 \tAverage score: 0.07\n",
      "Episode: 151 \tEpisode max: 0.09 \tAverage score: 0.08\n",
      "Episode: 161 \tEpisode max: 0.10 \tAverage score: 0.08\n",
      "Episode: 171 \tEpisode max: 0.20 \tAverage score: 0.10\n",
      "Episode: 181 \tEpisode max: 0.20 \tAverage score: 0.12\n",
      "Episode: 191 \tEpisode max: 0.20 \tAverage score: 0.13\n",
      "Episode: 201 \tEpisode max: 0.00 \tAverage score: 0.12\n",
      "Episode: 211 \tEpisode max: 0.10 \tAverage score: 0.12\n",
      "Episode: 221 \tEpisode max: 1.10 \tAverage score: 0.21\n",
      "Episode: 231 \tEpisode max: 2.70 \tAverage score: 0.32\n",
      "Episode: 241 \tEpisode max: 2.60 \tAverage score: 0.44\n",
      "Episode: 251 \tEpisode max: 1.00 \tAverage score: 0.52\n",
      "Episode: 261 \tEpisode max: 2.60 \tAverage score: 0.63\n",
      "Episode: 271 \tEpisode max: 0.20 \tAverage score: 0.66\n",
      "Episode: 281 \tEpisode max: 0.50 \tAverage score: 0.67\n",
      "Episode: 291 \tEpisode max: 0.10 \tAverage score: 0.71\n",
      "Episode: 301 \tEpisode max: 0.70 \tAverage score: 0.77\n",
      "Solved! Episode: 304\tAverage score:0.80Average: 0.80\n"
     ]
    }
   ],
   "source": [
    "scores, scores_window = ddpg()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f8d43ce2940>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJztnXecVNXZ+L/PLktHhAUiAkqxxI4I2A0mxp6oUSOaWGKPGsureX+WxBKjSYzRREnsGitibOGNvWGLDRURwQgaVBApuxQpC1ue3x9nzs6du/dO2Z3ZmVme7+czn9vOPfe5M7vnuU87V1QVwzAMwwCoKLYAhmEYRulgSsEwDMNoxpSCYRiG0YwpBcMwDKMZUwqGYRhGM6YUDMMwjGZMKRhGOyEiT4nI8cWWwzDSIVanYKwPiMhc4GRVfb7YshhGKWOWgmHkARHpVGwZDCMfmFIw1mtE5GARmSYiy0Tk3yKyfeDYhSLyqYh8IyIzReSwwLETROR1EbleRGqByxP7XhORa0VkqYj8V0QOCJwzRURODpyfru0wEXklce3nReSvInJfO30txnqMKQVjvUVERgF3AqcB1cAtwGQR6ZJo8imwJ9AbuAK4T0QGBrrYGfgMGABcFdj3H6AfcA1wh4hIjAjp2j4AvJ2Q63Lg2Lbcq2FkiykFY33mFOAWVX1LVRtV9W5gLbALgKr+Q1W/UtUmVZ0EzAbGBs7/SlVvVNUGVV2T2Pe5qt6mqo3A3cBA4Fsx149sKyKbAGOAS1V1naq+BkzO870bRiSmFIz1mU2B8xOuo2UisgwYAmwMICLHBVxLy4BtcU/1ni8j+vzar6jq6sRqz5jrx7XdGKgN7Iu7lmHkHQuOGeszXwJXqepV4QMisilwG/A94A1VbRSRaUDQFVSo1L0FQF8R6R5QDEMKdC3DSMEsBWN9okpEuvoPbtA/XUR2FkcPETlIRHoBPXCD/mIAEfkZzlIoOKr6OTAVF7zuLCK7Aj9oj2sbhikFY33iSWBN4HMoLq4wAVgKzAFOAFDVmcCfgDeAhcB2wOvtKOtPgF2BGuC3wCRcvMMwCooVrxlGGSAik4CPVfWyYstidGzMUjCMEkRExojICBGpEJH9gUOAx4stl9HxsUCzYZQmGwGP4uoU5gE/V9X3iyuSsT5g7iPDMAyjGXMfGYZhGM2UnfuoX79+OnTo0GKLYRiGUVa8++67S1S1f6Z2ZacUhg4dytSpU4sthmEYRlkhIp9n087cR4ZhGEYzphQMwzCMZkwpGIZhGM2UXUwhivr6eubNm0ddXV2xRSkbunbtyuDBg6mqqiq2KIZhlBAdQinMmzePXr16MXToUOLfZ2J4VJWamhrmzZvHsGHDii2OYRglRIdwH9XV1VFdXW0KIUtEhOrqarOsDMNoQYdQCoAphByx78swjCg6jFIwDKM8+cc/YPHiYktheEwp5InKykpGjhzJNttsww477MB1111HU1MTAFOmTKF3797suOOObLnlluy1117861//aj738ssvZ9CgQYwcOZJtt92WyZOTr+O977772H777Zv7Pfnkk1m2bFm7359hFILZs+HHP4aTTy62JIanQwSaS4Fu3boxbdo0ABYtWsQxxxzD8uXLueKKKwDYc889mxXBtGnTOPTQQ+nWrRvf+973ADjvvPO44IILmDVrFnvuuSeLFi3i2Wef5frrr+epp55i0KBBNDY2cvfdd7Nw4UI23HDD4tyoYeSRTz5xSwtvlQ5mKRSAAQMGcOuttzJhwgSiZqEdOXIkl156KRMmTGhxbKuttqJTp04sWbKEq666imuvvZZBgwYBzho58cQT2XLLLQt+D4bRHvz3v245fHhx5TCSdDhL4dxzIfHAnjdGjoQ//zm3c4YPH05TUxOLFi2KPD5q1Cj++Mc/ttj/1ltvUVFRQf/+/fnoo48YNWpUa0Q2jLLgs8/ccsiQ4sphJDFLoYCke1dF+Nj111/PyJEjueCCC5g0aVKL7KAPP/yQkSNHMmLECCZNmlQQeQ2jvfFKoVOHezwtXzrcT5HrE32h+Oyzz6isrGTAgAHMmjWrxfH333+frbbaqnnbxxSCbLPNNrz33nvsvffebLfddkybNo2zzjqLNWvWFFx+w2gPvPsokZNhlABmKRSAxYsXc/rpp3PWWWdF1gNMnz6dK6+8kjPPPDNtPxdddBEXXHAB8+bNa95nCsEoVZYtc4O7X8YdD+IthaDhvGwZNDZGn5/LfoDly1t3rJRYsQIaGtrveqYU8sSaNWuaU1L32Wcf9t13Xy677LLm46+++mpzSuqZZ57JDTfc0Jx5FMeBBx7I2WefzQEHHMDWW2/NbrvtRmVlJfvtt1+hb8cwcqK2Fvr0gfPPh002cbUHQdasgU03haDnc9UqWLnSrXulsHYtDB0KEyemnr96dbL/IPX1Lkh9330tZWpocMfuvrvlscZGGDEC7rorp9ssCtttBzfe2H7X63Duo2LRmOaRY9y4cSxfvjz2+OWXXx577Pjjj+f4449vi2iGUXCWLHHLu+6Cb76BBQtSj69Z4554v/46uW/duuS6tyDWrHFP8MF2fj/APfekuojXroWlS1u29/3X1kYfq6+HmproY6XGggXtK6dZCoZhtJmKxEhSX++WYTeR3w7uD657SyGqXRDff7p+Pf45LSrfI9N1SommpvaV05SCYRhtxofO/KAdNpzDg354PZNS8P2FlYI/L9eB3+9LkyBYMqi2r5ymFAzDaDOZLIW2KgUfaI1TCh3ZUlDtIJaCiAwRkZdEZJaIfCQi50S0GSciy0VkWuJzaaHkMQyjcFRWpm7n6j4KHw8P5H6Az6bf8LF0SsEshZYUMtDcAJyvqu+JSC/gXRF5TlVnhtq9qqoHF1AOwzAKTDjzurXuo7jBPy4lM537KK6vTMdKiXT3VygKZimo6gJVfS+x/g0wCxhUqOsZhlE8woNWNk/0wXOyjSmE6eiWQjHcXO0SUxCRocCOwFsRh3cVkQ9E5CkR2Sbm/FNFZKqITF1cwhOvP/bYY4gIH3/8cbFFMYx2JZNSyGQphAe/QlsK5RJT6FCWgkdEegKPAOeq6orQ4feATVV1B+BG4PGoPlT1VlUdraqj+/fvX1iB28DEiRPZY489ePDBB9vcV7q6B8MoNTI92ecaaI6LKYRJF2juCJZCuvsrFAVVCiJShVMI96vqo+HjqrpCVVcm1p8EqkSkXyFlKhQrV67k9ddf54477mhWCkcddRRPPvlkc5sTTjiBRx55hMbGRn75y18yZswYtt9+e2655RbAvYxn77335phjjmG77bYD4NBDD2WnnXZim2224dZbb23u64477mCLLbZg3LhxnHLKKZx11lmAm2Lj8MMPZ8yYMYwZM4bXX3+9vb4CYz2mNe6j1mQfhUk3uHeE7KNiKK+CBZrFTfpzBzBLVa+LabMRsFBVVUTG4pRUTZsuXKS5sx9//HH2339/tthiC/r27ct7773H+PHjmTRpEgceeCDr1q3jhRde4KabbuKOO+6gd+/evPPOO6xdu5bdd9+dfffdF4C3336bGTNmMGzYMADuvPNO+vbty5o1axgzZgyHH344a9eu5corr+S9996jV69efPe732WHHXYA4JxzzuG8885jjz324IsvvmC//faLnJDPMPJJnBLwZOs+inP5tMVSyDVdtZQohqVQyOyj3YFjgQ9FxI/SFwObAKjqzcARwM9FpAFYA4zXdPNNlzATJ07k3HPPBWD8+PFMnDiRK6+8krPPPpu1a9fy9NNPs9dee9GtWzeeffZZpk+fzsMPPwzA8uXLmT17Np07d2bs2LHNCgHghhtu4LHHHgPgyy+/ZPbs2Xz99dd85zvfoW/fvgAceeSRfJJ4hdXzzz/PzJnJBK8VK1bwzTff0KtXr3b5Hoz1k0zunnzVKcRdt6PWKXQoS0FVXwNaThGa2mYC0PL1Y22hCHNn19TU8OKLLzJjxgxEhMbGRkSEa665hnHjxvHMM88wadIkjj76aMC9S+HGG29sMbHdlClT6NGjR8r2888/zxtvvEH37t0ZN24cdXV1ad/T0NTUxBtvvEG3bt0Kc7OGEUEmSyHX7KNsYwqtjRtYTCEeq2jOAw8//DDHHXccn3/+OXPnzuXLL79k2LBhvPbaa4wfP5677rqLV199tVkJ7Lffftx0003UJ8ozP/nkE1atWtWi3+XLl9OnTx+6d+/Oxx9/zJtvvgnA2LFjefnll1m6dCkNDQ088sgjzefsu+++Ka/5nJZvV5phRNDW7KNCWgqWfZQbphTywMSJEznssMNS9h1++OE88MAD7Lvvvrzyyivss88+dO7cGYCTTz6ZrbfemlGjRrHtttty2mmn0RDxV7///vvT0NDA9ttvz69//Wt22WUXAAYNGsTFF1/MzjvvzD777MPWW29N7969Aedumjp1Kttvvz1bb701N998c4Hv3jBaZynkkpK6vloKxVBeNnV2HpgyZUqLfWeffXbzek1Nauy8oqKCq6++mquvvjpl/7hx4xg3blzzdpcuXXjqqacir3nMMcdw6qmn0tDQwGGHHdYcqO7Xr5+9rtNod7KNKQTbpatoDvcXfGZSTVZQd/SYglkKRtZcfvnljBw5km233ZZhw4Zx6KGHFlskYz2m0O6joJKpq0vfb7j/cp7mokMFmo3Ccu211xZbBMNoptDuo6ClUFcHPo+irXUKpe4+skBzGyjTTNaiYd+XkU9ak5La2rmPgq8pb2tFc6lbCuY+aiVdu3alpqbGBrosUVVqamro2rVrsUUxOghttRQypaQGLYWgUsjGUshVYZQSFmhuJYMHD2bevHmU8mR5pUbXrl0ZPHhwscUwOgj5mhAvm4rmUrcUFi+G3r0hkWzYJophKXQIpVBVVZVSBWwYRvuS7wnx0sUUslUK6WIKhZzmYsAA+PGPIR9JgMWwFDqE+8gwjOJS6Anx4rKPWmsNFGqw9XI89FB++7OYgmEYZUVbJ8RrbUyhrXUK+R5sC6VkzFIwDKOsyJR9lGnuo1wqmrMNNBcjplCo/sxSMAyjrMiXpZDNO5pztRTaM/uoUP2ZpWAYRllR7IrmXC2FQlU0m6VgGIZB69xH6Sqac61T6OgxBVMKhmGUFfkONOezTqEY2Uf57s/cR4ZhlBXt+Y7mxGtIUtqVytxH5j4yDMOg7cVruWQfBddLraLZUlINwzBoXaA5akK8bN6nEKUUSmXuo0JZHmYpGIZRVuR7QrxsLYVs4gblnH1UDEuhQ8x9ZBhGccnXm9fSxRT829aydR8Vw1Kw7CPDMAzabilkE1Po1AkqKqLPK5WK5kIpGYspGIZRVuS7eC0qplBZ6T65Wgrlln10330wZ45bN0vBMIyypK3ZR5mmufCWQpxSKJVZUvPR30knwV13pfZnMQXDMMqK1tQp5DIhnrcUIPtAczHep5CP/urrk7UY9pIdwzDKkrZUNFdUZJd91KlTcj1dv+H+y6lOQdV9fAquKQXDMMqStsx91KlT9jEF1WhLoVTqFPKhFKCl7B0i0CwiQ0TkJRGZJSIficg5EW1ERG4QkTkiMl1ERhVKHsMwCkdbAs2VldlnH+USaC7H7CN/bx3VUmgAzlfV90SkF/CuiDynqjMDbQ4ANk98dgZuSiwNwygj2uI+8hYApH+fglceuVY0l5P7yJ/fIS0FVV2gqu8l1r8BZgGDQs0OAe5Rx5vAhiIysFAyGYZRGNriPgoqhVwthdZmGJWq+8ifX0xLoV1SUkVkKLAj8Fbo0CDgy8D2PFoqDkTkVBGZKiJTFy9eXCgxDcNoJa2xFPy+KKVQ6DqFQk1zkS/3UVj2DmEpeESkJ/AIcK6qrggfjjilxdeqqreq6mhVHd2/f/9CiGkYRhtoy9TZbYkptLWiudQthULJmY6CKgURqcIphPtV9dGIJvOAIYHtwcBXhZTJMIz805bitWzcR22xFMqpeC0cU+hQloKICHAHMEtVr4tpNhk4LpGFtAuwXFUXFEomwzAKQ1uyj4IpqZkqmsNzH7X2Hc2FegLPd/ZRMSyFQmYf7Q4cC3woItMS+y4GNgFQ1ZuBJ4EDgTnAauBnBZTHMIwCkSmmkKv7KNuYQmsrms1SiKdgSkFVXyM6ZhBso8CZhZLBMIz2Idups/OdfdTauY8KNc2FxRQMwzBonaWQLvso25hCqVkKHSH7yKa5MAyjzWQbU4iaBK89K5qbmmDffQuX/98R6hRMKRiG0Wby5T5K947mdNlH2dYirFsHL7wAPXu2PJYPClXRbO4jwzDKirYGmgtR0RylYHy7detaHssHhZr7qEOkpBqGsf6Qr5TU1tYpZJt2GlYKZim0xJSCYRhtJlPxWqErmrMtXotTVvki3bVyOd8sBcMwypq2WApRL9nJx9xH6SyFuO22ErxWWDFmQ1z2UclZCiLSTUS2LLQwhmGUJ5kG23QT4uVS0dyaQHM6pVBIS8E/7bfm/JKeOltEfgBMA55ObI8UkcmFFswwjPKhrVNnZ3IfeUshPM1FukEz3TXjtttKvpRCqU+dfTkwFlgGoKrTgKGFE8kwjHKjNZZCoSuaoyyFcLtCZh+1RimE3UclaSkADaq6vOCSGIZRtrRl6uxc3tHc1pfsmKWQmWyK12aIyDFApYhsDpwN/LuwYhmGUU5kyj4qxjuaO0JMoVSzj34BbAOsBR4AlgPnFlIowzDKi7ZkH4m0f51C3HZbyVf2UclOnS0ilcAVqvpL4JL2EckwjHIjylev6gZ8iB7sVV3gOKgUcs0+KuU6hXxaCiUTaFbVRmCndpLFMIwyJZ1PH+IthYqK/NQplIqlkO+YQjECzdnEFN5PpKD+A1jld8a8XtMwjPWQuEG5sjK5Hlz6dW8pFLKiuT1jCvnOPirVQHNfoAb4bmCfAqYUDMMA2mYpFDqm0BGyj0rKUlBVe0WmYRhpSVcnEDyerfuovbKPSlUplPSEeCIyWEQeE5FFIrJQRB4RkcHtIZxhGOVBJksh7qldJNV9lOv7FLKpUyhH91GpT4h3FzAZ2BgYBPxfYp9hGAaQ3n0TPB6uLs7WfdSWiuZydB+VtKUA9FfVu1S1IfH5O9C/wHIZhlFGpHPfBI+3xn2kmgxat2buo3IsXit1S2GJiPxURCoTn5/iAs+GYRhA6uDaubNb5hJoTpd95JVLIWZJLWTxWrlmH2WjFE4Efgx8DSwAjkjsMwzDAFIH19YohXR1Cn5wbe37FIpVvNaaiuagzN5CCvdbaDIqBVX9QlV/qKr9VXWAqh6qqp+3h3CGYZQHwcG1qsoto9xHDQ0wZgy88EK0UoiKA6SzFLKpU2hshN12gyeeKJ+YAji5S9JSEJG7RWTDwHYfEbmzsGIZhlFOZOs+Apg6FaZPT2YfZYopZGMppAt019XBG2/ABx+UT/aRP79UA83bq+oyv6GqS4EdCyeSYRjlRnCw9ZZCOrdNY2N+YgrZWAp+cPbXjJM7HxTKUigp9xFQISJ9/IaI9CW7SmjDMNYTMrmPomZRTZeSmmtMIZ2lEHRJFfolO/lUCg0NpTvNxZ+Af4vIw4ntI4GrCieSYRjlRraB5mD7bFNSw5aCH9yjlEmQqLeYlUv2kV8v1UDzPcDhwMLE50eqem+m80TkzkQV9IyY4+NEZLmITEt8Ls1VeMMwSoNMMYVs3UdRgeawpRA8nk2dQjr3USlbCiUXaBaR7iJS5QTSmcBzQBXw7Sz7/juwf4Y2r6rqyMTnN1n2axhGiREVU8jkPkqXkprOUgjuy6ZOITgNdTnFFIKB5lKxFJ4GhgKIyGbAG8Bw4EwR+X2mjlX1FaA2DzIahlHi5Gop+AE6LvtIFf7wB/jss2hLIZsX20dZH+WUfVRylgLQR1VnJ9aPByaq6i+AA4CD8nT9XUXkAxF5SkS2iWskIqeKyFQRmbp48eI8XdowjHzR1OQshMMPh732Su7zhAe1TNlHqnDhhfDoo9GWQjggnY2lUG7ZR8FAc6lYCsGv+bs49xGqug7Ih4jvAZuq6g7AjcDjsYKo3qqqo1V1dP/+Nu2SYZQaqk4pPPywK06DtmUfeerrUy2FiorUvrOpaA62KZeKZn9+qdUpTBeRa0XkPGAz4FmAYCFbW1DVFaq6MrH+JFAlIv3y0bdhGO2LdwVBcuDOxn0UV9HsaWhIH1NIN2iG+yq37KOgpRDuu5CkUwqnAEtwcYV9VXV1Yv/WwLVtvbCIbCTi/oxEZGxCFptozzDKEP/UDy1dPP54kKD7KDjzaSZLIS7QnM5SCF8zLHc+yXf2UbrvsFDE1imo6hqgRUBZVf8N/DtTxyIyERgH9BORecBluOwlVPVm3MR6PxeRBmANMF61PY0kwzDyRZSlEFV5HNxOl33kydZSSFenEL5meF8+KVRMAUpAKbQVVT06w/EJwIRCXd8wjPbDF5NBtPso15RUT0NDdpZCppf8BK8ZljufFCr7CJzs/v4LSTbTXBiGYaSlte6juJRUT3192+sUwtcMEt6eORNWrWrZVxSzZsHKlfH95bNOAUojppCCiPQopCCGYZQvrXEfBbOPwjUFnkyWQiZrJLwd5S7y5/lpvW+7Lfoew+eMGQM33xx/zfr6zP2EiatoDh8rJNlMnb2biMwEZiW2dxCRvxVcMsMwyoagpZBv91E2lkLUudnEFIJ9NDTA6tWwbFnLNmEaG51FsXx5dF9R18+GuKmzw30XkmwsheuB/UhkBqnqB8BehRTKMIzyImgpRLmP0s19lMl9lK2lkMk1FOU+CrYLT6CXjmBBXNw181GnUJKWAoCqfhna1YrbNQyjoxJlKeQy91FcFlEulkJU3CLqmlGyB9tnEwuIUyDrS/bRlyKyG6Ai0hk4m4QryTAMAzJbCpmmuWitpZDuSTrbmEJrLIW4tl6eioq2u4/Clk0pWQqnA2cCg4B5wMjEtmEYBpA5ppBuQrxcKprDfWdSPOHt9rIUqqo6sKWgqkuAn7SDLIZhlCmZso/SzX2UrqI52zqFqHPjMp6iZA/22RZLIagU8jn3UfhYIcmoFETkhojdy4GpqvrP/ItkGEa5kalOIV2gWQTWrEkqlSBxdQqjRrl00EzXCJLPmEJcoNn31blzx5z7yNMV5zKanfhsD/QFThKRPxdQNsMwyoRME+Jlm5IaJs5SAPjtb9MPmoXMPiqGpVAy7iPcDKnfVdUGABG5CTdj6veBDwsom2EYZULUNBfZZB8F4wRRxGUfAXTtmt86BX8sl5hCnMsqHzGFUk5JHQQEq5l7ABuraiOwtiBSGYZRVrTVfRRHXPYROKWQj4rmQmQftdZSKAX3UTaWwjXANBGZAgiucO3qxLQXzxdQNsMwyoTWuo989lGQqqrkFBGttRRUs3cflWr2UckGmlX1DhF5EhiLUwoXq+pXicO/LKRwhmGUB5mK1+Ke2n32UZBOnZJKobWWQlyWUXvFFDp1antMoZQDzQB1wAKgFthMRGyaC8MwmmlN8VpwQrwgwYG/tZaCPyfYvr2yjyoqnKz5mDq7JIvXRORk4BXgGeCKxPLywoplGEY50dYJ8YJ0Cvgv0mUfpVMKwSd2T3tlH4k4ObPp55prYNq0lrJAaVsK5wBjgM9VdW9gR2BxQaUyDKOsyFS8ls59lE4pxNUpQHr3UfCc8DXD5DumkIulcNFFMGlS6vnBa5Rq9lGdqtYBiEgXVf0Y2LKwYhmGUU609R3NQfJhKfhzOndOPd4e2UcVFdlZCj4YHlQejY0uSO3voVTrFOaJyIbA48BzIrIU+CrDOYZhrEdkyj7KNPdRkLBSiJr7CJxSiKuF8AOtH2ChpfvIP83n21IQyc5SiLpeU5NTZN5CKoalkE320WGJ1ctF5CWgN/B0QaUyDKOsaOvU2UHC7qM4SyHcb5SlEFQKYUvBD9ytsRQyBZqzsRT88eAb2rxSWLWqRC0FEakApqvqtgCq+nK7SGUYRlnRluyjsPsoKvvIv8s5eCzdk3Q27iOvfAoVU1izJrs+wu4jL3NJxhRUtQn4QEQ2aR9xDMMoR6KmucjGfRRlKQTbekvBK4OwwohTPNm4j8LKK2wxpCMf2UfpLAV/D6WafTQQ+EhEXhCRyf5TaMEMwygfotxH998PK1cmjwdJpxTWBibP8ZaCf6rP1VLI5D4KytYaSyGs7HKpU4iLKXi5SnlCvCsKLoVhGGVN0H3Us6dbf/11uO8+OP309HMfhd1HdXXJdZ99FGcpBOMPUSmp2SiFfNcpZBtT8Mog7D7ysZOwpVAS7iNojiPMBaoS6+8A7xVYLsMwyoigpdCtG8yd69ZXrUoeD5LOUggqBZ+Fkw9LIdy+LZZCXKC5NdlHYfdRRYWTe9264lgK2VQ0nwI8DNyS2DUIl55qGIYBpFoKAP37u6UfGHOZEC/sPkpnKbQl0ByOKbR3nUKc+6iy0inWNWtK1FLAvY95d2AFgKrOBgYUUijDMMqLoKUAyadwP+BFuY/i5j4Kz/8TdBOFLYVcAs3ZxhTy4T5qbUyhsdGdH6UUSsZSANaq6jq/ISKdgHYSzzCMciBsKfgB17tG0rmP0r1kB5zl0FpLIVPxWvC8fBavtSX7yCuFuroSnRAPeFlELga6icj3gX8A/5fpJBG5U0QWiciMmOMiIjeIyBwRmS4io3IT3TCMUiFsKfiBMc5SSBdTCFNX13pLIZs6hUK4j7KxFKICzencR6VkKVyImwDvQ+A04EngV1mc93dg/zTHDwA2T3xOBW7Kok/DMEqQsKUAqe9FCCuNbN+8Bk4peGUQ7CNXSyEuppDvQHNbYgph91GpWgqHAPeo6pGqeoSq3qaaWWep6iu49y9k6ldV9U1gQxEZmJ3YhmGUEuFBH1KfllWj320Q5T7q0iV1O2gpBBVIOJtI1e3bcsvkzKNx7qPKSujePSmLPx5cpiObuY+i+tlhB7j77tRz49xHpWwp/BD4RETuFZGDEjGFfDAI+DKwPS+xrwUicqqITBWRqYsX26zdhlFqRFkKwVdSerdIsH1U9tGVV8KDD6b2E7QUeveGp5+GUaNaVjQ3NbmB9JNPYMaMpAzha3bpAk89BSed5Pbnc5qLYPZRuB9VmD4dPv44/nplkX2kqj8DNsPFEo4BPhWR2/Nw7SijMVIXquqtqjpaVUf397luhmGUDHGWQtBk6HSOAAAgAElEQVR9FH7hTVT20YEHQt++qf0ELQWA/fZzKa9RdQp+gPXzDkUphYoK+P73oV+/5H4vU3CZjmyyj8LHwjGEKEshnfuolCqaUdV6EXkKN2h3w7l+Tm7jtecBQwLbg7EpuQ2jLAlbApCdpRB2H1VWRlc4h/v2T+LhQTOdUgjGMSCpjPxg6/vKV/ZRuJ+wMoizFErefSQi+4vI34E5wBHA7bj5kNrKZOC4RBbSLsByVV2Qh34Nw2hnghPiedJZCnHZR1ExhrVrU8/1fUdZCn6g9UohKvso7rWhrbEU0s19lMlSyCb7qBiB5mwshROAB4HTVHVthrbNiMhEYBzQT0TmAZcBVQCqejMui+lAnLJZDfwsF8ENwygdcg00x2UfVVZGT3vRs2fqvqi5gTJZCmGlELYU8p19FO7HK8hM7qNip6Rm85Kd8cFtEdkdOEZVz8xw3tEZjiuuWtowjDKntYHmKKUQ5T6KsxTCT9K5uI/yYSnkkn0UF1MIWwpVVSUeaAYQkZEico2IzAV+C3xcUKkMwygLmprg/PNdxk8ugeZg9lHwvLhZU+NiCsFB83//F6ZNc+vZuI/aYilkO/dRUL44SyGd+yjd2+sKRaylICJbAOOBo4EaYBIgqrp3+4hmGEaps2QJXHedW8/FUkjnPmqtpfDaazBiRPIcL4OnEDGFdNlHftvfd67ZR8H7CMpZaNK5jz4GXgV+oKpzAETkvHaRyjCMsiD4lJtr8VpUSmpcTCEbS8G39deD3LKPgsHjqMB5kGyyjyB1htdsA81BpbB6dfJYKWQfHQ58DbwkIreJyPeIri0wDGM9JTigtTb7KFNKavhc33fYPQMt34vcGkshvB5FXKA5mH0UPp5NoDnoPoJUpVD0mIKqPqaqRwHfBqYA5wHfEpGbRGTf9hHPMIxSJjighQfz1riP4mZNzaZOAVLdLV4GT5xSCFsKkP201+myj8L9ZBNoDruP/EuKgnIWmmwqmlep6v2qejCuwGwabpI8wzDWc3KxFILkkn3k+wpvR1kK6ZRCsIo6KG9rLIVsso/Cx/13kW3xGpSYpRCFqtaq6i2q+t1CCWQYRvmQLqYQthTC6ZVx7qMoX346SyF4LOw+iso+8v0XwlIIZh+F+8km0Bx2H5WkpWAYhhFHcECLshSCgebgk64fEMMT4sW5j9JZCsH2ubiPCmUpxMUU4pRCcErvsPto9Wo4lVvYko9NKRiGUfrk4j7yc1mKJM/L1n2ULvsoqDDWhuZcCCoFL29cTCHqfQxxBJVd2AIKZx95woHmcCzBnx9UCt9a9Rl/4wx+xl2l6T4yDMMIkov7qFcvN4BedFHuSiFdnUI691E6pZAPSyF4PqTPPoqzFCCpMMLuozPWXU8jlfyFc4pfvGYYhpGJTO6jqDevpatgjnsTWzpLIXgs6D4K+vY96SyF1sQU/Lq/Trrso7iU1PC+ClGqpz7Lk/yZA3iaO/kZC9i4JIrXDMMw0pJt8VowwBscqNtiKXhffJylEBW0rq/Pv6UQthriso8yWgqrVzN++b2c+a+/MOj+WXRiIJfwW27gHMACzYZhlAHpYgpB91GwQjidZZBLTMFfPy7Q3KlTy74KZSl4WpN9tDHz6fqbi2HIEK6uPZ36Tt1YPuFehjKXq7mENZVuiliLKRiGUfKkK16Lcx8FB/hsJsTzfUVt19enHgsOwp06tVRU+YopRAWJfV+ZKpr9vr6fvsN9/IS5DKX7hD/AuHEcVv0Kvz1kKnLsT6nH5dP678tiCoZhlDzZWgpR9QF+PZySmm1MAdxAGz7mibIUgu6jdNNctNZSSJd91FCv7MAHnLjgCRj7T4545x2WswE3cDZHTTmLwXsO482B8INAoDl4rxZTMAyj5MkUU8gUaA7XKUT14/uK2s6kFLKxFBobYehQWLw42c5nNg0fDldeCccem9pPOvdRZWVSPv1mJfzzBXjiCX7w8JOMZz4sAobsxJRD/8wPHj+RlfTikMS7LH2MpKoqOf22WQqGYZQN2RavxQWaO3duqQT8do8ebv2bb1pvKaRTCn65ejV8/nnLdnV1bv8nn7TsO52lMLhuDsP/7wme4QlG7vMy1K+DXr1YvMW+nPXuQcza9ADenLoR7/4JVj6evJ7vy8vVpYuTzSwFwzDKhmzrFOICzZ07x1sKlZXuSX3atPSWQtDVEm6Tzn3krxuubQA3OPv9a+sUvpwHX30FNTVQU8O+H9eyFTVUU8MGp9fCarf/0RkLGbBuPrwBFWzJgh+dxaBTD4I99uC5uzvz91Nh08A1gnJBasV1586pSsEsBcMwSp5cKpqj3EdRSiGoPEaMcEohnaXQq1e0bHGWQpcuqXKEp8boziq6vzOVTo+9yaO8yXf/9iZc+3VKm58BTQhL6UPVB33hW9Ww0Ua8+8XW/Ld6DDv+6iB2O3YEU34Og76TlNXLANExjGCKrZ+3KZwlVWhMKRiG0WrSuY+8peDnPYpzH6VTCsOHR/fdlpiCtyzcMaXrF7M5ljfZJfHZnul0OtON2NuyGbMGfZ9dztnZBR6qq6G6mp+e3ZeJT29IE5V89iIMG+b6vHxn6NsXtt3EbaerU4gqbAu6j7xSMPeRYRhlQ6ZAM6S+ZS3cLiqm4J+IKyvdOAywcGFqm2xiClE1D0H3UY8Zb/EZ4xl2zVxOAlbQi7fYmd9xEYdcvQtVe+zM1nv144TdYZczU/tZVgl+jA4O1mmzj9IUrwUtBS+fn6LD3EeGYZQNmdxH4AbiuJTUqqqW5/lBtqIiqRTmzo3vu1WB5rvuYuhppzOXQTx2wK38+qldmcVWNOE6220sbNDVnROeZA8yF6/l8ua14L4o91F7WwpWvGYYRqvJ9OY1SLqQsnUf+cEvaCksWZLaJmgpRKWwQrRSkIZ6zv30F3DiiawZvRdjeIfXtjqFj9i2WSF4mZsDzTkohWzevBY17XY27iOb5sIwjJxYudINoi+/3H7XzNZSyMV9NHAg/M//wLPPwre/DeedB3ffHd131Pubg22CffdnEfd+vQ+HL5gA55/P/Nufopbq2OwjH4DOVSlkevNanKXQ2OjaeGVgloJhGG1i4UKXVz9jRvtdM1NKqm8T5z6KCzT/6U+wzTau7XXXwXbbpbYJWhvZuI9G8S5TGc3IdW9z1Vb3wbXXIlVu5A5nH3mZ01kKcdNc5DL3UfiYf/Wmz6YyS8EwjDbhB6+oJ99Ckal4DZLuo6i5j6KUQjYErYNMSuEn3Mdr7IEiHLjB67y08U+ApDzB78unq7bFUsjmfQqqLa0I/+rNHj3c0iwFwzDaRDGUQjbZR959FBdojosJpCMbS6FzRQOb3XQ+9yUSTkczlXd1VIvitaCl4JVCW2IKmd685tuFFcbKlW69p5sUtWjZRwVVCiKyv4j8R0TmiMiFEcdPEJHFIjIt8Tm5kPIYRkem2Eohqk7Bt8nFfZQNQUshSqn0pYY/ztifwQ9dx184m315liX0j5zmIvh9+afz1loK2bx5za9nUgrh4rWytxREpBL4K3AAsDVwtIhsHdF0kqqOTHxuL5Q8htHRKbb7KJOlkE/3UbCP1ECzcgiP8x6j2HbZa3xy0V2cy19owGmoqAnxgpaCH4izsRS80ss2+yj4XYWVQtB9FFYKHclSGAvMUdXPVHUd8CBwSAGvZxjrNX5wKxVLIRhTKKSl4AfNrZjJs+zL4xzGN/Ti/+36KksOPiHlvKips1tjKQSny/CD+5QpsGhR5jev+fXwtrcUih1TQFUL8gGOAG4PbB8LTAi1OQFYAEwHHgaGxPR1KjAVmLrJJptoa7nssssUaP5MnTpVp06dmrLvsssuU1XVgQMHNu8bNWqUqqqecsopKW3nz5+vkydPTtl3yy23qDqhmz8HH3ywqqoefPDBKftVVW+55ZaUfZMnT9b58+en7DvllFNUVXXUqFHN+wYOHGj3ZPeUsu+MM27RxPNkUe7pJz9peU9wmU6bplpRkbynYcNGJeQ8JdQ2+9/p5ZdVIfWe6qnU6+iWsu+aayYrpN7TJpu4e9puu1GB/QMVVKurLwvJNFX79Wv5O22zjapI6u9UVdXynq6+uuU9wS1aU5N6T9ttd7A+/HDLewLVYcNuSbl2awGmajZjdzaNWvMBjqSlUrgx1KYa6JJYPx14MVO/O+20U6u/FMPoyDz0kPuPPvLI9rvmaadpYoBXPeec1GOTJ7v977yjOmCA6umnu/2PPpo8Z/Fi1VdeSW5ny+uvqwqNeiK369LO/bUR0Zs4TfuxqLmvww9XfeONZN/+c8ghro/589325psnj223nVvecovqBRdoQlG0vP63v6260Ubu+HPPqS5fnuzjpz919wWqN96YPOenP022WbjQbffo4bb/+lfVu+92659+6tqfeKLb3m03t/zb37L/fqLIVikUcpqLecCQwPZg4KtgA1WtCWzeBvyhgPIYRoem2IHm8Cssswk0R01zkQ29P/o3b3M2o3mXmb32YO+aZ5jGjiltoqbODl4/U/ZRpkCzd+80NSVdP77fdHUKfr2x0V1v1apo91E4+6jsA83AO8DmIjJMRDoD44HJwQYiMjCw+UNgVgHlMYwOTbGVQvgVlnF1CpkqmtPy1Vdw7LFsc+rubMTXHM0DXDbulRYKwV8/SuFkG1PIFGgOtvVBYt9vuopmSCqFrl2Tx+KyjzrMhHiq2iAiZwHPAJXAnar6kYj8BmfGTAbOFpEfAg1ALS7GYBhGKyh29lHYUoibEK/V2UcvvACHHw5r1vD1SRfz7TsuYhU9OSrNNBfplEKm7CO/38sfVF4NDcnK48bGVEshU0WzXw8qBV/RLJKc2rtDTp2tqk8CT4b2XRpYvwi4qJAyGMb6QrGzj8KWQnhCvChLobIyS6Vwzz1w0kluMqTHHmNJ3WasuiPZRxRB91FFRersq8Fl8PsKvrc5uH/duuQA7o8HLYWw+yjb7KOgu2rlSujeveWEeO39kh2raDaMDkKx3UfpLAXV6JhC1HYKqvDb38Lxx8N3vgOvvQabbZbz3EdeQQWvF1QAwcv5+wpaEGEXUib3UbZ1Cl4pePeRdx1BByxeMwwjv7z8crR/G+CVV2DFCrdeLPdROkshzn0EaSyF+no49VT49a/huOPgySehd28gtU4h/ATtjwVfshNVAZ1OGQWL1yCzUgi7j3zfcd+PVwqdOjk5ffFalFLwysAsBcMwmpk/H8aNg4ceij72ne/Avfe67VKzFNK5jyBGKXzzDfzwh3D77fCrX8Hf/54cJUlVLN4HP2BA6na2lkKYqirn32+tUhBxnw02SCpqiFcK3bu7661cmcw8Csrsv1ezFAzDaKYmkby9YEHLY19/nXqsVGIK2bx5DSIG5wULnJZ77jm49Va48soWjYJP/l27umv/7nduO5NSiJPD30PPnm6ATuc+Clc0h91HAH36wLJlyf3hoHxjo1Nu/npx7iOvFMo++8gwjPzhn0Rra1se8/v8k2SpZB/FBZrD7qOUwXnmTDjgAKcF/+//3HoEYXdQZWXyekGlkM59FGUp1Ne7p/WVK9336APUUZZC8Ek+7D4CpxSWLk3ub2hw11RNBpq9Uli1yn022CDZPuw+MkvBMIxm/KBTU9PyWHhfeF6dQpKtpZAu0Oz3f4cpsNtuLtXnlVdiFQKkKpawNRBlKWQbU/CWwqpVzlLwg3Qu7iM/eG+4YapSqK9PyubdR5WVSSUUdh8Vy1IwpWAYZUA2lkKQ9rIWso0pZHIfjWciz7AfbLwxvPkmjBqV9rrBQT488PvU0UwxhUzuozVr3MAOLd/Oli77yL9BLcpSiFIKmdxH/js2pWAYRjPpLIViKoX6+tTBP0hW7qPGRgZOuISJHMMb7Aqvvw6bbprxusE+wi6iKPdRpkBz8B78k3tdXXOyU06WQlApBGMK4boEH2j2SiGcfWSBZsMwYklnKUQpiva0FIJVuUH8QPvznzvlEbYUerMMDjmEAbdfzW2czH4840bSLIiyFMLuo2BhXCb3kR/8g+6jNWtaKgVV+MlP3AAdN/eRVwpR7qPgdxV0H61aZe4jwzAysGQJ9O0Lzz9fupZCUCmE3UfV1XDCCcntoFL4NrN4m7HwzDN89eubOJVbWUeXrK/rB/5gv37g32ADV9pw6KHplUJVVXIQ9rEDrxSWL3eDuHcfHXAA/P73bvB+4AG379BDk/cd5z5avdqFSHzfce6jpUtTlRBYoNkwjBAffOAGi9deK11Lob4+1SUSRATuvLNlZW6fV/7JW+xMb5bDiy+y/OjTgdymSq2sTA6gYfdRVRX85jew9daZ3UcjRrj1oFLo0QPmzXPb3/pW8rxrrkl+/7fdBmPHuvV07iNIupDClkIw+8inFfftm+zHLAXDMFL4z3/ccvbs5KBTV9dywC9VSwHcwOsHxwqa4IorGHruofyHLRnNVNhzz+an+Vyn0Pb9ht1H6YLQkOo2ilIKPXsmA8sbb5xs269f0h3Up08yruGVgrcqgu4jSJ4TdrUF3Ud+wA96z8KBZrMUDGM9J0opQEvLILjtB7dSiCl4NtwQerGC8Q/9CC6/nGWHHM+evMq8xOtWWqsU/FN12FJIF28ItoekUvABXq8UPAMDk/v3759UwH37piqFVauSVkXYUggqBe8+8sVrPtDsiVMKvr6hPTClYBglyiefuOXs2W7WB0/YMghu+6fT9nQfpbMUALbt/AlvsgtbzP4X/OUvLPz9XawlOeVoviyFKKWQzn0EMHy4W3ql691HnqBSgPSWQpxSiHMfBWMKnqD7yMvsg/RmKRjGeo63FJYtg88/T+4PWgZNTcVVChkthSef5O+zxjKARdx33HNw9tlUVKaO/tlMUBeFH0Dz4T5atCh5D8FBOug+qq2NtxTSKYUoSyFOKURZCvX1TmazFAyjyKjCxIktC5fCx4MDsCo8+GByX10d3Hef2/bH/KARPGfSpGQV7U03wVVXwdy5sNNOrs0HHyQDq3fcAVdfDddeC1984RSDPxalFB5+ONXSeOQRN1Hbo4+mTtiWjkcfdRk5YWJjCt98A2eeCQcdxOKewxjNVL7cbG+gZZ1AWy2FbNxHcUrBWwoLFybvJ859VFMTbymsWgUbbeS2wzGF2293/ccFmoOWSVSg2buPzFIwjCLz/vtwzDHu/S4nnghffpl6/IMP3PHgzKWffAJHH51UJttuC8ceC/ffDx995I75lEbPrFkwfrzL1NljDzjjDDcxaEVFMqVz9WrYZhs3aNx/P1xyCfzyl3D99e74sGFu6WcK9YPXvHlw5JFuklGAzz6DI46AP/3JvcQsLEsUX3/t2t5xR8tj9fXJgfWCCxI7n3sOttvOabdzz+XaQ1/nc4a2mDrbD9TpJqhLRz7cR75O7oorkks/SIskn/4rKpyVUFOTjANUVLiBe+FC91t7pfCb36TK9+KLcOmlbnAP1kNEWQpekYDFFAyj5PA+/fvug7vugn/+M/X47Nlu+dlnyX2LFyePvfsufPqp254zJ9neL8PX+cc/3DkTJrhiqbo69zoBP4gNGuTcHP5Yjx7wzDPu2Pbbu+XgwW6/lykoT3A5c2bq8XQsWRItN7gBq08fN2CdePhyOOUU2Hdf90j82mtw/fV0q+6eck74yT7TVNZx+EHXP5m3xn1UVeVkP/tstzz//OQg3aePO0/VpaM2NjrLrE+f5PTYw4a5OhJwrihVuPBCt92lS3J9yhS33HxztwxXNINTGOFXlfq233zjrMf2wJSCYcTgB/R//9stvY8/fHzu3OQ+73P+9NNk+06d3LZv75dx19lnHzcgdOrklv5ptmdPN2h07uwGnOHDk9fYYQe37NLFDU6+z6A8UcuodNYw4T6CNDQkBtwnnnCmzJ13wv/7fzBtmpvcjuTTry/wCk930Vr3ke/XB3KDL9fxZMo+isIP0kFXTnW1W86Zk7p/xIikUvfxiSC/+x384AfJNltu6ZbhlFRoWcwdjCl07hz/hrl8Y0rBMGLwg6D3lccphWAQ2LttvFKoqoLvfjc7peCfHL07xuOfLoNuBkgOQj16JM8JK4WgPFHL4DQMcYT78KhCr4ZajvzXcXDwwW5Ue/NNV/obeKGxd5n4wdsPbmGlkKv7KE4p5OI+isIP0l4RQFIRRCmFqPUgwf1bbOGWUe6jYL9BmYPTk7cH9j4Fw4ghPAjmaikMGQKbbeYGgjffdMVP/lhwKungdYYPTx3AwCmFZ59tqRS8IhgxAnr1cutdu8KI4corT6yk6bMaKt6vYR9q6ffpMhond6fynWo2o5raFX2pYENqazM/fvp7+vxzN0B5+ZoeeYyZ/Jz+M2qc0/zii5PlzQG8UvCB6ny5j8JKIVv3UabrRA3Sfn3p0tQnej/g9+kTP21TsEDOxyjCL9nxfQQJVzS3F6YUDCOGsFL44gvnv+7ePfX4vHlJN4p/ql6+HN54w3lQRoxwWT5vv+2OffON89P379/yOt69EMRbCr164dKKPv0UZs/m8M9msyVz2GHxIra6oJYZ1DD0dzV0XV3LHxvrYQT8CPehATgErsF9AJoQvnmqD2ze1z0WV1e70S+klcbMgDsAGmHNMVDVC5g/n8pnn+UrduTJM5/mxCtGxn6PfvD2SsEHTAvlPsolphCFH6SDlkKU1QDJAT/OSgi3Cc4cG84+ClsKgbePtiumFMqUzz93f1CDB7usGBG3vmJF6tubcmHVqtT0uCi+/NK169bN+bp9Rs6QIW5wbGxM+sCXLHFPx+FlsB9wT7dDh7rzg5W7Qbp2df2KJNv5ffPntzzPy/fVV+478dfw23F06eLaffGF63eHHVyW0RZbOL/wCy+4Qbqhwd3DoEGu3dtvu/TFYA3BokVukPeDwvLlyf6mTHEJOk1N7rf0+7fcEvc4/t//ugvOns2hz89mGz5h7B9nwyVfNo+quwEjGEBjw0ZUVvVlFlvB9tV0H9yXv06q5uT/reat2X257bFqlrEhf7xsFbf9vobua2uppoZqatisVy3HjK5xgn/9NU0fzUSaGlNmIhq8HPyfVMWLUN8FtLITS874LTv/7X/53SYh0yZE2FLwT75tzT7KxlJoi/soylKAaEshW6XgFeGCBckHiThLIWwxthuqWlafnXbaSYvBunWqkyerNjUV5fLN1NerHn+8qhsZVG+7Lbl+552qlZWqH36Ye78ffOD6OPJI1YaG5P7GRtXHH3fXPe205LVAdcyYaDmOOUb1xBNT2/jl0UernnRSaj/hvuI+Rx2lesop2Z93662qIsntW25J3Y77jB6dXL/8cre89NLotv4+/ae6WrVLl+T2PfeofvJJctv3U0GDbsp/9fs8o2cwQd/Y+Rx9ggN0+bc2cz9ioNOG3n30DXbW/+7xU9UrrlCdOFF16lT9/MPlzb/711+7e3vgAdVPP3Wn3n67+746d06VMSjfwIHJ37qmRrVHD9VJk1L/Ns44Q7WqKv77+tOf0v9tvf++a7f99m57yRK3/eMfu+0FC9x2nz65/c0uXZraT12du7cbb0y9J1A999zs5V23TrVrV9U//CF1n/8OrroquX/NGnfNK66I72/tWtVu3dxvr6ravXtSliuvdGNKr17uby0MqI4dm17ebAGmahZjrFkKWfLYY3DUUfDWW8nZEYvB00/D3Xe7vPYHH3Q59J677nJPYe+95/Ljc+Hdd93yH/9wOfn77++2X3rJTRF8/vlwyy1w+ununervvQd//jP87Gfuul6OE05w66pw4IEuZT24fOgh93T8i180J6fwwQcub/6ss2D33aPlmz49mRZ4xhmw557w4Yfwxz8mtz11dU6ue+91cvzmN87lfd99bvv3v49/j8vMmfCHP7hU0AMOcDLv9506xgyvYb+Na6mbX0Pnb2rosrKG7nW1DKmqYe+qGrbov5QFXzVBjXuC3WoUNDbAgIeg8mFYtKt7MhzwdgO/3HQu3ebPobJhXfN1dUYPVm+xOd12GAVbHOXMkS22gC22oLK6msp3YOMdgIBLYRPc3+OOO7qnyrffdqmpFRXJjKfaWhfXmDDB5dNXVTkrZcIE10cw++jjj531NnUq/PjHyf21te77uv1294TrmT3bfa++RiIOH/s4+WS3rK52NSDeVdYWS2HaNHd/4Ky8t95Kbgf77NrVvb9n/vzk33YcVVWun2DAv6rKvSH0yy9hv/2S+7t2bdk2TOfOLqY0dKjbfukllzJcUeH6EnGuxiFDWp47Y0b0/kIiToGUD6NHj9apU6e2+3Uvu8wNLnfdlTpHfHtzzTUu42/hQlcsU1mZnF6gUye3fumlyWKcbPnVr5J50NddB+ed59ZvuAHOOcf9Qc+d6wYFX6Tjy++7d3cDfWOjc3n7d/L26pUMTPrlypWubdjFFQxgxhF1btx5ffq49g0Nzs8/fDisWtFIr8alfPpWDb0bapLVSP6T2G5aUkNFbWBfuAQ5SNeuaHU10qcPMz7uRH2Du+/N4twJFRXuv3yLLZID/+abO99Trk71NGy2mauGXrzYzef/2mvJY7/7nYsJe1avdu62e++F446DH/3IVT179t/ffYdvvdXyOtn8bplYtMgFYPv1y65uIheWL3fK45JL4Le/zW/f5YaIvKuqozO1M0shS3zmSTgDpRhyfOtbrnLVxxO23NINxl984dpE5ZNn4tNP3cC5bFnqPfr1uXNTsycgORgMG+baDRnintaCCSi+jV+GM2jC7dLRfK6qCwzU1lIVGtD9+v0VNUhDDQMqa9lwdA1zV9SwQWPC+bxzROeVlc5x3LcvFdXVsMkm7hG8byAI6wOxwe1u3Zr97z/f0w2+4w9yFc3FxKelNja2fNIM+65ra11sJC5ltra2ZRDUkw+/t38uzdVSyIbWWiHrM6YUssQXn/hlsfjPf5Jm94gRTimMGJEfpTBihHu6DiqF4P1uuWX0w+yIEe6cdMG2WOrqIp/W027X1sbP0wzQuzc7NfTlC6pZ16Maxo7g31LNm279XtcAAAs1SURBVHOq6TmkLxf8LmJw32CDNo8cI0Y4pZDlGyULyogRzp20wQbJamePH+C7dnVf/9KlLZWCBlJmly5NdcnkGz+nTx4NpWZaO9ne+kxBlYKI7A/8BagEblfV34eOdwHuAXYCaoCjVHVuIWVqDarJwbEULIXDDnPrI0Y4/7BXCi+95Pa3RinMmeNiJnV1yakT/PU8UemSXo5KGthh46XwcfSTe+x6OtdMt26pT+fbbtvyaT283qcPVFXxl4udm2T8gbDrA/DqJW4SuaN2A36S+/eTDV4pxj1VtycjRjirb9mylvJ4pTV8uIuhhCuWV650bhw/j1JtbWEVnbcUCqEUzFLInYIpBRGpBP4KfB+YB7wjIpNVdWag2UnAUlXdTETGA38AjiqUTK1l/nwXgOvTxw2evuikvamtdWmdQUsBlK02XgG1NYymhs361iKLa6j7Yw1dV9W4USFD3KiuDi5bCnvMAG2CkQtg7c+hQuD8L6FrF6hbC7vOAc5qdI+OgQH+j1/VcAPL4QHcJ4x3zfjB27tmotwxwe3gi3hzJJwqmE3qYFsJFjIVm+B9huXxSmLECKcUghXLAwe6uNGnnzql0NQUrVjyibmPSotCWgpjgTmq+hmAiDwIHAIElcIhwOWJ9YeBCSIiWqDo9zPPwP/8T+Z2u658jgsXntu83dQEM4DeTbB8LXzWvTBPNZnQhByb/AW4tYHzvl7KL6ml00Uu8fvnAD6b5H/dYmVFLxrJrMGOA3q879ZHAHW3Jvd3boJ1QI8PgU8qXOTOD9ybb85XK6r5+7+qOeSEvoz6fsTgvsEG7f6FmVJIrofl8du+zRlnuMDzokVw/PEuu+2oo1zAvKnJfdrjnjLVyLQGcx/lTiGVwiAgONnwPFqG+JrbqGqDiCwHqoElwUYicipwKsAmm2zSaoE22MC9zDsT/Ws2YHFjasOqzjD02/DVR7CincvOg3Sqgq4jgc4VVPTow5SPqtnz0L40bVjNA89U88OfVXPd3/uyqLGalVV9aKrI7ifu2dNNwywC/3NushisRw+XtfHnP7usps4RhXEDVsHKS2HEpUDvvN1qm9h1VzeV8w9/6LZ32SV1uxCMGuUyww46qHDXyJatt3YpvrW1blqiIEOHwq9/DSed5Kxen2a6ww5OOfTu7Yr8PDvuWNjvbeON3d/Y0Ufnv++uXV0Ksne5GpkpWEqqiBwJ7KeqJye2jwXGquovAm0+SrSZl9j+NNGmJqpPKF5KqmEYRjmTbUpqIY2qeUAwGW4w8FVcGxHphHvOzGIyX8MwDKMQFFIpvANsLiLDRKQzMB6YHGozGTg+sX4E8GKh4gmGYRhGZgoWU0jECM4CnsGlpN6pqh+JyG9wc3BMxk2+eK+IzMFZCOMLJY9hGIaRmYLWKajqk8CToX2XBtbrgCMLKYNhGIaRPZaoZRiGYTRjSsEwDMNoxpSCYRiG0YwpBcMwDKOZsnufgogsBj5v5en9CFVLlykd5T6g49yL3UdpYffRkk1VtX+mRmWnFNqCiEzNpqKv1Oko9wEd517sPkoLu4/WY+4jwzAMoxlTCoZhGEYz65tSuLXYAuSJjnIf0HHuxe6jtLD7aCXrVUzBMAzDSM/6ZikYhmEYaTClYBiGYTSz3igFEdlfRP4jInNE5MJiy5MLIjJXRD4UkWkiMjWxr6+IPCcisxPLEngJZCoicqeILBKRGYF9kXKL44bE7zNdREYVT/JUYu7jchGZn/hNponIgYFjFyXu4z8isl9xpG6JiAwRkZdEZJaIfCQi5yT2l9VvkuY+yuo3EZGuIvK2iHyQuI8rEvuHichbid9jUuLVA4hIl8T2nMTxoQURTFU7/Ac3dfenwHCgM/ABsHWx5cpB/rlAv9C+a4ALE+sXAn8otpwRcu8FjAJmZJIbOBB4ChBgF+CtYsuf4T4uBy6IaLt14u+rCzAs8XdXWex7SMg2EBiVWO8FfJKQt6x+kzT3UVa/SeJ77ZlYrwLeSnzPDwHjE/tvBn6eWD8DuDmxPh6YVAi51hdLYSwwR1U/U9V1wIPAIUWWqa0cAtydWL8bOLSIskSiqq/Q8k16cXIfAtyjjjeBDUVkYPtImp6Y+4jjEOBBVV2rqv8F5uD+/oqOqi5Q1fcS698As3DvSS+r3yTNfcRRkr9J4ntNvA2dqsRHge8CDyf2h38P/zs9DHxPRCTfcq0vSmEQ8GVgex7p/4hKDQWeFZF3ReTUxL5vqeoCcP8kwICiSZcbcXKX4290VsKtcmfAfVcW95FwPeyIezot298kdB9QZr+JiFSKyDRgEfAczopZpqoNiSZBWZvvI3F8OVCdb5nWF6UQpU3LKRd3d1UdBRwAnCkiexVboAJQbr/RTcAIYCSwAPhTYn/J34eI9AQeAc5V1RXpmkbsK5l7ibiPsvtNVLVRVUfi3mE/Ftgqqlli2S73sb4ohXnAkMD2YOCrIsmSM6r6VWK5CHgM98ez0JvyieWi4kmYE3Fyl9VvpKoLE//QTcBtJN0RJX0fIlKFG0jvV9VHE7vL7jeJuo9y/U0AVHUZMAUXU9hQRPxbMYOyNt9H4nhvsndrZs36ohTeATZPRPU744I0k4ssU1aISA8R6eXXgX2BGTj5j080Ox74Z3EkzJk4uScDxyUyXnYBlnuXRikS8q0fhvtNwN3H+ESmyDBgc+Dt9pYvioT/+Q5glqpeFzhUVr9J3H2U228iIv1FZMPEejdgH1x85CXgiESz8O/hf6cjgBc1EXXOK8WOwLfXB5dJ8QnOZ3dJseXJQe7huMyJD4CPvOw4X+ILwOzEsm+xZY2QfSLOjK/HPeWcFCc3zjT+a+L3+RAYXWz5M9zHvQk5p+P+WQcG2l+SuI//AAcUW/6AXHvg3A3TgWmJz4Hl9pukuY+y+k2A7YH3E/LOAC5N7B+OU1pzgH8AXRL7uya25ySODy+EXDbNhWEYhtHM+uI+MgzDMLLAlIJhGIbRjCkFwzAMoxlTCoZhGEYzphQMwzCMZkwpGOsNItIYmEFzmmSYLVdETheR4/Jw3bki0q8V5+2XmPmzj4g82VY5DCMbOmVuYhgdhjXqphTIClW9uZDCZMGeuEKmvYDXiyyLsZ5gSsFY7xGRucAkYO/ErmNUdY6IXA6sVNVrReRs4HSgAZipquNFpC9wJ67YaDVwqqpOF5FqXMFbf1yRkQSu9VPgbNwU7m8BZ6hqY0ieo4CLEv0eAnwLWCEiO6vqDwvxHRiGx9xHxvpEt5D76KjAsRWqOhaYAPw54twLgR1VdXuccgC4Ang/se9i4J7E/suA11R1R1xl7SYAIrIVcBRugsORQCPwk/CFVHUSyfc3bIerdt3RFILRHpilYKxPpHMfTQwsr484Ph24X0QeBx5P7NsDOBxAVV8UkWoR6Y1z9/wosf8JEVmaaP89YCfgncQ0+N2In8hwc9y0DADd1b03wDAKjikFw3BozLrnINxg/0Pg1yKyDemnMo7qQ4C7VfWidIKIe+VqP6CTiMwEBibm3P+Fqr6a/jYMo22Y+8gwHEcFlm8ED4hIBTBEVV8C/hfYEOgJvELC/SMi44Al6ub1D+4/APAve3kBOEJEBiSO9RWRTcOCqOpo4AlcPOEa3CSII00hGO2BWQrG+kS3xBO352lV9WmpXUTkLdyD0tGh8yqB+xKuIQGuV9VliUD0XSIyHRdo9tMaXwFMFJH3gJeBLwBUdaaI/Ar3Fr0K3KyrZwKfR8g6CheQPgO4LuK4YRQEmyXVWO9JZB+NVtUlxZbFMIqNuY8MwzCMZsxSMAzDMJoxS8EwDMNoxpSCYRiG0YwpBcMwDKMZUwqGYRhGM6YUDMMwjGb+P5GDWoLmZZfFAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8d45f7fe10>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(1, len(scores)+1), scores, c='b', label=\"DDPG\")\n",
    "plt.plot(np.arange(1, len(scores_window)+1), scores_window, c='r', label='Average')\n",
    "plt.plot(0.5*np.ones((305, )), 'k', linestyle='dashed', linewidth=1.0)\n",
    "plt.ylabel('Average Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.title('Learning')\n",
    "plt.legend(loc='upper left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.savetxt(\"09May_scores.csv\", scores, delimiter=',')   # solved scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}