05_RNN.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "05_RNN.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/MatchLab-Imperial/deep-learning-course/blob/master/05_RNN.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kiobin3g2wyW"
      },
      "source": [
        "# Recurrent Neural Networks\n",
        "\n",
        "A Recurrent Neural Network (RNN) is a specific type of architecture that is widely used to deal with sequential information. So far, the introduced CNN architectures from the previous tutorials treated inputs as independent objects, however, many applications need to deal with data that is interconnected. For instance, if you are translating a sentence from English to Taiwanese, and you are predicting the next word, it is useful to know which words came before the last one.\n",
        "\n",
        "RNNs are called recurrent since they apply the same operation to each of the input sequences, with the output of an individual element being dependent on the previous one. Theoretically, RNNs establish a connection between the actual input and ALL the previous ones. Although this is assumed, in the practice, RNNs have proven to only *remember* a limited number of inputs. In other words, RNNs have a *memory* that allows them to *remember* previous elements and use their information to deal with the current input. \n",
        "\n",
        "RNNs can be split into multiple types depending on their applications. For instance, if we want to predict one word given only the previous one, the topology of our network is a *One to One*. Another example is image captioning, where we can design a *One to Many* architecture to obtain a description from a single input image. The following diagram shows the different types of problems we can face ([Source](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)):\n",
        "\n",
        "\n",
        "![alt text](http://karpathy.github.io/assets/rnn/diags.jpeg)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9t6BQY9B21nV"
      },
      "source": [
        "**A Closer Look to RNN**\n",
        "\n",
        "The figure below shows the simplest version of an RNN, which can be easily derived from a simple feedforward architecture by adding a single loop: \n",
        "\n",
        "![alt text](https://i.ibb.co/qnGH6RT/vanilla-rnn.png)\n",
        "\n",
        "During training, the hidden state $h$ is iteratively updated based on the input value $x$ and the learned weights $W_h$ and $W_x$. The final output $y$ is estimated from the current state $h_t$ and the matrix $W_y$. \n",
        "\n",
        "Although RNN can assure short-term dependencies within the network, simple RNNs become unable to learn to connect information as the gap between past and present information grows. To overcome this limitation, in practical applications LSTM unit is adopted, that is a special RNNs architecture composed of multiple interacting layers ([Source](https://colah.github.io/posts/2015-08-Understanding-LSTMs/)):\n",
        "![alt text](https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-chain.png)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EOUUKOPH2_SG"
      },
      "source": [
        "In this tutorial, we will use LSTMs straight away as a black box."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iMJ28M_227fj"
      },
      "source": [
        "# ***Many to One* RNNs - Regression**\n",
        "\n",
        "In this section, we are going to implement an easy example of an RNN in Keras. Given the records of a local newspaper, which indicates the number of new subscriptions per month, we want to predict the number of members that will join next month.\n",
        "\n",
        "First, we load the data with the utility method in Pandas `read_csv`, which allows us to load directly the dataset from an URL address. Let's print some rows of the imported record to see the format of the data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0PZRejNk29jN"
      },
      "source": [
        "# install ktrain first due to some dependencies needed for the transformer sec.\n",
        "!pip install ktrain\n",
        "import math\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "from IPython.display import SVG\n",
        "from keras.utils.vis_utils import model_to_dot\n",
        "import io\n",
        "import requests\n",
        "data=pd.read_csv(\"https://drive.google.com/uc?id=1Hv2nuwVXO_aZN89llGva0hSH6k3kKAVm\",usecols=[1],engine = \"python\")\n",
        "data.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hd3kk4Zh3CA-"
      },
      "source": [
        "Moreover, let's plot the number of subscriptions per month."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uEmJqK1z3Dei",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 350
        },
        "outputId": "47e8c470-2d87-4f45-ce25-78c17bc94b39"
      },
      "source": [
        "plt.figure(figsize = (15, 5))\n",
        "plt.plot(data, label = \"City Newspaper Subscriptions\")\n",
        "plt.xlabel(\"Months\")\n",
        "plt.ylabel(\"1000 Subscriptions\")\n",
        "plt.title(\"Monthly Total Subscriptions to City Newspaper 1949 - 1960\")\n",
        "plt.legend()\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3sAAAFNCAYAAAC5cXZ6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeXxcdb3/8dcnW5O2WZo0bZJuaVNooU0XWmjLjooIckURWQQBkesPxR1R0KsiV6+4X1HUiyKboly4KrigyL4U6Eage5u0adO0SbM1+57v749zJp2mWSbpTJJJ38/HI49mzpxz5jszJ33MZz7f7+djzjlERERERERkbIkZ6QGIiIiIiIhI+CnYExERERERGYMU7ImIiIiIiIxBCvZERERERETGIAV7IiIiIiIiY5CCPRERERERkTFIwZ6IHHfMzJnZ3D7uu97MXhnuMYXCzHL9scdF4NwvmNmN4T5vj8c4y8y2H8PxXzGzX4dzTCPNzJ4ys+tGehwiIjI2KdgTkRFjZsVm1mZmk3tsf9MPanLD8BjDEcTMNLOGoB9nZo1Bt8/q47iwBpZmdqaZrTazWjOrNrNXzezUcJ3/WDnnXnbOzQtlXzM718z29Tj+v5xzEX0vexlHn18MhHh8gpndYWY7/Wui2Mx+E7i2nXMXOuce9Pc9puvBzB7wx3ta0La5ZqaGuoCZ3Wtm282sy8yu73HfODP7sZntN7MaM/u5mcX3co4TzKzFzH4btM3M7KtmttfM6szsD2aWcgzjPM/Mnvf/jot7uf90M1tjZvVm9raZndnj/kwze8Q/vsbMftfjef7GH2eZmX1hqOMUkeigYE9ERtpu4KrADTPLB8aP3HAGzzm31zk3MfDjb14ctO3lSI/B/3D5V+CnQDowDfgm0Brpxw5FJLKRUeJx4H3Ah4FUYDGwHnhnhB6vGvhWhM4dFfq51t4CPgls6OW+24DlwELgROAU4D962e8eYG2PbdcCHwHOAHKAJLy/w6FqBH4D3NrzDjNLB/4CfB9IA74H/MXMJgXt9kegDJgJTAF+EHTfHcAJwCzgPOBLZvaeYxiriIxyCvZEZKQ9jPdhKeA64KHgHcws1cweMrMKM9tjZv9hZjH+fdeb2Stm9gP/W+zdZnahf9+3gbOAn/kZtp8FnfZdfrblkJndY2bWc2D+9h/22PakmX0+1CfX19jN7CTgl8Aqf2yH/P3f62c268ysxMzuCPGhTgRwzv3eOdfpnGt2zj3tnHvbP+8dPbIRvU0JzfMzBnVm9oT/wRIzSzSz35pZlf96rTWzqf596WZ2f1BG5M/+9nPNbJ+ZfdnMyoD7rUe2zs9y3W5mW/xj7/cfawLwFJBjh7OjOb08h/eZ2WZ/TC/4r2nwub/oZz5qzexRM0v075tsZn/1j6s2s5cD11OP9+4l/9e3/DFc4W//dzMr9I990sxyentDzOxdwPnAJc65tc65DudcrXPuHufcff4+L5jZjb1dD2Z2qpmVm1ls0DkvNbO3+rkOHgQWmdk5fYwp1czuM7MDZlZqZt8KnN+/Ppf5v1/tXx8L/NsfC3pvTzOzdf51Um5mP/K3B66pj/vXwwEz+2LQY59mZq/5z+2Amf3MzBKC7ndm9hkz22VmlWb2/eD3xcxuMLOt/rXyTzOb1ePYm81sJ7Czt+fuv+7PAi293P1vwN3OuWrnXAVwN3BDj9fuSuAQ8Gwvx97nnCtxzjUA3wWuMLMhfWnlnFvjnHsY2NXL3acDZc65x/y/898CFcCl/hjfDcwAbvWvtXbn3JtBx18H/KdzrsY5txX4FXD9UMYpItFBwZ6IjLTXgRQzO8n/0Hkl8Nse+/wULysyBzgHLzj8aND9K4DtwGS8b7rvMzNzzn0VeBn4lJ9h+1TQMRcDpwKLgMuBC3oZ24PAVXY4sJwMvAt4ZBDPr9ex+x+0bgJe88eW5u/f6O+TBrwX+ISZvT+Ex9kBdJrZg2Z2oR35TX+orsX7gJsNdOB94AXvA2Iq3ofIDH/czf59D+NlYhfgZRF+HHS+LLws4yzg43085tV4r30eXsD6H865RuBCYH9QdnR/8EFmdiLwe+BzQCbwd7wMR0LQbpcD7wFm473P1/vbbwH2+cdNBb4CHDXV0Tl3tv9rIEv7qJm9A/iOf+5sYA/whz6e27uANc65kj7uD36so64H59xaoAp4d9CuH6HHlyE9NAH/BXy7j/sfwHtv5wJL/XMHpsa+CJzr/34OXrBxdtDtF/3ffwL8xDmXgve+/W+PxzgPL3v0buDLftAL0Al8Hu/vdBVedvOTPY79AF6G7RTgEvyAy8wuwXufLsV7317Ge/+DvR/v/4KT+3juA7Eev083s1T/8VOAO4G+pj32PHYc3msQCT2/mDK8jCTASrz/Cx8078uZtYHA3/8/IRsvwxnwFt7froiMUQr2RGQ0CGT3zge2AqWBO4ICwNudc/XOuWLgh3gfegP2OOd+5ZzrxAvQsvE+xPfnLufcIefcXuB5YEnPHZxza4BaDk+5uxJ4wTlXHsqTCnHsPR/zBefcRudcl5+V+z3eB+1+OefqgDPxgpZfARV+1mmg1yHYw865TX6w9TXgcv85tOMFeXP9bMJ651ydmWXjBWU3+ZmCdufci0Hn6wK+4Zxrdc41H/Vonp/5GZFqvADlqj726+kK4G/OuX8559rxpqol4WU+Au52zu33z/0XDr/H7XjXyCx/zC8750Jd13Y18Bvn3AbnXCtwO142LreXfTOAAyGety8PAtdA9xS+Cxj4y4b/AWaan+EO8K+Fi4DPOecanXMH8YLzK/1dXuTwtXYWXlAbuB0c7LUDc81ssnOuwTn3eo/H/6Z//o3A/fjvqX/dvO5nOIv9cfa8tr/rZ9f2Av/N4evhJuA7zrmtzrkOvIB2SXB2z7+/up9rrT//AD5r3nq3LOAz/vZAdu4/8bJ3+/o49kY/s5kKfLnHseH0Gl7G+yozizevuE9e0GNNxwuyn8f7suWHwBP+F1WBKea1QeerBZIjME4RGSUU7InIaPAw3pqm6zk6azEZiMfLoATswVuTFlAW+MU51+T/OpH+lQX93tTP/t0ftv1/Hx7gvMFCGfsRzGyFecUZKsysFu9D7uS+9g/mfxC+3jk3He+b/hy8D8yhCs5A7fHHPhnvOf8T+IM/Pe975hWvmAFUO+dq+jhfhXOutylz/T1mr1Mie5FD0OvqnOvyz9XrdcGR7/H3gULgaX/K4G0hPmZvj9uAl33r7T2twgsqj8VvgX8zb2rr5cDLzrl+A0g/CP1P/yfYLLz39IA/lfIQXsA1xb//ReAsP4iPxcvYneEHsqlAgb/fx/CysNv8zNHFPR6n1/fUzE40b/psmZnV4QVsPa/tvq6HWcBPgsZdjZfRmtbHsYP1beBNvOe4GvgzXlBbbmZL8LK0P+7j2N/gfSnzArAZL9ACL3t8BH96bGBq8lODHaRzrgov4/kFoBwvc/1M0GM1A8XOufv8LzL+gPe6nAE0+PsEF49JAeoHOw4RiR4K9kRkxDnn9uAVarkIr7hAsEq8D13B3+DPJCj7N9Dpj3F4vwUuMbPFwEl4HwJDNdDYexvbI8CTwAznXCreOq6j1hMOxDm3DW/KXmB6VyNHZhqyejlsRo9xtgOV/ofGbzrnTsbLnF2Ml4ktAdLNLO3oU3nDCGGoPR8zMF1zoGP3E/S6mpn55xrwuvCzrLc45+bgFU/5gpmFWjCl5+NOwMvg9fa4zwCnmdn0EM/d21TSUrxszqV4GeFQv2y4H28q8KVB20rwCvZM9qeJpjnnUpxzC/zHKsQLij8NvORni8vwpuC+4gfUOOd2OueuwgsSvws87r8OAX29p78AtgEn+FNAv8LR13Zfx5YA/y9o3GnOuSTn3Oqg/Yf8t+68Na6fcs5N86+LKmC9/5zPBXKBveatP/0i8EEz2+Af2+Wc+4ZzLtf/omUz3vVw1DXhnPtd0NTkC3veH+JYX3TOneqcS8e7JuYDa/y73+bo18H5x9XgZZoXB9232B+viIxRCvZEZLT4GPAOfwphN39q5v8C3zazZH/a1hc4el1fX8rx1ssNiT9tay3eh+z/G8wUsRDGXo63Lih4nVkyXrasxbwS+h8O5bHMbL6Z3RIILMxsBt4UuMAUuwLgbPPaRKTiTT/s6RozO9kvLHEn8LhzrtO8UvD5/pTOOrwgsMvPMD0F/NzMJvnTys7u5bz9udnMpvtTFL8KPOpvLwcyAmumevG/wHvN7J1+lvEWvEBmdR/7dzOzi81rSWB409g68aac9qbn9fN74KNmtsTMxuFlp97wpyUewTn3DPAv4E9mtszM4vzr4CYzu6Hn/vR+PYCX7f4SkM/RX4b0yp/q+A0OTynEf7+eBn5oZinmFQrKsyOLubwIfIrDUzZf6HEbM7vGzDL9QOiQvzn49fuamY03r7jLRzn8nibjXT8NZjYf+EQvQ7/Vv5ZmAJ8NOvaXwO12uGBMqpl9KJTXImjcCeYV6TEg3rxiQIH1uNPMKwJkZrYSbxrzN/xD78WbKrnE//kl8Df8db7mFSnK8489GfgRcGcgOB4s/31JxMvCmj/O4EI2S/2/tRS86cslzrl/+nf/CZhkZteZWayZXYY3tfNV//6HgP/wX+P5wL/jfSkkImOUgj0RGRWcc0XOuXV93P1pvMzULuAVvOzXb0I89U+Ay8yr4Hf3gHv37kG8D9qDmcIZ0N/Yn8P7Vr3MzCr9bZ8E7jSzeuDrHF38oi/1eMUp3jCzRrwgbxNeEIRz7l94H5zfxiv9/9dezvEw3ge/MiCRw+uWsvBaCNThral8kcOvxUfwgr9twEG8gimD8QheALILKMJvG+BnJn8P7PKn7h0xvdM5tx1vWu1P8TKo/wb8m3OuLYTHPAEv69aAlzX7uXPu+T72vQOv2MUhM7vcD+C+BvwfXpYkj8Nr3npzGV7xmEfxAstNeAVInull396uB/A+wM8C/hQ0TTkUv+foNYPXAgnAFqAG730Nnmr6Il5Q9lIft8GbOrjZzBrw/r6u7PElyIt402SfBX7gnHva3/5FvC8v6vHWlT7K0Z7Auz4L8AKq+wCcc3/CyyL+wZ8CuglvvehgPI03zfF0vACumcMFaPLwvihoxPt7vy0wbudck3OuLPCDd920OK9qJ3hTUf/uH/sU3prOewc5tmBn+2P7O152s9kfe8CX8K75Erz37gOBO/z1qe/De61r8VpKXOKcC1xP38D7O9uD9z593zn3j2MYq4iMchb6mnQRkeOTn636LV5BD/2nGSbmNYy+0Q+gpB9mVoQ3jXHUvlb+2r7dQLyfWRzs8Q5vimdhmIcmInLcUmZPRKQf/hTBzwK/VqAnI8HMPoi37uq5kR6LiIhEl7iBdxEROT6Z1+h6HV4vqo8OsLtI2JnZC3h94z4y1DVgIiJy/NI0ThERERERkTFI0zhFRERERETGIAV7IiIiIiIiY1BUr9mbPHmyy83NHelhiIiIiIiIjIj169dXOucye7svqoO93Nxc1q3rqy2XiIiIiIjI2GZme/q6T9M4RURERERExiAFeyIiIiIiImOQgj0REREREZExKKrX7PWmvb2dffv20dLSMtJDEQm7xMREpk+fTnx8/EgPRURERERGuTEX7O3bt4/k5GRyc3Mxs5EejkjYOOeoqqpi3759zJ49e6SHIyIiIiKj3JibxtnS0kJGRoYCPRlzzIyMjAxlrUVEREQkJGMu2AMU6MmYpWtbREREREI1JoO9kVZWVsaVV15JXl4ey5Yt46KLLmLHjh3s37+fyy67DICCggL+/ve/D+q8DzzwADExMbz99tvd2xYuXEhxcXE4hz8qlJeXc/HFF7N48WJOPvlkLrroogGPyc3NpbKyMqzj+PrXv84zzzzT7z5//vOf2bJly6COERERERGJNAV7Yeac4wMf+ADnnnsuRUVFrF+/nu985zuUl5eTk5PD448/Dgwt2AOYPn063/72t8M97BHX0dFxxO2vf/3rnH/++bz11lts2bKFu+66a9jH1NnZyZ133sm73vWufvfrGeyFcoyIiIiISKQp2Auz559/nvj4eG666abubYsXL+ass86iuLiYhQsX0tbWxte//nUeffRRlixZwqOPPsoJJ5xARUUFAF1dXcydO7f7drCLL76YzZs3s3379qPue/rpp1m1ahWnnHIKH/rQh2hoaGDt2rVceumlADzxxBMkJSXR1tZGS0sLc+bMAeDuu+/m5JNPZtGiRVx55ZUA3HHHHXzkIx9h1apVnHDCCfzqV78CoKGhgXe+852ccsop5Ofn88QTTwBQXFzM/PnzufrqqznppJO47LLLaGpqAmD9+vWcc845LFu2jAsuuIADBw4AcO655/K5z32O5cuX85Of/OSI53LgwAGmT5/efXvRokUAvPDCC1x88cXd2z/1qU/xwAMPdN/+3ve+R35+PqeddhqFhYUAPPbYYyxcuJDFixdz9tlnA14g98UvfpGFCxeyaNEifvrTnwJedvDLX/4yp5xyCo899hjXX399d4Cem5vLl770pSPOv3r1ap588kluvfVWlixZQlFR0RHHPPvssyxdupT8/HxuuOEGWltbu8/1jW98o/t13LZtGwAvvvgiS5YsYcmSJSxdupT6+vqj3mcRERERgdaOTp7fdpCuLjfSQxm1FOyF2aZNm1i2bFm/+yQkJHDnnXdyxRVXUFBQwBVXXME111zD7373OwCeeeYZFi9eTGZm5lHHxsTE8KUvfYn/+q//OmJ7ZWUl3/rWt3jmmWfYsGEDy5cv50c/+hFLly6loKAAgJdffpmFCxeydu1a3njjDVasWAHAXXfdxZtvvsnbb7/NL3/5y+5zvv322zz33HO89tpr3Hnnnezfv5/ExET+9Kc/sWHDBp5//nluueUWnPP+wLZv384nP/lJtm7dSkpKCj//+c9pb2/n05/+NI8//jjr16/nhhtu4Ktf/Wr3Y7S1tbFu3TpuueWWI57PzTffzMc+9jHOO+88vv3tb7N///6QXv/U1FQ2btzIpz71KT73uc8BXqbtn//8J2+99RZPPvkkAPfeey/FxcUUFBTw9ttvc/XVV3efIyMjgw0bNnQHvv2d//TTT+d973sf3//+9ykoKCAvL69735aWFq6//noeffRRNm7cSEdHB7/4xS+67588eTIbNmzgE5/4BD/4wQ8A+MEPfsA999xDQUEBL7/8MklJSSE9bxEREZHjzdOby/noA2u5f3XxSA9l1BpzrReCffMvm9myvy6s5zw5J4Vv/NuCsJ4T4IYbbuCSSy7hc5/7HL/5zW/46Ec/2ue+H/7wh/n2t7/N7t27u7e9/vrrbNmyhTPOOAPwgqhVq1YRFxdHXl4eW7duZc2aNXzhC1/gpZdeorOzk7POOgvwsmZXX30173//+3n/+9/ffc5LLrmEpKQkkpKSOO+881izZg3vfe97+cpXvsJLL71ETEwMpaWllJeXAzBjxozux7/mmmu4++67ec973sOmTZs4//zzAS+jlp2d3f0YV1xxRa/P8YILLmDXrl384x//4KmnnmLp0qVs2rRpwNfxqquu6v7385//PABnnHEG119/PZdffnl3lvOZZ57hpptuIi7O+xNIT08fcEx9nb8v27dvZ/bs2Zx44okAXHfdddxzzz3dQWhgLMuWLeOPf/xj91i/8IUvcPXVV3PppZcekd0UERERkcOKKxsB+O4/tnHm3MnMy0oe4RGNPsrshdmCBQtYv379oI+bMWMGU6dO5bnnnmPNmjVceOGFfe4bFxfHLbfcwne/+93ubc45zj//fAoKCigoKGDLli3cd999AJx99tk89dRTxMfH8653vYtXXnmFV155pTvY+9vf/sbNN9/Mhg0bOPXUU7vXz/Ws/Ghm/O53v6OiooL169dTUFDA1KlTu1sB9La/c44FCxZ0j2vjxo08/fTT3ftMmDChz+eZnp7Ohz/8YR5++GFOPfVUXnrpJeLi4ujq6urep2cbguAxBH7/5S9/ybe+9S1KSkpYtmwZVVVVfT7mQGPq7fxDNW7cOABiY2O7X/PbbruNX//61zQ3N3PGGWd0T+8UERERkSPtq2kmJTGOlMQ4PvdoAa0dnSM9pFFnTGf2IpGBG8g73vEOvvKVr3Dvvffy8Y9/HPCmQ9bW1jJjxozu/ZKTk49aj3XjjTdyzTXX8JGPfITY2Nh+H+f666/ne9/7Xvc5Vq5cyc0330xhYSFz586lsbGR0tJSTjzxRM466yyuvfZarr32WjIzM6mqqqK8vJyFCxfS1dVFSUkJ5513HmeeeSZ/+MMfaGhoALw1frfffjuNjY288MIL3HXXXTz22GNMmTKF+Ph4nn/+efbs2dM9pr179/Laa6+xatUqHnnkEc4880zmzZtHRUVF9/b29nZ27NjBggX9vzfPPfccK1euZPz48dTX11NUVMTMmTPJyspiy5YttLa20tzczLPPPsuZZ57Zfdyjjz7KbbfdxqOPPsqqVasAKCoqYsWKFaxYsYKnnnqKkpISzj//fP7nf/6H8847j7i4OKqrq4/I7vWlt/P39l4CzJs3j+Li4u735OGHH+acc87p9/xFRUXk5+eTn5/P2rVr2bZtG/Pnzx9wXCIiIiLHm5KaJvKmTOTmc+dy40Pr+PG/dnLbhfrcFEyZvTAzM/70pz/xzDPPkJeXx4IFC7j99tvJyso6Yr/zzjuPLVu2dBdoAXjf+95HQ0NDv1M4AxISEvjMZz7DwYMHAcjMzOSBBx7gqquuYtGiRaxatao7K7RixQrKy8u7i5MsWrSI/Px8zIzOzk6uueYa8vPzWbp0KZ/5zGdIS0vr3u+8885j5cqVfO1rXyMnJ4err76adevWkZ+fz0MPPXREIDJv3jzuueceTjrpJGpqavjEJz5BQkICjz/+OF/+8pdZvHgxS5YsYfXq1QM+v/Xr17N8+fLu53LjjTdy6qmnMmPGDC6//HIWLlzI5ZdfztKlS484rqamhkWLFvGTn/yEH//4xwDceuut5Ofns3DhQk4//XQWL17MjTfeyMyZM1m0aBGLFy/mkUceGXBMfZ3/yiuv5Pvf/z5Lly6lqKioe9/ExETuv/9+PvShD5Gfn09MTMwRhXt689///d/dRWPi4+P7zfCKiIiIHM9KapqYMWk87zp5KledNoP/eamINburR3pYo4oFimtEo+XLl7t169YdsW3r1q2cdNJJIzSiY7Nu3To+//nP8/LLL4/0ULjjjjuYOHEiX/ziF0Pav7i4mIsvvjikdXXRKjc3l3Xr1jF58uQRHUc0X+MiIiIi4dDR2cW8r/2Dm86Zw60XzKextYOL7n6Zzi7HU589i+TE+JEe4rAxs/XOueW93afM3ihx11138cEPfpDvfOc7Iz0UEREREZFR7UBtC51djumTxgMwYVwcP7p8CfsPNfPNv2wZ4OjjR0SDPTNLM7PHzWybmW01s1Vmlm5m/zKznf6/k/x9zczuNrNCM3vbzE6J5NhGm9tuu409e/Ycsf5sJN1xxx0hZ/XAy3qN5aweeNnLkc7qiYiIiIg3hRNghh/sASybNYmbz5vL4+v38fy2gyM1tFEl0pm9nwD/cM7NBxYDW4HbgGedcycAz/q3AS4ETvB/Pg784ujTiYiIiIjI8W5fTTMAM9KP7En8mXeewPiEWF7eWTkSwxp1IhbsmVkqcDZwH4Bzrs05dwi4BHjQ3+1BINDY7RLgIed5HUgzs2yGIJrXIYr0R9e2iIiICOyrbiLGICftyGAvPjaG7NREDtQ2j9DIRpdIZvZmAxXA/Wb2ppn92swmAFOdcwf8fcqAqf7v04CSoOP3+dsGJTExkaqqKn0oljHHOUdVVRWJiYkjPRQRERGREVVS00x2ahLxsUeHMzlpSew/pGAPIttnLw44Bfi0c+4NM/sJh6dsAuCcc2Y2qKjMzD6ON82TmTNnHnX/9OnT2bdvHxUVFUMeuMholZiYyPTp00d6GCIiIiIjqqS6iemTknq9Lzs1kW1lR/dAPh5FMtjbB+xzzr3h334cL9grN7Ns59wBf5pmYPVkKTAj6Pjp/rYjOOfuBe4Fr/VCz/vj4+OZPXt2+J6FiIiIiIiMKiU1TZw5N7PX+3LSkqhsaKWto4uEuOO7+UDEnr1zrgwoMbN5/qZ3AluAJ4Hr/G3XAU/4vz8JXOtX5VwJ1AZN9xQREREREaG1o5PyutajirME5KQm4RyU17UM88hGn0hm9gA+DfzOzBKAXcBH8QLM/zWzjwF7gMv9ff8OXAQUAk3+viIiIiIiIt1KA5U4g9ouBMtO8+ob7D/UzIz03vc5XkQ02HPOFQC9dXN/Zy/7OuDmSI5HRERERESiW0l324U+gr1UL+N3oFaZveN7EquIiIiIiESVkmq/oXpf0zgDmT21X1CwJyIiIiIi0aOkpon4WGNqcu/tqMYnxJGaFM+BQ8rsKdgTEREREZGosa+mmWlpScTEWJ/7ZKcmqtceCvZERERERCSK7KtuGrDwSk5aEvu1Zk/BnoiIiIiIRI+Smmam91GJMyA7NZEDWrOnYE9ERERERKJDY2sH1Y1tfRZnCchJS+JQUzvNbZ3DNLLRScGeiIiIiIhEhZIavxLnAJk9VeT0KNgTEREREZGosK+6/x57Ad299o7zipwK9kREREREJCoEMnvTJw0wjdMP9pTZExERERERiQIl1c0kxceSMSGh3/2mpo4DlNlTsCciIiIiIlGhpKaJGelJmPXdYw9gXFwskyeOO+577SnYExERERGRqFBS3TRgcZaAnLRETeMc6QGIiIiIiIgMxDlHaU3zgMVZArxee5rGKSIiIiIiY8T2snoO1o29IKe2uZ361o4Bi7ME5KQlceBQM865CI9s9FKwJyIiIiIyRjyzpZyLf/oy33lq20gPJexK/LYL00OdxpmaRGNbJ3UtHZEc1qimYE9EREREZAz45+YyPvG79bR3OvZUNY70cMKuu6F6emiZvWy/sfqB43jdnoI9EREREZEo99TGA9z8uw0snJbKBQumjsm1aiXVgWAv1DV7aqyuYE9EREREJIr97e0DfOr3b7J4RhoP3XAa86YmU17XQntn10gPLaz21TSTmhRPSmJ8SPvn+Jm947kip4I9EREREVGDqNoAACAASURBVJEo9eRb+/nMH97klJlpPHjDaSQnxpOTlkSXg/IxVqQl0GMvVFOSE4mNMWX2REREREQkupRUN/G5P7zJslmTeOCjpzFxXBzgVaEE2D/GgpzB9NgDiI0xpiYf343VFeyJiIiIiEShzfvr6HLw1YtOYoIf6EFwsDd2ghznHPsG0WMvIDstSdM4RUREREQkuhT7FTdnZ044YvtYXKtWUd9Ka0dXyD32AnLSksZksZpQKdgTEREREYlCuysamTwx4aiCJeMT4kgbHz+mMnvdbRcGMY0TICc1kQO1LcdtY3UFeyIiIiIiUWh3VSO5GRN6vS8nNWlMrdnbV+MFroMp0AKQnZpIW0cXVY1tkRjWqKdgT0REREQkCu2ubGT25D6CvbSksZXZ83vsTR9kZi877fjutadgT0REREQkyjS0dlBR33rUer2AnLTEMRbsNZOZPI7E+NhBHZfjN1YfS+sXB0PBnoiIiIhIlCmu9Iuz9DWNMy2JupYO6lvah3NYEbPzYD1z+shi9ifbL1ZzYAwFvoOhYE9EREREJMrsquy9EmdAoP3CWKhE2dXl2F5Wz0nZKYM+NmNCAglxMewfA6/DUCjYExERERGJMoHM3qz03oO9aX5Gq3QMZLRKDzXT2NbJvKzkQR9rZmSnjq0prYOhYE9EREREJMrsrmwkJzWRpITe17Blp46dwiTbyuoBhhTsgbdubyxkOIdCwZ6IiIiISJTZXdlIbj9r2KYkjyM2xsZERmvbgToA5k0dWrCXnZaoNXsiIiIiIhId+mu7ABAXG0NWytiYvritvJ6Z6eOZMC5uSMfnpCZRXt9KZ9fx11hdwZ6IiIiISBSpaWyjtrm932APvPYLY2HN3vay+iFP4QQvs9fZ5ThYf/xN5VSwJyIiIiISRborcQ4Q7GWPgbVqLe2d7K5s5KRjCPa6e+2NgfWLg6VgT0REREQkiuwOMdjLSUviQG0zXVE8fbHwYAOdXY55WYNvuxDQ3WvvOGysrmBPRERERCSKFFc2EhtjzEgf3+9+09ISae90VDa0DtPIwm/7MVbihMOVScfC+sXBUrAnIiIiIhJFdlc2MmNSEvGx/X+UDzRWj+Z1e9vK6hgXF0NuRv+BbX9SEuOYkBCraZwiIiIiIjK6DVSJMyAQ7EVzkLOtrJ4Tpk4kboDAtj9m1j2l9XijYE9EREREJEo45yiu6r/HXkCgMEk0Bznby+qZN3Xo6/UCstOS2FcTva/DUEU02DOzYjPbaGYFZrbO35ZuZv8ys53+v5P87WZmd5tZoZm9bWanRHJsIiIiIjI2Pb/9IM9sKR/pYUTEwfpWmto6mRNCsJeS5E1fjNZpnNWNbRysb+Wk7KGv1wtYkJPC9rJ6mts6wzCy6DEcmb3znHNLnHPL/du3Ac86504AnvVvA1wInOD/fBz4xTCMTURERETGkA17a/h/D63ne//cNtJDiYhdFV4lzlAye4Hpi9FamGRbWR1wbMVZAk7LTaejy/FmSc0xnyuajMQ0zkuAB/3fHwTeH7T9Ied5HUgzs+wRGJ+IiIiIRKGD9S184rfraevs4kAUr1PrT6htFwK8YC86X4twVOIMOGXWJMxg7W4Fe+HkgKfNbL2ZfdzfNtU5d8D/vQyY6v8+DSgJOnafv01EREREpF9tHV3c/LsN1Da384Gl06hv7aC+pX2khxV2xVWNJMTFdK/HG0hOWmLUrtnbdqCejAkJZE4cd8znSk2KZ35WCmuLq8MwsugR6WDvTOfcKXhTNG82s7OD73TOObyAMGRm9nEzW2dm6yoqKsI4VBERERGJVt/62xbWFtfwvcsWc+68TADK66Izo9WfXRWN5GaMJybGQto/JzWJyoY2Wtqjb63atvJ65mUlYxbacx3IqbmT2LC3ho7OrrCcLxpENNhzzpX6/x4E/gScBpQHpmf6/x70dy8FZgQdPt3f1vOc9zrnljvnlmdmZkZy+CIiIiISBf53XQkPvbaHj589h/ctzglqoj32gr3iqtDaLgQE2i8cqI2u16Kry7GjrD4sUzgDTs1Np6mtky0H6sJ2ztEuYsGemU0ws+TA78C7gU3Ak8B1/m7XAU/4vz8JXOtX5VwJ1AZN9xQREREROcpbJYf4jz9v4oy5GXzpgnkAZKcmAlAWZQHOQDq7HHurmkIqzhJwuNdedE3l3FvdRHN7JydlHXvbhYDTZqcDsGb38TOVM5KZvanAK2b2FrAG+Jtz7h/AXcD5ZrYTeJd/G+DvwC6gEPgV8MkIjk1EREREolxHZxef/N0GMieO46dXndLdeHtqihfsRVs2ayD7DzXT1tkVUtuFgJy0xO5jo8m2MBZnCZiaksjM9PHH1bq9uEid2Dm3C1jcy/Yq4J29bHfAzZEaj4iIiIiMLTsPNlB6qJkfX7GY9AkJ3dsT4mKYPHEcZXXRFeAMZJdfiTM3I/RgLys1EOxFV+C7vaweMzhxaviCPYDluZN4cXsFzrmwrQUczUai9YKIiIiIyDHbWFoLwKLpaUfdl5OWGHUBzkCKA20XMkMP9sbFxZKZPC4KM3t15GZMICkhNqznPS03narGtu7AeaxTsCciIiIiUWlTaS0Tx8Uxu5dMV1ZK4phbs7e7spGJ4+IG3YogJy2J/VHWfmF7WT3zwpzVAzjVX7e39jhZt6dgT0RERESi0sbSWk7OSem1DUF2avT2l+vLrspGciePH/T0w5zUxKjK7DW3dVJc1RjW9XoBcyZPYPLEBNYcJ+v2FOyJiIiISNTp6Oxi64E68qel9np/VmoSdS0dNLZ2DPPIIqe4spHZkycO+rictCT2H2rBK5Ex+u08WE+Xg5Oywx/smRnLZ6UfN0VaFOyJiIiISNQprGigpb2rz2AvUIVyrFTkbOvoYl9NE7Mzxg/62Jy0JJrbOznU1B6BkYXf4Uqc4Wu7EGx57iRKqpvH3DTf3ijYExEREZGos3GfV5xlYV+ZvZSx1Wtvb3UTXW5wxVkCpvmBb2mUTOXcXlZPYnwMM9MHH9iGItBv73jI7inYExEREZGos6m0lgkJsX32nMtO9ZqJj5V1e7uH0HYh4PBrER2B77ayOuZNTSa2l7WY4XBydgoTEmIV7ImIiIiIjEYbS2tZkJPaa3EWgKmpXsXKaAlwBrK7sgGA2YNoqB6Qk+YFe9FQpKWmsY3N++siUpwlIC42hlNmTWLNcVCRU8GeiIiIiESVjs4uthyo63MKJ3j95SZPTBgzwd6m0jqyUhJJG58w8M49ZExIICEuZtQHe1UNrVz1q9dpauvkQ8tnRPSxTs1NZ3t5PbXN0bGOcagU7ImIiIiMEe2dXewor+eJglJe3FEx0sOJmKKKRq84y/T+C3hkpSZSNkamcRaUHGLJjKObx4ciJsbISU0c1Wv2Khta+fCv3mB3ZSP3XbecU3PTI/p4y3Mn4Rys3zO2s3txIz0AERERERm6f2w6wNOby9laVk/RwQbaOrsAGBcXw6ZvXkB87Nj7bn9jqVecpa9KnAFZKUnsq2kajiFFVFVDK3urm/jwiplDPkd2atKozexV1Lfy4V+9TklNE7+5/lTOmDs54o+5dMYk4mONtcU1vGP+1Ig/3khRsCciIiISpdo6uvj8o2+RGB/D4hlpnHNiJidlJ1NS3cQPnt5BcWUjJ0yN3NqnkbKptJbxCbED9pzLSUscE0U43tp3CGDImT3w1u2tLqoM15DC5mBdC1f96nX2H2rh/utPY1VexrA8blJCLAunpbJ2jK/bU7AnIiIiEqXe2neI5vZOfnzFEt6zMKt7+5b9dfzg6R1sK6sfk8GeV5wlZcBqjVmpidQ2t9PU1sH4hOj92FtQUkuMDZzJ7M+sjPH88c0WGlo7mDhudLwWh5rauPLe1ymra+HBG07rbokwXE7LTef+V4tpae8kMT52WB97uIy9vL6IiIjIceLVwkrMYNWcI7MheVMmEBtjbPebU48lnV2OLfv7L84SkJ06NnrtFZQc4sSpyUw4hiBtQU4KznlfBIwWf337ALsqG/n1dcuHPdADWDozjTZ/netYNWCwZ2bfM7MUM4s3s2fNrMLMrhmOwYmIiIhI31YXVbEwJ5XU8fFHbB8XF0te5gS2lY2eD/bhUlTRQHN7Z0hZrqyU6Oov1xvnHG+VHGLpzKFP4YTDWcHAesfR4NXCSqalJR31ZcVwmTvFmwZcVNEwIo8/HELJ7L3bOVcHXAwUA3OBWyM5KBERERHpX1NbB2/ureH0PtY4zctKYdsYzOxt3BdacRbw1uxBdAd7uysbqW1uP6b1egBTUhKZkjyOTaMk2OvscqwuquLMuZMxi0zz9IHMTPcy4EUHG0fk8YdDKMFeIF/8XuAx59zouEJEREREjmPrimto73Sc3kflwvlZyeyraaa+ZWz1EdvoF2eZk9l/cRaAqSmBaZyjswplKApKAsVZJh3zufKnpY6azN7m/bXUNrdzxgmRr7zZl4S4GGaljz/uM3t/NbNtwDLgWTPLBKL36xERERGRMeDVokriY41Tc3sPAuZneYVZxtp6pE2ltZycPXBxFoDE+FjSJ0R3Y/WCkkNMSIjtnnJ4LBZOS6WoooHG1o4wjOzYvFLoVQbtKzM9XOZkTjy+gz3n3G3A6cBy51w70AhcEumBiYiIiEjfXiuqYumMSX1WmZznB3tjaSpnZ5djc4jFWQKyUhKjPtjLn54aUnA7kPxpqV6RlgMjv5bz1cJKTspOYfLEcSM6jrlTJrK7spEOvz/lWBNqNc75wBVmdi1wGfDuyA1JRERERPpT29TOptLafnuSTUtLInlc3JiqyLlrEMVZAnLSojfYa2nvZOuBurBM4QTIn+4Xadk3slM5W9o7WVtcw5lzRzarB5CXOYH2TkdJTfRO9e3PgPVbzexhIA8oADr9zQ54KILjEhEREZE+vL67ii7X/xQ4M2NeVjLbDoydYC+w3iwQtIQiKzWR9XtqIjWkiNq8v472TnfMxVkCpqYkkjkKirSsK66hraOLM/pYbzqc8gIVOQ82MHvyhBEeTfiF0qxjOXCyc85FejAiIiIiMrDXiqpIjI9h6cz+Mz7zspL5y1v7cc6NWMXDcNpYWktSfCx5IRRnCchOTaKmqZ3mtk6SEqKrcXagOMuxtl0INhqKtLxS6K03HYneej0FrqWiigbexdQRHk34hTKNcxOQFemBiIiIiEhoXi2s5NTcdBLi+v8oNz8rmbqWDsrqonMaY0+bSms5OSe04iwB3Y3Vo/A1eKvkENmpid1VRcMh3y/S0tQ2ckVaXi2s5JSZfa83HU6pSfFkJo8bs0VaQgn2JgNbzOyfZvZk4CfSAxMRERGRox2sb2HnwYaQpsDNy0oBGBNTObuLs+SkDOq4rNRAr73oW5NVUHIobFM4A/KnpdLlYMv+kSnSUtPYxqb9tZw5CqZwBuRlTqDw4NgM9kIJp++I9CBEREREJDSvFVUBoZWsD67Ied78KREdV6Ttrmygqa1zUJU4wZvGCVAWZUVaqhpa2VvdxNUrZob1vN1FWkprWZ47/NMoX9tVhXOMaH+9nvIyJ/LXtw+MmenOwUJpvfAisA1I9n+2+ttEREREZJitLqwiJTGOBTkDBz2pSfHkpCayvWzkS+0fq81+JmowxVnAa70ARF1Fzrf2BZqphzezFyjSMlLr9l7eWUnyuDgWDTJoj6S8zInUNrdT1dg20kMJuwGDPTO7HFgDfAi4HHjDzC6L9MBERERE5Gird1Wyck5GyOvW5mUlj4lee0UHG4iNsUFXTExKiGXS+Piom8ZZsPcQMTb44DYU+dNSR6wi56uFlazMyyAuNtQOcJEXXJGzL//51y3c/seNwzWksAnlVf4qcKpz7jrn3LXAacDXIjssEREREemppLqJkurmkKZwBszPTqGoooH2KG8aXVTRyIxJSYyLG3xFzazUpKibxvlmySFOnJockSImC6elUnhw+Iu07K1qYm9106harwdeY3XwrrHeOOf4+8YD1DW3D+ewwiKUYC/GOXcw6HZViMeJiIiISBitLqoEGFR/svlZybR3Onb18UE2WhRVNAyq5UKw7NToaqze1eV4q+RQWFsuBAsUadl6YHin9746hOt3OGSnJJIUH9tnkZZdlY0cqG0ZdeMORShB2z/8SpzXm9n1wN+Av0d2WCIiIiLS0+qiKiZPHNediQjF4SIt0btur7PLsbuykTmZQ2t6nRVlwd7uqkbqWjrCvl4vIN9fL7dx3/BO5XylsJKslETyhvg+RkpMjDEnc0Kf7RdeLfSC1NGWkQxFKAVabgXuBRb5P/c6574c6YGJiIiIyGHOOVYXVXF6XsagKgbOmTyR+FiL6nV7+w8109rRNeTMXk5qItWNbbS0d4Z5ZJFRsDdQnGVSRM4/NWUckyeOY2Pp8H0B0NXlWF1YyRlzJ4/Kipd5mRP7DPZe2VnJjPQkZmaMH+ZRHbuQJgE75/4P+L8Ij0VERERE+lBU0UBFfeug1usBJMTFkJc5ke1RHOwV+h/C8waR0QyW5bdfKK9rYVbG6Moq9aag5BATEmIHlcEdDDMjf1rKsBZp2XKgjpqmds48YXDX73DJy5zIX97eT3NbJ0kJh9eFdnY5XttVxXvzs0dwdEPXZ2bPzF7x/603s7qgn3ozi955ACIiIiJRaIvfGH3R9MFP7ZuXlRzVwV5gveGcQVbiDMj2G6vvPzT6p3I653hjdxWLpqeFXHF1KPKnpbLzYD3NbcOT7QxMhTwjb3ROhZw7ZSLOwa7KI7N7G0trqW/piMr1etBPsOecO9P/N9k5lxL0k+ycSxm+IYqIiIhIYXk9McaQ1q3Nz0qh9FAztVFYTRC8rGba+HjSJyQM6fhAsFdWN/rbL7xaWMWO8gYuWZIT0cdZ6Bdp2XIMRVoqG1r52XM7aWgduKrn3zYe4OTsFKb4fQ9Hm7wp3t9Vz4qcgSB1sBn10SKUPnsPh7JNRERERCJnR3kDszImkBg/+NYD8/0iLTvKozO7V3TQq8Q51LVeWanR01j9ly8WMSV5HB84ZVpEHyfQv2+oUznrWtq59r41/ODpHTz82p5+991UWsvb+2q54tQZQ3qs4ZCbMQGzo3vtvbKzkpOzU8iYOG6ERnZsQqnGuSD4hpnFAcsiMxwRERGRwXmioJSfv1AY9X3kBrLjYD0nDHEN1+GKnNEZ7O2qbBzyFE6A8QlxpCbFj/peexv31fJKYSU3nDl7SP0EByMrJZHJExPYOIRgr7mtkxsfWMeO8npmT57Aw68V09HP39/v1+xlXFwM718a2QD2WCTGxzJj0vgjirQ0t3Wyfk8NZ54QnVM4of81e7ebWT2wKGitXj1QDjwxbCMUERER6YNzjv/6+1a+94/tXPaL1RRXRncvub60dnSyp6qJE6cmD+n47NREUhLj2DbMfdXCoba5nYr61iEXZwnITk0c9Wv2fvlSEcnj4vjwipkRfywzY+G01EFn9to7u7j5kQ2s3VPNj69Ywu0Xzmd/bQv/2lLe6/6NrR08UbCfixflkJoUH46hR8zcKROPmMa5triats6uqF2vB/2v2fuOcy4Z+H7QWr1k51yGc+72YRyjiIiISK+2HqinvK6VDyydRnFVExfd/TKPrSvBOTfSQwur3ZWNdHY5Tpg6tIDHzJiflRKVRVp2BSpxDrHtQkB2auKoXrO3p6qRpzYe4OqVs0hJHJ6gyCvS0hByS4quLscXH3uL57Yd5FvvX8i/Lc7hnSdNZfqkJO5fXdzrMX99ez8NrR18eMXoncIZkJc5gV0VDXR2ef9/vFpYSUJsDKfmRqYFxnAIZRrnV8zsUjP7kZn90MzeP5gHMLNYM3vTzP7q355tZm+YWaGZPWpmCf72cf7tQv/+3EE/GxERETmuvLDjIAC3Xzifpz57FvnTUrn18bf59O/fjNpiJL3ZWe4FPCdMGVpmDw5X5Iy2QLi7EucxNuLOSk0a0WmcLe2dFJQc6vP+e1/aRVxMDDeckTtsY1o4LZXOLhdSkRbnHHf8ZTNPFOznS++Zx9UrZgEQG2Ncu2oWa3ZXs2X/0ed5ZE0JJ06dyCkzR3/AlJc5kdaOLvYf8r4UeKWwklNmpTE+IaRudaNSKMHePcBNwEZgE3CTmd0ziMf4LLA16PZ3gR875+YCNcDH/O0fA2r87T/29xMRERHp0wvbKliQ41X4y0lL4pF/X8mtF8zjH5vKeN/PXomaJtoD2XkMlTgD5mcnU9/aQemh0Zvd6k1RRQNxMcbM9GNraJ2dmkhlQxutHSNzTdz3ym7ef8+r3PXUtqMC7or6Vh5bv48PLps2rNUql8702nisK64ecN/fvr6Hh17bw/87ew6fOCfviPuuWD6TpPhYHuyR3du8v5a3Sg5x1WkzR2Uj9Z4CU4ULKxqobmxj8/46zoziKZwQWrD3DuAC59z9zrn7gYv8bQMys+nAe4Ff+7fNP/Zxf5cHgUCm8BL/Nv7977RouCpERERkRNQ2t7N+bw3nzsvs3hYbY9x83lx+ePli9lQ19ZtJiSbHUokzYH6W1zlr64HomspZVNHArIzxxMeG8rG1bzPSvcbquypGZl3nptJaYsyrtnnL/751REGhB1bvpr2zi38/a86wjmlKciJzMifwxq6Bg70nCvazICeF2y6cf1Tgljo+ng+cMo0/F5RS3djWvf0Pa0oYFxfDB0ZxYZZgganCRQcbWF3k9wU8DoK9QiB4legMf1so/hv4EhC4mjOAQ865QDOOfUDg3Z8GlAD499f6+4uIiIgc5dXCSjq7HOfOm3LUfWef4AWA6/fUDPewImLnMVTiDDgpO5kY87It0WRXRSNzjnG9HsDyWekArNk9cGATCdvL6jn/5Knccv6J/PHNUj724DoaWztoaO3g4df28J4FWWF5noO1YnYGa3ZXd69T601jawcFJYc4+8TMPjN015+eS2tHF39YuxeAprYO/vxmKe/NzyZt/ND6Iw639AkJpE9IoKiigVcLK0lOjCN/WupID+uYhBLsJQNbzewFM3se2AKkmNmTZvZkXweZ2cXAQefc+jCNNXDej5vZOjNbV1FREc5Ti4iISBR5YftBUhLjWDoj7aj7Jk1IIC9zAm/ujf5gr7Wjk+KqpiEXZwkYnxBHXuZENpVGT0XOjs4uiqsaj7k4C8D0SUlMS0vijd1VYRjZ4LS0d1Jc1ci8rBQ+/c4T+O4H83m1sJIr732dnz1XSF1LBzf1mBo5XFbOSae+tYOt/azbW1NcTUeX67ex+IlTkzljbgYPv7aHjs4u/vr2AepbO7hqGCqLhlNe5gSKDjbySmElq+ZkEHeMGeWRFspqw68P8dxnAO8zs4uARCAF+AmQZmZxfvZuOlDq71+KlzXc5/fySwWO+mt0zt0L3AuwfPny6FphLCIiImHhnOOF7RWcdWJmnx/GTpk5iWe2luOci4r1Qn0JVOIcatuFYAunpfJa0fAHO0NVUtNMe6cj7xiLs4BXkXTF7HRe3FEx7NdE4cEGuhzM89/DK06dyeSJ47j5kQ1sLK1l1ZwMFvfypcVwWDnHC+Be31XFwj6yWKv9qpSB7Ghfrj99Nv/+0Dqe3lLO79fsZe6UiSyfNfoLswTLy5zIH98spa1j+KfVRsKAoapz7sX+fvo57nbn3HTnXC5wJfCcc+5q4HngMn+36zjcs+9J/zb+/c+5aCsXJSIiIsNiy4E6Dta3cu6JmX3us2zWJGqa2tkd5b33wlGJM2BBTgpldS1U1Lce87mGQ6DtQrimN66ck0FVYxuFBxsG3jmMAi0vAs3tAd550lR+/+8rWTQ9lS9eMG9YxxNsakoiuRnjeb2fdXuri6pYOjONpIT+14y+Y/4Upk9K4q6ntvHm3ugpzBIsL3MibR3eCrRoX68H/TdVf8X/t95vqh74qTezY8n/fxn4gpkV4q3Ju8/ffh+Q4W//AnDbMTyGiIiIjGEvbPeWcpwzr/9gD2DD3ugu0hKOSpwBgcxNtKzbK+rusXfszx1gxRwvM/X6ruHNbm4vrychLobcjCMrii6dOYknP3Vm97U6UlbOyWBtcTVdvazbq2lsY8uBupACn9gY47pVueytbiIhLoZLo6QwS7C5/trY7NRE5kwOz3U3kvprqn6m/2+y31Q98JPsnEsZzIM4515wzl3s/77LOXeac26uc+5DzrlWf3uLf3uuf/+uY3liIiIiMna9uN1vuZDcd5n6vMyJpCTGRX2Rlp0Hj70SZ8DJOd5HuM299EMbjYoONpIxISFsBT5mpo8nOzWR14e5SMu2snrmZk4cteu/VsxJp7a5na1lR18Xr++qwjn6Xa8X7PLlM5iQEMvF+dlMmhAdhVmCBdaHnjF3ctRlJXvT7xXnN0TfNlyDERERERlIby0XehMTYyydOYkNUR7s7Sg/9kqcASmJ8eRmjGdTafRk9sJRnCUgsG7vjV1Vw9pcfkdZPfOzjn0abqSsmO0Fcr21YFhdVMX4hNiQ1xSmjo/nb585i29esiCsYxwu0yclcdVpM7hm5ayRHkpY9BvsOec6ge1mFl1ldERERGTMCrRcOK+Xlgs9LZs1iR0H66lraR+GkYVfuCpxBlswLZVNUTKNc1dlI3lTwjuVbuWcDCob2igapn57tU3tlNW1cOIoDvZy0pKYmT6+1+mtrxZVctrs9EH1OcydPIHkxPhwDnHYxMQY37l0EUtGqGBOuIXyrk0CNpvZs4F2C/21XBARERGJpOe3eS0XQvkwtmzWJJyDgihdt1dc2RS2SpwBC3NSKaluprZpdAfA1Y1tVDe2MWdyeHvPrfCrTw5XC4bt5UcXZxmNVs5JZ02PdXtltS3sqmgMeQqnjD6hBHtfAy4G7gR+GPQjIiIiMqycc7y4o/+WC8EWz0gjxqK3ufoOP1AIRyXOgIXTAuv2Rnd2L1CJM9yZvdyM8UxNGddv9clw2u6vg5sXxoA9ElbMzuBQU3t3cArw2q5KAE7Pi/6qlMerUPrs7QUOOOdaAMwsCZga0VGJx5xhJwAAIABJREFUiIiI9CKUlgvBJo6LY15WChuitLl6OCtxBizI8Spybiyt5fRRXFp+lz/NMpxr9iCwbi/DLzwS+X5728rqSU6MIzu172JCo0GgUukbu6o4Kdv7QuDVwirSxsdzcvagajPKKBJKZu8xoCvodqe/TURERGRYhdJyoadls9Io2HuIzl7Kyo924azEGZA+IYFpaUlsGuUVOYsqGkiIjWH6pPED7zxIK+akc7C+dVh6MO4o94qzjPbKjtMnjWf6pCTe8CuVOud4raiKVXMyiIkZ3WOXvoUS7MU559oCN/zfo6+OqoiIiES9F7dXsHBa/y0Xejpl5iTqWzvYebB+4J1HmR3l9d19v8JpQU4Km0d5Rc6iigZyJ48nNgKBxsrudXuRncrpnGNbWX1Y11xG0orZGbyxuxrnHHuqmig91Kz1elEulGCvwszeF7hhZpcAlZEbkoiIiMjRGlo7WL+3hnNCnMIZEGhYHW3r9to6uiiuauLEMFbiDMiflsquykbqR3GV0l0VjWGfwhkwZ/IEJk8cF/Hm6mV1LdS3dIzqtgvBVs5Jp7qxjZ0HG1hd5L02q7ReL6qFEuzdBHzFzPaaWQnwZeD/RXZYIiIiIkfaXFpLZ5dj+az0QR03M308kycmsGFPdFXk3F3ZGPZKnAELp3nr9rYeGJ3ZzraOLvZUN4V1rWIwM2PlnHTe2FUd0X5728q81zdaMnuBjOfru6pYXVTJ1JRx5EXoPZDhMWCw55wrcs6tBE4GTnLOne6cK4z80EREREQO2+hPOwwEKqEy85urR1mRlkAlzohM4/Qrco7W5up7q71AN1KZPfBaMJTVtbCnqilij7HdD/bmZ0VHgZPpk5KYlpbEa0VVvFZUxel5k0f9WkPp34DBnpl91sxSgEbgv81sg5m9O/JDExERETlsU2ktWSmJZCaPG/Sxy2ZNYndlI1UNrREYWWTsPNhAjIW/GiXAlOREpiSPG7XN1YsiVIkz2KpA9ckI9tvbUVZPVkoiqeOjo8G4V6k0nWe2llPV2Kb1emNAKNM4b3DO1QHvBjKAjwB3RXRUIiIiErKaxjYKo7D4yGBtLK0ddFYvILBu780oaq6+s7w+7JU4gy2clsrm0tFZkbPI77EXqWmc4AWSkycmRLTf3rayek6MkvV6ASvnZNDe6U1tXaVgL+qFEuwFcrcXAQ855zYHbRMREZER9p2ntvLeu1/pbkI9FjW0drCrspH8IQZ7+dNSiYsx1kfRVM6dBxsiMoUzYGFOCjsP1tPc1hmxxwjW3tnFSzsq6AqhBcbGfbVMSR5HcmLkMmKBfntv+P32wq2js4vCioaoKc4SEOi3NytjfETaXsjwCiXYW29mT+MFe/80s2SO7LsnIiIiI2jN7mpaO7q47f82hvRBOhpt2V+Hc5A/fWhrnxLjY1kwLTVqKnK2dXRRXNkYkUqcAQumpdLlYGvZ8GT3nizYz7W/WcNPn+u/9MM/N5fx1KYyLlmSE/ExrZiTzv7aFkqqm4d0vHOO1YWVNLV1HHVfcVUTbR1dUVOcJWBm+nhOmDKRd588daSHImEQSrD3MeA2+P/s3Xd81eX1wPHPk733JBsCBJIAGewlioKCimLds85WW1e1trX+WmurtrbWqnXUvbXugSB77xlCIDskIXvvdZ/fH0kQyE7uzc2l5/163RfJ/Y57EpHck+c85zBVa11P+4y9W0walRBCCCH6pbS2ieyyeiYHu7Mru5z3duaYOySTGGxzllMlhHpyKK+SlraR/zvrrNI6Wg2asX6mSxQ6v5fDNW9va0b75K5n16SyNqWo23PyKup56L8HiQ1y51eLxps8ppPdJwe5b29LeinXvraTR7883OXYj81ZLCvZU0rx7S/n8MiFE8wdijCC/nTjNADhwGNKqb8D87TWh0wdmBBCCCH61rlS9fulE5k3zpenvj9Kbrnpuguay+H8Kvzd7Ac0TP1MCWGeNLYYSCkwzz41rTX/3ZNLYVVjn+d2DoAfa8KVvVHuDng62XJ4GPbtaa3ZkVHGuVF+xAS5cd9HB7qUHbe0GfjFh/sxaHjh2jjsbUyzV/FUkb4ueDjZsjd7cCu+nauUn+/LZ1vG6WOojxXVYKVM003V1OxtrE0yzF4Mv/504/w37bP2koDDwJ1KqRdNHZgQQggh+rYvpwI7aytigtx58vJYFPCbz5NMOjvMHJLyqwa9X69TfJgHYL7h6ntyKnjo00Msfm4Tq5ILezwvt7yet7ZmY22lTNqNUilFTJD7sHTkPF5ez4mqRhaM9+Xl6xOwsVbc8e5eapt+LH98ZtUx9h+v5KnlsYR5D89sNysrRXyo56D2cu7MLGNXVjmPXBhFmLcTj35xmKbWH/c/HiusJtyEDXaE6I/+lHGeCyzSWr+ptX6T9r1755k2LCGEEEL0x96cCmKC3HCwtSbIw5HfXDSBLemlfLIn19yhGU1dUysZJbVDKuEECHR3ZJS7g9mSvYO57Z1AA9wcuPPdvfz2i6TTmqO0tBl4aUMG5z+7kZSCap68LNbkiUL0KHdSi2pOS1JMYUdme5nkzDHeBHs68eK18WSW1PKrTw6itWbd0SJe2ZTJddNDWTrJ9Hv1TpUQ5kl6cS2V9c0Duu6F9en4uNhx86xwHr80hszSOl7ZmHnyeGpRLeMtrIRTnH36k+ylA6GnfB4CpJkmHCGEEEL0V1NrG4fyq0gM9zr53LXTQpkx2osnvk3pV7mgJThS0NGcZYjJHkB8mKfZxi8k5VcR6O7A1/fM4c75o/lg53EufmELySeq2He8gouf38LTK48yb6wvax6cz5VTQ0weU0yQGy1tmrQi03Zy3Z5Rho+L/cmVylmRPvz2ogmsTC7kT9+m8OAnB4kKcOX3SyeaNI7uxIe2j+XYN4DVvf3HK9icVsrtc0fjYGvN/HG+LJ0UyAvr08kqraOhuY3ssjqLa84izj49JntKqW+UUl8DrkCKUmqDUmo9kNLxnBBCCCHM6HB+Nc2thpNvVqG9LO3p5ZNoMRj43RdnRzlnUl57maFRkr1QT/IrGyioGlz3xaFIymsvRbWzseI3F07gvVunU93QwrIXt7L8pW1U1rfwyg0JvHpjIoHujsMSU8yo9u/pYRM2adFasz2zjBmjvVDqx31gt86J4JLJo3hjaxZNrQZevC7eLCWPU0I8sLZSA1rxfXF9Oh5Otlw/I+zkc48tnYi9tRW///IwacU1aG15zVnE2ceml2PPDFsUQgghhBiwfR1vTjv3onUK83bmVxeM54nvUtieWcasMT7mCM9oDue3z1zzcxt8c5ZOncPV9+VUsmTS8CRUADWNLWSW1nF5fNDJ5+aM9WHlffN44tsjeDnbcd/543Cx7+2tmfGFejnh6mDDgdxKrp4W2vcFg5BVWkdRdVOXAd1Ktf9iQilYEhto0v2JvXG0syZ6lFu/k73kE1WsSSnmwfPH4XzKfy8/NwceWjyex75KxuqH9qRWyjiFufX4L4rWeuNwBiKEEEKIgdmbU0Gol1O3HSqvmx7GMz8cY+XhQotP9ozRnKXTxFFuONhase94BUsmBRrlnv3R2fEyNvj0xNzL2Y5/XDVl2OI4k5VV+2DxbRmDGz3QH9s79+uN9u5yzNHOmueujjPZa/dXfKgnH+0+TkubAVvr3nc5/Xt9Bq72Ntw4K7zLseumh/HZ3jw2pZZgb2M1bI1mhOhJf7px1iilqjsejUqpNqWUeXoWCyGEEAJoL43bk1NBYphnt8cd7dr3Ea1KLrToQev1zcZpztLJ1tqKSUEew96kJSm/fZ+gsZJWY5oT6c3x8nqOl5lmZMf2jDL83eyJ8Bm5iU9ieP/GcqQX17DicAE3zQrH3dG2y3FrK8WfL4vFSrWPzZDxBcLc+jNnz1Vr7aa1dgMcgeXAv00emRBCCCF6lFveQGltE/E9JHsAi2MCKKpu4mCeeRqSGMORE9UYjNScpVN8mCfJJ6pobDFtB8pTHcqrItjTES9nu2F7zf6aM7Z95XfrGXPi+nKisoGXNmRw6Ytb+WxvXrfnaK3ZkVnOzNHep+3XG2k6y3v7+iXAi+szcLS15qdzIno8p3MMyj0LIo0aoxCD0Z9unCfpdl8Ci0wUjxBCCCH6Ye/xcuDHN6ndOXe8PzZWipW9zHQb6ZI6GofEBhsv2UsI86SlTZu0KcmZkvKrmGTEr8GYxvi64O9mz5b0vpO9qoYWPtp1nKte2c6sp9bx9MqjZBbX8uT3KdQ3t3Y5P724ltLarvv1Rpr+jOXIKavjqwP5XD8jrM+k/aqpoSyOGb4yYSF60p8yzstPeVyhlHoKODt6OQshhBAWak92Ba72Nr22dnd3smXmGG9WHS602K6cSXlV+Lra42+E5iyd4kKHd7h6VX0LOWX1xAZ59H2yGSilmB3pw7b00l5Lfo+cqGbmk2t55PMkSmqaeOD8cWx86BzevGUqpbXNvLs9p8s1P+7XG/n7RhPCvXr9O/Hm1mxsrKy4rZdVPSFGmv6s7F18ymMRUANcasqghBBCCNG7vTkVTAn16HNP0KLoALLL6kk18Rw1UzFmc5ZOPi72hHs7DVuyd3J1cgTu1+s0J9KHivoWjvSyZ+39nTkYtOaLn89i7YPz+eV5YwnzdiYx3It543x5eWMGtU2nr+7tyCwjyMOREK/h63w6WAmhHhRUNXKisutYjrqmVj7bm8eSSYFG6QorxHDpz569W0553K61/rPWung4ghNCCCFEVzWNLRwrqum1hLPTBRP9UQpWWWApp7Gbs5wqPsyTfccrhmXF89AIbs7SaXZkx769Hko5m1sNfHuogAsmBhAX6tll/939C8dSUd/C29uyTz5nMLTv15t+xny9kSohzAvofsX3qwMnqGlq5foZphlPIYSp9DZU/Xal1NiOj5VS6g2lVJVS6pBSKn74QhRCCCHEqfYfr0Tr3vfrdfJzcyAh1JOVhy0v2TNFc5ZO8aGelNY2k1tu+uHqSXlVhHk74e7UtXvjSOHv5sBYPxe29jCCYcOxYqoaWrgsLqjb43Ghnpwb5cermzKpaWwBILW4hvK65m5HLoxEUYGuONpad0n2tNa8tyOHCYFuxIf2/f+cECNJbyt79wLZHR9fA0wGRgMPAM+ZNiwhhBBC9GRvTgVWCqaE9G8P2KLoAI4UVJNbbprW+qZiyvLHk90XOxrdmNKhPOOXoprC7EgfdmWV0dTatUvplwfy8Xa2O9m5szv3LxxHVUMLb27NBtpHLgAjvjlLJ1trK6aEdB3Lse94JUcKqrlhRphFrFAKcarekr1WrXVLx8dLgXe01mVa6zXAyB2UIoQQQpzl9h2vYHyAG64O/VspWhQdAFheKWdSfhU+Lvb4u9kb/d7j/F1xsbcx+b698rpm8isbRmwnzlPNifShscXAvpzTR3VUNbSwJqWYiyeP6nXgeGywO+dP9Oe1zZlUNbSwPaOMEC9Hgj2dTB260SSEeXKkoPq0zqLv7cjB1d6GS6eMMmNkQgxOb8meQSkVqJRyAM4D1pxybOTvshVCCCHOQm0Gzf7jlSSE9b+zY6i3ExMC3SyulPNwfhWxQW4mWU2xtlLEhXp0SWyM7cfVyZHZifNU00d7YW2luuzbW3m4gOZWA8t6KOE81X0Lx1Ld2MrrmzPZmVVuMSWcnRLCPGkzaA7mtv93K6tt4rtDBVweH4SzvY2ZoxNi4HpL9h4D9tBeyvm11joZQCk1H8g0fWhCCCGEONOxwhpqm1r7tV/vVIujA9h7vILiGsuYnlTf3Ep6sWmas3SKC/XkaGF1lw6SxpTUMdA+JsjNZK9hLK4OtkwJ8egyb++L/fmM9nFmcj9WJ6NHubM4OoB/b8igqqHFYko4O3Xuydt3vH3F979782huM3D9jDBzhiXEoPWY7GmtvwXCgAla69tPObQHuMrUgQkhhBCiq70db0ITOzoH9teiGH+0htVHikwRltEdzK3CoDFpspcQ5olBw8Fc063uHcqrYrSvc79Lbs1tdqQPh/IqqWpo38mTX9nAjsxylsUF9XuF9b7zx9LaMa/PEubrncrdyZaxfi7syS6nzaB5f2cOM0Z7MbaXeZZCjGS9jl7QWrdqrSvOeK5Oa22Zw3qEEEIIC7cvpwJfV3uCPQe2o2K8vyvh3k6sSjZfstfaZuDqV7dz57t7WHm4gMaWro1AUgqq+dV/D3LTG7twsrM2affDKSEeKNX+PTWVpPwqJllAc5ZOcyJ9MOj2+XgAXx84AcCyKX2XcHaKCnDj8vggYoLcCHC3vJl0CWGe7DteyYZjxeSWN3DDjHBzhyTEoEnxsRBCCGFB9uSUEx/qMeB9bEopFsUE8PrmLKoaWnB3HP6Vpv25lezILMfJzppVyUW4OtiwJDaQZXFBNLUaeG1zJpvTSnG0tebqaSH8dHYEvq7Gb87Syd2xfRWnc7V0MAqqGrj/4wOEeTnz1PLY0/67FNc0UlDVSGzwyN+v12lKiAdOdtZsTS/lgon+fLE/j4QwT0K9B9Zk5a/LJ2Ew/QhDk4gP8+Sj3bn8ZUUKvq72XBDtb+6QhBg0SfaEEEIIC3GisoHc8gZunhUxqOsXRQfwysZM1h8t7lezDWPbeKykvQHIr88lKb+KL/fn8/XBE3y0OxcAP1d7Hlo0nuumh+LhZDcsMSWEefLdoQIMBo2V1cAS6D3Z5dz13j4q65vZkVlOhK8zd80fc/L44Y7mLJbQibOTnY0V0yO82JJeypGCalKLavnTspgB38eml66dI11ix37YjJI6fnluZK8dSIUY6XpN9pRS7sBioPMnQj6wSmtt2tZVQgghhOhiZ1Z7ad30iIHt1+s0JdgDP1d7Vh8pMkuytymthLgQDzyd7Zg3zpd543x5ormVNSnFQHsTGTub4X1jHR/qyYe7csksrSXSr//7sj7YeZz/+/owQR6OvH/bXP61No2/rjzKxEA35o3zBdr36ykFEwNHfnOWU82O9GH9dyn8e0MGttaKpbGB5g5pWEX4OOPpZEt1YyvXTA81dzhCDEmP/6IqpW4E9gHnAE4djwXA3o5jQgghhBhGOzPLcXOwYcIgkwcrK8WC8X5sSi2hpc1g5Oh6V1bbRFJ+1clEqJOTnQ2XTB7FJZNHDXuiB+0le0C/5+01txr43RdJ/PaLJGaO8eGru+cwPsCVv14xibF+rvziw/0cL2sfXp+UV0Wkr4vFtezvHJz+3aECzhnvh6fz8KyyjhRKKa5MDOGGGWEEusu0MWHZevtX9XdAgtb6Z1rrJzoedwGJwKN93Vgp5aCU2qWUOqiUSlZK/bHj+Qil1E6lVLpS6mOllF3H8/Ydn6d3HA8f+pcnhBBCnD12ZJYxLaJ9FtpgnTvBj5qmVvZkm3aY+Jm2pJeiNcw/I9kzt9E+zng42fbr+9HcauD613fy/s7j3DV/DG/ePBV3p/a9j872NrxyQwJaa+54dw/1za0cyq8i1oJKODuN93fFx6U9wbvMDCvAI8FvLprAHy6JNncYQgxZb8meArrbWmvoONaXJuBcrfVkYAqwWCk1A3gaeFZrHQlUALd2nH8rUNHx/LMd5wkhhBACKKxqJLusnhlDHFI9J9IHO2sr1h8rNlJk/bMxtQRPJ1uTjlIYDKUUMyK82ZxWiqGPjiKbUkvYlVXOE8tieOTCqC5Jd7iPM89dE8exohrufHcvJTVNFtWJs5NSinnjfHF3tOXcKD9zhyOEGILekr0/A/uUUi8ppX7b8XiZ9tLOP/d1Y92uc0SDbcdDA+cCn3Y8/zawrOPjSzs+p+P4eWqgrcaEEEL8Tzpyor1df2GVZQwMH4wf9+sNLdlztrdh+mgv1qYM3wgGg0GzKbWUOWN9h7QqaSoXRPtTWN3IwbzeWxKsSi7E1cGGKxNDejxnwXg/fnXBeDantQ8mt6ROnKd6bOlEvrx7Ng621uYORQgxBL0NVX+b9pLNjbSv0jUBG4BErfVb/bm5UspaKXUAKAZWAxlApda6teOUPH5s/hIE5Ha8ditQBQztJ5oQQoj/Ca9vyeLTvXlc/MIW9uaUmzsck9iRWY6rvQ0TRw292ceC8X5klNSd3FtmaimF1ZTWNo24Es5O50X5Y2Olep1B2NpmYHVKEQsn+Pe5t/Dn54zhwpgAnOysLa45SycPJzsifJzNHYYQYoj6GqpeAaw/9XHmkPU+rm/TWk8BgoFpQNQQYgVAKXWHUmqPUmpPSUnJUG8nhBDCwhkMmo2pxUyL8MLJzpqrX93Bh7uOmzsso9uZVcbUIe7X69RZmrfu6PCs7m1KbV/lmtfR+GOkcXeyZeYYb1YlF6J196Wcu7LKqaxvYVF0QJ/3U0rx/DVxrHlgPo52sjImhDCf3rpxTlFK7aB9Ne9p4K/ARqXUDqVU/EBepGNUw3pgJuChlOpsSxVM+zgHOv4M6XhtG8AdKOvmXq9qrRO11om+viPzN4RCCCGGz6H8Kkprm7l2Wihf3z2HmWN8+M3nSTz6ZRLNrcPbcdJUiqsbySypY8bowY1cOFO4jzOjfZxZd2x4fmm6MbWYqABX/NwchuX1BmNRdABZpXWkFdd2e3xVciEOtlb9Xp20sbZilId0chRCmFdvK3tvAfdqrSdorc/XWi/UWkcB9wFv9nVjpZSvUsqj42NH4Hwghfak74qO024Cvur4+OuOz+k4vk739Os1IYQQosO6o8VYqfYuj+5Otrx581TunD+a93Yc5/rXdlJV32LuEIdsZ1Z7aepQ9+udakGUHzsyy6hvbu375CGoa2plb04F88eP7F/QXjDRH6Vg5eHCLscMBs2q5CLmj/OVlTohhEXpLdlz1lrvPPNJrfUOoD9F3IHAeqXUIWA3sFpr/S3wa+ABpVQ67XvyXu84/3XAu+P5B4BH+v9lCCGE+F+1/mgx8aGeJ2eBWVspfnPhBJ67egq7c8p5e3u2WeMzhh2ZZbjY2xBthP16nc6L8qO51cDW9C5FNEa1PaOMljbN/LEjO9nzc3MgPtSTVcldk72DeZUUVjf2q4RTCCFGkt6Sve+VUt8ppa5SSs3qeFyllPoOWNnXjbXWh7TWcVrrSVrrGK314x3PZ2qtp2mtI7XWP9FaN3U839jxeWTH8UzjfIlCCCHOVsXVjSTlV7Ggm/bwl04JIjHMkxVJBWaIzLh2ZpWTGO6JjbXxho4nhnvhYm9j8n17G1NLcLS1JiHc06SvYwyLowNIPlFNbvnpjWtWJRdhY6U4L8rfTJEJIcTg9NaN85fAC8AC4DcdjwXAi1rre4YnPCGEEKJnGzr2nPU0C+zCmECOFtaQWdL9PixLUFLTRHpx7ZDn653JzsaKuWN9WH+0pMemJMawKa2EWWO8sbcZ+eWPnSt3p67uaa1ZlVzIzDHeJweoCyGEpeirG+f3Wuu7tNYXdzzu0lqvGK7ghBBCiN6sPVpEoLsDUQGu3R5fHNP+5v37bvZhWYpdJ/frGac5y6kWRPlRWN3IkYLqId2nqbWNxpa2Ls9nl9aRU1bPvBE6cuFMod5OTAh0Oy3ZSyuuJau0Tko4hRAWqbdunO5KqaeUUilKqXKlVFnHx091Nl4RQgghzKWptY0taaUsiPJDqe7HEYzycCQu1IPvD1tuKefOrDKc7ayJCXI3+r3P6Wiasv5o8aCub2418M72bGY/tY7EJ9bwzvZs2gw/rhJuSmtfeR2p8/W6syjanz05FZTUNAHtDVuUam/gIoQQlqa3lb1PgApggdbaS2vtTXsZZ2XHMSGEEMJsdmdVUNfcxrnjuy/h7HRhTACH86uHbYC4se3ILCMh3AtbI+7X6+Tn6sCkYHfWDTDZMxg0Xx88wfnPbuSxr5IZ4+vClBAPHvsqmeUvbSOlY6VwU2oJoV5OhFvQcO7FMQFoDauPtO9lXHm4kPhQzxE9NkIIIXrS20+OcK3101rrk7UMWutCrfVTQJjpQxNCCCF6tu5oMXY2VsyK7H0v24UxgQAWubpXVttEalGt0ebrdefcKD/251ZSXtfcr/N3ZpZxyYtb+OWH+3G0tebNW6by0R0zePfWafzzqinkltez9PktPLkihW0ZZcwbNzIHqfdkvL8rYd5OrEwuJLe8niMF1SyWEk4hhIXqLdnLUUo9rJQ6WbeglPJXSv0ayDV9aEIIIUTP1h8rZtYYb5zsbHo9L8TLidggd1ZY4L69XSaYr3emc6P80Bo2HOt7da+2qZUb39hFRV0L/7hyMt/9ci4LxreX0SqlWBYXxNoH53NFfDCvbMqkvrmN+eN6X3kdaZRSLI4OYHtGKf/d0/52R/brCSEsVW/J3lW0z8Hb2LFnrxzYAHgBVw5DbEIIIUS3skrryCqt67EL55kujA3gYG4leRWWVcq5M6scR1trJgUbf79ep5hR7vi42PerlHNHRhlNrQb+dsUkLo8Pxtqq615JDyc7nr5iEh/fMYPb50Ywd6xlrewBXBAdQEub5uVNmUwIdCPU28ncIQkhxKD0NnqhQmv9a611VMeePS+t9YSO58qHM0ghhBADp7VmbUoRpbVN5g7F6DoTkwV97Nfr1FnKudLCVvd2ZJaRGO5pkv16naysFAvG+7IxtYSWNkOv525JL8XB1qpfM/Omj/bmd0sm4mA78kcunCkuxAM/V3uaWw1SwimEsGiD+umhlLrF2IEIIYQwnrLaJm57ew+3vr2HJ1ccNXc4Rrf+aDFj/VwI8erfikuEjzMTAt1G9AiG+uZWjpfVszenglXJhby7PZujhTUmGblwpvMm+FPT2Mru7N5/l7sprYQZoy1jZt5QWFmpk6WbneM7hBDCEvW+0aFnfwTeNGYgQgghjGNTagkP/vcgVQ0tRPq5sPZoES1tBpOuDg2n2qZWdmaV8dPZEQO67qKYAP6+OpXCqkYC3M3XWbGlzUBaUS1HC6s5WlhDSkE1xwprKK7pugJrZ2PFuVGmb/k/d6wPdtZWrE0pZtaY7ssu8ysbyCyp49ppoSaPZyS4e0EksUHujPN3MXcoQggxaD04bXNEAAAgAElEQVQme0qpQz0dAmTYjBBCjDBNrW38beUxXtuSxVg/F9756TRyyuq567297MoqZ3ak5e2d6s6WtBJa2jQL+rlfr9OFsYH8fXUqKw8XcPMAE0Vj0Vpzzas72JNTAYCdtRVj/V2YO9aXMX7O+LrY4+Nq3/6niz3eLnbDkqQ729swc4w3a1OKeHTJhG7nFm7pmJlnKQPShyrA3YErp4aYOwwhhBiS3lb2/IFFtM/aO5UCtpksIiGEEANWWtvETW/sIvlENTfODOO3F03AwdaacG9nHGytWHm48KxJ9tYdLcbVwYaEsL73jZ0q0s+Fcf4urDhcaLZk72hhDXtyKrh9bgRXTQ0h3NsZmxGy4rpwoj+///IwGSV1RPp1Xc3alFaKv5s9Y7s5JoQQYmTq7SfMt4CL1jrnjEc27V05hRBCjBD/WJ1KalEN/7kxkccvjTnZFMPRzppzxvnxw5FCDAZt5iiHrqm1jVXJRZwb5TeoFa8LYwLZnV1OcU2jCaLr2/dJBVgpuHP+GCL9XEdMogdwXsdK6ZqUoi7H2gyaremlzB3r2+2qnxBCiJGpt26ct2qtt/Rw7FrThSSEEGIgskrr+Hh3LtdOC+X8iV2r7BfF+FNU3cSBvEozRGdc61KKqWpoYXl88KCuvyg2EK1hVXLXhGY4rDhcyPQIb3xc7M3y+r0Z5eHIxEA31naT7CWfqKKyvsUixygIIcT/spHzK0UhhBCD8vcfjmFvY8U9547t9vi5Uf7YWClWJY/cTpT99enePALcHAZdkjrO34XRvs58e/CEkSPrW1pRDenFtVwUO3K7Oy6c4MfenAoq6ppPe35zWinAWVMKLIQQ/ysk2RNCCAt2OL+Kbw8VcOucCHxdu18tcne0ZeYYb1YdLkRryy3lLKlpYkNqCcvigrod5t0fSikumxLEzqxycsuHd8D6iqRClOJkS/+R6LwJ/hg0rD92+oD1zWklRI9yG5ErkkIIIXomyZ4QQliwv646hoeTLbfPG93reYtjAsguqye1qHaYIjO+rw7k02bQXJEQNKT7XBbffv3n+/KNEVa/fX+4gKlhXvi5mW/sQ19ig9zxdbU/bd9eXVMre3MqmCMlnEIIYXEk2RNCCAu1LaOUTakl3H1OJG4Otr2ee/5Ef5SClSN4qHhfPtuXz+RgdyL9XId0n2BPJ2aO9ubz/XnDttKZUVLL0cIaLhzBJZzQPkx84QQ/NqWW0txqAGBnVhktbZp5Y/83Ri4IIcTZRJI9IYSwQFpr/rryGIHuDtwwM6zP8/1cHUgI9WSlhe7bSz5RRUpBNcsTBteY5UzLE4LJKas/Oe/O1DqT7MUxIzvZAzgvyv/k4Hpo36/nYGs14FEXQgghzE+SPSGEsEA/HCniQG4l9y8cd3LMQl8WxwSQUlDN8bLh3atmDJ/tzcfWWnHxpFFGud+FMQE42Vnz6Z48o9yvLyuSCogP9SDQ3XFYXm8oZkf6YG9jxdqU9n17m9NKmRbh3e+/Z0IIIUYOSfaEEMLCtBk0f1t1jDG+zlwe3//9a52NQSytK2dLm4GvDuSzcII/ns52Rrmns70NF8YE8l1SAQ3NbUa5Z0+Ol9WTfKKai2IDTfo6xuJoZ82cSB/WpBRxorKB9OJa5sl+PSGEsEiS7AkhzmqtbQZa2wzmDsOoPt2bS3pxLQ8tGj+godwhXk5MDHSzuGRv47ESyuqaBz1bryfLE4KobWrlhyND/35oralvbu322PeHCwDLKOHsdN4Ef/IqGnh9SxYAc2W/nhBCWCRJ9oQQZ60TlQ0s+PsGHvk8ydyhGE1aUQ2Pf3OExDDPQbXwXxQdwN7jFRTXNJogOtP4bF8e3s52zB9v3IRjRoQ3QR6OfLp3aKWcBoPm/o8PkPCnNXx3qKDL8RWHC5kc7E6wp9OQXmc4nTfBD4C3t2Xj52rPOH8XM0ckhBBiMCTZE0Kclcpqm7jh9Z3kljfwzcET1DZ1v+piSaobW7jj3b042tnwwrXxKDXwWXOLYwLQGlYfKer75BGgsr6ZtSnFXDolCNsBrGL2h5WV4vL4ILaml1JYNfjk99k1qXx54ASeTrbc/cE+/rbqKG2G9i6feRX1HMytZHGMZZRwdvJ3cyA2yJ1Wg2bOWJ9B/V0TQghhfpLsCSHOOjWNLdz85m7yKhp4ePF4mloNrE0ZOclNbVMru7PLSS+upaq+pV/t/w0Gzf0fHSC3vJ5/XxdPgPvgZrWN83chwsfZYkYwfHPwBM1tBpYPcbZeTy6PD8ag4Yv9g5u59989uTy/Lp2rEkNY/9A5XD01hBfXZ3D7O3uoamg5+X2+0IJKODstnOAPICMXhBDCgtmYOwAhhDCmxpY2bnt7DykF1bx6YwLnjPPjnW05fHOwgEunmCZhGKg/f5fCh7uOn/zcztoKbxc7At0duHFmOJdMHoWV1ekrKc+tTWPt0WIevzSaaRFeg35tpRQXTx7F8+vSyCmrI8zbedD3Gqz3duSQXlzLrxaNx8W+5x9DWms+3ZtHVIAr0aPcTRJLhI8ziWGefLo3l7vmjx7QCta2jFJ++0USsyO9eeKyGGytrXjy8liig9z549fJXPbiVmysFRMD3Qj3Gf7v81BdOTWYrNJazu0o6RRCCGF5ZGVPCHHWaGkzcM8H+9iVXc7fr5zMuVH+WFkpLooNZFNqCdWNLeYOEa01644WMTvSm+eunsKjSybw0zkRzI70oa6pjfs+PsDS57ewMbXk5Irf6iNFPLc2jSsSgrlhRt8z9fpy3fRQrJXine05Q77XYLy0IYO3tmWz5F+bOZBb2e05x8vqufGNXRzMq+LqqSEmjWd5QjAZJXUczKvq9zXpxbXc9e5ewr2d+fd1CSdLTJVS3DAjjPdvm05VQwupRbVcNMIHqfck0N2Rf14dh5uDrblDEUIIMUiS7AkhzgoGg+bXnx5iTUoxj18ac9oq3tLJgTS3GVidbP5SztSiWoqqm7hk8igunRLEbXNH88iFUTzzk8l8f+9cnrt6CjVNLdz0xi6ue20n3x0q4P6PDzAp2J0nlsUYZe+Uv5sDF8UG8snuXOqGeS/jicoG8isbWB4fTGub5oqXtvHi+vSTe9xa2gy8tCGDC/65kf3HK/njJdHcMDPcpDEtmRSIvY0Vn/WzUUtZbRO3vLULOxsr3rh5Ku6OXZOh6aO9+eYXc7hj3miunT70BF0IIYQYDEn2hBAWT2vN498e4fP9+Ty0aHyX1a+4EA+CPBz5Lqlrp8ThtjG1fVD1vHFd90FZWSkunRLE2gfO4Q8XT+RYYQ13f7APexsrXr4+wahDrW+eHU5NUyufD3Kv2mDtzi4H4JbZ4ay4dy6LYwL426pjXPufHaw+UsTFz2/h6ZVHmT/Ol9UPzOOmWeFYW5m2OYibgy0XRAfw9cETNLb0PXPvvo8PUFzdxH9uTCTEq+cOm6M8HPntRRPwMtJsQCGEEGKgJNkTQli859am8da2bG6fG8HPzxnT5bhSiiWTAtmcVkJVvXlLOTelljLO34VAd8cez7GzseLm2RFsfHgBjy6ZwFu3TGOUR8/nD0ZciAeTg915a2tWvxrEGMuurHJc7G2YEOiGu6Mtz18TxzM/mUxSftXJpiav3pDAKzck9vo9MrZrpoVQ1dDC1wdP9HpeUl4Vm9NK+dUF44kL9Rym6IQQQojBkWRPCGHR3tyaxT/XpHFlYjC/vWhCj2WOS2IDaWnTrDLCAO3Bqm9uZVdWeb+7G7rY23Db3NHEBhu/OYlSiptnh5NRUseW9FKj378nu7PLiQ/zPLlap5TiioRgVvxyLo8tncjqB+ZzwSDmBw7VzNHejPd35a2t2b0mv29ty8bJzpqrppl2H6EQQghhDJLsCSEs1uf78vjjN0dYHB3AXy6L7XU/26Rgd0K8HPm2m6HXw2VnZjnNbQajDwcfrItiA/FxseetrdnD8nqV9c2kFtUyLbzrili4jzM/nRPRa3dOU1JKcdOscI4UVLM7u6Lbc0prm/jm4AmuSAiWpiVCCCEsgiR7QgiLtPpIEQ99eqi9q+U1U7DpY+C2UoolsaPYml5KRV3zMEV5uo2pJTjYWjE1fPCjE4zJ3saaa6eHsu5YMdmldSZ/vT0dSdRI+frPtCxuFO6Otry1Lavb4x/tOk5zm4EbTdwwRgghhDAWSfaEEBbnQG4ld3+wj9ggd169IRF7m/41Llk6KZA2g2ZlsnlKOTelljA9wtuojVaG6vphHMOwO7scO2srJod4mPy1BsPJzoarp4awKrmIE5UNpx1raTPw7o4c5o3zJdLPxUwRCiGEEAMjyZ4QwuK8uTULZztr3rx5Ks4DKPuLHuVGuLcT3x7qvQmHKeSW15NZWsf8brpwmpOfmwNLJgXy3z251Jp4DMOu7HJig91HVLJ7putnhKG15r0dpye/Kw8XUlTdxC2zws0TmBBCCDEIkuwJISxKc6uBdSnFXDAxAM8BtrRXSrF00ii2Z5RRWttkogi7tymtBOh+5IK53TyrYwzDvv7NmRuMhuY2kvKqRmwJZ6cQLyfOn+jPh7uOnzaG4a1t2YR7O424ZF0IIYTojSR7QgiLsi2jlJqmVhbF+A/q+iWTAjFo+P7w8JZybjxWQpCHI2N8nYf1dfsjLtSTySEevL0tG4PBNGMYDuRW0mrQTIsY+eMKbp4VQUV9C18faF8BPpRXyd6cCm6cGY6ViWf+CSGEEMYkyZ4QwqKsSi7Cxd6GWWN8BnV9VIArY3yd+W4YSzlb2gxsyyhj3jjfXjuGmtPNs8LIKKljZ1a5Se6/O7scpSAhbGSv7AHMGO3FeH9X3tzWPobhrW3ZONtZc0VisLlDE0IIIQZEkj0hhMVoM2hWHynknPG+g973pZTi0ilB7MgsZ29O9y32jW1fTgW1Ta3MHze4BHU4LI4OxNnOmi/355vk/ruzyxnv74q748gfWdA5gzCloJrvDxfy7cECGbcghBDCIpks2VNKhSil1iuljiilkpVS93Y876WUWq2USuv407PjeaWU+pdSKl0pdUgpFW+q2IQQlmlvTgWltc0sjhna0O1b50QQ6O7A775IoqXNYKToerYprQRrK8WsyJGb7DnaWbM4JpAVSQWn7VUzhtY2A/tyKkb8fr1TLZsShLujLQ9+crB93II0ZhFCCGGBTLmy1wo8qLWeCMwA7lZKTQQeAdZqrccCazs+B7gQGNvxuAN4yYSxCSEs0KrkQuxsrDhnvN+Q7uNsb8MfLonmaGENb27tfqaaMW1KLSU+1GPErwxdFhdETVMr644W9+v8xpY2tqaX8tT3R1n6/Gbi/7SatKKaLucdKaimrrmNqRGWk+w52llz9bQQGlramD/OlzG+Mm5BCCGE5TFZsqe1LtBa7+v4uAZIAYKAS4G3O057G1jW8fGlwDu63Q7AQykVaKr4hBCWRWvNysOFzI30wWUA4xZ6sig6gIUT/Hl2dRp5FfVGiLB7pbVNJOVXMW/syO/iOHOMN36u9nzRRynntoxSbnh9J1Me/4HrXtvJa5szcbK1QWvNfR8foLn19NXS3R3D1KdZ0MoewE0zwwn1cuLn54wxdyhCCCHEoAz9HVM/KKXCgThgJ+CvtS7oOFQIdLbUCwJyT7ksr+O5AoQQfdJa8/y6dI6Xd01cZo72ZnmCZTeXSD5RTX5lA/cuHGu0e/7x0mgW/n0jf/g6mf/cmGiS5ilb0koBmD9+5Cd71laKS6eM4q1t2VTUNXc72qK2qZVffLAfW2srrp4aytyxPkwf7Y2LvQ2rkgu58929PLc2lYcWRZ28ZndWOSFejgS4OwznlzNkozwc2fTwAnOHIYQQQgyayZM9pZQL8Blwn9a6+tQ3U1prrZQaUJ9vpdQdtJd5EhoaasxQhbBo644W84/Vqfi62mNn/eOifWNLG5/uzcPL2Y4FUUMrfzSnVcmFWClYOGFwIxe6E+ThyP3nj+UvK47yw5EiFkUPfi9gY0sbOWX1tBpOX9VakVSAl7MdMaPchxrusLgsLpj/bM7i26QCbpgR1uX465uzKKtr5su7ZzMlxOO0Y4uiA7gyMZiXNmSwYLwfieFeaK3ZnV0u8+mEEEIIMzBpsqeUsqU90Xtfa/15x9NFSqlArXVBR5lm5+aQfCDklMuDO547jdb6VeBVgMTERNMMhBLCwrQZNH9deYxwbydWPzAf2zOSvWUvbuWBTw6w4t65BLo7mjHSwVt5uJDpEd54DXCQel9umR3B5/vy+cPXyczuZ4loS5uBjcdKOFpYTUphDUcLqskqraOnEXWXxQVZzHy2CYGujPd35cv9+V2SvbLaJv6zOZPF0QFdEr1Oj10czfbMMh745CAr7p1LUXUjZXXNFrVfTwghhDhbmCzZU+1LeK8DKVrrf5xy6GvgJuCpjj+/OuX5e5RSHwHTgapTyj2FEL34+mA+x4pqeP6auNMSPQAHW2tevC6ei5/fwr0fHuCD26djY23eqSs7Msv4ZHcuj108EQ+nvpO3jJJa0oprub6blaahsrW24i+Xx7L8pW08uzqV3y+d2Oc1v/siiU/25AEQ4uVIVIAbS2IDGePn0mUkhAKL6kKplGJZXBBPrzzK8bJ6Qr2dTh7794YM6ptb+dWi8T1e72Jvw7NXTuHKV7bzp2+OEBfanhRa0vdACCGEOFuYcmVvNnADkKSUOtDx3G9pT/I+UUrdCuQAV3YcWwFcBKQD9cAtJoxNiLNGc6uBv/+QSvSo9oSjO2N8XfjzZTHc//FBnlubxoMX9Pxm3dSOFdZw+9t7qGlqJausjvdvm46TXe//FK1KLgTggmjjlXCeKj7Uk2umhfLm1iwujw8iupeSy8ySWj7dm8d100P5zUUTjNIsZqS5dMoonl55lC8P5PPL89r3SOZXNvDu9hx+khBCpF/vnSkTw7342TljeHF9Bruyy/FytmOMr/NwhC6EEEKIU5iyG+cWrbXSWk/SWk/peKzQWpdprc/TWo/VWi/UWpd3nK+11ndrrcdorWO11ntMFZsQZ5MPdx0nr6KBhxdH9VoqeFlcMFcmBvPC+vSTTUOGW3FNIz99azeOdtY8fmk0B3MrufPdvTS19j7XbVVyEZNDPExagvrrRVF4ONnxh6+T0brnCvF/rU3D3saa+xaOOysTPWhvTDJjtBdf7s8/+b345+pUUPS7Qc69540jJsiNrNI6EsM8TdL8RgghhBC9M28tlxBiSOqaWnl+XRozRnsxb2zfA7v/cEk0kb4u3PfxfoprGochwh81NLdx+9t7KK9r5vWbpnLjzHCeWj6JzWmlPPDxQdp62PB2orKBg7mVLDLRql4ndydbHl40nt3ZFXx98ES356QX1/DVwRPcOCsMX1d7k8ZjbpfFBZFZWsfBvCrSimr4bF8eN80MY5RH/xJuOxsr/nnVFJztrIc8F1EIIYQQgyPJnhAW7I0tWZTWNvPw4qh+rZw42dnw4nXx1Da1ct9HBzD01FHEyAwGzf0fH+BQfhXPXT2F2OD2MskrE0N4dMkEvksq4NEvk7pdUfuho4Rz8RA6ZfbXTxJDiA1y5y8rUqhrau1y/J9r0nC0tebOeWf/3LXFMYHY2Vjx5f58nvnhGM52Nvz8nMgB3SPSz5Xdjy7kmmkhfZ8shBBCCKM7O2uQhPgfUF7XzKubMrlgoj/xoZ79vm6cvyv/d3E0v/k8iW+TCrhk8igTRtnu6ZVHWZlcyO+XTuSCM5K22+aOpqqhhefXpePmYMuSSYEcLaghpbCaowU1HMqrZKyfC6N9e98nZgzWVoo/XBLN8pe28eL6dB5e/OOsuGOFNXyXVMDP5o8xekfQkcjd0ZaFE/z4ZE8u9c1tPHj+uG7n7vWlr/2YQgghhDAd+SkshIV6aUM6dX10RuzJVYkhvLk1i+fWpLIkNhBrE44FeHtbNq9syuTGmWH8dHZ4t+c8cP44KutbeGVTJq9sygTA0daacQGuXDx51LAOhE8I8+Ty+CBe25zFlYkhhPu0NxZ5bm0qznY23D539LDFYm7LpgSxIqkQHxc7fjonwtzhCCGEEGKAJNkTwgxqGlt4f+dxrp0eipuD7YCv35tTwdvbc7g8Pphx/q4Dvt7KSnH/wnH87P19fHPwBMviggZ8j760tBn483cpvLUtm4UT/Hhs6cQeS02VUvzxkmgSwz2xt7EiKsCNUC8ns82me2RxFKsOF/LEd0d47aapHDlRzYqkQn55buSgVrcs1Tnj/Zgc4sGNM8JwPkub0QghhBBnM/npLYQZvLU1m7+vTuVQXiUvXhvf706FGSW1PLPqGN8fLsTX1Z77zx836BgWRQcwIdCN59amsXRSoFFn75XVNnHPB/vZnlnGT2dH8NuLovq8v5WV4tIpxk86B8PPzYFfnjeWJ78/yvpjxXy48ziuDjbcOud/Z1UP2pusfHX3bHOHIYQQQohBkgYtQgyzNoPmo925uNrbsCKpkPd2Hu/zmuLqRn77RRIXPLuJTakl3LdwLOt/dQ5B/eyM2B0rK8V9C8eSVVrHlwe67z45GMknqrjkha3sPV7B338ymccunmj2Ie6DccvsCEb7OPPrTw/xw5EibpszGnenga/CCiGEEEKYi+W9AxPCwm1OKyG/soE/Xx7L/HG+/OnbIySfqOr2XK01r27KYN7f1vPJ7lyunx7KhocWGG3G2wUT/Yke5cbz69JoaTMM6V5aa77cn8/yl7Zh0JpP75o5rHvtjM3OxorHLp5IcU0Tbg423DIn3NwhCSGEEEIMiJRxirPKn749wp7s8i7PR/g485fLY0dEZ8APdx3H29mOxdEBzB7jzUX/2swvPtjP17+Yc1oCV9/cysOfHuLbQwWcP9GfR5dMIMzb2aixKKV44Pxx3Pr2Hr7Yl8+VUwfeIr+51cA3B0/w2pYsUgqqSQzz5KXrE86KOXTnjPfjvoVjGe/vOqi9lUIIIYQQ5mT+d75CGMnOzDJe35LFpGD301rjGzR8ffAEZXXNvHZTIvY21maLsbi6kTUpxdw2JwI7Gyu8Xex57uo4rv3PDh79Iolnr5qCUorc8nrueHcvRwur+fXiKO6aP7rf+/oG6twoPyYHu/OvdWksiwvCzqZ/C/5V9S28vyuHt7dlU1TdxFg/F566PJbL44P7fQ9LcN/Cwe+LFEIIIYQwJ0n2xFlBa81fVx3D382ej++YiaPd6QndJ7tzefizQzzw8UH+dU2cSUcN9Oa/e/NoM2iuOmUFbcZob+5bOI5/rE5l1hgfgj0dufuDfbQZNG/ePJVzxvuZNCalFPefP46b39zNp3vzuHZ6aJ/XbEsv5bZ39lDf3MacSB+eWj6Jc8b5miwhFUIIIYQQAyfJnjgrrE0pZm9OBX+5LLZLogdw5dQQqhtbeOK7FNwcbfjLZbHDnpgYDJoPdx1n5mjvLgPC714QyY7MMh796jBtBs1oH2devTGRCB/jlm32ZP44X+JCPXhhXRrLE4J6Xf1saTPw+68O4+dqz7+vS2DiKLdhiVEIIYQQQgzM2VNrJf5ntRk0f1t1jAgfZ36S2HNDkNvmjuaeBZF8uCuXp1ceG8YI221JLyWvooFrulk5s7ZS/PPqKfi62HP+BH++uHv2sCV68OPevRNVjbyzLafXcz/adZyMkjp+t2SiJHpCCCGEECOYrOwJi/fVgXyOFdXwwrVx2PbR4v/BC8ZR2dDMyxszcHe05WfnjBmmKNsbs3g62bIo2r/b436uDmx+eIHZBonPifRh4QQ//vbDMc4Z78vYboa1Vze28OyaNKZHeLFwgmnLS4UQQgghxNDIyp6waE2tbfxjdSoxQW5cFBPY5/lKKR6/JIaLJ4/i6ZVH2ZpeOgxRQnFNI6uPFHFFQnCvJZLmSvSg/Xvz5OWTcLW34d6PDtDc2nUUw0sbMiiva+bRJRNlf54QQgghxAgnyZ6waB/uPE5eRQMPL4rqd6JkZaV45ieT8HO15+WNGSaOsN2ne/NoNWiuntZ38xNz8nW156nlkzhSUM0/16Sediyvop7Xt2RxeVwQscHuZopQCCGEEEL0lyR7wmLVNrXy/Lp0Zo72Zu5YnwFda29jzU2zwtmcVsrRwmoTRdjOYNB8tCuX6RFejDmjMctIdP5Ef66eGsLLGzPYfcrMwr+tOoYCfrVovPmCE0IIIYQQ/SbJnrBYb2zJoqyumYcXjx9USeF100NxtLXm9c1ZJojuR9syyjheXt+vkQYjxaNLJxLs6cT9Hx+gprGFA7mVfHXgBLfNjWCUh6O5wxNCCCGEEP0gDVrOMmW1TRzMq+zyvLezPZNDPMwQkWlU1DXz6qZMFkX7ExfqOah7eDjZcUVCMB/vzuWhxePxc3UwaoxZpXW8viWTT/fm4e1sx6LoAKPe35Rc7G149qrJ/OTl7Tz+zRFyyurxcbHjZ+dEmjs0IYQQQgjRT5LsnUWKqhtZ9uJWCqoauz3+2NKJ/HROxDBHZRpvb8+mtqmVB84fWknhLbPDeW9nDu9tz+GBC4Zenqi1ZldWOf/ZnMXao0XYWlmxLG4Ud80fg4Ntz41ZRqKEMC9+fk4kL6xPB+DPl8XgYi//ZAghhBBCWAp553aWqG9u5da3d1PV0MLrNyXi42J/2vF/b0jn8W+P4O5oy/KEnmfRWYL65lbe3pbNeVF+jA/oOh5gIEb7unBelD/v7sjh5wsiB52QtbYZWHG4kNc2Z3IorwpPJ1t+sSCS62eGGX3FcDjdu3AsW9JLaW41cFViiLnDEUIIIYQQAyDJ3lmgzaD55YcHOHKimtduSuTcqK5z3J67Oo5b397Nw58dwtXBhgssqKTwTJ/szqWivoW7jDQj7/a5EaxJKeKzfXlcNz1sQNfWNLbw8e5c3tyaTX5lA6N9nHliWQzL44NxtLOslbzu2Fpb8eldM2k1aGz6mGEohBBCCCFGFnn3dhb4y4oU1qQU8X8XR3eb6AE42Frz6g2JxAa5c8+H+9mWMTzz5XqSUVLLgmc28PsvD1NS09Tv61raDPxncwNQIisAABHGSURBVBYJYZ5MDfcySizTIryIDXLn9S1ZGAy6X9c0txp4ckUKM59cxxPfpRDs6chrNyay5oH5XD8j7KxI9DrZWFtZXAmqEEIIIYSQZM/ivbs9m9e3ZHHzrHBumhXe67nO9ja8efNUwr2duP3tPRzqppHLcGhsaePu9/dRXN3Ih7uOM/9v6/nHD8eoaWzp89rvDhWQX9nAXfONs6oH7cPEb5sbQWZJHRtSi/t1zcd7cnllUyYLovz45p45fHznTBZO9DfrUHQhhBBCCCFOJcmeBVt/tJj/+zqZhRP8+P3Sif26xtPZjndvnY6nsx03vbGLzJJaE0fZ1Z++PcLRwhpeuDae1Q/MZ0GUH/9al878v23gjS1ZNLW2dXud1pqXN2Yw1s+F86L8jBrTRbGBBLg58Fo/xjBorXlraxaTgt3519VTZMC4EEIIIYQYkSTZs1BpRTXc88E+JgS68dzVcVgPYEXJ382B92+bDsD9nxykrZ+li8bw7aETvL/zOHfOG82CKD8ifJx58dp4vr5nNlEBrjz+7RGueGl7t6WdG1NLOFpYwx3zRht9Bc3W2oqbZ4ezLaOM5BNVvZ67Jb2UjJI6bp4VPqj5fkIIIYQQQgwHSfbMpM2g+evKozy3Jo29ORW0thn6fW11Ywt3vLsXRztrXrspEedBtMMP83bmj5fGcDC3kje2mHaoeKecsjoe+SyJuFAPfrXo9DEHk4I9eP+26bx8fTxpxTVc8fI2skvrTjvn5Y0ZBLo7cOmUIJPEd820UJztrHlhXXqv5721NRsfFzuWTAo0SRxCCCGEEEIYgyR7ZvLMD8f494YMnl2TyvKXthH3p9Xc8c4e3tmeTUFVQ4/XGQya+z86QG55Pf++LoFAd8dBx3DxpEDOn+jPMz8cI+uMxMrYmlrbuOeD/VhbKZ6/Jg7bbjo7KqVYHBPIh7fPoLqhheUvbTu5r/BAbiU7Msu5dU4Edjam+Wvr7mjLnfPH8P3hQrald9/AJqesjnXHirl2ehj2NtK0RAghhBBCjFyS7JnBd4cKeGlDBtdMC2X/78/nhWvjWDopkCMF1Tz2VTLnPrORj3cfR+uu5ZXPrU1j7dFifr90ItMihtaNUinFE8tisLex4tefHup3J8rBeHLFUZLyq/jbFZMI9nTq9dy4UE8+/dksHO2sufrVHWxMLeHlDRm4Odhw9bRQk8UIcMe80QR7OvKHb5K7XW19Z3sO1kpx/XTTxiGEEEIIIcRQSbI3zI4V1vDQpweJD/XgD5dMxNPZjqWTRvHk5ZPY/PAC1jwwn7hQD379WRJ3f7CPqvofO1SuPlLEc2vTWB4fzI0zBzYPrif+bg78fulEdmWX897OHKPc80xfHcjnrW3Z3DI7vN/z/cb4uvD5z2YR7u3MrW/tZtWRQm6cGY7LIEpWB8LB1ppHl0wktaiW93ac/v2oa2rlk925LJkUiJ+b5Q5KF0IIIYQQ/xsk2RtGVfUt3PHuHpztbXjp+oQuZYBKKSL9XHjv1uk8cmEUPyQXsfi5TezILCO9uJb7Pz5AbJA7f74sxqiNQa5ICGbeOF+e+v4oueX1RrsvtCd69398gGkRXjxyYdSArvVzc+DjO2cwfbQXLnY23Dw73Kix9WRRtD9zx/rwj9WplNX+2Cjm83151DS19jniQgghhBBCiJFAkr1h0mbQ3Pvxfk5UNvDSdfH497IyZGWluGv+GL74+WwcbK255j87uPrVHdjZWPHyDQlGH3CtlOLJy2NRwG8+T+q2fHQwvtifdzLRe+uWqYPa4+bqYMu7P53OlkfOxcfF3ihx9UUpxf9dPJH65jae+eEY0L5X8q1t2UwOdicuxGNY4hBCCCGEEGIoJNkbJv9ck8qGYyX838XRJIb3b69dbLA73/5iDlcmhFDb1MIL18YR5DH4hiy9CfJw5DcXTWBLeikf784d8v0+3ZvHA58cZOYYb968eRpOdoMvv7SyUrg72g45poGI9HPlplnhfLQ7l6S8qh/HLcyWcQtCCCGEEMIyKGOt4phDYmKi3rNnj7nD6FVxTSPPr03n3R05XJUYwlPLYweVLLS0GbrtYGlMBoPmutd2cjCvkm9+MYcxvi49nptRUsv1r+3Ez82BuZE+zBnrQ3yoJ3Y2VnyyO5dff36IOZE+/OfGRKOvRA6X6sYWzn1mA6FeTrg72pKUX83WRxZIF04hhBBCCDFiKKX2aq0Tuz0myZ5p1DS28J9Nmby2JYumVgPXTgvld0smjPjEp6CqgYue24y/mwNf3j2723hrGltY9uJWKupbCPd24mBeFW0GjZOdNZOC3dmRWc78cb68YoKS0+H2yZ5cHv70EAD3njeW+88fZ+aIhBBCCCGE+FFvyZ5pWxv+D2pqbeODncd5fl065XXNLIkN5MELxjG6l1WykSTQ3ZF/XDmFW97azRPfHeGJZbGnHTcYNA9+cpDssnreu3U6M8d4U93Ywo6MMrakl7Ito4xlU0bx1PJJFp/oAVwRH8z7O3I4UlDNdTJuQQghhBBCWBBJ9ozs3g8PsDK5kFljvPn14igmW2AzjwVRftw5bzSvbMpkxmhvlk4adfLYi+v/v737j/Wrru84/nzRUqEglNKuQFsBsZMVBrQQh0EJQyegDYVlcyDEDt3IoovMaRYqGcofW1ggOglbFyMVcKzMVCeNgpGBi86kIFosjB/aIEK70haZldgM1/HeH+eUfnO5l/7Yvff76/lIbr7nfM653/u+eef9vd/3/X4+52zgm49t4dolC3nrCUcCcNhBB/Kuk47a69sq9JMDDggrLj+dZ17Y4e0WJEmS1Fds9sbZH5/9Rt73W2/g7Qtm9fWFPD5+3pt58OkXWP7lR/jNuYdz7JGHcP8TW/j0v/6IixfN5YpJug1CLzhmxsEcM0EXxpEkSZImilfjHGenH3sEZ//67L5u9AAOnHIAN12yiAT+9J/W8eRzL3LVqodZePRh/PXF+3eRGUmSJEmTZ8KavSQrk2xN8mjH2Mwk9yb5cft4RDueJDcl2ZBkfZLFExWX9t78mdO54fdP5ZFN27nw5n9n6pTwD5efzsHT+n8tniRJkjToJvKTvVuB80eMXQ3cV1ULgPvafYALgAXt15XAigmMS/vgvJOO4gNnHc/Ol4ub37eY+TOndzskSZIkSXthwpq9qvo28MKI4aXAbe32bcBFHeO3V2MtMCPJ0RMVm/bNXy75Db53zTs5602zuh2KJEmSpL002Wv25lTV5nb7OWBOuz0XeLbjvI3t2KskuTLJQ0ke2rZt28RFqlckYeYh07odhiRJkqR90LULtFRzN/d9vqN7VX2uqs6oqjNmz549AZFJkiRJUv+b7GZvy67pme3j1nZ8EzC/47x57ZgkSZIkaT9MdrO3BljWbi8D7uoYf397Vc4zge0d0z0lSZIkSftowm6qnmQVcA4wK8lG4JPA9cCXknwQ+Cnw3vb0u4F3AxuAHcAVExWXJEmSJA2DCWv2qurSMQ69Y5RzC/jwRMUiSZIkScOmaxdokSRJkiRNHJs9SZIkSRpANnuSJEmSNIBs9iRJkiRpANnsSZIkSdIASnMhzP6UZBvNLRx6zSzg+W4Hob1irvqDeeoP5ql/mKv+YJ76h7nqD4Oap2OravZoB/q62etVSR6qqjO6HYf2zFz1B/PUH8xT/zBX/cE89Q9z1R+GMU9O45QkSZKkAWSzJ0mSJEkDyGZvYnyu2wFor5mr/mCe+oN56h/mqj+Yp/5hrvrD0OXJNXuSJEmSNID8ZE+SJEmSBpDN3jhLcn6SJ5NsSHJ1t+NRI8n8JN9K8liS/0hyVTs+M8m9SX7cPh7R7VgFSaYkWZfka+3+8UkeaOvqn5NM63aMgiQzkqxO8kSSx5O81ZrqPUk+2r7uPZpkVZKDrKnekGRlkq1JHu0YG7WG0ripzdn6JIu7F/lwGSNPN7SvfeuT/EuSGR3Hlrd5ejLJed2JejiNlquOYx9LUklmtftDUVM2e+MoyRTg74ALgIXApUkWdjcqtXYCH6uqhcCZwIfb3FwN3FdVC4D72n1131XA4x37fwN8pqreBPwX8MGuRKWRPgt8o6pOBE6lyZk11UOSzAU+ApxRVScDU4BLsKZ6xa3A+SPGxqqhC4AF7deVwIpJilGj5+le4OSqOgX4EbAcoH1vcQlwUvs9f9++P9TkuJVX54ok84F3Ac90DA9FTdnsja+3ABuq6qmq+hVwJ7C0yzEJqKrNVfWDdvtFmjelc2nyc1t72m3ARd2JULskmQe8B/h8ux/gXGB1e4p56gFJDgfOBm4BqKpfVdXPsaZ60VTg4CRTgenAZqypnlBV3wZeGDE8Vg0tBW6vxlpgRpKjJyfS4TZanqrqm1W1s91dC8xrt5cCd1bVS1X1E2ADzftDTYIxagrgM8BfAJ0XKxmKmrLZG19zgWc79je2Y+ohSY4DFgEPAHOqanN76DlgTpfC0m5/S/OC/HK7fyTw844/qtZVbzge2AZ8oZ1y+/kkh2BN9ZSq2gTcSPPf7M3AduD7WFO9bKwa8j1G7/oAcE+7bZ56TJKlwKaq+uGIQ0ORK5s9DZUkhwJfBv6sqn7ReayaS9N6edouSrIE2FpV3+92LNqjqcBiYEVVLQJ+yYgpm9ZU97XrvZbSNOfHAIcwyhQn9SZrqPcluYZmqcgd3Y5Fr5ZkOvAJ4Npux9ItNnvjaxMwv2N/XjumHpDkQJpG746q+ko7vGXXR/bt49ZuxScAzgIuTPI0zTToc2nWhc1op6CBddUrNgIbq+qBdn81TfNnTfWWdwI/qaptVfU/wFdo6sya6l1j1ZDvMXpMkj8ElgCX1e57mZmn3nICzT+7fti+t5gH/CDJUQxJrmz2xtf3gAXtVc6m0SzQXdPlmMQr675uAR6vqk93HFoDLGu3lwF3TXZs2q2qllfVvKo6jqZ+7q+qy4BvAb/XnmaeekBVPQc8m+TN7dA7gMewpnrNM8CZSaa3r4O78mRN9a6xamgN8P72CoJnAts7pntqkiU5n2bJwYVVtaPj0BrgkiSvS3I8zcU/HuxGjIKqeqSqfq2qjmvfW2wEFrd/w4aipryp+jhL8m6aNUdTgJVV9VddDklAkrcB3wEeYfdasE/QrNv7EvAG4KfAe6tqtIW9mmRJzgE+XlVLkryR5pO+mcA64PKqeqmb8QmSnEZzIZ1pwFPAFTT/RLSmekiS64A/oJlqtg74I5p1KdZUlyVZBZwDzAK2AJ8EvsooNdQ26zfTTMPdAVxRVQ91I+5hM0aelgOvA37Wnra2qv6kPf8amnV8O2mWjdwz8jk1MUbLVVXd0nH8aZqrEz8/LDVlsydJkiRJA8hpnJIkSZI0gGz2JEmSJGkA2exJkiRJ0gCy2ZMkSZKkAWSzJ0mSJEkDyGZPkjSUklSSf+zYn5pkW5Kv7efzzUjyoY79c/b3uSRJGg82e5KkYfVL4OQkB7f7vwNs+n883wzgQ3s8S5KkSWKzJ0kaZncD72m3LwVW7TqQZGaSryZZn2RtklPa8U8lWZnk35I8leQj7bdcD5yQ5OEkN7RjhyZZneSJJHe0N/ElyfVJHmuf+8bJ+VUlScNmarcDkCSpi+4Erm2nW54CrATe3h67DlhXVRclORe4HTitPXYi8NvA64Enk6wArgZOrqrToJnGCSwCTgL+E/gucFaSx4GLgROrqpLMmPhfU5I0jPxkT5I0tKpqPXAczad6d484/Dbgi+159wNHJjmsPfb1qnqpqp4HtgJzxvgRD1bVxqp6GXi4/Vnbgf8Gbknyu8CO8fuNJEnazWZPkjTs1gA30jGFcy+81LH9v4w9U+ZV51XVTuAtwGpgCfCNffi5kiTtNZs9SdKwWwlcV1WPjBj/DnAZvDIl8/mq+sVrPM+LNNM6X1OSQ4HDq+pu4KPAqfsTtCRJe+KaPUnSUKuqjcBNoxz6FLAyyXqaqZbL9vA8P0vy3SSPAvcAXx/j1NcDdyU5CAjw5/sbuyRJryVV1e0YJEmSJEnjzGmckiRJkjSAbPYkSZIkaQDZ7EmSJEnSALLZkyRJkqQBZLMnSZIkSQPIZk+SJEmSBpDNniRJkiQNIJs9SZIkSRpA/wcH+sRCSl9ljQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 1080x360 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Z8TX3X53FQT"
      },
      "source": [
        "Now we are ready to start importing Keras' modules."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0zIY8Fz_3Gar"
      },
      "source": [
        "from keras.layers.core import Dense, Activation, Dropout\n",
        "from keras.layers.recurrent import LSTM\n",
        "from keras.models import Sequential\n",
        "from sklearn.metrics import mean_squared_error\n",
        "from sklearn.preprocessing import MinMaxScaler\n",
        "from sklearn.utils import shuffle"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7x55PWxD3IFE"
      },
      "source": [
        "First of all, we need to prepare the data for the training and test stage. Once the data is loaded, we normalize the input values and split them between training (70%) and testing (30%).\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wIrOh9Pi3JZg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9acffd6c-cd97-49d7-e205-d39b3faac8b0"
      },
      "source": [
        "# convert pandas data frame in numpy array of float.\n",
        "data_np = data.values.astype(\"float32\")\n",
        "\n",
        "# normalize data with min max normalization\n",
        "normalizer = MinMaxScaler(feature_range = (0, 1))\n",
        "dataset = normalizer.fit_transform(data_np)\n",
        "\n",
        "# Using 70% of data for training, 30% for test.\n",
        "TRAINING_PERC = 0.70\n",
        "\n",
        "train_size = int(len(dataset) * TRAINING_PERC)\n",
        "test_size = len(dataset) - train_size\n",
        "train, test = dataset[0:train_size, :], dataset[train_size:len(dataset), :]\n",
        "print(\"Number of samples training set: \" + str((len(train))))\n",
        "print(\"Number of samples test set: \" + str((len(test))))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of samples training set: 100\n",
            "Number of samples test set: 44\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JI5UamBO3KrU"
      },
      "source": [
        "We also arrange the dataset (input and labels) in the appropriate Keras' format by using the helper function `create_dataset()`. \n",
        "\n",
        "`create_dataset()` takes as argument the variable `window_size`. This variable is highly important when dealing with sequences since it is going to determine the length of our input data to the network. For instance, by setting `window_size` to 5, we will be using the last 5 monthly subscriptions values to predict the next one. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DvGh-1db3MOy",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7ceecaf8-52bb-48eb-8af3-1331c05eb412"
      },
      "source": [
        "# helper function to read data.\n",
        "def create_dataset(dataset, window_size = 1):\n",
        "    data_x, data_y = [], []\n",
        "    for i in range(len(dataset) - window_size - 1):\n",
        "        sample = dataset[i:(i + window_size), 0]\n",
        "        data_x.append(sample)\n",
        "        data_y.append(dataset[i + window_size, 0])\n",
        "    return(np.array(data_x), np.array(data_y))\n",
        "\n",
        "# Create test and training sets for regression with window size 5.\n",
        "window_size = 5\n",
        "train_X, train_Y = create_dataset(train, window_size)\n",
        "test_X, test_Y = create_dataset(test, window_size)\n",
        "train_X = np.reshape(train_X, (train_X.shape[0], train_X.shape[1], 1))\n",
        "test_X = np.reshape(test_X, (test_X.shape[0], test_X.shape[1], 1))\n",
        "\n",
        "print(\"Shape of training inputs: \" + str((train_X.shape)))\n",
        "print(\"Shape of training labels: \" + str((train_Y.shape)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Shape of training inputs: (94, 5, 1)\n",
            "Shape of training labels: (94,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QlB3Edd03NwG"
      },
      "source": [
        "Once all the data is ready, we create the model as a `Sequential` object including 16 LSTM units and a dense layer outputting a single scalar.\n",
        "As mentioned, we specify a window size equal to 5, so that the prediction of the current element depends only on the previous five ones."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IuT4lcHt3Qqp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "92e2b3dd-b32c-4401-f0c1-1c7768ed780e"
      },
      "source": [
        "batch_size = 32\n",
        "rnn = Sequential()    \n",
        "rnn.add(LSTM(16, input_shape = (window_size, 1)))\n",
        "rnn.add(Dense(1))\n",
        "rnn.compile(loss = \"mean_squared_error\",  optimizer = \"adam\", metrics = ['mse'])\n",
        "\n",
        "rnn.fit(train_X, train_Y, epochs=500, batch_size=batch_size, verbose=0)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f9026445da0>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 102
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PrMqbfrK3Nz8"
      },
      "source": [
        "Now we can compute the MSE on the training and test data:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qkhF1mrr3Tor",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "52316edf-0fe5-49be-d1b6-e95a2871cc94"
      },
      "source": [
        "def get_predict_and_score(model, X, Y):\n",
        "    # transform the prediction to the original scale.\n",
        "    pred = normalizer.inverse_transform(model.predict(X))\n",
        "    # transform also the label to the original scale for interpretability.\n",
        "    orig_data = normalizer.inverse_transform([Y])\n",
        "    # calculate RMSE.\n",
        "    score = math.sqrt(mean_squared_error(orig_data[0], pred[:, 0]))\n",
        "    return score, pred\n",
        "\n",
        "mse_train, train_predict = get_predict_and_score(rnn, train_X, train_Y)\n",
        "mse_test, test_predict = get_predict_and_score(rnn, test_X, test_Y)\n",
        "\n",
        "print(\"Training data error: %.2f MSE\" % mse_train)\n",
        "print(\"Test data error: %.2f MSE\" % mse_test)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training data error: 23.40 MSE\n",
            "Test data error: 56.09 MSE\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SVOaCHMc3W6I"
      },
      "source": [
        "Moreover, we can plot the predictions and actual values in a graph to check visually the performance of our predictor."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kUjXRdOg3Vjj",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 350
        },
        "outputId": "b62f7123-e704-48fd-ba02-953c41405f10"
      },
      "source": [
        "# Training predictions.\n",
        "train_predictions = np.empty_like(dataset)\n",
        "train_predictions[:, :] = np.nan\n",
        "train_predictions[window_size:len(train_predict) + window_size, :] = train_predict\n",
        "\n",
        "# Test predictions.\n",
        "test_predictions = np.empty_like(dataset)\n",
        "test_predictions[:, :] = np.nan\n",
        "test_predictions[len(train_predict) + (window_size * 2) + 1:len(dataset) - 1, :] = test_predict\n",
        "\n",
        "# Create the plot.\n",
        "plt.figure(figsize = (15, 5))\n",
        "plt.plot(normalizer.inverse_transform(dataset), label = \"True value\")\n",
        "plt.plot(train_predictions, label = \"Training predictions\")\n",
        "plt.plot(test_predictions, label = \"Test predictions\")\n",
        "plt.xlabel(\"Months\")\n",
        "plt.ylabel(\"1000 member subscriptions\")\n",
        "plt.title(\"Comparison true vs. predicted in the training and testing set\")\n",
        "plt.legend()\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3sAAAFNCAYAAAC5cXZ6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeXxU9bn48c+TfU9ICGQDwho2IQJBwQ1csFWquNWtClpbsVZ6vdfa2l4tttp6q61e9bZW+3Op2oq1datLccOFRTYj+5KQSALZl8meTCbf3x/nTExgZpJA1uF5v17zYuacM+c8c+ZMmGe+3+/zFWMMSimllFJKKaX8S8BAB6CUUkoppZRSqvdpsqeUUkoppZRSfkiTPaWUUkoppZTyQ5rsKaWUUkoppZQf0mRPKaWUUkoppfyQJntKKaWUUkop5Yc02VNKqS6IyLUisnqg41CdiUi6iBgRCbIfvyMiS/vhuCtF5AUv684Qkb29eKxnReS+3trf8RKRn4nIn3t724E0UOdYREaLSJ2IBPb3sZVSJw5N9pRS/UZErhGRzfYXnCL7y/npAx1XV4wxLxpjFvXnMUUkX0TO7c9jDnXGmG8aY57raru+PLfGmE+NMRnH8lwRWSYin/V2TB32v0ZEbjqefRhjfm2M6dY+erLtYNWb78mR150x5qAxJsoY4+qN/fcG/bujlP/RZE8p1S9E5D+BR4BfAyOB0cAfgIsHMq6uuFuNBpvBGtfx8MfXNJTo+VdKKf+jyZ5Sqs+JSCzwS+BWY8w/jTH1xhinMeZNY8yP7W1CReQRETls3x4RkVB73QIRKRSRO0Wk1G4VXCIiF4jIPhGpFJGfdTjeShF5RURWiUitiGwVkZkd1v9URHLtdbtE5JIO65aJyFoReVhEKoCVHX/dF8vDdhw1IrJdRKa7X6eI/EVEykTkKxH5bxEJ6LDfz0TkIRGpEpE8Efmml/P1PFYy/KbdCnpnhy6L3xWRg8CH7vNyxHPbf5kXkYAOr7VCRF4WkXgvx9wtIos7PA6yX8csEQkTkRfsfVSLyCYRGdmN990d8/ft97RIRO7w8D69ICI1wDL7HP4/e9tDInKfu5ubiATa569cRA4AFx5xvE4tVyLyPft1ud/nWZ7Orb3tqSKyzn59X4rIgg77GSsiH9v7eQ8Y7uM1d3pP7PfjDhHZJiIO+5oM8/C8KcATwDw7ruoOq4eJyFv28T8XkfEdnjdZRN6zPwN7ReTbXuK6HzgDeNze/+P2ciMit4rIfmC/vex/RaTAvr63iMgZR7xnL9j33e/vUhE5aL8vPz/GbcNF5Dn7s7HbvuY7XdtHvJ6uYnxZrM9irYjsFJE5HdafLNbfhFoRWQUc9X74ek/E+lv1kP06SkTkCREJt9cNF5F/2ddRpYh8Ktbn0Ndn2t0NeY2I/Eqsvz+1IrJaRIZ3iOd6sf6uVIjI3eKjFU6sv4277P0cks6fu8Uikm3HuE5EZtjLPX42lFJDnDFGb3rTm9769AZ8A2gFgnxs80tgAzACSATWAb+y1y2wn38PEAx8DygD/gpEA9OARmCsvf1KwAlcbm9/B5AHBNvrrwBSsH7wuhKoB5LtdcvsY90GBAHh9rLP7PXnA1uAOECAKR2e+xfgdTumdGAf8N0O+3XasQcCtwCHAfFyPvKBczs8TgeMfYxIO64FQKG35wE/ss9pGhAK/An4m5fj3QO82OHxhcBu+/7NwJtAhB37bCCmG++7O+a/2TGfZL9v7vjc79MS+70IB16144y0r4WNwM329suBPcAoIB74yN5/kL1+DXBTh/f4EJBlv08TgDFezm0qUAFcYMdxnv040V6/Hvi9fQ7PBGqBF7y85k7viX2sjVjXWzywG1ju5bnLsK+zDsuetWOZi3U9vgi8ZK+LBAqAG+x1JwPlwFQv+28/Px2WGeA9O7Zwe9l3gAR7n/8FFANhHd6zF454f5+y37uZQDMw5Ri2fQD4GBiGdb1u44hr+4i4u4qxyX4/A4HfABvsdSHAV8DtWH8bLse6Bu/rwXvyMPCGfc6isT4bv7HX/QYrQQy2b2dgf8bx/pnueP3mApPsc7QGeMBeNxWoA063X8NDdtzneom7CDjDvj8MmGXfPxkoBU6xz81SO65QTzHqTW96G/o3bdlTSvWHBKDcGNPqY5trgV8aY0qNMWXAvcB1HdY7gfuNMU7gJazWlf81xtQaY3YCu7C+QLptMca8Ym//e6xf708FMMb83Rhz2BjTZoxZhdWiMbfDcw8bYx4zxrQaYxqPiNOJ9QVvMtaXuN3GmCKxWp+uAu6yY8oHfnfEa/jKGPOUscboPAckY3Vp7YmVxmoZPTIuT5YDPzfGFBpjmrG+BF8unrvr/RW4SEQi7MfXYCVpYL3mBGCCMcZljNlijKnpQcz32jFvB54Bru6wbr0x5jVjTBsQg/UF/T/s7UuxvlhfZW/7beARY0yBMaYS64u1NzcBvzXGbDKWHGPMV162/Q7wtjHmbfuaeA/YDFwgIqOxEsa7jTHNxphPsL7c98Sj9vVWaT83s4fPf9UYs9H+/LzY4fmLgXxjzDP2tfoF8A+sRLcnfmOMqXRfU8aYF4wxFfY+f4eV5Poah3ivMabRGPMl8CWdP4fd3fbbwK+NMVXGmELgUV8BdyPGz+z30wU83+E4p2IlYY8Yq3fBK8AmX8fqSEQE+D5wu33OarG6pruvUSfW53qMvf9PjTGmu/sHnjHG7LPfi5f5+r2+HHjTGPOZMaYF68cZX/t1AlNFJMY+p1vt5d8H/mSM+dz+LD+HlXSf2oMYlVJDiCZ7Sqn+UAEM95JkuKVg/eLu9pW9rH0f5utCBu5Ep6TD+kYgqsPjAvcdO5EodO/P7g7l7sZUDUync9e8ArwwxnwIPA78H1AqIk+KSIz9/GAPryG1w+PiDvtpsO92jLk7vMbmwRjg1Q6vczfgwkOCaYzJsdd/y074LsJKAMH6svxv4CWxumP+VkSCjzHmI9/XjuvGYJ3Dog4x/wmrhQ/7eUfuy5tRWK0k3TEGuMJ9TPu4p2N9aU8Bqowx9d08rifFHe430PP33NvzxwCnHBH3tUBSD/ff6ZoSq9vpbrG6nVYDsfjouuojvp5se+R76/M670aMRx4nzP77kwIcOiIB68n7mYjVwr2lwzl/114O8CCQA6wWkQMi8tMe7NtT3B7Pj/33o8LHfi7D+uHkK7G6IM+zl48B/uuIa2YUnT+TSik/osmeUqo/rMf69XiJj20OY30RcRttLztWo9x3xBo3lwYcFpExWF3JfggkGGPigB1YXf3cfP4Sb4x51BgzG6tr1STgx1jd55weXsOhY4zfWwwdl9djffEErDFtfP2lE6wvh980xsR1uIUZY7zF9DesVreLgV12AojdQnGvMWYqMB+rRen6HryWUR3uH/m+dnw9BVjXyfAO8cYYY6bZ64s87MubAmC8l3VHntsC4PkjzlOkMeYB+5jDRCSym8c9Hj1pAQIr7o+PiDvKGHNLD/ffvtwe+3YnVkvbMPvz4aDz56MvFGF9Rt1GedvwOGMsAlLtFjo3X+/nkeesHOuHpWkdznmsMSYKwG7V/y9jzDisH0z+U0TO8bKvnuh0fuwxggleg7ZatC/G+qHkNaxWQrCumfuPuGYijDHuVvzjiVEpNQhpsqeU6nPGGAdWt6P/E6uwSoSIBIvIN0Xkt/ZmfwP+W0QS7aIE9wAe5zLrptkicqn9a/5/YCURG7DGORmssWOIyA1YLXvdIiJZInKK3bJVjzU2qM1udXwZuF9Eou2k8j+P4zWUAOO62GYfVovFhXY8/43Vnc3tCTueMXbsiSLiq/rpS8AirPGE7lY9RGShiJxkJ5M1WEltWw9ey932ez4Na3zZKk8bGWOKgNXA70Qkxi5sMV5EzrI3eRlYISJpIjIM8NVq8mfgDhGZLZYJ7vPA0ef2BawWzfPFKgITJlahlTS76+dm4F4RCRFrqpBv9eC190QJkCYiId3c/l/AJBG5zv48BdvX5xQf++/qmorGGrNaBgSJyD1Y3Wv72svAXSIyTERSsX6M6YsY19vPXWGfr0vp3IX7SJ3eE7uXwFPAwyIyAkBEUkXkfPv+YvtaE6wE1MXXn5XunH9vXsG6RufbsazES3JrX6fXikissbqx13SI4Slguf03TEQk0v77Ed0LMSqlBiFN9pRS/cIeV/OfWAlJGdYvzD/E+tUZ4D6sL9XbgO3AVnvZsXodq/hKFda4uUvtFqpdWGPp1mN9sTkJWNuD/cZgfWGqwur+VYHVdQusoi71wAHgM6yE6eljjP83WMlvtXSopNeRnUT/ACuxOWQfu2MFw//FKiSxWkRqsZLdU7wd0E621mO13nVMyJKwvmzWYHX1/BiraydiVSJ8oovX8jFW17YPgIeMMb4mqL8eqwDFLqxz/ApWd0qwzvu/scZ6bQX+6eO1/B24H+s9qMW6ztyVSDudW2NMAVZr5s/4+tr8MV//H3kN1nmrBH6BVSSnL3wI7ASKRaS8q43t8WKLsMaLHcbqAvg/dE74O/pfrDGbVSLibUzcv7G6Je7Dur6b6FnX4WP1S6xrNw94H+t9b+7tGO3xbpdiFV6pxPob4fU6wvN78hOs63mDWFVk3+fr8YIT7cd1WJ+lPxhjPrLXdfmZ9hH3Tqy/Ly9htfLVYRVa8XaOrgPy7fiWY3XvxRizGatI1ONYn68crHPhdswxKqUGJ3eFKKWU8hsishKrmMh3BjqWE5mIpPN1FVRfxXmU6kREbgGuMsac1eXGJyARiQKqgYnGmLyBjkcpNXhpy55SSimlBpSIJIvIaXbX3Qys6RReHei4BhMR+ZbdHToSa+qF7VhTJSillFea7CmllFJqoIVgVV6txeo6+TrwhwGNaPC5GKu77mGs7qJX9XBaB6XUCUi7cSqllFJKKaWUH9KWPaWUUkoppZTyQ5rsKaWUUkoppZQfChroAI7H8OHDTXp6+kCHoZRSSimllFIDYsuWLeXGmERP64Z0speens7mzZsHOgyllFJKKaWUGhAi8pW3ddqNUymllFJKKaX8kCZ7SimllFJKKeWHNNlTSimllFJKKT80pMfseeJ0OiksLKSpqWmgQ1GDRFhYGGlpaQQHBw90KEoppZRSSvUbv0v2CgsLiY6OJj09HREZ6HDUADPGUFFRQWFhIWPHjh3ocJRSSimllOo3fteNs6mpiYSEBE30FAAiQkJCgrb0KqWUUkqpE47fJXuAJnqqE70elFJKKaXUicgvk72BVFFRQWZmJpmZmSQlJZGamtr+uKWlZUBiWrBggc5HqJRSSiml1AnG78bsDbSEhASys7MBWLlyJVFRUdxxxx3t61tbWwkK0tOulFJKKaWU6lvastcPli1bxvLlyznllFO48847WblyJQ899FD7+unTp5Ofnw/ACy+8wNy5c8nMzOTmm2/G5XJ12te7777LFVdc0f54zZo1LF68GIBbbrmFOXPmMG3aNH7xi194jCUqKqr9/iuvvMKyZcsAKCsr47LLLiMrK4usrCzWrl3bGy9dKaWUUkqpPtHc6uKjPaW0tZmBDmXQ0mSvnxQWFrJu3Tp+//vfe91m9+7drFq1irVr15KdnU1gYCAvvvhip23OPfdcPv/8c+rr6wFYtWoVV111FQD3338/mzdvZtu2bXz88cds27at2/H96Ec/4vbbb2fTpk384x//4KabbjqGV6mUUkoppVT/WL2zhBue3cQz6/IHOpRBy6/7E9775k52Ha7p1X1OTYnhF9+a1uPnXXHFFQQGBvrc5oMPPmDLli1kZWUB0NjYyIgRIzptExQUxDe+8Q3efPNNLr/8ct566y1++9vfAvDyyy/z5JNP0traSlFREbt27WLGjBndiu/9999n165d7Y9ramqoq6vr1BKolFJKKaXUYJFfbjV+/M+7ezh9wnAykqIHOKLBx6+TvcEkMjKy/X5QUBBtbW3tj93TAhhjWLp0Kb/5zW987uuqq67i8ccfJz4+njlz5hAdHU1eXh4PPfQQmzZtYtiwYSxbtszjdAMdK1N2XN/W1saGDRsICws75teolFJKKaVUfymsaiQmLIiQoAD+Y1U2r906n9Ag340rJxq/TvaOpQWuP6Snp/Ovf/0LgK1bt5KXlwfAOeecw8UXX8ztt9/OiBEjqKyspLa2ljFjxnR6/llnncWNN97IU0891d6Fs6amhsjISGJjYykpKeGdd95hwYIFRx175MiR7N69m4yMDF599VWio61fQBYtWsRjjz3Gj3/8YwCys7PJzMzsq1OglFJKKaXUcSmoamD8iChuXTCBm/6ymYff289Pvzl5oMMaVHTM3gC47LLLqKysZNq0aTz++ONMmjQJgKlTp3LfffexaNEiZsyYwXnnnUdRUdFRzw8MDGTx4sW888477cVZZs6cycknn8zkyZO55pprOO200zwe+4EHHmDx4sXMnz+f5OTk9uWPPvoomzdvZsaMGUydOpUnnniiD165UkoppZRSvaOgqoFRwyI4d+pIrp47ij99ksvGvMqBDmtQEWOGbvWaOXPmmCPnj9u9ezdTpkwZoIjUYKXXhVJKKaWU/2h1tZFx97ssP2scPz5/MvXNrVzw6Ke42gzv/OgMosOCBzrEfiMiW4wxczyt05Y9pZRSSiml1JBS5GjC1WZIGxYBQGRoEL//diaHqxu5981dXTz7xNGnyZ6IxInIKyKyR0R2i8g8EYkXkfdEZL/97zB7WxGRR0UkR0S2icisvoxNKaWUUkopNTQVVDUAMMpO9gBmjxnGrQsn8MqWQj7aUzpQoQ0qfd2y97/Au8aYycBMYDfwU+ADY8xE4AP7McA3gYn27fvAH/s4NqWUUkoppdQQVFjVCMCo+PBOy1ecM5GIkEA+3V8+EGENOn2W7IlILHAm8P8AjDEtxphq4GLgOXuz54Al9v2Lgb8YywYgTkSSUUoppZRSSqkOCisbCBBIieuc7AUHBpAcG0aRo3GAIhtc+rJlbyxQBjwjIl+IyJ9FJBIYaYxxl5gsBkba91OBgg7PL7SXKaWUUkoppVS7gqpGkmPDCQ48Op1JiQvncLUme9C3yV4QMAv4ozHmZKCer7tsAmCsUqA9KgcqIt8Xkc0isrmsrKzXglVKKaWUUkoNDQWVDaQNC/e4Ljk2jMOOpn6OaHDqy2SvECg0xnxuP34FK/krcXfPtP91j548BIzq8Pw0e1knxpgnjTFzjDFzEhMT+yz4Y1VRUUFmZiaZmZkkJSWRmpra/rilpcXnczdv3syKFSu6PMb8+fN7K9xetXLlSh566CEA7rnnHt5//32v22ZnZ/P222+3P37jjTd44IEH+jxGpZRSSik19BVUNbRX4jxSSlw45XXNtLS29XNUg09QX+3YGFMsIgUikmGM2QucA+yyb0uBB+x/X7ef8gbwQxF5CTgFcHTo7jlkJCQkkJ2dDVjJT1RUFHfccUf7+tbWVoKCPJ/2OXPmMGeOxykyOlm3bl3vBNsNxhiMMQQE9Ox3gV/+8pc+12dnZ7N582YuuOACAC666CIuuuiiY45TKaWUUkqdGJpbXZTUNB9VnMUtJTYcY6CkpolR8Z4TwhNFX1fjvA14UUS2AZnAr7GSvPNEZD9wrv0Y4G3gAJADPAX8oI9j6zfLli1j+fLlnHLKKdx5551s3LiRefPmcfLJJzN//nz27t0LwJo1a1i8eDFgJYo33ngjCxYsYNy4cTz66KPt+4uKimrffsGCBVx++eVMnjyZa6+9FqtnLLz99ttMnjyZ2bNns2LFivb9dvTss89y8cUXs2DBAiZOnMi9994LQH5+PhkZGVx//fVMnz6dgoICHnzwQbKyspgxYwa/+MUv2vdx//33M2nSJE4//fT21+F+za+88goAmzZtYv78+cycOZO5c+ficDi45557WLVqFZmZmaxatYpnn32WH/7wh+3HP/vss5kxYwbnnHMOBw8ebN/nihUrmD9/PuPGjWvff1FREWeeeSaZmZlMnz6dTz/9tBfeNaWUUkopNRgdclfi9NKylxwXBqDj9ujDlj0AY0w24Kmp6hwP2xrg1r6MZyAVFhaybt06AgMDqamp4dNPPyUoKIj333+fn/3sZ/zjH/846jl79uzho48+ora2loyMDG655RaCg4M7bfPFF1+wc+dOUlJSOO2001i7di1z5szh5ptv5pNPPmHs2LFcffXVXuPauHEjO3bsICIigqysLC688EKGDx/O/v37ee655zj11FNZvXo1+/fvZ+PGjRhjuOiii/jkk0+IjIzkpZdeIjs7m9bWVmbNmsXs2bM77b+lpYUrr7ySVatWkZWVRU1NDREREfzyl79k8+bNPP7444CVeLrddtttLF26lKVLl/L000+zYsUKXnvtNcBK7D777DP27NnDRRddxOWXX85f//pXzj//fH7+85/jcrloaGg41rdJKaWUUkoNcgXt0y54SfZirRa/Ih2317fJ3oB756dQvL1395l0Enyz52PLrrjiCgIDAwFwOBwsXbqU/fv3IyI4nU6Pz7nwwgsJDQ0lNDSUESNGUFJSQlpaWqdt5s6d274sMzOT/Px8oqKiGDduHGPHjgXg6quv5sknn/R4jPPOO4+EhAQALr30Uj777DOWLFnCmDFjOPXUUwFYvXo1q1ev5uSTTwagrq6O/fv3U1tbyyWXXEJEhPVB89QNc+/evSQnJ5OVlQVATExMl+dq/fr1/POf/wTguuuu484772xft2TJEgICApg6dSolJSUAZGVlceONN+J0OlmyZAmZmZldHkMppZRSSg1NBZX2hOreunG6W/Z0+oU+78apbJGRke337777bhYuXMiOHTt48803aWry/KtDaGho+/3AwEBaW1uPaRtfRMTj447xGmO46667yM7OJjs7m5ycHL773e/26Di9pePrdXdZPfPMM/nkk09ITU1l2bJl/OUvfxmQ2JRSSimlVN8rqGogOFAYGR3mcX1ESBCx4cEUVWvLnn+37B1DC1x/cDgcpKZaUwh27L7YWzIyMjhw4AD5+fmkp6ezatUqr9u+9957VFZWEh4ezmuvvcbTTz991Dbnn38+d999N9deey1RUVEcOnSI4OBgzjzzTJYtW8Zdd91Fa2srb775JjfffPNRsRQVFbFp0yaysrKora0lPDyc6OhoamtrPcY0f/58XnrpJa677jpefPFFzjjjDJ+v96uvviItLY3vfe97NDc3s3XrVq6//vpunCmllFJKKTXUFFY1khoXTkCAeN0mOTZMx+zh78neIHXnnXeydOlS7rvvPi688MJe3394eDh/+MMf+MY3vkFkZGR7F0pP5s6dy2WXXUZhYSHf+c53mDNnDvn5+Z22WbRoEbt372bevHmAVSDmhRdeYNasWVx55ZXMnDmTESNGeDxOSEgIq1at4rbbbqOxsZHw8HDef/99Fi5cyAMPPEBmZiZ33XVXp+c89thj3HDDDTz44IMkJibyzDPP+Hy9a9as4cEHHyQ4OJioqCht2VNKKaWU8mOFlQ1dVtlMiQvXufYAcXeFG4rmzJljNm/e3GnZ7t27mTJlygBFNHjU1dURFRWFMYZbb72ViRMncvvtt3fa5tlnn+1UJMWf6XWhlFJKKeUfZv3qPc6flsRvLj3J6zY/f3U7b20vIvueRf0Y2cAQkS3GGI/zt+mYPT/11FNPkZmZybRp03A4HEd1r1RKKaWUUmqoqW9upbK+xWtxFreUuHCqG5w0trj6KbLBSbtx+qnbb7/9qJa8Iy1btoxly5b1T0BKKaWUUkodp4IquxKnlzn23DpW5ByfGNXncQ1W2rKnlFJKKaWUGhIKK33PsefWPtfeCV6RU5M9pZRSSiml1JDgbtlLG9ZFN0472TvR59rTZE8ppZRSSik1JBRUNhIeHEhCZIjP7UbGWnMza8ueUkoppZRSSg0BBVUNjIoPR8T7HHsAoUGBDI8KPeHn2tNkr5dVVFSQmZlJZmYmSUlJpKamtj9uaWnp8vlr1qxh3bp1/RCpVaDllVdeAeCmm25i165d3Y7riSee0PnslFJKKaVUvyqobOiyOItbSlzYCd+NU6tx9rKEhASys7MBWLlyJVFRUdxxxx3dfv6aNWuIiopi/vz5x3T81tZWgoJ6/rb++c9/7lFcy5cvP6b4lFJKKaWUOhbGGA5VNXLquIRubZ8cG0ZuWX0fRzW4acteP9iyZQtnnXUWs2fP5vzzz6eoqAiARx99lKlTpzJjxgyuuuoq8vPzeeKJJ3j44YfJzMzk008/7bSflStXct111zFv3jwmTpzIU089BViJ2BlnnMFFF13E1KlTcblc/PjHPyYrK4sZM2bwpz/9CbA+ID/84Q/JyMjg3HPPpbS0tH3fCxYswD1B/bvvvsusWbOYOXMm55xzjse4Vq5cyUMPPQRAdnY2p556KjNmzOCSSy6hqqqqfZ8/+clPmDt3LpMmTWp/PTt37mTu3LlkZmYyY8YM9u/f34dnXymllFLqxLK3uJbSGv8bq+ZodFLb3NplcZaqpipa21pJiQunqLoRY0w/RTj4aMteHzPGcNttt/H666+TmJjIqlWr+PnPf87TTz/NAw88QF5eHqGhoVRXVxMXF8fy5ct9tgZu27aNDRs2UF9fz8knn8yFF14IwNatW9mxYwdjx47lySefJDY2lk2bNtHc3Mxpp53GokWL+OKLL9i7dy+7du2ipKSEqVOncuONN3baf1lZGd/73vf45JNPGDt2LJWVlcTHxx8V1wcffND+nOuvv57HHnuMs846i3vuuYd7772XRx55BLBaGjdu3Mjbb7/Nvffey/vvv88TTzzBj370I6699lpaWlpwuU7syS6VUkoppXrL+7tKuOXFLSyekcLDV2YOdDi9qsCediHNRzfOFlcLF756IUunLiUl9jzqW1zUNLUSGx7cX2EOKn6d7P3Pxv9hT+WeXt3n5PjJ/GTuT7q9fXNzMzt27OC8884DwOVykZycDMCMGTO49tprWbJkCUuWLOnW/i6++GLCw8MJDw9n4cKFbNy4kbi4OObOncvYsWMBWL16Ndu2bWsfj+dwONi/fz+ffPIJV199NYGBgaSkpHD22Wcftf8NGzZw5plntu8rPj7eZzwOh4Pq6mrOOussAOIuWQ4AACAASURBVJYuXcoVV1zRvv7SSy8FYPbs2eTn5wMwb9487r//fgoLC7n00kuZOHFit167UkoppZTy7t87i/nhX7fidBm+qvC/7ovtE6rHe2/Zy3PkUdtSyzt573DTuG8BUORoPGGTPe3G2ceMMUybNo3s7Gyys7PZvn07q1evBuCtt97i1ltvZevWrWRlZdHa2trl/o6sPOR+HBkZ2emYjz32WPsx8/LyWLRoUS++qu4LDbXK3gYGBra/vmuuuYY33niD8PBwLrjgAj788MMBiU0ppZRSyl+8s72IW1/cyvTUWM6fNpIih/914yyodCd73lv2cqtzrX8duZgga8jSiTz9gl+37PWkBa6vhIaGUlZWxvr165k3bx5Op5N9+/YxZcoUCgoKWLhwIaeffjovvfQSdXV1REdHU1NT43V/r7/+OnfddRf19fWsWbOGBx54gH379nXa5vzzz+ePf/wjZ599NsHBwezbt4/U1FTOPPNM/vSnP7F06VJKS0v56KOPuOaaazo999RTT+UHP/gBeXl5nbpxeosrNjaWYcOG8emnn3LGGWfw/PPPt7fyeXPgwAHGjRvHihUrOHjwINu2bfPYyqiUUkoppbr21rYiVrz0BZmj4nj2hiye+uQA7+0qwelqIzjQf9p2CqusFrqYMO+tdDnVOQRIAG2mjf3164HRJ3RFTr9O9gaDgIAAXnnlFVasWIHD4aC1tZX/+I//YNKkSXznO9/B4XBgjGHFihXExcXxrW99i8svv5zXX3+dxx57jDPOOKPT/mbMmMHChQspLy/n7rvvJiUl5ahk76abbiI/P59Zs2ZhjCExMZHXXnuNSy65hA8//JCpU6cyevRo5s2bd1S8iYmJPPnkk1x66aW0tbUxYsQI3nvvvaPi6ui5555j+fLlNDQ0MG7cOJ555hmf5+Tll1/m+eefJzg4mKSkJH72s58d49lVSimllDqxvfHlYW5flc2s0XE8c8NcokKDSIkLp81ASU2Tz/FtQ417jj1fcqtzGR09muiQaD4v+ZjAgOtP6JY9GcrVaebMmWPcFSTddu/ezZQpUwYoor51LFM5KIs/XxdKKaWUOjEVVDZw1oMfMSc9nmeWZREZarXjfLKvjOuf3sjLN89j7ljf9ReGknN+t4ZJI6P543dme91m8auLmRg3kenDp/PI1keILPkF88dM4Pd+VqymIxHZYoyZ42md/7TrKqWUUkopdQLZebiGNgM/v2BKe6IHkBJntX4drvaf7ovGGAqrGn2O12t2NVNQW8D4uPGcM/ocACKG7dZunGpoWLly5UCHoJRSSimlBol8u+Lm2MTITstT4sIA/CrJKattprm1zecce3mOPNpMGxPiJpAem86EuAmUObZRVHJ6P0Y6uGjLnlJKKaWUUkNQXlk9w6NCjipYEhESRFxEsF+17LVPu+BjDGJOdQ4A4+PGA3DumHOpMXspqis/YSdW98tk70R9M5Vnej0opZRSyh/lVdSTnhDpcV1KbDiH/agwSWGVlbj6KtCSW51LkASRHpMOwLmjz8VgMOE7qKhv6Y8wBx2/S/bCwsKoqKjQL/gKsBK9iooKwsLCBjoUpZRSSqlelVdez9jhXpK9uHD/atmz59jzVV00pzqH0TGjCQ60WjonDZtEfEgSQdE7T9iKnH43Zi8tLY3CwkLKysoGOhQ1SISFhZGWljbQYSillFJK9Zq65lbKapuPGq/nlhIXxsa8in6Oqu8UVDaSGB1KWHCg121yq3OZHD+5/bGIMC9pAf9q/ju5FeWclBbbH6EOKn6X7AUHBzN27NiBDkMppZRSSqk+k19uF2fx1o0zLpyaplZqm5xE+5iEfKjYX1rLOC+tmACNrY0U1hayeNziTsu/Me483jr4EuuLPmXJzPF9Heag43fdOJVSSimllPJ3B8o9V+J0c0+/UOQY+t0X29oMe4trmZIc43WbPEceBtNenMXtjFGzMa3RbK9a29dhDkqa7CmllFJKKTXEuFv2xsR7TvZS7ekXDvnBuL1D1Y3Ut7jISIr2uk1udS4AE+ImdFoeGBBIWMtMDrVspal16Ce+PaXJnlJKKaWUUkNMXnk9KbFhhId4HsOWHGu37PlBYZI9xbUAPpO9nOocggKCGB0z+qh1yUFZtNHCusPr+izGwUqTPaWUUkoppYaYvPJ60n2MYRsRHUpggPhFRc49RTUAZIz03bKXHpNOcMDR4xPHx8yAtgje/+r9PotxsNJkTymllFJKqSHG17QLAEGBASTFhPlHsldSy+j4CCJDvdeWzKnOOaoLp1taXDSt9eP5ovSLvgpx0NJkTymllFJKqSGkqr4FR6PTZ7IH1vQL/jBmb29xrc8unA3OBg7VHTqqOItbclwYrubhFNUX0drW2ldhDkqa7CmllFJKKTWEtFfi7CLZS44NH/LVOJucLvLK65niI9nLc+QBRxdncUuJDcc4h+EyLkoaSvokzsFKkz2llFJKKaWGkLxuJnspceEUORppazP9EVafyCmtw9VmyEjyPu1CTnUOgM+WvbaWeAAO1R7q/SAHMU32lFJKKaWUGkLyy+sJDBBGxUf43C41Lgyny1Be19xPkfW+vd2oxJlbnUtwQDCjokd5XJ8cG06b00r2CusKez/IQUyTPaWUUkoppYaQvPJ6Rg0LJzjQ91d598TqQ3nc3p7iGkKDAkhP8J7Y5lTnMDZ2LEEBngu4xIQFESHxCAEU1mqyp5RSSimllBqkuqrE6eZO9g4P4bn29hTXMnFkFEE+Etvc6lyvXTgBRISUuChCSNCWPaWUUkoppdTgZIwhv8L3HHtuKe6J1R1Dt2Vvb3EtGSO9j9drcDZwuP6w1+Isbslx4YgzQcfs9SYRyReR7SKSLSKb7WXxIvKeiOy3/x1mLxcReVREckRkm4jM6svYlFJKKaWUf/pobynv7/LPqoultc00tLgY5yPZ+9eBf/Hy3pcJCW4lMiRwyHbjrKxvobS2mSnJvsfrgffiLG7TUmKoq4+hoLagV2Mc7LpM9kTktyISIyLBIvKBiJSJyHd6cIyFxphMY8wc+/FPgQ+MMROBD+zHAN8EJtq37wN/7MExlFJKKaWUYuvBKm7+yxZ+++89Ax1KnzhQZlXi9NayZ4zh1xt+za82/Ipv/PMbxCR9ysGqiv4MsdfsKa4BfBdncVfi7Kplb256PK6WeKqaq2hwNvRekINcd1r2FhljaoDFQD4wAfjxcRzzYuA5+/5zwJIOy/9iLBuAOBFJPo7jKKWUUkqpE0hpbRO3vLCFFlcbRUN4nJovXU27UFxfTK2zlismXcGU+CnURb7Bprb/4tGtj1LROLSSvu5W4gwNDCUtKs3nvmaNGYY5AStydifZc5e1uRD4uzHG0YP9G2C1iGwRke/by0YaY4rs+8XASPt+KtCxXbXQXqaUUkoppZRPLa1t3PriVhyNTi45OZXa5lZqm5wDHVavy6+oJyQooH083pH2Vu0F4KLxF/HEeU9wesR9SGMGf97+Z6575zqMGTpz7u0pqiUhMoTEqFCv2+Q4rEqcgQGBPvcVGx7M6BhraoYTqSJnd5K9f4nIHmA28IGIJALd/ankdGPMLKwumreKyJkdVxrrauvRFSci3xeRzSKyuaysrCdPVUoppZRSfuq+t3axKb+K314+kwUZiQCU1Phf696BsnrSEyIICBCP6/dV7QNg4rCJAEyNn4rjq6u5NXMFBbUFOJp70m4zsPaU1JKRFI2I59cKXVfi7GhOqrXdwZoTZ9xel8meMeanwHxgjjHGCdRjdbnskjHmkP1vKfAqMBcocXfPtP8ttTc/BHScCTHNXnbkPp80xswxxsxJTEzsThhKKaWUUsqPvby5gL+s/4rvnzmOi2amkBw79Kcc8Ca/wve0C/uq9pEWlUZksLWNe/qF6EBrdNTh+sN9H2QvaGsz7Cuu9dmFs66ljuL64i7H67mdNnY0xhXK9pK83gpz0OtuNc7JwJUicj1wObCoqyeISKSIRLvv28/ZAbwBLLU3Wwq8bt9/A7jersp5KuDo0N1TKaWUUkqpo3xZUM1/v7aD0yYkcOf5GQAkx4YBUOzwr2TP1WY4WNHgc9qFfVX7mDRsUvtjd7InrdZ4tcN1QyPZO1jZQKPTxZQk79Mu5DrsSpyx3WvZO2VcAm3OePZVftUrMQ4FnqeZ70BEngfGA9mAy15sgL908dSRwKt2s2sQ8FdjzLsisgl4WUS+C3wFfNve/m3gAiAHaABu6NlLUUoppZRSJ5JWVxs/eHEriVGhPHb1rPaJt0fGWMlekZ8le4erG2lxtXmddqGptYmvar7i/PTz25elxFnnwtkcC8ChuqExz9yebhRn2VtpjU90d1ntysiYMMJIpKRhaCS8vaHLZA+YA0w1PRzNaYw5AMz0sLwCOMfDcgPc2pNjKKWUUkqpE9f+0joOVTfy8JUziY8MaV8eEhTA8KhQimuG5vxy3hywK3GmJ3hO9nKrc2kzbWQMy2hflmS3clbWBBEVHDVkWvb2FtciApNGek/2dlXsIi40jtSo7td0TIlK5auWXbS1tREQ0KdTjg8K3XmFO4Ckvg5EKaWUUkqpnth+yCo2MiMt7qh1KXFhfjdmL9897UKi52TPXZylYzfO0KBAEqNDKXI0kRyVPGSSvT3FNaQnRBIe4r3K5o7yHUxLmOazgMuRJiekQ0ArWw4d7IUoB7/uJHvDgV0i8m8RecN96+vAlFJKKaWU8mXHIQdRoUGM9dDSlRQT5ndj9vLK64kKDfI6FcG+qn2EB4WTFt15zrmUuHAOOxpJjUwdMgVa9hbXkuGjVa+xtZGc6hymJkzt0X7njrK6fK7J3XNc8Q0V3enGubKvg1BKKaWUUqqnth9yMDUlxuM0BMmxYWw4MLQmEe/KgfJ60odHeG3J2le1j4lxEwmQzu05KbFh7CupZWpUCptLNmOM6VFrWH9rbHGRX1HPt2ameN1mb+VeXMbF9OHTe7TvOanjYTNsOZxzvGEOCd2ZeuFjYA8Qbd9228uUUkoppZQaEK2uNnYX1XBSaqzH9Umx4dQ0tVLf3NrPkfWd/PJ6xg6P8rjOGMPeqr1Mip901LqUuHAOVzeREpVCnbOOmpaavg71uOwvraXNwJRk7y17Oyt2AjAtYVqP9p0anQoIuVXajRMAEfk2sBG4Aqty5ucicnlfB6aUUkoppZQ3OWV1NDnbvCZ77iqU/lKRs6W1jcKqBsYmRHhcX9pQiqPZ0Wm8nltKXDiNThexwSOBwT/9wteVOL1Pu7CzfCeJ4YmMjBzZo32HBoYSFRhPbWuJ33Xz9aQ7Y/Z+DmQZY5YaY67Hmhj97r4NSymllFJKKe+2F1rFWaZ7a9mL8a+59g5WNtBmelacxS3VTnwDXUNjrr29xbWEBQcwOt5zYguwo2IH04b3rFXPLTU6DQmpZFN+5bGGOGR0J9kLMMaUdnhc0c3nKaWUUkop1Sd2HHIQGRLodc655FhrMvEih39Mv5DXxbQL7mTP05xz7nPhclpVSwd7kZY9xTVkjIwm0MNYTIC6ljryHfk97sLpNil+DIEhVZrs2d61K3EuE5FlwFtYE6ArpZRSSik1ILYfcjAtJdZjcRaAkbFWxUp/6caZV14HwFgvye2+qn0kRyYTE3J018eUOCvZq64NJiIoYlC37FXVt7DzcI3PydR3V+7GYHpcnMVtVEwaEljD53klxxrmkNGdAi0/Bp4EZti3J40xP+nrwJRSSimllPKk1dXGrqIar104wZpfbnhUiN8kezsO1ZAUE0ZcRIjH9fuq9nWaTL2jhMgQQoICKHJYRVoO1R3qy1CPWUVdM1c/tYGGFhdXzBnldbsd5TuAnhdncUuLSgMx7K8qwNHoPKZ9DBXdmXoBY8w/gH/0cSxKKaWUUuo4OF1t5JXXs7uohriIEM6alDjQIfWJ3LJ6qzhLmvcCHgBJsWEU+0k3zuyCajJHHT15PECLq4U8Rx4LRy30uD4gQEiJDeNQdSOpI1MHZcteeV0z1z71OfkV9fy/pXPISo/3uu3Oip2kRqUyLGzYMR3LPQ+hBFWy5atKzp7csyIvQ4nXZE9EPjPGnC4itYDpuAowxhjfny6llFJKKdXn3t1RxOqdJewuriW3tI4WVxsAoUEB7Lj3fIID/a/UwvZDVnEWb5U43ZJiwimsauiPkPpURV0zBysbuOaU0R7XH3AcwGVcHqddcEuODedwdSNzxqewtWRrX4V6TMpqm7nmqQ0UVDXw9LIsTpsw3Of2O8p39Hgy9Y7SoqxkLzi0ik35VSdmsmeMOd3+13uHWaWUUkopNWBaWtu4fdWXhAUHMHNUHGdNSmRKcjQFlQ08tHof+eX1TBzpf1/ldhxyEBES6HXOObeUuDA25VdSXF/M+sPrqWqu4oZpNwzqCcU9+bKwGsBry56vSpxuKXHhrMstJyUyhVpnLTUtNR7H9/W30pomrn5qA4erm3hm2VzmjU/wuX1VUxWH6g7x7YxvH/Mxh4cPJzQwlLj4Ojbl+XeRli67cYrI88aY67pappRSSiml+teXhdU0Ol08fGUm35ie1L581+EaHlq9jz3FtX6Z7FnFWWK8VmtsdjWz9tBa9rv+jTN5E+e9Uta+buGohYyNHdtfofaK7AIHAeK9JXNv5V5CA0MZEz3G6z7GJETwzy+aiA+zWrGK6oqIiR/YZK+6oYWrntxAcU0Tz904l7ljvXfddNtVsQuA6QnHVpwFQERIjUqlJbCGbTscNDldhAUHHvP+BrPutOt3GvkoIkHA7L4JRymllFJKddfanHJEYN64zq0h40dEEhgg7LUnp/YnrjbDrsO+i7Pc/dnd/OijH7G37gOMM54bJ69g5byVABTVF/VTpL0nu6CaSSOjiQz13E6zr2ofE+ImEBjgPWGZlhKDMdDUYJ23wVCk5V/bijhQXs+fl87pVqIHXxdnmZIw5biOnRadhiuwnBZXG/tK/O9z4uY12RORu+zxejNEpEZEau3HJcDr/RahUkoppZTyaF1uBdNTYomNCO60PDQokPGJkewprhmgyPpOblkdjU6X11YuZ5uTNYVrWDxuMY/Me5PGghuYm3AJ81LmAVaL1lBijOHLgmpOHu25CydYyZ6vLpzwdatgaZU1dcNgKNKyNqec1Ljwo36s8GVnxU7SY9KJDjm+Fuu0qDQczhLAkFtWd1z7Gsy8JnvGmN/Y4/UeNMbEGGOi7VuCMeaufoxRKaWUUkodoaGllS8OVjHfyxinjKQY9vhhy972Qt/FWXaW76SxtZEFoxYwxu6mWORoIjEikQAJGHIte3nl9TganV7H65U3llPZVNllsjciJowR0aHkFLURHhQ+4C17rjbDutwKTp8wvEdjKHeW72Ta8GObcqGj1KhUGlrrCQxqJLe0/rj3N1h1pxvnz0TkUhH5vYj8TkSW9HlUSimllFLKp835VThdhvleKhdOToqmsKqR2ib/mkdsu12cZVyi5+Ism4o3AZCVlMXImDAAih2NBAcEMyJixJBL9rIL3MVZPE8zsK+y6+IsbielxrLjcA0pkSkD3rK387ADR6OT0yb6rrzZUWlDKaWNpcc1Xs/NPf1CyvCGE7Nlr4P/A5YD24EdwHIR+b8+jUoppZRSSvm0Nrec4EAhK91zEjA5yerm5m/jkXYccjA12Xtxlk3Fm5gQN4H4sHjCggOJj/x6YvXkyGSK64v7M9zjll1QTWRIIBNGeE5uu1OJ0216aiy5ZXWMjEge8KT3s5xyAK8t057sLN8JwPThvZfsDR9Wf8Ine2cD5xtjnjHGPANcYC9TSimllFIDZH1uBSePGkZEiOeiHRl2sudPXTldbYadPoqzOF1OssuymZs0t31ZUkxYe7KXFJk04ElOT2UXVHNSWqzX5HZf1T5GRIwgLsz7mD63k1JjMQZCZfiAd+Ncm1POlOQYhkeFdvs5Oyp2ECiBZMRnHPfx3XPtRUU6yCuvp9Wen9LfdCfZywE6zuA4yl6mlFJKKaUGgKPByY5DDp9zkqXGhRMdGuRXFTkPdFGcZXv5dhpbGzsleylxYUe17LWZofHFvsnpYndRjdcunNC94ixuJ6VZ5621KY6alhpqWwbm2mhyutiUX8XpE7rfqgdWcZbxceMJDwo/7hgigiOID4snIKQSp8tQUNV43PscjLqT7EUDu0VkjYh8BOwCYkTkDRF5o2/DU0oppZRSR9qQV0Gb8d0FTkTISIpmT5H/JHvbD9nFWdI8J3sbizciCHOS5rQvS4oNo9hhfZFPjkzG2eakorGi74PtBTsP1+B0Ga/FWZxtTnIdud1O9kbGhJEYHYqj1uoSOlDj9jbnV9HS2sZpXsabemKMsYqzJBx/cRa3tKg0mrDmYMwt9c+unF1Oqg7c0+dRKKWUUkqpblufW0FYcAAnj/be4gNWV843vzyMMaZHFQ8Hq+2HHIQHBzLeR3GWjPgMYkO/TgaTY8OpanDS2OIiJSoFsObaS4xI7JeYj4e7OIu3aRdyq3NpbWslY1j3uzWelBpLbnkYxFrJXm90ieypz3Ks8abdnVsPrHkBq5ure2W8nltqVCpflm0HrCk9zmVkr+17sOiyZc8Y87GvW38EqZRSSimlvrY2p5ys9HhCgnx/lZucFE1NUyvFNU39FFnf2nHIwdQUz8VZml3NZJdmk5WU1Wl5cqxdkbOmiaTIJGDoTKz+ZUE1ybFh7VVFj7S5eDMAs0bO6vY+T0qNpaDU2t/h+oFp2VubU86s0d7Hm3qys8IqztKrLXvRaZQ0FDE8Oshvi7T4mlT9M/vfWntS9ZoOk6v73wydSimllFJDQGltE/tL67rVBS4jyZpnzh+6crYXZ0mJ8bh+W9k2WtpayBrZOdlLspO9IkcjyZHJAEOmImd2QbXXLpxgdVsdFT2qPYntjpNSY2lrjSQkIHRAunFW1bew47CD03vQhbO8sZx38t4hOCC4211Wu2NU9ChcxkXqCAc5ftqN09ek6qfb/0bbk6rHdJhc3fOnTCmllFJK9an1udZ4s+6UrPenipx55XU0tLi8VuLcWLyRAAlgdtLsTsuTY61iHsWOJqJDookKjhrwOea6o6KumYOVDV6TPVebi83FmzsVo+kOa7yjEBU4YkDOw/oDFRhDl/PrOducfHjwQ2778DbO/fu5fHDwAy6ZcAnBgcG9FsuCUQsICwzDGfkxuWX1GGN6bd+Dhc+2UxEJBHYaYyb3UzxKKaWUUsqHdTkVxIQFMS3Fc9LTUWx4MCmxYewtHvqdsnYetl6D1+IsRRuZHD+ZmJDObRJJMe6WvaE1/cKXhe7J1D0ne3uq9lDrrD2q22pX3EVaxDVsQKZf+HR/OdGhQczwkrQDPLfzOZ7Z8QwVTRUMDx/O0mlLWTJhCWNjx/ZqLMPChnHZpMv42+6XqHHOp6K+pUdTQQwFPjt6G2NcwF4RGe1rO6WUUkop1T/WHSjn1HEJXuddO1JGUrRftOzlltYRGCCMHR551LrG1ka2lW/z2MoVHhLIsIhgijpU5BwK3TizD1YTIN6T201FmwB63LIHVlfOxoaYARmztzannFPHJxAU6DkNKaor4qHND5Eem85jZz/Ge5e/x+2zb+/1RM/t+qnXg0BI/Kc+K3L+6l+7uOuf2/skhr7UnakXhgE7ReQD93QLOuWCUkoppVT/K6hsoKCysVtdON0mJ8eQW1aHc4hPGp1bVs+oYeGEBgUetS67NJvWtlavrVxJseEU2y17KVEpQ6Jl74uCaiaNjPZaxGRj8UbSY9KPqaro9NRYqmuicDQ7qHfWH2+o3XawooGDlQ0+x+t9WPAhACvnrWTBqAUEBXS/iMuxSIlKYWHqIoKHbWR7sefrwhjD29uLqGl09mksfaE7yd7dwGLgl8DvOtyUUkoppVQ/WpdbDtCj+ckmJ0XjdBkOlPXfl/q+kFtW53PKhUAJZPbI2R7XJ8eGderGWd1cTYOzoc9iPV5tbYYvC6q9TrnQ2tbK1tKtx9SqB1bLnqvFmrajP8ftre3G9fvRwY8YFzuO9Nj0fooKbsm8CQlw8l7hPzyuP1BeT5GjqUefu8GiO8neQeDzDlMtbAS+6tuwlFJKKaXUkdblVjA8KpQJIzwnPZ58XaRl6I7bc7UZ8srrGZd4dBdOsJK9aQnTiAz2vD6pQ7I3FCpy5lXUU9PU6nW83q6KXdQ768lK7tl4PbeTUmNpc/Z/svdZTjlJMWGM9/I+OpodbC7ZzNmjz+63mAAyEiYR7jyJvQ3vevwRYG2OlaT2pILoYNGdZO/vQMd2f5e9TCmllFJK9RNjDOtyK5g/PqFHE6SPGx5FcKAM6XF7h6sbaW5t89iy1+BsYEf5Dp+FSlJiw6isb6HJ6WpP9gZzV87sg+7iLMM8rt9YvBHgqGkmumtkTCjDQqzpGvqrSEtbm2FdTjmnTRju9fr9pPATXMbF2aP6N9kDmBx+MS6p49WcV49a99n+ckbFhzM6IaLf4zpe3Un2gowxLe4H9v2QvgtJKaWUUkodKbesjrLa5h6N1wMICQpgfGIUe4dwspdjT3g93kOL5helX9BqWn12aUyyp18oqWkaGsleQTWRIYFeW3A3FW9iQtwEEsJ7di24iQgnJaeACe63lr1dRTVUNTg5faL3mD88+CEjIkYwbXjvTZzeXbNHzsLVkM6zO57D2fb12DxXm2H9gQpOGz/0WvWge8lemYhc5H4gIhcD5X0XklJKKaWUOtIue2L0GWneJ9n2JiMpekgne+7xhuM8VOLcWLyRoIAgMkdken1+sj2x+uHqJhIjEgmUwEGb7Blj+DyvghlpcR4rrjpdTr4o/aLHUy4caUZqHG0tcRTU9k/LnrsrpLekqam1ibWH17Jw1EICpDspSu+aMCKK5ooFFDcU8W7eu+3Ltx9yUNvUOiTH60H3kr3lwM9E5KCIFAA/AW7u27CUUkoppVRHOSW1BAhex635MjkphkPVjTiGYDVBsFo14yKCiY88unPZDvyzUwAAIABJREFUpuJNnDT8JCKCvXexcyd7xTWNBAUEMSJixKAds7c2p4J9JXVcnJnicf328u00tjYec3EWt+n2uL286sJj3kd5XTP/91EOdc2tXW771vYipibHMMKe9/BIG4o20Nja2O/j9dzGj4jEVZfByLB0nt7xNG3GGsXmTlJ72qI+WHSZ7Bljco0xpwJTgSnGmPnGmJy+D00ppZRSSrntK6ljTEIkYcFHTz3Qlcl2kZZ9JUOzdS+31KrEeeRYr8bWRnZV7GLOyDk+n58U23li9eTI5H4tTNITT3ycy4joUC6Zlepx/cbijQjS5WvuyklpVrJX3HBs56GmycnSpzfy4L/38vx637UbdxxysK3QwZVZo7xu88HBD4gOjj7mcYjHKz0hEhEhI+wicqpz+LTwU8Aarzc1OYaEITrZepfJnoj8SERigHrgERHZKiKL+j40pZRSSqmuvZ59iD+syRny88h1ZV9pLRN7UIWzo68rcg7NZO9Aeb3HLpy51bm4jIupCVN9Pj8iJIjY8OD2ufaSIpMGZTfO7YUOPssp58bTx3qcTxCslsyM+AziwnrenbejpJgwwmU4ja6aHk9D0eR0cdOzm9lbXMvY4ZE8vz6fVh+fv79tPEhoUABLTvacwLa2tbKmYA1npJ1BcGBwj2LpLWHBgYwaFoGpm8mM4TOod9bT2OJiy1dVnD5xaHbhhO5147zRGFMDLAISgOuAB/o0KqWUUkqpbjDG8Ou3d/Pbd/dy+R/XkV8+tOeS86a51cVXFQ1MGhl9TM9Pjg0jJiyIPUVDb/oFR6OTstpmj8VZ9lftB2DisIld7ic5NozD1V+37JU0lOBqc/VusMfpiU9yiQ4N4ppTRntc3+xqJrs0+7jH64FVpGV0jJV89aSV0+lq4wcvbmXTV5U8fGUmd31zMocdTby3q8Tj9vXNrbyefZjFM1KIDfecyGWXZlPdXD1gXTjdJoyIIq+8mRcueIELxl3w/9m77/C4qjPx498zM+q9S1ZvlmXJvXdsBwOmGXCAQIAEQgjJpvxI2ECSTVsCCZuyG0IoCSEhdEwzGNvBDfcmN8lW75LVe5dGc35/3BlLo+axNKNCzud55rF07507ZyRj9Oq8hRNF9XT3mqZsvR7YFuxZ9ss3Aq9IKc/3O6YoiqIoijJhMitaqGru4pZ54RTVtbPxjwd452QpUsqJXppdFda20WuSJIaMbmdPCMGMUO8p2aSlwNKJc4ixCzkNObjqXYnwjLjsfcJ8XKls7gBgmuc0jCYjdZ119l3sGBTXtbE9vYK7l0bj7Tp0UHSu5hzdpu4x1+tZpATFAXChLsem600myQ/eOcuerGqe2JTKjXOmsT45hAg/N14+XDTkcz4+d5HWLiN3LRk+hXNP6R6cdE6sDF95xe/BnuKDPCioacVk/ufjUF4tznodi2KGHoExFdgS7KUJIf6FFuztFEJ4YT13b0RCCL0Q4rQQ4mPz57FCiGNCiDwhxFtCCGfzcRfz53nm8zFX/nYURVEURfl3si+nGoDHr5vB9u+uYla4D49uOce33zg9ZZuRDCW3Sgt4EoNHt7MHfR05p1ogfKkT5xCNaXIbc4n3jUevu3wdY6iPm1UaJ4zv+IXOnl7OlDYOe/7F/QUYdDruXxEz7DXHK4+jEzoWhCywy5pWx8zFZPTgo9xPL3utlJKff3SeD89c5D+vTeLuJdEA6HWCe5dFc7ywngsXB+8cv368lOkhnsyPGjpgklKyp2QPS8OW4uF05c2H7Ck+yJMuo4mLjdovBQ7m1TI/2hd3Z8OErmssbAn2HgAeAxZJKdvRZux99Qpe47tAZr/PfwP8QUqZADSY7295nQbz8T+Yr1MURVEURRnWvqwaUqZpHf6m+brx+oNLefSaJHZkVHLTnw7S2TO50vRGK3cMnTgtZoR50dJlpNz8g+xUkV/TikEniPIf3G0ztyHXphRO0Hb2alu76TL2G6zeOn7B3ksHC9n07CF+vT1rUMBd09LFO2ll3LYgfNhulQDHK46T7J+Ml/Pog/7+FkQHYGxN5lTNYbp7u0e89tWjxbxypJiHVsfx8Jp4q3N3LIzCzUnPPwbs7p2/2MTZ0ka+tDhq2EHqOQ05lLeWT3gKJ/TNccyraaW+rZvzF5tZOYVTOMG2bpwmIAb4qRDid8BqKeU5W24uhIgArgf+av5cAOuALeZL/gFsMn98s/lzzOfXi+H+ViiKoiiK8m+vqaOHtJIGrkoKunRMrxN8a20Cv7t9DsV17SPupEwlY+nEaTEj1BvQUl+nkvyaVqID3HHSW//YWtdRR31nPYm+tgV7kf7aYPWCmrYJGayeUd6ETmjdNr//9lmrhkJ/P1xIT6+JB1fFDfv8DmMH52rP2S2FEyDYy5VAsZBu2c6ximMjXvvhmYukTPPmsetmDArcfNyduGV+OB+cKae+rS9ofPN4KS4GHbcM05gFtBROgeCqyKvG9F7swZIqnF/dyuF881zAz3uwJ4T4M9qsvXQgA3hICPGsjff/X+A/6Uv7DAAapZSWYRxlgOW7Hw6UApjPN5mvVxRFURRFGeRQXi29JslVScGDzq1O1ALAtOKG8V6WQ+SOoROnRXKYFzqh7bZMJQU1bcQNUa+X16hNAkvwS7DpPguj/QE4XliPp7MnXk5e4xrsZVe2cPXMEL5/9XTeO13OA/84SVuXkdYuI/88Usy1KaFDvk+LM9VnMJqMdmnO0t+K8GVgcmFX8e5hr2nrMnKmtJHV04OG3aH7yvIYuowm3jxRAkB7t5EPTpdz/awwfN0Hz0e02Fuyl7nBcwl0m/igyt/DGX8PZ/JrWjmUV4uXq4FZ4T4TvawxsSWNcx1wjZTyZSnly2i1e+sv9yQhxA1AtZQybYxrHHjfrwshTgohTtbU1Njz1oqiKIqiTCH7sqvxdjUwL3JwC3o/D2figzw4XTL1g70uYy9Fde2jbs5i4e5sID7Ik4zyqdOR09hroqiubcjmLJZOnNP9ptt0rwg/N8J93ThWqDVlCfMMG7dgr7Onl6K6NpJCvfn2+kR+c9ssDuXVcueLR/nTnjyaO418Y0Bq5EC7indh0BmYHzLfrmtbER9CT8sMdhXvHrY76fGieowmOeJg8ekhXqxICOCfR4ox9pr4+FwFLV1GvjRMZ1HQuoBm1meyLnLiUzgt4oM8yK9u42BeLcviAjDobQmXJi9bVp8H9P8uRQK5NjxvBXCTEKIIeBMtaPw/wFcIYalyjADKzR+Xm++N+bwPMKhFkpTyRSnlQinlwqCgoIGnFUVRFEX5NyClZF92DaumBw39w1hNDk8YXiK3qGTKNSQZyNKJc7RjF/pLDfcho3zq7OyVNnTQ0yuJH6Y5i5+LHwGutiWCCSFYEuvPsYJ6pJSEeYRR2VZp7yUPKa9a6/CYZP4e3rEoihfvWUBudQvPf5bPsrgA5gzxSwuLi60XeS/vPTYlbLJ7E5OlcQEYW1Jo7mnkdPXpIa85bO5KadkdHc5XlsdS2VbB/dv+H39J20pCsCcLo4fvZPlh/ocArI++7D7SuIkP8uRMWSOl9R1Ter6exbDBnhDiIyHEVsALyBRC7BNC7EVrtnLZf22klI9LKSOklDHAncAeKeXdwF5gs/my+4APzR9vNX+O+fweOdX/dVYURVEUxSEuVDRT3dLFVdOH+cXvof9lWcNWnun9b4rLbZ8hNhnZoxOnRco0byqbO6lp6RrzvcaDZezCUOmNluYsV9LiYWlcAHVt3eRVtxLqEXpF8+XGwjLywjLcHmB9cghvPLiU2RE+/OCapBGf/+K5FxEIHpr9kN3XFuLtSrjLPAQGdpcMncp5OL+OeVG+uDkPXzMqpaRRfwDP+P/ldMMeLure5o5F4cN+f3pMPWzJ3sKKaSuI9Bp+LMN4iw/ypNuoVaBN9Xo9GHln77fA74CfAtcBPwN+3u/j0foh8IgQIg+tJu8l8/GXgADz8UfQOoAqiqIoiqIMsi9bK+VYkzREsGfsgsyP6QxMZYYowXvLHdA5dXazBrJHJ06LVHP90VSp28u/NGPP+r2bpIm8xjybO3FaLInTdqaOFtQR5hFGc3czbT1t9lnsCLKrWnA26IgJsO4oOi/Kj+e+Esv8qOF39UqbS/kg7wM2T998aWSEvS2LnYZsn87ukt2DdsIb2rq5UNE8YuBT2VbJw7se5pdHf0GE+3Q6q65H51xPUEjesM/ZV7qP6o5q7pxxp93ehz0kmGtjw3xciQuc2FEQ9jBssCel/Gykx5W8iJRyn5TyBvPHBVLKxVLKBCnlF6WUXebjnebPE8znC8b21hRFURRF+bz6LNs8csFriDb1+Xugqwnnq3/GD8Qj+DRegFc3Q9fU6kJpkVs99k6cFjOnaR05zw8xD20yyq9uI8DDeVCDj/LWcjqMHTZ34rSI8ncnzMeVo4X1lzpyjkcqZ1ZlCwlBnoNSjktbSrnu3et4+sTTwz73+XPPY9AZeHDWgw5b35I4fzoaZ1LRVsGF+gtW544W1CElw9brfZD3Abd8eAunqk/x4yU/5vUbXsa5dTVuIoj3898Y9jXfynqLMI8wVoWvsut7GStLfeiKhMAr2jWerGzpxtkihGg2PzqFEL1CiKnxL4SiKIqiKJ87Q41csJLxHrj5oUtYS1PUBp50fxTK0+C1L0JX6/gu1g5yqsbeidPC29WJmAD3KVO3l1/TOmJzFls7cVr01e3Vjev4hZzKFmaEDk7DTa9JRyJ5NfNVtuRsGXS+sKmQjws+5o6kOwhyd1yviiWxAfS2JiPQsXtAV87D+XW4O+uHrCk8WXmS/zr0XyT5J/HuTe9y54w78fNwYdt31vD1OfdyqvoUGbUZg55X0FjAscpj3J50O3rd2H+JYU8Rfm58aXEkX14aPdFLsQtb5ux5SSm9pZTegBtwG/Bnh69MURRFURRlCJaRC2uHGLlATwdkfwLJN4LeiQXRfvytYTbtN70Apcfg9Tugu338Fz1K9urE2V9KuA8ZUySNs6C2jfjgIZqzWII93ysL9kCr26tt7aarU0tpdXSw19TeQ2VzJ9OHCPYy6zNx0jmxfNpyfnX0V5yoPGF1/rmzz+Gid+H+1PsdusZpvm5E+gThRdKgur1D+bUsjvUfNOcQ4EjFEfRCz7Prn7Wqu4sJ9ODO5M14OnnyyoVXBj3vrey3cNI5cUvCLfZ/M2Ok0wmeunU2c0domDOVXFEvUan5ALjGQetRFEVRFEUZ0d4sbeTCkD+M5X4K3a2QcisAC6L9kBJOelwFt7wIxQfh1D/Gd8FjUFTbbrdOnBap03wore+gqb3Hbvd0hPq2burbuokLHGJnrzGXcM/wUXWmXBKnpSPmVoBe6KlodWywl101uDmLRWZdJtP9pvPbNb8l0juSR/Y9QmlLqba+hlx2FO7grhl3EeDm+NHTS+P8aamfQUFTAQVNWjVVZVMnBTVtw6Zwnqw8ycyAmUN+HzydPbk18VY+LfrUKlW2vaedrflb2RCzYVze1787W9I4b+332CyE+DXQOQ5rUxRFURRFsSKl5LOcEUYunH8P3AMhRqsDmhPpi06Yh6vP/iL4xULh/nFe9ejlmAMFe3TitEgNt9TtTe7dPUsnzuF29q60OYtFTIA7Id4uHC9sIsQ9xOE7e9mVWvVT0oCAXUpJZn0myQHJeDl78cy6ZzBJE9/e/W1au1t57uxzuDu585WUrzh0fRZLYgNoqZsBwJ6SPQAcKagFYHn84OYsXb1dpNemsyBkwbD3vCv5LkyYeD3r9UvHthVuo7WnlTuTJldjls8rW3b2buz3uAZoAW525KIURVEURVGGMuLIhe42yNkJM28GvTbS19PFQFKoN6csw9VjVkLxYTCZxnHVo2fPTpwWKdO09MX0SV63V1CjdckcWLPX3dtNcXPxFTdnsdDq9gI4VlBHqEeow4O9rMoWvFwNhPlYNxO62HaR5u5mkv2TAYj2jub3V/2eouYivv7p1/m0+FPumXkPvq7jk064JM4fafQh1GU6u4p3AXAorw5fdydmhnkPuv5czTl6TD0jBnvhnuGsj1rPlpwttPe0I6Xkzaw3SfJLYk7QHIe9F6WPLTV7X+33eFBK+SspZfV4LE5RFEVRFKW/EUcu5OyAnnZIvdXq8IJoX86UNNJrklqw19kI1efHY7ljZs9OnBb+Hs6E+7qRMck7cubXtOKs1xHhZz2uoKCpgF7Zy3S/6aO+95I4f6pbuvA2BDu8G2dOldacZWBnx8y6TIBLwR7AkrAlPL74cdJr0/Fy9uKemfc4dG39Rfi5E+Hnhmv3HM7Xnediy0WO5NexLC4AnW5wV8q0qjQEgnnB80a8770z76Wlu4UP8z/kTM0ZchpyuHPGnZ+LTpdTgS1pnE8LIbyFEE5CiN1CiBohxJfHY3GKoiiKoij9fZZdQ2r4MCMXMt4Dz1CIWmZ1eH6UHy1dRnKrWyB6hXaw6OA4rHbscqpaLs39sqeUad6cL2/CaDLyZtabHL542O6vMVb5Na3EBLqjHxBojKU5i8VSc91ed5c3VW1V9Jp6R7/QEUgpyapsGbLmMrM+E73QD0pHvWPGHTy2+DGeWPEE3s6Dd9QcaUlsAGXl8QBsydpBeWPH8PV6VSeZ7jcdHxefEe85N3guswNn8+qFV3kj8w08nTzZGLvR7mtXhmZLGucGKWUzcANQBCQAjzpyUYqiKIqiKAO1dhlJK2lgzVApnJ3NWnOWlE0woJX7gmg/wFy35xsJvtFTItjrNpooqmtnuh07cVrMCvehoLaN9i4Tf0n/C+/lvmf31xirgpq2occuNOZi0BmI9hl9a/y4QA8CPV2oa3LHKI3UdNSMZanDqmzupKXTOOTYhcy6TGJ9YnE1DP7Fxd3Jd7Muap1D1jSSpXH+NDT5EuUZz/bCnQAsG6Jer6e3h7PVZ0dM4ezvnpn3UNJSwvai7dyccDPuTu6Xf5JiF7YEewbzn9cD70gpJ3eCt6IoiqIon0vny5voNUkWRvsPPpm9HXq7LnXh7C/K351AT2dOFTdqB6ZI3V5hbZvdO3FapIZruzFZla0sDl3MicoTSCnt/jqj1W00UVzfPmStYm5DLnE+cTjpnEZ9fyEES+P8KarUhrU7KpUzq1JrsDPczt7MgJkOed3Rsux4RruuoKzjAkF+bcQP8T24UH+Bzt5OFoYutOm+X4j+AqEeoQDckXSH/RasXJYtwd7HQogsYAGwWwgRhOrGqSiKoijKOLM0FLEEKlbOvwfeERCxaNApIQTzovysm7R01ENNpiOXO2aWTpwOSeM0d+TMKG9icehi6jvryW/Mt/vrjFZJvRboDjdQfbSdOPtbEhdAfZMWyDiqSUu2OdibEWqdjlnTXkNtR61Vvd5kEOHnRrivGx31s7TPI3KGrK07WXkSgPnB8226r0Fn4LHFj/HNOd8k1ifWfgtWLsuWBi2PAcuBhVLKHqAd1Y1TURRFUZRxllHeRKi3K0FeLtYnOhogb7c5hXPoH20WRPtRWNtGXWtXv7q9Qw5e8djkVreiE4O7UdpDsJcrwV4uZFxsYlGoFiAfrzxu99cZrfxhOnE2dTVR1V416k6c/S2L88fU44deOJFWlTbm+w0lp7KFUG9XfNytdyEz67VfNMzwn+GQ1x0trVOpPwezTPS2R9GiH/rvRFpVGnE+cVc0J2991HoenvuwvZaq2MimoepSynopZa/54zYppWPbFimKoiiKYrOGtm7yqlsmehkOl17eNPSuXtY2MPUMmcJpYanbO13SCH7R4BOpDVifxHKrWuzeibO/1HAfzpc3E+EVwTSPaZyoPOGQ1xmNfPOMvYFpnHmNeQB22dmLD/Ik0MOTILGIbQXb6DB2jPmeA2VVtjB9mHo9mHzBHmipnD29kp7muVR1FZLXkGd1vtfUy+nq0zbX6ykTy6ZgT1EURVGUyeup7Zlc/8eDl4ZQfx61dhkpqG1j1lDB3oWt4BsF4cOnlM0K98GgE6T1T+UsOgSTqE5toNzqVoekcFqkTvMmt7qFju5eFoUu4kTVCUzScXWMPb0m9ufUYDJd/mueXtZEsJcLXq7WO2KWwMMeO3uWeXtN1Qto7WllR+GOMd+zP2Ovibya1qGbs9RnEu0djaez476/o7UkTquJDdUvRid0fFL4idX57IZsWntaVbA3RYwY7AlN5HgtRlEURVGUK3e8sJ4uo4nH3k236QfpqejCxWakhFkRA1rRm0xQchTi18EIc7tcnfSkhPtoHTlBS+Vsr4WabAeuevS6jSaKatsc0onTIiXcB5OEzMpmFoctpqmr6dJYA0fYeuYi9/7tOM/syRvxup3nK9meUcnNc6cNOpfbmIuXk9elZh9jtSTOn+qaaUR4xrAld8sVP19KyeG8Wtq7jYPOFdW10200DdmcJas+a9LV61lE+buTGOzJtcmJLA1byieFn1g177HU66lgb2oYMdiT2nf2k5GuURRFURRl4tS2dlFU186cCB+OF9Xz6rHiiV6SQwzbnKU2G7qaIGLxZe+xIMqPc2WN9PSaIMZctzdJUzkLa9swmiSJwfbvxGlh+VqeNzdpAcfW7R3KrwXgD7ty2J1ZNeQ1ZQ3tPPrOWWaF+/CDa5IGnc9tyCXBL8FuA7m17pOCZI+rOVdzjuz6Kwv+D+bVctdfj/GTDzIGnetrzmL9PWzqaqK8tXxSpnCCtuP58XdW8th1yWyM3Uh5aznnas9dOp9WlUaEZ4TdAm7FsWxJ4zwlhBjc2kpRFEVRlAln2an6rxtmsnp6EL/enkVpfbt2chKnKF6pjPImQrxdBg9TLz2m/Rm55LL3WBDtR2ePicyKZvCLBe/wcZ23J6XknZOlVDZdvql5rrkGM9GBO3vTfFzxc3cio7yZUI9QIr0iHRbsSSk5ml/HuhnBpIZ78703zwxKO+7pNfHtN05jkvCnu+bhYtAPukduQ65dUjgtEoI88XV3QrYswFnnzLu5717R8y27lO+dKuewOZi1yK5qQScGd1O1NGdJDpicO3sALgY9ep1gfdR6nHXObC/cDoBJmjhVfUrt6k0htgR7S4AjQoh8IcQ5IUS6EOLcZZ+lKIqiKIrDnSpuwFmvIzXch6dunYUAHn8vHVlxDp6O1ebJfQ6klzcNXa9XegLc/CEg/rL3mB/tC5gDZCG0VM5xrNs7WdzAo1vOce3/7Wfn+eF73ZXWt/P3Q0XodcIhnTgthBCkhvuQcVHbNV0cupi0yjR6Tb12f62S+nYuNnWyNimI57+8AINe8PV/ptHa1Zf++Nud2ZwuaeTXt80iOmDwbLeq9ipaelrs0pzFQqcTzI/yI73UyNUxV/Nx/sc2N2o5VlDH8cJ6HrtuBtEB7vzk/Qy6jH1fu+zKZmKGaLCTVZcFMGnTOPvzdPZkTeQadhTuwGgykt+YT2NXo83z9ZSJZ0uwdw0QD6wDbgRuMP+pKIqiKMoESytuIDXcG1cnPeG+bjy+MZmDebVk7H1TG0nwwTehu32ilzkmbV1G8mtah+7EWXpM29WzIa0vzMeNaT6ufXV7MSugrRrqRq4hs5ezpdpQ91BvVx76Zxo/ej+dju6+4KCn18Rz+/K5+g+fkVnRzFO3zHJYJ06LlGk+5FS10GXUmrS09LSQ1ZBl99c5WlAHwLL4ACL83Hn2rvkU1LTy/bdO87293+Ppg6/zwv4C7l4SxQ2zB9fqAZyr0fYa7Bnsgbbjm1fdyrVRm2jpaWFn0U6bnvenvXkEejrzleUx/PLmVApq23jhswJA61iZXdVC0hDNWS7UXyDUIxQ/Vz+7vg9H2Ri7kbrOOo5XHr80okLt7E0dtszZKwYigXXmj9tteZ6iKIqiKI7VZezlXHkTC2P8Lx27a3EUS+P8ac/eh8ktABoKYc8TE7jKsbtQYW7OMjDYa6+HulyItL3aZH60nzZ+ASBmlfZn0QE7rXRk6eVNhPm4svU/VvLQmjheP1bCjX86yPmLTZwqaeDGZw7ymx1ZrE4MYtf313D7Isf3yEsN96anV5Jb1Xqpbu9Ehf1HMBzJryPQ0+XSTuXyhEB+tDGZTwtOs7tkN//Me5q48Fr+64aZQz6/qq2Kp44/RaRXJCkBKXZd2/woLejqbY8h1ieWLTmXb9RyuqSBA7m1PLgqDlcnPWumB3HD7DD+tDeP/JoW7t/5AFVuz5IYPHiHMrMuc0rs6lmsiliFp5Mn2wu3c7LqJCHuIUR4Rkz0shQbXTZoE0L8DPgh8Lj5kBPwqiMXpSiKoijK5WWUN9NtNF36YRW0tLTf3DydOeSwy+kq5IL74eifoeTYBK50bNLLtDTDQcFemTkosaFez2J+lB/ljR1UNHWAfxx4ho7bcPX0Mi0V1dmg4/Hrknn1gSU0d/Sw6dlD3PbcYRrbe3jhngW8eO9CwnzcxmVNqdO0r2lGeRNB7kHEeMfYvW5PSsmRgjqWxvlbNVZ5YGUsqQkV2jVGL4xBf6elp37Q87t6u/je3u/R3tPOH9f+EVeD66BrxmJupC96neBUSSObEzdztuYsOQ05Iz7n2b15+Lo78eWl0ZeO/fSGmbjodXx76185VZ2GwTObZifrv1vtPe0UNxdP6nq9gVz0LqyPWs+u4l2cqDzBgpAFdmuQozieLTt0twA3AW0AUsqLgONaQymKoiiKYpNT5nRESy2aRXRHJq6ih7drYziW+F3wiYAPvwU99h8aPR4yyrWZa8HeQzRnEXqYNvx8vYEsw9VPFTdqqZ8xK6HY8XV7LZ09FNS2MTuiL2BdmRjIju+t5sbZ03hgRSy7vr+Ga1LGt8NhlL87Xq4GzphTTBeHLuZU9SmMpsGjBEarsLaNquYulsUHWB0XQuDtn4+XiOHROU/T2dvG/9v3/+ju7b50jZSSXx75JRl1GTy56kkS/BLsti4LN2c9KdO8SStu4Kb4m3DSOY24u3f+YhO7Mqt5YEUsHi6GS8eDvV357oZoiuUW3GUsxrY4dlz8K9Xt1ZeuyW7IRiKn1M5ccEHPAAAgAElEQVQewMa4jbT2tFLfWa9SOKcYW4K9bvMIBgkghBi8H60oiqIoyrhLK24gyt99cIfKooNIBGf1M/kkuxVu+qOW7rjvqYlZ6BgN35zlOITOAmd3m+81c5o3rk46TpX0q9trqYD6AjutdmgZ5c0AzIqwDsz9PZz5/R1z+ckNM/HsFziMF51OGyx+OF+rqVsUtoi2njYu1F2w22scsdTrxVkHe83dzWTUnuPO1Ku5b9EynljxBGdrzvLU8b6/p69lvsbW/K08POdh1kett9uaBpof5ceZ0kY8nLy5OnrkRi1/3puPl4uBe5fHDDrX7bEXnVMztcXXYKrZjFH28Kujv7o0p87ydZ1qwd7i0MX4u2rp4qo5y9RiS7D3thDiBcBXCPEgsAv4i2OXpSiKoijKSKSUnCxuYGH0EE0eCg8gwmYzf3oMO89XYopdC/PvhcPPQFna+C92DNq7h2nO0muE8rQrSuEEcNLrmB3u22+4+krtTwePYEgv13bOhgxaJ9jKhABK6tspqWtnUYhW/2jPVM4j+XWEeLsQG2i9X3Dk4hF6ZS+rIrTayQ0xG/jarK+xJWcLb2e/zdGKo/z25G9ZF7mOb8z5ht3WM5SFMX1jOTZP30xLTwv/KvrXoOvyqlv4JKOC+5bH4OPmZHWuur2av59/mSXBa5GdMST6x/DNud9kT+kedpXsArR6PX9Xf4Ldgx36fuzNoDNwa+KtRHlFEesdO9HLUa6ALQ1afgtsAd4FpgM/lVI+4+iFKYqiKIoyvNL6Dmpbu5g/MNjr6dRq2WJWcW1qKFXNXZwta4QNT4BXGHz4TTB2TcyiR+HCxWZMQzVnqcqAnnaIvPww9YHmR/tx/mITnT29EJgIHsEOD/bOlTUR4eeGv4ezQ19nNFYmBgLa0PMAtwASfBM4UXn5Ji0XGzt4bl8+Nz97iHfTyoa8RkrJ0YJ6lsUFDKrzOlR+CC9nL2YFzrp07D/m/gcrw1fy1PGn+P6+7xPjHcOTq55EJxzbG9CS3ptW3MDCkIXEeMfwQd4Hg657dm8+bk567l85OOB55vQzGE1GfrbyUZ66dRb/sTaBe2feS7J/Mk8ee5Kmriay6rNIDkiekjVv3573bT7Y9MGUXPu/M1v/y0kHDgD7zR8riqIoijKB0kq0RhYLBgZ7ZcehtwtiVrEuKQSDTrDjfCW4+sCN/wc1WXDk2QlY8eikl5ubs0QMCPZKzTtPowj2FkT70dMryShv0ur2YldDwT4wmca42uGllzdZ1etNJvFBnoR4u3AwTxsKvih0EaerT9PT2zPo2qaOHt48XsIdLxxh+a/38JsdWRRUt/LU9kzauwfX+eVVt1LbOrheT0rJofJDLAtbhkHXl76q1+n59apfM81jGhLJH9f9EQ8nx1cQ9R/LIYTg6uirOV19mqaupkvXFNe18eGZcr68NHpQ0J5Vn8WHeR9yd/LdRHpFcseiKK5NDcOgM/CL5b+gobOBp44/RX5j/pRL4bTQCR1OOqfLX6hMKrZ04/wacBy4FdgMHBVC3O/ohSmKoiiKMryTRQ14uRiYHjKgZ1rRQRA6iF6Gj7sTy+ID2JlRqdUMJV6tBTan/zlug8THKr2siSAvF0IGNmcpO67tVPpc+XiCeVH9hqsDJKzX5u1VZYx1uUNqau+huK6dWeG+l794AgghWJEQyOG8WkwmyeLQxXQYO8ios/56XLjYzLKndvPYe+nUtHTxyNXT+ezRq3j5q4uobe3mn0eKB927r14v0Op4TkMO1R3VrAxfOeg5Pi4+vH7967x303tEeUfZ8Z2ObEGM/6W/E6sjVtMrezl88fCl8y8fKsKg0/G1Abt6Ukp+e+K3+Lj48ODsBwfdNzkgmftS7mNbwTaM0jhlgz1larJlZ+9RYJ6U8itSyvuABWijGBRFURRFmSBpxQ3MjdJaxlspPABhc7SdPOCalFCK6trJqWrVzs++U2tGUj41aveGb85yTNvVG0VKWaCnCzEB7n3BXvw67c+8XWNY6fAu7U5Owno9i5UJgTS093ChopmFIQsRCI5XWNftvXasGJOUvP/N5ez+/hq+sz6R6AAPFsb4s3p6EM9/lk9jRydnqs/w5zN/5s2sNzlaUEe4rxuR/tajJA6Ua7MNhwr2QAv4Qj3GtzPpgihfKpo6udjYwazAWfi6+LK/bD8AbV1G3k0r4/rZYYO6wn5W9hnHKo/xzbnfxNvZe8h7PzznYaK8tMB1Ko1dUKY+W4K9OqCl3+ct5mOKoiiKokyAls4esqtaBqdwdrdD+cm+YeHAhpkhCAE7z1dqB5JvBIMrnHtrHFc8OsM2Z2mphMaSK27O0t/8aD9OlTRoO55eoRAyC/L3jHHFQzs3iZuzWKxIMNft5dXi6+rLdL/pVnV73UYTH5+rYMPMUOZF+VnVbdV21DJvZjYdfn9n/TtXcc/2e3ju7HP86tivOFJ6liUD5usBHCw/SJJfEkHuQePzBm2wIFrrNplW3IBep2dF+AoOlh+k19TLh2cu0tJl5MtLrXcajSYjvzv5O2J9Ytk8ffOw93Y1uPL0mqf5aspX1UByZVwNG+wJIR4RQjwC5AHHhBA/Nw9YPwqMPGlSURRFURSHOV3SiJTD1et1WwV7wd6uLIjyY0eGOdhz9Yak6yDjXRiiJmsyGbY5i6VeL+LK6/Us5kf5UdvaTWm9ub1+wnooOQJdLSM/cRTSy5qIDnDHx33y1juFeLuSGOzJIfMIhmXTlpFWnXapZm1fdjVNHT3cMi/c6nmdxk6++NEX+Vv207h7ldHdnMJ/L/s122/djpeTL53eH7A01t/qOS3dLZytPjvsrt5EmRHmhZuTvi+VM3w1jV2NpNem8+rRYpLDvJkfZf3f3P6y/RQ1F/Gded+5bD1bSkAKjyx8RDU4UcbVSDt7XuZHPvAB5jl7wIdAoYPXpSiKoijKMNKKG9AJmBs5oAas6KA2ZDxqqdXha1JCuVDRTGl9u3Zg9h3QXuewnSx7GTb9sfQY6F0gbPao732p+6K50Q0J68Fk1NJg7exc2TCpqJPMioRAjhfW0WXs5drYazGajOwu2Q3AB2fKCfBwvtS502Jf2T5qO2r53Zrf8ff1W2kpu5WS0ulEeEWwyPdODB6FCI/zVs85VnEMozROumDPSa9jbmTfWI4V4SvQCR1vn/8XFyqauWdp9KBA7b3c9whyC+KqyKsmYMWKcnnDBntSyl+M9BjPRSqKoiiK0udUSQNJod54uQ7YSSg8ANPmart3/VyTotU+XUrljF8Pbv6TPpUzvbyJQE8XQrxdrE+UHodp88DgMvQTbTA9xAtPF0Nf3V7kUnDysHvdXn1bN+WNHZO2E2d/KxMC6ewxcaq4kZn+M4nxjuGTgk9o6uhhV2Y1N86ZhpPe+kfHbfnbCHYPZn3UemZH+nL1zBD+eqCApo4e2msWojOG8Er2n+kx9e0iHyw/iKeTJ3OC54z3W7ysBdF+XKhopr3biI+LD3OD5rKvdD9eLgZunjvN6tqqtioOlB9gU8Imq46iijKZ2NKNc6EQ4n0hxCkhxDnLYzwWpyiKoiiKtV6T5HRJIwuiB+zqdbdpTVf6pXBaRAW4kxzm3ZfKaXCG1Fshaxt0No/Dqkcno7yJWeHe1rspxi6oOAORi8Z0b71OMC/Kl1PFWj0dBmetU2neLrt2Ku3bnZycnTj7WxLnj14nOJRXixCCjbEbOV55nHdOZ9BtNLFpQApnQ2cDB8sPcn3s9eh1egC+94VEmjuNvHSggONFTczz+DJFzUW8nf02oHWuPFB+gKVhSydlG/8F0X70miRnS7Xv24Lg5bTIYq6b64aHi3VA92H+h5ikiVsSbpmIpSqKTWxp0PIa8DJwG3Bjv4eiKIqiKOMsu7KF1i7j4Hq90mNg6hky2AO4NiWUtJIGqls6tQOz7wBjJ2R97OAVj057t5G86iGas1Sc1eoSx9CcxWJelB9Zlc20dpnnwyWsh8ZirVupnaSXacFkavjQXRonEy9XJ+ZG+l6at3dd7HVIJG9mfkRcoAdzBuxO7izaiVEauT7u+kvHUqb5cG1KKH/el09TRw83TV/PktAlPH/2eZq7m8lrzKO6feiRC5OBpSbvVIm249tSnwBATGSp1XUmaeL93PdZHLqYSO8rH/+hKOPFlmCvRkq5VUpZKKUstjwcvjJFURRFUQZJM/8QujDauunFcPV6FtekhiAlfHqhSjsQsQj8YiZtKufZ0iZMksHBXukx7c8xNGexWBDth0nC2VLz7l7Ceu3PvN1jvrfFubIm4oI8BqfcTlIrEgI5V9ZIU0cPMT4xJPomc7HnCJvmhQ+qV/uo4CMS/RJJ8k+yOv69qxMxmrTd0eXxQfxg0Q9o6mriL+f+wsHyg9rrhK8Ynzd0hXzcnUgM9uRkUT29Jsn20+Ak/chuth5DcaLyBGWtZdyaeOsErVRRbGNLsPczIcRfhRBfEkLcank4fGWKoiiKogxyqriBIC8XIvys55ZReADC54OL55DPSwrxIibAnZ3nzcGeENruXsFn0Fzh4FVrjL0m7nzxCA/98yQ7Miro7OkddE1mRTM/eOcs9/3tOO7O+kHdDyk9Dr7R4BUy5vXMjfRFCO1rCoB/nPawY91eenkTs6dAcxaLlQmBmCQcNQ9DDxbL0LuVsSjB+ntV0lzCuZpz3BB3w6B7zAj15tb54aSGexPq48oM/xncFH8Tr2W+xtb8rST6JY77DL0rsSDaj1MljezLrqasvpP5QSs4UnGE7t7uS9e8m/su3s7efCH6CxO4UkW5PFuCva8Cc4Fr6UvhHPxftqIoiqIoDneyuJ75Ub7WuyxdrXDx1LApnABCCK5JDeVwXi1NHeZmGbNuByRkbHHsos1OlzZytKCeA7m1fOPVUyz61S4ee/ccRwvq+CynhnteOsZ1/3eAbecquHNxJJ98ZxVBXv2asLTVmYepjz2FE8DHTdvFseyWAlrzmqIDWm2gDSqaOrjzxSP8cMs5bWZfP9UtnVQ0dTIrYvLX61nMjfTF3VnPobxapJTkFsSBFJxp2Gt13baCbQi0ur6hPH3bbN57uG/37tvzvo1BZyCvMY+V0yZnCqfF/Gg/mjp6ePKTTIK8XPhS6gY6jB2crDoJQGNnI7uKd3FD3A246EffJEhRxoMtwd4iKeVCKeV9Usqvmh/3O3xliqIoiqJYudjYQWl9B4tjA6xPlB7VxgbEjPxD9DUpoRhNkr1Z1dqBwAQIXzBuqZyfZddoDUB+uI5X7l/M1ckhbD17kd0v/QSXV29kRfnf+J9lPRz54Rp+eXMqMYEeYDJpaZVv3we/S4LWKki23++cF0T7caq4AZM57ZCE9dDTrs3cu4yTRfXc+MwhThY18NbJUl7Y36/Wry6fqiNvAnJKdOK0cDboWBLrz8G8Wi5UNJNfaSDKYxafFH5yKZiVUvJxwccsDl087A6dQa/D2dD3Y2aIRwj3pdwHwKqI4X8pMRksNNfD5te08aVFkSwPX4qL3oUDZdpYjm2F2+gx9agUTmVKsCXYOyyEmOnwlSiKoiiKMqJjhVpq3ZIBQ6opPAA6w7D1ehZzI3wJ9nLpq9sDLZWzMh2qLth7uYPsz61hXqQvfh7OrJ4exO/vmMvJH13FDzw+YYFLGQ+Z3uKLp+/D99lkeOcr8OlP4f9mw6u3QuFnsPhBePgIzLzZbmuaH+VHc6eRgtpW7UDMKtA5XTaV8/VjJXzpL0fxdNGz7TuruH5WGE/vyGJ/djWceR2eX8Wsw9/lIcPHzAyb/M1Z+luREEhBTRt/3pePk15wZ/LNFDUXcaFO+zuSXptOSUuJVWMWW3x91td57gvPsTBkoSOWbTexgR74uTuh1wm+tCQKN4Mbi0IXsb9sP1JK3s19l5SAlEG1iooyGdkS7C0Fzgghss1jF9LV6AVFURRFGX/HCurxdjWQPDB4yN+tpTY6e4z4fJ1OsDYpmP05NfT0mrSDKbdqjV3S33bQqjV1rV2klzexenqQ1XH3i0dx6W7E6ZY/Ix7Ng9tegqSNUHIUDv0RAhNh88vw/Wy49ikIse/vn+dbhqtb6vZcPLWgOW/ogfPdRhM/fj+dH72fzrL4QD781kqSQr14evNs5gTpaXnjq/DBwxA+nzT3lTxueAOPnA/sumZHswxO33augquSgrkp8RqcdE5sK9wGwMcFH+Oid7niejUnvRMrw1cOavQy2QghuH1hJPcsjSbMR6uNXR2xmpKWErYVbiO3IVft6ilThi3B3rVAIrCBvnq9y45eEEK4CiGOCyHOCiHOCyF+YT4eK4Q4JoTIE0K8JYRwNh93MX+eZz4fM9o3pSiKoiifR0cL6lgcq81Cu6SlUtuZS7DtB+91ycG0dBk5WWQObjyDtNTF9C1ayqSDHMyrRUpYMyDYI/MjcHLX1u8RCLM2w6Y/wyOZ8Hgp3PO+NhNwDAPURxIX6IGvu1Pf1wO0tVSfh+aLVtd2G018+aVjvHashG+sieflryzCx13rsulRc4a3+U+u4TB/d/0y7Xe+y3e6v0W++xwt+Cs66JD1O0JSiBeBns4A3DIvHB8XH1aFr2JH4Q66ervYUbiDqyKvwsvZa4JX6jiPb0zm5zelXPp8dcRqAJ489iRuBrdhaxUVZbK5bLBnHrMQCawzf9xuy/OALvNz5mBu8CKEWAr8BviDlDIBaAAeMF//ANBgPv4H83WKoiiKogCVTZ0U1bWzNG5AvZ4l3dDGYG9lQiDOeh17s6v7DqZuhqZSKDthp9UO9llODX7uTtajFEwmyPxYW7uzu/UThAAXxwcTQgiWxgZwILfWum4PIN96d29/Tg3HC+t5YlMqj103oy/oPvk3+Ns1OAkTZ7/wBr9o2shDr52hvFVybPEfwS8W3rwLqrMc/n7sQQjB6ulB+Lg5sW5GMAAb4zZS01HDH0/9kYauhiG7cH6ehXuGE+8TT0t3CxuiN+DpPHTXW0WZbC4btAkhfgb8EHjcfMgJePVyz5Oa1n7PcQIksA6wtP36B7DJ/PHN5s8xn18vJvs+v6IoijIpXLioteuvbOqc6KU4TF+93hDBnmcohM6y6T4eLgaWxPmzO7Nf3V7SdWBwhYx37bVcKyaTZH9OLSsTg6x3JcuOQ2ulXWvwRmNDSgiVzZ2cNQ9AJyQVPEMG1e3tPF+Jl6uB2xf2G6Ld0wH/+i+IWgbfOMCCldfygw1JHMjVBpMnxUbDl7doX9/XNo/bmIux+ukNM/ngWytwddIDsCZiDe4Gd1658Aq+Lr6Tdk6eI1l2926bftsEr0RRbGfLDt0twE1AG4CU8iJg06/ahBB6IcQZoBr4FMgHGqWURvMlZUC4+eNwoNT8GkagCRjwfzRFURRFGeylg4VsSSvjxj8dJK24vu9EfSEYu4d/4hRytKAeLxcDM6f1q9frNWq7Twlf0HbCbLQ2KZj8mjZK6tq1A67ekLgBzr8PpsGz78Yqs7KZ2tauwSmcF7aC3ll77Qm0fkYIBp2wnkEYvx7y92pfY7QZgZ9mVvGF5BCrLpPk7ITuVlj9A3DT6v++eVU816WG4u6s15qz+EbB3e9ARwO8/kXoahnvt3jFfN2diQ3sqwF1NbheqtG7NuZanHRTY0i8Pd2Xch+/XP5L5gbNneilKIrNbAn2uqXWa1cCCCFGrv7uR0rZK6WcC0QAi4EZo1plP0KIrwshTgohTtbU1Iz1doqiKMoUZzJJPsupZnGsP+7Oeu588ShvHC+B2lx4Zj48v1LrVjnFHSusY9HAer3yk9DZBIlX1ijDkpq3J6vf7l7qbdBW7ZDasv052i7XanPjDwCk1Or14tZqweYE8nF3Yll8ADvPV/bNyku+ATobIfdfABwvrKexvYdrUgaMGkh/R9sF7DfjUAjBM1+ax65H1uDmrO2METYHbv8HVGdC1rbxeFt2tylhEwadgU2Jmy5/8edQgFsAtyTeMukbzChKf7YEe28LIV4AfIUQDwK7gL9cyYtIKRuBvcAy830M5lMRQLn543K02kDM532AuiHu9aJ57t/CoKCggacVRVGUfzPnypuobe3mrsVRbP3WSpbFB/L4e+l8uuUFkCYtze4fN8C7X9OamUxB1c2dFNS0sTRuwMiF3E+1Tppxa6/ofjGBHsQFerAnu98vTadfA86eDknl/CynmhmhXgR7u/YdrDgDTSUw8ya7v95oXJMSSmFtG7nV5gqUxGu09Ni0vwNaCqerk856d7LDHAym3gY6vdX9DHod03zdrF8k4QvwreMw504HvhPHWRS6iMNfOkxKQMrlL1YUZVKwpUHLb9Fq6N4FkoCfSimfudzzhBBBQghf88duwNVAJlrQt9l82X3Ah+aPt5o/x3x+j7z06zVFURRFGdqerGp0Quvy6OPuxMtfWcRDa+IIvribXKckmu4/BKv/Ey58CH9aBEefv5SaN1UcK9RSU4es14tYBG6+V3zPtTOCOVpQR3u3+Wvh5AYzrofMrXZNfW3rMpJW3MCapCFSOIVeG7MwCWyYGYIQsCPD/AsBvQHm3wN5n2JqKGHn+SrWTA/q26kDyPoYeru1Bje2Coi378LHmZvB7fIXKYoyadiys4eU8lPgv4EngTQhhP9lngIQBuw1z+Q7AXwqpfwYrdnLI0KIPLSavJfM178EBJiPPwI8dkXvRFEURfm3tDermvlRfvh5aK3i9TrB48u9mKMr4L2OefzjZBWs+zF88yhELIQdP4SPvjPBq74yRwvq8HQxkNK/Xq+1Wtsdu8IUTov1M4LpNpo4lNcviSb1Nq2urGDf2Bbcz5H8Onp6JWsS+wV7UmpBZcxKcLflRwrHC/Z2ZX6UHzvP99v9nXcPSEnVZ3+hsrlz6BROv1gInz++i1UURbGRLd04HxJCVALngJNAmvnPEUkpz0kp50kpZ0spU6WUvzQfL5BSLpZSJkgpvyil7DIf7zR/nmA+XzC2t6YoiqJ83lU3d5Je3sRacw3aJeaaqIuh6/kk3dz9MCAevvweLPiq9kN6ez1TxbHCehbG+GHQ9/vfdt5u7c+Eq0d1z4Ux/ni6GKzr9uLWgquvXVM5P8upwc1Jz4IYv76D1ZlQlzdpUjgtrk0J5fzFZkrrzY1r/KIhYT0e59/ARWdi/YyQvotbqqBwP8z64hU1x1EURRlPtuzs/QBIlVLGSCnjpJSxUso4Ry9MURRFUS5nn7nmbN3AYC/zIwhMYs7cRWRVtlBQY67DEgIWPaCl3qVvYSqoaekir7p16Pl6HsEQOntU93U26FiVGMjerJq+piQGZy0Ay9qm1Trawf7cGpbHB+Bi6Jf+mPkRIGDGjXZ5DXux7Nz1392T8+/Du6eGh6blXxqgDmidS6VJGwKvKIoySdkS7OWjDVJXFEVRlElld1YVYT6uzAjtNxGovR6KD0PyDVybqv3wvj2jX2pe6Cztcea1cV7t6By/VK/XL93R1Av5u7Xh3zqbKjKGtHZGMJXNnVyoaO47mHobdLdozV9s1GXspbNnwMiG3h7Kcs9RXNfG6oEjFzK3QtRS8AphMokKcCc5zNsq2Mv1XUm19OV2sdv64vR3tL9HQUnjvEpFURTb2fJ/iMeBw0KIF4QQf7Q8HL0wRVEURRlJl7GXg7m1rJ0RbN0KPXs7yF6YcQPTfN2YF+XL9owBg6znflmrd6s6P76LHoVjhXV4OOtJDffpO1h+SqutSxhdvZ7FVeamKXuzqvsOxqzSdgxtSOXsNpp45UgRK369h4VP7OKVI0X0mqQWKD63nIjXVvGu88/Z4NmvMqMuH6oyIHly7epZXJMSwsniBmpaugDYkVnHO71rCK89CE1l2kX1BdrYi1lfnMCVKoqiXJ4twd4LwB7gKFq9nuWhKIqiKBPmRGEDbd29rEsaWK/3MXhHwLR5AFyXGkpGeXPfAHHQfkjXOcGZ18dxxaNztKCOBTH+OFnV630KQgfx68Z072AvV2ZH+LCnf7Cn00PKJsjZMezwb5NJsvXsRa7+w2f89MPzxAd5MjfSl39u3cnZp9bDa5vB1MsHvvcRra8l7L1b4LXbteA6c6t2k0ka7F2bGoqU8OkFrZZxR0YlGaGbENIEp1/VLko3B8Ipt07QKhVFUWxjS7DnJKV8REr5spTyH5aHw1emKIqiKCPYk1WNs0HH8oR+tWzdbZC/RxshYN7tuy41DMB6d88jAJKuhXNvQW/PeC77itS1dpFT1Tp4vl7eLghfaJdOlutmBHO6tJH6tn7jFlJvA2Ontks6wLGCOm569iDfeeM0bk56Xv7qIt68J4l/hr7NTtfHSejJ4r+N9/CbuL/zo/qNPDvrHVj/Myg5Cs+tgAN/0AJx36gxr90RkkK8iA5wZ8f5Skrr27lQ0cz82XO1wPrUK1oKbfo7ELUcfCMnermKoigjsiXY2y6E+LoQIkwI4W95OHxliqIoijKCvdnVLI8PwN3Z0Hcwb5cWpCTfcOlQpL87s8J9+CRjwED1uXdDW80V1aaNt+NDzddrq9XSOMeYwmmxbkYwUsK+7H67exGLtd3RAamcrV1G7v3bcRraevj97XPY9p1VrE0KRnz0HUTay+gWPYD89ila536d5w6W0t7dy/IZUbDqEfjuGVj+be37M/duu6zdEYQQXJsSypH8Wt45WQqYG7cs+Ao0l8PBP0BttmrMoijKlGBLsPclzHV79KVwXnb0gqIoiqI4SmFtG4W1bUN04fwY3Py1XZd+rpsVytnSRsoa+qVyJnxBq02bxI1ajhXW4+akZ3ZEv3q9/D2AHPV8vYFSp/kQ6OkyIJVTB6m3aOMd+o2oOJpfR5fRxP9sns2t8yPQ64SW6pnzL1jyEGz8H3wCQvnN5tm89fWlPLgqllWJgdqT3f1hw3/Djy7Coq/ZZe2OsiEllJ5eyfP7C0gO8yYqwF0b/u4RDHt/BToDzNw00ctUFEW5rMsGe+ZRCwMfavSCoijKJCelZHdmFbWtXX0HjV2w+5ew6+cTti57sAQma/vX6xm7IcRwB8QAACAASURBVGcnJF0HeoPV9ZZUzh39d/f0TjD7dq02ra3W4WsejaMFdSyM8eur1+tu18YiuAdC2Dy7vIZOJ1ibFMRnOTX09Jr6TqTeBqaevho74GBeLa5OOuuZeTk7obcLkq1n5i2JC+DH18/E1UlvdRy9YdLPpZsX6UuwlwvdRhPXWgap651g3t3auIX4dVoqsKIoyiQ3+n7NiqIoyqRV19rF1/5xkgf+cZKnPskyH8yHl66GA7+Dg/8LjSUTu8gx2JtVTWKwJ5H+7n0Hiw5AVxPMuGHQ9bGBHiSHeVuPYAAtndBk1GqwJlh7t5GSunbSihv4V3oZn378JrfWPs/P25+CF1bD03HwZBhc+AASrx7TyIWB1ieH0NJp5ERRv0HzYXMhIBHO9X1t9ufWsDRuiJl5niEQucRu65loOp24NHPPMr4D0FI5nTxg/n0TszBFUZQrZLj8JYqiKMpUsj+nhu+/c5amjh4Sgj3ZnVWF8fTrGLY/qqWfbfwtfPIDOPsmrPnPiV7uFWvtMnKssI77V8Ran8j6WPtBPH7tkM/bmBrK7z7NobKpk1AfV+1gyEytWcjp12Dpww5euaan10RuVStZlc1kVbaQWdFMdmULDS1tLNed5zrdMTboT+IvWunUO4EpFjyiYdp8rSGIT6Td6vUsViUG4qzXsTuzmuXx5rRLIbSdz72/gqYyymUABTVt3LW4X2OVng6t5nHOHXYNPieDb61NYFa4D9NDPPsO+sXAY8XaLp+iKMoUoII9RVGUz4kuYy//syObvx4sJDHYk1fuX0xZZTUt730Xw4cHIXoF3Poi+ERoqXlnXoPVj076lLqBDubW0NMrWdu/Xs9kgqxPtCHjTm5DPu+6WWH87tMcdmRU8JX+geLcu7Xgt+IchM126NqllHzpxaOcLG4AwFmvIzHEk2+EZnGX/DWuxhaMBg9ao6+mYebNeKZswMnV8zJ3HTsPFwPL4gPYnVnFT65P7ptbOGuzFuylb+GgizZmwGpAev4e6GmbtGMUxiLUx5XbFw3RbVMFeoqiTCEjBntCCB/gWiDcfKgc2CmlbHT0whRFURTb1bZ2cd/fjnP+YjP3LovmRxuTcaWbpHfvRuqy2R36AOvv+x9thhpoAc77D0HJEYhePvLNJ5k9WdV4uRpYEN2vbqw8DVorRww6EoI9mR7iyScZldbBXuptsPNHWvDr4GAvq7KFk8UNPLgqljsWRRIT4IFBJ+DPj4JPCGz4C4a4tfg6uTp0HUP5wswQ/uuDDPJr2kgINgeY/nEQsQjS32G/z2pCvF1IDO4XfF7YCq6+2iB2RVEUZdIZNudCCHEvcAq4CnA3P9YCaeZziqIoyiTx+09zyKlq4S/3LuSXN6dqTTEO/wldbRbPhz3Bj+o3Yur/T37yjeDspaUvTiFdxl52nq9i3Yxg6yHjZ98AvQskbhjx+delhnGiqJ7qls6+g+7+WqfFc29rDWwcaHt6BToBD62JJyHYC4NeB2UnoSZTG0uQdB1MQKAHsN68U7ors8r6xKzboSqDytxTrEoM6tv1M3ZDznbta6d2uxRFUSalkRLsfwwskFI+LKV8wvz4BrAQ+Mn4LE9RFEW5nMLaNt46Ucpdi6O4emaIdrCpHA7+HpJvYtqSTVQ1d3GmrF9ShrMHpGyC8+9DV+vELHwU9mRW09TRw23zI/oOdjRq9YezNoOb74jP3zgrDClh5/kBAc2C+6CjHs5/4IBV9/kko5IlsQEEerr0HTz9ilZrmHqbQ1/7cqb5ujEzzJvdA4O9lFuQQs/6ns/6xigAFO2HziaYad2FU1EURZk8Rgr2BCCHOG4yn1MURVEmgd/9KxsXg47/WJfYd3DXz8DUCxueYN2MEAw6wc7zQ3Si7Gmzaq0/2W1JKyPU25UVCf2CjjOvae9j8f9n777jsizXAI7/bvZSNogMQXGBC8E9cmtuzZy5smzYqWx4Oo3TOO1hWZZmSzM1Tcud5t4DN05ARQFlKwIq633OH8+rgCCJMu36fj7vB7if533eCx5f5fK+7+ua+LfPr+duR21XW1YevlDwgF8nvfLk3lmlG3A+EfFpRCak07txvuqOmWkQtkTvaWdZrcxe+051a+jG/nOXuJSRlTdo58p5x9b0N91JuzpOeeMnVoCFHdQuuiCOEEKIildcsvcecEApNUMp9arxMRN9aed75ROeEEKI4hyNTWXlkYtMaO+HazXjbNH53XorgXbPgmMt7K3NaVPHmbVH49C0fP+H59Na35N1aH7FBF9CiWmZbA5PZGCQp97MG/SEdu8s8G4NNZv97TWUUgxq5smesylEp+RrsG5iojf6jt0HsQfKJP7VYXEoxc2S/oA+s5qdAUGVY3dE14buGDTYdCqhwPhyQzu8VBIuKQf1AUOu3u+vbo8KW3YqhBDi79022dM0bQ76ks0tQKbxsRkI0TRtdnkEJ4QQongfrz2Fg405j3esrQ8YDPDnFKjuCe0n3zyvV6MaRCVfJTw+35JNpaDZSL0/3aWo8g38Liw7FEuuQWNIsGfeYMQ6PfZWfz+rd8Og5vrzfz8QW/BAsxH6csrQ70sh2sL+PHqRFrWccKueLzk68DO41AfvlmXymiXV2NMe12qWBfbtZWTm8H1iQ7JMrPR9jaD/h0JG4n1ZhVMIIe4nxTbF0TTtErAp/8M4JoQQooLtPJ3E1vBEJnXyp7qVsUDGoV/g4mHo/o6+L8+oe4A7SsGaW5uKNx0BKDi0oPwCv0tLDsTS1Msef7d8yx33fgvVPKDhne8b83K0oU1tZ34/GFNwptPKXu8Xd3QJXE25/QXuwunEdE7GpfFg/iWc8cchJhSaj6k07S9MTBTdGrqxNTyJrBwDAHvOJpOaa0mqTw99JjInS1/6ewcFcYQQQlSs4qpxNlNK7UafzfsI+BjYopTarZRqXk7xCSGEKIKmaXy85hQe9laMblNLH7yeChve0Zc03lLsw62aFcE+jqy5dd+evRfU7gSH5+uzgpXUsQupnLh4hYeC8xVmSTyl93kLmVDiapAPBXtxLvnqzX53N7V4HHKuw8G5pRB1nhtJdq9G+ZK9g3PBxByaDi/V17pXXRu432xcD7AtIgkrcxPsW42C65ch4i99v55/V7As+x6AQggh7l5xM3uzgec0TWuoaVp3TdO6aZrWAHge+KlcohNCCFGkv47Hcyj6MpO71dPbLABs+RgykuDBD4ucKerVqAYnLl7hfPLVggeCHoHL5+Hc9nKI/O4s2R+LuamiX5OaeYN7Z4GpBQSPK/H1HmxUAxsLUxbviyl4wD0AarXXl3Iacu8t6HxWh12kuY8DHvbGhu85mXq7iAZ9wNal+CeXs3b+LliambDhhL5vb1tEEi39nLGo1xVsnGHD23AltkSzqUIIISpGccmeraZpe24d1DRtN2BbxPlCCFF1aEUVG64acg0an6w9RR1XWwYb959x8TDsmaknbjWDinzejcIghapyNugDltUrbaGW7FwDyw7F0q2hO462Fvrg9VR96Wmjh8DOtcTXtLU048FGHqwKu8i1rFuSupaP68lvxLp7C9xggDObiY5L4tiFK/Ru7JF37ORKuHZJX8JZyVhbmNLe34X1J+K5cPkakQnpdKzros+eBg6GpHAwMYP6vSo6VCGEEH+juGTvT6XUKqXUMKVUW+NjmFJqFbCmvAIUQoh7kZNrICc33/JETYPVL8OXQZCVUXGB3YPF+6OJTEjn5Z719abcCSdh7iCwqwFd37zt87ydbAjwqF442TO3hkaD4fgyuH6ljKMvuS2nEknOyCrYW++gsd1Cqyfu+roPBXuSnpnDX8eLSH6reZSoDYOmaVzNysk/AH+9Dj8PwGF2e3qYhNIr0D3v+IG5YO9dadsWdG3oTsyla/yw/SwAHeoaE+omQ/WPfh3B2rGCohNCCHGniqvG+SwwHegM/Mf46Ax8rWnaM+UTnhBC3L0Ll6/R+bPNvPJ7WN7ghrf1X+IvndXbE1QxEfFpvLPiOCG1HPWZuuTT8HN/faZl7PK/neXqGViD/ecvkZB2veCB4HGQfRX2zy6z2O/WkgMxONta8EB94/dmMOj30KvlbWcx70RrP2c8HaxZvP+WpZym5hDyKJzeoP98/4bBoDF54SGC/7eeVUcu6oM7voDdX0Pjh0nOtmSWxed4rR6jX+9SFJzZpM/CmhRbJ63CdG3oBsCcnVG4VbOknrtxb55XC/3PSrvnKi44IYQQd+zvqnH+qWnak5qm9TM+ntQ0bXV5BSeEEHcrOT2T0T/sITrlGisOXyA9Mwd2TIPtn0PweHBvDHu/q9zLOXOzIWYfpOkzT1euZzNx7n6sLcyYPrI56vJ5mNMfDDkwZjk41/nbS/ZqVANNg3XH4wseqBkEfg/Arq8h+3rRT64Al69mseFEAgOaeWJuavwnK3Kdnqzfw6we6JUnBzf3ZEdkEnGpt3zPzcfqxVPuoA3D5+vDWXroAo425kyaf4A1P38M69+CRkOI6fwFXTP+x3b/lyB6L3zTGn4bDyi9qX0l5V7disae9uQYNNrXdUHd2AOqFPSbphf1EUIIUekVV43TXin1oVLqhFIqRSmVbPz8Q6WUQ3kGKYQQJZF2PZtxP4USc+kaU3rVJzPHQMSfX8O6/+p7jvp8Bi0fg/ijer+wcpaemUNoVAqRCemkXs3OK/+vaZAUAXu+hfnD4SM/+L4rTA1AmzeUn3/4mospV/hmVHNqqBSY0w+y0mD0UnBrcEevXc/dDj8X28ItGAA6vAjpcXplzkpixeELZOUaeOhGb73Tm2DNK/oyy4AB93z9wc29MGjwx8Fbeu5Vc9evf3Besct9f9sXzVcbIxkW4s2mlzvxTr2zdD/9PmFWIaT2+pI1xxLIxRSvXi/AM/v0P38XDoB/N3Dwvuf4y1K3hvqy0451S74nUgghROVgVsyxRcBGoLOmaXEASqkawDjjMWmuI4SodK5n5/LYnH2cuHiFWWOC6VTPjejtC2h6aKr+C/agb8HEFBo/rCd/e2dBrTblGuN7q06wYO/5m19bmJoQYJvKN7lvUzP3gj7o6AdNHgbfDhB3hIw9c3kmey0T7JyxjhgJK9foveDGLAOPJnf82kop+jWtyVcbIziXnEEt53z1tvw6gmewPgMaNAZMi/sn4u79svsckQnpvNSzPnaWxtfIzYHEE+BSD8wsAX0f3OL9MTSoUY1A83iYPxHC14BDLf0+lrDdQlH8XGwJqeXI4v3RPPlA7bwZLICWE+HoYr1wTcvHCz135+kkXv0jjHb+zrw7qBHm0bsYHfsOyQ6BjEqchMvMUMxMFQEe1fF1sQVsYfC3+hJIO7d7jr2sDW3hxdmkdLo0rPyxCiGEKFpx/5L7apr2Uf4BY9L3oVJqfNmGJYQQJZeda+CZ+QfYG5XCF8Oa0aWBO5zeyLu5X3DQ4E/dAT9S3cxYzdHCFpo9ojflTouDajWKv3gp0TSNjSfjaefvzNAQbxLTMklKz6L5mRnUSLzI69njueDcjrEPdqKjcfncOpN2PJnWnNfqxTDeehvs+kZPiB75HbyCSxzDqFY+fLMpkp93neONvgF5B5SC9i/AwlFwfCk0HlKK33meGZtPE3v5GptOJTBteBDNvB1g/0+w+iW9Ubdnc1JdQ/junDvJsXZ8578bvvlNv2fd34FWT95MCEvDQ8Fe/Of3MA7HpOqx3ODdErxbwY4v9X1q+ZLLyIR0npy7H19nW74ZFYx50klYMALlWAuX8cv5Lh6enneA5IwsXupRr+ALugdQFXjYW/PF8LvfEymEEKLiFbdn75xSaopS6mb5MKWUu1Lq30B02YcmhBB3zmDQ+PfiI6w/kcA7AxoxoJknJIbDwjFkOvgzPusl1oWnFXxSiwn6frf9c8otzvD4dOKvZNK/aU0GNPPksQ61eeXBBvRgN6pWW1o8/DIROc6M/XEvo77fw6ojF5m88BCBXk6MHP0EasQCePEkPL37rmck3atb0buxB4tCo8nIzCl4sH5vcG0A26aWyX7GC5evEXv5Gg819yInV2PIjJ18vSkS7cQKcKhFbsvHib+Uhs2+b3gp6XW2Wz5Pg5hFerL17EF9VqwUEz2APk08sDQzYcmthVqU0pe2pp6HsMU3h5PTMxk/ey8WZib8OK4F9lZmsOJZMLPQE3BbZ1rVdmbFv9ozsWNtRraqVarxCiGEEHequGRvGOAMbDHu2UsBNgNOwNByiE0IIe6Ipmm8s/I4vx+M5eWe9Rndupbeh+3XEWBmifXYxVRzcGVV2MWCT3Suoy/t3PejXgylHGwJ1xtVd6yXbx9UwklIPIkKGMiAZp5seKETb/UL4FRcGpPmH8DSzISZjwTnNU+3cwPHe0sgxrXzJS0zh99v3atmYgLtJ0PCMQhfe0+vUZTQqBQAxrfzZfVzHejVqAYz1h4k9+x2zrr3oM+JnrRKfJUXai8jechi6PEe6qmd0HdqmTUfr25lTo/AGiw/fIHr2bf03KvbQy/ms32qXgUUeH7hIRKuZPLdmBC8nWzg6BKICYVubxXYh1fTwZpXezfE6UZvQCGEEKKcFdd64ZKmaf/WNK2BpmlOxkdD41hKeQYphBDFmbYhgtk7o3i8gx9Pd6qj/1K+5HG9xP3Qn1EO3vRp4sG2iERSr96S1LV4XC9KcnJlucS6NTyJeu52eNhb5w0eXwYoCOgPgIWZCePa+bFlSmde79OQ2eNbUtPBuugL3qUgbweaetkze8fZvAIxNzR6COx9YNtnpT67t/dsCnaWZjT0qI69tTlfjQjiu3ZXMCOXKUc8SL2WzazRwXw1tgPOjbpD22fArWGpxlCUES29Sb2WzfLDFwoeUAo6vKA3Ej+5grCYVLZFJPFSj/oE+ThC9jW98maNxtB0RJnHKYQQQpTEXTX4kT17QojK4qcdZ/lifQRDQ7x4tXdDvcDG5vchYi30+hB82wHQp7EH2bkaa29toF23u17wY+93ZR7r1awc9p5NKVzd8PhS8GlTaN+gnaUZj3WoTWMv+1KPRSnFuHa+nE7MYHtkUsGDpubQ7lmI2QvndpTq64ZGpdC8liOmJupmHG1y95Nr6UDvB/uz7oUH6BFYPvsn82tT25n67tWYvSOqcPIbMACc6sC2z5i94yw2FqYMa2mcwdv9DaRGQ8/39cI/QgghRCVyt91c3y7VKIQQ4i78fiCGt1ccp1dgDd4f1FhP9I4vg62fQNBoaPHYzXObeNnj7WTNyiO3LOU0MdX37p3bAfHHyjTePWdSyMo15DUHB31fYcJxCBxYpq9dlN6NPXCxs2T2jqjCB4MeAVtXfe9eKbl8NYvw+HRa+jrmDRoMEL4W07rdGN+xbl51znKmlGJsW1+OX7xCaNSlggdNTPWlrRcPcylsDUOCvahuZQ5p8frPp34fvZKpEEIIUckU12fvyG0eYYD77Z4nhBDlYd3xeF5efIR2/s5MG9EMM1MTiD8OfzwFXi30Xnr5yugrpejTuCY7IpO4lJFV8GJBo8HMqsxn97aEJ2JlbkILX6e8weNL9Y8N+5XpaxfF0syUka182HgqgaikW3rJmVtD66fh9Aa4cLBUXm+fMYkq8P1fOABXk6Ber1J5jXsxMKgm9tbmzN55tvDBJsNIs3TnCZM/GNPGVx/b9B7kXIce/yvXOIUQQog7VdzMnjswBuhXxCO57EMTQoiiHYq+zKT5B2jsac+s0SFYmpnqja9/HQmWdjB0bpEVG/s28SDXoLHm2C1LOW2coNEQOLJIL+xSRraGJ9LKzzmv0AroM5HeraF6zTJ73eI80soHU6X4ede5wgdbTACLanqT91IQGpWChakJTfO3NwhfC8oE/LuWymvcCxsLM4a38GbtsXguXL5W4Fi2MuPbnL60MjmJ/7UjEHcUDs7Ve/E516mgiIUQQojiFZfsrQTsNE07d8sjCr0qpxBCVIifdpzF1sKUn8a1wPbGsr9jf8ClszBoJlT3KPJ5gTWr4+tsw8ojFwofbDURsjNgy8dlEnN0ylXOJGXwQP4qnEmREH+0QpZw3uBW3Yo+TTz4bV806be2YbCyhyZD9Z/ttUtFX6AE9kal0NjLvmCyG75G72Vn43T7J5ajR1rXQtM0ftldMPldczSO7zPak2XppBeuWfuq/vN5YEoFRSqEEEL8veKqcU7QNG37bY6NLLuQhBDi9rJyDGw8kUCPgBo45i9pf3AeOPtD7c63fa5Sir5NarLrdDJJ6ZkFD3o0heDxesGNmH2lHvfWiETglpYLx//QPzbsX+qvVxLj2hrbMByIKXwweJy+VPHwwnt6jWtZuYTFpBZcwnnlAsQdgXo97+napcnbyYbuAe4s2Hu+QBuG2TujqOHsiFm7ZyByPZzdAp3+A9aOxVxNCCGEqFh3W6BFCCEqxM7TSaRl5tCzUb6tw0mRcH4nNBtVYJ9eUfo08cCgwZ9H4wof7P4OVPOAZZMgJ7Pw8Xuw5VQing7W1HG1zRs8tkyf1bL3LNXXKqkgH0eaejswZ2cUBsMtlSg9mkDN5rB/9j21YTgUfZkcg0ZLv3zJUcRf+se6lSfZAxjX1o9LV7NZfkifAT4Sc5n95y4xpo0vJi0fA0t7cK4LIY9WcKRCCCFE8STZE0JUKWuPxWNnaUbbOvkabB+ap+/7uoM+Zw1qVKOOqy2rilrKaVUd+k2DxJN6Rc9Skp1rYOfpZDrWc9UrhgIkn4b4ML2sfyUwrm0tTidmsOdsEW1Ug8dB4gmI3luyi165AEd+g4xkQqNSUAqCa+Wb2Qtfq/fzK4c+eiXRurYT9d2r8dNOvQ3D7J1R2FqYMiTES1+6OXYZjFyot6gQQgghKjFJ9oQQVUauQWPd8Tg61XfN2/dlyIXDC8C/+2336uWnlGJAM092n0lh/7ki9qHV7Q5NR+ol9S8eLpW4D5y7RHpmDg/Uy5eg3qjCWUmSvV6BHthamLL0YGzhg40e0gu17J9d/EUMuXB+N2x4B2a2h6kN4ffH4JfBHDkTS333athbGxOk7OtwZrO+hPNvZmPL240ehCcuXuHPo3GsPHwxr90CQM0gKcoihBCiSiizZE8p5a2U2qSUOq6UOqaUes447qSUWqeUijB+dDSOK6XUl0qpSGOLh+ZlFZsQomraf+4SSelZ9GqUr+n26Y2QdhGCRt3xdSa098PD3orX/ggjO9dQ+ISe74Gti76cMzf7nuPeGpGIqYmirX++ZO/YUr1FhL3XPV+/NFhbmNKrkQerwy4W2KsG6BVOmzwMx36/faGWyA3wiT/82BO2fwGW1aHbW9BvGlrcEUbFvEPLWvmaw0dth+yrlWq/Xn4Dm3lib23Oi4sOk5VrYExb34oOSQghhCixspzZywFe1DQtAGgNTFJKBQCvABs0TasLbDB+DfAgUNf4mAjMKMPYhPhnunYJwv/SZ15WToaMpIqOqETWHovDwsyETvXd8gYPzgUbZ6j34B1fx9bSjLf6B3IyLo2fdhTRU83GCfpMhbgw2DHtnuPeGp5Ecx+HvJmhlDN6YZKAiqvCWZRBQZ6kZeaw8WRC4YM3CrUcWXRz6Hp2Ljsik5j5x3rS543m9DUbLnafAVNOw/jVeiPy4HFcaP0mndV+xqXn62MYvgbMbcC3Q9l/Y3fB2sKU4S29uZadywP1XKnjalfRIQkhhBAlZlZWF9Y07SJw0fh5mlLqBOAJDAA6GU+bg97G4d/G8Z81TdOA3UopB6WUh/E6Qoi7FRcGoT/oy+sST+hjJmaAguhQGLu80pS9L46maaw5GkcHfxfsbrRbyEiGk6uh5eNgZlH8BW7RM7AG3Rq68/m6CHo39sDL0abgCQ37QuBg2PIRNOgLbg1KFrAhF+KOkHFqEwPj99La0wJ++w6y0uHyef2cgIqtwnmrNnWccatmyR8HY+nd+JYlsR5N9eWL+2ez03kwM7acITQqBbKv8YfFm2CqeE69grbfmz9a2ZP/bqyxHQA5u5lwei7sbgStntD369XuBOZW5fgdlszYNr78GRbH051kyaYQQoiqqcySvfyUUr5AELAHcM+XwMWhN28HPRGMzve0GOOYJHtC3AFN0/hqYyTnU67eHLPIvcqUyNHYchVz3zb63iuf1uDZHM7vggUj4JeHYMxSvfBEJXbswhViL1/juW518wbDfgNDNgQ9clfXfHtAIN0+28Jby4/x3ZiQvOIpN/T+RC+xv/xf8OhaMPmbxRApZ+H0BjizBc5uheuXsQWGmlpjlVYdcqqBhR3YuED7F8DB567iLiumJooBzWoye2cUlzKyCra2AL01xYpnmTV/IRFmDRke4s2TyR/ifj4aNXIx/8pqxBNz9zNtQzgv98xLjkPPpnCi2gQm1DKFNa/o/QxTz0OHF8r5OyyZmg7WbJ1y+1YeQgghRGVX5gValFJ2wBLgeU3TruQ/ZpzFK1Etb6XURKXUPqXUvsTExFKMVIiqbePJBKauC2dLeCK7Tiez63QyARHf4pCTyNCMKWwKmQEPvAx+HcDCFvy7wdCf9eWE8x6GzPSK/haKtfZYHCYKujU0/v+QpulLOD2agXvgXV3T08Gayd3rsv5EAn8djy98gq0L9HgPYvbCgTnFXivr6Aq0r5rDqhfJit7PJZ8exHSexss+i3jAbC6mL52Cf+2HJ7bA+FXQ7c27irmsDQryIjtXY2VYEf/P1ughskxt6JO1lpmjg3nLfRs1zq9AdXkN6najZ2ANhoZ4MWPzafZF6VU9NU0jNCqFYF9XGPydPju44R39epV0v54QQghxvyjTZE8pZY6e6M3TNO1343C8UsrDeNwDuLE5JBbwzvd0L+NYAZqmzdI0LUTTtBBXV9dbDwvxj5Rr0Ph4zSl8nW3Y+UoXdrzShR0TPHlEW0FO01Fcc2/OC4sOcTH1WsEn1n8QHvoBYkJhwXDIulr0C5SH2APwWQNY/XKRcaw5GkcrP2ecbsw2XTwM8UfvelbvhvHt/GhQoxpvLT9GemZO4ROaDtf3iYmRHAAAIABJREFUla1/E9L1v66ycw2sPx7P9I0RTJp/gGGfLCHjtyc5muvDA5lTqZf0MUFHBtL+T1d+C8/hgXqumJhUroqTt9PQoxr13asVWZUzOducpTltGWC+h2YZO2Hta1C/D7R/8eY5/+0XiKejNS8sOkx6Zg5nkjJIzsiihZ8TWNjoLQvsfcCrJVSvWZ7fmhBCCPGPU5bVOBXwA3BC07Sp+Q4tB8YaPx8LLMs3PsZYlbM1kCr79YS4M8sPx3IqPo0Xe9TH3NREn/Va/TJY2GLW4x2+HtWczBwDzy04RM6t1ScDB8Kgb/XqiAtH6SXxy9juM8m8sPAQl69m6QOJ4TBviF4AZO8s+LYDxOy/ef7pxHQiEtILVuE8NA9MLaHxkHuKxdzUhPcHNybuynU+Xxde+ASloO/nkH0N1r4KwGt/hPHYz/v49K9wwqJTeCv3K+xMc4jv8Q2vPtKHb0eH8O3oYL4dHcys0cH8t2/APcVYnpRSDAzyZP+5S5xPLph0f7P5NHOzO2OhZcKvI8HJDwbNKLC81c7SjM+HNiPm0lX+t+I4oca+fS18jftC7dxg0m4YtQghhBBClK2ynNlrB4wGuiilDhkfvYEPge5KqQigm/FrgNXAGSAS+A54ugxjE+K+kZVj4LO/wgmsWZ0+N4pqHF+q7zXr8gbYulDH1Y73BjVib1QK0zZEFL5Ik6HQ/yu9jcHG/5VpvKfi0nh8zj5+PxjL+NmhXEs6B3MH6U3RH9sAY5brCecP3WHje5CbzdpjcQD0CDQu4cw2VoVs2BesHe85puY+joxo6cNPO85y7EJq4RNc6up77MJ+4+KB1SzeH8OoVj4cfbsnW9sdpeH1g5j3/YRu7dvRM7BGgUePwBqF975VcgOa6TNuSw/lze7FXr7G3F3nCGjeEWo21ytpDptX5F7PEF8nnupUh4X7ovl26xmcbC2o42qbd4KFbancNyGEEEIUryyrcW4HbrduqWsR52vApLKKR4j71YK954m5dI33BjXWlwpmpuvL62o0hpBHb543KMiLXaeTmb4pklZ+zrSv61LwQs1H63vT9szUy+y71KW0JaRd59HZoVhbmPJyr/p8sXw3KTOfpKZpKmrcKr1RtXMdeHon/PkKbP0YItZCWgfecU7HY/dOuJoMl6Lg+uV7XsKZ3797NmDN0TjeWn6MRU+0KVyspf1kCPsNsz9fpLrZRzzfrR52SUf05DhgAASNLrVYKlpNB2ta13Zi6cFY/tXFH6UUX6wLB4VeIMdkvt4jr5jG4s91rceW8ESOxl6hR4B74Z+nEEIIIcpcmRdoEUKUnYzMHL7aGEHr2k50vJG8bfsUrsRC78/AxLTA+W/1D8Tf1Y7nFx4kIa2I5Zpd3tBnbNa+VuqxXsvK5fE5+0jJyOKHsS0Y09yFde5f45J9kS9c3iHXvUneyVb2+vLAYb+Qezmap9OnMyZjNuz7EaJ26Esqm40CvwdKLT57G3Om9KxPaNQllh++UPgEcyti27+Pa/YFvvbeiKtFNix5DOzcod80fbnnfWRQkCdnkjI4HJNKRHwaSw7EMLZNLWo6WEN1j2ITPQALMxO+GNYMWwvTgn0RhRBCCFFuJNkTogr7cftZktKzmNKrgT5zkhQBO6dD05Hg06rQ+TYWZnw9qjnpmTk8/+shDIZbiuHauUHHl/XZtIj1pRanwaAxeeEhjsSmMm14MxrXsIZFo3FOPcrWph8x7bQ7ry8NQ5/gz6dhPxa0WUXb619ydmIkvHYRJofpFS0HflMomb1XD4d409jTnvdXnyCjiGItH5x0Y5nWgbZxv8Di8Xpz9MGz7sslib0aeWBhZsLSg7F8+tcpbC3MeLqTf4mu4e9WjdDXuzGipfffnyyEEEKIUifJnhBVVEpGFrO2nqFHgDvNfRzhagqseF6fmev+9m2fV8+9Gm/2C2Tn6eSiy+u3ehKc6sDa/0BudqnE+tGak6w5FsfrfQLo4ZUDc/rq+wP7f0X3wRP4Vxd/FuyN5sM/T3Ik5jKLQqN5e8UxRszazQfrz2Pr5otfzbKvvmtqonirfyDxVzL5elNkgWOn4tJYFXaR6JBXURa2EPGX3ifOt32Zx1UR7K3N6dbQjUX7oll7LJ6JHWvf1d5DGwszWcIphBBCVBBJ9oSoomZsjiQjK4eXu3jBlk9gWlM4vxN6vqfP0BVjWIg39dztmLY+nNxbZ/fMLPRrJIVD6Pf3HOecnVF8u/UMY9rU4lHXUzCzPcQf01s+GPfcvdC9HqNb1+LbrWfoP30HU5Yc4de90VzNzqVf05q8P7jxPcdxp4JrOTK4uSffbztLVFLGzfFpG8KxtTBjVJcQvXpp87HQ6T/lFldFGNjMk6tZubjYWfBoe7+KDkcIIYQQJVRmBVqEELeXdj2beXvOM7KVD9WtzPMOGHLvaGni/nOXmL/rNJ/VCqXugucgIxEa9IUur4Nbw799vomJYnK3ejw17wArDl9gYJBnwRPq9YI6XWDzB9B4KNg6l/RbJDvXwHurTjB7ZxQ9GzjxltVC1IIvwb0xPDwbXPKWBCqleLt/ICG+jliamdCgRnV8nGwqrDfdK70asPZoHO+uOs73Y1tw/MIVVofF8WwXf312q34v/XGf61TfjabeDoxpXQtbS/nnQgghhKhq5F9vISrA7B1RfLYunCMxl/l6ZHN9mVtWBnzdChxqwYCvwKl2oeedTkzn0zUnyT2xkrUW8/GKi4da7WD4fPBuWaIYegbWoKFHdaZtiKBvEw/MTPNN9CsFPT+AGW1h03vQd+rtL1SE5PRMnpl/kF1nknm+hTXPXfovatdevTpozw/A3KrQc0xMFAOaeRZxtfLnVt2KZ7vW5YM/T7LpVAIL9pynmpUZE9oXvif3MwszE5ZNalfRYQghhBDiLskyTvHPlBQBB+bq+9zKWa5B49fQaKpZmrE6LI5f9pzXDxyYC6nRcOEgzGgHe74Fg94APeHKdV79I4zHPv+NEZEvMsviczxcnWDUEhi3qsSJHujJ1fPd6nI2KYOlh4qoPunWAFo8Bvt/0pdd3qFjF1LpP30H0edPs77hap4/PhyVcFxfttn38yITvcpofDs/arvY8u/FR/jreDyPta+NvY353z9RCCGEEKKSkGRP/HNkJMOeWTCrM0wPgeXPwLcdIWZfuYaxLSKR2MvXeG9wYx6o58r/Vh7nWEwS7JoOPm3gmVCo1Rb+nII2py8L1m6h+ydrqXHgc/6ynEJ7y0jo+T6mT26Dut3uqeR/jwB3AmtW56uNEWTnGgqf0OkVvQ3CqpduJp63o2kaSw/G8tSMlTyfNYutlpPxj1oATR6Gp3ZC4yF3HWdFsDAz4b/9AkhIy6S6lRnj2/tWdEhCCCGEECUiyzjFfeV/K4+zL6rgbJ1nTjRP5/xM4NW9KEOO3my8x3v63raVz8OPvaD7O9D6qXLplbZg73mcbS3oFViDdnWc6f3lNpbP/YrAzGjo/SnYe8KoxWTu+xnDn/9hQNTDdLd0wCUnHgKHQI939T5npUApxQvd6zFhzj7+OBDL0Ba3lMi3cdJfb9kkPRlt92yha2TlGFhx+AI/botkcNJMNpitwwxQTUZAhxfBqeoW9uhU343nu9Wlvnu1gnsrhRBCCCGqAEn2xH1jz5lkfth+liZe9jgZS8QrzcBLcVNxzoxhdfVBdB/xHBY181V2fGIrLJ2ktxk4twMGfA3WDmUWY8KV66w/kcBj7f2wMDPB2c6SacOa4TjneS5Y++FRtzsKiL50jYk7a5N67UPm11xELbMU6Pkd1C69JuI3dGngRlMve77cGMHAIE8szG6Z8G82CsLXwIZ3oE5nPVkGUq9mM2/vOebsjCL+ynWmV5tLX7M15DZ9BNXpZXD0LfVYK8Lz3epVdAhCCCGEEHdFkj1xX9A0jY/XnsK9uiULJ7bB2sJY0TJsMSwJZ0+z95i0x48+m7L5coSG6Y0qj9aOMHwe7P4G1v1XX9Y5fD7UaFQmcf62P4Zcg8awfDNorXP3g0k0k9Oeos3+C3g5WjNp/gFyDRpfjuuFb/0xZRLLDUopJnevx7ifQlm8P4aRrXxuPQH6ToPotrDkcZi4iZ3nMnjs531czcqlvb8L8wL24H9oDbSfjGm3t8o0XiGEEEIIcWdkz564L2w4kcD+c5d4rmu9vEQvJxM2vA3ujWk14Gle79OQVWEXeX1pGJqWr7ecUtBmEoxfoz/n94l/uz/tbhgMGgv2nqdNbWdqu9rlHdj+BZq9F0m+fXl92VFG/7gXFztLlj3Tnk71i++XV1oeqOdKkI8D0zdGkJmTW/gEW2cY+DUkniB33Zu8sewobtUsWf1sB35pcxH/Qx9D4CDo8t9yiVcIIYQQQvw9SfZElZdr0Phk7Sn8XGx5OMQr70Do93D5PPR4B0xMeKxDbZ7p7M+CvdF8tOZU4Qt5t9CbiSccg7DfSj3O7ZFJxFy6xoj8M2fn98D5nag2z/DZiBBc7Szp3tCdPya1w8/FttRjuJ0be/cupF7n553nij7Jvxu0fALTvd/imbyT1/oEEJB7Cv54ArxawsAZYCJ/pQghhBBCVBbym5mo8pYdiuVUfBov9qiH+Y1ecdcuwZaP9cbgdbrcPPfFHvV4pLUPM7ecZsbm04UvFjgYajSBTe9CTlapxrlg73kcbczpGeieN7jjC30pafMxuFWzYtuUzswcHYxdBTSwbu/vQreGbnzy1yki4tOKPOdKh9c5gxfTrGbRrXoMLBgO1WrAiAVgbl3OEQshhBBCiOJIsieqtMycXKauC6eRZ3V6N8pXoXLbVLieqlfZzEcpxTv9G9GvaU0+WnOSHZFJBS9oYgLd3tRnBPfPLrU4E9Kus+54PEOCvbA0My4zTTgJp1ZDy4lgYWt8+bKvBno7Sik+GNyEapZmPPfrIbJyCi9lnbHjAv/KfBp70lE/dANDDoxaDLYuFRCxEEIIIYQojiR7okpbsOc8MZeuMaVng7xE6fJ5vSF50xE3K0fmZ2Ki+PThJrhVs2TmliJm9+p0Bd8OsPVjyEwvlTgX748hx6AxvGW+JZw7vwQza2j5RKm8RmlwrWbJhw814fjFK3yxPrzAsZhLV/lh+1nqN2uHSfe39diHzwOXuhUUrRBCCCGEKI4ke/9k+YuUVEHpmTl8tTGSNrWd6VA338zSxnf1oitdXrvtcy3NTBnb1pdtEUmcjLtS8KBS0PVNyEiE3TPuOU6DQePXvdG08nOijqud/nOPWAdHFkLzMXrxk0qke4A7w1t4M3PLaULz9Sz8ZO0pFPBSz/p6QZt/R4Fv+wqLUwghhBBCFE+SvX+S66l6krH+bfihJ3zgBSsnQ/b1io7srvy4/SzJGVlM6VUfdaMZ+sXDehLV+imw9yr2+aNa+WBtbsoP284WPujdAhr0hR3TICP5nuLceTqZ8ylXGdnSG079Cd93hXlDoJpHkU3KK4PX+wbg5WjD5IWHSLuezaHoyyw7dIHHOvhR08G4N8/MomKDFEIIIYQQxZI+e/eZ5PRMDsdcLjBWPekgAQf/h03KcdAMYGIGHs306or7foQLB+HhOeBYq4KiLrlLGVnM2nqGnoHuBPk46oOaBmtfA2snaD/5b6/hYGPBkGAvFoZG83Kv+rhVsyp4Qpc39D1126fqVTpL6GxSBj9sP8OS/ecZZrOPfrveh4Sj4OADfb+AZiPBzLLE1y0PdpZmfD6sKQ/P3MU7K45zLvkqLnYWPNXJv6JDE0IIIYQQd0iSvftI/JXrDPx6BxdT82bq6qoYFlu8RbJmy6k6jxPUoQ94hdwsCMKJlbD0KZj1AAz+Hup2q6DoS2bOrijSM3N4oXv9vMHDv0LUNujzGVjZ39F1xrfz5Zc95/hl1zle6FG/4EG3BtB0JOz9Dlo9CQ7eRV8kH03T2Hs2he+2nWXDyXjam55gvd2veF6PgFx/GDgTGg8BU/MSfLcVI7iWE0938mf6pkgA3hvUqEKqhAohhBBCiLsjv7ndJ65m5TBhTiip17L5YWwILnaWmKdfwH/liyiDLR+5fs78EyZ81qgOD9XO17+tYV9wawiLxuhLCzu9Ah2nVOp+aVezcpizM4quDdyoX6OaPpiRBGtfBe9WEPzoHV+rtqsdXRu4M3f3OZ7u7I+VuWnBEzq9AmGLYPOHelPx28jJNbD6aBzfbzvDkZhUmlgnsc5jCf4pW8DSG3p/D40Gg4npba9RGT3XrS7bI5PIyjEwLOTvk10hhBBCCFF5SLJ3H8g1aDy74BDHL1zh+7EhdGngDldTYNk4yMmA8av5r3MA5+aEMmXJEapZmdEjsEbeBZzrwIR1sOoF2PwBJJ6CIT/qhUoqoUWh0Vy6ms2TnerkDa59FTLToN+0Eieqj3fwY/2JeJYciGFUq1uWsjp4660Rdn0NLR+Hms0KHE67ns3C0Gh+2hFF7OVrNHaG1Q3W0vD8fFSahb4UtM2kKtuDztzUhMVPtiHHoGFmWnn/A0AIIYQQQhQmv73dB95ffYL1J+J5s1+gnuhlXdWbXV86C8PnQ43GWJmbMmt0CI097XlmwUF2nr6lv5yFDQycoScnx34vlSqUxTmdmE7nTzfzxtKjJKZl6oOaBhv+B3MHw/JnYesnxqWZ2yE9EYDsXAPfbTtLcC1HWvg66c+L3KAXZWk/WZ+lLKGWfk409rTnh+1nMRiKqFDa8WWwcYY/p9ysYJqVY+CD1Sdo88FG3l11Ai9Ha2YPr8ty81cIiPoZ1WQYPHsAOr5UZRO9G8xMTQrPeAohhBBCiEpPZvaquLm7ovhh+1nGtfVlbFtfyM2BxY9C9F54eDb4dbh5rq2lGT+Na8GwWbt4fM4+FkxsTRMvh7yLKQUdXtQLtqx7A7xb6vv7Stn17FwmzTtAwpXrLNh7niUHYnisvR+TzFdgue1TcG0AcUf01gc3mFpA1zdZZTWA2MvXeKt/oD6elaFXFHWuq8d+F5RSPNbBj+d+PcTm8AQ9Yc7P2gG6vQXLn9GTyqbDWbgvmm+3nqFf05pM7FCbxl72sHgCpMbC2BUFfu5CCCGEEEJUBJnZq8I2nUzgzeXH6NbQjTf6BkBuNiybBOF/Qu9PIHBgoec42lowd0IrHG0tGPvjXs4k3tI0XCkYMB2q14TfxunLQUvZ/1Ye52RcGtNHNmfdCw/QuYEbRzcvwnzLu5xx70XmxB3wciS8FgfP7IfRf0DdHvDXa/itHkE7l6t0beCmX2zzB3D5nL5809yq+BcuRu/GHtSobsX3RbVhAGg2CjyDYd1/0a6nMnvHWZp42fPl8GZ6onfkNzi6GDr9RxI9IYQQQghRKUiyV0VFxKfxzPwDNPSozrThQZhmZ+hLN4/8Cp1f1/eX3YZ7dSvmPdYKgMmLDpN769JFa0d9VjAtDpY+XarN11ceucC8Ped5omNtOjdww8/Flq+72/Kd7UzOmdeh97lhDJm5W1/aaW4NLv5QpwsM+4WTrT6gdnYks68/j8mRX+HCIX0vXfOx4NvunuIyNzVhXDtfdp5O5tiF1MInmJjoCXR6ArHL3uZ0Ygbj2vrq/f0uR8OqF/XiMHfQ8kEIIYQQQojyIMleBck1aHy85iTT1kew/9wlcnINd/zcK9ezmTh3P9YWpnw/NgTb7BSY3QdOb9RnuB54+W+vUcvZlrcHNOJw9GV+3F7EbJZnMPR4V58l3DW9JN/abZ1LzuCVJWEE+TjwUk9jm4Nrl2DBcEwtrPGdtJQvHmlDREIaQ2buJCopI+/JSvFWdBBjLadi6tEYlj4Js/uCjQt0f7tU4hvR0gdbC1Omb4ws+gTPYAh6hBonZhNim0CfJh5gyIU/ngQtFwZ9C6ayMloIIYQQQlQOkuxVkE//OsU3m0/z+fpwHpqxk6D/rWPiz/v4eVcUF1Ov3fZ5BoPG5F8PEZ1ylW9GBeORcwF+6K5X0By+AILH3XEM/Zp40D3AnU//OsXZ/InVDa2egIb9YP1b+h7Ae5CZk8sz8w9iaqL4akQQ5qYmefsLL0fDsF9QDt70auTBgsdbc+VaNg/N2MkRY4P4Q9GX2X0mhd4dWmMyfhV0e1tfctp3qj4TWQrsrc154oE6/Hk0jp2RSUWeEx08hQzNkk+rLcDS1ERPhM9thwc/Bie/UolDCCGEEEKI0iDJXgVYdeQiMzafZkRLHw6+0Z3pI4Po28SD4xev8N9lx+jy6RYWhp5HK2L55LQNEWw4mcAbfQNoaX5WT/Qy02DcSqjfq0RxKKV4d2AjLM1M+PfiI4UrUSoF/adDdU/4bfw97d/7YPVJwmJT+WRIE7wcbfTB9W/qs5F9p4JP65vnBvk4sviptlhbmDJ81m62hCcyc/NpqluZMbylj96rrv3z8Mp5PRktRRM71sbL0Zq3VhwrcrZ19qF0vsh9GN/Le2DLR3r10Ib9oNnIUo1DCCGEEEKIeyXJXjk7FZfGy4sP09zHgbf6B+Boa0HfJjX5YHATtk3pzPoXHiDIx4F/Lwlj0vwDpF7NvvncdcfjmbYhgoeaezEmwBTm9AXLanqPvLusmule3Yo3+gawNyqFX/acK3yCtQMMnQMZCfpyRcOdLze9YdmhWGbvjGJ8O1+9v19mOvzxlD4r1vIJaD6m0HPquNrx+1Nt8XW2ZcLsUNYej2NMG1/sLPMtkyyDPoBW5qa83ieA8Ph0ftld8OeRkZnDotBoLgeOBrdAvTiMjTP0nVZpexIKIYQQQoh/Lkn2ylHq1Wwmzt2HraUZMx4JxtKsYO8ypRT+bnb8MqEVrzzYgL+OxdNr2lZ2n0kmMiGdyQsP0djTnvcGNUI5eEPP9/REz7nObV7xzgwJ9qJjPVc+/PMk0SlXC59QMwh6vAcRa2HnlyW69rJDsUxeeIiWfk688mADiDsKszrB4QXwwCvQ64PbPtetuhULn2hNq9pO2FmYMa6db8m+sbvUM9CdDnVdmLounOT0zJvjvx+IIS0zh9Ht/KHPZ2BXAwbNAFvncolLCCGEEEKIklBFLRWsKkJCQrR9+/ZVdBh3JNegMWFOKDsik1jweGtCbjQEL0ZYTCrP/nqQqOQMnG0tMWgaK/7VHk+H0m/SHXv5Gj2mbiHIx5G5E1rqVSbz0zS9FcOJFTBuFdRqU/SFDAa9WImpOX8cjOHFRYdp6efEj2NDsDkyB9a8qu+xe+j7O25RYDBopGXmYG9tfm/fZAlEJqTR64ttPBzixQeDm2AwaHT/fAt2lmYsndRO//lomszoCSGEEEKICqWU2q9pWpHL/GRmr5x8sT6czacSebNf4B0legCNvexZ+a/2DA32Jj0zm+kjg8ok0QPwdLDmP70bsj0yiYWh0YVPUAr6fwkOPnpRlYzkwueE/wVfNoN33Un/OBCXP4Yzy2kecxvswWbpo3p7Ar+O8NSOEvWiMzFR5ZroAfi7VWNsW19+DY0mLCaV7ZFJeruFdr55ibAkekIIIYQQohKTmb0ylpB2na82RDJ39zmGhXjz4UONC8+a3YHsXINewbIMGQwao77fw+GYy6z4V3vquNoVPunCIfihO1c929EtbhKu9jb08tEYmjQd53N/gkt9jtt3JCL8GI2sk6ltmoC6dglMzKDrm9DmGb1nXRVw5Xo2XT7djI+TDfbW5oTFXmHHK50LLb8VQgghhBCiohQ3sydNwcpI2vVsvtt6hu+3nyUzx8Do1rV4rU/Du0r0gDJP9ECfQZs6rCm9p21j0rwDLJ3UDivzWxKbms243vVdbP6awijljuU1B4btm4M5OXyhDSfUfBQ7jqXxQL2h9BwdjDI31XvpGQxVbm9bdStzpvRqwJTFRwB4rmtdSfSEEEIIIUSVIcleKcvMyWX+nvN8tTGSlIws+jT24MUe9ahd1CxZJeRhb83Uoc0YPzuUd1cd592BjQscNxg0no1oTv/cNkwy/RXSIad2Z3Y1eJWUOBviTyczsFlNPnyoSV6iWEp98CrCkOZezNt9juMXrzCqlU9FhyOEEEIIIcQdk2SvlD234BBrjsXRto4z/+7VgKbeDhUdUol1buDGEx1r8+3WM7Su7UzfJjVvHvt6UyR/nUigQ69P4PIMqNMFs0YP0UEp7nwXXtVhYqKY8Ugw51Ou4lbdqqLDEUIIIYQQ4o7Jnr1Stv/cJTIyc+hQ1+Wul2xWBtm5BoZ+u4vI+HRWPtueWs62bDwZz4Q5+xjYzJOpQ5tW6e9PCCGEEEKI+4FU4yxHwbUc6VjPtconQuamJnw5PAil4Jn5BzkVl8ZzCw4R4FGd9wfdXZEZIYQQQgghRPkps2RPKfWjUipBKXU035iTUmqdUirC+NHROK6UUl8qpSKVUkeUUs3LKi5x57ydbPjk4aaExabSf/p2zEwVMx8JxtpCipQIIYQQQghR2ZXlzN5soNctY68AGzRNqwtsMH4N8CBQ1/iYCMwow7hECfQMrMGj7fzIMWhMH9kcbyebig5JCCGEEEIIcQfKLNnTNG0rkHLL8ABgjvHzOcDAfOM/a7rdgINSyqOsYhMl80bfhoS+1o12/i4VHYoQQgghhBDiDpX3nj13TdMuGj+PA9yNn3sC0fnOizGOFaKUmqiU2qeU2peYmFh2kYqblFI42VpUdBhCCCGEEEKIEqiwAi2aXga0xKVANU2bpWlaiKZpIa6urmUQmRBCCCGEEEJUfeWd7MXfWJ5p/JhgHI8FvPOd52UcE0IIIYQQQghxF8o72VsOjDV+PhZYlm98jLEqZ2sgNd9yTyGEEEIIIYQQJWRWVhdWSi0AOgEuSqkY4E3gQ2CRUmoCcA4Yajx9NdAbiASuAuPLKi4hhBBCCCGE+Ccos2RP07QRtznUtYhzNWBSWcUihBBCCCGEEP80FVagRQghhBBCCCFE2ZFkTwghhBBCCCHuQ5LsCSGEEEIIIcR9SJI9IYQQQgghhLgPSbInhBBCCCGEEPchpRfCrJqUUonoLRwqGxcgqaKDEHdE7lXVIPepapD7VHXIvaoa5D5VHXKvqob79T7V0jQTSxJhAAAH60lEQVTNtagDVTrZq6yUUvs0TQup6DjE35N7VTXIfaoa5D5VHXKvqga5T1WH3Kuq4Z94n2QZpxBCCCGEEELchyTZE0IIIYQQQoj7kCR7ZWNWRQcg7pjcq6pB7lPVIPep6pB7VTXIfao65F5VDf+4+yR79oQQQgghhBDiPiQze0IIIYQQQghxH5Jkr5QppXoppU4ppSKVUq9UdDxCp5TyVkptUkodV0odU0o9Zxx3UkqtU0pFGD86VnSsApRSpkqpg0qplcav/ZRSe4zvq4VKKYuKjlGAUspBKbVYKXVSKXVCKdVG3lOVj1JqsvHvvaNKqQVKKSt5T1UOSqkflVIJSqmj+caKfA8p3ZfGe3ZEKdW84iL/Z7nNffrE+HffEaXUH0oph3zH/mO8T6eUUj0rJup/pqLuVb5jLyqlNKWUi/Hrf8R7SpK9UqSUMgW+Bh4EAoARSqmAio1KGOUAL2qaFgC0BiYZ780rwAZN0+oCG4xfi4r3HHAi39cfAZ9rmuYPXAImVEhU4lbTgDWapjUAmqLfM3lPVSJKKU/gWSBE07RGgCkwHHlPVRazgV63jN3uPfQgUNf4mAjMKKcYRdH3aR3QSNO0JkA48B8A4+8Ww4FA43O+Mf5+KMrHbArfK5RS3kAP4Hy+4X/Ee0qSvdLVEojUNO2MpmlZwK/AgAqOSQCapl3UNO2A8fM09F9KPdHvzxzjaXOAgRUTobhBKeUF9AG+N36tgC7AYuMpcp8qAaWUPdAR+AFA07QsTdMuI++pysgMsFZKmQE2wEXkPVUpaJq2FUi5Zfh276EBwM+abjfgoJTyKJ9I/9mKuk+apv2laVqO8cvdgJfx8wHAr5qmZWqadhaIRP/9UJSD27ynAD4HpgD5i5X8I95TkuyVLk8gOt/XMcYxUYkopXyBIGAP4K5p2kXjoTjAvYLCEnm+QP8L2WD82hm4nO8fVXlfVQ5+QCLwk3HJ7fdKKVvkPVWpaJoWC3yK/r/ZF4FUYD/ynqrMbvcekt8xKq9HgT+Nn8t9qmSUUgOAWE3TDt9y6B9xryTZE/8oSik7YAnwvKZpV/If0/TStFKetgIppfoCCZqm7a/oWMTfMgOaAzM0TQsCMrhlyaa8pyqecb/XAPTkvCZgSxFLnETlJO+hyk8p9Rr6VpF5FR2LKEwpZQO8Cvy3omOpKJLsla5YwDvf117GMVEJKKXM0RO9eZqm/W4cjr8xZW/8mFBR8QkA2gH9lVJR6Mugu6DvC3MwLkEDeV9VFjFAjKZpe4xfL0ZP/uQ9Vbl0A85qmpaoaVo28Dv6+0zeU5XX7d5D8jtGJaOUGgf0BUZpeb3M5D5VLnXQ/7PrsPF3Cy/ggFKqBv+QeyXJXukKBeoaq5xZoG/QXV7BMQlu7vv6ATihadrUfIeWA2ONn48FlpV3bCKPpmn/0TTNS9M0X/T3z0ZN00YBm4AhxtPkPlUCmqbFAdFKqfrGoa7AceQ9VdmcB1orpWyMfw/euE/ynqq8bvceWg6MMVYQbA2k5lvuKcqZUqoX+paD/pqmXc13aDkwXCllqZTyQy/+sbciYhSgaVqYpmlumqb5Gn+3iAGaG/8N+0e8p6SpeilTSvVG33NkCvyoadp7FRySAJRS7YFtQBh5e8FeRd+3twjwAc4BQzVNK2pjryhnSqlOwEua9v/27idEizqO4/j7A5IKmosKXcUuQqLbpUsGFnTSg3WJKAihg3gI6lQE5d6EPEXQafdgiR48qOTaKYIQJIKVLVIv0kGDdD1UJC5Y3w4z1cO2f3RbfWLm/TrNzO/3/OY3PMzh88z3N0/tTbKV5knfRmAKeLWqZoc5P0GSUZoX6TwCXAX20/yI6D31P5JkDHiJptRsCnidZl2K99SQJTkO7AY2Az8B7wOnmOceasP6RzRluLeB/VX1zTDm3TcLfE/vAKuBW223C1V1oO3/Ls06vrs0y0bOzR1TD8Z831VVjQ+0/0DzduKZvtxThj1JkiRJ6iDLOCVJkiSpgwx7kiRJktRBhj1JkiRJ6iDDniRJkiR1kGFPkiRJkjrIsCdJ6qUkleTTgf1VSW4m+WyZ440kOTiwv3u5Y0mStBIMe5KkvvoN2J5kbbv/PHD9P4w3AhxcspckSQ+JYU+S1GeTwJ52+2Xg+F8NSTYmOZVkOsmFJDva44eSTCT5MsnVJG+0HzkMPJ7kYpIP2mPrkpxMcjnJsfZPfElyOMn37dhHHs6lSpL6ZtWwJyBJ0hCdAN5ryy13ABPAM23bGDBVVfuSPAccBUbbtm3As8B64EqSj4G3ge1VNQpNGSfwJPAE8CNwHng6ySXgBWBbVVWSkQd/mZKkPvLJniSpt6pqGthC81Rvck7zLuCTtt8XwKYkj7ZtZ6tqtqpmgBvAYwuc4uuqulZVfwAX23P9DNwBxpO8CNxeuSuSJOkfhj1JUt+dAY4wUMJ5D2YHtn9n4UqZf/WrqrvAU8BJYC/w+X2cV5Kke2bYkyT13QQwVlXfzjn+FfAK/F2SOVNVvywyzq80ZZ2LSrIO2FBVk8CbwM7lTFqSpKW4Zk+S1GtVdQ34cJ6mQ8BEkmmaUsvXlhjnVpLzSb4DzgFnF+i6HjidZA0Q4K3lzl2SpMWkqoY9B0mSJEnSCrOMU5IkSZI6yLAnSZIkSR1k2JMkSZKkDjLsSZIkSVIHGfYkSZIkqYMMe5IkSZLUQYY9SZIkSeogw54kSZIkddCfN3mCGgwGpxwAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 1080x360 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rMzQDr20HPbw"
      },
      "source": [
        "#Sequence Modelling: Text\n",
        "\n",
        "Now we will use RNNs to tackle text examples. Text is one of the modalities that have been widely tackled with deep learning improving upon past methods. We will present two problems: classification and generation.\n",
        "\n",
        "A small summary of both problems is given below.\n",
        "\n",
        "**Classification**\n",
        "\n",
        "Classification is a standard problem, where we have some input data $x$ and try to classify it as one of the available classes $y$. In the case of sequential data, though, $x$ will be a sequence of elements that will be processed by the RNN to return the label $y$. The image depicts a sequence classification problem, where the red blocks are inputs, the green blocks the RNN model, and the blue block is the output. An example of a text classification problem that can be tackled with RNNs is \"Hate speech detection\", where the architecture must identify if an input text contains racist or sexist language, among others. \n",
        "\n",
        "![alt text](https://i.ibb.co/TtZPpZr/Capture.jpg)\n",
        "\n",
        "\n",
        "**Generation**\n",
        "\n",
        "In a generation problem, we aim to generate a sequence $y$ following the same distribution as the real data $x$. We will input the sequence into the model, and we will output another sequence $y$. Text translation is a typical example of many to many RNNs.\n",
        "\n",
        "![](https://i.ibb.co/7gSwnT2/Capture.jpg)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mc5bCFhzQnQi"
      },
      "source": [
        "## Preliminary imports\n",
        "\n",
        "Here we include some of the used imports during the tutorial. Just run it to import the necessary modules."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K2uDXpT1Qsqz"
      },
      "source": [
        "## These are just a bunch of imports\n",
        "## Just run them\n",
        "import tensorflow as tf\n",
        "from keras.preprocessing import sequence\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense, Embedding, Dropout, Flatten, Add, Lambda\n",
        "from keras.layers import LSTM\n",
        "from keras.datasets import imdb\n",
        "\n",
        "from keras import __version__\n",
        "from keras import backend as K\n",
        "\n",
        "\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L4kJNRmcq3DL"
      },
      "source": [
        "# Text Classification "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6fLVCgocM6Kk"
      },
      "source": [
        "**References:** \n",
        "\n",
        "https://colab.research.google.com/github/csc-training/intro-to-dl/blob/master/day1/keras-imdb-rnn.ipynb#scrollTo=BYu8ql1nG7xc\n",
        "\n",
        "For the text classification example we will use the IMDB dataset available in Keras. The dataset includes tens of thousands of movie reviews taken from the IMDB website with a corresponding label for each review. The label is binary, and indicates if the review has a positive (label=1) or negative (label=0) sentiment. This problem is a quite standard Natural Language Processing (NLP) problem, and it is called sentiment classification or sentiment analysis.\n",
        "\n",
        "Before loading the data and building our model, we will explain a common part of NLP models, the embeddings."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CAzSKba3J5NA"
      },
      "source": [
        "### Embeddings\n",
        "**References:** \n",
        "\n",
        "[1] https://nlp.stanford.edu/projects/glove/\n",
        "\n",
        "[2] https://colab.research.google.com/drive/1oXjNYSJ3VsRvAsXN4ClmtsVEgPW_CX_c?hl=en#scrollTo=p9q7qfXrvq-J\n",
        "\n",
        "\n",
        "Usually, the first step of text modelling is transforming the words into a numerical vector that represents the meaning or some properties of the word. This vector can be then processed by the network. To do so, we first will encode the word sequences in integer numbers, and we will also have a dictionary that contains the relationship `(actual word, integer)`. For example, the sentence \"the cat is on the table and the dog is on the mat\" can be encoded in the form $(7, 1, 3, 5, 7, 6, 0, 7, 2, 3, 5, 7, 4)$, with the corresponding dictionary $(and, 0), (cat, 1), (dog, 2)\\dots (the, 7)$:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jhRu4KoqSVvH",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "991db374-dcef-4e98-acbc-7a053d07b1a2"
      },
      "source": [
        "sentence = 'the cat is on the table and the dog is on the mat'\n",
        "## We form a list of unique words by using a set\n",
        "## which returns only unique elements\n",
        "## we also sort them, which is not necessary\n",
        "sentence_set = sorted(set(sentence.split(' ')))\n",
        "words = list(sentence_set)\n",
        "## We now form a dictionary in the form of \n",
        "## e.g. dict_words[and] = 1\n",
        "dict_words = dict((word, i) for i, word in enumerate(words))\n",
        "## We now encode the sentence in a list of integers\n",
        "encoded_sentence = [dict_words[w] for w in sentence.split()]\n",
        "print(sentence_set)\n",
        "print(encoded_sentence)\n",
        "print(dict_words)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['and', 'cat', 'dog', 'is', 'mat', 'on', 'table', 'the']\n",
            "[7, 1, 3, 5, 7, 6, 0, 7, 2, 3, 5, 7, 4]\n",
            "{'and': 0, 'cat': 1, 'dog': 2, 'is': 3, 'mat': 4, 'on': 5, 'table': 6, 'the': 7}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CoHyEZdVzHFD"
      },
      "source": [
        "Keras contains a class called `Tokenizer` that helps in performing this process. We will not use it for the examples in this tutorial, as we want to show all the steps involved in the tokenization process, but the `Tokenizer` class provides a high-level way to handle this preprocessing step. As it can be quite useful, we provide a small overview of it. [Here](https://keras.io/preprocessing/text/) you have a link to the Keras documentation explaining the different arguments of the class."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z5EL3-o5wsQl"
      },
      "source": [
        "import keras\n",
        "## First you define a tokenizer object\n",
        "## If you want a character level tokenization we set char_level=True\n",
        "## If set to false, it will tokenize in a word level\n",
        "## You can check in the documentation the different arguments\n",
        "tokenizer = keras.preprocessing.text.Tokenizer(char_level=False)\n",
        "## Then you fit it on the data you have\n",
        "## The method expects a list of sentences\n",
        "tokenizer.fit_on_texts([sentence])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0RBQrbI10CuR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ced67e60-e57c-4a0c-d17e-16dadb6f3e61"
      },
      "source": [
        "## Now, the class has formed a dictionary with all the words\n",
        "print(tokenizer.word_index)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{'the': 1, 'is': 2, 'on': 3, 'cat': 4, 'table': 5, 'and': 6, 'dog': 7, 'mat': 8}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CKshU7Lj0Sgl",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a08fa22f-57d8-4693-fbe6-71f34c478a9c"
      },
      "source": [
        "## You can also check the word count\n",
        "print(tokenizer.word_counts)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "OrderedDict([('the', 4), ('cat', 1), ('is', 2), ('on', 2), ('table', 1), ('and', 1), ('dog', 1), ('mat', 1)])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xNI-js25161w",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "851d62aa-d5cf-438f-ced6-f6d4911b019e"
      },
      "source": [
        "## You can transform the data to sequences of integers\n",
        "## We print the first 10 elements\n",
        "tokenizer.texts_to_sequences([sentence])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[1, 4, 2, 3, 1, 5, 6, 1, 7, 2, 3, 1, 8]]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 109
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3lITR0l3SUAU"
      },
      "source": [
        "Now that we have the sequence in the form of a list of integers,  we could input that directly to the RNN. However, as those numbers are arbitraryly chosen, it would be really hard for the RNN to understand the relationships between the words. To give the model more representation power, we first want to transform the integer to a vector of dimension $d$ which represents the semantic meaning of the words. Each of these vectors is called an Embedding. The actual values of the vectors are not important, only the relationships between them. For example, `dog` and `cat` vectors are probably going to be closer than `dog` and `the`. Or if you are doing sentiment classification, probably two words encoding similar positive sentiments will be closer than a word expressing positive sentiments and a completely neutral word.\n",
        "\n",
        " The embeddings can be initialized randomly and the model will learn suitable values to reduce the loss during the training process. However, embeddings which have already been trained in a large corpus of text such as Wikipedia can also be used. Examples of these pretrained embeddings are [Word2Vec](https://en.wikipedia.org/wiki/Word2vec) or  [GloVe](https://nlp.stanford.edu/projects/glove/). These methods are trained using the context of the words, for example predicting the surrounding words when a specific word is given. As a result, words that appear in similar contexts have closer embeddings.\n",
        "\n",
        "![alt text](https://i.ibb.co/s5sg6dZ/Screenshot-from-2019-02-08-15-13-41.png) \n",
        "\n",
        "The image shows relationships between words in the embedding space.\n",
        "\n",
        "\n",
        "\n",
        "We now download the GloVe pretrained vectors trained in Wikipedia and Gigaword. [Here](https://nlp.stanford.edu/projects/glove/) you have an overview of what GloVe is. We will see how the embeddings contain a semantic meaning that allows us to model semantic relationships."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ie3ToDUgKWN4"
      },
      "source": [
        "## Download and unzip glove pretrained embeddings\n",
        "!wget https://imperialcollegelondon.box.com/shared/static/c9trfhhwl9ohje5g3sapu3xk2zoywp3c.txt -O gensim_glove_vectors.txt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xDs0EebuUJ5x"
      },
      "source": [
        "We will now load the GloVe embeddings. We use `gensim` to manipulate the embeddings, which is a nice tool that can be used to play with GloVe or Word2Vec embeddings. This piece of code takes some time to load, as the embeddings file is quite large."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RK0qwB--K0Bp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b71f0186-7b70-4b32-8d08-1ca7a1d09f15"
      },
      "source": [
        "## We load the embeddings\n",
        "## Gensim is a really useful module that provides high-level function\n",
        "## to use with the embeddings\n",
        "!pip install gensim\n",
        "from gensim.models.keyedvectors import KeyedVectors\n",
        "glove_model = KeyedVectors.load_word2vec_format(\"gensim_glove_vectors.txt\", binary=False)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: gensim in /usr/local/lib/python3.6/dist-packages (3.6.0)\n",
            "Requirement already satisfied: six>=1.5.0 in /usr/local/lib/python3.6/dist-packages (from gensim) (1.15.0)\n",
            "Requirement already satisfied: scipy>=0.18.1 in /usr/local/lib/python3.6/dist-packages (from gensim) (1.4.1)\n",
            "Requirement already satisfied: numpy>=1.11.3 in /usr/local/lib/python3.6/dist-packages (from gensim) (1.19.5)\n",
            "Requirement already satisfied: smart-open>=1.2.1 in /usr/local/lib/python3.6/dist-packages (from gensim) (4.1.2)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yrj-dPMbSsrc"
      },
      "source": [
        "Now we will do some operations using the embeddings of the words. First, we will do some words arithmetics based on the embeddings, which it will show us that the words encode some semantic meaning. For example, the distance between words with similar semantic meaning but different genders is approximately fixed. Meaning that the vector resulting from doing $man - woman$ should be similar to $king - queen$, hence:\n",
        "\n",
        "$king - queen \\approx man - woman \\rightarrow woman + king - man \\approx queen$\n",
        "\n",
        "We can check this using the method `most_similar` and the `positive` and `negative` arguments as following:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0PYEzmorMFD_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5ec97994-63fa-4632-b6a2-989051c18ff1"
      },
      "source": [
        "## We can do word arithmetics. \n",
        "## We look for the nearest neighbour of the vector resulting on doing\n",
        "## the operation 'king' + 'woman' - 'man'\n",
        "glove_model.most_similar(positive=['woman', 'king'], negative=['man'])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('queen', 0.6713277101516724),\n",
              " ('princess', 0.5432624220848083),\n",
              " ('throne', 0.5386104583740234),\n",
              " ('monarch', 0.5347574949264526),\n",
              " ('daughter', 0.498025119304657),\n",
              " ('mother', 0.4956442713737488),\n",
              " ('elizabeth', 0.4832652509212494),\n",
              " ('kingdom', 0.47747087478637695),\n",
              " ('prince', 0.4668239951133728),\n",
              " ('wife', 0.4647327661514282)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 112
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O1bgFWT3aGUi",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "145cad9e-a9fc-4127-ef3b-8ab4f8824654"
      },
      "source": [
        "## Similar examples\n",
        "glove_model.most_similar(positive=['google', 'ios'], negative=['apple'])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('android', 0.5720318555831909),\n",
              " ('apps', 0.5662358999252319),\n",
              " ('app', 0.5322650671005249),\n",
              " ('facebook', 0.4770544171333313),\n",
              " ('smartphones', 0.47069066762924194),\n",
              " ('gmail', 0.4677259922027588),\n",
              " ('firefox', 0.4666402041912079),\n",
              " ('skype', 0.4491073191165924),\n",
              " ('iphone', 0.44582629203796387),\n",
              " ('web-based', 0.44464385509490967)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 113
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J09zgTciaHU4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f2238345-fdd6-47bb-8263-c65fd132ce23"
      },
      "source": [
        "glove_model.most_similar(positive=['england', 'paris'], negative=['france'])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('london', 0.6710129976272583),\n",
              " ('manchester', 0.5267472863197327),\n",
              " ('birmingham', 0.49347949028015137),\n",
              " ('liverpool', 0.49033817648887634),\n",
              " ('oxford', 0.4811282157897949),\n",
              " ('middlesex', 0.47516536712646484),\n",
              " ('surrey', 0.46148526668548584),\n",
              " ('sussex', 0.45899876952171326),\n",
              " ('leeds', 0.4551934599876404),\n",
              " ('essex', 0.45398280024528503)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 114
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p7LMqhpATsAL",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 91
        },
        "outputId": "1613700c-0d2d-441c-f440-27a834a6ae63"
      },
      "source": [
        "## We can also check the word that does not match the rest\n",
        "glove_model.doesnt_match(\"breakfast cereal dinner lunch\".split())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/gensim/models/keyedvectors.py:895: FutureWarning: arrays to stack must be passed as a \"sequence\" type such as list or tuple. Support for non-sequence iterables such as generators is deprecated as of NumPy 1.16 and will raise an error in the future.\n",
            "  vectors = vstack(self.word_vec(word, use_norm=True) for word in used_words).astype(REAL)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'cereal'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 115
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g85rTdpOY-HZ"
      },
      "source": [
        "To check out other functions you can call use [this](https://radimrehurek.com/gensim/models/keyedvectors.html#module-gensim.models.keyedvectors).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AjgjAlaBQvaH"
      },
      "source": [
        "In http://projector.tensorflow.org/ you can visualize the words projected in the $\\mathbb{R}^3$ space using either PCA or tSNE, which are both techniques for dimensionality reduction. There, you can see how words are clustered by meaning or topic.\n",
        "\n",
        "\n",
        "![](https://brenndoerfer.github.io/deep-sentiment-analysis-distill/img/tensorboard_projector.gif)\n",
        "\n",
        "Image taken from [here](https://brenndoerfer.github.io/deep-sentiment-analysis-distill/index.html)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "30w3Y4oHJy79"
      },
      "source": [
        "### IMDB\n",
        "Now let's start tackling a text classification problem. We first load the IMDB dataset for sentiment classification. As we said, it contains movie reviews with a corresponding binary sentiment label (label=1 corresponds to positive sentiment, and label=0 to negative sentiment). The words are already encoded as integers in order from most common words to less common (e.g. `the` is a common word so it should be encoded as a small integer). That makes it easy to filter the non-common words by using the argument `num_words` when loading the data. The filtered words will be all encoded as a special token `<UNK>`, which means unknown. For example, if we want to load the dataset with only the top $5000$ most common words we can do this:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GWU0zG7AHd3u",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "64a5af13-3129-4314-97f1-6081eb0318d6"
      },
      "source": [
        "# number of most-frequent words to use\n",
        "nb_words = 5000\n",
        "\n",
        "print('Loading data...')\n",
        "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=nb_words)\n",
        "print('x_train:', x_train.shape)\n",
        "print('x_test:', x_test.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loading data...\n",
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb.npz\n",
            "17465344/17464789 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "<string>:6: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
            "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/datasets/imdb.py:159: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
            "  x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "x_train: (25000,)\n",
            "x_test: (25000,)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/datasets/imdb.py:160: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
            "  x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BiYpN-jwcGd-"
      },
      "source": [
        "We see that the dataset contains 25000 examples for both train and testing. Let's print an example."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WF2twk31cNiN",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "031ea0c8-6035-4b55-f3e6-c9ba3af15c4d"
      },
      "source": [
        "print(x_train[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 2, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 2, 8, 4, 107, 117, 2, 15, 256, 4, 2, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 2, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 2, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 2, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 2, 19, 178, 32]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hqxzLNgAcUmL"
      },
      "source": [
        "The printed example is a sequence of numbers. It is important to note that there are three special integers in this IMDB dataset: $0$, $1$ and $2$. $0$ will be used to pad the sequences, which we will explain now. $1$ is used as the `<START>` token (you can see how the printed sequence starts with a `1`). $2$ is the token `<UNK>` which is used for all the filtered non-common words. If we want to retrieve the actual sentence, we can use the dictionary given when using the method `get_word_index()`. However, the given dictionary when calling `get_word_index()` does not take into account the three mentioned tokens, so we need to modify it a little bit."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y26JIPAadBtX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "52570657-33a2-4231-e817-a3129c2b047e"
      },
      "source": [
        "# get_word_index retrieves a mapping word -> index\n",
        "word_index = imdb.get_word_index()\n",
        "# We make space for the three special tokens\n",
        "word_index_c = dict((w, i+3) for (w, i) in word_index.items())\n",
        "word_index_c['<PAD>'] = 0\n",
        "word_index_c['<START>'] = 1\n",
        "word_index_c['<UNK>'] = 2\n",
        "# Instead of having dictionary word -> index we form\n",
        "# the dictionary index -> word\n",
        "index_word = dict((i, w) for (w, i) in word_index_c.items())\n",
        "# We now retrieve the sentence\n",
        "sentence = ''\n",
        "for index in x_train[0]:\n",
        "  sentence += index_word[index] + ' '\n",
        "print(sentence)\n",
        "print(y_train[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb_word_index.json\n",
            "1646592/1641221 [==============================] - 0s 0us/step\n",
            "<START> this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert <UNK> is an amazing actor and now the same being director <UNK> father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for <UNK> and would recommend it to everyone to watch and the fly <UNK> was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also <UNK> to the two little <UNK> that played the <UNK> of norman and paul they were just brilliant children are often left out of the <UNK> list i think because the stars that play them all grown up are such a big <UNK> for the whole film but these children are amazing and should be <UNK> for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was <UNK> with us all \n",
            "1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iyhd3lQefNH4"
      },
      "source": [
        "We can see how we just recovered the sentence that we had printed before as a sequence of numbers. We can also see how there is no punctuation in this dataset. The review is clearly positive, so the corresponding label is \"1\".\n",
        "\n",
        "One of the problems with the given text data is that the sequences all have a different length. We want to give Keras a batch of inputs with fixed dimensions. To do so, we define a maximum length `maxlen`, and truncate the sentences longer than that, and also pad with $0$'s at the beginning the sentences shorter than that length. We use the method `sequence.pad_sequence` from `keras.preprocessing`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CGcgjhlVbyOv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "63efb918-8a3e-4c91-b0a2-413791b12b24"
      },
      "source": [
        "# Truncate sentences after this number of words\n",
        "maxlen = 80\n",
        "print('Pad sequences (samples x time)')\n",
        "x_train = sequence.pad_sequences(x_train, maxlen=maxlen)\n",
        "x_test = sequence.pad_sequences(x_test, maxlen=maxlen)\n",
        "print('x_train shape:', x_train.shape)\n",
        "print('x_test shape:', x_test.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Pad sequences (samples x time)\n",
            "x_train shape: (25000, 80)\n",
            "x_test shape: (25000, 80)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sXCucplf0M5O"
      },
      "source": [
        "We can check how some sentences are padded at the beginning to be of length `maxlen`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HeBf1oOJzr4d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2ecdebcd-9a00-44d2-93f4-4639f5bc28d3"
      },
      "source": [
        "x_train[5]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([   0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
              "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
              "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
              "          0,    0,    0,    0,    1,  778,  128,   74,   12,  630,  163,\n",
              "         15,    4, 1766,    2, 1051,    2,   32,   85,  156,   45,   40,\n",
              "        148,  139,  121,  664,  665,   10,   10, 1361,  173,    4,  749,\n",
              "          2,   16, 3804,    8,    4,  226,   65,   12,   43,  127,   24,\n",
              "          2,   10,   10], dtype=int32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 120
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N2ljchrWPMz3"
      },
      "source": [
        "We will now build the model we will use for sentiment classification. The model is formed by an Embedding layer, where the model will learn a vector of dimensionality `embedding_dim` for each of the words; an LSTM layer and a linear layer that maps the output of the LSTM to 1 value. We train this with the `binary_crossentropy` loss and `sigmoid` activation as we only have two classes. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fh2cHXD5HjQG",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f8e8139d-d6b8-466b-dd22-15c874990e7f"
      },
      "source": [
        "## Model parameters:\n",
        "# Dimensions of the embeddings\n",
        "embedding_dim = 300\n",
        "## LSTM dimensionality\n",
        "lstm_units = 128\n",
        "\n",
        "print('Build model...')\n",
        "text_class_model = Sequential()\n",
        "\n",
        "text_class_model.add(Embedding(nb_words,\n",
        "                    embedding_dim,\n",
        "                    input_length=maxlen))\n",
        "\n",
        "text_class_model.add(LSTM(lstm_units))\n",
        "\n",
        "# To stack multiple RNN layers, all RNN layers except the last one need\n",
        "# to have \"return_sequences=True\".  An example of using two RNN layers:\n",
        "# model.add(LSTM(lstm_units, return_sequences=True))\n",
        "# model.add(LSTM(lstm_units))\n",
        "\n",
        "\n",
        "# You can add some dropout if you want\n",
        "# model.add(Dropout(0.5))\n",
        "\n",
        "text_class_model.add(Dense(1, activation='sigmoid'))\n",
        "\n",
        "# try using different optimizers and different optimizer configs\n",
        "text_class_model.compile(loss='binary_crossentropy',\n",
        "              optimizer='adam',\n",
        "              metrics=['accuracy'])\n",
        "print(text_class_model.summary())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Build model...\n",
            "Model: \"sequential_36\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding (Embedding)        (None, 80, 300)           1500000   \n",
            "_________________________________________________________________\n",
            "lstm_36 (LSTM)               (None, 128)               219648    \n",
            "_________________________________________________________________\n",
            "dense_36 (Dense)             (None, 1)                 129       \n",
            "=================================================================\n",
            "Total params: 1,719,777\n",
            "Trainable params: 1,719,777\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "None\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MTDlUcEF0mqn"
      },
      "source": [
        "Let's train it for 5 epochs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sJfDwWRlHoEm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fc5b6284-3592-4848-e296-8b5ba2e8d218"
      },
      "source": [
        "## We train the model for 5 epochs and check the accuracy in a validation split\n",
        "epochs = 5\n",
        "validation_split = 0.2\n",
        "\n",
        "history = text_class_model.fit(x_train, y_train, batch_size=128,\n",
        "          epochs=epochs, validation_split=validation_split)\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/5\n",
            "157/157 [==============================] - 8s 39ms/step - loss: 0.5508 - accuracy: 0.6928 - val_loss: 0.3686 - val_accuracy: 0.8330\n",
            "Epoch 2/5\n",
            "157/157 [==============================] - 5s 34ms/step - loss: 0.3080 - accuracy: 0.8741 - val_loss: 0.3914 - val_accuracy: 0.8252\n",
            "Epoch 3/5\n",
            "157/157 [==============================] - 5s 33ms/step - loss: 0.2452 - accuracy: 0.9026 - val_loss: 0.3998 - val_accuracy: 0.8278\n",
            "Epoch 4/5\n",
            "157/157 [==============================] - 5s 33ms/step - loss: 0.1857 - accuracy: 0.9307 - val_loss: 0.4578 - val_accuracy: 0.8272\n",
            "Epoch 5/5\n",
            "157/157 [==============================] - 5s 32ms/step - loss: 0.1456 - accuracy: 0.9466 - val_loss: 0.5097 - val_accuracy: 0.8260\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5tVyZLyv0rbO"
      },
      "source": [
        "This is a classification example, so let's print the loss and the classification accuracy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k3D78e9yHo8l",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 437
        },
        "outputId": "d03f36d4-8995-4672-f607-f578dbae42af"
      },
      "source": [
        "plt.figure(figsize=(5,3))\n",
        "plt.plot(history.epoch,history.history['loss'], label='training')\n",
        "plt.plot(history.epoch,history.history['val_loss'], label='validation')\n",
        "plt.title('loss')\n",
        "plt.legend(loc='best')\n",
        "\n",
        "plt.figure(figsize=(5,3))\n",
        "plt.plot(history.epoch,history.history['accuracy'], label='training')\n",
        "plt.plot(history.epoch,history.history['val_accuracy'], label='validation')\n",
        "plt.title('accuracy')\n",
        "plt.legend(loc='best');"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUIAAADSCAYAAAA/vMlrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU1f3/8ddnkpAQSIAkQIAEEiRA2CEJRFFEFMUNLIvgjlVRq7XaFb/2q9Vq69daS/1ZF0TaamUzYtW6o7iWJQn7pqySEISQQNhCIMnn98cdQogBRpjJzGQ+z8cjDzJz751zLo5vzr3n3HNEVTHGmFDm8ncFjDHG3ywIjTEhz4LQGBPyLAiNMSHPgtAYE/IsCI0xIc+C0AQcEdkiIhf5ux4mdFgQGmNCngWhMSbkWRCagCUikSIyRUSK3D9TRCTSvS1BRP4jIntEpFREvhARl3vbb0Rkm4jsE5GvReRC/56JCXTh/q6AMSfxAJAN9AMUeBP4LfC/wC+AQqC1e99sQEWkG3A3kKWqRSKSAoQ1bLVNsLEWoQlk1wGPqOpOVS0GHgZucG87ArQDOqnqEVX9Qp0H56uASKCHiESo6hZV3eiX2pugYUFoAll74Ntar791vwfwJ2AD8KGIbBKRyQCqugG4F/gdsFNEZolIe4w5CQtCE8iKgE61Xnd0v4eq7lPVX6hqZ2Ak8POj9wJVdYaqnus+VoH/a9hqm2BjQWgC2UzgtyLSWkQSgAeBfwGIyBUi0kVEBCjDuSSuFpFuIjLM3alyCCgHqv1UfxMkLAhNIHsUyANWACuBJe73ANKAecB+YAHwrKrOx7k/+DiwC/gOaAPc37DVNsFGbGJWY0yosxahMSbkWRAaY0KeBaExJuRZEBpjQp4FoTEm5AXcs8YJCQmakpLi72oYYxqZ/Pz8Xaraur5tAReEKSkp5OXl+bsaxphGRkS+PdE2uzQ2xoQ8C0JjTMizIDTGhDwLQmNMyLMgNMYED1UoWAzv/hq+fs9rHxtwvcbGGHMcVdixGlblwKrXYc9WCIuE2PbQ7VKvFOFREIrICOCvOGs/TFPVx+tsn4gzY/A291vPqOo097abcNaZAHhUVf/phXobYxq70k2w8nUnAIvXgYTBWRfA0P+B7pdDVKzXijplEIpIGPA3YDjOYjm5IvKWqq6ps+tsVb27zrFxwENAJs5MwfnuY3d7pfbGmMZl73ZYPRdW5kDREue9jufA5X+GHldBswSfFOtJi3AgsEFVNwGIyCxgFFA3COtzCfCRqpa6j/0IGIEz87AxxsDBUljzpnPZu+VLQKFdXxj+e+g1Glok+bwKngRhB6Cg1utCYFA9+40RkSHAN8B9qlpwgmM71D1QRCYBkwA6duzoWc2NMcGrYj98/a7T8tv4MVRXQnwaDJ0MvcZCQpcGrY63OkveBmaqaoWI3A78Exjm6cGqOhWYCpCZmWlTZhvTGFVWwPqPnHt+X78PleUQmwTZP4HeYyGxD4j4pWqeBOE2ILnW6ySOdYoAoKoltV5OA56odezQOsd++kMraYwJUlWVsOVzp9Nj7dtQUQbRCdD/OqfllzwIXP4fxedJEOYCaSKSihNsE4Bra+8gIu1Udbv75Uhgrfv3D4A/iEgr9+uLsYV0jGncjo71W5UDq9+AA8UQGQvdr4DeYyB1KIQF1si9U9ZGVStF5G6cUAsDpqvqahF5BMhT1beAe0RkJFAJlAIT3ceWisjvccIU4JGjHSfGmEZEFXascu75rZoLZVshPAq6XuK0/NIuhogof9fyhAJuFbvMzEy1abiMCRIlG53e3pU5sOtr91i/Yc49v26XeXWs35kSkXxVzaxvW2C1T40xgW9vkdPqW5UDRUud9zoNhkG3u8f6xfu3fqfBgtAYc2oHSmDtm06nx7df4Yz16wcXPwo9R0OL742KCyoWhMaY+lXsg3XvOi2/jZ84Y/0SusLQ+6HXmAYf6+dLFoTGmGOOHIINHzn3/L55HyoPQYtkOPsup9Mjsbffxvr5kgWhMaGuqhI2f+Z0eqx9Gyr2usf63eB0eiQNDIixfr5kQWhMKKquhsLFTstvzb+PjfVLv9K57E09P+DG+vlS6JypMaFOFb5b4bT8Vs2FsgL3WL8RTsuvy/CAHuvnSxaExjR2uzYcm9R01zfgCnfG+g37X+h+GUTG+LuGfmdBaExjVFZ4bKzf9uWAOGP9su+E9FFBOdbPlywIjWksDpTAmjecsX5b/+u8174/XPyYM69fbHv/1i+ABX0Qbi8rZ2vJQQZ1tn/hTAiq2Afr3oGVr8HG+aBVkNANLnjA6fSIP8vfNQwKQR+ED725ms++KWbaTZmcl9ba39UxxveqKmHDPFg+A775wD3WryOc81On06Ntr0Y51s+XPBocJCIjRORrEdkgIpNPst8YEVERyXS/ThGRchFZ5v553lsVP+qPo3uTmtCMW/6Zx6df7/T2xxsTOEo3wcePwJReMHM8bPkKBtwIP/4Q7l0Bwx9utAOefc1rizeJSAzwM2BRnY/YqKr9vFTf74lvHsmM27K5ftoiJr2czws3ZHBB9za+Ks6YhnWk3BnkvORl2PIFiMsZ5nLZn5xhL2ER/q5ho+BJi7Bm8SZVPQwcXbyprt8D/wcc8mL9PBLXrAkzbhtE18Tm3P5KPvPW7GjoKhjjXduXwzu/hD93g7m3OWP+hv0W7lsN181xBj5bCHqNJ0F4ygWYRGQAkKyq79RzfKqILBWRz0TkvPoKEJFJIpInInnFxcWe1v04LaOb8Oot2XRvF8Odr+bz4ervTutzjPGb8j2w+EV4YYjzs+RlZ0LTG9+Cny6FIb+ynl8fOePOEhFxAU/hnpW6ju1AR1UtEZEM4N8i0lNV99beyVuLN7WIjuCVWwZx0/TF/OTVJTxzbX9G9Gp3uh9njO+pOktYLn3FWdKy8hC07Q2X/snp+IiO83cNQ4I3Fm+KAXoBn4pzkzYReEtERqpqHlABoKr5IrIR6Ar4bArqFk0jePmWgUycvpi7Zizl6QlweR8LQxNg9m53en2X/svpBImMhX7XOp0f7fpZh0cDO+PFm1S1DKhZfl5EPgV+qap5ItIaKFXVKhHpDKQBm7xY/3rFRkXw8i2DuPnvi7ln1lKqVBnZ1y4pjJ9VVcL6D51L3vUfOmP+Og2G838D6SOhSbS/axiyvLV404kMAR4RkSNANXBHQy3e1DwynH/cPJCb/5HLvbOWUl2tXNU/uGfRNUGqZKMTfstnwv4d0LwtDL7HmebKBjwHhEa/eNPBw5Xc8o88Fm4u4cmxfRmTkeS1zzbmhA4fhLVvOQH47VfOokZpFzuXvmnDrcfXD0J68aboJuFMn5jFrS/n8suc5VSpcnVm8qkPNOaHUoXty2DJK84jbxV7oVUqXPgg9L0WYu1edaBq9EEI0LRJGC/dlMVtL+fx65wVVFcrEwZ29He1TGNRvhtWvOa0/nasdOb46zHKaf11GmwdH0EgJIIQICoijBdvzOT2V/KZPHclVapcN6iTv6tlglV1tfOkx9JXYM1bUFUB7frCZU9C73HQtKW/a2h+gJAJQnDCcOqNGdz5ryU88MYqqqqVG89O8Xe1TDDZWwTLXnWGvezeAlEtnJbfgBucIDRBKaSCECAyPIznrh/AXa8u5cE3V1NVrdw8ONXf1TKBrOqIs6LbklecFd60GlLOc6a6Sr8SIpr6u4bmDIVcEIIThs9eN4CfzlzCw2+voapaufW8zv6ulgk0u9YfG/ZyoBhi2sG590H/6yHOvi+NSUgGIUCTcBfPXDuAe2Yu5dF31lJVrdx+vo3pCnmHD8Dqfzv3/rYucIa9dLvUGfPX5aKQWtktlIT0f9WIMBdPX9Of+2Yv44/vraNKlZ8M7eLvapmGpgpFS5zW38rX4fA+iDsLLnoY+l4DMW39XUPjYyEdhOCE4ZTx/QhzCU+8/zVVVcpPL0zzd7VMQzhYCitmO/f+dq6G8KbQ8yqn86Pj2TbsJYSEfBAChIe5eOrqfoSJ8OePvqFKlXsv6urvahlfqK6GzZ85rb91/4Gqw84CR1f8xVnjI6qFv2to/MCC0C3MJfxpXF9cLmHKvPVUVyv3De+KWKugcSgrhGUznHt/e7ZCVEvIuNkZ9pLY29+1M35mQVhLmEt4YkwfwkR4+pMNVFYrv7qkm4VhsKo8DN+857T+NnwMKKSeDxc+BN2vgIgof9fQBAiPglBERgB/xZl9ZpqqPn6C/cYAOUCWey5CROR+4BagCrhHVT/wRsV9xeUS/ji6Ny6X8OynG6lSZfKI7haGwaT4a/ewl1lwcBfEtIchv4R+10GcjRk13+fTxZtEpAfO/IU9gfbAPBHpqqpV3jsF73O5hMeu6kWYC174bBNVVcoDl6dbGAayiv2w+g3n0rdgEbjC3cNeboQuF4IrzN81NAHMkxZhzeJNACJydPGmNXX2O7p4069qvTcKmKWqFcBmEdng/rwFZ1pxX3O5hN+P6kW4y8W0LzdTpcqDV/SwMAwkqlCYB0tfhlVz4fB+SOgKw38PfSdAc1vN0HjGkyCsb/GmQbV3qL14k4j8qs6xC+scGzSzo4oID13ZA5cI07/aTFW18vDInhaG3lJ5GCr2QUWZ8+ehve7X+5wprCr21vPe0f32wqEyOLQHIqKh52in4yN5kA17MT+Yrxdv8vQzJgGTADp2DKzpsUSE/70inTAXvPiFE4a/H9ULlyuE/2erqnQGHR8XXrX+rPve0eCq+15VxanLCmvirOcRGQNRsc7vLZOPvZfYG3r+yNlmzGny6eJNHhwLeG8VO18REf7nsnTCXC6e/2wj1ao8dlXv4AvD6mrn8vGEra5ara+a8Kr73j44cuDUZUnY8eEVGeNMUR/fxf1ejPunRa39Yo7tGxnrvBce6fu/FxPyfL14UzkwQ0SewuksSQMWe6/6DUdE+M2IboS7hGfmb6CqWnl8dB//heGBEihZD6WbncvDE7W66l5W4sG/M03qBFNUS2jZsVZQxdYTXnXei4i2S1QTNHy6eJN7vzk4HSuVwF2B3mN8MiLCLy7uisslPP3xeqqq4YmxfQjzVRhWHobdm51ZUErWw64N7j+/cWZFrisi+vhWVVSs85ysp+EVGQtNmoPL5ZvzMSZANfrFm3zlr/PW85d53/Cj/h14clzf0w9DVdi/0x1w66Fkw7Hg2/2ts+TjUc3bQnwaJHRx/5nmTA4QHecEmS0IZMwJhfTiTb7ys4vSCHPBkx9+Q1W18tTVfQkPO0lL6sghKN1YT+tug9NrelR4lHMfLbGP8+xrTfB1sedgjfERC8IzcPewNMJcLv7v/XVUVStTxvcl4sB39bfu9hRw3P252A5OuPUZd6x1l5AGsUl2aWpMA7MgPB2HD9SE3J1sYHinpZR/vY6qx3YQoeXH9oto5rTmkgY6j3fFd3HCLr4LNGnmv/obY45jQXgi1dWwt9DpmKi5jHW38vbWHgEkdGmZTGHrJGbu6EaTxO5cfckwItp2daZ2t55TYwKeBeGhvXXu2bnDrmQjVNZq3UW2cFp3KefV6azoDBFNSQLkq8088PYa5v+3GX+7ri2RFoLGBIXQCMLqKtjzrRNydTsr9u84tp+EQatOTsh1Huq+jHUHXrPWp2zdTRycSphL+N83V3PHK/k8d30GURH2sL8xga5xBeHB0uM7KI627ko3OTMRH9U0zgm3LsOPb921SoXwJmdUhRvOTiHM5eJ/3ljJ7a/k88INFobGBLrgD8KFzzmrjpWsh4Mlx953RThzz8WnQdcRx7fuouN8WqVrB3UkzAWT567ktpfzePHGTAtDYwJY8AfhoTJnrrnuVxwfdi07+XXpxfFZHXGJ8OvXV/Djf+Ty0k1ZNG1iYWhMILInS3xs7pJCfvnacgamxjF9YhbRTYL/3x5jgtHJniyxkbs+NnpAEn8Z34/Fm0uZOD2X/RWV/q6SMaYOC8IGMKpfB/46oT/5W3czcfpi9h064u8qGWNq8SgIRWSEiHwtIhtEZHI92+8QkZUiskxEvnSvVYKIpIhIufv9ZSLyvLdPIFhc2bc9/++a/iwr2MNN0xez18LQmIBxyiCstXjTpUAP4JqjQVfLDFXtrar9gCdwZqw+aqOq9nP/3OGtigejy3q345lrB7CisIwbXlpMWbmFoTGBwJMWYc3iTap6GDi6eFMNVd1b62UzPJr9MzSN6JXIc9dnsKaojBteWkTZQQtDY/zNkyCsb/Gm7y3AJCJ3ichGnBbhPbU2pYrIUhH5TETOO6PaNhLDe7Tl+eszWLd9H9dOW8juA4dPfZAxxme81lmiqn9T1bOA3wC/db+9Heioqv2Bn+NM2/+9VXZEZJKI5IlIXnFxsbeqFNAuTG/LCzdmsH7nfq6dtohSC0Nj/MaTIPRoAaZaZgFXAahqhaqWuH/PBzYCXeseoKpTVTVTVTNbt27tad2D3gXd2jDtxkw2Fe/n2hcXUrLfg1XdjDFe50kQ1izeJCJNcBZvOm6dEhFJq/XycmC9+/3W7s4WRKQzzuJNm7xR8cZiSNfWvHRTFltKDnDNiwsp3mdhaExDO2UQqmolcHTxprXAnKOLN7mX7AS4W0RWi8gynEvgm9zvDwFWuN/PAe5Q1VKvn0WQOzctgekTsygoLeeaFxeyc98hf1fJmJBij9gFkIWbSvjxP3JJbBHFzNuyaRsb5e8qGdNo2CN2QSK7czz//PFAdpQdYsLUhXxXZi1DYxqCBWGAyUqJ4+VbBlK8r4LxUxdQtKf81AcZY86IBWEAyugUxyu3DKR0/2HGT11A4e6D/q6SMY2aBWGA6t+xFf+6dRBlB48w/oWFFJRaGBrjKxaEAaxvcktevTWb/RWVTJi6kK0lFobG+IIFYYDrndSCGbcN4uDhSsZPXcCWXQf8XSVjGh0LwiDQs30LZtyWTUVlNeOnLmBT8X5/V8mYRsWCMEikt4tl5m3ZVFYpE6YuZMNOC0NjvMWCMIh0S4xh1qRsqhUmTF3I+h37/F0lYxoFC8Igk9bWCUOXOGH49XcWhsacKQvCINSlTXNmTcomPEy45sWF/OOrzew5aNN4GXO6LAiDVOfWzZk96WyS46L53dtrGPiHj/nZrKUs2FhCoD0/bkygs0kXGoHVRWXMzi3gjaXb2HeokpT4aK7OSmbsgCTa2MQNxgAnn3TBoyAUkRHAX4EwYJqqPl5n+x3AXUAVsB+YpKpr3NvuB25xb7tHVT84WVkWhKfv0JEq3lu1nZmLC1i8uZQwlzCsexuuGZjMkLTWhIfZBYAJXWcUhO6JVb8BhuOsV5ILXHM06Nz7xB5dwMk9R+FPVHWEe7W7mTgLQLUH5gFdVbXqROVZEHrHpuL9zM4r4PX8QnbtP0xibBTjMpO4OjOZ5Lhof1fPmAZ3ptNwnckqdqOAWe4p+zcDG9yfZ3ysc+vm3H9pOgvuv5Dnr8+ge7sYnpm/gfOemM/10xbxnxVFVFSe8N8jY0JKuAf71LeK3aC6O4nIXTizUzcBhtU6dmGdY+tbAW8SMAmgY8eOntTbeCgizMWIXomM6JVI0Z5yXssrZE5eAXfPWEqr6AhGD0hiQlYyaW1j/F1VY/zG16vYeXpsSC7e1NDat2zKzy5K4/NfX8DLPx7I2WfF8/KCLQz/y+eMee6/zMkr4ODhSn9X05gG50mL8HRWsXvuNI81DSDMJQzp2pohXVuza38FbyzZxqzcrfw6ZwWPvL2Gkf3aMyErmd4dWiAi/q6uMT7nSWdJOE5nyYU4IZYLXKuqq2vtk6aqR1euuxJ4SFUzRaQnMINjnSUfA2nWWRJ4VJW8b3cza3EB76ws4tCRatLbxTIhK5mr+nWgRXSEv6tozBnxxvCZy4ApOMNnpqvqYyLyCJCnqm+JyF+Bi4AjwG7g7qNBKSIPAD8GKoF7VfW9k5VlQeh/ew8d4c1lRczO3cqqbXuJDHdxWe92jM9KZlBqnLUSTVA64yBsSBaEgWXVNmew9r+XOYO1UxOacXVmMmMyOtAmxgZrm+BhQWjOWPnhKt5duZ3ZuQUs3lJKuEu4ML0NE7I6MqRra8Jc1ko0gc2C0HjVxuL9zMktICe/kJIDh2nXIopxGUmMs8HaJoBZEBqfOFxZzcdrdzArt4DP1xcDcG6XBCZkdeSiHm2IDA/zcw2NOcaC0Pjctj3lvJZXwJzcAorKDhHXrAmj+3dgwsBkurSxwdrG/ywITYOpqla+WF/M7NwCPlqzg8pqJbNTK8ZnJXN5n3ZEN/Fk6Kox3mdBaPyieF8Fc5cUMju3gE27DtA8MpyR/dpzTVZHenWItWE4pkFZEBq/UlVyt+xmVu5W3lmxnYrKanq0i2XCwGRG9bXB2qZhWBCagFFWfoS3lm1j5uIC1mw/Nlh7QlYyA22wtvEhC0ITkFZtK2NW7lbeXFrEvgpnsPb4rGTGDEiidUykv6tnGhkLQhPQDh6u5N2V3zE7dyu5W3YT7hIuSm/LePfM2jZY23iDBaEJGht27meOe2btmsHamclcnZlEUisbrG1OnwWhCTqHK6uZ5x6s/YV7sPZ5aa2ZkJXMReltaRJu66+YH6YhFm/6OXArzgwzxcCPVfVb97YqYKV7162qOvJkZVkQmroKSg/yWn4hr+UVsL3sEPHNmjB6QAfGZ3WkS5vm/q6eCRINsXjTBcAiVT0oIncCQ1V1vHvbflX1+NtqQWhOpKpa+Xx9MbMXFzBvrTNYO6NTK8ZlJHF5n3bERNkwHHNiZxqEZwO/U9VL3K/vB1DVP55g//7AM6o62P3agtB43c59h5i7ZBuv5RWwsfgAUREuLu3VjrEZSZzdOR6XdbCYOk4WhF5bvKmWW4Dak69GiUgezmXz46r673oqaIs3mR+kTUwUd5x/FrcP6cyygj3k5Bfy1vIi3li6jQ4tmzJ6QAfGZiTRKb6Zv6tqgoAnLcKxwAhVvdX9+gZgkKreXc++1wN3A+eraoX7vQ6quk1EOgOfABeq6sYTlWctQnO6Dh2p4sM1O8jJL+SL9cWowsCUOMZmJHFZn3Y0j7TnnEPZmbYIPVqASUQuAh6gVggCqOo295+bRORToD9wwiA05nRFRYQxsm97RvZtz/aycuYu2cbr+YX8+vUVPPTWai7tncjYjCSyU+3S2RzPW4s39QdycFqO62u93wo4qKoVIpIALABG1e5oqctahMabVJUlW51L5/8sd55gSWrVlDEDkhgzIImO8TY2MVQ0xOJN84DewHb3IVtVdaSInAO8AFTjrKE8RVVfOllZFoTGV8oPV/Hhmu/IyS/kyw27UIVBqe5L597taGaXzo2aDag2po6iPeXMXVJITn4hW0oOEt0kjMt6O73OA1Pi7NK5EbIgNOYEVJX8b3c7l84rtrO/opLkuKaMHZDM6AEdbA2WRsSC0BgPHDxcyQernUvnrzaUAHB253jGZSYxoleiza4d5II+CI8cOUJhYSGHDh3yU60an6ioKJKSkoiIsKcx6lO4+yBzl2wjJ7+QraUHadYkjMv7tGNsRjJZKa1s3sQgFPRBuHnzZmJiYoiPj7cvoBeoKiUlJezbt4/U1FR/VyegHZ1d+7W8At5ZuZ2Dh6voFB/N2AFJjM5IokPLpv6uovFQ0Afh2rVr6d69u4WgF6kq69atIz093d9VCRoHKip5f5Vz6bxgUwkicM5Z8YzLSOaSnok0bWLLlwayMx1QHRAsBL3L/j5/uGaR4YzJSGJMRhIFpQd5fUkhry8p5N7Zy2geGc4VfZxe54xOdukcbGxSNw/t2bOHZ5999gcfd9lll7Fnz56T7vPggw8yb968062a8YPkuGjuvagrn/3yAmZNymZEr0TeWl7E2OcXMOzPn/G3+Rso2lPu72oaDwXNpbG/L+G2bNnCFVdcwapVq457v7KykvDwoGlYHycQ/l4bkwMVlby7cjs5+YUs2lyKCJzbJYGxGUlc0jORqAi7dPank10aW4vQQ5MnT2bjxo3069ePrKwszjvvPEaOHEmPHj0AuOqqq8jIyKBnz55MnTq15riUlBR27drFli1bSE9P57bbbqNnz55cfPHFlJc7LYaJEyeSk5NTs/9DDz3EgAED6N27N+vWrQOguLiY4cOH07NnT2699VY6derErl27GvhvwZxMs8hwxmUmM/v2s/n8Vxfw02FpbCo+wM9mLSPr0XncP3clS7buJtAaHyaI7hEe9fDbq1lTtNern9mjfSwPXdnzpPs8/vjjrFq1imXLlvHpp59y+eWXs2rVqppe1+nTpxMXF0d5eTlZWVmMGTOG+Pj44z5j/fr1zJw5kxdffJGrr76a119/neuvv/57ZSUkJLBkyRKeffZZnnzySaZNm8bDDz/MsGHDuP/++3n//fd56aWTPqlo/KxjfDQ/H96Vey9MY+HmEnLyCnljaSEzF2+lc+tmjM1IYnT/JBJbRPm7qgZrEZ62gQMHHjf05Omnn6Zv375kZ2dTUFDA+vXrv3dMamoq/fr1AyAjI4MtW7bU+9mjR4/+3j5ffvklEyZMAGDEiBG0atXKi2djfMXlEs45K4Gnxvcj94GLeGJMHxKaRfLE+19zzuMfc9P0xby9vIhDR6r8XdWQFnQtwlO13BpKs2bHJvz89NNPmTdvHgsWLCA6OpqhQ4fWO/g7MvLYWr1hYWE1l8Yn2i8sLIzKykov19z4S0xUBFdnJXN1VjJbdh1wep3zC/npzKXERoVzZd/2jMtMpm9SC+t1bmAetQhFZISIfC0iG0Rkcj3bfy4ia0RkhYh8LCKdam27SUTWu39u8mblG1JMTAz79u2rd1tZWRmtWrUiOjqadevWsXDhQq+XP3jwYObMmQPAhx9+yO7du71ehmk4KQnN+MXF3fjyN8N49dZBDOvehteXFHLV375i+F8+5/nPNrJjrz1J1VBO2SJ0L970N2ot3iQib9WZU3ApkFlr8aYngPEiEgc8BGQCCuS7jw26/4vj4+MZPHgwvXr1omnTprRt27Zm24gRI3j++edJT0+nW7duZGdne738hx56iGuuuYZXXnmFs88+m8TERGJiYrxejmlYLpcwuEsCg7sksPfQEd5d4fQ6P/7eOp54fx3nd23N2IxkLurRhshw63X2FZ8u3iQi1+CsaHe7e9sLwKeqOvNE5QXq8Bl/q2UnpgEAAAzRSURBVKioICwsjPDwcBYsWMCdd97JsmXLzugz7e81cG0q3s/rSwqZu2Qb28sO0aJpBKP6tWdsRhK9O9il8+nw5+JN9R3bwYMyTR1bt27l6quvprq6miZNmvDiiy/6u0rGhzq3bs6vLunOz4d346sNu8jJL2R2bgEvL/iWrm2bMzYjiav6d6BNjPU6e4NXO0vcizdlAuf/wONsFbtTSEtLY+nSpf6uhmlgYS5hSNfWDOnamrLyI7yzYjs5+QX84d11/OHddXRPjCG7czyDUuMYmBpHfPPIU3+o+R5fL960DRha59hP6x6rqlOBqeBcGntQJ2NCToumEVw7qCPXDurIhp37eX/VdhZuKmV2bgH/+O8WALq2be4OxngGdY4jwYLRI54EYS6QJiKpOME2Abi29g7u+4Iv4CzetLPWpg+AP7gXcQK4GLj/jGttTIjr0qY5dw9L4+5hcLiympXb9rBwUymLNpeSk1/Iywu+rdkvu3NcTTDapXT9ThmEqlopInfjhNrRxZtW1168CfgT0Bx4zX0Td6uqjlTVUhH5PU6YAjyiqqU+ORNjQlSTcBcZneLI6BTHXRfAkapqVm0rcwdjCW8s2ca/Fm4FoHPrZgxKjSe7cxzZneNpG2vBCDbpQkizv9fQUFlVzeqivSzcVMKizaXkbi5lX4UzUD81oRmDUp1QHNQ5jnYtGu9Eszbpgh80b94cgKKiIsaOHVvvPkOHDuVU67NMmTKFgwcP1rz2ZFovY2oLD3PRN7klt59/FtMnZrHsoYt5++5z+e3l6ZzVuhnvrtzOvbOXcfYfP2HIE/P5dc5yXs8vZFsITSMWdI/YBZv27dvXzCxzOqZMmcL1119PdLSzmtq7777rraqZEBXmEnontaB3UgtuPa8zVdXK2u17WbS5lEWbSvhg9Q7m5BUCkNSqaU2vdHbn+Ea7qp8FoYcmT55McnIyd911FwC/+93vCA8PZ/78+ezevZsjR47w6KOPMmrUqOOOqz2PYXl5OTfffDPLly+ne/fuxz1rfOedd5Kbm0t5eTljx47l4Ycf5umnn6aoqIgLLriAhIQE5s+fT0pKCnl5eSQkJPDUU08xffp0AG699VbuvfdetmzZwqWXXsq5557Lf//7Xzp06MCbb75J06aN95LHnJkwl9CrQwt6dWjBLeemUl2tfL1jn3MpvamUj9fuICffCcYOLZsedyndMS66UQzuDr4gfG8yfLfSu5+Z2Bsuffyku4wfP5577723JgjnzJnDBx98wD333ENsbCy7du0iOzubkSNHnvCL8dxzzxEdHc3atWtZsWIFAwYMqNn22GOPERcXR1VVFRdeeCErVqzgnnvu4amnnmL+/PkkJCQc91n5+fn8/e9/Z9GiRagqgwYN4vzzz6dVq1YeT/dlTH1cLiG9XSzp7WK5ebATjOt37nffYyzhs2+KmbvUGUHXrkVUrWCMJyU+OIMx+ILQT/r378/OnTspKiqiuLiYVq1akZiYyH333cfnn3+Oy+Vi27Zt7Nixg8TExHo/4/PPP+eee+4BoE+fPvTp06dm25w5c5g6dSqVlZVs376dNWvWHLe9ri+//JIf/ehHNbPgjB49mi+++IKRI0d6PN2XMZ5wuYRuiTF0S4zhpnNSUFU27NzPws2lLNxUwpcbSvj3siIA2sRE1rQWszvH0zmhWVAEY/AF4Slabr40btw4cnJy+O677xg/fjyvvvoqxcXF5OfnExERQUpKymmtvbx582aefPJJcnNzadWqFRMnTjyjNZw9ne7LmNMhIqS1jSGtbQw3ZHdCVdlYfIBFm0tYuMkJx7eWO8GY0DyyJhSzU+Po0qZ5QAZj8AWhH40fP57bbruNXbt28dlnnzFnzhzatGlDREQE8+fP59tvvz3p8UOGDGHGjBkMGzaMVatWsWLFCgD27t1Ls2bNaNGiBTt27OC9995j6NChwLHpv+peGp933nlMnDiRyZMno6q88cYbvPLKKz45b2NORkTo0qY5Xdo057pBTjBuKTnovsfohOM7K7YDkNC8CQOPXkqnxpPWpjkul/+D0YLwB+jZsyf79u2jQ4cOtGvXjuuuu44rr7yS3r17k5mZSffu3U96/J133snNN99Meno66enpZGRkANC3b1/69+9P9+7dSU5OZvDgwTXHTJo0iREjRtC+fXvmz59f8/6AAQOYOHEiAwcOBJzOkv79+9tlsPE7ESE1oRmpCc24ZmBHVJWtpQdrOl8Wbirh3ZXfARDXrAkDU+JqWo3d2sb4JRhtQHUIs79X4w+qSuHucha6W4uLNpdQuNu5fdMyOsIdjM7TL+mJsV4LxkaxwLsxpnEQEZLjokmOi2ZcpjOfS+HugzWtxUWbS/lwzQ4AYqPCGeh+JHBQajw92scS5oMWowWhMcbvklpFk5QRzZiMJACK9pSzaPOxS+l5a51gjIkKJysljuzOcVzQrQ1pbb0zS7sFoTEm4LRv2ZQf9U/iR/2dYPyu7FBNr/SiTSV8sm4n+w5V8ouLu3mlPI+CUERGAH/FmX1mmqo+Xmf7EGAK0AeYoKo5tbZVAUdHQG9V1ZGnU1FVDchu92AVaPeGjTmZxBZRjOrXgVH9nAnud+49BF6MA28t3rQVmAj8sp6PKFfVfmdSyaioKEpKSoiPj7cw9AJVpaSkhKgom4LJBKc2Xp4+zJMW4UBgg6puAhCRWcAooCYIVXWLe1u1V2vnlpSURGFhIcXFxb74+JAUFRVFUlKSv6thTEDwxeJNdUWJSB5QCTyuqv/+AccCEBERQWpq6g89zBhjPNIQnSWdVHWbiHQGPhGRlaq6sfYOtniTMcafPJmY1aPFm05EVbe5/9yEs3BT/3r2maqqmaqa2bp1a08/2hhjvMKTIKxZvElEmuAs3vSWJx8uIq1EJNL9ewIwmFr3Fo0xJhB49IidiFyGMzzm6OJNj9VevElEsoA3gFbAIeA7Ve0pIufgrG5XjRO6U1T1pVOUVQycfPaC70sAdv3AY7wplMsP5XP3d/mhfO6nU34nVa33kjPgnjU+HSKSd6JnCK38xlt2qJcfyufu7fJt8SZjTMizIDTGhLzGEoRTrfyQLDvUyw/lc/dq+Y3iHqExxpyJxtIiNMaY0xZUQSgiI0TkaxHZICKT69keKSKz3dsXiUhKA5c/UUSKRWSZ++dWL5Y9XUR2isiqE2wXEXnaXbcVIjKgvv18VPZQESmrdd4Peqts9+cni8h8EVkjIqtF5Gf17OOT8/ewbJ+dv4hEichiEVnuLv/hevbx2ffew/J99r13f36YiCwVkf/Us807566qQfGDM4ZxI9AZaAIsB3rU2ecnwPPu3ycAsxu4/InAMz46/yHAAGDVCbZfBryHMzlRNrCoAcseCvzHh//t2wED3L/HAN/U83fvk/P3sGyfnb/7fJq7f48AFgHZdfbx5ffek/J99r13f/7PgRn1/R1769yDqUVYMwuOqh4Gjs6CU9so4J/u33OAC8V783Z5Ur7PqOrnQOlJdhkFvKyOhUBLEWnXQGX7lKpuV9Ul7t/3AWtxJgOpzSfn72HZPuM+n/3ulxHun7o39n32vfewfJ8RkSTgcmDaCXbxyrkHUxDWNwtO3S9kzT6qWgmUAfENWD7AGPelWY6IJNez3Vc8rZ+vnO2+fHpPRHr6qhD3pU9/nJZJbT4//5OUDT48f/el4TJgJ/CRqp7w3H3wvfekfPDd934K8Gucp9Pq45VzD6YgDAZvAymq2gf4iGP/UjV2S3AeX+oL/D/gB0+15gkRaQ68Dtyrqnt9UcZplu3T81fVKnUmN04CBopIL29+vhfK98n3XkSuAHaqar43Pu9kgikIPZkFp2YfEQkHWgAlDVW+qpaoaoX75TQgw0tle+KMZgk6E6q69+jlk6q+C0SIM8mG14hIBE4Qvaqqc+vZxWfnf6qyG+L83Z+9B5gPjKizyZff+1OW78Pv/WBgpIhswbkVNUxE/lVnH6+cezAFoSez4LwF3OT+fSzwibrvojZE+XXuSY3EuZ/UUN4CbnT3nmYDZaq6vSEKFpHEo/dlRGQgzvfKa/8juj/7JWCtqj51gt18cv6elO3L8xeR1iLS0v17U5wlM9bV2c1n33tPyvfV915V71fVJFVNwfn/7RNVvb7Obt45d1/19PjiB6dn8Buc3tsH3O89Aox0/x4FvAZsABYDnRu4/D8Cq3F6lOcD3b1Y9kxgO3AE5/7XLcAdwB3u7YKztsxGnMWyMhuw7LtrnfdC4Bwv/72fi3ODfgWwzP1zWUOcv4dl++z8cRZEW+oufxXwYEN+7z0s32ff+1r1GIq719gX525PlhhjQl4wXRobY4xPWBAaY0KeBaExJuRZEBpjQp4FoTEm5FkQGmNCngWhMSbkWRAaY0Le/wd66zhJ3x53AwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 360x216 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUkAAADSCAYAAADHXDKcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXhV5bX48e/KQEIggYQwJSEEFWQeAgQURetUQAQFnFArtmod6tThV7x6naqtvddaa1vtVetYQJFJbMUZZ2UKYZ5lSgIkkIEASciwfn/snXgIITmEc3KSk/V5nvNwsve73yEcFu9+9z57iapijDGmdiGB7oAxxjRlFiSNMaYOFiSNMaYOFiSNMaYOFiSNMaYOFiSNMaYOFiSNMaYOFiSNMaYOFiRNiyEO+8ybk2IfGNPoRGS6iGwTkSIRWS8iV3jsu0VENnjsS3W3dxOReSKSKyIHRORv7vZHRORfHseniIiKSJj782ci8oSIfA0cAU4TkZs82vheRH5eo38TRSRDRA66/RwjIleKyIoa5X4pIu/47zdlmoKwQHfAtEjbgHOBvcCVwL9E5AzgHOAR4HJgOXA6UCYiocC/gU+BG4AKYNhJtHcDMBbYBAhwJjAe+B4YDSwSkWWqmi4iacDrwBTgE6ArEA1sB/5PRPqo6gaPeh9vyC/ANB82kzSNTlXfVtVsVa1U1beALUAacDPwP6q6TB1bVXWnuy8B+I2qHlbVElX96iSafFVV16lquaqWqep/VHWb28bnwIc4QRvgZ8DLqvqR278sVd2oqqXAW8D1ACLSD0jBCd4miFmQNI1ORH7ins4WiEgB0B+IB7rhzDJr6gbsVNXyBja5u0b7Y0XkOxHJc9sf57Zf1VZtfQB4DZgqIoIzi5ztBk8TxCxImkYlIt2BF4FfAB1UtT2wFuc0eDfOKXZNu4HkqnXGGg4DUR4/d6mlTPWjrkQkApgLPAV0dtt/z22/qq3a+oCqfgccxZl1TgXeqH2UJphYkDSNrQ1O0MoFEJGbcGaSAC8BvxaRoe6V6DPcoLoU2AM8KSJtRCRSREa5x2QAo0UkWUTaAffX034rIMJtv1xExgKXeOz/J3CTiFwoIiEikigivT32vw78DSg7yVN+00xZkDSNSlXXA38CvgX2AQOAr919bwNPADOBImABEKeqFcBlwBnALiATuNo95iOctcLVwArqWSNU1SLgbmA2kI8zI1zosX8pcBPwZ6AQ+Bzo7lHFGzhB/V+YFkHsobvGeE9EWgM5QKqqbgl0f4z/2UzSmJNzO7DMAmTLYfdJGuMlEdmBc4Hn8gB3xTQiO902xpg62Om2McbUwYKkMcbUoVmtScbHx2tKSkqgu2GMCTIrVqzYr6oda9vXrIJkSkoKy5cvD3Q3jDFBRkR2nmifnW4bY0wdLEgaY0wdLEgaY0wdmtWaZG3KysrIzMykpKQk0F0JCpGRkSQlJREeHh7orhjTJDT7IJmZmUl0dDQpKSk4j/kzDaWqHDhwgMzMTHr06BHo7hjjteKjFazcnc/yHfkcLC7jwfF9fVZ3sw+SJSUlFiB9RETo0KEDubm5ge6KMXU6cKiU5TvzWb4jj6U78lmXVUh5pSICAxPbUVmphIT4JiY0+yAJWID0IftdmqZGVdmdV8yyHXnVr225hwFoFRrCoG7tuGX0aaSlxJGaHEu7KN8uFQVFkAykgoICZs6cyR133HFSx40bN46ZM2fSvn37E5Z56KGHGD16NBdddNGpdtOYZqOiUtmw5yDLd+SxbEc+y3bkkVPkZMmIiQxjWEock4cmkZYSR//EdkSGh/q1PxYkT1FBQQHPPffccUGyvLycsLAT/3rfe++9eut+7LHHTrl/xjR1JWUVrNxV4ATFnfmk78znUKmTziihXSRnnd6BYSlxDE+JpVenaJ+dRnvLguQpmj59Otu2bWPw4MGEh4cTGRlJbGwsGzduZPPmzVx++eXs3r2bkpIS7rnnHm699Vbgh28PHTp0iLFjx3LOOefwzTffkJiYyDvvvEPr1q2ZNm0a48ePZ8qUKaSkpHDjjTfy7rvvUlZWxttvv03v3r3Jzc1l6tSpZGdnc9ZZZ/HRRx+xYsUK4uPj6+m5MYGRf/goy3fmV586r80qpKzCeRrZmZ2jmTg4gbQecQxLiSOxfesA9zbIguSj765jffZBn9bZNyGGhy/rd8L9Tz75JGvXriUjI4PPPvuMSy+9lLVr11ZfHX755ZeJi4ujuLiY4cOHM3nyZDp06HBMHVu2bGHWrFm8+OKLXHXVVcydO5frr7/+uLbi4+NJT0/nueee46mnnuKll17i0Ucf5YILLuD+++/n/fff55///KdPx2/MqVBVMvOr1hOdwLg15xDgrCcOTGrHz845jbQesaQmx9I+qlWAe3y8oAqSTUFaWtoxt888++yzzJ8/H4Ddu3ezZcuW44Jkjx49GDx4MABDhw5lx44dtdY9adKk6jLz5s0D4Kuvvqquf8yYMcTGxvp0PMacjIpKZdPeoupZ4vId+ew96NzDHB0ZxrDusVwxJJHhKXEMTPL/eqIvBFWQrGvG11jatGlT/f6zzz7j448/5ttvvyUqKorzzz+/1pveIyIiqt+HhoZSXFxca91V5UJDQykvb2gKamN8p6SsglW7C1i+M5+l2/NI35lPkbue2CUmkuE94khLiWVYShy9OkcT2sjrib4QVEEyEKKjoykqKqp1X2FhIbGxsURFRbFx40a+++47n7c/atQoZs+ezW9/+1s+/PBD8vPzfd6GMVUKjhxl+Y58lu3MY9n2PNZ4rCf26tyWywYnMDwlluHuemIw3FJmQfIUdejQgVGjRtG/f39at25N586dq/eNGTOGf/zjH/Tp04czzzyTkSNH+rz9hx9+mGuvvZY33niDs846iy5duhAdHe3zdkzLo6pkFRSzfEc+S3fksXxHHpv3OeuJ4aHCwKT2/PScHgzvHsfQ7rHEtml664m+4FWOGxEZA/wFCAVeUtUna+zvDrwMdATygOtVNVNEBgPPAzFABfCEqr7lHvMqcB5ObmOAaaqaUVc/hg0bpjWfJ7lhwwb69OlT7xiCVWlpKaGhoYSFhfHtt99y++23k5FR56+xXi39d9pSVVYqm/YVHXN/4p5Cdz0xIozU7rHOVefusQzq1r5ZrCd6S0RWqOqw2vbVO5MUkVDg78DFOEnhl4nIQjfJfJWngNdV9TURuQD4A3ADcAT4iapuEZEEYIWIfKCqBe5xv1HVOQ0fmtm1axdXXXUVlZWVtGrVihdffDHQXTLNRElZBaszC90LLHks35lPUYmzntg5JoLhKXHVrzO7NM/1RF/w5nQ7Ddiqqt8DiMibwETAM0j2BX7pvl8MLABQ1c1VBVQ1W0RycGabBRif6NmzJytXrgx0N0wzUHikjBW78li63fnO8+rMQo5WVALQs1Nbxg/8YT0xKTY41hN9wZsgmQjs9vg5ExhRo8wqYBLOKfkVQLSIdFDVA1UFRCQNaAVs8zjuCRF5CPgEmK6qpTUbF5FbgVsBkpOTveiuMQZw1xPd7ztvz2fTPucCY1iIMCCpHdNGpTA8xVlPjAvS9URf8NWFm18DfxORacAXQBbOGiQAItIVeAO4UVUr3c33A3txAucLwG+B476Hp6ovuPsZNmyYJQk35gQqKpWvt+7nnYxsvvv+AFkFzq1kbd31xPEDuzK8RxyDktrTulXwrCf6mzdBMgvo5vFzkrutmqpm48wkEZG2wOSqdUcRiQH+Azygqt95HLPHfVsqIq/gBFpjzEnavK+IuemZLFiZxb6DpcREhnFur47ccm4PhqXE0adrTItdT/QFb4LkMqCniPTACY7XAFM9C4hIPJDnzhLvx7nSjYi0AubjXNSZU+OYrqq6R5yFj8uBtac6GGNaigOHSlm4Kpt56VmsySokLEQ4/8yOPHJZEhf06UREmM0UfaXeHDeqWg78AvgA2ADMVtV1IvKYiExwi50PbBKRzUBn4Al3+1XAaGCaiGS4r8HuvhkisgZYA8QDj/tqUE1Z27ZtAcjOzmbKlCm1ljn//PPrTZ37zDPPcOTIkeqfx40bR0GBXQ8LZqXlFSxas4ebX1vOiN9/wqPvrkdRHr6sL9/914W8dONwxg7oagHSx7xak1TV94D3amx7yOP9HOC4W3lU9V/Av05Q5wUn1dMgk5CQwJw5Db/76ZlnnuH6668nKioK8O7Ra6b5UVVW7i5gXnom767aQ2FxGZ2iI/jZOT2YlJrEmV3siwP+Zt+4OUXTp0+nW7du3HnnnQA88sgjhIWFsXjxYvLz8ykrK+Pxxx9n4sSJxxy3Y8cOxo8fz9q1aykuLuamm25i1apV9O7d+5jvbt9+++0sW7aM4uJipkyZwqOPPsqzzz5LdnY2P/rRj4iPj2fx4sXVj16Lj4/n6aef5uWXXwbg5ptv5t5772XHjh0nfCSbaXqyCoqZn57JvPQsvt9/mMjwEH7crwuTU5MYdUa8rTE2ouAKkoumw941vq2zywAY++QJd1999dXce++91UFy9uzZfPDBB9x9993ExMSwf/9+Ro4cyYQJE05439nzzz9PVFQUGzZsYPXq1aSmplbve+KJJ4iLi6OiooILL7yQ1atXc/fdd/P000+zePHi454buWLFCl555RWWLFmCqjJixAjOO+88YmNjvX4kmwmMQ6XlLFqzh3npWXz7vXP33Igecdx2/umM7d+F6EjLYBkIwRUkA2DIkCHk5OSQnZ1Nbm4usbGxdOnShfvuu48vvviCkJAQsrKy2LdvH126dKm1ji+++IK7774bgIEDBzJw4MDqfbNnz+aFF16gvLycPXv2sH79+mP21/TVV19xxRVXVD+NaNKkSXz55ZdMmDDB60eymcZTUal8u+0Ac9MzeX/tXorLKkjpEMUvL+7FFUMS6RYXFegutnjBFSTrmPH505VXXsmcOXPYu3cvV199NTNmzCA3N5cVK1YQHh5OSkpKg/KCb9++naeeeoply5YRGxvLtGnTTim/uLePZDP+tzWniDkrsliwMou9B0uIiQzjitREJqcmkZrc3r7t0oTUe3Xb1O/qq6/mzTffZM6cOVx55ZUUFhbSqVMnwsPDWbx4MTt37qzz+NGjRzNz5kwA1q5dy+rVqwE4ePAgbdq0oV27duzbt49FixZVH3OiR7Sde+65LFiwgCNHjnD48GHmz5/Pueee68PRmobKO3yU177ZwYS/fcVFT3/Bi19+T7+EGP4+NZWlD1zE768YwNDusRYgm5jgmkkGSL9+/SgqKiIxMZGuXbty3XXXcdlllzFgwACGDRtG79696zz+9ttv56abbqJPnz706dOHoUOHAjBo0CCGDBlC79696datG6NGjao+5tZbb2XMmDEkJCSwePHi6u2pqalMmzaNtLQ0wLlwM2TIEDu1DpDS8goWb8xlbnomizfmUF6p9EuI4b/H92Xi4ATi20bUX4kJKK8eldZU2KPSGof9Tk+NqrIqs5C5KzJ5d3U2BUfK6BgdwRVDEpmUmkjvLjGB7qKp4ZQelWaM8U52QTHzV2YxLz2TbbmHiQhzbtuZlJrIOWfEExZqq1vNkQVJY07B4dJy3l+7l7npmXz7/QFUIa1HHLeOPo2xA7oSY7ftNHsWJI05SRWVynffH2DuikwWubftdO8Qxb0XOrftJHew23aCSVAESVW1K4I+0pzWqBvb1pxDzEvPZP7KLPYUlhAdGcblQxKZnJpoV6WDWLMPkpGRkRw4cIAOHTrYh/QUqSoHDhwgMjIy0F1pMvIPH+Xd1dnMXZHJqsxCQkOE83p15IFL+3BRn85BlefF1K7ZB8mkpCQyMzPJzc0NdFeCQmRkJElJSYHuRkAdLa9k8aYc5qVn8unGHMoqlD5dY3jw0j5MGJxAp2j7T6Ql8SpINjRborvvRuBBt+jjqvqau30o8CrQGucJQ/doA871wsPD6dGjx8keZswxVJXVmYXMS89k4aps8o+UEd82gmlnp3DFkCT6JthtOy2VX7Mlikgc8DAwDFCcbIkLVTUfJ9XsLcASnCA5BliEMY1oT2HVbTtZbM05RKuwEC7p25nJQ5M4127bMfg5WyLwY+AjVc1zj/0IGCMinwExVekcROR1nKeTW5A0fnfkqHPbzrz0LL7eth9VGJ4Sy5OTBjB2QFfatbbbdswP/Jot8QTHJrqvzFq2G+MXlVW37aRnsWjtHo4crSA5Lop7LuzJFUMS6d6hTaC7aJqoRsmWeCospaw5Fdty3dt20rPILiwhOiKMCYMSmDw0iWF2247xgl+zJYpIFk7+G89jP3OPT6qx/Zg6Peq2lLLmpBQeKWPhqizmpmeRsbuAEIHRvToyfVwfLulrt+2Yk+PXbIk4ycN+LyKx7s+XAPerap6IHBSRkTgXbn4C/PWUR2NarKpcMDOX7OLfq7MpKaukd5doHhjXh4mDE+gUY7ftmIapN0iqarmIVGVLDAVersqWCCxX1YU4s8U/iIjinG7f6R6bJyK/wwm0AI9VXcQB7uCHW4AWYRdtTAMUlZSxICObmUt2sWHPQdq0CmVSahJT05Lpn9gu0N0zQaDZPyrNtExrMguZuXQn72Rkc+RoBf0SYrhuRHcmDE6gbUSz/46EaWT2qDQTFI4cLWdhRjYzl+5idWYhkeEhTBiUwNQR3RmU1M4uwhi/sCBpmrwNew4yc8kuFqzMoqi0nDM7R/PohH5cPiTR7mk0fmdB0jRJJWUV/Gf1HmYs2Un6rgJahYUwfkBXpo5ItifumEZlQdI0KVtzDjFzyS7mpmdSWFzGafFtePDSPkxOTSK2TatAd8+0QBYkTcCVllfw/tq9zFiyi6Xb8wgPFcb078rUtGRGnhZns0YTUBYkTcDs2H+YWUt38faKTPIOHyU5LorpY3szZWiSZRE0TYYFSdOoyioq+Wj9PmYu2cVXW/cTGiJc3Kcz141MZtTp8YSE2KzRNC0WJE2j2J13hDeX7eKtZZnsP1RKYvvW/PqSXlw1rJt9G8Y0aRYkjd+UV1Ty6cYcZi7dxeebcxHggt6dmDoimfN6dSLUZo2mGbAgaXxuT2Exby7dzVvLdrP3YAmdYyK464KeXDO8GwntWwe6e8acFAuSxicqKpUvNucyY8kuPt24DwVG9+zIoxP7cWHvTvaEb9NsWZA0pyTnYAmzl+9m1tLdZBUUE982gtvOO51r05LpFmf5p03zZ0HSnLTKSuWbbQeYsWQnH63fR3mlMuqMDvzXuD5c3LczrcJs1miCh6+yJSYDrwHt3TLTVfU9EbkO+I1H0YFAqqpmuHluugLF7r5LVDXnVAZj/OvAoVLeXpHJrKW72HngCLFR4fz0nB5cm5ZMj3hLf2CCk6+yJT4IzFbV50WkL072wxRVnQHMcOsZACxQ1QyP465TVXv2WROmqizZnseMJbv4YO1ejlZUktYjjl9e3Isf9+tiT/k2Qc9X2RIVqEpM3A7IrqWea4E3G95V05gKjhxlbnoWM5fsZFvuYWIiw7huZDJT05Lp2Tk60N0zptH4KlviI8CHInIX0Aa4qJZ6rsYJrp5eEZEKYC7wuDanJwAHIVUlfVc+M77bxX/W7KG0vJIhye156spBXDqgK61b2azRtDy+unBzLfCqqv5JRM4C3hCR/m7OG0RkBHBEVdd6HHOdqmaJSDROkLwBeL1mxZYt0f8OlpSxYGUWM77bxaZ9RbSNCOPKYUlMTetO34SY+iswJoj5JFsi8DNgDICqfisikUA8UHUh5hpglucBqprl/lkkIjNxTuuPC5KWLdE/VJXVmYXMXLKLhauyKS6rYEBiO56cNIDLBiXQxlIgGAP4KFsisAu4EHhVRPoAkUAugIiEAFcB51YVFpEwoL2q7heRcGA88PEpjsV44XBpOe9kZDNz6U7WZh2kdXgoEwcnMHVEMgOT2ge6e8Y0Ob7Klvgr4EURuQ/nIs40j/XF0cDuqgs/rgjgAzdAhuIEyBd9NipznHXZzqzxnYxsDpWW07tLNL+b2I+JQxKJibQUCMaciGVLDGLFRyt4d7WTbjVjdwERYSGMH+jMGlOT29vDbI1xWbbEFui5z7by/GfbKCop54xObXlofF8mpybRLspmjcacDAuSQeidjCz+5/1NXNC7Ez8ffRppPSwFgjENZUEyyGzLPcR/zVvDsO6x/N8NQwm3p+8Yc0rsX1AQKSmr4M4Z6bQKC+GvU4dYgDTGB2wmGUQefXcdG/cW8cpNw+nazh5ua4wv2FQjSCxYmcWspbu5/fzT+dGZnQLdHWOChgXJILA15xD/NX8NaSlx/OriXoHujjFBxYJkM1d81FmHjAwP5dlrh1iaBGN8zNYkm7lHFq5jc04Rr96URpd2lprVGF+zaUczNi89k7eW7+bO88/gvF4dA90dY4KSBclmamtOEQ/MX8uIHnHce1HPQHfHmKBlQbIZOnK0nDtmpBPVytYhjfE3W5Nshh5+Zx1bcg7x+k/T6Bxj65DG+JNNQZqZOSsyeXtFJnf96AzO7WnrkMb4m1dBUkTGiMgmEdkqItNr2Z8sIotFZKWIrBaRce72FBEpFpEM9/UPj2OGisgat85nxZ7AUK/N+4p4cMEaRp4Wxz0X2f2QxjSGeoOkR0rZsUBf4Fo3baynqpSyQ3CeXP6cx75tqjrYfd3msf154Bagp/sa0/BhBL8jR8u5c0Y6bSPCePaaIYSG2P8pxjQGb2aS1SllVfUoTlrYmlkPvUkpW01EugIxqvqd+wTz14HLT6rnLcx/L1jH1txD/OWaIXSydUhjGo03QbK2lLKJNco8AlwvIpnAe8BdHvt6uKfhn4tIVZ6bRLeeuuoEnGyJIrJcRJbn5uZ60d3gM3v5buamZ3L3BT0ZdUZ8oLtjTIviqws3VSllk4BxOCllQ4A9QLJ7Gv5LYKaInFSOUlV9QVWHqeqwjh1b3oWKTXuLeOidtZx9egfuvtDuhzSmsfk1payq5gCl7vYVIrIN6OUen1RPnS3e4dJy7pixgrYR4TxzzWBbhzQmALyZSVanlBWRVjgXZhbWKFOVUhbPlLIi0tG98IOInIZzgeZ7Vd0DHBSRke5V7Z8A7/hkREFCVXlwwVq27z/Ms9cMplO0rUMaEwh+TSkrIqOBx0SkDKgEblPVPLfqO4BXgdbAIvdlXLOX72b+yizuu6gXZ9s6pDEBYyllm6CNew8y8W9fMzwljtd+mman2cb4WV0pZe0bN03MoVLne9kxrcP589W2DmlMoFmQbEJUlQfmr2HH/sM8e80QOkZHBLpLxrR4FiSbkDeX7eadjGzuu6gXZ53eIdDdMcZgQbLJWJ99kIcXruPcnvHc+aMzAt0dY4zLgmQTcKi0nDtnphMb5axDhtg6pDFNhj1PMsBUlfvnrWHngcPMumUk8W1tHdKYpsRmkgE2c+ku3l2Vza8uOZMRp9k6pDFNjQXJAFqbVcij765ndK+O3H7e6YHujjGmFsEbJMtLoWgfVFYGuie1Kiop4xcz04mLasWfrxpk65DGNFHBuyaZvRJe/jGEhEN0V2iXCDEJ7ivR45UAbTtBSGijdU1VmT5vDbvzi3nz1pF0sHVIY5qs4A2S7ZNh3FNwMAsOZkNhFmSlw4Z/Q0XpsWVDwpxAelwQdd+3S4S2nX0WSP+1ZBf/Wb2H/zfmTIanxPmkTmOMfwRvkIxJgLRbjt+uCkfy4GCmEzw9g+jBLNizGjYtgvKSY4+TUIjuUksQTYB2Se6MtAuE1v0rXZtVyO/eXc/5Z3bkttG2DmlMUxe8QfJERKBNB+fVdVDtZVShON8jgFYFVDeo7lsLmz+A8uIadYc4M07PWWhMgnuqn0hRRCfunrGNDm1b8fRVdj+kMc2BV0FSRMYAf8F5VNpLqvpkjf3JwGtAe7fMdFV9T0QuBp4EWgFHgd+o6qfuMZ8BXYGqSHOJ+5DewBOBqDjn1WVA7WVUoaTg2FlodSDNhNyNsPUTKDtcfUg08LEK5VEdaTUj6djTec+ZaXQChLVqnLE2F6pQUQZlR5xZflmx8yp3/ywr8XhfDBVHnb9HCfF4hdb42d0fUnN7Pa/q8jXrr9mG1HJMA9owAVVvkPTIlngxTi6aZSKyUFXXexSrypb4vJtJ8T0gBdgPXKaq2SLSH+eZlJ65bK5T1eb57DMRaB3rvDr3q72MKpQUwsFsPl6yko+WrOTKnjAsttgJpge2wvefw9Gi449t06lGEE04foYaFuALPqrOXQTHBa4SZ1vNwFUd1ErqPuZE+7Rp3qngdw0JxCGhzlp7SJhz8bLq59Bwd1uou90tExp2fPlQj/2er9rKHlP3icqfqD3PsifoX0jgbsTxZiZZnS0RQESqsiV6BslasyWq6kqPMuuA1iISoao1rpwEKRFo3Z41B4Q7luzknJ5Xk/qTYVDzNLvkoMf6aNaxa6V538OOr6C08Pj6o+KPXROtuV4aFlEj6NQXuIq931cVxGjg80jDoyAs0vkzPBLCWkN4a+d96/buPneb577q42ru8/w5EkIjnL5ppfOqrPjhvVY6AV4rQWturypfy7bqV4XH8ZUnaENrlK9Zh9bSp3rqP6aNmvs96qp0f64sd2bfleXOcZUe78uPQuURd1tdZcuhovyH91rRsL/vUyUhdQTmGgE1ujPcMN9nTXsTJGvLljiiRplHgA9F5C6gDXBRLfVMBtJrBMhXRKQCmAs8rs3pCcBeKiwu446ZK4hv24o/XXmC+yEjY5xXp94nrqi0CA7u8bjg5LFWmr8Tdn7jnP43lITUEnDcoNQqCqI61LKvnkB1XECr2hZhp5HNVX0BuMIj6HoG5QqPoOv5Opny3gTxynKIbOfTIfvqwk1VtsQ/ichZONkS+6s650ci0g/4I3CJxzHXqWqWiETjBMkbcPJvH0NEbgVuBUhOTvZRdxuHqvLbOavZU1DCWz8/i9g2p7DOGBENHaOhY68Tlyk9BEV7nFloYZbzQao5Y6s1iEU5/yNb4DL1CQkBQpzPS3jrQPemUfg1WyKQIyJJwHzgJ6q6reoAVc1y/ywSkZk4p/XHBUlVfQF4AZz0DV6Oq0l49ZsdvL9uLw+M68PQ7rH+bzCiLUT0hHhLPWuMr/g7W2J74D84V7u/riosImEiEu++DwfGA2tPdTBNyYNT2MgAAApgSURBVKrdBfz+vQ1c1KcTN5/bI9DdMcY0UL1BUlXLgapsiRtwrmKvE5HHRGSCW+xXwC0isgqYhZst0T3uDOAhEclwX52ACOADEVkNZODMTF/09eACpfBIGXfOTKdTdCRPXTkIsdNYY5oty5boY6rKz99Ywacbc5h921mkJjfCabYx5pRYtsRG9PLXO/hw/T6mj+1tAdKYIGBB0ocydhfw5KINXNy3Mz87x9YhjQkGFiR9pODIUe6c4a5DTrF1SGOCRct7wIUfqCq/fns1OUUlvH3b2bSLCg90l4wxPmIzSR/451fb+XjDPu4f24fB3doHujvGGB+yIHmK0nfl8+Sijfy4X2duGpUS6O4YY3zMguQpKDhylLtmrqRLu0j+x9YhjQlKtibZQKrKr2avIqeohDm3nU271rYOaUwwsplkA7345fd8sjGHB8b1YZCtQxoTtCxINsCKnXn88f1NjO3fhRvPTgl0d4wxfmRB8iTlH3bWIRPbt+aPUwbaOqQxQc7WJE9CZaXyy9kZ7D90lLm3n01MpK1DGhPsbCZ5El748nsWb8rlwfF9GJDk26cfG2OaJguSXlq2I4///WATlw7oyg0juwe6O8aYRuJVkBSRMSKySUS2isj0WvYni8hiEVkpIqtFZJzHvvvd4zaJyI+9rbMpyXPXIZNiW/OHyQNsHdKYFqTeIOmRUnYs0Be41k0b66kqpewQnCeXP+ce29f9uR9OeofnRCTUyzqbhMpK5b63Msg7fJS/T021dUhjWhhvZpLVKWVV9ShQlVLWU60pZd1yb6pqqapuB7a69XlTZ5Pwjy+28fnmXP77sr70T7R1SGNaGm+CZG0pZRNrlHkEuF5EMoH3gLvqOdabOgEnW6KILBeR5bm5uV5013eWbs/jTx9uZvzArlw/onllajTG+IavLtxUpZRNAsbhpJT1Sd2q+oKqDlPVYR07dvRFlV45cKiUu2alkxwXxR8m2TqkMS2Vv1PK1nVsfXUGTGWlct/sVeQfKePlacOJtnVIY1osv6aUdctdIyIRItID6Aks9bLOgHn+8218sTmXhy/rS78EW4c0piWrdyapquUiUpVSNhR4uSqlLLBcVRfipJR9UUTuw7mIU5VSdp2IzAbWA+XAnapaAVBbnX4Y30n77vsD/OnDTUwYlMDUNFuHNKals5SyHvYfKmXcX76kbUQYC+86h7YR9q1NY1qCulLKWhRwVbj3QxYWl/HaT9MsQBpjAAuS1Z5bvJUvt+znD5MG0KdrTP0HGGNaBPvuNvDNtv38+ePNXD44gWuGd6v/AGNMi9Hig2RuUSn3vJlBSnwbnrjC7oc0xhyrRZ9uV1Qq9761koPFZbzxszTa2DqkMaaGFh0V/vrpFr7eeoA/Th5A7y62DmmMOV6LPd3+Zut+/vLJFiYNSeSqYbYOaYypXYsMkjlFJdz9Zganxbfhd5f3t3VIY8wJtbjT7YpK5Z5ZGRwqLWPGzSNsHdIYU6cWFyH+8skWvv3+AP87ZSBndokOdHeMMU1cizrd/mrLfv766RYmpyZxpa1DGmO80GKCZM7BEu59ayVndGzL7y7vF+juGGOaiRZxul1eUclds1ZyuLSCWbekEtWqRQzbGOMDvsqW+GcRyXBfm0WkwN3+I4/tGSJSIiKXu/teFZHtHvsG+3ZoP/jLJ1tYsj2Pxy/vT8/Otg5pjPFevVMqj8yGF+PkolkmIgtVdX1VGVW9z6P8XcAQd/tiYLC7PQ4nEdiHHtX/RlXn+GAcJ/TF5lz+tngrVw5NYvLQJH82ZYwJQr7KlujpWmBWLdunAItU9cjJd7Nh9h0s4b63MujZqS2PTezfWM0aY4KIr7IlAiAi3YEewKe17L6G44PnEyKy2j1dj/CiL16rWocsLqvguetSad0q1JfVG2NaCF9f3b4GmFOVoqGKiHQFBuCka6hyP9AbGA7EAb+trcKGppT9fHMuS7fn8cQV/Tmjk61DGmMaxpsg6U22xCq1zRYBrgLmq2pZ1QZV3aOOUuAVnNP64zQ0peyFfTrz7i/O4Yohtg5pjGk4X2VLRER6A7HAt7XUcdw6pTu7RJwvTl8OrD25rtdvQJJlOjTGnBpfZUsEJ3i+qTUyi4lICs5M9PMaVc8QkY6AABnAbacyEGOM8QfLlmiMafHqypbYYr6WaIwxDWFB0hhj6mBB0hhj6tCs1iRFJBfYeZKHxQP7/dCdpt52S2+/JY890O03x7F3V9Va7zFsVkGyIURk+YkWZIO57Zbefksee6DbD7ax2+m2McbUwYKkMcbUoSUEyRdaaNstvf2WPPZAtx9UYw/6NUljjDkVLWEmaYwxDRYUQdKL9BIRIvKWu3+J+33yxmx/mojkeqSquNmHbb8sIjkiUusDQsTxrNu31SKS6qu2vWz/fBEp9Bj7Qz5su5uILBaR9SKyTkTuqaWM38bvZfv+HH+kiCwVkVVu+4/WUsYvn30v2/bb596jjVARWSki/65ln2/GrqrN+oXz0I1twGlAK2AV0LdGmTuAf7jvrwHeauT2pwF/89P4RwOpwNoT7B8HLMJ5kMhIYEkjt38+8G8/jb0rkOq+jwY21/K799v4vWzfn+MXoK37PhxYAoysUcYvn30v2/bb596jjV8CM2v7Hftq7MEwk/QmvcRE4DX3/RzgQvcRbY3Vvt+o6hdAXh1FJgKvq+M7oH3VY+oaqX2/UeeZpOnu+yJgA8c/Nd9v4/eyfb9xx3TI/THcfdW8yOCXz76XbfuViCQBlwIvnaCIT8YeDEHSm/QS1WVUtRwoBDo0YvsAk93TvTki0q2W/f7idfoNPzrLPS1bJCJ+SXrunkoNwZnReGqU8dfRPvhx/O7pZgaQA3ykqiccv68/+160Df793D8D/D+g8gT7fTL2YAiSzcG7QIqqDgQ+4of/3VqCdJyvfA0C/gos8HUDItIWmAvcq6oHfV3/Kbbv1/GraoWqDsbJGJAmIo2W8c6Ltv32uReR8UCOqq7wVZ0nEgxB0pv0EtVlRCQMaAccaKz2VfWAOmkqwDk1GOqjtr1xMuk3fE5VD1adlqnqe0C4iMT7qn4RCccJUDNUdV4tRfw6/vra9/f4PdopABYDY2rs8udnv862/fy5HwVMEJEdOEtcF4jIv2qU8cnYgyFIepNeYiFwo/t+CvCpuqu5jdF+jTWwCThrV41lIfAT9yrvSKBQVfc0VuMi0qVqHUhE0nA+cz75R+rW+09gg6o+fYJifhu/N+37efwdRaS9+741cDGwsUYxv3z2vWnbn597Vb1fVZNUNQXn39ynqnp9jWK+Gbs/rzw11gvnCuZmnKvMD7jbHgMmuO8jgbeBrcBS4LRGbv8PwDqcK9+Lgd4+bHsWsAcow1lv+xlOKozb3P0C/N3t2xpgmI/HXl/7v/AY+3fA2T5s+xyciwWrcVKAZLh/F40yfi/b9+f4BwIr3fbXAg811mffy7b99rmv0Zfzca9u+2Ps9o0bY4ypQzCcbhtjjN9YkDTGmDpYkDTGmDpYkDTGmDpYkDTGmDpYkDTGmDpYkDTGmDpYkDTGmDr8fwDlgWFhHNE6AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 360x216 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mjoId37R0yVI"
      },
      "source": [
        "It looks like we could have stopped the training earlier as the validation loss did not decrease afterwards."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nOHn1utQHqHu",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a6a84e70-2f4c-4e76-c40d-f3a5517844e3"
      },
      "source": [
        "# Let's see the accuracy in the test split\n",
        "scores = text_class_model.evaluate(x_test, y_test, verbose=2)\n",
        "print(\"%s: %.2f%%\" % (text_class_model.metrics_names[1], scores[1]*100))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "782/782 - 3s - loss: 0.4924 - accuracy: 0.8165\n",
            "accuracy: 81.65%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w3HtK17c1S_F"
      },
      "source": [
        "### Saving best performing model\n",
        "\n",
        "Now we save the best performing models using the validation loss as a metric. As you already know, lower training error does not mean in some cases better performance in the validation or test split.\n",
        "\n",
        "In the last example we ran the model for 5 epochs, however after the first epoch the validation loss increased. We want to use the model performing the best in the validation set. To do so, we can use a ModelCheckpoint callback as was explained in past tutorials. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SzVNbh5a1xsq",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4879e0a8-9e39-4b58-e468-c449b02f662a"
      },
      "source": [
        "###### We reset the model #######\n",
        "text_class_model = Sequential()\n",
        "text_class_model.add(Embedding(nb_words,\n",
        "                    embedding_dim,\n",
        "                    input_length=maxlen))\n",
        "text_class_model.add(LSTM(lstm_units))\n",
        "text_class_model.add(Dense(1, activation='sigmoid'))\n",
        "text_class_model.compile(loss='binary_crossentropy',\n",
        "              optimizer='adam',\n",
        "              metrics=['accuracy'])\n",
        "#################################\n",
        "\n",
        "\n",
        "from keras.callbacks import ModelCheckpoint\n",
        "## Here we define the checkpoint callback. We first give the name it will use \n",
        "## to save the model. We also specify what metric should monitor, in this case\n",
        "## the validation loss (it could be validation accuracy too for example)\n",
        "## save_best_only means that the models will only be save when the monitored \n",
        "## metric improves with respect to the past best one\n",
        "checkpoint = ModelCheckpoint('model-{epoch:03d}.h5', verbose=1, \n",
        "                             monitor='val_loss', save_best_only=True)  \n",
        "## Now we pass the callback to the fit method\n",
        "history = text_class_model.fit(x_train, y_train, batch_size=128,\n",
        "                               epochs=epochs, callbacks=[checkpoint],          \n",
        "                               validation_split=validation_split)\n",
        "                             \n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/5\n",
            "157/157 [==============================] - 8s 39ms/step - loss: 0.5464 - accuracy: 0.6982 - val_loss: 0.3668 - val_accuracy: 0.8374\n",
            "\n",
            "Epoch 00001: val_loss improved from inf to 0.36677, saving model to model-001.h5\n",
            "Epoch 2/5\n",
            "157/157 [==============================] - 5s 34ms/step - loss: 0.3045 - accuracy: 0.8746 - val_loss: 0.3802 - val_accuracy: 0.8318\n",
            "\n",
            "Epoch 00002: val_loss did not improve from 0.36677\n",
            "Epoch 3/5\n",
            "157/157 [==============================] - 5s 32ms/step - loss: 0.2359 - accuracy: 0.9072 - val_loss: 0.3858 - val_accuracy: 0.8286\n",
            "\n",
            "Epoch 00003: val_loss did not improve from 0.36677\n",
            "Epoch 4/5\n",
            "157/157 [==============================] - 5s 32ms/step - loss: 0.1814 - accuracy: 0.9294 - val_loss: 0.4745 - val_accuracy: 0.8224\n",
            "\n",
            "Epoch 00004: val_loss did not improve from 0.36677\n",
            "Epoch 5/5\n",
            "157/157 [==============================] - 5s 32ms/step - loss: 0.1429 - accuracy: 0.9465 - val_loss: 0.4762 - val_accuracy: 0.8092\n",
            "\n",
            "Epoch 00005: val_loss did not improve from 0.36677\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wrtRFGUF22tf"
      },
      "source": [
        "We now saved the model after every epoch if the validation loss decreased. In this case, as the best epoch is the first one, it only saves the model after the first epoch. Let's load the model now."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YwHZ54_22vSa"
      },
      "source": [
        "from keras.models import load_model\n",
        "text_class_model = load_model('model-001.h5')  # load model weights"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1KLudp1u3CHY"
      },
      "source": [
        "And now we check if the accuracy is better compared to when we used the model trained for 5 epochs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZsuLgHQu2zqu",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7fa2e62b-70ea-457f-8a03-4f23025824b6"
      },
      "source": [
        "# Let's see the accuracy in the test split\n",
        "scores = text_class_model.evaluate(x_test, y_test, verbose=2)\n",
        "print(\"%s: %.2f%%\" % (text_class_model.metrics_names[1], scores[1]*100))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "782/782 - 3s - loss: 0.3680 - accuracy: 0.8382\n",
            "accuracy: 83.82%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Ufb6B68hUhl"
      },
      "source": [
        "### Importance of embeddings\n",
        "Now, let's check quickly if using Embeddings provide any benefit. For this experiment, we remove the Embedding layer, meaning we will be inputting the word index, i.e. an integer, to the LSTM. Additionally, we need to vary the shape of the input data as the LSTM needs a third dimension with the number of channels per input (in this case 1, as each word is represented by an integer)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w1NY5IhRhppg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0ec572a4-b07b-4f70-e7b8-9d530925368a"
      },
      "source": [
        "# model parameters\n",
        "# Same model as before but without embeddings\n",
        "lstm_units = 128\n",
        "x_train_r = np.reshape(x_train, (x_train.shape[0], x_train.shape[1], 1))\n",
        "x_test_r = np.reshape(x_test, (x_test.shape[0], x_test.shape[1], 1))\n",
        "\n",
        "\n",
        "print('Build model...')\n",
        "text_class_model = Sequential()\n",
        "\n",
        "text_class_model.add(LSTM(lstm_units, input_shape=(maxlen, 1)))\n",
        "\n",
        "text_class_model.add(Dense(1, activation='sigmoid'))\n",
        "\n",
        "text_class_model.compile(loss='binary_crossentropy',\n",
        "              optimizer='adam',\n",
        "              metrics=['accuracy'])\n",
        "print(text_class_model.summary())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Build model...\n",
            "Model: \"sequential_38\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "lstm_38 (LSTM)               (None, 128)               66560     \n",
            "_________________________________________________________________\n",
            "dense_38 (Dense)             (None, 1)                 129       \n",
            "=================================================================\n",
            "Total params: 66,689\n",
            "Trainable params: 66,689\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "None\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qZaz4eYyiBPc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cbaacf6b-fd24-439d-ffd4-c47eb00a640d"
      },
      "source": [
        "epochs = 5\n",
        "validation_split = 0.2\n",
        "\n",
        "history = text_class_model.fit(x_train_r, y_train, batch_size=128,\n",
        "          epochs=epochs, \n",
        "          validation_split=validation_split)\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/5\n",
            "157/157 [==============================] - 3s 10ms/step - loss: 0.7036 - accuracy: 0.5130 - val_loss: 0.6980 - val_accuracy: 0.4938\n",
            "Epoch 2/5\n",
            "157/157 [==============================] - 1s 7ms/step - loss: 0.6887 - accuracy: 0.5374 - val_loss: 0.6826 - val_accuracy: 0.5676\n",
            "Epoch 3/5\n",
            "157/157 [==============================] - 1s 8ms/step - loss: 0.6847 - accuracy: 0.5545 - val_loss: 0.6832 - val_accuracy: 0.5696\n",
            "Epoch 4/5\n",
            "157/157 [==============================] - 1s 8ms/step - loss: 0.6841 - accuracy: 0.5535 - val_loss: 0.6809 - val_accuracy: 0.5690\n",
            "Epoch 5/5\n",
            "157/157 [==============================] - 1s 7ms/step - loss: 0.6837 - accuracy: 0.5506 - val_loss: 0.6853 - val_accuracy: 0.5584\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oZcRp4nXiN84",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "800026b2-fe7e-40bb-e23b-4465dc2629b1"
      },
      "source": [
        "scores = text_class_model.evaluate(x_test_r, y_test, verbose=2)\n",
        "print(\"%s: %.2f%%\" % (text_class_model.metrics_names[1], scores[1]*100))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "782/782 - 2s - loss: 0.6901 - accuracy: 0.5391\n",
            "accuracy: 53.91%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uyOqn7yginQg"
      },
      "source": [
        "We see how the accuracy is just slightly better compared to doing a random guess as there are only 2 classes. That is because the integer that encodes the word is chosen arbitrarily and does not encode any property of the words nor any relationship between words. Hence, using learned embeddings results in models with better capacity/accuracy for this kind of tasks.\n",
        "\n",
        "Instead of inputting directly integers to the RNN, we could argue that another way of representing the input words is to input the one-hot representation of the word directly to the RNN instead of using embeddings. This method has two problems. First, the resulting dimensionality would be too large. In this case we are loading the $20000$ most common words in IMDB, so each word would be encoded in a vector of dimensionality $20000$. Secondly, the encoding does not give any notion of similarity between the different words. For these two reasons, embedding the words to a common lower dimensionality space is also better than using one-hot encoding."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-EwrgAmcM9na"
      },
      "source": [
        "# Text Generation\n",
        "\n",
        "We now will present an example of text generation using an RNN. We will use the Tiny Shakespeare dataset, which contains samples from Shakespeare works.\n",
        "\n",
        "We pose the problem as a classification problem as in the Sentiment classification example. In a text generation setting, given a sequence, we aim to predict the next one. During the training step, the process would be quite similar to the classification task, i.e. given a sequence you predict a label/word/character. However, in the prediction setting, we aim to output a whole sentence, not only a word or character. For that reason we will follow the procedure explained in the image.\n",
        "\n",
        "![](https://cdn-images-1.medium.com/max/1600/1*XvUt5wDQA8D3C0wAuxAvbA.png)\n",
        "The image is taken from [here](https://medium.com/@david.campion/text-generation-using-bidirectional-lstm-and-doc2vec-models-1-3-8979eb65cb3a)\n",
        "\n",
        "As in the text classification section, we encode the raw sentences in the form of a sequence of integers. In this case, however, we have a text with a really wide vocabulary, and we aim to predict the following element of the sequence. In this dataset the number of words is really large, so if we used a word-level model the number of available classes would be quite large if we did not filter the number of possible words. For this example, instead of encoding on a word level, we will do it on a character level, which results in a more limited vocabulary. We aim to predict the next character when inputting a sentence of length $d$. In evaluation mode, i.e. when we are predicting the next character, we then input a seed of $d$ characters as the first input, and afterwards we use the last $d$ characters as input to predict the next character. \n",
        "\n",
        "The advantages of predicting characters instead of words is that character-level models have a limited vocabulary/classes compared to the number of possible words in a text. They are also more flexible (for example, they can generate \"fake\" url links if trained in e.g. wikipedia data). However, word-level models usually present higher performance as it easier for them to keep track of the long-term meaning of the sentence, and also can avoid any spelling mistakes that may happen in character-level models.\n",
        "\n",
        "**References:** Some code snippets taken from [here](https://www.analyticsvidhya.com/blog/2018/03/text-generation-using-python-nlp/)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4_-da7NsDrJf"
      },
      "source": [
        "We first download the data we use for the example, and then read the file. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zcR0v2GGNFCd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "74480023-cb6f-4ba1-b5d2-0f5121ffd510"
      },
      "source": [
        "!wget https://raw.githubusercontent.com/jcjohnson/torch-rnn/master/data/tiny-shakespeare.txt\n",
        "## We read all the raw data in the variable data\n",
        "data = open('./tiny-shakespeare.txt', 'r').read()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-02-14 19:12:04--  https://raw.githubusercontent.com/jcjohnson/torch-rnn/master/data/tiny-shakespeare.txt\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 1115394 (1.1M) [text/plain]\n",
            "Saving to: ‘tiny-shakespeare.txt.1’\n",
            "\n",
            "tiny-shakespeare.tx 100%[===================>]   1.06M  --.-KB/s    in 0.02s   \n",
            "\n",
            "2021-02-14 19:12:04 (44.9 MB/s) - ‘tiny-shakespeare.txt.1’ saved [1115394/1115394]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WxFUMjE9kP4C"
      },
      "source": [
        "Let's print a subset of the data for visualization purposes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7c8ZSkLKkX6Y",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1f545d80-626a-4b5f-cb92-2ad8a996bcec"
      },
      "source": [
        "print(data[:364])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "First Citizen:\n",
            "Before we proceed any further, hear me speak.\n",
            "\n",
            "All:\n",
            "Speak, speak.\n",
            "\n",
            "First Citizen:\n",
            "You are all resolved rather to die than to famish?\n",
            "\n",
            "All:\n",
            "Resolved. resolved.\n",
            "\n",
            "First Citizen:\n",
            "First, you know Caius Marcius is chief enemy to the people.\n",
            "\n",
            "All:\n",
            "We know't, we know't.\n",
            "\n",
            "First Citizen:\n",
            "Let us kill him, and we'll have corn at our own price.\n",
            "Is't a verdict?\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HhZmAeuzD0Wt"
      },
      "source": [
        "We see in the raw text that the name of the characters is printed, that there is a blank line between the different character lines, and also that lines are usually not longer than ~80 characters. We hope the network is capable of learning all of this.\n",
        "\n",
        "Now we will structure the data so we can input it into our RNN model. We aim to encode the text using a sequence of integers as we explained in the classification section.  Hence, we form a dictionary with all the different characters appearing in the text (including whitespaces and punctuation) and its corresponding integer. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zs3tRv_1whDc"
      },
      "source": [
        "characters = sorted(list(set(data)))\n",
        "n_to_char = {n:char for n, char in enumerate(characters)}\n",
        "char_to_n = {char:n for n, char in enumerate(characters)}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MEfzwgCnwjwo"
      },
      "source": [
        "Now we want to split the text in examples of length `seq_length` where the label is the next character. As we mentioned, we pose the problem as a set of successive classification problems, where we try to predict the next word given an input sentence. To split the text in pairs of (sequence, next word), we use the dictionary `char_to_n` to encode the different elements as integers.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gulHCFMJ_tEE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a31831b0-73e5-4c7b-f731-798048d14732"
      },
      "source": [
        "x = []\n",
        "y = []\n",
        "length = len(data)\n",
        "seq_length = 100\n",
        "for i in range(0, length-seq_length, 1):\n",
        "  sequence = data[i:i + seq_length]\n",
        "  label = data[i + seq_length]\n",
        "  x.append([char_to_n[char] for char in sequence])\n",
        "  y.append(char_to_n[label])\n",
        "n_samples = len(x)\n",
        "print(\"Total Samples: {:d}\".format(n_samples))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Total Samples: 1115294\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ehank2xQFGOo"
      },
      "source": [
        "We now form a test split by using 5% of the available data. To do so, we just take the last 5% of the data. Usually, you need to randomize the data before splitting in test/train to avoid having a different distribution of the data in both splits. However, as this is a sequential data where we want to predict the next character, if we shuffle the splits we would have this in training:\n",
        "\n",
        "``\n",
        "x = 'the boy is tal_' y = 'l'\n",
        "``\n",
        "\n",
        "and this in testing:\n",
        "\n",
        "``\n",
        "x = 'the boy is ta_' y = 'l'\n",
        "``\n",
        "\n",
        "which would contaminate the splits."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qbBqXbtpfa5T"
      },
      "source": [
        "import keras\n",
        "x_train = x[:int(n_samples*0.95)]\n",
        "x_test = x[int(n_samples*0.95):]\n",
        "y_train = y[:int(n_samples*0.95)]\n",
        "y_test = y[int(n_samples*0.95):]\n",
        "\n",
        "## Transform the list to a numpy array\n",
        "x_train = np.reshape(x_train, (len(x_train), seq_length))\n",
        "## Onehot encoding of labels\n",
        "y_train = tf.keras.utils.to_categorical(np.asarray(y_train))\n",
        "\n",
        "x_test = np.reshape(x_test, (len(x_test), seq_length))\n",
        "y_test = tf.keras.utils.to_categorical(np.asarray(y_test))\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U63V-XoLkzD2"
      },
      "source": [
        "Let's print a sequence and the shape of the labels."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E_KvSAVQzOvq",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e57669b7-fada-40b5-a435-92bc7d6c9950"
      },
      "source": [
        "print(x_train[8000])\n",
        "print(y_train.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[ 0 13  1 57 47 41 49  1 51 39 52  5 57  1 39 54 54 43 58 47 58 43  6  1\n",
            " 61 46 53  1 42 43 57 47 56 43 57  1 51 53 57 58  1 58 46 39 58  0 35 46\n",
            " 47 41 46  1 61 53 59 50 42  1 47 52 41 56 43 39 57 43  1 46 47 57  1 43\n",
            " 60 47 50  8  1 20 43  1 58 46 39 58  1 42 43 54 43 52 42 57  0 33 54 53\n",
            " 52  1 63 53]\n",
            "(1059529, 65)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Uudj2qT_FNJG"
      },
      "source": [
        "Here, we define the model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "13TcalgkqHsM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a2f27118-8316-4497-a31e-9f2cef4815e2"
      },
      "source": [
        "# define the LSTM model\n",
        "embedding_size = 100\n",
        "lstm_units = 128\n",
        "\n",
        "text_gen_model = Sequential()\n",
        "text_gen_model.add(Embedding(y_train.shape[1],\n",
        "                    embedding_size, input_length=seq_length))\n",
        "text_gen_model.add(LSTM(lstm_units))\n",
        "text_gen_model.add(Dense(y_train.shape[1], activation='softmax'))\n",
        "\n",
        "text_gen_model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
        "text_gen_model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_40\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding_3 (Embedding)      (None, 100, 100)          6500      \n",
            "_________________________________________________________________\n",
            "lstm_40 (LSTM)               (None, 128)               117248    \n",
            "_________________________________________________________________\n",
            "dense_40 (Dense)             (None, 65)                8385      \n",
            "=================================================================\n",
            "Total params: 132,133\n",
            "Trainable params: 132,133\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nC2BaDcumZnm"
      },
      "source": [
        "The model is a little bit complex, so training takes some time."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z2EPi_Fee4dj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b204dfdb-fa81-474e-a471-811f43b02588"
      },
      "source": [
        "text_gen_model.fit(x_train, y_train, epochs=10 , batch_size=128)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "8278/8278 [==============================] - 97s 12ms/step - loss: 2.2288\n",
            "Epoch 2/10\n",
            "8278/8278 [==============================] - 96s 12ms/step - loss: 1.6515\n",
            "Epoch 3/10\n",
            "8278/8278 [==============================] - 95s 12ms/step - loss: 1.5489\n",
            "Epoch 4/10\n",
            "8278/8278 [==============================] - 95s 11ms/step - loss: 1.4949\n",
            "Epoch 5/10\n",
            "8278/8278 [==============================] - 94s 11ms/step - loss: 1.4622\n",
            "Epoch 6/10\n",
            "8278/8278 [==============================] - 95s 11ms/step - loss: 1.4349\n",
            "Epoch 7/10\n",
            "8278/8278 [==============================] - 94s 11ms/step - loss: 1.4188\n",
            "Epoch 8/10\n",
            "8278/8278 [==============================] - 93s 11ms/step - loss: 1.4062\n",
            "Epoch 9/10\n",
            "8278/8278 [==============================] - 93s 11ms/step - loss: 1.3934\n",
            "Epoch 10/10\n",
            "8278/8278 [==============================] - 95s 11ms/step - loss: 1.3847\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f9010341588>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 144
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-5XsfQRNr9An"
      },
      "source": [
        "### Prediction\n",
        "\n",
        "We now take any of the sequences from the test split and use it as an initial pattern for our model. Then we enter a loop where given an input sequence, we predict the next character and then form a new input sequence by appending the predicted character and dropping the first character.\n",
        "\n",
        "Whenever we input a sequence, we obtain as output a probability distribution of the possible characters. For example, given the sequence `the cat and the do` the model will output a probability distribution of the next character where probably the character `g` will have a high probability (forming then `the cat and the dog`). However, the character `c` can also be a possibility, as the sentence formed may be `the cat and the doctor`. When deciding what character to predict, a strategy is to just take the character with the maximum probability at all times, but then the variability of the formed text is then smaller. Another option would be to sample following the same probability of distribution as the model outputs. However, this might result in a excessive variability with some sentences not making any sense.\n",
        "\n",
        "This trade-off of variability against more probable sequences is controlled by what it is called the temperature of the sampling process. The temperature controls the smoothing of the probability vector. A temperature close to 0 will result in taking always the safest (i.e. that with the highest probability) as the next element. A temperature close to 1 will decide on the next element following the same distribution of probability as the original output of the model.\n",
        "\n",
        "In a more formal way, being $p_i$ the probability of the element $i$ output by the RNN model and $T$ the temperature, the probability after applying the mentioned smoothing $\\hat{p_i}$ is:\n",
        "\n",
        "$$\n",
        "\\hat{p_i} = \\frac{e^{\\log(p_i)/T}}{\\sum_j e^{\\log(p_j)/T}}\n",
        "$$\n",
        "\n",
        "You can check how a small $T$ is going to make the element with the largest probability be close to 1 after the process.\n",
        "\n",
        "Before starting the text prediction, let's show an example of how the sampling temperature affects the model choices."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "loPFAdDvIsWO",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "82b25aa6-461d-4d78-dc89-5117d5d03a7b"
      },
      "source": [
        "def temperature_smoothing(prediction, temperature=1.0):\n",
        "  # This function computes the temperature smoothing function\n",
        "  # we explained\n",
        "  prediction = np.log(prediction + 1e-7) / (temperature + 0.01)\n",
        "  exp_preds = np.exp(prediction)\n",
        "  prediction = exp_preds / np.sum(exp_preds)\n",
        "  return prediction\n",
        "\n",
        "# We have this vector of probabilities\n",
        "prediction = np.asarray([0.2, 0.3, 0.1, 0.4])\n",
        "## Temperature = 1.0\n",
        "print('Output probabilities with temp.=1')\n",
        "print(temperature_smoothing(prediction, temperature=1))\n",
        "## Temperature = 0.5\n",
        "print('Output probabilities with temp.=0.5')\n",
        "print(temperature_smoothing(prediction, temperature=0.5))\n",
        "## Temperature = 0.25\n",
        "print('Output probabilities with temp.=0.25')\n",
        "print(temperature_smoothing(prediction, temperature=0.25))\n",
        "## Temperature = 0.0\n",
        "print('Output probabilities with temp.=0')\n",
        "print(temperature_smoothing(prediction, temperature=0))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Output probabilities with temp.=1\n",
            "[0.20065194 0.29977201 0.10101691 0.39855913]\n",
            "Output probabilities with temp.=0.5\n",
            "[0.13579339 0.30071527 0.03488383 0.5286075 ]\n",
            "Output probabilities with temp.=0.25\n",
            "[0.04948661 0.23537565 0.00344097 0.71169677]\n",
            "Output probabilities with temp.=0\n",
            "[7.88880627e-31 3.20722891e-13 6.22348202e-61 1.00000000e+00]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ST0l8UWDJy-t"
      },
      "source": [
        "You can see how a lower temperature makes the small probabilities become smaller."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cnpdKtGwIjue"
      },
      "source": [
        "Here you can set the temperature and check how the output varies. You will see that for $T\\approx0$ the generated text has low variability."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "REoPwrqieFLK",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c05752ca-8787-4d34-c970-29b56ae3aa10"
      },
      "source": [
        "import sys\n",
        "# Vary the temperature here\n",
        "temperature = 0.5\n",
        "\n",
        "# We select a random element from the test set as seed\n",
        "start = np.random.randint(0, len(x_test)-1)\n",
        "pattern = x_test[start].tolist()\n",
        "print(\"Seed:\")\n",
        "print(\"\\\"\", ''.join([n_to_char[value] for value in pattern]), \"\\\"\")\n",
        "\n",
        "print(\"\\nPredicted:\")\n",
        "# generate 300 characters\n",
        "for i in range(300):\n",
        "  x = np.reshape(pattern, (1, len(pattern)))\n",
        "  prediction = text_gen_model.predict(x, verbose=0).astype(np.float64)\n",
        "  ## We put the constant 0.02 to avoid dividing by zero\n",
        "  ## We sum by 1e-7 to avoid log(0)\n",
        "  prediction = np.log(prediction + 1e-7) / (temperature + 0.01)\n",
        "  exp_preds = np.exp(prediction)\n",
        "  prediction = exp_preds / np.sum(exp_preds)\n",
        "  ## We applied the smoothing with the temperature\n",
        "  ## Now we predict following the probabilities in the variable prediction\n",
        "  prediction = np.random.multinomial(1, prediction[0,:], 1)\n",
        "  index = np.argmax(prediction)\n",
        "  result = n_to_char[index]\n",
        "  seq_in = [n_to_char[value] for value in pattern]\n",
        "  ## Print the result\n",
        "  sys.stdout.write(result)\n",
        "  \n",
        "  ## Create the input sequence for the next character by appending the predicted\n",
        "  ## character and dropping the first one to always have constant seq. length\n",
        "  pattern.append(index)\n",
        "  pattern = pattern[1:]\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Seed:\n",
            "\" nse. Would I had never\n",
            "Married my daughter there! for, coming thence,\n",
            "My son is lost and, in my rate \"\n",
            "\n",
            "Predicted:\n",
            " and straight.\n",
            "\n",
            "ROMEO:\n",
            "What you must me think you some son?\n",
            "\n",
            "DUKE VINCENTIO:\n",
            "My lord, the crown to for the country.\n",
            "\n",
            "First Servingman:\n",
            "What shall be so? he whom I melt them with me some\n",
            "To the son of the say as they shall procent,\n",
            "And you mean me with the master to me and play\n",
            "To see thee with thy h"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sSQs-l7OmNoI"
      },
      "source": [
        "Notice how the network has learnt the structure of the text, to start a new line every few words and to put the name of the characters too. The text itself seems grammatically correct in most cases but it fails to make much sense in most examples. Generating sentences with actual semantic meaning is harder for these type of models compared to generating grammatically correct sentences. Test different temperature settings to see how it affects the generation\n",
        "\n",
        "If you want to know more about text generation, along with some extra generated examples, we refer you to [Andrej Karpathy's blog](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5hY4m63VgiM9"
      },
      "source": [
        "### Quantitative evaluation of the generated text\n",
        "\n",
        "For a quantitative evaluation of the generated text we will use a metric commonly used in image captioning and translation tasks, BLEU. BLEU looks for matches on a word level between the generated text and the reference text. Specifically, BLEU looks for matches in n-grams up to $n=4$, where an n-gram is defined as a contiguous sentence of $n$ items. For example, in the sentence `the sky is blue`, an n-gram of $n=2$ would be `is blue`. BLEU scores range from [0,1].\n",
        "\n",
        "As we mentioned, BLEU is usual in more constrained text generation tasks, such as image captioning. In our text generation task there is a large number of both grammatically and semantically correct possibilities when generating new sentences, and one of the issues for this task is to develop a metric that can properly evaluate the semantic and syntactic quality of the generated text. Human evaluation is quite common when evaluating the quality of generated sentences in these less constrained tasks due to the difficulty of finding proper automatic metrics. However, we hope to see a correlation between the BLEU score and the quality of the generated text. Here we take an input sentence from the test data and generate a sentence. Then, we compare the generated sentence to the real one from the corpus. We do so 20 times and provide the average BLEU score.\n",
        "\n",
        "The used function is integrated in the package `nltk` and it is called using the following syntax `sentence_bleu(reference, candidate).`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LtnzNZZGr11D",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c740682c-79df-41c5-f374-f5bed61f3954"
      },
      "source": [
        "from nltk.translate.bleu_score import sentence_bleu\n",
        "import sys\n",
        "import time\n",
        "import string\n",
        "a = time.time()\n",
        "characters = sorted(list(set(data)))\n",
        "n_to_char = {n:char for n, char in enumerate(characters)}\n",
        "char_to_n = {char:n for n, char in enumerate(characters)}\n",
        "temperature = 0.5\n",
        "bleu_score = 0\n",
        "n_eval = 20\n",
        "seq_char_length = 100\n",
        "for _ in range(n_eval):\n",
        "  start = np.random.randint(0, len(x_test)-seq_char_length-1)\n",
        "  pattern = x_test[start].tolist()\n",
        "  reference = x_test[start+seq_char_length].tolist()\n",
        "  reference = ''.join([n_to_char[value] for value in reference])\n",
        "  # generate characters\n",
        "  output_sent = ''\n",
        "  for n_char in range(seq_char_length):\n",
        "    x = np.reshape(pattern, (1, len(pattern)))\n",
        "    prediction = text_gen_model.predict(x, verbose=0).astype(np.float64)\n",
        "    prediction = np.log(prediction + 1e-7) / (temperature + 0.01)\n",
        "    exp_preds = np.exp(prediction)\n",
        "    prediction = exp_preds / np.sum(exp_preds)\n",
        "    prediction = np.random.multinomial(1, prediction[0,:], 1)\n",
        "    index = np.argmax(prediction)\n",
        "    result = n_to_char[index]\n",
        "    output_sent += result\n",
        "    pattern.append(index)\n",
        "    pattern = pattern[1:]\n",
        "  # We filter the reference and candidate words to remove any punctuation\n",
        "  # We also use only lower case words\n",
        "  reference = reference.translate(str.maketrans('', '', string.punctuation)).replace('\\n',' ').lower().split(' ')\n",
        "  # We filter any empty element in the list\n",
        "  reference = list(filter(lambda x: x is not '', reference))\n",
        "  candidate = output_sent.replace('\\n',' ').translate(str.maketrans('', '', string.punctuation)).lower().split(' ')\n",
        "  candidate = list(filter(lambda x: x is not '', candidate))\n",
        "  # We remove the first and the last word for both the reference and candidate\n",
        "  # because they may not be completed words\n",
        "  bleu_score += sentence_bleu(reference[1:-1], candidate[1:-1])\n",
        "print(bleu_score/n_eval)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/nltk/translate/bleu_score.py:490: UserWarning: \n",
            "Corpus/Sentence contains 0 counts of 2-gram overlaps.\n",
            "BLEU scores might be undesirable; use SmoothingFunction().\n",
            "  warnings.warn(_msg)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "0.3012470740715931\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UEPP1gUthOI0"
      },
      "source": [
        "# Transformers\n",
        "RNNs are still quite used in several architectures and Natural Language Processing (NLP) tasks. However, during the last 3 years (since the paper [Attention Is All You Need](https://arxiv.org/abs/1706.03762) was published) a new architecture, the Transformer, has started to outperform RNNs in several text/NLP benchmarks. A Transformer does not use any recurrence, it instead uses attention to focus on specific parts of a sentence. If you want an in-depth explanation of the architecture, you can take a look to the [Illustrated Transformer](http://jalammar.github.io/illustrated-transformer/) blog post.\n",
        "\n",
        "Both text classification and text generation tasks have been succesfully tackled by transformers. Transfomers seem to scale better than recurrent neural networks, they can have great results when using large architectures and large datasets, whereas RNNs seem not to benefit as much from using more parameters and training examples. State-of-the-art transformer models can have even billions of parameters and can benefit from training in TBs of text data. \n",
        "\n",
        "We will show examples for both text classification and text generation of models pretrained on large datasets. The aim of the examples is to show you how these large models trained in diverse/large datasets can transfer their knowledge to several tasks (similarly to the transfer learning/finetuning section in the CNN Architectures notebook). However, we will not implement any transformer model from scratch. If you are interested in doing so, besides the theoretical explanation of the [Illustrated Transformer](http://jalammar.github.io/illustrated-transformer/) blog post, you can also check the [transformers example](https://keras.io/examples/nlp/text_classification_with_transformer/) in the Keras docs.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VPNbRN4lCkkL"
      },
      "source": [
        "##Text Classification\n",
        "\n",
        "For the text classification task, the model called BERT from Google, and all the subsequent variants have achieved state-of-the-art results. BERT is a method to train language representations, meaning that you then can use those represenations/embeddings to predict the label of the sentence in a text classification task.\n",
        "\n",
        "We will now show a quick example of finetuning BERT in IMDB to show you the capabilities of this model. We take part of the code from [this link](https://colab.research.google.com/drive/18SVeIFXWCiA9HL4WVCAFxlfH59ez6atc#scrollTo=IhHGCuPhv5Ew). To run the following piece of code you may need to restart your environment due to some dependencies issues (it will tell you to do so)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yDbm3oUxB7Ly"
      },
      "source": [
        "# install ktrain\n",
        "!pip install ktrain==0.26.3\n",
        "# import ktrain\n",
        "import ktrain\n",
        "from ktrain import text"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "objbxd0QCAnR",
        "outputId": "aa4685a1-8666-4802-8524-c1c24bd9f685"
      },
      "source": [
        "# download IMDb movie review dataset\n",
        "# As we take the code from another notebook, the way to load the\n",
        "# data is different, but the splits are the same as when using imbd.load_data\n",
        "import tensorflow as tf\n",
        "dataset = tf.keras.utils.get_file(\n",
        "    fname=\"aclImdb.tar.gz\", \n",
        "    origin=\"http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\", \n",
        "    extract=True,\n",
        ")\n",
        "# set path to dataset\n",
        "import os.path\n",
        "#dataset = '/root/.keras/datasets/aclImdb'\n",
        "IMDB_DATADIR = os.path.join(os.path.dirname(dataset), 'aclImdb')\n",
        "print(IMDB_DATADIR)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n",
            "84131840/84125825 [==============================] - 1s 0us/step\n",
            "/root/.keras/datasets/aclImdb\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Bm4kamkbCFIE"
      },
      "source": [
        "trn, test, preproc = text.texts_from_folder(IMDB_DATADIR, \n",
        "                                          maxlen=500, \n",
        "                                          preprocess_mode='bert',\n",
        "                                          train_test_names=['train', \n",
        "                                                            'test'],\n",
        "                                          classes=['pos', 'neg'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "83bECt0r36jW"
      },
      "source": [
        "The code below will load a pretrained BERT model and it will start finetuning it in IMDB. We also provide a model finetuned for IMDB as training for 1 epoch takes 1 hour."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KuCksFRVCHQA",
        "outputId": "47e16f6d-8fe9-44a9-d346-c45f4f0dc41a"
      },
      "source": [
        "model = text.text_classifier('bert', trn, preproc=preproc)\n",
        "learner = ktrain.get_learner(model,train_data=trn, val_data=test, batch_size=6)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Is Multi-Label? False\n",
            "maxlen is 500\n",
            "done.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lgBlcgve4I3t"
      },
      "source": [
        "We train for one epoch (you can skip this step as we already provide a trained model)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k4mcTiBLCMtl",
        "outputId": "af8ca658-32a4-4388-8826-221124b129a3"
      },
      "source": [
        "learner.fit_onecycle(2e-5, 1)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "begin training using onecycle policy with max lr of 2e-05...\n",
            "4167/4167 [==============================] - 5547s 1s/step - loss: 0.1335 - accuracy: 0.9528 - val_loss: 0.1641 - val_accuracy: 0.9405\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7fa73a97b390>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aemVsYcj6swJ"
      },
      "source": [
        "As mentioned, we provide a model already trained. We first download it, load it and test it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Koan7lanDB9L"
      },
      "source": [
        "!wget https://imperialcollegelondon.box.com/shared/static/w4n0r0l8tm2txvvisq64ysgrgvuwrd4h.zip -O bert.zip"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V8Rk9MN4DrZU",
        "outputId": "da9993fb-688a-4b53-873c-bc5e1e5f1f3a"
      },
      "source": [
        "!unzip /content/bert.zip"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Archive:  /content/bert.zip\n",
            "   creating: bert/\n",
            "  inflating: bert/tf_model.preproc   \n",
            "  inflating: bert/tf_model.h5        \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8wHbNXzL3V68"
      },
      "source": [
        "# reload Predictor and extract model\n",
        "model = ktrain.load_predictor('/content/bert').model\n",
        "\n",
        "# re-instantiate Learner\n",
        "learner = ktrain.get_learner(model, train_data=trn, val_data=test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sjejlAca4MOl"
      },
      "source": [
        "We evaluate now the performance of the loaded model. The model is quite big, so this evaluation step also takes a bit of time."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 324
        },
        "id": "pNOAf77VhsnB",
        "outputId": "a88a1e2e-1ee0-4762-aa82-61b92d38a7c8"
      },
      "source": [
        "learner.evaluate()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-55-33b23d2ae399>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mlearner\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mevaluate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/ktrain/core.py\u001b[0m in \u001b[0;36mevaluate\u001b[0;34m(self, test_data, print_report, save_path, class_names)\u001b[0m\n\u001b[1;32m    114\u001b[0m         \u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    115\u001b[0m         \"\"\"\n\u001b[0;32m--> 116\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalidate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mval_data\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtest_data\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprint_report\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mprint_report\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msave_path\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msave_path\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mclass_names\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mclass_names\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    117\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    118\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/ktrain/core.py\u001b[0m in \u001b[0;36mvalidate\u001b[0;34m(self, val_data, print_report, save_path, class_names)\u001b[0m\n\u001b[1;32m    157\u001b[0m             \u001b[0mwarnings\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwarn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'multilabel confusion matrices not yet supported'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    158\u001b[0m             \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 159\u001b[0;31m         \u001b[0my_pred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mval_data\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mval\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    160\u001b[0m         \u001b[0my_true\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mground_truth\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mval_data\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mval\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    161\u001b[0m         \u001b[0my_pred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqueeze\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my_pred\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/ktrain/core.py\u001b[0m in \u001b[0;36mpredict\u001b[0;34m(self, val_data)\u001b[0m\n\u001b[1;32m   1096\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1097\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1098\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mval\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0meval_batch_size\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1099\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1100\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py\u001b[0m in \u001b[0;36merror_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     62\u001b[0m     \u001b[0mfiltered_tb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     63\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 64\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     65\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# pylint: disable=broad-except\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     66\u001b[0m       \u001b[0mfiltered_tb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_process_traceback_frames\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__traceback__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/keras/engine/training.py\u001b[0m in \u001b[0;36mpredict\u001b[0;34m(self, x, batch_size, verbose, steps, callbacks, max_queue_size, workers, use_multiprocessing)\u001b[0m\n\u001b[1;32m   1787\u001b[0m           \u001b[0;32mfor\u001b[0m \u001b[0mstep\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mdata_handler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msteps\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1788\u001b[0m             \u001b[0mcallbacks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_predict_batch_begin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1789\u001b[0;31m             \u001b[0mtmp_batch_outputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0miterator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1790\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mdata_handler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshould_sync\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1791\u001b[0m               \u001b[0mcontext\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masync_wait\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/traceback_utils.py\u001b[0m in \u001b[0;36merror_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    148\u001b[0m     \u001b[0mfiltered_tb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    149\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 150\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    151\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    152\u001b[0m       \u001b[0mfiltered_tb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_process_traceback_frames\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__traceback__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m    908\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    909\u001b[0m       \u001b[0;32mwith\u001b[0m \u001b[0mOptionalXlaContext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jit_compile\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 910\u001b[0;31m         \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    911\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    912\u001b[0m       \u001b[0mnew_tracing_count\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexperimental_get_tracing_count\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m_call\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m    947\u001b[0m       \u001b[0;31m# In this case we have not created variables on the first call. So we can\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    948\u001b[0m       \u001b[0;31m# run the first trace but we should fail if variables are created.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 949\u001b[0;31m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_stateful_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    950\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_created_variables\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mALLOW_DYNAMIC_VARIABLE_CREATION\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    951\u001b[0m         raise ValueError(\"Creating variables on a non-first call to a function\"\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   3129\u001b[0m        filtered_flat_args) = self._maybe_define_function(args, kwargs)\n\u001b[1;32m   3130\u001b[0m     return graph_function._call_flat(\n\u001b[0;32m-> 3131\u001b[0;31m         filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access\n\u001b[0m\u001b[1;32m   3132\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3133\u001b[0m   \u001b[0;34m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_call_flat\u001b[0;34m(self, args, captured_inputs, cancellation_manager)\u001b[0m\n\u001b[1;32m   1958\u001b[0m       \u001b[0;31m# No tape is watching; skip to running the function.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1959\u001b[0m       return self._build_call_outputs(self._inference_function.call(\n\u001b[0;32m-> 1960\u001b[0;31m           ctx, args, cancellation_manager=cancellation_manager))\n\u001b[0m\u001b[1;32m   1961\u001b[0m     forward_backward = self._select_forward_and_backward_functions(\n\u001b[1;32m   1962\u001b[0m         \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36mcall\u001b[0;34m(self, ctx, args, cancellation_manager)\u001b[0m\n\u001b[1;32m    601\u001b[0m               \u001b[0minputs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    602\u001b[0m               \u001b[0mattrs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mattrs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 603\u001b[0;31m               ctx=ctx)\n\u001b[0m\u001b[1;32m    604\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    605\u001b[0m           outputs = execute.execute_with_cancellation(\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py\u001b[0m in \u001b[0;36mquick_execute\u001b[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[1;32m     57\u001b[0m     \u001b[0mctx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mensure_initialized\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     58\u001b[0m     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,\n\u001b[0;32m---> 59\u001b[0;31m                                         inputs, attrs, num_outputs)\n\u001b[0m\u001b[1;32m     60\u001b[0m   \u001b[0;32mexcept\u001b[0m \u001b[0mcore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_NotOkStatusException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     61\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5yfZcSi56QIl"
      },
      "source": [
        "The accuracy in the IMDB test set is 94% after finetuning for one epoch, which shows the transferability of the knowledge learned with a large transformer in a large dataset. State-of-the-art results have been achieved in several tasks by transferring the knowledge of BERT and similar models to other NLP dataset/task. Even when using only the resources available via Colab we can get state-of-art results in IMDB with a quick finetuning step."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OPToW5bT5LG6"
      },
      "source": [
        "Let's now predict scores for an example review."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xKNST0CU0D4b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a01425cd-af22-496d-b6b3-7dc20d8af075"
      },
      "source": [
        "predictor = ktrain.get_predictor(learner.model,preproc)\n",
        "predictor.predict_proba('Even though the acting was not superb, the photography was great and the script deserves praise.')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0.02212544, 0.9778746 ], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "flANhpoe5c_W"
      },
      "source": [
        "We can also use the following built-in functionality that allows us to see what words affected the score. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "483ugzLxb_2K"
      },
      "source": [
        "# To run \"explain\" you need the following version of the package eli5\n",
        "!pip install git+https://github.com/amaiya/eli5@tfkeras_0_10_1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7DtXqEoV0Zdj"
      },
      "source": [
        "predictor = ktrain.get_predictor(learner.model,preproc)\n",
        "predictor.explain('Even though the acting was not superb, the photography was great and the script deserves praise.')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5CfPJQK7j3JO"
      },
      "source": [
        "## Text Generation\n",
        "\n",
        "In text generation, the GPT variants from OpenAI have shown impressive results. We will now show an example from a pretrained [GPT2](https://openai.com/blog/better-language-models/) using the [Hugging Face](https://huggingface.co/transformers/index.html#) library, which it contains several state-of-the-art Natural Language Processing (NLP) models. The code to generate text is extracted from this [Colab Notebook](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/02_how_to_generate.ipynb#scrollTo=HBtDOdD0wx3l).\n",
        "\n",
        "First, we install the Hugging Face library and load a GPT2 pretrained model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m4-nL-6fVRSZ"
      },
      "source": [
        "!pip install ktrain==0.26.3"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zj4IgSwrZ58G"
      },
      "source": [
        "import tensorflow as tf\n",
        "from transformers import TFGPT2LMHeadModel, GPT2Tokenizer\n",
        "tokenizer = GPT2Tokenizer.from_pretrained(\"gpt2\")\n",
        "# add the EOS token as PAD token to avoid warnings\n",
        "model = TFGPT2LMHeadModel.from_pretrained(\"gpt2\", pad_token_id=tokenizer.eos_token_id)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jbs0t7nmkKgi"
      },
      "source": [
        "Now, we generate text using as input a part of the Tiny Shakespeare dataset. Note that this model has not been finetuned using the Tiny Shakespeare dataset, instead it has been trained in a larger more diverse dataset. You can run the following code to get different outputs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MzzXQEW0Z9LO"
      },
      "source": [
        "# encode context the generation is conditioned on\n",
        "input_ids = tokenizer.encode('First Citizen:\\nBefore we proceed any further, hear me speak.\\nAll:\\nSpeak, speak.\\nFirst Citizen:\\nYou are all resolved rather to die than to famish?', return_tensors='tf')\n",
        "\n",
        "# activate sampling and deactivate top_k by setting top_k sampling to 0\n",
        "sample_output = model.generate(\n",
        "    input_ids, \n",
        "    do_sample=True, \n",
        "    max_length=200, \n",
        "    top_k=50\n",
        ")\n",
        "print(\"Output:\\n\" + 100 * '-')\n",
        "print(tokenizer.decode(sample_output[0], skip_special_tokens=True))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fBWzhH76cZTr"
      },
      "source": [
        "What is quite impressive about the output of the GPT model is the capability of matching the style of the input text. The model is not finetuned in our dataset, but it still generates characters names and similary grammar/vocabulary to the given input. This is the result of the large capacity of the model and the diverse training data it used.\n",
        "\n",
        "Let's try to change the input to a completely different one, using in this case an input from a rugby game."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-RN1CBKGdN8W"
      },
      "source": [
        "# encode context the generation is conditioned on\n",
        "input_ids = tokenizer.encode(\"40 mins. Tom Curry receives Stuart Hogg’s kick-off and Youngs kicks poorly to put Scotland back on the attack.\\n\", return_tensors='tf')\n",
        "\n",
        "# activate sampling and deactivate top_k by setting top_k sampling to 0\n",
        "sample_output = model.generate(\n",
        "    input_ids, \n",
        "    do_sample=True, \n",
        "    max_length=100, \n",
        "    top_k=50\n",
        ")\n",
        "print(\"Output:\\n\" + 100 * '-')\n",
        "print(tokenizer.decode(sample_output[0], skip_special_tokens=True))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z956_MZLetBY"
      },
      "source": [
        "We see how the model understands that the input is related to sports and changes the output style compared to the Tiny Shakespeare example.\n",
        "\n",
        "We just showed examples with GPT2. However, the newer version of GPT, GPT3, which is trained using a quite larger model and dataset, shows largely improved results. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OwawQJMZ3aQM"
      },
      "source": [
        "# Coursework\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EKfTmNC7Bj5E"
      },
      "source": [
        "### **Task 1: RNN Regression**\n",
        "\n",
        "In this task, you are asked to estimate the next value of a time series. Specifically, we have selected the popular airline passenger dataset. This dataset contains the number of passengers that travels with a certain airline company. The data contains 144 entries, each entry corresponds to the number of the passengers that travel in a given month. The dataset starts in 1949, and it lasts until 1960. \n",
        "\n",
        "Similarly to the previous example, we import the data and plot it to see the structure."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cRf3UGQ33bnN"
      },
      "source": [
        "import pandas\n",
        "import matplotlib.pyplot as plt\n",
        "from keras.layers.core import Dense, Activation, Dropout\n",
        "from keras.layers.recurrent import LSTM\n",
        "from keras.models import Sequential\n",
        "from sklearn.metrics import mean_squared_error\n",
        "from sklearn.preprocessing import MinMaxScaler\n",
        "from sklearn.utils import shuffle\n",
        "import numpy as np\n",
        "\n",
        "!wget https://raw.githubusercontent.com/jbrownlee/Datasets/master/airline-passengers.csv\n",
        "data = pandas.read_csv('airline-passengers.csv', usecols=[1], engine='python')\n",
        "plt.plot(data)\n",
        "plt.xlabel(\"Months\")\n",
        "plt.ylabel(\"Passengers\")\n",
        "plt.title(\"Airline Passengers from January 1949 to December 1960 (12 years)\")\n",
        "plt.show()\n",
        "\n",
        "# convert pandas data frame in numpy array of float.\n",
        "data_np = data.values.astype(\"float32\")\n",
        "\n",
        "# normalize data with min max normalization\n",
        "normalizer = MinMaxScaler(feature_range = (0, 1))\n",
        "dataset = normalizer.fit_transform(data_np)\n",
        "\n",
        "# Using 70% of data for training, 30% for test.\n",
        "TRAINING_PERC = 0.70\n",
        "\n",
        "train_size = int(len(dataset) * TRAINING_PERC)\n",
        "test_size = len(dataset) - train_size\n",
        "train, test = dataset[0:train_size, :], dataset[train_size:len(dataset), :]\n",
        "print(\"Number of samples training set: \" + str((len(train))))\n",
        "print(\"Number of samples test set: \" + str((len(test))))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pOi9udyY3dHS"
      },
      "source": [
        "First of all, you need to train an RNN on the airline passenger dataset. This exercise expects you to study the impact of the `window_size` variable when defining the `train` and `test` dataset splits. Remember that the `window_size` variable indicates the number of past observations used for predicting the current value. Here, we treat the `test` split as a validation set. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WIB08JxD3ezu"
      },
      "source": [
        "def create_dataset(dataset, window_size = 1):\n",
        "    data_x, data_y = [], []\n",
        "    for i in range(len(dataset) - window_size - 1):\n",
        "        sample = dataset[i:(i + window_size), 0]\n",
        "        data_x.append(sample)\n",
        "        data_y.append(dataset[i + window_size, 0])\n",
        "    return(np.array(data_x), np.array(data_y))\n",
        "\n",
        "window_size = 1 #Use this variable to build the dataset with different number of inputs\n",
        "\n",
        "# Create test and training sets for regression with different window sizes.\n",
        "train_X, train_Y = create_dataset(train, window_size)\n",
        "test_X, test_Y = create_dataset(test, window_size)\n",
        "train_X = np.reshape(train_X, (train_X.shape[0], train_X.shape[1], 1))\n",
        "test_X = np.reshape(test_X, (test_X.shape[0], test_X.shape[1], 1))\n",
        "\n",
        "print(\"Shape of training inputs: \" + str((train_X.shape)))\n",
        "print(\"Shape of training labels: \" + str((train_Y.shape)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hH6W6t5q3gfN"
      },
      "source": [
        "\n",
        "**Report**:\n",
        "\n",
        "*   Create a plot showing the test curves of models trained with different `window_size` values. Report the plot and discuss the main differences you observe between the predicted curves. You can use the style proposed on the Many to One RNNs - Regression section to plot your curves."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mwb9IfiCLKuT"
      },
      "source": [
        "### **Task 2: Text Embeddings**\n",
        "For this task, we tackle a classification problem using the IMDB sentiment dataset as done in the example in the notebook. Labels in IMDB are 0 for negative reviews and 1 for positive reviews. The definitions of the models you will use for this task are given the code below. This task is similar to the transfer learning/finetuning task in the CNN Architectures notebook, however we now test the effect of transfer learning in the embeddings. In this task we use train, validation and test splits with Early Stopping. That means that we will take the best performing model in the validation set and use it in the test set to get a final performance.\n",
        "\n",
        "**Report**\n",
        "* Using embeddings of dimensionality 1, train a model without using any LSTM, only using an average pooling of the input embeddings (called `embeddings_model` in the code given below). Then train another model with an LSTM and trainable embeddings initialized at random (called `lstm_model`). Finally train a model with an LSTM with non-trainable embeddings initialized with GloVe embeddings (called `lstm_glove_model`). The code to train the three models is given below. Report in a table the test accuracy obtained after training with the given code for the three models. Also attach in the Appendix the training and validation accuracy curves for the different models trained. You can report the curves after using EarlyStopping with patience 10 (already given in the code), so you don't have to train for the full 50 epochs the three models. Discuss the results.\n",
        "\n",
        "* Predict the sentiment of the two given example reviews in the code below for the model trained without a LSTM (`embeddings_model`) and for the model trained with a LSTM and GloVe embeddings (`lstm_glove_model`). Report the predictions (you can use the same table as when reporting test accuracies). Discuss the results. Also discuss the differences you can observe between the GloVe embeddings and the embeddings learnt in `embeddings_model` (e.g. what kind of properties the embeddings encode, or differences in the closest words).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4Gq0lcyLnaSN"
      },
      "source": [
        "We provide the training code you need to use for this exercise below. First we load the dataset as we did in the tutorial. In this exercise, we will use of train, validation and test splits, which are defined in the next cell.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E0ibhzVdnZ0X"
      },
      "source": [
        "import tensorflow as tf\n",
        "from keras.preprocessing import sequence\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense, Embedding, Dropout, Flatten\n",
        "from keras.layers import LSTM, GlobalAvgPool1D\n",
        "from keras.datasets import imdb\n",
        "from keras.utils import np_utils\n",
        "\n",
        "# number of most-frequent words to use\n",
        "nb_words = 5000\n",
        "n_classes = 1\n",
        "print('Loading data...')\n",
        "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=nb_words)\n",
        "print('x_train:', x_train.shape)\n",
        "print('x_test:', x_test.shape)\n",
        "# get_word_index retrieves a mapping word -> index\n",
        "word_index = imdb.get_word_index()\n",
        "# We make space for the three special tokens\n",
        "word_index_c = dict((w, i+3) for (w, i) in word_index.items())\n",
        "word_index_c['<PAD>'] = 0\n",
        "word_index_c['<START>'] = 1\n",
        "word_index_c['<UNK>'] = 2\n",
        "# Instead of having dictionary word -> index we form\n",
        "# the dictionary index -> word\n",
        "index_word = dict((i, w) for (w, i) in word_index_c.items())\n",
        "# Truncate sentences after this number of words\n",
        "maxlen = 100\n",
        "print('Pad sequences (samples x time)')\n",
        "x_train = sequence.pad_sequences(x_train, maxlen=maxlen)\n",
        "x_test = sequence.pad_sequences(x_test, maxlen=maxlen)\n",
        "print('x_train shape:', x_train.shape)\n",
        "print('x_test shape:', x_test.shape)\n",
        "\n",
        "x_val = x_train[23000:]\n",
        "y_val = y_train[23000:]\n",
        "x_train = x_train[:23000]\n",
        "y_train = y_train[:23000]\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4EDi21HPGU3h"
      },
      "source": [
        "The following code includes the model that uses embeddings of size 1 (so each word is only represented by a single digit) and averages them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7421cnIYoPQ-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "82d6cf6b-2178-4b59-a2ae-fd586cd76da4"
      },
      "source": [
        "# Dimensions of the embeddings\n",
        "embedding_dim = 1\n",
        "\n",
        "print('Build model...')\n",
        "embeddings_model = Sequential()\n",
        "\n",
        "embeddings_model.add(Embedding(nb_words,\n",
        "                    embedding_dim,\n",
        "                    input_length=maxlen,\n",
        "                            trainable=True))\n",
        "embeddings_model.add(GlobalAvgPool1D())\n",
        "embeddings_model.add(Dense(1, activation='sigmoid'))\n",
        "embeddings_model.compile(loss='binary_crossentropy',\n",
        "              optimizer='adam',\n",
        "              metrics=['accuracy'])\n",
        "print(embeddings_model.summary())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Build model...\n",
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding (Embedding)        (None, 100, 1)            5000      \n",
            "_________________________________________________________________\n",
            "global_average_pooling1d (Gl (None, 1)                 0         \n",
            "_________________________________________________________________\n",
            "dense (Dense)                (None, 1)                 2         \n",
            "=================================================================\n",
            "Total params: 5,002\n",
            "Trainable params: 5,002\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "None\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pRm3ql3UE9w7"
      },
      "source": [
        "We use Early Stopping, so the best validation model is then used to compute the result in the test set. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gom37wigpZAn"
      },
      "source": [
        "from keras.callbacks import EarlyStopping\n",
        "\n",
        "## We train the model for at most 50 epochs\n",
        "epochs = 50\n",
        "early_stop = EarlyStopping(\n",
        "    monitor='val_accuracy',\n",
        "    min_delta=0,\n",
        "    patience=10,\n",
        "    verbose=0,\n",
        "    mode='auto',\n",
        "    baseline=None,\n",
        "    restore_best_weights=True\n",
        ")\n",
        "history = embeddings_model.fit(x_train, y_train, batch_size=32, epochs=epochs, callbacks=[early_stop], validation_data=(x_val,y_val))\n",
        "print('Final test accuracy is:')\n",
        "embeddings_model.evaluate(x_test, y_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z_DTYhcl68GF"
      },
      "source": [
        "Now we have `embedding_model` trained. The code below will print the embedding of any `query_word`, which in this case is a single number. We also give you the code to compute the `top_k` closest embeddings to `query_word`. The metric used is the L2 distance."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V9uJoVAt7EWk"
      },
      "source": [
        "weights = embeddings_model.layers[0].get_weights()[0]\n",
        "query_word = '8'\n",
        "dist = ((weights - weights[word_index_c[query_word]])**2).sum(1).argsort()\n",
        "top_k = 10\n",
        "print(\"Embedding value of {:s} is {:f}\", query_word, weights[word_index_c[query_word]][0])\n",
        "print('Most {:d} similar words to {:s}'.format(top_k, query_word))\n",
        "for k in range(1, top_k+1):\n",
        "  print(\"{:d}: {:s}\".format(k, index_word[dist[k]]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EhD9hosS7cKQ"
      },
      "source": [
        "The code below gives the prediction for two example reviews we input. Remember that predictions close to 0 refer to a negative review, and predictions close to 1 refer to a positive review."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mqatHlu8glYQ"
      },
      "source": [
        "from keras.preprocessing.sequence import pad_sequences\n",
        "negative_review = \"the movie is boring and not good\"\n",
        "words = negative_review.split()\n",
        "seq_input = [word_index_c[w] for w in words]\n",
        "seq_input = np.array([seq_input])\n",
        "seq_input = pad_sequences(seq_input,maxlen=100)\n",
        "negative_review_score = embeddings_model.predict(seq_input)\n",
        "\n",
        "positive_review = \"the movie is good and not boring\"\n",
        "words = positive_review.split()\n",
        "seq_input = [word_index_c[w] for w in words]\n",
        "seq_input = np.array([seq_input])\n",
        "seq_input = pad_sequences(seq_input,maxlen=100)\n",
        "positive_review_score = embeddings_model.predict(seq_input)\n",
        "\n",
        "print(\"The score for the negative review is:\", negative_review_score)\n",
        "print(\"The score for the positive review is:\", positive_review_score)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6FzItW89aLVj"
      },
      "source": [
        "With the above code, we trained a model that classifies the sentiment of the sentence using the average of all the embeddings, which were only of size 1. Now we will increase the capacity of the embeddings to 300 and will also add a LSTM to process the embeddings. Hence, the model has a much higher capacity."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R-z4K9iAaKR1"
      },
      "source": [
        "## Model parameters:\n",
        "# Dimensions of the embeddings\n",
        "embedding_dim = 300\n",
        "\n",
        "## LSTM dimensionality\n",
        "lstm_units = 50\n",
        "\n",
        "print('Build model...')\n",
        "lstm_model = Sequential()\n",
        "\n",
        "lstm_model.add(Embedding(nb_words,\n",
        "                    embedding_dim,\n",
        "                    input_length=maxlen,\n",
        "                            trainable=True))\n",
        "### Do not modify the layers below\n",
        "lstm_model.add(Dropout(0.2))\n",
        "lstm_model.add(LSTM(lstm_units, dropout=0.2))\n",
        "lstm_model.add(Dropout(0.2))\n",
        "lstm_model.add(Dense(1, activation='sigmoid'))\n",
        "lstm_model.compile(loss='binary_crossentropy',\n",
        "              optimizer='adam',\n",
        "              metrics=['accuracy'])\n",
        "print(lstm_model.summary())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "msBDHkgfFhtq"
      },
      "source": [
        "Similarly, we use EarlyStopping for this model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cXpHfGa-aoWW"
      },
      "source": [
        "from keras.callbacks import EarlyStopping\n",
        "\n",
        "## We train the model for 50 epochs\n",
        "epochs = 50\n",
        "early_stop = EarlyStopping(\n",
        "    monitor='val_accuracy',\n",
        "    min_delta=0,\n",
        "    patience=10,\n",
        "    verbose=0,\n",
        "    mode='auto',\n",
        "    baseline=None,\n",
        "    restore_best_weights=True\n",
        ")\n",
        "history = lstm_model.fit(x_train, y_train, batch_size=32, epochs=epochs, callbacks=[early_stop], validation_data=(x_val,y_val))\n",
        "print('Final test accuracy is:')\n",
        "lstm_model.evaluate(x_test, y_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N3F0IRz2FqVX"
      },
      "source": [
        "We just trained a model with a large number of parameters in the IMDB, which is a small dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a7dnHR3za1xB"
      },
      "source": [
        "The last model we train is the same model as the `lstm_model` above, but in this case we use the embeddings from the GloVe method (which were introduced in this notebook) without any finetuning. First, we download them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t7ZjGnuWb4H9"
      },
      "source": [
        "!wget https://imperialcollegelondon.box.com/shared/static/c9trfhhwl9ohje5g3sapu3xk2zoywp3c.txt -O gensim_glove_vectors.txt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2vhK-7g7GIZX"
      },
      "source": [
        "Then we load the GloVe embeddings with dimensionality 300 we just downloaded. This takes some time."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5KeP7idDb-Wj"
      },
      "source": [
        "from gensim.models.keyedvectors import KeyedVectors\n",
        "glove_model = KeyedVectors.load_word2vec_format(\"gensim_glove_vectors.txt\", binary=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BV_QslV7ctj4"
      },
      "source": [
        "We now load the pretrained embeddings in a numpy array to then pass it to the Keras Embedding layer. Change the shape of `embedding_matrix` to the corresponding embedding dimensionality you are using."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "npBJO9pOcA9D"
      },
      "source": [
        "import numpy as np\n",
        "embedding_matrix = np.zeros((nb_words, 300))\n",
        "for word, i in word_index_c.items():\n",
        "    if word in glove_model:\n",
        "      embedding_vector = glove_model[word]\n",
        "      if embedding_vector is not None and i < nb_words:\n",
        "          # words not found in embedding index will be all-zeros.\n",
        "          embedding_matrix[i] = embedding_vector"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rXEgmD5CdbEo"
      },
      "source": [
        "To initialize the Keras Embedding layer with the embeddings we loaded, we can use the argument `weights=[embedding_matrix]`. Also, to freeze the embeddings during training, we use `trainable=False`. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FTDZJhNqbFiz"
      },
      "source": [
        "from keras.layers.wrappers import Bidirectional\n",
        "from keras.layers import GlobalMaxPool1D, Flatten, GlobalAvgPool1D, Dropout\n",
        "\n",
        "## Model parameters:\n",
        "# Dimensions of the embeddings\n",
        "embedding_dim = 300\n",
        "\n",
        "## LSTM dimensionality\n",
        "lstm_units = 50\n",
        "\n",
        "print('Build model...')\n",
        "lstm_glove_model = Sequential()\n",
        "\n",
        "lstm_glove_model.add(Embedding(nb_words,\n",
        "                    embedding_dim,\n",
        "                    input_length=maxlen,\n",
        "                    weights=[embedding_matrix],\n",
        "                            trainable=False))\n",
        "### Do not modify the layers below\n",
        "lstm_glove_model.add(Dropout(0.2))\n",
        "lstm_glove_model.add(LSTM(lstm_units, dropout=0.2))\n",
        "lstm_glove_model.add(Dropout(0.2))\n",
        "lstm_glove_model.add(Dense(1, activation='sigmoid'))\n",
        "lstm_glove_model.compile(loss='binary_crossentropy',\n",
        "              optimizer='adam',\n",
        "              metrics=['accuracy'])\n",
        "print(lstm_glove_model.summary())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N7c-u98zbJaf"
      },
      "source": [
        "from keras.callbacks import EarlyStopping\n",
        "\n",
        "## We train the model for 50 epochs\n",
        "epochs = 50\n",
        "early_stop = EarlyStopping(\n",
        "    monitor='val_accuracy',\n",
        "    min_delta=0,\n",
        "    patience=10,\n",
        "    verbose=0,\n",
        "    mode='auto',\n",
        "    baseline=None,\n",
        "    restore_best_weights=True\n",
        ")\n",
        "history = lstm_glove_model.fit(x_train, y_train, batch_size=32, epochs=epochs, callbacks=[early_stop], validation_data=(x_val,y_val))\n",
        "print('Final test accuracy is:')\n",
        "lstm_glove_model.evaluate(x_test, y_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g0v3mUULGkfV"
      },
      "source": [
        "We can also compute the closest words in the GloVe embeddings to any `query_word` using the code below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IQDqhKtOc56X"
      },
      "source": [
        "query_word = '8'\n",
        "weights = embedding_matrix\n",
        "\n",
        "dist = ((weights - weights[word_index_c[query_word]])**2).sum(1).argsort()\n",
        "top_k = 10\n",
        "print('Most {:d} similar words to {:s}'.format(top_k, query_word))\n",
        "for k in range(1, top_k+1):\n",
        "  print(\"{:d}: {:s}\".format(k, index_word[dist[k]]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IXiBKBgtG2xf"
      },
      "source": [
        "We use the same example reviews as for the `embedding_model` case and we compute the predictions using the `lstm_glove_model`. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z10tY2Nyg9px"
      },
      "source": [
        "from keras.preprocessing.sequence import pad_sequences\n",
        "negative_review = \"the movie is boring and not good\"\n",
        "words = negative_review.split()\n",
        "seq_input = [word_index_c[w] for w in words]\n",
        "seq_input = np.array([seq_input])\n",
        "seq_input = pad_sequences(seq_input,maxlen=100)\n",
        "negative_review_score = lstm_glove_model.predict(seq_input)\n",
        "\n",
        "positive_review = \"the movie is good and not boring\"\n",
        "words = positive_review.split()\n",
        "seq_input = [word_index_c[w] for w in words]\n",
        "seq_input = np.array([seq_input])\n",
        "seq_input = pad_sequences(seq_input,maxlen=100)\n",
        "positive_review_score = lstm_glove_model.predict(seq_input)\n",
        "\n",
        "print(negative_review_score, positive_review_score)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CywiPmy1eZoF"
      },
      "source": [
        "### **Task 3: Text Generation**\n",
        "In this task we focus on the text generation problem. For this purpose, we will download the scripts of the TV show Game of Thrones and try to generate some text resembling the style of the scripts.\n",
        "\n",
        "\n",
        "**Report**\n",
        "* Plot the retrieved BLEU for different temperature values (from 0 to 2 in the x-axis) for both the character-level model and the word-level model. To compute the BLEU score, use a minimum of 20 generated samples per temperature used to reduce variability (you can increase it at the cost of higher computational time for lower variability). Each sample should contain 100 characters for the char-level model or 30 words for the word-level model (the code given uses these parameters by default). Do you see any relationship between the obtained BLEU score and temperature used? If you generate sentences at different temperatures what differences can you observe? Are the generated sentences grammatically correct? Do the generated sentences make sense? \n",
        "\n",
        "We give below the code needed to download the dataset and to compute the results."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bD9gEFKXqukY"
      },
      "source": [
        "We first download and read the dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4_I1pRickXKl"
      },
      "source": [
        "!git clone https://github.com/shekharkoirala/Game_of_Thrones"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D1i9e9nekb70"
      },
      "source": [
        "data = open('./Game_of_Thrones/Data/final_data.txt', 'r').read()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oHqzKZQ8aYkp"
      },
      "source": [
        "**Character-level model**\n",
        "\n",
        "We first include the code to build the character-level dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cnwDgj_unBhW",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c4cca5c0-01fc-4119-f0f8-cdd9af987e8e"
      },
      "source": [
        "characters = sorted(list(set(data)))\n",
        "n_to_char = {n:char for n, char in enumerate(characters)}\n",
        "char_to_n = {char:n for n, char in enumerate(characters)}\n",
        "x_char = []\n",
        "y_char = []\n",
        "length = len(data)\n",
        "seq_char_length = 100\n",
        "for i in range(0, length-seq_char_length):\n",
        "  sequence = data[i:i + seq_char_length]\n",
        "  label = data[i + seq_char_length]\n",
        "  x_char.append([char_to_n[char] for char in sequence])\n",
        "  y_char.append(char_to_n[label])\n",
        "n_samples = len(x_char)\n",
        "print ('Total Samples:' , n_samples)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Total Samples: 2474358\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZcmI8iV7afK5"
      },
      "source": [
        "The splits used for training are given below, although we already give the model trained."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3VkGsFkMnCuB"
      },
      "source": [
        "import keras\n",
        "import numpy as np\n",
        "\n",
        "x_train_char = x_char[:int(n_samples*0.8)]\n",
        "x_test_char = x_char[int(n_samples*0.8):]\n",
        "y_train_char = y_char[:int(n_samples*0.8)]\n",
        "y_test_char = y_char[int(n_samples*0.8):]\n",
        "## Transform the list to a numpy array\n",
        "x_train_char = np.reshape(x_train_char, (len(x_train_char), seq_char_length))\n",
        "## Onehot encoding of labels\n",
        "y_train_char = tf.keras.utils.to_categorical(np.asarray(y_train_char))\n",
        "\n",
        "x_test_char = np.reshape(x_test_char, (len(x_test_char), seq_char_length))\n",
        "y_test_char = tf.keras.utils.to_categorical(np.asarray(y_test_char))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hcyzP955aukz"
      },
      "source": [
        "The definition of the model is the one given below. You will not train the model, so this piece of code is only for you to know what kind of model we trained for this task."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LO3FoRM1nEOe"
      },
      "source": [
        "from keras.models import Sequential\n",
        "from keras.layers import Dense, Embedding, Dropout\n",
        "from keras.layers import LSTM\n",
        "# define the LSTM model\n",
        "embedding_size = 300\n",
        "lstm_units = 256\n",
        "\n",
        "char_gen_model = Sequential()\n",
        "char_gen_model.add(Embedding(y_train_char.shape[1],\n",
        "                    embedding_size, input_length=seq_char_length))\n",
        "char_gen_model.add(LSTM(lstm_units))\n",
        "char_gen_model.add(Dense(y_train_char.shape[1], activation='softmax'))\n",
        "\n",
        "char_gen_model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
        "char_gen_model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jSgGhwym_WLx"
      },
      "source": [
        "As the training takes a while, we include a saved model that you can load to skip the training step. Use this model to compute your results."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xcasWx79nFcY"
      },
      "source": [
        "!wget https://imperialcollegelondon.box.com/shared/static/1ffasfm5bx691allukv4y8n0tglr5c06.h5 -O ./char_model.h5\n",
        "char_gen_model = keras.models.load_model(\"./char_model.h5\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MsuUE8vua77e"
      },
      "source": [
        "The code you need to evaluate the BLEU score is given below. Vary the temperature to the different needed values. It takes around 1 minute in average per temperature if `n_eval` is set to 20."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s5ufMSdgp0A6"
      },
      "source": [
        "from nltk.translate.bleu_score import sentence_bleu\n",
        "import sys\n",
        "import time\n",
        "import string\n",
        "a = time.time()\n",
        "characters = sorted(list(set(data)))\n",
        "n_to_char = {n:char for n, char in enumerate(characters)}\n",
        "char_to_n = {char:n for n, char in enumerate(characters)}\n",
        "temperature = 1.0\n",
        "bleu_score = 0\n",
        "n_eval = 20\n",
        "seq_char_length = 100\n",
        "for _ in range(n_eval):\n",
        "  start = np.random.randint(0, len(x_test_char)-seq_char_length-1)\n",
        "  pattern = x_test_char[start].tolist()\n",
        "  reference = x_test_char[start+seq_char_length].tolist()\n",
        "  reference = ''.join([n_to_char[value] for value in reference])\n",
        "  # generate characters\n",
        "  output_sent = ''\n",
        "  for n_char in range(seq_char_length):\n",
        "    x = np.reshape(pattern, (1, len(pattern)))\n",
        "    prediction = char_gen_model.predict(x, verbose=0).astype(np.float64)\n",
        "    prediction = np.log(prediction + 1e-7) / (temperature + 0.01)\n",
        "    exp_preds = np.exp(prediction)\n",
        "    prediction = exp_preds / np.sum(exp_preds)\n",
        "    prediction = np.random.multinomial(1, prediction[0,:], 1)\n",
        "    index = np.argmax(prediction)\n",
        "    result = n_to_char[index]\n",
        "    output_sent += result\n",
        "    pattern.append(index)\n",
        "    pattern = pattern[1:]\n",
        "  # We filter the reference and candidate words to remove any punctuation\n",
        "  # We also use only lower case words\n",
        "  reference = reference.translate(str.maketrans('', '', string.punctuation)).replace('\\n',' ').lower().split(' ')\n",
        "  # We filter any empty element in the list\n",
        "  reference = list(filter(lambda x: x is not '', reference))\n",
        "  candidate = output_sent.replace('\\n',' ').translate(str.maketrans('', '', string.punctuation)).lower().split(' ')\n",
        "  candidate = list(filter(lambda x: x is not '', candidate))\n",
        "  # We remove the first and the last word for both the reference and candidate\n",
        "  # because they may not be completed words\n",
        "  bleu_score += sentence_bleu(reference[1:-1], candidate[1:-1])\n",
        "print(bleu_score/n_eval)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gVEF_WPycOnD"
      },
      "source": [
        "The code below allows you to generate sentences for different input patterns and different temperature values. You can test how the temperature values affect the quality of the output sentences for the character-level model by generating a few examples."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Yshz7OZGnIWX"
      },
      "source": [
        "import sys\n",
        "# Change the temperature here\n",
        "temperature = 0.5\n",
        "# You can modify the input pattern here.\n",
        "pattern = [char_to_n[value] for value in list(\"TYRION pours himself some wine and drinks it down. He pours another glass, and walks back to CERSEI \")]\n",
        "pattern = pattern[:seq_char_length]\n",
        "print(\"\\nPredicted:\")\n",
        "# generate 300 characters\n",
        "for i in range(300):\n",
        "  x = np.reshape(pattern, (1, len(pattern)))\n",
        "  prediction = char_gen_model.predict(x, verbose=0).astype(np.float64)\n",
        "  ## We put the constant 0.02 to avoid dividing by zero\n",
        "  ## We sum by 1e-7 to avoid log(0)\n",
        "  prediction = np.log(prediction + 1e-7) / (temperature + 0.01)\n",
        "  exp_preds = np.exp(prediction)\n",
        "  prediction = exp_preds / np.sum(exp_preds)\n",
        "  ## We applied the smoothing with the temperature\n",
        "  ## Now we predict following the probabilities in the variable prediction\n",
        "  prediction = np.random.multinomial(1, prediction[0,:], 1)\n",
        "  index = np.argmax(prediction)\n",
        "  result = n_to_char[index]\n",
        "  seq_in = [n_to_char[value] for value in pattern]\n",
        "  ## Print the result\n",
        "  sys.stdout.write(result)\n",
        "  \n",
        "  ## Create the input sequence for the next character by appending the predicted\n",
        "  ## character and dropping the first one to always have constant seq. length\n",
        "  pattern.append(index)\n",
        "  pattern = pattern[1:]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CeNwJWRN0Mue"
      },
      "source": [
        "**Word-level model**\n",
        "\n",
        "We now give the code to run the word-level model. The code is similar to the char-level model. The main difference is that we only try to predict the 2000 words most commonly used in the dataset. The reason for this limitation is to limit the size of the output layer and number of input embeddings for memory constraints."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Z-XfzutEv9Ta",
        "outputId": "1d8300cc-89e5-482c-d521-2dd6ca540ddc"
      },
      "source": [
        "from collections import Counter\n",
        "n_words = 2000\n",
        "# We preprocess the data to separate the punctuation from the words so e.g.\n",
        "# \"sword.\" and \"sword\" are not treated as separate words. \n",
        "# We also use only lower-cased words\n",
        "data_p = data.replace('.', ' . ').replace(',', ' , ').replace(':', ' : ').replace('?', ' ? ')\n",
        "data_p = data_p.replace('!', ' !').replace('\\n', ' \\n ').replace('[', ' [ ').replace(']', ' ] ')\n",
        "data_p = data_p.replace(')', ' ) ').replace('(', ' ( ').lower().split(' ')\n",
        "data_p = list(filter(lambda x: x is not '' and x is not ' ', data_p))\n",
        "common = dict(Counter(data_p).most_common(n_words))\n",
        "n_to_word = {n:word for n, word in enumerate(common.keys())}\n",
        "word_to_n = {word:n for n, word in enumerate(common.keys())}\n",
        "x_word = []\n",
        "y_word = []\n",
        "length = len(data_p)\n",
        "seq_length = 30\n",
        "for i in range(0, length-seq_length):\n",
        "  sequence = data_p[i:i + seq_length]\n",
        "  label = data_p[i + seq_length]\n",
        "  seq = []\n",
        "  if label in common:\n",
        "    for word in sequence:\n",
        "      if word in word_to_n:\n",
        "        seq.append(word_to_n[word])\n",
        "      else:\n",
        "        # If the word in the input sequence is not from the most common 2000 \n",
        "        # words, we use a special token. We use the same token for all the \n",
        "        # non-common words.\n",
        "        seq.append(len(word_to_n))\n",
        "    x_word.append(seq)\n",
        "    y_word.append(word_to_n[label])\n",
        "n_samples = len(x_word)\n",
        "print('Total Samples:' , n_samples)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Total Samples: 544224\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "teAOmchqxvQd"
      },
      "source": [
        "import keras\n",
        "import numpy as np\n",
        "\n",
        "x_train_word = x_word[:int(n_samples*0.8)]\n",
        "x_test_word = x_word[int(n_samples*0.8):]\n",
        "y_train_word = np.array(y_word[:int(n_samples*0.8)])\n",
        "y_test_word = np.array(y_word[int(n_samples*0.8):])\n",
        "## Transform the list to a numpy array\n",
        "x_train_word = np.reshape(x_train_word, (len(x_train_word), seq_length))\n",
        "\n",
        "x_test_word = np.reshape(x_test_word, (len(x_test_word), seq_length))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iJvV2NaP0yam"
      },
      "source": [
        "The definition of the word-level model we train is given below. The model is the same as in the char-level case, the only difference is the size of the output vector and the number of input embeddings."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "myJRka3BxyUf"
      },
      "source": [
        "from keras.models import Sequential\n",
        "from keras.layers import Dense, Embedding, Dropout\n",
        "from keras.layers import LSTM\n",
        "# define the LSTM model\n",
        "embedding_size = 300\n",
        "lstm_units = 256\n",
        "\n",
        "word_gen_model = Sequential()\n",
        "word_gen_model.add(Embedding(n_words + 1,\n",
        "                    embedding_size, input_length=seq_length))\n",
        "word_gen_model.add(LSTM(lstm_units))\n",
        "word_gen_model.add(Dense(n_words + 1, activation='softmax'))\n",
        "\n",
        "word_gen_model.compile(loss='sparse_categorical_crossentropy', optimizer='adam')\n",
        "word_gen_model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s10IJIxrzEUE"
      },
      "source": [
        "!wget https://imperialcollegelondon.box.com/shared/static/h38nnk3mbi0zl1w6x76uilawivpeklhz.h5 -O ./word_model.h5\n",
        "word_gen_model = keras.models.load_model(\"./word_model.h5\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GgvJiBOs5CEF"
      },
      "source": [
        "from nltk.translate.bleu_score import sentence_bleu\n",
        "import string\n",
        "import sys\n",
        "temperature = 1.0\n",
        "bleu_score = 0\n",
        "n_eval = 20\n",
        "seq_length = 30\n",
        "for _ in range(n_eval):\n",
        "  # We look for references that do not contain any non-common words as we only\n",
        "  # learnt to predict the 2000 most common words\n",
        "  while True:\n",
        "    start = np.random.randint(0, len(x_test_word)-seq_length-1)\n",
        "    pattern = x_test_word[start].tolist()\n",
        "    reference = x_test_word[start+seq_length].tolist()\n",
        "    if n_words not in reference:\n",
        "      break\n",
        "  reference = ' '.join([n_to_word[value] for value in reference])\n",
        "  # generate words\n",
        "  output_sent = ''\n",
        "  for i in range(seq_length):\n",
        "    x = np.reshape(pattern, (1, len(pattern)))\n",
        "    prediction = word_gen_model.predict(x, verbose=0).astype(np.float64)\n",
        "    prediction = np.log(prediction + 1e-7) / (temperature + 0.01)\n",
        "    exp_preds = np.exp(prediction)\n",
        "    prediction = exp_preds / np.sum(exp_preds)\n",
        "    prediction = np.random.multinomial(1, prediction[0,:], 1)\n",
        "    index = np.argmax(prediction)\n",
        "    result = n_to_word.get(index, '')\n",
        "    output_sent += result + ' '\n",
        "    pattern.append(index)\n",
        "    pattern = pattern[1:]\n",
        "  # We filter the reference and candidate words to remove any punctuation\n",
        "  reference = reference.translate(str.maketrans('', '', string.punctuation)).replace('\\n', ' ').split(' ')\n",
        "  # We filter any empty element in the list\n",
        "  reference = list(filter(lambda x: x is not '', reference))\n",
        "  candidate = output_sent.replace('\\n',' ').translate(str.maketrans('', '', string.punctuation)).split(' ')\n",
        "  candidate = list(filter(lambda x: x is not '', candidate))\n",
        "  bleu_score += sentence_bleu([reference], candidate)\n",
        "print(bleu_score/n_eval)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hf1cYUeC27W3"
      },
      "source": [
        "import sys\n",
        "# Vary the temperature here\n",
        "temperature = 1.0\n",
        "pattern = \"TYRION pours himself some wine and drinks it down. He pours another glass, and walks back to CERSEI placing his cup on her desk. He takes another glass.\\nTYRION: \"\n",
        "# Process input pattern\n",
        "pattern = pattern.replace('.', ' . ').replace(',', ' , ').replace(':', ' : ').replace('?', ' ? ')\n",
        "pattern = pattern.replace('!', ' !').replace('\\n', ' \\n ').replace('[', ' [ ').replace(']', ' ] ')\n",
        "pattern = pattern.replace(')', ' ) ').replace('(', ' ( ').lower().split(' ')\n",
        "pattern = list(filter(lambda x: x is not '' and x is not ' ', pattern))\n",
        "\n",
        "pattern = pattern[:seq_length]\n",
        "\n",
        "print(\"\\nInput Pattern:\\n\", ' '.join(pattern))\n",
        "pattern = [word_to_n.get(value, n_words) for value in pattern]\n",
        "print(\"\\nPredicted:\")\n",
        "# generate 100 words\n",
        "for i in range(100):\n",
        "  x = np.reshape(pattern, (1, len(pattern)))\n",
        "  prediction = word_gen_model.predict(x, verbose=0).astype(np.float64)\n",
        "  ## We put the constant 0.02 to avoid dividing by zero\n",
        "  ## We sum by 1e-7 to avoid log(0)\n",
        "  prediction = np.log(prediction + 1e-7) / (temperature + 0.01)\n",
        "  exp_preds = np.exp(prediction)\n",
        "  prediction = exp_preds / np.sum(exp_preds)\n",
        "  ## We applied the smoothing with the temperature\n",
        "  ## Now we predict following the probabilities in the variable prediction\n",
        "  prediction = np.random.multinomial(1, prediction[0,:], 1)\n",
        "  index = np.argmax(prediction)\n",
        "  result = n_to_word[index]\n",
        "  ## Print the result\n",
        "  sys.stdout.write(result+' ')\n",
        "  \n",
        "  ## Create the input sequence for the next character by appending the predicted\n",
        "  ## character and dropping the first one to always have constant seq. length\n",
        "  pattern.append(index)\n",
        "  pattern = pattern[1:]"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}