Neural Attention Mechanism.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sequence To Sequence Attention Models In DyNet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the most coveted AI tasks is automatic machine translation (MT). In this task a sequence of words in a source language are translated into a sequence of words in a target language (usually those sequences are of different lengths). In order to succeed in this task, the model needs to generalize language patterns from a relatively small dataset of translated examples (the number of possible sentences in each language is large enough to be considered infinite). \n",
    "\n",
    "Sequence to sequence neural models have been shown to make such generalizations for input and output of variable lengths. \n",
    "For example: [Bahdanau et al.2015] Neural Machine Translation by Jointly Learning to Align and Translate in ICLR 2015\n",
    "(https://arxiv.org/abs/1409.0473) achieves nearly state of the art results in MT.\n",
    "\n",
    "In this tutorial, we solve a toy problem on a synthetic dataset to learn the reverse function (map \"abcd\" to \"dcba\") using the same models developed to solve MT.\n",
    "\n",
    "We review three models of increasing sophistication: \n",
    "* a simple Recurrent Neural Network (RNN)\n",
    "* a sequence to sequence encoder-decoder model (https://www.tensorflow.org/versions/r0.9/tutorials/seq2seq/index.html)\n",
    "* finally, an Attention Based model as introduced by Bahdanau et al.\n",
    "\n",
    "This is a hands-on description of these models, using the DyNet framework.\n",
    "Make sure you first install Dynet following these instructions: http://dynet.readthedocs.io/en/latest/python.html\n",
    "\n",
    "The task we address is the following: the input is a random string $(c_0, c_1, c_2 ... c_n)$ of random length n and the output is the reversed string $(c_n, c_{n-1}, c_{n-2} ... c_0)$.\n",
    "\n",
    "Let's build a function that generates an $(input, output)$ instance of our reverse function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from random import choice, randrange\n",
    "\n",
    "EOS = \"<EOS>\" #all strings will end with the End Of String token\n",
    "characters = list(\"abcd\")\n",
    "characters.append(EOS)\n",
    "\n",
    "int2char = list(characters)\n",
    "char2int = {c:i for i,c in enumerate(characters)}\n",
    "\n",
    "VOCAB_SIZE = len(characters)\n",
    "\n",
    "def sample_model(min_length, max_lenth):\n",
    "    random_length = randrange(min_length, max_lenth)                             # Pick a random length\n",
    "    random_char_list = [choice(characters[:-1]) for _ in range(random_length)]  # Pick random chars\n",
    "    random_string = ''.join(random_char_list) \n",
    "    return random_string, random_string[::-1]  # Return the random string and its reverse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us generate a couple $(Input, Output)$ examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('cbab', 'babc')\n",
      "('cbbaadbac', 'cabdaabbc')\n"
     ]
    }
   ],
   "source": [
    "print(sample_model(4, 5))\n",
    "print(sample_model(5, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to compare different models under identical conditions, we create a dataset in advance. \n",
    "The dataset contains 3,000* random strings with lengths ranging from 1 to 15.<br><br> \n",
    "\n",
    "*Note that there are more than $\\sum_{n=1}^{14} 4^{n} >> 4^{14} = 268,435,456$ possible random strings so a dataset containing only 3,000 strings means our model will really need to generalize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_STRING_LEN = 15\n",
    "\n",
    "train_set = [sample_model(1, MAX_STRING_LEN) for _ in range(3000)]\n",
    "val_set = [sample_model(1, MAX_STRING_LEN) for _ in range(50)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As usual, We split the data set into train and validation subsets. \n",
    "\n",
    "We define a $train$ function to optimize a model on our training dataset and plot training errors as learning proceeds over the validation set.  \n",
    "\n",
    "This method uses generic interface of the PyCNN network class which is used to encode any neural network model:\n",
    "* network.get_loss(input, output)\n",
    "* dy.SimpleSGDTrainer(network.model)\n",
    "\n",
    "This applies a backpropagation training regime over the network for a set number of epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import dynet as dy\n",
    "from tqdm import tqdm_notebook as tqdm\n",
    "def train(network, train_set, val_set, epochs = 20):\n",
    "    def get_val_set_loss(network, val_set):\n",
    "        loss = [network.get_loss(input_string, output_string).value() for input_string, output_string in val_set]\n",
    "        return sum(loss)\n",
    "    \n",
    "    train_set = train_set*epochs\n",
    "    trainer = dy.SimpleSGDTrainer(network.model)\n",
    "    losses = []\n",
    "    iterations = []\n",
    "    for i, training_example in enumerate(tqdm(train_set)):\n",
    "        input_string, output_string = training_example\n",
    "        \n",
    "        dy.renew_cg()\n",
    "        \n",
    "        loss = network.get_loss(input_string, output_string)\n",
    "        loss_value = loss.value()\n",
    "        loss.backward()\n",
    "        trainer.update()\n",
    "\n",
    "        # Accumulate average losses over training to plot\n",
    "        if i%(len(train_set)/100) == 0:\n",
    "            val_loss = get_val_set_loss(network, val_set)\n",
    "            losses.append(val_loss)\n",
    "            iterations.append(i/((len(train_set)/100)))\n",
    "\n",
    "    plt.plot(iterations, losses)\n",
    "    plt.axis([0, 100, 0, len(val_set)*MAX_STRING_LEN])\n",
    "    plt.show() \n",
    "    print('loss on validation set:', val_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple RNN Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let us test our toy problem using a simple Recurrent Neural Network (RNN). \n",
    "This model can handle inputs of variable lengths, but it cannot produce outputs of variable lengths. \n",
    "Luckily for us, in our problem the length of the input and the output are equal.\n",
    "\n",
    "Specifically, we will use LSTM units in our RNN - read background information about LSTMs: http://colah.github.io/posts/2015-08-Understanding-LSTMs/\n",
    "\n",
    "Then make sure to read about RNNs in pycnn: https://github.com/clab/cnn/blob/master/pyexamples/tutorials/RNNs.ipynb\n",
    "\n",
    "If you want, compare it with the Tensorflow corresponding tutorial: https://www.tensorflow.org/versions/r0.9/tutorials/recurrent/index.html\n",
    "\n",
    "Note that in all of our models, we have character units - that is, we consider sequences of characters.\n",
    "In many other models, we could consider sequences of words instead of characters, and sometimes sequences of morphemes or word segments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/rnn.jpg\" width=\"50%\" height=\"50%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will develop three models that are all refinements of the same structure.\n",
    "We therefore define a base class that has the following interface:\n",
    "* A constructor with the hyper-parameters of the network (number of layers, sizes of embeddings and for the latent state of the units.\n",
    "* Pre-processors for input and output to normalize the sequences coming in and out of the network.  In our case, these simply add the special <EOS> tokens.\n",
    "* The method $get\\_probs(rnn\\_output)$ maps the computed output of the RNN to a probability distribution over candidate characters by applying a simple softmax transformation - and the corresponding $predict(probs)$ method which picks the maximum likelihood character given the prob distribution.\n",
    "* $get\\_loss(input\\_string, output\\_string)$ computes the overall loss of the network on a single pair.  It runs the network over the input, for each input char, it computes the distribution of possible output chars, and then computes the cross-entropy loss for each character: $-\\sum_{output\\_char} log(prob(output\\_char))$.\n",
    "* $generate(input\\_string)$ which generates a candidate output given the input based on the current state of the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "RNN_BUILDER = dy.LSTMBuilder\n",
    "\n",
    "class SimpleRNNNetwork:\n",
    "    def __init__(self, rnn_num_of_layers, embeddings_size, state_size):\n",
    "        self.model = dy.Model()\n",
    "\n",
    "        # the embedding paramaters\n",
    "        self.embeddings = self.model.add_lookup_parameters((VOCAB_SIZE, embeddings_size))\n",
    "\n",
    "        # the rnn\n",
    "        self.RNN = RNN_BUILDER(rnn_num_of_layers, embeddings_size, state_size, self.model)\n",
    "\n",
    "        # project the rnn output to a vector of VOCAB_SIZE length\n",
    "        self.output_w = self.model.add_parameters((VOCAB_SIZE, state_size))\n",
    "        self.output_b = self.model.add_parameters((VOCAB_SIZE))\n",
    "    \n",
    "    def _pre_process(self, string):\n",
    "        string = list(string) + [EOS]\n",
    "        return [char2int[c] for c in string]\n",
    "    \n",
    "    def _embed_string(self, string):\n",
    "        return [self.embeddings[char] for char in string]\n",
    "\n",
    "    def _run_rnn(self, init_state, input_vecs):\n",
    "        s = init_state\n",
    "\n",
    "        states = s.add_inputs(input_vecs)\n",
    "        rnn_outputs = [s.output() for s in states]\n",
    "        return rnn_outputs\n",
    "    \n",
    "    def _get_probs(self, rnn_output):\n",
    "        probs = dy.softmax(self.output_w * rnn_output + self.output_b)\n",
    "        return probs\n",
    "    \n",
    "    def __call__(self, input_string):\n",
    "        input_string = self._pre_process(input_string)\n",
    "        \n",
    "        embedded_string = self._embed_string(input_string)\n",
    "        rnn_state = self.RNN.initial_state()\n",
    "        rnn_outputs = self._run_rnn(rnn_state, embedded_string)\n",
    "        return [self._get_probs(rnn_output) for rnn_output in rnn_outputs]\n",
    "\n",
    "            \n",
    "    def get_loss(self, input_string, output_string):\n",
    "        output_string = self._pre_process(output_string)\n",
    "\n",
    "        probs = self(input_string)\n",
    "        loss = [-dy.log(dy.pick(p, output_char)) for p, output_char in zip(probs, output_string)]\n",
    "        loss = dy.esum(loss)\n",
    "        return loss\n",
    "\n",
    "    def _predict(self, probs):\n",
    "        probs = probs.value()\n",
    "        predicted_char = int2char[probs.index(max(probs))]\n",
    "        return predicted_char\n",
    "    \n",
    "    def generate(self, input_string):\n",
    "        probs = self(input_string)\n",
    "        output_string = [self._predict(p) for p in probs]\n",
    "        output_string = ''.join(output_string)\n",
    "        return output_string.replace('<EOS>', '')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to run the first RNN network over our sample dataset - and test it on a simple \"ab\" string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b3ccff796b71405ca0749d389a1f276a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=15000), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl4VdW9//H3NzkZyDwQAiTMMsggU4qgVK1Thaqotf60g9RLL63Xtna4bWnvbbW/tl7rrfZqa52qlWrrhLZQS1GL03VADMg8hjkhI2QOmdf942xCwEASMh3dn9fz5Dl7r73OPuvsrJzPWWvvc2LOOURExH/C+roBIiLSNxQAIiI+pQAQEfEpBYCIiE8pAEREfEoBICLiUwoAERGfUgCIiPiUAkBExKcCfd0AgPikFFffL5UzByUQCLO+bo6IyEfCmjVrSpxzaad7/5AIgPTBQ2i88g6e+/Z5jE6P7+vmiIh8JJjZvq7cPySmgALhwXf9h6rr+7glIiL+ERoB4E37HFYAiIj0mpAIgPCwYDM0AhAR6T0hEQAtI4AqBYCISG8JiQAwg/joAIer6/q6KSIivhESAQCQGhupKSARkV4UMgGQEhupk8AiIr0ohAIgSgEgItKLQiYANAUkItK72g0AMxtrZuta/VSY2bfMLMXMXjGznd5tslffzOw+M8sxsw1mNq0jDUmNi6S0uh79k3oRkd7RbgA457Y756Y456YA04Ea4C/AImClc240sNJbB5gDjPZ+FgIPdKQhKbGRNDY7Ko40dv5ZiIhIp3V2CugiYJdzbh8wD1jslS8GrvKW5wF/dEGrgCQzG9TejlPjIgE4pEtBRUR6RWcD4HrgKW853TmX7y0XAOnecgZwoNV9cr2yU0qJjQL0dRAiIr2lwwFgZpHAlcBzJ25zwYn7Tk3em9lCM8s2s+zi4mJSY4+OABQAIiK9oTMjgDnAWudcobdeeHRqx7st8srzgCGt7pfplR3HOfewcy7LOZeVlpZGihcAGgGIiPSOzgTADRyb/gFYBsz3lucDS1uV3+hdDTQTKG81VXRSCgARkd7VoX8IY2axwCXAV1sV3wk8a2YLgH3AdV75cmAukEPwiqGbOvIY0RHhxEaGc0hfCCci0is6FADOuWog9YSyQwSvCjqxrgNuOZ3GpMRF6gvhRER6Sch8EhiCVwLpJLCISO8IqQBI1RfCiYj0mpALAJ0DEBHpHSEVAMFzAPo+IBGR3hBSAZAaG0l9UzNVdfo+IBGRnhZSAaCvgxAR6T0hFQBp8cEAKKnSpaAiIj0tpAIgPSEYAAXlCgARkZ4WWgEQHw1AYUVtH7dEROTjL6QCICkmgshAGIWVCgARkZ4WUgFgZqQnRFFUoSkgEZGeFlIBAMFpoIJyjQBERHpa6AVAQrSmgEREekHIBcAATQGJiPSKkAuA9IRoquoa9WlgEZEeFnIBMDAheClokS4FFRHpUSEXAAOOfhhMASAi0qNCLgDSW0YAOg8gItKTQjYA9GlgEZGeFXIBEBcVIDYynEKNAEREelSHAsDMksxsiZltM7OtZjbLzFLM7BUz2+ndJnt1zczuM7McM9tgZtM626j0xGiNAEREelhHRwD3Aiucc+OAycBWYBGw0jk3GljprQPMAUZ7PwuBBzrbqPR4BYCISE9rNwDMLBE4D3gUwDlX75wrA+YBi71qi4GrvOV5wB9d0CogycwGdaZR6QlR+jSwiEgP68gIYARQDPzBzD4ws9+bWSyQ7pzL9+oUAOnecgZwoNX9c72yDktPiKawok7/G1hEpAd1JAACwDTgAefcVKCaY9M9ALjgK3WnXq3NbKGZZZtZdnFx8XHbBiREU9/YTPmRhs7sUkREOqEjAZAL5Drn3vPWlxAMhMKjUzvebZG3PQ8Y0ur+mV7ZcZxzDzvnspxzWWlpacdtO/ppYH0YTESk57QbAM65AuCAmY31ii4CtgDLgPle2Xxgqbe8DLjRuxpoJlDeaqqoQ47+a0hdCioi0nMCHaz3DeBPZhYJ7AZuIhgez5rZAmAfcJ1XdzkwF8gBary6naIPg4mI9LwOBYBzbh2Q1cami9qo64BbutKotPjgCEBfCCci0nNC7pPAANER4STHROgcgIhIDwrJAIBjl4KKiEjPCNkAGJAQrSkgEZEeFLIBkB4fpRGAiEgPCt0ASIimuKqOpmZ9GlhEpCeEbgAkRtPU7DhUpVGAiEhPCN0AiNeHwUREelLoBoA+DCYi0qNCPwD0tdAiIj0iZAOgf1wkgTBjW35lXzdFRORjKWQDIBAexrwpGTyTfYC8siPHbduQW8Z1D77L3pLqbnms5mbHu7sOUd/Y3C37ExH5KAjZAAD4zqVjALjn5R0tZbUNTXzrmXWs3nuYO5Zv7ZbHefDNXdzwyCp+/NdN3bI/EZGPgpAOgIykftx0znBe+CCXbQUVANz98nZ2F1dz8ZkDeHlLIe/tPtShfe07VM3Tq/ez+WD5ceVr9pVy98s7GJQYzTPZB3hq9f5ufx4iIqEopAMA4OYLRhEfFeCX/9hG9t7D/P6tPXzh7KH85oZpDEqM5o7lW2k+yYfFDlfX89O/beaC/36N8//7dRa9sJGr73+Hp70X+fIjDXzzqQ8YnBTNilvP45Oj+3Pb0s2sO1DWm0+xT7y1s4QDh2v6uhnyEbWnpJrahqa+boZ0UcgHQFJMJLd86gxe217M155cQ0ZSP34490z6RYbz75eOZX1uOX/bcPBD91t/oIwrfvMWT67ax4j+sdx+xXhe/MZszh6ZwqIXNvIff9nIouc3UFhRy29umEZiTAT3XT+VAQlR3PzkGko68AG0gvJa/ndnMW/nlPBOTgn7DnXPOYnO2F1cxb3/3ElVXWOH7/Ps+wf44qPvcfXv3mZ7gU6yd9Shqjrfnydqanbc88oOLrz7db773Pq+bo50kYXCP17Pyspy2dnZJ91e29DEp371Ovnltfz5X8/mnFH9geDJ2yt++xZlNQ2s/O75REeEA/D06v38ZOlm0uKjeOCL0zgrM6llX03Njrte2sZDb+wG4IdzxvHV80e1bN+UV85nH3iHqEAYnzlrEPOmZDBjeAphYXZcm97OKeGrT6w57oU3PMx4/uZzmDIkidORW1pDcWUdEwYnEhk4dTY753g2+wC3L9vCkYYmZp/Rn0e/nEVUIPyU91uxKZ9/+9NaZoxIYXdxNY3NjicXnM34wQltPkZxZR2JMRHt7vejyjlH9r5SYiLDGT8oATNrs86f3tvPz17cwriB8fzhphmkxEb2QWv7VlFlLbc+tY53dx9iVFosu4qrefars5gxIqWlTlOzY82+UiZlJNIvMnT6TH1jc7t/U92ttLqeJ1ftY2JGIjNGpBAb1dH/v9VxZrbGOdfW/2rp2P0/CgEAwXf0+w7XcOXkwceVv7OrhM8/8h794yIJMwt+fUR1PbPP6M99N0w96R/qik35bMqr4DuXjPnQi/sH+0t54t19rNhcQE19E0NS+nHz+Wdw7fRMIgNhLF2Xx78/t56R/eO47YrxBMLDaGxq5rvPrSc2KsCL35jdEkYdUVnbwG9ezeEPb++hockRHRHGlCFJTBicSG1DE2VHGqg40sCA+GhGp8cxKi2Ov3yQy/KNBcwamcqF4wbwi+VbufysQdx3/dQPPZ+j3s4p4aY/vM/EjASe/MrZFFbU8flHVnGkoYm7PzcZ5yC/opbc0hq2HKxg88EKDlfXMzAhmm9dPJprp2cSCG/7j2jNvlJW7znMgdIaDhyuITYywM+vnkj/uKgOH4eC8lpS4yKJOMljHNXY1HzSdnTGnpJqblu2mTd3FAPBf0V6/pg0zj2jP5MyEhmeGktFbQOLnt/Iis0FZA1LZmNeOZnJ/fjjgrPJSOp3yv03NDWzdl8pYWHGJ4annLJua83NjjX7SxmTHk9iv4guPcfucLi6nhfW5vLgG7upqmvgZ/MmcvlZg7nw7tdJjYtk2S2zW/rc7cs28/g7e4mLCjB30kCumZbZ5huojiqqrOU3K3OYf85wzhgQ96Htzrk2Q7u1X720nUf+dzcLzxvJzReMIiaye1+I22qDc44Fi7N5dVvwX6UHwoxpQ5P51/NGcsn49G57bN8EwKk88e5eNuSWEx5mhIUZo9Li+PI5wwk/zU53VE19I69sKeSxt/ey/kAZgxOjOX9sGk+tPsDZI1J4+Mas4/5A39hRzPzHVvO180exaM64lvL6xmZKa+ox41hIVdVzqLqOnKIq7n9tF4eq6/jc9EzOHzOA7H2HeX/vYXYUVhEfFSCxXwTx0QEKKmpbvhojEGZ899KxLDxvJOFhxoNv7OLOf2xj/qxh3H7lhOM6ZG1DE0+t3s+vXtrOkJQYnlk4i8SYYLv3H6rhhkdWHXepbUS4MSY9nomDExmdHsffN+bzwf4yRqXFcuvFY7h0fHpLwJVW1/Pzv2/l+bW5ACTHRDAkJYYdhZUMSuzHEwtmkJkcc8rjnFd2hJ+/uIV/bCpg9IA4brtiArNH92/ZXlpdz9u7Sli95zCr9xxmR2Ell44fyC+unkiqFzDOOf62IZ8X1x9kRP9YJmYkMikjkWGpMR/646xtaOL+13J46I3dRAXC+NYlY0iIDvD6jmLe3FFMZW1wVBcbGU5kIIzK2ka+f9lYvjJ7JO/vPcxXFmcTFx3gsS9/gnED41v239Ts2FFYydr9pby1s4S3dpZQ6Y0Qb7tiPDedO6LdPre3pJrvLVnP+3tLSYgOsPC8kXz53BHERQUoqqjlrZwSthdWEhsZID46QFJMBBefmU589LF+6Jzjt6/m8OjbezgrM4kLx6bxqXEDGJYae9LHdc7x/t5Snl+TS0NTM8mxkSTHRLC9sIqXNhVQ39RM1rBkfnH1JMYOjAdg6bo8bn16HXddexbXZQ3hsbf28P9f3MK10zMxYPnGfKrrmxiUGM1lEwcyd9Igpg9N7nAYHKqq4/qHV7GzqIq0+CieXjiTUWnBEKhrbOInf93MS1sK+NLMYdx07og23+wdbeMZA+LIKapiUGI0i+aM48rJg9sNjpPJKaritW1FbM2vYGtBJXtLqrkuK5OfXDGh5TXn8bf3cPvftvCjueOYMDiRt3JKeGlzAbuLq/nK7BH8YM64dt/otOXENz8KgF7gnOPNnSXc+88drN1fxtxJA7nnuiltvsv/wZINPLfmAM/ffA5nZSaxZM0B7n55B0WVJz+nMG1oErdfOeG4qaqTKT/SQE5RJamxUQzvf+wP2jnHHcu38sj/7mFyZiKfGJ7C9GHJ5JUd4aE3d1NcWcfMkSneeY7o4/Z5qKqOtfvLGBAfxaDEaFLjoo4LT+ccL28p5L9f2k5OURWxkeFcdGY6EwYn8NCbu6k40sBXzx/JwvNGtQRi9t7D/Mvj7xMTGeCJBTMYnR6Pc8HRWVlNA/WNzdQ1NvF2Tgm/fS0HgC+cPYxXthSy/3ANl45PZ9qwZF7dWkT2vsM0O4iJDGfa0GSGpsawJDuXhH4R/PKzk8hI7sdtSzfz3p7DpCdEUVrdQH1TcK5+xvAUfnz5eCZlJgKwavchfvjCRvaUVHP11Ax+OHccA+KPHY/GpmZ2FFax6WA5m/PKKayo4+YLRjG51bTe5oPlzH/sfUqq6ogKhDEoMZrEmEh2FlZSUx88MZqeEMUFYwZwwdg0lq47yIrNBfz0ygnMP2d4m7/X5mbH4+/s5a6XthERHsatF43m3V2HWLmtiOSYCNITotnmna8JhBmNrS58GNE/lge/OJ2xA4PH+K6XtvPA67uYOTKFoso6dhcHz01dOz2Tn1818bh+W9vQxJI1uTy5ah/bCiqJjwqQ0C+Cspp6quubSIgOcM20TG6YMbTlhb91v/jsA+9woPQIP5o7ju88u55Lx6fzuy9MJzzMOFLfxMtbCnhxQz5v7CimvrGZEf1j+fX/m9LuNGl5TQM3PLKKXcVV3H7lBO5+eTvhYcYzC2cRExnOV59cwwf7y8galkz2vlL6RYRzw4yh/Mvs4S1vODbklvG5B99lcmYST37lbDbklnH73zazKa+Cq6YM5q5rJ590WmhXcRV3/mMb4WaMGhDLqLQ4Cipq+dv6fLbmB69IHJgQzbhB8cREhrN8YwGXnzWIe66bwq7iKubd/3ZwWnZ+VkvQ1DU2ccfft7L43X1MG5rEbz8/jcHtjCJbW7u/lG8/s47ffWEaEwYH+7MCoBc559hTUs3w1NiTvoupqG3g079+k6hAGNER4WwrqGTq0CSumRZ8V3R0uNg/LpLUuCj6x0UxvI13qaejudnx0Ju7Wbm1kA155S0nLGeNTOXWi0czc2Rql/bf5H1g7u8bD7JiUwGlNQ1MGZLEnZ+dxLiBHz6HsDW/gi89upr6xiYGJESTW1pDbcOHT6LOmTiQ/7x8PBlJ/ahtaOLRt/bw21dzONLQxJmDErj4zAF8atwAJmUktrxr2ppfwbefWce2gkrCDBL6RfC9T4/l+k8MbXknvmr3IR54fReHa+q5ZmomURFh/Pm9/QxNieG/rpnEuWf0/1BbOupg2RFWbCogv/wIB8trKa2uZ/SAOKYMTWLqkOTjRh71jc18/c9reXlLIT+bN4EvzRresp+GpmaWrjvIg2/sIqeoigvHDeCOqycxMDEYSusOlHnHopHZZ6TxydH9GT8ogSbnqKptZGNeOd99bj1VtY3ccc1ENuVV8Ohbe/j82UP5+byJhIUZ+w5V89TqAzz4xi4mDE7gwS9OJzO5H8vWH+SuFdvJKzvChMEJ3DhrGFdMHtwyRVLX2ESY2Snfqa4/UMa8+98GYHJmIk8vnNXm3H9VXSMrtxZy14rtFFbU8u1LxvC180e1OUovqapjwePvszW/kkfmZ3H+mDS2F1RywyOriAqE0ewcFUcauee6ycyZNIidhZU88MYulq47iHOOS8anc+30Ifxk6SbCzFj69XNbpiKbmh2/ey2Hu1/ZwayRqTz4penHjeKdczyXncttyzYTGQgjNS6SfYdqWr6WfurQJK6cPJg5Ewe1/I4AHnpjF//1j22cPyaNg2VHKDvSwIpbP9kyQm3txQ0HWfT8RgLhxi8/exafnjDwpMf3qKXr8vjekg0MTIjmsS9nccaAYBj3SgCY2V6gEmgCGp1zWWaWAjwDDAf2Atc550ot2OvvBeYCNcCXnXNrT7X/j0oAdNSbO4q58bHVDEnpxw8uG8dnJg3qlhf4zqhvbGbzwXICYWEt7367U0NTM3tKqhmVFnfKqbZ9h6r52YtbCISFkZncj8zkfiTHRhIVCCcqEMaAhKiWdzOtlVbXU9fYfNwf2YnqGpu4/7VdVNY28M0LR5PcxhRARW0D97+awx/e3ktjczMLZo/gO5eM7fUTlPWNzfzbn9byz62FpMVHMaJ/LENTYnh31yHyyo4wbmA837hwNHMnDex0XymqqOXrf/6A1XsPA/Dlc4Zz2xXjP7SflVsL+dYz6wgPM4alxLA+t5wJgxP4j8+cyayRqafdR3/81028vauEZxbOIi3+1Od8yo808J9/3cTf1h8ka1gys0f3JyU2kqSYSHIKK3ljZwkbcssIN+PBL07n4lbz5VvzK7jhkVXERgZ45MasD124kFd2hCdX7eOp1fspq2mgX0Q4S26e1Wb/+ssHuXx/yQZG9I/ltism0NDUTHVdE8s35fP3DfmcMyqVe66bwsDEaOobm9l/uJp+kYFTnvd5evV+fvSXjTQ7eGLBDD45Ou2kdfeUVPPNpz5gY145nz97KD/+zPg2+2Rzs+N//rmD+17NYcaIFB784vTjprp6MwCynHMlrcruAg475+40s0VAsnPuB2Y2F/gGwQA4G7jXOXf2qfb/cQsACA4hM5P7fWyvnvmoyS2toaa+iTHp8e1X7iF1jU08uWo/2/Ir2Huomj0lNYxMi+Xm80dxwdi0Lr1JaGhq5v7XcogKhPO180eedF97S6r52pNrKK2p53ufHsc1UzNO+wTtUc45nKPD+3HO8cLaPH65YttxU6NhBlOGJHHemDQumziwzVFlaXU9kYGwU15RU9vQxN835JOZ3I+zTzHqfce7kq+y1ZV8gTDjO5eO4avntT06ac8bO4opra7nqqkZ7datb2zm7le289AbuxmWGsO4gfEtb4xKaxrIKztCXmkNFbWNfG56Jr+4etKHpqz6MgC2Axc45/LNbBDwunNurJk95C0/dWK9k+3/4xgAIqGqqdnhnOuWK6m6qqEpeIFEaXUDAxOiWy5O6C0F5bXsKKwkNipAXFSAtPioXr/E962dJdy3cicVtQ3UNjRR19hMQnQEGcn9GJwUTdawFOZNafukdVcDoKPXQzngZTNzwEPOuYeB9FYv6gXA0bFaBnCg1X1zvbKTBoCI9J7gO9venZI8mYjwMAbERx93Ir43DUyMPuU0Y2+YPbr/cVe99aaOBsBs51yemQ0AXjGzba03OuecFw4dZmYLgYUAQ4cO7cxdRUSkG3RoDOicy/Nui4C/ADOAQm/qB++2yKueBwxpdfdMr+zEfT7snMtyzmWlpZ38ZImIiPSMdgPAzGLNLP7oMnApsAlYBsz3qs0HlnrLy4AbLWgmUH6q+X8REekbHZkCSgf+4p2ACAB/ds6tMLP3gWfNbAGwD7jOq7+c4BVAOQQvA72p21stIiJd1m4AOOd2A5PbKD8EXNRGuQNu6ZbWiYhIj+n768BERKRPKABERHxKASAi4lMKABERn1IAiIj4lAJARMSnFAAiIj6lABAR8SkFgIiITykARER8SgEgIuJTCgAREZ9SAIiI+JQCQETEpxQAIiI+pQAQEfEpBYCIiE8pAEREfEoBICLiUwoAERGf6nAAmFm4mX1gZi966yPM7D0zyzGzZ8ws0iuP8tZzvO3De6bpIiLSFZ0ZAdwKbG21/kvg1865M4BSYIFXvgAo9cp/7dUTEZEQ06EAMLNM4DPA7711Ay4ElnhVFgNXecvzvHW87Rd59UVEJIR0dATwP8D3gWZvPRUoc841euu5QIa3nAEcAPC2l3v1RUQkhLQbAGZ2OVDknFvTnQ9sZgvNLNvMsouLi7tz1yIi0gEdGQGcC1xpZnuBpwlO/dwLJJlZwKuTCeR5y3nAEABveyJw6MSdOuceds5lOeey0tLSuvQkRESk89oNAOfcD51zmc654cD1wKvOuS8ArwHXetXmA0u95WXeOt72V51zrltbLSIiXdaVzwH8APiOmeUQnON/1Ct/FEj1yr8DLOpaE0VEpCcE2q9yjHPudeB1b3k3MKONOrXA57qhbSIi0oP0SWAREZ9SAIiI+JQCQETEpxQAIiI+pQAQEfEpBYCIiE8pAEREfEoBICLiUwoAERGfUgCIiPiUAkBExKcUACIiPqUAEBHxKQWAiIhPKQBERHxKASAi4lMKABERn1IAiIj4lAJARMSnFAAiIj7VbgCYWbSZrTaz9Wa22cx+6pWPMLP3zCzHzJ4xs0ivPMpbz/G2D+/ZpyAiIqejIyOAOuBC59xkYApwmZnNBH4J/No5dwZQCizw6i8ASr3yX3v1REQkxLQbAC6oyluN8H4ccCGwxCtfDFzlLc/z1vG2X2Rm1m0tFhGRbtGhcwBmFm5m64Ai4BVgF1DmnGv0quQCGd5yBnAAwNteDqR2Z6NFRKTrOhQAzrkm59wUIBOYAYzr6gOb2UIzyzaz7OLi4q7uTkREOqlTVwE558qA14BZQJKZBbxNmUCet5wHDAHwticCh9rY18POuSznXFZaWtppNl9ERE5XR64CSjOzJG+5H3AJsJVgEFzrVZsPLPWWl3nreNtfdc657my0iIh0XaD9KgwCFptZOMHAeNY596KZbQGeNrOfAx8Aj3r1HwWeMLMc4DBwfQ+0W0REuqjdAHDObQCmtlG+m+D5gBPLa4HPdUvrRESkx+iTwCIiPqUAEBHxKQWAiIhPKQBERHxKASAi4lMKABERn1IAiIj4lAJARMSnFAAiIj6lABAR8SkFgIiITykARER8SgEgIuJTCgAREZ9SAIiI+JQCQETEpxQAIiI+pQAQEfEpBYCIiE8pAEREfKrdADCzIWb2mpltMbPNZnarV55iZq+Y2U7vNtkrNzO7z8xyzGyDmU3r6SchIiKd15ERQCPwXefceGAmcIuZjQcWASudc6OBld46wBxgtPezEHig21stIiJd1m4AOOfynXNrveVKYCuQAcwDFnvVFgNXecvzgD+6oFVAkpkN6vaWi4hIl3TqHICZDQemAu8B6c65fG9TAZDuLWcAB1rdLdcrExGRENLhADCzOOB54FvOuYrW25xzDnCdeWAzW2hm2WaWXVxc3Jm7iohIN+hQAJhZBMEX/z85517wiguPTu14t0VeeR4wpNXdM72y4zjnHnbOZTnnstLS0k63/SIicpo6chWQAY8CW51z97TatAyY7y3PB5a2Kr/RuxpoJlDeaqpIRERCRKADdc4FvgRsNLN1XtmPgDuBZ81sAbAPuM7bthyYC+QANcBN3dpiERHpFu0GgHPuLcBOsvmiNuo74JYutktERHqYPgksIuJTCgAREZ9SAIiI+JQCQETEpxQAIiI+pQAQEfEpBYCIiE8pAEREfEoBICLiUwoAERGfUgCIiPiUAkBExKcUACIiPqUAEBHxKQWAiIhPKQBERHxKASAi4lMKABERn1IAiIj4lAJARMSn2g0AM3vMzIrMbFOrshQze8XMdnq3yV65mdl9ZpZjZhvMbFpPNl5ERE5fR0YAjwOXnVC2CFjpnBsNrPTWAeYAo72fhcAD3dNMERHpbu0GgHPuTeDwCcXzgMXe8mLgqlblf3RBq4AkMxvUXY0VEZHuc7rnANKdc/necgGQ7i1nAAda1cv1ykREJMR0+SSwc84BrrP3M7OFZpZtZtnFxcVdbYaIiHTS6QZA4dGpHe+2yCvPA4a0qpfplX2Ic+5h51yWcy4rLS3tNJshIiKn63QDYBkw31ueDyxtVX6jdzXQTKC81VSRiIiEkEB7FczsKeACoL+Z5QK3AXcCz5rZAmAfcJ1XfTkwF8gBaoCbeqDNIiLSDdoNAOfcDSfZdFEbdR1wS1cbJSIiPU+fBBYR8SkFgIiITykARER8SgEgIuJTCgAREZ9SAIiI+JQCQETEpxQAIiI+pQAQEfEpBYCIiE8pAET2CJlVAAAEf0lEQVREfEoBICLiUwoAERGfUgCIiPiUAkBExKcUACIiPqUAEBHxKQWAiIhPKQBERHxKASAi4lM9EgBmdpmZbTezHDNb1BOPISIiXdPtAWBm4cD9wBxgPHCDmY3v7scREZGu6YkRwAwgxzm32zlXDzwNzOuBxxERkS7oiQDIAA60Ws/1ykREJIQE+uqBzWwhsNBbrTOzTX3VlhDTHyjp60aECB2LY3QsjtGxOGZsV+7cEwGQBwxptZ7plR3HOfcw8DCAmWU757J6oC0fOToWx+hYHKNjcYyOxTFmlt2V+/fEFND7wGgzG2FmkcD1wLIeeBwREemCbh8BOOcazezrwEtAOPCYc25zdz+OiIh0TY+cA3DOLQeWd+IuD/dEOz6idCyO0bE4RsfiGB2LY7p0LMw5110NERGRjxB9FYSIiE/1eQD49WsjzGyImb1mZlvMbLOZ3eqVp5jZK2a207tN7uu29hYzCzezD8zsRW99hJm95/WNZ7yLCj72zCzJzJaY2TYz22pms/zaL8zs297fxyYze8rMov3UL8zsMTMran2Z/Mn6ggXd5x2XDWY2rb3992kA+PxrIxqB7zrnxgMzgVu8574IWOmcGw2s9Nb94lZga6v1XwK/ds6dAZQCC/qkVb3vXmCFc24cMJngMfFdvzCzDOCbQJZzbiLBi0qux1/94nHgshPKTtYX5gCjvZ+FwAPt7byvRwC+/doI51y+c26tt1xJ8I88g+DzX+xVWwxc1Tct7F1mlgl8Bvi9t27AhcASr4ovjoWZJQLnAY8COOfqnXNl+LRfELxQpZ+ZBYAYIB8f9Qvn3JvA4ROKT9YX5gF/dEGrgCQzG3Sq/fd1AOhrIwAzGw5MBd4D0p1z+d6mAiC9j5rV2/4H+D7Q7K2nAmXOuUZv3S99YwRQDPzBmw77vZnF4sN+4ZzLA34F7Cf4wl8OrMGf/aK1k/WFTr+e9nUA+J6ZxQHPA99yzlW03uaCl2h97C/TMrPLgSLn3Jq+bksICADTgAecc1OBak6Y7vFRv0gm+K52BDAYiOXD0yG+1tW+0NcB0KGvjfi4MrMIgi/+f3LOveAVFx4dtnm3RX3Vvl50LnClme0lOA14IcF58CRv6A/+6Ru5QK5z7j1vfQnBQPBjv7gY2OOcK3bONQAvEOwrfuwXrZ2sL3T69bSvA8C3XxvhzXE/Cmx1zt3TatMyYL63PB9Y2ttt623OuR865zKdc8MJ9oFXnXNfAF4DrvWq+eVYFAAHzOzol3xdBGzBh/2C4NTPTDOL8f5ejh4L3/WLE5ysLywDbvSuBpoJlLeaKmqbc65Pf4C5wA5gF/Affd2eXnzeswkO3TYA67yfuQTnvlcCO4F/Ail93dZePi4XAC96yyOB1UAO8BwQ1dft66VjMAXI9vrGX4Fkv/YL4KfANmAT8AQQ5ad+ATxF8PxHA8HR4YKT9QXACF5VuQvYSPDqqVPuX58EFhHxqb6eAhIRkT6iABAR8SkFgIiITykARER8SgEgIuJTCgAREZ9SAIiI+JQCQETEp/4P1v+upH6YZ10AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss on validation set: 516.9060791730881\n",
      "aaa\n"
     ]
    }
   ],
   "source": [
    "RNN_NUM_OF_LAYERS = 2\n",
    "EMBEDDINGS_SIZE = 4\n",
    "STATE_SIZE = 128\n",
    "\n",
    "\n",
    "rnn = SimpleRNNNetwork(RNN_NUM_OF_LAYERS, EMBEDDINGS_SIZE, STATE_SIZE)\n",
    "train(rnn, train_set, val_set, epochs = 5)\n",
    "print(rnn.generate('abc'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We observe that the loss does not improve over time - training does not converge.\n",
    "We also see that the network did not learn how to reverse 'ab'.\n",
    "\n",
    "Our RNN model has no chance to learn the $reverse\\ function$ since it predicts each output character $o_i$ immediately after reading input character $i_i$. So when predicting the 1st output charcter the RNN didn't even see the last input charcter.\n",
    "\n",
    "No magic going on here - so let's improve.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bidirectional RNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can improve the model slightly by using Bidirectional RNN, One RNN read the input string from left to write and another will read it from right to left, for each step we will concatenate the output of the two RNNs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/birnn.jpg\" width=\"50%\" height=\"50%\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BiRNNNetwork(SimpleRNNNetwork):\n",
    "    def __init__(self, rnn_num_of_layers, embeddings_size, state_size):\n",
    "        self.model = dy.Model()\n",
    "\n",
    "        # the embedding paramaters\n",
    "        self.embeddings = self.model.add_lookup_parameters((VOCAB_SIZE, embeddings_size))\n",
    "\n",
    "        # the foreword rnn\n",
    "        self.fwd_RNN = RNN_BUILDER(rnn_num_of_layers, embeddings_size, state_size, self.model)\n",
    "        # the backword rnn\n",
    "        self.bwd_RNN = RNN_BUILDER(rnn_num_of_layers, embeddings_size, state_size, self.model)\n",
    "\n",
    "        # project the rnn output to a vector of VOCAB_SIZE length\n",
    "        self.output_w = self.model.add_parameters((VOCAB_SIZE, state_size*2))\n",
    "        self.output_b = self.model.add_parameters((VOCAB_SIZE))\n",
    "    \n",
    "    def __call__(self, input_string):\n",
    "        input_string = self._pre_process(input_string)\n",
    "        \n",
    "        embedded_string = self._embed_string(input_string)\n",
    "        \n",
    "        #run the foreword RNN\n",
    "        rnn_fwd_state = self.fwd_RNN.initial_state()\n",
    "        rnn_fwd_outputs = self._run_rnn(rnn_fwd_state, embedded_string)\n",
    "        \n",
    "        #run the backword rnn\n",
    "        rnn_bwd_state = self.bwd_RNN.initial_state()\n",
    "        rnn_bwd_outputs = self._run_rnn(rnn_bwd_state, embedded_string[::-1])[::-1]\n",
    "        \n",
    "        #concataenate the foreward and backword outputs\n",
    "        rnn_outputs = [dy.concatenate([fwd_out, bwd_out]) for fwd_out, bwd_out in zip(rnn_fwd_outputs, rnn_bwd_outputs)]\n",
    "        \n",
    "        loss = []\n",
    "        probs = [self._get_probs(rnn_output) for rnn_output in rnn_outputs]\n",
    "            \n",
    "        return probs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "316abec6298d47e896f256de84e6d1a3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=15000), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl8FfW9//HXJzsJCUkghABBQFmVPSIuVdTWIlixtlJtVapUeltq7a1d7G393S632721Km1FqRvUfataS60K7gISQCEsSkAgCYHsIfty8v39cQYIEEhCtoPzfj4eeWTmOzPnfM8wzDvf73xnjjnnEBER/wnr6QqIiEjPUACIiPiUAkBExKcUACIiPqUAEBHxKQWAiIhPKQBERHxKASAi4lMKABERn4ro6QoA9OvXzw0dOrSnqyEiclJZu3ZtkXMu5US3D4kAiEkeQGZmZk9XQ0TkpGJmuzqyfUh0AZVXN/R0FUREfCckAqChqQk9lE5EpHuFRAA4B5V1jT1dDRERXwmJAAAoqKjr6SqIiPhKyARAoQJARKRbhUwAqAUgItK9QiYA1AIQEeleIREABhRU1PZ0NUREfCUkAiAiPEwtABGRbtZqAJjZKDP7oNnPfjP7npklm9mrZrbN+53krW9mttDMss1sg5lNbu09IsJMASAi0s1aDQDn3EfOuYnOuYnAFKAa+DtwG7DcOTcCWO7NA1wKjPB+5gOLWnuPSLUARES6XXu7gC4GtjvndgGzgSVe+RLgCm96NrDUBa0CEs0s7XgvGhFuGgUkItLN2hsAVwOPe9Opzrl8b3ovkOpNDwJymm2T65UdU0SYUVJVT0OgqZ3VERGRE9XmADCzKOBy4Okjl7ngg3za9TAfM5tvZplmlllXUw1AUaVaASIi3aU9LYBLgXXOuX3e/L4DXTve7wKvPA9Ib7bdYK/sMM65xc65DOdcRmKfeED3AoiIdKf2BMA1HOr+AXgRmOtNzwVeaFZ+vTcaaBpQ3qyrqEWRYcFqFOxXAIiIdJc2fSGMmcUBnwO+2az4d8BTZjYP2AXM8cqXATOBbIIjhm5otRLhBkChuoBERLpNmwLAOVcF9D2irJjgqKAj13XAgnZVQi0AEZFuFxJ3AptBUmwkhZV6HISISHcJiQAA6B8foxaAiEg3CpkASImP1jUAEZFuFDIB0D8+Wi0AEZFuFDIBkBIfTWFFnb4cXkSkm4RUANQHmthfoy+HFxHpDiEVAKAvhhER6S4hEwD942MAPQ5CRKS7hEwAHGoBKABERLpDyARA/4RgAKgFICLSPUImAOKjI4iOCNM1ABGRbhIyAWBm9E+IVgtARKSbhEwAAKT0jtY1ABGRbhJSAdA/PkYtABGRbhJSAZASrxaAiEh3CakA6B8fTXlNA3WNgZ6uiojIp15IBcCBewHUDSQi0vVCKgBGDgh+OfyanSU9XBMRkU+/kAqAiYMTGdgnhpc+PO53yIuISCcIqQAICzMumzCQt7YVUl7d0NPVERH5VGtTAJhZopk9Y2ZbzWyLmZ1tZslm9qqZbfN+J3nrmpktNLNsM9tgZpPbU6HLxqfREHD8e9PeE/k8IiLSRm1tAdwNvOycGw1MALYAtwHLnXMjgOXePMClwAjvZz6wqD0VGjeoD0OSY/nHhj3t2UxERNqp1QAwsz7A+cADAM65eudcGTAbWOKttgS4wpueDSx1QauARDNLa2uFzIzLxqfx3vZiivUdwSIiXaYtLYBhQCHwkJmtN7P7zSwOSHXOHbhauxdI9aYHATnNts/1ytrssvEDCTQ5XlY3kIhIl2lLAEQAk4FFzrlJQBWHunsAcMEv8m3Xl/ma2XwzyzSzzMLCwsOWjUmL59SUOP7xobqBRES6SlsCIBfIdc6t9uafIRgI+w507Xi/C7zleUB6s+0He2WHcc4tds5lOOcyUlJSDlsW7AYayOpPSijYr8dDi4h0hVYDwDm3F8gxs1Fe0cXAZuBFYK5XNhd4wZt+EbjeGw00DShv1lXUZl+YkIZz8K8sdQOJiHSFiDaudzPwqJlFATuAGwiGx1NmNg/YBczx1l0GzASygWpv3XY7rX88aX1iWL+7lLnnDD2RlxARkeNoUwA45z4AMlpYdHEL6zpgQQfrBcCYtAS25Fd0xkuJiMgRQupO4CONSYtne2Glng4qItIFQjoARg9IoLHJkV1Q2dNVERH51AnpABiTlgDAVnUDiYh0upAOgKF9Y4mOCGNL/v6eroqIyKdOSAdARHgYowbEs3WvWgAiIp0tpAMAYPSAeLbuVQtARKSzhXwAjElLoKiynoIK3REsItKZQj4ARg/QhWARka4Q8gEwJi34PcG6ECwi0rlCPgASY6NI6xOjC8EiIp0s5AMADjwSQi0AEZHOdFIEwOgB8WQXVFLf2NTTVRER+dQ4KQJgTNrxHwnhnCP4DDoREWmrkyQAgheCW7of4F8b85n22+Xc8crH3V0tEZGT2kkRAEP7xh31SIiiyjoWPLqObz26jsraRu59c7seGici0g5t/UKYHhURHsbI1HiWbymguj5AbmkNH+SUUVMf4IefH8WXpwzms3e8yf/8czMP3zC1p6srInJSOClaAADThiezo6iKZRvzKamq5/yRKbz03fNYcOFppCbEcMtnR/DGR4W8vrWg9RcTEREsFC6eZmRkuMzMzOOu45yjtqGJXlHhLS6vb2xixl1vgcHLt5xPVMSJZVtjoImlK3dR0xDgsvFpnNI37oRe50SUVNUTaHKkxEd323uKyMnLzNY651r6tsa2bX+yBEBbrNi6jxsfzuTaaUOIi47gw5wyckpqmDU+jW98Zhj942MOrtsQaKK2IUB8TOTBsoL9tdz8+HpWf1JysGxCeiJfO2sIczLSO1y/43ll015+8PSHJPSK5LXvX0BMZMtBJyJyQEcD4KS4BtBWF47qz/RRKTyyajdR4WGMSYtn9IB47n97Bw+/t5Orz0wnOS6K9z8pYf3uMuoaA2QMTeaSsakMSuzF7S9sorKugTuumsC0U/vyzw17eG5dHj96ZgNDkmOZNrxvp9e5vrGJ37+8lQfe+YTh/eLYUVTFg+9+wrenn9bp7yUi0lybWgBmthOoAAJAo3Muw8ySgSeBocBOYI5zrtTMDLgbmAlUA193zq073ut3VgsAoKK2gV3F1YxI7U10RPCv6J1FVdz75naeXZdLY5NjzIAEpg5Lpnd0BK9t2XfwMRPD+8Wx6NopjBoQf/D1ahsCXPSHN+jbO5oXFpxLWJh1Sj0Biivr+MbSTNbvLmPu2afwX7PG8J3H1rNyezGv/2C6uoJE5Li6pQvIC4AM51xRs7L/BUqcc78zs9uAJOfcj81sJnAzwQA4C7jbOXfW8V6/MwPgeEqr6gkLM/r0ijysPKekmg255VwwKoXe0Uc3iv6+Ppf/fPJD/jhnAldOHtwpdckrq+G6+1eTV1bDHXMmcNn4gQDsKKzkkjvf4qqMwfz2yvGd8l7tEWhyrNlZQsYpSUSEnzRjBER8qaMB0JH/4bOBJd70EuCKZuVLXdAqINHM0jrwPp0mKS7qqJM/QHpyLLPGp7V48geYPWEQ4wf34f/+/RE19YF2v29tQ4CCitqDdytnF1Ty5UXvUVhZxyPfOOvgyR9geEpvrj97KE+uyWnx+UcNgSZ+/uImLv/zO/z8xU38c0M++/Z3zncllFXX8/WH3ufqxau4+fH11DW2/7OKyMmjrdcAHPCKmTngPufcYiDVOZfvLd8LpHrTg4CcZtvmemX5nKTCwoyfzRrLnPtWcv/bO7j54hGHLS+vaeDpzBx2FVfznYuCw1IP+GhvBTc89D57ymuJj45geEocu0qqiQgL48n5ZzN2YMJR73fLxSN4bn0uv3ppMw/dcObBrqzKuka+/eg63vq4kAnpiTy5JoeH39sJQL/eUYxJS2D0gHgmpidx5rCkwy56t+ajvRXM/1sme8pquHLSIJ5bn0f10rXce+2UgyOvgo/coFO7wQ5oDDTxVGYuFbUNxEZHEBsZzrRT+zIosVenv1eoaww08cSaHC4YmUJ6cmxPV0c+xdoaAOc55/LMrD/wqpltbb7QOee8cGgzM5sPzAcYMmRIezbtEVOHJTPj9AEsenM70ZFhJMUGWxPvZhfx9NpcqusDRIQZyzbmc+dXJnL+yBTeyy7im4+spVdkOD+dOYbc0mq2F1YxZkACv71yHEP7tTzEtE9sJLdeMorbn8/i3N+t4KtTh3DJ6QO47bkNbMmv4H+/NJ45Z6bTEGhi0579rNtVypb8/WzZu58lK3fx17c/AWBYvzguGZvKzRePOGbrBuDlrL18/6kPiIuO4In5ZzPllCTOGp7Mbc9tZO5D73PN1HTezS7mvewiiqvqmXJKEtOG92XSkESq6hrJLa1hb3kt00f157wR/dq9b2vqA3znsXUsP+IejpT4aP51y2fo17vnr4U0NTmy9pQTGR7GqSm9T3iYcVv8Y8MefvZ8Fr2jI/jF5adz5eRBBC+tiXSudg8DNbOfA5XATcB051y+18XzhnNulJnd500/7q3/0YH1jvWa3XUNoKN2FlUx576VFFTUHSyLCg/jsglp3HjuMKIjwljw2Do+3lfJ5RMG8q+sfIb1i+OhG6a2+y9Z5xzvZBex5L2dLN9agHPQKzKce66dzIWj+h9zu4ZAE1l55azZWcKqHSW8/lEBaQkx/PrKcUdt19TkuGv5NhYu38bE9ETuvXYKA/ocajX848M9/OeTH9DY5EiMjeTs4X1JTYhhzc4SNufvp/mhEx5mhBn89foMph+nfkcqr27gG0vXkLmrlF9efjpXTh5MVX0jOwqrmPvg+0wb3peHvn5ml7Q6WtPU5HhrWyH/3rSP17bso9D7d48IM05N6c3AxBjqGoPDiSPCwrht5mgmD0nq8HvOuPstAk2OvnHRvL+zhFnj0/j1FWeQGBvVGR/rpLKruIprH1jNDy4ZxeyJg3q6Oj2qtKqenz2fxYILTzvYc9DlF4HNLA4Ic85VeNOvAr8ELgaKm10ETnbO/cjMZgHf4dBF4IXOueM+n+FkCQAInpgr6hopq2qgtLqeQUm9DvsLtaY+wC/+sYkn1uRw9vC+3HvdlBavO7THruIqnluXx+fGpnLGoD7t2nbtrlJue3YD2woquWx8Gp8bm8roAQmkJkTzw2c28OrmfVw1ZTC/uuKMFu892F5YSU19gLFpCYedhMurG8jaU06fXpEMSuxFWJhxzeJV7Ciq5JF5Z5ExNLnVuuWV1TDv4TVsL6zkrq9MYtb4wy8V/W3VLm5/PoufzhzDTecPP2zZ/toGXtu8j5ez9pIYG8lPLh1DUlzbTpDl1Q2s+Ggfs8YNPOZf8iu3F/PrZZvJyttPXFQ400f153NjUwkLM7bm72fr3goKKmqJiQgnJjKc7YWVlFU3sOjaye0KwCO9unkfNy3N5K6vTOQLEwZy31vb+eMrH5MxNInHb5rW6S2ByrpGckqqGZN2dFdkKPjVS5t54J1PiAw3ltw4lXNObX8L89PiyTW7+fGzG3np5vMOnge6IwCGA3/3ZiOAx5xzvzazvsBTwBBgF8FhoCXeMNA/AzMIDgO9wTl33LP7yRQAbfXR3gqGp8QRGQIjaeoaA9zz+nYWvbn9sO9UCA8zbp81hrnnDO2UE0tRZR1z7l1JYWXdMa9vQPCv3Eff383vlm0B4L7rMlrsOnLO8R+PrGXF1gKe/dY5JMVG8cbHwcd9vLOtiPpAEwMSYiiuqiM5Loo7rprYahdUVl4533p0LTklNXxmRD8WXTvlsO6x7YWV/HbZFl7bUsDAPjH84POjmDkurdUb8woqavn6g2v4eF8Fd8yZcEJ/rTrnuHLRexRV1vH6rdMPjsJ6+N1P+Pk/NvPoN87i3NM6fgIsr27g1S37eDkrn7e2FVHf2MQdV03gS1M6Z4RbZ6ltCHDWb5YzaUgieaU17N1fy7PfOoeRqfGtb/wpdN0Dq9lVXM2bP5x+8P+r7gSWNqtvbGJHUSUf7a0gu6CSC0amtOkv9fbIK6vhy4veo6y6ganDkpk2vC9nDk0iJjKc+kATVXWN/GlFNu9/UsJ5p/Xjt1eOO+6FzrLqembe/TZFVfUHw2twUi8+f/oAZo5LY1J6Ipvz9/PdJ9azo7CKq6YMJi46gsLKOoor6xie0pvzR6Rw7ml9+VfWXm5/Pouk2CjmnJnOX17PZmxaAg9+/UyiIsJYuHwbS97bSUxkON++8FRuPHdYu+7I3l/bwE1LMnl/Zwm3zxrLjecNa9e+W7m9mGv+uopfXXEG10075WB5bUOA6f/3BoOTevH0f5x9QmFdUFHLK5v28e9Ne1m5vZjGJsfAPjHMOCONrD3lfJhTxjP/cQ7jBrevhdmVnl2by61Pf8hjN53FkORYvnjPe0SFh/H3b59D/4TDBzg451i5o5iiynqiwsOIjgzj9IEJ7RoIEWhyFFfVtWub7lJaVU/Gr19j/vnD+fGM0QfLFQAScnYXV3P/OztYtaOYj/cd/Yju+JgIbp81lqsyBrfpZPZBThn3vJ7N1GHJTB/Vn1NT4o7arqY+wK+Xbeax1buJi44gpXc0fWIj+XhvBVX1AcLDjECT45xT+7Lwmkn06x3Niq37+Paj6+jXO5qa+gAl1fVcfWY6t14y6oQvPNc2BLjlifX8e9M+bjh3KD+bNZbwFq5f7C2v5btPrCfM4MZzh/HZManMfeh9tuRX8M6PLzwqeP62cie3v7CJpTdO5fyRKUDw/pUfP7uBPWU11DQEqG1oYmBiLy6fMJDLJw5kQEIMb20r5LHVu1mxtYBAk2No31hmnJHGjDMGMGFwH8yM4so6Lv/zuzjnePHm8w777NX1jeSU1LCruIqCijqSYqPo1zuKvr2j2FVczfrdZazPKaW0qoHBSb1IT44lPakXQ/rGMiQ5jsFJvU74sSZX3vMuZdUNLL/1AsyMjbnlzLlvJRPTE3nsprMOOwaWbczn248efr9p7+gIbr9sDHMy0ls9znYXV/P9pz5g3e5S/javc1pamTtLuOOVj7nt0tFMSE/s0Gs98f5ubnvu8O4fUABIiCuqrOOD3WUEnCMqIozo8DBGpyWQ3Mb++vYKNLnDTrj1jU2s313K29uKSIyN5OvnDD3sBrf1u0u5aWkmw/v15v99YWy7r7Ecqw6//ucWHnz3Ey4Zm8rdV0867CGG63eXMv9va6mua6RPr0j2lNeSntyLnJIafjRjVIuPAalrDHDh/71Bap8YnvvWOewuqearf11NRW0D00f1p1dkONGRYWzMK2f97jIA+sZFUVxVT7/eUXx5SjpfnDSIkam9WzwZZuWV86VF7zExPZEbzxvGu9lFvJNdxI7CquN+1vAwY/SAeFLio8krrSGntJrahkPdjGZw02eG85NLR7er5bJ5z35mLnybn80awzc+c+j6z6Ord/HTv2dx11cmcsWkYDdbTX2Az/7xTRJ6RbLw6onUNTZRWdfI3a9tY+WOYi4YmcJvrxzHwBYGYjjneGJNDr96aTPh3k2itQ0Bln33M0e1Mg6sX1hZx/6aBk5NaXlfQvCGziu9lnBMZBgLr57EJacPaPPnP9J1D6xmd0k1b/xg+mHvqQAQ6aDGQFOX3PX80Luf8MuXNnNKcixnn9qXsQP7EAg08Zt/bWVAQgz3z81geL84Xt60l7++/Qn7ymt55fvnkxDT8qCBAye/X1x+Ove+uZ2ahgCPzDvrqNDaXVzNCx/ksXVfBTPPCF74b8uw1QN3vAPERoVz1rBkJg9J4pR+cQxJjiU1IZrymgYKK+oorqxnYGIvxg3qc1i4Oecoqqxnd0k1OSXVvP5RAS98sOeoE/mRXs7KJy46gvNO64eZ8dO/b+SZtbms/q+LDxv9FGgKXifJK61h+a0X0KdXJH985SMWrsjmqW+ezdRhh7o0m5ocj6zexW+XbaWmIUBUeBix0eHERoZjZjjnqA84iirrOOfUvvzhqglU1jUy+8/vMiG9D4/MO4uI8DBqGwIHR+Nt21dBaXUDEBwa/sPPj+LMI7pRiyvruHLRe1TUNvLX6zP45Uub2ZBbxv+7bCw3nNt6t2BDoImIMDt4oj9W9w8oAERC2oqt+7j/7U/YtGc/5TXBE8fZw/tyz9cmHzVqyTl33L+S6xubuPAPb5BXVkNyXBSPzDvrmBfaT9TrWwuIi45gYnpip9zr0NTkWPDYOl7etJdFX5vCjDOO/it48Vvb+c2y4K1FU4cm860LT+U7j65jxhlp3DFnwlHrb8wtZ/Zf3uHaaafwjfOG89k73+TSMwZw99WTWqzDruIqXtqQT0VtIzX1jVTVB3Au2DoxYNKQJK4+M/3gKLdn1ubyg6c/5OaLTmNMWgK/WbaF3NIaJgzuw9iBCYxMjSfQ5LjvrR0UVtRxwcgUZo4bwPCU3qQnxbLgsXVk5ZXz2E3TmHJKEjX1Ab73ZLBb8NbPjTzqRtLm9u2v5Yt/eZczBvXhz1+dTFRE2DG7f0ABIHJScM6RV1bDnrJaJg1JPOHRYcs25rNw+TYWXjPppBkNU9sQ4Jq/rmJL/n4ev2kak5rdK3Hg5D9rfBrThiXzpxXZB++zee7b5xzzvor/fiGLpat2MW5QH7ILKllx6/TD7mHpqB8+/SFPr80FYPSAeG6/bOxR1wVq6gMsXbmTe9/cfrBVAMFgueerk7l03KFhzYEmxw+f/pDn1uex+LopLXYH1Tc2cfXilWTt2U99YxOzxqWx8JpJfP2h91vs/gm+lwJAREJcUWUdX7znXfaU1TJ5SCIXjEyhrrGJP63IZtb4NO7+ysSD3S2PrNpFYWUdt8049nWD/bUNXPSHNymqrDvmdZOOqKkP8N8vZjExPYmvnJne4oX8AwJNjtzSanYUVrG9sJLT+vdu8V6Q2oYAX7lvJdkFlTy/4FxGHBHgP3t+I4+s2s1fvjqZvLLqg8H4ctZevnn+cH50RPcPKABE5CSxp6yGR1fv4s2PC8nKCz7osPnJv73e/LiQ59fn8bsvjTv4vKxQl19ewxf+9A7xMZE8v+DcgzeJPpWZw4+e2cA3zx/OT2aOAeCu1z7mrte2AbTY/QMKABE5CRVW1LGzuIpJ6Ym+e+z4mp0lXLN4FakJMST0iqS+McDukmqmDktmyQ1TD+4P5xx/WpHNlvz93PO1yS22hhQAIiInmWUb83kqM4fI8DCiwsNIjoviPz83st3Do/WVkCIiJ5mZ49KYOa7nvybFX20vERE5SAEgIuJTCgAREZ9SAIiI+JQCQETEpxQAIiI+pQAQEfEpBYCIiE8pAEREfEoBICLiU20OADMLN7P1ZvaSNz/MzFabWbaZPWlmUV55tDef7S0f2jVVFxGRjmhPC+AWYEuz+d8DdzrnTgNKgXle+Tyg1Cu/01tPRERCTJsCwMwGA7OA+715Ay4CnvFWWQJc4U3P9ubxll9s7fk2aBER6RZtbQHcBfwIaPLm+wJlzrlGbz4XGORNDwJyALzl5d76IiISQloNADO7DChwzq3tzDc2s/lmlmlmmYWFhZ350iIi0gZtaQGcC1xuZjuBJwh2/dwNJJrZge8TGAzkedN5QDqAt7wPUHzkizrnFjvnMpxzGSkpKR36ECIi0n6tBoBz7ifOucHOuaHA1cAK59zXgNeBL3urzQVe8KZf9Obxlq9wofC1YyIicpiO3AfwY+D7ZpZNsI//Aa/8AaCvV/594LaOVVFERLpCu74S0jn3BvCGN70DmNrCOrXAVZ1QNxER6UK6E1hExKcUACIiPqUAEBHxKQWAiIhPKQBERHxKASAi4lMKABERn1IAiIj4lAJARMSnFAAiIj6lABAR8SkFgIiITykARER8SgEgIuJTCgAREZ9SAIiI+JQCQETEpxQAIiI+pQAQEfEpBYCIiE+1GgBmFmNm75vZh2a2ycx+4ZUPM7PVZpZtZk+aWZRXHu3NZ3vLh3btRxARkRPRlhZAHXCRc24CMBGYYWbTgN8DdzrnTgNKgXne+vOAUq/8Tm89EREJMa0GgAuq9GYjvR8HXAQ845UvAa7wpmd783jLLzYz67Qai4hIp2jTNQAzCzezD4AC4FVgO1DmnGv0VskFBnnTg4AcAG95OdC3MystIiId16YAcM4FnHMTgcHAVGB0R9/YzOabWaaZZRYWFnb05UREpJ3aNQrIOVcGvA6cDSSaWYS3aDCQ503nAekA3vI+QHELr7XYOZfhnMtISUk5weqLiMiJassooBQzS/SmewGfA7YQDIIve6vNBV7wpl/05vGWr3DOuc6stIiIdFxE66uQBiwxs3CCgfGUc+4lM9sMPGFm/wOsBx7w1n8A+JuZZQMlwNVdUG8REemgVgPAObcBmNRC+Q6C1wOOLK8FruqU2omISJfRncAiIj6lABAR8SkFgIiITykARER8SgEgIuJTCgAREZ9SAIiI+JQCQETEpxQAIiI+pQAQEfEpBYCIiE8pAEREfEoBICLiUwoAERGfUgCIiPiUAkBExKcUACIiPqUAEBHxKQWAiIhPKQBERHyq1QAws3Qze93MNpvZJjO7xStPNrNXzWyb9zvJKzczW2hm2Wa2wcwmd/WHEBGR9mtLC6ARuNU5NxaYBiwws7HAbcBy59wIYLk3D3ApMML7mQ8s6vRai4hIh7UaAM65fOfcOm+6AtgCDAJmA0u81ZYAV3jTs4GlLmgVkGhmaZ1ecxER6ZB2XQMws6HAJGA1kOqcy/cW7QVSvelBQE6zzXK9MhERCSFtDgAz6w08C3zPObe/+TLnnANce97YzOabWaaZZRYWFrZnUxER6QRtCgAziyR48n/UOfecV7zvQNeO97vAK88D0pttPtgrO4xzbrFzLsM5l5GSknKi9RcRkRPUllFABjwAbHHO/bHZoheBud70XOCFZuXXe6OBpgHlzbqKREQkRES0YZ1zgeuAjWb2gVf2X8DvgKfMbB6wC5jjLVsGzASygWrghk6tsYiIdIpWA8A59w5gx1h8cQvrO2BBB+slIiJdTHcCi4j4lAJARMSnFAAiIj6lABAR8SkFgIiITykARER8SgEgIuJTCgAREZ9SAIiI+JQCQETEpxQAIiI+pQAQEfEpBYCIiE8pAEREfEoBICLiUwoAERGfUgCIiPiUAkBExKcUACLcvgtIAAAF6klEQVQiPqUAEBHxqVYDwMweNLMCM8tqVpZsZq+a2Tbvd5JXbma20MyyzWyDmU3uysqLiMiJa0sL4GFgxhFltwHLnXMjgOXePMClwAjvZz6wqHOqKSIina3VAHDOvQWUHFE8G1jiTS8BrmhWvtQFrQISzSytsyorIiKd50SvAaQ65/K96b1Aqjc9CMhptl6uVyYiIiGmwxeBnXMOcO3dzszmm1mmmWUWFhZ2tBoiItJOJxoA+w507Xi/C7zyPCC92XqDvbKjOOcWO+cynHMZKSkpJ1gNERE5UScaAC8Cc73pucALzcqv90YDTQPKm3UViYhICIlobQUzexyYDvQzs1zgv4HfAU+Z2TxgFzDHW30ZMBPIBqqBG7qgziIi0glaDQDn3DXHWHRxC+s6YEFHKyUiIl1PdwKLiPiUAkBExKcUACIiPqUAEBHxKQWAiIhPKQBERHxKASAi4lMKABERn1IAiIj4lAJARMSnFAAiIj6lABAR8SkFgIiITykARER8SgEgIuJTCgAREZ9SAIiI+JQCQETEpxQAIiI+pQAQEfGpLgkAM5thZh+ZWbaZ3dYV7yEiIh3T6QFgZuHAX4BLgbHANWY2trPfR0REOqYrWgBTgWzn3A7nXD3wBDC7C95HREQ6oCsCYBCQ02w+1ysTEZEQEtFTb2xm84H53mydmWX1VF1CTD+gqKcrESK0Lw7RvjhE++KQUR3ZuCsCIA9IbzY/2Cs7jHNuMbAYwMwynXMZXVCXk472xSHaF4doXxyifXGImWV2ZPuu6AJaA4wws2FmFgVcDbzYBe8jIiId0OktAOdco5l9B/g3EA486Jzb1NnvIyIiHdMl1wCcc8uAZe3YZHFX1OMkpX1xiPbFIdoXh2hfHNKhfWHOuc6qiIiInET0KAgREZ/q8QDw62MjzCzdzF43s81mtsnMbvHKk83sVTPb5v1O6um6dhczCzez9Wb2kjc/zMxWe8fGk96ggk89M0s0s2fMbKuZbTGzs/16XJjZf3r/P7LM7HEzi/HTcWFmD5pZQfNh8sc6FixoobdfNpjZ5NZev0cDwOePjWgEbnXOjQWmAQu8z34bsNw5NwJY7s37xS3AlmbzvwfudM6dBpQC83qkVt3vbuBl59xoYALBfeK748LMBgHfBTKcc2cQHFRyNf46Lh4GZhxRdqxj4VJghPczH1jU2ov3dAvAt4+NcM7lO+fWedMVBP+TDyL4+Zd4qy0BruiZGnYvMxsMzALu9+YNuAh4xlvFF/vCzPoA5wMPADjn6p1zZfj0uCA4UKWXmUUAsUA+PjounHNvASVHFB/rWJgNLHVBq4BEM0s73uv3dADosRGAmQ0FJgGrgVTnXL63aC+Q2kPV6m53AT8Cmrz5vkCZc67Rm/fLsTEMKAQe8rrD7jezOHx4XDjn8oA/ALsJnvjLgbX487ho7ljHQrvPpz0dAL5nZr2BZ4HvOef2N1/mgkO0PvXDtMzsMqDAObe2p+sSAiKAycAi59wkoIojunt8dFwkEfyrdhgwEIjj6O4QX+vosdDTAdCmx0Z8WplZJMGT/6POuee84n0Hmm3e74Keql83Ohe43Mx2EuwGvIhgP3ii1/QH/xwbuUCuc261N/8MwUDw43HxWeAT51yhc64BeI7gseLH46K5Yx0L7T6f9nQA+PaxEV4f9wPAFufcH5stehGY603PBV7o7rp1N+fcT5xzg51zQwkeAyucc18DXge+7K3ml32xF8gxswMP+boY2IwPjwuCXT/TzCzW+/9yYF/47rg4wrGOhReB673RQNOA8mZdRS1zzvXoDzAT+BjYDvy0p+vTjZ/7PIJNtw3AB97PTIJ938uBbcBrQHJP17Wb98t04CVvejjwPpANPA1E93T9umkfTAQyvWPjeSDJr8cF8AtgK5AF/A2I9tNxATxO8PpHA8HW4bxjHQuAERxVuR3YSHD01HFfX3cCi4j4VE93AYmISA9RAIiI+JQCQETEpxQAIiI+pQAQEfEpBYCIiE8pAEREfEoBICLiU/8fybWkqUhVdMQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss on validation set: 492.0908612012863\n",
      "aaa\n"
     ]
    }
   ],
   "source": [
    "RNN_NUM_OF_LAYERS = 2\n",
    "EMBEDDINGS_SIZE = 4\n",
    "STATE_SIZE = 64\n",
    "\n",
    "\n",
    "birnn = BiRNNNetwork(RNN_NUM_OF_LAYERS, EMBEDDINGS_SIZE, STATE_SIZE)\n",
    "train(birnn, train_set, val_set, epochs = 5)\n",
    "print(birnn.generate('abc'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the BiRNN improved over RNN it didn't manage to generalize the problem, we can see clear overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encoder Decoder Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now try to solve our problem using the encoder-decoder model. An encoder RNN will read the entire input string and generate a fixed length encoded vector to represent the entire input string. The decoder RNN will then use the encoded string to predict the output.\n",
    "\n",
    "This strategy should address the concern we identified above - which is that to generate a reverse string properly we should first look at the whole input string - then start generating. \n",
    "\n",
    "Naturally, we are still skeptical: can it be that the encoder will \"remember\" the input string sufficiently to generate the whole output strings?  Let us try it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/encdec.jpg\" width=\"50%\" height=\"50%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We prepare a new class EncoderDecoderNetwork which derives from SimpleRNNNetwork.\n",
    "\n",
    "In the constructor, we build two RNNs - encoder and decoder.\n",
    "We define a new method $encode\\_string(embedded\\_string)$ which runs the input string into the encoder.\n",
    "We then adapt $generate(input\\_string)$ and $get\\_loss(input\\_string, output\\_string)$ to perform the \n",
    "encoder / decoder choreography.\n",
    "\n",
    "Note that the decoder receives the same encoding of the whole input string for all slices.\n",
    "Each slice of the decoder must learn to decide what \"part\" of the encoding it should look at to decide\n",
    "which output to produce - based on the latent state the previous slice produced (remember that all slices are\n",
    "repetitions of the same network unit with the same parameters)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EncoderDecoderNetwork(SimpleRNNNetwork):\n",
    "    def __init__(self, enc_layers, dec_layers, embeddings_size, enc_state_size, dec_state_size):\n",
    "        self.model = dy.Model()\n",
    "\n",
    "        # the embedding paramaters\n",
    "        self.embeddings = self.model.add_lookup_parameters((VOCAB_SIZE, embeddings_size))\n",
    "\n",
    "        # the rnns\n",
    "        self.ENC_RNN = RNN_BUILDER(enc_layers, embeddings_size, enc_state_size, self.model)\n",
    "        self.DEC_RNN = RNN_BUILDER(dec_layers, enc_state_size, dec_state_size, self.model)\n",
    "\n",
    "        # project the rnn output to a vector of VOCAB_SIZE length\n",
    "        self.output_w = self.model.add_parameters((VOCAB_SIZE, dec_state_size))\n",
    "        self.output_b = self.model.add_parameters((VOCAB_SIZE))\n",
    "\n",
    "    def _encode_string(self, embedded_string):\n",
    "        initial_state = self.ENC_RNN.initial_state()\n",
    "\n",
    "        # run_rnn returns all the hidden state of all the slices of the RNN\n",
    "        hidden_states = self._run_rnn(initial_state, embedded_string)\n",
    "\n",
    "        return hidden_states\n",
    "\n",
    "    def __call__(self, input_string):\n",
    "        input_string = self._pre_process(input_string)\n",
    "                \n",
    "        embedded_string = self._embed_string(input_string)\n",
    "        # The encoded string is the hidden state of the last slice of the encoder\n",
    "        encoded_string = self._encode_string(embedded_string)[-1]\n",
    "\n",
    "        rnn_state = self.DEC_RNN.initial_state()\n",
    "\n",
    "        probs = []\n",
    "        for _ in range(len(input_string)):\n",
    "            rnn_state = rnn_state.add_input(encoded_string)\n",
    "            p = self._get_probs(rnn_state.output())\n",
    "            probs.append(p)\n",
    "        return probs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to run it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d6671540c6a7424ca166c9e8edfa3667",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=60000), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XmYnFWV+PHvqare97073el0ku7OvjdLCHuAYVGCIwqiggyacWQctxnFn+OP0Rnncfm5wOigLCooAopKEBEIISCLhCQkpLOnk3TSW3rf1+qq+/uj3qqu3qv36q7zeZ48XfXWW9W3i+I9dc+991wxxqCUUir02Ga6AUoppWaGBgCllApRGgCUUipEaQBQSqkQpQFAKaVClAYApZQKURoAlFIqRGkAUEqpEKUBQCmlQpRjphsAEBWXaBwJGSzJjJvppiil1Kyxd+/eOmNM2nifHxQBICkjm7SP/5A9914z001RSqlZQ0TOTOT5QZECstuEli4nbrfWJVJKqekSNAHAGGjt7p3ppiilVMgImgAA0NzhnOGWKKVU6AiOACBWAOjUAKCUUtMlOAKATQOAUkpNtyAJAJ5mNHX2zHBLlFIqdARJANAegFJKTbegCAAODQBKKTXtgiIAiEC4w6azgJRSahqNGgBEZImI7Pf71yIinxeRZBHZLiInrJ9J1vkiIveLSImIHBCR9YE0JCEqTHsASik1jUYNAMaYY8aYtcaYtcAGoAP4I3APsMMYUwDssO4DXAcUWP+2Ag8E0pBEDQBKKTWtxpoC2gycNMacAbYAj1rHHwVusm5vAR4zHm8DiSKSNdoLJ0SF0aQpIKWUmjZjDQC3Ak9YtzOMMVXW7XNAhnU7Gyjze065dWxEmgJSSqnpFXAAEJFw4EbgdwMfM8YYYEyV3ERkq4jsEZE9tbW1JERrAFBKqek0lh7AdcC7xphq6361N7Vj/ayxjlcA8/2el2Md68cY86AxpsgYU5SWlqY9AKWUmmZjCQAfoS/9A/AscId1+w5gm9/x263ZQBcCzX6pomElRIXR1t1Lr8s9hiYppZQar4ACgIjEAFcDf/A7/G3gahE5AVxl3Qd4HjgFlAAPAZ8J5HckRoUB0NKlJaGVUmo6BLQjmDGmHUgZcKwez6yggeca4O6xNiQh2hMAmjp6SI4JH+vTlVJKjVFQrAQGTwoItByEUkpNlyAKAJ5v/RoAlFJqegRRANAegFJKTScNAEopFaKCLwBoOQillJoWQRMAwh02osPt2gNQSqlpEjQBAKyCcBoAlFJqWgRdANAegFJKTQ8NAEopFaKCLwDoILBSSk2L4AsA2gNQSqlpEVQBIFH3BFBKqWkTVAEgISqMTqeL7l7XTDdFKaXmvKALAKCrgZVSajoEVQBIsspAH6xonuGWKKXU3BdUAeDyJeksTovhX393gLKGjplujlJKzWlBFQBiIxw8dHsRTpebTz22h44e3R1MKaWmSlAFAIBFabH8+Lb1HK9u5V9/9x6eDcaUUkpNtqALAACXFabx1euW8XzxOR7fdXamm6OUUnNSUAYAgE9espClmXH86b3KmW6KUkrNSQEFABFJFJGnReSoiBwRkY0ikiwi20XkhPUzyTpXROR+ESkRkQMisn48DRMRNi9LZ8+ZRlq6dFqoUkpNtkB7APcBLxhjlgJrgCPAPcAOY0wBsMO6D3AdUGD92wo8MN7GXbEkHZfb8MaJuvG+hFJKqWGMGgBEJAG4FHgEwBjTY4xpArYAj1qnPQrcZN3eAjxmPN4GEkUkazyNWzs/kYSoMHYerRnP05VSSo0gkB7AQqAW+IWI7BORh0UkBsgwxlRZ55wDMqzb2UCZ3/PLrWNj5rDbuLQwjVeP1+J262wgpZSaTIEEAAewHnjAGLMOaKcv3QOA8czVHNMVWkS2isgeEdlTW1s77HlXLEmjtrWbw1UtY3l5pZRSowgkAJQD5caYXdb9p/EEhGpvasf66c3TVADz/Z6fYx3rxxjzoDGmyBhTlJaWNuwvv7QwDRE0DaSUUpNs1ABgjDkHlInIEuvQZuAw8Cxwh3XsDmCbdftZ4HZrNtCFQLNfqmjMUmMjWJ2TyM5jGgCUUmoyOQI877PA4yISDpwC7sQTPH4rIncBZ4APW+c+D1wPlAAd1rkTcsWSNO7bcYKG9h6SrYJxSimlJiagAGCM2Q8UDfHQ5iHONcDdE2xXP1csSedHL5/g9RO1bFk7rvFkpZRSAwTtSmB/q7ITSIkJ13EApZSaRLMiANhswqb8VN453TDTTVFKqTljVgQAgLzUGM61dOF0uWe6KUopNSfMmgCQnRiJ28C55q6ZbopSSs0JsyYAzEuMAqCyqXOGW6KUUnPDrAkA2VYAqNAAoJRSk2LWBADtASil1OSaNQEgMsxOamw4FU06BqCUUpNh1gQA8PQCNAWklFKTY3YFgIQoTQEppdQkmVUBIDspiorGTjzVJpRSSk3ErAoA8xKj6HS6aOrQPYKVUmqiZlUA0KmgSik1eTQAKKVUiJpdASBJ1wIopdRkmVUBICk6jMgwGxWNGgCUUmqiZlUAEBHmJUZR2awBQCmlJmpWBQDwjANoD0AppSZudgYALQehlFITNisDQF1bN11O10w3RSmlZrWAAoCIlIpIsYjsF5E91rFkEdkuIiesn0nWcRGR+0WkREQOiMj6yWywtypold/GMI3tPXT3Dg4IVc2dbPnJm5Q1dExmE5RSak4YSw/gCmPMWmNMkXX/HmCHMaYA2GHdB7gOKLD+bQUemKzGwuCy0B09vWz+wWv86OUTg859/Xgd75U18drx2slsglJKzQkTSQFtAR61bj8K3OR3/DHj8TaQKCJZE/g9/eRYawG8A8HP7Kukob1nyA3jD1e1AFBc3jxZv14ppeaMQAOAAV4Skb0istU6lmGMqbJunwMyrNvZQJnfc8utY5MiIz4SEc9qYGMMj/2tFIDDlS243P2LxB2q9Fz4iys0ACil1ECBBoCLjTHr8aR37haRS/0fNJ7ynGMq0SkiW0Vkj4jsqa0NPEUT7rCRERdJRVMne840cvRcK+fnJdPpdHGyts13ntttOFzZgk3geHWrDhorpdQAAQUAY0yF9bMG+CNwPlDtTe1YP2us0yuA+X5Pz7GODXzNB40xRcaYorS0tDE1el5iJJVNnTz2tzPERTr42g3LgP6pnjMNHbT3uLh8STq9bsMRKx2klFLKY9QAICIxIhLnvQ1cAxwEngXusE67A9hm3X4WuN2aDXQh0OyXKpoU8xKjOHaulRcOVvGhDfNZmZ1AVJi9X6rHm/659TxPLNI0kFJK9ecI4JwM4I8i4j3/N8aYF0RkN/BbEbkLOAN82Dr/eeB6oAToAO6c7EZnJ0VR394DwMc3LsBuE5bPi+dgvwDQgsMmXLYkjZSYcB0IVkqpAUYNAMaYU8CaIY7XA5uHOG6AuyeldcPwloW+pCCVhakxAKzKTuCp3WW43Aa7TThU2UJBRhwRDjurchK0B6CUUgPMupXAAItSYwG4Y2Oe79jK7AQ6nS5O1bZhjOFwZTMr5sUDnuBwvLqVzh4dCFZKKa9ZGQA25afw7D9v4qrlGb5jq7ITADhY2UxNazd1bT39AoDb9K0LUEopNUsDgIiwOiex37HFaTFEhtkoLm/xDQCvmOcJCt5zi8ubprehSikVxAIZBJ4VHHYby7M8A8HJMWEALMuKAyAjPoK0uAgO6DiAUkr5zMoewHBWZidwqLKZ4opm8lKiiYv0BAIRYVV2gs4EUkopP3MuALT3uHjteK0v/eO1KjuBk7VttHf3zlDrlFIquMypAOAdCO5yulluDQB7rc7RgWCllPI3pwJAQXosEQ7Pn7RiQADwBocDmgZSSilgjgUAh93GsizPhX9gCig9PpLM+EjeK9OZQEopBXNoFpDXJQWptHf3khYXMeixtfMT2a8BQCmlgDnWAwD4wlWF/PlfLhnysfULEjnb0EFdW/c0t0oppYLPnAsANpsQ7hj6z1qXmwTAvrPaC1BKqTkXAEayKjsBh01492zjTDdFKaVmXEgFgMgwO8vnxbNPA4BSSoVWAABYn5vEgfJmel3umW6KUkrNqJALAOtyE+nocXGsunWmm6KUUjMq5ALAeh0IVkopIAQDQE5SFKmxEToQrJQKeSEXAESEdbmJ7NcegFIqxIVcAABPGuhUXTuN1sbySikVigIOACJiF5F9IvKcdX+hiOwSkRIReUpEwq3jEdb9EuvxvKlp+vity/XsEKZlIZRSoWwsPYDPAUf87n8H+KExJh9oBO6yjt8FNFrHf2idF1RW5yRg1wVhY1bX1s3eM/qeKTVXBBQARCQHuAF42LovwJXA09YpjwI3Wbe3WPexHt9snR80osMdLM2M05lAY/TQ66e4/ZFdM90MpdQkCbQH8CPgy4B39VQK0GSM8W6vVQ5kW7ezgTIA6/Fm6/ygsmFBEu+ebaS50znose5eFy63mYFWBbe61h7ae1x097pmuilKqUkwagAQkfcBNcaYvZP5i0Vkq4jsEZE9tbW1k/nSAbn1vFw6elw88sbpfse7nC6uv+91rvrBa/z1+PS3K5g1d3oGzdu6dFtNpeaCQHoAm4AbRaQUeBJP6uc+IFFEvPsJ5AAV1u0KYD6A9XgCUD/wRY0xDxpjiowxRWlpaRP6I8Zj+bx4rluZyS/eOE1TR99soJ+9doqTte309Lq5/efv8JnH91LV3Dnt7QtGTR2e3lKrBgCl5oRRA4Ax5qvGmBxjTB5wK/CKMeajwE7gZuu0O4Bt1u1nrftYj79ijAnKfMrnriqgtbuXh1/39ALO1nfwv6+W8L7VWbzyr5fxpasL2XGkhi0/flNrBwFNVrqsrVsDgFJzwUTWAXwF+KKIlODJ8T9iHX8ESLGOfxG4Z2JNnDpLM+O5YXUWv3jzNI3tPXzzuUPYbcK/37CcCIedz24u4D+3rKSmtZszDR0z3dwZpz0ApeaWMW0JaYx5FXjVun0KOH+Ic7qAD01C26bF5zcX8HxxFVt/tYfdpY189bqlZCZE+h737jF8/Fwri9NiZ6qZM84YQ4v2AJSaU0JyJbC/gow43r96HrtLG8lPj+XOTQv7PZ6fHosIIV89tNPposdKg7V2DZ45pZSafebcpvDj8YWrCzlR08Y3blwxaDvJqHA7ucnRnKhum6HWBQdv+ge0B6DUXKEBAFiYGsNfPjf0RvIAhRlxId8D8A8AOgag1NwQ8imgQBRmxFJa1x7SC6CaOvumymoAUGpu0AAQgMKMOHrdhtN17TPdlBnT0umfAtIxAKXmAg0AASjMiAPgeAiPA3hTQOF2m64EVmqO0AAQgEVpMdhtwvFzoTsO4F0Elp0UpYPASs0RGgACEOGwszA1ZtBA8PdfOsYv3zw9zLPmlqYOJ2F2IS0ughbtASg1J2gACFBhRiwn/AJAQ3sPD7x6kodeP02QVrqYVM2dPSREhRMf6dAUkFJzhAaAABVmxHGmoYPOHs9MoD8fqKTXbaho6uTULBgcLm/s4Jaf/Y36tu5xPb+500lidBixEQ5NASk1R2gACFBhRhzGwMlaz0DwM/srSYuLAOCNE3Uz2bSA7Dxaw67TDRyoaB7X85s6nCRGhREb6dCVwErNERoAAuSdCXTsXCtn6zvYe6aROzflMT85itdnQQA4VNkCQG3r+HoATR2eHkBcZBht3b0hkfZSaq7TlcABykuJJtxu43hNK5VNnv0Bblwzj7KGTv70XiVOl5swe/DG04kGgOZOJ8uy4omNcOB0Gbp73USG2SeziUqpaRa8V6wg47DbWJQWw7FzrTyzv4Lz85LJSYrmkoJU2rp7ea8sePcXdrrcHLOmsNa0dI3rNZo6ekiICiMu0vOdQVcDKzX7aQAYgyWZcbx1sp6Tte3ctM6zBfJFi1MQIajTQCeq23yVPGvHMQjsdLlp73FZKSBPANCBYKVmPw0AY1CYEUdPr5swu3D9qkwAEqPDWZ2dwBslwRsADlV6Bn4z4iOoaRl7AGi2FoF5ZgGFAbovsFJzgQaAMfAOBF++JJ3E6HDf8YsLUtlf1kRLkM6OOVTZQnS4nfPykqkZxxiAtwxEQpRnGihAq9YDUmrW0wAwBmtyEogJt3PbBbn9jl+cn4bLbXj7ZP0MtWxkhytbWJYVT0Z8JLWt3WOewdNsVQJNjA7XMQCl5hANAGOQHh9J8X/8HVcsSe93fP2CRKLC7EE5DuB2Gw5XtbBiXjzpcRF0Ol1jzt97ewCJfoPAmgJSavbTaaBjZLPJoGMRDjsXLErmjZI6jDGIDD5nppxp6KCtu5cV8+J9u53VtHYTFxkW8GsMlQLSQWClZr9RewAiEiki74jIeyJySES+YR1fKCK7RKRERJ4SkXDreIR1v8R6PG9q/4TgcPXyDE7XtfO/r56c6ab04x0AXjEvgbRYz2b3Yx0I7jcI7EsB6RiAUrNdID2AbuBKY0ybiIQBb4jIX4AvAj80xjwpIj8F7gIesH42GmPyReRW4DvALVPU/qDxkfNy2VPayPdePEZCVBgfu3DBhF6vprWLB187BUBEmI2oMDs3b5hPZkLkmF7nUGULDptQkBHLmfoOYOxTQZs6nYhAXGQYdpsQ7rDRqj0ApWa9UQOA8YwYendCCbP+GeBK4Dbr+KPAf+AJAFus2wBPAz8WETFzvHaAzSZ89+bVNHc6+fq2gyREhfH+NfPG/Xrb9lXy8BuniQm30+Ny43QZqpq7+NYHVo3pdQ5VtlCYEUeEw066VbtorIvBmjt6iLcu/gBxEVoRVKm5IKBBYBGxi8h+oAbYDpwEmowx3qtAOZBt3c4GygCsx5uBlMlsdLAKs9v4yW3rKVqQxBd/u5+XDp0b92sdqGhmXkIkh755LSe+dT3Xrsjk5SPVuN2Bx1FjDIcrm1kxLx7w5PDD7bYxl4NosiqBesVGakVQpeaCgAKAMcZljFkL5ADnA0sn+otFZKuI7BGRPbW1tRN9uaARFW7n4TvOY3lWPJ/+9V5+u7tsXK9zsKKZldkJvvtXL8+guqWb4jFU86xp7aaurccXAEQ8G7qMOQBYlUC9YiMcOg1UqTlgTNNAjTFNwE5gI5AoIt4UUg5QYd2uAOYDWI8nAIMmyBtjHjTGFBljitLS0sbZ/OCUEBXGbz51IZvyU/ny7w/wk50lY5p739Ll5HRdO6v8AsCVS9Ox24Tth6sDfp2DVrBY4fc6aXERY14M1tzpJN4vAMTppjBKzQmBzAJKE5FE63YUcDVwBE8guNk67Q5gm3X7Wes+1uOvzPX8/1BiIhw8csd53LhmHt978Rj/76VjAT/3UIWncufKnL4Ld1JMOEULksYUAA5VtiACy7LifcfS4yKoaR3jGECns9/K59iIMB0EVmoOCKQHkAXsFJEDwG5guzHmOeArwBdFpARPjv8R6/xHgBTr+BeBeya/2bNDuMPGj25Zy4eLcvjJzpO8eqwmoOd5v7n79wDAkwY6Vu3ZjyAQhytbyEuJ8c3dB8aZAurplwKK001hlJoTRg0AxpgDxph1xpjVxpiVxphvWsdPGWPON8bkG2M+ZIzpto53WffzrcdPTfUfEcxsNuGbW1ayJCOOf/3dAer8pmBWNHXyuSf3+S74XsUVzWQlRJIaG9Hv+DXLPQXoXjoc2ODykXMtLMuK63csPS6Sxg4nPb3ugF7D7Ta+7SC94nQQWKk5QUtBTIPIMDv3fWQtLV1OvvL0AYwx7DpVz43/8wbb9lfys7/2j5EDB4C9clOiWZIRF1AaqK27lzP1HSzLjO93PD3eE1QCXQvQ2t2L23jGNbxirWmgIZjZU2pO0QAwTZZmxnPPtUvZcbSGzzz+Lh99eBcJ0WFcuTSdHUeqfZvNt3Y5OTVgANjf1csz2F3aQGN7z4i/z7sBjH/+HyDN6lUEmgZq8a0C9hsDiHTQ6zZ0OQPrRQSq1+XWoKLUNNIAMI3u3JTHZYVp/OXgOS4rTOOZuzfxyYsX0tHj8o0PeLduHCkAuA28cnTk8YQjVZ7XWTowBRQ/tsVg/nWAvOKmoCR0c4eTdd/czs4Ax0mUUhOnAWAaiQj3f2QdP/3Yeh66vYj4yDDOX5hMSkw4fy6uAvoGgIdKAYEnMGTER/D03vIR8/BHqlqIj3SQnRjV73h6nFUPKMAeQJOvFLT/GMDkbwpzoqaV1u5e3wwopdTU0wAwzRKiwrh2ZZavqqjDbuPalZnsOFJDZ4+L4opmMuMjSYuLGPL5Nptw18UL+dupei7/3qs8+c5ZXEOsDj5S1cLSrPhBlUlTYsMRCTwF5F8K2msqKoKWWjObxrNhjVJqfDQABIEbVmfR6XSx81gNxcMMAPvbeuli/viZi1iQEs09fyjmxh+/0W9aptttOHquleUD8v/gKVeRHB0+hh6AlQIaUAoCJndTmLP17QBUj3PTeqXU2GkACAIXLEwhNTacp3aXDVoBPJx1uUk8/emNfPeDqzlU2cKLh/pmBp1t6KCjxzVoCqiXZy1AYBda7yBwwoB1ADC5AeBMg/YAlJpuGgCCgN0mXLsyk9eO12IMrMoZ/M19KCLCh4pyyEqI5EW/wnNHz1kDwJlDv066tTVkIJo6eogOtxPhsPuOxXk3hp+CFNBYF6nNZq8eq6G4PPDaTkpNNg0AQeKGVX2lo0dLAfkTEa5ZnsHrJ2p9U0kPV7ViE1iSOUwPIDbwekBNHc5+3/6BMW8KU9bQMer0Tm8KaDx7Fs9Gxhi+9Nv3uG/H8ZluigphGgCCxPkLk0mNjSAzPtI3UydQf7ciky6nm9eOe6qqHqlqYWFqDJFh9iHPT4/3lIMIpLR0U+cQASAi8H2BS+vauex7O/v1UAZq6XLS2OEkIz6CHpfbN/A8l1U1d1Hf3kNVs455qJmjASBI2G3C125YyueuKhjzc89bmExCVJhv/4EjVS2DFoD5S4+LoNdtfAO8I2nu6F8GAjw1jiIctoBSQO+ebcRt4OAI0zu9tY3Oy0sGgmccoMvp4qG/nsLpmtwFb9A33VcHvdVM0gAQRD6wLoePnJ875ueF2W1sXpbOjqM1NLb3UN7YOWIA8E4xDaQqaFNnD4lR4YOOx0U6AqoIesDKcZ+sbRv2nFIr/XP+Qm8ACI6L4itHa/jW80d4s6Ru0l/7oLXgr66th+5e16S/frBo7Qq87pSafhoA5ohrlmfS3Onk0b+VAgw7Awj8FoONsjl8c6eT8sZO3+phf4FuCuP9pjtSAPDuVbxhQRIA1WPctH6qeANTaV37pL+2fwHA0f47zGY3/vhN/ueVEzPdDDUMDQBzxGWFaUSG2fj5G6eBwTWA/Hn3Bh5txs2jb5XS0ePi1vMG90riIsNoG2UQ2OU2vj0JSus66B0mlXK2voPU2HAWpsYAwdMDOFPnCUynpygAeHti5+ZoGsjpcnO6rn1Mu9ip6aUBYI6ICrdzaUEaLV29JEaHkRk//ECy98JT2dQ57Iyb9u5efv7maTYvTWf5vMHBJDaif0no5k4nXc7+qYyTtW10Ol1sXJRCj8tNeWPnkL/rTEM7C1JiiA53EBvhCJpvxGcaPBf+0wHuvxCompYualq7uWpZOgDn5uhAsLdgYaD7V6jppwFgDrlmhWe/gGWZg0tA+IuJcBAf6eD724+z9OsvcPF3XuHzT+7zTSMFeHzXGZo6nNx9Zf6QrxEb2ZcCcrsNH/jfN/nK7w/0O8c7x/2mtdnA8GmgM/UdLEiOBvpmKAUDb2rqdN3w6avxOFjpeV+uWpYBzN0AUNfmCQBljR1DlitRM08DwByyeWk6YXZhZfboC8l+ced5/PsNy/jERXmsy01i23uVbP3VHrqcLs/sl9dPsyk/hfW5SUM+P84vALx9qp5Tte28dKi6Xy+guKKZ6HA7Vy/3XOiGCgBdThfnWrrITbECwChbVh6pamHLj9/gbycHbTM9qbqcLqqau4hw2Kho7JzUgVrvjKgLFqUQHW6fsymg+nZPIHe6zJz9G2c7x+inqNkiKSacpz/tqRE0mg0LktmwINl3/9KCVP7t6QP806/3sik/ldrWbu67de2wz4/zSwE9tacMEeh0unj9RJ3vgl9c0cyKefEkxYSTGhvByZrBufTyxg6Mwdfm9LhI3itvGvJ37i5t4B9+uZvWrl62H65m4+KUUf/O8SqzSlNsXJzCq8dqKWvoID99+IH1sSiuaGZRqmerzsyEyDnbA6hv69uz4kx9+6DKtGrmaQ9gjlkzP7Hf5i2B+lDRfP77A6vYeayW//rzETYsSGLjouEvsLHWtpDNHU7+cvAct543n7hIh2/BV6/LzaHKvsJ2i9NihuwBeNMsC1I8A8DpcRFUt3QNGpt4+XA1H3t4F2mxEeSlRHOipnXMf+NYeEtTXF6YBsCp2skbCD7kV/AvMz5yzn479t/+VMcBgpMGAOVz2wW5fOPGFYTbbXz+qoIRxxFiI8JwuQ1P7TlLT6+bj16wgM3W7ma9Ljcna9vpcrp9he0Wp8cOGQC8F1r/MYAup7vfGoN3Tjfwj7/ey5LMOH736Y2sz02ipGZy8/IDnbGmgF6+JN1q5+QEgPq2biqbu3xpujndA2jvIcwuOGziK/angsuoAUBE5ovIThE5LCKHRORz1vFkEdkuIiesn0nWcRGR+0WkREQOiMj6qf4j1OS546I8DvzHNVxSkDbied6KoI++dYZlWfGsmBfP363IpLHDye7SRt/Uv9U53h5ALI0dThoGbGV5tr6d2AgHyTGeXktG/OA1CtsPn8NuEx7/5AWkxEaQnxFLVXNXwLWIxuNMfQdxkQ4WpESTHBM+aVNBvQvA/HsA1S1dAZXlGM0Dr57kf18tmfDrTJb6tm5SYiLISYqasz0AYww/eOkYhytn50ZGgfQAeoEvGWOWAxcCd4vIcuAeYIcxpgDYYd0HuA4osP5tBR6Y9FarKTVcDSF/3gBQ0dTJLUU5iAiXFqYR7rDx4qFzFJc3ER1uZ2FqLOBJAcHggeAzDR0sSIn29TaGWqVcXNHMsqx4305kBVYufip7AaX17eSlxCAiLEyNmbwAYAXGFfM8ASArIZJet6GufWIzn1q7nNy34zg/2n6iX+plJtW39ZASG05uSgxn52gP4FRdO/e/UsK2/RUz3ZRxGTUAGGOqjDHvWrdbgSNANrARQZFdAAAeD0lEQVQFeNQ67VHgJuv2FuAx4/E2kCgiWZPecjWjvAXhwu02tljTPGMiHFxakMr2w9UcqGhm5bwE7NbOZ4vTPIHg5ICL9pn6jn6D1t5Vyt6poG634VBFC6v8ZjYVpHteayoDgH+78lImNwDkJkf7Cux5ezzVzX0X7Y6eXu5+/N0x/c7ni6vocrrpcbl5YtfZSWnrRNW195ASG8GC5GhfSm2u8ZYJma01ncY0BiAiecA6YBeQYYypsh46B2RYt7OBMr+nlVvH1Bzi/TZ+9YoMkmL6Bp2vWZFJRVMn+8429StrnZ0YRYTD1q8H4HIbyhs7yE2O8R3r27Tec0E809BBa3dvv01y5idHE+6wjTkAGGMoa+jgt3vK+MFLx6hsGnphmtPlpqKp0xcAFqXFUN3STfsk7H9wsLK539+SleCZGVPV3NeWPaWN/Lm4iu2Hh6+gOtDTe8tZnBbDJQWp/HrXmSkpYDdW9W3dpMaEk5scTUtXL00dPaM/aZbxBoBgKWA4VgEHABGJBX4PfN4Y0y/hZTxTNsaUxBSRrSKyR0T21NbWjuWpKggsSIkmKTqMT1yU1+/4VcsysL7099vYxmYTFqXFctJvNk1lUydOl+nXA4iLcBAZZvN9o/KOJfgHE7tNWJQaw4kBAeCtkjrO/9bLQ17Yn9lXwaZvv8Il393Jl58+wP2vlHDVD17jwb+eHHSxrGjsxOU2vplJ3hIVEx0Ibu5wUtbQyQq/3kxGgifg+X+D9P7NJ6oDC3Clde3sLm3kgxtyuHNTHtUt3bxwMPDgMVX6UkCe/75n5tg4gMttfOtR5nQPQETC8Fz8HzfG/ME6XO1N7Vg/a6zjFcB8v6fnWMf6McY8aIwpMsYUpaWNPOCogk9GfCTvfv1qXwlnr+SYcN+xVdmJ/R5bnBbT71u7Ny/sHwBEhPS4SN83qoMVzYQ7bBRm9J+DX5ARN2gq6HPFVdS0dvP4rjP9jnc5XXzjT4eIjwrjP7es4KUvXMpf/+0KNi5K4b+fP8r77n+DEr/X8s5YybMCgPfnRNNA+631Dav93pfUmAgcNum3L8B7ZZ7zBga44fzh3XJsAn+/LofLC9NZkBLNo2+VTqitE9XR00un0+VJAXkDwBwbBzhU2UxLVy+psRFBU75krAKZBSTAI8ARY8wP/B56FrjDun0HsM3v+O3WbKALgWa/VJGaQ4abJnrHRXlcuCiZRakx/Y4vToulrLHDt1r4kFUSwftN2ysjvm81cHF5M8sy4wiz9/+oFqTHUt7YSUdPX1rG2x1/8p2yfit3t+2voLHDyb3vX8HHN+ZRmBFHbko0j3ziPB66vYhzLV1894VjvvO9+WrfGECq56d/VVBjzJh3Lttb2oBNYG1uXwCw2YSM+P5TQb09gJKatlF/h9tt+P27FWzKTyUzIRKbTbh9Yx57zjTO6HaT3kVgKVYKCPoW180Vb5Z4vv2/f00Wrd29/T6Ls0UgPYBNwMeBK0Vkv/XveuDbwNUicgK4yroP8DxwCigBHgI+M/nNVsHs+lVZPLl1IzZb/wCxOD0WYzyplOLyZn6w/TgbFiSRNaBwnbcHYIzhoN9iMn/51mt5F2iVNXRwpr6DzUvTqW/v4S/FnhSIMYZfvFnK0sw4LlyUPOh1rl6ewS3nzWfnsRrfFNXSug4iw2y+qqnR4Q4y4yM55RcAvv/ScS773qtjmv63u7SR5fPifQPoXpkJfYvBalq7qGruYmFqDG3dvaMuEnv7VD0VTZ3cvCHHd+xDRTlEh9v55Qz2ArwzkVJjI4gOd5AWFzHnBoLfOllHYUasb0bXbOwFBDIL6A1jjBhjVhtj1lr/njfG1BtjNhtjCowxVxljGqzzjTHmbmPMYmPMKmPMnqn/M9Rs4J0K+mZJPZ98bDcpMRH89GMbBgWKtLgIalu6OVPfQWtX/wFgr4Ezgd6wvv3fc91S8lKieexvpQC8faqBo+dauXNT3rA9lr9fn43TZXjWmsp3tqGdBckx/c5fmBrj6wGUNXTws7+epKyxgw/99C12Hq0Z8nX9OV1u9pc1UbRgcBDK9OsBeL+1//06z7yJgeMAr5+o5dof/ZVv/OkQb5+q57d7yoiLcPB3ViFAgPjIMG7ekMOf3qucsYFXXw8g1jNBwDMTaO70ALqcLt453cCm/FQy4geP48wWuhJYTZtF1pqAb/35MG1dvTzyiSLfvH9/6fERtHb38k5pA8CQPYAFKTE4bOIbB3ijpI7M+Ejy02P52IULePdsEwcrmvnFm6dJig7zTVUdytJMz0K237/rCQClA6amAuT5rQX4wfbj2ETYdvcm8lJjuOvR3fzq7TODXtff4coWOp0uivIGF9fz9gCMMRwob8YmcJM3AAwYB9i2v5JTte08vusstz74Ns/sr+R9a7IGrd34wLpselx9+0SPx6naNvaeaRzXc72F4FJiPf99c1Oi59RagHfPNtLd62bT4tS+xYuzcCaQBgA1baLC7b6CYP9z2zqWZg5dtdS7FuCVIzWE2wcPAINnX+K81BhOVLfhdhveKqljU34qIsKHNswnMszGd144ystHqrntgtxRF7d9cH0OxRXNHD3XwtmGDvIGjF8sSo2hscPJ26fqeWZ/BXduWsjqnER++48buWJJOl9/5iDbD1cP+/p7rAvpcD2Ajh4XLV29FFc0k58ey/xkzwrkkgED3XvPNHJpYRr7vn41D3x0PbdvXMA/XTa4ZPeanERSYsLZcWT03slwvvGnw3zs4V3jKlVR5zcGAJCbHM25lq5Be0bMVm+V1GO3CRcsSvalCrUHoNQovnB1IT+8ZS1XLs0Y9hxvl/r1E7UszYoj3DH0x7QgPZaSmjYOV7XQ2OHk4gJP8bqE6DBuWpvN6yfqEBE+duGCUdt149p5OGzCT3aepKfX7Ru49PJOBf23p98jLsLBP122GPAsfvvZxzewMDWG7790bNiSDntKG8hJiiIzYfBGPd5j55q7OFDe5Js9lZ8e2y8FVNfWzem6dorykoiJcHDdqiy+uWWlb5qlP5tNuGJpOq8eqxl2J7aRGGN4r7yJTqeL7754dMzPr2vrJjbC4Qu8C1KiMcZT/XUuePNkHatzEoiLDCMhKoxwh017AEqN5uYNOSOmY6CvB9De4xoy/eOVnx5LaX27Lwe/aXGq77GPb/Rc9K9bmelbbDWS1NgILl+SxnMHKoG+qZ9e3h5BWUMn/3R5PgnRYb7HHHYbn9tcwNFzrbxwaPD8e2MMu0sbB02Z9fIGgH1nG6lr6/HVTypIj+WE30ygvb5exNB7NAy0eWk6LV2940rjnG3ooKnDSW5yNH94t4L9ZUOX6B6Odw2Al3ex31xIA7V0OTlQ3uz7vImIZ+baFPcAmjucfPGp/RyvnrxKuBoAVNBJ9xsXGGoA2Cs/PRa3gd+8c5bCjFjS/WYTrZiXwP0fWcfXblgW8O/94PocvLMuB44B5CZHYxNP72Tg4jeA96+ZR356LD/cfnzQ7ldnGzqoa+v2bXo/kHf7Tm8KyT8ANHc6qbVm1Ow900i43TZiUPR3cUEqYXbhlQAGqQc6YA1Gf/fm1aTGRvDNPx0a07TX+vZuX/oH+t7PuTAQvOtUAy634aL8vnLp6XGRVA8xC2hg8cOJ+MvBKv6wr6Lfzn0TpQFABZ3E6DDCrXn/IwUAb1G4quYuNuWnDnr8xjXzAvr273XlsnQSosIIswvzBmxeEu6wcfcV+Xz771cTFT54PMFuEz5/VQEnatr4c3H/ZS+7Sz3fwIfrAXgHEV8vqcNhE5ZlecZGCqyxjxIrDbSntIFVOQkBFesDT7mOCxamsGNcAaCJcIeNDQuS+PLfLeHds0386UDgy3k8PYC+QJ4SE05MuH1OBIBXjlYTG+HoF9Az4iOoHrCT3VsldRT91/ZJqxT6zP4KFqbG+L4gTAYNACroiAhpcRHDDgB7LUqL8ZWduHiIADBWEQ47d27K49KCNF8RO39fumYJVyxNH/b516/MYklGHD96+Xi/vPveMw3ERzp8U1cHCnfYSI0Np6fXzZLMON8F3nv+iZo2upwuDla0BJz+8bpyaTolNW1jnoP/Xnkzy7PiCbPbuHlDDivmxfPt548E/O2zrq2HVL8UkIjMiaqgbrdhx5EaLi1MJcLRF4jT4yKpHdADePdsI24Dfy6uHPE1K5s6+dofi0csb17V3Mmu0w1sWTtvxH06xkoDgApKWQmRLBthABg8Zatzk6Ot2RiTsz3k568q5JFPnDeu59pswheuLuBUbTtP7O6rh7i7tJENC5IGrXfw5+0F+H+7S4uLID7SwYmaVoormulxuYdNIw1n8zJPwBpLGsjlNhysaGaN1RabTbj3/SuobO7ioddPjfp8t9vQ0N5Namz/Kb5zoSpocUUzNa3dXLWs/yQG79Rl/4KBx62e20h1mbqcLj796708vuusb2XxUJ7dX4kxcNMo42djpQFABaVvfWAV3/vQmlHPuyg/lSuXpg9aXTtTrlmeyfkLk/n6Mwf5j2cPUd3SRUlNG0XDpH+8sqyBYP/6SSLiqXlU3cYeK4001gCwICWGxWkxYwoAJ2vb6OhxsTqnry3nL0zmupWZPPDqyVGnOzZ1OnEb+o0BABRmxFJa30Fz59Rt5DPVdhypxiZwxZL+PcGMuMFrAY5Xt2K3CSdr2wdN5wXP5ID/u+2gb7zlxAiDu8/sr2TN/MRB05MnSgOACkpLMuNGTP94/fcHVvHQ7UXT0KLA2GzCr+46nzs35fHLt0q5/r7XgdFn7gzVA4C+qa57zzSwKDWmX149UJuXZfD2qXraAixn7b0gDWzLV69bhstt+N6Lx4Z6mk99W/9FYF6XFKbhchveOFEXaNODzvYjNRQtSO5XAh38d7LzBMdel5tTte3cuGYeIvhKk/h74p0yfrunnM9emU9OUhTHhyn+d7y6lSNVLdy0dt4k/zUaAJSadBEOO/e+fwU//0QRBogMs7FmfuKIz1mdk0BmfOSgoJefHkt9ew9vnawf87d/ryuXpuN0GV4PcFXwgfImYsLtLErrP2aRmxLNnRfn8fTe8hELzdUNKAPhtW5+IvGRDl49Nv7FaTOpoqmTI1UtvrSaP+8+FtWtfftY9LjcbMpPZX1u0qDpwfvONnLvswe5tDCNz19VSGFG3LA9gG37K7DbhPet1gCg1Kxx5dIMtn/hUrbdffGoM3duOS+Xt+65ctCYh3cmUEfP0GUkArFhQRLJMeE8VxzYLJ73yj0F+IYaCP/nK/JJiQnnP587POy0UG8ZiIFjAA67jUsK03jteO2YK6kGgx1HPNN0r1o+eBGjLwVk9QC8F/PCjFiuXZHJocoWXzXUli4nn31iHxnxkdx/61rsNqEgPZZTte2DFu0ZY9i2v5JN+alDlk2ZKA0ASk2hlNgIlmSOnsoChhwk9p85tGGIMhKBCLPbuGFVFi8frh5xpglAT6+bI5Utw/ZY4iLD+OI1hbxT2sDXnjnIjiPVg3L69QPKQPi7vDCNmtZuDlfNvk3UXz5Sw8LUGN/2pv7ioxxE+K0G9g4A56fH+gr1vWj1Au7ddoiq5i7uu3UdidGe96ggI44el3vQngl7zzRS3tg5Jekf0ACgVFDLSogkJtxOYnSYr5rqeNy0LpvuXjcvHhq+XhHAsXOt9LjcI841v6VoPjeszuLpPeXc9ege1n7zJb7w1H7f4/Vt3dgE38XN32VLPJs/vXpsdu0C2Nbdy9sn67lqiPQPWBsZxUf4BsiPV7cyPzmK6HAHuSnRLM+K54WD59i2v4I/7qvgs1fm90vpFWZYU34HpIH+XFxFhMPGNX7VXidTcEydUEoNSUQoyksmOSZ8QvO/1+cmkpsczTP7KvrtHTDQgQpPyYc1OcOPWTjsNn5y23q6nC72lzXx291l/GFfBZ+6ZBHL58VT195Dckz4kCmk9LhIVsyL59VjNdx9xeAidjOpp9fN9sPV1LV109zppK27l5ykKFZlJ3Cm3pPTHzj9019GXGS/AFCY3tfzu3ZlJj98+TjHzrWyPjeRfx7wt+dbPb3j1W1cu7Lv+N9O1nP+wuQpm+WmAUCpIPfwHUVMdOmPiLBl7Tx+srOEmpaufmUz/B0oayYpOoycpNFXUEeG2blwUQpLM+N4rriKJ3ef5ZtbVlLf1k1KzPD56suXpPHT107R3OkkISps2POm238+d7hfWe9wu40ev5x8QlTYiAPxGfGRHDnXgtPl5nRde7+Ch9euzOQH249jgB/dsg7HgB3uosMdzE+O6lfnp6G9h6PnWnn/mqlJ/4AGAKWC3sDtMMdry9ps/ueVEp59r5JPXrJoyHPeK29iVU7imHobidHhXLcykz/uq+D/XL9sUCG4gS5fks5Pdp7kjRN13LA6y3e8qrmTt0/V8/bJBmw24b8/sHJSV72CZ6/irz9ziA8X5fRbPFhc3syvd53hoxfk8oWrC4mP9JQEqWzuori8meKKJpZmxg+6cPtLi4vgtePdlNa143QZX1oHPGM5H7swl0sL0oas3gpQmB7Xr/rrO6c9+2FcsHB8Yz+B0ACgVIjIT49lVXYC2/YPHQBeOFjF8epWrl059nzzLefNZ9v+Sp4vrqK+vWfEgnX+00FvWJ3Fu2cbuXfbId9eyFFhdjqdLq5als7mEVIu4/GbXWf5/bvlbD98jj/evYnFabG43YZ/33aQlJgIvnLdUuIj+3ol2YlRZCdGBfSeZMRH0tbdyz6rcqr/lF4R4b9uWjXi8/MzYvnriVp6XW4cdhtvn6onMszWb0HeZNNBYKVCyJa18yiuaPZtpen1fHEVd/9mH2vnJ3LXxQvH/LobF6WQlxLNk++UUdfWPeQMIC/vdNBXj9fytT8W88EH3qK2tZuvXb+M5z57MfvvvZqcpCh+vLNkUqeLdva4+Olrp1iTk0CY3cYnH91DU0cPT+4u472yJr52Q/+L/1h597F4s6QOEYacLTSSwvQ4nC5DqVUw7+1T9RQtSB6xHMpEaQBQKoTcuGYeNoFH3jhNiVVk7rkDlXz2iX2sm5/IY3ddQNw4LoIiwi3n5fJOaQOtXb39CsEN5fLCNGpbu3ninbP8w6aFvPyly/jUpYtYmZ1AhMPOpy9bzL6zTfzt5PD1cQZyutzUtHRxpKqFN0vqfBvTe/3mnbPUtXXztRuW87OPb6CisZOtj+3luy8e5fyFyROus+Pdx+LNkjoWJEcPWTV2JN4ew4nqVhqt/P+Fi6Yu/QMBpIBE5OfA+4AaY8xK61gy8BSQB5QCHzbGNIonYXcfcD3QAXzCGPPu1DRdKTVW6fGRXFaYxhPvnOWJd876jp+fl8zP7zxvQrNNPrghm++/dIxetxm1ZMX1q7IoqW3j/avnDZkuunlDDvfvOMGPd5ZwUQCVXv92sp6tv9pDa1dfuYvU2HAe/+SFLMmMo8vp4qevnWTjohTOt3Lq37l5FV946j3sNuE/t0x8vMHbA6hr62Fd7tgX7eWnxyLimQnkbctkFTkcTiD/tX8J/Bh4zO/YPcAOY8y3ReQe6/5XgOuAAuvfBcAD1k+lVJD4349u4FBlM2WNHZQ1dOJyG7ZeuoiYCU41TI+LZPOydF48VD1iCgg8W2l+9brhN+uJDLPzqUsW8a3nj/Du2UbWj3BB3VPawF2P7iY7MYovX5tHSkw44XYbX3ummFsf/Bu//uQFvHO6gdrWbu6/dZ3veR9Yl0NnjxuHTQJerDcS/5lV/gPAgYoKtzM/KZrjNa00dfZY+f/Jq/0/lFH/ixtj/ioieQMObwEut24/CryKJwBsAR4znsTd2yKSKCJZxpjAd5JQSk2pqHA7RXnJo1YoHY87Nubx8pGaQXWExuO2C3L5yasl/OSVkmFLdO8va+ITv9hNZnwkj3/qAl8aBjzfqG976G1ue2gXYXbh/IXJbFzc/xv1bRfkTridXvGRntXA3b3ugAoZDqUwI5YT1a2cqm1nw4KkfnsOTIXxjgFk+F3UzwHeofpsoMzvvHLrmFIqBFyUn8qBe6/xLWyaiJgIB/+waSE7jtbw6Ful9PT2zcl3uw0vHTrH7Y/sIjkmnN986sJ+F3/w7OP81D9uJDbCQV1bD5/fXDDhNo3Eszewpw0F6eMLAPnpcZyqbefouRYuWDi16R+YhGmgxhgjImMeqheRrcBWgNzcyYvCSqmZNdFUkr9PbMrjjZI67n32EA+/cYp/ubKAHpebR944zanadhamxvCru84nM2HohW3zk6P5w2cuYt/ZpkHf/qdCelwE5Y0dLBpn2Y7CjFh6rT2lL5zi/D+MPwBUe1M7IpIFeOu7VgDz/c7LsY4NYox5EHgQoKioaPaVBlRKTbn4yDCe2nohrx2v5fsvHeffnj4AeMpn3/+RdVy/MnPExVngmZ8/nrUN45GfHktXryvgfZsH8qaOIhw21syf2vw/jD8APAvcAXzb+rnN7/g/i8iTeAZ/mzX/r5SaCBHh8iXpXFaYxusn6jxjGAuSJn2V8GT4v+9fjrN3/N9nF6d5ZgKtz536/D8ENg30CTwDvqkiUg7ci+fC/1sRuQs4A3zYOv15PFNAS/BMA71zCtqslApBIsKlhWkz3YwRRYc7YOQJUCOKCrdz50ULfVNVp5oEw8YMRUVFZs+ePTPdDKWUmlVEZK8xZtx7oupKYKWUClEaAJRSKkRpAFBKqRClAUAppUKUBgCllApRGgCUUipEaQBQSqkQpQFAKaVClAYApZQKURoAlFIqRGkAUEqpEKUBQCmlQpQGAKWUClEaAJRSKkRpAFBKqRClAUAppUKUBgCllApRGgCUUipEaQBQSqkQpQFAKaVC1JQEABG5VkSOiUiJiNwzFb9DKaXUxEx6ABARO/AT4DpgOfAREVk+2b9HKaXUxExFD+B8oMQYc8oY0wM8CWyZgt+jlFJqAqYiAGQDZX73y61jSimlgohjpn6xiGwFtlp3u0Xk4Ey1JcikAnUz3Yggoe9FH30v+uh70WfJRJ48FQGgApjvdz/HOtaPMeZB4EEAEdljjCmagrbMOvpe9NH3oo++F330vegjInsm8vypSAHtBgpEZKGIhAO3As9Owe9RSik1AZPeAzDG9IrIPwMvAnbg58aYQ5P9e5RSSk3MlIwBGGOeB54fw1MenIp2zFL6XvTR96KPvhd99L3oM6H3Qowxk9UQpZRSs4iWglBKqRA14wEgVMtGiMh8EdkpIodF5JCIfM46niwi20XkhPUzaabbOl1ExC4i+0TkOev+QhHZZX02nrImFcx5IpIoIk+LyFEROSIiG0P1cyEiX7D+/zgoIk+ISGQofS5E5OciUuM/TX64z4J43G+9LwdEZP1orz+jASDEy0b0Al8yxiwHLgTutv72e4AdxpgCYId1P1R8Djjid/87wA+NMflAI3DXjLRq+t0HvGCMWQqswfOehNznQkSygX8BiowxK/FMKrmV0Ppc/BK4dsCx4T4L1wEF1r+twAOjvfhM9wBCtmyEMabKGPOudbsVz//k2Xj+/ket0x4FbpqZFk4vEckBbgAetu4LcCXwtHVKSLwXIpIAXAo8AmCM6THGNBGinws8E1WiRMQBRANVhNDnwhjzV6BhwOHhPgtbgMeMx9tAoohkjfT6Mx0AtGwEICJ5wDpgF5BhjKmyHjoHZMxQs6bbj4AvA27rfgrQZIzpte6HymdjIVAL/MJKhz0sIjGE4OfCGFMB/D/gLJ4LfzOwl9D8XPgb7rMw5uvpTAeAkCciscDvgc8bY1r8HzOeKVpzfpqWiLwPqDHG7J3ptgQBB7AeeMAYsw5oZ0C6J4Q+F0l4vtUuBOYBMQxOh4S0iX4WZjoABFQ2Yq4SkTA8F//HjTF/sA5Xe7tt1s+amWrfNNoE3CgipXjSgFfiyYMnWl1/CJ3PRjlQbozZZd1/Gk9ACMXPxVXAaWNMrTHGCfwBz2clFD8X/ob7LIz5ejrTASBky0ZYOe5HgCPGmB/4PfQscId1+w5g23S3bboZY75qjMkxxuTh+Qy8Yoz5KLATuNk6LVTei3NAmYh4i3xtBg4Tgp8LPKmfC0Uk2vr/xftehNznYoDhPgvPArdbs4EuBJr9UkVDM8bM6D/geuA4cBL42ky3Zxr/7ovxdN0OAPutf9fjyX3vAE4ALwPJM93WaX5fLgees24vAt4BSoDfAREz3b5peg/WAnusz8YzQFKofi6AbwBHgYPAr4CIUPpcAE/gGf9w4ukd3jXcZwEQPLMqTwLFeGZPjfj6uhJYKaVC1EyngJRSSs0QDQBKKRWiNAAopVSI0gCglFIhSgOAUkqFKA0ASikVojQAKKVUiNIAoJRSIer/A6SyY5LrPWxXAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss on validation set: 167.73892472702437\n",
      "abcd -> dcba\n",
      "abcdabcdabcdabcd -> dcbbcdadcaaddba\n"
     ]
    }
   ],
   "source": [
    "ENC_RNN_NUM_OF_LAYERS = 1\n",
    "DEC_RNN_NUM_OF_LAYERS = 1\n",
    "EMBEDDINGS_SIZE = 4\n",
    "ENC_STATE_SIZE = 64\n",
    "DEC_STATE_SIZE = 64\n",
    "\n",
    "encoder_decoder = EncoderDecoderNetwork(\n",
    "    ENC_RNN_NUM_OF_LAYERS, DEC_RNN_NUM_OF_LAYERS, EMBEDDINGS_SIZE, ENC_STATE_SIZE, DEC_STATE_SIZE)\n",
    "train(encoder_decoder, train_set, val_set)\n",
    "print('abcd', '->', encoder_decoder.generate('abcd'))\n",
    "print('abcdabcdabcdabcd', '->', encoder_decoder.generate('abcdabcdabcdabcd'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The loss curve indicates this model is doing much better than the hepless SimpleRNN.  It is \"learning\" something.\n",
    "But it hits a plateau and remains unstable after the 10th iteration.\n",
    "\n",
    "Since the encoder decoder uses a fixed length vector to represent variable size input strings it can't properly represent long input strings. \n",
    "\n",
    "Lets test it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matches for short strings 99\n",
      "Matches for medium strings 89\n",
      "Matches for long strings 27\n"
     ]
    }
   ],
   "source": [
    "short_strings = [sample_model(1, 5) for _ in range(100)]\n",
    "medium_strings = [sample_model(5, 10) for _ in range(100)]\n",
    "long_strings = [sample_model(10, 15) for _ in range(100)]\n",
    "\n",
    "def count_matches(network, val_set):\n",
    "    matches = [network.generate(input_string)==output_string for input_string, output_string in val_set]\n",
    "    return matches.count(True)\n",
    "\n",
    "print('Matches for short strings', count_matches(encoder_decoder, short_strings))\n",
    "print('Matches for medium strings', count_matches(encoder_decoder, medium_strings))\n",
    "print('Matches for long strings', count_matches(encoder_decoder, long_strings))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Overall, the encoder-decoder model does improve over the Simple RNN but hits a plateau because it has limited memory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attention Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The *attention model* was introduced to address the limitation we just observed:\n",
    "* How does the decoder know which part of the encoding is relevant at each step of the generation.\n",
    "* How can we overcome the limited memory of the encoder so that we can \"remember\" more of the encoding process than a single fixed size vector.\n",
    "\n",
    "The attention model comes between the encoder and the decoder and helps the decoder to pick only the encoded inputs that are important for each step of the decoding process.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/att.jpg\" width=\"50%\" height=\"50%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each encoded input from the encoder RNN, the attention mechanism calculates its importance:\n",
    "\n",
    "$importance_{ij} =V*tanh(encodedInput_iW_1+decoderstate_jW_2)$<br>\n",
    "$importance_{ij}$ is the importance of encoded vector $i$ at decoding step $j$<br>\n",
    "$W_1$, $W_2$ and $V$ are learned parameters <br>\n",
    "\n",
    "Once we calculate the importance of each encoded vector, we normalize the vectors with softmax and multiply each encoded vector by its weight to obtain a \"time dependent\" input encoding which is fed to each step of the decoder RNN.\n",
    "\n",
    "Note that in this model, the attention mechanism computes a fixed-size vector that encodes the whole input sequence based on the sequence of *all* the outputs generated by the encoder (as opposed to the encoder-decoder model above which was looking ONLY at the last state generated by the encoder for all the slices of the decoder).\n",
    "\n",
    "We prepare a new class that refines the EncoderDecoderNetwork.  AttentionNetwork adds the units of the attention model ($W_1, W_2 and V$) in its constructor. \n",
    "\n",
    "The method $attend(input_vectors, state)$ computes the weighted representation of the whole input string for each slide of the decoder.\n",
    "\n",
    "$get\\_loss(input\\_string, output\\_string)$ and $generate(input\\_string)$ are adapted to introduce the call to $attend()$ in the overall choreography."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AttentionNetwork(EncoderDecoderNetwork):\n",
    "    def __init__(self, enc_layers, dec_layers, embeddings_size, enc_state_size, dec_state_size):\n",
    "        EncoderDecoderNetwork.__init__(self, enc_layers, dec_layers, embeddings_size, enc_state_size, dec_state_size)\n",
    "\n",
    "        # attention weights\n",
    "        self.attention_w1 = self.model.add_parameters((enc_state_size, enc_state_size))\n",
    "        self.attention_w2 = self.model.add_parameters((enc_state_size, dec_state_size))\n",
    "        self.attention_v = self.model.add_parameters((1, enc_state_size))\n",
    "\n",
    "        self.enc_state_size = enc_state_size\n",
    "\n",
    "    def _attend(self, input_vectors, state):\n",
    "        attention_weights = []\n",
    "\n",
    "        w2dt = self.attention_w2 * state.h()[-1]\n",
    "        for input_vector in input_vectors:\n",
    "            attention_weight = self.attention_v * dy.tanh(self.attention_w1  * input_vector + w2dt)\n",
    "            attention_weights.append(attention_weight)\n",
    "        attention_weights = dy.softmax(dy.concatenate(attention_weights))\n",
    "\n",
    "        output_vectors = dy.esum(\n",
    "            [vector * attention_weight \n",
    "             for vector, attention_weight \n",
    "             in zip(input_vectors, attention_weights)])\n",
    "        return output_vectors\n",
    "\n",
    "    def __call__(self, input_string):\n",
    "        input_string = self._pre_process(input_string)\n",
    "        \n",
    "        embedded_string = self._embed_string(input_string)\n",
    "        encoded_string = self._encode_string(embedded_string)\n",
    "\n",
    "        rnn_state = self.DEC_RNN.initial_state().add_input(dy.vecInput(self.enc_state_size))\n",
    "\n",
    "        probs = []\n",
    "        for _ in range(len(input_string)):\n",
    "            attended_encoding = self._attend(encoded_string, rnn_state)\n",
    "            rnn_state = rnn_state.add_input(attended_encoding)\n",
    "            p = self._get_probs(rnn_state.output())\n",
    "            probs.append(p)\n",
    "        return probs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ready to try it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a95572745e9c4a6f9e66f200f31d4953",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=18000), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztvXl8XHd97/3+zb5otK+W5H13EhzHJM6+AwFuk7aUC5SSctOmPKQtvelz2/D0oS3tcwvdoOUFT9pASgNlKU2BBBogwTgkgTjBThwn3h1vsqx9HUmzz+/+cc4ZzaoZyWNpJH3fr5deM3PmnJkz4/Hvc7670lojCIIgLD9sC30CgiAIwsIgAiAIgrBMEQEQBEFYpogACIIgLFNEAARBEJYpIgCCIAjLFBEAQRCEZYoIgCAIwjJFBEAQBGGZ4ljoEwCw+2p0e8dKmqvdC30qgiAIi4b9+/cPaq2b5np8RQiAp66FX//01/nUr1y+0KciCIKwaFBKnb2Y4yvCBeSw2+gdCy30aQiCICwrKkIAXHZFz1h4oU9DEARhWVERAuCw2+gbFwEQBEGYTypCAJx2GyNTMcKxxEKfiiAIwrKhQgRAAdArbiBBEIR5o0IEwDgNiQMIgiDMHxUlABIHEARBmD8qRAAMF5BYAIIgCPNHRQiATSkCHofUAgiCIMwjRQVAKbVJKXUg7W9cKfUHSql6pdQzSqkT5m2dub9SSn1OKXVSKXVQKbWjlBNpq/HQKy4gQRCEeaOoAGitj2mtt2uttwNXAVPAd4CHgN1a6w3AbvMxwF3ABvPvfuDhUk6kpdojWUCCIAjzyGxdQLcDb2qtzwJ3A4+Z2x8D7jHv3w18RRvsBWqVUm3FXritxiMxAEEQhHlktgLwPuAb5v0WrXWPeb8XaDHvtwNdacecN7fNSGuNl4GJCLFEcpanJAiCIMyFkgVAKeUCfgn4j+zntNYa0LN5Y6XU/UqpfUqpfQMDA7RWe9AaBoKR2byMIAiCMEdmYwHcBbyite4zH/dZrh3ztt/c3g10ph3XYW7LQGv9iNZ6p9Z6Z1NTE201HkBSQQVBEOaL2QjA+5l2/wA8Cdxr3r8XeCJt+4fMbKBdwFiaq6ggraYASCBYEARhfihpIIxSyg/cCfxO2uZPA99SSt0HnAXea25/CngncBIjY+jDpbzHtAUgtQCCIAjzQUkCoLWeBBqytg1hZAVl76uBB2Z7IjVeJ26HtIUWBEGYLyqiEhhAKSWpoIIgCPNIxQgAGHGAUmIA4ViCx35+RlJGBUEQLoLKEoDq0tpB/OCNHv7syUPsPtJfdF9BEAQhP5UlADVe+sbDJJMzlxS81jUGwN5TQ/NxWoIgCEuSihKAthoPsYRmaDI6434Hz48C8PM3B+fjtARBEJYkFSUAqxp8ABy6MFZwn1giyaEL4/hcdo73TUjlsCAIwhypKAHYtbYBv8vOD9/oLbjP8b4gkXiSD1y9EoAXxQ0kCIIwJypKADxOO7dvaeHpw33EC2T4HDxvWAfvv2YlAY+DF8UNJAiCMCcqSgAA3nl5K8OTUV4+PZz3+YPnR6nxOlnb6GfX2gZ+/qZYAIIgCHOh4gTg5o3NeJ12nnojf/ug17rGuKKjBqUU161r4OzQFOdHpub5LAVBEBY/FScAXped2zY388M3+khkpYOGYwmO9QW5oqMGgOvWNQLwolgBgiAIs6biBADgrstbGZyIsO9MphvocM84iaTmio5aADa2VNHgd4kACIIgzIGKFIBbNzXjdtj4QVY20MEuI///LaYAKKW4dp0RBzB60EEkniAYjs3vCQuCICxCKlIA/G4Ht2xq4gdv9GRUBR88P0ZzwJ2aHQCGG6h3PMwb3eP880/f5PpP7+Ed//B8jvtIEARByKQiBQDgnZe30Tce4dWukdS2186Pptw/FtetM7pU3/P//4xP/eAoNV4H3aMh9p8dYTZ0DU+lrAhBEITlQMUKwG2bm/E4bfzpE4foGw8TDMc4NTjJW8wAsMWqBh+3bGri1k3NfPeB6/nuA9fjstt4+lDhYrJsTg1McOPf7GHvqfypp4IgCEuRihWAgMfJwx+8itODk/zyF37G4/vPozVc0ZlpASil+NcPX82X7t3J9s5aAh4n169v4OnDfSVf0Z8bnjJvJ8v+OQRBECqVihUAMILB3/qda0lozSe/dxiAK9prihwFb9vWyrnhKY72Bkt6n6EJo/nc4MTMTegEQRCWEhUtAACXtdfw3QeuZ3NrgMvaq6nzu4oec/uWZpSCpw/1lfQew2b30eEiXUgFQRCWEiXNBF5o2mq8PPX7NxKJlzYBrDngYcfKOp4+3MvH7thQdP8hEQBBEJYhJVkASqlapdTjSqmjSqkjSqlrlVL1SqlnlFInzNs6c1+llPqcUuqkUuqgUmpHWU7UpvC67CXv//ZtLRy6ME7XcPE2EUMTRkvpwQlpLS0IwvKhVBfQPwI/1FpvBt4CHAEeAnZrrTcAu83HAHcBG8y/+4GHy3rGJXLn1lYAnjlc3A0kLiBBEJYjRQVAKVUD3AQ8CqC1jmqtR4G7gcfM3R4D7jHv3w18RRvsBWqVUm1lP/MirGn0s7GliqcPF08HtVxAQxIEFgRhGVGKBbAGGAC+rJR6VSn1JaWUH2jRWlstO3uBFvN+O9CVdvx5c9u88/Ztrbx8erjolf3QpOH6GZ6MSjGYIAjLhlIEwAHsAB7WWl8JTDLt7gFAG6vmrFZOpdT9Sql9Sql9AwMDszm0ZG7d3ExSw0tFpoYNT0Sx2xTRRJKJSPySnIsgCEKlUYoAnAfOa61fMh8/jiEIfZZrx7ztN5/vBjrTju8wt2WgtX5Ea71Ta72zqalpruc/I1vbqrEpONIzXnCfcCzBZDTBanMesbiBBEFYLhQVAK11L9CllNpkbrodOAw8CdxrbrsXeMK8/yTwITMbaBcwluYqmlc8TjtrGv0c7ilcEGb5/ze2BDIeLxa01pzsn1jo0xAEYRFSahbQ7wFfU0odBLYDfwV8GrhTKXUCuMN8DPAUcAo4CXwR+GhZz3iWbF1RM6MFMGxe8W9orjIeLzIBeP7EIHd85qclpbsKgiCkU1IhmNb6ALAzz1O359lXAw9c5HmVjS1tAb732gXGpmLU+Jw5z1sB4A2WBbDIagEGgtM1DJ31vgU+G0EQFhMV3wriYtnaVg3Akd78VoDl89/QYlgAi80FFIolMm4FQRBKZdkIwOEL+QXAcvmsqPXid9kXnQsobC78YREAQRBmyZIXgKaAm8YqV8E4wNBkFJfdRsDtoL7KtehcQNbCH4qW1idJEATBYskLgFKKLW3VHC4kABMR6v0ulFLU+92L1gUkFoAgCLNlyQsAGG6gE30TxBK5V8nDk1HqzRbTjX7XonMBWVf+EgMQBGG2LAsB2NJWTTSR5M2B3Hz5ockoDVWGANT7XYuuECwcFwtAEIS5sWwEAPJXBA9NRmgwLYD6Ktei6wcUjloxABEAQRBmx7IQgLVNflwOG0fyVAQPT0Sp97sBaPS7iSaSBBdRPyBJAxUEYa4sCwFw2m1sbKnKSQW1+gClu4Bgujp4MSACIAjCXFkWAgBGIPhIz3iGe8fK+El3AaVvXwxM1wFIGqggCLNj2QjAlrZqhiaj9Aen8/ytK/3pLCDDFbSYagFC5sIvQWBBEGbLshGAVEVwWiDY6gPUUGUs/JYFsJhSQSUILAjCXFk2ArA5T0sIK+XTcgFZt4vJBSQxAEEQ5sqyEYAar5OV9T5e6xpNbbOu9K0rf4/Tjt9lX1S1AGERAEEQ5siyEQCAq9fU8/KZYZJJIxCc3gfIwqgFWEwxACkEEwRhbiwrAdi1toHRqRjH+ox6gPQ+QBaLrR/QdDM4EQBBEGbHshKAa9bUA7DXHBKf3gfIonERtYOIJ5LEEoY1Y7WEEARBKJVlJQCd9T466rwpAUjvA2RRv4gawoXj07n/0g5aEITZsqwEAODatQ28dNqIA6T3AbJYTP2ALLePy2GTGIAgCLNm2QlAehwgvQ+QxWLqB2Qt+nU+J6FYYlGIliAIlUNJAqCUOqOUel0pdUAptc/cVq+UekYpdcK8rTO3K6XU55RSJ5VSB5VSOy7lB5gt16w14gDPHhvI6ANksZj6AYVSAuAikdSpeIAgCEIpzMYCuFVrvV1rvdN8/BCwW2u9AdhtPga4C9hg/t0PPFyuky0HHXU+Ouu9PPV6D0BeFxAsjmIwywKwREtqAQRBmA0X4wK6G3jMvP8YcE/a9q9og71ArVKq7SLep+zsWtPA691jAHmygBZPPyArBlDnMz5DRARAEIRZUKoAaOBppdR+pdT95rYWrXWPeb8XaDHvtwNdaceeN7dVDLvWNqTuW32ALBZTP6CUC8jvzHi80CSSmmA4ttCnIQhCEUoVgBu01jsw3DsPKKVuSn9SG9HHWTmglVL3K6X2KaX2DQwMzObQi8aKA0CuC2gx9QNKuYB8leUC+vpLZ7nlb58lkZSYhCBUMiUJgNa627ztB74DXA30Wa4d87bf3L0b6Ew7vMPclv2aj2itd2qtdzY1Nc39E8wBKw4A01f8FoupH5A1A6DOigFUSDXw+dEQQ5NRJhZBJpUgLGeKCoBSyq+UClj3gbcBbwBPAveau90LPGHefxL4kJkNtAsYS3MVVQzXrm3A48zsA2TRUOVmcDHEAGKZMYBKsQCsFtXiBhKEyiZ39culBfiO2S/HAXxda/1DpdQvgG8ppe4DzgLvNfd/CngncBKYAj5c9rMuA3/4tk3cs709ow+QRVuNh+7R0AKc1eywrvhrfUYMoFKKwaZSAiAWgCBUMkUFQGt9CnhLnu1DwO15tmvggbKc3SWkpdpDS7Un73Od9T6ePzG/cYm5EMpOA62QdhDWeYkACEJls+wqgUuhs85H33ikYq6oCxGJJVDKmHUAlWMBhGPiAhKExYAIQB6sAHGlu4FCsQQehx2vy556XAmIBSAIiwMRgDx01vsA6BqeWuAzmZlQLIHXZcfrNASgUiyAkASBBWFRIAKQh846UwBGKtwCiCbxOu14TAGolDRQKwg8LhaAIFQ0IgB5aA64cTlsnK9wCyAcT+Bx2nDabThsqmJcQGFxAQnCokAEIA82m6Kj1kvXSKYADE9GufFvfsJzxysjQygcTaSu/r1Oe8UIQEiCwIKwKBABKEBHvY+u4UwX0CtnR+gaDvH//dfhimhzEIolUv5/j8tegTEAsQAEoZIRAShAZ12uBXC4ZxyA430TfPfVnO4W844VBAbDArBaQyw01nmIBSAIlY0IQAE6632MTsUyFrHDF8ZZ3eDj8vYaPvPMcSILPIg9HEtmuoAqIAgcTySJJiwBEAtAECoZEYACpDKB0txAh3vG2dZewx+9YxPdoyG+tvfcQp0eYARbPWkuoEqIAaSfgwiAIFQ2IgAFsIrBLDfQeDjGueEptrZVc+OGJq5f38Dn95xc0I6XoWgCr9P4J/Q6bRUoAOICEoRKRgSgANMWgCEAR3uCAGxdUQ3AH719M8OTUR59/vTCnCBZQWBnZQSBw2Y/omqPQ+oABKHCEQEoQK3PSZXbwXmzGOzwBWOE5LY2QwDe0lnLjRsa+c6r5xfsHMOxBB5XZcUALAugpdrDRCReEdlSgiDkRwSgAEopOuq8KQvgcM84jVUumgLTIyTv3NrCmaEpTg9Ozvv5JZOaSDyJx5GWBbTAQWmYFoDmauN7kqEwglC5iADMQGe9LxUDONwzzpa26oz5AbduagZgz9H+vMdfSqzF3koD9bjsFdEO2rJCmgNGq22JAwhC5SICMAOddUYxWCyR5HjvBFtN90/q+Xof65ur2HNs/gXAWmi9zvQ6gEqwAIwr/mbTUpJMIEGoXEQAZqCz3ksoluDl08NEE8lUADidWzc18dKpYSbn2dURjhtX+96sVhDGPJ6Fw7JCmkQABKHiEQGYASsT6EeHegFyLAAw3EDRRJKfvzk0r+dmWQBuKw3UZSeR1MQSCywAaUFgEBeQIFQyIgAzYM0FePpQH26HjTWN/px9dq6up8rt4CfzHAew3D3paaCw8ENhUkFgsQAEoeIRAZiBjjqjGKx3PMzm1gAOe+7X5XLYuGF9I88e659X94u10KaCwKYlsNBxgLAVBBYLQBAqnpIFQCllV0q9qpT6vvl4jVLqJaXUSaXUvyulXOZ2t/n4pPn86ktz6pcev9tBgzlwPZ//3+LWzU30jIU51hecr1PLsQAqZSpYtgUgxWCCULnMxgL4GHAk7fFfA5/VWq8HRoD7zO33ASPm9s+a+y1aOkw3UD7/v8UtZjrofLqBrBiAJ0sAFtoFNBVN4LLb8LnsOO1KXECCUMGUJABKqQ7gXcCXzMcKuA143NzlMeAe8/7d5mPM529X6cnzi4xO0w00kwXQUu1h24pqnj06f4NirIU+vRkcLPxYSKNBnQ2lFAGPU1xAglDBlGoB/APwR4BVadQAjGqtrcu780C7eb8d6AIwnx8z91+UrG3047ApNrUWFgCAWzY1sf/cyLwtwOGsGEClWACh6PSMgoDHIRaAIFQwRQVAKfVuoF9rvb+cb6yUul8ptU8ptW9goDJGLObjvhvW8q2PXEuV2zHjfhuaAySSmu7R+Rkkbw1dqcQYgM9lfFeGAIgFIAiVSikWwPXALymlzgDfxHD9/CNQq5SyVsUOwBqR1Q10ApjP1wA5SfJa60e01ju11jubmpou6kNcSmp8TnasrCu6X7vpKpovAZh2AU3XAQAL3g4ilDajIOB2igUgCBVMUQHQWn9ca92htV4NvA/4idb614E9wHvM3e4FnjDvP2k+xnz+J3qhy1PngRW1hgBcmC8BsILAZjM463ahLYBwbHpGQcDjYFwsAEGoWC6mDuCPgQeVUicxfPyPmtsfBRrM7Q8CD13cKS4OWgJu7DZF98h8uYASuB02bDYjvu5xGf+UCx0DmEqLAVR7xQIQhEpmZsd2FlrrZ4FnzfungKvz7BMGfq0M57aocNhttFZ75s0CCKcNhIcKigFEE9T5nIAEgQWh0pFK4DLSXuvl/DzGACy3D6S1gqiINFArC8gpQ2EEoYIRASgjK2rnzwIIxZIZFoDTbsNpVwvuAjKygEwXkMcwMGUojCBUJrNyAQkz017n5fsHe0gkNXbbdO1bfzDMgXOjjExFGZmK4XHY+NC1q1P++7mQfqVt4TFbQi8k6XOKA6YABMMxarzOhTwtQRDyIAJQRlbUeoknNf3BMG013tT23/v6q7x0ejhj3/Y6H3dubZnze6Vn21hUwlCYUHR6TnHAYyz6EgcQhMpEXEBlpN1MBU3PBNJac+jCOL98ZTsv/PGtHPzzt9FZ7+Xze05eVPfQUDS/BWAViC0ECXNOca4FIAIgCJWICEAZSQlAWhzg/EiIiUict66up6POR7XHyUduXsdrXaP87OTch8iku1osvE77ggaBszuUTlsAUgsgCJWICEAZWZFHAI6bLaI3tValtr3nqg5aqt18fs+J1LZwLMFvf2Uf//d/vFbSe4Vj064WC49rYWMA1nv7XGIBCMJiQASgjPjdDmp9zoxMoKO9hgBsbAmktrkddn77xrXsPTXM/rPDROIJPvJv+3nmcB+7j/SV5BoKx5IZaaAAXqdtYQUgq0V1ehBYEITKQwSgzLTXerkwGk49PtYbpL3Wm3KHWHzgmpXU+138w49P8MDXXuHZYwNcvbqekakYAxORou8TiiXwuiorCJzdobTa/MwyFEYQKhMRgDKzotabEQQ+3hdkU2sgZz+fy8H/uH41z58Y5MdH+vnLu7fxB3dsMI7pnSj6PqFonhiAa2FjAKGsGIDbYZOhMIJQwYgAlBnDAjAEIJZI8ubARF4BAPjQdau5cmUtf3n3Nn7j2tWp/YqNltRaE47nCoDHaSccL68A/PqX9vKFPSdL2ncqmikAMhRGECobqQMoM+21XoKROGOhGL1jYWIJzaaW/AJQ7XHynY9en3rcUOWmscrF8d6ZBSAST6I1uPMVgpWxHXQyqXn59DB2m40Hbi2+f6pFdVpwWvoBCULlIgJQZqy5ABdGQ5zoN1w5hSyAfGxsCRS1ALLTLS3KHQMYnIwQS2jOD0+VtH84mpkFBIbIiQUgCJWJuIDKzIq0YrBjvePYbYq1Tf6Sj9/YEuB4X5DkDA3UQlnBVguv2QqiXOMXesxg9vmR0Iznk3NezkwLQILAglCZiACUGasY7MJYiGO9QdY2+nFnpWvOxKbWAFPRxIyTxbLHQVp4XXYSSU0sUSYBGDMEIJpI0h8sLTMp+7xkLKQgVC4iAGWmwe/C5bAZFkCBDKCZSAWCZ4gDTOfbZ/7zeco8GL5nbFqEzpXgBkqdV0YMQIbCCEKlIgJQZmw2RXutl+N9QbqGQwUDwIXY0GxUDM8UB5ieB5zrAoLyDYXpHZuuZ+iahQDkWgAiAIJQiYgAXAJW1Hp48ZTR52e2FkDA40wJSCEKBoHNwrByCcCFsTBtNR6Ugq6REgQglsBhUzjt0z8rGQojCJWLCMAloL3Wm/LTz1YArGNmcgFlV9xaWK0hyuUC6h0LsbLeR0vAQ9dw8UE3oawxlSBDYQShkhEBuARYmUA+l53OOt+sj9/YEuDNgQliifw5/YVcQJbvvVzVwBdGw6yo9dJZ7y3JAgjn6VAq/YAEoXIpKgBKKY9S6mWl1GtKqUNKqU+a29copV5SSp1USv27Usplbnebj0+az6++tB+h8rAygTa0BOY09WtTaxWxhObM4GTe5/P52tMfl8MCSCY1feNhWms8dNb5SqoFCEVzLQAZCiMIlUspFkAEuE1r/RZgO/AOpdQu4K+Bz2qt1wMjwH3m/vcBI+b2z5r7LSssAdjUUlVkz/xsaqkGCgeCw/MQBB6ciBBPalbUeOio99EzHiYan7nKON+MAmkJLQiVS1EB0AZWdzKn+aeB24DHze2PAfeY9+82H2M+f7tSau7DbxchnfWG22dLW/Wcjl/b5MduUwVbQqTqALILwVIuoItvB2HVALTWeOms86I1RQfeT+WZUiZDYQShcikpBqCUsiulDgD9wDPAm8Co1tq6rDsPtJv324EuAPP5MaChnCdd6XTW+/jih3by39/aOafjPU47qxt8BS2AVAzAkdsOGspjAVg1AG01npSgFYsD5IsBWMPgR6dEAASh0ihJALTWCa31dqADuBrYfLFvrJS6Xym1Tym1b2Bg4GJfruK4c2sLPtfcWy3NlAkUiiVw2hUO+6UrBLMsgAwBKJIJFIolMvoAAbRWe8zXK55FVIjXz4/x4S+/TCRPp9MnDnTz5Z+dnvNrC8JyZlZZQFrrUWAPcC1Qq5SyVrgOoNu83w10ApjP1wA5w2+11o9orXdqrXc2NTXN8fSXLhtbApwdnsqb0ZNvIDxMVwbnswC01rw5UHzOgEXPWBiXw0a930VrtQenXRW1AELR3DGVXpedBr+L7rQhObPlhZOD7Dk2kFeAvvriWf7i+4d5rWt0zq8vCMuVUrKAmpRSteZ9L3AncARDCN5j7nYv8IR5/0nzMebzP9Hl6k62jLhqVR1aw9OHe3Oei+SZBQBpFkAe0dhzrJ/b//6nfP/ghZLevydVBKaw2xQrar1F20GEY8m857Wi1jtjb6NiDJkT0vqDuSLSH4ygNXziiTek2EwQZkkpFkAbsEcpdRD4BfCM1vr7wB8DDyqlTmL4+B81938UaDC3Pwg8VP7TXvpcv66RdU1+vvj8qZzunoUsAKfdhtdpz9u47ZnDfQB86qmjJdUJ9IyGaKvxpB6Xkgo6FY3nFYD0ITlzYXgyCsBA1ufSWtMfDLO6wcfB82N88xfn5vwegrAcKSUL6KDW+kqt9RVa68u01n9hbj+ltb5aa71ea/1rWuuIuT1sPl5vPn/qUn+IpYjNprjvhrW80T3Oy6eHU9vDsQSvdo3SWOXKe9x16xr4ydH+DNHQWrPn6ABrG/10j4Z45Lni/ySGBeBNPTaKwYrHALIzk8CwAC6MhubcpnqwgABMROKEY0k+cM1Kdq2t529+eCxlLQiCUBypBK5gfmVHO/V+F198fjrI+fmfnOTs0BQP3rkp7zFv39ZK92iIQxfGU9uO9gbpHQ/zkZvX8a7L23j4pyczrsj3nx1m95G+1OOEWQSWbgF01PkYnowyWaClQzKpCceSeS2TFbUepqKJOWcCDU9aLqDMxd163Bzw8Jd3X8ZkJM7f/PDYnN5DEJYjIgAVjMdp54O7VrH7aB+nBiY41hvkn376Jr+yo50bNjTmPeb2Lc3YFDx9aDp28OwxI8vq5k1NPHTXZrSGT//gKEMTEf7wW6/xqw+/yO98dT9jIWOBtorAMlxARVJBI2aRWHYWEECHOSVtrnGAoQnDAugfz4wB9I9bAuBmQ0uA37xuNf++rytnP0EQ8iMCUOH8xq5VOG02vvTCaR769kGqvU7+33dtLbh/Q5Wbnavr+dGh6Sv6Pcf62dpWTUu1kdL5Ozet5cnXLnDL3z3Lk691c/f2FcSTmj1H+4H0FNBpF9DKIqmg+YbBWKSmpM1BALTWDJkuoFwLwDjPpoAbgJ2r6wEYEDeQIJSECECF0xRwc8+VK/j6S+d49dwon3j3Fur9+f3/Fm/f1sqxviBnBicZC8XYf3aEWzdPp9p+5JZ1rG3yc9mKGn7wsRv57Hu30xxw88M3DKuhx1yo22rTg8DGIl5oLsBMApCakjYHAZiIxFMtKLIFYCDNBQTTnUel7YQglIYIwCLgvhvWAnDjhkbu2d5eZG9429YWwEghfeHEIImk5tZNzannfS4Hux+8mW/cv4v1zUbDurdta+GnxwcIRRN5LYB6vwufy17QBRSKGotudh2AdazHaUxJmy1WBlCV25HrAgpGcDlsVHuNhb/Kaj0tAiAIJSECsAjY1BrgPz5yLZ//wA5KaavUWe9ja1s1PzrUx55j/dR4nWzvrM3YJ/t13rGtjVAswXMnBugZC+F22KjzOTP276zzFXYBRfPPKbaOXVHr5cIcqoEHTf//5tYA4+F4RpFb/3iY5oA79Vmq3DJ7QBBmgwjAIuGtq+tTfXVK4e3bWnnl3AjPHO7jxg2NOW0jsrlmrfH6PzrUm1EElk5nvZfzhSyAGVxAYLiB5lINbFkAm9uMwTrpqaD9wQg6mJwDAAAgAElEQVTNpv8fpPGcIMwWEYAlytsva0FrGAvFMtw/hXDabdy+pZndR/rpGglluH8sOut9nBueIpmn4jZUYEqZRXutd04uICuv3+qs2p8jANNxilTrabEAysLr58f46fGl16dLmEYEYImyqSWQyty5eVNpvZbesa2VsVCM17pGM1JALTa2BJiKJvLGAQoNqbFYUetlcCIy606lVgbQ5lZDAAbS2kH0j4dprp62ANwOGw6bkhhAmfjCnpN88nuHFvo0hEuICMASRSnFR25ex69fs5LGKnfxA4CbNjalFvD0DCCLreZV+JGe8ZznCs0ptrAygawAc6kMTUTxuewpMbMsgHAswXg4nuECUkoR8DgkC6hMjIViBQv/hKWBCMAS5gPXrOR///LlJe/vcdq5xbQWWvO4gDa1BrApOHwhVwCmSrAAYPapoMOTERqqXNT7XdhtKlX8lZ0CalHlcUgQuEwEIzGmIuWZLy1UJiIAQgZv39YKQHseC8DjtLO2qYrDPblzCooFgVPVwLOMAwxNRqn3u7HbFA1+V6r4K1UEVp1p3VS5nWIBlIlgOM5kND7nHk5C5TP3iSXCkuRdV7QRjiW4cUP+uMGWtmpeOTuSsz01p9iV/5qipdqDUrOvBh6aiKbiEc3V7tSVf3obiHQMF5BkAZWDiXCcpDbafOTr8SQsfsQCEDJw2m287+qVOAukjW5tq6Z7NMRYVmO3UDSB3aZwFTjO5bDRHHDP2gU0NBlJVT43BzypGEB/ARdQwC0uoHJhWVJTJbQPFxYnIgDCrNi6wggEH84KBIfMecAzFaq1z3IwjNaa4ckoDWYQuzngThOAcMotlI7EAMpDOJYgmjCK+yQQvHQRARBmxRazICufABRzE6yY5WCY8XCcWEKnFvnmgJuhiQiJpKZ/PEJjlQubLVNwJAuoPKR/h+WYMS1UJiIAwqxoDnhorHLnpIKGogm8Bfz/Fu11Xi6MhvMWkuXDqgJuMIffNFV7SGqjOCy7CMyiyu2UOoAykB5HEQtg6SICIMyarSuqc1JBQ9H8c4rTaa/1Ek0kGZwsrV2zVQVcn2YBgOH/z24DYRHwOIgmkkTictV6MaS70SQGsHQRARBmzZa2ACf7J1JtmmE6BjATVjGYlQp6YTTEEwe6C+5vVQFbhWxNKQEIMxDMrAK2CEhL6LKQ/v2JACxdRACEWbO1rZpoIsmbAxOpbYXmAaczXQwWpmt4il/7pxf52DcPpFw92ViTwLItgJ6xMEOTUZryuoAuriX0L84M8619XXM6dimR7gKaioqYLlWKCoBSqlMptUcpdVgpdUgp9TFze71S6hml1Anzts7crpRSn1NKnVRKHVRK7bjUH0KYX/K1hAiXYgGYxWAvnx7ifY/sTWUEFQoMW7OALQGwLIAjPeNonVsDABffEvqrL57lb354dE7HLiXG0wR0UqqBlyylWABx4A+11luBXcADSqmtwEPAbq31BmC3+RjgLmCD+Xc/8HDZz1pYUNY0+nE7bKk4QCiaoG88jM81c11htcdJwO3gsRfPEgzH+CuzTUUhARiciBJwO1LZRW6HnVqfkze6jffNHwMwWkKPz7EYbDwcY3QqNqvqV601L5wYnHXF7Eunhjjel1tVXQlkuoDEAliqFBUArXWP1voV834QOAK0A3cDj5m7PQbcY96/G/iKNtgL1Cql2sp+5sKC4bDb2Nwa4EjvOImk5ve/+Sr9wQjvuaqj6LErG3wEPA7+7beu4U5zclmhBnHDk1HqqzLz/JsD0xlIzdW5LqDARU4FGw/FiCf1rCyIfWdH+OCjL/HiqaFZvdcf/+dB/vZHx2Z7ivNCpgtILIClyqxaQSilVgNXAi8BLVrrHvOpXqDFvN8OpDtRz5vbehCWDFvaqvnRoV7+4nuHeOZwH5/8pW3curn43IHPvHc7boeN1Y1+kkmNy24raAEMTUZyCr2aAx6O902Y98vvArKufEenYilrohinByeB6fYUpTI0EaXKM/sZCfPBRDiO32UnltBMigWwZCk5CKyUqgL+E/gDrXVGDqA2bN9Z2b9KqfuVUvuUUvsGBmToxGJj64pqRqZiPPbiWX77xjXce93qko7b1BpgdaMfAJtN0Vbr4UIBC2BowmgEl076op+vzXXVRWYBpQtAqVhZTSNT+YPZ+YjGkwQjcXpn2R57vgiG41R5HPjc9tSsB2HpUZIAKKWcGIv/17TW3zY391muHfO239zeDXSmHd5hbstAa/2I1nqn1npnU1NpA0uEymHbihoA3nl5Kx+/a8ucX6etxkNPwSBwlMYsF5AVCK73u3A5cn++KRfQHC0AK3Ywm8XcCmaPzEI0Rs3XH5yIZqTTVgrBiGEB+V0OCQIvYUrJAlLAo8ARrfVn0p56ErjXvH8v8ETa9g+Z2UC7gLE0V5GwRNixspYvf/itfPa/b89pxzAbVtTkbw9h9QGq9+cXgHzuHzACxS67bU4WQDyRTPm7ZyMA1pzk0VkcM5y2b9945VkBwXCcgMeB12UnFBMX0FKlFAvgeuA3gNuUUgfMv3cCnwbuVEqdAO4wHwM8BZwCTgJfBD5a/tMWFhqlFLduasbtuLg2wStqvfQFjf4+6YyH4sSTOtUIzsIK/DYVEAAw3EBzaQmdbjWMhWbhApqDBZBe+9A7RwHQWvPVF8+kKqbLyXg4bloAdrEAljBFg8Ba6xeAQpd4t+fZXwMPXOR5CcuEtlqP0dwtGM4YRG+1i8gNAlsWQG4GkEVgjh1Bx0PTx4xMlraYJ5KanlFjAZ+NBZD++rMdk2nRNRziE08cIp7UfPj6NXN6jUJMhGN01HmJxZOSBrqEkUpgYUFZUTNdHZxOdiM4i5QA5GkDYVHldswpDTS9dqBUF1B/MEzctF5m4zZKdwH1js0tE8iaimZVTJeTYDhOwO3A57JLGugSRgRAWFAKzQrObgRn0VbjpdbnTFUj56PKPbeW0OkCUKoLyMoAavC7SrYaAEZMgfM4bXO2AKzZCEMlNtebDVYMwOd2iAAsYWQkpLCgtJmzh3uyroKzG8FZeF12fvEnd+CYIfAc8DhzBs+MTkUJx5K01hR2HVmi4bSrkq/mrffZ1l7D/jPDJR0DhoUT8DhoDrjnHATuN48bLLMFEEskCcUSBDzGfGVpB710EQtAWFCqPU6q3I4cF5Dl1qjzuXKOcdptM04eyzcX+M+fPMT7v7h3xnYNlgB01PlKDuieNy2AbSuqmYwmSk7pHJkyMpxaazwXbwGUOQhsuc9SWUBiASxZRACEBaetxpPjAhqejFLtceTN9S9GVZ65wKcGJzk9OMnZoamCx42bbp/Oeh9jJVoA50dC1PtdrDAti1IDwcOTUep8LlqrvXMuBrMEoFA31bkSTAmAWQcQjc+6z5GwOBABEBacFbXenKvgwYlITgpoqQQ8RhA4fdGyBOb5E4Wrzq2Fr7POW7IF0D0aor3WS61pqZR6nGUBtNUYg+7jidkXgw2kLIAyC0DE+AxVbqMSOKkhUoHFasLFIwIgLDgraj05MYD+YG4foFKp8jiIJzXhmLFohWOJlJ/8+RODBY8bD8fwu+w0VLkZD8dyahPy0T0yRUedN+WqKjV2MDIZMyyAGiMNdi5+fMsCCEbihMs4t9cSwmqPA5/ZiVUCwUsTEQBhwWmr8TI4EU0tYpF4goPnR7msvWZOrxcwG8JZV7KWiyXgcfDim0MFr7aDYaP9QZ3PidbTLqFCaK3TLACjcdxsXED1fidtNfmD4KUwEAzjsttSr1cu0l1APvO7lEDw0kQEQFhwrFRQa6F+9dwo4ViS69Y1zOn1rC6eVjDTcv/cs72dYCTOa+dH8x5npT5ai3mxq/nhSSOzqL3OS52/dBdQKJogFEtQZwaBYfbtIOKJJEOTUdY3VwHldQNZAfSAx4HfnPEgFsDSRARAWHCsAOoF8yr45ycHsSm4Zu3cBMBqCW1dyVrdRn/1qg6UKuwGGg/HqPY6S/bnWymg7bVe6lIWQHEBsISl3ueitdqyAGYnAIMTUbQ22nLDdOV0ObC+tyqPUQgGMhRmqSICICw4baYFYLVU+NmbQ1zeUUuNt7R+/NlUZXUEtSyALW0BrmivKSgAlgVg+fPHQjNfVVspoO11XrxOOy6HrSQXkOWuqfO7jK6mdtusM4GsAPCWtgBQXgvA+t4CGQIgFsBSRARAWHAsP/iF0RATkTivdY1y/RzdPzDdEjqY5gJqrHLjdti5YUMjB7pG846MHA/FqPY4qTWFp1hlr1UF3FHrQylFnc9ZUhA4ZQH4XSil5lQLYLWBsCyActYCjIdjuBw23A47/hJiAN8/eIFP/eBI2d5fmD9EAIQFx+O00+B3cWEszMunh4gnNdevb5zz6wXcxgJu+bIvjIVpNyuOb9zQRCKpefHN3PGN2RZAscW8ezREldtBtddYJOt8rpJiACkLwHyf1hrPrC0AKwNotTmfeajMQeBqU0S9pgUQmiHL6L8O9vDo86fLmokkzA8iAEJF0Gamgv7s5BAuh42rVtXN+bXyuYCsTqM7Vtbhc9l5IcsNpLU2BcBJwOPApor788+PhOio86aqkmt9zpJcQFYfIKvPUVuNh57x2WUBWeMnm6rcNFa5GSyjBWB9D0AqCDxTS+ihySjxpE7NahYWDyIAQkXQZg6G+dnJQXauqsPjnPucgdRcYLMY7MJoKJVp5HLYuGZNPS+czBSASDxJNJGk2uvAZlPUeJ2MFokBWCmgFiVbAFMxlCIV42it9tA3FplVte3ARJg6nxOXw0ZDlavMaaCx1HfocxcPAlvv/Xr3WMnvEY4l+LsfHZvz5DahPIgACBVBe62XM0NTHO0NXpT7BzD91zaCkTjjoThT0QQraqebwN24oYnTg5MZDePGU6mPxqJcymLePTJFe920ANT6XCVbALVeJ3azoV1rjYdoIjmrRbx/PJKaidDgd5U5DTSeiqOUUghmnfdrXaULwHPHB/j8npPsPtJ3EWcqXCwiAEJF0FbjSTVSm2v+fzpGQ7h4apFfkXalbgVOzw5OprZZw2As33cxd854OMZ4OJ5lATgZnYoVvZIfnoqm6gaAtGKw0uMA/cFIaiZCQ5W7rEHgiTQBcNhtuBw2JgtYAImkTsVKXu/OX1+Rj+N9QQBO9k9c5NkKF4MIgFARWKmgAbeDy+dYAZxOwONkIhJPpYCmC0BrngXXChhXmxaAcTVf2ALoTksBtajzuYgnNcEibo2RySj1aV1OW2syC+FKYSAYoanKEgAXg5PRsjVssyqiLfwuO1MFYgCjU0Y9Qp3Pycn+iZIrho/2GgJgCYGwMIgACBWBlaVzzdoGHPaL/1kaQ2FiqRYL6S4gq/gqfRZvMK0FMlgWQAkCUJvuAjKLwYqkjw5PFrAASqwG1lobAmBZAH4X0XiybP70dBcQgM9VeCiM5f65eWMTSQ2HSwwEHzMF4ESfWAALiQiAUBGsrPdjtylu2dRUltezxkJ2j4Zx2hWN/unOol6XnVqfM+OK24oBVHvTYwCFXUCWa6mjzpfaVmr66MhUpgXQWOXGblMlj4YcC8WIJpJpMQDjs5UjDpBMaiai8VQ/JcAcC5lfXKz001s3NwPwWldxN1AknuD04CQuh40zQ5OSPrqAFBUApdS/KKX6lVJvpG2rV0o9o5Q6Yd7WmduVUupzSqmTSqmDSqkdl/LkhaVDU8DNj/7gJt5/9cqyvJ41GN5KAbVlTRBrrfZkuYCyLACvk6logkg8/+LUPRrC5bDRmDazuM5fvIeQ1troBJpmAdhtiuaAu+QYgFUDYM1HtuYml2M05EQ0jtZkuIB8bgeTRSyAjS0B2mo8JWUCnRqYJJ7U3GJaDacGJoseI1waSrEA/hV4R9a2h4DdWusNwG7zMcBdwAbz737g4fKcprAcWN9clcqMuViqzCBwz1gow/1j0VrjyWjAZnX+TMUAzAV6rIAbqHs0REetN2MymdVDaCbX0WQ0QTSRpN6f2eYi+3xmwqoBsATAGptZjtGQE1lCCEYMIFTEAmjwu7i8vYbXzxcXAMv989/esgKAE/0SB1goigqA1vo5IHvY6d3AY+b9x4B70rZ/RRvsBWqVUm3lOllBKJWAGQO4MBpmRY035/m2mlwLwG5Tqd43damOoPkX8wujodQ8Y4tSXEAjWVXAhc5nJqw2EE3ZFkAZBCC9FbSFz2UvWAg2PDHd1+iKjhpODU7mbbORzrG+IE674vYtzThsSgLBC8hcYwAtWuse834v0GLebwe60vY7b24ThHnFygLqHQ9nZABZtFR7GJyIpFJPreKnVFWv17qaz7+oXhgN5QhLjdeJUjN3ER3OqgK2aK320jMaLimTJ+UCMoPZ1muVIxU0vRW0hREEzm8BDE9GqPY4cNptXNFRC8AbRayAY71B1jVV4XM5WN3o57gEgheMiw4Ca+MXO+v8M6XU/UqpfUqpfQMDhcf0CcJcqPI4SGojTz2fALRl9eEfD8dTPX2AtJkAuYt5NJ6kPxjJeV27TVHtmbl+IL0TaDrrm6sIxRK8WYI/fCAYweeyp6p13Q47AY+jLP2A0ltBW/jd9oJZQEOT0dToTit992CROMCx3iAbW4wuphtbqjghFsCCMVcB6LNcO+Ztv7m9G+hM26/D3JaD1voRrfVOrfXOpqbyZH4IgkVVWhZLtqsGpnPvLQEIhmOpJnIwvUDnW8z7xsNonZkCmjrO5yzNAshyAd1gVj+/MMPMYov+YCTl/7dorHKXRQBS2VBpAuB1zpwGalkgdX4XnfXeGeMAwXCM7tEQm1oNAVjfHODc8JRkAi0QcxWAJ4F7zfv3Ak+kbf+QmQ20CxhLcxUJwryR7sLIt1BnV9+OhzJz362W0KN5xkLmKy5LHVekHYQVH8i2AFY2+FhZ78vpUZSP/vFwyv9vUe93lckFlBsD8LvtTEbjed1T6QIAcEVHLQdnqAi2/P2bW6ctgKSGNwfEDbQQlJIG+g3gRWCTUuq8Uuo+4NPAnUqpE8Ad5mOAp4BTwEngi8BHL8lZC0IR0hdza7FPp8UqBrMEwJwGZuFz2XHZbXkDuhfyFJdZFJsJMDwZNV1FjpznbtjQyN5Tw8QKzCy2GAhO9wGyKFc/oPRhMBY+lwOtjYZ52QxNRmlIF4D2GrqGQ6lgdzZWBfCmlAAYt1IQtjCUkgX0fq11m9baqbXu0Fo/qrUe0lrfrrXeoLW+Q2s9bO6rtdYPaK3Xaa0v11rvu/QfQRByqTLdOQGPI+Nq1qLanHbVm3IBZVoASimjGjhPVe8Fc3JZW57sojqfa8ZBMiNTUep8roz0UYsb1zcyEYlzoEgxVX8wkmMBNFS5y1IHEAzHsNsU3rRurFZmVHabB6OmIdMC2Lm6HoD/2N9FPo73BqlyO1JW2eoG/yXPBJqIxHnpVO78B0EqgYUlirWY53P/AKlJXBkWQJZQ1PlceVtCd4+GqPe7UsNS0inmAjJcJvlHXV63rhHbDDOLwWjLPBGJpxrBWTSaLaETyYvrBxQMxzOyoYCCYyHHQ3HiSZ0hAFetquP2zc38449P5K1rONobZGNLVer1XQ4bay5xJtDX9p7lfV/cW9aGeUsFEQBhSWIFgfO5fyyMauCQ0f4gEs9xy9QUCOga8wXyv26dz2kUe+Vxl4AxZjK7BiD9/S7vqJ0xEDyQqgLOdQEldeG01VLJtoSA1FjIbAGwLI6GqszP82f/bRuxpOavnsocE6m15lhfkE2t1RnbN7YELmkx2Mn+CbSG04NScZyNCICwJLEWsXyBWgvLAsjX/gCs9s65C2pPgeIymK4gLrQQD09Fc2oA0rlxfSOvnR8rWExl1QDkcwEBF50JlN0JFKbHQma3hJ6uacg8l5UNPj5y8zqeOHCBvWmul/5ghNGpGJtaqjL239BSxbnhKUKXaPD8mSFj4RcByEUEQFiSBDxOmgJurugo3Fq6rcZDfzCSaveQXgcAhYfCpE8Yy6ZYBfFIVifQbG7Y0FhwZjHktoGwsK7CL3Y0ZF4LwBwLmd0SOr0NRDYfvWUdHXVe/uyJQ6mg9rFUADjXAtCXMBPozNAUAGfNW2Ga3FQEQVgC2G2KFx+6bcbeQq01XuJJnboyzL7yrfE5GTMHvFg+6/FwjGAkXjC2MFM7iKQ5PCW7BiCd9JnFb9/WmvHcQDDC42Zw1cpisrD6AV1sJlAwHM9xm03HAApZALmfx+O086fv3sr9X93Prz78c9Y2+hkwxcnKALLYaFoEJ/qDXFaGWRDpTETiKbfZ6aG5WwBdw1NEE0nWNVUV33kRIRaAsGRx2G15s20srLkAVgZKviBwNJHM8H1bNQD5issgbSZAHgEYD8dI6twagHTyzSxOJDVfffEMt/39s7xwcpD/9fZNOYtuudpBBCOxHAugUBB4JgEAuHNrC//zjo24HTb2nR3h5dPDbGmrztl/VYMfp12lUkTLyRlT3O02xdmLEID/5zuv88DXXinXaVUMYgEIyxbrStfKQc9e+KbdOdFUIHSmIjDjGMsCyHQBDU5E+M/95wEKZgFZ3LChiT3HDvOZp49xon+CA12j9IyFuX59A39x92V5r0KN1FIueji84QLKPD/rs2fHAIYmovhddjzO3GwoMDKtPnbHBj52xwbAsIDy4bTbuHpNPd/6RRcfuWndjAI5Wyy3z1Wr6jh8YTzDmisVrTWvd48RDMcJxxIFP+9iRCwAYdlijYY8ZloA2QJQk2oIN72Yd5s1AKW4gELRBN/a18X7H9nL1f/7x3zqB0dZ2+hnx8q6Gc/rlk1NKAWf+8lJDl0Y56pVdXzhAzv4t/uuKeiCsNsU9T5jNGQ+nj7UWzRDSGudMQ/YwrIAsoO0w5MR6qtKX6xtNpUzl8HiE+/eyng4zl//8GjJr1cKVgD45o1NTETic2qZ3TseZnQqRiKpL1nB2sUK91wRC0BYttT7XDjtKjWYPL0SGGBdkx+AN7rHUr7pntEQTrtKzePNxuuy43bY+M/95/nnn55iLBRjbZOf3711PXdd3sbm1kDRK9B1TVXsfvBm6v2u1IyBUmioyt8O4vzIFPd/dT+/f9t6HnzbpoLHd4+GiCd1nhiAaQHkCQJnZwDNlc2t1fzWDWv45+dO8Z6rOlIFZRfL6cFJmgNutq4wAs9nhyZzMqiKcSRtzOWR3nEunyGxYC7sPzvMe/7pRb73uzeUPQZSDLEAhGWLzaZoqfbkbX8ARofO5oCbn6Vl5FwYDdFa4yl4JQvGoPizQ1PcsKGRb96/i90P3syDb9vElrbqkt0Pa5uqZrX4gxHUzpfp8uo5o7L41SIVxtZ+2zszLRS7TeF22PIGgfNlAM2V3799AytqPPzJd94o2g6jVM4OTbK60c+aBkPMs1NB954a4udvztx/6UiPYSG6HLYMMSgXL5wYQmuKnselQARAWNZYV7tuhw23I9O3q5TihvWN/PzkYMp/XWjATDpf/61dvPjx2/nCB3awa23DrH3Oc2XHylqO9wVzagishf3AudGCfniAA12juB02NrcFcp7zu3M7gmY3grtY/G4Hf/5L2zjWF+TLPztdltc8PTjF6gYf7XVe7DaVcglZWMHdmWoQjvSM017rZUtbNUd7yh+ofrVrxLg9V3yecrkRARCWNVZb6Hz9ggCuX9/I0GQ0laHSPUMNwPRrembtZigHb11dT1LnLiTWAhOMxDk1WNiHfaBrlMvaa3Dac5cFr9OeEQTWWuc0gisHb9vWyu2bm/nc7pN55zGf7J9Ite8oRjAcY3AiwupGP067jc46b6omAIyuqqcGJhmZivH4K+cLvs6RnnG2tFWztS3Akd7xkob2lEoyqVP/Xq+cGynra5eCCICwrGk1e+rk684JhgAA/OzkIImkNieMFW4vsZBs76zFblPsOzM9wTUST3Coe5zbNjcD8EqBq8xoPMkb3WNs76zN+7zfbc+4SrbaXZTTArB4/9UrmYjE2X9mJGN7Mqn54Jde4n89/lpJr2O5w1ab7p/Vjf5UWijAS6eN76mxysWjz5/K20cpHEtwenCSrW0BtrRVMzoVo2+8fD2FTg9NMhaKsW1FNX3jkZLHgpYLEQBhWZOyALz5LYDWGg/rm6t44eQg/cFwwQljlYDf7WBrWzX70hbOIz1Bookkv7qjg4DHUbDT6NHecSLxJFeuzC8APpeDyTQBsGYBXwoBuHZdA0674rmspniHLozTOx5m76mhVNxmJix3T0oAGgwBsK6y954aosrt4BPv3sqZoSmeOdyX8xrH+4IkNWxuq2azWcFczjjAK2eNf6v7blhjPD43MtPuZUcEQFjWWDGAQhYAGNO6Xj49zJlB44qyUgUAYOfqOl7tGkkFUV81F5Qdq2rZ3llb0M9sCUMhC8DnsjOVtugWagRXDvxuBztW1vHc8cymeLuPGgt0LKFLmpxmXe2vbvQZtw0+JqOJVCroS6eH2bm6jndd3kZnvZcvPn8q5zWsxX5LW3UqNnK4jALwatcoAY+Dd13Rhtthm/c4gAiAsKyxWipkVwGnc926BkKxBP/1+gWgcA1AJbBzVT3hWJJDF4xF6tVzo7RWe2ir8XJlZy3HesfzDng/cG6Uxip3wc9mDIZPswAKNIIrFzdtbOJwz3iqjQPAnqP9XNFRQ7XHwe4j/TMcbXBmaIrmgDuVxrq60W9un2RwIsLJ/gmuWdOAw27jvuvXsP/sCPvPDme8xpGeID6XnVX1Pqo9TtprvWWtWH7l7AjbO2txO+xc0VEjFoAgzCeWBZCdAprOrnUN2BQ8ceBCxjGVyM7VRgqnFQc40DWacutsX1lLUsPBPDN7D3SNsr2ztmDGkjEYPt0CKNwIrhzctMGYE/7CSeNKvz8Y5rXzY7xtaws3b2pmz7GBGTOawLAArEUfpl1BZwYneemU8f3sWmvUG7z3rZ3UeJ088lymFXCkZ5xNrYFU2u+WtuqyuYAmInGO9wW50iwM3LGyjkPd43mD35cKEQBhWdMccGO3KWp8hVfp054AAAlxSURBVC2Aao+Tt3TWEgwbMwMKZQxVAi3VHlbW+9h3ZoTBiQjnhqemBcDM78+OA4xORTk1OFnQ/w+GC2gyrwVwaQRg2wqjZ9Dzx404wLNHDSG4bXMLt21uYnAiwuvdhYfPg3Glb+X/A3TUeXGYqaAvnR7C57KnCq98Lge/sWsVTx/uS/WG0lpzpGc85fsH2NoW4NTARFmG2B88P0pSG+m7AFeurCWaSHL4QvlrDQohAiAsaxx2G//0wav40LWrZ9zvBjMbqJL9/xY7V9ex7+xwTmFXvd/FqgZfKi5gYQnClQX8/2AskKEsAXA7bKk2EeXGZjNqMJ47YdRg/ORoP201Hra0Bbh5YzM2BbuPFnYDGSmgUVaZ/n8w/q07632cGZxi76khrlpVl5Hyet8Na6hyO/jrHxjtKHrGwoyH42xNq4vY3FZNUpc2w3gqGuebL5/jGy+f43uvXeDZY/0ZwmH9+1xp/vtYlkChTK1LgQiAsOy5c2tLUb/+9YtJAFbVMzgR5bsHurHbFJentRe40gwEp+ebH+gaRSlmbHHgdxl1ANZxQxNGDcClLHK7cUMjgxMRDnaP8fyJAW7d3IxSinq/ix0r69gzgwBYKaDpFgDAqgYfB7pGOd43wa61DRnP1fldfPSW9ew+2s/eU0MZAWAL634xN5DWmj96/CAPfft1Pv7t1/m9b7zKb375F3z4y78gnhagX9fkT1mfLdUe2mu9OQJ9KbkkAqCUeodS6phS6qRS6qFL8R6CMJ9cubKWgNvBqgZf8Z0XmLeacYAfvN7DlrZAxuzi7Z219Acz880PdI2ysTkwo2vL63KgNYRjxuI120Zwc+GmjUYc4O+fPsZkNMHtZi0DwK2bm3m9eyzv3GGYbvmQHgMAIw7QbXZ0tfz/6Xz4+tW01Xj41FNHUq6Y9PkFK+t9eJ12jvTOLABff/kc3z/Yw4N3bmTvx2/nxw/exJ++eysvnhrib58+htaaV86Npq76LbavLJypdSkouwAopezAF4C7gK3A+5VSW8v9PoIwn7gddr790ev42O0bFvpUirKuqYpan5OknnYvWFgLjrXIaK1TAeCZ8Lszh8IMl7ERXCFaqj1sagnw/IlB3A4b161rTD13+xZDDApZAVbv/2zBXmMKgtdp5/L23M/scdp58M6NvHZ+jMdePENnvTdDGO02xabWwIwWwKELY3zye4e5aWMTv3vrerOWJMD/uGENH9y1kn/+6Sn++blTDE9Gc+IuO1bW0T0aKihs5eZSWABXAye11qe01lHgm8Ddl+B9BGFe2dASmHWDtoXAZlPsXGX5lTMXmC1t1bgcNl4+PUR/MMwvzowwOhVj+wwBYJjuCDoWihFLJC9JG4h83LTRWPSvW9eQYclsagnQXuvlx0cMv7r1F40niSWSnBqcpKV6OgXUwhKEq1bV4XLkX/5+ZUcHm1sDDE5E2ZI1vhKM7/BobzBv24ZgOMbvfv1V6nxOPvvet+Q0DfzEu7eyvbOWT5txhuzW4Na/13y5gS5FO+h2oCvt8XngmkvwPoIgFOCtq+v58ZH+HBeDy2HjshXVPPbiWR578Wxq+1WrZp5RUGUOhbnt73+a2nbXZfMhAE188fnT3LalJWO7UorbNjfz1b1n2fyJH+Y99po1uS4ea55CPvePhd2meOiuzfzml3+R4f+32NoW4Bsvn2PNx59KOx/jVmuwKfjGb++iIU/LcLfDzsMf3MG7P/cC4ViCjS2Zjfe2rajGZbfxf33tFWwF4itf/s23ptxjF4sqd/MhpdR7gHdorX/LfPwbwDVa69/N2u9+4H7z4WXAG2U9kcVLIzD/fWErE/kuppHvYhr5LqbZpLXObd9aIpfCAugGOtMed5jbMtBaPwI8AqCU2qe13nkJzmXRId/FNPJdTCPfxTTyXUyjlNp3McdfihjAL4ANSqk1SikX8D7gyUvwPoIgCMJFUHYLQGsdV0r9LvAjwA78i9b6ULnfRxAEQbg4LslMYK31U8BTRXec5pFLcR6LFPkuppHvYhr5LqaR72Kai/ouyh4EFgRBEBYH0gpCEARhmbLgArBc20YopTqVUnuUUoeVUoeUUh8zt9crpZ5RSp0wb2dO0F5CKKXsSqlXlVLfNx+vUUq9ZP42/t1MKljyKKVqlVKPK6WOKqWOKKWuXa6/C6XU/zT/f7yhlPqGUsqznH4XSql/UUr1K6XeSNuW97egDD5nfi8HlVI7ir3+ggrAMm8bEQf+UGu9FdgFPGB+9oeA3VrrDcBu8/Fy4WPAkbTHfw18Vmu9HhgB7luQs5p//hH4odZ6M/AWjO9k2f0ulFLtwO8DO7XWl2EklbyP5fW7+FfgHVnbCv0W7gI2mH/3Aw8Xe/GFtgCWbdsIrXWP1voV834Q4z95O8bnf8zc7THgnoU5w/lFKdUBvAv4kvlYAbcBj5u7LIvvQilVA9wEPAqgtY5qrUdZpr8LjEQVr1LKAfiAHpbR70Jr/RwwnLW50G/hbuAr2mAvUKuUapvp9RdaAPK1jWhfoHNZMJRSq4ErgZeAFq11j/lUL9BS4LClxj8AfwQkzccNwKjW2hpDtVx+G2uAAeDLpjvsS0opP8vwd6G17gb+DjiHsfCPAftZnr+LdAr9Fma9ni60ACx7lFJVwH8Cf6C1zmgxqI0UrSWfpqWUejfQr7Xev9DnUgE4gB3Aw1rrK4FJstw9y+h3UYdxVbsGWAH4yXWHLGsu9rew0AJQUtuIpYpSyomx+H9Na/1tc3OfZbaZt8WnXy9+rgd+SSl1BsMNeBuGH7zWNP1h+fw2zgPntdYvmY8fxxCE5fi7uAM4rbUe0FrHgG9j/FaW4+8inUK/hVmvpwstAMu2bYTp434UOKK1/kzaU08C95r37wWemO9zm2+01h/XWndorVdj/AZ+orX+dWAP8B5zt+XyXfQCXUqpTeam24HDLMPfBYbrZ5dSymf+f7G+i2X3u8ii0G/hSeBDZjbQLmAszVWUH631gv4B7wSOA28Cf7LQ5zOPn/sGDNPtIHDA/Hsnhu97N3AC+DFQv9DnOs/fyy3A9837a4GXgZPAfwDuhT6/efoOtgP7zN/Gd4G65fq7AD4JHMXoFvxVwL2cfhfANzDiHzEM6/C+Qr8FQGFkVb4JvI6RPTXj60slsCAIwjJloV1AgiAIwgIhAiAIgrBMEQEQBEFYpogACIIgLFNEAARBEJYpIgCCIAjLFBEAQRCEZYoIgCAIwjLl/wCy3PdZCoPp5QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss on validation set: 0.10626037213660311\n",
      "dcbadcbadcbadcba\n"
     ]
    }
   ],
   "source": [
    "ENC_RNN_NUM_OF_LAYERS = 1\n",
    "DEC_RNN_NUM_OF_LAYERS = 1\n",
    "EMBEDDINGS_SIZE = 4\n",
    "ENC_STATE_SIZE = 32\n",
    "DEC_STATE_SIZE = 32\n",
    "\n",
    "att = AttentionNetwork(\n",
    "    ENC_RNN_NUM_OF_LAYERS, DEC_RNN_NUM_OF_LAYERS, EMBEDDINGS_SIZE, ENC_STATE_SIZE, DEC_STATE_SIZE)\n",
    "train(att, train_set, val_set, epochs=6)\n",
    "print(att.generate('abcdabcdabcdabcd'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems like the attention mechanism solved our problem! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matches for short strings 100\n",
      "Matches for medium strings 100\n",
      "Matches for long strings 100\n"
     ]
    }
   ],
   "source": [
    "print('Matches for short strings', count_matches(att, short_strings))\n",
    "print('Matches for medium strings', count_matches(att, medium_strings))\n",
    "print('Matches for long strings', count_matches(att, long_strings))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loss went down on the same dataset from 200 to 0.  We now can predict correctly long strings.\n",
    "\n",
    "How did this happen?  Let us zoom in on the attention part of the model and try to illustrate what was learned."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attention Weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make a version of the attention network that plots the attention of each encoded input at each decoding step when generating.  We want to look at what was the relative weight that was computed for each of the outputs of the encoders \n",
    "$encOutput_i$ when generating each of the output characters $Output_j$\n",
    "\n",
    "This has the shape of a matrix NxN which will tell us where the model is \"focusing\" on the input encodings when it decides to generate each output char."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "class AttentionNetworkWithPrint(AttentionNetwork):\n",
    "    def __init__(self, enc_layers, dec_layers, embeddings_size, enc_state_size, dec_state_size):\n",
    "        AttentionNetwork.__init__(self, enc_layers, dec_layers, embeddings_size, enc_state_size, dec_state_size)\n",
    "\n",
    "        self.should_print = False\n",
    "        self.att_mat = []\n",
    "\n",
    "    def _attend(self, input_vectors, state):\n",
    "        attention_weights = []\n",
    "\n",
    "        w2dt = self.attention_w2 * state.h()[-1]\n",
    "        for input_vector in input_vectors:\n",
    "            attention_weight = self.attention_v * dy.tanh(self.attention_w1 * input_vector + w2dt)\n",
    "            attention_weights.append(attention_weight)\n",
    "        attention_weights = dy.softmax(dy.concatenate(attention_weights))\n",
    "        if self.should_print:\n",
    "            self.att_mat.append(attention_weights.value())\n",
    "        output_vectors = dy.esum(\n",
    "            [vector * attention_weight \n",
    "             for vector, attention_weight \n",
    "             in zip(input_vectors, attention_weights)])\n",
    "        return output_vectors\n",
    "\n",
    "    def _plot_attention(self, matrix, max_weight=None, ax=None):\n",
    "        \"\"\"Draw Hinton diagram for visualizing a weight matrix.\"\"\"\n",
    "        # I stole this code for matplotlib tutorial\n",
    "        ax = ax if ax is not None else plt.gca()\n",
    "\n",
    "        if not max_weight:\n",
    "            max_weight = 2**np.ceil(np.log(np.abs(matrix).max())/np.log(2))\n",
    "\n",
    "        ax.patch.set_facecolor('gray')\n",
    "        ax.set_aspect('equal', 'box')\n",
    "        ax.xaxis.set_major_locator(plt.NullLocator())\n",
    "        ax.yaxis.set_major_locator(plt.NullLocator())\n",
    "\n",
    "        for (x, y), w in np.ndenumerate(matrix):\n",
    "            color = 'white' if w > 0 else 'black'\n",
    "            size = np.sqrt(np.abs(w))\n",
    "            rect = plt.Rectangle([x - size / 2, y - size / 2], size, size,\n",
    "                                 facecolor=color, edgecolor=color)\n",
    "            ax.add_patch(rect)\n",
    "\n",
    "        ax.autoscale_view()\n",
    "        ax.invert_yaxis()\n",
    "        plt.show()\n",
    "    \n",
    "    def generate_and_plot_attention(self, input_string):\n",
    "        att.should_print = True\n",
    "        att.att_mat = []\n",
    "        output_string = self.generate(input_string)\n",
    "        self._plot_attention(np.array(att.att_mat))\n",
    "        att.should_print = False\n",
    "        att.att_mat = []\n",
    "        return output_string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b07266e9e555436488bb681e83c10748",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=18000), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztvXmQJHd55/156j76vuboOTUaSQiJ0UiDEIfASGADNkjGrJHAL3qN3hhvmF3jtWNZHBvhtcPvu4DffZfD4dCuDNjCL8ZgrbEElsGyOCRkdMwgadA9o9GMpnuunr6mr7p/+0dmVmVVZVVl9Vnd9XwiOqoyKyvrN6VSfvO5xRiDoiiK0n4E1noBiqIoytqgAqAoitKmqAAoiqK0KSoAiqIobYoKgKIoSpuiAqAoitKmqAAoiqK0KSoAiqIobYoKgKIoSpsSWusFAPT29Zv5aB87+xJ0xcNrvRxFUZR1weHDhy8YYwYX+/6WEIAdO3cy/Ut/wh/96tV85E071no5iqIo6wIRObmU97eECygUsJYxNpNe45UoiqK0Dy0hACLQmwgzNpta66UoiqK0DS0hAAADHVG1ABRFUVaRlhGAwc4oF2Yza70MRVGUtqGlBEAtAEVRlNWjdQTAdgHpgBpFUZTVoWUEYKAzykI2z1wmv9ZLURRFaQtaRgAGO6IAXFA3kKIoyqrQOgLQaQnA2KwKgKIoymrQUABE5HIRedr1d1FEfldE+kTkQRE5aj/22seLiHxJRI6JyBERudbPQooCoBaAoijKqtBQAIwxLxljrjHGXANcB8wD3wY+DTxkjNkLPGRvA7wX2Gv/HQTu8rOQgQ4VAEVRlNWkWRfQzcArxpiTwC3APfb+e4Bb7ee3AF8zFo8BPSKypdGJ+5IRAqICoCiKslo0KwC3Ad+wn28yxpyxn58FNtnPh4FTrveM2PvqEgwI/R1RLmgMQFEUZVXwLQAiEgE+APxd5WvGSt5vKoFfRA6KyCEROTQ2NgaUagEURVGUlacZC+C9wM+MMefs7XOOa8d+PG/vHwW2u963zd5XhjHmbmPMAWPMgcFBq531YGdUs4AURVFWiWYE4HZK7h+A+4E77Od3APe59n/Mzga6AZh2uYrqog3hFEVRVg9fA2FEJAm8G/gt1+7PAt8SkTuBk8Cv2/sfAN4HHMPKGPpNv4uxGsJZ7SBExO/bFEVRlEXgSwCMMXNAf8W+caysoMpjDfCJxSxmsDNKNm+YXsjSk4gs5hSKoiiKT1piJKTDlu4YAL/4+Ye5bmcv1+3s5bbrd9ARballKoqibAha6sp68+uG+L9vvYpDJyY4/Nok//TsWV4+N8OffmjfWi9NURRlw9EyvYAAoqEgv3HDTr5w234e+dRN3PbG7XznmTPMpLJrvTRFUZQNR0sJQCW3Xb+DhWye7zzjK4lIURRFaYKWFoB927q5YnMn33zytbVeiqIoyoajpQVARPjwG7fzzMg0z5++uNbLURRF2VC0tAAA/Or+YSKhAN86dKrxwYqiKIpvWl4AehIR3vP6zXz7qVFSWR0XqSiKsly0vAAA3PbG7UwvZPn+c2c9X//RS+c5NTG/yqtSFEVZ36wLAbjhkn529CU83UDZfIGDf32YP7zv2TVYmaIoyvplXQhAICD84pWbOHRikly+UPbaqxfmyOQKPHz0AudnUmu0QkVRlPXHuhAAgNdt6SKdK3BivNzV89LZGQDyBcN9T51ei6UpiqKsS9aNAFyxpROAF8+Wp4O+dHaGYEB4/dYu7j08gtWLTlEURWnEuhGAS4c6CAWEF8/MlO1/6dwMuweS3Hb9Dl46N8NzWi+gKIrii3UjANFQkD2DHbxwptoCuHxTJx94w1YiwQD3Hh5ZoxUqiqKsL9aNAIDlBnrxbMkCmM/keG1inss3d9KdCPPu12/ivqdHyeQKdc6iKIqiwHoTgM1djE4tML1gdQd9+dwsAJdtsuIDH7puG5PzWX7w4vma51AURVEs1pUAvM4JBNtuoJdta+CKzdb+Gy8dYLAz6ssN9OOXx/jj7zy3QitVFEVpfdaZAHQBFN1AL56dIRYOsL0vAUAoGOCXr97CI0fHyOZru4GMMXzmgRf4y0dPcPTcTM3jFEVRNjLrSgCGOqP0JsLFVNCXzl3ksk2dBAOlAfL7d/SQzhU4aruHvHji1YmiiPzTs97tJRRFUTY6vgRARHpE5F4ReVFEXhCRN4tIn4g8KCJH7cde+1gRkS+JyDEROSIi1y7XYkWE123p4nk7FfSls7Ncbvv/Ha4e7gbg2dHpmuf52mMn6Y6H2betmwd+rsNmFEVpT/xaAF8EvmeMuQLYB7wAfBp4yBizF3jI3gZ4L7DX/jsI3LWcC75icxcvn51hbCbNhdk0l28uF4Bd/Uk6oyGOjE55vv/cxRTff/Ysv35gG7fuH+bFszMcH6ttLSiKomxUGgqAiHQDbwe+AmCMyRhjpoBbgHvsw+4BbrWf3wJ8zVg8BvSIyJblWvAVWzpZyOZ58PlzAFUCEAgIrx/u4uej3gVhf/P4a+SN4Tdu2Ml7rtoMqBtIUZT2xI8FsBsYA/5SRJ4SkS+LSBLYZIxx/CdngU3282HA3bZzxN63LLxusxUI/oenRwGqXEAAb9jWwwtnLlYFgjO5An/zxGv8wmWD7OxPsqU7zrU7etQNpChKW+JHAELAtcBdxpj9wBwldw8AxmrA01QTHhE5KCKHROTQ2NiY7/ft3dRBQKxAbm8izGBntOqYq4e7yeQKvFyR4fP9584yNpPmY2/eVdz3vqu38Nzpi5wcn2tm+YqiKOsePwIwAowYYx63t+/FEoRzjmvHfnSqr0aB7a73b7P3lWGMudsYc8AYc2BwcND3gmPhIJcMdgBWAZiIVB3jBIJ/PlIeCP7rn55kR1+Cd1xW+jx1AymK0q40FABjzFnglIhcbu+6GXgeuB+4w953B3Cf/fx+4GN2NtANwLTLVbQsOIVfV2yudv8A7OxP0BkLccSVCXR8bJYnTkxw+/U7CLjSRrf1JjQbSFGUtsRvFtC/B74uIkeAa4D/CnwWeLeIHAXeZW8DPAAcB44BfwH89rKumFJB2GU1BEBEuHq4uywV9N7DIwQEfu3a6nDE+67ewpGRaR0rqShKWxHyc5Ax5mnggMdLN3sca4BPLHFdddm/oweAfdt6ah5z9bZu/vInJ8jkCgQDwt//bJR3XDbIUFes6tgb9w7ymX96kadPTRWrihVFUTY6vgSg1XjLngEe+dQ7616srx7uJpO3AsETcxnOXkzxh++/0vPYLd2WKJyfSa/IehVFUVqRdSkAQMM79TcMW9bBkZFpfnp8nJ5EmJtfN+R5bE8iTCQY0JnCiqK0FeuqF1AzbO+L0x0P8+grF/j+c2e5Zd9WoqGg57EiwmBnlLGLagEoitI+rFsLoBFOIPgfj1jZPR+6bnvd4wc7o+oCUhSlrdiwFgBYgWCw0kWvGu6qe+xQZ1RdQIqitBUbWwDsgrAPXbfNs2DMzVCXWgCKorQXG1oA3nn5EJ+8eS+3Xb+j4bFDnTGm5rOkc/lVWJmiKMras6EFIB4J8h/efRkd0cahjk1dVk+hMbUCFEVpEza0ADTDUKfWAiiK0l6oANg4XUXPayqooihtggqAzVDRBaSZQOuJ//+xk9z1o1fWehmKsi5RAbDpT0YJiLqA1hvff+4s9z1d1W1cURQfqADYBAPCQEdUXUDrjEyuwEJWM7cUZTGoALiwagHUBbSeyOYLzKVVABRlMagAuBjqjKkLaJ2RzRvmM7m1XoairEtUAFwMaT+gdYfjAioUmhpJrSgKKgBlDHVGGZ9Nk9eLybohmy9gDKS0gltRmkYFwMVgV4yCgfFZtQJWg+89e5a3fvYHZHKFRZ8jk7feO59RAVCUZlEBcDFkF4Od00ygVeGVsVlGpxaYTS/eh591BEADwYrSNCoALhwB8JsJ9N++/xJf++mJlVvQBidt3/kvJY0zm7fcdfNZDQQrSrP4EgAROSEiPxeRp0XkkL2vT0QeFJGj9mOvvV9E5EsickxEjojItSv5D1hOnIHxfgLBuXyBrz76Kv/83LmVXtaGxem8urAE903WFhFNBVWU5mnGAninMeYaY8wBe/vTwEPGmL3AQ/Y2wHuBvfbfQeCu5VrsSjPY4b8f0HOnLzKfyS/JfbGRuPvhV3jg52eaeo/j+08twQJwYgBLERFFaVeW4gK6BbjHfn4PcKtr/9eMxWNAj4hsWcLnrBqRUIDeRNiXC+jJExMAKgA2f/3YyaZbMiyPC8i2ALQWQFGaxq8AGOCfReSwiBy0920yxji3fGeBTfbzYeCU670j9r51gd9isCdetQUgpRcegFS2+YpcxwJY7N17vmBwMnbVAlCU5vE7FP5txphRERkCHhSRF90vGmOMiDSVPG8LyUGAHTsaT+xaLfyMhjTGcOjkJKAWgEMqm2/6LnypFoBz9w9qASjKYvBlARhjRu3H88C3geuBc45rx348bx8+Cmx3vX2bva/ynHcbYw4YYw4MDg4u/l+wzAx2Rhm7WN8F9MrYLBNzGbZ0x5hN57QKFUhnC02nYmbsIPBiYwAZlwBoGqiiNE9DARCRpIh0Os+BXwSeBe4H7rAPuwO4z35+P/AxOxvoBmDa5SpqeYY6Y4zNpjGm9kX9yRPW3f87rxgC9O6zUDBk8oVFWwCLLeLKugrItBBMUZrHjwtoE/BtEXGO/xtjzPdE5EngWyJyJ3AS+HX7+AeA9wHHgHngN5d91SvIUGeUbN4wOZ+lLxnxPObJVycY6Ihw1dZuwHIDdcbCq7nMlmKxF/KlxgCcGgDrs9tbhBVlMTQUAGPMcWCfx/5x4GaP/Qb4xLKsbg1wJoOdn0nVFIAnTkzwxl19dMSsr2+uzeMAjgun2e9hqTGAjFoAirIktBK4guJw+Bq1AGemFxiZXOCNu/rojFoCMNPmmUBOI7Z0rkAu77+vz1LrADIaBFaUJaECUEGpHYS3ADjpn9fvLlkA7Z4JlMq6L8T+L+ZLrQR2ZwFpGqiiNI8KQAVuF5AXT56YoCMa4orNnXTYFkC71wK47+Cb8cVnljUNVAVAUZpFBaCCRCRERzRU0wX05KuT7N/RQygYKArATNtbAKWLbzPFYMtZB7CgLiBFaRoVAA92DST45+fOcq6iHuDk+BwvnZvh+l19AHTG1AKAchfQYiyARccAclYWUDIS1GZwirIIVAA8+OwH38D0QpY7vvoEF1NZwCr++vD/fIyuWIhf2bcVgGRUYwBQPo1rURbAEmMAPYmIpoEqyiJQAfDgquFu7vqN6zh2fpbf+tphjoxM8eH/+VOy+QJ/e/DN7B5IAhAOBoiFA22fBppe4xhAdzysaaCKsghUAGrw9ssG+dMPvYGfHh/nlj9/lFAgwLf+7Zu5cmtX2XEd0ZDGABaRBeRUDwMsZP2njropWQAqAIqyGPw2g2tLPnjtNqbms9z/zGn+7Pb9bO9LVB3TEQ21fQwgXeYC8vdduHP4U4u8eGfsSmBLAHIYY7Ar1hVF8YEKQAM+/rbdfPxtu2u+3hELaQzAbQH4/C7SrireRbuAciUXUMFY54yFg4s6l6K0I+oCWiJqAVTWAfi7mGeWQwCKMYBIU5+tKIqFCsAS6YiGNQZgWwDhoPhuyeC4jTpjoUW7gNxBYNCeTIrSLCoAS6QzFmI2nfV9/OGTE3z0y4+V+c3XO6lcnlBA6IyFfffldyyAnkR40RZA2nUOWNpoSUVpR1QAlkhHNNRU7vs3nzzFo8fGOT3VeO7weiGVzRMLB0lEgk1YAKW791zBlLmE/OK0g1YLQFEWhwrAEkk2EQMwxvDI0QsATMw1nju8XkhlC8TCAZKRUPMWgO2/X8zdezEN1BYAbQinKM2hArBEOmMhMvmCL5fOsfOznJm27vwvzGbqHjufyfGdZ04vyxpXmnQ2TzQUJBFdnAUAi2sHkc0XEKE4jEcbwilKc6gALJFmOoI+bN/9A4w3EIB/PHKGf/+NpzhxYW5pC1wFUrl80QLwXQfgCEBi8XfvmXyBcDBAImqlfmo7CEVpDhWAJdLRRD+gR46Osb0vDsD4bH0XkGMhjDU4rhWwXEBWDMBvKqZjMTkWwKJcQDlDJBggEXEEQC0ARWkGFYAl4gyFaTQVLJXN89jxcW6+YhOdsRDjc/UtgKl56/VGlkIrkM5ZQeCOaMi3C6gUA1iCAOQLhINCIqKjORVlMagALJFOnxbA4ZOTpLIFbtw7wEBHtKEATNoC4Dy2Mk4QOBEN+g4CV8UAFnH3ns0XiIRKFoAGgRWlOXwLgIgEReQpEfmuvb1bRB4XkWMi8k0Ridj7o/b2Mfv1XSuz9Nagw+dMgIePjhEOCjdc0k9/MtLQBTQxl7Uf14MA5ImFglYMoFkLYAk5/E4MIBwMEAkGNAisKE3SjAXwSeAF1/bngM8bYy4FJoE77f13ApP2/s/bx21YnBhAowvfwy9f4LqdvSSjIfqSkYaunfXkAirVAYRIZf0NhndiAF1LcgFZMQCAeCSoU8EUpUl8CYCIbAN+GfiyvS3ATcC99iH3ALfaz2+xt7Ffv1k2cIvG4ljIOhbA+ZkUL5y5yI17BwHob8IFtB7qBVLZAtFQgKSTjePjYp6urANYjAsoZ1kAYE8FUwtAUZrCrwXwBeBTgHNr1w9MGWOcq94IMGw/HwZOAdivT9vHb0iKLqA6MYBHj1npn++4zBKAgY4IE3NpCgVT8z2T85YLqJFQtALpXJ6obQEAvuIA6Yo00MXWAYRD1r1FIhrSGICiNElDARCRXwHOG2MOL+cHi8hBETkkIofGxsaW89SrSjwcJCD1YwCPvHyBvmSEK7dYw2T6kxEKBqYWvHsIFQqm6AJaHzEAuxLYtgD8xAEyFUHgpcQAgKbaUCiKYuHHAngr8AEROQH8LZbr54tAj4g48wS2AaP281FgO4D9ejcwXnlSY8zdxpgDxpgDg4ODS/pHrCUiYrWErmMBPHf6Ivu39xAIWHerfR1RoHYtwEwqh2McTK4LASjFAMC/BRAJBoiHnQye5nsBZXLlAuA3A0lRFIuGAmCM+QNjzDZjzC7gNuAHxpiPAj8EPmQfdgdwn/38fnsb+/UfGGNq+zo2AJ2xcM0YgDGG0amFsmliA0nL713LvTNh3/0Pdlqxglb++nL5ArmCsbKAmrQAoqEAwYAQCQUWXQcQKQpAiPmsWgCK0gxLqQP4T8DvicgxLB//V+z9XwH67f2/B3x6aUtsfSwLwNudc3Ehx2w6x3BPvLivv2gBeAuAEwDeM5gknSu0dIVrynblOK0gwF9LhnQuTyRUuntfXAzAEA5K8RxqAShKczQ1EtIY8yPgR/bz48D1HsekgH+zDGtbN3TEareEHpmaB2C41y0AjgXg7QJy/P+XDnXw2PEJJuYyJKOtOb3TuXDHwiULYNbHhdixAMCKoyymj0+2IgbQykKpKK2IVgIvA8loqOZUsNHJBYAyC6A3EUGkdkdQpwhsz2AH0NqZQGmXBVCKAfixAApFCyAeDrKQXUQMIF8gHCq5gDQIrCjNoQKwDHRGQ8ymvF1AI7YAbHNZAMGA0JuI1Mzxnyq6gCwBaOVAcJkF4PTk8XEnblkAweJ7F1UHUBYDsM7RyvESRWk1VACWgXpZQKNTC8TCAfrswK9Df51q4Mn5DKGAsMMOHLeyBeAIQDQUJO505fRlAZRiAPHFxgBypRhAMhqyJov5qEJWFMVCBWAZ6IjVngo2OrnAcE+cymLo/o7aAjAxl6UnES7GClq5GtgZCB8LB4iE/PfkyeTLYwCLzgJynQO0IZyiNIMKwDJgtUHOk/eo7B2dWmC4N1G1vz8Z5UIdF1BvIkJHNEQkGGhpCyDtsgAAqyOonyygbOnivVgXkLsQrJSCqgKgKH5RAVgGOmO1G8KNTi2UBYAd+jsiNat8J20BEBH6khEmWrghXCrnxACcnjy1M6LclFkAi04DLbiawVn/DbQhnKL4RwVgGSh2BK3wfc9nckzMZcoCwA79yShT89niYHM3U/PZYpvkvmRtoVgNPvH1n/FXj75a8/WSC8i6A09Gg74Gs7gtgHh4sYVgpqwZHOBLfBRFsVABWAZqzQQY9cgAcnD8+14ZPhNzmWLQuL8jUqwMXm0KBcODz5/jEdcs40rcWUDgPx3TsgCs9ywmBpAvGPIF46oD8NeWW1GUEioAy4BTpFVZCzAyVV0D4NBvX+ArawGMMbYFYL1upYuujQBMzGfI5Aucnk7VPMYdBAbLAvBTkJXOlrKAYpHmYwCO5VTsBqpTwRSlaVQAloHiWMgaFsCwpwVgtYOovLjPZ/Jk8gV63S6gNYoBnJmyLvynbSHzomgBhFwWgA8XUGUWUDpXqNse2+v9QDEGoEFgRWkeFYBloNZMgNGpBUIBYagzVvWeWu0gHEHodVxAyQgz6VxxgtZqcnrauvBPL2RrZvaUgsB2DMBnS4byGECw7Fx+yNoVyGENAivKolEBWAY6algAI5MLbO2JEwxUD0QbSFoWQKULaMoeBNNru4D6irEC70rjleSsy/VzesrbDZS2XUDO3XwiGvKXBuqOASzCfZPNW9aCBoEVZfGoACwDnVHLXVMZAxidnPf0/wN0xUOEAlI1E8DpBOq4gJxYwVrEARwLAODMtLcbKGVX9DqzDjqijdNAjTFkcuV1ANDcUJhiDMCuBC6KyCKyiRSlXVEBWAaK/mcPF5CX/x8o5fhXXNiLAmBf+PuS3rGC1eDMVKp4ka4VB0hnC8RCpZ9RImJl9HgVxTk4/vtopQuoiYt3MQZgnyMSDBAKiK/4g6IoFioAy0DInmzljgFkcgXOz6RrWgBgBYIrXUBOWmjRBZS0LIFaraNXkjPTC1y1tQuR2i4gZxqYg5+ZAE4H0UoBaKadc8kCsM4hIsS1JbSiNIUKwDKRjIbKpoKdmV7AGO8MIIeBjkjVhX1yPotIaVbumloA0yl29CUY7IjWdgFl80TDLgsg2vhinqkUgMXEAHKWheFkAYElPouZK6Ao7YoKwDLRGSvvCOrVBrqSPo+OoJPzGbrj4WLguCceJiCrLwCFguHcxRRbeuJs6YlzpkYtQCpbKKaAQskCqOeKcSyApcQAMsU6gHL3k6aBKop/VACWiY6KmQDFKuCe6kZwDv3JqEcMIFt0/wAE7NkBq90Q7sJsmmzesKU7xtbuGKM1YgCpXLkLyCnI8mcBBMve00wMoDIIDJb1oYVgiuIfFYBlonImwMjUAiKwubu6BsChvyPCbDpXduGbms8U+wA59CUjqz4Uxrnj39IdZ2tPnDNTKc9hK1YMwOWGqdEXyY1T01DVynkRAuB2ASXC/orQFEWxUAFYJjpi5TGA0ckFNnXGihc5L5wUT/fd/cRchr5E+fCYvuTqWwCOz39Ld4wt3TEWsnmmF6prEVLZgqcFUK8nT+0YgP9hLpVBYLAtAE0DVRTfNBQAEYmJyBMi8oyIPCcif2zv3y0ij4vIMRH5pohE7P1Re/uY/fqulf0ntAadFRbA6NR83QAwlNpBuGsB3H2AHNaiI6iT9bOlO8ZWO5PJKxMolc0XXTng7oxa+0K8LDGAXHkhGDitqNUCUBS/+LEA0sBNxph9wDXAe0TkBuBzwOeNMZcCk8Cd9vF3ApP2/s/bx214OmLlF5/RqYW6AWBwt4MoXdytWQDVLqDVFoAz0wtEQ9Yoyy22G8srEyiTK5S5gBLRxmmglTGApdUBlGIA8UU0lVOUdqahABiLWXszbP8Z4CbgXnv/PcCt9vNb7G3s12+WynmIG5AOOw10Yi7D8bFZzkyl6tYAQKkdhJMJlMrmmc/ki0VgDv3JCJPzmbrFVcvNmekUW7pjiIjLAqgWgOo6gMYtGSpjAOGgEAxIk2mg1S6gpGYBKUpT+IoBiEhQRJ4GzgMPAq8AU8YY5zZvBBi2nw8DpwDs16eB/uVcdCvSFQ+TKxiu/ZMHuen/+zG5gmH3QLLue4a6okRCAZ45NQVU9wFy6EtGMAZPH/xKYQmAdeEf7IgSDopnW+hUpQXgoxCsMgYgIk3PBPCKAcQjIbUAFKUJQn4OMsbkgWtEpAf4NnDFUj9YRA4CBwF27Nix1NOtOR/cP0zBGBLhIN2JML2JCG/ZM1D3PbFwkPddtZl/eGqUP3jfFVV9gBz6iq2j08VBMSvNmakFbrjE0u1AQNjUFeNMLQvAFQOIhAKEg1L3TrwyBgD2XOAlCkAyEiSTL5B1zQpWFKU2vgTAwRgzJSI/BN4M9IhIyL7L3waM2oeNAtuBEREJAd3AuMe57gbuBjhw4MDq+TZWiKGuGL/9C5c2/b6PvGkn//D0ab77zJlizMDLBQSWq+jSoaWvtRH5guHcTJotPaUU1q3d8SoLwBhT5QICywqY91EIFg25794DpJq4e8/kqyuB464ahO64CoCiNMJPFtCgfeePiMSBdwMvAD8EPmQfdgdwn/38fnsb+/UfGK8EcgWAN+7q5dKhDv7mideYrOECcrZXKxA8NpMmXzBFFxDAlp5YVQwgmzcUTPmFHKw78dkmsoCg+bGQlRPBoFSDoO0gFMUffm6TtgA/FJEjwJPAg8aY7wL/Cfg9ETmG5eP/in38V4B+e//vAZ9e/mVvHESEj1y/g6dPTfGvr1izdytdQF7ZQiuJ0wZ6q9sC6Ilz7mKqbGpX5TAYh0YzASqzgGARApDzKASLeHdlVRTFm4YuIGPMEWC/x/7jwPUe+1PAv1mW1bUJH7x2mM9+70X+7vAIgGcdQGc0xP/48StcNdzNNdt7VnQ9ziCYzV0lC2Brd4xs3nBhNs1QlyUMpYHwFRZANNQgBmC9Fq2MATTZDVSEsmE7pT5EGghWFD+oo7QF6ElE+JWrt5DJFeiIhqqqh8PBAH/18esxBj50179y98OvNDU/t1kcV4/bAnDcQe44QHEaWIUFkIwE68YAMh537/FIsMk6AEM4GMCdYeynDQXA06emuPqPvs/5mdrD7hWlHVABaBE+8iYrE6qyD5DDdTt7eeB3buRdr9vEf33gRX71rn/lKz95lZHJ+WVfy5npFPFwsNiSGigGhN1xgJIFUB0EbpQFFA5KcYoYWC6gZucBRCoyfYpVyA3Oc+jEBDOpXLFhn6K0KyoALcJ1O3u5fFMnQ51ucT2bAAAchUlEQVTRmsd0J8Lc9RvX8pkPXk0qk+dPvvs8b/vcD7nlzx/lYqq6RuAL//Iyt9/9WNNrOTO9UCwCcxj2KAZL2RZArDIIHA02jAG4/f+wuCCwuxMolGYRNLIATo5boqk1A0q701QaqLJyiAh/8bED5Ar1G6KJCLdfv4Pbr9/Bqxfm+PpjJ/nyT17l2ZFp3nJped3BY8fHeez4hNVgron6gTPTqbIUULAG1MTDwbK5AOlaQeBI/bnAaXuOsJtYky4gr1x/xwKYbSQAE5YA6PQwpd1RC6CF2NGf4JLBDt/H7x5I8ptv2w3AifFqV5Bzp3v45GRT6zgzlSpLAQVLeLb0xMr6ARUtAI8YQL27cMsCKP/pxZsMAmdypkoA/KaBnhyfs47TzqFKm6MCsM7Z0mW1nHYuag6pbL54t37o5ITv8+XyBc7PpNjqMcdga3ecUVdH0FpZQB2xEAvZfDHYW0k6V6iyABwXkN+SkUy++hwJW4jq1SBk84Wi739hheoFHjk6xrHzs40PVJQ1RgVgnRMICDv7EpyoEIDXJkoWweET/i2A8zNpCgY2d1c3stvaU94OolYdwLZeawparQC1pwUQCVIwpS6fjcjmqmMAgYBYYyHrWB+npxbI2RlUK+UC+tS9R/gfP35lRc6tKMuJCsAGYGd/sujucXC2r9/Vx5HR6aK/vhFPnrCsBa9W1sM9Cc7PpIt3/qUgcLkA7OpPlK2hkloWAEDK51CYWv1+kg2K0NxrWikBmEnlmE1pMZrS+qgAbAB29VsWgNt94riEfu26YTK5As+OTjc8z7mLKf7L/c/xhm3dvHlPdQPXnf3ld/aOEEQrXEC77C6olVaJg2cWUKS5oTCZWgLQoA2F21W2EllAxhjmMzmNLyjrAhWADcDOgSSpbIHzM6XJYifG5+iOh7npik0AHGrgBioUDL//rWdIZwt84cPXeF5cHQE4caFcACotgP5khI5oiBMXvAUgnctX5fA3OxfYqw4A7CrkOi6gk+PzxMIBEpHm6g78ks4VKJiViy8oynKiArABcFwur7ouuCfH59nZn2CwM8qu/gSHGmQCffXRV/nJsQv84fuvrJmJtLPfurN30iiLXT0rLAARYWd/wjMzCWwLoOI9xbGQPi/K2bzxnLfcSABOjM+zoy9BIhJiIbv8F2lHVLQdhbIeUAHYAOxyLszjlQJg7b9uZx8/OzlZM8Pm+dMX+dPvvcS7r9zEbW/cXvNzehNhOmOh4ueksnlEqruBguUGqsxMckjnqu/em3UBeRWCgVULUG8g/WsTc+zsT66YBeCIjw6nV9YDKgAbgC3dMcJBKd5xZ3IFRibni5bBgV29jM9lyiwEN1/4l5fpiIX43K+9gXrTO507eyeQag2ED3i+Z1d/glOTC8W2zW4sC6C6Etg5px8yOe8YgJUF5H2OQsFYwtiXWDEBcC782pFUWQ+oAGwAQsEA23sTxTvu0akFCgZ29NkCsLMXwNMNNL2Q5UcvjXHrNcO+qoWtjCPrc9K5QlUKqPu4fMF49tvxtACadgEVCHtYHh11XEDnZ9KkcwV2DiSbbj7nl6IFoFXGyjpABWCDsLM/UQzOOtk3TjbOnsEOuuNhz3qA7z93lky+wPv3bfH1Obv6E4xMLpDLF6rGQbpx5iG/6uEGSnvEAOIRa9u/C8g0HQR2hGtFLQD7nPNNFLUpylqhArBBcHzuxhhes100TtZOICBct7PXsyL4O8+cZkdfwveMgZ19SXIFw+mpFKlsoaoKuHicUwvg4XbyygJyLAm/07xqxQCSkSBzmbxnu2zHdbWrP0k8HFqZGIB9znzBFIPkitKqqABsEHb1J5nL5Lkwm+HE+ByJSJDBjlJn0et29vLK2FzZWMkLs2kePXaB9+/bUtf376Z4YZ+Y85wH7DDYESUZCXpmAnllAQ11xkhEgvzcR70C1C8EA29L4uTEHKGAsLXH+qyVSNV0C5i6gZRWRwVgg1C8MI/PcdJOdXRf1N9x2SAAX/yXl4v7Hvj5GQoGPrBvuInPcYq85kl5BHMdrIBxsqoYzBjrzjhacfGOhAK8+ZJ+Hjl6wdc60jWCwPWGwpwYn2dbb5xQcOXqANznrJeNpCitgArABsFJBX31whwnxueK2w5XDXfz8bfu5p6fnuRHL50H4P6nT3PZpg4u39zp+3OGOqPEwgFeG7ctAI9AbHFNA4mqdhDZvOWa8RKOG/cOcHJ8vmb6aPl5qttJgDWLALxbQr82Ps8O+3uJR5rrPuoXtwCoBaC0OioAG4Th3jjBgHD8whwjEwvsHEhUHfOp91zOZZs6+I/3HuHZ0WkOnZzkA/u2NvU5gYCwo88q8kpn8zUtALBE6dTEPDlXKqjTk8grgHujbaX4sQKyeVMjBuA9F9gYYwuj9b0kIsEVCdS6R2HqvAGl1WkoACKyXUR+KCLPi8hzIvJJe3+fiDwoIkftx157v4jIl0TkmIgcEZFrV/ofoVhzg7f1xnns+DiZfIGdfcmqY2LhIF/48H6m57N89MuPA/D+JgUALDfQa+PzVhC4ngXQbwWMR10dRDM1qocBLhlIMtwT5+GXx+p+fr5gyBeq5wGAeyxkuQUwNZ9lJpUrpsYmIiHyBeO7+6hf3D2A1AWktDp+LIAc8PvGmCuBG4BPiMiVwKeBh4wxe4GH7G2A9wJ77b+DwF3LvmrFk539SZ45NQWU2kNUcuXWLv7jL13O9EKWfdt7ij79pj6nL8HJiTkW6gSBrfXYvYNcbqC0x0B4BxHh7ZcN8NNXxj0LyByc15qJATixCOff22zdgV/cFoC6gJRWp6EAGGPOGGN+Zj+fAV4AhoFbgHvsw+4BbrWf3wJ8zVg8BvSIiL8kc2VJ7OpP4GQ/7hyofWG/8227+T/fsovfvXnvoj7HaT53ZnqhZhoolGoB3E3h6lkAADfuHWQmnSsKmReOAHi1oKgVA3DmI7hdQLD8bpryILAKgNLaNBUDEJFdwH7gcWCTMeaM/dJZYJP9fBg45XrbiL1PWWGcu9tIKMCWruqJXg6BgPBHH3g977xiaHGfY7tRsnlT1wIY7IwSDwfLMoFKFoD3+966Z4CAUNcN5ASS61kAlRd2p0huu732+AoKQMm6UBeQ0tr4FgAR6QD+F/C7xpiL7teMFUlrKpomIgdF5JCIHBobq+/zVfyx2w78bu+NEwj4y+tfDO4Mo3oCUNk7CFwWQI3YQXcizL7tPTxcJxC8GBfQyYk5NnfFiutN2MHiZXcBZXIMdEbs52oBKK2NLwEQkTDWxf/rxpi/t3efc1w79uN5e/8o4G4puc3eV4Yx5m5jzAFjzIHBwcHFrl9x4VgAlSmgy83WnhghW2DqBYHBcgO5XUDFLKA673v73kGOjEwxNZ/xfN0REa8soNJc4HIBGJtJs9k15zjeZOWxX+YyeQbsAjwVAKXV8ZMFJMBXgBeMMf/d9dL9wB328zuA+1z7P2ZnA90ATLtcRcoKsq03TjgoRd/7ShGyM47AO5/fzc7+JKcmS6mgjSwAgLdfNkDBwKPHxj1fdywALxEJBQPEwoGqi+/4bIZ+V7O7ogtomRvCLWTy9MTDBAOy7OKiKMuNHwvgrcD/AdwkIk/bf+8DPgu8W0SOAu+ytwEeAI4Dx4C/AH57+ZeteBENBbnn49fzW+/Ys+Kf5Vgb9VxAYLmlsnnDmekU4IoB1BGAfdt66IyFasYB6sUAwEoFrbQAJuYyZd1OnSDwcruA5jI5EtEQiXDtttSK0iqEGh1gjPkJUMuhfLPH8Qb4xBLXpSySt+wZWJXPcVI862UBWceVKpS39yVKU8RqdBEF6y7+Tbv7iwPqK6kXAwDLv++OARhjLAHoqBaA5XbTLGTyJMJBEtGVqTRWlOVEK4GVRVG0AOpcyAEuHbLGS750dgbwFwMAuHZnD8cvzDE5Vx0HyORrxwDAaQlduvjOpnNk8gVPF9ByZ+rMpXMkoyESkZAOhldaHhUAZVE4qaC18vkdBjqiDPfEeXrEyuv3EwMAiu2pnfe5ydQpJgPoiAbLLACnA2pfstQdtZgFtNwxgGyeeCRotZrQqWBKi6MCoCyKfdt72DOY5AofjeT2be/miH0hT/sUgDds6yEg8NRr1QJQdAHVOEciUj4XeNwWgDILIOzPBXTWjl34IZMrkM0bko4AqAtIaXFUAJRFMdgZ5aHf/wUuHfIhANt6ODWxwPhs2mUB1HcddURDXLapk6deq55i1igGUDkWcmLWsQBKAhAMCNFQoK6f/qnXJrnhMw8VxasRzrnikRDxSEizgJSWRwVAWXH22e6cIyPTRf99oxgAwP4dvTxzaqpqulcm52QB1YoBlGfgjM+lAapmHje6S3faR/idUeBYHclIkKRaAMo6QAVAWXGuGu5GBJ4ZmSKdbUYAeriYynG8YqxkvV5AUD0XuOgC6qgUgPpjIafmswA88ap3NlIl80ULIEhcBUBZB6gAKCtORzTE3qEOnjk1RSafJxQQgj5aVVy7w7IcKt1AjVxASTsG4PT6n5jNEAsHioFfh3gkyEK2tpvGEYDDJyfJe8wYrmS+aAGESKoLSFkHqAAoq8K+bT08MzJNOltoGAB2uGSgg85YiKcqOoM2FIBoiIKBlG1tTMxl6HdlADk0cgFN2q0oZtM5XjhzseZxDs65EhoEVtYJKgDKqrBvew8TcxmOX5jz5f4Bq2vpNdt7qjKBMg0rgcv7AY1XVAE7xMP1L9LTC1mSdr3A4z7cQM4dfyIaIh4Jks4VfFkOirJWqAAoq8K+bZY758kTEw0zgNzs39HLS2cvlvn0sw3qABKR8o6glW0gSsfVr9adms9wyWAH2/viPOlLAEoWgDOaUt1ASiujAqCsCpdv7iQSCjCTyvm2AMAKBBeMlUHkUKoDqF0JDKWsHMsF5CUA9f30k/NZehJh3rirjydPTDScHzyfLglAfIV6DSnKcqICoKwKkVCA12/tAhoXgbm5xrYcnjpVCgT7qQOA0mD48bm0twuogQUwvZClJxHhTbv7GJ/L8MrYXM1jweUCioSKk8l0KpjSyqgAKKuG4wZqxgLoTUbYPZAsiwM4MYBQjUyihHPxTeeYz+RIZQtljeCKx0WCdfv1TM5n6IlbFgBQszmdw5zLBRQPqwtIaX1UAJRVY9/2bqA5CwBgvx0Idlww2XyBSDCANaqimg6XC2h8troNhEO9IHChYJheyNKbCLN7IMlAR6RhPcBCJk9ArH/fSnUbVZTlRAVAWTUWYwGAlUF0YTbN2YtWX55MrlCzChjKx0J6NYJziEeCZGpk6sykchgD3YkIIsL1u/saCsBcJkcyEkJEii4gFQCllVEBUFaNXf1JumKhprKAAPZuslpKv3Le8sFn84WajeCAYurmbDrvEgBvFxB4u2mcGoCeeBiAN+7qY3RqgdGphZqfu5DJF4O/jgtIB8MrrYwKgLJqBALC/3XjJfzi6zc19b5LB20BGJsFbAGoEQCGkgUwn855dgJ1iNcZDD+1YFUB9yYtAbh+tx0HqGMFzGXyxc8uBoF1KpjSwjScCKYoy8nv3Ly36fcMdkbpjIaKApDJmZo1AGBlB0VCAWYzOSacRnBeQeA6LaGdgfTdcet9V2zuIhkJ8vSpKW7dP+z5uQuZXLHN9ErNHFaU5UQFQGl5RIRLhjrKLIBGcQR3S+hwUOiMVv/Ui3OBPS7STh+g3oRlAQQDwnBvnDPTtV1Ac+l88c7fKUbToTBKK6MuIGVdsGcwWR4DqBMEBjvFM51nYtaqAvbKGIrXydRxLICeRMly2NQV49zFdM3PnM/mi24lvwNnFGUtaSgAIvJVETkvIs+69vWJyIMictR+7LX3i4h8SUSOicgREbl2JRevtA97Bjs4ezHFbDrXMAYAlgUwa2cBeWUAgWsspMdFetK2ALpiJcthqDPGuYu1J4TNp3PFAHQwIMTCgWUfOakoy4kfC+CvgPdU7Ps08JAxZi/wkL0N8F5gr/13ELhreZaptDt77EDw8bFZMnnTUACSUasl9HiNNhBQPwtoeiFLVyxEyPU5m7qinJ9JVw2ocZh3ZQGB3ZZaXUBKC9NQAIwxDwOVqQ+3APfYz+8BbnXt/5qxeAzoEZEty7VYpX25dCgJWJlA2VyhbhAYrIv7nJ0G6pUCCiUXkHcMIFPm/gHLBZQvmGJmUSXzdh2A+/zaC0hpZRYbA9hkjDljPz8LOHl9w8Ap13Ej9j5FWRI7+pIEA8Ir5+fsOoD6MQAnCFxPAOpV607OZ4sBYIdNXTGAmm6g+Uy+eE7n/HNaB6C0MEsOAhurPr/ppuciclBEDonIobGxsaUuQ9ngREIBdvYlLAvARwwgGQ0xOZ9hNp2r7QIq9uvxrgPorrIArFjC+ZlqAcgXDOlcoWzqWKORk4qy1ixWAM45rh378by9fxTY7jpum72vCmPM3caYA8aYA4ODg4tchtJOXDJopYL6igFEglyw+wB51QCAywXkcZc+ZTeCc1OyAKozgUqdQMstAHUBKa3MYgXgfuAO+/kdwH2u/R+zs4FuAKZdriJFWRJ7hpKcuDBPKptvGANIuvL+a1kAkVCAUEBqpIFWu4AGOy0L4Ox0tQVQHAYTdQtASNtBKy2NnzTQbwA/BS4XkRERuRP4LPBuETkKvMveBngAOA4cA/4C+O0VWbXSluwZ7CCTL3BqYr5hHYBbAGqlgYJlBVQKQL5guJiqdgGFgwEGOiKeLiD3NDAHywLQGIDSujSsBDbG3F7jpZs9jjXAJ5a6KEXxwkkFzRX8uYAcagWBwdtNc3EhizFUuYCgdjGYk+5ZHgMIqgWgtDRaCaysG/YMJovPG7WC8OMCAnsmQEUaaGUjODeWAFRbAE4qabkFENIYgNLSqAAo64aeRIQBO6DrpxIYrIrcbo87eYd4JFTlpim2gYhXC8emrmhTFsB8JtdwlrCirBUqAMq64hLbDeTXAuhNhAnUGB0JzkW6wgKw20D0JKqFY6gzxvhcujiX2GHBKwYQDVIwkM6VH6sorYIKgLKucOIAjYPA1oW4nv8fagjAQnUjOIdNXTGMgbGZcivA8fW7K4HrtZtWlFZABUBZVzhxAD+FYNBYAOLh6iDw5JxtAXi4jjZ3WxlFlXEAx40Ur4gBANoPSGlZVACUdcWeIccCaJQFZF18++ukgIKdBeQRBBaBLg8BGOr0LgYrWgDRchcQePcaUpRWQAVAWVc44yH9FoI1tAA82jVMz2foioUJesQOnGrgyloA5xyxUHkdgPs1RWk1VACUdcW23jh/8N4reM9Vm+se1xENEQkG2Nwdq3ucV7HW5HzWMwAMVkppMCBV1cDz6RyJSLAs4KxTwZRWR0dCKusKEeG33rGn4XGRUIBv/ds3l9UOeJGIWHUAxpji1LCphaxnABiswfZDndWpoPPZ8k6gzrlBLQCldVEBUDYs12zvaXhMPBLE2KmaMTtrZ2o+Q28NAQDLDVTlAkrnymoAoCQA2hJaaVXUBaS0NV6pml6N4NxYxWDVMYBqC6D2yElFaQVUAJS2puind92le00Dc+PVD8hbANQFpLQ2KgBKW1OaCWBdpHP5AhdTubrtIzZ1xZheyJJypXfOZ7xcQNXioiithAqA0tZU3qVfTFkX63ouoKHO6mIwLwug3rwBRWkFVACUtiZeIQDFRnANXEBQXgzmJQDg3WpCUVoFFQClrSkGarPWnf+k3Qiuu44F4NQWlFsAORLR6qQ6ay6wuoCU1kQFQGlrKl1A03YjuLppoJ1eApAvZhRVnl8tAKVVUQFQ2pp4RRpovUZwDl3xENFQgPN2R9BCwVgC4GUBRFUAlNZFBUBpayqzgIrTwOpYACLCpq5YsR1EKlc9C8AhEVYXkNK6qAAobU2VC2g+gwh0xuoXybuLwebSdQRALQClhVmRVhAi8h7gi0AQ+LIx5rMr8TmKslSc7p33PT3K0XMz/Hx0mu54/SliYGUC/ezkJBdTWdc0MK8gcJCRSRUApTVZdgtARILAnwPvBa4EbheRK5f7cxRlOQgEhPfv20o6V+CJExPMpHLcdPlQw/e9Zc8Ap6dTvOUzP+D//eeXAG8LIB7WwfBK67ISFsD1wDFjzHEAEflb4Bbg+RX4LEVZMn92+/6m3/ORN+3g6uFu7n7kOP945DTg7TZKRoNML2T58ctj9MTDdMXDxQKxYEAIihCwnwcEBMFuSoq4th17xOlY6rZPRNzP61suiuJmJQRgGDjl2h4B3rQCn6Moa8rV27r5s9v386lfupxHjl7ghkv6q47Z2hNnNp3jjq8+sQYrrE+ZcNQ5zni9l2qxqSVKdddQ95N90gKat5pL+PIdB7hx7+CynGvN2kGLyEHgoL2ZFpFn12otLcYAcGGtF9EirKvv4qMre/p19V2sMG39Xbz9/ynbvHwp51oJARgFtru2t9n7yjDG3A3cDSAih4wxB1ZgLesO/S5K6HdRQr+LEvpdlBCRQ0t5/0qkgT4J7BWR3SISAW4D7l+Bz1EURVGWwLJbAMaYnIj8O+D7WGmgXzXGPLfcn6MoiqIsjRWJARhjHgAeaOItd6/EOtYp+l2U0O+ihH4XJfS7KLGk70KM8YrxK4qiKBsdbQWhKIrSpqy5AIjIe0TkJRE5JiKfXuv1rBYisl1Efigiz4vIcyLySXt/n4g8KCJH7cfetV7raiEiQRF5SkS+a2/vFpHH7d/GN+2kgg2PiPSIyL0i8qKIvCAib27X34WI/Af7/49nReQbIhJrp9+FiHxVRM670+Rr/RbE4kv293JERK5tdP41FYA2bxuRA37fGHMlcAPwCfvf/mngIWPMXuAhe7td+CTwgmv7c8DnjTGXApPAnWuyqtXni8D3jDFXAPuwvpO2+12IyDDwO8ABY8xVWEklt9Fev4u/At5Tsa/Wb+G9wF777yBwV6OTr7UFUGwbYYzJAE7biA2PMeaMMeZn9vMZrP/Jh7H+/ffYh90D3Lo2K1xdRGQb8MvAl+1tAW4C7rUPaYvvQkS6gbcDXwEwxmSMMVO06e8CK1ElLiIhIAGcoY1+F8aYh4GJit21fgu3AF8zFo8BPSKypd7511oAvNpGDK/RWtYMEdkF7AceBzYZY87YL50FNq3RslabLwCfAgr2dj8wZYxxmum3y29jNzAG/KXtDvuyiCRpw9+FMWYU+G/Aa1gX/mngMO35u3BT67fQ9PV0rQWg7RGRDuB/Ab9rjLnofs1YKVobPk1LRH4FOG+MObzWa2kBQsC1wF3GmP3AHBXunjb6XfRi3dXuBrYCSardIW3NUn8Lay0AvtpGbFREJIx18f+6Mebv7d3nHLPNfjy/VutbRd4KfEBETmC5AW/C8oP32KY/tM9vYwQYMcY8bm/fiyUI7fi7eBfwqjFmzBiTBf4e67fSjr8LN7V+C01fT9daANq2bYTt4/4K8IIx5r+7XrofuMN+fgdw32qvbbUxxvyBMWabMWYX1m/gB8aYjwI/BD5kH9Yu38VZ4JSIOE2+bsZqpd52vwss188NIpKw/39xvou2+11UUOu3cD/wMTsb6AZg2uUq8sYYs6Z/wPuAl4FXgP+81utZxX/327BMtyPA0/bf+7B83w8BR4F/AfrWeq2r/L38AvBd+/klwBPAMeDvgOhar2+VvoNrgEP2b+MfgN52/V0Afwy8CDwL/DUQbaffBfANrPhHFss6vLPWbwGrK/Wf29fSn2NlT9U9v1YCK4qitClr7QJSFEVR1ggVAEVRlDZFBUBRFKVNUQFQFEVpU1QAFEVR2hQVAEVRlDZFBUBRFKVNUQFQFEVpU/43kXJnLRn7/BEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss on validation set: 0.04799825516238343\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAO4AAADuCAYAAAA+7jsiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAACCFJREFUeJzt3TF240YSBmCIb6N5EyiZZJK9hHMcRwfAAUY6wBxKB9hDOJrADhkLjmCTXNFEkyigC/19iRM8GkPxZ1WjS62ncRw7IJfD1jcAlBNcSEhwISHBhYQEFxISXEhIcCEhwYWEBBcS+k/JxV++fBmfn5+j7gWa9+vXrz/Hcfx267qi4D4/P3cvLy/33xXwr15fX3+fc51WGRISXEhIcCEhwYWEBBcSElxISHAhIcGFhIoGMKAVwzB0X79+nXXt8Xjsfv78GXxH51Rc+MTc0JZeuxTBhYS0yrCwz9rspdtpwaVZU8CWDtVnrfPS7bRWmWZNYdpijfoowaVZx+Px7L+ZaJVp1tpbOEtScWFhn1Xwpau6igsLW6OSq7iQkIpLE0pHGI/HY9H1axNcmlA6wvj29hZ4N4/TKkNCggsJaZXhhsv18a0RSbPKUGCt2eNb62Wzyqyu7/uu7/utb+MumWePS6m4nHl/f9/6Fu42beFknD0uJbjsRtTE0uWebg1fDIILN5R+IXw2vGFWGSpnVhn4lIpLSnubPS4luKS0t9njUlplSEhwISGtMk06XSNH/AmR6NcXXKoVNXvcdedr5IgRyejX1ypTrZZmj0sJLtWKPPf49DUzbP9c0ipTrcgJpOjpptN944gvBsGFANFfDFplSEhwISHBJaWSdWPGh0+3WONSjbm/OBCxr5uNiks15u7X2tcVXEhJq8wuRM8G10bFZVVRR79GzwbfYxiGbhiGkNcWXEhIq8yqos5tjh4xvEdkuy647MLe17SXtMqQkOBCQoJLNeauTWtZw27JGpdqtLZOfYTgEmrO/HELAxNL0yoTas4wRC0DE5mouFTnWpVWmf8huFTnWgXOVJmjZ6e1yjwkavY4O+cqA/9HcHlI1OxxdtHnNlvjUp1rf8s20+BF9EM0waU6nhzfplUm1JwqmamS1kLFJZTqGUPFhYQEFxLSKhPq1i8ZGGO8j4pLqFtTQ5nGGGui4nK31s4yLjG9N1Hvi4rLmb7vZ88f13iWcS2m9yPqfVFxOWOEcRnT9FfUHrXgcrcazzKuhZFHqmVNux1rXELdqsQq9X1UXEKpyjFUXEhIxWW2OUetTuzrxlJxma1kT9K+bizBhYQEFxISXM4Mw9ANw7D1bXCDh1OciV6bTl8KLTy4mma+I8ZIBZczBiKWEzn3Lbicia6ELVTaNVjjQkKCCwkJLiRkjduw0hHGa38a5Nr1xBHchpWOML69vQXeDSW0ypCQ4EJCggsJCS4PMdu8DQ+neIjfu92G4PIQ2z7bEFweYvZ4G9a4kJDgQkKCCwlZ4+6I2eN2CO6OmD1uh1YZEhJcSEirzN9O18j+hEjdBDeZKVwRwTpdI89dL7d03GpNtMrJTIEyI9w2FTeZaQsnYnvmdHto7uurtNsQ3GQigyKEeWiVISHBhYQEFxKyxq1Yyexx13Xdx8dHdzjM+y42e5yb4FasdMvncDiYP26EVhkSElxISKu8Y2aP90vF3bF7Z49Lzkl2rvI2BBcS0irv2Bqzx9rvbQjujgnVfmmVISHBhYQEt2KlY4nGGNthjbui0tnj4/FohJFPqbgrKp09djwN1wguJCS4kJA17oIu17Dmg4myWsXt+z7t9XOvvVyTRq1RI/+t0bPKNf1co0Xei1YZEnoax3H2xd+/fx9fXl4Cbye3W63yjx8/il/TdlBbXl9f/zeO42+3rrPGXZD1LGvRKkNCggsJCe6KzB6zFGvcB5XMH5s9Zikq7oNK9mrNHrMUwYWEBBcSElxIqMmHU9MM6fv7+6zrp1nciAGL6eFWLb+QEP3e9H0/+7Wzi/zcNBncVj449/De5NBkcEtFVsIaquwjSu+/pS+GyJ+tNS4kJLiQkOBCQoL7oJJ5YrPHLMXDqQuls8fZHy6Rk4p7wewxGQguJCS4kFBza9zTNaw1KlntouL2fX/X2ceR5x7XdL5vicz3XpvSM6dL7KLilozRHY/Hs4q79f3UJvO9t2QXwS2hNWYtZpWBM4ILCQkuJLT7NW7pCOPpw6s518MWdh/c0hFG5x6TgVYZEhJcSGj3rfKjjEhSo/TBjQ7WGiOS7JPjWf+FYC2r9Fxlrov8PKYP7hqzxy0R2OVEfh7TBzd6zemLgXtFfjbTBzeah1HUyHYQJCS4kFC6Vrlk9rjruu7j46M7HOZ9P1nDkkW64JY+Yj8cDuaP2R2tMiRUVcW91gYbNYRzVVXca22wiSg4t0rFnSqpykkroj/zq1TcqWKqnPVzrvIyoj/zq1TcaWzQdkv9zCovI/ozv0pwtce0JvozX9XDqWvfTio1nKtqO0hlhnmqqrjAPCEVt/Qs466b9/St9Nzj09eHPQkJbulZxiXXmjsGrTKkJLiQUFVPlWEPLp/xRIw97qbiGtPbp4w/18vnNhFjj7sJLsswq3xdTe/NblplM7bLqO19rOl+aroXFRcWdjk7EDFLsJuKC7VYY3RXxYWEQoJb0hpMY4xLvy7sWUir7Ld8IJZWGRLycAoWtNYRwyouLGitI4bDK+4ac5vQmvCKe8/c5jAM3TAMRf+fklG00te/536yqmmsr+vKZ5Ujr6/pc2CNy5maxvq4Ljy4l0fNzNmLvaeVLvnAlb6+1n47pV8kkdfX9DkID25N/1iIdu1MtKWHh7TKsKC1CpXtIEhIcCEhwYWEBBcSElxISHAhIcGFhAQXEnoax3H+xU9Pf3Rd93vc7UDz/juO47dbFxUFF6iDVhkSElxISHAhIcGFhAQXEhJcSEhwISHBhYQEFxL6C59KEwQFaZwpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dcbadcbadcbadcba\n"
     ]
    }
   ],
   "source": [
    "ENC_RNN_NUM_OF_LAYERS = 1\n",
    "DEC_RNN_NUM_OF_LAYERS = 1\n",
    "EMBEDDINGS_SIZE = 4\n",
    "ENC_STATE_SIZE = 32\n",
    "DEC_STATE_SIZE = 32\n",
    "\n",
    "att = AttentionNetworkWithPrint(ENC_RNN_NUM_OF_LAYERS, DEC_RNN_NUM_OF_LAYERS, EMBEDDINGS_SIZE, ENC_STATE_SIZE, DEC_STATE_SIZE)\n",
    "train(att, train_set, val_set, epochs=6)\n",
    "\n",
    "att.att_mat = []\n",
    "print(att.generate_and_plot_attention('abcdabcdabcdabcd'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When plotting the attention weights of each input stage we can see that for we get a diagonal line. When generation stage $i$ the most important input was $|input|-i$\n",
    "\n",
    "In other words, the attention model learned pretty well the structure of the \"reverse\" function on our test sample by looking at 3,000 sample strings.\n",
    "\n",
    "We recommend to read the following book for an extended introduction to NLP using neural networks: http://www.kyunghyuncho.me/home/blog/lecturenotefornlpwithdistributedreponarxivnow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "About me: https://www.cs.bgu.ac.il/~talbau/"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}