pytorch_introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Introduction to\n",
    "![](pytorch-logo-dark.svg)\n",
    "## Thomas Viehmann, MathInf GmbH\n",
    "## [3rd Advanced Course on Data Science and Machine Learning](https://acdl2020.icas.xyz/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "These slides are (c) 2020 by MathInf GmbH except graphics with attribution (and the PyTorch logo above).\n",
    "\n",
    "Please use what you learn but don't just take my slides without asking. Thank you!\n",
    "\n",
    "Thanks to: Giuseppe Nicosia for the invitation and virtual hospitability. Piotr Bialecki for feedback on the draft of these slides and many interesting perspectives on PyTorch and teaching PyTorch. The participants of the ACDL2020 for their interest and questions. Luca Antiga made the autograd graphic for our book."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Device 66af'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib import pyplot\n",
    "import numpy\n",
    "\n",
    "torch._C._jit_set_profiling_executor(False)\n",
    "torch._C._jit_set_profiling_mode(False)\n",
    "torch._C._jit_override_can_fuse_on_gpu(True)\n",
    "torch._C._jit_set_texpr_fuser_enabled(False)\n",
    "\n",
    "assert torch.cuda.is_available(), \"Some examples need the GPU\"\n",
    "torch.cuda.get_device_name()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Hello\n",
    "\n",
    "I'm Thomas Viehmann.\n",
    "\n",
    "I founded MathInf GmbH, a boutique research and development consultancy & PyTorch training company\n",
    "\n",
    "<img src=\"deep-learning-thumbnail.png\"  style=\"width: 200px; float: right;\" />\n",
    "\n",
    "\n",
    "I'm a PyTorch core developer, and with ~140 patches one of the most prolific independent contributors to PyTorch\n",
    "\n",
    "With Eli Stevens and Luca Antiga, I wrote [Deep Learning with PyTorch](https://pytorch.org/deep-learning-with-pytorch), currently available for free download from PyTorch.org.\n",
    "\n",
    "\n",
    "Background: PhD in pen and paper Mathematics, been an Actuary & Consultant for 10 years, Debian Developer emeritus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Plan\n",
    "\n",
    "Conundrum: Introduction to PyTorch @ Advanced Course on Data Science and Machine Learning\n",
    "\n",
    "We take a tour of PyTorch, nominally from 0, but we put best practices and pitfalls into the spotlight.\n",
    "- Tensors\n",
    "- Functions\n",
    "- Autograd\n",
    "- Models\n",
    "- Optimizers / Data\n",
    "- Excursion: Efficiency and Benchmarking\n",
    "- Beyond Python / JIT\n",
    "- All of this is in Jupyter: https://github.com/t-vi/acdl2020/ has the notebook\n",
    "\n",
    "Your favourite topic?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# PyTorch is ... Tensors\n",
    "\n",
    "At the heart of PyTorch is the Tensor data type. It is a multidimensional array.\n",
    "\n",
    "When we speak of the dimension of tensors, it is the number of axes (or, in PyTorch, dimensions) - so dimension is in the sense of arrays, not the mathematical one.\n",
    "\n",
    "Tensors can be 0d (i.e. scalars - single numbers), 1d (vectors), 2d (matrices), 3d..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.7071, 1.4142, 1.6180],\n",
       "        [0.8346, 2.7183, 3.1416]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t = torch.tensor([[0.7071, 1.4142, 1.6180],\n",
    "                  [0.8346, 2.7183, 3.1416]])\n",
    "t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Tensors are arrays\n",
    "Tensors have a shape (the size of each dimension):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, torch.Size([2, 3]))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.dim(), t.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indexing - also with assignments - works as expected. Use .item() to get numbers from tensors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(3.1416)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t[1, 2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[0.7071, 1.4142, 1.6180],\n",
       "         [0.8346, 2.7183, 3.1416]]),\n",
       " 3.1415927410125732)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t[1, 2] = 3.141592653589793\n",
    "t, t[1, 2].item()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\"Advanced indexing\" with masks also works"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.0000, 1.4142, 1.6180],\n",
       "        [0.0000, 2.7183, 3.1416]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t[t < 1] = 0\n",
    "t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "so does slicing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[1.4142, 1.6180],\n",
       "         [2.7183, 3.1416]]),\n",
       " tensor([[0.0000, 1.6180],\n",
       "         [0.0000, 3.1416]]),\n",
       " tensor([3.1416]))"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t[:, 1:], t[:, ::2], t[1, 2, None]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Tensors always hold elements (numbers) of a given scalar type, the \"data type\" or dtype.\n",
    "For PyTorch (with its GPU focus), `float32` is perhaps the most common dtype.\n",
    "Apropos GPU, they do have a device."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.float32, device(type='cpu'))"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.dtype, t.device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can move between devices and convert between dtypes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.0000, 1.4142, 1.6180],\n",
       "        [0.0000, 2.7183, 3.1416]], device='cuda:0', dtype=torch.float64)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.to(dtype=torch.float64, device=\"cuda\")  # if we have a GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Tensors - a scenic view of memory blobs\n",
    "\n",
    "Internally, Tensors are stored as a blob of memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       " 0.0\n",
       " 1.414199948310852\n",
       " 1.6180000305175781\n",
       " 0.0\n",
       " 2.7183001041412354\n",
       " 3.1415927410125732\n",
       "[torch.FloatStorage of size 6]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.storage()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((3, 1), torch.Size([2, 3]), 0, 78994688)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.stride(), t.shape, t.storage_offset(), t.data_ptr()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.0000, 1.4142, 1.6180],\n",
       "        [0.0000, 2.7183, 3.1416]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Indexing and Slicing (and taking Diagonals, too) do *not* copy, but create views:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "v = t[:, 2:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((3, 1), torch.Size([2, 1]), 2, 78994696, 78994688)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.stride(), v.shape, v.storage_offset(), v.data_ptr(), v.storage().data_ptr() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[1.6180],\n",
       "         [3.1416]]),\n",
       "  0.0\n",
       "  1.414199948310852\n",
       "  1.6180000305175781\n",
       "  0.0\n",
       "  2.7183001041412354\n",
       "  3.1415927410125732\n",
       " [torch.FloatStorage of size 6])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v, v.storage()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(`data_ptr` already subsumes the storage offset...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Generalized tensors are tensors, too.\n",
    "\n",
    "- These were strided tensors\n",
    "- There are other Tensor types: Sparse, on TPU,...\n",
    "- Quantized is a bit special, but similar in spirit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.strided"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.layout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(indices=tensor([[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]]),\n",
       "        values=tensor([-1.9326, -0.6894, -0.6399,  0.6636, -0.0204, -0.6179,\n",
       "                       -0.7636,  1.5748, -1.0660,  1.0288]),\n",
       "        size=(10,), nnz=10, layout=torch.sparse_coo),\n",
       " torch.sparse_coo)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = torch.sparse_coo_tensor(torch.arange(10)[None], torch.randn(10))\n",
    "s, s.layout"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Where to get Tensors?\n",
    "\n",
    "PyTorch has ~90 factory functions:\n",
    "\n",
    "- torch.tensor, torch.sparse_coo_tensor\n",
    "\n",
    "- torch.zeros, ones, randn, ...\n",
    "\n",
    "- torch.zeros_like, ones_like, randn_like, ...\n",
    "\n",
    "Conversion to and from Numpy works without copying (there also is `torch.utils.dlpack` for a similar effect with GPU tensors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.numpy()\n",
    "t2 = torch.from_numpy(a)\n",
    "t2.data_ptr() == t.data_ptr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Careful: Numpy uses a dtype of float64 by default, PyTorch float32 (as you typically want that for GPUs)\n",
    "\n",
    "Sometimes you see FloatTensor, .data, ... in old code or copied from old code. Don't do that.\n",
    "\n",
    "There also are `torch.utils.dlpack.to_dlpack` / `from_dlpack` for interchange with GPU tensors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Saving and loading tensors\n",
    "\n",
    "Saving and loading tensors and dicts, lists of them (and other types) can be done though `torch.save` and `torch.load`. Optionally, you can specify where the tensors should be loaded to (the default is to put GPU tensors back on the GPU)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.0000, 1.4142, 1.6180],\n",
       "        [0.0000, 2.7183, 3.1416]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.save(t, 'my-tensor.pt')\n",
    "torch.load('my-tensor.pt', map_location=\"cpu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "**Style advice**: Do use `.pt` as an extension, not `.pth` or anything else (`pth` collides with Python \"path\" files)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# PyTorch is ... operations on tensors\n",
    "\n",
    "PyTorch mostly follows numpy in naming. Most things are available as methods (`t.tanh()`) and functions (`torch.tanh(t)`). More neural-network specific things are in `torch.nn.functional`.\n",
    "\n",
    "I'm not going to list them, but generally we follow NumPy naming, the [documentation](https://pytorch.org/docs/) has all the details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.0000, 0.8884, 0.9243],\n",
       "        [0.0000, 0.9913, 0.9963]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.tanh()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Inplace operations are signaled by a trailing underscore `_`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.1275,  0.4194,  0.7845,  0.6627,  0.3954],\n",
       "        [ 0.6835, -0.7321,  0.9535, -0.7996,  0.0952],\n",
       "        [-0.7956, -0.3266, -0.9758,  0.9814, -0.0405],\n",
       "        [-0.4738,  0.9680,  0.0144, -0.5584, -0.2566],\n",
       "        [ 0.0568,  0.9590,  0.5114, -0.2172,  0.7584]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = torch.randn(5, 5)\n",
    "z.tanh_()\n",
    "z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The `torch.nn.functional` namespace (sometimes mapped to `F.`) contains functions that are more neural-network related.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.3343,  2.0150,  0.3340,  1.2196,  5.7402, -1.3553,  3.2065, -3.1894,\n",
       "         -2.1207,  0.4356],\n",
       "        [ 0.9437,  0.3812, -0.2400,  0.7666,  1.9859, -1.8378,  0.5127,  0.1119,\n",
       "          1.2590, -0.3914]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w = torch.randn(10, 5)\n",
    "i = torch.randn(2, 5)\n",
    "torch.nn.functional.linear(i, w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# PyTorch is ... autograd\n",
    "\n",
    "PyTorch has a backpropagation-based autograd mechanism. It computes the gradient of *scalar* functions (and, by proxy of Jacobian-Vector products) at the point where is was evaluated. (Image from Deep Learning with PyTorch)\n",
    "\n",
    "<img src='backprop.png' style='width: 100%' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Autograd in action"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.linspace(-3, 3, requires_grad=True)\n",
    "f = 10 * torch.sin(x)**2 + torch.tanh(x)\n",
    "gr, = torch.autograd.grad(f.sum(), [x]) # gr always returns a list of gradients\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5jcVdn/8feZsr1lS7Ynm7rpdQnpPZACCUhRBAFBEQuIDyoIj/pDhccGj8gjAgqI0lRqgCChJYSQttn0vkm292zvOzPn90eCIiYkuzszZ74z9+u6cpFdJvP9TLJ775nzPec+SmuNEEII67GZDiCEEKJvpIALIYRFSQEXQgiLkgIuhBAWJQVcCCEsyuHPiyUnJ+ucnBx/XlIIISxv+/btdVrrlE9/3q8FPCcnh/z8fH9eUgghLE8pVXy6z8sUihBCWJQUcCGEsCgp4EIIYVFSwIUQwqKkgAshhEVJARdCCIuSAi6EEBYlBVwI4RPbmtr4bXG16RhBTQq4EMIn3qxt4r5jleQ3tZmOErSkgAsheqWjo5yenuazPu72nFTSw53cebgMtxwc4xNSwIUQvXLo8I/Ylr8KrT2f+bhoh517hmeyt7WDP5XX+SldaJECLoQ4Z83NezhxYh0Z6Veg1NnLx8Up8cwbEMsvjldS293jh4Shxa/NrIQQ1na86P9wOOLIyvrSv32+5EQ7O0obcLk1Lo+HgXERzB6ejNNu476RmczfeoifHK3godGDDSUPTlLAhRDnpKX1IHV17zAk51Ycjlhcbg89bk1kmJ3d5Y18+/md//b45Jgwnv3qdEamxvKNQQN5sLiaq9OTmJ4QY+gVBB8p4EKIc1JU9Dvs9hiys6+nqqmTm5/ezqJRA7ll0QguHJvGW7fNJdJpx2FX7K9oZs3eSoYkRwPw+QFxvFBVz52Hy3g7LxenTRl+NcFBCrgQ4qxa245QU/MmOYNvZme5m68//SEd3S5unjcUAKfdRm5a7D8fn5EQyeIxqQDUtHRy2UMbGTY+hQ8TFE+U1/K17IFGXkewkZuYQoizKip6GLs9kr2Ny7jqsc3ERjh45ZuzWDou/ax/Njk6nBtmDWH71griml388ngVVV1yQ9MbpIALIT5Te/txqqtfJzHlC9zxSgl5OQN45ZuzGJEae/Y/DNhsilsWjeDx6/Lo2X2Cjh43Pz5S7uPUoUEKuBDiMxUV/R6bLYzc4V/jyeun8eiX8oiPdPb6eRaOSuUXy8agjrXwam0jHza0+CBtaJECLoQ4o46OEiqrXsETuZLwsGRmDEvqU/H+2JV52fx++ggGRYTxg8NldHs+ezOQ+GxSwIUQZ1RU9AhubeOutROobenyynOuHJ/OvSMyOdLexX0HyrzynKFKCrgQ4rQ6Oysor3yR9aXnc+O8aaTEhnvtuZckx5PY4uLRyhMcbenw2vOGGingQojTOlT4MG6Ppqjzc9wwa4jXn/++kVloNNduPuz15w4VUsCFEP+hq6ua6uoX2FQ5jR9esgCH3ful4pJRaeT1ODjq0DyyT1al9MVZ/1WUUk8opWqUUns/8blEpdTbSqkjp/47wLcxhRD+VFz8GDblJivrZkalxfnsOn9ZMApnp5v7Sqpo7XH57DrB6lx+rP4JWPqpz90JvKu1HgG8e+pjIUQQ6Oquo7zieTLSLuErC2b79FoDIsL4TloK3RF2flskp/f01lkLuNb6A6D+U59eBTx16vdPAZd4OZcQwpD12x/E7ekme9DX/XK9/5o8iItS4vlDRR0lHd5Z6RIq+jqxlaq1rgQ49V9pbCBEEGhpr8XV+iKHms4nJtr7Ny7P5J7hmQDctP2Y364ZDHx+E1MpdZNSKl8plV9bW+vrywkh+mHt1t/itHUzYdStKOW/joGZEWGMaPKws6eLv5XI6T3nqq8FvFoplQ5w6r81Z3qg1voxrXWe1jovJSWlj5cTQvhaQ2sdEd0vUdx2HnNGT/P79X8/YyS21h7uOlxGh1t2aJ6Lvhbw1cB1p35/HfCqd+IIIUz5x5aHiHB0MnnMd4xcf1hKNMtVOK12+J/DsqzwXJzLMsLngE1ArlKqTCl1I/BzYIlS6giw5NTHQgiLcrlaGGh7lQb3dKYO9//o+2M/njkce2U7j1eeoEhuaJ7VWQ900FpfdYb/tcjLWYQQhpSV/QV0Cxecb3ZFcHZiFJdHRvMymrsOl/HMhKF+nYu3GtmJKUSI6+pu5fCxPxAbP4e4uPGm4/CbSyZw94gM3qtv4R91TabjBDQp4EKEuPU7H8NOM7X6i6ajAKCU4obMFAY7ndx9uJw2t9t0pIAlBTzIdRcVUf2rX+Hp7jYdRQQgt7uDrsanOdo0mgsnLzYd558Kq1uo/LCCiu4eHpQdmmckBTyI6e5uym//Lk0vvIi7/tObaYWALXufJMrRRGzyV33SsKqvRqfHMX1ADDG1Xfy+tJbC9k7TkQJS4PyLCa+r+c2DdO7bR/q9P8OZlmY6jggwbncX9TVPUtg4gpXTVpiO8x+un5lDz956nMBdh8vQWpuOFHCkgAep1g0fUv/EEyRc9QViFwfOW2MROCoq/06kvZ7O8OuIDj/rgjS/WzImlbQIJ4Nqe/igoZXXauWG5qdJAQ9Crro6Ku68k/ARw0m94w7TcUQA8ni6KS5+hPi4ydyy7POm45yW027jqmmDaD7YwOioCH5cWE6rS25ofpIU8CCjPR4qfnAXntZWMu6/H1tEhOlIIgBVVLxEV1clQ4bcgs0WuGXgK3OG8tGdC/nVqGwqu3q4v6jKdKSAErj/cqJP6p/6M20bNpB65x00pGbz5MbjFNa0ANDY3s3f8ktPezhtXVkrbz+5D1ePjHCCncfTw+Gjv+N40yD2nxhjOs5nigl3EOG0MyU2ijtz0liWHG86UkCRAh5EOvbuo+aBB2D2PH5mG82cX7zPPa/tp6C4EYANR+r4/gu7mXbfO1zxyEfsKfvXnGJnWw+Ht1RzcJOMcIJddfVqtLuC98uXc96QRNNxzqqmuZMF968jva6baQkxpuMEFCngQcLT1kb57bfTGR3HlXELeWNvJddMH8z6783nirwsAC6akM4bt87mvxaPpKS+nUse3sgDaw/hcnvIHJnAwMGx7Hi7BI9H7vYHK63dFB5/mNKWTEblLCfCaTcd6axSYsMJd9h4dkuJ6SgBRwp4kKj62b30lJTw3ue+yZLzR/LRnYv4fyvHMjgp+p+9JJRSjM2I55ZFI1h72zxWTcxgV1kTdptCKcWUpYNpru3gaMEZuwMLi6uufoPuziJeO3ohV00bbDrOOVFKcWVeNrvKmjhc3WI6TkCRAh4EKl58laaXXybp5q/x7ds/z/1XTiQxOuwz/0x8lJMHPj+Jx66dilKKhrZuEobHk5AaRcFbxbLmNghp7aGo+GHqOjMhch7DB1pnOuLSyZk4bIq/55eajhJQpIBbXNvxYqrvuYfjqUNJ+vo3cNhtvereFu6w4/Forn9yK9f9aSuj5mdSV9pK2YEGH6YWJtTUvkVb2xEmj7mNn10ywXScXkmKCWfR6IG8vKOcHjns4Z+kgFuYp7ub7Td9C5dHo+/+CfYwZ5+ex2ZT3LZkJIeqWrh3XzFR8WFsf6vYy2mFSVpriop+R1TUEEYMXsXo9DjTkXrt5nnD+Nkl45Hmsv8iBdzC3r3zPlJKCzl09TdZccHUfj3XgtyBPPiFyRSUNVGTHkb5oQZqipu9lFSYVlf3Lq2tB1hXtpxNx6z57mryoAEsHZcWUD1bTJO/CYsqeOVtMtb8jb0T5/HFO270ynMuH5/O9TNzeLyyFluYjQIZhQcFrTXHix4CeyZ/2D6c1k6X6Uh9VtvSxf++fZi6VjmtB87hRB4ReFwNDUT/+qc0pqSz9JFfYrN5703lnctG0e32MLw7jMPrymmsbichNcprzy/870T9elpa9lLQdDPxkRHMzx1oOlKfNbZ38+C7R4iNcPCVOUNNxzFORuAWo7Wm8q670U2NTHz0IWIHeHcuM8Jp575LxzNr2RDsdhs73pZRuNVFRw0jNf16nigYxcqJGYQ5rPttPyI1lonZCbxUIIcegxRwy9n/8OO0vv8+zm/cSsQY322D7nEoKhJt7P+oirYmebtqZZGR2exrvY6OHhuXTskyHaffLpmUwf7KZo7ImnAp4FbStv8A7ocfZGfGGJKuvdan14qNcHAoXuHxeMhfK6Nwq4uPdLJ0bBoTs6zfS2TFhHRsCl7dWWE6inFSwC3C09HBoW/dRosjkoR7fkrCWTbq9JfNpvj+5eM45HSzZ305Xe09Pr2e8K2l49J55EtTg+KE94GxESzIHUhLp3xNSgG3iPJ7/4fIihJWL/sqS2eP9ss1pw4eQMykRJRLs+HNIr9cU3hfUV1b0BW7P1ybxz2rxpmOYVy/CrhS6jtKqX1Kqb1KqeeUUtJ82gea31pL6wt/5+8j5nP5zZf5dRT1X1eMoyTMw771ZdJq1qLufGk3Vz662XQMr/p45VVzkP1g6q0+F3ClVCZwK5CntR4H2IEveCuYOKmnooLKH/6QsHHjmHLPnZyX49/2nwPjIlh0yXAc3VpazVpQTUsnW47Xs2RMqukoXnf/2kMs/PW6kN5a398pFAcQqZRyAFGA3FXwIu1yUf6974PLRfYD93PRFDPd45YsGCytZi3qzT1VaA0XT0g3HcXrxmXGU9fazcbCOtNRjOlzAddalwO/BkqASqBJa732049TSt2klMpXSuXX1tb2PWkIqnvkUTq2b+ep8z/Pbo+5znFKKdKnp9Jc28E7a48ZyyF67/XdFeSmxjIiNdZ0FK+bn5tCXISD1SG8GqU/UygDgFXAECADiFZKXfPpx2mtH9Na52mt81JSUvqeNMS05+dT9/DDlOTN4/mEsQyMNXt7YdKMdJocmh1rS6XVrEXUNHeyraiBi4Jw9A0nO2leODaNt/dX0xWihx33ZwplMXBca12rte4BXgJmeidWaHM3NVH+ve9jz8zk7kHLWDEhg0FJZrezx0Q4SZySREy7h/fWy8koVjAwLoK135nL56dlm47iM8snpNPS5WLD4dCcRulPAS8BpiulotTJZRGLgAPeiRW6tNZU/vBHuGpr2XbNd6jzOPja3MDo+XD158fQbtNsWXPcdBRxjkamxhp/9+ZLs4Ylc9+l45kyeIDpKEb0Zw58C/ACUADsOfVcj3kpV8jqLiqidcMGEm+9ld+WO5k1PIlxmYGxey42Oozo8QOIb/awKT905x2toK61i+/8dWfQbzcPc9j44vmDznoCVbDq1yoUrfWPtdajtNbjtNZf0lpL04x+Ch8yhKGvrSbphhv40UVj+a8luaYj/ZsrvjAaFWajYfsJ01HEZ3h7fzUv7yjHHQL3Kzp73DyzpZjtxfWmo/id7MQMQGFZWTgcdlZMSGdqgL01TBkQyeSF2RzfWUtjdbvpOOIM3txbRU5SFLlBuPrk0+w2xa/eOsTTm0Pv3owU8AC0+dgJfvvuEdq6ArPx/oQFWaDg+b/sMx1FnEZTRw8fFdZx4di0oOh9cjZOu40Lx5xcjdIZYruFpYAHoD9uOM6fNxXhDNCjo6Ljw6lNcdJd2Ex9nYzCA817B6txeTQXjkszHcVvVkxIp7XLxQeHQ2uvSWBWiBBW3tjBeweruTIvO6Ab78+7eCgKeOWvB01HEZ/i9pxsRDYpK8F0FL+ZMSyJAVFO1uypNB3FrwK3QoSov24tQQNXTRtkOspnmj81g8o4G837GumUVrMB5fKpWbz49ZlePWov0DntNi4Yk8aJtu6Q2mgmBTyA9Lg9PL+tlPkjU8hODOxzKJVSjF2QhdMDb60uNB1HnNLU3oM7RPvV3HvpOP5y4/khMe//MSngAaSxvYcJWQlcOyPHdJRzcumiIbQkOKjeViutZgPEPa/vY8kD60NqFPoxx6l7Rq4Q6k4oBTyApMSG88fr8lgwyhqnhkeFObjm+nH0tLk4tFlazZrmcnt4/2ANE7MTQmoU+kl/3HCMOb98P2SKuBTwAFHT0knJCeut6MjMHUBCVjSb1xRJq1nDthc30NDeE5S9v89VZkIklU2dbC0KjU09UsADxFMfFbHg/nU0tHWbjtIrSik+sHfR2dDF0YIa03FC2tv7qwmz25g7MnS7fs4dmYzTrnjsg9BoeywFPAC4PZqXCsqZMyKZARbs6bBwYQ71Ng8fvnYsJOdeA4HWmrcPVDNjWBIx4Q7TcYyJDncS6bTz4ZG6kHhHKAU8AGw6eoLKpk4un5plOkqfXDw5g93RHtqrOyg72GA6Tsi679LxfGvhcNMxjJszIgWX5+QPtGAnBTwAvLC9lLgIB4tHW3PuMirMQe6MNFptmq1yer0RSilmDU/2+5mpgeimeSfbLx+sbDacxPekgBvW5XLz7sEaVk7KIMJpNx2nz644bxAF4S6qDjdSUxz83ziB5k8bj7Ovosl0jIAwMSuBEQNj2HQs+DtmSgE3LNxhZ9135/OtBSNMR+mXCVnxPHj3bMIiHRS8FXpd4Uyqaenkntf3894BuYn8scVjUtlyvJ49ZY2mo/iUFPAAkBQTTlq8tU9NUUqRlhzNuLmZHN1RI61m/ei9AzVofbJoiZNmDUtCa/jtu8G9S1gKuEHFJ9q44pGP2F8RHFMOLreHR6tq0Ap2vCOjcH9550ANmQmRjEoL/t7f52rW8GTC7Lagn0aRAm7Q6p0VbCtqID7KaTqKVzjsNtxhNgqj4OCmStqa5IAmX+vscfNhYS2LRg8M2d2Xp6OUYvKgBFq7XEF9M1MKuCFaa1bvquC8nAFkJkSajuM1l0/NYp3qxOPW7H6v1HScoFdY04rDZmORRVcw+dI100929Hz0g6OGk/iOFHBDDla1cKSmlZWTMk1H8aoLxqThirLTmhJG8b56dAhspjBpXGY8BT9cwqxhSaajBJzl4zOw2xSbjgbvtnop4Ia8urMCu02xPMhOTYkMs7NifDrPe1q5+PYpqBDqSW1KmMP2z0584l/sNsWqiRk0d/YE7VFr8q9uSG5aDDfOHkJSTLjpKF539fRB3LI0V766fGxveROLH1jPnjJZ/30ml0zOpL3bzYYjdaaj+ES/vsWUUglKqReUUgeVUgeUUjO8FSzYXTo5i7uWjzYdwycmZCVww+whRIdwTw5/WLt1P/H1B0gID43WqX0xfWgS4Q4b976x33QUn+jvGOlB4B9a61HAROBA/yMFv4KSBpo6gvsYstYuF3/PL6Wx3VrdFa2k8NB+xjpqSI2PNh0lYIU5bGQNiKLoRDsNQfi12OcCrpSKA+YCjwNorbu11sG97ckLXG4PN/05n7tf3mM6ik8V1bXxvRd2s2aPHPTgC9VNHUS21xCRmE5YmPU6WPrTpZMzAHjiw+OGk3hff0bgQ4Fa4Eml1A6l1B+VUv8xFFBK3aSUyldK5dfW1vbjcsFhy/F66lq7uWhCuukoPjU2I46hKdG8urPcdJSg9I/th4m1dTF+bHBOw3nT9TOHAPDa7grDSbyvPwXcAUwBfq+1ngy0AXd++kFa68e01nla67yUlNBtNP+xN/ZUEhVmZ36uNY5N6yulFKsmZrK1qJ6Kxg7TcYKOo7kSgHnnTTCcJPDFRDjITIikuK6dzu7gWo3SnwJeBpRprbec+vgFThZ0cQYut4e39laxcNRAS3cePFcrJ2WgNbwehCMf09prS8nMzCQuLs50FEtYNSkDDWw+Hlxb6/tcwLXWVUCpUir31KcWAcF5q9dLdpU1caKtmxXjg3v65GNDkqOZmBUfNL1eAsWh0hrKy8vJzc09+4MFAF+bNwynXfHR0eAq4P1d53UL8IxSKgw4Bny5/5GC19TBA3jv9nlkBNHW+bN59qvTZTmhlz27djMKyBkmp++cq/hIJzOHJbN6ZwV3XJiLPUg2PvXrVWitd56a356gtb5Eay3naZ1BU1cTN799My0cDYnpk499XLzlrEzv0FpTV16EyxFJdkZovJPzlvT4CKqaO1m9K3im9ILjx5AFPLXzTTZWbKSxzWU6it/99t0jXPrwR6ZjBIWD5Q0McDeQnDlEug/20sdHrT29JXhaHUsB95PXC99Gu+KZmj7edBS/i490srO0kSPVLaajWN5bm3fjUJrZebL6pLeGJscQF+lgT1lj0LwjlALuB23dHVR27yTDOZWY8NDbdLFsfBo2Ba/trjQdxfKOHz2CSzmYPMbaR/CZMnt4Mt1uzbpDwbEnRQq4Hzy7+12wdbN06GLTUYwYGBvB+UOSeH13RdCMfEzweDxkqAaGDh2G3R4691G86aa5J6dRnvwoOHZlSgH3g9cK16LdEXx5SmgWcICLJqZzrLaNA5UyjdJXZWVldLS3c96kcaajWNak7AEMTY6mqT04ehFJAfcxt8dNVU8B2RFTGRAVOssHP23ZuHS+Pn8YCUFyfJwJz7+9BZRi+HBZPtgfl03NYldZE5VN1t8hLAXcx3bV7qLD08RtMy81HcWoxOgw7lg6KqTWwHuTy+2hqvQYrqhkIiPl77A/Fo8+2cbi/rWHDSfpPyngPvbK4bdw2pzMzpxtOopxPW4P6w7VUFTXZjqK5WzYW0QcHQwbPtJ0FMvLTYsjzK5Yu8/6nTKlgPuQ1prVR9YS7RlNtFN6Nrd2urjxqXye3yaHHffWh9t3A7BkxiTDSYLDhKwEmjtdFNZY+56MFHAfeu/YLtz2OqYNnGs6SkAYEB3GzGFJrNlTKatReqm+vIguZyxZadLR0xuumT4YgEfXHzOcpH+kgPvQM3vWoLXixikrTEcJGCvGp1NS387ecmlwda5ONDYT525kYNYQ01GCxsqJ6diV4r2DNaaj9IsUcB/aVf8hkZ4hjEnNMh0lYFw4Ng27TfHGHtnUc67Kio+jgCsXn286StCw2WyMy4yjob2blk7rLimUAu4jOyqO020vZXLyHNNRAsrH0yj5RfWmo1jG/gMHiY2NJT1dmld5090rxuDRWHpXpvT59JF9jSebN30tb6XhJIHnN5+fxICo0Gsp0BeNbZ3sPXCIpEEjsNlkvOVNUwcPIDkmjNd2VXDxxAzTcfpEviJ8ZF3p+wyLH8bUDFn29WlJMeHYbNJJ71z8Y9NunMrD8BHydeRtdpsiMyGStfuraWjrMh2nT6SA+0BJYy1bq7YxJVnWfp/JM1uKufKRTbIa5Sx27zuACxtLzg+9Lpb+sHRcGgCPfWDN3ihSwH3gj/lr0HgYN2CW6SgBy64UW4vq2SfHrZ2Ry+3B3VCOJyaVyBDsYukP183IQWHdc1ulgPvA+vL3Ue54Vo2eZjpKwLrg1GqUNbIa5YzW7zxCJN0yfeJDUeEOBidFUdrQQasFV6NIAfeyhvY2Tnh2kxM5DbtNWn6eSaJs6jmrjtoSQLFs1mTTUYLaxzcwn9xYZDZIH0gB97Kndr6NsvVw8fALTEcJeMvHp1N0op39lTKNcjplRUcZNCib9OQE01GC2lfmDMFpV5Q1Wq87oRRwL8uv+QDlieTqiQtMRwl4F45N44qpWYQ75J3Kpx0tr6Gqqoohw+XkHV+Ljwxj4aiBrDtUg8djrXeDUsC9yO1xU9KZz7JhC4gKCzcdJ+AlRofxqysmMnxgjOkoAeeNDdsBSEzPMRskRCwalUp1cxfPbbPWgcdSwL1oW1UBDV0NLBy00HQUy9Bas7vkBBUWfPvqS6XHj9KuIpk4Itt0lJCw6FSP8Cc3Wms5Yb8LuFLKrpTaoZR63RuBrOxXG14E7WBGuiwfPFfv/OVPPH/Xd3h2c7HpKAGjrqmViM4TxKYNMh0lZCTFhJMWH8HRmjY6ul2m45wzb4zAvw0c8MLzWJrH46GwdTPxjCEuXKYEzlVaViZJPfV8tHmHrEY5Zc3GndiVZpqcfelXy8elo4Fnt1hnGqVfBVwplQWsAP7onTjWtfbILjyOE8xKn2c6iqWMnD4bHE5iy3bKgcenHCs8TDcOFk4dYzpKSLlp7sl2vX/Nt86BI/0dgf8G+D7gOdMDlFI3KaXylVL5tbXW7fp1Ns/tO9n7+wbp/d0r4VFRDJ06nZFthazZYZ2Rj6+43W4i2msYP3YUTlmd41dp8ZGkxIZT3tBhmXeDfS7gSqmLgBqt9fbPepzW+jGtdZ7WOi8lJXhPE9nTsJFIz1ByUzJNR7GcyYuWEOHpYs/mTaajGFdaWkpnZycTx8ro24Sb5gylrdtN8Yl201HOSX9G4LOAlUqpIuB5YKFS6mmvpLKYipYKehylXJAjq0/6YtD4iUQmJHJxVLnpKMY989YmtLIxbNgw01FC0sfNraxy4EifC7jW+gda6yytdQ7wBeA9rfU1XktmIe+XvQ/ATXmrDCexJpvNzvh5CynZXUBbY4PpOMZ4PB6aKovpikgiPFz2EZiQnRhFZkIk9689RGN7t+k4ZyXrwL3gb/v/QXbMEAbHDTYdxbLGzFuE9ni453//Ypn5R2/btK+IKDoZOlyaV5m0cNRAPBqe3Rr492S8UsC11uu01hd547mspqSxlqMtu0nQk0xHsbSkzGycaTnYj+azp6zJdBwj1m/bBcDSWVMMJwlt1844ORD769bAX40iI/B++mP+GpTycMWYZaajWN55Sy4guaeeNes/87540DpRfpwOZxyD0pJMRwlpI1JjSYwOo7i+nbKGwL6ZKQW8nz7u/b1ylPT+7q9J8xfgUXaKN68LuWmUxqZmYt3NDB4qzasCwYrxJw+Qfj7AR+FSwPvh497fQ6T3t1dExsQSM3IimScOUHC8znQcvzp2tBCAzy04z3ASAXBFXhYAMRGB/X0tBbwfXtz/HsrWw0UjpPe3t8xevpRITyf1h3abjuJX23btJT4+ntTUVNNRBDA+M56M+AjyiwJ7VZQU8H4o6dpGjDOGayZI729vGZM3jaj4BNr3bTYdxW9K6popLSrCFZuGUsp0HAEopbhwXBrrDtXy501FpuOckRTwPnJ73KwrXcfcrLlESu9vr7E7HIyePZ9jBds4WlplOo5fvPbhThzKw4wpE0xHEZ+wdGwaLo/mF28epMvlNh3ntKSA99EzOz+goauBCXLyvNeNnL0Aj9vF08+8ajqKXxw+dAgXdqZPyDUdRXxCXk4isREO2rrdvHugxnSc05IC3kcvH/4HWtu5cOh801GCTsbQYXTHp9N1YL+ppUgAABNASURBVAvdrjP2SQsK1c0dRLRXE5mcicPhMB1HfILdplg+Ph0FPBegLWalgPeBx+PhaNsWBqixJEfHmY4TlIbMmEdSZw3vbNxpOopPvb5pH5HKRd5E6f0diJaNS0MDGwrrAvLUKCngffD6oQK04wRzMuabjhK0lq1agQcbW99eazqKT2XbGlFKMWeqFPBANHNYMtfPzCE9PoKyhsAr4PKerQ/+uu9NtFZ8ZWpIdg/wi/jEAfRk5GI/VkBHVzeR4WGmI/nEkcOHGDx4MFFRUaajiNMIc9j4fyvH8uOLxwTkCiEZgfdBrWc7KWEjGZqYbjpKUFu0cgVR7naq9gfnmvCXNh2ktraWYSOkeVWgU0rR7fJQ19plOsq/kQLeSxWtFVR2HOXaCXLyjq/lzZlFRGwc+9a/azqKT7yz+eT8/tjRowwnEWejtWbpgx/w3y/vNR3l30gB76Vn964BYOEgObzB1+wOJwMnTufAlo8orQqurfU1zZ24G8qxRcWTmJhoOo44C6UUS8aksnZ/FeUBdDNTCngvPbf3TcI96dL7209yZszF5nGz+uU3TUfxqtXbi0hVLYwZJaNvq/jS9JPf809vLjac5F+kgPfCzvJyuhyFTEycbTpKyMibOpG2qGSqt28wHcWrNu7Yi03B9CnjTUcR5yhrQBSLR6fy/NYSOnsCY2emFPBeeHLHyd7f18j8t98opUiePIv4lgp27D5kOo5X9Lg9pOl6bGERZGRkmI4jeuH6mTk0tPcEzM5MKeC9sLnqA+yeBOblTDYdJaSsuPRiPCjeeW2N6SheobSH2K46Jo0bg80m34JWMmNYEi9+fQbLx6eZjgJIAT9nxfWNtNn3MTpuBjYlf23+NDg7DU/6CJzHC9Ae62+t37rnEN3d3eTmSu8Tq1FKMXVwIkqpgDh0RCrROTrWugNl6+H6STJ9YsKqKy/B3dJAyT5rrwk/XN3Cgy+uR9ntDB061HQc0UcPvnOEW57bYTqGFPBz9V7pe8Q6Y1mYM9N0lJA0PG86YVFRrHv9DdNR+uWF/FKy7Y3kDBmK0+k0HUf0kU3B67sr2V/RbDaH0atbxPHaZl4vfIfJyTNw2uWbzgRHWBgdWeOp3LWNqtpG03H6xOX28M6Ow8SobsaPGW06juiHa2fmEBvu4HfrCo3m6HMBV0plK6XeV0odUErtU0p925vBAskT29fhopXZmfNNRwlpc5cvx6ldvGTRNeEbjtQR23Fy9cLIkbJ93sriI51cO3Mwa/ZUUljTaixHf0bgLuB2rfVoYDrwTaXUGO/ECizvlryL0g5WjpTdlyadP30K7ZEDKN/6QUDcQOqtl3eUM8TZRGZWFjExMabjiH66YdYQIhx2HjY4Cu9zAddaV2qtC079vgU4AGR6K1igKKprpUntZHDUJKKd0abjhDSlFAOnzCaxpZTNu46YjtNrdy4aRAJtjJbdl0EhKSacn182npvnDTOWwStz4EqpHGAysOU0/+8mpVS+Uiq/trbWG5fzq78UbMYWVs+qkXLyfCC4+LKL0cDmtW+ZjtJrackD+PKXv8yECXL2ZbBYNSmTkamxxq7f7wKulIoBXgRu01r/xy1ZrfVjWus8rXVeSkpKfy/ndxU9+YDiktwlpqMIICMzg9Tc8cSW7bLUNMp3/76LNXurGDx4MHFxcopTMKlt6eLmv2xne3GD36/drwKulHJysng/o7V+yTuRAssJTwGTB04iOTLZdBRxytTFS2iqrqL0wD7TUc7JwapmXtheRlVTp+kowgeiwuwUlDTw09f34/H4d1DRn1UoCngcOKC1fsB7kQLH5pIjHKg/wMJsuXkZSEZMmwnOcB78/TN+/4bpi6c3FxPmsHHZlCzTUYQPRIc7+N6FuewsbeS13RV+vXZ/RuCzgC8BC5VSO0/9Wu6lXMZprbn11WcAWDBogeE04pOcERHEjJpCcu1BPjhQbjrOZ2rp7OHlgnIunpDBgOjgPBZOwGVTshiXGccv3jxIR7f/OhX2ZxXKh1prpbWeoLWedOpXcHQbAgpKGmmx7yAlfLD0/g5AS1ZdRJjuYfWrgX3o8cs7ymnrdnPtDPkaCmY2m+KHK8ZQ0dTJox8c9d91/XYli3llRxmqexDXjL3SdBRxGkPGjkfHDMB9eCul9e2m45xR9oAorpqWzcTsBNNRhI+dPzSJX18xkS/PGuK3a0oBP40et4c39lSxcOCXuWH8tabjiNNQNhvj5y0iu6OMp9/bZTrOGS0YNZD/+ZwsGwwVl0/NIj7SSbfLQ2uXy+fXkwJ+GpuOnqC+rZtLJgXdvqSgcv4FF6KAyV3+e8vaG2/tq6K+rdt0DOFnLreHKx7dxN0v7/H5taSAn8bs4cn8/eYZzB1pvXXroSQhLZ3MUWOpyP8w4NaEVzZ18M1nCnhkfWD+cBG+47DbWDRqIK/urOCpj4p8ei0p4KdhsynOy0kkzCF/PYFu7LxFNFSUcdejrwfMOYUAf/jgOJp/HYQrQss35g9jyZhUfrx6H6/s8N1KKalQn/LqznJ+9OregCoG4sxGTp+NzRlGZf4GXioIjCWF9W3dPLe1hFUTM8hOjDIdRxjgsNt46KrJzBiaxO1/38X7h3xzhqYU8E95cmMRW47VEy6jb0sIj4pi5LQZjOo4yh/WHcIdABt7/vRRER09br4+31yTI2FehNPOH67LY/n4dIan+Kb7pFSpTzhc3cLO0kauyMvi5EZTYQVj5y/G6erCXnaAN/dWmo5DWUM7F45NZYTBJkciMMSEO3joqsk+eyfm8MmzWtTf80tx2BSXTpbVJ1YyaNwEYhKTmNp1hAffOcLSsWk47ObGJg9cOYket/UPXxaBT0bgp/S4PbxUUM7i0akkxYSbjiN6wWazM2bOAgY2F7N0aBQuQ9MonT1uSk6c3FTkNPgDRIQO+So7pa3LxQVjU7l6+iDTUUQfjJm3CLSHefZSIpx2Ixme3FjE/F+/b/SILRFaZArllISoMNkxZ2FJmdmkD89l3/p3qR86k6aOHq7Iy/bb9U+0dvHw+4UsyB3I8IFyXJrwDxmBA9XNnRSUNATcZhDRO2PmLaKutJhX39nKT17fT4Mfd0H+5p0jtPe4+cFyOW1e+I8UcOCJjce5/PcfUdvSZTqK6IfcmXOwOxwsdJTQ1uXiwXf9c25mYU0Lz24t4YvTBsnoW/hVyBfwzh43f91WygVj0hgYF2E6juiHyJhYLvjarcxftZKrzx/MU5uKKCjx/TFXO0ubSIh0ctviET6/lhCfFPIFfPWuChrbe7huZo7pKMILxsxdSFLWIO5YNoqM+Ei+/8Juuly+3VV7+dQsNtyxQFYvCb8L6ZuYWmue+qiI3NRYpg9NNB1HeFFMuIOfXzaeqqZOwny0pK+muZO9FU0sHJVKVFhIfysJQ0L6q66yqZOyhg6+vzRXdl4GoTkj/tVNUmvt1X9jrTXff3E3m4+dYMP3F5ISK6Nv4X8hPYWSkRDJ5h8sksNmg9xLBWVc9NCHXl2V8uzWEtYdquXOpaNIiQ2XFUzCiJAt4K1dLtweTWSY3djGD+Efg5OiOFLTys1Pb6fb1f8t7jtLG/nZ6weYPTyZL50/mPZdtVT/bwHdlW1eSCvEuQvZAn7fmgNc9NCHuKRnRdCbOjiRX142gS3H6/nvV/b0a7Tc2N7Nl5/cSkpMOL+eNIi6/9tJ/XMHQYH2wxFaQnxSSM6BVzV18kJ+GZfnZRlteiT855LJmRyrbeW37xWSGhfB7Rfk9ul54iOd/GLyYEYXtuJ6oRBHciSJX8glckIKyib3UYR/9auAK6WWAg8CduCPWuufeyWVj/1hwzHcWvP1edKvOZTctngkVc2dfWp2tb+imdbD9eTsb2RMSQv2xAjiLh9J1OSBKLsUbmFGnwu4UsoO/A5YApQB25RSq7XW+70Vzhfq27p5douclhKKbDbFLy6bwMczKHvKmoiNcJCTHH3GP+PxaF5cfYCIzdVMxoErPoyES4cTPTUVJYd+CMP6MwKfBhRqrY8BKKWeB1YBAV3A/5ZfSqfLzTcWyOg7FCmlUArcHs2tz++g+EQby8al89W5Q5mUnfDPxzW197DhgyLCN1cxoxNaHQ6ciwYzcHYWyimFWwSG/hTwTKD0Ex+XAef3L47vfWX2ECZnJzB8oJyWEsrsNsXzN03nyY1FPLOlmDf2VBLptHPLouF8ZWQaJ1YXMqmohRalKRqfyIzLR2EPD8lbRiKA9ecr8nQTf/8xuaiUugm4CWDQILO9trtcbsIdds4fmmQ0hwgMqXER3LlsFN9aOJxXdpTTVNLEwoMt1PyjkvAIO50z0hh5YQ72CKfpqEKcVn8KeBnwyYbLWUDFpx+ktX4MeAwgLy/P2G6HA5XNfOnxLTx89VSmDZFt8+Jfwpu7WVbUQceuJlSYnZiF2cTOycIWKSNuEdj68xW6DRihlBoClANfAL7olVReprXm3jcO0OPWjEyNgW1/hJw5kNK3pWQiOLjqO2l+t4T2HdUou43YuVnEzM3CHi0jbmENfS7gWmuXUupbwFucXEb4hNZ6n9eSedH7h2r4sLCOH180moSN98LG38B5X4EV95uOJgzQHk3j6qO0ba0CG8TMyCB2fjb22DDT0YTolX69R9RarwHWeCmLT7R3u/jJa/sZkRTOddU/hz1/hbwbYdkvTUcThiibwtPeQ/S0NGIXZOOIl0ZUwpqCfpLvH3urqK2vZ+OQP2Hb8wEs/G+Y812Q7oMhLfGqUdKBUlhe0Bfwz40MY3nWb4io3AcrH4Ip15qOJAKAFG8RDIK2gNe3dXOi9CAj3rqWiJYquOo5GHmh6VhCCOE1QbmlzOX28H9P/43E51bg6WyC616T4i2ECDpBNwLXWvPknx/n9oq70VFJ2G5cDcly2KwQIvgE1Qhca83qPz/A9UV30BYziJhvvC/FWwgRtIKngGvN4Zd+xqrjP6E0dhIpt7wLsWmmUwkhhM8ExxSKxwNv/YDcPY9QkrGMnOv/hAqLMJ1KCCF8yvoF3NUFL38N9r0M07/JoAt+BrbgeWMhhBBnYu0C3tkEz18NRRvggp/BzFtMJxJCCL+xbgFvroRnLofaQ/C5P8CEK00nEkIIv7JmAa89BE9fBh0NcPXfYNhC04mEEMLvrFfAS7fCs1eCzQnXvwEZk0wnEkIII6x1t+/gGnhqJUQOgBvXSvEWQoQ06xTw7X+Cv14NA0fDjW9D4hDTiYQQwihrFPAPfg2vfRuGLTrZ1yQ62XQiIYQwzhpz4EnDYfI1cNFvwC7HXQkhBFilgI+95OQvIYQQ/2SNKRQhhBD/QQq4EEJYlBRwIYSwKCngQghhUVLAhRDCoqSACyGERUkBF0IIi5ICLoQQFqW01v67mFK1QLGPL5MM1Pn4Gr4mryEwyGsIDMHwGqB/r2Ow1jrl05/0awH3B6VUvtY6z3SO/pDXEBjkNQSGYHgN4JvXIVMoQghhUVLAhRDCooKxgD9mOoAXyGsIDPIaAkMwvAbwwesIujlwIYQIFcE4AhdCiJAgBVwIISwqKAu4UuqnSqndSqmdSqm1SqkM05l6Syn1K6XUwVOv42WlVILpTL2llLpCKbVPKeVRSllqGZhSaqlS6pBSqlApdafpPL2llHpCKVWjlNprOktfKaWylVLvK6UOnPo6+rbpTL2llIpQSm1VSu069Rru8erzB+McuFIqTmvdfOr3twJjtNY3G47VK0qpC4D3tNYupdQvALTWdxiO1StKqdGAB3gU+K7WOt9wpHOilLIDh4ElQBmwDbhKa73faLBeUErNBVqBP2utx5nO0xdKqXQgXWtdoJSKBbYDl1js30EB0VrrVqWUE/gQ+LbWerM3nj8oR+AfF+9TogHL/ZTSWq/VWrtOfbgZyDKZpy+01ge01odM5+iDaUCh1vqY1robeB5YZThTr2itPwDqTefoD611pda64NTvW4ADQKbZVL2jT2o99aHz1C+v1aOgLOAASql7lVKlwNXAj0zn6acbgDdNhwghmUDpJz4uw2KFI9gopXKAycAWs0l6TyllV0rtBGqAt7XWXnsNli3gSql3lFJ7T/NrFYDW+m6tdTbwDPAts2lP72yv4dRj7gZcnHwdAedcXoMFqdN8znLv4oKFUioGeBG47VPvri1Ba+3WWk/i5LvoaUopr01pWeNU+tPQWi8+x4c+C7wB/NiHcfrkbK9BKXUdcBGwSAfozYpe/DtYSRmQ/YmPs4AKQ1lC2ql54xeBZ7TWL5nO0x9a60al1DpgKeCVm8uWHYF/FqXUiE98uBI4aCpLXymllgJ3ACu11u2m84SYbcAIpdQQpVQY8AVgteFMIefUDcDHgQNa6wdM5+kLpVTKxyvIlFKRwGK8WI+CdRXKi0AuJ1dAFAM3a63LzabqHaVUIRAOnDj1qc0WXElzKfAQkAI0Aju11heaTXVulFLLgd8AduAJrfW9hiP1ilLqOWA+J1uYVgM/1lo/bjRULymlZgMbgD2c/F4GuEtrvcZcqt5RSk0AnuLk15EN+JvW+idee/5gLOBCCBEKgnIKRQghQoEUcCGEsCgp4EIIYVFSwIUQwqKkgAshhEVJARdCCIuSAi6EEBb1/wFNJP/EwZ7YoQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pyplot.plot(x.detach().numpy(), f.detach().numpy(), '--')\n",
    "delta = 0.2\n",
    "for xi, fi, gi in zip(x[::10].tolist(), f[::10].tolist(), gr[::10].tolist()):\n",
    "    pyplot.plot([xi-delta, xi+delta], [fi-delta*gi, fi+delta*gi]) # tangents at every 10th point"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "A mickey mouse example of optimizing a model (actually 100 times, starting from random initializations).\n",
    "\n",
    "- We have 100 mini batches à 10 items.\n",
    "- Each is, for a (normally) random $x$ the value $f(x) = \\sin(2x)$.\n",
    "- This we approximate by a 2-layer network with 10 hidden units. Our loss is the squared euclidean distance.\n",
    "- By trying various starting values, we represent the uncertainty coming from the random initializaiton of the weights (of course, we would expect that to go away with more iterations)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([100, 100])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_preds = []\n",
    "for no in range(100):   # 100 experiments\n",
    "    w1 = torch.randn(1, 10, requires_grad=True)   # initialise random paramters\n",
    "    b1 = torch.randn(10, requires_grad=True)\n",
    "    w2 = torch.randn(10, 1, requires_grad=True)\n",
    "    b2 = torch.randn(1, requires_grad=True)\n",
    "    params = [w1, b1, w2, b2]\n",
    "\n",
    "    for i in range(100):  # fitting 100 minibatches\n",
    "        x = torch.randn(10, 1)   # input value\n",
    "        target = torch.sin(2 * x)   # target value (usually from dataset)\n",
    "\n",
    "        pred = torch.tanh(x @ w1 + b1) @ w2 + b2   # our model\n",
    "\n",
    "        loss = ((pred - target) ** 2).sum()\n",
    "\n",
    "        grads = torch.autograd.grad(loss, params)\n",
    "\n",
    "        for p, g in zip(params, grads):\n",
    "            with torch.no_grad():\n",
    "                p -= 1e-2 * g\n",
    "\n",
    "    with torch.no_grad(): # no gradients here...\n",
    "        x0 = torch.linspace(-2, 2)\n",
    "        final_preds.append(torch.tanh(x0[:, None] @ w1 + b1) @ w2 + b2)\n",
    "\n",
    "final_preds = torch.cat(final_preds, dim=1)\n",
    "final_preds.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f65d4385fd0>]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOy9eZBk2XWf9923Zr7c18ral97X2XpmMDMABztBkCBAiUFKFklTIYsRJOWwHbQZDi+SLVmiLNG2aJKiSNoUBcskJJokSIALSOwDYDBL79M93V3d1bVkVeW+v8yXb/UfNVtNdQ8Gg9n7fREZlfWW+25mRf3uueece64IgoCQkJCQkHc/0lvdgZCQkJCQN4dQ8ENCQkLuEELBDwkJCblDCAU/JCQk5A4hFPyQkJCQOwTlre7AK5HP54OFhYW3uhshISEh7xhOnz7dCIKgcKtzb2vBX1hY4Omnn36ruxESEhLyjkEIsXa7c6FLJyQkJOQOIRT8kJCQkDuEUPBDQkJC7hBCwQ8JCQm5QwgFPyQkJOQOIRT8kJCQkDuEUPBDQkJC7hDe1nn4ISEhbw2B7+PYYxzLwrEsfN/bOR4EyIqKFomg6hEUXUcI8Rb3NuTVEgp+SEgIAJ7rMux1GLTb1GsdxpaH5wQwFgR+gAQIQJZAVUFTQYuqGBMpjFSaWDqNrKhv9ccIeQVCwQ8JucNxxhbl7W1uVusMzQAsCV3WUTQdkdTA0JAiKj4BIPAdF8wxYmgT9G2Udhs10iQelygWc6QKE0Ti8bf6Y4XcglDwQ0LuUKzxmEtr69xstQlMSPsRJo04iVKcVM7ASOlENAVVetFl4wcBYz/A9HyGrsugb9NvDBm3Taq9IWuDFrFag5lEjJnZOaKJ5Fv4CUNeTij4ISF3GK7vc2W7wsWtbcTQZzaIM51OUSjEiRZjSJHby4IAIgKimgKaAkYEJpIMLYfGVp92w6RumZx2h1zuX2F/OsX+uVn0qPHmfcCQ2xIKfkjIHcTWwOTxGzcxzSGzjs6RRJ5MOoYyEUOO7fW/O/YYs91mbA5wLAt7PCLwfQCEkJAVFT0WQ4/FKE4kmS7GMbcGNAYW1zE53e1x6eJljpeKHJ6ZQZLCxMC3klDwQ0LuABw/4EK9waW1dTRzzH0kyWgRbNliSxkzrG/Sr/RBA6SAYauN3xuhuRKGEiUdy2EYCZKJIrKiEAQBgR/g2mMsc4DZaQGgRQ2SuSJTukHJ1GjpGS5YDZ7aqnK11eHexQXmkokws+ctIhT8kJB3OY2RxZeWb1De3KTUG7LfcbHFDW4adfr1Cs6yjQhkJCL4jo7vxYnqGZR0Ajst42QlhpERpViK2USRqBLd8wzPdTDbbbr1Ko3yKkKSSRtFspbBh+JTbGVHnCuX+dqVa0yXStw7VSKjhvLzZhN+4yEh70Jc16Xb7XKz3eUbN9eQa9e5z+oyHbEIEgP6Wh/Zcch5EaJqFN9yMAd1bH+MkEARcbTBLGn9AdLmEu3xgNqwwnZ/m8nEJPtS+5Al+YXnyYpKslAkWSgyGvRplTdoD7ZRiZCyc0xlk8ycPMGlletcqWzxxeGIA1OTHI4bROTQzfNmIYIgeKv7cFtOnToVhBughIS8ehzHodVq0e12WR3ZrJavUGhd46hmk8gZDBWZJjJQIh9bJBEp0G6W6berSIFJXB0j3Bq2WWY4buEJH6GkSUWPky88QFeDSmyEVkxyNH+MmBq7bV96jTqt8jpOf0xKyZOemUSZitHY2uDSdpWqFiU7Oc3hZIzZiBa6eV4nhBCngyA4dctzr4fgCyF+B/ghoBYEwfFbnBfArwAfB4bATwdBcOY7tRsKfkjIq8P3fVqtFq1WiyAIqHsW9eXHSQwqzKUUoukZ+p5Oz4uiSDGyiSitTpnW+lX88ZBEUiWeiaGrMaJaFk1L4dsWnepl+r1rOPSQRYKUdBxdn6OdlLCn4yztu4/JxPRt++W5DtWVGwy2GsRIkN+/hDYdx2y3WF9d4aZQERPT5Iwox+NRYop827ZCXh2vJPivl0vnd4FfAz59m/M/ABx47vUg8BvP/QwJCfkeGY1GVCoVbNsmkUjQs1v0z3+dhNVibqaIrhRp9VQ6fpeB2GSkDLiy3MDt9EASqJMpOqqE7YzZHKTYGEis9SxMV8NyTjEanwLfxpD6RGWTjNrgoCo4WN6ku7yMff8jzM8/CLew0GVFZfLAIZrRKM3r69iXLzOlHiM+kWNJ0zGuX6VWWac1McPjns8BQ2cutPbfMF4XwQ+C4OtCiIVXuOSTwKeDnenEt4UQaSHEZBAE26/H80NC7lSazSaNRgNFUZienKBRu8L6ldPg9piamiLi5dna6nJNnKOvegQ+SCsOcj/A8JNEI2kuN3I8bhW4aqfwb1tPUaNLDsgBcPq5o8n1EUdunOFTR87xIx/4YSLJqT13CiHIz86jRaNsn3mWrXOXmHnoJJFknOnDxxDLV0hU12lPznE1CGjYLsfi0dC3/wbwZgVtp4GNl/xefu5YKPghIa8B3/epVqv0ej0SiQQThRy11Se4ceMKpmuzlMuj9Q2eqT/LVWUZ14uRqUVJbHTQLZtAy/ClyBG+IpUwxWuXgZ4X5YnOPE88Dr95/kv85Kk0P/WRj6Kq+p5rk/ki3Cux9cQFNp+8yOx770GNRJg+fIzt5Svo2+t4s4usuYLHOwOOJ6IUtLA2z+vJmzWE3mp+dsvggRDiZ4QQTwshnq7X629wt0JC3nm4rku5XKbX65HL5ZiaKNJbf4rl9es0bIfZuIxTG3J65SzXt8+Qftrj6Je65M+uQ9viK9pJ/lvjA3xenvmexP7lrA6z/JOvS3zsX/4xXzh/4ZbXJPN5SvccZWT22XziIr7roagqUwcPo0YiyBs3uUvy0CWJs70h10wL/22cWPJO43XL0nnOpfP52wRtfxP4ahAEv//c71eB938nl04YtA0J2Y3rumxsbOA4DqVSiWTMYLj2BJcrG1xtDZgRNdjo0F6u0WvVyXpZFElirAScSy7xB/rdDII3x2p+eM7j//iJDzOR3FtWoXWzTPXcFTIL05TuOQLsBHi3rl3BsSwK+w6wIeuULZu0InNX0kAPV+m+Kt6MoO134k+BfyCE+Aw7wdpu6L8PCfnu8DyPcrmM4zhMT08TMwzsjae40apwbSBI1y4yXq8xXg/o4xNLL2FpARXJ4XPqKa7aU7eZV79IQnKZU4bMSiYz/pCMPyLhDnB9iZYSYy0e5YYcpWYatOzEK/j84VvrMh/+5b/mn/7IMX74nv27zmUXZxi3B7RXy0TzaVKzk8jKjqW/de0K9ZVllg4cIZOIcmlg8e2OyclENFys9T3yeqVl/j7wfiAPVIF/BKgAQRD8m+fSMn8N+Bg7aZl/NwiC72i6hxZ+SMgOnuexsbGBbds7Yh+LEdSucKP8LBcrNuLSnyH1GpidDO10ClU3wOtwnQSfG38fQ1+7bduy5LOYHHA8NWC/NCQ+dPEGEsEQxFigBhKy5CEUGykIGHsuA8OjnnC4PMpybTxDx799Pj7AJ4/H+V9//L1E1BfTLn3XZ/3rT2MNTeYfPUU0kQDAtW02r17G9zymDx1hrOqc7w8Zej6HYxHmonvjAyEv8obn4b9RhIIfErIToC2Xy1iWxdTUFPF4HHpbbK88yfkLa9RvfAVF6WM7+/DSizDuYbjbnJYO8Yf9E/i3DKGBJHwOTTQ4NNVCQcKzIgRuFMWRSTqClGkTG/QRowB55BH1ZXTZRo35DBSfvu3iSBamOmBZlDjtH6Th7S278DxHChL/7u9/gGIy8sIxuztk7bGnkeIq84+cQlF33E3O2GLzymUQgulDR0DVuTgY0rBdZiMah2IRpDB185aEgh8S8g6mUqnQ7XaZnJwkmUyC1aX9zBd54ptPcLN6BTXTxxAniGkH6VvXQTT5I+v9nDFLt20zZ4w4lm2Rl0zinoPhe0TFCKF6uDGZUVTBV3VUVyc5stE6A5RtiUhHQx8HxBSLZDZCWR7RdwaM6dMSCjciJ3jCmWHg3TpOkI/6/O7fey/HZzIvHBus1tk4d4HU/hJTx4+9cHw8HLJ17VlkRWX68FEkWWZ5OGZ1NCarKpxMRNFCv/4eQsEPCXmH0mq1qNfrZLNZCoUC+B7Vb/4eZ7/2Za71h2iTNab0Q0jSSa46FUzJ5wvNe9g0I7dsTxMOH06scCK2jS2rOJKGKWm4SoArufiSh8yYKC5C8/DjCUTUICrrpFwHozXAf1YlWlNRbZO44hBk4gyUAYPRDeqSRDe+nyfFES4Ps7fsgy4H/OZP3Mf7j0wCEPgBtaeu0apvMvXAMVLFiReuHfa6VK5fIxKPM7n/EEKS2LJsLg1GRGWJe5IGMTlcnftSQsEPCXkHMhgM2NzcJB6PMzW1s6Dp5mP/Lytf/jOuOVGk0iapVIme/xCrXgcGFl9qnqRm39rHPat2+DvJ88SFizdKILwI+AqK7GDLAU1Z0EZioKoMJQlbAwmLiBgix2yUmEZKCcg6FtmKRHQ9jehY6G4dYhHceBzbuUrT62PmimzHTvLH9YN4twjsqlLAv/7bd/GRE7MAuL0xG4+fw4t6zD94L6r+4oDVa9Spr62QyBUoLiwB0HFczvaGBMA9SSMM5r6EUPBDQt5hOI7D2toaiqIwNzeHG7hc/vp/pPf1P2XNj9Kb6+An84xGdzEYOijVLp+17qYV3Frs3xO5zKeCpzEHJZrjSbqKjimDK2RUIaEJn7hwScgOkuZjRmV6qoSpCEaKwPRVAuEgR8ckjSEx1WJmrDJ5PUmk6iMNy4xFHS8xhaBK32nhZON0J/bx6e1HGPh7XTyyCPjVHzvOx+9ZAMC80WDj8kXiSwWmjx7bVV6htVWmvb1JdnqWTGln8DM9j7O9ISPP53giyqR++8D0nUQo+CEh7yCCIKBcLjMajVhYWMATHqe//Dt43/5renKUi3MJnJhGrHcAUUszaJT5Y/kIHfaKvcDn49pf81Bvk6p7kL5uMJItJGGjCx8VHzcQeJ5C4KmovkQCyAUOEQWGUYVqQqOvQSeiMxA6fVlH0Yfk1DpJecxMZ4Z9GwmC9jq29yz9ZBqCIY7ZJhJXGMxk+b9rn6Dq7s3Hl0TAr/7YCX7wnnl826N59iat7iYTdx0iPbE7BlFduc6g3aS07yCx9E4MwPEDzvWHtB2Xg7EIC2EGTyj4ISHvJJ6vj1MqlRCaxZPf+HeIJ58CxeDrk4sIzaQwmMDfnKU6WOEv5X002JsWKeHxSeULzPcUbDeOLwJ81UGKSqDIuJKEJ2kIoSGQ8B0H13EILJuoY5O1xsw6Y5J+gKsq9GIB6zGFSjJBM5qmrxnElDpZqULUTzJXPcxEt4XnnGMQB1MLkJo90nEJd1rjV+s/Ttne209VBPzbn7qH9x6ZxqkN2Tz3DF7SZ/7ee1G0F6123/fYuvIszthi+sgxtMhORpAfBFwcjKiOHWYjGodjkTu6+Foo+CEh7xBGoxHr6+vE43EUo8G5M58nOLdC3Nd5bGIRK7DIDWXsRolGf4NvSIuUnyto9lJkHD4aPMZSVxBIGn5CJl4ISMccoopD4DuM/TEj36VPhK5I4Ig4Qz/G2JaxLbDHPppjUbSHHDSH5McSupARSoeyZrKSKVIxikiSzZS/QsbpobcXKfQMhL+MawxoJQRi4FGMCyiq/HL3J9gY7Q0oG5LHpz9Y4MT+KaxGQKW+SvrQDBNLuxdsOfaYzcvPICkK04ePISs7vvsgCLg6tFgf2UzoKifi0Ts2bTMU/JCQdwC+77O6ukoQuBipJhcufw0u1Ej5UR6PZ2k4Ghmrgz/SGbc7PMYi18Te6pQyLg8ZTzLjpZCiaYpFl0TUwXI8RraH63houMQVl7TuE5f6yF4HOwgYazHGkRw9OUfP1ygPFdqdAMdyyVs9Dg8sslYMWY0iiy2qRpXTuSN01AQ5f4ulwXWS3QSprQwxexWhtOlHxki2RiHqoiQ1/qn8d1lx91r6GcXl378nzowk0a73GcYcZh44RWJmFvGS9MtRv8fWtSsYqRSlfQd3WfOrozHXTIusqnB3wkCR7jzRv+MEv1MbokQV4okwiBPyzqFWq9FsbhNL1FmuXsY7XQEnxxVdpenIqP0aUeETbw752niSb+lH9rQh8Lknd5oTdo6SnmUmO8LCxrYtAs9FoKAoUSQ5ihOoOJ6MEArIEsIbIrstIsoIVQZL0hnKBiNFpzaGxggcb0hm0Gau65Jyk6AqeKxxqRhwNn8URVjM2csUBx56bYGMuY4s1hkLk5glMy0P0UWMfxj9SdZEYk//5w2fP/jUPrStAZsrV5DiATMLi6iFAko+j5zLISSJbq1CY2ON7NQMmcndG7BsWTbPDEYkFZl7k8Ydl6t/Rwn+cOzw3//eeVY6Q/7jzz+MroXpWiFvf0ajEWtry0jyChW7TueJKoNOikZSxR8N6A/qZIRHsdHmG+Ykn0s+SCD2CtmRxFketLJMRnRS6T6+cJFdUGQDPZJBM9LIkTi6rmJEJFRN4HpjHMfGHI/pWWN6/R6qPCQb90gqEuNxQG2s7oj+UMJ0A0Rgk+0PyHU0krKGr9hU4hW+VZykq+sU3S0mrRF6PUlyMEREruAGDilfZz7ZR3Gn+e+Gn6Rxi+ydhxMWv3Z3gnFtQGdUI1nIEtMjCFlBihkohQJqsUij18HstJk8cAgjmdrVRt12ONcbEpUlTiVjd1Rt/TtG8DdaQ37+985wodwF4CfvmuEf/62Td3QAJ+Ttj+/7rK2tYA7Ps+UPqJ2uIlU1WrkEhrVJvdMgNXApjdqctmf4D8lHsMVeobw/ssqHbJlkREWJuBAoyHIcOVFEK06g5GKoSRXNUPEFOJ6P7794vyZDzO/jD9vUtjfoDXrIukMuK1GKCeJynLW2w/naiK22he+4JIIxiYZMepgkIikMtB5nSj5rMZlE0Cbj+0TbMrGuSVy5CI4gq0SYSQ4ZjQ/zD80P0vf3Lpz6Tw+o/MJcisb1a3jKiML8AlgWfreLb1kIRYVohFbgIgp55h55FC26u6xD23E50xuiCsF9qTtngdYdIfhfvVrjv/jMObojZ9fxX/nh43zy4fk3onshIa8LjUaNzc1vsOFYbC3XyG0GDHJFGK3Q29yCvsec0uWqMsP/Iz1Mn71Bz0WtzIfdFiUjT0SPoKlJctk8k7NT5JIGsYiMJARCFgSajKdLOAL63S69vslgMKQ/GDKwXXxAyCA7PXy3i5AcgoiPHPWZyCWYSRdZbXp8faVMvdskypDCaIjRSpEYFXEkidWMz6ph4qp1dFlFGmpE223y4gJJ0yOX0plICtabJ/gl+z04wV4L/Jc+Os8PZjNsN66Tmc6TyebxTROv38et1nArFcatJq16DTUSIX/4KPrsDEqxiBSLIcViDFSNMwMLIeC+ZIzEHbBn7tuhPPIbjq7I9C1nz/H/4S+f5ch0ioPz6begVyEhr4xt26yVn+Rqx6JlmizUXNxMEZxN/Bs3GFsRFhJjWokZvjB4kH6wV+zTSo37/S1yiSVimRylUorDd+1jYmESIQS+52M2LQaNIYNyH7PSxDZN3PEIR/g4ik+gSiiaRkZI2L7PyPUYWODYCoHvEtNBi8N6rcpqtEZxZoofe+huzm4NeHr5MhuiTVFvIPXWSY6K7OvPovtZmlIUVy8zMjzacoFO/x7mgkvoLYuIBvsmb/DT5TS/7R7h5fsk/U9fXOHop06QU1P0BwMyi/vQJ14suxD4Pt5ggHb5ErVzZ+ltbxJrNBACpFQaJZNG0nROahpXAomzkSjHJnKkkwkkw0Bod16M711j4QP8+leu8y+/cHXP8UO5GH/0848QM8Lt0kLeXly69jRP3LiKqfscWWuBGWUjYqI/e4HqUCOT18gnEnzWPMoXx5k99xuSycekqxxLH2JptsTCoSwzRw+gajqW6dCpD2ltmgzafcx2Dc/pIauCiBYjGYkT0+Ioqo6kKYi4hheV8bwAZ+zhjH1a3RGVWpNOq4YsLHKxABE06XodREohO7tAZOIoj6/coFJdJzlqMtfbJOWNEcxzM7LAwPHRg3X6hkyFJAOnw2L3Cge6faYXLJLp/fz+8n38ubc342g2avGZ77+HobVNYrZIad+BW36PtdUVetUKhWwepbczAwgsC6FrSLEYtqqx3Ori+j77DJ24LCNUBckwdl7RKOKl79/B7p87wqUD4PsBf+93n+Qr1xp7zv3NoyV++SfvDf35IW8bytVt/urM1xjpNocHYxLXRlw2XMTWTcyOg1dMcFxJ8QQZfrs3t6cmjYzLR5UrnMrNc8+ROWYOlyjMLWCZLpWVLu3qEHs4xrUbCGESS2lkJifIz05jpOIIIQj8AN90cNsW/sABSSCndOSMjhfAeORij1zK5TZXlzcZmi5RRSItdXBHa7TdNmSixOYOUibOcmuLYNBhsbLG5GgbxTBYTRxi5CkYgyau7HFDLlDHZrp3g+PdLRb2mUTj9/MrV09w1kvu+Z4+nDX5Rw8eZBy1mD56jGh8b3aP73tsPnsJz3WZOXoCyfNwKxWcShV/OETIEk4uxwXNwBOC4zIkHRvfHOKPhgRje1d7QteQogaSEd0ZAKLRncEgEkEob2/HyB0j+ADNgcUP/m9fpjLa+7n+908c5288EvrzQ956ar0hf/HtL2IHbRYiMpNPVbgoAlreAFFp4KTi3B3kKcfG/LPuUUx3b8mAh5QVHsmpPHDiCAdPHCKayLF5rUNr28R3fVRtgKT0iBgy+dlp0qWdXaVuh297uI0RXs+GIEBO6iiFKJK2Y+32uyaXL92kvD3E92PofkB8VGbYX6bj9XBzadqZWTaFT81xmSpvcbjyLKoxplKYR5J0Iv0BnhtwjRRloZMftjk6Xmb/dJNR5CP8i2uL1G9RD+i/mrP42OEi8en8Tn38W2CPhpSvXCISizN54PALxp3X7eJUKri1Gpbjcl4zGGcynJqbopiIAxC4Lv5ohD8cEYyG+MPhi787u13FQtOQohFEJIIUfW5AeO690PW33Ki8owTfsU3+v8/+Lv/j2TnclwWCUrrC5372YeZKey2EkJA3i+7I4a/PPsWwv0K+qLLw1Zus9BxWcwpurYcuYB9xHMPmXwSLrHUKe9rYL9X56MQ2d+0/wcMPvgezp7J5rY3n+iSyKorWQgQWkWQSvTCFhcrY9bAcH9vzef7/PghAlgSqLKHKEhFVIiIJ9L6DZroIQE7pKPkd4bdtm42NDSqNId2RgT3wiTs2ke5N2p1l2pJNM5ulrCfZ1uMovRH3XL1AWmrQnEqQVhQSIwvT1Thv+WyJDGkb5uQaR4w1bkQ+wW+v5nBfNpuJCo9fWuxx+PgCc8dP7EnDfJ5eo0Z97Sa56TnSpcld5wLPw200MDe3eLozYAjcnU4wOTOFks/vWty16z7HwbesHfG3RjsDwWhEYFn4lrV720jBzixAj+wMCnoEKaLvzBB0fWdAeIPdRXdE0PZ5XKHRTE/yQ9ObfLY8u+tcd+zyi39wgX//s+9BuQOi9SFvPyzH46kbZUaDdfKFgMnzFdpbXcoLGbquTMGXmVR9fBHw2ZjBWnmv2GfEkIeK15iZPMLdx+9l/dkx/XaPWEpjYkGjXl2j2hvipibx/BRBxQIsJAk0WUZTJAQBTjDG8oZY7piRM2bs2TieB+yUKlCQyI81MtsKiUiE7GSK7EyW2dlZhCiTc1xGSpLtio1jHKWQnSW59Qzpxhay0kZK5tkycjx213s4euU6E2tb1KZsNM1lShoSV3QeM6/TCdKU/TxDP8V87ywfTN3DX3V3f+5RIPMb6zo/6z6NEo2y/577bvn9JvNFht0ura0y0WQS3XhxRa+QZdSJCdITE7zPHPLk2ibnm03cZy5TVBXUyRLq5CRSbPcqYKGqyKqKnNhrKAa+/4LwP//TH40IxmPcVmuPq+j59qSIjohEEJqOpGs773UdSdN2BoU3yG30rrPwPdvli18/zZnaNZ7YUDnX3vtH+sXv28/PffzQ69XNkJBXhev5PL3WZq18hoy2itH0yXzxKk9MTbA9lSP97CYz0hgklZWJAb9Rfh/2y3aOUvD4ZPZplkp5Hj35EUabO1k7xQMpum6H1Rs38CSV2NQC+VyKVFQlGVGJ6RK2b9Iet+mMOwzsAV7g7WpblVSCQDB2fcaujzke0x/bjEYORk/GGElIikDNa6QmEkhDm7hsUCruY7Xl0K2PSLkBem2N7vZltoM6l/UMW8kiDSXGvtVtJvpVRvkh+7QmM4FHX/g8MXAx6yAnCmjpNHFT4puDRa66e/35f0Ov88h0i7sf/TgL+w8g38Ja9lyXjcsXkSSJmaPHkaRbG3eOH3C6O6DT6XKk1ybbakAAcjqNOj2FUijc1up/tQS+TzAe7wwI4/HOoPDSn+MxgePuuU+K6MQefvg1PfOOsvBlTeHhE4cof6vDyek6a6ZH2979B/9X37zBo4cLHFu69Y48ISGvN0EQ8MxWj0a3yoS2zXDokP/KNZYTabbn82jLHSbwgSFmKuA/tB/YI/YAD8dWyKUl5otHGazryFEJZTHGpcoGo3qFfD7L0qEDZA0VaWTSHmyxWdmmYTbw8RBAQjLIqHHiikFMixPRYmgRA1mPvOCPfqkf2vVdmkOTar1Dd7NDr9Wn3uzSMEzMoE+ifoWTiweIlZJ0+hEMfYlioYS+fIb0aIULzSHR1BzX5vIMyhLFXpvLaZ2RXGUusDmV9LkeCJzNMoINhrl5DshNNlsGg2C3RH1unOdQrcUzT34V2w8olUqk0+ld/ZUVheLCEtvLV2isr72wacrLUSXBqVScM0JwNZHg+NIS+VYDZ2sL69JlhKqglkqoU1N7rP5Xi5CkHXdO9PZ7/QaetzMoPD8AjMc7vrY3gHefhe95bK6tMOgGfO7KebbHgs9e2ftlnywm+MOffwRVD107IW88G60hVypd9PHTjPtrTHxpFceU+ep9Bxl2bBbLHVLyJpYx5k+jd/HY9uKeNhbUFh8qXiKXPcxR/RSBITGIWdi1MvqwTUFXyGga/tCkbtWo0sXGRUYiQ4y0iJOUDFRFR1YUZElGkiUURUVRNRRdQ5IUkMROhkoshpyIIyWTyPE44vkNxntjOhs92p0ha1aHy9Y6A9GnWC5aGOoAACAASURBVDCIaCqWrZPw0xyUCkjlG2ytPc0NyaWcWOCyGiFVbZP0RhjRNnM0Kcg2vtSl04zitfqksxVupvdzYfAeTrf3pmouKja/kLtM/MhRCrMHMQyDYrFI7GWi3Cxv0Klu7aqffytcP+BMz6TjepxMGExoCl6ng7O5hVuvve5W/xvNHWXhS4FL5/q3kSNJ7ppaxF5f5uH5GN9a83ddd6HW59/81TX+80/cOuIfEvJ6MRi7LNf6RIIaulNBPbOB2nF48ugBhoFPqWqRkNrYeo0r8bv59vbcnjYiwuF9iWv4chZjkKLmbiMLF2OjTMrqktUN4nqKWtCmnrHxoxkmkgeZTMyQjRWQtQi+7+G7Hp7r4No2rm0zdsaMHAeee0m+ixYIVLOP0u+iVMQLLhEpHkNOp1EyGXL7U2QGcWYbCU4OClztNlnrmZCQ8IIOK8NV1qV1FqfyzCUfQnn2KfTOKsnkEmeLabx6QNcr4qqg2S0CLUEka+JZWRxT8F71CeJym0rsU2yau8s/33Q1vtqd5RPlNdziBI6qUi6XicfjFAoFtOcWVGWnphn1utRXV9CPnURRb52hpEiCe5MxzvRMLvSHnEwYlDIZlEwG3z6Au739EqtfRZ2a3PH1G3s3dHm7864TfBuFZW2GWOUypZJMIZHGp8PNZJbt3m5f2a9/e5WPnChxeOH2o39IyPeC7wdc2uwi4ZHyr9J4ZpVSechGaYF6RiZVHhFxx/jKdczYDH/R2Yfj7Z11PqpdxZJ8sn6cWMxgMuaQHjYIlID0/oM4C3nWlB5+dIoFo8hscpakttcH/nKCIMB1bJzRCNuyGA9NxqbJYDzaOe96aH6ABmhjC2VzE6e8CQLkVAo5mSGVjHG3WmC6LehKMlJhH5u9PteaG5xpVlmNwMLxfeSvX4HODd4T28/F4gR2tcqaMomlSZwa1ehGR+j5EcF2iYYFj8TOEXcH/J/K38d82W5Zf2JluafeJ7l2Aw7FSWYKDAYDVldXyWQyZLNZZFmmuLhE+dlL1NdWmNx/+7jdy0UfDEq6iqRpaPPzqHNzeO02zuYm9vo69to6SjaDOj2NnM+/5amYr5Z3nUvHNEf83qc/j62YLMV7GIUlvlUf0R0rfObK3qXU95SS/ME/eDjM2gl5Q7heG7DaMJmNVWg+9SdELzaQgixfXpoiCFwSay5Z/zwi1uSPlB/hG/X8njYOSjXujd9Ezqa421jkZCKCwZgBHuriNK0JmaE7JK2n2Z/eT1yLf8/99j0PyxxgDQZYgx7WYEAQ+BCAFgiiAWiui7DGOzfIOo5j0LYcpGKSiaPzNG2fi5sdzm5uMPJqTEZGqFuXMKs+jn6A9UyJzvYGfQNSos4Do1WsaAd1GEWuFlmaWCOvXuLrlUP8evDTBC8rvbBfGvE/T9TIHJlCnTnA1PwCvV6PXq+HLMsUCgWSySS9epXGxhqFuUWSheIrfu6Xu3dK+t5ZgT8e42xt4WxtEYxthK6hTk3t+Pr1t36LxTvKpRMRcKJZ5dLY5sZimsXqCrNGCc+1eN9slMc2dmcmnK30+N0v3eA/+/6Db1GPQ96t9C2HtaZJKangXX8C/0YFNchyLZfHVl3SmwLhtNGMNc4ZH+Kp7b0zzQg2pyJVxkbAIUnm4WICPZ+m6ViYhkc/YxElyvH8cfLRvYPFa0WSZYxk6rl892kC32c06DPq9Rj2OnRHQ5BllEiCKKA7HorfITEe0zu/RW29ysT9h/mB45PcNZPmK1frbHbbaBNJhPQk9vYzpCo9RG6OSK1ONZHn8YjC0fElpNgAJyNxo7Gf2EzAI4V1ztef4rHggV19vO5H+eOqwU8ZHSRW2Aok5vYvkU6nqdVqVCoVOp0OxWKRaCJFo7xGNJFEjeytR/Q8t7P0d303uo6+uIi2sIDXaOxY/TdXsVdXUfIF1JlplMzb02vwrrPwHc/n9DcvYn3hr7mpKozmE5REjxWlgIXKn9xMUTd3i35Sk/mLn3sv06Xv3TIKCXme02tt+pbDIeUmq3/5GSJ1GUcv8VgxguSqqFtjJvkyZsrgd/o/yupor3X4/dIG6dgapYLGjz34CTIzC9y4/gxVv4U8lWEmOcNSagn5NqmHbxTO2GLY3alHP+r3gQBFVjCEjNfq0VuuoqGRni4QOVhCTE9ytjJkudonsJtYlcfZXN+i5WUYpfLQHdOMy+hBm3n/AjPqmH4jTcov8sDENfodm1/o/TjtYPeCKw2PX9QqPDCpo80kkJeOMbd/AU3T6PV61Ot1PM8jbhiMattEY3GmDh35ji6Yl1r6dyUMJm5h6b8Ufzjcsfq3KwSOg2REUaenUUulF4LdbxavZOG/vcPNrwFFEkQX57Ef/D6WfJ9kdUg7MJgYbTG2LT44Ze25p2d7/JPPXcL3376DX8g7i1rPom3aLKZVWk99haBrg5Llhi7h6zJq10e4q6APeTL4PlZHe92NS36PCbFBvCTz/g//AMUjJ3h2+Sxrow2iM0XunribA5kDb7rYA6h6hFSxxNTBIyzcdQ+FuUVUw6DnjjGTOv7Jabp5mW5ngHlmjdGXvs2Jzjqn4i6lzCQLhx7lvqMnmVE95GaTfmRMYtgjCAxWxEHWgyi5bIt60OD8YJZsIsLP6nuNPxuZz3hJrrZ92jf72OfOsn7lGrZtk0qlWFxcJJPJYI5GDJGp16q0K1vf8fM9b+mnFJnz/SG18d5KvC9FMgz0/fuJPfIwkaNHEKrKePk65re+hfXss3i93mv+rl9P3nUuHS/wqLhnMRcyCOshpp59nHpLws7FmRpus6EG3Fcocrq+W9z/8kaDLz61wUcf3JshERLy3eD7Acu1ATFdQb36NXrVTRRlguEY6jkNaagy6vZY1K9RMxZ5rD7Fy0sDq4HLI8FN5P1RHnz0AfZNHeGpM1+i3q8yc+QYJyfvQZVvbzkGfkDgeAS2T+DuvPADAh/wg53HCYEQgCwQsoRQBEKREJqMUF69LSgrKslCkWShiGvbDFpN+q0G1mjImtSnqOQpyDEoD5nWO0Q8j3U5Tmxyhg9kY+RP1zjbsdjUBfp4gKbI3GQCVekwm62z2lTQo2nek6zyIXuNL/m762Fd9xJ8RerxPlshX7PJPX6ejX6b2VMPoEciFItFUqkU9Xqd8vUeVy5e4Iiqk8m/sgvsedE/3TM53x9ytzAoaK9srQtJ2sndL5XwBgOczc2dIm7bFeRkAnV6GqVYfMuqcb4uLh0hxMeAXwFk4P8KguCfv+z8+4E/AW4+d+iPgiD4x9+p3dfi0mlbFv/NE58npY541Lif6DM1WDtDN6HhxCy2LIdNeZYvlNOY7u7PPheDP/8vP0Q8cXsfX0jId+Jmw+RGbcBJzaT6jU/T6ZhE+xmux3TWSimMNRtn+AwT8Q0+6/woTw32Cs8HnW2Wllrs/+ACdxdPUr15g3azwqGj93N45sQel0Tg+fimg9W3cUcu3tgl8AMkIZAQyJJAlsXOfc/fGuxk6eDdQgNkgaTKiIiMpMuIiIIUURDyq89GscwBa8vXqJU3SKpx0iJNVFKIaB69QZP1xgBiXTJFg+s3Jb5Z7rEse0hSE0VU8FSPk6pHfFSlNZjiSKHHXFvm5/o/RJfd7q+YbPOfzFVY9ALilkFe1cnO6yx83/uIpF8s09DttLn01JMEAuYPHaU4MfFCGuftcPyA0z2Tvutxd/I7i/7LCRwHp1rF2dzCN80XF3RNT78hqZ1vaPE0IYQMXAM+ApSBp4C/HQTB5Zdc837gvw6C4Ie+m7Zfi+BblsM//cvTXBY3OVr0OCUeInl1FWf7In1DxY12uO7oPGMt8nhtr7D/3FGZX/w7Hwb5XTf5CXkTGLse37reJCv7FK5+lZur34ZRAUyXMwcKjPsa1kaTfdFvsBI9yL+tfwD3ZdZ90R/xqfwKUx+eYDo7hTxwGGxWOHbgfvYtnXjhOstxaTRGDDojxqaD5fm4ksDXJTxdxtdkfFUQKBKBLFAkCUPeeSUliZQkkUAgAnasf9eHICDwANfHtz0Cy9s5/hxCl5GiCnJMRTIUhPrKlmoQBGxubtLYKmMIgdLzESNBNBkHA1ZurkP/WdJGhHI/z5Mtk4toSGzhBZfQ9STHIxZ+zwQ/zVKqz/bmPP/Ke2jPsw4lWjw6NWbGH5Ic6CS8GLkJmf0PHia+eOKF/+lBu8XyhXMEepR4Nkc2myWbzSK9woIqxw94qmtieh73JGPkX+Ne2W67vWtB1xuR2vlGZ+k8AFwPgmDluYd9BvgkcPkV73qDkIHj7Qh1JrgUbBEtPc2x2fvJBB5B/VkGfoS5xBgnWOdmZD8Va/cf+XeuuPzomcdYuve98ApT5pCQW7HeHOL7PrO9MlvN64yJkeg7bOZTdIMo0eYQTb7JWNf5cvv+PWIPAR80KszfX0I2NLqDNpHtEYenT7C0eBzHD9jqDmlUTcbdnZRISZfRsxGSCQ0jpqJIEooQSGLHeLfHDrbpYlouw7FDdeyy9ly8SgIyQiInyeRlCeP5jdHlHfeOZCigCJ6fFgS2h9+38To7zxaajBxXkZ4fAF62WbgQgqmpKXzfx7Is0gsZnEaH4WoTugGTk9OsFTI0mqeZjtV50Ewi2j2e1SeQJA/659n0JplImPS6Y7atKFPFMse3qzzDxK5nLfdSzBhlvGyBUr6P1+tibcewv3KRhVqNwrG7EKkS8UyWqbl5uvUauqLQbDbpdrsvpHHeip0yDAZPd4ec6w+5N2mQVb97+VSeX9A13v9Caqd78Zk3LbXz9RD8aWDjJb+XgQdvcd1DQojzwBY71v6lWzUmhPgZ4GcA5ua+e3+6H0BO0rirHucJUeRMUCVSPMfSxEnykoNXuYrUl5nN2DzgbPG5ysyu6qaWL/hn3+jxW/knkeYeCEU/5FVjuz7l9ogJs4VdXaXpmahdHUWRWC8lkOoubbPNklHjkns3N9y9hf3uosORQz5O0ac/6jDbS7Ive4DSvhNcaQ5oVkwk08FQJCaLcbIFg2RM22UdPr+pidez8YcOgbNjoQtZ7NTJSSmMpYCegG7g03A9VnyflQBiAkqSQkmSiLjgWx6B471QAlhoMiKuImsyIAgsF7czhpa1UxrYUHcGgIT2Qh19SZKYnp5mfX2dZqfD7OIsuaV5usubDNYbTPsB69H9bGTLzEzFeOiSiluuc0PPgLaI1CjjZ5LEEw1uDnIoMYmPpS5xpZvH5cUZhi9krlcUMqzgFI4wmnDJdzt4bRn7zBqj9oDJw/Noc8fIz84z6veQRgNmF/ZRq9fZ3t5+IY0zcovUTU2SOJUyeKprcqY35L6kQeY1iD68itTO6SnkTOZ1X9D1egj+rXr0cj/RGWA+CIKBEOLjwGeBW+5VFgTBbwG/BTsune+2M3pURS+eZcqc51Qnwzckhyf8Onr2OnL+EDkxprJ2jaICh9MdrpsZLvd31+D4YlXhK5eafEg8CbOh6Ie8OtZbJv5wSKlToeJUsAcK6aHD9r45WpaE1jaJq2vYeowvt47vuT+Cw4fzG0QOz7AxrDLn5pnXZ3GKC5xfaaMMXPK6wsR8hnQhtsef7tseXtvC69oEro+QBVJcQ4opSIb6ggADaECCHWsNYOj51G2H6thh1fVYxSdjKMxEdIqKghh7+CMXf+jiDRx8z96puWMoKCUDIUsEIxevb+NUh1AdInQZOaEhJzXkiMLMzAzr6+tsbm4yNzdH7vgiqYUp+ssV9FqE1abJSmSN0qkTvC+bx35mmWUxhRnpcXDZRiwq+Kk6616SmUSNR8w1vubuLoy2JqY5UruO4j1JZ+Ju3Nw0sdQYZ7vLaKWB0+2Sr2yROnSU4tw82zeWGbVbzM/P0+12aTQarK2tkUqlyOfzKC8rU7wj+rFdop9+jaIPOzMgpVBAKRR2pXZ63S6xhx+Ct6Hgl4GXFp6fYceKf4EgCHovef/nQoh/LYTIB0Gwdy/C75FBr8JG588ItCwl70e4v1fgm7h8O9hCN3TUwgkKnsXWjWVmNJUPFFe5OTzCyNs9Ff1fLkR4ZKFLZOMJmHkAlDtvw+OQV4/t+my0R0y0NvH9EZVhA73rYyQSXEsqUPExR00WYm2+OXiU5i12dfqgskX+pMGKXSYnpZh2Z9lWEhgVn7wsMzufIVYw9rhNfMvd2amqb4MQyHEVOaUjxVSE9OoEw5Al5qM681GdoedTGTtsWjYX+yNUIZiKaMyldaK56M4MYuTiD2y8no27Ze5sjZjQ0CZjoEj4Awevb+M2R7iNEUKTkJM6k7kJNuvbbG5uMjs7ixLXSd81R7yeJ7FR4HpZpbr5DEZ2iUeOLTG4sc2ydgAjf4n3P5MmdWCdaxlBT9PYVzzNma0p+uy2xp8K7uZvmp9nWB8ydk/ipyYY7l8iu1XH6nZZWqkxrNUpHJwhGS/SqW4TS2dIp9MkEgmazSbtdpt+v08ulyOdTu/y7+uSxKlkjKd7O6J/KhUj+Tqs1H8+tVNbWtrZlvENKNL2erT4FHBACLEohNCAvwX86UsvEEKUxHNzEyHEA889t/k6PHsPbhDH2vwQVk/C0v+QklPj7uEE/VGKJ/s3qA6qBAsPkCpNMtwYs6gHvC+3vaedmz2H3742AbYJ5SfB3buRQUjI86y3hgT1BhPeiKrUxqu5xGyf7YUS9R7IHYu0skJTmufx8d5tNifEgPtnt9hMjTFkg7Q5SbsFOSnH4WyMQ8eKxEvxXWLv2x52uc94pYtnOii5KJH96Z0FSAntVYv9yzFkiSVD572ZOPcmY2RUhbXRmMfafS70h/R9Hzmmok7E0Pen0ReSyCkdf2AzXu1hb/QB0GYTRA5kUKdiCE3GbY4QW2MK4wR21WRrtYzv+whJoE7ESB0tceL4e1lKnCAYdVBSAe8tGGTdOBcmjvDMhEJiY5ITtSb6wCYWGXAke2FP/5uk2LSPkXGrWMMnEa1LeL067bkZVnPznPeTbI4MNs+toK0+gdG9Sm3lKr7v7dTfKRZZWFggEolQ///Ze9MYSdLzzu8Xd+R9Z1VW1l1dR5/T091zkDMjkhJXIkWtrpV12IIW/kIIxsIfjf1IfbA/GAvDlr2wl5JhY3ehcyUDOkhRpIbn3N3T91n3kVWVlXdmZMYd4Q/ZHE5PkhxSnJnuGfYfKDS6MiLet7Mj/vG8z/N//0+txvb2NoZhPDCGLomcT8aQBYGLnaGC572CIIpI8fdnE+hPTPhhGHrAvwG+AtwG/iIMw5uCIPy+IAi/f/+w3wBu3M/h/yHw2+H7tMVX11XExDRe/zz0fELt75g2dlg0x6l6KS5Zd6gcNcg99SnUWJxwy+L5fIOSZo5c64uXaxxoxx+T/mP8ULh+wG69S6GxhxxV2d9fRzd80uUyVwILwQhxzSpqVOSl3mlsRqPBX4qs05gNsQkRehn0A4X5wgynlvLkFjKIb7PxDoMQ92gwJHrDRS4MiV4pRn8s/fy7QRAE8qrM2WSU5zMJpiMqNcfj1bbBpU6fpushCAJiVEEtxdC+S+6igHvYx15t4TUtpLiKNp18i/wjiShZKYG90aH65hZewyR0A8SogjafYf7kOaZSZWJAcWmez6QiRGyVb0zNUNHGEfpljrWrJLsuH0/dJK02R+b+Vf8Uhd4EedemG6yhNF5HqF5HyMWpJEtcltNsJBaodVP4B4eod/6Kzs1vv+VDr2kaU1NTlMvlt5RGe3t72Lb91hgRaZjeke6TvvEekv77hY+ctUIYBLz0J3/FzSMdyTHIqC/iSiJd+3nezE5RjRyy7A54Sj/DRDlg58Wvouo+11Jx/t+tpRGDpt84Mc6/+/Vp2H8T1BhMPfM4p/8YD2CnMWDjjWscF3rUIhaV11+haEL7+An+sW2iVS2SwetUxeP8Re/MyD12WjrkhaXXcZYWUMwcU4c658unWHz2JKL2YNY1GLg4+wahEyCltCHJKx/chnkvCNm1HLYtGycIScsSx2L6iGIlGLh4DWuYZhIF5IyGnI28NdfQC6htHdCpNEnpCRKJBGJMQUprSHEVq3fE3s0rNDpR5GSZS1e+zZ9bHWKuzW/eqhGZWCXQm7iZGNfFef5s/zO8s5z4jFDnXys3WM26tJIhGRckfRzGz9O3ouiexWwmwTHXQapcIiFsUVg5iXrylyD5vX64QRDQbrdpNBoEQUA6nSafz7/Vbavv+1zsDAgJeSoZI/aQjRh/qqwVeu0O126s41ev4cgxetLn0EOBtPQdTjVWiVt51lSdm52btKoa6TNPYA8ETppdTqdGI4W/vnXItaoEE+eGkf7u6+D/8G3Wj/HTgzAM2a3USXVqREpj7N69jG75FBZWuNhvEpgiirmPpRd41ZgbIXsFj2cz1+mNZQjNJPGaxdmxWY59/EGyfyuq3xqWw9SZJGo5/oGSPQx3n85FNV7IJFiO6ZhByMVOn4udPq23teoTowrqVAJtIY2UVPGaFtZ6G7faHxaUZZHisTKxxRytmIUVGaqB3IqBtdpC6scYW5whXxrgGU0uHH+On1FCelqEr82N4RyWEP0YctfhnLzO8eTOyFxfC3Js2hMsN+MUBjnqShHPriFVXiQhbuJIHre7XS7rGr2lz1IVn6Zyew/v9T+Gm/8fmO3hv0UUyWazzM3NkUqlaLfbbGxs0Gw2CYKAmCRxITncQHWxO6DvP7qR/keO8DVNJzM2RdcC+/AapgWD1K+jKhFK0kWeOLqLaOW4nZC4XruJKk0SmZ/H78j8oraOKjz4nxUA/9PfXCeM5u+TvgF7bzwm/ccAoGbYeOtrFNJRdnoN6JgkFYUtIaRmq2h9k0i0xs5glu1wVOP9jLrOoNxDEqbJdOFkssTshZNIb4uYQ9fH2e7i1U2ktIY2l0aKPdxVpiQIzEQ0XsjEWY7pGL5/X7nyYD5b1CTUifgo8ddNwvstCiPJKDW3TTChDesBSRXfcJBreVJKhPzYIa7n8HNzL/BMaHGYSvJ66RhetYjgiQhNl99KfRVVfEdvWEHgT4I8Zi/C3JHAgq/R1pbpuRJS7RqZYAM96LLVOuBNb5/GzAkqsU+w3l7AqdyD1/4DrH4V7GH+XpZlxsfHH8jvb21t0ev1iMkS55Mx/DDkUmeA6T/YcOlRwUeO8GVN5vjpGJmxHI4nYFRvYRz0CCZ+A2IF5pQbnD66i2umuRVzubq3Sjwzh1wokOrJPJ/cG7nmq1WDL72+DfECTDwJdg/2LoI/2nz4MX66sLtRQR90Sc1NsX3jFaKuT256kVc6NXxHJW4d0JEned0cbdWXFAYslG6DNkMxiFDUJJaXTxNNpd86xu+72JtdAsdHnUqgTsR/LHuD9xviW8SfYDGq03Z9XmkbXO89SHqiep/459NIUQXvaIC93iboukxMTKAoCpVKBU8OUSfi6IsZtMkE8cQJYoLIWHoTXYzwVHaWC4HLYWaMW9EVwk6K0BVJdbt8OvXayPwqUpYXvR5B1Saz7bEUugTReZp+lrB3QM7dJetaNNoNbnpr7CUM9plgK/I5uuLsMLi7+H/D1svD554H8/uCILC/v8/Ozg6y63AhFcMLh6se6xEk/Y8c4SMEDMI6x+ZdEvkcvh/SrK/TvntAdOpz+JlZTqp3WWysY1gJbusGq5UqydIiWjLDc84hKckeuez/+A+3sW0P4kUonQWrA5XHpP/TjJ7l0r+7Sj6fZnNrC7HfJ6WprA5sjoQocdMm0E3u9cs0Ge2r/HTkFmFGp6CWyWge07kpClOzb33uNS2cnS5IAtpsCinx6EqDJeF7qZ7ZiMah7fKdVo+7fQv3bS60oiYNUz2zSQRFxN038Hf7lDJFBEFgb28Pz/MQRAEppRGZyZNbepZoQaQ0VWeiuMy0HnJC8GgVZ9l1lsDS8DyFnxUukpU7I3P7evwstyLrKDsHpO50WRyYKEqBulvE9Hpk3T1KgxCnJ7GumNyTb7NZX6Uae5Zq5lfw5RRsfhOu/hnsvgHWMK0Wj8eZnZ1lbGwMx3HY2dnBOKpyJqLghCEXu33s4NEi/Y8c4fseBB0N2zGZm3GI5fMEvk29tUfr+h7p0sex8yd4Ultn0tinZUW4LbXZbRjEJxdJR7J8Wl4fue6+FfA//+WloYVyYgwmzg5zfJVLEDy6ObvHeP+wd3sDyTJJT42xs3aFaBAQyZW4ahzhezpRt44RFLliF0bOLUl15qaOSKuzFJISqiSxMnf2rb6r7tEA97A/VLjMph5Q6TzKUESBpZjOC5kEJU15S865bdoEbxOIiFHlrTpE6AWE+xYFIYVrO+zv7xO8jSjVaJbE+Am0ssPieQ1teRolUmNCkujnz9DoroALiifwa7FvjczJCBN8O3uW66UaVnWbxKbB6VaTlGFz1J2kJoREpQ1m2gaJbpqKXuaacMid6jc4GnTZUp/FLH8aBBE2vwG3/3b43JttBEEgnU4zNzdHNpvFMAyae7vM2H0GrsfFTh/nESL9jxzhy4LAslAiZUUIAoOpCYtItoBld6g0qhjX9smmV3ByT/CkvknGbVE3RW67hzT6AYnyMc4ILpPSaKTwF7frvHxz/z7pj0PpDJgtqLwJj9B/6mO8/3Bsh+adVdLjOSrr60h2n4QC6x2XQzVJzHFxJY9rg3FMRvPtzyRWScdSpNNppMBhNjVPKj9GGIY4+8YwX5/RUSYfrRTOjwpdEjmViPKxdJyULHG3b/Fy26DmfK/2JQjDKF5bSCPnI6iuRNaKYhx1ODw8fOB60egMmlpAkCt87tkLGCdEWsldhJiEkziN3T0NvsJ5YZcFtTIyn7vWCpVskdWyza6xhtiPciYMWKyv4t2Ls+HG8eIVJltrzLRUutIxXncj3LJ3aBrbrB4N2E+cwSudAc+C7Vdg7Z+Gqd1B862WinNzcySTSfxel1yjSqPR4I2W8cAq52HiI0f4fgivckgiWmbMSyKHbcoFi1i2QN9qstNoEt49IhcpI2We4FRij4ho0eyH3OhtYIsxxsvH+JQwmsvv+fCfvnaTxY8UOQAAIABJREFUq3vtIeknJ2D8NAzqsH/5Men/FOHg1hqh45LOJtmrbBAVPUIlyYbVwA9iaN4Aw4lxy8uNnLus7HCq5KCn88hygIzM8cXzEIK7Z+C3beRCBLUU+9A0x/5BSMgS51MxnryvYrncHXDpHZp1QRRQilG0uRTJbJq0H6W5dshRpfrAteLx44iiiuRt8AuLFxDnTBqZHtt5iYE4j2suIoYRfkd7HeEd7i5OIHPdPk83kqMVC7jZuUojVmZ5YpIlZYf8VYPdXZ2m2iXVv87ZZgvNTXDRSHNVjjJQJWr1HqtdiVa+hJPKE9ptOLgK2y8N1Xv9BoqivFXYHU8mKNkD9re3+PbePs4jkNP/yBF+zxrw5c42f9q9jBifZEYrEhebjGdd9FSKllVno94jslEj6aVIR04wla4S6gKdjsPFoxtExqY5n8uyJNRGrv/NI4e7t7a5VukMST9VhrGT0D+Cg8ek/9OA0HGo39tAy2dp7G4SegYxQvZNhWo0hhyA6Ftcsibw3vGISfh8IlZBzSkIWgLRdjlWOk4kEsetGPg9B2U8hlJ4733SHyYKqsLH00NFT8fzebltcNswH0h3iJqENpNkbGWShBanenuXxvoh4XedPUWFROI0QeBwIhFltlRkfKqHkRC4mlNoD8oYbpFZpcfT6ubIHHbMDJvqLN1IEcKQexvf5pqeZubkCkvTIeN1n86mzkGnj9m+w9naNlNdk9WWw7c9nWZyEsIS+9UEVT1DJ6lgKj5e/xAaG7D72jDyN47QNI1yuczJuVmWU3Fq1Spfu3WHZrvDw9z79JEjfEWVKEY99jSBP6m/TlMrspApkdMajKcD5IjGgdVgrWGTO6ijGzrTyjypXJt+OobRMnh18zLl5Sf5rG4g8iCBW6HA169VOKp1uP5d0k9PQ/E4GEdweO2t3XqP8dFEc20T03KJayJ7jRpx0cN0RWp2A1PMovkBdVtlzU+PnHte2WBpwkLIFAgHJhEtysrME8Pi5X2yl7MfzQY831X0PJ+JM6Wr7FoO32kZbJv2AyQopTSmzi0QyyU52NyjdeuQYDBMBSlKklhsgdDvcDZbJjIGL0xF6aVVrscFmu0CDT/Jv4pcR+HB2pofChyF02xkSjSjY0SskIM7r/ItB+JzeY4fUxlLFbCCWVoDgWptl+mtm5zd3cE0DF5sNrgqiciJFaz2Ej3xDP1CmZ7mYnj72MY2fq8yzO9vfQe6+0QjEc4szPPMwhz9IOSljS02t7ZGrBo+KEhf+MIXHsrAPwq++MUvfuHzn//8j3dSv0Xq0t9iKxJbYoKdzjaR6AQriQi+V8MLdLqWi+F4iEGE6bCLGSaQozLN6IABMcR6A8cZcHZhmc2DfQ540E1zy4RfTln0Y0kGrk8hriFEM8OiTnsbXHOo5vmQL8cfYxSBbbP72mW6soY8aNIzm8StDg1PY10T6csFUt0W3zQn6IQPErcu2Pxu9DrqcQ3LzyJ6PqfnL1Cws/hdB2UsipwbVfP86JMLhjWlfh16h9DZG/7ZOxwGI3YXPBvCACTtod2fkiBQUBXGVIWeH7BnOVQdl5gkEb3vFSRIIslimr5j0qk30QYikiAiRhUUNY3vGcQx2TU7SBmYHRS5EXjQ66H4IWq0hxJEWPWzD4zdNgWeHE9ypCvIHuTaPbq9Ljv5LKWUxZhvIcenqIytEAgeplFHrzeZqzfwsknuOR4H2Iwnsmh2GpgkUp4nUAVc8wDXPsIPTPBdBKOB0DsEQSKdKpDNZNj3oW0YCP0eg34fRVHetePWj4s/+IM/OPjCF77wxe/73X/UCF9Qohys9SkY1wlVlYqYp9arYMppTuRiKHSwLZWWNWAQyERCjTGvhe8n8RLQ1kNcQcKt18mIKstJhW91JPy3LYZCwOjZ/PJ8jIqnMHB8igkNIXr/5mpvDzdmxUfVGY/x4Ya1vsHm5gGyrtDsN9C9Dr5h0rRNNhNlFC9KwzC56I3q7n9OvcmZ2Rru5DyDhkUum+ep/Meg6/3zyd53obsPjTWo3oTO7jC9aHUg8IYE79ng9qFfA6M6PKa1PXwBBD4oUXgfnBnfDaooMqGrJCSJuuOxYzn0vICkLKGIAqIoEk8lMHwLw+wTtRVCw0PUJdRoAdc5QgoGrPUbPLd8ktb+gJ0gJNKyCFSbCX2LK/YKzgPeRQLJUORkOqSix5F8kWKnh9X12YjHSKXbZAdtsn6C+vxp5NI0TfMIv9OjvFclFTjsej6rvomWTpCyJdyeSrK4TLR8llCQCPoVXKeOHRr4vonQPUDsHZGQJZLpAkdqFFeUiLo2nXabwWCAoigoynuzme6nivAHvR53v/lNfCvKuLBKqOjUpQm6xj51V2G5mCSluHR7ULd6uGKUjKARt5sofoxmBnqKjOA5tFt1TibGaFt9Nr0HH8ZNC35eN5ieHWPP8DBdn0JCQ4jlhg9Re2sYScV+eKPkx/jwILAsqm9eo+qCHAzoByZ6r45h++wqPo3IHKlum69bJfrhg1FbWuzxu9LL+KfHGVhx8HyeKF8gZ6eRc/qPn7N3rSHJH14D43B4zyVLkDsG+SUorEB2DjKzw5/sHGQXhsdEMsPoflCH3j60d4YvBSXyUGzAY7LEpK4iCQIV22XHtAmBlCyhyDKRaISubWBJHjFU/KaNEAqoyTyRsMdOb4cDb8B/e+qTvLR7QNdxSXcDwpiBJlqsu5MPjLdvCfz2soAbmBypcUInpGAM0AcJDvUMUmKb2KBNzhBopMYpHDvDUdiha5skeyblboeWNWCt3aCtCxTVBE7LQRAUEqVFtOI5JEFFNNt4XgdLCXCdOkF7k4SxT1JS2FcTiIkME1GdQb9Pu93GNM33hPh/qghfVhX2HZlO20a1oBDcxVZketosllGjagaMFxOUowr1lk3d6uFLScZFEbnfIeJpVNIatqYguz1q3TY/n83z9a6I+w6Xw20j4HfzNtrkBLstC8u7n96JF4YPUHt7mOaJZn/AbB/jwwR7fZ3drQN6SASYCHYLsdOmb5uspkv4QY5mt8c1b2zk3F9W32BmziScnGdQc0gns1yIXUBNR1HGfww1ju9B7c5QHWK1IT42VIoVjw/TiGpsaO73/a4nCCBroCWGe0kycxDNQegPVwnt7eFuUj0J0gdL/KIgkFFkSpqCFYbsWQ4HtosmimR0DVVVafc6BDGRWCSG37JhICAnIki0WO3sksiW+RfRIl9rd/AHFkVbJBtf55a9iP2OF/B6W+b3Tlo0nQGtWBx34BCz+qS8PKI8hx1bRTHrJDs2bTlFZuYEtmfQwwRVZtLxsIwBlU6HnW6VmKigOjKuFRBJRpAz0yjpY6ihjmLbIIhYEQ3X66C1b5LobnNk9nFiBRZLE8iShGEYtN+DiP+nivB9P+DWtQ2seAZTmSTmDyg4d+n7Af3oPIHVodntoxViHItlOKwZNJwOaBkmBBCNHlroshXP4GshctBlYDlMSDo3nQdzsgc2nNI9nsiryNksu00Txw8oJDSIFcAdQGsLRHkYVT3GhxaBbdO7cZO1poUs+/QVD615gGcOOMJkO3GKlNHjxUER6x26+zG5wW/xbQYnJ/HdDGbf4mzhAqVceWiA9qP61ncPYP8SDBqQmoLyk5CeAuWfWeQVhGFUnxgfXk+UoHcwvGd9G/T08HcfIBRRYFxTyCgyTddn13Jouj75aARdlmi324hxhXg+SdBzELoqcU1iz7zHTr/Ox898htLeIa84Dn7fYxIfRa1z1517YJyWA3PRCHMFi9AysaI6fcPCcw0kKUMu+gSitkfgVNHrTRwLxMnj+HZAKFoIikQurqB6PgeWx1anjtNrEmvZ2D0fLR1FiUYRUpNI0SKK66O7IpKaxstMotBD6W7QPLxBp7fPeDpNfmwGSZIxDINer0c6nf5nyXJ/GOF/5FQ61qCPfLiNulelHwTsJX4OIXWW0+EW+f5dnFgaMxDZ3KmyFe1xdmUeRRDZ7VbYQiMTKkwfmqy0D+kxRiceo6Z3mIt2yAqjfvj/62aItb3DjOIxm49RaZncPewNH6bxM8MIrHYH2rvfZ7aP8WGBs71No97FFCSISPhGF8W08ByLneQEgqex0/doh6N5+F+SX8OfSSDHCphNi6SWYj67gDr5I5K97w439x1cGRZbpz8G46eGZP1eQdYgvwhzPzN8ibR3Yetbw5fMQ0BWkflYOsZKTKfnDf15qmqEaDJJs9mk5w3Q5od2E1pvhieEJQb9HS4dvsFTn/0UPxON04vF2bZKfFzepSiNOuH+yW0ouQVSWZ1x3SM5HqUfeFR6u6wbLfzspxkvldFjRyRqN4hfe4MYUUJ9GjuWwhdEplIS5zJD98+XLYu/7+yyefcO2196jdqb9wgsa7jymn0eYfwMKipJwyEdP83k8i9TLC3T7e6zdvOv6d78YxT7GhMlnYmJ4gNdtt4rfOQIX9FErKkGod4h1e1itzqsOxcIC09yQqqQMQ5x4yoeMs39OhW9yrH5ZdxQYK9T5VDWSQYqK3smJfuAhjBHPyqwpx3yC+nByHj3uh5fqQtYd+6wkIswnYuy2xywdnSf9Etnh9F+9cZw2fwYHzoEjoO7v8+RYSMoAj05JDLoEjoW3WDAkTpHxDK46I6PnDun7HPa26FfyhH2NRzL5vj4aRKz+ZFWhd8XVhe2Xx6qbArLMPNxiIzKPd8zyNpwX8nsc8Ni7sGV4cvGG/WXer8hCALT943ZvivjvKdE6ag6h4eHDCwTdTKBPplmNvIsE3ac61tfw9YEfv2Tz1JMJOnoEar2Er+mvTRy/aon8LVVmWUzB/EYYwmZclFBsAccto+4t19jJ36cmeVlitMBqnyAWtskfVgn0othKwX6nkJa0vlk1mRFNdmWJP5SF/iWY7F/Y4ftv3uZ/qXLuEc1wmQZ5j8J+SUEs4V2tMliYpqpk79DrfACB26UsHIJZ/W/YFe/Qhi89z5dHznC1wSRj0f7pPJryIkaGcGFnsm92hJOdollpUJiYOKkZGRZZHB4REvbIl9eph0KVLoNOopCylN5YrdFwjniKDyOqduo8VXKyqgt8v++5mEbA+z1dZbGEpQzEbbqAzbr/aH6YeLJYUrn4NrwwX2MDxXcnR2MWpOGFEGMy/TbLSKWiWd3WE0VEIMo24ZP//v0qf2XyncYlFPIWgar5hKNpllZOfNAQ/EfiO4B7Lw6zLFPPQ3Z+Q9OSqklhiuJ/NJQ3bP1HRiMRskfBBRR4Hg8MrRpUGQasSS3QplbuxUsy0JKacSPTfNk8VP43Q6vXv47IkuL/N5MGTkS5VCMM4HMMWnUcuGvqw7mIMJiP8VAjFFIJ1jMhsTaR+x3WqztGFzp5UiPT3PizBjJqQGWYuB1KyQOB4hGCqMrYpsxzqQl/mW0Rs7v8FJU5P+Jx7geRtnfNOheuUH/pZew1tbx1SLMfQIyM9DdZ7Z2mUU9SqX8GSoTv0osfoKYIyMI73067SNH+KhRimf/G14Yn2AlcYVkcpOc7qF6ChvVRaxkgdlwB3UgYuaTpHQBv1nF1dZQCrMcBhJVo8tAkchbGmeqB4h2jyPvJP34gHOZrZEhd3sOf9OL4+5V8FotVsYTjKd01o8MdhqDYR60fGFYDNu//NAenMf48RE6Ds7uLvWuiaUr2FKIbvVQTIOu43CoLSO4FledUQuFE9oGC04LYzKNYkQxfZulY6eIp1PvPnBrexhd60mYee7hFP4FAXILw1WFKA/tA5obH/w87iMhS1xIxTiXipPJ57jlhnxtY4euZSMoEnMnnmOiMM9q/Q2qt+9RePZZfi0SI1Bk7qkL/Jp2ZeSaXVT+cm2dUNEpeREMK8JYYYyTeZGxwx0O+xabjZBXGxpOGOOJmUkWn4hgZAWOxCqe3yLsCti1gG5VQvfSfCrZ4xeEfdygyV/JHn8aRrgTjmEqKdz9fQZvXKR/6QqOFSOYeAYSJabMA07VL9Lod7ibfwZl/rPvy8v9o0f4ANk51Kc+z8LpX+Wp1B1mY69T0jpExRS7zVO4qSRldwOvK9AuTTKVEYgMqmjqGlamxEGg0DUtLEWh1FVY7Ozi2R51d5FIfo8prTcy5L+/3sZRNew7dyAIODmRpJjUuFftUWmbIMlD0leibzntPcajD2dvD/ewyqGagohMo9chaw7wrDZ3E0kgxW43wArfqagI+SX1JXoTMcQwhtsR0bNpzqyce/dBG+twdAtiRZh8aphmeZjQEsOXTrwItbv3faMenkPsmKbwiXyapydLND2fv93Y4U6vT4DACyd/Ez8p8kbjS/hNn+NnznA+1PBDn25siSfl0RfWS+40d7Zfwo+qKLpKu60ylp7kZFFkavsudcNlr6/xrbpA3ZCYS6T45LkCYTZKSx5g5Vw6kku362DVXaxGjKQD/0o85IJ4xI7S5T+3u/z1fshGZhnp2DEEWcJeW6f/xmXMaoAbWWQ8Nc68uYu9/g3WDtcJ34fv+KNJ+ABqFHn5c6Sf+x9YmYKz0ReZV7bJhCkq3ScI0zGmrHW6TY/twiLzRYmCf0hcWmc/nuUQFc+18GWN+aZL0TrA6as0mGZl4u7IcLWBy591EwSmhb2+jiAInJpIkYur3N7vctixhhrnyaeGsrm9i281VHiMRxOh5+FsbWEMLDqxKFboEbG7xMwBLavPQfQ4gedxyxqN2M9F7lG2B3QnYkS7SfqKz/zCCZL6aNerB1C7C/V7Q2O+iSc/cJXMD4QkQ/ncsI7QOxw2BvFGRQwfFERB4Hgmyedmy6QCjzcrh3y71WMgplkqPc+WUuFAvIaYKvPCWImyJ9EO4alUf8RYbYDCtU6Z2vZrDFSHVlqg1RMppxc5nZWY3LhJv+vTDOP8U8dn80ggicS/OFdAjUfo+x6xsTi9HOxi4ZgeGBJGX2d20OY3/RpT0R7X/DZ/eWObv79hsFdaRLnwFOrMDIFpYq3t0N8wyAYT5EMNt7VHwOMI/8eDICAVjhN/5t+SOXGS8+mXOK3dpORHqBkn8TMJ5gbrtOoe1xInWRhXWJAPSEub3IrEaIdRpMAjJMJSrYXuNvHqKUjEmEqMGqv9h5d2sQvjuHsV/HYbURQ4M5kmE1O4ud/hqGcNJXSTTw9z+7uvD/vkPsYjCffgAHd3l1osx0AW6dh9Jhwb36ixFssQCFm2eozszxAJ+Jz6Cr1shNCKIgUxtLEMxyeO//ABG+vDlEl6eqjwegi7X98V2fnhi8jqwM4rD/3+zSWTPD9ZYil0sTptbhkmYvJZPLnARes1KNvE5xf5lBojY4X0tTHO66OKuW+G8+i9Hv7uDQy/y2q+T9P2mBk/yfm4RnHrNmHLw1YSfL3vcrUiojvwiZNRUnGNrheSTSSJFBJsRwV6hEieiGdJtK0+zzpH/JxqosoDLtYbfPkb93jxToPtWAHx/FNEzz2JPD6O17PIdjWmutH3hZwfwTvqJ8erF/+GvYPvReFirEDiyf8e+dznWBhb42n9IvOuwKA3xyCdZcbcwOh5vJa4wExe5IxeISvt8rKq4IZRoqGA6mksdKqEgUF4kGVqsj4SKXT9kD965QBBU7Hu3CH0fSRR4InJNAld4UalQ7PvgBodRvqEw0jJtT7gb+gx3g1hEGDduYPvuByk8xiWieYbJAYmDctgT5/FCWDVjI2c+0zsFlnToTmpkzYzWBmV8fwk4+lRFc9baG1/L7Ivnni0fZgS48P7N3CHReWHvFJNp9PM5HPMuyZTno0i6sjJj3Gzb3HTeAXt6WnKi3M87UjoRof5bAzpHcZqDhJfDp4jY9aJH25imQMuR+sc+QNm51Z4SlFJ7WwSqwVEYknesB2+uSUhezLnpl0m4iJmJElUjjARi3GYkKjqGlEtgepIHPYHEB7w6Wyf2YRDyzW4eXOXyy+t89L1A24b4M7ME3vuOSKnT6FNzyA8lmW+O/qDFluVL/PVb/1vfOXFP6bbGqpiRFknufRfI1/4LTKzAy6krrIcOCi9Ar1InuJgG2vg8XLh00zmZM7Ju6TlCl/XZBQiJNBI9TyKZgM7dEk045Ty9ZHx/+PGgHrDxO/1cDaG+UJZEnlyOk1Ekbm626YzcId50fKFocZ67/WHujx+jFF41SrO1jb93BgN38fwTKZDl7BdZVVNEAo5VvvyAx5LADIen9Fep5dUCUyNuD6OkI6wXFz+wYN194c5+3gRxk4/2mT/XUSzMPXscK67rw0j/oeIfD5PIpFA7LQ4LQV8ZvIcnjrDVw/XuNG+gvzsCVZWyqyYMknb5mRidL6v+Hm64iJpr0fmaJf+oM/18IAjxWR2eY5z+EQrVVKHARPpGKuhw5c3FSxHZybTYDEa4OZyhHqSRT2Bg8WdKESLk+SkDFbNYa1eYTlf5+yshxtzWOt1OLi9x+HlXS5frHBltUFbT6BOT78v39NHjvCjkSTPPf2bTJbzVFqX+POv/Du+/cqf0+93EASRxORn0C78ayILUU7k1jlJh+IghhVmiRgVnIHNG9O/ytREjPPiLpq0y4txmaQUJxnEGG80ifo9XFdkXuohCA/aJ5sIfPGmiddoY29s4reHxVnlPulrssjl3RY9yx3qqcvnh+6ae2887o/7iCAMQwbXroHvczQxy1Grg6q6ZHt9WoMuNXmcviizNRjd+PRc/AaJvk99Qibr5LCKEVKRLNOFH/AAD5pweB0iWSg9+WimcX4QtDhMPQOCNOz1+hCFCIIgMD4+TiQS4ahapSxK/Fezz+IKEa7V7/GaX6G5OMfJuSRTXYeTuoIsPPi8BQj86eAksp4ljUO2vk/NMFgzqnTSKjMnxlnx2kiVFsltkRM5hZrs8bVKlEovQkLa4JlCFLGQpJfIspgukR143Ogf0illmSkvEzdjVFYbOMYmT092iZc81pQBt4wGTr1Bf7XOvVcrvPn6Hr733vfW+BDdXT8awlAgcOY5MfeLfOKpn2N8PMWt7Tf463/4X7hy9R8xbYN47mm0J38PbXmMuYkGJ5QjjnkiohUlbO/R7hhcX/h1FhbGOOsd4QTbvBaFopol5Uco1yogOCQskXK8NTKH/9IVOeirOJvbmFeuEt5v8qArEk9OZ5BEgcs7bQaON4yUJp4cLosf98d9JODVajgbG0jT06z1TEzPZUaxodFmCxlPTnOnHyF8R1FNFVw+rV2ip6mEvkY+NYsXkVjILSDL8uhArgn7b4KsDwuiHyay/y7U2JD0JXkYtDzESF8URcrlMoqiUKlUWEkeYzk9iel1iXqH7OZENiYnGZtJstg1OBEb3Ux2N9R5o7WEkkhRVGRSrUPu1g7YOtzBKZWYPp5lMtjDqjbQ1yI8lxTxowJX2hmuVBXatdf52flxMrkE+8kYU+OzLJKkcVjlqtAkv3iMscQkQl2lUakxI+2wkOrT0lxe9pscqkfIygAt8JDkxymdHwkSIoaRQRWmefbkMzz39HNosSQXb/8T//hPf8Td1VdQ44tETv0O2sljTE4OmI0ccTLwSHUlhPoOW7Ue12d+keXTZU64PWpscTcRUtKmyVhQaG4zEGVOeG3Ed0T5Tgj/VzsBeobBteuYN2689VlElTg3nSEELu+0sVx/uJQvnQGzCftXHnfNesgwL12CEHpLJ9g+qKHGPIodi16vzpE8RlvU2bVG/Wt+JnGN2AAaEwLZoICdjRNRYswWZkcH8b3hCz5kuMqT3htr3IcCNTokffE+6dsPp7kHgCRJTE5OIooi+5V9ni0+iyfomG6TE7EGqWKCbi6FlFP5WcFBER4MsEIEXvQzbDfGEDMpSnqUhNHk8u499vY20OaPM7+QIc0WtWYNeyPCJxWPTFxhzxnn1QOBq3f+iU8tTbIyUWQ7KiCWSpzJTBFv+FyvbtMuyBQKk8T9PFJLJ0aNlcgRcc/matvitUGFtrJP4D3eafuuEELIWTGipkzfSGCaacYSCV648CzHz34K23e4dO0f+eY3/iO1loU+9znkM+eZnvYZT9Y5KXqMtXwi1XVuHfW5Wf4UKyfLzNg+G94azZTLpL5CzjBJGTu4qsqCNJrL//uayVb2GIKSwPjmN7F3dt76LKbJnJ1K4/gBl3faOF4wLNZ9t1Xi465ZDw1evY69toa2MM/1ozZ9y2U2LcBRm6rn4yNzzc7AO6J7TXD4lHqFnqjiSTLjsXnsqMh0Ypp4PD460OG1ITGWnhimRj7sUCLD3cAIw5qUM2pD8oFNRVEol8sEQYDQFZiJz3C3b0DgcKJkciEZI1mKkibgSX10nnvo3Oml2KvFEPN5SokUmtXjjVuX2d3ZILV4luWJNAl5k53+EUd7Gk95JsdjOn13gjd3Hb50+aucnMjwM4sLdCISu2mVE+VZZoIM3f0eq2KLIKOgkiTrlsnrCaaLbaYZ0G76vLLXw/YeUR2+IAifEQThriAIa4Ig/Nvv87kgCMIf3v/8miAIP8Luk3/mXCQRRTwkj0iir9LvROn3s0iByfGxDOee+xXyc8dpdqtcefPLXLl2ETdxBumJ55iehXy2xpLiM9U0yR7c5dqRzb2Z55mfK5K1VO54q4Qph2n9FLluC908oiz3kXlnPhD+j3sNOPkMoS/T+4d/wD36nq1CKqJwdjKN6Xpc2W3j+cFQjpdfGjoWHt16v76ix/gh6L/+BiEC8rnz3NjYR46HFPomdrtJTUhxKKc5sEdz959IXEU3oTYRUhDGCHNZZFFhvjg/Okhzc9iIpLD00WqSo8buq3eC+0KED95/57vQdZ2JiQls22ZOnANEbhgGoeSRnvT5WERnfi7FzwshmvDOSFrgupagU4uwfSQi5YoUMwXcYMAbb36HncMdxpeeZCUeJyVVWHdrbLcU5vweH4um0ewJVrc6/NXFF5Fkj188dYJoKsXtCCRKKU4mJ0k2o9QGDodRg65nIncjzEkzzE+rzBUsppDRtffeovonJnxhaPjw74HPAieA3xEE4cQ7DvsssHj/5/PA//mTjvsD4bu4VFGFO+TsKumnmmP8AAAgAElEQVSuz6Cu0O8X8DyDnNDi/MoLTF34BGo2S7W6wZVL32avIcCJ55lakMnlG0xpMrPNLqW9G9w8GrC79HFy4+OI/SibrBHL+Mzqx0kPamh+h3lhNMr/Rt3kcr2HfOY5goFL78UXcavVtz7PxFROlVP0LJere/f74+YWhj7l7R2o3XvfvqbHGIXXaGCvraKvrHD1sEV3YDM9JhHsNumaA1xB5M1gVFqpixaf0q7S9XV8TaCsLzKIwER0glTqHZuyzPZQfhkvDjXtHzXoSZi8MFSdPeSaVCwWY3x8HNVXKYtlNo0qrUDHTwuomYDJiM+Jgs7PyqMpqN0gRieq4FZFdo581PQ4mew4HWyuvPIiW0c7TK+cY1nSyQk1doIGtwaQVus8m8hT7pWp7TT46tWXuNM45NPLS8yVy2xrCrUMHCtmKQdF9H6SpuCy41dptHtMOCWOJ8eYWlYIwkezaPs0sBaG4UYYhg7wZ8CvvOOYXwH+YzjEq0BaEITSezD2CBxB4iVtmevZYygFi1y4T65ZY7A7wOgW8Xwbwd7gyeJJpleeJnXiDKGqsLu3zc07u/THnqS8rJLL1cjrOnOdHjM7N1mt9+icPIOcKdPuRmmFq2TTErPKLHGnSVmsoYbvNFYT+MPbLdqSiDRzCr/ZZXDpTdzK90ycigmdExNJWn2HG/v3O9oXVyA1Cc31h+pd8tOG/muvIYgS+oXzvH57C0EXKAg+QqND2xVYV7IcuaOdqT6RuIJkizSKUBALCNkCoSAyX5hHkt62Kct3h7YEkjrcWPVRRSQNE2eHTp/7Vx5qejKVSpHP55mVZ8GCi41tRDWNM+YRj4qkCzK/nouTZFQW/bKSoyA40HDZbjlEkhNEM2NURI/bF7/F2v4aC0unmHdFskGDtm/wpuEhpyucyedYaEzBXocbq5f42vodFvI5nl1apB9PshqFWMZlMlGgGEyi+kn2zBo3mqtIrsqcPYn4PmTc34srloG3b13bu/+7H/eY9wRSECDeuca99X1eVY/B8UWyuZBCr8Hg1ha9apzADxgY11hOTXJi4izxlRPoU9M4vsidtR123DxjxzXG8g3ius5Ut8fCzm32Wz2804u0ItNstmOE4SZjKZlJZYy032NWqI7M50bP5cXNHkYuh5CcwGu2MO/cxXlbTr+UirA0luCoa3P74P4mlrFTww0utbuPvfQ/ALhHR9jrG+jHV1hrGtQ7A8ZLEvZWC7fbwZBk3mBUWhkVB/yMdp2eG8FLeEzpy/S0gDF9jGzmHYZnh9fAd4aqrA9zkfZHQbwIYyeGNanqzYc6lVwux3h+nHllnv3mPttWiJIt4qfbRAWXsXGNz0VGXXD3nTiv5suUTB+t2Wer4xCNjqOliqzLsHXnIne2r3N8fokpUyLhNfGxudT0sEu7LE6lKTemyFds2ju3+NadG3T9kE8sL5Ecn2AzGqentMilAwrRacbkBRwj5GrtNjf763iPqD3y99sl8s5X+o9yzPBAQfi8IAgXBUG4WKuN2he8G8IwQPNcItU9du/c4uubLcxj58mdWaKkypi3N+jc6xN6AZ3OZbKKzIWxC2TGZ9BXVogUp2kYPndqkJoPGct20CMqk0aP+f1Ver0u2olp9rQZ1toKkeCIiVhIXk+xQhU9HI0U/tN2nYrhYuWKQAw8H3ttHXtj861jpnNR5gox9tvm97z0x594m5f+w2lE8dOAMAwZvPY6giyhnTvHqzc3EGWBSERAP6xhWQ439CJ1f3RX7fPJK4iWSDsjUpDSKJlJfCFkJjuDqr4tB9veHVpj55feXz/7Rwnp6WHaqrM7rFs8RBSLRU6NnSLiRnh9701EbRZxPIeSaqKGNr+2mCHHaM3hqqWxOjVBuh8Q7bbZMUGVC4h6mlVVoXqwwZ2NyywXi4xbMhG7hSgFXN1zMEp7zM5rxLrjTDREtMoGq6u3uFqtsTxWZH5+nlZ+gn3RQ5H2SWSjlNInSZlFzCMTIXw0vXT2gKm3/X0SeGenjx/lGADCMPxiGIYXwjC8UCj8+AUtQZSYWD5OOp0j77u0G3W+cfkKFVci9fHnmSjPYVV6tC9XodOg17mK6Lc5N3aO6fQs8mSZyOJpwliJm00fsdSlmLBQIiJlo8fUwRaC20Gem+CmOsdhNyQZGMxEbGLRGMvC4cicdkyRv90+oqpr2LKO13YR02mcrS3s1dW3jlsoxJnMDr30tx7w0s8Oe5gaP/4L8DHeHW6lgrO7i7a0zKbhcNjskCwo2Ps91EaXlgSvMztyXkLq8bx2na6r48ZsphIrtCWXYmycXOZtdslOH45uD/vHZkav85FGful7LpsP8f4VBIHJ8iQXShdot9u8sneNROEMZCV0tU1c9fmN0uiq68hNcRAa1ErjZPsButViL1ARxByuGGMzGqNlNljfvsqSopC1I0SsNqoqcHXTopnZZ2ZGQHYLZO0oucoe9uYGV7d3ECWZE3NziDML7CWzOIM9lGiTZHmWVGIZQXo0/fDfABYFQZgTBEEFfhv4m3cc8zfA791X6zwLdMIwfF9CVkmSmJmbZ2blOLFIhLIi4cgql3Z2uX33NsLcFBOnn8Ky4jRveojVDXpHL2GZ2xzLHON0/jSRZAZ96TTxqXPsuBLW2AE5yUXWA8qDFsXaPmmxiVsq8W11DrsHKc9mJtrhWMwl9n0ihW8e9Lix26BeyGH3bbz6ALk8gbO7N/TduZ/nXB4beumvHRlDW2VRGuq0tcRwk85jL/33FKHrYl6+gqAo6KdO8eatNbxQxMkIZHar0B/wamyKZjgqnfxY6gqCJWPEdQp6DC05h4PHZHLye1LMMBw2vhHEYd7+w2Cb8F7iuytVLT7093+IGn1RFDm3cI7p5DQ3dm+wP3CJTZ9GzLQRrC6fmVEpSaNplCtGEcQakViKuW6faNhmW0pjB2k6rsJ+Iktfs9k7vM1cr0nCjBJxesSiEjd2LQ4ie0zmPaLqGJFInHTriNR2hYO1TSqdLhOZLIVjSzRnFmk6A+T+Ktkxf8Sr6z35Dn7SC4Rh6AH/BvgKcBv4izAMbwqC8PuCIPz+/cO+BGwAa8AfAf/dTzruu2FyaorppRVURWHSsxDSWdYDkY3tTfpBj8KJUzjqJPXtMeR6k/7WlzAal8jqWS6MXyAXzfP/s/emP5Lk553fJ+6IvO+r7ru6u/ruObpnhkNyNEORoo5dLuzVysbagLGAsfB7/wMG/Afsi4UNLCAbtqHFUhK5EiWNOBxyLk5P30d1131n5X3fmXH4RTY55CQpSmLP9HBYH6CQWd0RkRFZkU/+fs/veb5fLTmDb/FLlAwXtckMnraJJg+YbhXw1coklBKNYJw39UmEhkqsZ5IM5DmtjbaYH/cNPqoX2C52KQcDdI9yMJBRp6cYHGfoPnqE4zgIgsDppI+QR2UtUx8qbErysNxNMU609J8y/f19Btks2uwMewOBXKmKEVGplxtE8hVKksP1X5C7D8oVXlAe0TA1bJfFRHCRqm0ScceJhn9mZlregW51mM/+55qN/6YjycNBiyAO719rNF/+WSHLMm+ceQNVVnl79W0EYxYtPo7uLqJ2e/x3c6Md0QUzwK4lU3D3mOh5eLGQJUmRYzVKfeBnsyFS8Y3Tj2iUWkekjnZQawYuq43Po7CWM9ljh4DUYtw3hxR1o1hVorkqyuYBmeMMCAKRRArxzEXywQj17CH251UP33Gc7zmOs+g4zpzjOP/bk3/7j47j/Mcnzx3Hcf79k/8/6zjOzafxur8I27bZ2d+nVCqRTKWYmF9E0XSSlQKybnAcTpDt9mg2chgRg44ap1A6g9Lx0jl8i8bed1Bth/PR80z7pzH8KfyLX6Ud9VI7XUSq99DMNnPNY/ztOimtRsUd5l13Atp+plsmS8l9fL/A8Pxm2mKjd8y+rVERBJq3VxF9EbTZGcxcnu7Dhzi2jfhEYdNn/IzCpqwOZZUlBdInWvpPA7vdpruxgaipKPML3FnbxXYcih6T8aMCcsPkB54xavyi0f19hK5AW9MJuwW04CI9e8C4fxyv1zvcqFuH4iZ4k8PGut9mFGOYnjS7wxnPM6zciXgivLTwEqVBiffXPsQ7+SWUIIi9Ai+HYNY1em7Xm3P0lQIPxkVi7Qiv5fc44+Qoy0E6XYN3KwJ7rkmEyTA9pUF4dxU7Cz76BPwyWxXY6D7CbhY44z6DPuHB9lYx2n2iByV6G/t02h1kXUdfOEX7zKVh5/JT5gvXaTuwHTb6NrfTGfb29wkEAozPzqF5fcRLOaRGjXx8knowhiDZSHqPSrdLoXEOVTpPr7ZJfeNPcWr7zPhnOB89T8A1TnDsJfrjbtpXenS7Fq5mjYX6IaF+g4DaJ6OHWHP5MVthTjd6rMSPRs4t5/jZypZYbxxxqPmp1Gs03ruDnBxHW1zALBTp3r//U1nlCxNPFDaPqtS7gyda+s8NR0pHN55pN+MXgd72DlaphDw2xqGkUy6XsH065VaFmXSDMj0+EKZH9oupeS6q6zRMFUFTGEvMUO/ZhIww0XAUQRCGzUfZB8Mv6PiZz/zaPpe4QhBdHlbulLaf6amcS5xjJjnDZm2TjXQa79SLGP4uNCr8ycxoWqdqebjfG6crFbk/o+DvxXihccgVJ4eNAQ2bH1fgupCiMZmCkEDgYIv2dpug7RAKa+y1RVbrd2hWMqzIZ4mPBxkk8vRliNQG+NYOsI8LWI6DruuIn0L27wsX8FVJZHFijEEowoNak429PSRJIjkxiR4KEa9XUDKHHKkG9ckFQqkkhk8ik9/hOBNE079GH4vawXex9t4hKMhcSVwhGThDLPIC5pjA4JpKAQdPtcSZ+g4xGigyHClejhUDuxnlq2KWgDZqDnG34KNul9gYFNkXDUoHe7Rub6MkUujLS8OyzXv3cUzzpwqbiih+LLb2yW7GEy39fxZmpcIgnUZQVZTxCe5tHQEOR3qDVLGNu2ryPXecFqN19y8G7mN3bPqqh6hbQA+fotPvMRaa+LjRqrILvfow2H/RSzD/KQSnhrOd0uYzXcTVJI1rE9dQfSqrhVWK+DEicQyanNe7rPwC2+E7zVlaiBQo8jjlI9IJcM7JcUqpow1EtHKLzSo8GPjJhxN0xr3I5TzltRLhTp9oxGCvbXMrd51atcikeYqleAI1maUZlhEFjXi2gu/xPlb90zGW+cIFfEEQmHfpXE3E8I6NsysobGRydDodwtEYejhKpNvCe3zAXr1BJjrG9PmLhCdiZMt7bD/OITtfxfQkqDXuYe7+ALWyz/nwWZYSr5IIXsGOtnBeDrHrEtHLRS6V1ojSoiUblF0GBWS0SoTXIumR8ytIAY7X6tTFKkcum71Wj+Mb1+ntlpETSfQzp7FqVTr37uEMBuiKxKWpAAJwe/+J2JrmHXYznmjp/7NwHIfexiZ2q4kcjZLWfRSLBbpuk0KvwfnDLiW7ywfSzMi+Y1qaM9IOzYGCrLqIjYdotCUCaohoJIYoisN0W2lr2Efh/QdMT35bia8M7+HMvWc6Sx3zjnE6fpqKWmG7uE0vsIzLo6N0KvyrRH1k+5apcqN/Fkuy2JCybAfCxNoGK2qJaa2PNrCRy3WOWwp5W8dWNaxUgJwks7lbJ1Rpk4x62G/1+PHeD+h0GwSaCyyFEiSix3QTDnVvCFfXJLxzjGN9PjttP5eEVZlrIT/jySRlX4CtRot6o4HhdqOHwgTMPtFCmv1cjoeOzOyVF5k+f4a6XWfzwTr97Bymf5GaVGFQuI9w8D7Tio9LU99iKnQRNVhCfCHGakRFruW4VrhHyO5x5ASQXFAdwEpVIuUerUq47kzSy1bJqxUqYR9HxTKbP/oB3XQdJR7HWFnBajRo37mD0+/jUmUuTAYY2D8jtvZTLf3uMOg/w4Ww3zQG6WOsRgNB05AiUe7u57BokVH6xGoW/mKX77rCdPjkIqvDi/77WN0BA9lLQtdQovO02x3GQuMEg8Fhbjr7cFhdFfukwsgJwPC9ST2R08o8O3VYURA5Fz1HzB8jK2bJ2m0s9xg+UWVBLXA5OLpoulMPkJWnEZQuN408aTlGvGWz4mswpgpo/S6DYp29vp+OYJC0Wix5JYqBCB9UwSx0iXvc7DQavL36VwysHkZ1lqQrylIkjyfao5ZK0YpEPxW57C9kwDcHw1GDLolc9rlYioQxYwl2bIFGtweCiOIL4AUmqnkyx2k+qreILiyz8MILWGGD46NjqqsyXSdOzSfTG1Tg8Drhepbnp/8N87FL+IMlxDNhbo67ETs5vly8i8d2uCeNEVDr9LsWX5NHF1erko/MQZN2w+TIW6Mf8VPcT/P4/fepHhaRo1GMc+dwOh3at+9g93r49I/F1u4dPRFbc4WGOur91tAU/cRA5VfiDAb0d3fAthBdbo4NP7n8EQNXj6LZ52zapGp2+UCeGtl3St9nQUlT6ynoaoBQSqHdD+AT/MRTyaGMQnV/WJUTOw2y9gyu8DcE1QWJs0P9/OL6r97+UyKoB1mJrtBX+mSFLM1YGMHx43dU/iA02uHeNmUOuilqygSoTX7krVEyoyS7Hc6EWyQVGaPXoFWostqPk1bDJOsZ/siq4w5GWXcMcj0RSfGzVqrx/ft/gckAvTKLWwgxE8gwH23hDvmQPoUk/hcu4NuWyb27/xfba39Bq7WP41gsuHWuBP0YsQRHLi8NR2BgmtiqjltRmW2VKRwd8mG5hhaLM7WygjoTp2s51Fahctil4lHpuF1QPcBI3+FS7Hc5k7rMVKSNOOPhvXEXonPIl6oPUE2RD92LBDhgst5gXh+ty3/PWMFY26PZtdgK9RFcGs31XbbvPiK9c4jg96OfO4fT79G5fRu70/mp2Fq9M+B++onYmjsCyQvDD86JgcqvpLeziz0YICgKgsfD3WyJll2k43bjafVJ5jv8uRGkz88rFQrYXAvcp9/pYilexjQ3jI3TrncYD04OR/eDzrAqxxU5qcr5x+CNDxvRKnvQGG1Y/KxYCCww45+hJJSoGV26kSCunp95V4cXg6O59I2SgmOP0ZYT9FwFfuBzqLb8jJkNlqMmKVHB2ytQKlVY78XZ9kzSLaX5vfIhK94ojqbjKDKmGuTHxw3+7NZf0qGPVplGNoN4lUNmg59O6fUXLuD3rT4ZRB5m77P28LscH/4tzeY6AbHL1YCHSCBAORSlrruwbJt6r49muDnVb1E/OuCjUhVCYUKJJOrcGK5UikE2ROFhjnynQDMcA0FEza6xrJ7i7Ph5ViIy8rjB23EFVVnnhfoWdl/nZuAsbmuHb3QzfFJJoiW6eWzqSEd1ak6f7aiFyID+dprC1jG7mzv0NQ3jwgUc06R9+zZ2q0XMq3Mq6aPc7LN6XB82bHnjHxuopG+fGKj8EqxGg8FxGsnjAQcyupeD3BqKV6Nhy0ymLSrdNtfliZF9511bTEtZ6l0ZjxLGFxPo23H8tp/k5NhwdJ9/NEzpnFTl/OOJLIHuH6bBnlE+X5EUzkTO4FbcZMUs5kIE09EIWiG+GdwbqZbpWDIFU8bXn8SSY5S9h7zt8tCsu5kUqkzEBVK2TKiXIV2usNELUwwtUKvmmD16xGk1QSoSYiqsEtRkHh1U+D/f/3Pudhs0S2PI/QDd7hH251RL53OFrrq4OPGH+FIvs9dX2DrcJ737EcXC+3Sbdzint5j2uuiEY1QDYSRVpVCu4Gg6K86AfvqAW/ky/WAE3TDoBw2SF8/icpaobbQ43LtLye3CCUyhtRqM911cmJjnhWgEJeHi7/0CbuM255oZ2maAR8FlJgaPuSiMpnY+NC4iZrdwSg45Q+DI28WsZLFyTfq5Fgf7+5S6XbTz58F2aN+5i9VskgoMxdZy9S5r2SfH9aWGi2HtImTunAT9TzBcqN1AkBUEUcRRFG7kd2k4A6KxaXrFApPZLn+hBzD5+fpnEYurgXt0On0sycuE5seM+WjXB4z7xghGQ8MRajMPkflhuuKEfxyiOJyhwnAR9xndtwl3goXAAo1+g5xcRF4aQ256mdYlXvaPSp9/cKzijXmJNsdAC3Ec3OM9JUCzJjGrVwnGDCZMkcQgzU69wb1ekH5oEbtVx7N7B1/LSyI5ycJSgll3l3o6yzvvfZsb5Sw3c3HyvaWhV/BT5gsX8C3LxizJTPVXmJu6RtObYKNikT9s0ShnaDYeEO/fYVkpgs9DJTaG6g9SKJVpCRJnVREpe8SDfIFGIIwkSZQGTcZeOE8q+hJ23kN64zqH9QPM1EVccpxQ2+RiIsCXY3PocS9vuUx8rg9YaFeokGDdN803u3cRPzHK74ka96QpxOMDxBrsJ91UhDKN7VXsqoXRlimXyqTLZaSVMyBA584drFqNybCL6YibdOWJ2BpAYAJip4aBJ3vvxDXrZzCzWaxaHSWVxKo3OKDBdr1C0BemZVt48gPq7Ra35FER1yX3Y8bEIrWuiF+LYQRFBkaM4MBPYmIM0bGGo3vNO/QyOOGfhuoazoq61WF10zNiKbTEmGeM4/YxlTkRzecl2I7zdX8aRfz5z9LAkbl1mGM8GGO8GgO3h41wjhtOkGatz6K7jBzxMWM6TA322ajX+WE/iBCcx9vr4jq4R/egQyq4zMSVyySSCk7piOL1/4J98JhMpo3wKXz3feECviSJ+FMFRKmKr5LgVPB5vBOTbDsdDjLQzhvYpoDL3GfJvoshpqnFw0jxBJVqlUK7xylDw1vMsp7NUfKFsGyHTCWH/+wk07Ov4TGXqKS32Np4i6Z/EW/kGi7T4XzY5o3oMnrExw/1OoZ+m4luh4o6SdHl46XBaDPWTeMcQq1It9hArsPufIROP0129S6dXJeYHMA0TQ4LBfqzswiyTOfePcxKhfmYh7HgUGxtv/Qk1xichujScMR5YpUIDBdqe1vbSAE/dr9P36zwQbPKwNFYmZzjML3P2HGP72oh7E98JBT6XA3eo90eYEs+xt0hen7otVTGXClCqegwb2/2hjOs3zatnKeFL/mxB0Sr9ExOwaW4WAou4VW8HLbTmOdiuEQvKcnHq75R6a/3Cgla2gELwjizpQBCQORuuMlqP0in2WbWU6TnDTNnCSxZe2w36nzXCuC4JxgbOATy2+Qeb+FXpjm1chV9eow8XTJrf8vUzo8Y+uY9Xb5wAd+2TWynjCt8CNodzGKeaWuRmekz1PwWj6tpigcgdCfwaXEWlSph8x5dTxk7GaLRbpOu1Zn1ukk2Suxls2RcPlq9HplCFmM+yNjcCyTdr2C1Wuxt/Q2ZJrhTv4cma5wL9HgjsoQRCnLT2EWQVomZNiXvPHNSGcX5+UVVU5C5qZ9BLexTqw5wuiLHMyHs+iY7j1cp75ZIeWO4XC4K9TrleBxbVoZBv1hkOeEl7tPZzD0RW4OhJG1kEerHw27P3/Kg39vZxTEHKNPTdNNbbA9K7AzcTAbjDIQOVrZHo9XivjTqyXPa85C4UBnW2htJVLfNwBUiNAiQmhhHHNSH7mSByd8e2eNPi9jpYWNh9t4z6y0Z944z6ZukM+iw6y2jj0VICTFec1cwxJ//7NqIvLnt4My2WG5NcKqoIUYtPgzBRseLYzYY9xWoq0EWbJnzzgH7tSr/WQrTNCJMDxTi1Rz1x7cxnTCnp58nmJzmwK/x/dpjuv2nLzT3hQv4oigT1JYI+M4RTsYxAnla1Zu4cy0WQ7NoYxG2hDR7R3vU0g5e/QKnQ8uMK11s7QAz2aM1qHCQL5LweJju1MkWchyqLirNFvliAXXcS3Binong67gED+XSdQ73D3ACX0H2hbgcUHgtPIvmC/LY84C+uU3Ygk5olgv2qEnKHX0Zqd3BapSpVwa0JBe1lI5YesSjx5tkHqRJ+KPEYjE6pkkuGKAvSXQePMDM5zmT8hF+IraWqz/pvA3PQXge6umhAcVvadC3ajUG6TTK2Bj9UppWY5v3iGDg5spYjNu7u4QyVf5WiY/sqwsdngs+oNkyceQAYwE/bc3BEfxMGZP4k8Hheyspwy/YE349RGmYz7cGkHvwTE5BEASWQ8sk3AlKvRKH4yZ+b5h5T4yveEYriW7UpznM3UNf9rKQn+RixUJO9nk/6GG34UIWagS8RQp4WBA0npeOSFcq/L8EqegukgMX0VYHefs2tmQQC5wm6l1ESp5C1XxP/fq+cAEfa4CQvoV69AC/HWBs4hWS04tYVoVeeotJyyIW81L0d9mq73C0vk63IrMUe4Vz0fNoLg070afp7LJ3vIHX0DlldamXi+w4AkelMpVqBSXhxjWWJO59hbAxjiUeUsoe0bBmsSJhrvojfCk4juox2PLeoddNE7IFxsMe9E9YITqCyFvBKyQzO9QsgXaxTdYVwAnbCI1NHjxc4/DGLn6Xl6mpKRTDoBAOU7cdOg8fYh4fc248gN9QWD2uUWw+KQONLEBobmhA8ZMKkt8iHNumu76OoKmIE1Fqm++y5bjJSnHOBD20ZIvWUYN6rcumFBnZ/7znNhGhRqOt4PNMIMo2tstF1EmQjCcQu8dD+YTYqRP5hKeF7hvet838M3N6cyku5gJz+DQfWaNG0Wsz5U7w9YiNTxptcPze0RyN/tuEFyeYOhznuUYVJdXnXX+Ug7qGR6+ju0rkLJV5ycfLyhGlao3/u+ejoIuETQ+eNoQP7xHzSmhqHE87hDl4+s2UX7yALylD2QHVA/nHSAcfEZLczJ95Ha9vmV7FJljvM2Z0sENNdqU0R8dbpB8/xk+Qq+OvEoxcRhgfp6WXOcjcwBJqnHK6WK0mm50ej9MZGo0GckjHmIzi164QlKcxfB3MXpdqTacRcvNy8BTPB5Jo7j4PvQ8ZtPMkkVh2j5ZbbcizpA0Xqf1V0nKQbrHDnsuPz1XHtgrcubPKwUdbqJLC5OQkwXCYRjJB0TRpPVrF3N/j/EQAtyrz4KhGtf1kShxdHKZ4qgfDoP9bxODwELvZQpmfpLr7Dt3OgA+UecKOzUoswL2DQ6TjDM72T1QAACAASURBVG+ro/LHPrHJ+dBjmg0TSQ4RD3poSQMUNcKsMYURUodm5Cc190+f4MzQLCb/eNhU+AyY8E6QcCdwRIdcwqJlWZyOxPnGL6iPf9ROsVUZ0JA+IJRaYGxrjKuDY7SUxfu+CbJNCZe/gSUVyfUt5tUkrxo5mq06/09V45geASlAv+simtnklYhDcnYWTVN/wZn9enzxAj4MO1AnXxjKCSsuyD9CPvyAiZib1MRLiINT6PUUU6oLtztPztjisL1Nev0RjfQhV4LjzCReQpx8lWYgTKGyQ8vcY9Y6whg02arWuLm7T6fTQfKoGDMhvPoKrn4Kb0jA5fPTaUHVbfKq/3kuB5K4vVluuDdx6gWe0yxc4miD1I98L+I4JRLpVXbdSdr1AduiTsrIYyldbl1/wP5HmwgIxONxxiYmMKenKdg21dVV7J1tLkwE0BSRO4dPFDZhuIj7k6D/jP1FPyvsdpve7i5ixE9LOWRwlOPhIEVD9fBcyKAsQn4nTbXucCSO5t4veT4iYNVotnV8vjlsqQOiRso1RyQQRurtgGMPde5PeLoIwhOzGPGZlWr+JLUT0kNUPW0KflAHEt+a1ImroyPv7x5fpmVtonrW8QYWSK3HeFncxRiz+cA9R70joEeaNM0ix90GC8YMX/FUGfRbfLsIG40yHtlNyY4hlvK80NrCMU9G+P803GGYfHGoLinrCLlVAq2bjMUUdOk0Yv0KM8aLRDw6Xc8+B+ItMpmHHK0+YMrucyk6iWvyS5QjV2gOFHq0SVqbRIQDdnJp3t3coT8YIGoy+kwQr/c0WiOFrthEJiZAEmm6OrygXeO8K4bLu8N77jRGtcxzxuii1LGdZD+YwunkCBZ3SesxKm2H3Y7JXKyCrdnceP8u+zc2cRwHr9fLzOws2qlTVDSN3P37mGuPuDTuR5VEbu9XaPaezCZ+NuhnH37h0zu9jQ0cwaabaGGWilTaQR5pIeYVSPrdPDw6op9N8440N7JvRKqzGFij2QBdjREIaTQtE78nyYJ7FlXrI7Syw/dTHfW5PeEpoOhPSjVrw8qdZ4AhGywEF/Dpfkpxi5LdIyiI/PHE6Ax9rxfkbmmavHiHWLCBKi8QX3fzJXkdV9LhQ32ZpqkgxusU2zn2mscsek/xWqCHKPX5fr7Hg6NjFAsy6jRFNYIgP/004Rc74P8EdwSmrsLYFRAV9MZjUuoDPGKbfnmCcfGbzEdfRvZIFIx1st0fc7j+HlLmgKsBD/HUDJngWeriFKKWIiL0GZO3ODy8yZur9+laFoIsok768IQX0dsz0OwRHh/HFXUhhx0uyddYUd0o3g1+qJe42C4RkkZH+de5iqx3UCrbCN0meT1CtiVwmM5zasVB0AQ++tEtdm8N9UcUZZjiiVy8SDsUIvNwle69O1xIeZBEgdv7laGsMjwJ+k9y+rkvbtAfpNP0ywW6yQ6OaDIo+Vit2dhuP6cDOhUEjre2yFQlKuJowH7O/QFeu0mvZ+AKLdMX6siCwmzkIl7NhTzYHhp6hGafwdX9FuF7YhxT2n5mLm8Jd4KEJ4EYUKnHDJrtFq8FLeY9o7OO72TO0QaK8ruEgypib4nIns6r6hremM097TQtwQ9jLfLNYzbKj5n1n+OrQQXNgNulNje39umX62T6Hkzzc+p49RuDJwrTL0HqEpIiE5W3iVr36GWzSMVZziX+DYHwBdp+gZy0Tib7FvnHb3FRtlhMJci5I2SkJIbvLFH3HGNKk/z+e/zt7e9R7dYQRAF1zIMnOY1rsIRTMnF5XARm3CRmIlzWX2ZJdLC967yv1PmKM1pvnHeC1JUZJKGNUrxHV5ApaUF2apC594iVlwNIqsLNH9xh4+YwPSMIApFIhInnn8eZmiS3uUnrow+5kHDj8DOyyjDM6YfnoXb0hazTtzsdOlvrdDx5hKCByiwbB2Wyho9TLg2fLnN/Z5dBPsOH4vLI/uNqlcnABu2ahKFN4PIINAZdku4pZoLTiGYeUWhB7MywquSET5efiNBl7j0znajF4CJ+V4BOGHouL51Wkf9xdjTglyydd48vUVM6SMYP8Xn9iOU5AnmVV1wbhEM9ttVFqkoca2xAunPAeuEmcf8ZvhIO4PLJHFVr3FjbpFOrIUknaplPB28cpl5CSF3EE1SJ69uouevU14+ZU66ykPomTniJil+kPtjiaOPbJIu3uJjwU1V11k0ZzTvFWOIaY1qE8tEOP7j3HQ5KtzHNBnLEwD01gUc4j1Bx41hdPNMWZ89d4qrvJWbEFi3XGiUaTAmdkdP7a+cay3YTrV/Hqd6m4QpSMwI8Kjnkb37Exa+lUFSNO2/f49HN+z81QHe73cw8/zz6ygrldJrKj97mbEjGtG1u7Vc+DvqRhY/r9J+hPO3TxnEc2o8f0GQLcTyCz3eW9c0c+UYbTyhG0iNRshyy26ts1310hE8KpDk87/oQfdDCMt0o4UU6Ygm3oHF67Hnk/gDJPgJPfDh4OOHTR1KG+fxBGwrPRlVTERVOh0+jBz3U4iqOoDFrHfNceHSw9HeFcXL9WXJaiaDnDroeRE6n8Ld0XnTvkfLXKSjT5PRJBgmdA2uHnfx7uH3zvBibxB3y0G7W+OC9D2g2Turwnx6CMJwyTr+COn2J8JiBt7tK++E7qOkBF6Kv4AldoB6do+X2UC2vI+79Vy67cwxki7t9G8fwMjFxgTFtllrG5qPNVTZy71Gv38cxerhm4/i0i6j1cXrNMnZ4j5ef+zpfjl0lqtYoGI+YsCojp1YT3dyULrLSbOKvZWm27lHxJanLHu4fdSje/ZAr35hFVwwe/PAh9z+6jWUNg7ksy0yeO0f42jXajQblH77FkmHSt2xu/2zQD8993JF7fOcLobLZO9ylVr2FlAwTiFxhr6mQ39nBcftJBf24VYH7jx5g1uvcYH5k/1lXlZjnIb2Gitu9iKybNPst5r1zxH0TCI0jJBfDMswTPjvc4WEHeXUfWqO6Np8FPtXHXGgeMajSi4To9Zv8t2ONEWG1tiPx48MFSlKSinubiPsIiSDKjg/vwOC8u8iCUaAvx8m4ZugEw+wLW+yX3sLjS7CYOIUnEcXvE1DVEy2dp48ggH8Mae5Vgmefxx8U4OAGnZsfcloYZzKwRDcwTj22gC0EsXOPOOPcx+Vsc7dVpmu4mZyZZ0yL0s55uJ922KoeU6lcp9l/hDKp4vddwNM+Q79eomF8yFcv/AveGL+E4arTcN1jyh511/lz7RpR2WS52iNS2qPee0QhPEEVF3c2ipQ33uW5byzhUTysv7vO3Y9u0u/3n1ySQHRmhrHXfwdbEKi/90OmBhV65tBApfeT3GBodrgw1soPpZV/g/X0B/UCpZ03ETwaoZnXqDo+dh6vIXV6aGMzhFSLg3KN8tE2dxoprE8IU0mCzWX3TeR+H40QYnCWul0kILs4N/YytCpIchkhOj/M35/w2RJZfNKFe/+Zmf2Me8dJxsbpBhQUfxxPfYvXUqOz47dqPkqVM2TUIH3/PUKuFnbbh7qn4kVl3tvmnHqMLBhkjTka3nmO5AN2639HwqUSSZzFSK6Ac7Jo++khihCYwH3+dYLnriDZHRp33yG6n+GsZwrJcFOORrBCZ1E7Pub6BQL9Bzwu36SkDhifniWhagg5i9VsmI1OiE6vTLV5k17wAHd4noB5lX61RE17mzdOfYtXx+axfHWC6n0k5+dvnIEg85983+CsU+FUroO3skFrsEM5NE7B9HDrXo7K7g+59PVlvIqX7Q+2uXPjJq3Wx3XL3kiE6a9/HS0YpH/nI2KlPdr9Abf3n7hmwVASIHke2uXfWLvEQb9G/uF3cASb8Jlv0hW83CvXEHe2kX0h3H4fst1j/f5tWm2TVcZHjrHoqxBRHmA3dXTfabpik4HT5nLwLG4thNA4Qg65ITD92V/gCcP1ksS54f2Zf/zMTmMpuown6Kcd8WAYHr5qbGJ8YiBuIfD3WQ8l63nyuoocvYtfEzDLbtSsglvSSPnhRf0Qv9OloI1Tc10gLZTYHHyfKbXK5NgMknIywv/0EUXU5Bzhl38Pdfw0nXwe7tzkQkcgorioGgPqiSi6tsK8HcHfrXJYvMmxsEN0LEBAkfEWC+xkbe63pxG0KUyrSstYRQxBQLhKv1qmqf6Qf7H0L7kYjtMO50iJo/6370jLbESmOWu3OXvQQGo9pkmWpifIQd/LnYdZGvs/4OIbZ/ArPg6v73Pv1h3K5fJPj6G6XEy+/jr+mVmknXWCew+pt9rc2q98HPR9qaFzVq8Bhx8OjTx+Q+j3yxQf/xVOt0d0+ZtgRLhTb9Pb2CBimQzGFwhIfdYfPaLXLPB+a7QMUxdNLrjuIXR7+NQ4PXeCplUmbrhZTjwPtSyS1kMYO/Op2M6d8I/ECAxTkfU0NEYlSj4LZFHm/OwlJMPATCQID1q8lqyNbHezaVDLaeyLl6noPTyJdVyiiHUoYbRcuCQ3QZ+Ll11pEmaeohSiql4mLcKhdAexfQPskzr8zwxRlgmdOYv3ytfouaZp7KSZOcqwbNr0nTa5UA8hMsmieppAy0+mXCQrHeEJtzCkNvFqiXwmx3vHYLuew2XMYLla2KEmhjhPv15ioHzInyx9g1mPByV8H5XRP/B/kH8fKSJx3jE5t1fB6jyiobQYyCprLR9313K0jt7k3FdP41MCZG+nefTgIZlMBvvJYqwoyyReukbi4iU85TzBjdsUSxVuH/xM0PfEhv0KZh8OfjwM/p9zut1jKkfvYJfqhMe/jByb4G69Ta1eZ+z4gL4/huLz0s3sc5zJkuu7OGK0yepUqEiINcSeC9F/irrTQJK7XI2cRxpo0CsgjyWGDX0nPFtCc0MZ6tzDZzYbdWseVibOY7ldyPEor0m7BLXR1M638yJyc5Jj9RR1T5FgNI9igrXWwOX40MUALl+Ia54M04M96oJByVpiFx+NmISsPn2LzJOA/ytwhzzELl/CmX6Vci+OelTjYqOFt1kgJxVoJhXmw1Ok2nGOi25Kbh3N38K2DxjrZOgVc7y1sUPBiRMKXsMTmkNPBFDlCO3GAYq6yv8w/zIxt0PAMyp9sCf4+I70BoRlLls2S5kyZvchPbmD4AjcaEe4v3VML/sm57+0hF/2UbqXY3NtjYODg5/L6wdWzjDx6peIOH1im7dJHxxxc6/0cU7fFYKJ54fPD64/M5naX4XjODRbm9RL97APq/jc59Hmllltdih1+8T2t9Fth/bYLO5mnscbG2A3ebs5WjcfULosuTdwWg3inhQ1KUCbGhM+H/P+CzjVNJJLRBw/Waj9XCCKw9SObQ6D/jMimkgy65+jHwrh8+h8MzY64zjoudjIHVLsvUJJH6cV2iUU7kHDxlrN4PcE0eUUhi/Gc54M89YqoqCR7cS5fdyn2X36shInAf8fgaxIxOcj+JbP0/A+T6MRZbmjMlsv0KxtkA82mZhOsGTK5I50sq4lxKCP3iBLytrBqG/z3uM17pebGK45wvGXicy9iuGapt3dIaBv8ccTi8R9+2jS6ALuf1GWSDvnEPwKz/dM4s0GbWsDxayg9bq8bU5zfztLP/f3rFxdwCv7qD0qc7C7y97e3s/l9Y3xcSZef53xoIfx/VX21zb5cKvwcfWO7oPJq8Pa5/TNYenm5wjbHlBv3KfT3EM4aOMRZnGfvcB6p0+m2yfQqBLIZjFDCZAkjtdv0e4NeNQMU0MfOd6paIZg7wCf5Kahn6JIB59h8kr0NE7bQrBbyDNzID99XZMT/pnovmEvSTP3zO5PQRSYHp8n6ZmiH/PxslJk2jc64/jLSgJX6Q579u9T90QZxLcIBiXMfJ/egz0CAQ+atozHP8GKnmNWuktUVnHbE6jSyQj/mSEIAr6IQXI5jhM9TY4LeOw5zmOg5h5x3F3FNx/knCrSO6pw5MzT9y8zsCRiYp1I5w6rmx/ww/09Bih4fUtMnP7vCUWuYtFg1pPn95MeEoFRWdgGMn9hnCLnXMSQZV5stXEJffLKMdHaId5Klb8SF7izn8Msvc3pC9N4RC/NxzWyR8fs7+9TLBZ/Wq+vBAIkX3uNublJ5kt7HN69zzuPjj4O+ooxlKTQA8OGl9KzaW3/JKbZoFq9waBfQs5J6J0Qxpkz7CJy2O0T6HcJ5Y5xaTolbxRn6xZHtR4OFu90RxdqE0aNCSON0iwQ8U5TwoWl1Jnz+Jnwn8EuFhCDXsTU1DO42hP+QUKzQy/c3KOh+cwzQPJrLAYWiQQW6AZE/nVkdJRfNxXeb+rIhW2Oxd+j6vUgj+/idcn0dxu01/YJ+CU0/RLB4DxTYoZx+TqzCQlZOlm0feaoukxyLoA/FaIuzdOwnuNM4DLjgz7l7AdYsQpnIm685SyHDYeCOoOtTBJxJUiZGYrpt3hz9UdUex0UxSC18C+Jjb+BZkR4LmTwe6k8Pn3UGes9J8yBK8SB+AKBrsy1ThXBY7PrqTOVWyNcLPNdZYkPskWc+rucOp3ELXlordcoF4rkcjmOjo4wzWHppahphK9d49RzF1gxy+Tv3OXNG+s0Ok9GKZIyzOn7UkNVyOO7z7RWv9s9plq9iYON0YwjFwW02RmOXR622z2CtoWvXsFVrdLWPNiZA3areVRV5gfFMOYnbnUJm+XoEcFmhbhbpyAuUJQ7RHV4IbqIWWyD5KDMLyOcuFh9/viJwJpjDbWhnsUpiAJq3M2y5xTh1AIxOc+L0dEvn78rJhDtY8rZBmX9NYoBBe9UHrcsMljN0trP4Q1YGN4vEw2cIm5l6df+jp759Isnfq2ALwhCSBCEvxcEYfPJY/CXbLcnCMIDQRDuCoJw89d5zc8DgigQTLhJzvtB95DrzBH0f42z4XMIzTRN5w4ToT5Tgy7FSpWNuo2jzxALXWZCCtKvPeate3/GVv4hjmMTHrtGePp5wsEVXg0t8drYXQQ+6a4j8G0liWnopKUXCWZVnhsU6ERhKzBgafsWoXyFv2OW75fq9Fs3WJoN4pE8tNaq1MtVKpUK+/v7tNvtJ9ch4jl9mtO/82WueKHx8CF/8/Ytyo3h/w8Nps8Pa6AbGTi8DoPuZ/peD1M4D2k0H6MofryDGezdPHI0Si6RYq3VJSiAv1pEazZxSzLFYo18a5+O6KNaHfDQCo8cd8qbJqbUCHaPkYwFco4b1eiw5PaT9I5hlZtIsRhS5MTF6nOL5hnem638UCrkGSB5VVSPwdnQi/gScd5wb6BKP9+BazkCf1ldIMEambRG2f0ihZhNeLaHjo15a5NOpoLL28EX+SaxwFmSHQnZfvqyJ7/uCP9/Bd5yHGcBeOvJ77+MrziOc8FxnCu/5mt+btBcCqmFAP6IQb2p0h5cYGXqW8Q9Kbrs4DJ2WJabdJs1bu8fUhUMQsEVFnzPIVs6t7fe58b292h3jvCFzuGaSDI5vsIfhJ5jKThaa7zX9/BhwEPP7aEsXCSyqXFpUCA7JbETsDi38WNi+QbvWAnebLVpDlZZSBp4RA/1R2W6tRatVovDw0NKpdJPUzxqIsHi11/nldkY1uEBf/PX73GY/ZkF2/AcpC5Brwn77w9r9j8DBoMK1epH9Ht53K5Z3PYMvcebSF4v+ZlZHjc7hCSRUG14rqFul9J+mqJZoSh68Ztdvl0bdbJyCR0WY4cEGx0iXpGCs0DF3SQuO1wMT+KU2ziOijI/g/DJVsoTPl8Ep8EIQn7tMx+M/AQl7kJF5fLMG8S9Em/ERruBbxdVcmocj7NLIx2i6D5NcaxLZEJH7pv0r9+lUyqhGU1C4/+K2ef+GxT96Sux/roB/w+BP33y/E+BP/o1j/cbhygKhFJuknN+BAGKOY2o/+ssz34D1a8yUPZZkLIorRL3t3fY7/RQVC/LwfOE1DNs5zt8uP0hxcpNFDWIFJc5tTLNn0TH0KX2yOv9sBkjGxnQ9ofpOSsk191cruXYXtLY95icWXuHVK7N7baPN3tN0vI+81EJr+ii9rhCr96l3++Tz+c5PDxk8MRVR3S5mPjql3jj2hn0Ro0f/NWPePhg/adfCkP9oasgynD4EZR3P7X31LYHNJvrVGu3AQG//wqaHaP78CGirpNbWGa9OyCqyMSaVczBgBgC/cdrHNs2Rx4FZSByIwtFxzVy/MXQY/ymyLiTpqNdJCtJaGqHFU+IqEtnUHSQEink8Oi+J3zOEARInB16E2SfjS2iqMtIAQ1d8PPCqa/wZdcRMWO0a/0/bcVJhm36VoF2NknOPUVjrkso6UNs9eh9dJtWKYMkdXB649jW58/EPO44TgbgyWPsl2znAG8KgnBLEIR/9w8dUBCEfycIwk1BEG4WCoVf8/Q+O3S3QmoxiD9i0Cx3GRTjnJn+Y8bnn2Pg7xNRsviah+wdbLNardMdDJjyxVgMnCXbGuf6YZ1iM43tdDHdOV59KcVXfKM3Tc3WuNn3kYv16QSi2IMFJo9jXCpnWJ+TOTT6LGx9wFi2x3ZN43q/yAOtxGQQPLZIba1Mr9rBtm1arRZ7e3vU68PKIEEUCZ9d4Xf/4FUiLpWb797lx2++S7/3JC+peWHq2lA4rLAG6dtPvc2918tTqV6n201j6BMEgy8gmQrd+/dwEDlcWGZzYBFTZWKtGp12m4jXh/X++1RbXR5NejA7Okq5xN/2R03J43KBSX+JSM9Ed3soW9PUXDVmJIUFlxe5JWMLQZS5BMKnoFZ4wqeA6h6qwLaLz8wWUYm6QABvcI7nxs/xR+H9kW1yLZvvl+eYDLZwBm2q5THSnjDWGQdvJAjFJr3bt+mUtzB8HcRnoZYpCML3BUF4+At+/vCf8DovOY5zCfg68O8FQfjSL9vQcZz/w3GcK47jXIlGf7MUCX8y2k/M+RElgcphl7hwiXPnvoUxM4GsN/E2Dmnlt3hYLpKvlPFKEs/Hx+lbE1xPeyh2A4iSwkC5z//0ZYGoPDpNvdFK0pI67Mcdel4/QmucxeYiK90SuylIyw0mD64TS5tkCxIbg30+CjTwx1S8PZP6epVWrokgCNi2TSaT4fj4+KcCbO54jN/91hvMzk+yuZPh7f/811QPn3yQJAXGLg+F15p52HvvqaR4TLNBrXabeuMBoqDg91/B41nE6fRo376NaVpsLixy4AhM6CqJdoNmo0FQ05Bu3MBqNLg7n6TVVonVGvxZOYDJz1c5CDicCV/HPfAxJuRoqFfJqE28isOKy0dEF+lX3YihKGr8xNjkN4rA1NAWsbD2TDrFBUVEiRjYHZv40mX+KBJgxT9aR///PTJp6wlC/i6BlkWulmTPreG6YOAOBCFTp3fvHqW9W1ifgmbQrwz4juP8juM4K7/g5ztAThCEJMCTx/wvOcbxk8c88BfA80/vEj5/6O5hbj8QM2jVelhpnYvjX2PqwkvYPhWhlUNpH7FfS7N/tEOv0eTFZJyA6uNWRme7sYTbfZpAOM3/fDGHwCcWgRB5pzSGqpfZGNMwVQ2zEOSC9SJTrhaZiEPaqRDL3cJ7DK1jkWrjMXeDBTrjOlqzSXurSu2ojm3bCIJAvV5nb2+PZnMoySppKq+8/iIXXnmO/EDk7e+9y8GP3sXsPPkwhWaHpZuCOEzxFDb+WTLLltWl0XhMpfoRptnA414gEHgORfFhNVu0b9+hZzk8WlimKKssuXUinSa1Wg1fr4exu4uVz3M4EWdfkvCV+6wVmqw7o12xp1wP8asCMSxsfYaKo9LQ2izLGikJdDmERQxl3IcgneTuf6MQBIivDJ9nHzwTnwcpqCNoElguJubP8L9EWyjiJxdw4T888OILuXEFJGINmXQrzKbHJHI5jO4NYB3XsW8/gP7nT1rhu8C/ffL83wLf+eQGgiC4BUHw/uQ58Abw7FrkPiN+UsmTWgggaxKN4wFT9ikuX/gKoj9Mu9VE03qUrQp7+/fJba+xEvJzKhRgp9zlw0wCw/dlri4LvJwcHUFv21FKeRHR1WJ90o0t2bSPRa4Ovko4aVIN2aSdIp7yHZysQDuto5cfcezPcLyk0u026DzO0jxqY5k2oihi2zbpdJpMJoNpmgiCwPmVWb78+1/FTE7y47Vj1v7yv9JYX8cxzaG2ydRL4B8b2tDtvwedUbnnX4RltYeBvvIBvV4Ww5gkGLyGYUwiCCJWrUbnzh1KDtydW6Slapz3GOj1KqVcDncmg6dYwhmYdJNhbmkWQg7chQzf7o3W3HvENqe8NwgSJGz3aemLHEtVorLGoiIRcqsMOikw3KgTvl/773/CM0B1DWee7dLQyvMzRhAFlIQbZ2Cjxme5NJHkX8dHZxur2S7vlqK4fRaRUIBk3c1B189jX4vEuTCaK4Bp2vA5rMP/34HXBUHYBF5/8juCIKQEQfjek23iwHuCINz7/9t78yC5sus+87tvzz2zsiqrKmsv7A2gu4EGge5mk2yRFClTokhqlx1jhekRLSs01igmZkaWLIU8ETMea8IeLbZiRFu2bIVsyiOJQ4YWiru49b5ga+xA7Vtm5b699c4fWb2gq9AAGqgqAPW+iIx6VXnz3ZO3Mn/vvnPPPQd4HvhLKeWX7rDf+4Zu3H6KbD6G0/axKlme2Pt9WNEeGlUHvS9JKxVjrnyV6Re/TsZp8r7xEdquyzenwI88yj8+qpLQ1/vzv+AdIl9YpRPXuTScwvea1Jc9nmq8n/SIQaNHsKRVsGqnqC15LEwnyRcvE40WWD2sMmc0KZy8QutKDd+VBEGAqqpUq1WmpqaoVrtJoUb6knz4Q8dJHDnKySDB6e+9yMpXv4ozO9udSA0c7paPDHyYeba7GWaD21EpJY5TpFY7Rbn8LLa9hGUNkck8QTy2B0XppoN1FxdpvPwKV1Wdc5N7iESjnEhGcIsrlC9fJj49TUqCMTGOZ2qccYoUKzEGVwp8oRmnIdbvUDyR/CammqVHAcc8wnKwiqvBQV3QK23i6XF8twc9F0Mxw0pW9y3pUYj2doulOHc/NcHNUGM6asokaEpS+w7yM4Mw5sriOgAAIABJREFUGl1/t/HZ5xuU1QR6tMFgzxADzRTXXItzuQ75PVHiVj8Ed/8uU8h7uMTdsWPH5Isv3vdh+2/guT6lhSbNqkPHa3F+6WWaooE53o9r6+jzs/RWC/T0DjN85CmeXSxRbLXp95c4Ndvk915NrDvnceUyP5xa4mxsH5lyiYmFJSK9A8RSHqeMkyyXOnjSZLAdpaFPkoxJTkxIrKFJLhspiudLRIsBY/1Z8kf3EWjd2b4QAt/3iUQi5HI5LMui4/q8PLXK9LUZMqvz7FVdsrlerNFR9OHh7iJT8UJ34UzVILsH0qN4fgPbXsG2l/CDDoowsKw8kcgwivKmOMsgwLlyhcLsPBfjKdzRMYbjEXabGksXL9K+eJGUqpLK5zH37cMuLHP+la/zHTuBPiso1qb4A//wujEaN2d4Kv5F0slHSDQHqPfGmQ3ajMSifFCrMJLNEks+jbsqiB7JofeE+e7va9xOd23JjMPIia67ZwuRXkDnSgVhqriFS3z51Uv8T1fW3zUeGoF/+p4Wg36UaquXufYlirEl9qkKj6X2Yx76SPd7dJsIIV66Ufh7KPjbQKvmsLrQoLS6ynTxMk6siZpPUtYzRAoleqbOk1Uk44eOMGUNcnq5jFK5zJ+cM3mtuF6MfjHxRazMCHPeIOlSmZGVFVKD4xhWi6vBSeY7dUqxFGMVjVYwgRUJOJJ1OXDgCMtDu3jl1Yu0Fzr0GZI9B7NYuQTIGJqWeCMsM5VK0dvbi6KoXFiucX62QFAuststMSggEo2g9faiDwxARBIsv4TfmMcVLm4yi4xl0Y0MljWEYfQhxPU3l36jSeO117jcaLGY7SM+PMyBmEVkaZGl06eQ1SqZ/n4yhw+j9ffj2x1e+/KfcL7qMLMyQGb1ZX43OEiF60MpNTx+NPlHZJNR9GAvenSMFb2KrST4YLzChHAY3fcR3FIvSkQj+khfGJ3zIFCd6/ry+/ZDz8SWd+9VOrgLTdS0QuPsC/zTky3+pri+oMmPvxc+2VtmlzrMQjXBjDxNxVhmsucA7z34g6jvwq3zToJ/+5ePkDsmmjSw4hniKyau77CwPIeGpH+wxGJfHDv1PjqXzuGeeoHhvhSDo4f4SmeUE0NXuVox6HjXfwj+sPEh/kXsP+Jmn6Qg08wFvfgr0/Tn9zOhHiMizjDdXGS6P8No+TJ2Zw8vrSq0nvke+/eU+JETH+WlSzNcvDhP6VSd/GCV4XFQdR8hYgjiLC3pFIsWPdkM4+k0MS3CGT3JqVWfgtZknFVic1PIay2kEqBkEmiWQkTRSNRdNB+U3hzovd2F3jVkENCenubK1Cwzig4jY4xFLcYLC9Sev8ZCsYgWi9F/4gTxiQmE1v3IXnj5a8wvrbBUn6R39TRflj3rxB7gRPQ5+rUaMAFBP/VIhaYTZTLiMOLVSQ49hCQP0kbPRUOxf1BIDXeTqxUvQqyvO9vfQrS0hV918OsesT37+aXqKzxfg/Lb8qv95YsBQx+JYsmrjGQOQeUIrv88i61FfOmicnfdi+EMf5txOh5nXrnA4sIiZsxE9nssJhQaXozelToDy5fp11tke/v4TiHCny91+Nb0+tvD94pz/IPRb/Ja+jCr5RjxJZ3+aoXxkQN4XoeV+jmmItdYzEUYXg2w24cxLY1JZ5mx/n4GHn4vXqSXV09fo1DzsVI6qT6fTJ9NTLMhcPE8DSlVNE0lmbIwoyaXCpK5oo3uuezqMRmJRYg6AlF1Eb6GkKDQRJOrKKoPhgWpIWRymEa5yfTsArOOh2daDKaTjLsd1GaTar2GH4uRnNxF//591810zl19iZm/+DzVSo5qx6MoVvis/xSS62/dB7QVPhn5f0mnYpTc96JlVNpGjMCN8ZHIFfqiSYb2fwqqKkJXMSdTqPEwK+YDg2fD1LdBj3YzwG6xaydwfOyrVZSIhr96hT996Rq/fmX9+tL+MY+ferTGcdkmqR9nqaKip2scOvI4qhbO8B8oDEvj8LG9GOcUlmYL6EtRxloBK30N5nIGrehDuIUyndIyj3gFgkSSqaTOTO1618535QFOnJ9iYm8RkfWoaoIlNY2/dJF9oxMM8DBKJYWiXmFuMGBo+SXa7UNcjQ7BapHgua+TGDvAwfE85YUKy01BZyXBYiOBlhFYUYN4RCWpCgg8Wk2FwLc42N/D5GCMS8sNrpbK1CXsy/fQ/1Afqm3jVyoEjQZevR9ZW8ZdWaBx8RuUWm0qaDhWllx6gNFYBEsJqKkKnd4sxr695AcHicXW4uF9D79R4OLVZ1j+9tfozOcoqBa6cpE/kU+sE3sFn6f1b9Mfs1l034M0NETEwGtlmDRmySoQHz2O4lhIzUeJaCixu19DNGQb0UzIHYTFV6F0tZsiZAtRDBW9L4K73EIfmuSTtSpfK9f429L1In5+WuM7+ShGT4OjwXfIJN6PHt2P2ITqaqHg3wPous7k7jF0Q6FWbtKu+eQ7Jok+h/NRm1pfjN2dXbRqNfLNBX6yX+O3GwZOcP0H5/cjH+K3z36O3N59aNlVKvklVsp9iMJV9g/kGGQUUTbRxBJTQw3Glk7TaE9yIdqP59fZd+UMQbNJdGCQrF1EejGQQ7Trkma7w5IZMK0IVFUhomtY7TZmbZF0xGJ3MkXKzDJfbvLdSwXyKxV2j/QT6R+gEW9RjdYo6RFKShoRmSRJk9GMzkA0QNptGq05Wm0f3YjSE8uSMCSiVIaCTeC0aTWqTBWmWFqYpTI7zLIYR9Mu8dfiABV7fZ77o/oZdkVLNIM8tpYjFvfwvBymbPAerYzI7iNpTYIPaCpqygyzYj6IJAfXXDuXuq4da2tDbtUeC7/u4FV8rMnd/Fr9LK8871F7WxDbd1406PuBLKZzld3uV+nJ/hBC3P0Ka6Hg3yNEo1H6B/pBLJPJalSKDZSCxfFEgleMVc5YNkMiyoC7i55Sg4+PePzZ9PWC38Tgt5Lfz69NfR3b/xQkXqOaXGbRyBKUF9mbXWRQjkDZQKfKtXyMsdIC9YrN+dQgDaXD0YUzZHyJNTrEymIRe+UUydQo+cFRAiPAVVtU7QZL9RYlz8UGHCnxmEVTVaTQqdsKLy66aOfn6TUhH9NIGDoxVWU8m6Enm8EydNrtNnPtNgpNrIhHOqISMwSKDED6eG6Hesuh3vaY7zic6qQwLyRwvRRKcorLhs4r9YF1Y5lVSjwdu4KpNJlRHscyPCKpUQorAU/GZtEiaaJ9x1A9gRLXCRouauruF5sIuUfoP9iNzV882d03soV1iYUQ6Pk49rUqMkiQH+nnfykt8M/OXe9K77gKz7wQI/uBXZilV/ELX2Rw4hfQtPWTmTshFPx7iHQ6jW3bVCoVhif7KZeqNEs272eU0xSYUssUY20GvAgHoiZHmm1eKV7/wTnDIH9h7uNjha9y0fsEfvQVGsoSC+ksQdNmV/o0Q+ogopRACaIsDZrktBKt4jxTqTyVtMZjyy8y0Vxk4sBhVrIZKjMz2JdX0OMDxAb72Ts8wvG4xLVblGs1ipUq1XabartNw66QDAJ6fMFqR1CqCtqqwnBPlPRIDtNQaTcbtJsQiUTo7esjFhvHNE08z6Nl27RaLVqtFrZr08bnkl/kWtEh9ZJEb3RwstN0YhX+ovrkujFU8fh47BRRWWNWPISvRenJJSjVTLLBNfbHoDN4gqSMd+OlOx5KVAtj7x9kVL2bO3/+xe4ibm7/lnavGCr6QAx3voHWP8knJup8Y7XB11au35k+tQRXpgfJ7rGIiRZCufsuxlDw7zH6+vpwHIdarcZgvp+SVaJVb3Fc5Bl0Mjyrz3JOKRANVD5+aJir31ul6lwv+v9FHGEfRXa1vs015XE8KWgrBeYjPbiuYLd1jaFMEqXeh7qYo5KJkOgvY6zMUXUH+NbgGCulJR767ldJTwyTfHQv5SUPp1SgPdfi6twMZiZFdqSf3Pgk44++uT/AdV06nQ61Wo16vc5Suc50ocZStcPyhSUilsFQb4p8Nkk0cGh0XPRyBc/z8AOJH0hansQTGh2pcWV1hdXpEtlph55ymWpvC5Gp8IXqY7T99SL9vsg5+mRA1RA09CH6TIEwR3Hn5ngqV8buP4ZFD6ZloqYN/GkbbTDMm/PAE++D1AiUr0E8t+UF6bWUSVB38CsO5uRe/nn7LKdqHQqd67+7X36+yqGJA+TGk6jK3Z+EhFE69yCe5zEzM0MQBAwNDVEoFGi328TMJPWWxteWr3Ft7gq6hJ7sCL/37eq6c2So8ZvKl7CMfSxEJljwL9EK6riZPjKizV73GpmmStHpZymawo50UPUmTqFMS+mjOdpLr9vhyNwy/UYE9UAfnWyaoBXH9Ppp1yWNdhthKcR6EmSH+0jleogkE+jG9e4R3/ep1htcml1idqVMqekQoBKJRDBNE0VRUDUVTdVQNRVDN2hXasxcO0+wXKSvESHabFCI1IiMOHytNMg3yusTs45qy/ykdY2GPkUjMoCijDG+b4i5Kw7jcomnD+WomwcZtnLEJnsIGi5excbakw7DMXcCvtet5yADGH/fu9rUdCdIP8C+WgUB0l3m26dn+czL6+vg9iQU/vnPHOdjQz2o72JdKdx4dR9i2zYzMzPous7w8DCFQoFarUYikcAyUvzNKzN8d+Yclt5hrpXm2an1/8f9yjV+hWfRjYcpWAmu2LPUXHD6hukxKuzxLpJteLT8AS7H0mhIpNEgqFexOxEawwP4PXH2LpTYt1THSlv4e+L42Shxo48IQ9SLCvWqi+13EJZCJGEQTUWIpRNYsSiGFUHVdTTDQFmLoa/V6iwXS9RaHRwvQFENTF1DCyR2sU5pdo5SaYHACUgbo/hqwGpzllS2zAVH5w8WDq+LyjGx+dnEaSqOj9Izi1AeZrgnQUPL4s8V+NRDLk7ufcQci8GJIbRcFPtSGSVuYAxtbYx2yDbSLsPMc93SnYMPb3n3QcvFnqmjWAru4nl+83Sd/3x1veif2B3jc//wA+8qkCAMy7wPMU2TfD7P3NwcS0tLDA0NYRgGxWIR13L5ofeMYQiT7y1cZTxR4nI0QrF1/Sz1fDDBH+rL/D3vHIOtgyR1izOiQqEwx3LPXhwzyb7Yq2Q6VzgcDPNCZBCjk8SM6hhGBWP+Cp1GP5fGcywM9bL/0ipD36tBukNtT4Az3iSxq4eEo+PWIrRqOu061BsO1fkVhB6gWwqaoaJqAkXr2icEqBIijTZepUGz1qLSdPHsgLZ06agusWQPvfFJpjpF7LmT9EVWaPgun1v64DqxB/h49BKVdhqv92Vi7CKlRhG9Udpnihzv8zDGn6a9aNM71ouWi+LXHKQvUdPhYu2OIpLpZnotXYF4f7ewzxaiRHX0XBR3qYk+tJt/0jrN86WA85Xrc2U9d7nJZ799hX/0/t13tf9whn+PUy6XWVlZIZPJkMvlun7xpSWEEGhugmfmG8wHNUrlRf7byQAvWB+P/snIV/lA4DEU7EWo1zjZ7jCtpKnmjpBWBfuc75HxpohH45yL76ZQt8j4AbqsIu0yUtVpDuVw+4fobekMv7ZAz2odGY2i7u4n+9gu4r0WbrOG13RxGhK3beJ2LDxHQ0qBDARCCGQAMhAQSARKN0unlLRosSQX6CgNIkEc24szbVeJz51j1FvE7XP57ZWPMuut972e0K6xG0k9eY28Dqayi8QkVKZUMoHKJz7+BIszNtlEhsFHxhGagj1VRfoSa1dYs3bHEQQw80w3b/74U6Df3UiYW8GZb+DXbITW5MqFS/zk8w4N5/pFXE0R/Mk/epzHxm5vvSF06dznLC8vU6lUyOVyZDIZbNtmfn6eVqNDaVWjYJoYPfCVl6/xxVPrMwRGRJsfjf81e/00+4LdRHiJ81Wby1ovS6MniIkE+6qnyLovE01Boy/LWSePWtVJuD56UEHxKgRZg1auFzU3SaoaIXt2htRyEUXR0MfHSB97lL7RflTDw/PreE4Nt9PEd3w8u+tCBYGUIHyBIhRc32XBLlKWdQhi4OdZacGiWyV35QwT5TncfIs/qr+XF9rrZzvDYpX3q0U6Vp1ctEBMTKKmdUyjTX0xy4ef3E9E68Fudth7/CBazCDoeNhXq+j9UbRsmChtR2I3YPp73Rn/8LGtT7AWSOxrVfACgtY8f3NhiV96db1rZyBp8Zf/5Cmy8Vu/Ew1dOvc5uVwOz/NYWVlB0zQSiQRjY2MsLi7SKK/QXGmSy+b5+Y88zELtJC9OXV8Lty0jfKX1AeKJv6ItNQ65J9iVeA6ztETk8ne4tus4p/seZd9qH7mZ72G06zw6dIXFXIxyK4OopjHdDHqhQqqxgqgtUU9l6HzgAGVnH9rZGWIz0zQuX6SY6cHcNUlkbJxovB/TsLBUBSUWoAQS6brIIKDsVph3lliUNWpGBN8dw2lrVKoVXGeZ/VdfI9Vexu+r83X70IZiH8HmEXUJ33CJGQ49DGPrCun4HMXZ3QwP5cmlx1heWGJo3yharJs2wa/YoIgw9n4nY8a74ZnLZ6E8teUJ1oQiMEYSOFM1lOgg3z9S57+rCf7oqn1du6Vah29cKPBjj62v8fCu+g1n+PcHQRAwOzuLbdsMDw8TjUaRUrK0sMLzL12jIHQ++sQeVEPhp3//GaZX188W9urTfCT+VXRljKP2BKZ7jqXlGnPEOb/3GDI1zERtlclLzyFiAf4kdGINGloUrxXBLsVQmjF00cRIVdEyNYgnUPp3oUVG6cxX8aYXiNY6mJ4gmohjZDOIRBQ/HqNtQpE2ZdGhQYBjG8Q6CayGiqiW0doLpJ1FepwWfqeJkihxRh3m35U+gXxb6QZBwAn1LLtNHyVQGU4KOppONrlMp53H0/bw8eOPUFldwepPMPHQrjWXkgwXa0PeZO6lbi3c0Se2fBcuvLmIS2DTWjzPp192Obna3Yabiuj86594hA8duL11htCl84Dwerim7/uMjo5imt0Z6vT5Jf7m5FXSuSgfPTpB1VP54X/zHSptf905jpoXeCr2DQJtmGP2GFF7lqXlDsttlfN7DtPq38WgU+PwxWfQXVgdy6OkFvDMKlLVCWyJV87gdRLgGwSJKlp0BTMq0TJpSOVpuxbNWhO/3gHXI6BbxV5KFeGb6IFF1DWIOgG600SRq+h6HTOm0wSoLWHoZeZknt8t/SjuBjeiB5RLvFdvoXkmSsbAMB0GKGEkJPPOU7x3che9MUFDt5k4uJtIpOu68So27kIDYyyJGubOCfHsbu581YCxJ2ETYt9vhl+1ceYbSKfK3Pw1fuJ5l7G+OP/m7x5lpGd9BtibEQr+A4TjOMzMzCCEYHR0FF3XcR2fl16c52yxyIERk8nBLPMdnZ/6d8/h+uv/v09bJ3k09l2aZi+POKP0NsusrkjqNY8zw5MsjT9EWvF4aPplxsrTFAYfYyETIS4vY0RqSLWJaEpw4uCm8YnhmApuUEMTbQwDdNMi0CwIdIQr0D0dU+pYnkDzPXzpYGsdnIiLFgUHB7vTwCgXSNg1loNB/lXrx+iwXpSHlAU+GRTRpE6lN07UajHiVhgcKPBa54cYtvp5375BlimTGeplYODN9AvdLe7hYm3IW2gWYe6F7sasgUPbYoJbbOOttPArC1worLD/xGFiA+v3mtwKoQ//AcIwDIaHh5mdnWV2drYr+obG7tEMiw2XVSegp1KhxzT5Pz7xEP/zn59dd45vdh4hKgIO8xyvxGqMWX0MKRpCT/DE7BSX6y1O7XuUZ0cfZznSy96VV3ioGefc2AkKniDtLBDVlvGVCkowhYlJ1LMQuoVrWCiWgubaKO02iq8gAx/P83ECn47iInUf1VRQrIAAhYotCFxJcrFMX73MfDTH73if2lDs00qDH2tXiSsKMz0mQdylt9Nkd2qeU8oPYwVpjo31U4t10FST3t7eN14btFyCtoc+EO6sDXkLsV7ITHR34cZ6IbE+P9Nmo/dGwA+QwQB7Oy3kpYvI3p436j/cLULBvw+xLIuhoSHm5uaYm5tjZGSEzECMscUoVxpttJEegnaFR9I+v/T0GP/3N6fXneOv2kcAi8e9Z5lPzVIZtMjEEiT0AQ6sFMi/+hwv7D7ExZ59FKwsk/MvMHjl69Ty4yzGR5FymB69Qyq7Squ1gqg2MNoummojNZe24SISKmrURDdB0wJUDVRFx3UUmk1B07XwXIFVbZC/ukgyaHAlPcy/8n6cButD5eKKy6dbBQb8Bqd7UhT6dB5uVXlUWeBM/P041R7eN5zDnDBp1FsM5AbQ3vKF8co2qOFibcgG9O6FdgmWzoCVAn3ro7f0/hjSl+CNoiTFXRd7CF069zXNZpP5+XkMw2BkZIRGyeEbJxeJ90V4/4FeCoUVGo0Gf/hKhf/6anHDc/xA5Cof5GUWe1Zw4zauY5FZnGBiEbANLg3t5vLAOG7gkV88TdaeJshEuZobw1OiGK5Cqu2gGS6q3iTSaGN2FIJA4tKhrtdpqzaKaiA0E1e1UFFRAonVbpJcLZMrr2KpLi/ldvHH3vfTkevFPiJ8/rGzxK7mLM9lo1wbTXC04/FBf4YruUEWO49xdGCIA8cHWaksEY1GGR5+M7JBugGdy2W0Hgu9P5zhh2yA0+yGapoJGD6+pVk1X0dKiTvfIGh7mJNphBrutA1ZIxaLkc/nWVhYYG5ujqH8ELv64pxdqbE8nGRkaIhqtco/OCooNmy+crm+7hxfak/SjBn8SPEcjfYK5cw8C2OvMZfpY89snN2LZxhsrnJ+4jDl0RN4q2MMr5zlseYs8/kcV1K9lKJRIo5DxNZxtQyNhI7lK/T4EjNwUaRNSVZpuDUUu43heeiuR0/VJms3qac0vpmb4EvVD+HI9bNvUwT8vKywpz7FM8mAK6NxDjsWH/DmKPcaLPr72Z8fYvfhQartMoqiXOe3h26NUSSoma3fZBNyn2DEuqmUF0/C6mXo27vlJryeThlfviuxvxmh4N/nxOPxN0R/fmGe8dEc14pNzlwtk89ESaVSRKNR/pkVIZDX+NqV9aL/7eYwK5Eon26/xlgzQzZ7kcvJZU4eaNFb0BhbKHPwXIGVoQPMD+7majZHfuEKR+cWeaq+wkxfD1OxNG5UIdIJyLgOMdvFFQo+cTTRi9AmaGk6zYiC5zWIFKYYTF/Gyfj8rTbOl2aP48v1ERIRAv4HUWeyeJbnrSaX9j3KQS/D9zlTyHiBc8aTjGYm2DvahzRtOuUOg4OD17lyZCDxyzZqwkAxwjTIIe9AMt/NnV+60s2oGeu9+WvuMkIRoGzORrDQpfOAUK/XWVxcxDAMWg2LV+cavOeRHHsG34wtLleq/Mqfn+KvL9Y2PEev0eLnrNfIt8oo5iKXe6vMmlGE7ZEte+RWoqjKEIWRo5QS/YiOw1hphn1qnWRSUI6bLCajdHQLNYCk55J2a8ScJorjoNUdZKmO3V6lZZRY7Q/4884H+W7l0Q3tSRDwC6JGduW7nDU9Lj/yQQ6R5sOdWfToq7yc2kM29308lMmTGtBZrSyTSCTI5/PXncer2rjzDYzRRFizNuTmBH7XtePbMLY9qRfuhDAsc4fwuk9fSIVLcwJbV/k7T4xg6W/Oah3H5Vf+7BX+9GRhw3Nois8notf4kDuNSoXlaJOpqIWrugi/RbLmEqsLpLGLSmYvbSOO6nkM2x12m4KY7iHVDg3NoWoIgsAlaNVQa1Wwq0i/jJv0WVH6+dPVH2TGy25oRxKfn6WAXn2J2WiC4t7H2afH+WiwCup3OWekMEY/xsHEHqJJlWZQQlVVRkdHryt4DmBfrSJlGIoZchu8nnrBSm6bP//dEgr+DqLdbjM3N0dltcP5gsKBvTmO7eu7ro2Ukt/5ynl+6+tXudF/f1ekwqc5Q59foWFKLurQ9n1MFVTFQ3NbaLZPoI5QjQ7iCtDcVRKdObKKjeYFKIGPDHxk4IHho5sBQovyt7Un+LJ7CE9s7F5J4fKzYpa6ssKSHkMZ3MujiTRPuEXawTPMqArByPs53PckqiLwrTq+7123Ge11/IaDM1NHz8fQ0vfXTC1km6ktdP35mYktr5J1J4SLtjuISCTC6OgoipgjUVzh/MWAXUNJMm9JviSE4Bc/coD9+TS/+LlX6XjBuvNcaaf5NfEkT+uz/LBzgUelQtFSmXFKNDyB5fWS8GNEFJNEu0HDjFMyx1gxR1gM6piywgB1+kSLmCKRrsozzUm+XN9HSUbZIMMxACOizUfiFU5GdCx3kMFoH09ls4w7iyz6r1DSVIzRo+wbPAqOQETauI5DPp9fJ/YA3moHoSuoyTAUM+Q2Sea7+fPL1yCS3pb4/LtNOMN/QHFdl6tXpvnOK3PkBvv5+NMHNmx3dqHKz/7hiyzUOjc8lyk8nlLm+YC1TF/KpOEWKbZXaXg2gVQQ0iAq48RFFGn0UbVi1DVwsLEDm6lmnEudHGX/ncIhJXtMm+Gkh7Q8RmyXQ6rB0dFBlPYlZrxFXLVDdGgXExNP41TTSLONS4tsNnvdBqvXCdoe9rUwK2bIHRAEMPtsN2Rz7MluJM89TujS2aEEQcB3nj3PuWvLPHZwiCOHd63zbwPUOy6//qen+fyZxZuec0JpciDqszfRIK8u4jdXsDsNgsDBDySB1CiQY0aOck3kuObF8Xhn/6elBhyKB/SbHknL5ZDX5kg8QbInQql0milTJ2bY9PTGGdz7PjqlEeygCVaHdDpNf//GyaWcuTpB08XcndmUELeQHYLb7pZGVM1ukrUtLo14u2ya4Ashfhz4DeAAcFxKuaE6CyF+APhtQAX+vZTy/7yV84eCf+d4ns8XvvIa1VqJJx/JMz4+gmVt7Mv+wnOz/MZfnqHsrHfx3AhTgZgmEfh0PEE7UAhu5K/ZgEd1m/cbNo7pkh43eDQaJ1+uY3cKrIoqs6ksQxFJX9onMf4w7fJDdOwmxLpiPzAwsGEZuMD2sa9U0Hoj6LnbT0AVEnIdzdVuvp14DoaObrc178g7Cf6dLj2fAX7RhJwZAAAUrElEQVQE+NY7dK4C/xb4O8BDwE8LIR66w35DbhFNU/m+J/agR7Ocv1pienqa1dVVgmC9qH/ixAh/9fPv46fHExji1iYCdgAlR7DqaDQD9ZbFflTv8N8PVXjfgTJzewSZp/fyg0PDZC+cobzwCldMh9mhvezuiTOeC0iN76dd2kO9UUVGOqRSqRuKPYC32gZFoPWEC7Uhd4FYFvr2QWMZVq9stzXvmju6N5FSngNuVmj3OHBZSnl1re3ngE8Ar91J3yG3Tk/a4vC+Ac5cLFJvABSp1+sMDAysm+0PDsT59b/3OJ/83gX+8JUpvlpTcG99wn9TBqKSD++CiVEd25XY7V4eUeI88vKzVGcvUknGWNp/hMTAOEf9RTJmAW9gD63KAVYrFayMINubJZfL3fBzFzg+ftVGy1gI7f4Jpwu5x+mZgE4Vihe76Rfi7y6b5XayFc6oIWD2Lb/PASdu1FgI8RngMwCjo6Oba9kO4vDuLHOFJjOrHk8NZ6g3ykxPT5NKpchms+j6m5kpIwmDIx84wEDS4mfPnuNbMsIXyypXS+131belwdF8wOG+JvlEG1MXKAsO8Zqg4ZYYL12m6bms7jmMOHCMCV2yW5lHUQq0MpPUintYXS0R69EYGh4kk8m8Y39esQ1ChAu1IXefgcPdBdzFkzD6eFf47yNuKvhCiK8CG8Uj/aqU8gu30MdG07Ab+guklJ8FPgtdH/4tnD/kFlAVwfHDOb7+3AJnLzV5/MgwtUaFarVKrVYjk8mQyWTeSElgWBrD79mFFrH4oTNn+bG0S/kje3mupvC11xZ4bbFOzQ6QG/x7TeExopcZ10sM6xWGtBqWDbElhb6ihuEo2NLkotZLzHap9kzC5EP0jo2zS87SI5doqjUa8UmWZ3LUGyV6BhOMT44Qi71zlERgr83ueyyEHs7uQ+4yitr14U9/D+ZfgtEnQbt/dm/fVPCllB++wz7mgJG3/D4MLNzhOUPeBblkhIcPZDl5psjFi2UOHe4WRV9dXaVUKlEul4nH46TTaaLRKKqmkH94mHJvkvLzJ0mcPs9P7Bvi03//MVpC4eL0EudmZplbXcTwa4xaNkNJEyORoil6qNViCKefuK7TE40gA1hehZJiMN0WiEqLvuE4+fF+JnsdMt5ZXK9KzZKUvREKF5J4gcfo7gGGRq/Pj3Mjwtl9yKajR7qiP/s8LLwCw++5b3biboVL5wVgjxBiApgHfgr4u1vQb8gG7M+nWK3ZXJitk7xsMLY3w+DgINlslkqlO+Ov1+tomkYsFiMWi5EciGF9+AQrL16gcG6G5lyB9KFJjuwa5+Hdw8wtLnJx6iSrpRJXCnWiywtEFEE0EkXJjKDEsiwToVSUSKWD3yoyUFnh4UGNiX1JTHUe33WoCJeiarK6YGA3Y0QiFvsPjZHJpm7pvQUdrzu7742EvvuQzSWS6bp3Fk/Cytnu8X3AnYZlfgr4XaAPqACvSik/KoTI0w2//Nhau48Bv0U3LPM/SCn/91s5fxiWuTm0HI+/Pb2MV3F4bDRN/2QKVe0KZBAE1Ot1Go0GrVbrjWgeRVHQVJ1GoUn76hxKo4yVMtD6MpBKISMR6o1lGvV5/MAjppmYUiEqQHVd7EIdo91Ak03qjiQ9lCM7ksfRDeq+Q8V36DgCt5nBUHLkBnOM7s29Ydet4MzW8Vsu1u404jZeFxLyrile6qZS7t0L2V3bbQ0QbrwK2YCFSpuXL60S6wTs7U8wMJFCfZvPOwgCOp0Otm3jOA6O4+D7PnbLozpXwisWMToN4jGDRCZKPNODkYjQ8YtUKvM02y2aJYfmchNNCETUoKBGieQy5LMGnuLjSwcZCLSgF03miUaS9I+kSfbenkvGb7o40zW0vgh6Xxh3H7KFLJ7s5t0ZfKSbjmGbCXPphKwjn45QH0lxZb7GdLGJDCS58SSG9eZHQlEUotEo0eh6AZUHJM2KQ2WxjrNSwG3XaZdtgkoNTRX0MIBaKaLaNTK7osiBKGcaFoahsndAoKKgyF5we/DacRRhEu8xSeeiaLeZs15KibfcQuhK6LsP2Xr6D4PbgaXToFndPPr3KKHg72D29sfpuD7zxSZW3ca7VKFnMEYia91sbwVCCOIZk1jaoDWUoFGyaTcc2hI8x6NddxF9e0nlIphpk9MLJRKWy8FcAiVQcNoS3xcIAYmMSSoXue5iczv4VZug42EMxbvFI0JCthJF6S7izjzTjdwZOdFNq3wPEgr+DkYIwaGhFI4fsNqwiSqwutCkVXPoycduSYCFEMRSJrGUid12Wbpao1l1kVISi+m0Wh6npysEEnbn4rh1Bd1QicRVYimTSNJAuQORlr7EW2mjRLWwOHnI9qHq3WidmWe6KRhGnwDj3nMthoK/w1EVwSPDaV6aLrPgeowkTTp1l/mLFaIJg2SfReQdqkRJKbFbHvXVDs2KDQJG9mfIDERpOj4vXS0RG4nzyHCKuKWjGcpN7x5uB2+1jfQCjJH7awNMyAOIHlkT/efWRP9x0O6tSUgo+CEYmsKx8Qyn5ipMN20m+qJkhUqt2GbpqoOiCsyIhhnVUFSBDLpC77R9Ok2XIJAIAcneCMk+C01XWap2OLdYQ9UVjo5liJt3/6MW2D5eqYOaMlEi4Uc55B7ATMDwYzD7Qlf0R050Z//3COG3JAQAXVU4MpLhtcUa10otGgmTfbtTeC2PTtPtRuastN/YIi0AzVCIpU0icR0roaOqCn4geW2hxkKlTTqqc2godV2JxbuFlBJ3sYEQoPffe7fOITuYSKbr059/CeZe7M7675GUyveGFSH3BIoiOJhPEjc1rhWbPHOtxGRfjJF8HEURyEAipUQIsW5xNAgks6UW06stOq7PeG+MXX2xu+q+eSt+2SZoeej5eLjJKuTeI9YLg492d+LOvwTDx7ppGbaZUPBDrkMIwXhvjP6kxYXlOpeWG0yttuiLmwykLBKWhram4a4fUG27VFoui9U2thuQjuo8lE/SE9u8/CLSDXALLZSYjpa+t3ykISFvkOiHwYe7cfrzL3dn/dss+qHgh2xIxFB5dCRNsWGzVO2wXOuwUHkzW6aqCny/6+ARAtJRg4P52KYK/eu4S02QoA/e++XmQnY4yTzIoBujfw+Ifij4Ie9Ib9ykN27iB5LVpk3HCXCDAD+Q6KpCKqKTtDS0LUpl4FU6+HUHvT+KcpsbtEJCtoXUcPfn0umue2fosW0T/VDwQ24JVRHkEttbPSqwfdylritHDStZhdxPpIYBAUunugu5Q49ty0LuA7na5S418UodAtvfblNC7hIykLjzdYQiMPLxTVsMDgnZNFJD3Xw77TLMPQ+es+UmPHAzfBlI/IaLdDoACF1BiRuocR0lqiPUUCjuR7xCi6DjY4wkwsImIfcvyTwIFRZfhdnnuiGb+tbdrT5wgi8UgbU7TeD4BE2XoOHi12z8cgcEKFEdNaajJHSUTdgMFHL38Ws23moHrcdCTdw/1YVCQjYk0Q/Ksa4/f/bZrugbWxOA8MAqnmKo3UW9jIUMJEHbW7sAOLgrLVh52+w/poeJt+5Bgo6Hs9BEiWpouXCDVcgDQiwLI8dh/sVu/p2hx7obtjaZB1bw34pQBGqsO7MnF0W6AX7TIai7+NW12f9aGyWuo8aN0G1wDyDdAGe2jlAFxnAivCCHPFhE0t0ka3MvdFMx5B+FeG5Tu9wRgv92hK6gpS1Ir83+W2uun4aDX3dwaaJYGkpCR00YKO8ybW/Iu0cGEmeujvQl5ngy3E0b8mBixLqiP/9SN06/bx/0TGxadzteyYQiUOMGatxAJ0Zge92Zf8PBK7TxCm2ErnSFP2GgRLRwprnJvC72QdvDGEmEF9yQBxvN7CZZWzwJhfPgNCB3cFMKo4ffpLehmBqKqaH1RpBegN/oun68ig2lDkIVXb9/wuj6/cOon7uKlBJ3vkHQcNHzsXCRNmRnoKiQP9KtkVu6Am4b8kfveqx+KPjvgNDe5vppuvh1h6Du4FftN/3+ie7Cb+h2uDOklLgLze5O2oFYd+xDQnYKQkDf3q6bp1XclI1ZoeDfIkIRqInuzF5KSdDyusK/9nBfD/lcaxMu+t4eMpC4Cw38moOWi6KFO2lDdiqpoe5jEwgF/10gxJtRP/pAjKDj4dfWhH+pibvURIloXbdP0ghzvtwE6Qc4cw2CpoveHw0LkYeEbBKh4N8FFEtDsTT0XJTA9t9w+3Tj/VsIU0VNGmHEzwZI18eZrRM4AcZQPKxLGxKyiYTqc5dRTBXFjEBvBOn6+PWu398rrkX8GMqbbp+ItqNzwvgNB3e+gQSM4TjqO9TODQkJuXNCwd9EhK6i9ahoPdYbET9+zcErd1MFCE3pbvR6PeJnh4R7Sim7Ia/FNoqlYgwnQrdXSMgWEAr+FvF6xI+WtpC+JGg6b/j9/crOifgJOh7uQpOg46GmTfSB2I650IWEbDeh4G8DQhWoSRM1aV6/0/f1iB9AiWqocQMlrj8Qfn8ZSLxCC6/UQahK14WTDP31ISFbyR0piRDix4HfAA4Ax6WUL96g3RRQB3zAk1Ieu5N+HySu2+n7esRPvbvZa12St5iOEtMQW1Rd6m4gA4lfsfGKbaQXdGf1/dH76j2EhDwo3OnU8QzwI8Dv30Lb75NSFu+wvwee1yN+6OP6JG9vTfEc0VBi3fz+92qqB+kH+FUHb7WNdAOUmI4xHEeJ6tttWkjIjuWOBF9KeQ7Y0ZEmm8nbk7zJjoffcAmaLl6xDbINiuheAKJa9wJgaduW7kFKiWx7eNW1nciBRIlq6IOxMAInJOQeYKucwxL4shBCAr8vpfzsjRoKIT4DfAZgdHR0i8y79xGKQET1N2bI0g+6u32bLkHLwyu0gXb3DsBUEWt3CorVPd6suwDpB91aA2trENINugvQSQOtx3og1h9CQh4UbvptFEJ8FRjY4KlflVJ+4Rb7ea+UckEIkQO+IoQ4L6X81kYN1y4GnwU4duyYvMXz7ziE+mY8P7xFeF9/vB7983p7Q0EYavdioKsIXek+VAVUcdMLgvQDpCeRro+0fQLbR3Y8gs5a3eDXo4xya1FGoY8+JOSe46aCL6X88J12IqVcWPu5IoT4PHAc2FDwQ94dQlXeWPx9Hen6BB2foOMhbR/p+HgtD4INrqOq6LrmFLpJnCQgJcjuwuvbXyM0BWGpaH3GPb2WEBIS8iabfr8thIgBipSyvnb8EeB/2+x+Q7obv1RdXZdiWHoB0l17+AHSl+AHyICusEvZFX0BCIFQ1x7a2l2BoT6w+wRCQh5k7jQs81PA7wJ9wF8KIV6VUn5UCJEH/r2U8mNAP/D5tYVdDfgvUsov3aHdIXeA0JSuYIc5ykJCdhR3GqXzeeDzG/x9AfjY2vFV4JE76SckJCQk5M4J78tDQkJCdgih4IeEhITsEELBDwkJCdkhhIIfEhISskMIBT8kJCRkhxAKfkhISMgOIRT8kJCQkB2CkPLeTVcjhCgA0+/y5b3AvZiOObTr9gjtuj1Cu26PB9GuMSll30ZP3NOCfycIIV68FwuthHbdHqFdt0do1+2x0+wKXTohISEhO4RQ8ENCQkJ2CA+y4N+wyMo2E9p1e4R23R6hXbfHjrLrgfXhh4SEhIRcz4M8ww8JCQkJeQuh4IeEhITsEB4YwRdC/F9CiPNCiFNCiM8LIdI3aPcDQogLQojLQohf3gK7flwIcVYIEQghbhhmJYSYEkKcFkK8KoR48R6ya6vHq0cI8RUhxKW1n5kbtNuS8brZ+xddfmft+VNCiKObZctt2vW0EKK6Nj6vCiF+fQts+g9CiBUhxJkbPL9dY3Uzu7Z8rNb6HRFCfEMIcW7tu/iLG7S5u2MmpXwgHnRLJ2prx/8S+JcbtFGBK8AkYAAngYc22a4DwD7gm8Cxd2g3BfRu4Xjd1K5tGq/fBH557fiXN/o/btV43cr7p1vo56/pFoR8HHhuC/53t2LX08BfbNXnaa3P9wNHgTM3eH7Lx+oW7drysVrrdxA4unacAC5u9ufrgZnhSym/LKX01n59FhjeoNlx4LKU8qqU0gE+B3xik+06J6W8sJl9vBtu0a4tH6+18/+nteP/BHxyk/t7J27l/X8C+M+yy7NAWggxeA/YteVIKb8FlN6hyXaM1a3YtS1IKRellC+vHdeBc8DQ25rd1TF7YAT/bXya7lXx7QwBs2/5fY71A7xdSODLQoiXhBCf2W5j1tiO8eqXUi5C9wsB5G7QbivG61be/3aM0a32+YQQ4qQQ4q+FEAc32aZb4V7+/m3rWAkhxoEjwHNve+qujtkd1bTdaoQQXwUGNnjqV6WUX1hr86uAB/zxRqfY4G93HJd6K3bdAu+VUi4IIXLAV4QQ59dmJttp15aP122c5q6P1wbcyvvflDG6CbfS58t0c6o0hBAfA/4/YM8m23UztmOsboVtHSshRBz4M+B/lFLW3v70Bi9512N2Xwm+lPLD7/S8EOJngB8CPiTXHGBvYw4Yecvvw8DCZtt1i+dYWPu5IoT4PN3b9jsSsLtg15aPlxBiWQgxKKVcXLt1XbnBOe76eG3Arbz/TRmjO7XrrcIhpfwrIcTvCSF6pZTbmShsO8bqpmznWAkhdLpi/8dSyj/foMldHbMHxqUjhPgB4H8FflhK2bpBsxeAPUKICSGEAfwU8MWtsvFGCCFiQojE68d0F6A3jCjYYrZjvL4I/Mza8c8A6+5EtnC8buX9fxH4+2vRFI8D1dddUpvITe0SQgwIIcTa8XG63/XVTbbrZmzHWN2U7RqrtT7/ADgnpfzXN2h2d8dsq1emN+sBXKbr63p17fH/rP09D/zVW9p9jO5q+BW6ro3NtutTdK/SNrAM/M3b7aIbbXFy7XH2XrFrm8YrC3wNuLT2s2c7x2uj9w/8HPBza8cC+Ldrz5/mHSKxttiuX1gbm5N0gxie3AKb/iuwCLhrn61/eI+M1c3s2vKxWuv3KbrumVNv0a2PbeaYhakVQkJCQnYID4xLJyQkJCTknQkFPyQkJGSHEAp+SEhIyA4hFPyQkJCQHUIo+CEhISE7hFDwQ0JCQnYIoeCHhISE7BD+f6GyV823znheAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x0 = torch.linspace(-2, 2)\n",
    "pyplot.plot(x0, final_preds, alpha=0.3)\n",
    "pyplot.plot(x0, torch.sin(2 * x0), linewidth=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f65d41d1dc0>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd3ib1dn/P0dbtmTZlrx3bGcHQiBhhBFKgTBbCmF0AF2UAv31fTvpbhkF3lLopruUUtqwyip7jwAhgRCSOHE8421LHpJla5/fHx5kSLIdy5bH+VxXrsR6Hp/njsZX93OfewgpJQqFQqGY+2iSbYBCoVAopgcl+AqFQjFPUIKvUCgU8wQl+AqFQjFPUIKvUCgU8wRdsg2Ih8PhkKWlpck2Q6FQKGYNW7dudUops6Idm9GCX1paypYtW5JthkKhUMwahBCNsY6pkI5CoVDME5TgKxQKxTxBCb5CoVDME2Z0DD8awWCQ5uZmfD5fsk2ZVZhMJgoLC9Hr9ck2RaFQJIlZJ/jNzc1YrVZKS0sRQiTbnFmBlBKXy0VzczNlZWXJNkehUCSJWRfS8fl82O12JfYTQAiB3W5Xd0UKxTxn1gk+oMT+MFDPmUKhmJWCr1AoFIqJowR/Crn77rtpbW0d/fkLX/gCu3btmvS6DQ0N3HfffZNeR6GIRzgcZmBggO7ubtra2ti3bx91dXXU1NTQ0NBAa2srTqeTwcFB1FyN2cGs27SdTdx9990sX76c/Px8AP785z8nZN0Rwf/kJz+ZkPUUChja3Pd6vXg8HjweD4FAAI1Gg5QypqALIUbDhVarlezsbIxG43SarZgAysM/DO69917WrFnDypUr+dKXvkQ4HObKK69k+fLlrFixgjvvvJMHH3yQLVu28KlPfYqVK1cyODjIunXrRltFWCwWvv3tb3P00Ufz0Y9+lM2bN7Nu3ToWLFjAY489BgwJ+0knncSqVatYtWoVmzZtAuD666/ntddeY+XKldx5552Ew2G++c1vsnr1ao444gj+8Ic/JO25UcwupJQMDAzQ0tJCVVUV+/btw+VyEQgEAIhEInG9dyklkUiESCRCX18fNTU1dHR0EIlEpuu/oJgAs9rD/8njO9nV6k7omkvz0/jRectiHq+qqmLjxo288cYb6PV6rrnmGm666SZaWlrYsWMHAL29vaSnp/Ob3/yG22+/nWOOOeaQdbxeL+vWreO2227jggsu4Pvf/z7PPfccu3bt4oorruD8888nOzub5557DpPJxN69e7nsssvYsmULt956K7fffjtPPPEEAH/84x+x2Wy88847+P1+1q5dyxlnnKFSMBVx8Xq9tLW14ff7ExaSkVLidDrp7e2lpKQEk8mUkHUViWFWC34yeOGFF9i6dSurV68GYHBwkPXr11NXV8dXvvIVzjnnHM4444wx1zEYDKxfvx6AFStWYDQa0ev1rFixgoaGBmCoyOy6665j27ZtaLVaqquro6717LPPsn37dh588EEA+vr62Lt3rxJ8xSFIKRkcHKS9vX3KYu9SSoLBIHV1dRQXF2OxWBJ+DcXhMasFP54nPlVIKbniiiu45ZZbDnj85ptv5plnnuG3v/0t999/P3/961/jrqPX60djnxqNZjTuqdFoCIVCANx5553k5OTw/vvvE4lEYnpLUkp+/etfc+aZZ072v6eYo4yEXLq6uggGg9OyyRqJRGhsbCQ/P5+MjIwpv55ibFQMf4KcdtppPPjgg3R2dgLQ3d1NY2MjkUiECy+8kBtvvJF3330XGNrE8ng8h32tvr4+8vLy0Gg0/OMf/yAcDkdd98wzz+Suu+4iGAwCUF1djdfrPezrKuYW4XCYuro6WltbCQQC05pRI6WktbUVl8s1bddUxGZWe/jJYOnSpdx0002cccYZRCIR9Ho9d9xxBxdccMHoRtWI93/llVdy9dVXYzabefPNNyd8rWuuuYYLL7yQBx54gFNPPZXU1FQAjjjiCHQ6HUceeSRXXnklX/3qV2loaGDVqlVIKcnKyuKRRx5J3H9aMWsJhULU19dPu9Dvj5SS9vZ2DAYDVqs1KTYohhAzOX/2mGOOkQcPQKmqqmLJkiVJsmh2o567+UUoFKKurm404ybZCCFYsGABZrM52abMaYQQW6WUh2aKkKCQjhDir0KITiHEjhjHhRDiV0KIGiHEdiHEqkRcV6FQRCcSiYx69jMFKeWMs2m+kagY/t3A+jjHzwIqh/9cBdyVoOsqFIqDkFLS1NQ0I4V1ZCN3JkcW5jIJieFLKV8VQpTGOeVjwD1y6FV+SwiRLoTIk1K2JeL6CoXiQzo6Oujv7z8sUdX6e0nt3IKl810M3ha0/l50gT6k0BE0ZxEyO/ClLcCTdzy+9IUgJu4zBgIBOjo6yM3NnfDvKibHdG3aFgBN+/3cPPzYIYIvhLiKobsAiouLp8U4hWKu0Nvbi8vlmpjYR0LYml/CXvMAKd07AQjrLfitJQRTcvBlLEKEA+h8Tky91aQ1v0TOrj8TNNlxF5yCs/JSgpaCcV9uZD6D1WodTURQTA/TJfjRevNGfUdKKf8I/BGGNm2n0iiFYi4RDAZpbW0dt9iLSJDM2oexV2/EMNiB31pMx7Iv0p+9msGMRaCJLg9aXw/Wjrewtm0io/5xMusepa/oNLoWX44/bXzFflJK9u3bR2VlJTqdShacLqbrmW4Givb7uRBojXGuQqGYICNx+/H2sDG7dlCw9TZM7jq8jqNoO+rrePKOH1eIJmzKoLfkLHpLzkI32IWjeiMZdY9ga3qBrkWfomvJlUjt2A3UwuEwra2t6k5+GpmuwqvHgMuHs3WOA/pma/y+t7eX3/3ud8k2Q6E4gN7eXgYHB8c8T4T95L13BwteuhptsJ/GE26lft1v8OSvPax4fMicRfuR11F99kP0Fp9J9u57KH/+c5hdURP2DsHj8agiwWkkUWmZ/wLeBBYJIZqFEJ8XQlwthLh6+JQngTqgBvgTcE0irpsMYgn+SBWsQjHdBINB2traxgzl6Ac6KHv5Guy1D+GquIi9Z9yLJ/+khNgQNtpoWf09Gk68A014kAUvX0Nm7X/G/D0pJc3Nzaq75jSRqCydy8Y4LoFrE3GtZHP99ddTW1vLypUr0ev1WCwW8vLy2LZtG08++STnnnvuaNfM22+/nf7+fn784x9TW1vLtddeS1dXFykpKfzpT39i8eLFSf7fKGY74w3lpDjfp/jN7yHCfhpPuDVhQn8w/bnHUnP6Pyh8+8fkv3c7RncdbUd+NeZ+AAwViDmdTrKzs6fEJsWHzO7dkqeuh/YPErtm7go469aYh2+99VZ27NjBtm3bePnllznnnHPYsWMHZWVlo10uo3HVVVfx+9//nsrKSt5++22uueYaXnzxxcTarph3tLW1jRnKSWt+icK3f0wwNY99p/x63Burh0tEn8q+tbeS88FdZFX/C6NnH/tOuIWILiXq+VJKurq6SE9Px2AwTKlt853ZLfgzgDVr1ozZhri/v59NmzaxYcOG0cf8fv9Um6aY43R3d9PT0xM3lJPW9DxFm29gMGMJDSf+jIghbXqME1o6jrgOf1oZBVtvo+T1b9J44s/iin5rayulpaXTY988ZXYLfhxPfLrYP49Yp9MdcGvt8/mAoerC9PR0tm3bNu32KeYm/f39Y8btbfueoXDzTQw4VtC49mdE9EPv1U5viIaeAPv6gjS7g/hDH66RadZSkKanIE1HeaYRi2Fy23y9pecQ0RopevsnY4q+1+tlYGCAlJToxxWTZ3YLfhKI1/I4JyeHzs5OXC4XFouFJ554gvXr15OWlkZZWRkPPPAAGzZsQErJ9u3bOfLII6fZesVcwOfzsW/fvvieffNLFG6+EW/WUTSu/T/afDpe29PH640D1PZ82HIhw6QlRS8QAqSEdwbD+Ia/ADQClmYZWV1g5sSSVHIshycX7qKP0gSjot9w0h1R0zZHvPzy8vLRWRGKxKIEf4LY7XbWrl3L8uXLMZvN5OTkjB7T6/X88Ic/5Nhjj6WsrOyATdl//vOffPnLX+amm24iGAxy6aWXKsFXTJhgMEh9fX3cTVqz8wMKN9/AgH05m1b8lHvf7Oe1xgEksMhh4HOrMliSZaTIpj/Eg5dS4hoM09wX5P12H++0DPK393q5+71e1hSaOW+RlSNzTRMWZHfRR2kGCt/+MYWbb6TpuBuipoH6/X76+/tVG+UpQrVHnkeo5252Ew6HqampGR10Ew2Dp4kFL32JoN7Kd6y38FijFoNWcN4iK+srrYflpXf2h3impp+n93ro80dYkKHnc6syWJk38TbH9up/k7f913Qt+jQdK74c9Ry9Xs/ChQuVl3+YxGuPrDx8hWIWEIlEaGhoiCv2Wn8vJa9/nVAENni/we4eLectsrJhuY10k/awr51t0fGZlelcssLGqw1e/rW9l++/0MnR+SY+tyqDkvTxZ9a4Ki/B0N9E1p57CaQW0LPg/EPOCYVC9Pb2qrGIU4ASfIVihiOlpLGxcTQJIPpJYQre+hEabycX+7+Px1bAL05zUJqRuDRHg1bw0XILp5Sm8vgeDxt39PH//tvGpStsbFhuQ6cZh0cuBG0r/xeDt538927Hby1mIGvlgf8VKWlra8Nqtao+OwlmVs60nclhqJmKes5mJyOVqAMDA3Ffw/QP/kpa1xa+H7yS4kWruPOsvISK/f7otYJPLE3jj+fnc2JJCv/c3sfXnmqjrmec/fc1OpqOu4FAaj5Fb/8Ira/nkFMikQgtLS0Jtlwx6wTfZDJNvP3rPGekHa3JZEq2KYoJ0t7ejtvtjvt+jzS+QX7133kwfDLFx13EF4/JxKCd+vi3zaTlmydm8b1TsugeDPO1p9p4qtozrs9mRJ9K03E3og24Kdr8E5CHtibp7+/H7XZPhenzllm3aRsMBmlubo5/e6s4BJPJRGFhIXq9PtmmKMZJb28vLS0tcQXU1dHMka99nnaZyXvH/44VhcmJe/f5wtyxycnWVh+nlqVy7bGZmHRj+5MZ9Y9RsPU2OpZ+ga6lnz3kuEajYeHChSq0MwHm1KatXq8fs7JVoZjt+P3+scXeG8Dy2k/QyxDNx9+cNLGHIW//R6dms/GDPu7b3kd9T4AfnppNdmp8iekpPY/Urm1k7/oL3qyjosbzGxoayM7Oxmq1qsydSTLrQjoKxVxnJCMnntj3+8N88NzdHM0u9iz5CrmFC6bRwuhohOCyI9L58Uey6fKG+MbT7dS4xmghIgStq75BIDWfwi03owke2CpZSonP56O5uZmqqira2trG3M9QxEYJvkIxw2hpaSEUCsU87g9F+NsL7/LF4H00ZxyPYdnHptG6sTk638z/nZmLVgPXP9fB5uaBuOdHdCm0rP4+em8budt/G/2cSIRIJILL5aKhoYGqqiqamppwu92qtfIEmHuCHxiAp78DNS8k2xKFYsK4XK64m7RSSn77ZgfX9f+CiD4Fz9rvwgwMc5SkG7hjfR6FaXpueqWLF+v6454/4DgC58LLyKx/FEv7W3HPHRH/vr6+Uc9/3759SvjHwdwTfKGBvc/CE/87JP4KxSzB6/XS3t4eN1zx3+p+ljf/mxWaBjpXf5uwKXMaLZwYGWYtt5yew4ocE3dscvFkdfQeVCN0LvsCvrRSCrbcgiYwvuycSCSClBKPx0NdXZ0aRDQGc0/w9SY475fQ2wgv35JsaxSKcREIBGhsbIwr9ru7/Ly49QO+on+E3sLT8BScMo0WHh5mvYYfnZrNmgIzv9vczUM7+2KeK7VGmlf/AJ2/h9wPJjZGVEqJ3+9Xoj8Gc0/wAUpPhFWXw5u/hbb3k22NQhGXkU3aeCGJPl+Y217t5BbD3xB6E20r/2caLZwcBq3gu6dkcVJJCn97rzeu6PsyFuOsvJjM+sdJcU7ssyulJBAIKNGPw9wUfIDTb4AUOzz2/yAcewNMoUg2Tqczbo8cKSW/esvFacEXOZpddKy4dkaHcqKh0wi+sdYxKvqP7o4dsulc+nkCKTnkv/szRCT28xINJfrxmZOCL6UkYrTBWbdB2zbY/Idkm6RQRCUQCNDV1RU3lPNyvZea5nZ+ZLgPr/0IesrOnUYLE4dWI/j6WgfHF5n505aemDF9qTPTtvJrmNz12Kv/PeHrjIh+fX29Ev2DmJOC7/f72bNnD+HF50PlGfDSLeDpSLZZCsUhtLa2xhX77oEQf9jSw62WjRilj9ajvxW1j/xsQacRfOvELFYPx/Rfqo+evePJP5G+glPIrvob+v6J99QZiemPFSqbb8zed04cIpEI4XCYtvZ2WH8rhHzwwk+SbZZCcQAejwev1xvzuJSS37zdzeLIXk4PvYyr8pIpH0A+Hei1gu+cnMWKHCO/2ORia2v0IextK/8XKTTkbf/NYV1npGhLif6HzEnBBxBC0NfXh9eUC8dfC9v+Cc1bxv5FhWIaGOkGOVYo550WL7+03EvQZKdryRXTaOHUYtAKvn9KNiXpen76Shd7nIdW5IbMWXQtvpy01ldJ7Ti8z66UksHBwTFHQs4X5rTgj7SWjZz4NbDkwpPfBPVNr5gB9PT0xI0v9wci/HlrD9fY3qLAt4eO5VePDiGfK6QaNPzkIzlkmrX8+MVOmvsO3aB1VV5CIDWfvPd/CZHDS76QUuL1emlqapr3oj9nBX+EUChEl9s3lLXT+u6Qp69QJJFIJEJnZ2dc8blvey9hv5evyPsYyFhKb8n6abRw+sgwa7nxtGy0GvjxS530+Q78EpRaI+1HXIvJXUdm/eOHfZ2R4qzm5uZ5LfpzXvCllDidTgYrz4PCNfDijRCIHTdVKKaa7u7uuDHlht4AT+zxcHvWU5gC3UM597N4o3Yscq16frAum+7BMDe83Ik/dOBz484/hf6sVWTv/NO4K3CjIaXE7XbPa09/7r6L9kNKSeO+fUROvwH6O2DT4W0CKRSTJRKJxE3DlFLyh3e6WaDv5nTvY/QWn8Ggfdk0Wzn9LHIY+cZaB9XOAD/f5CKy//MjBG1HfhVtwEN21d2Tus6Ipz9fY/rzQvABwuEwLZpCWHI+vPFL6O9MtkmKechY3v1rjQN80OHnDvtjCCQdy66aRuuSywnFKXxuVQab9g3wj229Bxzzp1fQU3o2mbUPo/e2Teo6Ukr6+/upq6uL25V0LjJvBH/kds5z7Ncg7IeXb022SYp5xlix+2BYcvd7PZxma2VFz3N0l19EMDVvmq1MLh9fYmV9hYUHdrp5uf7A0Gvn0s8Dguydf570dUayd2pqaubV9Lx5I/gw9CI3DRgJH3UFbL0buqqTbZJiHtHd3R03jPDkXg+d3jA/SbmfiN5C15LLp9G6mYEQgi+tzmR5tpFfvuk8IF0zlJKNq/Ji0vc9g6l3b0KuFwqFqK2txeOJ38lzrjCvBB+G858rPoXUp6hiLMW0IaWMG7sfCETY+EEfn7LXUNizma7FnyFsSJtmK2cGI4VZmSk6bnq5C+fAh2GXrkWfIqy3kLPj9wm7npSSffv20dcXu6nbXGHeCT6AJ2LCf/QXYfcT0LI12eYo5gG9vb1xvfv/VLlx+8N8XfMvguZsXBUXTaN1Mw+bScsP1mXhC0X46StdBMJDz13EkEbX4suxtr9FamfiPrsjNTs9PT0JW3MmMi8FX0pJQ+7ZSHMmvHhzss1RzHGklHR0dMTcrO31hXmkys11ObvI9FTRufSzSK1xmq2ceZSmG/jfExxUuwLctdk1+oXZXXEhQXM2OTv+AAnMtJFS0traSnd3d8LWnGnMS8EHCGnN9C67EmpfgMZNyTZHMYdxu91xq2o3ftBHIBzmqshG/KkF9JScPY3WzWxOKE7hkuU2nqv18mT1UKM1qTXSueQKUrp3jjkOcaJIKWlra8PpdCZ03ZlCQgRfCLFeCLFHCFEjhLg+yvF1Qog+IcS24T8/TMR1J0tb0blEUrLghRsT6ikoFCNIKeOOLXQNhHh6r4dv5n1AmqdmKBNFo5tmK2c2nzrSxuoCM3/c0s3OzqGMmt7Scwik5JG9688J/+yO3JF1ds691O1JC74QQgv8FjgLWApcJoRYGuXU16SUK4f/3DDZ6yaCiNZIx6LLYd8mqH0x2eYo5iD9/f1xvfuHd7mRMsKn/RvxWUvpK/5owq6t0Whi/hEzcPB5LDRiqI9+jkXHra866R4IITV6Opd+lpSe3VjbXk/4NUc22Ts6OuZUgVYiPPw1QI2Usk5KGQD+DXwsAetOC92l5xJKzUO+dLPy8hUJx+VyxY3dP723n2/lbMXibaBz2edBaA/rOiMibjAYSE9PJzc3l8LCQkpLS1mwYAHl5eUsWLCAkpISCgsLyc/PJzs7m/T0dCwWC2azGaPRiE43dHchhJhRXwwWg4bvnpzFQDDCba87CUUkvcVn4rcUDeXly8Q3RRxpyzLWYPnZRCLuHQuApv1+bgaOjXLe8UKI94FW4BtSyp0JuPakkVoDnYsvJ3/rbVDzAlQmzsNSzG9CoVDcfvf/2eUmEgnxqcADDNoqcBesm9D6I2JssVjIyMggNTUVrfbwvjD2R0pJKBQiEAgwODiI1+tlYGBg9IsrWeJXmmHguuMy+fkbLu5+r4cvHJ1J59LPUrT5BtJaXsZd+JGEX1NKOVodnZ+fP2O+AA+XRAh+tGfg4HfEu0CJlLJfCHE28AhQGXUxIa4CrgIoLi5OgHlj01NyFo6qu9G9fAuaitNglr+oiplBb29vzGNuf5j/Vnv4eva7pPY1s+/4m8fdIE0IgRACh8NBZmbmqFeeKIQQ6PV69Ho9qampOBwOYGgcY39/P263G6/XixBi2geLnFpmYXdXgEeqPCx2GDmx+KNkVd1D9q6/DX1hTkGTOSnlaFptQUHBrBb9RDw7zUDRfj8XMuTFjyKldEsp+4f//SSgF0I4oi0mpfyjlPIYKeUxWVlZCTBvbKRGT9eiz6Bp2YKse3larqmY+8SrrH20ykMgFOZToYfxpS3AnX/yuNYUQmC321m0aBHZ2dkJF/t4GAwGMjMzKS0tZcmSJRQWFmK1Wke/gKaLLxydwSKHgV++5aLFE6FryeWY3HVYW1+bsmtKKenr65v17ZUTIfjvAJVCiDIhhAG4FHhs/xOEELli+B0hhFgzfF1XAq6dMHpLzyFoziL4nMrYUUyewcFBgsFDB3oADAQjPLHHzVezt2H1NtK55MoxPVMhBKmpqSxcuJDc3NyEhG4mg0ajIS0tjZKSEhYvXkxubi4Gg2FahF+vFVx/UhZ6jeCWV7voyD0Vv6VwqJPmFH52R/pxzeZOm5MWfCllCLgOeAaoAu6XUu4UQlwthLh6+LSLgB3DMfxfAZfKGfaMSa2BrkWfxtC+lcHdzyfbHMUsJ553/2xNPwPBMFeGH8JnLcFduC7uWkIIcnJyKC0tRa/XT4G1k0Or1WK321m4cCFlZWWkpaVNudeflarj62sdNPYG+f0WN12LL8fcW421bWprakY6bTY2Ns7KObkJCXhJKZ+UUi6UUpZLKW8efuz3UsrfD//7N1LKZVLKI6WUx0kpZ2SlU0/ZeQRNdiIv/TRuKp1CEY9IJBKzL0soInl0t5svZGzH5q0fmlMbJzNHq9VSXl6Ow+GYFbHjlJQUiouLWbhwIZmZmVMq/Efnm7lkhY3n67w8GDyBQEoeWVPs5cOHIxNno+jP20rbaEitEefCT5La+S6d7/531t62KZJLvM6LbzQO0OUN8SXxMH5LEX2Fp8U8V6PRsGDBAkwm01SYOaXo9Xry8vJYvHgxWVlZU5biedkKG0fmmvjdFg+7iy4lpWcXlo7NCb/OwUgpGRgYoKGhYVaJvhL8g+hZcD4hg43Ud/8QN8tCoYhFV1dXVBGQUvLwLjcXWnbiGKiha9GnY1bVCiEoLS3FaJzdPXW0Wi3Z2dksWrRoSoRfqxF8Y62DVL2Gr9YdQ8CUTdbuvyds/XiM9NSvq6ubNREBJfgHEdGl4Kq4iLS213FVvUYgEEi2SYpZhM/nw+/3Rz22vcNHbU+A/zE+TtCcTV/JmVHPE0JQUlJCSkrKVJo6rewv/Ha7PaGin2HW8s0THTT2Cx40nEeq833Mzg8Stn48pJT4/f5ZI/pK8KPQXXERYZ0ZR9U9s3pHXjH9OJ3OmO+Xh3e5OcW0lyLvBzgXXorURN+AtdlsWCyWqTQzaWi1WnJzc1m4cCE2my1hwn9EronLVti4sXMtg1orWXv+kZB1x8OI6NfW1s74kYlK8KMQNqTRveDj2JpeIOKqm7Od8xSJJRwOx9ysbeoLsrXVx/Wp/yVksNFddn7U80YycuY6er2eoqKihG5GX7zcxuK8dP4UOIO0tjcw9tUlZN3xEggEZrzoK8GPgavyEqRGi2PPvXR2dqrQjmJM4g3PeGKPh6XaJpZ4N+OquAipMx9yjhCCzMzMGZl6OVXk5OSQnZ2dENHXagRfO8HBw9qzGMBIRtW9CbBwYgSDQWpqamLWYCQbJfgxCJmz6C09m/SGJ9EOdM36CjvF1DLSaCvae8QbiPBCXT8/SHuSsNZMd4xpVkIIsrOzp9rUGUdWVhY5OTkJEf0Ms5YvnljKfaGPkNH8HHpvWwIsnBihUIiampoZ6SQqwY+Dc+EnEZEQ9poHGRwcVFk7iph4vd6Y6XnP1/ZjD3VyrO81ehZ8LOqs2pHeOMmuoE0WDocjYZ7+kblmuiouJSwFg+9MT8bOwYTDYWpra2Nu4CcLJfhxCFgKcRecQmbdI4hAP21tbTM6PqdIHp2dnVEFPyIlT+zx8E3rswgErsqLo/7+iODPZ7KyshIW0z97VTmvGk5hUdcztHV2JcC6iTMi+j6fLynXj8acFPxH3m+j3ZOYGJpz4SfRBj1k1D9OJBKho6MjIesq5g6Dg4MMDg5GPba1ZZCB/l7OCb9Ab9HpBFMO3ZAdaYim0czJj+OEyM7OHq3QnQxajcB23BWYRYCmTffhCyWnOCoSiVBXV8fAwEBSrn8wc+4d1uMNcNuzNXz/+XZ6fZPPix20L8PrOAr73vshEqK3t3fGbsgokkNnZ2fM/Z3H9ni4yvwC+ogP56JPxlzDbrdPlXmzCiEEubm5CUnZNOdU0JRxLOcHn+bud9oTZOHEiUQi1NfX09/fnzQbRphzgp+RauB3l1u0aIQAACAASURBVB6BayDMT17sZDA4+W/2rkWfxDDYga3phdHRZwoFgN/vj/lBbnEH2dXm5nLNM3hyjsNvK496ns1mm9Y2xzMdIQQFBQVYLJZJi37wiM/gEG5sDU/xSn3sYTRTjZSSxsZG3G530myAOSj4AEcV2bj+5GxqewL89NUuguHJZdf05x6PL62MrOp/gpT09PQoL18BDLVRiOXdP1nt4ULt61jDvTG9exW7j44QgqKiIsxm86REf8CxEm/GEq4xPMnv3u6ixZ28z62Ukqamprjpu1PNnBR8gGOLUvjKcXbea/Pxyzddk0upFALnwssw9dWONmZSXr4iGAzGLLTyhyK8WOfhOtNTDKYvwpu1Kup5ZrN5VjZHmw40Gg2lpaUYDIbDX0QIXIs+SYFs53TNVm57rYvAJB3AySClpLW1NWn6MWcFH+D0cguXr0zn5QYv92ybXEplX9HpBE127Hv/jRz28lXGzvwmXgX2a40DrAltJT/cgnPhZVHHZmo0mnmZdz8RNBoNZWVlkypGc+efTCA1n++mPUVdT5C/bE2ehw1Dot/Z2UlbW9u01/bMacEH2LAsjfUVFh7Y6ebJ6thta8dCag10l1+ItWPzaMl2Z2dnosxUzDLC4XDcISdP7fVwrfFpAuZs+gpPjXqOVqslNTV1Ks2cE+h0OsrKyg6/RkGjw1l5Cdn9VfxPWQv/rfbwxr7kxfPhw+HoTU1N09peec4LvhCCL6/JZHWBmd+/083bzYefHtVdfgERrQmH8vLnPfHisLXdAXSuPaySO4fy7qO0QB6J3c+GoSYzAYPBQFlZ2WGnrvaWnk1Yb+UK8V8W2g386k1XwlK3DxcpJR6Ph+rqarze6fkCmvOCD0M5ud860UF5hoH/e81Jjevwqt/ChjR6Ss/Gtu9ZdL6hkbyqsdr8I14bBRjarP2S/klCuhR6ys6LuU56evpUmTgnMZlMlJSUHNaXZESXQveCj5He+io/PHpI6G973TnphI7JIqUkFArR0NBAS0vLlDuQ80LwAcx6DT88NZs0k4YbXu6iy3t4T6yr8hJEJERmzUNIKXG5XLOiD7YicXg8npi34d5AhKr6fZyteYvesvOI6KO3ObbZbPO2jcJkSE1NpbCw8LBE31V+IQjB4tb/8NXjHex1Bbj7veTG80eQUtLb28uePXtobW2dMuGfN4IPQ42VfnRqNr5QhJ+81MnAYeToByyFePJPIrPuP4jQUMm0y+VKtKmKGUysNgoArzR4+aR4Bi0SV8WGqOeoVMzJYbPZyM3NnbDoh1Ky6Ss6jYyGxzkxL8J5i6w8utvDW00zowpWSjkaKt6zZ8+UiP68EnyA0nQD15+Uxb6+ILe91kU4MvFbOmflJegCbtIbnxq9vZ9Ncy0Vh8/g4GDMhlhSSl6u7uTTuhfoKziFYGpe1POMRqNKxZwkdrv9sJqtOSsvRRsaJKP+cT63KoOKTAN3vumio3/m7MVJKRFCTEnkYN4JPsCqfDNfXp3J1lYff3134rd0A44jGcxYhKPmfpCR0dCOYu4Tr9CqpjvAas8LWBjAtfDSqOdoNBqysrKm0sR5w+G0VfZlLKI/6yjsex9AL8Jcf5IDpOTW1yZfoDkbmJeCD3DWQivnLx66pXt67wTTNYXAWXkJRs8+LO1vj7ZbUBk7c5tIJILHE/u98my1m8/pnsaTsYxB+/KY51mt1qkwb17icDgmHN5xVV6CYbCDtJZXyLXqR+P5h+P8zTbmreADfH5VBkfnm7hrczfb2yfWwtRd+BGCJgeOvRuBodsw1UlzbtPf3x9TWAaDEcS+1ygRHfQsvCTqOaor5tRgt9spKioat+h78k7An1qAY+/9AJxQnML5i608vif5+flTzbx+5w2la2ZRkKbnp6920TaBvFyp0eOquBBL5zsY++pGd9ln2sADReLo7u6OuVfzaoOXz/AUA8Zs3AWnxFxDdcWcGtLS0igvLx9fEzqhxVW5gZTuHZhdOwH47FEZLLQb+OWbLlqTnJ8/lcxrwQdINWj4wbqhmOqNL3dNKHOnZ8HHiWiN2Pfz8ltbW6fETkVyCYfDcYtj9u75gOO1u+irvChqoRUMiZLqijl1mEwmKioqxtWGobfkbMK61NHPrl4ruP6kLDRCcOurXfiT1D9/qpn3gg+QZ9Vz/UkOmt1Bfv6Gk8g4+1uEDWn0lJxF+r5n0fqH4n8DAwMzou+1IrF4PJ6YIYO6ngAf9T5OQGOiZ0H0QishhNqsnQZ0Ot24irMi+lR6ys7H1vIy+oGhUGy2RcfXT7BT1xPkj1vmZjxfCf4wK/PMfPHoDN5uHuSf74+/0Zqr4mI0kQCZdY8CQ15+c3OzKsaaY8QL57y5p4nzNZvoLj6LSJR5tTDkfapUzOnBZDKNK3vHVXEhSElmzUOjj60uTGHDsjSeqennhbq557gpwd+PcxdZOaPCwsYdbl5vHN/mTSCtBE/OcWTWPoyIDMX+wuEwbW1tU2mqYhoJhUIxRxj6QxEKmx7FIEK4F0WfV6tSMacfu90+5hdsMDVvaGZ1/WOI0Iev76ePTGdFjpHfvd1NQ09gqk2dVpTg74cQgi+vzmSxw8gv3nTR0Du+F9tVuQG9z0Va84vAkJff19enQjtzBLfbHdNbfLuxj4t5juaMYwlYi6Oeo9FoVCrmNCOEoLi4eGwvv/JitEEP6Y1Pjz42ksyRatDw01e78AbmTjxfCf5B6LWC75zswKzTcNPLXXj8Y4dm+nPW4LOWDM29HY7/j0y3UaGd2U+8cE5gzzNkCTeBZdELrUZSMVVXzOlHr9eP+dwP2FcwmLEIe80DID98jTPMWr59koOO/hC/eDN2o7zZhhL8KNhTdHz3lCycAyH+73Xn2O0XhIbuig2k9OwmxfXB6MORSISWlpYptlYxlQSDwZiptm3uAKf3P0aHoYSBnNUx18jIyJgq8xRjMGbPIiFwVlyCydOIpeOdAw4tyzbxuVUZvNk0yEO7kjuLNlEowY/BkiwjV6/O5L02H//cPvYmbk/JesJ665CXP8xIv+tkDy5WHD6xRhgC1O58m2WaRnoXbog60QrAYrGoVMwkotPpxmxD7S76yPA0u/sPOXb+YisnlaRwz7beCRdnzkSU4MdhfaWVM8ot3L/DPWZHPakz0112Hmktr6D3tn/4+HDWjmq7MDvp6emJejsfjkgqWx7GIyyEKs6K+rsajUYVWs0AsrKy4oZ1pEZP94ILsHa8hcHdeMAxIQRfOc5OYZqeW1/rovMw26rPFJTgj8HVazKpzDRwxybnmBPvuyuG+m1n1j50wOMqtDM7CQQCBALRN+6r6+pZJ9+hJvdcpC56NohGo1EjDGcABoNhzE3z7gUfJ6LRY6998JBjKXoN3z05i1BEcssryR2CPlkSIvhCiPVCiD1CiBohxPVRjgshxK+Gj28XQqxKxHWnA4NW8J1TstBpBDe/0sVgnErcYEou7vyTyax//IA0LxjqwxIvPKCYecR7vcx7HkQKgeHI6KmYarN2ZjFWK+WwKYO+otPJaHgSTeDQEGyhTc/XTnCwtzvAXZtds3YTd9KCL4TQAr8FzgKWApcJIZYedNpZQOXwn6uAuyZ73ekkO1XHt04cqsT9zdvxX2xX5YZD0rxgKLTT0tJCMDh3+3TMNWKFc/o8Hk4eeI7tqWuRlpyYv682a2cOJpOJlJSUuOe4Ki9GE/aRUf9E1OPHFaVwyXIbz9V6eWrv7Ey5ToSHvwaokVLWSSkDwL+Bjx10zseAe+QQbwHpQojo0yFmKCvzzHz6yHReaRjgierYLXIH7EcwmL4Ie82DoymaI0QiEZqbm2etdzCf8Pv9Mb+c+95/jDQxgH9p9FRMGBrFpzZrZxZjtVH2pVfidRyFvfYhiESP1X/yCBurC8z84Z1udnTMvk3cRAh+AdC038/Nw49N9BwAhBBXCSG2CCG2dHV1JcC8xHHRsjRWF5j5y9YeqrpidMUUAlflBkyeBiwdmw85PDAwQE/P3OzTMZfo7Y2emSUjYZa3/4c9mgqsxUdGPUej0ZCZmTmV5ikOA7PZPKaX76zcgGGgnbS216Me12oEX1/rINei45ZXZ98mbiIEP9pX5sEu7HjOGXpQyj9KKY+RUh4z08rRNULw9RPs2FN03PpaF32+6EVVfYWnETRmDhVzHISUkra2tpibgYrkMzJXNNqdWG/16xTTRm3hJ2KmYkopsViiDy9XJJexvHxP/okEUvKw7z30szuCxaDh++uyCUQkN7/ShW8WddZMhOA3A0X7/VwIHNwjeDznzAosRi3fPTkLty/M7W9EL8qSWgM9Cz6Otf1NDJ7GQ48PV+Gq0M7MxOv1xqyszdj7AJ0ynawj18f8/bS0NDXkZIYyppcvtLgqLiTVuQ1TT3XM04pser6x1kFdd4BfvzV7NnET8a58B6gUQpQJIQzApcBjB53zGHD5cLbOcUCflHLWdhcrzzTwpeGirPt3RM/kcJVfMJTmVXNomheAz+ejs7NzKs1UHCYulyuq4Ee661nuf49NaedgNhqj/q5Go1GbtTOcsbz8ntJzCWvN2GsOLcTan2MLU/jMyqF9vQd2zo7iykkLvpQyBFwHPANUAfdLKXcKIa4WQlw9fNqTQB1QA/wJuGay1002Z1ZYOLUslfu297Gt7dBOimFTJn1Fp5He8BSawKGbvFJKnE6narA2wwiFQjFfk8j7/8Yv9WiWXxB3DZV7P7MZy8uPGKz0lp6Frel5tL7uuGttWJbGKaVDlbhvjlGcORNIyH2nlPJJKeVCKWW5lPLm4cd+L6X8/fC/pZTy2uHjK6SUWxJx3WQihOCaNZkU2vT87HUnzoFDN29cFRejDQ+S0fDfqGtIKdm3b5+qwp1BxNpQ1wTcLHY9x/PatZTl58b8/fT0dJV7PwsoKCiI+zq5Ki5CEwmSWfdI3HWEEPy/4+wstBv4+RtO6md4O2UVaJwEZr2G75zkwBeS3B6lyZovYxFex5HDKZrRN3gjkQiNjY2zJgY4l5FS4nJFj8eKXY9gwk9z6YaYQiGEUOGcWYLBYIhbjBWwluDJPY7M2v8gwvFF3KjT8P1Thtop3/BSJz2DM7dDrhL8SVKcbuCaYzPZ0ennvihN1lyVF2MYaMPaGj3NC4bi+R0dHVNppmIcxNysjYTIrX+ItyJLWbZ0Rczf12q1aqrVLMLhcMSdf+uquBi9vxvb8JyLeGSm6Pjhumzc/gg3vNw5YzN3lOAngNMWWDi9PJX7d7jZ2npgPN+dfxKBlFwccdK8RjxL1VUzuTidzqiCn9r8KhlhJ29kfIx0kzbm72dkZKhwzixCCEFRUVHM12xozkXpAXMu4lGeaeBbJzqocQW4c5Nr3LOxpxMl+AniS6szKU7X8/M3DornCy2uiotIdb6HqTd2mtdIV02Vn58cgsEgXm/0sZamXf9mXyQLx5J1MX9fCDFmG17FzMNsNsfedxkuojT37iHFtX1c6x1blMJnV2Xwxr4B7tk2/tnY04US/ARh0mm4/qQsAuFD4/mjaV5xvHwYiuc3NDSoKVlJwOVyRX3c1F1FXv9OHtCexcr82Nk3BoMBY4xUTcXMJjc3N2bdRG/xekIHzbkYiwuWWFlfaeHBnW6eitOGJRkowU8gRTY916wZiuf/64MP8/OH0rzOxtb0HDpfdGEZIRAIUFdXp0R/GolEInR3d0fdrLXs3ki/NOEuOxetRm3WzkW0Wi2FhYVRvXypM9Gz4GOktbyK3ju+0qGR2djH5Ju5651u3mk5NG07WSjBTzAfWWDho+WpbPzgwPx8V8WGoTSv2vhpXjAk+vX19Ur0p4lYeye6wS6yWl/k/vA6TlqYHXcNFc6Z3Vit1pg987vLh9po2Gseino8GlqN4NsnOSjLMHDba13UuGL03ppmlOBPAVevHsrPv/0N52iKVsBahDtvLZl1DyPC8V98KSV+v5/6+vqYJf6KxNHV1RX1ec6oeQhBhM2Z55NriZ3NYTabVWfMOUB+fn7U0E4wJYe+wlPJqH8MTTD6Pk80zHoNP1qXhdWo4ccvddLmSX5rdCX4U4BJp+HbJzkYCMoD+u24Ki9B5+/F1vT8mGso0Z8eBgcHo26Ui5APW+0jPBc+mpWLKmL+vuqMOXfQ6XQxC7JcFRejDXlJb3hyQmtmpui44SM5hCPwwxc76Y3RcHG6UII/RZSmG/jSMRm83+7jweE+G96sVfhs5Tj2bhxXmpeUEp/Pp0R/CnE6nVFj9+n7nsEU8rBRew7HFsYuw5dSkpaWNpUmKqaRtLS0qK0xBu3LGMhcPtQBN0YRZSyKbHp+eGo23QNhfvJSZ9ypeVONEvwp5IwKCyeXpPDP7b3s7PSBEDgrLsbUV0tq59ZxraFEf+oIBALR4/dSkl69kR2RUjLKjkGvjZ1bb7VaVWfMOYQQgsLCwqivqXPhJRi9LVjbNk143SVZRr51koPa7gA/fbWLYJLm4qp36hQihOC6Y+3kpOr42etO3P4wfcWnEzKmY9+7cdzrKNGfGjo6OqJn5nS8TWp/I38JncUZlbGHXwshcDgcU2miIgnECu24808mkJKDo3r8n939ObYwheuOtfNem487NkVvrT7VzEnB12g0M6Y3TYphKJ7f6wvzyzddRDQGXOUXkta+CYP70F75sVCin1h8Pl/M7Bx79UacpNPoOIWCtNibtUajccwJSorZic1mOzS0o9HhKh8uoozTKz8eZ1RY+NyqdF5rHOCud6KnAk8lc1LwTSYT+fn5M6bMvcJu5LOrMni7eZDH93joXnABEY0BR83EPAUl+omjvb096ofN2FeLtXMzfw2ewWmVsTdjhRBkZ8dP1VTMbvLy8g7RkJ6y8wjrzDj2/vuw1/3EUhsXLUvj6b39/H2aq3HnpODDUF+TyspKTCbTjBD+8xdZWVNg5q/v9rDHm0JvyXrSG55C65/YfFsl+pNncHAwZhsFx96N+DDymO4Mji+O7b1rtdqYeduKuYHRaDzkNY4YrPSUnout6Xl0A4c/wOiKlemj1bixhihNBXNW8GGo3L28vByHw5F00RdC8D8n2Ek3abntdSfNpRehiQTGVYh1MEr0J0dra2tU717nc2Hb9ywPhE9mTXkuhhibtRqNJm5rXcXcIScn55DX2VV5MUgZc5rdeBiZp3FqWSr3bOvl0arpaZw4pwUfhp7YnJwcysrK0Ol0Sf2Qphm1fONEBx39IX5elYYn5zjstQ+NWYgVDSX6h4fX68Xn80U9llnzEERC/CW0nrMq4w8hV5W184NoXn4wNR934Toy6x+dUCHWwWiE4H+Ot3NCcQp/2tozLX135rzgj5CSkkJlZSVpaWlJFf3l2SYuW2HjlYYBnrF+HJ2/B9u+5w5rrRHRb2hoUKI/DqSUtLW1RR9wEvKRWfcIr3I0Gbll5Fmjb9YKIbDb7SoVcx4Rzct3Vl6KNtg/4UKsg9FqBN9c62B1gZnfbu7m2ZqpHXk6r961Wq2WoqIiCgoKkvqBvXi5jSNyjHyvqhi3pRzH3n+BPDzBllIyODioRH8c9Pf34/dHv5tKb3wKXaCP3/rP5uyF8WPzdrt9KsxTzFCiefmD9mV47UfgqLl/woVYB6PXCr5zchZH55v49Vsunq+dOtGfV4I/Qnp6OhUVFRiNxqR4+1qN4OtrHZh0Gn7tPwuTuwFL+1uHvZ4S/bGJ590jwzj2/pu92grqTUtZU2COuY7ValV9c+YhUWP5Cy/B4G0lreWVSa9v0Aq+d0o2K/NM/PJNFy/UTk14Z14KPgxt6FZUVGC325Mi+vYUHf97goO/edbQrXXgqL5vUusp0Y+P2+0mGIzevCqt9TWM/c3cOXg2Z1Za47ZBzsrKmkozFTMUo9F4SAsNd/5J+C2FOPbcN65WKWNh0Aq+f0oWR+Sa+Ou7PfT7Q2P/0gSZt4IPQx/g3NxcSktL0Wpjj66bKo4pMHP+kkx+5zsTS9d7mLqrJrWeEv3oSClj5t0jJY499+HU5fKsXMPpFbE3aw0GA2ZzbO9fMbc5xMsXWpyVl5LSU0WKc1tCrmHUafjBuixuOzMPizHxd5LzWvBHSE1NTVrO/mdWprPFdiYeacay65+TXm9E9FX2zof09fXFnC2Q4tpOSvdOfh84i+OKLDhSon/INBqN8u7nOQaDAZvNdsBjvaVnEzKmk7Vn8p/dEUw6DYVxKrwngxL8YXQ6HeXl5bHnW04Req3gupOLeUCeRlb7Kwh386TXVCmbHzLi3cd6Hhx7/smANo17Aydz7uL4m7WqK6biYC9fao24yi/C2v4mxr66JFo2PpTg74cQgoKCgqgl1VNJrkWPftUniUhB91v/SMia+4v+fJ6c1dfXF1Psje4G0treYCNnkp9hYVlW9Jm0QggyMzNVKqYCvV5/iFPYXf4JIlrTpPfhpgP1Do5CZmYmZWVl0/oBP6qihC2WdRzT9yzb61sTsuZ8F/2xvHt79b8IaQz82nsa5y2OX5+hZtYqRsjJyTng57DRRk/ZuaTve3ZS7RamAyX4MRgp1JrO1E3bcZ/FKIJ4ttyH05uYHfqRyVnzcTB6PO9eN9BJeuPTvGA4jZAxnZNLYvfN0ev1GI3RvX/F/EOn0x3SWsNZeSkw1ItpJqMEPw56vZ7y8nIyMjKmRfQjGWV0ZZ/IpTzLr19rSli/7BHRr62tJRRKfKrXTGTM2P3efwOSm/rWc2aFBaMu+kdBCKHaKCgO4eBq62BqHn1Fp5FR9yha//Q1Q5soSvDHQKPRkJ+fT3Fx8bSEePqXX45NeDmq5ynufT+xrVMDgcC8Ef2urq6YdzRafx8ZdY+xNfVkWsgas7L24MwMhWJEF/bXhK5Fn0EbHiSz9qEkWhYfJfjjxGq1snDhQlJSUqbU2x/MXEp/1tFca3yKR3e62NIymND1g8EgNTU1UQd3zxX8fj9dXV0xh0vYax5EGx7kxr6zWFucQlZq7HxnnU6nwjmKqKSlpWEwGEZ/9tsW4M47Efve+9GEBpJoWWyU4E8AnU5HWVnZlLfG7Vr8aTIi3XzR8iZ3bHLSlaB4/gihUIja2tqYfWVmM1JKmpqaYoq9JjRAZs0D7LEcx/ZgIZ9YGjvVUgihNmsVMRnJ6ttfC7oWfwZd0ENG3WNJtCw2SvAnyEh5/YIFC6as3bI3ezUDGYu5Vv8YkXCI/3vdSSjB8y/D4TC1tbUxWwXPVrq7u+N+kWXUPYYu6OEW7zkszzZSaY/vvatwjiIeZrMZm802qgOD9uX0Z63CUf0vRHjm3UUrwT9MzGYzlZWVWK3WxIu+EHQtvoKUwVburNxOVZefe95L/Ci0SCRCXV0dAwMz8/ZzogQCgdgtFAAR9uOovo9myxG8PFge17sHFc5RjI+D63a6Fn8Gvc9JeuNTSbQqOkrwJ4FWq6W4uHhK5ud68k/EZyvnJOdGzq4w83CVmzebEi/MkUiE+vp6PJ6pH74wlUgp2bdvX9yh0Bn1T6D3ubgjcAFFNj3HxOmKqcI5ivGi1WoPEP2hO/QlZO25FyIzK0FCCX4CyMjISHy7ZaGhc/HlmDyNfCNvOxWZBn6xyUmbJ3rHx8kwIpa9vdM7UDmRdHR0xA3liHCArD330pm2nIfdC7lgiRXNGK+VSsdUjJf09PQP7waFoGvJZzF4W0nf90xyDTuISQm+ECJTCPGcEGLv8N9RXSIhRIMQ4gMhxDYhxJbJXHOmYjQaqaioIDMzM2Gi7y48Fb+1mPzqe/jOSUNtnG95tQt/KPH9caSUtLS00NXVlfC1pxqv14vL5Yrr3ac3Pol+sJPfRT5BuknLurL4IwwtFssBGRgKRTyEEBQWFo5+9j15JzCYvoisqr/PKC9/sh7+9cALUspK4IXhn2NxqpRypZTymElec8YihCAvL4+SkpLEtFsWWroWX4Gpr5YK91t8/QQ7dT1B/rClZ/JrR0FKSWdnZ8wh3zORUCg0ZiiHSIis3f/AZV3C3c5FfHxJWswB5fDhHGSFYiKYTKYPizSFoHPpZzF6W0hvej7Zpo0yWcH/GPD34X//Hfj4JNebE1gsFiorK0lNTZ20t99b9FH8qQVkV93N6gIzFy9P49mafp6tmZqYu5SSnp6esUV0BiClpLGxccyWEemNT2MYaOf38kLSjFrOGaPQymw2YzKZEmmqYp6wfzdNT96JDNoqyKq6e9JjEBPFZAU/R0rZBjD8d3aM8yTwrBBiqxDiqngLCiGuEkJsEUJsmY3hhRF0Oh2lpaVRR6NNCI2OriVXYO7dg7X1NT51RDpH5Zm4a3M3e11Tk0cvpaS/v3/GD1JpbW0dO600EiJ799/psS7kT84lfGxxGmZ97Le98u4Vk0Gr1X6YxCEEXUuuxNjfhK3phWSbBoxD8IUQzwshdkT587EJXGetlHIVcBZwrRDi5FgnSin/KKU8Rkp5zGwfOCGEwOFwTDpnv7f4TPyWInJ2/RmtkHxjrYN0s5afvtpFn29qPAcpJQMDAzO26Vp3dze9vb1j3oVkNPwXg7eVP4oNpBq0nLsovndvMBhISYndSE2hGAubzTa6gesuOAWfrZzsXX+dEbH8MQVfSvlRKeXyKH8eBTqEEHkAw39H7Q0qpWwd/rsT+A+wJnH/hZnPSM6+2Ww+PNHX6Ohc+jlMfbWkNb+EzaTleydn0TsY5mevOxPWZO1gZmrTtf7+/tgDyfdDhP1kV91Nd9pS7upcyvmLrKQaYr/lNRrN5O/IFPOeAzZwhYaOpV/A2N80IzJ2JhvSeQy4YvjfVwCPHnyCECJVCGEd+TdwBrBjkteddWi1WkpLSw87rt9XdBq+tNIhT0GGqbAbueZYO9vafdw9BUVZI0gpCQQC1NTUzIhWDAMDAzQ2No5rfyGj/jH0g538QXMJZr2G88eYaKXVarFa45+jUIwHk8k0WqXtyT+JgYzFZO/6GyKS+LTqiTBZwb8VOF0IsRc4ffhnhBD5Qognh8/JAV4XQrwPbAb+K6V8epLXnZVoNBpKSkpIj5pbcwAAFrRJREFUS4s/bCMqQkvn0s9j8jRgG971P73cwrmLrPynys1L9f1TYPGHjPTfGRxMbDO3ieDz+WhoaBiX2IuQj6yqe3CmH8kf2is4f1EaVmPszCnl3SsSTW5u7ocZO8uuwjDQRkb9E0m1aVKCL6V0SSlPk1JWDv/dPfx4q5Ty7OF/10kpjxz+s0xKeXMiDJ+tjNzuHU7XTXfBOgZtFQfEA79wdAbLs438+q1uaqZoE3eEkVYMyajK9fl81NXVjXsTObP2YfT+bu4IbSDNqOXCMdooaDQa1TdHkVB0Oh0OhwMhBP05a/DajyCr6m5EOHl3yqrSNgkIISguLkani92WN/ovauhcdhXG/mYy6x8HQKcRfOfkLGwmDTe90kXP4NRusI5U5fb0TE0tQDTcbje1tbXjFntNsJ+sPffSmn409zkXcPFyGylxYvcjDfGUd69INFlZWUM984WgY/mX0PucZNY+nDR7lOAniZGY/kSHqnjyTsDrWEnWrr+O9ty2mbT84JRsPP4IN73SOSWVuPsjpaS1tZWOjo4pzdWXUtLR0RG33XE0snbfiy7Qx42+S8hK1Y454ESj0ai+OYopQaPRjPbZGchaiSdnDVm770ETSE7vKiX4ScRoNFJUVDQxz1II2ld8Gb2/G3v1v0cfXpBp4OtrHexxBvjVW/HbDCQCKSVOp5P6+nqCwcRvRA0ODlJbW4vT6ZzQ/0U30Il970b2Zp7KU72FfOqI9DGrake9MIViCrDZbKNtOjpWfBltwDPUWC0JqHd5krFarYcMURiLQfty+gpOwVF9H1rfh6GVE4pTuHxlOq80DLBxx9TP1RzJ1d+7dy9utzshXzLBYJCmpibq6urw+XwTXjNn118AybfdF1Js03NqWWrc84UQZGZmTsJihSI++6dp+tIX0ltyJva996MfaJ92W5TgzwDS09MnLPody7+EJhwge/fdBzy+YVkap5alcu/7fbxc702wpdGJRCI0NTVRVVU12nVzIhW6Ukq8Xi8NDQ1UV1fT19d3WF8exr460hue5C3bObzbn8nnj85Aq1HevSL57D8opXPZFwHI3vmnabdDvdNnCOnp6RPqqx+wltBTei6Ztf/B4GkcfVwIwf87zs7ybCO/eNPJBx3TM9FKSkkkEsHtdtPa2sru3btjDhKXUhIMBunr66O5uZndu3fT2NhIf3//pO4Scj64i5Auhf/tPJvji8wcnR+73z0o714xvYykaQZTcnFVbCC98RlMvdXTaoMS/BlERkbGhGL6Hcu+QERrJPf93xzwuF4r+N4pWeRadNz0ShdNfdNb7BGJRIhEInR2drJ792727NnDnj17qK6upqqqip07d1JdXU1LSwu9vb2Ew+FJ9+yxtL9NWvsmHjJ9gh6sfOHo+EIuhMButyemq6lCMQ50Ot1orUfX4s8QNljJ3f5bmMYmhUrwZxhpaWmUlZWNK8wQNmXSueSzpLVvwtL+1gHHrEYtP/5INnoN/OjFDlwD098aQUo56s0Hg0ECgcCoxz9yR5AQIiFy3/8VHlM+P3SexkXL0sixjJ3y6nA4EnN9hWKc/P/27j067rJM4Pj3+c09mcwkzeQyuTXNhQpIW0ophZabiHBwFWEXj0ddWUVZdgEtKEsVV1bwBnS9Lq7iyirqInu8LD0IFhSBolxawNaWtqQtzaVJc2tzmSTN9d0/Mim1ncnMNHNL8nzOyckkvze/95m3mae/vPP+nnfBggU4nU4mnHl0nPYxvB1byGt9Lm39a8LPQjk5OdTV1eFwOGK2PVR/DcPeCkq3fvuE4kylXgd3Xjy5XPMLT3cQGs6+ImjJULj3l7j79/OVsQ9TkOuOeZOVXt2rTBGRo3/FH6q5iiO+GoJbv5O2m7E04Wcpp9NJXV1dzIJrxnJwcMnNuPsbKYxwQ0d9oYs7LiziQN8oX3ymkyMpXqOfbrbhwxS//iC7PWfycGgpN6wsxGWP/WutV/cqU9xu9+TOeDYHbcvW4hxsI7D7f9LStyb8LGaz2Vi0aFHM2jv9wdWEis+m+PUH/2qZ5pRlQQ+3rQmwu2uYrz7Xyeh4dm9skoiS7T9Axoa4qfeDvLPWy9nTbEwOb13dJ3yXs1JJVFJSgs1mY6D4LHor3kHR7p+kZZmmJvwsZ1kWFRUV09/6L0LrsluQsSGC274TscnqqlxuXLmAV1qPcO/znYylqKRyOnm6d1Dw5gZ+YV3OYXdVzDdq4a2lmEpl0tTrWkQ4eMaNACcsvkhJvynvQc2YiFBcXDztCp4R30K6Fn+I/KaN5LZH3if+svo8rl9RwAvNQ6xPYR39tJgYo+zVe+m1LeCugau4eVUh3mnq5cBbyV7n7lU28Hq9+P1+xrxBOt/2EfwH/oC37U8p7VMT/izi8/mora2NmrA6T72W4dxyyl5bH/VNoPe+zcfHzyrg+aZB/v2PszfpBxoewdO7h9uHPsJ5dcUx19zDW9M5SmWLYDCIZVl0Lf4QR3zVlL26/miNrFTQhD/LuN1uampqIi7bNDYXrcs/gyvUTNGun0Q9x/tO9fHRM/N5rnGQe5/vmnVz+o6BVop2/JCnzQp25p3H9StiFz6b2qtW76pV2cRms1FVVQU2JweWr8Mx1EHx9gdS1p/+9s9CLpeL6urqiNM7AyUr6am8lMDun+Lq3Rf1HH97up9PnFXAH5sG+dKzHbNn9Y4xBF9dz/CEcPfEP/C5C4pwx7Eqx263a0VMlZVyc3MpKCjgSNESDtVeReGeX+Du3pGSvjThz1I5OTlR5/Tbln6KCXsuFZu/NO3GyVee6uOTqwp5re0IX/h9BwMj2Z/08/f/Bl/7S9w7+n4+cO4pVPhj36swtf+AXt2rbFVaWordbqf97f/ImLuQsi1fg/Hk3yGvr4BZzOfzUVhYeELSH3cX0HrWbXh6dlO066Fpz/GuOi//sibAG93D3LbxIO2h7Nms/HiOgVaKXvsmL4yfRl/937G6avpKmDCZ7AOBAB5P7Dl+pTLFsiyqqqowzjxal3+G/vILgORPtWrCn+WKi4sjXrn2lV9ET9VlFO/8Ee7Du6Y9x5qFudz1jhK6h8b59G/b2N2V+c3KT2DG8W66i5FxeLh4Ldcuj6/omdPppLi4OMXBKTVzHo+HkpISQuUX0HH6x8HmTHofmvBnOcuyopZWbl22ljFXARWb74556/aSUjfrLyvFZbf47FPtaSutHK/BzT+lPPQXfphzHR+94LRpyx5PmZrK0a0L1WxRWFhITk5OyjYw0oQ/B+Tl5UWcsphw+jiw4nO4+/YT3PqtmOep9Dv4+uWl1C1wsv6PXXz35e6sWMGzd/c2ljb+N8/bVnLepe/HMc0OVlOmVuW4XK40RKhUckzV2nG5XCl5z0kT/hwgIlGv8kOl59C5+MMs2Pco/qaNMc/ld9v4yqUlXH2qj8ffCHHbxoO09ae3vPKxnt3VxrJtd9Jn5eF4x+fJccZ305Tb7dY192pWstvt1NfXx1U8MVGa8OcIl8sVddlh++mfYCCwlLJX7sPVtz/mueyW8LGzCvj8hUW0hca46bE2NuzqYyKNdbuNMTy89TD1W79KuXTTuebL5PrjS+DHViRUSr1FE/4cEggEIic5y07zOV/E2F1Uvvh5ZGworvOtqszh/ncHOaPEzQNbDrPuyfa0bKZyeGicu57pJPf1n/Eu2yscXHIjEyVL4vpZEaG0tPToptFKqbdowp9DnE4neXl5EY+NeYpoXnknrr5GKl/+Ipj4auMHcu3ceXERt5xXSGPvKDc+1sr9L3VzeCg1tfVfbB7kxsdayW1/mXWOR+gpv5ie+vfH/fNTpWeVUifShD/HTFdVc6BkJW3LPomvddPk1mpxEhEuqfHywHvLuOKUPJ7cE+ITjx7gR68dpmMgOev23+ga5s6n2/nSs52c427hB65vMeyvoXXFZyHOqRmdylFqeloUfI7xeDy43W6GhiJP2xyquwZn6ACBhkcY8VZwqPbquM/td9u44ewFvGdxHj/d2sOvXu/jV6/3saoih8vqvZxR4sYZxwqaKaPjhq0Hj/BEQz8vtQzhc1l88u1j3NRyDzi9NK5Zz4Qj9s1VMJnsg8GgTuUoNQ1N+HNQSUkJTU1NUfeMPbj0ZpwDrQRf+waj7kL6yy9M6PzlPge3n19Ee2iMJxr62dgQ4k/Ng7hswrKgmyUlbir8Dsp9dgo99qMX6IOjEzT3jtLcO8quzmFeaBliYGSCXKfFh5b4ubrW4rQ/3YxtbIB9F/0nY57469Z7PB6tlaNUDJKqBf7JsGLFCrNlS+Ta7io6YwwNDQ2MjIxEbWONDVL93Fo8h3fRtOruhJP+sUbCV+pbDgyy+cAQHQOx5/dzHcLKihzOX5jDmUEPrvF+qjfdgrtnD41r7mOgZGXc/VuWlbJlbErNNiLyijFmRcRjmvDnpt7eXlpaWqa9Y88aDVG96dakJP0pxhh6hydo6RvlQO8oh4+MI0xe4jttQqXfQaXfQVGuDSt86W8b7qV601pcfW/SvOrL9Jetjru/qakcfaNWqUnTJXyd0pmjfD4fDodj2qv8CYeX/ed/nepNt1L14r/ScvYd9FZdNqN+RYR8t418t423F7tjtrcf6Wbhpltx9TfRdN7XCJWuSqg/ncpRKn66SmeOmlqPHmvFylTSHyxcQuXLd1G8/ftg0lMm2X1oJ7W/vw5XqJnG1fcmnOxF5Oi+oEqp2DThz2F5eXlxrVqZcHjZf8E3OLToPRTveoiqF+7AGk1t8bT8xt9S88w/Y8TGvou/x0DJ2Qn9vN5gpVTiNOHPYfFe5QMYy0Hr8ttpXbqWvNbnqXvqWnLbNyc9JttwL+Wbv0zF5rsZLDydvZf8F0fyT0n4PHqDlVKJ04Q/x3m93vgrRopwqP4a3rzofoxlZ9GmtZRv+Sq2kb6ZB2IM/saN1G/8IPlNG+lc/PfsP/+bjLsSn3+3LEtvsFLqJMwo4YvINSKyQ0QmRCTiu8LhdpeLyG4R2SMi62bSp0rMycxzDwaWsOfSH9O5+MPk73+cUx6/mpJt/4F9qCvxACbG8Df/jtqnr6Ny812MeMvYc8mDtJ9xA1iJrxnQG6yUOnkzXaWzHbga+H60BiJiA+4HLgVagM0issEY8/oM+1ZxcrvdBINB2tra4t5YwdhctJ/xT/RUXUbRrocIvPEIhXt+QX/Z+fSXnkt/6SrG3ZGnVGR8hJzu7eR2bCa/aSPOwXaGvZUcOOt2Dle/GyS+EseR5Obmkp+ff9I/r9R8NqOEb4zZCcS6elwJ7DHG7Au3/TlwJaAJP40KCgoIhUL09/cntJvOsL+GlnP+jY7TP05hwyP4DjyLv+VpAEZyyxh1BxjzBDBiwzbSi324B1d/I9b4MEZsDASW0bbsFvqDq0FmNoNoWZauylFqBtKxDr8caD7m6xbgnGiNReR64HqAqqqq1EY2j0xN7TQ0NDA6mniJ4xFvBW1nfpq2Zbfi7m0gr+0FXH1vYj/ShbunATHjjLnyGXUHGAgsZaB4BQNFZ8ZdCyee+KuqqrDb9dYRpU5WzFePiPwOKI1w6A5jzKNx9BHpcizqJaYx5gHgAZi80zaO86s4WZZFdXU1e/fujVpnJyYRjuSfclIra07W1Gojr9ebtj6VmotiJnxjzDtn2EcLUHnM1xVA6wzPqU6Sy+Vi0aJF7Nu3L2UbJSeTiJCfn6/bFSqVBOlYlrkZqBeRRSLiBD4AbEhDvyoKj8fDwoULs34uXETweDyUlZVlOhSl5oSZLsu8SkRagHOB34jIxvD3y0TkcQBjzBhwE7AR2An8rzFmx8zCVjPl9Xqz+g3QqSv76urqrI1RqdlGq2XOc6FQiMbGxqya3rEsi/Lycvx+f6ZDUWrWma5apt5pO895vV5qamqw2U5+bXyyiAg+n4+6ujpN9kqlgK5xU3g8Hmpra2lqamJ4eDitV/uWZWGMwev1UlpaGn8ZCKVUwjThKwCcTie1tbV0d3fT3t6e0qQvItjtdvx+P16vl5ycHCxL/9hUKtU04aujRIRAIEBeXh4HDx4kFAolLfGLCJZlkZ+fT0FBAW537M1RlFLJpQlfncDlcrFw4UJGRkbo6uqip6cHYwwiEvcNW1NTNU6nE7/fj8/nw+Vy6YobpTJIE76Kyul0UlZWRjAYZHR0lKGhIYaGhjDGYIxhYmKCkZERRkdHGR8fx2azYbfbcblceL1evF6vlkJQKovoq1HFJCI4nc6jV+tKqdlJ3ylTSql5QhO+UkrNE5rwlVJqntCEr5RS84QmfKWUmic04Sul1DyhCV8ppeYJTfhKKTVPZHU9fBHpBBpP8scDQFcSw0kWjSsxGldiNK7EzMW4FhpjiiIdyOqEPxMisiXaJgCZpHElRuNKjMaVmPkWl07pKKXUPKEJXyml5om5nPAfyHQAUWhcidG4EqNxJWZexTVn5/CVUkr9tbl8ha+UUuoYmvCVUmqemDMJX0TuE5FdIrJNRH4tIvlR2l0uIrtFZI+IrEtDXNeIyA4RmRCRqMusRGS/iPxFRP4sIluyKK50j9cCEXlKRBrCnwuitEvLeMV6/jLp2+Hj20RkeapiSTCui0SkNzw+fxaRL6QhpgdFpENEtkc5nqmxihVX2scq3G+liPxBRHaGX4ufitAmuWM2tV3dbP8A3gXYw4/vAe6J0MYG7AVqACewFTgtxXGdCiwGngFWTNNuPxBI43jFjCtD43UvsC78eF2kf8d0jVc8zx+4AngCEGAV8FIa/u3iiesi4LF0/T6F+7wAWA5sj3I87WMVZ1xpH6twv0FgefhxHvBGqn+/5swVvjHmSWPMWPjLF4GKCM1WAnuMMfuMMSPAz4ErUxzXTmPM7lT2cTLijCvt4xU+/4/Dj38MvC/F/U0nnud/JfCQmfQikC8iwSyIK+2MMc8Bh6ZpkomxiieujDDGtBljXg0/7gd2AuXHNUvqmM2ZhH+cjzH5v+LxyoHmY75u4cQBzhQDPCkir4jI9ZkOJiwT41VijGmDyRcEUBylXTrGK57nn4kxirfPc0Vkq4g8ISKnpzimeGTz6y+jYyUi1cCZwEvHHUrqmM2qTcxF5HdAaYRDdxhjHg23uQMYA34W6RQRvjfjdanxxBWH1caYVhEpBp4SkV3hK5NMxpX28UrgNEkfrwjief4pGaMY4unzVSZrqoRE5Arg/4D6FMcVSybGKh4ZHSsR8QK/BNYaY/qOPxzhR056zGZVwjfGvHO64yJyLfA3wCUmPAF2nBag8pivK4DWVMcV5zlaw587ROTXTP7ZPqMEloS40j5eItIuIkFjTFv4T9eOKOdI+nhFEM/zT8kYzTSuYxOHMeZxEfmuiASMMZksFJaJsYopk2MlIg4mk/3PjDG/itAkqWM2Z6Z0RORy4HbgvcaYwSjNNgP1IrJIRJzAB4AN6YoxGhHJFZG8qcdMvgEdcUVBmmVivDYA14YfXwuc8JdIGscrnue/AfhIeDXFKqB3akoqhWLGJSKlIiLhxyuZfK13pziuWDIxVjFlaqzCff4Q2GmM+XqUZskds3S/M52qD2APk3Ndfw5/fC/8/TLg8WPaXcHku+F7mZzaSHVcVzH5v/Qw0A5sPD4uJldbbA1/7MiWuDI0XoXA74GG8OcFmRyvSM8fuAG4IfxYgPvDx//CNCux0hzXTeGx2crkIobz0hDTw0AbMBr+3bouS8YqVlxpH6twv2uYnJ7ZdkzeuiKVY6alFZRSap6YM1M6SimlpqcJXyml5glN+EopNU9owldKqXlCE75SSs0TmvCVUmqe0ISvlFLzxP8DlndJjdwDlMMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pyplot.plot(x0, final_preds.mean(1), label='estimate')\n",
    "pyplot.plot(x0, torch.sin(2 * x0), label='true')\n",
    "pyplot.fill_between(x0, numpy.percentile(final_preds, 2.5, axis=1), numpy.percentile(final_preds, 97.5, axis=1),\n",
    "                   color='lightgray')\n",
    "pyplot.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Leaves, intermediates, inplace and ...\n",
    "\n",
    "There roughly are these types of tensors from an autograd perspective:\n",
    "\n",
    "- \"Plain\" tensors, not requiring gradients (in particular all integral tensors).\n",
    "- Leaves are the initial tensors requiring grad, from the factory functions or by taking a non-grad-requiring tensor and saying `t.requires_grad_()`.\n",
    "- Computed tensors where an input requires gradients will, in general, require gradient.\n",
    "- \"views\", tensors sharing memory, are special."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's look at this in some detail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import graphviz\n",
    "import warnings\n",
    "def ptgraph(tensors=None):\n",
    "    g = graphviz.Digraph()\n",
    "    with warnings.catch_warnings():\n",
    "        warnings.filterwarnings('ignore')\n",
    "        names = {v:k for k,v in globals().items() if isinstance(v, torch.Tensor) and not k.startswith('_')}\n",
    "        for n, m in globals().items():\n",
    "            if isinstance(m, torch.nn.Module) and not n.startswith('_'):            \n",
    "                names.update({p: n+':'+np for np, p in m.named_parameters()})\n",
    "        new_names = names\n",
    "        while new_names:\n",
    "            new_names = {k.grad: v+'.grad' for k, v in new_names.items() if k.grad is not None}\n",
    "            names.update(new_names)\n",
    "        known_gradfns = {k.grad_fn: k for k,v in names.items() if k.grad_fn is not None}\n",
    "    seen = set()\n",
    "    def traverse_node(node):\n",
    "        node = known_gradfns.get(node, node)\n",
    "        node_id = str(hash(node))\n",
    "        nodename = node.__class__.__name__\n",
    "        if isinstance(node, torch.Tensor) and node in names:\n",
    "            nodename = names[node]\n",
    "            if node.grad_fn is None:\n",
    "                g.node(node_id, nodename)\n",
    "                return node_id\n",
    "            node = node.grad_fn\n",
    "            nodename = f'{nodename} ({node.__class__.__name__})'\n",
    "        if hasattr(node, 'variable'):\n",
    "            v = node.variable\n",
    "            vn = names.get(v)\n",
    "            node_id = str(hash(v))\n",
    "            if vn is not None:\n",
    "                nodename = f'{vn} ({nodename})'\n",
    "        g.node(node_id, nodename)\n",
    "        if node in seen:\n",
    "            return node_id\n",
    "        seen.add(node)\n",
    "        for n ,p in node.next_functions:\n",
    "            if n is not None:\n",
    "                other_id = traverse_node(n)\n",
    "                g.edge(other_id, node_id)\n",
    "        return node_id\n",
    "    if tensors is None:\n",
    "        tensors = names.keys()\n",
    "    elif not isinstance(tensors, (list, tuple)):\n",
    "        tensors = [tensors]\n",
    "    for t in tensors:\n",
    "        other_id = traverse_node(t)\n",
    "    return g"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"416pt\" height=\"188pt\"\n",
       " viewBox=\"0.00 0.00 416.03 188.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 184)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-184 412.03,-184 412.03,4 -4,4\"/>\n",
       "<!-- 140075326172672 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>140075326172672</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"203.69\" cy=\"-18\" rx=\"94.78\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"203.69\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">c (MulBackward0)</text>\n",
       "</g>\n",
       "<!-- 8754707876702 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>8754707876702</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"102.69\" cy=\"-90\" rx=\"80.69\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"102.69\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">AddBackward0</text>\n",
       "</g>\n",
       "<!-- 8754707876702&#45;&gt;140075326172672 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>8754707876702&#45;&gt;140075326172672</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M126.12,-72.76C139.56,-63.45 156.64,-51.61 171.35,-41.41\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"173.47,-44.21 179.69,-35.63 169.48,-38.45 173.47,-44.21\"/>\n",
       "</g>\n",
       "<!-- 140075326014848 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>140075326014848</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"102.69\" cy=\"-162\" rx=\"102.88\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"102.69\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\">a (AccumulateGrad)</text>\n",
       "</g>\n",
       "<!-- 140075326014848&#45;&gt;8754707876702 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>140075326014848&#45;&gt;8754707876702</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M102.69,-143.7C102.69,-135.98 102.69,-126.71 102.69,-118.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"106.19,-118.1 102.69,-108.1 99.19,-118.1 106.19,-118.1\"/>\n",
       "</g>\n",
       "<!-- 140075327278656 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>140075327278656</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"304.69\" cy=\"-90\" rx=\"103.18\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"304.69\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">b (AccumulateGrad)</text>\n",
       "</g>\n",
       "<!-- 140075327278656&#45;&gt;140075326172672 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>140075327278656&#45;&gt;140075326172672</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M280.76,-72.41C267.3,-63.08 250.32,-51.32 235.73,-41.2\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"237.67,-38.29 227.45,-35.47 233.68,-44.04 237.67,-38.29\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7f65d40cb550>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = torch.randn(5, requires_grad=True)\n",
    "b = torch.randn(5, requires_grad=True)\n",
    "c = (a + 1) * b\n",
    "ptgraph(c)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "A common mistake is to accidentally throw away the true leaf. Note the difference between `a` and `b`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"413pt\" height=\"260pt\"\n",
       " viewBox=\"0.00 0.00 413.43 260.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 256)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-256 409.43,-256 409.43,4 -4,4\"/>\n",
       "<!-- 140075326161856 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>140075326161856</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"201.09\" cy=\"-18\" rx=\"94.78\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"201.09\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">c (MulBackward0)</text>\n",
       "</g>\n",
       "<!-- 8754707886204 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>8754707886204</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"100.09\" cy=\"-90\" rx=\"80.69\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"100.09\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">AddBackward0</text>\n",
       "</g>\n",
       "<!-- 8754707886204&#45;&gt;140075326161856 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>8754707886204&#45;&gt;140075326161856</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M123.52,-72.76C136.96,-63.45 154.04,-51.61 168.75,-41.41\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"170.87,-44.21 177.09,-35.63 166.88,-38.45 170.87,-44.21\"/>\n",
       "</g>\n",
       "<!-- 140075326184896 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>140075326184896</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"100.09\" cy=\"-162\" rx=\"100.18\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"100.09\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\">a (CopyBackwards)</text>\n",
       "</g>\n",
       "<!-- 140075326184896&#45;&gt;8754707886204 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>140075326184896&#45;&gt;8754707886204</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M100.09,-143.7C100.09,-135.98 100.09,-126.71 100.09,-118.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"103.59,-118.1 100.09,-108.1 96.59,-118.1 103.59,-118.1\"/>\n",
       "</g>\n",
       "<!-- 140075326162368 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>140075326162368</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"100.09\" cy=\"-234\" rx=\"87.99\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"100.09\" y=\"-230.3\" font-family=\"Times,serif\" font-size=\"14.00\">AccumulateGrad</text>\n",
       "</g>\n",
       "<!-- 140075326162368&#45;&gt;140075326184896 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>140075326162368&#45;&gt;140075326184896</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M100.09,-215.7C100.09,-207.98 100.09,-198.71 100.09,-190.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"103.59,-190.1 100.09,-180.1 96.59,-190.1 103.59,-190.1\"/>\n",
       "</g>\n",
       "<!-- 140075325981568 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>140075325981568</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"302.09\" cy=\"-90\" rx=\"103.18\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"302.09\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">b (AccumulateGrad)</text>\n",
       "</g>\n",
       "<!-- 140075325981568&#45;&gt;140075326161856 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>140075325981568&#45;&gt;140075326161856</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M278.16,-72.41C264.7,-63.08 247.72,-51.32 233.13,-41.2\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"235.07,-38.29 224.85,-35.47 231.08,-44.04 235.07,-38.29\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7f65d40f0d30>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = torch.randn(5, requires_grad=True).cuda()\n",
    "b = torch.randn(5, requires_grad=True, device='cuda')\n",
    "c = (a + 1) * b\n",
    "ptgraph(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This would be a repaired version (but, if you can, working directly on the GPU is more efficient)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"416pt\" height=\"188pt\"\n",
       " viewBox=\"0.00 0.00 416.03 188.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 184)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-184 412.03,-184 412.03,4 -4,4\"/>\n",
       "<!-- 140075326036480 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>140075326036480</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"203.69\" cy=\"-18\" rx=\"94.78\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"203.69\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">c (MulBackward0)</text>\n",
       "</g>\n",
       "<!-- 8754707866718 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>8754707866718</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"102.69\" cy=\"-90\" rx=\"80.69\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"102.69\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">AddBackward0</text>\n",
       "</g>\n",
       "<!-- 8754707866718&#45;&gt;140075326036480 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>8754707866718&#45;&gt;140075326036480</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M126.12,-72.76C139.56,-63.45 156.64,-51.61 171.35,-41.41\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"173.47,-44.21 179.69,-35.63 169.48,-38.45 173.47,-44.21\"/>\n",
       "</g>\n",
       "<!-- 140075326162752 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>140075326162752</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"102.69\" cy=\"-162\" rx=\"102.88\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"102.69\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\">a (AccumulateGrad)</text>\n",
       "</g>\n",
       "<!-- 140075326162752&#45;&gt;8754707866718 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>140075326162752&#45;&gt;8754707866718</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M102.69,-143.7C102.69,-135.98 102.69,-126.71 102.69,-118.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"106.19,-118.1 102.69,-108.1 99.19,-118.1 106.19,-118.1\"/>\n",
       "</g>\n",
       "<!-- 140075326094848 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>140075326094848</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"304.69\" cy=\"-90\" rx=\"103.18\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"304.69\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">b (AccumulateGrad)</text>\n",
       "</g>\n",
       "<!-- 140075326094848&#45;&gt;140075326036480 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>140075326094848&#45;&gt;140075326036480</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M280.76,-72.41C267.3,-63.08 250.32,-51.32 235.73,-41.2\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"237.67,-38.29 227.45,-35.47 233.68,-44.04 237.67,-38.29\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7f65d40a4610>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = torch.randn(5).cuda().requires_grad_()\n",
    "b = torch.randn(5, requires_grad=True, device='cuda')\n",
    "c = (a + 1) * b\n",
    "ptgraph(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You can cut things from autograd with `detach()` (and `detach_()`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"215pt\" height=\"116pt\"\n",
       " viewBox=\"0.00 0.00 214.68 116.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 112)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-112 210.68,-112 210.68,4 -4,4\"/>\n",
       "<!-- 140075326207744 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>140075326207744</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"103.34\" cy=\"-18\" rx=\"94.78\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"103.34\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">c (MulBackward0)</text>\n",
       "</g>\n",
       "<!-- 140075326115072 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>140075326115072</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"103.34\" cy=\"-90\" rx=\"103.18\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"103.34\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">b (AccumulateGrad)</text>\n",
       "</g>\n",
       "<!-- 140075326115072&#45;&gt;140075326207744 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>140075326115072&#45;&gt;140075326207744</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M103.34,-71.7C103.34,-63.98 103.34,-54.71 103.34,-46.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"106.84,-46.1 103.34,-36.1 99.84,-46.1 106.84,-46.1\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7f65d40f0100>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = torch.randn(5, requires_grad=True).cuda()\n",
    "b = torch.randn(5, requires_grad=True, device='cuda')\n",
    "c = (a + 1).detach() * b\n",
    "ptgraph(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Inplace with autograd can be tricky"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([1., 1., 1., 1., 1.]),)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = torch.randn(5, requires_grad=True)\n",
    "b1 = a + 1\n",
    "b2 = b1 + 1\n",
    "\n",
    "b1 += 1\n",
    "\n",
    "torch.autograd.grad(b2.sum(), a) # works\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [5]], which is output 0 of AddBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-35-950f59f473fd>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mb1\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0ma\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# this would need the original b1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/torch/autograd/__init__.py\u001b[0m in \u001b[0;36mgrad\u001b[0;34m(outputs, inputs, grad_outputs, retain_graph, create_graph, only_inputs, allow_unused)\u001b[0m\n\u001b[1;32m    188\u001b[0m         \u001b[0mretain_graph\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    189\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 190\u001b[0;31m     return Variable._execution_engine.run_backward(\n\u001b[0m\u001b[1;32m    191\u001b[0m         \u001b[0moutputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad_outputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    192\u001b[0m         inputs, allow_unused)\n",
      "\u001b[0;31mRuntimeError\u001b[0m: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [5]], which is output 0 of AddBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True)."
     ]
    }
   ],
   "source": [
    "a = torch.randn(5, requires_grad=True)\n",
    "b1 = a + 1\n",
    "b2 = b1 ** 2\n",
    "b1 += 1\n",
    "\n",
    "torch.autograd.grad(b2.sum(), a) # this would need the original b1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [
     "raises-exception"
    ]
   },
   "source": [
    "We did not get a TraceBack in Jupyter before PyTorch 1.7 (PR #41116), but now we get too much (the crucial `b2 = b1 ** 2` is there, but off-screen)..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    },
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-36-fb1eaba42148>:1: UserWarning: Anomaly Detection has been enabled. This mode will increase the runtime and should only be enabled for debugging.\n",
      "  with torch.autograd.detect_anomaly():\n",
      "/usr/local/lib/python3.8/dist-packages/torch/autograd/__init__.py:190: UserWarning: Error detected in PowBackward0. Traceback of forward call that caused the error:\n",
      "  File \"/usr/lib/python3.8/runpy.py\", line 194, in _run_module_as_main\n",
      "    return _run_code(code, main_globals, None,\n",
      "  File \"/usr/lib/python3.8/runpy.py\", line 87, in _run_code\n",
      "    exec(code, run_globals)\n",
      "  File \"/usr/lib/python3/dist-packages/ipykernel_launcher.py\", line 16, in <module>\n",
      "    app.launch_new_instance()\n",
      "  File \"/usr/lib/python3/dist-packages/traitlets/config/application.py\", line 664, in launch_instance\n",
      "    app.start()\n",
      "  File \"/usr/lib/python3/dist-packages/ipykernel/kernelapp.py\", line 597, in start\n",
      "    self.io_loop.start()\n",
      "  File \"/usr/lib/python3/dist-packages/tornado/platform/asyncio.py\", line 149, in start\n",
      "    self.asyncio_loop.run_forever()\n",
      "  File \"/usr/lib/python3.8/asyncio/base_events.py\", line 570, in run_forever\n",
      "    self._run_once()\n",
      "  File \"/usr/lib/python3.8/asyncio/base_events.py\", line 1859, in _run_once\n",
      "    handle._run()\n",
      "  File \"/usr/lib/python3.8/asyncio/events.py\", line 81, in _run\n",
      "    self._context.run(self._callback, *self._args)\n",
      "  File \"/usr/lib/python3/dist-packages/tornado/ioloop.py\", line 690, in <lambda>\n",
      "    lambda f: self._run_callback(functools.partial(callback, future))\n",
      "  File \"/usr/lib/python3/dist-packages/tornado/ioloop.py\", line 743, in _run_callback\n",
      "    ret = callback()\n",
      "  File \"/usr/lib/python3/dist-packages/tornado/gen.py\", line 787, in inner\n",
      "    self.run()\n",
      "  File \"/usr/lib/python3/dist-packages/tornado/gen.py\", line 748, in run\n",
      "    yielded = self.gen.send(value)\n",
      "  File \"/usr/lib/python3/dist-packages/ipykernel/kernelbase.py\", line 381, in dispatch_queue\n",
      "    yield self.process_one()\n",
      "  File \"/usr/lib/python3/dist-packages/tornado/gen.py\", line 225, in wrapper\n",
      "    runner = Runner(result, future, yielded)\n",
      "  File \"/usr/lib/python3/dist-packages/tornado/gen.py\", line 714, in __init__\n",
      "    self.run()\n",
      "  File \"/usr/lib/python3/dist-packages/tornado/gen.py\", line 748, in run\n",
      "    yielded = self.gen.send(value)\n",
      "  File \"/usr/lib/python3/dist-packages/ipykernel/kernelbase.py\", line 365, in process_one\n",
      "    yield gen.maybe_future(dispatch(*args))\n",
      "  File \"/usr/lib/python3/dist-packages/tornado/gen.py\", line 209, in wrapper\n",
      "    yielded = next(result)\n",
      "  File \"/usr/lib/python3/dist-packages/ipykernel/kernelbase.py\", line 268, in dispatch_shell\n",
      "    yield gen.maybe_future(handler(stream, idents, msg))\n",
      "  File \"/usr/lib/python3/dist-packages/tornado/gen.py\", line 209, in wrapper\n",
      "    yielded = next(result)\n",
      "  File \"/usr/lib/python3/dist-packages/ipykernel/kernelbase.py\", line 543, in execute_request\n",
      "    self.do_execute(\n",
      "  File \"/usr/lib/python3/dist-packages/tornado/gen.py\", line 209, in wrapper\n",
      "    yielded = next(result)\n",
      "  File \"/usr/lib/python3/dist-packages/ipykernel/ipkernel.py\", line 300, in do_execute\n",
      "    res = shell.run_cell(code, store_history=store_history, silent=silent)\n",
      "  File \"/usr/lib/python3/dist-packages/ipykernel/zmqshell.py\", line 536, in run_cell\n",
      "    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)\n",
      "  File \"/usr/lib/python3/dist-packages/IPython/core/interactiveshell.py\", line 2857, in run_cell\n",
      "    result = self._run_cell(\n",
      "  File \"/usr/lib/python3/dist-packages/IPython/core/interactiveshell.py\", line 2886, in _run_cell\n",
      "    return runner(coro)\n",
      "  File \"/usr/lib/python3/dist-packages/IPython/core/async_helpers.py\", line 68, in _pseudo_sync_runner\n",
      "    coro.send(None)\n",
      "  File \"/usr/lib/python3/dist-packages/IPython/core/interactiveshell.py\", line 3062, in run_cell_async\n",
      "    has_raised = await self.run_ast_nodes(code_ast.body, cell_name,\n",
      "  File \"/usr/lib/python3/dist-packages/IPython/core/interactiveshell.py\", line 3254, in run_ast_nodes\n",
      "    if (await self.run_code(code, result,  async_=asy)):\n",
      "  File \"/usr/lib/python3/dist-packages/IPython/core/interactiveshell.py\", line 3331, in run_code\n",
      "    exec(code_obj, self.user_global_ns, self.user_ns)\n",
      "  File \"<ipython-input-36-fb1eaba42148>\", line 4, in <module>\n",
      "    b2 = b1 ** 2\n",
      " (Triggered internally at  ../torch/csrc/autograd/python_anomaly_mode.cpp:58.)\n",
      "  return Variable._execution_engine.run_backward(\n"
     ]
    },
    {
     "ename": "RuntimeError",
     "evalue": "one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [5]], which is output 0 of AddBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-36-fb1eaba42148>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      5\u001b[0m     \u001b[0mb1\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m     \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0ma\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# this would need the original b1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/torch/autograd/__init__.py\u001b[0m in \u001b[0;36mgrad\u001b[0;34m(outputs, inputs, grad_outputs, retain_graph, create_graph, only_inputs, allow_unused)\u001b[0m\n\u001b[1;32m    188\u001b[0m         \u001b[0mretain_graph\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    189\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 190\u001b[0;31m     return Variable._execution_engine.run_backward(\n\u001b[0m\u001b[1;32m    191\u001b[0m         \u001b[0moutputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad_outputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    192\u001b[0m         inputs, allow_unused)\n",
      "\u001b[0;31mRuntimeError\u001b[0m: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [5]], which is output 0 of AddBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!"
     ]
    }
   ],
   "source": [
    "with torch.autograd.detect_anomaly():\n",
    "    a = torch.randn(5, requires_grad=True)\n",
    "    b1 = a + 1\n",
    "    b2 = b1 ** 2\n",
    "    b1 += 1\n",
    "\n",
    "    torch.autograd.grad(b2.sum(), a) # this would need the original b1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There are some opportunities where inplace is actually very useful. One of my faviourite tricks is to copy things into a preallocated tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000],\n",
       "        [ 0.0000,  0.0539,  0.2435, -0.3037, -0.2411, -0.0319,  0.0000],\n",
       "        [ 0.0000,  0.7659,  1.2740,  0.2249, -1.1687, -1.7693,  0.0000],\n",
       "        [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000]],\n",
       "       grad_fn=<CopySlices>)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = torch.randn(5, requires_grad=True)\n",
    "b = torch.randn(5, requires_grad=True)\n",
    "c = torch.zeros(4, 7)  # padded torch.stack([a,b])\n",
    "c[1, 1:-1] = a\n",
    "c[2, 1:-1] = b\n",
    "c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "gr_a, gr_b = torch.autograd.grad(c.sum(), [a, b])\n",
    "assert (gr_a -1).abs().max().item() < 1e-6 and (gr_b -1).abs().max().item() < 1e-6  # float and \"==\" don't mix well"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Inplace Summary\n",
    "\n",
    "\n",
    "There are two main situations where inplace operations cannot be used:\n",
    "\n",
    "- When operating on leaf tensors, the leaf tensor would be moved into the graph, which would be bad.\n",
    "- When the operation before the inplace wants to have its result to compute the backward. Whether this is the case is not easy to tell from the outside, unfortunately.\n",
    "\n",
    "As a corollary, you should avoid using inplace on the inputs of your re-usable module lest in the future use could be in one of the two situations.\n",
    "\n",
    "Later: You might also be less eager to use inplace when planning to use the JIT, as it will fuse pointwise non-inplace operations like ReLU if there are several in a row.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For validation, inference or other things where the *end user* decides she does not want gradients, there is the `torch.no_grad` context manager.\n",
    "\n",
    "We'll use it below to evaluate models after training.\n",
    "\n",
    "- If you need to use `torch.no_grad()` somewhere where it isn’t because you’re evaluating something that’s written for training, you should ask yourself if you’re doing it wrong.\n",
    "\n",
    "- Using item and detach for things to keep around longer than the next backward is generally a good idea (e.g. when you record loss history, statistics, …), but be careful to not ruin your graph. (Targeted detach is good in nn.Module subclass code, with torch.no_grad() should be needed very rarely.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "It is not true that tensors created in `torch.no_grad` never require grad (as often views are special):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[ 4.5442,  2.7897, -2.8073,  1.9178,  2.6865],\n",
       "         [-0.4001, -1.4518,  2.4382,  4.3330,  2.4355],\n",
       "         [-1.9214, -0.5341,  2.4665, -3.8517,  3.4939],\n",
       "         [ 0.3584,  1.1731,  1.1691, -3.1120, -0.5954],\n",
       "         [ 0.7143, -0.2832, -1.8675, -3.3314, -0.5892]]),\n",
       " tensor([[ 2.2721,  1.3948],\n",
       "         [-0.2001, -0.7259]], requires_grad=True))"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = torch.randn(5, 5, requires_grad=True)\n",
    "\n",
    "with torch.no_grad():\n",
    "    b = a * 2\n",
    "    c = a[:2, :2]\n",
    "\n",
    "b, c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Second derivatives\n",
    "\n",
    "You cannot get Hessians, but you can get Hessian vector products or really derivatives of any scalar function of the gradient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([0., 1., 2., 3., 4.], requires_grad=True),\n",
       " tensor([  0.,  12.,  48., 108., 192.]))"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = torch.arange(5., requires_grad=True)\n",
    "b = a ** 4\n",
    "gr, = torch.autograd.grad(b.sum(), [a], create_graph=True)\n",
    "hvp, = torch.autograd.grad(gr.sum(), [a])  # what is the vector in HVP?\n",
    "a, hvp # 4 * 3 * x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "\n",
    "## `backward` as a convenience method\n",
    "\n",
    "Above we used `torch.autograd.grad`. More commonly, you find `loss.backward()`. This is a convenience method for computing the gradient of all leaves `t` and storing the gradient in `t.grad`.\n",
    "In particular, PyTorch's built-in optimizers use these.\n",
    "\n",
    "You can ask for a gradient of an intermediate to be stored by `t.retain_grad()`.\n",
    "\n",
    "My advice: Only use backward to get gradients to then use with the optimizer. Don't use backward with \n",
    "`create_graph`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## A fun trick\n",
    "\n",
    "Sometimes, we want the backward to not quite be like the forward. Round obviously has gradient 0, but wouldn't it be sometimes be convenient if..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0., 0., 0., 0., 0.])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = torch.randn(5, requires_grad=True)\n",
    "b = a.round()\n",
    "gr, = torch.autograd.grad(b.sum(), [a])\n",
    "gr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "...we can actually do that (not this construction, but the same rounding trick is the gist of \"quantization aware training\", other things like \"gumble softmax\" also work like this):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, tensor([1., 1., 1., 1., 1.]))"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def differentiable_round(a, ndigits=None):\n",
    "    if ndigits is None:\n",
    "        b = a.round()\n",
    "    else:\n",
    "        scale = 10 ** ndigits\n",
    "        b = (a * scale).round() / scale\n",
    "    return a + (b - a).detach()  # PyTorch's version of \"cleverly add 0\"\n",
    "c = differentiable_round(a)\n",
    "gr2, = torch.autograd.grad(c.sum(), [a])\n",
    "(c == b).all().item(), gr2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Fake quantization training the hard way"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "w1 = torch.randn(1, 10, requires_grad=True)\n",
    "b1 = torch.randn(10, requires_grad=True)\n",
    "w2 = torch.randn(10, 1, requires_grad=True)\n",
    "b2 = torch.randn(1, requires_grad=True)\n",
    "params = [w1, b1, w2, b2]\n",
    "\n",
    "\n",
    "for i in range(200):\n",
    "    x = torch.randn(10, 1)\n",
    "    target = torch.sin(2 * x)\n",
    "\n",
    "    xr = differentiable_round(x)\n",
    "    w1r = differentiable_round(w1, 0.5)\n",
    "    b1r = differentiable_round(b1, 0.5)\n",
    "    w2r = differentiable_round(w2, 0.5)\n",
    "    b2r = differentiable_round(b2, 0.5)\n",
    "    pred = torch.tanh(xr @ w1r + b1r) @ w2r + b2r\n",
    "\n",
    "    loss = ((pred - target) ** 2).sum()\n",
    "\n",
    "    grads = torch.autograd.grad(loss, params)\n",
    "\n",
    "    for p, g in zip(params, grads):\n",
    "        with torch.no_grad():\n",
    "            p -= 1e-2 * g"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Did it work?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f65d403e0d0>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd1zV1f/A8ddhDxHEgQsVB+JAAQG3WaapmZYjRzmyNNt7fn+Nb3sP7Zs50sqGmebWTK0cOVHBAe4FgggqSzac3x8fLMe9gnL5fO44z8eDB3A/h3vefOC+P+eezxlCSomiKIpi/5yMDkBRFEXRh0r4iqIoDkIlfEVRFAehEr6iKIqDUAlfURTFQbgYHcC11KpVSzZp0sToMBRFUWzGjh070qWUtU0ds+qE36RJE2JiYowOQ1EUxWYIIU6YO6a6dBRFURyESviKoigOQiV8RVEUB2HVffimFBUVkZSURH5+vtGhODQPDw8aNmyIq6ur0aEoilJBNpfwk5KS8PHxoUmTJgghjA7HIUkpOXv2LElJSQQFBRkdjqIoFWRzXTr5+fnUrFlTJXsDCSGoWbOmepelKDbG5hI+oJK9FVB/A0WxPTbXpaMoivUrKZUcS89hd1ImJ87m4iQELs4CLzdnQhv40raBLx6uzkaH6XBUwr8BkydPZurUqZw+fZoXXniBF198kUWLFhEcHEzr1q2NDk9RDBObmMHsv4+xJj6VC4UlZsu5OAnaNvBlSEQD7gxvgI+HuvmvB5Xwb8CXX37JypUrL7thuWjRIgYMGKASvuKQ1h9M45PVB4lNzKCauwsDw+rTobE/7Rr60rSWN0IIiktLycgtIi4xg12JGaw7kMYri/fx7sr9DAprwCM3N6NhDS+jfxW7Jqx5x6vIyEh55dIKCQkJtGrVyqCIYNKkScyaNYuWLVsyfvx4jhw5wqhRoxgwYAC+vr74+vqyYMECmjVrZliMejH6b6EYLyu/iLeWxTMvJolG/l6M79qEoZGBVHMvvy0ppWR3UiY/bD3B4thkAB68qRmTbmqKl5tqi94oIcQOKWWkqWM2fVb/u3Qf8clZFn3O1vWr89odbcwe/+qrr/jtt9/4888/WbZsGQBdunRh4MCBDBgwgKFDh1o0HkWxVpuPnOXpebGkZuXzUM9mPNGrxXX1ywshaB/oR/tAP564NZj3Vu5n8tpD/BKTyHtD2nFTsMn1v5RKsMlROoqiGGtJXDJjZm3F082ZBQ914YW+IZW6CdvAz5MpI8P5ZVJnfDxcGDtrG++uSKCwuNSCUSs23cK/VktcUZSqMWvjMd5YFk90kD8zxkTi62m5G65RTfxZ8mg33lwWz7T1R9ly9Cz/uydC9e1biGrhW4iPjw/Z2dlGh6EoVerT1Qd5Y1k8t7UJ4Lvx0RZN9hd5uDrz9l2hTL0ngqPpF7jzf5vYnZRh8XockUr4FjJixAg+/PBDwsPDOXLkiNHhKIrFfb/lBJ+vPcSwDg358p4OVT6Ovl9oPX59qAvuLk4Mn7aFtQmpVVqfI1CjdJQbpv4WjmNNfCoT58Rwc8s6TBvdARdn/dqKZ7LzeeDbGPaeyuSdu0IZEd1It7pt0bVG6agWvqIo1xSXmMFjP+2iTX1fpowK1zXZA9Tx8WDuxE70CK7Ni7/u4YetZjd0UsqhEr6iKGal5xQw4bsYalZz4+txkYaNj/dyc2Ha6A7cElKH/yzcy5wtKunfCJXwFUUxqbRU8uwvcWTkFTF9dCR1fDwMjcfdxZmp90Zwa6s6vLJoL9+rpH/dLJLwhRCzhBBnhBB7zRwXQojJQojDQojdQogIS9SrKErVmbnxKH8dSOOV21vRun51o8MBtKT/5T0d6BVSh1cW72X57hSjQ7IplmrhfwP0vcbxfkCLso+JwFQL1asoShWITczgg98O0LdNXe7t1NjocC7j5uLEF6Mi6NCoBk/9HMumw+lGh2QzLJLwpZTrgXPXKDII+E5qtgB+Qoh6lqhbURTLyi0s5vGfdlHPx5UP+tZDnEmA9MNQmGt0aP/wdHPm67FRBNXyZsJ32ggepXx69eE3ABIv+T6p7DHlBrz++ut89NFHVz2+aNEi4uPjr/v5jh8/zo8//vjP99988w2PPvpopWJUbFB+JsQvYf/08XyfM4H1BcOo/r9WMLUzfNEB3qkH7zWG2bfD35Mh/ZCh4fp6ufLt+Gj8vNwY/812UjLzDI3HFuiV8E1tj2RyAoAQYqIQIkYIEZOWllbFYVWd4uJi3eu8VsK/VjxXJnzFwWSchBXPw0ctYd5oWqT9TrZfCOKm56HfhzB0Ftw1DXq9Cm2HQH4GrH4FvoiEr7rBvkVQasyaN3V9PZg1LorcwhIe+DaG3EL9X3e2RK+EnwQEXvJ9QyDZVEEp5XQpZaSUMrJ2betcLe/NN98kJCSE3r17M3LkyH9a2z179uTll1/mpptu4vPPP2ft2rWEh4cTGhrK+PHjKSgoAKBJkyakp2v9jjExMfTs2RPQWu7jx4+nZ8+eNG3alMmTJ/9T59tvv03Lli259dZbOXDgwFUxbdq0iSVLlvDcc88RFhbGkSNHropn3LhxzJ8//5+fqVatGgAvvvgiGzZsICwsjE8//RSA5ORk+vbtS4sWLXj++ectfxIV4+WkwaKH4fMwiPmakjaDeabae9zm9g2BDy+Em1+GjhO1JN9+BHR/BgZ8Ag/9DU/u0S4GRfnwy1iY2gXil4ABEzlb1vVhyshwElKyeHJuLKWl1juZ1Gh6DapdAjwqhJgLdAQypZSVv72+8kU4vafST3OZuqHQ7z2zh2NiYliwYAG7du2iuLiYiIgIOnTo8M/xjIwM1q1bR35+Pi1atGDt2rUEBwczZswYpk6dypNPPnnN6vfv38+ff/5JdnY2LVu25KGHHmL37t3MnTvXbJ1gfonmi/EAjBs3zmSd7733Hh999NE/yz1/8803xMbGsmvXLtzd3WnZsiWPPfYYgYGBJn9esTFSwp75sPJ5KMyB6InQ5VG+jMljQfpBZowJp3p5O1D5NdIuBlH3w76FsO4DmDcaWt8Jt38C3jX1+V3K3BxSh5f7t+Kt5Ql89PsBnu8bomv9tsJSwzJ/AjYDLYUQSUKI+4UQk4QQk8qKrACOAoeBGcDDlqjXCBs3bmTQoEF4enri4+PDHXfccdnx4cOHA3DgwAGCgoIIDg4GYOzYsaxfv77c57/99ttxd3enVq1a1KlTh9TUVDZs2MBdd92Fl5cX1atXZ+DAgRWO92I816tXr174+vri4eFB69atOXFCjXm2C3nnYe498OsD4N8UHtwA/d7jRHENpvx5mNtD69G7dUDFn8/JGUKHwkOboNdrsH85fNkR9q+out/BjPu7BTEiKpAv/zrCb3vVcE1TLNLCl1KOLOe4BB6xRF2XuUZLvKqUt/aQt7d3ueVcXFwoLevzzM/Pv+yYu7v7P187Ozv/0/cuhKnbIOW7GM+V9UopKSwsNPtz5uJQbNjZI/Dj3XD+BPR5Czo9rCVs4K3lCbg4CV694wa36HR2ge5PQ/BtsPBBmDsSbvk/6P4s3OD/7vUSQvDfQW1ISMnimXlxNK/jQ/M61XSp21aombbXqVu3bixdupT8/HxycnJYvny5yXIhISEcP36cw4cPAzBnzhxuuukmQOvD37FjBwALFiwot84ePXqwcOFC8vLyyM7OZunSpSbLlbdE86X1Ll68mKKiogr9nGIHjm2AGbdA7jkYuwS6PPZPsl9/MI3V8ak8ektzAqpXcjZtQBt4YC2EDoM/3oIlj0JJkQV+gYrRZuNqK3k+OCeGnALVULmUSvjXKSoqioEDB9K+fXsGDx5MZGQkvr6+V5Xz8PBg9uzZDBs2jNDQUJycnJg0Sevheu2113jiiSfo3r07zs7lLzEbERHB8OHDCQsLY8iQIXTv3t1kufKWaJ4wYQLr1q0jOjqarVu3/tP6b9euHS4uLrRv3/6fm7aKHUlYBnPuhGoBMOEPaNzln0NFJaW8sSyexjW9uL9bkGXqc3GHwTOgx/Ow63v4YSgUXrDMc1dAfT9PpowK51j6BV6Yv7vcd+UORUpptR8dOnSQV4qPj7/qMb1lZ2dLKaW8cOGC7NChg9yxY4fBERnDGv4WSjn2r5TyvzWlnNFLytzzVx2eueGobPzCMrl63+mqqX/nHClf95Py20FSFuZVTR1mTP3rsGz8wjL53aZjutZrNCBGmsmpqoV/AyZOnEhYWBgREREMGTKEiAi1NJBihQ6t1kbO1G0L9y4AT7/LDp+7UMhnaw7SI7g2vVrVqZoYwu+FgV/A0T/hl3G6du9M7N6Uni1r8+ayBDUTt4xN72lrFDVJSbF6JzZpo3Fqh8DoheBxdbfjlD8OcaGgmFdub3XDgwIqJPweKM6D5c/Agge0iVxOVbtbFoCTk+CTu8Po//kGHvlxJ8se64ZPecNN7ZxNtvCl6pMznPobWLFzR7Vk7xcIYxaDZ42ripw8m8v3W04wPCqQFgE+VR9T1APayKD4RbD2v1VfXxl/bzcmjwwn6XweL/26x+H/b20u4Xt4eHD27FmH/8MZSUrJ2bNn8fAwdn10xYT8TPhxBMhSGDUPvPxNFvvw9wM4OwmevDVYv9i6PAaR98Pfn0PsT7pVGx3kz1O3tmDZ7hQW7DylW73WyOa6dBo2bEhSUhK2vM6OPfDw8KBhw4ZGh6FcqqQYfrkPzh2B0YugZjOTxeISM1gal8xjlhiGeb36vQ/pB2Hp41CzOQRG6VLtQz2bs+FQOq8t3ktk4xo0qeVd/g/ZIZvbxFxRFDPW/Bc2fgJ3TIYOY00WkVIycsYWDqbmsO65nsb0aeeegxk3a8stP7gequuzUnpyRh59P1tPUO1qzJ/UGVed9+bVi9rEXFHs3eG1sPFTiBhjNtkDrD+Uzpaj53j8lubG3cD08oeRc6EgGxZOhNISXaqt7+fJu4PbEZeYwedrjF3a2Sgq4SuKrctO1ZYzqB0Cfd83W6y0VPLhqv00rOHJqI4G72JVp5XWvXNsvdanr5Pb29VjWIeGfPnXYWKOX2vPJvukEr6i2LLSEvh1AhTkwLDZ4OZltuhv+06z91QWT90ajJuLFbz0I8ZAm7u0JRgSt+tW7WsD29CghidPz4vjgoMtvWAFf3VFUW7Y5i/g2DqttVynldlixSWlfPz7AZrXqcad4Vay2ZwQMOAzqN4AFozXRhjpoJq7Cx8PCyPxfC5vLU/QpU5roRK+otiqtIPwx9sQMkBrLV/Dwl2nOJJ2gWf7BOPspM/qlRXi6QdDv4bMJFj9qm7VRgf5M7F7U37adpI/9qfqVq/RVMJXFFtUWgKLHwFXT23DkWvMlC0oLuGzNYcIbeDLbW3q6hhkBQVGQ+dHYMc3cHSdbtU+3SeYkLo+PD9/D+cvmF8q3J7YZ8LPO6/r6nyKorstUyFpG/T7AHyuvWHJz9sTOZWRx7O3tazaJRQq4+b/gH8zWPKYbq9ddxdnPrk7jIzcQl5bsk+XOo1mfwk/7zx8EQXrzI9WUBSbdvYI/PEmBPeDdndfs2heYQlT/jhMdJA/PVrU0inAG+DqCQOnQMYJWPumbtW2rl+dx3u1YElcMiv32P8uWfaX8D1raLvubP4fnHGsGzKKA5ASlj0Fzu4w4NNyd5Oas+U4adkFPNvHilv3FzXpClETYOtXkLhNt2of6tmM0Aa+/GfRXtJzCnSr1wj2l/ABbn0D3H201fmseCaxoly3fb9qo3J6vVLuDNXs/CKm/nWEHsG1iQ4yvaaO1bn1NfCpp712dZqQ5ersxEfD2pOTX8wri/ba9Tpd9pnwvWvCra/Dib8hbq7R0SiKZRRkw6r/QN12EDm+3OKz/z7O+dwinumt4wJpleXuA7e9Bad3Q8ws3aptWdeHJ3u3YOXe0yy3464d+0z4AOFjoGEU/P5/Wr++oti6v96D7BRtVE4568ln5BYyY/1R+rQOoH2g3zXLWp02gyGoh3af4kK6btVO7N6Udg19eW3xPs7aadeO/SZ8JyfthZF3TpvJpyi2LDVeG5kTMaZCK0xOW3+UnMJinu5jQ637i4SA/h9po3XWvK5btS7OTnw4tD1Z+UW8vjRet3r1ZL8JH6BeO2397ZjZcGa/0dEoyo2REn57Uevu6PV6ucXPZOcz++9jDGxfn5C61as+vqpQuyV0egh2zYEk/VbMbVnXh8duacHSuGRW7TutW716se+ED9DzJXCrpnXtKIotOvS7dqO250va/alyfPHHYYpLJE/publJVbjpBagWAKte1nXwxUM9m9GqXnX+b9FeMnLta0KW/Sd875pw03NweDUcXmN0NIpyfUqK4fdXtElJFbhRm3gul5+2neTuqEDb3+TD3UebkJW4FRKW6Fatq7MTHw5tx7kLhbxtZ2vt2H/CB4ieCDWCYNX/aS8gRbEVO7+B9APQ+w1wcSu3+GdrDiGE4PFbWlR9bHoIvxdqt9L68ov1a223beDLgz2a8suOJNYftJ/d9Rwj4bu4ay+YtATY+a3R0ShKxeRnwZ/vQuOuEHJ7ucUPpWazcFcSYzs3pq6vnew37OQMfd7UNmbXcZgmwOO9WtC0tjcv/brHbpZRdoyED9DqDmjURRvaptbZUWzBxk8hNx36vFXujFqA9387gLebCw/1bK5DcDpqfis07Qnr3oO8DN2q9XB15oMh7UjOzOPDVQd0q7cqOU7CF0KbxXfhDGydZnQ0inJt2anaMMzQYdAgotzi246dY01CKpN6NsPfu/yuH5siBPR+U0v2Gz/VterIJv6M6dSYbzcfZ8cJ298hy3ESPkCjTtDiNvj7MzUZS7FuGz6CkkJtZE45pJS8uzKBgOrujO8apENwBqjXDkKHao21bH3Xr3++bwj1fT15YcEeCor1We6hqjhWwgdtDZL8TNg0xehIFMW08ye0uSMRo6Fms3KL/7b3NLtOZvB072A83a49A9em9XxJuwhu+FjXar3dXXjrrrYcPpPDl38e0bVuS3O8hF83FNoO0d4u69xSUJQKWfcBCCfo8Xy5RYtKSvlg1QGCA6oxJKKhDsEZqGYzCL8HdsyGjERdq765ZR3uDKvPl38d5mBqtq51W5LjJXzQxvYWF2hvmxXFmqQdhLgfIXoC+Ja/9+yPW09yLP0CL/QNwcXZAV7OFy+CBux38eodbfDxcOX5+bspKbXNFTUd4D/EhJrNtPG9O76BzFNGR6Mo//rrHXD1gm5PlVv0/IVCPll9kC7NanJLSB0dgrMCfoHQ4T6I/VHbCEZH/t5uvDqgNbGJGXy3+biudVuKYyZ8gO7PgCzVbuAqijU4kwD7FkHHB8G7/N2pPl1zkOz8Il67o431b25iSd2f0ebW/PWe7lUPCqvPTcG1+XDVAU5l5Olef2U5bsKv0RjCRmmt/Kxko6NRFFj/oda67/xouUX3n87i+y0nuLdTY1rW9dEhOCviE6AtM7F3vu6tfCEEb9/VFoD/W7jH5jZLcdyED/+28jeqVr5isLQDsPdXre/e69q7U0kpeWNpPD4erra/QNqN6vI4OLvBhk90r7phDS+e7dOSPw+ksSTOthqLFkn4Qoi+QogDQojDQogXTRzvKYTIFELEln28aol6K61GE2g/sqyVb7+73Cg2YP1H2kbeXR4rt+iqfafZdOQsz/QJpoa9TbKqKJ8A6DAO4n6C88d1r35slya0D/Tjv0vjOXfBdlbUrHTCF0I4A/8D+gGtgZFCiNYmim6QUoaVfbxR2XotpvszIEtUX75inPTDWvdE1APl9t1n5xfx+pJ4Qur6MCq6kU4BWqmuT2hr7RjQynd2Erw3OJSsvCLeWm47m6VYooUfDRyWUh6VUhYCc4FBFnheffgHQfsRWis/54zR0SiOaMNH4Oxeodb9x78fJDU7n3cHhzrGMMxrqV5f2wEs9kfIOKl79a3qVWfSTc34decpNhyyjRU1LfEf0wC4dBZEUtljV+oshIgTQqwUQrQx92RCiIlCiBghRExamk4nsetT2rj8LVP1qU9RLjp/AnbPg8j7oNq1h1bGJmbw7ebjjO7UmPBGNfSJz9p1fVL7/PfnhlT/6C3NaVrLm5cX7iG30PpX1LREwjc1HuzKW9c7gcZSyvbAFGCRuSeTUk6XUkZKKSNr165tgfAqoFZzaD0Its/Ull1QFL1smqLNqi1nZE5RSSkv/bqHOj7uPHdbS52CswF+gdo79F3fG/IO3cPVmXcGh5J4Lo/P1hzSvf7rZYmEnwQEXvJ9Q+CyW9dSyiwpZU7Z1ysAVyFE+QON9dT9aSjI0pK+ough54y2Z2v74eXOqp254RgJKVn8d6A221O5RNcntXfoW78ypPpOTWsyMjqQmRuOsifJuhuMlkj424EWQoggIYQbMAK4bD8yIURdUTYzRAgRXVbvWQvUbTn12mvrbm+ZCkW2N6FCsUFbpmqJ6mK3hBnxyVl8svoAfdvU5bY2dXUKzobUag6tB8K2mdqmMQZ4sV8ralZz54UFuykqKTUkhoqodMKXUhYDjwKrgARgnpRynxBikhBiUlmxocBeIUQcMBkYIa1xxkK3p+FCmvb2UFGqUn6m9m6y9UCoZX47wvyiEp76ORY/LzfeGRzqWDNqr0fXJ6EgU/ddsS7y9XTlzUFtiE/J4uuNxwyJoSIscptfSrlCShkspWwmpXy77LGvpJRflX39hZSyjZSyvZSyk5RykyXqtbjGXSCwI/w9We19q1StmFlaF2I5a+Z8tOoAB1Kz+WBoO/vb2MSSGkRou2Jt+RKK8g0JoW/bevRpHcCnqw9yPN06d9Vz8HFdVxBCaylknoR4s/eVFaVyLo4Ia3oz1A83W2zTkXRmbjzGvZ0acXNLB1kcrTK6PQU5qdpkLIO8Magtbs5OvPSrdS67oBL+lYL7Qq1g2DQZrPAPptiB3fO0xNT1CbNFzmTl88TcWG3IX/9WOgZnw4Ju0i6gmyZDqTE7U9X19eDF/iFsPnqWeTH6rtlfESrhX8mpbIhcShwcW290NIq9KS3VhmIGhGpdECYUlZTyyI87yckvZuq9HfByc9E1RJslhLbGzrmjcGClYWGMjGpEdBN/3l6ewJksY7qXzFEJ35R2w8G7jtZSUBRLOrwa0g9os2rN3IB9Z0UC24+f5/2h7RxvJczKajUQfBvB5i8MC8HJSfDukFDyi0t5bck+w+IwRSV8U1w9oONEOLwGUq3rD6bYuE1ToHoDaDvY5OHFsaeY/fdxxncNYmD7+joHZwecXaDzw3ByMyRuNyyMZrWr8USvFqzce5rf9p42LI4rqYRvTuT94OoNm4xrKSh25tROOL4BOj0EzldPntp69CzPzd9NdJA/L/UPMSBAOxF+L3j4wuYphoYxsUdTWtWrzquL95KZV2RoLBephG+Olz9EjIY9v6ilkxXL2PwFuFeHiLFXHTqYms2E72IIrOHJ9NEdcHX0hdEqw91H2wYxYSmcM25MvKuzE+8PCSU9p4B3VyQYFsel1H/VtXR8EEqLYfsMoyNRbF1GorZ9YcQY8Kh+2aHTmfmMm7UNd1dnvrkvGj8vNd6+0jo+CMJZG5dvoHYN/ZjQvSlztyey6XC6obGASvjX5t8UQm7XJskU5hodjWLLtk3TPnecdNnDZ3MKGDtrG5l5RXxzXxSB/l4GBGeHqteH0KGw6wfIyzA0lCdvDaZJTS9e/NX4FTVVwi9P50cg7zzsnmt0JIqtKsiBHd9pyyj4/bvO4NmcAkbN2MqJcxeYMSaSNvV9DQzSDnV6CIouwM7vDA3D082Z94a04+S5XD7+/aChsaiEX55GnaFemDYzstR6F0VSrFjsj9o6L50e+eehszkF3DNzK8fPXuDrsVF0aW5di8fahXrtoXE32Dbd8KVSOjWtyT0dGzHr72PsPHnesDhUwi+PEForP/0gHFlrdDSKrSktha1ToUEkBEYBkJqVz6gZWzmWfoFZ46LoqpJ91en0EGQmwv6lRkfCi/1CqFvdgxfm76ag2JiZwCrhV0TrO8GnHmz+n9GRKLbm4G/azM/ODwNwJC2HwV9uIul8LrNVsq96LftBjSZWsZudj4cr7wwO5dCZHKasPWxIDCrhV4SLG0RPgKN/whnrGF6l2IgtX0L1htBqELGJGQyduomC4hJ+frCz6sbRg5OzdqM8cSsk7TA6Gm5uWYchEQ2Zuu4Ie0/pv1mKSvgVFTEOXDxg6zSjI1Fsxem92kSr6An8vj+dkdO34OPhyvxJXWjbQN2g1U3YPeDmY/gQzYteGdAKf283npuv/2YpKuFXlHdNCB0GcXMh95zR0Si2YNs0pIsnc4p68uD3OwgOqMb8hzrTpJa30ZE5Fo/q2iTK+EVWMYnSz8uNt+9sS0JKFlP/OqJr3XaZ8KvsqtlxEhTnafuQKsq15J5D7p5HjG8fXll1ij6tA5g7sTN1fDyMjswxRU/Qlkw2aEesK/VpU5eB7esz5Y9DJKToty2j3SX8/KIShkzdxJS1hygttfB69nXbQpPusG2G4cO8FOtWsG02ojif/yR3YWKPpky9pwOebs5Gh+W4/JtC8G1awi8uMDoaAF4f2AZfT1ee/SVOt64du0v4pVISVMubj1cfZOzsbZzNsfAft+MkbZjXgRWWfV7Fbpw6l03mui/ZVNqGcXf25+X+rXByUnvRGq7jg5CbDnt/NToSAPy93XjrzlD2JWfx5Z/6dO3YXcL3cnPhs+FhvHNXKFuPnaP/5A1sP27BPveW/bT1ttXNW8WE+OQsJv/vM+rIdHx7Psqojo2MDkm5qOnNUKslbP3Kanaz69v2366d+OSq79qxu4QPIIRgVMdGLHy4C56uzoycvoXvNh+3zB6TTs5af+CJjXB6T+WfT7EbW46eZfi0zYwoXUGhTyBteg43OiTlUkJo+1ykxEKScWvlX+m/A9vg5+XGM7/EUVhctV07dpnwL2pT35fFj3ajR3BtXl28j+fm7ya/yAIz3MLvBRdPbcq2ogC/7U1hzKxtdK6WQriMx63TRK1xoFiXdiPA3Vdr5VuJGt5uvDs4lISULKb8cahK67LrhA/g6+nKzDGRPN6rBfN3JDF82ubK7zPp5Q/t7obdv6ghmgqLY0/x8A87aVu/OlOabdcaAxGjjQ5LMcW9mtZgi19sFUM0L+rdOjTPJ1cAACAASURBVIAhEQ358q8jxCZW3eqedp/wQdtj8unewUwb3YGDqTkM+t/f7Euu5Cy36AnaEM3YHywTpGKTlsYl89TPsUQH+fP9PcG4xy/QGgOeNYwOTTEn6n5tiOaOb4yO5DKv3tGaOj7uPDMv1jI9ESY4RMK/6LY2dZn/UGcAhn21md/3VWKvybqh0KiLNkSz1JiFkBRjrdyTwpM/xxLZ2J9Z46Lw2vuT1giInmh0aMq11GwGLXrDjtlQXGh0NP/w9XTl/SHtOJJ2gQ9XHaiSOhwq4UNZv/4jXWkR4MOD3+/gu83Hb/zJOk6EjBNwaLWlwlNsxIZDaTz20y7CAv2YdV8UXi4Cts+Exl21+RqKdYueCDmpkLDE6Egu0yO4Nvd2asTq+NQq2SzF4RI+QJ3qHsyd0IleIQG8ungf763cf2OTtEIGgE/9f3czUhzCodRsHv5+J83rVOOb+6Ko5u6iXfQzTmhdfYr1a9YLagRp79CtzMv9W7Hiie54ublY/LkdMuGDtgvNtNEduLdTI75ad4Sn5sVe/2w3Z1eIHA9H/oD0qr27rliHszkFjP92O+6uzswcG4mPh6t2YNs07eIfMsDYAJWKcXLSLs6JWyAlzuhoLuPl5qI1IqqAwyZ8AGcnwZuD2vLcbS1ZHJvMxO9iyCu8zv74DmPByVV7O6/YtYLiEiZ9v4PUrAJmjOlAwxpl+8+mH9Yu+pH3aY0AxTaE3QOuXg41vNqhEz5ok7Qeubk57w4O5a+DaYyZtZXMvKKKP0G1OtDmzrJt7HKqLlDFcO+vPMD24+f5aFh7whtdMgpn+0ztot9hnGGxKTfA0w/aDYc98x1meLXDJ/yLRkY3YsrIcGITMxgxfcv1rcETNQEKsmDPvKoLUDHUuoNpzPr7GGM6N2Zg+/r/HijI0S72rQdpF3/FtkRPgOJ82PW90ZHoQiX8SwxoV5+ZY6M4mpbDyBlbOJNdwQlagdHaMM1tM6xmjQ7Fcs7mFPDsL3EEB1Tj5f6tLj+4Z562QbkaimmbAtpow6tjvtb2H7ZzKuFf4abg2sweF0XiuTxGTN9CakVm5QqhtfLPxMOJTVUfpKIbKSXPz99NZm4Rn48Ix8PV+dKDsG2mdrEPjDYuSKVyoh+A88fh8BqjI6lyKuGb0KV5Lb4dH01qZj7Dp23mdGYFkn7oMPDwhe3WN8xLuXG/xCSxdv8ZXugXQqt61S8/eHIznNmnXeyFWv7YZoXcAdUCHOK1qxK+GdFB/sx5oCPpOYWMmrGl/PV33LwgfDQkLLWqNTqUG3f+QiHvrkwgqkkN7uvS5OoC22ZoF/nQYbrHpliQi5t2w/3Qajh3zOhoqpRK+NcQ0agG39wXxemsfEbN3Epadjk3ciPHQ2mx1a3RodyYD38/QFZ+MW8Manv1BibZp7VZmmH3ahd7xbZ1GAfCSevLt2Mq4Zcjsok/s8dFcep8HvfM3ML5C9dYe6NmM2h+q5bwS65jaKdideISM/hp20nGdm5ydVcOwI5vtYt71P36B6dYXvX60OoO2DkHivKMjqbKWCThCyH6CiEOCCEOCyFeNHFcCCEmlx3fLYSIsES9eunYtCZfj43k+Nlcxs7eRnb+NZJ51ATIOQ37l+kXoGJRJaWSVxbvpVY1d57s3cJEgSJt4a1mvbSLvGIfoidAfgbsXWB0JFWm0glfCOEM/A/oB7QGRgohWl9RrB/QouxjIjC1svXqrUvzWnw5KoJ9yVk88G2M+eVLW/QGv0ba6A3FJs3fkcjupEz+078V1T1MzJzdvxyyU9S6OfamcVeo3cquh1dbooUfDRyWUh6VUhYCc4FBV5QZBHwnNVsAPyFEPQvUratbWwfwyd3t2Xb8HI/8sNP02jtOzlpf/omNkBqvf5BKpeQXlfDZmkOEBfoxKKy+6ULbZ2r7Grfoo29wStUSQuuiS4mFUzuMjqZKWCLhNwASL/k+qeyx6y0DgBBiohAiRggRk5aWZoHwLGtQWAPeHNSWtfvP8OKCPab3yQ0fA87uan0dGzR320lSMvN5tk9LhKmhlmf2w/ENEDVebWFoj9qPADcfq1xF0xIskfBNDUC+MgtWpIz2oJTTpZSRUsrI2rVrVzq4qnBvp8Y8eWsLFuxMMr1RgXdNaDsYdv8M+VW/E71iGXmFJXzx5xE6BvnTtXlN04W2z9Qu5uFj9A1O0Ye7j5b09/0KF9KNjsbiLJHwk4DAS75vCCTfQBmb8kSvFoyMbsSXfx3hm79NjN2NmgCFORA3V//glBsyZ8tx0nMKeMZc674gW/t7th2sXdQV+xT1AJQUwq45RkdicZZI+NuBFkKIICGEGzACuHIbmSXAmLLROp2ATCmlTc9OEkLw5qA29G4dwH+XxfPb3it+nYYdoH641iK00xtA9iSnoJipfx2hR3BtooP8TReKmwuF2VpCUOxXnRBo0h22z7K77UsrnfCllMXAo8AqIAGYJ6XcJ4SYJISYVFZsBXAUOAzMAB6ubL3WwMXZiSkjwwkL9OOJubHsOnn+8gJREyD9ABxbb0yASoV9u+k453OLeKZ3sOkCUmoX73ph0KCDvsEp+ot6ADJPwqHfjY7EoiwyDl9KuUJKGSylbCalfLvssa+klF+VfS2llI+UHQ+VUsZYol5r4OHqzMwxkQRU9+CBb2M4eTb334NtB4Onv0Os0WHLCopL+GbTcXoE16Z9oJ/pQsc3QNp+bVVMtW6O/Qu5vWz7Uvt67aqZthZQs5o7s++LorhUMu6bbWTmlk3McvWEiNGwfwVknjI2SMWsxbHJpGUXMKF7kPlC22aAZw3tIq7YP2dXbQezI2u1Hc3shEr4FtKsdjWmj+5A4rlcHvnxkjH6keNBlmozMxWrI6Xk6w3HCKnrQ7fmtUwXyjylTbYKH61dxBXHEFG2fakdra+jEr4FdWxak3fuCmXj4XTeXFY26apGEwi+TVtfp/g6dtFSdLH+UDoHUrN5oHtT0yNzQPvbyVK1bo6j8QnQdjLb9QMUXjA6GotQCd/ChkUGMrFHU77bfII5W05oD0ZPgAtpEH/l4CXFaDM3HKWOj/vl2xZeqrhQS/gt+mgXb8WxRE/QdjTbbR/bl6qEXwVe6BtCr5A6vL5kH5sOp0PTW8C/qbp5a2USUrLYcCidsV2a4OZi5qWQsAQunFHr5jiqwI4QEGo3w6tVwq8Czk6Cz0aE0bSWN4/8uJPEjHxtiGbiVkiONTo8pczsv4/h6erMPR0bmS+0bbp2sW7WS7/AFOshhLYFYupebYczG6cSfhXx8XBl+phIikslD87ZQV6bEeDqrVr5ViIzr4glccncGV4fPy8304WSY7WLdNQEcFIvFYd1cfvSbdONjqTS1H9xFQqq5c3kkeEknM7iheUnkO2Gw575kHvO6NAc3qJdp8gvKmVUdGPzhbbPAFcvCBulX2CK9XHzvmT7UpteEUYl/Kp2c8s6PNunJUvikvnVpR8U59vlGh22RErJT9tOEtrAl9CGvqYL5Z7TLs7thoOnmclYiuOIekBbZiHGtodXq4Svg4d7NqNvm7o8v6GYrIBO2g0gO1ujw5bsPJnB/tPZjIy+Rt/9zu+0i7O6WasA+AeVDa+ebdPDq1XC14EQgg+HtaOxvxdvpXWDjJNwcJXRYTmsH7eexNvNmYHmNjgpLYHtX0PjbhDQRt/gFOtlB8OrVcLXiY+HK1+N7sCKogjSnWpRagc3gGxRZm4Ry3YnMyi8AdXcXUwXOrhKWzir40R9g1OsW9NbwL8ZbJtmdCQ3TCV8HQUH+PD2kDBmFdyC09E/td2TFF0t3JVEQXEpo67VnbP1K6jeAFrerl9givVzctIWz0vaDqd2Gh3NDVEJX2eDwhpQGj6GAunKyVWfGR2Ow5m7PZF2DX1p28DMzdozCXBsnbaMgrOZdwCK4wobCW7VbHaIpkr4Bnjqzi6s9+hJrSO/kpRi0/vA2JT45Cz2n85maIeG5gttnQYuHhAxTre4FBvi4asN0927AHLOGB3NdVMJ3wDuLs6E3vUcXhSw6vuPKChWI3b0sGBnEq7OgjvamblZm3de29UqdJjawlAxL3qitgWiDQ7RVAnfIHVDOnKuZgd6Zy/mveX7jA7H7hWVlLI49hS9QgKo4W1mZu3OOVCcBx0f1Dc4xbbUagHNe2vLJhcXGh3NdVEJ30D+tzxGI6c0Tm5dxOr4VKPDsWvrD6aRnlPIEHPdOaUl2iYnjbtB3VB9g1NsT8dJkJMK8YuMjuS6qIRvpJABSJ/6POa1mufmx5GckWd0RHZrwc4kanq70bNlbdMFDqwsG4qpWvdKBTS7BWo210Z02RCV8I3k7IqInkBY8W6Cio/x5NxYii/ulKVYTEZuIWvizzAwrD6uzmb+5bdMBd9AaNlf3+AU2+TkpLXyT+2AxO1GR1NhKuEbrcM4cPHk0yZb2Hb8HJP/sJ/9M63F0t0pFJaUMiTCTHdOShyc2KjdjFNDMZWKaj8C3H1hy5dGR1JhKuEbzcsfwkbS5NRyxrTz4os/DrH9uFpN05J+3ZlEywAf2tSvbrrAlqna0tURY/QNTLFt7j4QMRriF0NGotHRVIhK+Nag40NQUsB/ArYQ6O/Fk3NjycwrMjoqu3DybC67TmZwV0QD03vWZp/WVsUMv1etiqlcv44PAtJmJmKphG8NagdD896475zF5GGtSc3K5z8L9yDtYEs1oy2JOwXAHeb2rN3+NZQWq5u1yo3xawStBsKOb6Egx+hoyqUSvrXo9BBcOEP7jD94qncwy3anMH9HktFR2TQpJYtik4lu4k8DP8+rCxTlaWOpW/aDms30D1CxD50f1TY6j/3R6EjKpRK+tWh2C9RuBZu/YFKPpnQM8uf1Jfs4eTbX6MhsVnxKFofP5DAo3Ezrfs8vkHsWOj2sb2CKfQmMgoZRsHUqlFr3KDuV8K2FEND5EUjdi/PxdXwyPAwnJ8FT89RQzRu1JDYZFydB/7b1rj5YWgqbvtAmWTXppn9win3p9DCcOwoHVxodyTWphG9N2t0N3nVg0xQa+Hny1p1t2XHiPFP/OmJ0ZDantFSyJC6Zm4Jrm15K4fAaSD8AXR7XLraKUhmtBoJvI60RYcVUwrcmLu7aphtH1kJqPIPCGjAorD6frT1EbGKG0dHZlG3Hz5GSmW9+V6tNk7U179vcpW9gin1ydtHuw53cBEkxRkdjlkr41ibyfnD1gs1aS+GNQW0J8HHnqZ9jyS0sNjg427E4NhkvN2d6tw64+uCpnXB8g/YCdXbVPzjFPkWM0ZZP/vtzoyMxSyV8a+Plr40J3z0PslLw9XTlo7vbcyz9Au+uUDtkVURhcSkr96bQu3UAXm4mZs5u/gLcq0PEWP2DU+yXezWtwZawVOvPt0Iq4VujTg9pY8PL9s7s0qwWD3QLYs6WE/x1wPY2XdDbhkNpZOQWMchUd875E7BvkbakhYeZmbeKcqM6Pqi9a9z8P6MjMUklfGvk3xRaD4TtsyA/C4Bnb2tJcEA1np+/m/MXbGsNbr0tiUvGz8uVbs1NrIy5Zap2k7bjJP0DU+yfT11t8MWuH+DCWaOjuYpK+Naq65PaZI4d2q46Hq7OfHJ3GOdzC/m/xXvVLFwzcguL+X1fKv1D6+HmcsW/94WzsPNbbUcr3wbGBKjYv86PaRvpbJ9hdCRXUQnfWjWIgKY9tbeGRfkAtG3gy5O3BrN8dwpL4pINDc9arUk4Q15RCQNNLaWwbRoU5WoXU0WpKnVCILiftla+lS23oBK+Nev2lLarTtxP/zz0YI+mhDfy49XF+0jNyjcwOOu0JDaZutU9iG7if/mBgmxtg/KWt2svSEWpSt2f0fZI3vmt0ZFcplIJXwjhL4RYLYQ4VPa5hplyx4UQe4QQsUII6x2kam2CboL64dowr1Jto3MXZyc+uTuMguISnp+/W3XtXCIjt5B1B88woF09nJyumEy141vIz4DuTxsTnOJYAqOgSXfYNAWKC4yO5h+VbeG/CKyVUrYA1pZ9b87NUsowKWVkJet0HEJAt6fh/DFtze0yQbW8ealfK9YdTOOnbbaxDrceftt7mqISyaCwK/rniwu0oZhNukND9e+n6KTbU5CdAnFzjY7kH5VN+IOAi+9ZvgXurOTzKVcKGQA1W8DGT+CS1vzoTo3p1rwWby2PVwuslVkSl0xQLW/aNrhiuGXcXO2F1+0pYwJTHFOzW6BeGPz92T/v0I1W2YQfIKVMASj7XMdMOQn8LoTYIYSYeK0nFEJMFELECCFi0tLSKhmeHXBy0rohTu+Bg79d8rDgg6HtcBaCZ3+Jo7TUsbt2Tmfms/noWe5oX//yjU5KirUXXL322gtQUfQihPbaPXcU9i00OhqgAglfCLFGCLHXxMeg66inq5QyAugHPCKE6GGuoJRyupQyUkoZWbu2iXHUjih0GPg1hnXvX9bKr+/nyWsD27Dt+Dlm/X3MwACNtzQuGSnhzisnW+2dr73gejyvFklT9BdyB9QKhg0fW8XSyeUmfCnlrVLKtiY+FgOpQoh6AGWfTU4DlVIml30+AywEoi33KzgAZ1ftrn/yLji89rJDQyIacGurAD5YdYDDZ7INCtB4i2JP0b6hL01rV/v3wdISWP8hBIRCyO3GBac4LicnrbFxJh4SlhgdTaW7dJYAFxckGQssvrKAEMJbCOFz8WugD7C3kvU6nvYjwTcQ1r13WStfCME7g9vi7ebM0/PiKHLAtfMPpWazLzmLO8OvuFm791c4exhuek617hXjtB2s3Ydb94HhrfzKJvz3gN5CiENA77LvEULUF0KsKCsTAGwUQsQB24DlUsrfTD6bYp6LG3R7EpK2w9G/LjtUx8eDt+4MZXdSpkOunb8o9hTOToIB7S7pziktgfUfQJ3W2ttqRTGKkzP0eA7O7IP9y4wNpTI/LKU8K6XsJaVsUfb5XNnjyVLK/mVfH5VSti/7aCOlfNsSgTuk8NHgU19rKVwx/v72dvW4o319Jq89xN5TmQYFqL/SUsmiXcl0a16L2j7u/x6IXwTpB7UXmpOaX6gYrO0QqNlcuw9nYCtfvRJsiYu7dtf/5CY48sdVh98c1IYa3m48My+OgmLrGAZW1XacPM+pjDzuDL+idb/uA6gdAq3VSGHFCji7aI2P1L1wYLlhYaiEb2sixmh9+X+8eVUr38/LjfeHhHIgNZvP1hwyKEB9Ldx1Ck9XZ/q0rvvvg3vmQ9p+uOkF1bpXrEfboeDfDP5817BWvno12BoXdy2RJe+C/Ve3FG4JCWB4ZCDT1h1hx4lzBgSon4LiElbs0TY68XYv2+ikuBD+egfqtlOte8W6OLvAzS9rffl7FxgSgkr4tqj9SK0/8M+3Tc7g+78Brajn68nT8+LselvENfFnyMgtYmiHhv8+uGsOnD8OvV5VrXvF+rQZrA0T/vNtKCnSvXr1irBFzi7Q8yVtbO/eX6867OPhysd3t+fkuVy73hbxlx2J1PP1oGvzWtoDRXla332jztD8VmODUxRTnJyg1yva+lg7v9O/et1rVCyjzWAIaKt1X5hoKXRqWpP7u2rbIq47aH9LVJzOzGf9wTSGdmiI88WVMbfNgJzTWutejbtXrFWLPhDYSWucFOq7DpZK+LbKyUlLbOeOQsxsk0Weva0lzetU4/n5cWTk2te2iAt2JlEq+bc7J++8tsBcs17QuIuxwSnKtQihvXZzTsO26bpWrRK+LWvRR1vy9693If/qsfcers58encYZ3MKeWXxPgMCrBpSSn6JSaRjkD+Na3prD67/CPIyoPd/jQ1OUSqiSVdo3hs2fKLr3rcq4dsyIaDPW5B3DjZ+arJIaENfnry1BUvjklkce0rnAKtGzInzHD+by7DIQO2Bc8e0llLYPVA31NjgFKWi+rwJhdnacik6UQnf1tUPg3YjYPOXkGF6M5RJNzUjopEf/7doL8kZeToHaHnztifi7eZM/9CysfdrXgcnF7jl/wyNS1GuS51W0GEcbP8a0g7qUqVK+PbgYqL7402Th12cnfh0eBglpdLm187Pzi9i+Z4U7mhfHy83Fzi5VVtGocvjUL2e0eEpyvXp+TK4ecPqV3SpTiV8e+AXCJ0fht0/Q+J2k0Ua1/Tm1QGt2XTkLF9vtN2183/deYrcwhJGRDfSZiv+/h+oVhe6Pm50aIpy/arV1pY+P/gbHPmzyqtTCd9edH8GfOrBimfMbqc2PCqQ29oE8MGq/Ta5wFppqeTbzcdpH+hHWKAfxP2orR7a61WtlaQotqjjJPBrBKtervLJWCrh2wt3H+0Gbkoc7PjGZBEhBO8Nboe/txuPz91lc7NwNx5O52jaBcZ1aQy552D1qxDYUZt5rCi2ytUDbntHm0i5dVqVVqUSvj1pO0Qbprn2DbNDvWp4u/Hp3WEcS7/Am8sSdA6wcr7bfJxa1dzoH1oP/nhLG3t/+8dqCQXF9oUM0IZZ//UuZFbdaDr1SrEnQkD/D6EwB9a+brZYl+a1mNijKT9tO8nKPSn6xVcJJ8/msnb/GUZGN8I9NQ5iZkH0g2oYpmIfhIB+H0Bpsda1U0VUwrc3dVppfYI758DJLWaLPdO7Je0D/Xh+wW5OntV3eveNmLPlOM5CcE9UQ1j+DFSrAze/ZHRYimI5/kHQ/Vlt1NnhNVVShUr49qjnS9qa+Ysf0RYUM8HNxYkvRoYD8NhPOykstt69cHMLi/l5eyK3ta1L3YTZkLwT+rwNHr5Gh6YoltX1cW0l3OXPQlG+xZ9eJXx75F4NBn6ubeD9l/lZfIH+Xnw4tD1xSZm8/5v1rqr5w5aTZOUX81BbtLkGwf0gdKjRYSmK5bm4w4BPocuj4Oxq8adXCd9eNbtF2wN30xQ4tdNssb5t6zKuSxO+3niMVftO6xhgxeQVljBt/RG6N/OnbczL/74g1GqYir0K6gFRD2ibn1uYSvj2rM9bWl/34ke1naDMeKl/CO0b+vLMvDiOpOXoGGD5ftx2kvScQt5qsAlOboa+76kZtYpyg1TCt2eeflpr+Mw++OMNs8XcXZyZem8H3F2ceHDODnIKrGN8fn5RCV+tO8LgRrk03vWRtrqgGnOvKDdMJXx717IfRN6vde1c485/fT9PpowK51j6BZ77JQ4pjV9v56dtJ8nKzubNoo/BxQMGTlZdOYpSCSrhO4Lb3obarWDhJMg5Y7ZYl2a1eKlfCCv3nuaLPw7rGODVLrbuP/dfgPf5BLjrK6he39CYFMXWqYTvCFw9YegsKMjWkn6p+SGY93cL4q7wBny8+iBL45J1DPJy09cfJSxnA31zl0LnRyH4NsNiURR7oRK+owhora3XcWTtNTdcEELw3pBQopv488wvcew4cU7HIDUnzl5g8Z9/86nHTKgfAb1e0z0GRbFHKuE7ksjx2q5Q696HfQvNFnN3cWba6A408PNkwnc7OHH2gm4hSil5f9E2pjt/gIebq/bOxMVNt/oVxZ6phO9IhNBG7QR2hIUPQXKs2aI1vN2YNS6KUim59+utpGTqs1PW73tOMeL4KwQ5peI0fI423VxRFItQCd/RuLjD8O/BqybMHQVZ5hdPC6rlzXfjo8m4UMQ9M7aSll1QpaFdKCgma9Gz9HDeg7z9EwjqXqX1KYqjUQnfEVWrAyN/grwMmHMnXEg3W7RdQz9m3xdFSmY+987cyvkL5idwVYaUktUzXmRY6UpOt3kA58ixVVKPojgylfAdVb12MOpnOH9cS/p5580WjWziz8yxkRw7e4Hh0zdXyUbom75/nTvTZ3AooB91h3xg8edXFEUlfMcW1B1G/ABpB+D7oZCfZbZo1+a1+GZcFCkZ+Qz+chP7T5sve70OLf6Arkc+Y6fPzTSfOKdK1hBRFEUlfKX5rTDsW0iJhdn9r7nbTpfmtZg3qTMAw6ZuZv3BtMrVLSVnlr9Ni11v87drF1o9PBdRBSsEKoqiUQlfgZD+MGqe1r0zsxek7DZbtFW96vz6cBfq+3kyZtY23lwWT36R6U3Tr6m4gJRvxlJn+wesdOpB4wd/wtPT48Z/B0VRyqUSvqJp3gvuXwXCGWb1hb2/mi1a38+TRY90ZWznxny98RgDv9hIbGJGhauS2ac5PaUP9U4sZo7naMIfn0fDWn6W+C0URbkGYQ2LZJkTGRkpY2JijA7DsWSlwM/3wKkd2qbo/T8CL3+zxdcdTOO5X+I4k11A9xa1mHRTM7o0q4kwschZaUkpCaum02j7m7iUFjKn3kuMHv8Enm6qz15RLEUIsUNKGWnymEr4ylVKimHjp9oSDF41tXX12w4xezM1K7+IH7ac5OuNx0jPKaBJTS/aB/rRtr4vdaq7czozn/wzh+l04AM6FsewW4RwtOv7DLzlJpyc1OqXimJJVZbwhRDDgNeBVkC0lNJkdhZC9AU+B5yBmVJK84u5XEIlfIOl7IbFD8PpPdo+m92fhdBh4Oxisnh+UQkLd53ij/1n2Hsqk5TMfFqL40xyWcrtTlsoEm4cCn2aVgOfwcVV3ZxVlKpQlQm/FVAKTAOeNZXwhRDOwEGgN5AEbAdGSinjy3t+lfCtQGkp7F8G6z6A1D3gWUPbiKRlX2gYDT71/r0ASAmFF+D0bji8lqKDq3FNjUO6VUNE3Q+dHgafusb+Popi566V8E031SpISplQVsG1ikUDh6WUR8vKzgUGAeUmfMUKODlB64HQ6g449Lu26NrBVbBnnnZcOEG1AHBygQtpUJxf9rgzrg2joPcbiIix2u5biqIYqlIJv4IaAImXfJ8EdDRXWAgxEZgI0KhRo6qNTKk4IbQ16YNvg9ISSIqBtATISoasU9pj3rXAuzbUCNI2YlZJXlGsSrkJXwixBjD1Pvw/UsrFFajDVPPfbD+SlHI6MB20Lp0KPL+iNydnaNRR+1AUxWaUm/CllLdWso4kIPCS7xsCxm2lpCiK4qD0mHi1HWghhAgSQrgBI4AlOtSrKIqiXKJSCV8IcZcQIgnoDCwXQqwqe7y++r9fkAAABKpJREFUEGIFgJSyGHgUWAUkAPOklPsqF7aiKIpyvSo7SmchcNVeeVLKZKD/Jd+vAFZUpi5FURSlctRaOoqiKA5CJXxFURQHoRK+oiiKg1AJX1EUxUFY9WqZQog04MQN/ngtwPzu3MZRcV0fFdf1UXFdH3uMq7GUsrapA1ad8CtDCBFjbgEhI6m4ro+K6/qouK6Po8WlunQURVEchEr4iqIoDsKeE/50owMwQ8V1fVRc10fFdX0cKi677cNXFEVRLmfPLXxFURTlEirhK4qiOAi7SfhCiA+FEPuFELuFEAuFECa3WxJC9BVCHBBCHBZCvKhDXMOEEPuEEKVCCLPDrIQQx4UQe4QQsUKIKt/I9zri0vt8+QshVgshDpV9rmGmnC7nq7zfX2gmlx3fLYSIqKpYrjOunkKIzLLzEyuEeFWHmGYJIc4IIfaaOW7UuSovLt3PVVm9gUKIP4UQCWWvxSdMlLHsOZNS2sUH0AdwKfv6feB9E2WcgSNAU8ANiANaV3FcrYCWwF9A5DXKHQdq6Xi+yo3LoPP1AfBi2dcvmvo76nW+KvL7o60KuxJtZ7dOwFYd/nYViasnsEyv/6eyOnsAEcBeM8d1P1cVjEv3c1VWbz0gouxrH+BgVf9/2U0LX0r5u9TW3gfYgraz1pX+2VBdSlkIXNxQvSrjSpBSHqjKOm5EBePS/XyVPf+3ZV9/C9xZxfVdS0V+/0HAd1KzBfATQtSzgrh0J6VcD5y7RhEjzlVF4jKElDJFSrmz7OtstP1CGlxRzKLnzG4S/hXGo10Vr2RqQ/UrT7BRJPC7EGJH2Ubu1sCI8xUgpUwB7QUB1DFTTo/zVZHf34hzVNE6Owsh4oQQK4UQbao4poqw5tefoedKCNEECAe2XnHIouesUhug6K0iG6oLIf4DFAM/mHoKE49VelyqBTZ6B+gqpUwWQtQBVgsh9pe1TIyMS/fzdR1PY/HzZUJFfv8qOUflqEidO9HWVMkRQvQHFgEtqjiu8hhxrirC0HMlhKgGLACelFJmXXnYxI/c8DmzqYQvy9lQXQgxFhgA9JJlHWBXqJIN1cuLq4LPkVz2+YwQYiHa2/ZKJTALxKX7+RJCpAoh6kkpU8reup4x8xwWP18mVOT3r5JzVNm4Lk0cUsoVQogvhRC1pJRGLhRmxLkql5HnSgjhipbsf5BS/mqiiEXPmd106Qgh+gIvAAOllLlmilnlhupCCG8hhM/Fr9FuQJscUaAzI87XEmBs2ddjgaveieh4viry+y8BxpSNpugEZF7skqpC5cYlhKgrhBBlX0ejvdbPVnFc5THiXJXLqHNVVufXQIKU8hMzxSx7zvS+M11VH8BhtL6u2LKPr8oerw+suKRcf7S74UfQujaqOq670K7SBUAqsOrKuNBGW8SVfeyzlrgMOl81gbXAobLP/kaeL1O/PzAJmFT2tQD+V3Z8D9cYiaVzXI+WnZs4tEEMXXSI6ScgBSgq+9+630rOVXlx6X6uyurthtY9s/uSvNW/Ks+ZWlpBURTFQdhNl46iKIpybSrhK4qiOAiV8BVFURyESviKoigOQiV8RVEUB6ESvqIoioNQCV9RFMVB/D9HbAqLCxeo1AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "with torch.no_grad(): # no gradients here...\n",
    "    w1r = differentiable_round(w1, 0.5)\n",
    "    b1r = differentiable_round(b1, 0.5)\n",
    "    w2r = differentiable_round(w2, 0.5)\n",
    "    b2r = differentiable_round(b2, 0.5)\n",
    "    x0 = torch.linspace(-2, 2)\n",
    "    final_preds = torch.tanh(x0[:, None] @ w1r + b1r) @ w2r + b2r\n",
    "    gt = torch.sin(2*x0)\n",
    "\n",
    "pyplot.plot(x0, final_preds, label='fit')\n",
    "pyplot.plot(x0, gt, label='ground truth')\n",
    "pyplot.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Own Autograd functions\n",
    "\n",
    "Let's consider $f : \\mathbb{R}^n \\rightarrow \\mathbb{R}^m$ implicitly given by $F(x,f(x))=0$ for some function $F : \\mathbb{R}^n \\times \\mathbb{R}^m \\rightarrow \\mathbb{R}$.\n",
    "\n",
    "For nice (smooth, nondegenerate) functions, the [implicit function theorem](https://en.wikipedia.org/wiki/Implicit_function_theorem)\n",
    "gives us:\n",
    "\n",
    "Given $x,y$ such that $F(x,y) = 0$ there is a neighborhood $U\\ni x$ and a function $f$ such that $f(x)=y$ and $F(x,f(x))=0$. And if $F$ is nice enough, we can also compute the derivative of $f$ at $x$, namely  $\\frac{df}{dx}(x) = - (\\frac{dF}{dy}(x,y))^{-1} \\frac{dF}{dx}(x,y)$.\n",
    "\n",
    "Given $F$ in PyTorch, can we define $f$ and backpropagate through the computation?\n",
    "\n",
    "$\\Rightarrow$ Need to extend autograd."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "class Implicit(torch.autograd.Function):  # derive autograd.Function\n",
    "    @staticmethod\n",
    "    def forward(ctx, x, y0, F, max_iter=100):  # Context ctx as special first parameter to static method forward\n",
    "        with torch.enable_grad():\n",
    "            y = y0.clone().detach().requires_grad_()\n",
    "            xv = x.detach()\n",
    "            opt = torch.optim.LBFGS([y], max_iter=max_iter)  # numerical search for y s.t. F(x, y) = 0\n",
    "            def reevaluate():\n",
    "                opt.zero_grad()\n",
    "                z = F(xv,y)**2\n",
    "                z.backward()\n",
    "                return z\n",
    "            opt.step(reevaluate)\n",
    "        ctx._the_function = F                 # save non-tensors\n",
    "        ctx.save_for_backward(x, y)           # safely save tensors\n",
    "        return y\n",
    "    @staticmethod\n",
    "    def backward(ctx, output_grad):          # ouput_grad = dl / dy\n",
    "        x, y = ctx.saved_tensors             # get the saved tensors\n",
    "        F = ctx._the_function                # other arguments\n",
    "        with torch.enable_grad():\n",
    "            xv = x.detach().requires_grad_()\n",
    "            y = y.detach().requires_grad_()\n",
    "            z = F(xv,y)\n",
    "            z.backward()\n",
    "        return -xv.grad/y.grad*output_grad, None, None, None  # return dl / dx and None for all other inputs\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's try it on a simple case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.8660], dtype=torch.float64, grad_fn=<ImplicitBackward>) 0.8660254037844386\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(tensor([-0.5774], dtype=torch.float64),\n",
       " tensor([-0.5774], dtype=torch.float64))"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def circle(x,y):\n",
    "    return x**2+y**2-1  # 0-set = unit circle\n",
    "\n",
    "x = torch.tensor([0.5], dtype=torch.double, requires_grad=True)\n",
    "y0 = torch.tensor([0.5], dtype=torch.double) # starting value\n",
    "y= Implicit.apply(x, y0, circle)    # this is how to use our new function\n",
    "print (y, (1-0.5**2)**0.5)\n",
    "gr, = torch.autograd.grad(y, x)\n",
    "gr_exp, = torch.autograd.grad((1-x**2)**0.5, [x])\n",
    "gr, gr_exp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "We could now backpropagate through it, too."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# PyTorch is ... a Neural Network (and general Modelling) Library\n",
    "\n",
    "`torch.nn.Module` and subclasses play the central role as Neural Network building blocks.\n",
    "Modules are for bundling neural network *state*, i.e. Parameters, other Buffers, Submodules, with the rules to compute outputs from inputs.\n",
    "- Parameters: wrap in torch.nn.Parameter + assign\n",
    "- Buffers (Tensors noch trained via SGD, e.g. statistics): register_buffer\n",
    "- Submodules: assign"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here is the equivalent of our previous model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[ 1.1177,  1.2803, -3.9744, -2.1031, -1.7723, -0.8169,  0.2313,  1.3349,\n",
       "         -0.5564, -0.0529]], device='cuda:0', requires_grad=True)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class MyModel(torch.nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.w1 = torch.nn.Parameter(torch.randn(1, 10))\n",
    "        self.b1 = torch.nn.Parameter(torch.randn(10))\n",
    "        self.w2 = torch.nn.Parameter(torch.randn(10, 1))\n",
    "        self.b2 = torch.nn.Parameter(torch.randn(1))\n",
    "  \n",
    "    def forward(self, x):\n",
    "        return torch.tanh(x @ self.w1 + self.b1) @ self.w2 + self.b2\n",
    "\n",
    "model = MyModel()\n",
    "model.to(device=\"cuda\")\n",
    "model.w1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "A rich library of pre-made modules under `torch.nn`.\n",
    "- `torch.nn.Linear`, `Conv2d`, `MaxPool2D`, `Tanh`, ...\n",
    "Also loss functions:\n",
    "- `CrossEntropyLoss`, `MSELoss`, `CTCLoss`...\n",
    "\n",
    "Careful: Naming `CrossEntropyLoss` vs. `NLLLoss` etc. is particular to PyTorch. Check the documentation for PyTorch's expectation of whether input have `softmax` / `log_softmax`  / something else applied."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can make our model again with these:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MyModel(\n",
       "  (l1): Linear(in_features=1, out_features=10, bias=True)\n",
       "  (l2): Linear(in_features=10, out_features=1, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class MyModel(torch.nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.l1 = torch.nn.Linear(1, 10)\n",
    "        self.l2 = torch.nn.Linear(10, 1)\n",
    "  \n",
    "    def forward(self, x):\n",
    "        x = torch.tanh(self.l1(x))\n",
    "        return self.l2(x)\n",
    "\n",
    "model = MyModel()\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There also is a `Sequential` class for particularly simple \"chains\" of modules (with only one input/output)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"900pt\" height=\"332pt\"\n",
       " viewBox=\"0.00 0.00 899.62 332.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 328)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-328 895.62,-328 895.62,4 -4,4\"/>\n",
       "<!-- 140075256986688 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>140075256986688</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"473.64\" cy=\"-18\" rx=\"116.18\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"473.64\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">res (AddmmBackward)</text>\n",
       "</g>\n",
       "<!-- 140075256986176 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>140075256986176</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"219.64\" cy=\"-90\" rx=\"156.77\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"219.64\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">model:2.bias (AccumulateGrad)</text>\n",
       "</g>\n",
       "<!-- 140075256986176&#45;&gt;140075256986688 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>140075256986176&#45;&gt;140075256986688</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M276.96,-73.2C316.29,-62.36 368.54,-47.96 409.3,-36.73\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"410.43,-40.05 419.14,-34.02 408.57,-33.3 410.43,-40.05\"/>\n",
       "</g>\n",
       "<!-- 8754703564192 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>8754703564192</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"473.64\" cy=\"-90\" rx=\"79.09\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"473.64\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">TanhBackward</text>\n",
       "</g>\n",
       "<!-- 8754703564192&#45;&gt;140075256986688 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>8754703564192&#45;&gt;140075256986688</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M473.64,-71.7C473.64,-63.98 473.64,-54.71 473.64,-46.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"477.14,-46.1 473.64,-36.1 470.14,-46.1 477.14,-46.1\"/>\n",
       "</g>\n",
       "<!-- 8754703555518 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>8754703555518</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"443.64\" cy=\"-162\" rx=\"92.08\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"443.64\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\">AddmmBackward</text>\n",
       "</g>\n",
       "<!-- 8754703555518&#45;&gt;8754703564192 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>8754703555518&#45;&gt;8754703564192</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M450.9,-144.05C454.31,-136.09 458.46,-126.41 462.27,-117.51\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"465.51,-118.85 466.23,-108.28 459.08,-116.09 465.51,-118.85\"/>\n",
       "</g>\n",
       "<!-- 140075256985792 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>140075256985792</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"156.64\" cy=\"-234\" rx=\"156.77\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"156.64\" y=\"-230.3\" font-family=\"Times,serif\" font-size=\"14.00\">model:0.bias (AccumulateGrad)</text>\n",
       "</g>\n",
       "<!-- 140075256985792&#45;&gt;8754703555518 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>140075256985792&#45;&gt;8754703555518</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M220.33,-217.46C267.35,-206 331.14,-190.44 378.5,-178.89\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"379.46,-182.26 388.35,-176.49 377.8,-175.45 379.46,-182.26\"/>\n",
       "</g>\n",
       "<!-- 140075256984384 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>140075256984384</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"443.64\" cy=\"-234\" rx=\"112.38\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"443.64\" y=\"-230.3\" font-family=\"Times,serif\" font-size=\"14.00\">inp (AccumulateGrad)</text>\n",
       "</g>\n",
       "<!-- 140075256984384&#45;&gt;8754703555518 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>140075256984384&#45;&gt;8754703555518</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M443.64,-215.7C443.64,-207.98 443.64,-198.71 443.64,-190.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"447.14,-190.1 443.64,-180.1 440.14,-190.1 447.14,-190.1\"/>\n",
       "</g>\n",
       "<!-- 8754703532584 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>8754703532584</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"636.64\" cy=\"-234\" rx=\"62.29\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"636.64\" y=\"-230.3\" font-family=\"Times,serif\" font-size=\"14.00\">TBackward</text>\n",
       "</g>\n",
       "<!-- 8754703532584&#45;&gt;8754703555518 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>8754703532584&#45;&gt;8754703555518</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M599.42,-219.5C569.71,-208.73 527.76,-193.51 494.88,-181.59\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"496.02,-178.28 485.43,-178.16 493.63,-184.86 496.02,-178.28\"/>\n",
       "</g>\n",
       "<!-- 140075256985920 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>140075256985920</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"636.64\" cy=\"-306\" rx=\"168.97\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"636.64\" y=\"-302.3\" font-family=\"Times,serif\" font-size=\"14.00\">model:0.weight (AccumulateGrad)</text>\n",
       "</g>\n",
       "<!-- 140075256985920&#45;&gt;8754703532584 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>140075256985920&#45;&gt;8754703532584</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M636.64,-287.7C636.64,-279.98 636.64,-270.71 636.64,-262.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"640.14,-262.1 636.64,-252.1 633.14,-262.1 640.14,-262.1\"/>\n",
       "</g>\n",
       "<!-- 8754703564183 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>8754703564183</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"692.64\" cy=\"-90\" rx=\"62.29\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"692.64\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">TBackward</text>\n",
       "</g>\n",
       "<!-- 8754703564183&#45;&gt;140075256986688 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>8754703564183&#45;&gt;140075256986688</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M652.47,-76.16C618.69,-65.36 569.96,-49.79 531.89,-37.62\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"532.55,-34.16 521.96,-34.45 530.42,-40.82 532.55,-34.16\"/>\n",
       "</g>\n",
       "<!-- 140075256987584 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>140075256987584</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"722.64\" cy=\"-162\" rx=\"168.97\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"722.64\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\">model:2.weight (AccumulateGrad)</text>\n",
       "</g>\n",
       "<!-- 140075256987584&#45;&gt;8754703564183 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>140075256987584&#45;&gt;8754703564183</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M715.22,-143.7C711.84,-135.81 707.77,-126.3 704.01,-117.55\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"707.12,-115.92 699.97,-108.1 700.69,-118.67 707.12,-115.92\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7f65cfefd190>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = torch.nn.Sequential(\n",
    "            torch.nn.Linear(1, 10),\n",
    "            torch.nn.Tanh(),\n",
    "            torch.nn.Linear(10, 1))\n",
    "\n",
    "inp = torch.randn(5, 1, requires_grad=True)\n",
    "res = model(inp)\n",
    "ptgraph(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Task: Can you subclass sequential to get a model class?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Loading and Saving models\n",
    "\n",
    "While you can save and load models using `torch.save` and `torch.load`, it is generally not recommended.\n",
    "\n",
    "Instead, save the state dict and upon loading instantiate the model and load the state dict. This is much better for forward compatibility.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.save(model.state_dict(), 'saved_model.pt')\n",
    "sd = torch.load('saved_model.pt')\n",
    "model.load_state_dict(sd)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Some thoughts on functional vs. Module\n",
    "\n",
    "- If you write for re-use, the functional / Module split of PyTorch has turned out to be a good idea.\n",
    "\n",
    "- Use functional for stuff without state (unless you have a quick and dirty Sequential).\n",
    "\n",
    "- Never re-use modules (define one `torch.nn.ReLU` and use it 5 times). It's a trap!\n",
    "\n",
    "  When doing analysis or quantization (when `ReLU` becomes stateful due to quantization params), this will break.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# PyTorch has ... optimizers\n",
    "\n",
    "In `torch.optim`.\n",
    "\n",
    "- Most common probably: `torch.optim.SGD`, `torch.optim.Adam`, but also `torch.optim.LBFGS` etc.\n",
    "- Schedules for the learning are also available.\n",
    "\n",
    "Pattern:\n",
    "- `loss = ...`\n",
    "- `opt.zero_grad()`\n",
    "- `loss.backward()`\n",
    "- `opt.step()`\n",
    "\n",
    "Optimizers also have a `state_dict` for saving / restoring."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's see that in action with our toy model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "model = torch.nn.Sequential(\n",
    "            torch.nn.Linear(1, 10),\n",
    "            torch.nn.Tanh(),\n",
    "            torch.nn.Linear(10, 1))\n",
    "\n",
    "opt = torch.optim.SGD(model.parameters(), lr=1e-2)\n",
    "\n",
    "for i in range(200):\n",
    "    x = torch.randn(10, 1)    # input  from dataset\n",
    "    target = torch.sin(2 * x) # target from dataset\n",
    "\n",
    "    pred = model(x)\n",
    "    loss = torch.nn.functional.mse_loss(pred, target, reduction='sum')\n",
    "    \n",
    "    opt.zero_grad()\n",
    "    loss.backward()\n",
    "    opt.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3hkd3no8e87M+q9l5W0vfddedcNdzu2MSzGFBeMC+DrBCeQADfOhUCAy70GbkgCMTg2GAy4YDBg44JbjI37anvv0q5WWkmr3svM7/5xRkaac7QqU6V5P88zj6T3nDPn3VmN3jnn18QYg1JKqfjlinYCSimloksLgVJKxTktBEopFee0ECilVJzTQqCUUnHOE+0EpiI/P9/MmTMn2mkopdS0smXLltPGmILA+LQsBHPmzKGqqiraaSil1LQiIjVOcb01pJRScU4LgVJKxTktBEopFee0ECilVJzTQqCUUnEuJIVARB4UkUYR2T3GdhGR74vIYRHZKSLrRmy7UkQO+LfdHYp8lFJh5B2Ejjqo2w41b0HLMRgaiHZWKgih6j76M+A/gZ+Psf0qYKH/sRH4EbBRRNzAvcDlQC2wWUSeMsbsDVFeSqlgDQ1Azetw4I9w+EVoOeqwk0BGMVScDcs2wcIrIDEt4qmqqQlJITDGvCYic86wyybg58aa8/ptEckWkRJgDnDYGHMUQEQe8++rhUCpaOtpgbfuhc0PQF/7ODsb6KyHPb+zHp4UWPoBuPAfIX9BRNJVUxepAWWzgBMjfq71x5ziG52eQETuAO4AqKioCE+WSino77QKwFv3Qn/H1J5jqBd2PQ67n4B1n7QKQmZJaPNUIROpxmJxiJkzxO1BY+43xlQaYyoLCmwjpJVSoVDzFvzwXPjT/516ERjJeGHLT+EH66DqQdCFsGJSpK4IaoHyET+XAXVA4hhxpVQkeQfh1W/Dn/8VjG/8/VPzIK0QEpKhox66Tp15/8EeePrv4eir8MHvQ3JWaPJWIRGpQvAUcJe/DWAj0G6MqReRJmChiMwFTgLXAzdGKCelFFhtAY/dBMffHHsfdyIseT8svhoWXAapuaO3D/ZBzRuw90nY9wfobXF+nr2/h7ptcP3DULwydP8GFRQJxZrFIvIocBGQDzQAXwMSAIwx94mIYPUquhLoAW4zxlT5j70a+HfADTxojPnWeOerrKw0OumcUiHQfhJ++WFo2u+83ZVg3eN/3xcga9bEnnOgB965D17/d+gfo5E5KQtuetzqZaQiRkS2GGMqbfHpuHi9FgKlQqDpIPziWuiodd5evhGuvQ9y503t+Xta4KWvwdYxepV7UuDjv4SFl03t+dWkjVUIdGSxUvGo6QD89ErnIiAuuOif4NZnp14EwLp99MEfwEcehMQM+/ahXnj0etj71NTPoUJCC4FS8ab9JPziw9DTbN+WmAE3/x4uuhvcIWpCXHEd3Pka5C+2b/MNwhOfgmN/Ds251JRoIVAqnvS2wi+vc74SSCuA256BeReG/ry58+C2Z6FkjX2bd8BqrG7QcaTRooVAqXgx2AeP3gBN++zbsivg9uehZHX4zp+WD7f8AWafZ9/W3w4Pf8S6WlERp4VAqXjx3Jfg+Fv2eFqBdTsob374c0jOhJt+A2Ub7Ns6TsIjH4fB3vDnoUbRQqBUPNj6C+feO4np1h/mSBSB986ZCjf+CvIc5iBq2AXPfjFyuShAC4FSM1/9DnjmC/a4K8Ea2FXqcN8+3FJz4RNPWKOTA237JWx7OPI5xTEtBErNZL2t8Kubwdtv33b1d2HeRZHO6C9y5liDytxJ9m3PfAEa9kQ8pXilhUCpmeyZL0JbjT2+5iZYf2vE07EpXQtXf8ceH+qFxz9pjVJWYaeFQKmZas/vYPdv7PGilXD1/wNxmvw3CtbdAquut8ebD8PL34h8PnFIC4FSM1FnAzz9D/Z4UhZ8/OdWg22sEIFrvgcFS+3b3rkPqt+IfE5xRguBUjONMfD0551nAL36u8FNGxEuiWnwsYcc2gsMPPk3MNAdlbTihRYCpWaaHY/BgWft8aUfgFUfi3w+E1WwGC79Z3u8tRpe+pdIZxNXIrUegVIqEnpa4IUv28ImNZ/GC/4vbQ1ddPYNMuD1gQGfgQS3kJbkITXRTWZKAjmpibhdUWo/OPtvrPUMTrwzOv7u/bDiI1DhuJKtClL8FQJjwOcN3YRaSsWSl7/uOJnczrVfp6k1Eberl4xkDxlJCe+1FQ96fbT3DnKqvQ8At0vITUukMDOJooxkXJEsCi43bPoh3He+1XNopGe+AHf8Sd+7YRBfr+iJzfD8P8Hiq6yFNpSaSU5shi0P2cKNs68hY82HmJeRRHqSBxmjt5DXZ2jrGaCpq5/TnQM0dfZzyNPFrJwUZmWnkJzgDve/wJK/AC79qvVeHalhF2z+MZx9Z2TyiCOhWqHsSuA/sFYZ+7Ex5p6A7V8CbvL/6AGWAgXGmBYRqQY6AS8w5LRoQqBJL0zTWmN9Utr9hPVzYjr83TZIdxjVqNR05B2CBy6CU7tGhX2JGfDZzbiySib9lM1d/Zxo7eV0Zz8uF1TkpjI7L40EdwSaFn1eeOBia1T0SEmZcNdmyCgOfw4zUNgWphERN3AvcBWwDLhBRJaN3McY811jzBpjzBrgn4BXjTEjuzRc7N8+bhGYtJ4W+OE5fykCAANd8Mq4K2IqNX1U/cRWBABcl/7zlIoAQF56EmvKszl3QR6FGclUn+7hjcOnOd7cQ9hXNnS54f3fAwKuXvo74MWvhvfccSgUpX0DcNgYc9QYMwA8Bmw6w/43AI+G4LwTk5oLa26wx7f+XIewq5mhpwXfK//HHi9eBZWfCvrpUxM9rJiVxcZ5uWSmJHCwoZN3j7XQ0TcY9HOfUVmltV5yoJ2/ghqHWVTVlIWiEMwCToz4udYfsxGRVKwF7Ed8PMcAL4jIFhG5Y6yTiMgdIlIlIlVNTU2Ty/Cif7IG0oxkfPD8l63GY6Wmsd6X78HV1xYQFbjm30LasJqRnMC6ihxWlWUx4PWx+VgLhxo68fnC+B669GuQkmOPv/AVfe+GUCgKgVPL01j/Qx8A3gi4LXSeMWYd1q2lz4rIBU4HGmPuN8ZUGmMqCwoKJpdhWj5c4DC17dFX4PBLk3supWJI64l9JG170L5h7U3WJ+owKMxM5ux5eZRmp1DT3MM7x1ro6h8Ky7lIy4PL/sUeP1llTaGhQiIUhaAWKB/xcxlQN8a+1xNwW8gYU+f/2gj8DutWU+ht/B+QPdsef/7L4A3zJa5SYdDdP8TQC1/D5Qv4/U1IhYu/EtZzJ7hdLC3JZE1FNoNeH+8ea+ZES5gmiFt7MxQut8df/joMOcyqqiYtFIVgM7BQROaKSCLWH/unAncSkSzgQuDJEbE0EckY/h64AtgdgpzsPElwucMEVqcPwFZ7lzulYtmg18fRLS9RcOJ5+8Zz/w4yp9ZAPFn56UlsnJdLbloSB051squ2nSGvL7QncbnhCof3bms1bP5JaM8Vp4IuBMaYIeAu4HlgH/C4MWaPiNwpIiM7/F4LvGCMGTlpSBHwuojsAN4FnjHG/DHYnMa0bBOUn22Pv/odne5WTRs+n2HniTYqttxj35heDOf9XUTzSfK4WVOezcKidBo7+3i3uoXuUN8qWnAZzLvYHn/129aaCyooIekQbIx51hizyBgz3xjzLX/sPmPMfSP2+Zkx5vqA444aY1b7H8uHjw0bEbjSoXdFVwNsfiCsp1YqVI41d+M68iJZzdvsGy/5sjWBWxTMzktjbUUOg17Du9UtNHWG+LbNFd/E1iTZ1wZv/iC054lD8Tfp3Kz1sOxD9vjr/wZ9HZHPR6lJ6OgbpLqpiyV7v2/fWLjcWnAminLTEtk4N5fUBDc7TrRR0xzCWUOLV8KaG+3xt++D7tOhO08cir9CAHDxl0EC/um9rfD2D6OTj1IT4PMZ9tZ1UFL/EinNDk1pl3zFup8eZckJbirn5FKUmcyhhi721LWHrovpxV+2T1U92A1v/Htonj9OxWchKFjkvCLSm/9pjURWKgYdPd1NV08/i/Y63AopXWfNoRUj3C5hxaxM5hakUd/Wx7YTbQyGohE5axZU3maPv/tjazEeNSXxWQgALvpHcCWMjg10whv/EZ18lDqDjr5Bapq7WdLyIp7m/fYdLvlK7Cw96ScizC9IZ/msTNp6BthS00rfoDf4Jz7/H8CTMjo21Gvd3lVTEr+FIGcOrL/FHn/3Ab0qUDHn4KlOPGKYtd2hbaDiXJh/SeSTmqCSrBTWlGfTO+hlc3UIBp9lFMGGT9vjVQ9C+8ngnjtOxW8hAHjfF8GTPDo22A1v/yg6+SjloLGzj7aeQVa0v4K0HLbvEINXA4Hy0pOonG1NFVFV3UJbz0BwT3je5yEhoHeUt1+v6KcovgtBZgmsv9Uef+e/oK894ukoFcjnMxxu6CIt0UXuFoergbkXwpzzIp/YFGQkJ1A5O5dEt4ttx9s43RVE99K0fOd1CbY+BF2NU3/eOBXfhQCsUZjuxNGx/nZraTyloqy2tZeeAS/Lu95EGvfad7jwf0Y+qSCkJLpZPyeH1ESre+nwqmhTcs5d1toiIw31ae+/KdBCkDXLue/1Wz+E/q7I56OU36DXx9HTXeSmJZD5rkP3yIpzYPb0uBoYKcnjZv3sHLJTE9h9sp2Tbb3jH+QkNRcqb7fH3/2xjjaeJC0EAOd/HiSg/3Vvi7XYh1JRUtPczZDXsKRnC9Rtte/wvi/GfNvAWDxuF2vKc8hLT2RfXQfHm6c4xcs5d9nHFQx0Wp0+1IRpIQCrB9Gqj9vjb92rsxuqqBj0+jjR2ktRZjKpbzt0iyxZAwsujXxiIeR2CavLsinMTOJgQydHm6ZwBZ5RBOtutsff1iv6ydBCMOx9X8A2j0lXA+x8PCrpqPhW29qL12uYN7APat6w73DB9L0aGMnlElbOyqI4K5mjTd0cbpzCH+/zPgeugAV4elthy89CkmM80EIwLH+BNTtpoDd/AL4QT6ur1BkMeX0cb+khPyOJtM332ncoWAKL3x/5xMJERFhemklpdgrVp7s51NA5uSfIroCVH7PH3/6RrjUyQVoIRnKavvf0ATjkMOe7UmFS19bH4JCP+a4G2PcH+w7nfQ5cM+utKyIsK82kPDeVmuYeDpyaZDE4/++xXdF31MKe34csx5lsZv02BWvWepjzPnv8DYf+20qFgc9nqGnpJictkYzt92Nb9TWjFFZ8JCq5RcLi4gxm56VyoqWH/ac6MBNdl7hgESy+2h5/8/u6tvEEaCEIdN7n7LHjb8KJzZHPRcWduvZe+gd9zEvtg20P23c4+07wJNrjM8jCogzm5KdS29LL/lOdEy8G5/6tPXZqJxx7LbQJzkAhKQQicqWIHBCRwyJyt8P2i0SkXUS2+x9fneixEbfgMihcZo+/qUPXVfidaOklI9lDzp6HrInURkrMcB4JPwMtKMxgTn4aJ1t72Vc/wWJQcTbMqrTHdeGacQVdCETEDdwLXAUsA24QEYe/pPzZGLPG//jGJI+NHBFrtHGg/c9Ya6QqFSat3QN09w9RninOI9vX3wLJWZFPLEoWFKYzJz+NurYJFgMR56uCwy9Cg8OobPWeUFwRbAAO+5edHAAeAxy634T82PBZcZ11L3Yk49NBKiqsTrT24HELxTV/gJ7m0RtdHjj7r6OTWBQtKExnbsEkisHSD0D2bHv8LYfeV+o9oSgEs4ATI36u9ccCnSMiO0TkORFZPsljEZE7RKRKRKqamppCkPYZeBJh4x32+NafQ/8kezMoNQF9g16aOvuZlZWM653/su+w4jrIKot8YjFgfsEkioHLbY02DrTr17qc5RmEohA4jWoJ/J/aCsw2xqwGfgAM9+mayLFW0Jj7jTGVxpjKgoKCKSc7YetusS9+0d8B2x8J/7lV3Klt7cUYqOjYAo177DvE4dXASPMLJnGbaO1NkJw9Oubthy0/DW+S01goCkEtUD7i5zKgbuQOxpgOY0yX//tngQQRyZ/IsVGTmgtrbrDH3/6RDjBTIeXzGeraesnPSCJpi0PbQMU5ULo28onFmJFtBmfsTZSYBus+aY9v/okOMBtDKArBZmChiMwVkUTgeuCpkTuISLGINR5eRDb4z9s8kWOjaqPDfOetx3SAmQqpxs5+BoZ8zJZTcOA5+w5Ov4dxyioGqZxs7eXAmUYgb/gMSMCft8562PtkeBOcpoIuBMaYIeAu4HlgH/C4MWaPiNwpIsO/wR8BdovIDuD7wPXG4nhssDmNpba1hx0n2iYxSGWx1Z00kDY8qRA62dZDaqKb7F0/xXZnNKscllwTlbxildW1dHicQYfzTtkVzq+brj7oyDP+LuPz3+55NiB234jv/xP4z4keGy7GQFNnP0eaulhQmDGxg87+azj80uhY9Z+hcR8ULg19kiqu9A54ae0eZEE2iNMAsg2fAXdI3qYzyoLCDIyBmuYeXCIsKnJ4P2+8E/YF3GA4WQW1VVDmMN4gjsXVyOLy3FTKclOoPt1DffsEF8OYfynkL7LHtSupCoE6/+/hrJrfW/Poj5SQ6nyvWwHWCOSKvFSON/dw0Ok20exzoXilPa5XBTZxVQgAFhVmkJOWyL76Dtp7JtBwJAIbHLqS7nhM1zVWQTHGUN/WR25aAglbHBZBWn09pOREPrFpZFFRBuW5VjGwzVoqAhsdelvtfVLXNQ4Qd4VgeP7zJI+bHbVt9A16xz9o1cfta6MOdsP2R8OTpIoLLd0D9A16mdOxxZrlNpDTBxBls7g4g7LcFGqaezjcGFAMVlwHqXmjY75Ba5F79Z64KwQAiR4Xq8uz8RrDztp2fL5xGo+TM2G1Q1fSzQ9oV1I1ZfXtfXjcYs0rFGj2+doGNQmLizKYlWPd9h21uE1CsvPttaqfgncocgnGuLgsBADpSR6Wl2bS0TvI3voxeh6MtOEz9ljzYTj2p5Dnpma+Qa+Pxs4+ytytyAGHvhJOv29qTCLCkuKM9xa3GbXsZeXt9q6kHSfhoENX3TgVt4UAoDAjmXkFaZxq7xt/8eyCxTD3QntcG43VFDR09OHzQdmRX4EJuD2ZUQJLZs4KZJEiIiwtyaAk21r28tjpbmtDdgUsutJ+wOYfRzbBGBbXhQBgbn4aBRlJHGrspKV74Mw7O31KO/ActB0PT3JqxjrV3ke6x0fyzl/YN66/DdwJkU9qBhARlpVkUpyVzJHGLqqHi8FZn7bvfPRPcPpQRPOLVXFfCIbXS01N9LDrZDu9A2doPF50FWQGTvxlrMnolJqgvkEvbT2DzGt+BboDeq+4PNZ002rKht/TxVnJHB4uBvMuhtz59p31qgDQQgCAx+1idXkWxhh21LbhHavx2O2Bylvt8a0/1zlM1IQ1dPQBkLfPYQDZ0g9CRnGEM5p5AotBTWuv81XB9kdhYJzbwnFAC4FfaqKH5aVZdPUNse9Mjcdrb7Y+tY3U1WAtXKPUBJxq76Nw4Dju46/bNzr9sVJTMlwMijKTOdTQxYnZ11qD9Ebqb4c9v41OgjFEC8EIBRlJ7zUen2gZ41NCRrFzQ17Vg+FNTs0I3f1DdPYNMaf6V/aN+Yut0bAqZIaLQWFmEgfaXHQtdFj3St+7WggCzc1PIz8jiYMNnbT1jNF4XPkpe+zYq3D6cHiTU9PeqY4+XEO9ZOx/3L6x8nZrNKwKKZdLWFGaRWFmEntLr7PvcHIL1O+IfGIxRAtBgOFPECkJbnbWtjuPPJ57AeQtsMd14Qs1job2PuY1voAETk/iSYHVH49OUnFguBgkVVTSke2wLHpVfL93tRA4SHC7WFWejddn2H3SYeSxiNXFL9D2h2GwLzJJqmmno2+QngEvpYcdpiZZcZ3OKxRmLpewsiyb9uU32zfu+nVcL0OrhWAM6UkelpZk0tYzyOGRoxSHrbkR3EmjY72t9mlvlfJraO8js30viae22jdW3h75hOKQyyXMOv8TeBPSRm8Y6LKKQZwKSSEQkStF5ICIHBaRux223yQiO/2PN0Vk9Yht1SKyS0S2i0hVKPIJleKs5PdmNhzu8vee1FxY8WH7QVt+FpHc1PTT0NHP3Jon7BuKV8GsdZFPKE65UjJxrXK4DVf1oLVoSRwKuhCIiBu4F7gKWAbcICKBN+GOARcaY1YB3wQCF2a92BizxhgTc6tFLCxMJzs1gb31HXT3B0xStf5W+wE1b+hoRWXT3jvIQG8neUd/b99YeZs2EkeYVDrc2j21C+q2RT6ZGBCKK4INwGFjzFFjzADwGDCqj5Yx5k1jTKv/x7exFqmfFlwuYcWsLFwi7KhtY8g7YrbR8o1QsMR+kF4VqABNnX0U1/4Rl23xmTRY8ZHoJBXPSlbBLIfPnXE6PXUoCsEs4MSIn2v9sbF8Chg57Z8BXhCRLSIy5gTsInKHiFSJSFVTU1NQCU9WcoKblbOy6B3wsv/UiDeyCKxzmA5g+yMw1B+5BFXMa+zop6Laocvoig9b05yryHOYysO389fQ79AmOMOFohA4XdM63mgTkYuxCsE/jgifZ4xZh3Vr6bMicoHTscaY+40xlcaYyoKCgmBznrTctETmFaTbB5utvt6h0bgF9v0hsgmqmNXZN4g07SO90aGR2Kn3mYqM5R+2LTjlGuym+Z1HopRQ9ISiENQC5SN+LgPqAncSkVXAj4FNxpjm4bgxps7/tRH4Hdatppg0Jy+VvPREDjV20t7rn1soNReWOYxW1NtDyq+xs5+yow5XA0UrtJE4mpLSYeVHbWHP9l/8ZQrrOBGKQrAZWCgic0UkEbgeGNWHUkQqgN8CNxtjDo6Ip4lIxvD3wBXA7hDkFBbWYLMsEt1udtW2MzjcXuDUaFz9Z2g+EtH8VGxqammjpOZJ+4b1t2ojcbQ53B7KatlJw8Gq0YvbzHBBFwJjzBBwF/A8sA943BizR0TuFJE7/bt9FcgDfhjQTbQIeF1EdgDvAs8YY/4YbE7hlOhxsbIsiwGvlz11/snpZp8LeQvtO29zmGtexZXu/iHSjz6HZ8BhJLHDp1EVYaVrre67ARbW/pajTd2jl72cwUIyjsAY86wxZpExZr4x5lv+2H3GmPv833/aGJPj7yL6XjdRf0+j1f7H8uFjY11WSgILCzM43dlPTXO3f6TxGI3GOj11XGvs7Kf0qMNApeXXQkp25BNSdg5X9LlHfsusdKg+3c2hhpk/4lhHFk9ReW4qhZlJHG7ssianW3U9uAJWlepqgEMvRCdBFRPaa/eR2/SOfYPTguoqOlZ+1DY9tfR3sKT1FcpyU6hp7uHgDC8GWgiCsLTEmpxu18l2BpLzYMnV9p109bK41TvgJWu/03TTi6Di7MgnpJwlZ1o9iALItl+ypDiTijxrdoF99R2YGTryWAtBEBLcLlaUZTHo9bGnrh2z1uFT3qEXoMPWiUrFgaa2Lkqrf2ffsO6T2kgca9Y5TETn7/CxqCiDOflpnGztZU/dzCwGWgiClJlstRc0dw1Qk7UBsspH72B81qykKu4M7P8jSX0Bgx9dCdZtRBVbyjdaV2qB/B0+FhSmM7/QGke0+2SHfUbiaU4LQQgMtxccae6ld8UN9h22/gJ8PntczVgDQz6y9jtMN734KkiP/IBINQ4R53ab7Y+A15pjbG5+GouKMmjo6GOn0/T005gWghBZWpJJcoKb3QUfwAQOtm6rsVYwU3GjtaGa/HqH/3OnKUlUbJhAh4+KvFSWlFg9BredCJh7bBrTQhAiCW4XK2Zl0ZlcRGeZwywZOqYgrvi2PoKYgD8SmWUw/+LoJKTGl15gXbEFCujwUZaTyvJZmbT1DLDtRNtfBpZOY1oIQigrJYEFBRlUlzusi7rvaehpiXxSKuK8Xi9ZBxx6C639BLjckU9ITZzTFduh56GjflSoJCuFlbOy6OwbZEtNK/1DDkvaTiNaCEKsIi8Vs/gqBpIClh309sf1CkjxpPPAq6R2HQ+ICqy9KSr5qEmYf7F15TaS8cEO+0R0hZnJrC7LpnfAy5bqVuf1zacJLQRhsLQsn8a5H7Jv2PqLuF0BKa5sdbgNOO8iyK6IdCZqslxu54K97ZeO79289CTWVmQz4PVRVd1Kz8CQ/dhpQAtBGCR6XGSe67AGbcMuqN8e+YRUxJjeVjKOPWPf4NRPXcWmNQ6FoOUo1LzpuHt2aiLrZ+fgNYbN1a109E2/aWW0EIRJZsUq+orW2jc4fVpUM0bv1sdxewMWJUrJgSXXRCchNXk5s2Huhfb4GTp8ZCQncNacHNwibKlppbV7IIwJhp4WgjBKOutWe3DXb2CwN+K5qMhwbf+lPbjq4+BJssdV7HIaU7Dn99DXbo/7pSZ6qJyTQ5LHxbYTrTR29oUxwdDSQhBGsuLDmIDJrOhvh71POR+gprdTu0hu2mmPr9XbQtPOkmsgOWB22KFe2P3EGQ9LTnBTOTuX9KQEdtW2U9c2PT70aSEIp+RMZPm19riOKZiRBqscFj4vXQvFKyKfjApOQjKs+pg9PoFbu4keF+sqsslJS2RvXQfV02C1My0E4eb0abD6z1bjk5o5BvtwOXUP1quB6cvp/65uKzTsGfdQj9vFmrJsijKTOdzYxcGGzpierC4khUBErhSRAyJyWETudtguIvJ9//adIrJuosdOexVnQ94Ce3x7/C2QPaMdeAZ3f9vomCcZVn4kOvmo4JWsgpLV9vg2h3YgBy6XsGJWJuW51jTWe+pid7K6oAuBiLiBe4GrgGXADSKyLGC3q4CF/scdwI8mcez0JmKNKA1gtj8Cvuk7AEWN5tvq8Mdh2SZIzop8Mip0nK4KdjwGQ/32uAMRYXFxBgv8M5dur43N+YlCcUWwATjsX3ZyAHgM2BSwzybg58byNpAtIiUTPHb6W30DyOipBaTjJBx5JUoJqZBqO44cdfi/1NtC09/Kj4A7oMdXbwsceG5STzMnP41lpZm0dg/E5JQUoSgEs4ATI36u9ccmss9EjgVARO4QkSoRqWpqanLaJXZlFMPCK2xhx8ZFNf1sfxQh4JI/Zw7MPi8q6agQSsmBZR+0x6fQ4aM0O4XV5dn0DHipqm6luz92RiGHohA4LbUUeCNsrH0mcqwVNOZ+Y0ylMaayoGAazufuMFvbXKcAACAASURBVLLUfeg5hjoao5CMChmfD+M0dmDtJ8ClfTFmBIdbuxx+GdprJ/1U+elJrJudg9dn2FzdYq13HgNC8ZtaC4xclqsMCFybcax9JnLszLDwCkgrHBVy+QY5/aZ2JZ3Wjr2KtAVMMCcuWH1jdPJRoTfnAod5ogxsd1h4aAKyUhI4a04uiW4XW4+30tAR/YFnoSgEm4GFIjJXRBKB64HAEVNPAZ/09x46G2g3xtRP8NiZwZ0Aq+1LFKbve4xT02TQiXLg1INk/qWQ5XiHU01HLhescbgq2P7LKa88mJLopnJOLpnJ1sCzmubojjUIuhAYY4aAu4DngX3A48aYPSJyp4jc6d/tWeAocBh4APibMx0bbE4xy6HxML39ILV7X6d3ILYaj9QE9LbCvj/Y4063EtT0tuZGbHeyW6uh5vUpP2Wix8XaihyKMpM51NDF/lMdURtr4AnFkxhjnsX6Yz8ydt+I7w3w2YkeO2MVLLIWyT7xzqhwyZFfs6d0Petn5yDi1GyiYtLOX1vrTIyUmgeLr45OPip8ssth/iVw5OXR8a2/gLkOKxJOkNs/1iA5wUVNcw+9A15WzsrC445s+5K2ZkWaw1VByYln6Ojo4Ng0GIquRtj2c3ts1fXgSYx8Lir8nK709j0FvW32+CSICAuLMlhSkkFL9wBVNZFf5EYLQaQtvxYS00eFXANdLGp+mWOnu2nvmX5zmceluu1wapc9rusOzFxL3g8puaNjQ30hW3mwLCeVNeXZ9A562VzdQntv5P4WaCGItKR0qxgEKD32G5IT3Oyua4/JkYcqgFM/8lmVULg08rmoyPAkWVOKBwrhJJJ56UmcNScXQdha00pjhHoUaSGIBoe5zl3H32RlchN9g172n+qMQlJqwgZ7rfaBQNpIPPM5XfHV74B6h+nHpyg9ycNZc3NIT/aws7Y9IreMtRBEQ9lZkL/YFs7c/yvm5Kdxqr2PU+3R71usxrDvD9a6EiMlpMKK66KTj4qcouVQus4eD/HU8kkeN+srcijOSuZIYxe7T7aHdcI6LQTRMMZEdGx/hHm5SWSlJrDvVId2KY1VWx0aiZd9CJIzI5+Lijynq4Kdj8NgaD+8WbOXZjHfP2FdOBuRtRBEy+obwBXQe7erATn0IitKrRkr99S1x/Qc5nGp+Yi1nkQgbSSOHyuuA0/K6Fhfm/OYkhCYm5/GqvIsuvuH2FzdQkdf6BuRtRBES3qBc3/zbb8gJdHNkuIM2noGqW7uiXxuamxOI4nzFkDFOZHPRUVHcpY1xXggp+7EIVKYkUzlnBwS3C7cYRhrpIUgmtbdYo8dfB466inJSqE4K5mjTV0R7UamzsA75Lyg0Nqbrdt9Kn44LW5/7LWwrjyYkZzAxrm5pCWFZBzwKFoIomn+xZBZNjpmvLDD+mOzuDiDJI+bPSe1S2lMOPwidJ0aHXN5rNt8Kr7MPtd55cEJrl42VeGaeUALQTS53LD2Jnt86y/A5yPB7WJ5aSY9A14ONGiX0qhzaiRedCVkFEU+FxVdIs5XBdsetq4cpxktBNG25ibsk1kdg5o3AMhJS2ROfhr1bX0RG1yiHHTUW7ftAjnd3lPxwbHDxynrynGa0UIQbTmzYd5F9vjWv6xeNi8/jcyUBPbWd0R8DhLlt+MR67bdSBmlsODS6OSjoi+9EBZfZY87XTnGOC0EscDpEnPvU9DTAgz3J87EGNhTF72pauOWz+f85l77Cev2nopfZ+jwMZ1oIYgFS95vTV88krcfdjz23o+piR4WFWfQ2j3A8RbtUhpR1a9Zc88HcmrfUfFl/iWQGbAIkfHC9oejk88UaSGIBZ4k554nWx+CEZ/+Z2WnUJCRxJGmrrAMKlFj2PIze2z+JdYC9Sq+udzOswRsfWjKq5dFQ1CFQERyReRFETnk/5rjsE+5iLwiIvtEZI+IfG7Etn8RkZMist3/iN8VPZwuMZv2w4l3R4WWlmSS4Hax+2Q73jDOPaL8uk/DvqftcW0kVsPWfgJbh4+243D0laikMxXBXhHcDbxsjFkIvOz/OdAQ8AVjzFLgbOCzIrJsxPZ/M8as8T/iY6UyJwWLoOJcezzg02iix8Wykkx6+r0catQupWG3/RHwBVx9pY0xKlzFp+wKWHCZPe50JRmjgi0Em4Dh7i0PAR8K3MEYU2+M2er/vhNrbWJd2dvJeodPmXt+Z1sBKS89idl5qdS29NLU2W8/RoWGMaN6b71nzY26Cpkabf2t9tiBZ6GzIeKpTEWwhaDIGFMP1h98oPBMO4vIHGAtMHLR3rtEZKeIPOh0a2nEsXeISJWIVDU1NQWZdoxatsmax2SkoV7HFZDmF6STnuxhb30H/UPapTQsat6A5sP2uN4WUoEW/RWkBwws9A1Nm0bjcQuBiLwkIrsdHg6zLp3xedKBJ4DPG2M6/OEfAfOBNUA98K9jHW+Mud8YU2mMqSwoKJjMqaePhBRrzdtAVT8d1WgMf5mi1uvzaZfScNnicDUw9wLImx/5XFRscydM60bjcQuBMeYyY8wKh8eTQIOIlAD4vzY6PYeIJGAVgYeNMb8d8dwNxhivMcYHPABsCMU/alpzuj3UuAdqN9vC6UkeFhZm0NI1wImW3ggkF0e6m2Hv7+1xvRpQY3EaD9RaDcdejXgqkxXsraGngOF3xi3Ak4E7iDVL0k+AfcaY7wVsKxnx47XA7iDzmf6KlkP5Rnu86kHH3ctzU8nPSOJwUyed2qU0dHY8At6BUSFfSi4s/UCUElIxL2eO1a040JafRjyVyQq2ENwDXC4ih4DL/T8jIqUiMtwD6DzgZuASh26i3xGRXSKyE7gY+Psg85kZ1t9mj+353XsjjQMtK8nE43Kx+2SHdikNBZ/Puh0XwLX2E9aYD6XG4tRovP8Z6Dxlj8eQoAqBMabZGHOpMWah/2uLP15njLna//3rxhgxxqwK7CZqjLnZGLPSv+2Dww3PcW/5hyA5e3RsqG/USOOREj3WLKXd/UMcbuyKQIIzXPVr0HLEHnd6kys10uKrnRuNt4Z2TeNQ05HFsSghxT8raYAt9kbjYXnpSVTkpXKipUe7lAbL4Tbc4OwLtZFYjc+d4NxWsOVn4Ivd3n1aCGJVpcPtodMH35ue2smCEV1KdZbSKeo8ZV3KB/BsuD0Kyahpad0tIAF/Wjtq4VDsTk+thSBW5S+EOe+zxzf/ZMxDhruU+nyGvfXapXRKtv3CupQfYSi1EFny/iglpKad7HJY+Ff2+BgdPmKBFoJY5nRVsO8PZxytmJ7kYWFROi1dOkvppHmHHMcODK2+2brkV2qiKh2uIA+9AK01kc9lArQQxLIlH7DmtRnJN+g87cEIZTmpOkvpVBz8I7SfGBUy4iJ5o0NBVupMFlxqzUE0ionZrqRaCGKZJ9F5AFPVT8ddF/W9WUprdeH7Cdv8gC3UU3GJdamv1GS43M69zLb+HAZjb8lZLQSxrvI2e8NTZ501odUZJHpcrCjN0oXvJ+r0ITj6J1tYNt4R+VzUzLD2k+AOmJywZ4wR61GmhSDWZZU5T3ns8Ok10MiF70+1x96nkJiy+ce2UG/GbFKXXB6FZNSMkF4Ay2wTMsO790c+l3FoIZgOzvq0PXbsNWg6MO6h8wvSyE5NYN+pDnoHtEupo/4ua92BAN2rbwOXvkVUEDY4XFGe3GI9Yoj+lk8H8y6CvIX2uMOn2EAiVpdSAXadbMenU1DY7fwV9HeMCnndKaSc5TAwSKnJKKuEkjX2+Lvjv3cjSQvBdCACZ33KHt/+CPR12OMBkhPcLCvJpKN3kKOndQqKUYxxLKjN8zeRlpUXhYTUjCICGz5jj+9+wloGNUZoIZguVt8ACWmjYwNdE174ojAzmVk5KVSf7qG5S6egeM+x16Bxry3sW+9wO06pqVhxHaQErLnl7R+3G3gkaSGYLlKyYbXDojXv/NeE5zBZVJRBWpKH3XU6BcV73rnPFmrLX0/egvVRSEbNSAkpzvMPvftj8MbGOB8tBNPJxjvtsdZj1ojFCXC7hFVl1hQUe+radQqKlqNw4DlbuH31p0hw61tDhVDlp5y7ge+1LeESFfrbPp0ULIL5l9rjb/9owk+RluRhcXEGrd2DHD3dHcLkpqF37gdGF8Pe1FJSVzl0+VMqGDmzwWm+qkm8d8MpqEIgIrki8qKIHPJ/dVx8XkSq/QvQbBeRqsker0Y4+6/tsWOvQuO+CT9FaXYKJdnJHGvqpqV7YPwDZqK+Dtj2S1u4ftHN5GWkRiEhNeOd/Tf22MkqOGFfhjbSgr0iuBt42RizEHjZ//NYLvYvSlM5xeMVWFcEeQvscYd73WeypDiT1CQ3u0+2x2d7wfaHYWD0iGuvOwXfuptxuSRKSakZreIcKFltj78T/auCYAvBJmC46fshYLLX1MEeH39cLue2gh2PTao7mtVekI03HtsLfF6rkT1A3ZxrKSosjkJCKi6IOF8V7Pk9tJ+MfD4jBFsIioaXl/R/LRxjPwO8ICJbRGTkULuJHo+I3CEiVSJS1dTUFGTa09zqGyApa3RsqG9CA8xGSk/ysKTEai840hRH7QX7n7Ea2QO0r7ydjGSdblqF0fJr7UtZGi+8a/9gEknjFgIReUlEdjs8Nk3iPOcZY9YBVwGfFZELJpuoMeZ+Y0ylMaayoKBg/ANmsqR0qLzVHn/3fhiY3BoEJVkplGanUH26Oz6WuDQG3vy+LXy6+H3kz10ZhYRUXPEkOU8ZU/WzCQ0ODZdxC4Ex5jJjzAqHx5NAg4iUAPi/No7xHHX+r43A74AN/k0TOl452HgnuDyjYz3NsOPRST/V4uIM0pM97Klrn/nzER1/G2rtjXMnl32GwoykKCSk4s7628Ad8LvW3x7VAWbB3hp6ChieMP8WwNYpVkTSRCRj+HvgCmD3RI9XY8gshZUftcffunfSi2S7XcLqsmwAdtS24Z3J8xE5XA10Zi8lbfHF2kisIiO9ANbcaI+/9UMYik4vvmALwT3A5SJyCLjc/zMiUioiwxPmFwGvi8gO4F3gGWPMH890vJqgc+6yx1qOOA6SGk9KopvlpVl09Q2x/1T0LlHD6vQhx3Ucqhd/mlk5aQ4HKBUm5/4tEPDBo7POmoMoCoIqBMaYZmPMpcaYhf6vLf54nTHmav/3R40xq/2P5caYb413vJqg4hUw/xJ7/M3vW/fCJ6kgI4m5Bdb6BbWtM3C94zd/YAv1pZbiXbaJlER3FBJScStvvvMAsym+d4OlI4unu3P/1h478Q7UvDGlp5uXn0ZeeiIHGzpp65lBg806T1ldbAPULLqVsrzMKCSk4t55n7fHGvfC4ZcinooWgulu3sVQtMIe//O/TunphtcvSPa42Vk7gwabvfWf1oyPIwwlZNC6+OPkpSWOcZBSYVR+ljXILNCfvxfxVLQQTHcicP7f2+NH/nvKqyAluF2sKs/Gaww7a2fAYjY9LbD5QVv4xPwbKS8pQkQbiVWUnPc5e+z4m1A9tSv6qdJCMBMsvxZy59njQXyySE/ysLzUWsxm33RvPH77RzA4esCcz51Mw/LbKclMjlJSSgEL/woKltjjr303omloIZgJXG7nq4L9T09qMrpAhRnJzPM3HldP15lK+9odp5OonfdxSksrtMuoii6XC973RXv86CtQW2WPhyuNiJ1Jhdeq6yFzlj0e5P3GeQXpFGclc7ixi8bOvqCeKyo2/9garDOCz5XAyWWfojRbrwZUDFjxYcidb49H8KpAC8FM4UmEc//OHt/9G2g6GNRTLy3JJCs1gT0nO+joi40VlSZkoNsapBOgbs6HKSybj0cXn1GxwOWG9/2DPX7wj1C/IzIpROQsKjLWfRJS80fHjA9e/XZQTzu8slmC28WOE23TpyfRu/dDz+gZWY24ObH8DspzdM0BFUNWfRyyKuzxV78TkdNrIZhJElOdxxXsfgIa7Au0T0aSx83q8iyGfIZtx9sY9PqCer6w62uH1//dFj5V8X6KKpaQ6NFffRVD3AlwvsO4gv1PQ922sJ9e3w0zzYbPQFrg7KwG/vR/g37qjOQEVpdl0zs4xM7attjuVvr2j6CvbVTIiIuTq/6W2Xk6nYSKQWs/ARml9vh//++wn1oLwUyTmAbnO9xv3PcU1O8M+ulz0xJZVpJFa/cge+o6YnNBm54Wa/K9APWzr6Vk/krc2lNIxSJPElzg0IPo8EtQ82ZYT62FYCaqvB0ySuzxEFwVABRnJbOwKJ2Gjj721XeOf0CkvfkD6B899sHnSuDU2s9RmqU9hVQMW3sz5Myxx1/+ZljnINJCMBMlJMP7vmCPH3gWat4KySlm56UxJz+NurZeDjXEUDHobHBcv/nk3I9SMX+JjiJWsc2TCBf9kz1+/E048nLYTquFYKZa90nIKrfHX/znkH2yWFCYTnluKjXNPRxt6grJcwbtT/8HBkfPnOp1JdJx1ufIT9eFZ9Q0sPKjzqONX/o6+MLTSUMLwUzlSYKL7rbHazfD3t+H7DSLitIpyU7maFN39ItBw17Y+nNbuG7hjcyfvzAKCSk1BS43XPy/7PFTO2GnfQbdkJwyLM+qYsPqG6BwuT3+0tdDthKSiLCsJPO9YnC4MYrF4MWvWuMmRhhMyCDh4i+RnKDrDahpZOkHoWSNPf7yN6yBkiEWVCEQkVwReVFEDvm/5jjss1hEto94dIjI5/3b/kVETo7YdnUw+agALjdc8Q17vPUYVP0kZKcZLgal2SlUn+6OTpvBkf+Gwy/awo1r7qK4uCzy+SgVDBH4q2/Z4531jgssBSvYK4K7gZeNMQuBl/0/j2KMOWCMWWOMWQOsB3qwFrAf9m/D240x9nUEVXAWXOa8itmr37a6WYaIiLC0JOO9NoPdJyM4fbXPCy/8sy3cm1ZG3iUOA+yUmg7mnA9LrrHH3/gP6KgL6amCLQSbgIf83z8EfGic/S8FjhhjaoI8r5qMy7+JbX3U3lZ4+eshPY2IsLg4g/mF6Zxq72PbiQiNQK56EBp228JDF3+V5BQdPKamscu/Aa6E0bHBnpAPMgu2EBQZY+oB/F8Lx9n/euDRgNhdIrJTRB50urU0TETuEJEqEalqamoKLut4U7wC1txkj295CGqntnjNmczNT2P5rEzaegaoqm6lZ2Ao5Od4T1ej1cc6QH/xejLWfyx851UqEvLmw4Y77PHtj0Dd9pCdZtxCICIvichuh8emyZxIRBKBDwK/HhH+ETAfWAPUA2Our2iMud8YU2mMqSwoCJxCQY3r0q9CUuDavAae+Qfr1kqIlWSlsLYih/4hL+8ca6GxI0xTWL/wz7ZppgGS3n+PdZ9Vqenuwi9BSuBnZGM1HIfIuIXAGHOZMWaFw+NJoEFESgD8XxvP8FRXAVuNMQ0jnrvBGOM1xviAB4ANwf1z1JgyiuDiL9vj9dthy0/DcsrctETOnpdHWqKHnbXtHGzoDG27wbE/O3anM2s+AeX6q6RmiJQc+yCzlR+DD34/ZKcI9tbQU8At/u9vAZ48w743EHBbaLiI+F0L2G/0qtA569NQtNIef/kb0HkqLKdMTnBTOTuH8txUjjf38PbRZlq7Q9B1dWgAnrGPnjYpOcjlofukpFRMqLwd8hZA+dnw6f+G6x6ArND1hgu2ENwDXC4ih4DL/T8jIqUi8l4PIBFJ9W//bcDx3xGRXSKyE7gYcFhvUYWM2wPv/3/2eF87/OHzYZvLxOWyGpHXVmRjgC01reyt6whuXYPXvgunD9jCctnXIS1v6s+rVCxyJ8Ctz8Ltf4Sy9SF/eonJ2SPHUVlZaaqqIree54zz+7+B7Q/b49f+F6y+Pqyn9voMx053UdPcg4jVljAnL42UxIkP+PKe2ILrwcsRE1BIys6C21+w1oFVStmIyBZjTGVgXN8x8ejybzqsWQA89z9D3j85kNslLCjM4Nz5+ZRkpVDf3subR06zpaaV48099A44XyUYY2jpHmD/iQb6fv0ZexEQN7z/e1oElJoCT7QTUFGQlgfX/Bv86hOj433t8NTfwU2/DnuPm5REN0tLMpmbn0Ztay+NnX0cbOjkYEMnbreQ7HGTnODCZwz9gz76h3x4fYZFO+4hreOI/Qkv+BKUrAprzkrNVPrxKV4t/YA1y2Ggwy86LuoSLskJbhYUpnPu/HzOXZDHoqIMSrNSSE10MzBkDUbLTElgVk4K6307KD/g0MOpZLXzgh5KqQnRK4J4dtV34Nhr0NUwOv7S16z77RUbI5pOaqKHirwxfiXbT8KzdwIBbVruRKttw53geJhSanx6RRDPUnPhA/9hj/uG4Ne3QvfpiKfkaGgAfn0L9DTbt13yFShcGvmclJpBtBDEu8VXwca/tsc76+C3nwFvGKeHmKgXvmKtoxBozvvgnLsin49SM4wWAmVNbFV2lj1+5L/huS+Fda3UcW35Gbz7X/Z4Rgl85EFrqm2lVFC0EChrndSP/NRhPhOsmT1f/17kcwLY9zQ87TDG0OWBjz4E6ePNcaiUmggtBMqSXQ4ffgDbdNVgTUGxIzxL5I2p+g34ze22FccAuOJbEW/IVmom00Kg/mLh5c6rIoE1Gnnn45HJo3YLPHoDePvt21bfABv/R2TyUCpOaCFQo53zWecGWOOF394B7z4Q3vMffgkeusZxamkWXA4f/IFOL61UiGkhUHaXfxOWf9hhg4Fnvwivfic8Dci7fgOPfNxagSnQrPXwsYd0vIBSYaCFQNm5XHDtfTDvYuftr3zLunUTqjWPh/rhj/8LnviUNYYhUN5CuPHXkKjLTioVDloIlDNPEtzwmPPi2QAHn4MfnWctDhOMxv3wwCXw9hjTWhSthFuf1qmllQojLQRqbAnJVjfN1Tc4b++ss+7nP3qj9Qd9MtpPWgvL3He+48LzAMw+D257BjKKJ/fcSqlJ0bmG1Jm5PbDph5CWD2/+wHmfA89YVwiLr4bl18Kiv4KkDPt+g33W3Eb7/wA7fuXcK2jYkmvgup9YxUgpFVZBFQIR+SjwL8BSYIMxxnG1GBG5EvgPwA382BgzvJJZLvArYA5QDXzMGNMaTE4qDFwuuOJ/W8vkPflZ6Guz72N8sP9p6+FOtJbVyyqD1HzobYGuRmja79wQPOpcCXDZ1+Dsz+raAkpFSLBXBLuBDwMOcwBYRMQN3Iu1VGUtsFlEnjLG7AXuBl42xtwjInf7f/7HIHNS4bL0GmvK5yc+DSfeHns/7wA07rUek5G/yLoK0HUFlIqooD5yGWP2GWPsC8eOtgE4bIw5aowZAB4DNvm3bQIe8n//EPChYPJREZBdDrc9C5vuhcwQLZ6dlAkX3g13vKpFQKkoiEQbwSzgxIifa4Hh+QGKjDH1AMaYehEZc/IYEbkDuAOgoqIiTKmqCXG5Ye0nYMVHYPOP4Y1/h+6myT9PYro1Svicu6wpsZVSUTFuIRCRlwCnbhtfNsY8OYFzOA0DnfRoJGPM/cD9YC1eP9njVRgkJMO5d8HGO6Hmddj7lNVGELjQzUgpubDwClh8Jcy/FJIzI5evUsrRuIXAGHNZkOeoBcpH/FwGDK+Q3iAiJf6rgRKgMchzqWhwe2DeRdbj/f9qNQx31EJ7rTXoLDUX0ous2UKzZ+vU0UrFmEjcGtoMLBSRucBJ4HrgRv+2p4BbgHv8XydyhaFimQhkFFmPWeujnY1SagKCaiwWkWtFpBY4B3hGRJ73x0tF5FkAY8wQcBfwPLAPeNwYs8f/FPcAl4vIIaxeRfcEk49SSqnJExPN1aemqLKy0lRVOQ5ZUEopNQYR2WKMqQyM64gdpZSKc1oIlFIqzmkhUEqpOKeFQCml4ty0bCwWkSagZoqH5wOnQ5hOqGhek6N5TY7mNTmxmhcEl9tsY0xBYHBaFoJgiEiVU6t5tGlek6N5TY7mNTmxmheEJze9NaSUUnFOC4FSSsW5eCwE90c7gTFoXpOjeU2O5jU5sZoXhCG3uGsjUEopNVo8XhEopZQaQQuBUkrFuRlfCETkuyKyX0R2isjvRCR7jP2uFJEDInLYv35yuPP6qIjsERGfiIzZFUxEqkVkl4hsF5Gwz7Q3ibwi/XrlisiLInLI/zVnjP0i8nqN9+8Xy/f923eKyLpw5TLJvC4SkXb/67NdRL4aobweFJFGEdk9xvZovV7j5RXx10tEykXkFRHZ538vfs5hn9C+XsaYGf0ArgA8/u+/DXzbYR83cASYByQCO4BlYc5rKbAY+BNQeYb9qoH8CL5e4+YVpdfrO8Dd/u/vdvp/jNTrNZF/P3A18BzWCn1nA+9E4P9uInldBDwdqd+nEee9AFgH7B5je8RfrwnmFfHXCygB1vm/zwAOhvv3a8ZfERhjXjDWmggAb2OtkBZoA3DYGHPUGDMAPAZsCnNe+4wxB8J5jqmYYF4Rf738z/+Q//uHgA+F+XxnMpF//ybg58byNpDtX4Uv2nlFhTHmNaDlDLtE4/WaSF4RZ4ypN8Zs9X/fibWOy6yA3UL6es34QhDgdqwqGmgWcGLEz7XYX/hoMcALIrJFRO6IdjJ+0Xi9iowx9WC9UYDCMfaLxOs1kX9/NF6jiZ7zHBHZISLPicjyMOc0UbH8Hoza6yUic4C1wDsBm0L6ekViqcqwE5GXgGKHTV82xjzp3+fLwBDwsNNTOMSC7lc7kbwm4DxjTJ2IFAIvish+/6eYaOYV8ddrEk8T8tfLwUT+/WF5jcYxkXNuxZpvpktErgZ+DywMc14TEY3XayKi9nqJSDrwBPB5Y0xH4GaHQ6b8es2IQmCMuexM20XkFuAa4FLjv8EWoBYoH/FzGVAX7rwm+Bx1/q+NIvI7rMv/oP6whSCviL9eItIgIiXGmHr/JXDjGM8R8tfLwUT+/WF5jYLNa+QfFGPMsyLyQxHJN8ZEe4K1aLxe44rW6yUiCVhF4GFjzG8ddgnp6zXjbw2JyJXAPwIfNMb0jLHbZmChiMwVkUTgeuCpSOU4FhFJE5GM4e+xGr4dezdEWDReNrR0YgAAAQdJREFUr6eAW/zf3wLYrlwi+HpN5N//FPBJf++Os4H24VtbYTRuXiJSLCLi/34D1t+A5jDnNRHReL3GFY3Xy3++nwD7jDHfG2O30L5ekWwNj8YDOIx1L227/3GfP14KPDtiv6uxWuePYN0iCXde12JV9X6gAXg+MC+s3h87/I89sZJXlF6vPOBl4JD/a240Xy+nfz9wJ3Cn/3sB7vVv38UZeoZFOK+7/K/NDqzOE+dGKK9HgXpg0P/79akYeb3GyyvirxdwPtZtnp0j/m5dHc7XS6eYUEqpODfjbw0ppZQ6My0ESikV57QQKKVUnNNCoJRScU4LgVJKxTktBEopFee0ECilVJz7/6fy0/GF0zywAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "with torch.no_grad():\n",
    "    x0 = torch.linspace(-2, 2)\n",
    "    pyplot.plot(x0, model(x0[:, None]).detach(), alpha=0.3)\n",
    "    pyplot.plot(x0, torch.sin(2 * x0), linewidth=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# PyTorch has ... utilities for data handling\n",
    "\n",
    "PyTorch splits data handling in two parts:\n",
    "- Problem specific (DataSet)\n",
    "- Generic, e.g. Sampling, Batching (DataLoader)\n",
    "\n",
    "User's responsibility: Define `torch.utils.data.DataSet` subclass with\n",
    "- a `__len__` method (how many data points)\n",
    "- a `__getitem__` method to get the `i`th element, returning tuples of numbers and tensors\n",
    "\n",
    "`DataLoader` is more or less universal. In several parallel processes, get items from dataset and assemble\n",
    "them into batches (for numbers, tensors, make tensors, for other things, lists)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's try that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 9.5571,  7.1095,  7.1393,  7.4666,  6.7035],\n",
      "        [ 5.3171,  9.0150,  6.7444,  7.5128,  5.4807],\n",
      "        [ 9.7531,  7.9709,  8.2818, 10.1165, 10.7010],\n",
      "        [-1.1085,  1.3127,  0.5249,  1.1068, -0.7207],\n",
      "        [ 1.0284,  0.7190,  3.0181,  3.5250,  3.3695],\n",
      "        [ 3.6932,  0.9453,  1.3540,  0.8289,  0.6140],\n",
      "        [ 5.3443,  3.6160,  5.4480,  5.8480,  4.3890],\n",
      "        [ 1.3029,  1.7199,  0.7438,  0.8375,  2.5249]]) tensor([7, 7, 9, 0, 2, 1, 5, 1])\n"
     ]
    }
   ],
   "source": [
    "import torch.utils.data\n",
    "class DS(torch.utils.data.Dataset):\n",
    "    def __init__(self, shape):\n",
    "        self.shape = shape\n",
    "    def __len__(self):\n",
    "        return 100\n",
    "    def __getitem__(self, i):\n",
    "        return (i % 10 + torch.randn(self.shape), i % 10) # observation and label\n",
    "    \n",
    "ds = DS((5,))\n",
    "dl = torch.utils.data.DataLoader(ds, batch_size=8, shuffle=True)\n",
    "\n",
    "for inp, label in dl:\n",
    "    print(inp, label)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What happened here?\n",
    "\n",
    "The DataLoader provides an iterator and for each iteration\n",
    "\n",
    "- requested elements from the dataset,\n",
    "- took the 1d `5`-tensors and `stack`ed them to a 2d `batchsize x 5` tensor,\n",
    "- took the integers and put them into a 1d `batchsize` tensor.\n",
    "- gives the tuple of input and label to the for loop.\n",
    "\n",
    "**Good to know**: If `batchsize` does not divide `len(ds)`, the last batch will be shorter. This can be troublesome (e.g. batch norm running with a batch size of 1), so you have the `drop_last` optional command to just skip this minibatch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Performance of the DataLoader\n",
    "\n",
    "Using `Dataset` / `DataLoader` is simple, but keep in mind that it can greatly impact performance.\n",
    "\n",
    "How do you detect this? If, during training, your GPU is not constantly fully used (shown by `*smi` tools), you have a CPU bottleneck somewhere, and the data loading is one thing to check.\n",
    "\n",
    "- If more cores help, there is the `num_workers` argument to the dataloader.\n",
    "- Get a SSD.\n",
    "- If you can do preprocessing beforehand (no augmentation), try that. (But careful, I/O is often more expensive than processing.)\n",
    "- If you can do preprocessing on the GPU instead (augmentation in particular), do that.\n",
    "- Sometimes it helps to put tensors into _GPU pinned memory_, with the `pin_memory` argument for the DataLoader.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Excursion: GPU, efficiency, measurement\n",
    "\n",
    "One of my informal mottos is _It's not optimization until you measure_. I'll only discuss the most basic measurement here, PyTorch offers a capable profiling facility, too.\n",
    "\n",
    "When you think about code being slow, it's important to figure out what is slow and why.\n",
    "To my mind, a lot of measurement can be done with very basic tools, e.g. IPython's `%timeit` magic.\n",
    "\n",
    "As GPU computation is and shold be ansynchronous, avoid unneeded synchronization points. Synchronization happens when the CPU waits for the GPU (to get the results).\n",
    "- Synchronizations can happen because the program needs to know something (e.g. sizes of tensors depending on\n",
    "    the input). Often, these are unavoidable.\n",
    "- Typical sources of spurious synchronizations are too frequent \n",
    "    `.to(device=\"cpu\")`, `.item()`, `.to_list()`, `print`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If we want to time GPU kernels, we want to be sure to synchronize before taking the start and end times.\n",
    "Typically, we also want to have some \"warm-up\", i.e. run the measured function before timing.\n",
    "\n",
    "Let's take the uniformity loss from [Wang and Isola: Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://arxiv.org/abs/2005.10242) (a great paper!).\n",
    "\n",
    "The Uniformity loss is defined as a function of the pairwise distances over a largish set of vectors.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(-3.9367, device='cuda:0', grad_fn=<LogBackward>)"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def lunif(x, t=2): # copied from the paper\n",
    "    sq_pdist = torch.pdist(x, p=2).pow(2)\n",
    "    return sq_pdist.mul(-t).exp().mean().log()\n",
    "\n",
    "x = torch.randn(1024, 128, device=\"cuda\")\n",
    "x /= x.norm(p=2, dim=1, keepdim=True).requires_grad_()\n",
    "\n",
    "lunif(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "One would think that the specialised `pdist` function is the right tool for the job.\n",
    "But is it? Let's time it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "19.4 ms ± 103 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
     ]
    }
   ],
   "source": [
    "def totime(fn):\n",
    "    l = fn(x)\n",
    "    g, = torch.autograd.grad(l, x)\n",
    "    torch.cuda.synchronize()\n",
    "\n",
    "totime(lunif) # warmup\n",
    "%timeit totime(lunif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's use $|x-y|^2 = |x|^2 + |y|^2 - 2 <x, y>$ and compare."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9.313225746154785e-10\n",
      "2.26 ms ± 16.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
     ]
    }
   ],
   "source": [
    "def lunif2(x, t=2):\n",
    "    t=2\n",
    "    xnorm = torch.norm(x, p=2, dim=1).pow(2)\n",
    "    sq_pdist = xnorm[None] + xnorm[:, None] - 2 * torch.mm(x, x.t())\n",
    "    exp = sq_pdist.mul(-t).exp().tril(diagonal=-1)\n",
    "    N = x.size(0)\n",
    "    res = exp.sum().mul(2/(N*N-N)).log()\n",
    "    return res\n",
    "\n",
    "print((lunif2(x.to(torch.double)) - lunif(x.to(torch.double))).item())\n",
    "\n",
    "totime(lunif2)\n",
    "%timeit totime(lunif2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Even though we have stark inefficiencies (like taking tril and taking a copy to do so), this is almost an order of magnitude faster!\n",
    "\n",
    "Largely due to backward of `pdist` implementation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## But Python is slow...\n",
    "\n",
    "Uniformity loss: \"what formulas you use\" is the real bottleneck (unless you optimize pdist).\n",
    "\n",
    "The \"what\" typically should be the first optimization target.\n",
    "\n",
    "But when we fix the task (\"what\"), how can we optimize?\n",
    "\n",
    "Conventional wisdom: **Python is slow**\n",
    "\n",
    "- certainly, Python isn't fast (`for` loop vs C++ `for` loop)\n",
    "- but, if the GPU is saturated $\\Rightarrow$ Python isn't the bottleneck\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### How PyTorch programs spend their time\n",
    "\n",
    "At a very high level, you can divide time spent into these parts:\n",
    "- Python program flow,\n",
    "- Data \"administrative overhead\" (creating `Tensor` data structures, autograd `Node`s etc.),\n",
    "- Data aquisition (I/O),\n",
    "- Computation roughly as\n",
    "  - fixed overhead (kernel launches etc.),\n",
    "  - reading / writing memory,\n",
    "  - \"real computation\".\n",
    "\n",
    "**Thomas' rule of thumb**: As long as your operands are reasonably large (say 100s of elements, not single elements), Python and Data \"administrative overhead\" probably isn't your main problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Optimization potentials\n",
    "\n",
    "remember that modelling does have a huge lever regarding efficiency\n",
    "\n",
    "<table>\n",
    "    <tr>\n",
    "        <td>\n",
    "Millions of parameters vs. timeline for the popular transformer models\n",
    "<img src=\"distilbert.png\" width=\"80%\" />\n",
    "        </td>\n",
    "        <td><img src=\"efficientnet_flops.png\" width=\"80%\" /></td>\n",
    "    </tr>\n",
    "    <tr><td>\n",
    "Source: V. Sanh (Huggingface), DistilBERT\n",
    "        </td><td>Source: M. Tan, Q. Le: EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks</td></tr>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The PyTorch JIT\n",
    "\n",
    "So if we have our model in PyTorch. Can we export it?\n",
    "\n",
    "This is where the PyTorch Just-In-Time compiler comes into play. It provides a graphical representation \n",
    "called _TorchScript_  (used somewhat synonymously for the language and the representation).\n",
    "\n",
    "There are two main ways to get a TorchScript graph:\n",
    "- scripting and\n",
    "- tracing.\n",
    "\n",
    "We'll look at these in some detail."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Scripting\n",
    "\n",
    "Scripting compiles (mostly) a subset of Python.\n",
    "It takes the Python source code and transforms it. \n",
    "\"Here is what the function should do\", just like normal programming.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "@torch.jit.script\n",
    "def fn(x):\n",
    "    return x * 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<torch.jit.ScriptFunction at 0x7f65d43b2220>,\n",
       " graph(%x.1 : Tensor):\n",
       "   %2 : int = prim::Constant[value=2]() # <ipython-input-57-6dcc6c3c1c8e>:3:15\n",
       "   %3 : Tensor = aten::mul(%x.1, %2) # <ipython-input-57-6dcc6c3c1c8e>:3:11\n",
       "   return (%3))"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fn, fn.graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Tracing\n",
    "\n",
    "Tracing runs the code and observers the calls into PyTorch with some sample input.\n",
    "\"Watch me, now you know how to do the same.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "graph(%x : Float(5:1)):\n",
      "  %1 : Long() = prim::Constant[value={2}]() # <ipython-input-59-7d43d0af2e80>:2:0\n",
      "  %2 : Float(5:1) = aten::mul(%x, %1) # <ipython-input-59-7d43d0af2e80>:2:0\n",
      "  return (%2)\n",
      " def fn(x: Tensor) -> Tensor:\n",
      "  return torch.mul(x, CONSTANTS.c0)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def fn(x):\n",
    "    return x * 2\n",
    "fn = torch.jit.trace(fn, [torch.randn(5)])\n",
    "\n",
    "print(fn.graph, fn.code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "N.B.: The specialization for the Tensor shape isn't relevant here and will be erased e.g. during saving of the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### TorchScript\n",
    "\n",
    "One important difference between TorchScript and Python is that in TorchScript everything is typed. Important\n",
    "types are\n",
    "- `bool`, `int`, `long`, `double` for numbers (int = 32 bit integer, long = 64 bit integer)\n",
    "- `Tensor` for tensors (of arbitrary shape, dtype, ...)\n",
    "- `List[T]` a list with elements fo type T (one of the above)\n",
    "- Tuples are of fixed size with arbitrary but fixed element type, so e.g. `Tuple(Tensor, int)`.\n",
    "- `Optional[T]` for things that can be `None`\n",
    "\n",
    "`None` always is of type `Optional[T]` for some specific `T` (except in the rarest circumstances).\n",
    "\n",
    "PyTorch will mostly infer the intermediate and return types, but you need to annotate any non-Tensor inputs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Tracing vs. Scripting\n",
    "\n",
    "Scripting will process all code but may not understand all. This means it captures all constructs (like control flow) it understands, but it will fail if it doesn't understand something.\n",
    "\n",
    "Tracing doesn't see anything not calling into PyTorch and will happily ignore that (e.g. control flow). This is also the reason why it will loudly complain if you have non-tensor inputs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def fn(x):\n",
    "    for i in range(x.dim()):\n",
    "        x = x * x\n",
    "    return x\n",
    "\n",
    "script_fn = torch.jit.script(fn)\n",
    "trace_fn = torch.jit.trace(fn, [torch.randn(5, 5)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "def fn(x: Tensor) -> Tensor:\n",
      "  x0 = x\n",
      "  for i in range(torch.dim(x)):\n",
      "    x0 = torch.mul(x0, x0)\n",
      "  return x0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(script_fn.code)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "def fn(x: Tensor) -> Tensor:\n",
      "  x0 = torch.mul(x, x)\n",
      "  return torch.mul(x0, x0)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(trace_fn.code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Tracing and Scripting Modules\n",
    "\n",
    "But our models often are not functions. What now?\n",
    "\n",
    "With tracing, we can work just like with functions. We get a `ScriptModule` subclass that behaves much like a\n",
    "`Module` with parameters, state dict etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.jit._trace.TopLevelTracedModule,\n",
       " Sequential(\n",
       "   original_name=Sequential\n",
       "   (0): Linear(original_name=Linear)\n",
       "   (1): Tanh(original_name=Tanh)\n",
       "   (2): Linear(original_name=Linear)\n",
       " ))"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "traced_model = torch.jit.trace(model, [torch.randn(8, 1)])\n",
    "type(traced_model), traced_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Saving is a bit different, here we include the model on purpose:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.8112],\n",
       "        [ 0.3232],\n",
       "        [ 0.7830],\n",
       "        [-0.8254],\n",
       "        [ 0.7857],\n",
       "        [-0.3138],\n",
       "        [-0.7612],\n",
       "        [ 0.5701]], grad_fn=<DifferentiableGraphBackward>)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "traced_model.save('./traced_model.pt')\n",
    "loaded_model = torch.jit.load('./traced_model.pt')\n",
    "\n",
    "loaded_model(torch.randn(8,1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Scripting Modules\n",
    "\n",
    "Scripting modules is ... a bit tricky. We don't script the class in its entirety but instead take an instance (in particular past `__init__`) and process its data members and methods (the latters work like script functions)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "def forward(self,\n",
      "    input: Tensor) -> Tensor:\n",
      "  _0 = getattr(self, \"0\")\n",
      "  _1 = getattr(self, \"1\")\n",
      "  _2 = getattr(self, \"2\")\n",
      "  input0 = (_0).forward(input, )\n",
      "  input1 = (_1).forward(input0, )\n",
      "  return (_2).forward(input1, )\n",
      "\n"
     ]
    }
   ],
   "source": [
    "scripted_model = torch.jit.script(model)\n",
    "print(scripted_model.code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can also look at the graph including submodules, but it gets unwieldy rather fast:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "graph(%self : __torch__.torch.nn.modules.container.___torch_mangle_13.Sequential,\n",
       "      %input.1 : Tensor):\n",
       "  %2 : __torch__.torch.nn.modules.linear.___torch_mangle_10.Linear = prim::GetAttr[name=\"0\"](%self)\n",
       "  %3 : __torch__.torch.nn.modules.activation.___torch_mangle_11.Tanh = prim::GetAttr[name=\"1\"](%self)\n",
       "  %4 : __torch__.torch.nn.modules.linear.___torch_mangle_12.Linear = prim::GetAttr[name=\"2\"](%self)\n",
       "  %8 : int = prim::Constant[value=1]()\n",
       "  %9 : int = prim::Constant[value=2]() # /usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:1672:22\n",
       "  %10 : Tensor = prim::GetAttr[name=\"weight\"](%2)\n",
       "  %11 : Tensor = prim::GetAttr[name=\"bias\"](%2)\n",
       "  %12 : int = aten::dim(%input.1) # /usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:1672:7\n",
       "  %13 : bool = aten::eq(%12, %9) # /usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:1672:7\n",
       "  %input.3 : Tensor = prim::If(%13) # /usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:1672:4\n",
       "    block0():\n",
       "      %15 : Tensor = aten::t(%10) # /usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:1674:39\n",
       "      %ret.2 : Tensor = aten::addmm(%11, %input.1, %15, %8, %8) # /usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:1674:14\n",
       "      -> (%ret.2)\n",
       "    block1():\n",
       "      %17 : Tensor = aten::t(%10) # /usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:1676:30\n",
       "      %output.2 : Tensor = aten::matmul(%input.1, %17) # /usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:1676:17\n",
       "      %output.4 : Tensor = aten::add_(%output.2, %11, %8) # /usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:1678:12\n",
       "      -> (%output.4)\n",
       "  %input.5 : Tensor = aten::tanh(%input.3) # /usr/local/lib/python3.8/dist-packages/torch/nn/modules/activation.py:350:15\n",
       "  %21 : int = prim::Constant[value=1]()\n",
       "  %22 : int = prim::Constant[value=2]() # /usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:1672:22\n",
       "  %23 : Tensor = prim::GetAttr[name=\"weight\"](%4)\n",
       "  %24 : Tensor = prim::GetAttr[name=\"bias\"](%4)\n",
       "  %25 : int = aten::dim(%input.5) # /usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:1672:7\n",
       "  %26 : bool = aten::eq(%25, %22) # /usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:1672:7\n",
       "  %input.7 : Tensor = prim::If(%26) # /usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:1672:4\n",
       "    block0():\n",
       "      %28 : Tensor = aten::t(%23) # /usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:1674:39\n",
       "      %ret.1 : Tensor = aten::addmm(%24, %input.5, %28, %21, %21) # /usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:1674:14\n",
       "      -> (%ret.1)\n",
       "    block1():\n",
       "      %30 : Tensor = aten::t(%23) # /usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:1676:30\n",
       "      %output.1 : Tensor = aten::matmul(%input.5, %30) # /usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:1676:17\n",
       "      %output.3 : Tensor = aten::add_(%output.1, %24, %21) # /usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:1678:12\n",
       "      -> (%output.3)\n",
       "  return (%input.7)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scripted_model.forward.inlined_graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What can you do with scripted modules?\n",
    "\n",
    "- Run them as is, bypassing Python.\n",
    "  - not as much speedup as often is expected (maybe 5%-10% for some models I tested),\n",
    "  - but - sometimes crucially - it avoids the dreaded Python Global Interpreter Lock (GIL), so it is useful e.g.\n",
    "    for multithreaded things like serving PyTorch models.\n",
    "- Export and run in C++ / Mobile / ..., export to other frameworks like [TVM](https://tvm.ai/).\n",
    "- Apply holistic optimizations (this is what a submodule, the JIT fuser does)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Holistic Optimizations - JIT fusers\n",
    "\n",
    "So currently the fuser is a hotspot of development, look at the competition. We'll use `fuser0` here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function fuser in module torch.jit:\n",
      "\n",
      "fuser(name)\n",
      "    A context manager that facilitates switching between\n",
      "    backend fusers.\n",
      "    \n",
      "    Valid names:\n",
      "    * ``fuser0`` - enables only legacy fuser\n",
      "    * ``fuser1`` - enables only NNC\n",
      "    * ``fuser2`` - enables only nvFuser\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(torch.jit.fuser)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Example: Intersection over Union loss\n",
    "\n",
    "\n",
    "<img src=\"iou.svg\" style=\"float: right;\" />\n",
    "\n",
    "...for detection models.\n",
    "\n",
    "Many elementwise operations. Usually PyTorch executes them one by one, reading input from memory and writing back outputs.\n",
    "\n",
    "By fusing this into a single GPU kernel, we can save all that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def ratio_iou(x1, y1, w1, h1, x2, y2, w2, h2):\n",
    "    xi = torch.max(x1, x2)                                # Intersection (yi similarly)\n",
    "    yi = torch.max(y1, y2)                                # Intersection (yi similarly)\n",
    "    wi = torch.clamp(torch.min(x1+w1, x2+w2) - xi, min=0)\n",
    "    hi = torch.clamp(torch.min(y1+h1, y2+h2) - yi, min=0)\n",
    "    area_i = wi * hi                                      # Area Intersection\n",
    "    area_u = w1 * h1 + w2 * h2 - wi * hi                  # Area Union\n",
    "    return area_i / torch.clamp(area_u, min=1e-5)\n",
    "\n",
    "ratio_iou_scripted = torch.jit.script(ratio_iou)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This gives us a good speedup:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "238 µs ± 998 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n",
      "40.3 µs ± 113 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n"
     ]
    }
   ],
   "source": [
    "x1, y1, w1, h1, x2, y2, w2, h2 = torch.randn(8, 100, 1000, device='cuda').exp()\n",
    "\n",
    "def take_time(fn):\n",
    "    _ = fn(x1, y1, w1, h1, x2, y2, w2, h2)\n",
    "    torch.cuda.synchronize()\n",
    "\n",
    "take_time(ratio_iou) # warmup\n",
    "%timeit take_time(ratio_iou)\n",
    "\n",
    "\n",
    "take_time(ratio_iou_scripted)\n",
    "%timeit take_time(ratio_iou_scripted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can see in the graph specialised for the inputs which operations are fused:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "graph(%x1.1 : Float(*, *),\n",
       "      %y1.1 : Float(*, *),\n",
       "      %w1.1 : Float(*, *),\n",
       "      %h1.1 : Float(*, *),\n",
       "      %x2.1 : Float(*, *),\n",
       "      %y2.1 : Float(*, *),\n",
       "      %w2.1 : Float(*, *),\n",
       "      %h2.1 : Float(*, *)):\n",
       "  %32 : Float(*, *) = prim::FusionGroup_0(%w2.1, %h2.1, %w1.1, %h1.1, %y2.1, %y1.1, %x2.1, %x1.1)\n",
       "  return (%32)\n",
       "with prim::FusionGroup_0 = graph(%14 : Float(*, *),\n",
       "      %15 : Float(*, *),\n",
       "      %17 : Float(*, *),\n",
       "      %18 : Float(*, *),\n",
       "      %34 : Float(*, *),\n",
       "      %37 : Float(*, *),\n",
       "      %51 : Float(*, *),\n",
       "      %54 : Float(*, *)):\n",
       "  %4 : float = prim::Constant[value=1.0000000000000001e-05]() # <ipython-input-68-5fb00890f03c>:8:44\n",
       "  %42 : None = prim::Constant()\n",
       "  %41 : int = prim::Constant[value=0]() # <ipython-input-68-5fb00890f03c>:4:55\n",
       "  %55 : int = prim::Constant[value=1]()\n",
       "  %xi.1 : Float(*, *) = aten::max(%54, %51) # <ipython-input-68-5fb00890f03c>:2:9\n",
       "  %yi.1 : Float(*, *) = aten::max(%37, %34) # <ipython-input-68-5fb00890f03c>:3:9\n",
       "  %56 : Float(*, *) = aten::add(%54, %17, %55) # <ipython-input-68-5fb00890f03c>:4:31\n",
       "  %53 : Float(*, *) = aten::add(%51, %14, %55) # <ipython-input-68-5fb00890f03c>:4:38\n",
       "  %50 : Float(*, *) = aten::min(%56, %53) # <ipython-input-68-5fb00890f03c>:4:21\n",
       "  %47 : Float(*, *) = aten::sub(%50, %xi.1, %55) # <ipython-input-68-5fb00890f03c>:4:21\n",
       "  %wi.1 : Float(*, *) = aten::clamp(%47, %41, %42) # <ipython-input-68-5fb00890f03c>:4:9\n",
       "  %39 : Float(*, *) = aten::add(%37, %18, %55) # <ipython-input-68-5fb00890f03c>:5:31\n",
       "  %36 : Float(*, *) = aten::add(%34, %15, %55) # <ipython-input-68-5fb00890f03c>:5:38\n",
       "  %33 : Float(*, *) = aten::min(%39, %36) # <ipython-input-68-5fb00890f03c>:5:21\n",
       "  %30 : Float(*, *) = aten::sub(%33, %yi.1, %55) # <ipython-input-68-5fb00890f03c>:5:21\n",
       "  %hi.1 : Float(*, *) = aten::clamp(%30, %41, %42) # <ipython-input-68-5fb00890f03c>:5:9\n",
       "  %area_i.1 : Float(*, *) = aten::mul(%wi.1, %hi.1) # <ipython-input-68-5fb00890f03c>:6:13\n",
       "  %19 : Float(*, *) = aten::mul(%17, %18) # <ipython-input-68-5fb00890f03c>:7:13\n",
       "  %16 : Float(*, *) = aten::mul(%14, %15) # <ipython-input-68-5fb00890f03c>:7:23\n",
       "  %13 : Float(*, *) = aten::add(%19, %16, %55) # <ipython-input-68-5fb00890f03c>:7:13\n",
       "  %area_u.1 : Float(*, *) = aten::sub(%13, %area_i.1, %55) # <ipython-input-68-5fb00890f03c>:7:13\n",
       "  %6 : Float(*, *) = aten::clamp(%area_u.1, %4, %42) # <ipython-input-68-5fb00890f03c>:8:20\n",
       "  %2 : Float(*, *) = aten::div(%area_i.1, %6) # <ipython-input-68-5fb00890f03c>:8:11\n",
       "  return (%2)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratio_iou_scripted.graph_for(x1, y1, w1, h1, x2, y2, w2, h2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# PyTorch is ... Pythonic, but not Python only\n",
    "\n",
    "\n",
    "<img src=\"Ch15_F4_PyTorch_calling_pytorch.svg\" style=\"float: right;\"/>\n",
    "\n",
    "When we looked at _PyTorch is functions_, we actually conflated PyTorch the interface and PyTorch the backend.\n",
    "\n",
    "The JIT was another way of accessing the backend.\n",
    "\n",
    "Looking closer, we have `libtorch` (the C++-implemented autograd-enabled backend) calling down into `ATen` for tensor operations and there the dispatching to ATen's own kernels or libraries.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Beyond Python\n",
    "\n",
    "\n",
    "<img src=\"Ch15_F5_PyTorch_zebra_app.png\" style=\"width: 20%; float: right;\"/>\n",
    "\n",
    "We can also run our JITed models from C++ in a few lines of code or on mobile.\n",
    "\n",
    "The picture shows a CycleGAN running on PyTorch Mobile (from Chapter 15 of our book, just like the PyTorch sketch above).\n",
    "\n",
    "We also cover C++ there, but not now."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# PyTorch is ... much more\n",
    "\n",
    "We skipped `torch.distributions`, `distributed`, automatic mixed precision and more PyTorch has to offer. Check out the tutorials on PyTorch.org.\n",
    "\n",
    "## Forums\n",
    "\n",
    "Forums at [discuss.pytorch.org](https://discuss.pytorch.org/) probably are the best source of casual help, with a number of core developers also keeping an eye for the advanced questions. (I'm @tom there.)\n",
    "\n",
    "## Wider Ecosystem\n",
    "\n",
    "Libraries for many classes of models building on PyTorch (not complete)\n",
    "\n",
    "- Immediate \"Satellites\" like TorchVision, TorchAudio, TorchText, TorchServing\n",
    "- Extending capabilities: PyTorch-NLP, Graph NNs, Gaussian Processes, Pyro (Probabilistic)\n",
    "- Training helpers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><img alt=\"Thank you\" src=\"thank_you.svg\" width=80% /></center>\n",
    "\n",
    "<center><img alt=\"Your questions and comments\" src=\"your_questions_and_comments.svg\" width=80%/></center>\n",
    "\n",
    "[Graves style Handwriting in PyTorch](https://github.com/t-vi/pytorch-tvmisc/blob/master/misc/graves_handwriting_generation.ipynb)\n",
    "\n",
    "# Contact: Thomas Viehmann <tv@mathinf.eu>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "rise": {
   "header": "<img id='mathinf-logo-header' src='mathinf_logo.png' />",
   "theme": "white",
   "transition": "off"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}