hw3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "edLrpksDJfDg"
   },
   "source": [
    "# Intro\n",
    "This is a PyTorch Tutorial for UC Berkeley's CS285.\n",
    "There's already a bunch of great tutorials that you might want to check out, and in particular [this tutorial](https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html).\n",
    "This tutorial covers a lot of the same material. If you're familiar with PyTorch basics, you might want to skip ahead to the PyTorch Advanced section.\n",
    "\n",
    "First, let's import some things and define a useful plotting function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "dq6HedFjckGr"
   },
   "outputs": [],
   "source": [
    "#@title imports and define `plot`\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "import numpy as np\n",
    "\n",
    "def plot(xs, ys, xlim=(-3, 3), ylim=(-3, 3)):\n",
    "  fig, ax = plt.subplots()\n",
    "  ax.plot(xs, ys, linewidth=5)\n",
    "  # ax.set_aspect('equal')\n",
    "  ax.grid(True, which='both')\n",
    "  \n",
    "  ax.axhline(y=0, color='k')\n",
    "  ax.axvline(x=0, color='k')\n",
    "  ax.set_xlim(*xlim)\n",
    "  ax.set_ylim(*ylim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cMr7RToPzjn0"
   },
   "source": [
    "For GPU support, go to `Runtime > Change runtime type`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ynLs1RCduz_O"
   },
   "source": [
    "# PyTorch Basic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "U5rl_7Kx5vk8"
   },
   "source": [
    "## Tensors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rjXm2HUv_rBm"
   },
   "source": [
    "Numpy arrays are objects that allow you to store and manipulation matrices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "x55DbowZBcDY",
    "outputId": "2e5206f6-5881-4e51-a7ac-c7c2acc79499"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0. 0. 0.]\n",
      " [0. 0. 0.]]\n",
      "+\n",
      "[[1. 1. 1.]\n",
      " [1. 1. 1.]]\n",
      "=\n",
      "[[1. 1. 1.]\n",
      " [1. 1. 1.]]\n"
     ]
    }
   ],
   "source": [
    "shape = (2, 3)\n",
    "x = np.zeros(shape)\n",
    "y = np.ones(shape)\n",
    "z = x + y\n",
    "print(x)\n",
    "print(\"+\")\n",
    "print(y)\n",
    "print(\"=\")\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "g3Acexuuz0RS",
    "outputId": "0d2299a2-0915-4baa-f83a-dfa56007170c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1. 1.]\n"
     ]
    }
   ],
   "source": [
    "# print(z.sum())\n",
    "print(z[0, 1:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kPGVgAOZ_mIq"
   },
   "source": [
    "PyTorch is build around _tensors_, which play a similar role as numpy arrays. You can do many of the same operations in PyTorch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ujftyWqRCGtK",
    "outputId": "091e6c2a-524a-49bf-e1e9-88b5829c8aa7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0., 0., 0.],\n",
      "        [0., 0., 0.]])\n",
      "+\n",
      "tensor([[1., 1., 1.],\n",
      "        [1., 1., 1.]])\n",
      "=\n",
      "tensor([[1., 1., 1.],\n",
      "        [1., 1., 1.]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.zeros(shape)\n",
    "y = torch.ones(shape)\n",
    "z = x + y\n",
    "\n",
    "print(x)\n",
    "print(\"+\")\n",
    "print(y)\n",
    "print(\"=\")\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Nb6RWm3iAQjd"
   },
   "source": [
    "Many functions have alternate syntax that accomplish the same thing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ozmW7nxfKvue",
    "outputId": "a8f2f925-0596-4588-cdb9-fb68ceb16938"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.add(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "6l0MwUdvDNsU",
    "outputId": "dbd69b75-aa84-4865-f8f1-3e08ca16507e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z.min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Kv65CHa_D-WK",
    "outputId": "cafdde4f-4304-417d-bfe6-6258c58d129b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1.])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z[1:, 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "T6T5UEfJ0eKa",
    "outputId": "c8c813a1-ff3e-4026-e509-4c815a45f05d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(6.)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.sum(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NKs4pG0oAhro"
   },
   "source": [
    "Function that reduce dimenions will by default reduce all dimensions unless a dimension is specified"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rcLKnEh80gxY",
    "outputId": "05d0b50b-ec9f-43e6-e870-683ac3f446d3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([3., 3.])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.sum(z, dim=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "2hi_pF7a0i5Q",
    "outputId": "75ad0708-a8ec-44b2-fa26-af8ba04b2e9d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2., 2., 2.])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.sum(z, dim=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VUjHBWOHAaXZ"
   },
   "source": [
    "Like numpy, pytorch will try to broadcast operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ffy_uyGF0qZV",
    "outputId": "d01e12b0-d012-4b62-982f-1388d2ed2d4c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1.],\n",
      "        [1.],\n",
      "        [1.]])\n",
      "+\n",
      "tensor([[1., 1., 1.]])\n",
      "=\n",
      "tensor([[2., 2., 2.],\n",
      "        [2., 2., 2.],\n",
      "        [2., 2., 2.]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.ones((3, 1))\n",
    "y = torch.ones((1, 3))\n",
    "z = x + y\n",
    "\n",
    "print(x)\n",
    "print(\"+\")\n",
    "print(y)\n",
    "print(\"=\")\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "U8AQHu-K_6Z9"
   },
   "source": [
    "Operations that end with an underscore denote in-place functions. Use these sparingly as they easily lead to bugs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ewIwYOhnnCC5",
    "outputId": "512515dc-f962-4ac3-b651-83909841c79a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[2., 2., 2.],\n",
      "        [2., 2., 2.],\n",
      "        [2., 2., 2.]])\n",
      "tensor([[0., 0., 0.],\n",
      "        [0., 0., 0.],\n",
      "        [0., 0., 0.]])\n"
     ]
    }
   ],
   "source": [
    "print(z)\n",
    "z.zero_()\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "BsC0k_cgnFW-",
    "outputId": "d1d331a9-826a-496c-b50b-e2ca7dc788a9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[5., 5., 5.],\n",
      "        [5., 5., 5.],\n",
      "        [5., 5., 5.]])\n"
     ]
    }
   ],
   "source": [
    "z.add_(5)\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DwA9Pmj-K4Pr"
   },
   "source": [
    "### Moving between numpy and PyTorch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "id": "v3J1zw1WyHlG"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[ 0.02222857 -0.98373457  0.89597194  0.59514966]\n",
      "  [ 0.28770108 -0.33827953  0.05544195  0.2104092 ]]\n",
      "\n",
      " [[-0.166102   -0.42115955 -0.47166849 -0.06234568]\n",
      "  [ 0.50721528 -0.74923156  0.36231422 -2.02108458]]\n",
      "\n",
      " [[-0.516134    0.70809667  2.14771877 -0.03131667]\n",
      "  [ 0.09333847  0.83130934  0.26354267  0.03159759]]]\n"
     ]
    }
   ],
   "source": [
    "x_np = np.random.randn(3,2,4)\n",
    "print(x_np)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YkIYPgPzA2-C"
   },
   "source": [
    "numpy -> pytorch is easy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "id": "EMPx2iKzybuk"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[ 0.0222, -0.9837,  0.8960,  0.5951],\n",
      "         [ 0.2877, -0.3383,  0.0554,  0.2104]],\n",
      "\n",
      "        [[-0.1661, -0.4212, -0.4717, -0.0623],\n",
      "         [ 0.5072, -0.7492,  0.3623, -2.0211]],\n",
      "\n",
      "        [[-0.5161,  0.7081,  2.1477, -0.0313],\n",
      "         [ 0.0933,  0.8313,  0.2635,  0.0316]]], dtype=torch.float64)\n"
     ]
    }
   ],
   "source": [
    "x = torch.from_numpy(x_np)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vInPpem8ArmF"
   },
   "source": [
    "By default, numpy arrays are float64. You'll probably want to convert arrays to float32, as most tensors in pytorch are float32."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "xZSegeSByh9D"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.9208, -1.2725,  0.4922],\n",
      "        [-1.0813, -0.9124,  0.9050]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.from_numpy(x_np).to(torch.float32)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lUd3zm2yAzuI"
   },
   "source": [
    "pytorch -> numpy is also easy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "mjxo4Lwryu-6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-0.92079955 -1.2725363   0.49217486]\n",
      " [-1.0813323  -0.91243386  0.9050194 ]]\n"
     ]
    }
   ],
   "source": [
    "print(x.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4NDB2ypyy9eE"
   },
   "source": [
    "### GPU support"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "Kh-UHYiVzSKc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.cuda.is_available()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gvdb3bSIA7i-"
   },
   "source": [
    "The code below errors out because both tensors need to be on the same device."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "C2DzoBcizK7j"
   },
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "expected device cpu but got device cuda:0",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-25-7738a425635e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0my\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mones\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdevice\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mz\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m: expected device cpu but got device cuda:0"
     ]
    }
   ],
   "source": [
    "device = torch.device(\"cuda\")\n",
    "x = torch.zeros(shape)\n",
    "y = torch.ones(shape, device=device)\n",
    "z = x + y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "inYRaSWbBI5y"
   },
   "source": [
    "You can move a tensor to the GPU by using the `to` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "8jbmhtOY5Oj4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1., 1.],\n",
      "        [1., 1., 1.]], device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "x = x.to(device)\n",
    "z = x + y\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aXasNfBJBKFl"
   },
   "source": [
    "This code also errors out, because you can't convert tensors on a GPU into numpy arrays directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "ITCz002W5Vu0"
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "can't convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-27-545f895c5ddc>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mz\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnumpy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m: can't convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first."
     ]
    }
   ],
   "source": [
    "z.numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OEW3bs_ABaM6"
   },
   "source": [
    "First you need to move them to the CPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "id": "z00AQCfG5Xaw"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 1., 1.],\n",
       "       [1., 1., 1.]], dtype=float32)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z_cpu = z.to('cpu')\n",
    "z_cpu.numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PP-wQ8sDzIbn"
   },
   "source": [
    "### (homework aside)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "F5W8aKyiyPtc"
   },
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'cs285'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-29-303d2b6f387f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mcs285\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minfrastructure\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mpytorch_util\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mptu\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mptu\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfrom_numpy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_np\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mptu\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_numpy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'cs285'"
     ]
    }
   ],
   "source": [
    "from cs285.infrastructure import pytorch_util as ptu\n",
    "ptu.from_numpy(x_np)\n",
    "ptu.to_numpy(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lkH9ZA-s49Mg"
   },
   "source": [
    "## Neural-Network specific functions\n",
    "PyTorch has a bunch of built-in funcitons.\n",
    "See [the docs](https://pytorch.org/docs/stable/torch.html) for a full list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "id": "gpEeDQWcC2AT"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAZj0lEQVR4nO3deXzV9Z3v8dcnG4GwRCHs+xa2xI26VK2xbujgwjK9tTP2drwtbedatQ87LKWUqqiAvfZ6W/tomdGO3uu1M2URVBTXI+4iDiRhiSA7suMBAoQkJ9/7R8JcnOGXBM4vZ/nm/Xw8eDw4yed8f5+vJ7z5+Tu/88Gcc4iIiD8ykt2AiIiES8EuIuIZBbuIiGcU7CIinlGwi4h4RsEuIuKZuIPdzHLN7GMzW21ma8zs/jAaExGRs2Px3sduZgbkOecqzSwbeBe4xzn3YRgNiojImcmKdwFX/zdDZcPD7IZf+tSTiEiSxB3sAGaWCawEBgNPOOc+Ok3NJGASQG5u7kV9+/YN49Apqa6ujowMf9++8HV/27dvxzmHfjbTly/7OxGD/cfrqKn76terd2/c75wraOr5cV+K+cpiZvnAIuAnzrnyoLrCwkJXUVER2nFTTSQSoaSkJNlttBhf91dSUkI0GmXVqlXJbqXF+PranZTu+zt6opZfv1rBP7+/hdNF89Y5Y1c650Y3tU4oZ+wnOeeiZvYWMAYIDHYREfmqdzbsY9rCMnZ8eTzutcK4K6ag4UwdM2sLXAesj3ddEZHW4NCxGv7hL6u548mPA0M9K8O4+5ohzV4zjDP2HsDTDdfZM4B/dc69GMK6IiJee6V8FzMWr2HfkROBNef17sScicUM696R+5q5bhh3xZQCF8S7johIa7H3SBUzF6/h5fLdgTW52Rncd10hd14xgMwMO6P1Q73GLiIiwZxzzF+5g1kvrePQ8ZrAuksHnsvs8cX075J3VsdRsIuIJMCOL4/x80XlLP9sX2BNhzZZTLtpON/+Wh8yzvAs/VQKdhGRFlRX5/jfH25lzivrOVYdC6y7ZlhXZo0bRY9ObeM+poJdRKSFfL6vkqkLSlmx5cvAmnPzcph58whuOa8n9RNa4qdgFxEJWU2sjnnLN/H4Gxuorq0LrLv1/J78cuwIOrdvE+rxFewiIiEq33mIKQtKWfPF4cCa7h1zmXXbKK4d0a1FelCwi4iEoKomxv96YwN/XL6JWF3wqJbvXNKXqTcOo2Nudov1omAXEYnTii0HmTK/lE37jwbW9Ovcjtnji7lsUOcW70fBLiJylipP1DL3lfU888HWwJoMg+9fOZCfXjuUtjmZCelLwS4ichYiFXuZvqicndHgoV2F3Towd2Ix5/XJT2BnCnYRkTPy5dFqHnxpLQs/3RlYk51p3HX1EH5cMoicrMTPh1ewi4g0g3OOl8t388vF5eyvrA6sO79PPnMnFjO0W4cEdvdVCnYRkSbsPVzFjMXlLFuzJ7CmbXYm910/lL+7/MyHdoVNwS4iEsA5x19W7mDWi2s5XFUbWHf54M48Mq6Yvp3bJbC7YAp2EZHT2H7wGNMWlvHuxv2BNR1ys5h+03D+y9f6hDYOIAwKdhGRU8TqHE+/v4VHl1VwvCZ4aNd1I7ox67ZRdOuYm8DumkfBLiLSYMOeI0xZUMqn26KBNV3a53D/LaO4qah7Sp2ln0rBLiKtXk2sjj9EPue3b26kOhY8tGvcBb345dgRnJOXk8DuzpyCXURatbIdh/iH+atZv/tIYE3PTrk8NK6Iq4d1TWBnZ0/BLiKtUlVNjN+8/hn/uHwTjczs4o5L+zF5TCEdWnBoV9gU7CLS6ny06QBTF5axuZGhXQO65DF7fBGXDGz5oV1hU7CLSKtxpKqGOa+s5/98uC2wJjPD+MGVA7n32iHkZidmaFfYFOwi0iq8tX4vP19Uxq5DVYE1w7p34NGJ51HUu1MCOwufgl1EvHbwaDUPvLCG51d9EViTk5nB3dcM5odXDSI7M/FDu8KmYBcRLznneLF0F79asoYDR4OHdl3Yt35o1+CuyRvaFTYFu4h4Z8/hKqYvKuf1dcFDu9rlZDL5hkLuuKx/0od2hU3BLiLecM7x54+38dDSdRxpZGjXlUO68PC4IvqcmxpDu8IWd7CbWR/gGaAb4IB5zrnH411XRORMbDtwjLkrqlh3sCywpmNuFjPGjmDiRb1TdhxAGMI4Y68F7nPOfWpmHYCVZvaac25tCGuLiDQqVuf403ub+fWrFVTVBI8DuGFkNx68dRRdU3BoV9jiDnbn3C5gV8Pvj5jZOqAXoGAXkRb12Z4jTJ5fyqrtjQ3tasODt47kxqIeCewsuUK9xm5m/YELgI/CXFdE5FTVtXX8PrKRJ97aSE0seB7AhAt7M2PscPLbpfbQrrCFFuxm1h5YANzrnDt8mu9PAiYBFBQUEIlEwjp0yqmsrNT+0lA0GiUWi3m5t5N8eO02HYrxVNkJdlQGB3rnXON7I3MoKviSVR+/n8DuUoM518j0m+YuYpYNvAgsc8491lR9YWGhq6ioiPu4qSoSiVBSUpLsNlqMr/srKSkhGo2yatWqZLfSYtL5tTteHeOx1yp48t3NgUO7DPivX+/Pz24opH0b/276M7OVzrnRTdWFcVeMAU8C65oT6iIiZ+qDzw8wdWEpWw8cC6wZWJDH7QNj/OCWkQnsLDWF8Vfa5cAdQJmZnTzV+blzbmkIa4tIK3a4qoZHlq7nuY8bH9r1o6sG8pNvDuHD995JYHepK4y7Yt6l/v+ARERC88a6PUxfVM7uw8FDu0b06MjcicWM6pXeQ7vC5t9FKBFJawcqT3D/C2tZsrqRoV1ZGdx77RB+cOVAL4Z2hU3BLiIpwTnHktVfcP8LaznYyNCur/U/h9kTihlU0D6B3aUXBbuIJN2uQ8f5xaJy3li/N7AmLyeTKTcO428v6UeGZ0O7wqZgF5GkqatzPLdiG48sXU/lieChXVcNLeChcaPofY6fQ7vCpmAXkaTYsv8oUxaU8tHmg4E1ndpm88uxIxh/YS+vh3aFTcEuIglVG6vjqfc28z9e/YwTtcFDu24q6s79t4yioEObBHbnBwW7iCTM+t2HmTK/lNU7DgXWFHSoH9o1ZlTrGdoVNgW7iLS4E7UxnnhzI7+PfE5t0DwA4FujezP9phF0apedwO78o2AXkRb16bYvmTK/lA17KwNrep/Tltnji7liSJcEduYvBbuItIhj1bX8etln/On9zQTNGjSD7329Pz+7vpA8D4d2JYv+S4pI6N7buJ+pC0vZfvB4YM3gru2ZM6GYi/qdk8DOWgcFu4iE5tDxGh5+aR3/8sn2wJqsDOPHJYO465uDaZOVmcDuWg8Fu4iEYtma3cx4vpy9R04E1hT16sScCcWM6NkxgZ21Pgp2EYnLviMn+NWSNbxUtiuwpk1WBj+9bijfv2IAWRra1eIU7CJyVpxzLPq3nTzw4lqix2oC6y4ecC6zxxcxUEO7EkbBLiJnbGf0ONMXlRGp2BdY075NFlNuHMbfXNxXQ7sSTMEuIs1WV+d49uNtzF66jqPVscC6qwsLeGhcET3z2yawOzlJwS4izbJpXyVTF5Tx8ZbgoV3ntMtm5s0jufX8nhralUQKdhFpVG2sjn96dzO/ea3xoV03n9eTmTePoEt7De1KNgW7iARa+8VhJi9YTfnOw4E13Tq2YdZtRVw3olsCO5PGKNhF5D+pqonxuzc38oe3Gx/adfvFfZh203A65mpoVypRsIvIV6zcepDJ80v5fN/RwJq+57Zj9vgivj5YQ7tSkYJdRAA4eqKWR5dV8PQHWwKHdmUY3Hn5AO67vpC2ORoHkKoU7CLC8s/2MW1hGTujwUO7hnarH9p1QV8N7Up1CnaRVuzQsRpmvbSWv6zcEViTnWn8fclg/v7qQRralSYU7CKt1Cvlu5ixeA37GhnadV7vTsyZWMyw7hralU4U7CKtzN4jVcxcvIaXy3cH1uRmZ3DfdYXcecUAMjUOIO0o2EVaCecc7+6s4Z63l3PoePDQrssGdmb2hCL6dc5LYHcSplCC3cyeAsYCe51zo8JYU0TCs+PLY0xbWMY7G6oDazq0yWLaTcO5/eI+GgeQ5sI6Y/9n4HfAMyGtJyIhqKtzPPPBFuYuq+BYI0O7rh3elVm3FdG9U27impMWE0qwO+eWm1n/MNYSkXBs3FvJ1AWlfLL1y8Cac/NymHnzCG45T0O7fJKwa+xmNgmYBFBQUEAkEknUoROusrJS+0tD0WiUWCyW9nurrXO8vLmGxRtrqA2eBsClPTL5zvAsOkY38PbbGxLXYAvy9WfzTCUs2J1z84B5AIWFha6kpCRRh064SCSC9pd+8vPziUajab238p2HmDy/lLW7jgXWdO+Yy0PjRnHNcP+Gdvn6s3mmdFeMiAeqamI8/sYG5i3fRKyRoV1X98ni8f/2DQ3t8pyCXSTNrdhykCnzS9m0P3hoV//O7Zg9oZiqbWUK9VYgrNsdnwNKgC5mtgOY6Zx7Moy1ReT0Kk/UMveV9TzzwdbAmgyDH1w5kHuvHUrbnEwi2xLYoCRNWHfF3B7GOiLSPJGKvUxfVN7o0K5h3Tswd2Ixxb3zE9iZpAJdihFJI18erebBl9ay8NOdgTXZmcZdVw/hxyWDyMnKSGB3kioU7CJpwDnH0rLdzFxSzv7K4E+Pnt8nn7kTixnarUMCu5NUo2AXSXF7D1cxY3E5y9bsCaxpm53JfdcP5e8u19AuUbCLpCznHH/5ZAezXlrL4arawLrLB3fmkXHF9O3cLoHdSSpTsIukoO0H64d2vbtxf2BNh9wsfvFXw/nWaA3tkq9SsIukkFid4+n3t/DosgqO1wQP7bpuRDdm3TaKbh01tEv+MwW7SIrYsOcIUxaU8um2aGBNl/Y53H/LKG4q6q6zdAmkYBdJsuraOv749uf89s2NVMfqAuvGX9iLGX81gnPychLYnaQjBbtIEpXuiDJ5finrdx8JrOnZKZeHxhdxdWHXBHYm6UzBLpIEVTUxfvPaZ/zjO5toZGYXd1zajyk3DqN9G/1RlebTT4tIgn246QBTF5Sy5UDwaN0BXfKYM6GYiwecm8DOxBcKdpEEOVJVw+yX1/PsR8GTuDIzjO9fOYCfXjuU3OzMBHYnPlGwiyTAm+v3MH1RObsOVQXWDO/RkbkTiinq3SmBnYmPFOwiLejg0WoeeGENz6/6IrAmJzODu68ZzA+vGkR2poZ2SfwU7CItwDnHC6W7+NWSNRw8Gjy066J+5zBnQjGDu7ZPYHfiOwW7SMh2H6riF8+X8/q64KFd7XIymXxDId+9rD8ZGtolIVOwi4TEOcefV2zn4ZfWceRE8NCuK4d04eFxRfQ5V0O7pGUo2EVCsPXAUaYuKOODTQcCazrmZjFj7AgmXtRb4wCkRSnYReIQq3P86b3N/PrVCqpqgscBjBnZnQduG0nXDhraJS1PwS5ylip2H2HyglJWb29saFcbHrx1JDcW9UhgZ9LaKdhFzlB1bR2/j2zkibc2UhMLngcw4cLezBg7nPx2GtoliaVgFzkDq7ZHmTK/lIo9wUO7euW35eHxRVw1tCCBnYn8fwp2kWY4Xh3jsdcqePLdzYFDu8zgu5f2Y/KYYeRpaJckkX76RJrw/uf7mbqgjG0Hg4d2DSqoH9o1ur+GdknyKdhFAhyuquGRpet57uPGh3b96KqB/OSbQzS0S1KGgl3kNF5fu4fpz5ex5/CJwJqRPTsyd2IxI3tqaJekFgW7yCmqY3D3c//GktWNDO3KyuCea4Yw6RsDNbRLUlIowW5mY4DHgUzgn5xzsxurP1xd/y+x+2rD1hq2an9pZX/lCUp3RDlWHWs01L/W/xxmTyhmUIGGdknqijvYzSwTeAK4DtgBrDCzJc65tUHPOVjlmLlkTbyHTm3rtL90c6w6Fvi9vJxMptw4jL+9pJ+GdknKM+ca+QcXm7OA2WXAr5xzNzQ8ngbgnHsk6DkZOW1dTvfBcR1XJGzVezcBkNN14Fe+nt8uhwFd8miTlf6XXaLRKPn5+cluo8X4vr+33357pXNudFN1YVyK6QVsP+XxDuCS/1hkZpOASQCWrXkZkvoyDbq2y6BTmxjHKw9zPNkNhSAWixGNBo9ASHe+76+5EvbmqXNuHjAPoE2PIa77dxq9DC+ScLv/71QAbpjyB24c1Z1vje7DuXl+jQOIRCKUlJQku40W4/v+mjsVNIxg3wn0OeVx74avBeqQY3z3sn4hHDo17dy5k169eiW7jRbj4/4yzPiXV/PIqK1i8X+/PNntiMQljGBfAQwxswHUB/q3ge809oTOucYDt44K4dCpKRLZT0mJ9pduIo/lEo0G/2PTIuki7mB3ztWa2V3AMupvd3zKOeffLRMiImkilGvszrmlwNIw1hIRkfik//1bIiLyFQp2ERHPKNhFRDyjYBcR8YyCXUTEMwp2ERHPKNhFRDyjYBcR8YyCXUTEMwp2ERHPKNhFRDyjYBcR8YyCXUTEMwp2ERHPKNhFRDyjYBcR8YyCXUTEMwp2ERHPKNhFRDyjYBcR8YyCXUTEMwp2ERHPKNhFRDyjYBcR8YyCXUTEMwp2ERHPKNhFRDwTV7Cb2V+b2RozqzOz0WE1JSIiZy/eM/ZyYDywPIReREQkBFnxPNk5tw7AzMLpRkRE4hZXsJ8JM5sETAIoKCggEokk6tAJV1lZqf2loWg0SiwW83JvJ/n62p3k+/6aq8lgN7PXge6n+dZ059zi5h7IOTcPmAdQWFjoSkpKmvvUtBOJRND+0k9+fj7RaNTLvZ3k62t3ku/7a64mg905d20iGhERkXDodkcREc/Ee7vjODPbAVwGvGRmy8JpS0REzla8d8UsAhaF1IuIiIRAl2JERDyjYBcR8YyCXUTEMwp2ERHPKNhFRDyjYBcR8YyCXUTEMwp2ERHPKNhFRDyjYBcR8YyCXUTEMwp2ERHPKNhFRDyjYBcR8YyCXUTEMwp2ERHPKNhFRDyjYBcR8YyCXUTEMwp2ERHPKNhFRDyjYBcR8YyCXUTEMwp2ERHPKNhFRDyjYBcR8YyCXUTEM3EFu5k9ambrzazUzBaZWX5YjYmIyNmJ94z9NWCUc64Y+AyYFn9LIiISj7iC3Tn3qnOutuHhh0Dv+FsSEZF4hHmN/U7g5RDXExGRs5DVVIGZvQ50P823pjvnFjfUTAdqgWcbWWcSMAmgoKCASCRyNv2mhcrKSu0vDUWjUWKxmJd7O8nX1+4k3/fXXOaci28Bs+8BPwSucc4da85zCgsLXUVFRVzHTWWRSISSkpJkt9FifN1fSUkJ0WiUVatWJbuVFuPra3eS7/szs5XOudFN1TV5xt7EQcYAk4GrmhvqIiLSsuK9xv47oAPwmpmtMrM/hNCTiIjEIa4zdufc4LAaERGRcOiTpyIinlGwi4h4RsEuIuIZBbuIiGcU7CIinlGwi4h4RsEuIuIZBbuIiGcU7CIinlGwi4h4RsEuIuIZBbuIiGcU7CIinlGwi4h4RsEuIuIZBbuIiGcU7CIinlGwi4h4RsEuIuIZBbuIiGcU7CIinlGwi4h4RsEuIuIZBbuIiGcU7CIinlGwi4h4RsEuIuIZBbuIiGfiCnYze9DMSs1slZm9amY9w2pMRETOTrxn7I8654qdc+cDLwK/DKEnERGJQ1zB7pw7fMrDPMDF146IiMQrK94FzOwh4LvAIeDqRuomAZMaHp4ws/J4j53CugD7k91EC/J5f13MzNe9gd+vHfi/v8LmFJlzjZ9km9nrQPfTfGu6c27xKXXTgFzn3MwmD2r2iXNudHMaTEfaX/ryeW+g/aW75u6vyTN259y1zTzms8BSoMlgFxGRlhPvXTFDTnl4K7A+vnZERCRe8V5jn21mhUAdsBX4UTOfNy/O46Y67S99+bw30P7SXbP21+Q1dhERSS/65KmIiGcU7CIinklasPs8jsDMHjWz9Q37W2Rm+cnuKUxm9tdmtsbM6szMm1vLzGyMmVWY2UYzm5rsfsJkZk+Z2V5fPz9iZn3M7C0zW9vws3lPsnsKi5nlmtnHZra6YW/3N/mcZF1jN7OOJz+5amZ3AyOcc8198zWlmdn1wJvOuVozmwPgnJuS5LZCY2bDqX/D/I/Az5xznyS5pbiZWSbwGXAdsANYAdzunFub1MZCYmbfACqBZ5xzo5LdT9jMrAfQwzn3qZl1AFYCt/nw+pmZAXnOuUozywbeBe5xzn0Y9JyknbH7PI7AOfeqc6624eGHQO9k9hM259w651xFsvsI2cXARufcJudcNfBn6m/h9YJzbjlwMNl9tBTn3C7n3KcNvz8CrAN6JbercLh6lQ0Psxt+NZqXSb3GbmYPmdl24G/wd4DYncDLyW5CmtQL2H7K4x14EgytjZn1By4APkpuJ+Exs0wzWwXsBV5zzjW6txYNdjN73czKT/PrVgDn3HTnXB/qP7V6V0v2Eram9tZQMx2opX5/aaU5+xNJNWbWHlgA3PsfrgqkNedcrGGKbm/gYjNr9HJa3EPAmmjG23EETe3NzL4HjAWucWn4YYEzeO18sRPoc8rj3g1fkzTRcP15AfCsc25hsvtpCc65qJm9BYwBAt8IT+ZdMd6OIzCzMcBk4Bbn3LFk9yPNsgIYYmYDzCwH+DawJMk9STM1vMH4JLDOOfdYsvsJk5kVnLyzzszaUv8Gf6N5mcy7YhZQP4Ly38cROOe8OEMys41AG+BAw5c+9OWOHwAzGwf8FigAosAq59wNye0qfmZ2E/A/gUzgKefcQ0luKTRm9hxQQv1Y2z3ATOfck0ltKkRmdgXwDlBGfaYA/Nw5tzR5XYXDzIqBp6n/ucwA/tU590Cjz0nDqwQiItIIffJURMQzCnYREc8o2EVEPKNgFxHxjIJdRMQzCnYREc8o2EVEPPP/AGYoBWhxoofyAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "xs = torch.linspace(-3, 3, 100)\n",
    "ys = torch.relu(xs)\n",
    "plot(xs.numpy(), ys.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "id": "of1U9ahX46eL"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAZ4klEQVR4nO3deZRc5Xnn8e9TS+8bklpSaxdCNLsBgzDGNu0FwxhObHAywdiTscmMnJljxzke79hxEg9jOD7jOGfs2CaBGDIOOXYIOMZgloG22RdBI7QjoR1tLanUXb1Ud1U980dVg3DoRarbVd2vfp9z7qlbVW+993kp6afLW3cxd0dERMIRq3QBIiISLQW7iEhgFOwiIoFRsIuIBEbBLiISGAW7iEhgSg52M6sxs2fN7CUzW2tmfxlFYSIicnys1OPYzcyAendPm1kSeBz4nLs/HUWBIiJybBKlduCFfxnSxafJ4qKznkREKqTkYAcwsziwCjgF+IG7P/MWbVYCKwFqamrevmjRoig2PSXl83lisXB/vgh1fDt37sTd0Z/N6Sv08W3atKnb3VvHa1fyVMybOjNrAe4GPuvua0Zr197e7hs3boxsu1NNZ2cnHR0dlS5j0oQ6vo6ODlKpFF1dXZUuZdKE+t2NCH18ZrbK3S8Yr12k/7S5ewp4FLgiyn5FRGTiojgqprW4p46Z1QKXARtK7VdERI5PFHPsbcDtxXn2GPAzd783gn5FROQ4RHFUzGrgvAhqERGRCIT787GIyAlKwS4iEhgFu4hIYBTsIiKBUbCLiARGwS4iEhgFu4hIYBTsIiKBUbCLiARGwS4iEhgFu4hIYBTsIiKBUbCLiARGwS4iEhgFu4hIYBTsIiKBUbCLiARGwS4iEhgFu4hIYBTsIiKBUbCLiARGwS4iEhgFu4hIYBTsIiKBUbCLiARGwS4iEhgFu4hIYEoOdjNbaGaPmtk6M1trZp+LojARETk+iQj6yAL/w91fMLNGYJWZPeTu6yLoW0REjlHJe+zuvsfdXyiu9wLrgfml9isiIscn0jl2M1sCnAc8E2W/IiIycVFMxQBgZg3AXcCfuXvPW7y/ElgJ0NraSmdnZ1SbnnLS6bTGNw2lUilyuVyQYxsR6nc3IvTxTZS5e+mdmCWBe4EH3P2747Vvb2/3jRs3lrzdqaqzs5OOjo5KlzFpQh1fR0cHqVSKrq6uSpcyaUL97kaEPj4zW+XuF4zXLoqjYgy4FVg/kVAXEZHJFcUc+yXAfwLeZ2ZdxeVDEfQrIiLHoeQ5dnd/HLAIahERkQjozFMRkcAo2EVEAqNgFxEJjIJdRCQwCnYRkcAo2EVEAqNgFxEJjIJdRCQwCnYRkcAo2EVEAqNgFxEJjIJdRCQwCnYRkcAo2EVEAqNgFxEJjIJdRCQwCnYRkcAo2EVEAqNgFxEJjIJdRCQwCnYRkcAo2EVEAqNgFxEJjIJdRCQwCnYRkcAo2EVEAqNgFxEJjIJdRCQwiSg6MbPbgKuA/e5+VhR9iohUUi7vDOfyxcXJ5vIM5wuP2byTzTnZfJ5c3snmvfCYKzzm3Mnl8+TyvPHoTr7YLu8jC4XHfGF95D3gTe+7Qz7vE649kmAHfgJ8H7gjov5ERN7S4HCOdCZLejBLOpOlL5OlbyhLXybHi7uG2frEVvqHcgwO5xgYyjEwXFgyw3kGh3Nksm88ZrI5hrL5wpLLk8kWgnwom+cYcnTKiSTY3f23ZrYkir5E5MSQzeU51DfEgXSGQ31DHOob4mB6iFT/EKmBYQ73D3NkoLD0FJfewSxDufzYHa9ZV54BTGFR7bGPy8xWAisBWltb6ezsLNemyy6dTmt801AqlSKXywU5thHl+u4Gsk73gHNoMM+hAefQoJPKOIczTmowz5GMkx6GabxTPKWVLdjd/RbgFoD29nbv6Ogo16bLrrOzE41v+mlpaSGVSgU5thFRfXfuzoHeDK9297Gtu49tB/vZfrCPnYf72XlogCMDw6UXK8etbMEuItPT/t5BNu7tZePeXjbs7WXz/jRbDqTpHcxWurRJV5WIkYwZyUSMRCxGMm4k4kYyFiMeMxLxGImYFdZjRqz4GB9Z7I312FHPzXjTeqy4HjMDKK6DWeG1wjp86eaJ1a1gF5HX7T0ySNfOw7y8+whrX+thze4eutOZSpf1JvGY0ViToKH6jaW+OkF9dZyeQ92cvGg+tVVxapNx6qri1CQLS21yZD1GdSJOdSJGdTJGVTxGVeKox0SMZDGwrRi0U8WXJtguqsMd7wQ6gFlmtgv4prvfGkXfIjI5srk86/b08OzWQ6zafpgXd6TY2zNY1hpa6pLMaqhmZn0VM45aWuqqaKlN0lKXpLm2sDTVJmmqSVKTjI0auIWpJh1xHdVRMR+Loh8RmTy5vLPtSI4f/WYLT245yPPbDtE/lJuUbSXjRltzLfNaapjXXEtbSw1zm2uZ21TDnKZqWhurmVlfTVVC50hOBk3FiATsQG+G3246QOemAzz2ygFS/cPAhkj6rknGWDqrgZNn1bNkVh2LZ9azeEYdC2fUMaephnhsak1jnEgU7CKB2bw/zYPr9vLg2n107UyV3F8ybpwyu5HT5jbSPreR9jmNnDK7gfkttcQU3lOSgl0kAK/s6+WXq/fwq9WvseVA33H3k4gZp7U18rYFLZyzoJkz5zVz6pxGTZlMMwp2kWlq75FB7n5xN/e8uJuN+3qPq4+mmgQXLJnBhUtmcMGSkzh7fjM1yXjElUq5KdhFppFMNscDa/fx8+d38sTm7mO+nkl1HC4+pZVLls3i4mUzOaOtSdMpAVKwi0wDWw6kufOZHdz1wi4O9x/bWZ3LWut5b/ts3nvabPp3vMxl71sxSVXKVKFgF5micnnn0Q37+cmT23h8c/cxffb8RS1cfuZcLjtjDie3Nrz+eucu7Z2fCBTsIlNMXybLz57fyT88sY0dh/on/LlzF7Zw1TltXHlOG23NtZNYoUx1CnaRKaI7neH2J7dxx1PbJ3wRrYUzarnmvAVcc/58Fs+sn+QKZbpQsItU2J4jA/z4N69y57M7yGTHudY4hQtTXXl2G3944UJWLJmhHz/l31Gwi1TI7tQAf/voZn7+/K7xbx4BLJ1VzyfesZiPnj+flrqqMlQo05WCXaTM9vcO8rePbuGfntkxoUC/9NRWPnnJEi5d3qq9c5kQBbtImRwZGOaHnVv4yZNbGRweO9CTcePq8+bzX999MsvnNJapQgmFgl1kkmWyOf7xqe18/9HNxYtwja6+Ks4n3rGY69+1lDlNNWWqUEKjYBeZJO7Ovav3cPOvN7Dr8MCYbRtrEnzqnUv41CVLOale8+dSGgW7yCR4aWeKb927jue3Hx6zXX1VnD9+11L++N0n01ybLFN1EjoFu0iEDvRmuOn+Ddz1wq4x21UnYvzndy7h0+85mZkN1WWqTk4UCnaRCAzn8tzx1Ha+99AmejOj3+TZDH7//AV8/oOn6uxQmTQKdpESPbv1EN+4Z824l8699NRWvvqh0zhtblOZKpMTlYJd5DgdTGf4X/eNP+1yyuwGvn7l6XS0zy5TZXKiU7CLHKN83vnZ8zv59v0bxrymS3Ntks9fdirXXbSIZFx3IJLyUbCLHIPN+3v52r+u4dlth0ZtYwbXXriIL17ezgwduigVoGAXmYBMNscPHt3CDzs3M5wb/bZFb1vQzLc+chbnLGgpY3Uib6ZgFxnHc9sO8ZW7Vo95k+immgRf/g+nce2Fi4jrei5SYQp2kVH0Dg5z86838H+f3jFmu2vOm8/XrjydWToeXaYIBbvIW3h43T6+fs8a9vYMjtpmycw6brz6bC45ZVYZKxMZn4Jd5Ci5PHz2zhf55UuvjdomETM+fenJfPZ9y6lJxstYncjERBLsZnYF8DdAHPh7d78pin5FysXd6U5n2HokR+8YoX7uwhZu+ujZOslIprSSg93M4sAPgMuAXcBzZvZv7r6u1L5FyuG11ABfv2cNm/enR21TVxXni5e380cXL9GPozLlmfvoh25NqAOzi4G/cPfLi8+/CuDu3x7tM3V1db5ixYqStjuVpVIpWlrCPdwtpPHt6xlkx6F+cnlnaP+rAFTNPvlNbVrqqlg6q57qxPQ/ySik7+6thD6+3/zmN6vc/YLx2kUxFTMf2HnU813ARb/byMxWAisBkskkqVQqgk1PTblcTuOb4oZysLcvT3929B2buMHsuhjN1TkG0j2MfUX16SGE724soY9vosr246m73wLcAtDe3u5dXV3l2nTZdXZ20tHRUekyJs10Ht9wLs/fPfYq33v4FZqyeY6eKd/7T18BYO51N3Hl2W188/fOYHZjWHcxms7f3USEPj6ziU0DRhHsu4GFRz1fUHxNZEpZvSvFl+96mfV7ekZtk4gZP/rE27nirLllrEwkWlEE+3PAcjNbSiHQrwWui6BfkUj0ZbL89UObuO2JreTH+ElpdmM1zYmsQl2mvZJ/DXL3LPAZ4AFgPfAzd19bar8iUXhkwz4++Ne/5e8fHz3UF82o46f/5SJObm0grgNeJACRzLG7+33AfVH0JRKFfT2D/NUv1/Grl/eM2iZmcP0lS/n8B0+lrkrn6kk49KdZgpIt3qLuuw9tIj3GLepOb2vi5o+eraswSpAU7BKMVdsP84171rBujB9HqxMx/vT9y1n5npN18wsJloJdpr0DvRlu/vUG/mXV2Leoe/fyWfzPj5zF4pn1ZapMpDIU7DJtDWXz3PHUNv7m4VfoHWPaZWZ9Fd+46gw+fO68CR8HLDKdKdhl2nF3Htmwnxt/tZ5Xu0e/+YUZfGzFIr58+Wk01yXLWKFIZSnYZVpZ91oP375/PY+90j1muzPamrjx6rM4b9FJZapMZOpQsMu0sDs1wP9+cCN3v7ibsa5b11yb5AuXt3PdCt2iTk5cCnaZ0g71DfHDzs3c/tR2hrL5UduZwbUXLuSLl5/GjPqqMlYoMvUo2GVKOjIwzK2Pvcqtj2+lbyg3ZtsVS2fw51edwVnzm8tUncjUpmCXKSXVP8RtT2zjJ09spWdw9CNdABbOqOWGD53O5WfO1dEuIkdRsMuUcKA3w21PbOUfn9o+5hmjACfVJfns+5bz8Xcsojqhe46K/C4Fu1TUqwfS/N1jW7nrhV1jzqED1CRjfOqSpfy3jmU01ejwRZHRKNil7Nydp7Yc5B+e3MbD6/eNeZQLQFU8xnUXLeK/v3dZcDe+EJkMCnYpm3Qmyy+6dnP7k9vYtG/0G0ePSMaN33/7Aj7zvuXMb6ktQ4UiYVCwy6Ryd1bvOsI/P7eDX3S9Rv84R7gAVCViXHvhQv7k0mXMU6CLHDMFu0yKfT2D3PPibu56YdeE9s4BGmsSfPyixVx/yRJmN2nKReR4KdglMqn+IX69Zi/3rt7Dk1u6x7wN3dHammu4/pKlXLtiIY36UVSkZAp2Kcn+3kH+3/r9PLB2L4+/0k12omlO4cSiT71zCZedMYeEro0uEhkFuxyTfN7ZdiTH9x95hUc27OfFnalxj2o5WkN1go+cN4/rVizmjHlNk1eoyAlMwS7jei01wJNbDvLk5m5++0o33ekMsOmY+nj74pP4wwsWctXb2nR/UZFJpr9h8ibuzraD/Ty39RDPbSss2w72H1dfC06q5ZrzF3DNefNZMkt3LRIpFwX7CW5/7yBrd/fw0q4UXTtTvLQzxeH+4ePub25TDVee08ZV57Rx7sIWXcNFpAIU7CeIweEcW7v72Li3lw17e9m4t4e1r/WwvzdTct/LWuu57Iy5fPDMOZy7oIWYroMuUlEK9oBkc3n2HBlk+8F+th3sY1t3H1u7+9h8IM3OQ/0TPvxwPIkYXLxsFh3ts+lob2VZa0M0HYtIJBTs04S7c7h/mH09g+ztGWTvkUH2pAbYnRrktdQAu1L97EkNHtPhhhNlBmfNa+ady2byzlNmMbhjDZd/4KLItyMi0VCwV0gmm+PIwDA9A1lS/UMc7h/mcP8Qh/uGONQ3xMHi44HeDN3pDAfTQwzlxr76YVSqEzHOWdDMhUtmcOGSGZy/+CSaa984cajzNU21iExlCvYxuDuZbJ6hXJ7McJ7B4RyZbI7B4vrAcI6BoTce+4dy9A9lWf/KEI8eWUM6k6MvkyWdydKbydI7OEzvYOFxcLg8IT2eZNxYPruRM+c18baFLZy7sIX2uY0kdcKQyLRVUrCb2R8AfwGcDqxw9+cn8rn0sPMvq3aRd8fdyTu4Q969sOQLr73+3CGXL7ye8zcec3nI5fPk8oW22eJ6Lp8nm3dyeSebd7K5PNmcM/ym9eJjrhDcw7k8Q9k8wzlnKFtYL2kPeev24//sJJnbVMPyOQ20z2mkfW4jp7c1sXxOg25WIRKYUvfY1wDXAD8+lg91Dzhf+PlLJW5a3sqM+ioWnlTLkln1LJ5Zz5KZdSxrbWDZ7AYaqvU/aCIngpL+prv7ekDHKpdJfVWcOc01zGmsYU5TNW0ttcxrrqGtuZYFM2pZcFKdwltEyjfHbmYrgZUAVXNPKddmp6yYQV0C6pNGfdJoSBr1SWioMhqrjKajHpurC481iZF/QAeLyxHIAPth737YW6ba0+k0nZ2dZdpa+aRSKXK5XJBjGxHqdzci9PFN1LjBbmYPA3Pf4q0b3P0XE92Qu98C3AJQ3bY8+mPyJkkybtQk4lQnY1Qn4lQnYtQk49QkC491VXFqknFqi+t11Qn27d7Bme3LaaxOUF+doL46TmNNgsaaJA3VCRprEjRUJ6bt/+l0dnbS0dFR6TIi19LSQiqVCnJsI0L97kaEPr6JGjfY3f0DUW+0Pmlcc/58YmYYEI8ZZoYZxM2IWWF6J2ZGPAaxWHG9+F4sVlyPGYmYET9qiZmRjBvxWOz1945+nozHSMQLryXjseJSWK9KxF5/rToRoyoeO66zKDs799LxrqVR/2cTEZmQikzIttYa3/2P51Zi0yIiwSvpYGUzu9rMdgEXA78ysweiKUtERI5XqUfF3A3cHVEtIiISAZ1eKCISGAW7iEhgFOwiIoFRsIuIBEbBLiISGAW7iEhgFOwiIoFRsIuIBEbBLiISGAW7iEhgFOwiIoFRsIuIBEbBLiISGAW7iEhgFOwiIoFRsIuIBEbBLiISGAW7iEhgFOwiIoFRsIuIBEbBLiISGAW7iEhgFOwiIoFRsIuIBEbBLiISGAW7iEhgFOwiIoEpKdjN7DtmtsHMVpvZ3WbWElVhIiJyfErdY38IOMvdzwE2AV8tvSQRESlFScHu7g+6e7b49GlgQekliYhIKaKcY78euD/C/kRE5DgkxmtgZg8Dc9/irRvc/RfFNjcAWeCnY/SzElgJ0NraSmdn5/HUOy2k02mNbxpKpVLkcrkgxzYi1O9uROjjmyhz99I6MPsk8Gng/e7eP5HPtLe3+8aNG0va7lTW2dlJR0dHpcuYNKGOr6Ojg1QqRVdXV6VLmTShfncjQh+fma1y9wvGazfuHvs4G7kC+BJw6URDXUREJlepc+zfBxqBh8ysy8x+FEFNIiJSgpL22N39lKgKERGRaOjMUxGRwCjYRUQCo2AXEQmMgl1EJDAKdhGRwCjYRUQCo2AXEQmMgl1EJDAKdhGRwCjYRUQCo2AXEQmMgl1EJDAKdhGRwCjYRUQCo2AXEQmMgl1EJDAKdhGRwCjYRUQCo2AXEQmMgl1EJDAKdhGRwCjYRUQCo2AXEQmMgl1EJDAKdhGRwCjYRUQCo2AXEQmMgl1EJDAlBbuZfcvMVptZl5k9aGbzoipMRESOT6l77N9x93Pc/VzgXuDPI6hJRERKUFKwu3vPUU/rAS+tHBERKVWi1A7M7Ebgj4AjwHvHaLcSWFl8mjGzNaVuewqbBXRXuohJFPL4ZplZqGODsL87CH987RNpZO5j72Sb2cPA3Ld46wZ3/8VR7b4K1Lj7N8fdqNnz7n7BRAqcjjS+6SvksYHGN91NdHzj7rG7+wcmuM2fAvcB4wa7iIhMnlKPill+1NMPAxtKK0dEREpV6hz7TWbWDuSB7cCfTPBzt5S43alO45u+Qh4baHzT3YTGN+4cu4iITC8681REJDAKdhGRwFQs2EO+HIGZfcfMNhTHd7eZtVS6piiZ2R+Y2Vozy5tZMIeWmdkVZrbRzDab2VcqXU+UzOw2M9sf6vkjZrbQzB41s3XFP5ufq3RNUTGzGjN71sxeKo7tL8f9TKXm2M2saeTMVTP7U+AMd5/oj69Tmpl9EHjE3bNmdjOAu3+5wmVFxsxOp/CD+Y+BL7j78xUuqWRmFgc2AZcBu4DngI+5+7qKFhYRM3sPkAbucPezKl1P1MysDWhz9xfMrBFYBXwkhO/PzAyod/e0mSWBx4HPufvTo32mYnvsIV+OwN0fdPds8enTwIJK1hM1d1/v7hsrXUfEVgCb3f1Vdx8C/pnCIbxBcPffAocqXcdkcfc97v5Ccb0XWA/Mr2xV0fCCdPFpsriMmZcVnWM3sxvNbCfwccK9gNj1wP2VLkLGNR/YedTzXQQSDCcaM1sCnAc8U9lKomNmcTPrAvYDD7n7mGOb1GA3s4fNbM1bLB8GcPcb3H0hhbNWPzOZtURtvLEV29wAZCmMb1qZyPhEphozawDuAv7sd2YFpjV3zxWvorsAWGFmY06nlXwRsHGKCfZyBOONzcw+CVwFvN+n4ckCx/DdhWI3sPCo5wuKr8k0UZx/vgv4qbv/a6XrmQzunjKzR4ErgFF/CK/kUTHBXo7AzK4AvgT8nrv3V7oemZDngOVmttTMqoBrgX+rcE0yQcUfGG8F1rv7dytdT5TMrHXkyDozq6XwA/+YeVnJo2LuonAJytcvR+DuQewhmdlmoBo4WHzp6VCO+AEws6uB/wO0Aimgy90vr2xVpTOzDwHfA+LAbe5+Y4VLioyZ3Ql0ULis7T7gm+5+a0WLipCZvQt4DHiZQqYAfM3d76tcVdEws3OA2yn8uYwBP3P3vxrzM9NwlkBERMagM09FRAKjYBcRCYyCXUQkMAp2EZHAKNhFRAKjYBcRCYyCXUQkMP8ffAAT1hZzKoMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "xs = torch.linspace(-3, 3, 100)\n",
    "ys = torch.tanh(xs)\n",
    "plot(xs.numpy(), ys.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "id": "smjZcbzo4Ers"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXhV5bn38e+TeU4IBAJhnsIccEDrUIMjrQNO8NbaSU+lPZ08RcRaHOpYccC2nvq+pWqrx1YLKkpRnI2KCgqaQBjCDGEIEGBnHnee94+EHqpZSWDPK7/PdXFl2HfWuh92+Llce617G2stIiLiHlGhbkBERPxLwS4i4jIKdhERl1Gwi4i4jIJdRMRlFOwiIi7jc7AbYxKMMZ8aY4qMMeuMMXf5ozERETkxxtfr2I0xBki21lYbY2KB5cCN1toV/mhQRESOT4yvG7Ct/2Wobvsytu2P7noSEQkRn4MdwBgTDawGhgN/tNaubKdmJjATICEh4eSBAwf6Y9dhqaWlhago97584db1lZaWYq1Fv5uRK1LW19QCB2tbaGxxromPhqzEKGKOWc6mTZvKrbVZnW3f51Mx/7YxYzKAxcDPrbXFTnW5ubm2pKTEb/sNNwUFBeTn54e6jYBx6/ry8/PxeDwUFhaGupWAcetzd1S4r8/bYvnzh9uY/+YmGr3tp3pcdBQ3XTiSH549lOgo82+PGWNWW2tP6Ww/fjliP8pa6zHGvAdMBRyDXUSku9lRXsNNi4pYvfOIY83YfmnMnzGR3OxUn/blc7AbY7KAprZQTwQuAOb5ul0RETew1vLsip3c/9pG6pq87dZERxl+OmU4Pz93OLHRvp9K8scRe1/g6bbz7FHAQmvtUj9sV0Qkou2rqGPOC2v4cHO5Y82wrGTmz5hI3oAMv+3XH1fFrAEm+aEXERFXsNay+Is93LlkHVX1ze3WGAP/ceYQZl+US0JstF/379dz7CIi3V15dQNzF6/ljXX7HWv690jk4el5nD60Z0B6ULCLiPjJ68VlzF28lkM1jY4110weyNyLR5MSH7j4VbCLiPiooq6J3yxZx+Iv9jjW9E6NZ97VE5iS2zvg/SjYRUR88MGmg8x5YQ1llfWONZdP7MdvLhtLRlJcUHpSsIuInICahmZ+u2wDz67Y5VjTIymW+64YzzfH9w1iZwp2EZHj9tmOw8xeVMTOQ7WONeeP7sNvrxxPVmp8EDtrpWAXEemi+iYvj761iQUfbsNpGktqfAx3XjaWq07KoXX4bfAp2EVEuqB4TwWzFhayaX+1Y80Zw3ry0PQ8cjISg9jZVynYRUQ60ORt4fH3tvLYu5tpbmn/MD0hNopbvzGa754+iKio0BylH0vBLiLiYMuBKmYtLGLN7grHmkkDM3hkeh5Ds1KC2FnHFOwiIl/S0mJ56qPtPPhGCY3N7Y/XjY02/PKCkfzo68O+Ml431BTsIiLH2HWoltkvFPHp9sOONaP7pjF/Rh6j+6YFsbOuU7CLiNA6uOu5T0u579X11DS2P143ysBP8ofzi/NGEBcTvu/UpGAXkW6vrKKeW15cw/ubDjrWDM1K5pHpeUwa2COInZ0YBbuIdFvWWpYU7eX2l4updBivC3D9mUOYM9X/43UDRcEuIt3SoeoGbnu5mGXFZY41ORmt43W/Niww43UDRcEuIt3OW+v3c+tLayivdh6vO+OU/tx+yRhSE2KD2Jl/KNhFpNuorG/iriXrefHz3Y41WanxPHDleM4b3SeInfmXgl1EuoWPtpRz86Ii9lY4j9e9eEJf7p02jh7JwRmvGygKdhFxtdrGZuYt28jTn+x0rMlIiuWeaeO4NK9fEDsLHAW7iLjW6p1HuGlhITs6GK87JTeLeVdNoHdaQhA7CywFu4i4TkOzl9+9vZk/vb8Vh7ldJMdFc8elY5hxyoCQjdcNFAW7iLjKur0V3LSwiI1lVY41pw/N5KGr8xiQmRTEzoJHwS4irtDsbWHJ1kb++dZHNHnbP0yPj4liztRRXHfG4LAYrxsoCnYRiXhbD1Yza2ERRaVNjjV5A1rH6w7vHT7jdQNFwS4iEaulxfLXj3cw7/WNNHQwXvfG80bw43OGERMdvoO7/EnBLiIRafeRWm5etIZPth1yrBmVncojM/IY2y89iJ2Fns/BbowZADwD9AEssMBa+3tftysi0h5rLQtXlXLP0g1UN7Q/uCvKwI/OGcZ/nT+C+JjIGNzlT/44Ym8GbrLWfm6MSQVWG2Pestau98O2RUT+5UBlPb96aS3vbjzgWDO4ZxKPzMjj5EGZQewsvPgc7NbafcC+ts+rjDEbgBxAwS4ifvPPor3c/koxnlrnF0jPGxjDYz88m6S47n2W2a+rN8YMBiYBK/25XRHpvo7UNHL7K8UsXbPPsaZfegIPXp1H857ibh/q4MdgN8akAC8C/2WtrWzn8ZnATICsrCwKCgr8teuwU11drfVFII/Hg9frdeXajoq0567wQDN/WddIRYPD7aPAWTkxfHtUFM17iiNufYFirHX+C+vyRoyJBZYCb1hr53dWn5uba0tKSnzeb7gqKCggPz8/1G0EjFvXl5+fj8fjobCwMNStBEykPHdV9U3cu3QD/1hV6ljTKyWO+68Yz4Vjs//1vUhZ34kyxqy21p7SWZ0/rooxwJPAhq6EuohIRz7eWs7Ni9awx1PnWPPN8dnce/l4MiN8vG6g+ONUzJnAd4G1xpijhzq/tta+5odti0g3Ud/kZd7rG/nLRzsca9ISYrjn8nFcltfPdYO7/MkfV8UsB/Q3LCInrLDUw6yFhWw7WONYc87I1vG62enuGa8bKHr5WERCprG5hT+8s5nHC7Y4jtdNiovmtovHcM1k943XDRQFu4iExIZ9lcxaWMSGfV+5iO5fJg/O5OHpeQzs6c7xuoGiYBeRoGr2trDgw208+tYmx/G6cTFRzLkol+vPHOLq8bqBomAXkaDZXl7DrIWFfLHL41gzPied+TPyGNEnNYiduYuCXUQCrqXF8j8rdvLbZRuob2p/vG5MlOHn547gJ1OGEdtNxusGioJdRAJqj6eOOS8U8dEW5/G6I3qnMH/GRMb3717jdQNFwS4iAWGt5YXVu7n7n+upchivawzMPHsov7xgJAmx3W+8bqAo2EXE7w5WNXDrS2t5e8N+x5qBmUk8PD2PyUO673jdQFGwi4hfLVu7j7kvF3O4ptGx5trTBvLrb44mOV4RFAj6WxURv6iobeKOJcW8UrjXsSY7LYF5V0/gnJFZQeys+1Gwi4jPCkoOcMuLa9hf2eBYc8WkHH5z6VjSk2KD2Fn3pGAXkRNW3dDM/a9t4O8rdznW9EyO474rxjF1XN8gdta9KdhF5ISs3HaI2S8UUXrYebzuhWP6cP+V4+mVEh/EzkTBLiLHpb7Jy8NvlPDkR9txep+e1IQY7rpsLFdMytHgrhBQsItIl63Z7WHWwiK2HKh2rDl7RC/mXTWBfhmJQexMjqVgF5FONXlbeOzdLfzxvS14HebrJsZG8+uLR/Od0wbqKD3EFOwi0qGSsipmLSxk3V7n8bonD+rBI9PzGNwrOYidiRMFu4i0y9tieXL5Nh5+YxON3vYHd8VFR/HLC0Yy8+tDidZ43bChYBeRr9h5qIbZi4r4bMcRx5qx/dKYP2MiudkarxtuFOwi8i/WWp5duYv7X91AXZO33ZroKMNP84fxs3NHEBej8brhSMEuIgDsq6hjzgtr+HBzuWPNsKxk5s+YSN6AjCB2JsdLwS7SzVlrWfzFHu5cso6qeufxutefOYSbL8rVeN0IoGAX6cbKqxuYu3gtb6xzHq/bv0ciD0/P4/ShPYPYmfhCwS7STb1eXMbcxWs51MF43WsmD2TuxaNJ0XjdiKJnS6Sbqahr4q4l63jpiz2ONb1T45l39QSm5PYOYmfiLwp2kW6kuLyZXz36AWWV9Y41l+X14+5pY8lIigtiZ+JPCnaRbqC2sXW87rOrnOel90iK5d7Lx3PxBI3XjXQKdhGXW7XjMDctKmLnoVrHmvNH9+b+K8fTOzUhiJ1JoPgl2I0xTwGXAAesteP8sU0R8U19k5dH397Egg+2OY7XTYmP4Y5LxzD95P4a3OUi/jpi/yvw38AzftqeiPigeE8FNy0somR/lWPNGcN68uDVE+jfIymInUkw+CXYrbUfGGMG+2NbInLimr0tPF6wlT+8s5lmh/G6CbFR3PqN0Xz39EFEaXCXKwXtHLsxZiYwEyArK4uCgoJg7Troqqurtb4I5PF48Hq9Ebu2vdUt/HltA9sr2p/ECDA41fLjifFkN+7ggw92BK+5IHHr7+bxClqwW2sXAAsAcnNzbX5+frB2HXQFBQVofZEnIyMDj8cTcWtrabE89dF2HlxRQmNz+6EeG2345QUjGWVLOXfKlCB3GDxu/d08XroqRiSClR6u5aZFRXy6/bBjzei+acyfkcfovmkUFOwOYncSKgp2kQhkreW5T0u599X11Da2P143ysB/5g/jxvNGarxuN+Ovyx2fA/KBXsaY3cCd1ton/bFtEfl3+yvrueXFNRSUHHSsGdIrmUdm5HHSwB5B7EzChb+uirnGH9sREWfWWpYU7eWOV9ZRUdfkWPeDMwZzy9RRJMZpvG53pVMxIhHgcE0jt728ltfWljnW5GQk8tDVEzhjeK8gdibhSMEuEubeXr+fX720lvJq5zkvM07pz22XjCEtITaInUm4UrCLhKnK+ibu/ud6XljtfCVLVmo8D1w5nvNG9wliZxLuFOwiYejjLeXc/MIa9njqHGsuntCXe6eNo0eyxuvKv1Owi4SRukYv817fyF8/3uFYk5EUyz3TxnFpXr/gNSYRRcEuEiZW7zzC7EVFbC+vcayZkpvFvKsm0DtN43XFmYJdJMQamr387u3N/On9rTjM7SI5LprbLxnD/zl1gMbrSqcU7CIhtH5vJbMWFrKxzHm87mlDMnl4eh4DMjVeV7pGwS4SAs3eFv70wTZ+9/YmmrztH6bHx0QxZ+oorjtjsMbrynFRsIsE2daD1dy0sIjCUo9jTV7/dB6ZMZHhvVOC2Jm4hYJdJEhaWixPf7KDea9vpL6p/fG6MVGGX5w3gp/kDyMmWoO75MQo2EWCYPeRWm5etIZPth1yrMntk8ojM/IYl5MexM7EjRTsIgFkrWXhqlLuWbqB6obmdmuiDNzw9aHMumAk8TEa3CW+U7CLBMiBynpufWkt72w84FgzuGcSj8zI4+RBmUHsTNxOwS4SAEvX7OW2l4vx1DqP1/3e1wbxq2+MIilO/wzFv/QbJeJHR2oauWPJOv5ZtNexpl96Ag9encdZIzReVwJDwS7iJ+9tPMCcF9dwsMp5vO5VJ/Xnzss0XlcCS8Eu4qOq+ibue3UDz39W6ljTKyWO+64Yz0Vjs4PYmXRXCnYRH3yy9RCzFxV1OF536ths7rtiHD1T4oPYmXRnCnaRE1Df5OXB10t46qPtjjVpCTHcPW0c0yb20+AuCSoFu8hxKiz1MGthIdsOOo/X/frILB68agLZ6RqvK8GnYBfposbmFh57dzOPF2zF6zBfNykumrkXj+bbkwfqKF1CRsEu0gUbyyqZ9Y8i1u+rdKyZPLh1vO7AnhqvK6GlYBfpgLfFsuCDbTz61iYave0P7oqLiWLORblcd+YQojVeV8KAgl3EwfbyGmYvKmL1ziOONeNz0pk/I48RfVKD2JlIxxTsIl/S0mJ5duVOfvvaRuqavO3WxEQZfnbucH46ZTixGq8rYUbBLnKMphb43lOfsnxLuWPNiN4pzJ8xkfH9NV5XwpNfgt0YMxX4PRANPGGtfcAf2xUJppqGZnZUeqlxCHVj4IazW8frJsRqvK6EL5+D3RgTDfwRuADYDXxmjFlirV3v67ZFguWDTQdZv68Sh9dHGZCZyCPTJzJ5iMbrSvjzxxH7ZGCLtXYbgDHmeWAa4BjspaWl5Ofn+2HX4cnj8ZCRkRHqNgLGbesrr25g68EaGvZvBaDs77/6t8f7pCVAZhJzXov8K17c9tx9mdvX11X+CPYc4NjpR7uB075cZIyZCcwEiI2NxeNxfiPfSOf1erW+CFHRYNlX4/z+o32TDckxTVRVVgS5s8Bw03PXHrevr6uC9uKptXYBsAAgNzfXFhYWBmvXQVdQUODq/yNxy/qWrd3HT//+OdltN5EePVLP/vYDjMpO5enrJ7cerbuIW547J25fX1fvZvZHsO8BBhzzdf+274mErQ83H+TG5wtpbzLA5CGZ/Pl7p5CeqJnpEpn8cQHuZ8AIY8wQY0wc8C1giR+2KxIQn+86wo/+Z3W7d5KmxhmeuX6yQl0ims9H7NbaZmPMz4A3aL3c8Slr7TqfOxMJgDW7PXz/qU+pbfzqjUcZSXH0SfDqUkaJeH45x26tfQ14zR/bEgmU4j0VfOeJlVTVN3/lsVMH92B/nxQqK9zxIql0b7oXWrqFdXsruPaJlVS2E+pj+qbxxPdPJUpjdsUlFOziekeP1Cvqmr7y2LCsZJ7WOXVxGc2KEVdbteMw1/3lM6oavnqkPrRXMs/dcDpZqXovUnEXBbu41vLN5dzwzKp2JzQO6ZXMczNPp7fLrlMXAQW7uNQb68r4+XNf0Nj81UsaB/VM4rkbTnfdzUciRynYxXWeXbGTO14pbvfmoxG9U3j2h6cp1MXVFOziGtZaHn1rE394d0u7j4/tl8Yz10+mZ4rOqYu7KdjFFRqbW7jt5bUsXLW73cdPGpjBX67T1S/SPSjYJeJ5ahv58bOrWbHtcLuPnzMyi8evPYnkeP26S/eg33SJaNsOVvMfT69ie3lNu49fdVJ/HrhqvN6XVLoVBbtErPc3HeQXz33R7o1HAD+dMozZF+Z2edSpiFso2CXiWGt5vGArD79Zgm3nypfoKMNdl43lO6cPCn5zImFAwS4RpbqhmdkLi3h9XVm7j6cmxPB/rz2Zs0b0CnJnIuFDwS4Ro3hPBT/7++fsOFTb7uODeibx5PdPZXjvlCB3JhJeFOwS9qy1PLtyF/csXd/unaQAZ4/oxR++NYkeyXFB7k4k/CjYJax5ahuZu7iYV9fuc6z5z/zWF0mjo/QiqQgo2CWMLd9czuxFRZRV1rf7eFJcNA9Pz+Ob4/sGuTOR8KZgl7BT3+TloTdKeHL5dsea0X3T+OO3JzE0S+fTRb5MwS5hZfXOw9y8aA3bHG44Arj2tIHcfskYvTepiAMFu4SFukYvD79ZwlMfbW/32nSAtIQY7r9yPJdM6Bfc5kQijIJdQu69kgPc/nIxu4/UOdZ8bWhPHpmRR7+MxCB2JhKZFOwSMgcq67lr6XpeXeN8xUtcdBQ3XTiSG84eSpSuehHpEgW7BF1jcwt//Xg7f3hnC9XtvBfpUXkDMnjo6gmM7JMaxO5EIp+CXYKqoOQAdy9dz7aDzi+OxsVEMeuCkfzwrCHEaCqjyHFTsEtQbNhXyW+XbeSDTQc7rDtjWE/uuXwcw3QZo8gJU7BLQO2rqGP+m5t44fPdjle7AGQmx3HbxaO5YlKOxuyK+EjBLgFRXt3A4+9t5dmVOx3nuwBEGbj2tEHMumCk5ryI+IlPwW6MmQ78BhgNTLbWrvJHUxK5Dtc08sSH2/jrxzuobfR2WDt5SCa/uXQsY/qlBak7ke7B1yP2YuBK4E9+6EUi2IGqep74cDvPrtjZaaAPyEzklqmjuHh8X512EQkAn4LdWrsB0D/ObmxHeQ1PLN/GolW7aejglAtAemIsPz93ON/92iDiYzQOQCRQgnaO3RgzE5gJkJWVRUFBQbB2HXTV1dWuX98TL7/D69ubWL3fSweviQIQHw0XDIrlG0NiSfbu4pPlu4LS5/HyeDx4vV7XP3dan/t1GuzGmLeB7HYemmutfaWrO7LWLgAWAOTm5tr8/Pyu/mjEKSgowI3ra2xuYVnxPn7/yRq2VbQ/SvdYcTFRfOe0QfxkyjB6pcQHoUPfZGRk4PF4XPncHeXW382j3L6+ruo02K215wejEQlfezx1PP/pLp7/rJSDVQ2d1ifERvHtyYP40TlD6ZOWEIQOReRYutxR2tXkbeG9jQf4x2elvFdygJbOzrcAKfExXHvaQH549lCyUsP/CF3ErXy93PEK4DEgC3jVGFNorb3IL51JSGzeX8ULq3fz4ud7KK/u/OgcoE9aPNefOYRrThtIWkJsgDsUkc74elXMYmCxn3qREDlQWc+Sor0s/mIP6/ZWdvnnxuWkcd0ZQ7g0rx9xMZrpIhIudCqmmzpU3cCy4jKWrtnLyu2HO7zd/1gxUYaJWVHccsVkThnUQ5e6ioQhBXs3sq+ijjfX7eeNdWWs3H4Yb1dOnLfJyUjkmskDmHHKANZ/voJTB2cGsFMR8YWC3cWstazfV8k7Gw7wzob9FO2uOK6fj402XDgmm+mn9OfsEVlEt73RxfpANCsifqNgd5mq+iY+2lLO+5sO8n7JQfZ24XrzL8vrn860iTlcPimHTA3mEok4CvYI1+RtoajUw/It5SzfXE5hqYfm4zjFctSgnklcltePaRNzGN5bs9BFIpmCPcI0NrdQvLeCldsO88m2Q6zacbjToVtO+vdI5OIJfbl0Qj/G9kvTC6EiLqFgD3MVtU18XnqEL3Ye4dMdhyks9VDf1PGwrY6M6J3C1HHZXDQ2W2Eu4lIK9jDS2NzCxrJKinZXUFTq4YtdR9jawXuDdkVMlOG0oZmcO6oP54/uzaCeyX7qVkTClYI9ROoavZTsr2Ld3gqK91RSvKeCkrIqGr0nfjR+VE5GIufkZnHOyCzOHN6LlHg9zSLdif7FB1hLi2X3kTo2llWyaX8VG8uq2LCvku3lNV2av9IV6YmxnD40k7NGZHHW8F4M7pmkUywi3ZiC3U/qm7zsPFTLtoPVvLW1kZfLvmDzgWq2Hqz26Zx4ezKT4zh5UA9OH9qTrw3tyajsVKKiFOQi0krBfhzqGr3sOlzLzkM1bR9r2V5ew/byGvZW1H3ptvy9ftvv0KxkThrYg5MH9eDUwZkMy0rWEbmIOFKwH6Oyvol9nnr2eurY7aljz5E69njqKD1cy+4jtZRXNwa8h14pceT1z2BC/wwmDEhn0oAMMpJ0k5CIdF23CPbG5hYO1TRwoLKB/ZX17K9q4EBlPWUV9ZS1fdxXUU91Q3NQ+8rJSGR03zTG9EtjfE4643PS6ZMWr6NxEfFJRAa7tZbK+maO1DRyqKaRwzWNHK5poLy6kUPVjZRXNxzzp/XxUEpNiGFE7xRys9MYlZ3KyD6pjOmbRnqSZpeLiP+FLNittTQ0t1BZ30RlXXPbxyYq6v73Y0VdE57aJjx1TXhqGzlS2/rRU9t0QrfNB1p2WgLDeicT31jB2XkjGdE7lRF9UuidqqNwEQmekAT7rqoWRt62jCZv+IVzR6IM9E1PZGBmEoN6JjGoZzKDeiYxpFfrx6S41r/OgoIC8s8cEuJuRaS7Ckmwt1jCMtTjoqPITk+gb3oCOT0S6Z+RSE6PRHIykhiQmUjf9ES9U5CIhL2IPMd+Inomx5GVGk/vtAT6pMbTJy2BPmmtH/umJ9InPZ5eyfG6HlxEIl7EBntibDSZyXFkJsfRIzmOXm2fZ6bE0SslnqyUeHqlxNMrtfXr2GgdaYtI9xDSYI+NNqQmxJKeGEtaQsz/fp4YQ1pi6+c9kuLIaPs8IymOHsmt30uIjQ5l6yIiYSskwT4gNYrie6YSHxOlq0VERPwsJMEebdARt4hIgOjEs4iIyyjYRURcRsEuIuIyCnYREZfxKdiNMQ8ZYzYaY9YYYxYbYzL81ZiIiJwYX4/Y3wLGWWsnAJuAW31vSUREfOFTsFtr37TWHh1ivgLo73tLIiLiC3+eY78eWObH7YmIyAno9AYlY8zbQHY7D8211r7SVjMXaAb+1sF2ZgIzAbKysigoKDiRfiNCdXW11heBPB4PXq/XlWs7yq3P3VFuX19XGWt9G59rjPkB8CPgPGttbVd+Jjc315aUlPi033BWUFBAfn5+qNsIGLeuLz8/H4/HQ2FhYahbCRi3PndHuX19xpjV1tpTOqvzaaSAMWYqMAc4p6uhLiIigeXrOfb/BlKBt4wxhcaY/+eHnkRExAc+HbFba4f7qxEREfEP3XkqIuIyCnYREZdRsIuIuIyCXUTEZRTsIiIuo2AXEXEZBbuIiMso2EVEXEbBLiLiMgp2ERGXUbCLiLiMgl1ExGUU7CIiLqNgFxFxGQW7iIjLKNhFRFxGwS4i4jIKdhERl1Gwi4i4jIJdRMRlFOwiIi6jYBcRcRkFu4iIyyjYRURcRsEuIuIyCnYREZdRsIuIuIyCXUTEZXwKdmPMPcaYNcaYQmPMm8aYfv5qTEREToyvR+wPWWsnWGsnAkuBO/zQk4iI+MCnYLfWVh7zZTJgfWtHRER8FePrBowx9wHfAyqAKR3UzQRmtn3ZYIwp9nXfYawXUB7qJgLIzevrZYxx69rA3c8duH99uV0pMtZ2fJBtjHkbyG7nobnW2leOqbsVSLDW3tnpTo1ZZa09pSsNRiKtL3K5eW2g9UW6rq6v0yN2a+35Xdzn34DXgE6DXUREAsfXq2JGHPPlNGCjb+2IiIivfD3H/oAxJhdoAXYCP+7izy3wcb/hTuuLXG5eG2h9ka5L6+v0HLuIiEQW3XkqIuIyCnYREZcJWbC7eRyBMeYhY8zGtvUtNsZkhLonfzLGTDfGrDPGtBhjXHNpmTFmqjGmxBizxRjzq1D340/GmKeMMQfcev+IMWaAMeY9Y8z6tt/NG0Pdk78YYxKMMZ8aY4ra1nZXpz8TqnPsxpi0o3euGmN+AYyx1nb1xdewZoy5EHjXWttsjJkHYK29JcRt+Y0xZjStL5j/CZhtrV0V4pZ8ZoyJBjYBFwC7gc+Aa6y160PamJ8YY74OVAPPWGvHhboffzPG9AX6Wms/N8akAquBy93w/BljDJBsra02xsQCy4EbrbUrnH4mZEfsbh5HYK1901rb3PblCqB/KPvxN2vtBmttSaj78LPJwBZr7TZrbSPwPK2X8LqCtfYD4HCo+wgUa+0+a+3nbZ9XARuAnNB25R+2VXXbl7FtfzrMy5CeYzfG3GeMKQWuxb0DxK4HloW6CelUDlB6zNe7cUkwdDfGmMHAJGBlaH5vjC4AAAGKSURBVDvxH2NMtDGmEDgAvGWt7XBtAQ12Y8zbxpjidv5MA7DWzrXWDqD1rtWfBbIXf+tsbW01c4FmWtcXUbqyPpFwY4xJAV4E/utLZwUimrXW2zZFtz8w2RjT4ek0n4eAddKMa8cRdLY2Y8wPgEuA82wE3ixwHM+dW+wBBhzzdf+270mEaDv//CLwN2vtS6HuJxCstR5jzHvAVMDxhfBQXhXj2nEExpipwBzgMmttbaj7kS75DBhhjBlijIkDvgUsCXFP0kVtLzA+CWyw1s4PdT/+ZIzJOnplnTEmkdYX+DvMy1BeFfMirSMo/zWOwFrriiMkY8wWIB441PatFW654gfAGHMF8BiQBXiAQmvtRaHtynfGmG8CvwOigaestfeFuCW/McY8B+TTOtZ2P3CntfbJkDblR8aYs4APgbW0ZgrAr621r4WuK/8wxkwAnqb19zIKWGitvbvDn4nAswQiItIB3XkqIuIyCnYREZdRsIuIuIyCXUTEZRTsIiIuo2AXEXEZBbuIiMv8fwAtJxKgZbRNAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "xs = torch.linspace(-3, 3, 100)\n",
    "ys = torch.selu(xs)\n",
    "plot(xs.numpy(), ys.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X97sJV2bEqT2"
   },
   "source": [
    "## Automatic differentiation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-1wKYOcNc_I-"
   },
   "source": [
    "Given some loss function\n",
    "$$L(\\vec x, \\vec y) = ||2 \\vec x + \\vec y||_2^2$$\n",
    "we want to evaluate\n",
    "$$\\frac{\\partial L}{\\partial \\vec x}$$\n",
    "and\n",
    "$$\\frac{\\partial L}{\\partial \\vec y}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "id": "ceo_JWyUD0Py"
   },
   "outputs": [],
   "source": [
    "shape = (3, )\n",
    "x = torch.tensor([1., 2, 3], requires_grad=True)\n",
    "y = torch.ones(shape, requires_grad=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5f0cycqiBt0h"
   },
   "source": [
    "PyTorch makes this easy by having tensors keep track of their data..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "id": "GroZ0prJgAZ0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1., 2., 3.])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5Ozf4qtnByLZ"
   },
   "source": [
    "...and their gradient:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "id": "_9-FwL4wgCDG"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None\n"
     ]
    }
   ],
   "source": [
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sdnlnBFjB0KT"
   },
   "source": [
    "However, right now `x` has no gradient because it does not know what loss it must be differentiated with respect to.\n",
    "Below, we define the loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "id": "GN2CeNSOCZw9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(83., grad_fn=<SumBackward0>)\n"
     ]
    }
   ],
   "source": [
    "loss = ((2 * x + y)**2).sum()\n",
    "print(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OFfIwK0jB8T3"
   },
   "source": [
    "And we perform back-propagation by calling `backward` on it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "id": "UIr4j_hrgN7n"
   },
   "outputs": [],
   "source": [
    "loss.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sfEKcV4YB-s9"
   },
   "source": [
    "Now we see that the gradients are populated!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "id": "yUSm8_r5gOhi"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([12., 20., 28.])\n",
      "tensor([ 6., 10., 14.])\n"
     ]
    }
   ],
   "source": [
    "print(x.grad)\n",
    "print(y.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rUnvZN9Jg0mg"
   },
   "source": [
    "### gradients accumulate\n",
    "Gradients accumulate, os if you call backwards twice..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "id": "qwAZ8fyMgQ1z"
   },
   "outputs": [],
   "source": [
    "loss = ((2 * x + y)**2).sum()\n",
    "loss.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ndyknuW_CEGb"
   },
   "source": [
    "...you'll get twice the gradient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "id": "5bT1AWRngYVJ"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([24., 40., 56.])\n",
      "tensor([12., 20., 28.])\n"
     ]
    }
   ],
   "source": [
    "print(x.grad)\n",
    "print(y.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WJ3rei78gyxd"
   },
   "source": [
    "### multiple losses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "atVB4ihsCH7s"
   },
   "source": [
    "This accumulation makes it easy to add gradients from different losses, which might not even use the same parameters. For example, this loss is only a function of `x`...."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "n1Lf5NOcgfN3"
   },
   "outputs": [],
   "source": [
    "other_loss = (x**2).sum()\n",
    "other_loss.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j_TI2Db4CQ5N"
   },
   "source": [
    "...and so only `x.grad` changes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4NX0Oj1Vgjia"
   },
   "outputs": [],
   "source": [
    "print(x.grad)\n",
    "print(y.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_tVQv9nCgpxk"
   },
   "source": [
    "### stopping and starting gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cHO0aQioCXrC"
   },
   "source": [
    "If you don't specify `required_grad=True`, the gradient will always be `None`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 54
    },
    "id": "ueSASTvNg6mP",
    "outputId": "3829650d-f4fc-4173-f97a-e27de7f4e262"
   },
   "outputs": [],
   "source": [
    "shape = (3, )\n",
    "x = torch.tensor([1., 2, 3], requires_grad=True)\n",
    "y = torch.ones(shape)\n",
    "loss = ((2 * x + y)**2).sum()\n",
    "loss.backward()\n",
    "print(x.grad)\n",
    "print(y.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_o-HON3lCbjS"
   },
   "source": [
    "You can turn `required_grad` back on after initializing a tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tOUN_k0hhFU5"
   },
   "outputs": [],
   "source": [
    "shape = (3, )\n",
    "x = torch.tensor([1., 2, 3], requires_grad=True)\n",
    "y = torch.ones(shape)\n",
    "y.requires_grad = True\n",
    "loss = ((2 * x + y)**2).sum()\n",
    "loss.backward()\n",
    "print(x.grad)\n",
    "print(y.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yHfgIecCCeyq"
   },
   "source": [
    "You can cut a gradient by calling `y.detach()`, which will return a new tensor with `required_grad=False`. Note that `detach` is not an in-place operation!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "id": "_bQbj4u8hL3-",
    "outputId": "eaa75c2f-8258-4620-e09c-95658546531a"
   },
   "outputs": [],
   "source": [
    "shape = (3, )\n",
    "x = torch.tensor([1., 2, 3], requires_grad=True)\n",
    "y = torch.ones(shape, requires_grad=True)\n",
    "# y_detached = y.detach()\n",
    "loss = ((2 * x + y)**2).sum()\n",
    "loss.backward()\n",
    "print(x.grad)\n",
    "print(y.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "id": "RD-zF6Xna2GJ",
    "outputId": "60591077-350b-4112-bbde-3cf902c1251d"
   },
   "outputs": [],
   "source": [
    "shape = (3, )\n",
    "x = torch.tensor([1., 2, 3], requires_grad=True)\n",
    "y = torch.ones(shape, requires_grad=True)\n",
    "z = 2 * x\n",
    "z.required_grad = True\n",
    "loss = ((2 * z + y)**2).sum()\n",
    "loss.backward()\n",
    "print(x.grad)\n",
    "print(y.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 71
    },
    "id": "YsxDGYysafqT",
    "outputId": "8d5f0ede-82ac-445e-d007-0ec1854fa5d8"
   },
   "outputs": [],
   "source": [
    "z.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "id": "AkXdmy9Fah_1",
    "outputId": "aae3245b-3d60-4eab-f8b6-af7767ad459b"
   },
   "outputs": [],
   "source": [
    "torch.from_numpy(np.array([1,2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "g0nGMph3amPa"
   },
   "source": [
    "Any difference between p.data.add_(-0.001 + p.grad) and p.data+= -0.001 + p.grad?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "q8B8phm0hdrJ"
   },
   "source": [
    "## Modules\n",
    "![tmp.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA1IAAAFVCAMAAADSYi6kAAAABGdBTUEAALGPC/xhBQAAAAFzUkdCAK7OHOkAAAMAUExURf///+Grqf39//79/dHp8dfN4XKRt8Xl7a23manByf3//63Dy9flvdjlvsPl7c3r8WSlubK/mdvnxafAyYp5nqXR3ZSFp8vp8c3dq6vCyfz+/dnR42+QuGCgsf79+cnn8M/frmmourK6l2+svWSkt3mxwtvv9mKhtG2qvMXVpcDSoNfju22js46/zbPU3F6dsNbt9K/P2dnlv9Pr9HKTt9Xjt93V54G5yWalt6vDy97px7O/l4u9zc/p8dbo72ant8jn79PhtHSvwJzH1J+7xWihsczo76S+yEd8vIS4x322xbJlZL/Z4fr7/WOitdjU4Ozq8L7f6NHL2XCls418oaWXt5XF09fP4418o72zx7LGzWyltm+PtdXL4WyNtWemurTZ5M3D2ZeJrY3Bz+jx9E+BvK+5mZ/L2K7Fy4CcroO3xfT47F6JtnWVsvPo54OgvOLs8YGuvYm5x/b7/O30+JGtwrzY3//9/XSUuH2sufP3+rd0dN3q8MLj7K/Fy+Hqy////pe9zbjGmazL2rzMm5i6x/D8/ZezxeOvrcrc6FqKv/z9/+HrzZq42nOfyT93uiBHgtStq6HD1NukoIa9zVh2mxFDgKq2ml2Wt4d1luOpp8F6eDhzttr3/eWzsU5WiSpUi2yXxJK2wa7V4OjV0putoL7Y5//78VKIvP735jljlihYme/14zdbkJCxvqGwnWmw4/jp2sS+0LetwlaAsYeuvBZVnKHa+cKuqV+TxR5RjNPd6oKvz0V4sXKntYiGls/n8Z+RsnBsj9HfsEdmlLrN4IGjx+j4+0iJwLfi9rVsapWir9XO4CJpr+LGvsTVrry+v6SysdXL36mduzlzqZyFlESW0llnkHKAoJ+UmPrSzMDo+ydjn7WfqbLL4crw/TdNiuednIPF8IC24P9mZi55v2Snudi5tNHl6zRqp47F6oqsxlai1zGJyo+y1ImSq/fg1rq/0+DV18bN2dDt9fS3tPfQvZ7J55vQ8f+Fhb2YmKnDyfmtra/DyWaluau/yWP+01MAACAASURBVHgB7L0LfBXVtT8+9xyL5GD1dz2+cipHE0RSRdFya0UtUpqY1MuhoNZbIY20l1SCCUESCM0FepQaStI89DY87i80iQIJcCOgEUgAsWl4XUsu3vIUBD5Yf4o8xEfVlkv5r7Vfs+ecmTlnJnPC4z+Tk5k9e2b22rNnf+e719pr71EUd3FLwC0BtwTcEnBLwC0BtwTcEnBLwC0BtwTcEnBLwC0BtwTcEnBLwC0BtwTcEnBLwC2BC6YEwsrT911939Vk+eUFkys3I24JXKwlAIj65fduhuUPf/je964OX6y34ebbLYELpASQo7538+1f//q3vn3DDd+8+gLJlZsNtwQu1hKgHPWt3/zyN/d9+4bnbnMhdbE+SDffF0gJEI76w7d+c98vr/7DDb96zmWpC+S5uNm4WEuAtvoIor53w69+nupC6mJ9km6+L4wSkDgKEPWPve93IXVhPBg3FxdpCVA96upfQqsPOap3b5elLtIn6Wb7wigB2urjiPrH3r3nuyx1YTwZNxcXZwloEPUcIKo3NvzCF+fNuLl2S+C8l4AGUaBH4eI2/M77Y3EzcNGWAEHUH3irL5UgCljqWxftDbkZd0vgvJaAxFHQH0U5ymWp8/pIXOEXdQkAosCvj3MU2Po4S7m61EX9XN3Mn68SoK0+3sPLOcplqfP1PFy5F3sJSK2+791AbX2MpW5zWepif7hu/s9DCUgcJelRACq3X+o8PA1X5MVfAoyj0FMWfCaYrY+yFBjRwxf/Dbp34JZAj5aAFlGqHoWgcr0nevRRuMIuiRIw1KMQUqluv9Ql8ZDdm+jBEpA4SqtHIaJclurBJ+GKujRKQMNR6HuuXVyWujQes3sXPVYCMqL+W/hMqLBydakeexSuoEuiBGREgRfSfBVLLORa/C6J5+zeRE+VgIQo0KM01nMGKZeleupZuHIuhRKQEEXH8EaRlGvxuxSes3sPPVUCGkTJXkgSslyW6qmn4cq5+EsgHkS5LHXxP2f3DnqqBGREfVPH1keoyvXx66nH4cq56EtARpS+ZYJgChp+4Yv+Xt0bcEsg8SUgIcrA1kcVKnfuicQ/C1fCpVACEqJwfFR0fxS3UNzvjpe6FJ63ew+JLgEZUXo+ExxQro9fop+Em/6lUQIyokz0KASW6z1xaTxy9y4SWgISokz1KISU2y+V0EfhJn5JlICEKHM9irKUO4/fJfHU3ZtIXAlIiIrJUS5LJe45uClfKiUgISo2R7mjei+Vx+7eR8JKQELUHzSzi2ErT2dxdamEPQo34UuiBCREaefr00ETiXItfpfEc3dvIlElICEK9CjtXEi6oHJ9/BL1KNx0L5ESePo+Me/5c1HzTOiBCrwnLpFbd2/DLQHHS0DiqO+Z+0yo4HJel/I5fl9ugm4JnJ8SkBEVw2dChZTjupRXCRXFWgLnp4BcqW4JWCoBCVFx9EdxUDnNUl5lxpRnHp74zETD/2cm9pumVFi6NfdktwTOQwlIiIqnP4pDyuF5/BBR/VKehGWszv+9cCSt3/CxLqTOQwVxRVosAQlRFjjKae+JACKq/8vj7hynv0D8b9PGjhzhspTFx+ue3uMlICHKCkc57D1BEDV83K13GS233jq734BBGS5L9XgFcQVaLAFA1NXMem6Jo5xlKdLqGz7trjffvOyyy+D/w8veHCpvIO6plL5vZLgsZfHxuqf3eAnY5ihHWYpx1F0IKP1l6FNpfR9NenSA2/Dr8SriCrRUAvY5ykmWoogCjtKH02XfvWwwIipz1kCXpSw9XvfkHi8BCVHW9Ci0+jnWL0URtTQmopKHuJDq8SriCrRYAkKP+t43dec95wZzna1jPn5UjxoXG1H+Pi6kLD5f9/SeLQENR+l9m0MHSFKUQz5+lKNGxEZUUpILqZ6tIK40iyUgI8oyRwG0iPdEOBQKeRQFNrC2sxBE9TfjqMuIHpWc5ELKTvm61/RcCciI0v1+lERIukHUpcLddQ/irT4Dw4SwTCCiXJbqudrRQ5K88EaGlzFu4vaZxpOh2sGmu5XP4ZuUEWWHoxhLNT0x59RWJbQ6/dTnig2eoogys0yoHOVCyuEqcMEkZ6PiKHEjsMfuUkaULY4i/VK+ZR/X1KxZq4TX1az5qiJkOffM1jf4uwYkxa3nhKNcSFku3wv8Ao/y6b78fW2HlcX7WqsWheNBSdgTfAKuOVHq+2Bf64ayCwhZBFF/uPqX9/3yauu2PtYMRF3qg5aj3pIi3zv1Bce+8lnmYURUGuhRBoCCVh9w1IBHMxmi3IbfBQ4Rq9mrCHxwZHnn+593tSzvbDlRVBIHpgBS79Ys7zxZuq79WOeKj5QSqzITdb4DHAUsddvVwf11mz8bV9G0cE/XQYBUHEUi3xFv9cXiqD6oR+HiWvzk4rv4wxXKB0fWBn3LOnYcVk5XxtXICVcE3z22yBc+XX+yTPnpITvKRkLKTUbUN38V16h4HQMFsFTT/oaaeWvWdh1b9MXBtUpcZaLeEG31TbvLGFFEj/JzjnIhpZbdpRHyhT44sqDwva6WVUpFKBjXPYVDwXdbGgu+fGHNV2FvXFf0yEkyouKauUUHTxAF/VLB/e9/d3HHyd3ttbUtdQc8lu6RtPqGLzVBFPVCUhHlslSPVI+eE1KhdB1ZXnNy5/Kj3iJPKBjPKxkbfkcaGvasO7i2pMiOWSMhd+eEHgWIQu+J4Ltrtv+m48S61tra5XUHfFYgRTgK9KgYHCX0KLfhl5DKcF4TrQCWWhsMnq7fVqYoxXHxVAmw1LEDYd8XR1YpJcHi85p9SbjqhWTT1kdZC0b1VpzumDcPsOTzfbFmrTeetwzPBupRabE5Smr1uboUL7tLZosstbZCadq9purcx41FoTgqELLUigMh5emONafO7XvPcyGYJ+RWn73+KN4MBJaqCN/z2ea7At5QyaRDt3os8LB1PcplqUsGSeJGKio+aP9KUTzLNkFXzPZQPDSFLLXigOILnO6omVd3uMRCnRNiHQ7Irb5ucRTxRFe8YbgpvK1w2EpGAVET+/W3pEe5kLJSwBfJub7QqDKfAswzfnwZICoudaqkdFRZSPH5SsePKr0AAKXIiLJv66M8hf1SijcQIPOAha34+BHref9xZj2838fxUZJlAhHlmicuEqTEk00vLh6yeGG2ObLgdWb6uAeu8BJi8nkDYTwZYnDBC8/TIrf6uslRxHvC3m1QPWqESQ8vsfVp9SgXUvYK+yK5KohLXCx1Yd2QjKju6VHIU4SlbNxgIAxzIYFlwmQML3JUNKJclrJR2hfmJeHQq69plideo3+vvVpq1H0bLn5Cc4m688h5mzBVbvV1m6OILhW28cCIHjV8hIn1nHLUOdLW06xc7wkb5X0hXuJV/njdY9csvAZ/5B82dPnxj39o0IXrVV597DHpEn7pwmt+fN0o0/ZiIktAtZ53n6NIv5SNzHIvJOsc5epSNor7wrzEG37kmuv+RJYfwfpH7A821z32QwN4eJXXHvsZnIGna/5/dN4gJbf6HOAo4j1h/YHFwVEGrT7XPGG9tC/UK7zKI9f87B69BSBVom9t8CpPXPMjvUt+8th1o86LhUJu9TnAUTZ1KaJHgfXcmKMi/frklp/b8LtQMWIxXwRS//KT/yF/sFKX664xZCkfQoqcSa9jV3/tpwApA2azmC1rp8uIcoSjbOlS0OqbmNZ/mpn1HGbA1LNMEGS5kLL20C/YswmkvvGTP/4Efpq/fwGWMmAcylL/oz3/J3/82jcQUgbXJLYAHNWjSMeUdYsf1aNGm3DUYONWn9vwS2wF6cnUKaS+Fr38S0yWiroGIPXYeWApx/UoxJTlL3cwPcqMo0wR1S0jOnRJ21r0G/Y9Wf8uQVmGkPoasJShxe+Ja35/z//oQeo8sJTc6nNGj0JIWWUp2h9lxlFmehS2/LrR8DtvPReXICC6f0uGkMKGn4Fe5CW6lAGkDK7pfk4NUpAR5ZAeZYOlqB41YrCxZWKwmR7VPUh5lMlDbS0fjjfqeTQobDc6jhIwhBSylIFexHSpC4Kl5FYfcNR8hIMTizWWQkTZ85lAMNGFsFSANsUseUx6wzMmjk17I20s+b0R3+YNuKLfyDsVl+DiAIm1UwwhZa5LLfyRbsPvmp62+EkcBV+7+UcnwETTsDQnOvU9H2GiR8Vq9QGqeMMPKnlYCcc/4tGjDAU421rczy5aA0t8ZxtCKoYuBUb0P0bSVM9b/LSISnUOUZZ0qUB4KPHrMx7Da+ApywmKbAmkQk9f19C+yrO6owFmkImLqzwovP+4Nx+ytrx5+QMpGSPdb8TFhxJLZxlCypyloF8qWpf6l57vlwLrOZtd7Ibn7M7cogtECxY/2uobbcJR5tZzBiwCKRgDWljwZdPCg4Udxw7HM48bcBRMbbYUHd8HW/gN/vCOlL7uZxctQSXekw0hFYOlfn8BsJSkR/3hhuccbPXRuSfiLEJm6xtqPF9fXIjCht+0wOn6ozAX0xedq0q6lq+N48u9QJDQvWw8lMQoUw/d8eSg1x8d4LJUnA/ZymmGkLJn8evJfilNq89ZRFnw8UM9CmnC2NZ3mXl/FG/9IaSUrhW1nce+eqfzaKhr+dHY02JXAEeZDSDWR9Sblz10R8qgX/QZ6ELKClTiPdcQUjEtfudbl9Igyur3o3Qbe3JkvBY/Zj03G80RH6KIeSL8u4O1+fUnXwVIfbH8aMwZbjlHGcHZAFFDkaN+kexCKl6QWDvPEFI2dClinui5finVCwlafY5Zzxmu4rT4ET2q/9Lu6lHIVISlYC6m8H/s2Ll3lfJC7IYf16OMzCIGiBr80ANjgaNAoMtS1sAS39mGkIqpS0WbJ3rS4idxlO15z2VWigzHx1KgR8HMLWYjDuNs9TFIBVZ3HNvXue3phQdzWqoXxTBPIEdhq88iR9FWX3ISfHbRhVR8ILF2liGkbLFUj/VLCUT95mpn+6M4tOKy+DE9yoSj4kcUafj5Al17lzcf9q3u6Ny2KMaDBD2KIOohfTIyiH2TctRZgmEXUjHK2NZhQ0jFYCk9I3rPsZRA1C8BUc852R/FIRUPS3nQMjF8tAN6FGv4jYDppvBrBrjyeDBkvHgZR1lr9X136EMPoB5FBbqQMi5f+0cMIWVu8dP1nuixfimCqJv5124ctZ5zSMXBUkyPusuIJuD7UXFaJrCCE5ZSAoApiqoY/bxesPW90d8YzvokJXEUCnQhZR84xlcaQuoCtvhxjvolcpTjtj4KqtgsFSANrxHQwaq/AH1YQRSFVLxfsaogHDXaEM76WQLr+QNgPWczCLqQMoZFd44YQsqWLtUz46XCaOv79tdvv+L2hFgmCKZiWvyoX99oxxBlCVIUzqMNHd8NEKXqUW7DrzugMb/WEFIxdKnz6T0RvPp7N9x88w0333DDDb+632nrOWv5xWIpdP/uru85ae+JFXFIio+lwK/Phq3vMrD1/b0v5yi34WcODPtHDSFlj6V6xBM9eDviCRB1s5O+51yLotsYuhS19TljPWegih9SwFGoRxkSZFwc5ULKPmjMrzSEVAyWOp+e6MGv3/C922+//Ypv33Db/VogOLaH35cyKTjqtmA6hhf0qDd05pQVpBQViBtSVI+y2h8FHCXpUSid6lJemLAbbYuB2O5PJgXiHlJLwBBS5hY/XU/0b/w0Rr8UPD5zy7CaL7MQQOrbX7/iiq/ffvNtibCfE1zCVxCNs0BtfWZjeK1ZJgi84oUU16OsWc81tj4KZwIp8tWjsPGtukcsl4AhpGJY/OKceyIcCOl89YJCC47FsBQb3A1C6ltgnLj95l8liqVMx0tRPcqMoyxZz2kFj9c8YU+PQlsf749i8ihLeZb9aWplRYmv6dXjX8bw1jB4Gm50RAkYQsqGLhWjX8rjWTz1y7JwRAZs7AqWuuG2hPRJIU2ZWPyY9dzQEYhaz621+qCax8dSKBz7oyx6IWltfRJLnd7b0L7mI6/vdP28PT3nnmnjoV88lxhCKqYuFZeP37Ift7bCZ92wwe4NLOuYl7UoDF+vendf2yKPb/XHrV+V4Kd3fOFAheKNz9iFJUshBTR1XlgK/frSTCq1TUTFB6mKErT12emPGivZ+lRIBRafKnqnZVvZsk2ttXvcuV0cwa0hpMxZStd7AhySNOOlwr5lC1tqW/aUkTlKSnzv1B8Nn649qgR3t6/5yqesa6lZRVSrMNxJ/IDikLriitsTZ54wnscPp3rAQX9GNGG1h5c3w+KCFPhMIKJs9Efdi77n2oWaJzwlp+tXKb/eMWLhHuK24Uit+v91IoaQislSscdLlfhWd+4pafqkTJn0yZhDZU3vdp6Yu25529Ki3Rv3nvA1bdrWsSq0ePYnh7Y2vbo98OnLcaNKsNQNCTRPGFj8qO+5caW2jah4IMX9+vSt5EaxqEdFcxSz+IW8yxa+v2h17XtNwFLuTJlOvAsMIWXL4gcTOMvPJdy0uz3/vQrfYvg0+7xV36pvaOhsaal5f+vu/LcOLlpc/1T9qqaF82raVy1buGN0x4mKeD+gLelSCTNPGPRLMT3KsEvIbqsP+CO2LlWhfAjzTNjUoyI5ikHKN2n3yTLld0eO1S4/+J7P64RB1olqeTGnYQipmBa/2CwFs2c17axvbyvbf3D7jOtWbO9qOTq5q3PV5ODubZ/VH123Y0ZHVcWkzz5bWL319N7lWYvibncIlupxXSpRehS2yGJCio+PMnLT1WcpPVsfbQCSht/qjrrR93z46b5WGKD/nuJCygEsG0LKXJeKr1+qpGlRScULK2Y/drKk5Heda7/YezQM0yqEg7uztm7a1bFqWf2q4Lv5hZ25i4DNThTFGM2g3qzEUj1r8UNVBvSoRHBUbEiR+fpscdS9g57B8VGRC0LK97t5DfPmrTkQDC7rOEn6qNRidkP2SsAQUg7oUl7lL7ueT9908MD+FRvSOw4eBjgF3+lsu7YMIPVCy47tAKnTLSfOdVSXfdBZuGKt3GQ0vRmJpRLX1aujS1GaMG54vQmjOVIs+kyo9TwGS4XDH4JlwupcSEZ6FIolLDVp2mcjPptZ5itq+uEBt9VnWu3iPWgIKRssFdkvFfYFd9e07zhUsmxhe82O7UoXMNSyTfO2le3etvV0/Z6tq/euatpfU7Nr24yOE//ZsacoHOczZSzV0xY/1iU01NzWZ7k/ioPKHFKoR9m29elxFIOUqCc+X1iE3UB3SsAQUjFYKi7vieDpkuK/lIXeCQW/mLT19Gpl8Wll0jtbR71TNP6dYGlXMPiXSaF3Fpe9s7hi8eLQpC7rFj+HdSmJ8nR8/Ki5zdzWZ5+jYjT8AqQ/yqpf35tDh8HMLc+w8VEcu3xLWIp8Q6cC32XQN+guTpSAIaTsWfzkfqmwUvLXv/7vn//3f/8Ma/j9GX9/5T8e+9e/iFiYCzyuJVG61EoJU1E+fqhHgduCKUd1B1Gm5gnWH2XYGWZgmQCfCeiP0ucoxlIujuKqclZOMoRUDItfPHNPhBXlr38FWAX/98+hd/4cPP3nSU1/fif4578E//q/TRj71+Bf/hx8R42dFIwv55Iu5aARPXVlTr4KqkhPdKZHGXayovW8W4gyg1RAoXqUUZPTAFHIUVE+E5yjXEjFV9msn2UIKXNdysh7QtsvFQwFwf0oFKQ/2LIwDcAaT5A2cWZf0qXsWPzm99Ybtpi6siA/q0CMDonw8WN6lKmtr3uIMoGU/f4oHZ8JS5CqcHAJxPzujsdBaRWBOFyyAwH7Eo3byYaQiqFLxTVeisGFAAfBo/dHItmRYHxmXAKp28EhyY4ulXo/LAI5IpC6pCAnL29jHm/7aVkK3b9jzdLcTUQZQ8oDelTaw87qUQgsoku5Db843+Pxn2YIKXOWiqdfigCIo2jSYv3ldImAE3JWXBnvDkul7q+t7aiNwhRwVPaNefPzczmBabwnYnEUNMi62eojNXzkCD1XRy9p9Y3Gb3NYWHB8lIkeFQ+kfEpg2uy7HVtmzw2b8lRAGe+YLEjoqfExOmXAJDO3G7c3e5rRx4oMIRWDpeKYe4I08zgJ/e2/YPlnXOGa/nDzbx9w0LFtHKDqhi41//5NDXsbdqk6E6Up4KjsXtkFqUuWcNqSWSrxehSt4bqQotZzWxxlokfFASmPEpg9/MF7nVkeTOuX8ZTpJxcBUXekOSPs3ntT0voNHGoqDqvZiOEP2r2/tLQHpxhB1hBSNix+mn4pRj4IFGjPTSJg0lu9QE4g8MMz4+EpxlJfB090y7rU/Zs685Y8x4HDtshRvXpl58F3SQtY009iqVgc1X3LBFZwg4YfWCaIXx8IsbBQjnom2cB8TuTFaPj5fKHZw/v94AFHljse7tdv7N3hijAZjQ9DfaL4yguI6jfcEWGQSP9+aW/MAEh5cRQ5jiyK6vH0eT0j+vebaFPgD+B2nvV5YVASTV/TfDaEVEyLXwwfP04+BFrBSf/2z//+63+IWn73f//5BX6i2MbkqW6yVPu82pUaTBGO6tWrV0Fq79T86iVEnVJZCvujwFnV8PtRjrT6jCBFlTjLrT6w9UWO4aUwktemuhQg6u7haY8PvvUuB/5uxQ8bjLzbhDaAo57t1+/OWx0Qdtett47rn9F3LELKcIHvto7onzblIVvybr3r8bRBfZ81St8QUrZ0KTFeipITsg75C5lAip8DW/ozLAd+QNKlopQiDVb0dlbuX1LbsISrTHgGRxSyVOrK1lbSKhQWP+5aN3jwZYP1F6JHnTMlBLkmG4X1vCeo9dzYBUqfuOg8E7p+fbJwM0j5lIrZw1MeH3rZm91fHho8euLIvn0z7vYpq/d9FFKCr+1r/UrbGsFWHyDqroe6L+3NNwe/3G9A3wyAlCf4buPhsHJ6X35VmdYnG7iFIGqwDXkPwViZe+F2nvUpy67bsNUT7trXWlkhOagaQiqmLmU6qleAicEEIRXFUf/wH5ylKJbi5SnBUnbGS82//1f7GwZxwx5CamUBMBQsoEvB3hJiSRfeEz7gqOH97njqqe8b/t1hdS4kuV6rYR1IgR5FxvBa748CT9mYGDeBlEchHHXZZdYk6yN88MyJ9w566Y2+d/s+6DxyAobNtXS2HDsgDyEmiBr+ssWebH1plw1+efiAgS8hpFbvazi2XVm8t7OzfVWF7GQPc2QDon57k2G/vUHSNBpnmnr99UF3+L6oPXJyUXDn8s6WNV95VJu9IaTssRTtlyLEo4GJCUsRcuJoiounJF3KMkul1u7bvxdYSmCK6lEEUqhL9e6dR1p+wnsiMLHf2JQn7733/xgtT6YMsO3XpwJKT5cCWx/Roz40fcKRB7keJaetGzaGFNGjxgJHObC8eRci6vUxbwyYrTwxe+GJouAnh5V17V/xJgdsiR4FHGVNX9TP25t3vdwvIyPdDw2/8OnKd3cd8E360rN6b9ZWaa4aGBZGOGqoDYHfHfzQHfB1rjG/GPSs75HK/dsWBT9dGnqnYZWvRKhrhpCKwVKm3hMcILwpFzLQpf7j15IupYGgVODRQcFSNvqlUvc3tDcMUpHIW32IKcJSve9fWbAkVbCUEgJF6jtmy8P3ZqT38etWWkuRUSzFlDiL7+434dsc8XCUSb8UsfWlPX7ZZTaqXGRFf/Ou0RPv7fv62THAUuHwso4T2PUY3F93WGUpJznqu4CokRnpZzMBUr5A4N+PHQiXlChftLwnfVES2oCs1ReZ2Xj2yRckz816ZtCzJSUw8G8rfGJFeQGGUcTFUo/90MBM6PU9YeA9gfP4KQgo+Gn+TFiKni1Op7vROJJiJJaybPHrnbpyycrnhColEIVGdMpSvVduzJdmSApMvOWpYaNnGi2jZ14LjzBmIysOcEVCyrYeNeXJvoZ+fXI+jFjKg3rU2MfxC/XdXt4krb5X+vjPAqSUokkdJ4qgRr+w4qsKYSRDRL0x/E6Lbw79nCGigKOGJCGklFDRuuXbfSHl9N49W1VdStWj9NMwi4UWAHJUHz9CKhws2p21CLwoulpWVQQESQFoHrnmZ9/4ms5y3WM/9OpPRuBVnoCuXiOLH7yEBEoIUHClr0v9g8xSGgia+lFILCXab6gUxbXMT00l7Tt6stCjhC4FaMqrzksVFr/AxL9fe9XlhsuwYdcCSyUAUnb1qMuhpR/TMkGQZQApokelfN9BPeqVIUn+WQipYFPHqnBJcN3GtXIFROv5y461+uAFN8SfTCAVDr6w67DPt3jTCel97PN4poEe9ZCht6YJpGD0GfnKcZJ/zC/63hGGsbTbysKhLpihSF4MIWVDl6L9UoSlVHwQVCkmLMVbhzIOISxnMiLMWMrueClBUcBY2B+FTT5cAUtRnKUWVC+RdKn/c+31Mw0hdWWCIOUBywTOM2FHj4qLo4wafkSPQkQ5sJBW3yBAVBKw1IC7lU9f29v6/VvXNbR9kr6dTTTipB7FOcqfRCE16ZNNyzdsP713x4vnZvNvSnLruZ27Q46iHin+Wa8DS316tmPjA4O7jmwbc+5QmaqrGUIqhi5l7D2hEBAhpuiPbPUhpdGlVBTSUASOpF2JpVSlKC6K0p4kWn29GLA4pJbkrZRY6nxAKhC2Z+sjepTRaA651QdhXZZyVI8aipYJRBS81t/oOzu8e15DzZpxu9sbamAezjA+USf1KLD1oR6Fei2BVLDrSHvLvBNdDSAOBsCSlqYDrT5skQBLDXpWWVfT0DDv6Lp2mGrgJOhUvIYaQsoGS9E50QmiOJoYTmKylIo/jiyeQZ1tt3QpFVUCUYyquC4FZ6Quyfvm1WEiOjDxPECKtPqse8oOHhavHoXo0oMUQRToUU4sTI9CRDGWKh01fvL4otLJk8ePL6WF67AeRVp9II6yVPDpUZNHFYdGTR4/arxwyIZW35SH7KiJaEmF792R9xXTpeB2RpWWTh41flSxUA3RgGmiS4Vt6FKgSnFU0C3Z1WcpoUuxSygQ6VqUQTSmHGIpJRHl2QAAIABJREFU3uojFIUOSdTiRzCXV/32fbS9fz4gxTnKYsUeejmO4Y2eXQzrtM6iAylPuAh6eJ3Ro9RWH8gmulQYFnia4nUOlS8BehTeKYWUD/pfNeP/wQuJIspGd5tmFg/KUiXkdkrwnlTd0ARSdn38CDr4imPEjKW4dZBco6IR8mm0SLqUdYsfZymtHoXKlNCl4JT5rc3/TUF9HiDF9CiL3UJvDh6G1vM4W31Q76IhJSwTFrGsdzrnKNq9QHWpgBcWpQT+CKxIq8+h/ihVj1IhRcT5FJRJ5IHreHc4Ci0T7H2F5olnfZHps/pqylImFj/DuSfAOsEBpeLDwCFJ9p6QKI0C0QhQbAJnu+OlGKREq4/qUYSpVJaan7okq5Fyec9DinOUtVfpd4cOm2I2hhdrmnaJgpSjehT2R1E9CqWiLnV3xNCVQJhylLX71IMvxFE9agjrH6QsFeHjB0zSLY5Sv3KclHQWjOhAsrqLIaRs6FLE4ifhiEIDI8xYil/Az4YtuUQ3uyTSAZbiiGJqFDH4SbpU7/nPvXWcNlESDyk/rQe8X4qMj5o4eoaRS6G+o+HgwZeDp6wFjopmKeAo6I9yqNVH+6N4FWe6lKRwUMuEU/1R6DPB9ShEsB6kfD7W6jMApVk06lHQHyVaAKThB8ZD3cUQUjEsfobeE6hKEUxI2xjeE/xMRBK/1MyK3n1diiMKoQRLNvXyU1mqN3hP3Dd3PbYXEg6pZDYEg0MKrOfD+0156vtWlyng9oPWqLiXCJbyoR6FPhMOsAbTo1S1jvZLyZCi1nNn+qOYz4QAsB6kuqNHRX7lGFRD51lK13tCw1IEG5RwcLyU8eAO6UR6Ol3rvgBoJGMpW+OlSLtPIArbe4SoMCDrUvDJtivaykvhReQ4pPywqNXePyS98XgyRnBIBab0S0t70sij0DD+7yl9Y3vKqnKJwAHT1LYY56jBTnghRXAUt/hJkAI96oF+DvpMyBylx1LgPdEdPUrDUdiORR+/sH4dtc9SBt4TgqU0dnEDSKneE5Sg6JpylX5+Mba7LCUQhV28RItCqtJY/Obff9sv51afAmFOQ2rIrFljZomq7U86V5iVm46Y4pCqgC7eH5g5Feod+8F3+o8cKCFVCDAJaFgqgXoUZiFSl6J6lDNeSBF6FIqLavh1R4+K/sqxbZYy8/HTnXuCsBQjHYl7IGjGUhrokcvIyhhRDFK2vy/FECWpUTQo61L4YVFPZd1mj89hSJ1Jf33qK1PP4HPHJdlfmF2Y24aNdA6pwA9uMXPX0PfjuHLY4wMezSRJxr2SIcU5ysFWH+mP4pmhFj/BUtjqSxt+px2noCiVh/meq60+kBkJKZ/H6yBHEZYCi19Yv46aspSpxc+IpZg+xDYMWvosJSx+QoviTT+EoeEiWMrOeCkx4hB5CamJEhVqU5IuhR8W9ZRuOAROwI529fonLDwCk1lzPklOKszOeb6wessEv4BUeOLIa6+6Uh85xrEIKZ4qr8cxthKkEtAfpepRmA2tLuW09Vzb6gNxEZCC0RzdsfXd8ffIWTzQIemOHrP4MQyRjViZsRScxK4hAb5rCCje8IMvytv5CqLa6mOufdjmo+jiPn6ob4GPX4lS6plb6nEYUpuOPfPKK5Sl/MhR07PnJFWln03yC5YCgQ8ZOxXqo4qwlG1IMY5yQIsC4wbTo7R41rAUIGoK0aO6L/C739Vaz5lULaRoqw88ZaMYLnZEpK2PCvDPAl3KwBECkOa094TAB2cc3BqwVKQuJV9ijCnBUnbGSy1h1j1ikGBgIqDS6FL0i/KhudUvKo5CKmnCvpblnf0ppJCjCnrlzEk+m74lU2Ipuw0/u5CC2cWoz0T3q/hl36X9UVqO0upSjvZHRVjP9SAFLhTT+qdMGXaZjWH4D3Hf84g3xBjnLX6muhQnGnVrxlISSVFE0cuMEcV1KTsWP5mjKJAAYUyr0uhS0PCDHIS2ZK2fYlrDLXqi+yd8XAvfG3wArd3IUZCHbIDUluqqs8kD6aRjMZQ351nKp4D1vB/x6/tud5c3LyM9vFFDMiWW8gFHDQc9CtS2bkt78zJtfxSv+DJL4XSE/fs9DrO82Fn0Z0OkPn6yF5JUXe2ylJn3BCIDfgIgENBnKdnHj14jLotLl7r5VxbHS0mIAn5gehTWa4IqWZcinugeT3F5ef97zawFFiF15syYMZ993DIc7BHIUSg5Z85Z/7kFWXPGMEiFJ5pi2HlIYX8UzPd1hzPLw+AzEclRsi7lVSY/MDytvzPC7nhg+AAcH8WhxLcaSCGi+j1rvaOPdAz+VtcjBR2S7OhS9ix+HEpcMSJbM5ZS0Sdj0MQ6wVnK+nipaFsfZSq61vRL0Xn8PJ6lbVtSHIRU0rmC53fuW/FwH6pHoWCEVHJ6bnPmwAFktllgqaeu6kFdyqOM7zc8JcV4cg3DrjC9A0+m9CWjOXjlZluVpSqUaWn90lL0rrURlwLjP6MRJZsnAsrk4cPTUm4ZOXLkLdb/7k3RHSFNx0vZ8Z4wsfjF9p5QsWXIUsLipzIaCZGVRKURQbu6FCCKQQj5iSw8QFpgUSwFLkmeUP8BzkHqjH9nbUtDQy4MkoNWH81E9pw+fv/Z443Jj1JIhc2VN9ss5fdnat/mzOLnCY8fnvbbO/7VmeU7IzMydXw41H6pivCIlOH/6pC4O/oP+oXGWM8QLLFUQJmRltb/YZvL8JEDI14OZJd09Tps8YvlPQGgED9AlBlLcegRHPEds3YfZymrupTU6qMWPgoqrkqJuSeogwUfL+VprHMOUklJZ97OqX3yzJkkqkdBDkjDLynpXNKcBRmcpcwE2oWUf8yYMbNkTIGJkXhPeJTi4Q+OvmrYTQ4sw656auRAve4xmaVG3NvfIWnXX/7bQb/QAbCWpWb0S5t51U3Xw5/V37DB0DWhCyl0SHLex8+oXyoCGhRZ+roUG9VLT+Fo4nsR1CTt2mMpjijexUt1KZmsdFgKLBTDs6r0qzGJtaxLTYCqDU+J6FEEzTnAUhDhL8+l5okE6VJnPq5tnTJBrR/JYGKkkAKWShk3bOaV3V8un3n5U4KltH5XZO4J7OqtUEbc+/BNTki78srLZ35HQMo/ZIjKVzJLhYGlRl9uS+DM6x/vqw8pMqrXasOvO57oAlU8YMZS/BwKJbEnQSgySCFFvCfiHy/FEUUxBGvS1pP3dHQpkByauKB6wzDDrleLkAKaOnPGfwb7o1A05IGxlD/5eG7ziDCMR0iIxc8/a1NLS8Ou54Xbhv95/wTOUgRShndo8j6JOnTlMGQp5A3/kDFjoP+aL1qWevgmR6RdfvlMlaUmZGamq2YRMp0LGdwBDb9+aaOHRWU1noiZVz3e16DhlwCWMpx7gg7qpfigxIMo0WcpYfHjzUSJqWKN6rU6Xio1DzHEHCa4CkWNfkyxkkf10n4pxHJoYr/yqssNMWUZUlDJaH8U5gYXxlJJZxfUveiB93hCWMo/puBfd+5b8wAfntCnMGvLGAopRXEWUlSXSj439fWpKimq3hOUpZyClGCpM28daVj+MGsE+vv4cbZZCinCUjYhFS9L+eShWYZGdO2oXo9McjCPn2m/FAeGChQzlsKzBTdJl0Ryk7pvj6UKcnLQTZbjh9ZnQVYQHdEv9S0qMDDxwe9fP260EaZsQIr1R4FAzAtjKfTx21iJtSAhLJWUtLOgoHMBr+NDpvbKz5rzqNClUsZd70glV1kqvaOm/dhEdAcmC0AKZkhSG36OSJNYasjbe/P3Ld/1IhUG3v2D0mYg5UNxJoSl0CFJ6z2hTopuCCkcL6Wx+MlTBpjM48dH9aq4wpA+SzGLHwGUumIYUyEUGbKlS6XmZOXm5vbKySFkRbAluArhhW0wqZdLM0PSzGbDpp9lSPlZfxSHtGApUG1GzCjyJIalwCzS0t7SzGr4kKnZvdKbcweMxMEdaPEDXcqRSo6QQpY6M+ut2lc+bmiecI5BKqnPo2xUb2JY6kzSnJfXdTaTVwb2T2Tl9iBLQTfEXEAHnxbdEFKyLuX1zFgP9ZoDMR6WCoYmD0aUUHyYsRTBEp4myIqEI3Ek7TOWsmbxSy3Iz8/Nhe/xArCye+VkE2QBZ6m40telkDRuqqp+0eA1bhlSyFFUjwLZGpbKSDtefsjjSVC/VPq5MZsaniR1DhHVK2lO84KB06ClCRUCIGVwe/FoHNI5gqXOJKf79x/sz7QpGBRW3pyRSJZKOvtWQ8uxKWdBVyQ9foW5zYllKdClwug9UQEfrwqFhpY3zi1F0wsuhpDio3rxiqLQ5twtQ0MKm1gaZ5s19p4AGJFlae6GoQxThizFffwi4EQARrKnu7LHUnk3YqMvJ7+8NTcnpzortzkHKOtGbHsBrLByy7qUOic6Qurz0W25L+s3x61CirX6VCBLLDVy3JasyYo3Ef1SmTtrC17ft6IKIQWtPlgy/ecyFkzzYOMoASyVuXNfe/vGZ89SWkzOLMzJbabfl0oMS/mT3wZHr40oDks4O31B3SFyb0qgJ3Sp8eXVWVVLiyhRGUJKZin4qmluddYGYCpCbrFYinwjPrQeLxkMahIAJAZLEVqSSQoidMFEIyVdKv6pMZl5olcvpIjs/Pzm3PxeuVm5rfnToSk4nahYbE500i8FDb8wkUZY6qZrs07pD7iwBinyBsUKTRqa0NTMlnWpAXOLsxqLY4wmkThBCsb0RD/3cUNN+4L0M35AFCHozKRZA+oqw8hSxc6zlH/CWwV5Dc3EwAjTKRfmTG2se9aDJidnIcUtfkPerpr23x0HvwOOKNjjlz2nb/5cH5kXKTG61Bi0+IVBwOZTlfB3CmoSgGozmX3MEFKUpTzKi6cq4aItWblZgJC5pJbROdENvi8FutSXlbhsyUIxG9YTytHXpbQWPzyRgQuDRJTuSrCUlfFSAlKkQgOuoGJhUxCABUt+dnZ2Dg7uSKX61HzqNgviEVLXz7xp3OjRuk5CliDF9ShCjRxW3OJHuokO5c5QEuLjd+bcSzDyEXpukglHAUud6zMwt3m9EkiELpWU/FbaZ2/X73oWTHAEUTlTf7GxCttEDkOKW/yGvL28s7Nl4+PnaB96dnpG2txTxOaXKJYiPn4VStWx6rrq6uosqOwAq8bNcIeGkGIWP095HbkmF6+pzq1aD4P4jb7cwUb1huASXACFeFUV6FRmLCVwxODENrpgopGSLmWDpQikpBUwFkDqxma4v7z581euRFTB321Xk5ccg9TM679svvYmHVBZghRr9VEdinCFxuI3coSnori0OCGQSko+O2aWylEAKTAxjs0tnwwKcgJ0qbNvtTS0LP87UW2Ao3oBpAbNLgZ1w2FIcZYCXapzV2v6GVbC2c/3feNQbiO0xBJn8cPxUiFlC9T1uo1Y15F0yiuLTCBFLX5epbyubiNchvgA0mk7BCqY6Zc7YIRgW10d4pCgMKu67RBwjj5L9aSPXyroUrhw5wkVVNjqywbGam7NX1lQ3VpQsHL+kvn333AfPH8PHyh/5bBxueV6lnQrkKLPm2pvKJ4qVOjjh3YxMgQxpJxqm2jq+i619qRgzIYfCMBZZKgehcIBUhlvbKgCmkqALgWTjL21r5VobtDqg5dHzlSYE72qCvQGZxt+nKWSksArZRZHVK9eczLGjp9bd8oT8CSKpcg8fh7lEGnEbUD2qG6rLMZ3vglLXUO+LwWtPmz4wTXV1Y2HSAdsLF1KvSQLLvEAosxYijf2GDthmw//MHMGi8RSFrwnSFcvpQbJOIC1iy7Tswvm916Sl9O6ccmSrNacvDFBUkC04Xf5zIderN5webSl2QqksJVPF6nlJ/VLgUNSSNlc3TY2ET5+iFquR2EmkKUGTPOUgvk3ASxFvCfGgCsj6lFIxcBSdx+qq8R+InRIii5I6f0Qf1DynkgCnxT0SiEFDA2/sUMVmDtEgU7YRHhPqD5+HvgkXElxedbG8tnwbTVcDCHFLX5wDny3ZC6AsHFzsYc0hqjFz8jHDxpygCFFWY+XHCqGFxPEGLCUZPFDGBEs8S3mTn8RupSV8VKoSxEoMTzhRgstNKKn9p4/H4GV37xxzOTy3CpQBaEddtWwy4cNu74q6+VhUU2/+CHlnzA1hzMUhxY+f1WXAkh5PZ7KrAWJglQytUyQWkchpVRWL00ES4HflR+dg5OQo+BuCaRKNlRvrgg5CimVpZj1nNwbYakZJcWH4KXoSazFzxMCM7pnfW55JdT0GEZ0bvEj11RszmrbDC1h2ptlylIAjZAHxkTgJYfwEsSJKUshmCIVqnhYytp4KY15glRpuuKEgQATXb2pvZ/71X3Fp6oac78syc1q3FAJ/hOXzxx3+eio7htrkCJ2fIorbICSfxwvhQzC554oLd+YIEgxWx/tMsD5Y4ClihuzJpc6b/GD+wHFjXEUvjeQpZTSxlOesKOQEroUCOSaKkAYWCoFviivDN0yXkkMS8neEx5l8qFiqPaMAOJhKUDS+kMhYCjmQGGuSxGAwO3AJTCdPHIU/OmzlGTx06IKL9FnKIyVWMqKeeJG9gaDIldxhHFQuRFemq5e9J6AUppRGmhsLs9q+7yquWpD5YaqKHXKAqRmgdm0OR+6mUG8lAENS6Gm2v+Wp6KQG6stFFuXgiFayFFCMGMpz+Tctg/7pzjsPYGvCFiIHoVFyyDlKQ7NKHYUUhJLqZoqqMagSwGkPHOrq0oV9ETX71SMVarxeqJDRYGuIl5hDSGl9fHDBp8Yam/KUgRRAg9kz5ylGJjodVyxUjPIMyptu6NLMesEoArrF0ES1DJS0SSWIt/qxfeBp6RiyoNPLR33eWXVNrRifhk5d5EVSL3emp+blZODNnvQL4hQEBzJUhUTn2yutIqp2JBKpq0+UsGxkmcmE6s9KG9VMwBSVgXq10bhPYGA4noU3illqZAyuXyDo5ASLMV7/OD+8BbT8YvyAbAe1FV6ugEpQ0908q1e4fcq+IBUUkNIUYsfr8icoNg15t4TFBdE7eL6kT5LyT5+IlsYwD8uOnprk6VEbcIapV3IIZmluPeExxt6+JaXP5957YZxYPKr6hZL9Zma0ys7+8YcsNlnkd4wAFZ2dkFOOrg0wOAi3vCD2WZzc62yRkxI+TOnouYIP7xXCDKWUuD9OnRkIliK6lFEHjAzWPzAIckTqqw7NPcW58wTnKWgkSlsP3h/lKVgHMGpuqHjE8xSvMLClsHDYNIxrktF12jslzL2RKcCEBSyKH1I/cPv/u8/v8BPY0giIDRFFB/Vi7qUVYsfFDfjKSx4eQ+NF0KXIvP44Z0DOU988PENuXW5pzZUV35+fdTL2QJLgXkCBJIlO6dXfm5rVnOv1tz8/ELQ4mfNOjth4Mhp3gB03ExMeaC58Uod62KUcDUiBqT8s44DNebfCL6N00mrM5uzlAc+Vg4GkagGrZq0lZDMUmjr428xavEDA7ri2ZK1oV9/py1+ybPYmE72eNHihz5+4Bmy2TM0UeOlwhQbyCCk8uIKokxZKiyojV7M1ubeEyR1xAkLkI0+pKJG9apXxWQp6+OlbiQWPoIjCMGWvbIJtKCm67LU+NltC/61vKpyQ1v1hnE61dwCpM6qkAJp03vdCDUOGKu6PHNLeeOW45kDx47AAg5MvOXlyupKazO6xITUltbm3GpsdOajb2N2wfSkpFk4uANtuDNys15+yBGzNvdEp3oUvEG4TRUafsQTHSz2jVXD+zvj9w6DOyhLJadvmU6b0vBIcREsBRje3Dg8EbpUXzZDEqvlHFXwAA0hZc5S5nNPEDHyKj6Ln8gdApLBV2/DdCk7LEVfm+zlyTbYHsKglqW+eV+4QpncCP1xC669cubnYPu7Se/dagFSEkuhNHz0sNyYczx5S35WVnP6gtyq2XOLFe8PoKv38ZnWqngsSKHs7Jwbs0Wjs3VLcnoGDO4AWlRK/57VNtoKGRmeK7EUcBSp33Cf8KO6FBqZPUXTFvR3RnPj46WAg8EPIQuYH/zK8E1ZgLrU0EAIbg5NFOUpds0ThrrUWfyiPCEc3sgSJALvKIOGn9QvFVmt47D4adAB0vRZKsLiR3MHl5KrI6Wq+/HqUvN7zycusHR+Fjqql73GaHXmREWr9418Auf50D3189t2VuYuLW07NXfi/7t2adXL45Y+pMNRl19uAVLIUiiev7lZHsA8kZR8Zs7U5AW55bm564duAVqcedeXoLYZ1tzoA7EgNUtlSGx0gpmkcVZjdVbjqVDxjOIZ/RfAuOXoRK3HCJbKbCzPRxsMmIHwjcV1KXiGITAZNDrLUv6zc/LhlnKyAVdZ+dNBcM5U8J7ACuOp8HherIZ2rfV7gStiW/xIXaU1lld5M5ayN48fRwQXRrZmLCXgrV6BUSqCokJxWvxS74dFqEesX4qih1dssYfVm7NUamrqyvkrF4BD41yPp0SZmFK1LffLq2DyEL3FAqQISxGJsBLQZt4Tycl9Hs2YvX5zaH0VdJFvOJVbPc5KNYgFKS1D4t3OSZ6a31y+pejUjvK28ievrbRqENErDHjBPEXmnkj2t6IXaUE+KHD5YIOBid+hXwrcg6Bd653WnGWxWasrCyOp94S/zxwYgjYdaBik4fiCrKzyjAWVm+fOIPaC8c1ZNm/OeO4J7j3BcSS2pOaasJRmVK9as2N5T2hBQrhHn6VUix/kRM4VhlV5kaH4WCp1fy0s8zmmuI8fYQlCFLR207qNLT+iS8HpK8EpKW9+XvpkqADw699cXQUUZbBYgBSwFHljY4XG9zdbWL8UzgE2OhSC2ZSr/g5qTzV4FBqI1IuOBSkqm4vErT9pzKP33lnkmby0siq3ORfGg+m/M/SkGcYJlvJPz4EFVEXwJp2en9vcOn1g39n0OcI8fs1ZTzkhjetSwFLM8DMdR+7cmJOT35ixoK46q7x4/ZbKpZufHKk/isDwNviBGCxFu1xZvaU1GNbGDT9zXcrc4sekEDARqJixlIolCYrxsFQM74n7NzV0du4STT+N9wSr0QROtIsKHgVCChxml2zMys1ZAh8WxYa44pk8sXnDMOMKYAFSlCkQStToSGAsxkv5kwfCPH6lQytzy4e3ZeVusFYLjCEFE2KChZmxFHWcgBxAvxQKBO+J9acOgXmirTLK1YrXKytbzlJJfjJwGeVAOxOMIqCRjoROiMr1SnHRtJR+bZWfOwBgylIw5eeQOVRvQ3m4TEddav3kzV+WHoLGdHnjgrYvbZlfjFlqDI7qBacDASRa18laSYwuJTfiUKwBSxEfP54bmj1BV5HcpO4LljIdL3X/pvqM/YKkehNIATvhjxY8rNUQ7hQsyWuty+tdsKR3KvZLIUmN35LbP+VaE+ubBUgJix+VStYAasZSyWMeXXBIqaorr3oypbHtqeutcBSqdAbfl/LjvIHn/Chb6HAEy0mzxmQMmF1ZXZ3b1pbV/LIzBgOJpUQZY2D6jYVvLGjc0pi7eXx1c3lzv5tevtYKUg3PBYvfM3B/gqW4ULD4pUwG51RPaPLSuY3QDrTHwWYsBRY/OhkYrb6ERcjKTJeinuhqTRYh034pCgySOAqjgVgsJU5k18FGSIsOSLqUiUMSsFRDe+1KbqCIYClKFPwRUGzB8PmcvJUwXAqoDb4v5fF5NpdXV5qPtbAAKWAKQovqCjFNdKnkIUnPg9mvrXTuoeIZ5QueulzPuGhYseCAEaT8E97O2Td8CGcpvF+EckH29DlQwZtHvHhq6ajG8sYHx1kVqJ8ZylIw0FFlKVrEOXN+0Xeap2hGqPhQZSPQFUyNqJ+CtVjUpT7Oz0lPZw0/8YqEfqnJSmj9oS25VZvbAFHWkuVnG7MUzJAELCXQRKotX9llKeO5JyJJh+zrsxSz+DEcaTPYfV1q5ZKVHQ0Zqi4lypsSFdRsfG+Tf2otKFiycj4FFH5RHj6GMyOrbbJj35ciTIEVmuQDVgzV2XNmJT9/PLmxOrf5xdJQ6BBa7U1okT9u7dYAUv4zb9c3NKzoP4TrUigWYTU9vXxjbvOCuaEvtxRNHuqkj98tODXmEP90cpcUT7AG7wkwT0BDunRu5YsLsqqrDjkDqe8Mqm1oaGh+CTmYSsRNAbBUWuVmz4a6tlNQrk+mzLQnzYylBj0O1Zb+eI1nW0MjeoSPn8wV5iylAQiTasZSPEMROZQFRoQlljLznlh5/3P7WzKeYzRFzRPSgyZBXLFAARjRf96b291Tv3nf0kOlS0OeUPdZCr8hz/UZQlFUJKtv2a9vKa8uT5o6dWDfEYrnVF1V/ycd+woiTAFx491L2tsmJBMjOmn6oe44PT15y1R/RsazVXWNk0OOzj1xC3xHYeOLSf+EBUurN6IY+qUyZhd55rZBI6wxK/cUTBatfSfY2wOWqn18596DUxj/s8ebXVgIdpEqz4y5c0OVlUOhq9dm8ob9UshS0CEg1XWOLrSr2bP4GX+5Q8auEKnPUhqLX0T+4mMpzkG8eSdtU2tr9+9tX8LPoA0/okuxuiw2tJr36rVEvRq+KP96XWMpuM3GmF0ljoYffFIKZ/hHlhJ6HNIjqXSwAo+/qqn+ZHBIGrE+tLTSM2Wkg4M7zr6St2/X88m80UnuGRDVx392yKyBI6vLN8OYBCdnSPr+oOWttSsWTOBvDfbOypk6dkFjY/H6xsq5s6ubhztjnADzxA8G/SIvr/NJbvFjpVp4bkFu8+xi6JCoHhpUJifGxw9Ziv5xNDE1x3HvCV1BZiwVkS+evwhmkncZS8Ww+KXub2hvyHhuJcMUhRR7wrSVwF+irHYL9AGyUpfkV5/C4ZPgcmf6BbU4IHVuX35t6zOzhD5Dm5oM0JCTwuf9Z+FbvckDxzZmHQIl0vYX5cmwK/QBVxf/zo55DTCrCrrs4gL3TzjqHM4ivWDsqcnoiekoSw0a9NnKFWkT2O2xTfbUwqzq5i3Fnhmn1hedSnvY2IJqjU5m/rbvpvb2FWlEl8Jni83a7MIbBkaBAAAgAElEQVTMPjD3xPjSyqzcQ8UhD0LKWrL8bHNdCq0TvLZydOHWlKWoy4Vcm0k4Du8JVQTIgD8DluKjevnp9GRYwy9KrIgQFj/zb/Xev3LlyueEyU8yTzBWIo+APnOMkRCVOn/+kvxkGO0FItlAeV7KkduYkDoz4e29LZ3LF4zJFBY/KpNocRDMSR+Djuj+swtys2DyAo/HHMORGaD7BroUjANMz3zryDYqG/EENQ45yp+U3lbd/CwO8wkFHWWpjNrlLRvPnPsnoaeSSu6fs2DBbE/oEJh7YHCHc26z4OOXnt6xogqfJX+shTARFc6JXpq7oRjHBXZjcIfZnOgqS2EZSj8TXcq2xY8LUEWZsRQ7i1yEYRYQAIoOxKlLwVRHgA3emJMgRbFEngCHFRjQeQsRLluSn3P/bfcRRHUbUklDkva2vjR1whk2UB6qGnn+uIJ/IKn0czj+NTm5fOOLiGGPufJmCVL+sx83p791pPVxbp4giDrnTwavuKwFz5ZSO7Cjo3ofrO2sPZjSh94drDFQCIMRMwbc6dlS1whz0I1IcW5wx28HHXvl7drlaaw0UU8szEz2T3hpUHNVCIw9OB6iGw0/I10qOX1gxh3chs5rLK28hKUe+9k3vqazwJzoJixl/OUOgQ1ZlD5LqRY/AiUZig6wFAcT3TLzBLML4YPGp42vUIyariKq9/y8rI15xOKHaO4uSyUBS0HbpN8ZavFDofKSA60UaLDNmtOYPnYB+WSbo1/uOLekBaf7PyMYEjkqacKczMbWe18spfq1w7pU3zt31m98Vi1nRNSQzIED+1WWHjqErOHkdC4TB9XWtK9YwLp6KSEm+5NfAj/kygpSqRLBUuBxnPHwh7R+s5rL8QS7Jixl28ePJ8+3YAQxgBSOl0LOxD+24biKJicRI+lSZha/SEixvguo0Px5E1RhBddwVF5dzspUgFSYCOw2pM4kv/X6WzhrI/NgEPIhF/hOPQetsD7Hc7PmvJGAL8r7x7z9euHzs5hpBI0z6WeTMxtzF2SM/X4p3CApdEd1qdrltbUNC84Q7wlSztmFyYCojLTcLHCZxJbmUkdnSMp75fXCMcw8AWPeCuEbjH1eWlDdNhfa7Xh3wFIwXkqf22PF6utS8GXljAEPz+XdUqzmqjXYVJcy8fH70T1GMySRp6QKILtGuhQZgsiRJ6EKowSCogIEUtbHSzEAkYYXMScDR7E4zlHQLZW3ZGUe9Z6gcrsNKfAGunNnx647zoIhu4BQIhFKVsBRyX4YUN5YXT4HvqDmPEuRyb9guhjOUqhHHW/Oyh/0xuOs1Qcl7aQu9VTf2uXLm/2kXwpvEV4a/rPpj/ZtzmpbGoIhxPBYnWQp0KWo9wRVpEBY8tnMxgUZw8F3gtQ8WDmsSyGiMibCvMuUBIgUUXVxz4Sl7OlSiAZVDAvFYinNNRRiUUBSI2yylFqTkZjIHmUMlaNSV+bU5aEK1pt+UR5kdhtSfd5qff3jho1El0LBVI3GKkA5Khk+cnFjI3iigxEdx73GEKj/ZjUyT+AQfJyqiDIktPpmZWaWlw9EjiqByVLp43GUpQZkJE0AxZF39QJHJWcO7Nu/sbIU5q4DgaBLOcpSv8B3ErIUKqn4hoI3xoIp40sC/O4IS9ns6tXTpRhHUZIiBcgqOQ+bspSpLmU0JzoTwB4X7GGEfsOPjeqlp7Lr6AWwViEUGaKQuv2K280tfpENPwIiQUxEjSJVXLX1peblbsyjo6yc1KXg62Ubp5wVupSANnDUEPhgX2P582CdEHNPOKpLcXM6shTt4S3PTC9ERJG3OCls8pkBZ8za4JB0y8D0M0lnsKsX3xqgR/XJzMjNHVHMlXnHWQpmCwQfPyrMf9ZfVZ07YAqMlRL1KNSNqTF1LH6AqJHPfIhvB5DA/0QAIxxnKZY6bNSQuS6F2ZDOZcFIHEn7jKUsfl9KnW2WqlJYszEkOCp1fmpeK+/uhc8MhInIGKQR04iedGZC5iuvpM8iTEGbJ5QikaPA0jenPKsxHb8WKLHUU045JHFEEZYCjpoDfhr+VwZpOCoYdJilMnEmdGQpVG2S+6QvAFOBF8yn9Ck7rUvBF+VhvBTqydmF55KO+wuaH50yHr6Dw2u7s7oUafWBHuUTzUq15tJqHMN7woSlYnhP0BuiQjCsz1KqxU+CFS8LCUMRwe6wFOUlac0Rldp7CeCJd2PxGZJitsNiQwpmFp41CzDDWIpwFL7BgaNg7v/nc3PBxIydtBxS4QR8XwplA4Ln5OZugSHexDKhlnjIUV2Kff6alDBaJgBR5dPg7aQ+VicbfmTuCcpSIAxeT89kjL2D61FMpIMsRRDVHzgKOQB+9E8E6K4JS5lZ/IznnqCSIqSY6VJStuAq8YuAkbwr6VImnujadh8d3EFoCZkJKzZdCctEQXXzEtGLhZ7oVGS3WQrQQtwaqD4DWGIkCYg6+/zxpC0vkS8TqpCKIdCqLkWICmXnNU49t2XOmNcHvQF6FE6MhAVPVgljqcIx6WAqaATjGLzVmTTQpZ50sl+KsxT071VlNU/vO3bK+DBjRCYRLX4O6VJEj1qPnz+Uaq0oRxoH3QR2ffyMLH5CGH9i+Nj0WUr18WO5wVP5nwwibViwlOl4qShI3UgbXMAQCCpmn1gCpghYUnvnVxes5D6zEOGcLiUaX8AUiCZEM7ZSMpPPwkxgYNMmgFMhlQhdClgqG+bM9U8Ad0/V1kdKGhoqCCnHdCnGUvgBr8LkLc3VC/rPDYe5IoUYdtriBw0/HC9VmJmZ2/h6xtgHJvtIE5M6hmBtco6lEFETeauPVlRIn1VYGoK1TR8/B0f1RmYN9+GnRZFmT9KlrLIUqc6kSpMQa/WlzgfL+RLe6CNYdE6XEpBS+6WAqMBnIrMxq3zqEOiVogtv+CWEpfoUZuUWpvuTCUeFS0QzjBS+0yyF32mbnp39+jnwOc94Zm4AIEUfMl072y9FWSq7sLzRn/kSIioMdQWrEK/qIaf6pbgehQIEkFhtle/PlKVMdClj7wm4HfWPBQ10KXUeP3oFloTILGTcYBEsZcPihwQhcMURBT6y2P5Tm32JYikuGfSoCclzmhv9fTjgEsxSE46XzxmSPIRwFFZwXszk+SRGlyrc0ucX+Rn95wbQVADGc3zEKC4RLDUVvmo+/dxLg8ZOmexZ9tlWEHMPjOtgVckplgKOgv4o76SlZdCK/cZc/Lo7uyW2ZXsmupS9filIlxUel4fFqN/wI7PNSqfTS+naAE4YLbGUFe8JhBK2vJjZDbpdqR41P29jbh6hJmnlXL8UBw3qMyAaO3uh1XeuavqQOcnwPhdLIlkKaKPPBOQotEyEyVucPSTcOMpSIzNgCGJS0vR/as6a+gYiCuVB5SOPGaSBxe9Bx3Wp49WNU8e8lDEQ9Kiu2pqjoeC6I/PWHPXRmuSQLkX0qA+bdtavOOBbdl3NvINr+S1ptlDZlUds+fiZek9EyIBdhNQ/RC86nuhwMv0hePQXiaWEr6uEBv0g8fFjBEV8GLjveep8+PphVDqJ0aUoSwFHZeZXA0XxNh+BFYdUInQpEAAdorTVB4Aiz4essLAToEslp5dnNR/P6At6VNOPGg8HQ6f35VeV0QfrPEslJaUfzxwDHPXAeO+yfb061wZX57/8yN4di+gtOuI9wX0muvL37ToQWFw+94Mje8poRWUFKd5RJUbmCbujeumbgQpjUhhL/e7XkYuYEz0qb/HoUjHGS0UgS3iiU1gRjoKWXmpeznz4PG/Eyfjljm9RPMdQbeIwojMeohY/JMrCc1OzcrcIJYodH6J6TzjeL4UihB4FnEFeW2LjKEsNQJZKzixvTH8UOCq4el/L8gOh052dnStWkbZfAix+eHNnX+r7KOhRgWDRO/VfYe1p2lS3iNZFJ3QpRBT69YWDoRd2HfCB9+/qvScqSElyChDIMtWlTHz8jOeeEEmrD46y1L/rLOQzA+IK8aSxLPQZCmMllrJhnmDtPuJ7nroyu1qHokCtSgxLEYtf4ZzMOW0wxax28c8SPn74CXt9W7lhrLFDEpfCWn2i0ScA5bCPH7X4wfsCERVWFr/4LlTBSYeCq+u3wVsdpIYc75cCCk5GPWq84oW/ruVrQ0pYeWHFWoooR7wnAFHgM4Ft2PDfdh2ADxg07d9xQDUqMkmUhh3XpaRHpQZBl/qvf9Zb/uu/xJc7BJ4oFI0R1V1dCmmK6lGpS1qz8uZTM3oETwGkoPxgcZqlsgunlpcnC9M5r/H+Pi/1HUimKI8hUB9VMSFFOQr6o4itgD57/ipzlKWoLoV+fc+AuRnK8IVjB3zwFZx3Ot8j9oJQcLTDPn5QhFh4d4z3wt0BpDq/AhLpWn6U2+UcYKkhZDQHWuu8yjqAVEXw3WPbaT0lhaiuoM7b16VMffyoOApelFf6t//6N/2li7w3+fkkb/QyUqF1V/ZZiulS0PqC/qhUGPWbw3t6IxAFDT8Hu3oJbqj3RHZeVVbucXD15FiiW0QUfc0qCdGlVD1KvLmwlMnPeV0KfOsHIkcF8Nn+bdf2QIVyeu+eMtrwc16XIq8j8JkAcYo33NV51Ke823lUKWK32m2LH7Wef0jTLwGW8j29eyNwFKus2o3tfikT7wnpmTFhEBN8epLBInIWkUNdMNFIZvFDXcqqxY/YB6itDyiqtffPNZZzCVfgPREm0mKQhlVdqnBL1o34jTbtApUCGy4+/NBkDIG2WIohCt6ytJTJgxFPx1GWAl0KVClEVIAW4Au7DvsCi/edUIgBDoQ6bPEbQjgK9CgQF1aWddQ01Jz8tL69pWHFUXqH3WUpqketh84AcFRcXN/eUHOiq6a9oQFbfqwM1Q2ETHUpZDqdxXzuCfHQIHXx47IjtnxXkyWGSR3BLIpAyvp4qRsRT6jN3IjW8/kF1a1RpnMVUwnRpbKn50/1bznD/SVUWAGiBsJrlny6NQEsRfUoKD3+RFgRs43T/VLge44chX47yqRzm5ZvWLq6fteL577cSh630yyVBJYJ6oUE4jzLXjs+dWrlpPTCqYXHwdKIErvLUqBHwRhe8HICDgys/tPUqYUvT4LV1FNgUeQVWBQoRJjoUmY+fha8J6hUBhqRhahYij5x3MQ6wXUpOyxFWn6IqNQl1dkr1e96qFhiocR4T+RnNZ6NoijoM4JKQV+z8NicZynGUVAnNE9B7DjKUqBLMY4Ks7d6y7wTXQ0N0JFD3uqOW/wAUaBHibe/t6QEBmaRxUMrejdZqg/a+tZjIxYXnw/JykO5hqSvVlxSoDF0KVOLn5GPn3hSKjzMQjRH0kU0InEWPzKaoyBv5ZKfR5vOVWQlgqW2bMyqwkn/IxZs9SFHsYfmOEsxRIXVRh8vYnwuzvdLwRhe5KgwqYJNo0ZNHl8aGjW+eNRW+k532OL3DLyO1MJTwgGww+Hw4RDMakrrXTf7paitjz0cxYspV5TQ9LEVxv9IgKxM+qW64T0BSev+Fu9dBVlY17n8PdZNxjNEt/QaGiYPRH/FdCk746Wg4QccBQ5IBcZqFMGV8/1SSeBEOvWs7C/BoIV6lOAowlKO9kupepRcvFjIfB9ZathMR5Zh378lA1t9QbT1sYWBC/aoSLD4OSTtytG/HTSQ+Ezoayhwg/CHLHWlvbu7/l9HZgwcMPFD1KOiF5K8uqL1NhG6FL0P3fUHu+pPlCkfHGne1w7dcfgnnSaCLDr6FniMfYsfalMr78/LauUfPFRpKSLkPEuB0p4Z2RmFoKKWCaJ4kBt0eLyUpEfpFzfplxr9+fXXX39Vt/+u//ypkYNQj6JqId4PVHavT0H7NqtyqEuBtO4LgxSG/XYQ+kyA2SB6EfUJdKkr7Um76a7HBwzCMbxEyY0QIdInAVppydpEl+oGS6lvQIYbIlZ5ZMTuPWVNm94vW9ZxghWwlBVyDlnhsYgbkHcli5/Frl7gqJUre+cZms5VXCWApWCC16hGH9Wj7oBuf3GDzupShKO8qGlgycpPQwVYcf9+D3/HoaX/k8QyERa3owboowWLX9pwh4R95+HhAwY+gPNM6CyiDo7vZ//2ht878OEPvQbpqyUoai2tufZ8/Iy9JyRBIEkIw3CwqaMNILWnbNnCk/B45cPsYUsbnVJiUYKlLI+XAj1qSVbe/ToOSCqYaCgBLBWhQpHdCD0Kb9BRXYpw1LLfH2k5AS3t1R3HmF2Zlzzdjk9Ju9epJSVlOO3hjXx8IIo8cUWZNtIxcX9Pwc48vRqv1jxlxr39bN9dSgqM4TVMn98T2eL9EakmLGXP4kdBRBMXUmggVLKsY8/WZZtOlk3atEfcs5QfOS7ykUj7ki5lhaXwgylvFVTn8+klIlGk2Xfe4qeHKNLqk/QovEknWYpwVNO6XYX72j8KBfd3HjwqeofYI8HSL7r77jsdW+4Gt52w9LB4kNeFUHDobMeE3Xn33eOJdZtLEVtSq2jVKu6OvLs/hJmkdBZNrSX1FmJopGm/lInFz2zuCfVpcTEsA8Hw7hPhpt3v+wBakSexU3i+0BZptAiWsjZe6sZeBSsLqgs0Iw01MJJ2EuDjpwMp1sOrrRQOshTVo3z3bPV8ceSo0nXwtWPv4VR6+Ed/ZGtUzDbiS5Rw2BvWuVASCDwcLoFRkA78EUl6eo6mcoE8nRzFF1VCe9ciT+bpSwXJo2zPkGTqPcHLjzw7Lgqlv5pXv+ufDq9rH7ivHQeb0B8/Qd6HI5F3Ie1LLGXJe6Ige+XKuCgKsOXo3BM6aMIogijQo8LSvTnJUoSjSjF17/4dh5d1rFq9/KgPfBigdOkfDygVxkugqKgoUBEIFRWFKgIVZGN8MhwxMY4xsUrAMAEhCOQUgTgQGzI8mRwIGHEI3JvZ7WHaUC4V8L1xMJyAxxSsQWDUYng7UvpqkIRMWUr0n4FAaTH3nuA3QrZ8hdtQ8N0jyztb1j79syMH38MJAvCPrGQ0iShJYERQYimzriWJdCAIozjARzbu83tCl6K2vsimumMsRRGF1rDguwe/8r5w7KsPWlZt5YY3qeQjilezSxiUfsMEU7K3iEdNnrZpGh5gLwUt8EB4EDA91+ggFSeEGpzmC9/zYZGiPP0vc2+FErpn7nriDmZwsiZaTh/D5K741raPnwXvCSKQSA5CK5u8JXGMp05eRBQGNDeh3WEsZW28VO+c6ua34kYUus2GidQYqk38Pn5RVIWIgm7/yIoKAh3pl2IcBQ2xZe9mHVCUv9XMa29pP8p7h+It7KYn5lSWKSU/nTNnu1LiW51euQjd2a0tanWjlcH4ao/y6ZxTN4Ef02tzXizz+ZpenbMd7e/WFiqOiiJrncvBG/C1I7sOK5MWzpu340DTuzXz1nwV7ztDSp8GsSzpH27sslSMuSeoKLoWvAMBJlkOqXnhR+llOuXAoiSWit880TsHPh6gJS6zvYSzFLf1RTVcHOqXYojyKBXedTW7Clo3fPjZZztXnNjqE49AFLyx3hr2Ne2uqVmxSllXX1NTd6AkuL9mzXacZdrKgnIkoRA0vNpT0YWCFq3uqKlpf68ssLh+3omwfXFUsO7teZUPWjdtWxRct+baR+pXrW6dtrOzemtJPKxIi00UnhRgN+m8xU9Tflw+vzkmFaLZaSSCxWpOMix2YOmv3/Dtr19xhUXvCZjnXJ6vxQxP2E50fFSvlqWIHhVh66O3HIMW4/RE5xwFjafgumO1nS0nisLK6tqvKph5ghS/eBq6tQ6yA+/yHy1avff94ML3yz5AC0dnwY4DViFl4elWeJ/4smTdmrVfrPiqqePk1mULW2tP+CxCShUnVT6d2lRa9Le6RYG/7Tj8dEfbVjRGb9saV9NPU0tFZZekmrKUqcXPyMdPwFaSwoJ4iP6iACWdwY7plAKLssdS2hmQYiAqAfP4aSHF9KgojnKoX4rpUSx5GAoIRjZfEJrdRJXiz0A8KmPi8Dy9dN2uj4JPdF77+/JF0EP/6x0woND40UQfIQ9W83Tx+UafR2LCoNR8thD6LTuqxnR8GX5hxyNWISWLk8I68sLB/e8vCnS11NQ07NlaAu+dtfEgisOUVWWpBFm1TbQuBTfF74ts1QxF7Eo5ZEd0CoFHSbpU/Ba/WBiKOJ5glkJEPRCtR+EdOsFSjKNYgYUDXi/VSMA1SDx5VuTiQfDS1Ww9yu/mzTu2NrisY968VeF1dWW/23HYYwlSUTWACDQAVUnF6r3z2k+UedbNm/f+gdV7jy6rPUFscppMme2o4iKqVNRFPhjyvsgX/jT93N62MuWFg19FnaEXAbkXPx6ktZrdF96ZyWyzJhY/Y+8JkjSVol3TrGjvlJ3MD6m7xu9N1vCzOl4qAjKxdhOqS7H+qEhbH32EDlj8CKKK0L0uYmHly56B2CPFH3Eq2/U0zfhNR93Q/Sdv/aDlvYUHa2tXZC3yxW8wICkLOXJ91xcXDo4auq7haNeuL0ctPLmuZVft8mMfxWs1gBSpOCaPbGgMCI6WF9xfvSgcUoKnj7wXfHfXVz4j6pSulCovu5fIooRoE13Kto+fuDHppkhQK59F0bPxEP1jG+lGIoMXO0vR/qjx+q5jDrAU4yg+bEgqPrlO8/LmWz3iKFHeaZs8auHBuXtPjuqq3/NubW3n8m2HpQRjBTWPVDxiGht9bbiiqfDLUTuPvAfq1KjdO9L31SKkLJj8+L1I9V2N0sjzgPd2Q0P70a4jDTXbFr1TA6OAD66NqSVqq2iUFDgMP7ssFbf3hHpLmhLV7NBzaIYwTA5qSkCzY1eXisVM8vFEshTVo6Ks5+weu81SgKi0F0t1kmcFS0qXlrK81msXgEVj07x5a95DI/O8Opxs69drDseseOJhEYFEnCpalS5OEwGYM+wDEJR16+K98+a1v1fkDSxrOeGNu50pqo4qQw4JMRjwKp++VViYt/3pPxUeXxRePbWwsHDL4Rh8KNKXAiBAloFhE5Yy8/Ez8Z4gElAo/SMbNayG8DD9qVsWhxtNAWh3GEtZs/jJeIkjnLh+KcZRBo3q7o+XYhwV1pYZ7pGyV1f0+WjWUdf4lGWHPlkKnR+fbR7zueIL+CZdu9UTd7uPyJKeKZXFYmETJS4cCPxw86GtPmXSmE+WloVDgWWfbbcgLvreIEYVp9Oy84C3ks/ji89nSU1eDWluie6YspRNix+9D7oWBUpywfa0OYo6QoshqrxFRA+wVOJ8/Ez1KLzFbvZLIaJeLNIBLH3ekWv+lOhWFLEa8EGdhMofhhjotfH54q/hXBR/2nzLKgFsVCks5FMC4P2n+IgWGI6nl0hNQk2e3gsXr261EC6B/oRQRRhW4LEBa290dtTEIaRmm6OUwZUI5rQAOyYsZVuXUm9CvCPkKDXM751vWa5JHjW3o92RdCkLXb1xUJN0SsJ8/JjPhN6AAXqX3bP4AaLYvOfaMpNUd1K8tMhJUDwk3Iu8CiZaIDMbgWMeqeHxN8O0tgL2aLUbHXFhGIkOgrzgdofywsT7LjpTOjGYtPoHQfpjEtkhnetoVNjwCD8gEicBtqeNpOJNWUrnXYcC0McvhvcEuxH5fkC6+PEgzQNd04NqDL+V6K1gKSvjpSS4xBNMlC5lrkfhvXZLl2Ktvuj3O3sSoqjVcma1gm2iSpudGBUfK0JfoFaYTtMvVrJGx9UaxUTwG4RdSajR5THjSRpqWtLdqZEslBiW4mJgy/9EQMqNWg78LHYQTja5SUmXShhLJUaX4nqUMUd1p18KJtDDLwkURbi2k5JkxS+egghEPo5IniIn4srkgegdYgJgw1NQY2SZetfaiJMTj5aJAplQG2njJQKYvM5yITxpvsWCstcvZfzlDkm4CLI74mLZ3fFdNXdqyAxTgqWsjJeKh5qkcxLCUogomCBLvz+KP2v7LJXZB219enoUKX/2ENhGqhricZBDPB9kyx8RnK2Jj7UjC1TT0NYHOAd+sVKK6zgVoRVK9sQBKgx240ov8iRxtZQoJq3GSzsmX0E0s/gZe6IzoZHCuHhJtHqKNne0GCLvStqXWOri8p4getRkg+Y0v0H7ulQS/dpNdA8vf8eKwo8qevYEaJ3nOYGtuAJPkOJjBWlCrMpxabAVcmiQrGOlFcdxLiFCQLQ4jIkjvchTRLpaQXI0DePalKVMLH5xeU/oZUAVjSHxk6NpZORtqfsSS1nwLZcoKI5gAliK2frCFWCE9njAcqu/hCfe+9T1M6+0tswc9viAR5/BL1vr2Ml0C1ktfRZSn4DIl3QE0oi/LlKBaoIR6YiXe6wHLTJiHmDJy1LVMIToH4+y8magcqXsy0mJlCUJWKMToUvxzHNREnJINvgqKk/8RLzQpBQZS1kcLxUHjqRTnPfxY3oU8WnAnp2wwR0GJt5y7V3DbrK2DLvq8VsGGelR6nPQC0FZ8weCh0mYZY2EWRTZGGQ5MpqKYVerMjUR8k7k9Rb3paQgSH+aLMhReMBq+nCJNl2ROk9Z2qIuZfMriEZf7tATR+NYxqQNj4/IMeya3bfEUgkzTzjPUlSPKhm1fmgx+FuvH1phMGY1PDHl8XEvW/57YCRBVHSrTypuUcxygD8DtdIwPpIuJMdgP666KCfEKjiVxxJUd/iZcSVrdBIkx7JHQrhH/8gGViKCi4vvNrg86XImiSesEcR3QkGbs80ae08wsaoEKUssqI2BPSlC7PA70tlelLoU1aOCXfXz5h31fgCfkv2IdmhG3V9gYr+UlJT/l/Kghf8nn/x7Whq2+qJSA3WIlG5kKUc8H+kJ0DMhIX6deCQYoSMgMkpKS5WCoYgD8m5kGhb2WbpyatGyNOLhsKX0iQSSPhPCNtIdqUexiJy2+Al50m1AnIjm0uU4tVj4UbO7vghZiutRq/eenLv5rtNHmj9bWGcw4jw0cXh/O8vwF4t0fNREscu1mxe9OMhLXUTQs3m0VKdi1kVxJb9YkyaP1DxwiIyZrNEJcK0kgO7Ja1YJ1dPo2UbJRcWz63U3JCmaHli8gPoAACAASURBVE8ftvAQDCBl91u9XI4khESp94Yh+U/siAAejbo5NULSpRJo8XNy7gmmR4W9H6zYHgqV/O7YWhigs91gLF9pcamlHzm9uLQ4qOMpy91oSNHCiv5o6fM4aU/UT3JMfkhqWH0OuiFVgkiDRvEDJFpdYcK4p5tY7EiaL21yal5Z2uQwF8TiYidNzpABK0pHCpAgF0PlmLKUqcXPaFSvyL96Z1yudP8iSE7neWK3TeKM75lAKrHjpRz28eP9UeF1K47UrPmq69jawAuGYwlg0hschKv+IAi7sGKRMAue5ijuwqKjm7HylDa03LVr8cRYgGxgpdmKZ2j8XOCIkMSSYolo5Wn32JmmyRodFJmKzKssgt8JP5nellGSmniWN5oC29HckohTBYJzEbDU1/4nevnJdeY+ft+IvuRrP33sulGYNhOkhlSBNESPsNPUg/IFmlvT7vQESznq40f0qPElXl943cHjmR0n1x3b7iOfsNXeF9vzRS4kntnc6WBuH/ivsgUGweMfjoGPXviTEKWOAfnHnhV5EuqKPxESo17CQmaEwq7Q3fDLRfUXFYUGorMfM0ZkkOdYe3fSYQhq90zbQVywyDJPnydDtyxFdUNChKXu0Vuue+yHBp2SdB4/nWse+fF1o4RWGyFIe7NSFvkBHsVvg99W9PYi06Ww1Ud9Jiq8H6w5ULL7ZNeKo027dxyOb+6fcLjpZ7WtB5TF+1prt92qNL1Wu/FLHaUpupjok+DFSx4HrNhj4aUtdkWAX0DPkKLFJTqiaBQ5WVwh70GY/lgG1GNqyDBZowMih1LqPJN8S5LnO+ICEjBKVsTTZPnF2j0eq94Pk0T6pR677vd6y3WP/RE4TG/x+p748XU/0rsGIKVzD2pxajIi8kACsNJs9QSzOMZSiR4vRaXFcGaIPY8f748Cvz6v53QHfOT1vacXNjSseK+C+nab3CgeCvuWbeqsbXh/a1f9xtpth4PvtsPXuSpgEonYC5Y2KVO12HmIHmEH+WlkVxMXHUMTMJAd8QjVXS5WjlHTFiGDZI2i4TpyKU+d7ososUsDeFZEyChhFk/T1VwmJ6JJjOcBZCjKHx977Bq95TEzljK6BlhK3KcqUpInjkZkjp4Cl4iA8f32AEv93Ll+KcZR5O1UUfHpx9O/LAr/5+8L34sLUAippkNbA/vXHO7qfHzz1pLV9aug4RUPoLTFz/folpWyKGte7Jqno4lkV7BkdJ9NRNIaGSThiBg1fR7STdYoEi+SUpSC+sLk09nJRkmT+OhUaArsWrYReVAPKuHSnz7xqv5SasBSHmXUq/rXPPEI/T4qTZ+WFA/zLc8q7Gv/xC4JmNyupEslqqv3udtu/09K0d1kKc5RfCQsbgEQPo+eu7jBPXt88PGgMuzRqjvQtaLq46rt8TT85NJnRStKmBc8RJA4umWHI3ekk0WaevoUP0/dipRYyvzRq1LVED3FoAT0olmaQgaKJXHsANtj0SJSioZDegmzOHEFBESY3hvbjT7ADqNnDExKFbmAeUlnPhApC5EXcPOgRrzYEQEqVcoazRg7TnfIUUlURFCwVMLGSyGimNDuQco/4RWcpVn1PcdvvcJHAWEdcVPGuyW+4G6YkDJ4z4ydRz7qamlpadi2KOYUJ6QaaAo9ckcqanFIBPAJ8BNIpOYIHIvKLj+bXxdxgd5jJ6cySWwTlaxRhHQZT1qbTyZfzVZExugBo+SJ6ZJfwWSxFKm4yB2aHI0FPOHoHS+OpYTB/vgWhTHDOIWGviJFMuENwoccoF7ARx28+JGFkJftsKzwzDDBQj4VrCkOnWPkaiJHfyXpUglhqdT7BUfFHL4UQ5ditj6z8VH69yjF+ia9i7MSFSvKO/WrusD2/u9xzPhqVKzqE9CpGpoHQ49LUfRSkbCUQwyyE8VhXgVoMuJa6bg4ImTgwYhkjXZJOnwVkSjs8iNURuRaPt9AAE9D5I0FaLyavHyeegpM6j73wwql6Yf4CQNYDwUpTT9cP9hAGIvGLysAj8EkpthlAk1+Rmo6MlRRckg9kWQQD2n+jMULlkrQeCmJo7oJKapH6XXBGt9dxJGwp2nTmqpzcx76U+Pzm9ZsX13fltnxvoHjhbiUFqSmhDWFS3ZYDD1LrPFS+hOJsCi6L44KYRiQTpEvZtEQFflHzyLR7BjbaJI12lGvUy/mcSyGCoQd+afmQsTqisDz4Cdt1CvlEDtHjiJhaKnPO1kxCWaXqju87DH4kMH2Etx532TWdd/pOZVbladfm3OozNv0Wjp82sHX9ARueOIaWZodeoaIYgF5H8Lw071VGimxVAK8JzQc1S1IcZ8J8wa0yY3iIfhaRkdDQ828o6fr5x08Wta0rqYGerVi2CdoEUYXtShmVmOk/agnF5WG5mTckXLOK6CUCjldXCMHRFg6mwbJESlZo6Co8HiZSE4bkI+I1CMj8RIdISwldaNex0PSlmSB7sOa/K3rrD3pXdf+0c76VR+s+OiR+raivx386I9zy3RkkSh4ce6ugW+APVbTvuajpk3wkYVVFas3wedEDmBy9Me3TAbZ8CNULDvCioQeE3FwipF08Ocknxm4/Yrbb/6V8+OlUu//ptCjMAv2dSmmR4W71eqD1ni4eHxxcXFZIFhcDB4SJaXFZWHjssEj7ClIG1ridK19CuwkueDVOhp9UL5Yfeuxq+WD7FIhUq8+SAf5Ydya3xwcpXLEWpMM3WGViu/gFn78j++xiCh5JJ4dZBt6KewYRPOkyQnB4OnlpzadbNp9sqhp4Z51O7Y/vXvb+I49paXGMxJ6Ax90Zu+4tav9I/hAaFfL0eB1xxa9sOIrpUJNWA5pc6Fmi4rHM9kZ6kUQEXWjagRjqcSMl1JtfVSgfUgxPcpEJVVvySyEvkZgI/TBh8zCwE4lJWZaLibEilPdQEjdEQVOY9UDmrM0B9nl0aeiNFhEkvyRsjP5rvpgxZma9MXp7DBN1XCtTU4VoklTzTON1rlIZCZCkubSiB01GZJpnjbZUY/t36MAnnafLIMPUy+Gz/u0b/t07/KaefBtkAhRbNfrW7131Qs7Fv37jsPK397fuXzVZ/uPHd79/tLPDkDZSjIkMSwoYvhZaiaic64vHGMllnLcPKHRo1CYbUgBoh6FL7IZFCImHfcCY4BhEDCbQs94PDBNT1QUTUDaiXwY8kNgj0E3ilwnHiE5hUjUe5iadNQTtFeTPRal2ZgWjHQnanWLyrAmPdiRfyxzNAp3NPJ4tJwkxKl/6glSpCQhFHynfttbtTteJJDa41v8bFLHntWd2869tfyo/ky9MHH1u9XhdTu2vrDjcOhvOz7cDXPt7pq7CZr7aw7QjxuyLGukiLjoAMmYusK8w09zm9qdxOlSoEdd8Z/a7zfbhBTXo7rNUdpbj2NPLUkS4lWB7Wg3rKzVSAixHXod31fPYJfwDWRIPpOcRy8Sa/k4SZ4e4UEhkVzLYk3uk2aQp0ETZ7lhKfC4iF3DaDhPkkdPE9dCgIT5ih0wiKYnL97b0NDS0LZ/x6LT9ScC4fDq+hPL9p4IfbH8aFD3DetVlnXgFxza9tcd9v3t/UVNMybvrpux6f0Zi+Eq9YEw0ZoM8nywctPsqjlnV0p3GRFMHEs9p9WjUK49SHFEdVOPirjxOHaxvFkBqhsep8aw0/gB9YFoDrDzNc8w4jnRFKQTpSC9zmitOZHtiDjDOyVniNPUy+SoqHPEwYjMs5uFjZAnlwRcJl0ZsaPelnQOCQaD4WDRsn3vF3XVNzQcO3C6vqGmGj5dCl812LZI3+/FW7LsOvyCQ9vONR/9pn7b1hmTF++FduOOYYs7V/Ev8AopEGBhnSiaK3FAzSSGxE1GByRdylGLX4Stjwq2BymqR8XoLI++s+7HiNIkAbGHJSrHRB/Qlr52L+Li6GfKkmepRiZOLyexNMhT10SRHZG0Wsm1ZaK5lJzNrpPDalKakDhTzS6JgpUAlThHGyCnSCt2VIpRcw6RoJzsfC8UeqRwy/aSZX8qLIRuxaY/4XcMjJYSUJXX7Vi0bGENGNzxY14bwU8a3GWyDjNJapZpCagZ4CUi4skhWPGteqmRdKZLJWC8lA5H2WMp5CjwPe9Wf5Tx7Zsc4aVItqKMNU9b7EjnyHERScBu5IlsXzox8iRyDY1kJ0t5kQ6yh00Pih16oe5dinMiAxoxmoO6+Y86nb3C8UrN1ZH70kE4xI5qriFxYKeFBjH020L/CfM0CaNdyXjxBnyLv9yqLHv10OBw+NNPPlkEdqhPN3+yKEIIFRUtkOVLkyF2qZpPY+n8W71o8XOQpQhHBaAotIsNlvL3IV5IxFCnTSzRe9IDoCVJylizitwh++Q6NSQeGYvS36ex6hpOpj81KjoUmSKeIX7q6XiaTmHRSkzPl+5VkwheSv7UlSaG74iT+AUgD4L0p24lMfyQFIWJREVDREjByePh6wWw9gVwTb9noHNPkVElaGr3wacm4asiwpgqxMgBlg8Rpd4ZjVL32ZmRotT9hOhSkf1RXJwNSCGinpX8+nhSid6qBQkhvT/dyIgKgtdFnaeNo3tRcVyi9rC8B2H1jwSlfSYYYlhkVHFpTiY7/GTNEZIAP0zl0fPEWWI3OkbNn5qfqBCLUAWJPItAVOZjR5SAM6DHg/ALBIhjIHxLRHysXJsvdU/cAY+i9ybnMJ48JcTih9ZzPWOCdUj1eaUveMr2fKtPNb2JAmUBUu6i8Gmhq0UvilyNEqeQY7Cn/klBflBEiQA/PSoi6hIhXAqwXEfp0yw+IlG2q240x2HHdF/I4jnTEyJSYAHtBq/QuSo2huI5Q+LASClMrIF4niV2/2TXWGAiWIr0R4V1ZFqFFPc973HreUSdiHzIvIgNHgREsyOaE6QdeoZY8yepfdD0sIjjJ/Fo2PI/GkV2pSAepT/cah4Hj2an0HQg8v/r7otCdU2vsw4TUDjM6FU0zMUxqRWsc1EJVLyzVcSr0AvB23ghXrTWUDJyIFT5bUu8GAhToXNlMKhkzKQDjZVEZsCBsdqUMrEg1lSTWhxEKg2pNEzFC/V911rPs5613vf7zt77nJlz9jn743vXetbzPOv91vf/88+/z7/3Yd4IUXJCkBrmCgNZcYymDJRA8My2bP6GyfTPHmVrwDGvZDYEHst/oXRP8h2/R/RXvfZ/ffsnwTWfUuN9lL1G7V7w9BoefRwztYHGbdD7kZPNqEvIT0rYkYnKAALL6KiWxMlY+dD12mDKmJyVXukTWCFop2SiU0fowCD1BuUpClnPaJD8mND4ks3fMHSfk3MpWdKRkRM/3gVfpR7Zz0stn5nI5td8Svl3Jh70kaG0f1RR3EV5nLf7G3fbJ8zzlNnEI4iY9Z6jja9OW2MiSTcj5c8SwIgthcQTjod02a9TQ5VeIwpsS8hySjoxOV7BOXHTFhk4c/WDm79hkE5wXNe4VtlOILqdgE62Ie+lHsmrlH9m4qDh9Z5S8Rp14PU+wryPy0TLtOughUuWGymPJaN7XbBGjcoBYZJ5RCK7cC8nBDFmxqcfivDZm7DHCPJrDQ3JU1I9CsFYIpowMoME7+KRP9wNp3cEB3lWR2SJr4FjWyM93g5fpR7Rz0vZ30ddDvpd5ylln5l4HN/rizuN6bXbnmnwMHfn+w2IuwGLDWgq1PvNA791cJjntQn8wlxtvGS3xapB0Q4R03aVJ1vk7lyASNIAEda8YFezdengbJ4PHlNXgoeJ9bQVPR2S3lEtJJAhh+ikrbxKPYK/l9p+ZiK7X+cp9dj+PioHi/Ftp55FE2TqN8FzKxEInwBzca7T8izaBDMq5eI1Ob2KfNwLWADiykAMBlZgiOsjzChODTZFjqYMkfIeSA5RPpCuG8lV+IbYkhXdkGzSYNcA9fV4C/Iq9Qh+Xmr7mYlsfo2n1OP67jlGmPPLeVqN98AZlb8yXB04tcljZdr5oSwrs2JZENm+iVSdjohope3ouFh5kyomG8W+xi5jQZWd0eRoo1XgmWvygXS9aOmIFnULzOR6J3Mc+uXIyQ7iVeqR/LyUv0Zdjrtd+SmF7/UdW71flZjgdpKb+eqso1zmn/Xu2GhZtgKrDNyp0FqNNz/wpZybIdWCQkzxiCzp5W0OMrcJ59VCKLIPbM6NokdJBuNmd33X0J1nJVtFFoCrRJvcCR5vRV6lHvrbE6fvo+YWrvqUsk8hjfdRl+N9vz+VOjU8KvZjdy4UWPO2KNJuzygFL+kFkiqbHwRwWjzpMSp7krdkY/Ck0aJMibPNmALBokDzFM7oQJYVIdFiVG9y18Uq22a0lm17RhgnP1bSyU4e3Xspe42avxTq+M9Vn1KP7X0Ub5/MMB4A9SY4EefjGm+KUTwLbyAwwUqJPvKgNFEYiMBDdezEyMdSIs8DEhd45ypMD1EKcfFAba5Z9yRzqS6CYClu8fGD66jiG/De3E36Hm6n7ECScDpqF59Ef9Z+98RDvko94H3U3MLVnlL+f30vPpKf4T2+8E3FBq4jjulxnhpoPG+Q5Vj9jpGyTXFXg+XSJtk/BFIhdVFmPSN0wyp0QHMl7LGlxJwoKUMG4WH5OPlRjCUJVSxS4D5owWBz204hMZ8NpIlbVoAZiFhxKZZP1klTeS/1UN/xs9eo8RsdTv9c7SkVr1H7D2Cc+j9kkfNiECO2PMCx7KLJ9C8nZLJ4OGGBRb83CssUZo8WCVNcB2oFWTdVPHi4BwBiChuUli1FJ7GXMNVB8w2FvyfhCBXWybneTQ6jZh5p7j/rbetLiq0cb8OeUo/g56V++geexe+UPW52pafU3xufQnpcfx8VNyBHjQHayMdJ74XX9GZI/dzpoEHovY+2yiZES3f1M8Y4yVpEPQE1dixlC2ue5IyilW8od5IEs/DUQmEi9HVfdSW7HD+81sq0DdOM2Csq2IOlx2yhDdLai8ijeZWy16gLTY+Cqzyl/so/mb+l+TH8fNSYkx91djFjgLxFPahaqRaluTVLKBucuhqBDvRQVgqREBsB4+lUEksbgnYbMqFjHzFzUgAl4S4crbtIwTVep/pl+UXAHpe0yeOSvGKtJTTd0WP8Ub2XutJrFN5L/ZGjP3/3hz/8/X/5J8cnZV/84H+aI2boU7YZehi4DzTuOZJ6zylFGStoI+9fSaEa7AlkGdFYyQRmPJCtrKzwQX2xhWG4KC+52Spo0cHxTNwM7MCzQ41KWTQFp6MEx4/mWqERzPvF1dxZFOlmGzjS2qlkj+I7fv4adSm+2+T5H/v+D//ZH/7jR3/+xB/98F/4wR/8q3//bzzsb8DcNj8HY2hjsaivc75WyEFLiqJA4KHkuWc8gzRXEMeqaRa4ATCDJgulW65UqXGvpWfoA6ewdWZ5DTYbcLFakOQN+rkKIBzr6QNabvNkQqaRN8pK8Bw+Eg16KUmfFvpT6uG+43fF16j5KvWxv/bX/+nJn7/05+u/zdE2+76lmBbH1gLefNyHGLDeKEIKus/RGW3hesSbuHArzQq6QZSjULcDG5CwkjyBSAoGotQbZHRXSxj85uW0rIksQfXX+GrfoggfFSLO7hgOt5eljHayk4eivJe64TfRr/g+au7h+R/7kx/75P89+vPJT37yY3/6Xz+O/+vzX0OK8ZZ7YbOOgS9LAPoYHFDSSpSVaGDVoCTkNNSkjBsbDZy37hVXkXUXZm6envYQ4nYRQm7dvZJbWyNrAbjSazY5PCDAysIIjHbykI4S7gNc0x+mdXVfx6wLylrI8vEG+Cp145+Xsteoy3EHqXziJ//UA/78xfF/fR/4d8/Xmc3JJToiSzHjXWKcJCDqQtge0CGbKykeK1KL8Cwid6oikWUjtlGbjRUNxSW03iV7GZwnKg3qbTKPctnawPwo6EjkUbUNzayo04dwQmkvl6e9QcW6beqgvJe60avU982fOLzy9+c+o39efNGyWKLy6St7nVzUNUs+zzGs+YWZYcgBZykAW5SccUYwga2JNqArWIUBVigKbV8sjwjXVeLIsBuxRUhFZR1LzC7VToxNWG2RskWR9qTk6S/i85tcJSduo+THVCCSUI0G7F9jOW7PV6kb/rzUyc/w7pqOX/P/gD/b38q7s3pUWB9kjtYHmzkGGo8YLjXIqVvkJgHueqXrrlMqlLfYaRESkrzoG4kesRTME0Bj7ToYrzgRiPYrvLPqSOtVGltCGplnj2n+erHsY1ufqR9aQHxUAT5W/7Ll5NEnr1I3+PTE0W8XO2w4nzCXcYx/le6ffepTv3X37v1/8ak/mL9bYoBWGMsH/Iejwsj6SoKOlPcoglEjLw020CwS1oAx5I0YdaBCBxTWrDhuqWJlB0vi3avIMZzdtmbAUtciXrQzs1p9JNuEE/LDiscPFJ8w2ehmKyxYBVAaJjWatt5Xe5W6wc9LPfCz50dXfffuN17/wsu/8dm7v/f2K9/5wF+YZFcyx5xiRF5LmMMdUKKFrINvBYgIRwBc72xi2J8aA6OTA1VUipHUBTZhrSnitMwINd1vYjEZ61ROmTjZcoaSwUugEfohtQHIbSyhKCE4W4WOLgYhcS1ZDpeGNYlXqRv9vNR4HzU+hWT/BvX8BaDnn0Kvbe+8+vbXf+XOnZfuvfXz33ysTykfUMzLkzlCPeJRwgW3J0QrPZBdITRuvzimYkaZgQ607g9EKppvWoXjzthb8CwWjhUN+nvHzEAdaxhkBO7ibBSvukq0CFuZXu1BFeksQ8lwBIlqFHjdV81MK9CU7FtP9CHeS43/6xvPqBu9xFwu//m1z/2vf/i9+9/41hff+s61novHl3KDypzN+rWBOM9Rk3KEgljZ8nEqeKQdPuCmT92fy5sxOwWei9OnhWFZUNdAF2iDO4eFEfjBDmqiMZmmTRE4HTEainNtgCO7W17IRRWJS4N2kGTnFBXjXefAbv5eyt5HXe586df+0Oc+f//u7333c//l/pV/4d7l8t5r4x/R+vpPvf2L9956jK9SPlCbWs5WRgfQGLytjhoGgt+CmuG27NAitqQ9OAOLvbiFYNxL3eya+SZEGWEuIyqJKwJcKmBHoZBTs8jAq+titnMFaUqtjtW81oc2roaXFUJo6xY8sxoK8McquHk6vjYGcvNXKf/5qK98843Xv/yrd+79ny98dfxrqHB98Pret9598efe+bV//K1ffu213717/0avdQ/ucs6YsxpHXYBhZbUyXZbnRjeyY6kDJ5HUIwJn5jxQjO2GvhEMPS9FNZdG3zYdnI0AzdvWttx0KA20eRTgaqU8pYz+YPY7XE1H5kdu06MVDWEsSx/DWTz7/74b/8sd/j7qpcv/eP1r419GuPuN1777ztcuV35KXe6Mf8zuz337nZ/57r/85i//zrvPPJ6nVJ2izCtvwAAFZ2gBalhTNSOjVEjgqDqP55XuiLGdFcaGJD3SwqMU6AT8ONxdOga5AsjMMRhwFAHLFVMmnLiToqHOUNUBSH37S9+QLnaJp51HJWfnEsykf/Xncub2KnX9n5fy91EfeunV33zn//2D//npV9/+3fe+9bVrvEpd7v2rl195491P37v36tvfOfu3gnKnjzoqQ5OJWZgnH+XISYmgLni0OpHcVU2jxuF26FAIJQmuYVFY6vTzLaznFC4RtaPSikAChi1g8mUe4PhahExcb6mHVdQz0kegj4uqHRm+amGiLDkl0lwsGiceEg5Mu7Y43ktd89+X8r+Puox/aviPvfHbL7/xv3/uH33+va9e41Vq/B3UvR/9/Tc/Pf69oHtv/uH2H5q2w/crxcw4XR8ex211zJFzT9Qh0jNVbvEsSRN6LdzhRdBwnIQEiGZRG0tGETtmMMrB0YsUVt0FWzAINcyIq12AjVOatNpUqAMTBs6Qcz5CKF6vOVxFJiNCzw0JAqMgGWs27dHN3kvF5/ruPv/qW//9s196+zv/8Y3XX3v9nV+9xq8vn587Gv+w+/hL3xE8hj8cHqbJkTEgRQYJdtRGCsRDyzq/5kHJxZUg0YEBKnMN0BYhNFjT3CF8REdDr/VKz40u4AhrtiGgay/F1XR9wNyP1OEk0AjHgYdPtjCK8OjqAj8HWGhMkkIt+pee6C1rvEo9d51/BdFfo4bJ5aV73/7yH/z6f/rFX/+Jn/jma//8e+L7wPBy/+5L9k2J8W/UPYY/nF0OSmanA/WYZ+cfyHFTo0xR5tHEgECbI0Vm5sUIUxXSlRuFIlO1NHNx6cJk9YfjXKGsZsoIVqeuOUSlIbeBatUtO4iXjPBIEWR1v6gXelJ6FDSHs3j8mL3Jq1T+LqTL3a+Mf17469+7f7k//sfvGu+ljjf0gVRsTDjZOufcACC4Bb4qihirMjuGfKz1CwXc/13VJKuuUuMCHCyuyTuFo0Fyus4IwcpxTRaPJUkg3cDPWnVO/ChyL6jsdcqhtPYtAaXANLH5CfKIpAoo6/DJw1TeS13xk+j+vb6Lez5z52//1JufvfOhT3zix9/8lZv9te/J5t63Uow0x+uDzblFBRPPAqIUNCpTMFcPo7gB2QyOZBCpHUUj2MQNamlIVCou2qZsaSbhVJcoBOia5Kpf6gqZzkSbfPXkVkbg5NCqRZfBFavXcTZllhAJ7NDxY5OvUlf9eSn/Xl81fMY+9VqxJzjzmdiQMEismBxyrJQw8ApNGEzcjxoBDWLhixfVhSAJwwjGopDGaOn2sZ1MCpA24ZBLlkSRZQHVe6va7sisko60mWkKG6PWwswIR6DQygaSrANZwM4/eXjLe6krvUrxfRQ8P/T8M/aBosfznXDs4lorhog7M3MMtIwzk5QgiloqEaVIKZBFLy6Dw8NCJQYrHb24992jzbP2Mgl12AklSyWAKktwRDzyMhyiKS8hdHGNhJlHXeQInUvFMUwvBsn1SM9BGlDbLC6rwMePOL5KXfHnpa7xM7zHTR9rJQdmw8t5zVEu40y2EsELxaKCFQtpA6lTxAAaUH0lwYJaC/dCLDQ+Bgpl6QxX7CB8uZgnk64yTgAAFvdJREFUimiwjAMEX9lRxI7RNnZHQgTMtUOapwmv7rSjumQHaVKsM2mmWfANnDyE5VXqCj8vVd5Hnbg+yaU5HD9yTDq/qFdSCjKiaHtvw2bQ86uGkRG0oIPSJH0IUkskSaO2MzOMOg1GjCLW6WZxlrKBFMkKLDTJJZCGIk/wjMeadYusLmhoqPBTUUCnD4ioBQ4oDD15+Z379aEur1IP/nmp3fuo1fKJRuRGYk5YY151QZHTX++aCcaJR/QIo5qBXPxcaQ7B9nLxpL34gkY3OGVB2CgGhJR96bKoTYFTyENWXDIpFlCuYOkdxt0k1dvKbh8NC+Nsb3V3i1pab8hNONOTB3m8Sl3p56X8NerE7BaU2n+Qyi1lkmPWYXpstTytBLqgFG6Bd29WC32gfhRUEsrIXJFKZ33CTDI26AiHla8pYq4EOCkN2ORZzCWCIzDw9IdBF4Tzjg+xS2EANFdU6LETBOn4sS6vUg/69oS/Rl2OvZ78is2Kc8IsgXpuWd7nTF2XFaEHaSI1TKDiwSTovGS7t3RQAPGOTkfbycpIJKPpNw7uvUTEgRqTaCoDr8vk0TfCYGjWRZu82mRbi8SSHQ1joctThgqp1cEz0Ek6eaxf/b3U0/I+CjMqU7JRcV4MON0UZTRYKfMkhKiAW2BLApkEP0ANJKy3HZqmWqkPYjLWQNtblZQiJhrBWCwiHtvOsshXULsK8cBEmozQMoFc74WAl4V0BmvXaL5ubWiazNLj59SVX6WervdRbUgy4ajYMk6ZIsHKe0mSILxB4oww12oVGuuIk61RSF1GUR+LH1iRxd6kLrtNkoA7he6sUGvBtzVcGWhsQm9JGBgFokVNyEKjUSs7XmxKorqQHi/VTLpf9b3U6Xf8vu/Gv7nl+On8QVf4XzgbYkxyDqqODlkQkJaJVrB7FGUk1oUeqjeCAmQxAMWBE9iIYK2e3EQtUQWYAGZDgAEq283ABztZV2eEW9INCDRFXl9gbd2KrpgEK2BNS0RW1rG0gqeuh4tpjh+89ir14J+X8r/hvRz7PPkVG4SdMDU8KgLcEZI6IyXWPCsZpRZSX4ORZRqHJ+jOKHQ6dFQ8qjw9M9qKV5l4lq04M0zEy0Ivku9pq1jqFD2DZX23nAEmaUoLy3NFo0xW5hZlKl4DTNyTyHWvJ4/3eC/1gJ+X8teoE5tbUMJAbD4+KkD1RgkhaYis2IbuNlGxGthIspFwnQQqs+SiNBGNk5H4qFuCWmZWSSIIwY9qQ3f09FEymM2npbK1bjNyg3YKr4GBtqBTAQCEudYi86USvLqQno5GgHomx4/4K72XeireR2EeMnDMqU2uTVSFXQylOUkxQoUjVgjqvhon+SM6TqrWqcE2GWMlrjSvOr62a/gk84DvAEKIyFaiEkQBfPZLPaVwLw1ZhYMBSCBBbkXopdjCpHkh8r65yI+fUfjdE6c/L/U0fK8vB5YRpzUgP+YwtY6pY0XZOElEGgjYfR1lML1EvsIaqwPJutl0lPJUlRS5r6V0NQPZ7saDhhYwwz6BRqEu3OmAQcRVI0cJ+HJ9WXCqCRNkD4dgJ9wQ2EKYAS4kHUfp+Dl1lVep2/+5vhzGjMaxLjuscDOhuEIBOzjOuBMMvIWfrZ6c2FSherIVqVCvx1zDml6Rk4d9hcmCu9D7aqw7QYw61o4jz3Uwa8csCc7Q6CGJOJeMjE8usgH4gR41A+qrqGshM/E7fkbhVerk0xNPxWcmNrOVQcY0j5Y27JbSp+EtnTQ/bM2k3DEkFFMVCPPaVvnhD6u51nKV7rLQhMwNYNNX1x+5KC5KvXzAjcpdjzq+QD0GwKyr66iaRT92UaEpsXgeP6f4KnX481L2GnVscAsqOYoySt4znWvMs43VHUTtocLZJOwWQK3pT08i0C1A2e9RE5LcmLSWoknUW68kK49eJcBjLjUetXM3Qj5o/uV8ZFizyAtDaQ+4TcjonJoZ+VGj5kYWnVJk1JOHfXzHb76X2n4gabxGvfALt+snDJer5Xxysm1AHFzMOcdtFWUvjG4Ksq88J+0sIn0EvMsZLRBoSiHJAmOgZadX2cpPJJk0hBnMZccKeQxyqMcSkQTZrUXkiEszUsPYiEOQJLhujv6y12SltTg+xHf8nrb3UTmpGeW0EA8kCslULGK5CQPhkZp95EQR0A5Y10GhdRFNOtOjOCjkoYcALg3AlhWB7HArIQfPHbg/VoGnTyJxPcrtBqyxUAOpYyfV1QjjJGtoHIzECIzVasbLf7cTkFepzacnxmvUc7+Q5FsZcU4axJB9UG1ukdZFmKQz8CLOA7YKVkqDH4vCG+jojqaJRUU5Lf3Y7CWLorFQctG10H0LNZJSYZLFGAf8jlZXajUsHLLkxFTIIwS7rPAmlcGs9CRyr2Qd0fFzge+lPvqz689Ljb/hHf82x+VY/eRXlmHxBuXI+uOQ48X4fA1BOFoGgpuqD8lO0RTkupIXVCpav5MmO8NQy5J+2cmbw5qt1wAUW72fkhhLKbCx8ChbsHqQwrYT127KdzbsvXN0kCV71mhmgTCgO4pr5eSRH69S2+/4PQWfQrqjAxtxG1ZJvQxSKWG0pACYqx4N36bKHzEMlOtxK1raWS5Pl6ZJOgqJhCgBaQBbrI0bTFQjhRNg5G3svRzqurhWsYhpNnJCpZOhpaRIL4zcj1ynm9Gwpn1u6/g5Ja9S/dsTf/P2f68Pg+GQIjAcM4ykz20Za+GNpOSp9ooVk9TSZG9tmi42jf26ODlqRmoEYFn/xFQSpVzITdaADC2nwkNlrIF7lHm6rftISdToIoGFwTwzo4Q7qKaEt17pHOVK9+yG76Wehr+PiuliDlj7zVlxIGON+4E1R64ujb8n0UuVYQ8IRshtXe1sN+BcRVN8XFBksrduG70At80AJosX1IhsmnjuILZHEwtaWYsRHzHCDp2SDQQrPeUy3NQqUp4KpiO4yavU7X8fpUPADI9XZXN2K91pQtjqFIRHYlRvILB1dVqSvUYXpoIkuYLyuKBsBsl3GGdRCyToPpQ23Rk5hQDcv2YTq8jIOpFQZSdPdqN2SRhRMUEPXUmw4PgZpZ+eKN/xexp+PoqjZ+Cji+EwiRzzY2oBWXozEJM6tZFcLUqWCnWja4wN5rq0nKUAsUf1SX4QDaBAQZclspNWrHFH0Q84aZbKiDp5zcM+u8AX1MVfASPFqToVG3hlFzeJXYrj8XPK3kttfl7q9r+PWqdThhdDcixH1WYnrH0IT6uOk7C6Nah1DUEudBgBY2gc41mvcVL8aDqA7jEyZXqWSClXJW2DQyP0FVzC9EYvK46TkBi2Gli+8ixOJkDua4GKplcm37AosA48jRUZ8fEz6uBV6ml4H6Xj2YyakPIsJkAKBruWB+IHKFhXVBHKRsCHlsWi9xCM5IHSV+ohmYQKiiXUziENsK6FUpglmZICRELM60zJrzzW10A3xV6khZ/l4+RHSoJXKoZlgTtK1eoTtePn1PY7fk/J30dxQj7GSJFg9RHFXGNeXgODGQ0xaKpAzR4a7avZSx8eB82cvDfaSbR92BfacgW5G/JGwJ2xfxqjbGuX954ridZRgl3pVGzQYzD9WICwAp5rNXdcsRaHz1gigpMj43z8jMKrVPl5KX+NupyonvwSrx3DaNORUTm13Dyppp4OpSoJjFRCkQUkM0hYIDcIYIcTGwFjCQUTFPuqIjZz+FgalUI4Eh+SpsCKwahLVBOMKHEBzGjkBUpmRkJgyCBocxGrNVZkUo+fArtXqZ/+gdv+KaQ+Hh8YRzYmEow+qKgIf0VCCk64kmfBOGG1SKsRO4WFB7salXwGuY+wAC8YY/GjEjM7qoYeckuJAU2bjII0FqGjbFAyHAYo/H046SDDMVcvhhLMkCDtHG4yZTCEBLle8/EzCq9S8umJp+D/+uQzEzaonE1GOSdMyqh5xyKV6kpQJJRcJMi76l2HjO4RAvI1qragQk0WUfIVWV6ZIsppMQSgY7WOI6lfXg3Qk0nQg1s9hMlIA++i+YiDt1npkM3dgaJM91Har11SgdYdOX5O8VUKPy/ln5m4HCue/IrOSiZvMCaDSXXQ64GWYqnAhvZVMTIDsCa9R1WX1cAT8GhHX6gDaFhLu6vxyYmAOTsPpIFa6qYL1QD1EIbDBMAyQNFNE3Db5kKV1b7XrBg1+J2Fhsl25ORZ4E+pZ/nzUk/DZ89juDmFGFcuGSWXwyxBMiMywYg1FQVhx+Y5yUyMBdwS0CIJnaAsWMBMWpdwEPyw1UthSu1C0EqQ0wYeQYIYsK6oHVC9bMU8qX6NXRLXUhLlesE8dd+El0obzvQKDvSWFl0kx88pvkrFvy91+99HjUuOq+4Twpj8LnQSdFhpUwBRpR0YVvQEEO54z+leNhlJkkeUXwkXXjC8akkgriTugYKMw08W7i+8dNkZOcazO+eEwi5cUC39QKZHkD1PScJQ5ApWcJDuVqGgQ19dJkT4DOj4GYX3UvEdv6fhfdQyCEwKA8nVKzYzO/VKPBAS1ogKBlqdMQ8tOOhIxiNamhkWqFY1hnNwkdJtg0sfViOwJU/CtJB0tCHAQCqCRWjLCu+94xKEnrwCcpcDZSGChDzKvE8oBNh/XSnOLTzop3rHpyd+yF6l7DXqcvIMfOJLedHL+GxqnI5ldXKpVTyILpwFHpGAbEQ7rQXUDsoUsO6mkRIdAWNsJCCUfDWWUGkHGo1IUkmLlYPSWAPOBTXvlhtDjtW1YId+Fv0ALdelZJo8dWb16q6am4fLYedlPcNeyCdPhPJe6vZ/Cgnf65MhyBwkjKHXYTIL4lgsog4AiQySpxwK+fhjICXFXM2zlrxX7cg6YWVZlV7SMUJb8gRpIsGDz6YJGVJb9APQA33K5p0wS3pEngpqBktjS0HLChCsWemRGdgJ3FwFJu34OWVPqReefWH+Open4TVqzsEPDjwCmUuQFq5Q4IG5SknU2Uu6uQQGJqQ6glxGlAma9dUYYGLlNgAYS5sLkI4ELYhMFxJolWJuFlAKR2RJnrb6CW4oBrEGd6NqqYhhEwTUUhx+m3pAy+WkNt0WuQHHzyh7L/Xx5577+Asf/TO3/6c55mtUTiAmWhdnVJLe5KjkbPO2psiicSK5lFqh1OpmsgudgrCIQC1EgNzJqPqBSvOxtDJUexS7ArqaUYOWWEEPQqZUDIjxGrSakd1bQjTDuhcpKszm5izlIh4ruRaN0/Fz6t6zH/mRHxp/fuQjPzs/M3E5Zj75lXnZce0SBiIFjqcFTFMyIhHWzOmFAIhOLSDZTGEnCSAjGj4dgBIAJDVa5x6MDm34lGqrGT98NE7NLgomthR9FJWtDbg6O5CYRQ56M40p3jQJi53IMWwwDRH52g2g2juePBvuPftvPmJ/Pvrvb/unkPA+SiZkY+KskCmwjTF9THWQyIvQIRSY7RQmzVPxagnkupqSvKUTqNrA2KHzOpMImM+yH43IjhJsyGaUBhk1mbvjbKrCUCFYfTVVnmZZZYiNEcXKSENwE1kEasOYrOPv+N2999yzzz47vuP37Asfv+WvUXbZ5dp5/QONQi7CLGXB67z17hyS3IplT90nQFuA92STA+LFbHbFy8MeTUQFLAIdi9YZE04CIl+RhfFIj7+sBIGvknFr3UA4FB1e1GD4F4O4BJeWJg5lgcLQ1kWMo1A2MbCTV6nLKH/mM5+ZHies21A6HmcdX5sRppfrIBjHV8noMwOjEJGMmAeFW2j7JNBSFMssj8iP3G525P6aj6QQQxQlMhiAkCtLFowTj8nxA+yR6VckuYzIj2RFEYCkDDWwmABUOoFSHIkfYCILli3AwFnXk6fE5fKhy/hz93LCuQUln0FMJYeG0bDQZyME45DIYCpKwtQLrCXPIFYlKxtrifQxBXZKnwog0zV0lGdukeKMsYtkjCiqwCIt1ErCLlLL+hostq52LXsRhDcBUhjMkh6QkOBF2YrwQZZq6NSyQOevP5c7937p7/y7z9/50C143pxsMeeyRjGNWdAjiTmu4HhJdKI8rIW2qJzcz4WSzTNa2qV316ZoRH7kGrK6dIe+O8/TN9Q7mZJGLBQLAfkKSPsZFg0SJx+SCjiR0tI3vFzQjEWRvUIdta2O/ZJ09py6vPTqt3/75S/8xmfv3D15wD7ppTIJJL7GGOrDLAfP+xE8W1yZJHok1Mfs+R51O5ia2TYJd7Xq28saHLYX1vdRemrRChPQozdVUtlACGURo2T2yHsBRSYrSvvVieNcmkGuGuPE1VR+VOqVbkH1i5ZzOXlGXJ559T/811ffeufzd2/zUyqulTPhqHwgA5eSY6yEVhaWRGSh+mTsUeYq36BeXtoVWGUaq3UK1mhojr6sUKoj6fkUH2BaCUpd0DhRi8YJa7gDgUCaZik1iUFgNU8iTA4iK0QVumhUKHV3vHrn8AyHAZw9p+68+m9/6effvc0vUjEiXC/nkaNDibMZAWkW50gjrRKhG7ORVjIQXU00Tlirkxes5iIwNdthsCkGIBIcQXTmAuGyxy3DDEBdvGB/BZa466UFXJzhSgmq0oahBozTlR64hlyFbazICzzZPE6fUXfu33nv5Ve++t9u84tU/gfVr7ldvY8uS8yDJwtLg61fEBcwdCeYyAqrJ+CxffaupR1ezGJP4JVaJLn0albChgQLslzx9lgdRR7Yh2yrap0pZVVkbKJFyca10mRI4THW4GcU0ALgUuAh68mr1J3791788d/88tfuPHNGeoJrR/PJy9dJOZqamVsmKzmOWRluQkOIuRuP5FJtjtE0KXBvewHsRNCxhksleTYoYSVVYrXmmSmoiaybLcIDSzSNcsqyi0WRLlddBCTJbiSMMiS+8mzWqK3big2KnYQq09g8T54Q9+9+5W998c35lLqt76X6WMbV6xcTC8YpgLrIJMlQlxmH4oGh8GBR1cXqhNJVpTO7OCtSSEqaCZrRKUoLHluEndY93nkmG1Gw9hIrFoZ1DWQsrHmcSImEd7jNWeDRWVIY4cbO0G5w/Jy6PPOlb7/yylffva3f8cMIsJbhOZilEfnXsjSOpwU0SZ7gFP14I8CwFbdByWSGchD9cJLGRKA302RkBEJRGBuVSCApaUmmIAGRJwiGIUhWJi4MNHFmi8C6yc5MMA/NFyialfw4wZ6axxTw6GFxO35Gzcq9H/39L37itv5vX15njKMCmcXwfGDCDdyJJSEkaIRj0cMy8YyQOgso6FXvI/LClE5JZMMGsWMQIj9bom+l0N/grEWkXVPfipKmQbIjSqtB0qMXPE8ni7qg5kIpbjOxWgQRKyoEYRsx2GfPqcvz43/57t/O/+2Tu4Ch4bLjygHLYCQspBOluwS7iDb+rQ7bM+bg+KGkFQPHLCe1HQEZDqdFQ1KhgQ7Pc9ambwhGRaQSAg5CNB8ZCi7EeSkIYAoQz1ehuoEBRDWImFDfWa2fPaPu3Lk8/9JL54wntsrrH4EfMYg56PWoUxr1Qsubo8qkkM4ezkOnYCKlCXwJjGCxqCRkuprqUJfM5GUXr3qe9Ym2Y2eTNNCThShMV6oVpBoM6EraaLxWJWs8+JRk7NEmhzQ0dUFxrkUbPYytkif2OfFQG8MY4kpjGo4KBlopsK6BxQTID2QsUsuMoAXMtG2SG6rpjEPt0S4RUtV2SVZLxeF0TtqM/NBtJBZ1WRiaXXhOcPc1QME9SwwR8WAzZ+De4M8s44ywg4HolyQW4hT4WCIKX2iJMnioB+6TK/YJxFXyYjmWLGTkM8rcokhljAMREIPFulY7IuJe4g6Mg5OzkB01ctasgumhZRBlMRGPgrYsO/lmAEIboR8He0FR1mAC0b0NzL/WABWsLidPdimYkBHmfo23kAcQ2LJUgEb/H4lDxfQOIMDIAAAAAElFTkSuQmCC)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lLYr9XTskvut"
   },
   "source": [
    "`nn.Modules` represent the building blocks of a computation graph.\n",
    "For example, in typical pytorch code, each convolution block above is its own module, each fully connected block is a module, and the whole network itself is also a module.\n",
    "Modules can contain modules within them.\n",
    "All the classes inside of `torch.nn` are instances `nn.Modules`.\n",
    "Below is an example definition of a module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vwmkBLL0dQiw"
   },
   "outputs": [],
   "source": [
    "import pdb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LhkD9c-Uheqd"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "\n",
    "class Net(nn.Module):\n",
    "\n",
    "    def __init__(self, input_size, output_size):\n",
    "        super(Net, self).__init__()\n",
    "        self.fc1 = nn.Linear(input_size, 32)\n",
    "        self.fc2 = nn.Linear(32, 32)\n",
    "        self.fc3 = nn.Linear(32, output_size)\n",
    "\n",
    "    def forward(self, x):\n",
    "        pdb.set_trace()\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x\n",
    "    \n",
    "    def hook(self, gradient):\n",
    "      return 2*gradient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ulXK0yY-DCwt"
   },
   "source": [
    "The main function that you need to implement is the `forward` function.\n",
    "Otherwise, it's a normal Python object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 104
    },
    "id": "CvcCU1s1itC3",
    "outputId": "06896f68-b7da-4ebe-cf8d-04650f2d6c7d"
   },
   "outputs": [],
   "source": [
    "net = Net(input_size=1, output_size=1)\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RaYLDhamEMKw"
   },
   "source": [
    "Here we create some dummy input. The first dimension will be the batch dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "id": "H2AGmUDrhy7i",
    "outputId": "da201e8d-b17c-4a57-f03a-0e7b75887fed"
   },
   "outputs": [],
   "source": [
    "x = torch.linspace(-5, 5, 100).view(100, 1)\n",
    "print(x.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vXIho7F8D9_x"
   },
   "source": [
    "To evaluate a neural network on some input, you pass an input through a module by calling it directly. In particular, don't call `net.forward(x)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "JN4mWZlkiEmZ",
    "outputId": "6706faf8-dfd7-4a8e-8507-ed47d36a742e"
   },
   "outputs": [],
   "source": [
    "y = net(x)\n",
    "print(y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IHO5oRs1EQxz"
   },
   "source": [
    "Let's visualize what the networks looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 269
    },
    "id": "vJu-Tt2jiF2y",
    "outputId": "0c88791c-975d-4f30-c7e9-dfa2ebafee05"
   },
   "outputs": [],
   "source": [
    "plot(x.detach().numpy(), y.detach().numpy(), ylim=(-1, 1), xlim=(-5, 5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ccm0jfOcETl0"
   },
   "source": [
    "The network keeps track of all the parameters and gradients!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_nQx9_nDmKXs"
   },
   "outputs": [],
   "source": [
    "print(net.fc1.bias.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oirQcFSmDVW5"
   },
   "source": [
    "In the `__init__` function, any variable that you assign to `self` that is also a module will be automatically added as a sub-module. The parameters of a module (and all sub-modules) can be accessed through the `parameters()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 121
    },
    "id": "Vwrjulr8mlwR",
    "outputId": "6cc62d8f-491f-482e-ba48-e02f0745cd6a"
   },
   "outputs": [],
   "source": [
    "for p in net.parameters():\n",
    "  print(p.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RAkasCBWDd2D"
   },
   "source": [
    "WARNING: if you want to have a list of modules use\n",
    "```\n",
    "def __init__(self, network1, network2):\n",
    "  self.list = nn.ModuleList([network1, network 2])\n",
    "```\n",
    "and **not** \n",
    "```\n",
    "def __init__(self, network1, network2):\n",
    "  self.list = [network1, network 2]\n",
    "```\n",
    "In the later case, `network1` and `network2` won't be added as sub-modules."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PcP0KwA4EYH-"
   },
   "source": [
    "The output of the module is just a tensor. We can perform operations on the tensor like before to automatically compute derivatives.\n",
    "For example, below, we minimize the sum-of-squares of the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tqDLwkAXmSV3"
   },
   "outputs": [],
   "source": [
    "loss = (y**2).sum()\n",
    "loss.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mzAJE131EkW5"
   },
   "source": [
    "We can manually update the parameters by adding the gradient (times a negative learning rate) and zero'ing out the gradients to prevent gradient accumulation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1tT_gl-QmfMx"
   },
   "outputs": [],
   "source": [
    "for p in net.parameters():\n",
    "  p.data.add_(-0.001 * p.grad)\n",
    "  p.grad.data.zero_()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cBSnExmmEsy5"
   },
   "source": [
    "And we can do this in a loop to train our network!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mrhmNphPnWEQ"
   },
   "outputs": [],
   "source": [
    "for _ in range(100):\n",
    "  y = net(x)\n",
    "  loss = (y**2).sum()\n",
    "  loss.backward()\n",
    "  for p in net.parameters():\n",
    "    p.data.add_(- 0.001 * p.grad)\n",
    "    p.grad.data.zero_()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oGe1_gpCE1hs"
   },
   "source": [
    "Sure enough, our network learns to set everything to zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "YnZ9swUJnp1d"
   },
   "outputs": [],
   "source": [
    "plot(x.detach().numpy(), y.detach().numpy(), ylim=(-1, 1), xlim=(-5, 5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5onPqgL_o9N9"
   },
   "source": [
    "## Loss Functions\n",
    "PyTorch has a bunch of built in loss functions, which are just other modules that you can pass your data through."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KutB4zljpCJz"
   },
   "outputs": [],
   "source": [
    "y_target = torch.sin(x)\n",
    "loss_fn = nn.SmoothL1Loss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LBtKp7iro-VR"
   },
   "outputs": [],
   "source": [
    "for _ in range(1000):\n",
    "  y = net(x)\n",
    "  loss = loss_fn(y, y_target)\n",
    "  loss.backward()\n",
    "  for p in net.parameters():\n",
    "    p.data.add_(- 0.001 * p.grad)\n",
    "    p.grad.data.zero_()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8DHmqNfxpPoQ"
   },
   "outputs": [],
   "source": [
    "plot(x.detach().numpy(), y.detach().numpy(), ylim=(-1, 1), xlim=(-5, 5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WX-qNKKPn2Ft"
   },
   "source": [
    "## Optimizers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pPhBZESWE-9M"
   },
   "source": [
    "We can use more fancy optimizers with the `optim` package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "pZ0ptKURn5TE"
   },
   "outputs": [],
   "source": [
    "from torch import optim\n",
    "\n",
    "net = Net(input_size=1, output_size=1)\n",
    "\n",
    "optimizer = optim.Adam(net.parameters(), lr=1e-3)\n",
    "\n",
    "x = torch.linspace(-5, 5, 100).view(-1, 1)\n",
    "y = net(x)\n",
    "y_target = torch.sin(x)\n",
    "loss_fn = nn.MSELoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TTmTVBaDFDP-"
   },
   "source": [
    "Here's the network before training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vR278Ii7ooXa"
   },
   "outputs": [],
   "source": [
    "plot(x.detach().numpy(), y.detach().numpy(), ylim=(-1, 1), xlim=(-5, 5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CCCE8LCoFEiB"
   },
   "source": [
    "and here's how you can use the optimize to train the network.\n",
    "Note that we call `zero_grad` _before_ calling `loss.backward()`, and then we just call `optimizer.step()`. This `step` function will take care of updating all the parameters that were passed to that optimizer's constructor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "sJfVNGhPod5i"
   },
   "outputs": [],
   "source": [
    "for _ in range(100):\n",
    "  y = net(x)\n",
    "  loss = loss_fn(y, y_target)\n",
    "\n",
    "  optimizer.zero_grad()\n",
    "  loss.backward()\n",
    "  optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lOM7_Xr6FUaV"
   },
   "source": [
    "And we see that this trained a network quite well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "219Pojw0os9X"
   },
   "outputs": [],
   "source": [
    "plot(x.detach().numpy(), y.detach().numpy(), ylim=(-1, 1), xlim=(-5, 5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5lS-wv_ABTky"
   },
   "source": [
    "## Regression example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j1KeajKoec60"
   },
   "source": [
    "We're going to train two separate neural networks to solve a prediction task:\n",
    "$$f_\\theta(x) \\approx y$$\n",
    "First we generate the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0_TUikVIuBCA"
   },
   "outputs": [],
   "source": [
    "N = 100\n",
    "d = 1\n",
    "X = np.random.randn(N, 1)\n",
    "Y = X * 2 + 3 + np.random.randn(N, 1)\n",
    "plt.scatter(X, Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "EJ3h8uW1WWYg"
   },
   "outputs": [],
   "source": [
    "print(X.shape, Y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c-2yrp4uFa43"
   },
   "source": [
    "Next we convert that data to pytorch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5Q6Mil6uUoJs"
   },
   "outputs": [],
   "source": [
    "X_pt = torch.from_numpy(X).to(torch.float32)\n",
    "Y_pt = torch.from_numpy(Y).to(torch.float32)\n",
    "\n",
    "loss_fn = nn.MSELoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d5hhaIlsFd_4"
   },
   "source": [
    "Define the training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JbQaSvbruUjo"
   },
   "outputs": [],
   "source": [
    "def train(net: nn.Module):\n",
    "  optimizer = optim.SGD(net.parameters(), lr=1e-2)\n",
    "  losses = []\n",
    "  for _  in range(100):\n",
    "    Y_hat_pt = net(X_pt)\n",
    "    loss = loss_fn(Y_hat_pt, Y_pt) \n",
    "  \n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "  \n",
    "    losses.append(loss.detach().numpy())\n",
    "  return np.array(losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "J9UGOhquFfz-"
   },
   "source": [
    "Let's test two different networks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "VH4ARnKUqVt0"
   },
   "outputs": [],
   "source": [
    "linear_network = nn.Linear(1, 1)\n",
    "linear_losses = train(linear_network)\n",
    "\n",
    "non_linear_network = Net(1, 1)\n",
    "non_linear_losses = train(non_linear_network)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2kHLnn2TFiXO"
   },
   "source": [
    "and plot the losses and predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "N3tTaRBkv573"
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "t = np.arange(len(linear_losses))\n",
    "plt.plot(t, linear_losses, t, non_linear_losses)\n",
    "plt.legend(['linear', 'non_linear'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Wu_cKZ5PUzT0"
   },
   "outputs": [],
   "source": [
    "x_probe = torch.linspace(X.min(), X.max(), 300).reshape(-1, 1)\n",
    "y_lin = linear_network(x_probe).detach().numpy()\n",
    "y_non_lin = non_linear_network(x_probe).detach().numpy()\n",
    "\n",
    "plt.scatter(x_probe.numpy(), y_lin, s=3)\n",
    "plt.scatter(x_probe.numpy(), y_non_lin, s=3)\n",
    "plt.scatter(X, Y)\n",
    "plt.legend(['linear', 'non_linear', 'data'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "N7Faplaeu1Ur"
   },
   "source": [
    "# PyTorch - Advanced"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lSt5Pr5gu5qX"
   },
   "source": [
    "## Distributions\n",
    "PyTorch has a very convenient [distributions](https://pytorch.org/docs/stable/distributions.html) package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qaCv2BGmbXWF"
   },
   "outputs": [],
   "source": [
    "from torch import distributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "C4p2WPuCGxww"
   },
   "source": [
    "You create distributions by passing the parameters of the distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "790r1b6tduyY"
   },
   "outputs": [],
   "source": [
    "mean = torch.zeros(1, requires_grad=True)\n",
    "std = torch.ones(1, requires_grad=True)\n",
    "gaussian = distributions.Normal(mean, std)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6Hoepp6iG0tl"
   },
   "source": [
    "These distributions are instances of the more general `Distribution` class, which you can read more about [here](https://pytorch.org/docs/stable/distributions.html#distribution)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1RO2ebhxgYUx"
   },
   "outputs": [],
   "source": [
    "print(gaussian)\n",
    "print(isinstance(gaussian, distributions.Distribution))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4X2m8tlmd0Rb"
   },
   "outputs": [],
   "source": [
    "sample = gaussian.sample((1,))\n",
    "print(sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0UbTN8I5Fmxs"
   },
   "outputs": [],
   "source": [
    "gaussian.log_prob(sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "z3BYXDwOG8Z-"
   },
   "source": [
    "The log probability depends on the the parameters of the distribution. So, calling `backward` on a loss that depends on `log_prob` will back-propagate gradients into the parmaeters of the distribution.\n",
    "\n",
    "NOTE: this won't back-propagate through the samples (the \"reparatermization trick''), unless you use `rsample`, which is only implemented for some distributions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gLAaXWLHeLco"
   },
   "source": [
    "at 5:18, we can set loss to -log_prob to maximize the probability of an event. That makes sense to me because the higher prob is, the smaller loss would be. Now if we want to incorporate reward into this loss function, i think usually people do loss=-log_prob*reward? but that means the higher the reward is, the higher the loss is. Is this because we want to push the prob of the event to be higher with a higher loss when we have a higher reward? or should we have lower loss with higher reward (loss=log_prob*reward)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "azs4-dQ3fWoP"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IZ7F-74Be6Rg"
   },
   "source": [
    "$$s_1, a_1, s_2, a_2, ...$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "p1Hm8zTjd9pR"
   },
   "outputs": [],
   "source": [
    "loss = - gaussian.log_prob(sample).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oxkylpG9hGZU"
   },
   "outputs": [],
   "source": [
    "loss.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3Ad0bfIChIgS"
   },
   "outputs": [],
   "source": [
    "mean.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_KmS36DoeAgn"
   },
   "source": [
    "## Batch-Wise distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HB1GRLL3GZ-d"
   },
   "source": [
    "The distributions also support batch-operations. In this case, all the operations (`sample`, `log_prob`, etc.) are batch-wise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "DfPMQRfWeDRs"
   },
   "outputs": [],
   "source": [
    "mean = torch.zeros(10)\n",
    "std = torch.ones(10)\n",
    "gaussian = distributions.Normal(mean, std)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cFPZf9mBhXMa"
   },
   "outputs": [],
   "source": [
    "gaussian"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "EhwiqGtseGD-"
   },
   "outputs": [],
   "source": [
    "sample = gaussian.sample((1,))\n",
    "print(sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fGVHHImreH5n"
   },
   "outputs": [],
   "source": [
    "gaussian.log_prob(sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wUn4p3aReLeR"
   },
   "source": [
    "## Multivariate Normal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OUchz-ayHlVz"
   },
   "source": [
    "There are other distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Dv74f_XZbTv_"
   },
   "outputs": [],
   "source": [
    "mean = torch.zeros(2)\n",
    "covariance = torch.tensor(\n",
    "    [[1, 0.8],\n",
    "     [0.8, 1]]\n",
    ")\n",
    "gaussian = distributions.MultivariateNormal(mean, covariance)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "g1vfCg1McpOM"
   },
   "outputs": [],
   "source": [
    "gaussian.sample((1,))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "XMG929mzeqJz"
   },
   "outputs": [],
   "source": [
    "samples = gaussian.sample((500,))\n",
    "plt.scatter(samples[:, 0].numpy(), samples[:, 1].numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vXCJs0dCHn54"
   },
   "source": [
    "NOTE: if you want to use a batch of `MultivariateNormal` distributions, you'll need to construct a batch of covariance matrices (i.e. shape `[BATCH_SIZE, DIM, DIM]`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KsZZu33nfDB5"
   },
   "source": [
    "## Categorical Distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "IfoFevpBdy9X"
   },
   "outputs": [],
   "source": [
    "from torch import distributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ePKpc3akHxit"
   },
   "source": [
    "Another useful distribution is the categorical distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QgRw31OMfEcV"
   },
   "outputs": [],
   "source": [
    "probs = torch.tensor([0.1, 0.2, 0.7])\n",
    "dist = distributions.Categorical(probs=probs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "K5ucMwVKfNv9"
   },
   "outputs": [],
   "source": [
    "sample = dist.sample([20])\n",
    "print(sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ye8mqQa4iDa5"
   },
   "outputs": [],
   "source": [
    "dist.log_prob(sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lNEq80qTfZ6s"
   },
   "source": [
    "## Distributions and Modules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OJm0r8WFIwqf"
   },
   "source": [
    "Typically, your network will output parameters of a distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "TkKsDyCmIzvF"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "\n",
    "class Net(nn.Module):\n",
    "\n",
    "    def __init__(self, input_size, output_size):\n",
    "        super(Net, self).__init__()\n",
    "        self.fc1 = nn.Linear(input_size, 32)\n",
    "        self.fc2 = nn.Linear(32, 32)\n",
    "        self.fc3 = nn.Linear(32, output_size)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3nCEfh-II2zV"
   },
   "outputs": [],
   "source": [
    "mean_network = Net(1, 1)\n",
    "x = torch.randn(100, 1)\n",
    "mean = mean_network(x)\n",
    "distribution = distributions.Normal(x, scale=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G9FDsX3DIpFn"
   },
   "source": [
    "If you want, your nn.Module can return a distribution in the `forward` function!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Gz2eMYZydMyC"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "\n",
    "class Net(nn.Module):\n",
    "\n",
    "    def __init__(self, input_size, output_size):\n",
    "        super(Net, self).__init__()\n",
    "        self.fc1 = nn.Linear(input_size, 32)\n",
    "        self.fc2 = nn.Linear(32, 32)\n",
    "        self.fc3 = nn.Linear(32, output_size)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return distributions.Normal(x, scale=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "L5KpD2reccWJ"
   },
   "outputs": [],
   "source": [
    "distribution_network = Net(1, 1)\n",
    "x = torch.randn(100, 1)\n",
    "distribution = distribution_network(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4Wh09beAu6d5"
   },
   "source": [
    "## `gather`\n",
    "This function will be useful for the DQN assignment.\n",
    "It allows you to index into arrays in a batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "l0A4vz6iJKab"
   },
   "outputs": [],
   "source": [
    "x = torch.arange(6).reshape(2, 3)\n",
    "y = torch.tensor([0, 1]).reshape(2, 1)\n",
    "print(x)\n",
    "print(y)\n",
    "print(torch.gather(x, 0, y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "NfMgAVAYu7n_"
   },
   "outputs": [],
   "source": [
    "x = torch.arange(6).reshape(2, 3)\n",
    "y = torch.tensor([0, 1, 0]).reshape(1, 3)\n",
    "print(x)\n",
    "print(y)\n",
    "print(torch.gather(x, 0, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hUmPV_O_cFz4"
   },
   "source": [
    "For a 3-D tensor the output is specified by::\n",
    "\n",
    "    out[i][j][k] = input[index[i][j][k]][j][k]  # if dim == 0\n",
    "    out[i][j][k] = input[i][index[i][j][k]][k]  # if dim == 1\n",
    "    out[i][j][k] = input[i][j][index[i][j][k]]  # if dim == 2"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Copy of PyTorch_Tutorial.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}