notebooks/AI4Dev_05_dnns_mnist.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a href=\"https://colab.research.google.com/github/battistabiggio/ai4dev/blob/main/notebooks/AI4Dev_05_dnns_mnist.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Learning from images\n",
        "\n",
        "In this notebook, we will load a dataset of images and go back to the classification problem.\n",
        "\n",
        "We will load the MNIST dataset, a (labeled) dataset of images of handwritten digits. We can use the `torchvision` library to load and process data in the image domain. Additionally, we can use `matplotlib` for visualization and `pillow` for processing them.\n",
        "\n",
        "We will also inspect how data can be loaded in PyTorch, how to create batches of images rather than passing the whole dataset at once, and also how to implement the learning phase (this last part won't be much different than the previous notebook)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 820
        },
        "id": "_fkvuH4z65Z1",
        "outputId": "b680b6be-7e0d-477a-a22e-03a09caf0a13"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "samples in training dataset:  60000\n",
            "samples in testing dataset:  10000\n",
            "shape of sample\n",
            "(28, 28)\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGzCAYAAABpdMNsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAlZklEQVR4nO3df1TVdZ7H8ReiXFHhMshvQ8QfyaRiu6aMY6OZJLDlptlONeM5aK5tDbqTTtbi2aIfqzQ6/XJyHWdqcNotK9vUbMomEfBM/mj9NdZphxViRkogNbkXIdDgs394vNsN/PHFix/A5+Oczznc7/fz5vvm2zdefu/3y/cGGWOMAAC4zHrYbgAAcGUigAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggIB2CgoK0qOPPhqw7zdo0CDNnj3bcV1xcbGCgoL0xhtvBKyXtWvXKigoSH/5y18C9j2BbyOAYNVHH32k22+/XUlJSerdu7cGDBigm266Sb/85S9tt4ZLMHv2bAUFBbUaKSkptltDJ9LTdgO4cu3YsUOTJ0/WwIEDNW/ePMXFxamyslK7du3Sc889pwULFthuEZfA5XLphRde8FvmdrstdYPOiACCNUuXLpXb7dZ///d/KyIiwm/dF198YacpBEzPnj01a9Ys222gE+MtOFhTXl6uESNGtAofSYqJifF7XVBQoBtvvFExMTFyuVy65pprtHr16lZ1gwYN0i233KLi4mJdd911Cg0N1ahRo1RcXCxJevPNNzVq1Cj17t1bY8aM0f79+/3qZ8+erX79+unTTz9VRkaG+vbtq4SEBD3++OO6mAfHf/7557r77rsVGxsrl8ulESNG6Le//e3F75Rv+PLLL/XAAw9o1KhR6tevn8LDw5WVlaU//elPbc5vbm7WkiVLFBcXp759++rv//7vVVlZ2Wre7t27lZmZKbfbrT59+mjSpEn64IMPLtiPx+PRn//8Z3k8nov+GZqbm+X1ei96Pq4sBBCsSUpK0t69e/Xxxx9fcO7q1auVlJSkJUuW6KmnnlJiYqJ+8pOfaNWqVa3mlpWV6Uc/+pGmTZum/Px8nThxQtOmTdPLL7+shQsXatasWXrsscdUXl6uH/7wh2ppafGrb25uVmZmpmJjY7V8+XKNGTNGeXl5ysvLO2+PNTU1+t73vqetW7dq/vz5eu655zR06FDNnTtXzz77rKN9I0mffvqpNm7cqFtuuUVPP/20Fi9erI8++kiTJk3SkSNHWs1funSpfv/73+uhhx7SP//zP+v9999Xenq6vvrqK9+cbdu2aeLEifJ6vcrLy9OyZctUW1urG2+8UR9++OF5+9mwYYO++93vasOGDRfVf0NDg8LDw+V2uxUZGamcnBydPHnS2U5A92YAS/7whz+Y4OBgExwcbMaPH28efPBB895775lTp061mtvQ0NBqWUZGhhk8eLDfsqSkJCPJ7Nixw7fsvffeM5JMaGio+etf/+pbvmbNGiPJFBUV+ZZlZ2cbSWbBggW+ZS0tLebmm282ISEh5ujRo77lkkxeXp7v9dy5c018fLw5duyYX0933nmncbvdbf4M3+49Ozvb97qxsdE0Nzf7zamoqDAul8s8/vjjvmVFRUVGkhkwYIDxer2+5a+//rqRZJ577jnfzzFs2DCTkZFhWlpafPMaGhpMcnKyuemmm3zLCgoKjCRTUVHRallBQcF5fw5jjPmXf/kX89BDD5nXXnvNrFu3zrdfJ0yYYE6fPn3BelwZCCBY9eGHH5oZM2aYPn36GElGkomOjjabNm06Z01tba05evSoWbZsmZFkamtrfeuSkpLMNddc02q+JHPzzTf7LT9w4ICRZF588UXfsrO/KEtLS/3mvvvuu0aSWbdunW/ZNwOopaXFREREmHvuucccPXrUb5z9xf3HP/7xvPvi2wH0TV9//bU5duyYOXr0qElNTTXTp0/3rTsbQLm5uX41LS0tJj4+3mRkZBhjjNm3b5+RZH73u9+16vEf//Efjcvl8gVeWwF0qZYuXdpqH+LKxltwsGrs2LF68803deLECX344YfKzc1VXV2dbr/9dn3yySe+eR988IHS09PVt29fRUREKDo6WkuWLJGkVtckBg4c6Pf67J1XiYmJbS4/ceKE3/IePXpo8ODBfsuuvvpqSTrn38UcPXpUtbW1+vWvf63o6Gi/MWfOHEnOb6xoaWnRM888o2HDhsnlcikqKkrR0dE6ePBgm9dhhg0b5vc6KChIQ4cO9fV86NAhSVJ2dnarHl944QU1NTU5ur7j1MKFC9WjRw9t3bq1w7aBroW74NAphISEaOzYsRo7dqyuvvpqzZkzR+vXr1deXp7Ky8s1ZcoUpaSk6Omnn1ZiYqJCQkL0zjvv6Jlnnml1DSc4OLjNbZxruQnAp9Kf7WHWrFnKzs5uc05qaqqj77ls2TI9/PDDuvvuu/XEE08oMjJSPXr00P3339/qZ3bS44oVK3Tttde2Oadfv36Ov+/FCg0NVf/+/fXll1922DbQtRBA6HSuu+46SVJVVZUkafPmzWpqatJbb73ld3ZTVFTUIdtvaWnRp59+6jvrkaT//d//lXTmLru2REdHKywsTM3NzUpPTw9IH2+88YYmT56sF1980W95bW2toqKiWs0/e4ZzljFGZWVlvuAbMmSIJCk8PDxgPTpRV1enY8eOKTo6+rJvG50Tb8HBmqKiojbPPt555x1J0vDhwyX9/5nLN+d6PB4VFBR0WG/PP/+872tjjJ5//nn16tVLU6ZMaXN+cHCwZs6cqf/6r/9q866+o0ePOu4hODi41f5Zv369Pv/88zbnv/TSS6qrq/O9fuONN1RVVaWsrCxJ0pgxYzRkyBD94he/aPNutAv1eLG3YTc2Nvr1cdYTTzwhY4wyMzPPW48rB2dAsGbBggVqaGjQjBkzlJKSolOnTmnHjh167bXXNGjQIN+1k6lTpyokJETTpk3TP/3TP+nkyZP6zW9+o5iYGN9ZUiD17t1bW7ZsUXZ2ttLS0vTuu+/q97//vZYsWXLef70/+eSTKioqUlpamubNm6drrrlGX375pfbt26etW7c6fuvplltu0eOPP645c+bo+9//vj766CO9/PLLra5PnRUZGanrr79ec+bMUU1NjZ599lkNHTpU8+bNk3Tm2tYLL7ygrKwsjRgxQnPmzNGAAQP0+eefq6ioSOHh4dq8efM5+9mwYYPmzJmjgoKC8z6zrrq6Wn/zN3+ju+66y/fonffee0/vvPOOMjMzdeuttzraD+jGLN4AgSvcu+++a+6++26TkpJi+vXrZ0JCQszQoUPNggULTE1Njd/ct956y6SmpprevXubQYMGmZ///Ofmt7/9bas7tZKSklrd7WbMmTvWcnJy/JZVVFQYSWbFihW+ZdnZ2aZv376mvLzcTJ061fTp08fExsaavLy8VrdE61u3YRtjTE1NjcnJyTGJiYmmV69eJi4uzkyZMsX8+te/vuD+aOs27J/97GcmPj7ehIaGmgkTJpidO3eaSZMmmUmTJvnmnb0Lbt26dSY3N9fExMSY0NBQc/PNN/vddn7W/v37zW233Wb69+9vXC6XSUpKMj/84Q9NYWGhb86l3IZ94sQJM2vWLDN06FDTp08f43K5zIgRI8yyZcvavMUeV64gYwJwBRboJmbPnq033niDP5gELgOuAQEArCCAAABWEEAAACu4BgQAsIIzIACAFQQQAMCKTveHqC0tLTpy5IjCwsIUFBRkux0AgEPGGNXV1SkhIUE9epz7PKfTBdCRI0daPbUYAND1VFZW6qqrrjrn+k73FlxYWJjtFgAAAXCh3+cdFkCrVq3SoEGD1Lt3b6WlpV3w437P4m03AOgeLvT7vEMC6LXXXtOiRYuUl5enffv2afTo0crIyHD8gVwAgG6sIx4wN27cOL8HPzY3N5uEhASTn59/wVqPx+P7aGYGg8FgdN3h8XjO+/s+4GdAp06d0t69e/0+8KpHjx5KT0/Xzp07W81vamqS1+v1GwCA7i/gAXTs2DE1NzcrNjbWb3lsbKyqq6tbzc/Pz5fb7fYN7oADgCuD9bvgcnNz5fF4fKOystJ2SwCAyyDgfwcUFRWl4OBg1dTU+C2vqalRXFxcq/kul0sulyvQbQAAOrmAnwGFhIRozJgxKiws9C1raWlRYWGhxo8fH+jNAQC6qA55EsKiRYuUnZ2t6667TuPGjdOzzz6r+vp6zZkzpyM2BwDogjokgO644w4dPXpUjzzyiKqrq3Xttddqy5YtrW5MAABcuTrd5wF5vV653W7bbQAALpHH41F4ePg511u/Cw4AcGUigAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBU9bTcAdCbBwcGOa9xudwd0Ehjz589vV12fPn0c1wwfPtxxTU5OjuOaX/ziF45r7rrrLsc1ktTY2Oi45sknn3Rc89hjjzmu6Q44AwIAWEEAAQCsCHgAPfroowoKCvIbKSkpgd4MAKCL65BrQCNGjNDWrVv/fyM9udQEAPDXIcnQs2dPxcXFdcS3BgB0Ex1yDejQoUNKSEjQ4MGD9eMf/1iHDx8+59ympiZ5vV6/AQDo/gIeQGlpaVq7dq22bNmi1atXq6KiQj/4wQ9UV1fX5vz8/Hy53W7fSExMDHRLAIBOKOABlJWVpX/4h39QamqqMjIy9M4776i2tlavv/56m/Nzc3Pl8Xh8o7KyMtAtAQA6oQ6/OyAiIkJXX321ysrK2lzvcrnkcrk6ug0AQCfT4X8HdPLkSZWXlys+Pr6jNwUA6EICHkAPPPCASkpK9Je//EU7duzQjBkzFBwc3O5HYQAAuqeAvwX32Wef6a677tLx48cVHR2t66+/Xrt27VJ0dHSgNwUA6MICHkCvvvpqoL8lOqmBAwc6rgkJCXFc8/3vf99xzfXXX++4RjpzzdKpmTNntmtb3c1nn33muGblypWOa2bMmOG45lx34V7In/70J8c1JSUl7drWlYhnwQEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFUHGGGO7iW/yer1yu92227iiXHvtte2q27Ztm+Ma/tt2DS0tLY5r7r77bsc1J0+edFzTHlVVVe2qO3HihOOa0tLSdm2rO/J4PAoPDz/nes6AAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYEVP2w3AvsOHD7er7vjx445reBr2Gbt373ZcU1tb67hm8uTJjmsk6dSpU45r/uM//qNd28KVizMgAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCh5FCX375ZbvqFi9e7LjmlltucVyzf/9+xzUrV650XNNeBw4ccFxz0003Oa6pr693XDNixAjHNZL005/+tF11gBOcAQEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFUHGGGO7iW/yer1yu92220AHCQ8Pd1xTV1fnuGbNmjWOayRp7ty5jmtmzZrluGbdunWOa4CuxuPxnPf/ec6AAABWEEAAACscB9D27ds1bdo0JSQkKCgoSBs3bvRbb4zRI488ovj4eIWGhio9PV2HDh0KVL8AgG7CcQDV19dr9OjRWrVqVZvrly9frpUrV+pXv/qVdu/erb59+yojI0ONjY2X3CwAoPtw/ImoWVlZysrKanOdMUbPPvus/vVf/1W33nqrJOmll15SbGysNm7cqDvvvPPSugUAdBsBvQZUUVGh6upqpaen+5a53W6lpaVp586dbdY0NTXJ6/X6DQBA9xfQAKqurpYkxcbG+i2PjY31rfu2/Px8ud1u30hMTAxkSwCATsr6XXC5ubnyeDy+UVlZabslAMBlENAAiouLkyTV1NT4La+pqfGt+zaXy6Xw8HC/AQDo/gIaQMnJyYqLi1NhYaFvmdfr1e7duzV+/PhAbgoA0MU5vgvu5MmTKisr872uqKjQgQMHFBkZqYEDB+r+++/Xv/3bv2nYsGFKTk7Www8/rISEBE2fPj2QfQMAujjHAbRnzx5NnjzZ93rRokWSpOzsbK1du1YPPvig6uvrdc8996i2tlbXX3+9tmzZot69eweuawBAl8fDSNEtrVixol11Z/9B5URJSYnjmm/+qcLFamlpcVwD2MTDSAEAnRIBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABW8DRsdEt9+/ZtV93mzZsd10yaNMlxTVZWluOaP/zhD45rAJt4GjYAoFMigAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBU8jBT4hiFDhjiu2bdvn+Oa2tpaxzVFRUWOa/bs2eO4RpJWrVrluKaT/SpBJ8DDSAEAnRIBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArOBhpMAlmjFjhuOagoICxzVhYWGOa9pryZIljmteeuklxzVVVVWOa9B18DBSAECnRAABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAAreBgpYMHIkSMd1zz99NOOa6ZMmeK4pr3WrFnjuGbp0qWOaz7//HPHNbCDh5ECADolAggAYIXjANq+fbumTZumhIQEBQUFaePGjX7rZ8+eraCgIL+RmZkZqH4BAN2E4wCqr6/X6NGjtWrVqnPOyczMVFVVlW+sW7fukpoEAHQ/PZ0WZGVlKSsr67xzXC6X4uLi2t0UAKD765BrQMXFxYqJidHw4cN133336fjx4+ec29TUJK/X6zcAAN1fwAMoMzNTL730kgoLC/Xzn/9cJSUlysrKUnNzc5vz8/Pz5Xa7fSMxMTHQLQEAOiHHb8FdyJ133un7etSoUUpNTdWQIUNUXFzc5t8k5ObmatGiRb7XXq+XEAKAK0CH34Y9ePBgRUVFqaysrM31LpdL4eHhfgMA0P11eAB99tlnOn78uOLj4zt6UwCALsTxW3AnT570O5upqKjQgQMHFBkZqcjISD322GOaOXOm4uLiVF5ergcffFBDhw5VRkZGQBsHAHRtjgNoz549mjx5su/12es32dnZWr16tQ4ePKjf/e53qq2tVUJCgqZOnaonnnhCLpcrcF0DALo8HkYKdBERERGOa6ZNm9aubRUUFDiuCQoKclyzbds2xzU33XST4xrYwcNIAQCdEgEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFbwNGwArTQ1NTmu6dnT8ae76Ouvv3Zc057PFisuLnZcg0vH07ABAJ0SAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKxw/vRAAJcsNTXVcc3tt9/uuGbs2LGOa6T2PVi0PT755BPHNdu3b++ATmADZ0AAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUPIwW+Yfjw4Y5r5s+f77jmtttuc1wTFxfnuOZyam5udlxTVVXluKalpcVxDTonzoAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoeRopOrz0P4bzrrrvata32PFh00KBB7dpWZ7Znzx7HNUuXLnVc89ZbbzmuQffBGRAAwAoCCABghaMAys/P19ixYxUWFqaYmBhNnz5dpaWlfnMaGxuVk5Oj/v37q1+/fpo5c6ZqamoC2jQAoOtzFEAlJSXKycnRrl279P777+v06dOaOnWq6uvrfXMWLlyozZs3a/369SopKdGRI0fa9eFbAIDuzdFNCFu2bPF7vXbtWsXExGjv3r2aOHGiPB6PXnzxRb3yyiu68cYbJUkFBQX67ne/q127dul73/te4DoHAHRpl3QNyOPxSJIiIyMlSXv37tXp06eVnp7um5OSkqKBAwdq586dbX6PpqYmeb1evwEA6P7aHUAtLS26//77NWHCBI0cOVKSVF1drZCQEEVERPjNjY2NVXV1dZvfJz8/X2632zcSExPb2xIAoAtpdwDl5OTo448/1quvvnpJDeTm5srj8fhGZWXlJX0/AEDX0K4/RJ0/f77efvttbd++XVdddZVveVxcnE6dOqXa2lq/s6Camppz/jGhy+WSy+VqTxsAgC7M0RmQMUbz58/Xhg0btG3bNiUnJ/utHzNmjHr16qXCwkLfstLSUh0+fFjjx48PTMcAgG7B0RlQTk6OXnnlFW3atElhYWG+6zput1uhoaFyu92aO3euFi1apMjISIWHh2vBggUaP348d8ABAPw4CqDVq1dLkm644Qa/5QUFBZo9e7Yk6ZlnnlGPHj00c+ZMNTU1KSMjQ//+7/8ekGYBAN1HkDHG2G7im7xer9xut+02cBFiY2Md11xzzTWOa55//nnHNSkpKY5rOrvdu3c7rlmxYkW7trVp0ybHNS0tLe3aFrovj8ej8PDwc67nWXAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwol2fiIrOKzIy0nHNmjVr2rWta6+91nHN4MGD27WtzmzHjh2Oa5566inHNe+9957jmq+++spxDXC5cAYEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFbwMNLLJC0tzXHN4sWLHdeMGzfOcc2AAQMc13R2DQ0N7apbuXKl45ply5Y5rqmvr3dcA3Q3nAEBAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBU8jPQymTFjxmWpuZw++eQTxzVvv/2245qvv/7acc1TTz3luEaSamtr21UHwDnOgAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADAiiBjjLHdxDd5vV653W7bbQAALpHH41F4ePg513MGBACwggACAFjhKIDy8/M1duxYhYWFKSYmRtOnT1dpaanfnBtuuEFBQUF+49577w1o0wCArs9RAJWUlCgnJ0e7du3S+++/r9OnT2vq1Kmqr6/3mzdv3jxVVVX5xvLlywPaNACg63P0iahbtmzxe7127VrFxMRo7969mjhxom95nz59FBcXF5gOAQDd0iVdA/J4PJKkyMhIv+Uvv/yyoqKiNHLkSOXm5qqhoeGc36OpqUler9dvAACuAKadmpubzc0332wmTJjgt3zNmjVmy5Yt5uDBg+Y///M/zYABA8yMGTPO+X3y8vKMJAaDwWB0s+HxeM6bI+0OoHvvvdckJSWZysrK884rLCw0kkxZWVmb6xsbG43H4/GNyspK6zuNwWAwGJc+LhRAjq4BnTV//ny9/fbb2r59u6666qrzzk1LS5MklZWVaciQIa3Wu1wuuVyu9rQBAOjCHAWQMUYLFizQhg0bVFxcrOTk5AvWHDhwQJIUHx/frgYBAN2TowDKycnRK6+8ok2bNiksLEzV1dWSJLfbrdDQUJWXl+uVV17R3/3d36l///46ePCgFi5cqIkTJyo1NbVDfgAAQBfl5LqPzvE+X0FBgTHGmMOHD5uJEyeayMhI43K5zNChQ83ixYsv+D7gN3k8HuvvWzIYDAbj0seFfvfzMFIAQIfgYaQAgE6JAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCi0wWQMcZ2CwCAALjQ7/NOF0B1dXW2WwAABMCFfp8HmU52ytHS0qIjR44oLCxMQUFBfuu8Xq8SExNVWVmp8PBwSx3ax344g/1wBvvhDPbDGZ1hPxhjVFdXp4SEBPXoce7znJ6XsaeL0qNHD1111VXnnRMeHn5FH2BnsR/OYD+cwX44g/1whu394Ha7Lzin070FBwC4MhBAAAArulQAuVwu5eXlyeVy2W7FKvbDGeyHM9gPZ7AfzuhK+6HT3YQAALgydKkzIABA90EAAQCsIIAAAFYQQAAAKwggAIAVXSaAVq1apUGDBql3795KS0vThx9+aLuly+7RRx9VUFCQ30hJSbHdVofbvn27pk2bpoSEBAUFBWnjxo1+640xeuSRRxQfH6/Q0FClp6fr0KFDdprtQBfaD7Nnz251fGRmZtpptoPk5+dr7NixCgsLU0xMjKZPn67S0lK/OY2NjcrJyVH//v3Vr18/zZw5UzU1NZY67hgXsx9uuOGGVsfDvffea6njtnWJAHrttde0aNEi5eXlad++fRo9erQyMjL0xRdf2G7tshsxYoSqqqp8449//KPtljpcfX29Ro8erVWrVrW5fvny5Vq5cqV+9atfaffu3erbt68yMjLU2Nh4mTvtWBfaD5KUmZnpd3ysW7fuMnbY8UpKSpSTk6Ndu3bp/fff1+nTpzV16lTV19f75ixcuFCbN2/W+vXrVVJSoiNHjui2226z2HXgXcx+kKR58+b5HQ/Lly+31PE5mC5g3LhxJicnx/e6ubnZJCQkmPz8fItdXX55eXlm9OjRttuwSpLZsGGD73VLS4uJi4szK1as8C2rra01LpfLrFu3zkKHl8e394MxxmRnZ5tbb73VSj+2fPHFF0aSKSkpMcac+W/fq1cvs379et+c//mf/zGSzM6dO2212eG+vR+MMWbSpEnmpz/9qb2mLkKnPwM6deqU9u7dq/T0dN+yHj16KD09XTt37rTYmR2HDh1SQkKCBg8erB//+Mc6fPiw7ZasqqioUHV1td/x4Xa7lZaWdkUeH8XFxYqJidHw4cN133336fjx47Zb6lAej0eSFBkZKUnau3evTp8+7Xc8pKSkaODAgd36ePj2fjjr5ZdfVlRUlEaOHKnc3Fw1NDTYaO+cOt3TsL/t2LFjam5uVmxsrN/y2NhY/fnPf7bUlR1paWlau3athg8frqqqKj322GP6wQ9+oI8//lhhYWG227Oiurpakto8Ps6uu1JkZmbqtttuU3JyssrLy7VkyRJlZWVp586dCg4Ott1ewLW0tOj+++/XhAkTNHLkSElnjoeQkBBFRET4ze3Ox0Nb+0GSfvSjHykpKUkJCQk6ePCgHnroIZWWlurNN9+02K2/Th9A+H9ZWVm+r1NTU5WWlqakpCS9/vrrmjt3rsXO0Bnceeedvq9HjRql1NRUDRkyRMXFxZoyZYrFzjpGTk6OPv744yviOuj5nGs/3HPPPb6vR40apfj4eE2ZMkXl5eUaMmTI5W6zTZ3+LbioqCgFBwe3uoulpqZGcXFxlrrqHCIiInT11VerrKzMdivWnD0GOD5aGzx4sKKiorrl8TF//ny9/fbbKioq8vv8sLi4OJ06dUq1tbV+87vr8XCu/dCWtLQ0SepUx0OnD6CQkBCNGTNGhYWFvmUtLS0qLCzU+PHjLXZm38mTJ1VeXq74+HjbrViTnJysuLg4v+PD6/Vq9+7dV/zx8dlnn+n48ePd6vgwxmj+/PnasGGDtm3bpuTkZL/1Y8aMUa9evfyOh9LSUh0+fLhbHQ8X2g9tOXDggCR1ruPB9l0QF+PVV181LpfLrF271nzyySfmnnvuMREREaa6utp2a5fVz372M1NcXGwqKirMBx98YNLT001UVJT54osvbLfWoerq6sz+/fvN/v37jSTz9NNPm/3795u//vWvxhhjnnzySRMREWE2bdpkDh48aG699VaTnJxsvvrqK8udB9b59kNdXZ154IEHzM6dO01FRYXZunWr+du//VszbNgw09jYaLv1gLnvvvuM2+02xcXFpqqqyjcaGhp8c+69914zcOBAs23bNrNnzx4zfvx4M378eItdB96F9kNZWZl5/PHHzZ49e0xFRYXZtGmTGTx4sJk4caLlzv11iQAyxphf/vKXZuDAgSYkJMSMGzfO7Nq1y3ZLl90dd9xh4uPjTUhIiBkwYIC54447TFlZme22OlxRUZGR1GpkZ2cbY87civ3www+b2NhY43K5zJQpU0xpaandpjvA+fZDQ0ODmTp1qomOjja9evUySUlJZt68ed3uH2lt/fySTEFBgW/OV199ZX7yk5+Y73znO6ZPnz5mxowZpqqqyl7THeBC++Hw4cNm4sSJJjIy0rhcLjN06FCzePFi4/F47Db+LXweEADAik5/DQgA0D0RQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAV/wciOz2spatUFAAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from torchvision import datasets\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "\n",
        "data_path = 'data'\n",
        "mnist_train = datasets.MNIST(data_path,\n",
        "                             train=True,\n",
        "                             download=True)\n",
        "mnist_validation = datasets.MNIST(data_path,\n",
        "                                  train=False,\n",
        "                                  download=True)\n",
        "\n",
        "print(\"samples in training dataset: \", len(mnist_train))\n",
        "print(\"samples in testing dataset: \", len(mnist_validation))\n",
        "\n",
        "# first sample\n",
        "image, label = mnist_train[0]\n",
        "print(\"shape of sample\")\n",
        "print(np.array(image).shape)\n",
        "\n",
        "# visualize the image\n",
        "plt.imshow(image, cmap='gray')\n",
        "plt.title(f\"Sample label: {label}\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In torchvision we also have tools to preprocess the images, as well as converting them directly to PyTorch tensors.\n",
        "As we load the image and convert to tensor, we can also inspect what changed. First, the shape now includes an additional dimension that is the batch index."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AdbK9BL59sbQ",
        "outputId": "98f372b1-d645-4d7f-ec6d-6bd76c5ca948"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "torch.Size([1, 28, 28])\n"
          ]
        }
      ],
      "source": [
        "from torchvision import transforms\n",
        "\n",
        "to_tensor = transforms.ToTensor()\n",
        "image_as_tensor = to_tensor(image)\n",
        "print(image_as_tensor.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Additionally, we can also apply transformations to the images. These are in general useful if we want the model to generalize across different modifications of the samples, rather than learning only from well-placed input data.\n",
        "\n",
        "We also define a function to visualize the image so that we can easily display it by passing a tensor without repeating lines of code in our notebook (matplotlib requires images in the format CHW - channels, height, width while pytorch encodes as HWC)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 473
        },
        "id": "aZeHUguQ_p5B",
        "outputId": "a2774788-3a05-4466-d90b-dab129b67c89"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGzCAYAAABpdMNsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAmJ0lEQVR4nO3de3hU9Z3H8U8SyBAgGTaQK4YQLpJytQVMWRRUIkmqVBAfb/XZgBRWDbRAvWzwEsGVWOwqXii2VaHuClqoQLWClUvCgwRYERata5bEtKCQGNDMhAQCm/z2Dx5mHRMuJ8zwy+X9ep7zPMw5v++cb47nyccz5+Q3IcYYIwAALrFQ2w0AANonAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAghoppCQED3++OMBe7/evXtrypQpjusKCgoUEhKi1atXB6yX5cuXKyQkRH/7298C9p7AdxFAsOrjjz/WLbfcouTkZHXq1Ek9e/bU9ddfrxdeeMF2a7gIU6ZMUUhISKMlNTXVdmtoQTrYbgDt1/bt23XttdeqV69emj59uuLj43Xw4EHt2LFDzz33nGbNmmW7RVwEl8ull19+2W+d2+221A1aIgII1jz55JNyu936z//8T3Xr1s1v21dffWWnKQRMhw4ddNddd9luAy0YH8HBmtLSUg0aNKhR+EhSbGys3+tly5bpuuuuU2xsrFwulwYOHKilS5c2quvdu7duvPFGFRQUaMSIEYqIiNCQIUNUUFAgSXrrrbc0ZMgQderUScOHD9eePXv86qdMmaKuXbvq888/V0ZGhrp06aLExEQtWLBAFzJx/Jdffqm7775bcXFxcrlcGjRokF599dULPyjf8vXXX+v+++/XkCFD1LVrV0VFRSkrK0v/9V//1eT4+vp6zZs3T/Hx8erSpYt+/OMf6+DBg43G7dy5U5mZmXK73ercubPGjh2rDz744Lz9eDweffbZZ/J4PBf8M9TX18vr9V7weLQvBBCsSU5O1u7du/XJJ5+cd+zSpUuVnJysefPm6d/+7d+UlJSk++67T0uWLGk0tqSkRHfeeacmTJig/Px8ffPNN5owYYJef/11zZkzR3fddZfmz5+v0tJS3XrrrWpoaPCrr6+vV2ZmpuLi4rRo0SINHz5ceXl5ysvLO2ePFRUV+uEPf6iNGzdq5syZeu6559SvXz9NmzZNixcvdnRsJOnzzz/X2rVrdeONN+qZZ57RAw88oI8//lhjx47VoUOHGo1/8skn9ec//1kPPfSQfvazn+n9999Xenq6jh8/7huzefNmjRkzRl6vV3l5eVq4cKGqqqp03XXXadeuXefsZ82aNfre976nNWvWXFD/tbW1ioqKktvtVnR0tHJycnTs2DFnBwFtmwEs+ctf/mLCwsJMWFiYGTVqlHnwwQfNe++9Z06ePNlobG1tbaN1GRkZpk+fPn7rkpOTjSSzfft237r33nvPSDIRERHm73//u2/9b37zGyPJbNmyxbcuOzvbSDKzZs3yrWtoaDA33HCDCQ8PN5WVlb71kkxeXp7v9bRp00xCQoI5cuSIX0+33367cbvdTf4M3+09Ozvb9/rEiROmvr7eb0xZWZlxuVxmwYIFvnVbtmwxkkzPnj2N1+v1rf/DH/5gJJnnnnvO93P079/fZGRkmIaGBt+42tpak5KSYq6//nrfumXLlhlJpqysrNG6ZcuWnfPnMMaYf/mXfzEPPfSQefPNN83KlSt9x3X06NHm1KlT561H+0AAwapdu3aZSZMmmc6dOxtJRpKJiYkx69atO2tNVVWVqaysNAsXLjSSTFVVlW9bcnKyGThwYKPxkswNN9zgt37v3r1GknnllVd86878oiwuLvYbu379eiPJrFy50rfu2wHU0NBgunXrZmbMmGEqKyv9ljO/uLdt23bOY/HdAPq2//3f/zVHjhwxlZWVZujQoWbixIm+bWcCKDc316+moaHBJCQkmIyMDGOMMR999JGRZH7/+9836vGnP/2pcblcvsBrKoAu1pNPPtnoGKJ94yM4WDVy5Ei99dZb+uabb7Rr1y7l5uaqurpat9xyiz799FPfuA8++EDp6enq0qWLunXrppiYGM2bN0+SGt2T6NWrl9/rM09eJSUlNbn+m2++8VsfGhqqPn36+K27/PLLJemsfxdTWVmpqqoq/fa3v1VMTIzfMnXqVEnOH6xoaGjQs88+q/79+8vlcqlHjx6KiYnRvn37mrwP079/f7/XISEh6tevn6/n/fv3S5Kys7Mb9fjyyy+rrq7O0f0dp+bMmaPQ0FBt3LgxaPtA68JTcGgRwsPDNXLkSI0cOVKXX365pk6dqlWrVikvL0+lpaUaN26cUlNT9cwzzygpKUnh4eF699139eyzzza6hxMWFtbkPs623gTgW+nP9HDXXXcpOzu7yTFDhw519J4LFy7Uo48+qrvvvltPPPGEoqOjFRoaqtmzZzf6mZ30+PTTT+uKK65ockzXrl0dv++FioiIUPfu3fX1118HbR9oXQggtDgjRoyQJB0+fFiS9Pbbb6uurk5/+tOf/K5utmzZEpT9NzQ06PPPP/dd9UjS//zP/0g6/ZRdU2JiYhQZGan6+nqlp6cHpI/Vq1fr2muv1SuvvOK3vqqqSj169Gg0/swVzhnGGJWUlPiCr2/fvpKkqKiogPXoRHV1tY4cOaKYmJhLvm+0THwEB2u2bNnS5NXHu+++K0kaMGCApP+/cvn2WI/Ho2XLlgWttxdffNH3b2OMXnzxRXXs2FHjxo1rcnxYWJgmT56sP/7xj00+1VdZWem4h7CwsEbHZ9WqVfryyy+bHP/aa6+purra93r16tU6fPiwsrKyJEnDhw9X37599atf/arJp9HO1+OFPoZ94sQJvz7OeOKJJ2SMUWZm5jnr0X5wBQRrZs2apdraWk2aNEmpqak6efKktm/frjfffFO9e/f23TsZP368wsPDNWHCBP3zP/+zjh07pt/97neKjY31XSUFUqdOnbRhwwZlZ2crLS1N69ev15///GfNmzfvnP/3/tRTT2nLli1KS0vT9OnTNXDgQH399df66KOPtHHjRscfPd14441asGCBpk6dqn/8x3/Uxx9/rNdff73R/akzoqOjddVVV2nq1KmqqKjQ4sWL1a9fP02fPl3S6XtbL7/8srKysjRo0CBNnTpVPXv21JdffqktW7YoKipKb7/99ln7WbNmjaZOnaply5adc8668vJyff/739cdd9zhm3rnvffe07vvvqvMzEzddNNNjo4D2jCLD0CgnVu/fr25++67TWpqqunatasJDw83/fr1M7NmzTIVFRV+Y//0pz+ZoUOHmk6dOpnevXubX/7yl+bVV19t9KRWcnJyo6fdjDn9xFpOTo7furKyMiPJPP3007512dnZpkuXLqa0tNSMHz/edO7c2cTFxZm8vLxGj0TrO49hG2NMRUWFycnJMUlJSaZjx44mPj7ejBs3zvz2t7897/Fo6jHsX/ziFyYhIcFERESY0aNHm6KiIjN27FgzduxY37gzT8GtXLnS5ObmmtjYWBMREWFuuOEGv8fOz9izZ4+5+eabTffu3Y3L5TLJycnm1ltvNZs2bfKNuZjHsL/55htz1113mX79+pnOnTsbl8tlBg0aZBYuXNjkI/Zov0KMCcAdWKCNmDJlilavXs0fTAKXAPeAAABWEEAAACsIIACAFdwDAgBYwRUQAMAKAggAYEWL+0PUhoYGHTp0SJGRkQoJCbHdDgDAIWOMqqurlZiYqNDQs1/ntLgAOnToUKNZiwEArc/Bgwd12WWXnXV7i/sILjIy0nYLAIAAON/v86AF0JIlS9S7d2916tRJaWlp5/263zP42A0A2obz/T4PSgC9+eabmjt3rvLy8vTRRx9p2LBhysjIcPyFXACANiwYE8xdeeWVfhM/1tfXm8TERJOfn3/eWo/H4/tqZhYWFhaW1rt4PJ5z/r4P+BXQyZMntXv3br8vvAoNDVV6erqKiooaja+rq5PX6/VbAABtX8AD6MiRI6qvr1dcXJzf+ri4OJWXlzcan5+fL7fb7Vt4Ag4A2gfrT8Hl5ubK4/H4loMHD9puCQBwCQT874B69OihsLAwVVRU+K2vqKhQfHx8o/Eul0sulyvQbQAAWriAXwGFh4dr+PDh2rRpk29dQ0ODNm3apFGjRgV6dwCAViooMyHMnTtX2dnZGjFihK688kotXrxYNTU1mjp1ajB2BwBohYISQLfddpsqKyv12GOPqby8XFdccYU2bNjQ6MEEAED71eK+D8jr9crtdttuA0GSl5fnuGb+/PlB6ARAsHk8HkVFRZ11u/Wn4AAA7RMBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArGAyUlxSx48fd1yTlZXluKagoMBxDYDAYjJSAECLRAABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBXMho1Lqjmn27Zt2xzX/PSnP3VcI0nFxcXNqgPQGLNhAwBaJAIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBY0cF2A2i9VqxY4bimurracc0DDzzguIZJRYGWjysgAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCyUjRbPfff7/jmttuu81xTUJCguMaAC0fV0AAACsIIACAFQEPoMcff1whISF+S2pqaqB3AwBo5YJyD2jQoEHauHHj/++kA7eaAAD+gpIMHTp0UHx8fDDeGgDQRgTlHtD+/fuVmJioPn366Cc/+YkOHDhw1rF1dXXyer1+CwCg7Qt4AKWlpWn58uXasGGDli5dqrKyMl199dWqrq5ucnx+fr7cbrdvSUpKCnRLAIAWKMQYY4K5g6qqKiUnJ+uZZ57RtGnTGm2vq6tTXV2d77XX6yWEWonExETHNQcPHnRcc8sttziuWbNmjeMaAIHl8XgUFRV11u1BfzqgW7duuvzyy1VSUtLkdpfLJZfLFew2AAAtTND/DujYsWMqLS3lr9kBAH4CHkD333+/CgsL9be//U3bt2/XpEmTFBYWpjvuuCPQuwIAtGIB/wjuiy++0B133KGjR48qJiZGV111lXbs2KGYmJhA7woA0IoFPIDeeOONQL8lgqy5N+x/8IMfXLJ9AWh7mAsOAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwI+hfSoeUrLi5uVt2Pf/xjxzWPPPKI4xomMAXaJq6AAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAWzYUO1tbWXbF+vvvqq45oXX3zRcY3b7XZcA+DS4goIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwIMcYY2018m9frZSLJixAdHe24prKyMgidNO3kyZOOa0aMGOG45uc//7njGkmaMWNGs+ramp49ezquWb58ueOaJ5980nHN1q1bHdc0NDQ4rsHF83g8ioqKOut2roAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwIoOthtA+7JlyxbHNX/9618d17T0SUVDQkIc1zz44IOOa1577TXHNZL08MMPO6657rrrHNeMGTPGcc2nn37quOb73/++4xoEH1dAAAArCCAAgBWOA2jr1q2aMGGCEhMTFRISorVr1/ptN8boscceU0JCgiIiIpSenq79+/cHql8AQBvhOIBqamo0bNgwLVmypMntixYt0vPPP6+XXnpJO3fuVJcuXZSRkaETJ05cdLMAgLbD8UMIWVlZysrKanKbMUaLFy/WI488optuuknS6ZugcXFxWrt2rW6//faL6xYA0GYE9B5QWVmZysvLlZ6e7lvndruVlpamoqKiJmvq6urk9Xr9FgBA2xfQACovL5ckxcXF+a2Pi4vzbfuu/Px8ud1u35KUlBTIlgAALZT1p+Byc3Pl8Xh8y8GDB223BAC4BAIaQPHx8ZKkiooKv/UVFRW+bd/lcrkUFRXltwAA2r6ABlBKSori4+O1adMm3zqv16udO3dq1KhRgdwVAKCVc/wU3LFjx1RSUuJ7XVZWpr179yo6Olq9evXS7Nmz9a//+q/q37+/UlJS9OijjyoxMVETJ04MZN8AgFbOcQB9+OGHuvbaa32v586dK0nKzs7W8uXL9eCDD6qmpkYzZsxQVVWVrrrqKm3YsEGdOnUKXNcAgFYvxBhjbDfxbV6vV26323YbLUJ0dLTjmsrKyiB0Ejg7d+50XHP99dc7rqmpqXFccynNnDnTcc1zzz0XhE6a1pxfC82ZYLU56uvrHdds3LixWfv60Y9+1Kw6nObxeM55X9/6U3AAgPaJAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKxx/HQMuHY/H47jmj3/8o+OayZMnO65pLpfL5bimOTNbh4Y27/+t0tPTHdf85S9/cVwzYsQIxzWX0tSpUx3XvPzyy45rOnRw/isoLCzMcc1LL73kuAbBxxUQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFjBZKQtWH19veOaqqqqwDcSQFdccYXjmjvuuMNxzfTp0x3XSNLYsWMd13zxxReOa9xut+OaTZs2Oa659dZbHddIzTuP/v3f/91xzfr16x3XjB8/3nHNww8/7LhGkt555x3HNQ0NDc3aV3vEFRAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWMFkpC3YP/3TPzmumTZtWhA6CZxXXnnFcc1//Md/BKGTphUWFjquGT58uOOaKVOmOK7Zv3+/45pLOTntggULHNc0Z2LR5njqqaeaVcfEosHFFRAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWMFkpC3Y9u3bbbdwTj/72c8c1/z61792XOPxeBzXPP74445rJCk+Pt5xTWlpabP21dasXr3acU1zJs9tzn+jBx54wHGNJG3YsMFxzfHjx5u1r/aIKyAAgBUEEADACscBtHXrVk2YMEGJiYkKCQnR2rVr/bZPmTJFISEhfktmZmag+gUAtBGOA6impkbDhg3TkiVLzjomMzNThw8f9i0rV668qCYBAG2P44cQsrKylJWVdc4xLperWTcKAQDtR1DuARUUFCg2NlYDBgzQvffeq6NHj551bF1dnbxer98CAGj7Ah5AmZmZeu2117Rp0yb98pe/VGFhobKyslRfX9/k+Pz8fLndbt+SlJQU6JYAAC1QwP8O6Pbbb/f9e8iQIRo6dKj69u2rgoICjRs3rtH43NxczZ071/fa6/USQgDQDgT9Mew+ffqoR48eKikpaXK7y+VSVFSU3wIAaPuCHkBffPGFjh49qoSEhGDvCgDQijj+CO7YsWN+VzNlZWXau3evoqOjFR0drfnz52vy5MmKj49XaWmpHnzwQfXr108ZGRkBbRwA0Lo5DqAPP/xQ1157re/1mfs32dnZWrp0qfbt26ff//73qqqqUmJiosaPH68nnnhCLpcrcF0DAFq9EGOMsd3Et3m9XrndbtttBFyvXr0c1+zdu9dxzbkeeT+bPn36OK6RpGXLljmu+fYDJxeKR/NbhxdeeMFxzX333ReETgJn8uTJjmu+OztMe+bxeM55X5+54AAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFs2G3YFdccYXjmgMHDjiuGTNmjOMaiVl/4W/AgAGOa959913HNb1793Zcs337dsc1knT11Vc3qw6nMRs2AKBFIoAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVTEYKwJo5c+Y4rvnVr34VhE6adq6JNM+mpqYmCJ20TkxGCgBokQggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBZORAgiI+Ph4xzVffvllEDpprLa2tll1zz//vOOahx9+uFn7aouYjBQA0CIRQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAomIwVgzV//+lfHNampqUHopGndu3d3XFNVVRX4RlopJiMFALRIBBAAwApHAZSfn6+RI0cqMjJSsbGxmjhxooqLi/3GnDhxQjk5Oerevbu6du2qyZMnq6KiIqBNAwBaP0cBVFhYqJycHO3YsUPvv/++Tp06pfHjx6umpsY3Zs6cOXr77be1atUqFRYW6tChQ7r55psD3jgAoHW7qIcQKisrFRsbq8LCQo0ZM0Yej0cxMTFasWKFbrnlFknSZ599pu9973sqKirSD3/4w/O+Jw8hAO0HDyG0bUF9CMHj8UiSoqOjJUm7d+/WqVOnlJ6e7huTmpqqXr16qaioqMn3qKurk9fr9VsAAG1fswOooaFBs2fP1ujRozV48GBJUnl5ucLDw9WtWze/sXFxcSovL2/yffLz8+V2u31LUlJSc1sCALQizQ6gnJwcffLJJ3rjjTcuqoHc3Fx5PB7fcvDgwYt6PwBA69ChOUUzZ87UO++8o61bt+qyyy7zrY+Pj9fJkydVVVXldxVUUVGh+Pj4Jt/L5XLJ5XI1pw0AQCvm6ArIGKOZM2dqzZo12rx5s1JSUvy2Dx8+XB07dtSmTZt864qLi3XgwAGNGjUqMB0DANoER1dAOTk5WrFihdatW6fIyEjffR23262IiAi53W5NmzZNc+fOVXR0tKKiojRr1iyNGjXqgp6AAwC0H44CaOnSpZKka665xm/9smXLNGXKFEnSs88+q9DQUE2ePFl1dXXKyMjQr3/964A0CwBoO5iMFEBAxMXFOa4ZOHCg45qNGzc6rmmuhQsXOq559NFHg9BJ68RkpACAFokAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArmvWNqADwXRUVFY5rtm3bFoROGlu0aFGz6pjZOri4AgIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAK5iMFIA1e/fudVzTp08fxzU33nij4xpJys3NbVYdLgxXQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBZORAgiI6OhoxzVpaWlB6KSxgQMHXpL9wBmugAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACiYjBRAQX3/9teOaXbt2Oa6ZNGmS45qwsDDHNQg+roAAAFYQQAAAKxwFUH5+vkaOHKnIyEjFxsZq4sSJKi4u9htzzTXXKCQkxG+55557Ato0AKD1cxRAhYWFysnJ0Y4dO/T+++/r1KlTGj9+vGpqavzGTZ8+XYcPH/YtixYtCmjTAIDWz9FDCBs2bPB7vXz5csXGxmr37t0aM2aMb33nzp0VHx8fmA4BAG3SRd0D8ng8khp/Fe/rr7+uHj16aPDgwcrNzVVtbe1Z36Ourk5er9dvAQC0fc1+DLuhoUGzZ8/W6NGjNXjwYN/6O++8U8nJyUpMTNS+ffv00EMPqbi4WG+99VaT75Ofn6/58+c3tw0AQCsVYowxzSm89957tX79em3btk2XXXbZWcdt3rxZ48aNU0lJifr27dtoe11dnerq6nyvvV6vkpKSmtMSgFZm9erVjmv4O6DWw+PxKCoq6qzbm3UFNHPmTL3zzjvaunXrOcNHktLS0iTprAHkcrnkcrma0wYAoBVzFEDGGM2aNUtr1qxRQUGBUlJSzluzd+9eSVJCQkKzGgQAtE2OAignJ0crVqzQunXrFBkZqfLyckmS2+1WRESESktLtWLFCv3oRz9S9+7dtW/fPs2ZM0djxozR0KFDg/IDAABaJ0cBtHTpUkmn/9j025YtW6YpU6YoPDxcGzdu1OLFi1VTU6OkpCRNnjxZjzzySMAaBgC0DY4/gjuXpKQkFRYWXlRDAID2gdmwAVjz9NNPO64pKSkJQiewgclIAQBWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMCKZn8ld7B4vV653W7bbQAALtL5vpKbKyAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGBFiwugFjY1HQCgmc73+7zFBVB1dbXtFgAAAXC+3+ctbjbshoYGHTp0SJGRkQoJCfHb5vV6lZSUpIMHD55zhtW2juNwGsfhNI7DaRyH01rCcTDGqLq6WomJiQoNPft1TodL2NMFCQ0N1WWXXXbOMVFRUe36BDuD43Aax+E0jsNpHIfTbB+HC/lanRb3ERwAoH0ggAAAVrSqAHK5XMrLy5PL5bLdilUch9M4DqdxHE7jOJzWmo5Di3sIAQDQPrSqKyAAQNtBAAEArCCAAABWEEAAACsIIACAFa0mgJYsWaLevXurU6dOSktL065du2y3dMk9/vjjCgkJ8VtSU1NttxV0W7du1YQJE5SYmKiQkBCtXbvWb7sxRo899pgSEhIUERGh9PR07d+/306zQXS+4zBlypRG50dmZqadZoMkPz9fI0eOVGRkpGJjYzVx4kQVFxf7jTlx4oRycnLUvXt3de3aVZMnT1ZFRYWljoPjQo7DNddc0+h8uOeeeyx13LRWEUBvvvmm5s6dq7y8PH300UcaNmyYMjIy9NVXX9lu7ZIbNGiQDh8+7Fu2bdtmu6Wgq6mp0bBhw7RkyZImty9atEjPP/+8XnrpJe3cuVNdunRRRkaGTpw4cYk7Da7zHQdJyszM9Ds/Vq5ceQk7DL7CwkLl5ORox44dev/993Xq1CmNHz9eNTU1vjFz5szR22+/rVWrVqmwsFCHDh3SzTffbLHrwLuQ4yBJ06dP9zsfFi1aZKnjszCtwJVXXmlycnJ8r+vr601iYqLJz8+32NWll5eXZ4YNG2a7DaskmTVr1vheNzQ0mPj4ePP000/71lVVVRmXy2VWrlxpocNL47vHwRhjsrOzzU033WSlH1u++uorI8kUFhYaY07/t+/YsaNZtWqVb8x///d/G0mmqKjIVptB993jYIwxY8eONT//+c/tNXUBWvwV0MmTJ7V7926lp6f71oWGhio9PV1FRUUWO7Nj//79SkxMVJ8+ffSTn/xEBw4csN2SVWVlZSovL/c7P9xut9LS0trl+VFQUKDY2FgNGDBA9957r44ePWq7paDyeDySpOjoaEnS7t27derUKb/zITU1Vb169WrT58N3j8MZr7/+unr06KHBgwcrNzdXtbW1Nto7qxY3G/Z3HTlyRPX19YqLi/NbHxcXp88++8xSV3akpaVp+fLlGjBggA4fPqz58+fr6quv1ieffKLIyEjb7VlRXl4uSU2eH2e2tReZmZm6+eablZKSotLSUs2bN09ZWVkqKipSWFiY7fYCrqGhQbNnz9bo0aM1ePBgSafPh/DwcHXr1s1vbFs+H5o6DpJ05513Kjk5WYmJidq3b58eeughFRcX66233rLYrb8WH0D4f1lZWb5/Dx06VGlpaUpOTtYf/vAHTZs2zWJnaAluv/1237+HDBmioUOHqm/fviooKNC4ceMsdhYcOTk5+uSTT9rFfdBzOdtxmDFjhu/fQ4YMUUJCgsaNG6fS0lL17dv3UrfZpBb/EVyPHj0UFhbW6CmWiooKxcfHW+qqZejWrZsuv/xylZSU2G7FmjPnAOdHY3369FGPHj3a5Pkxc+ZMvfPOO9qyZYvf94fFx8fr5MmTqqqq8hvfVs+Hsx2HpqSlpUlSizofWnwAhYeHa/jw4dq0aZNvXUNDgzZt2qRRo0ZZ7My+Y8eOqbS0VAkJCbZbsSYlJUXx8fF+54fX69XOnTvb/fnxxRdf6OjRo23q/DDGaObMmVqzZo02b96slJQUv+3Dhw9Xx44d/c6H4uJiHThwoE2dD+c7Dk3Zu3evJLWs88H2UxAX4o033jAul8ssX77cfPrpp2bGjBmmW7dupry83HZrl9QvfvELU1BQYMrKyswHH3xg0tPTTY8ePcxXX31lu7Wgqq6uNnv27DF79uwxkswzzzxj9uzZY/7+978bY4x56qmnTLdu3cy6devMvn37zE033WRSUlLM8ePHLXceWOc6DtXV1eb+++83RUVFpqyszGzcuNH84Ac/MP379zcnTpyw3XrA3HvvvcbtdpuCggJz+PBh31JbW+sbc88995hevXqZzZs3mw8//NCMGjXKjBo1ymLXgXe+41BSUmIWLFhgPvzwQ1NWVmbWrVtn+vTpY8aMGWO5c3+tIoCMMeaFF14wvXr1MuHh4ebKK680O3bssN3SJXfbbbeZhIQEEx4ebnr27Gluu+02U1JSYrutoNuyZYuR1GjJzs42xpx+FPvRRx81cXFxxuVymXHjxpni4mK7TQfBuY5DbW2tGT9+vImJiTEdO3Y0ycnJZvr06W3uf9Ka+vklmWXLlvnGHD9+3Nx3333mH/7hH0znzp3NpEmTzOHDh+01HQTnOw4HDhwwY8aMMdHR0cblcpl+/fqZBx54wHg8HruNfwffBwQAsKLF3wMCALRNBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgxf8BPOWRUlBlNXEAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# random rotation between 0 an 90 degrees\n",
        "rotation = transforms.RandomRotation(90)\n",
        "rotated = rotation(image_as_tensor)\n",
        "\n",
        "def display_image(image, label):\n",
        "    # permute required to transform back in the PIL format\n",
        "    plt.imshow(image.permute(1, 2, 0), cmap='gray')\n",
        "    plt.title(f\"Sample label: {label}\")\n",
        "    plt.show()\n",
        "\n",
        "display_image(rotated, label)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We can also include the transforms in the dataset constructor, and then they will be applied to each sample upon loading.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Y7XmxStFBFZW",
        "outputId": "92cc95c3-58c7-405b-eca1-acb0a4e55266"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Sample type (as python object)\n",
            "<class 'torch.Tensor'>\n",
            "Sample dtype (of the elements)\n",
            "torch.float32\n",
            "Sample shape\n",
            "torch.Size([1, 28, 28])\n"
          ]
        }
      ],
      "source": [
        "tensor_mnist_train = datasets.MNIST(data_path,\n",
        "                                    train=True,\n",
        "                                    download=False,\n",
        "                                    transform=transforms.ToTensor())\n",
        "\n",
        "sample, label = tensor_mnist_train[0]\n",
        "\n",
        "print(\"Sample type (as python object)\")\n",
        "print(type(sample))\n",
        "\n",
        "print(\"Sample dtype (of the elements)\")\n",
        "print(sample.dtype)\n",
        "\n",
        "print(\"Sample shape\")\n",
        "print(sample.shape)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Also, note that  the rescaling automatic if the inputs are recognized as images. This means that the images are automatically converted to Tensor and the pixels are divided by 255 (as [0, 255] is the range of values for the pixels). In this way, the values will become floating points in [0,1].\n",
        "\n",
        "Reference:\n",
        "- https://pytorch.org/vision/stable/generated/torchvision.transforms.ToTensor.html"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ceECVslBBvG5",
        "outputId": "ba012e68-1ced-4771-cefd-039c83bd9ad0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "image min: 0, image max: 255, dtype: uint8\n",
            "tensor min: 0.0, tensor max: 1.0, dtype: torch.float32\n"
          ]
        }
      ],
      "source": [
        "import numpy as np\n",
        "print(f\"image min: {np.array(image).min()}, image max: {np.array(image).max()}, dtype: {np.array(image).dtype}\")\n",
        "print(f\"tensor min: {sample.min()}, tensor max: {sample.max()}, dtype: {sample.dtype}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If we want to normalize the data with zero mean and standard deviation equal to 1, we have to compute these values on the training dataset and then apply the same transformation to the testing data. \n",
        "Here is the code to compute the mean and standard deviations for all features of the MNIST dataset. Note that when images contain three channels (RGB images), the values of mean and standard deviation are computed for each channel."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Gp23ub73c9M9",
        "outputId": "95d1f958-ff51-47e7-ad35-d742951bd15d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "dataset shape:\n",
            "torch.Size([1, 28, 28, 60000])\n",
            "means: tensor([0.1307])\n",
            "stds: tensor([0.3081])\n"
          ]
        }
      ],
      "source": [
        "import torch\n",
        "dataset = torch.stack([sample for sample, _ in tensor_mnist_train], dim=3)\n",
        "\n",
        "print(\"dataset shape:\")\n",
        "print(dataset.shape)\n",
        "\n",
        "means = dataset.view(1, -1).mean(dim=1)\n",
        "stds = dataset.view(1, -1).std(dim=1)\n",
        "\n",
        "print(f\"means: {means}\")\n",
        "print(f\"stds: {stds}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we can initialize the transform with the correct values (and we can make sure they are correct after transformation). If we want to apply more than one transformation, we can use the `torchvision.transforms.Compose` to chain the transformations (which will be applied in sequence)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PdPQAvd1dg9S",
        "outputId": "c978dcbd-4db7-4fc3-be41-231e395c77a1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "mean: tensor([4.2517e-08])\n",
            "std: tensor([1.0000])\n"
          ]
        }
      ],
      "source": [
        "normalize = transforms.Normalize(means, stds)\n",
        "transformed_mnist_train = datasets.MNIST(data_path, train=True,\n",
        "                                         download=False,\n",
        "                                         transform=transforms.Compose([\n",
        "                                         transforms.ToTensor(),\n",
        "                                         normalize,]))\n",
        "\n",
        "dataset = torch.stack([sample for sample, _ in transformed_mnist_train], dim=3)\n",
        "\n",
        "# let's verify that the mean and standard deviation are now as we want them\n",
        "means = dataset.view(1, -1).mean(dim=1)\n",
        "stds = dataset.view(1, -1).std(dim=1)\n",
        "print(f\"mean: {means}\")\n",
        "print(f\"std: {stds}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Once we have our data normalized, we can create a model to classify the images.\n",
        "\n",
        "Here are some important details:\n",
        "- In the first layer, we have 784 features, while our images are 28x28. We first have to flatten the input (and take care that the batch size stays the same)\n",
        "- Note that the model can (and should) process data in batches, thus the code should be independent from the batch size. This is why in the first step of the transform we have a -1 as the first argument of the `view` method. This -1 means that we leave the first dimension as it is, and we reshape the others to fit the new shape.\n",
        "- The number of outputs is equal to the number of classes, as we want one node for each of the classes. We will pick the \"winner\" as the top score in the output of the model.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Q-BAOS89hThL",
        "outputId": "5c87739c-08c4-4494-c5ee-156123646138"
      },
      "outputs": [],
      "source": [
        "class MNISTModel(torch.nn.Module):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.fc1 = torch.nn.Linear(784, 512)\n",
        "        self.fc2 = torch.nn.Linear(512, 10)\n",
        "        self.relu = torch.nn.ReLU()\n",
        "\n",
        "    def forward(self, x):\n",
        "        # we have to flatten the samples that are 28x28\n",
        "        x = x.view(-1, 784)\n",
        "        x = self.relu(self.fc1(x))\n",
        "        x = self.fc2(x)\n",
        "        return x\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Then, we can create our model and predict the label for one sample. Note that the model is not yet trained, so the output will be basically random. Also, the output has shape (n_samples, n_classes). To get the prediction for each sample (in this case it's only one, but let's make it in a way that will work also for batches), we can use the `tensor.argmax()` method. If we specify the argument `dim=1` we can take the argument of the maximum for each row (i.e., the predicted classes, as desired)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "scores: tensor([[ 0.1693,  0.0982,  0.1845, -0.7322, -0.0712, -0.3645,  0.2340,  0.4162,\n",
            "          0.0799,  0.2491]], grad_fn=<AddmmBackward0>)\n",
            "predicted class: 7\n",
            "original label:  5\n"
          ]
        }
      ],
      "source": [
        "\n",
        "net = MNISTModel()\n",
        "sample, label = transformed_mnist_train[0]\n",
        "\n",
        "# prediction from untrained model\n",
        "out = net(sample)\n",
        "print(\"scores:\", out)\n",
        "\n",
        "# predicted class\n",
        "pred = out.argmax(dim=1)\n",
        "print(\"predicted class:\", pred.item())\n",
        "print(\"original label: \", label)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "For loading the data in batches, rather than single samples or entire datasets, we can use the `DataLoader` from `torch.utils.data`. We can specify the batch size of each batch here (i.e., how many images for each batch), and also whether to shuffle the indexes or not."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "H-zsg64UlSf7"
      },
      "outputs": [],
      "source": [
        "transformed_mnist_validation = datasets.MNIST(data_path, train=False,\n",
        "                                              download=False,\n",
        "                                              transform=transforms.Compose([\n",
        "                                              transforms.ToTensor(),\n",
        "                                              normalize,]))  # same normalize as train\n",
        "\n",
        "train_loader = torch.utils.data.DataLoader(transformed_mnist_train,\n",
        "                                           batch_size=64,\n",
        "                                           shuffle=True)\n",
        "val_loader = torch.utils.data.DataLoader(transformed_mnist_validation,\n",
        "                                         batch_size=64,\n",
        "                                         shuffle=False)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Then, we can write our training loop as usual. Compared to our previous example with regression, in this case we are going to use the cross-entropy loss.\n",
        "\n",
        "$$\n",
        "L = - \\sum_{c=1}^{C} y_c log(p_c)\n",
        "$$\n",
        "\n",
        "Where we loop through the classes and compare the prediction $p_c$ with the desired label $y_c$. This is implemented in PyTorch, so we only have to take care of instantiating the loss."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JuWkoFcqmBNH",
        "outputId": "7958470c-223b-46d8-9557-0284395564c1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch: 1, Loss: 0.35720548033714294\n",
            "Epoch: 2, Loss: 0.4035971760749817\n",
            "Epoch: 3, Loss: 0.35455450415611267\n",
            "Epoch: 4, Loss: 0.32169651985168457\n",
            "Epoch: 5, Loss: 0.2930167615413666\n",
            "Epoch: 6, Loss: 0.22188043594360352\n",
            "Epoch: 7, Loss: 0.2525811493396759\n",
            "Epoch: 8, Loss: 0.5798184871673584\n",
            "Epoch: 9, Loss: 0.39627477526664734\n",
            "Epoch: 10, Loss: 0.20599155128002167\n"
          ]
        }
      ],
      "source": [
        "net = MNISTModel()\n",
        "learning_rate = 1e-2\n",
        "optimizer = torch.optim.SGD(net.parameters(), lr=learning_rate)\n",
        "scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[5, 8], gamma=0.5)\n",
        "loss_fn = torch.nn.CrossEntropyLoss()\n",
        "epochs = 10\n",
        "\n",
        "# train\n",
        "for epoch in range(epochs):\n",
        "    for samples, labels in train_loader:\n",
        "        outputs = net(samples)\n",
        "        loss = loss_fn(outputs, labels)\n",
        "\n",
        "        optimizer.zero_grad()\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "        scheduler.step()\n",
        "\n",
        "    print(f\"Epoch: {epoch + 1}, Loss: {loss}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If we want to check how our model behaves, we cannot directly use the loss (it's not that informative). To compute a metric that has immediate meaning for us, we can compute the accuracy. In a multi-class setting, we have to count the number of labels that are equal to the targets, and divide by the total number of predictions. If we load the data in batches, we have to compute the accuracy as incremental."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KT6-lrXknHnp",
        "outputId": "6e69b6fc-01f8-4975-ef6f-840fe25562a0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 0.9348999857902527\n"
          ]
        }
      ],
      "source": [
        "accuracy = 0.0\n",
        "total = 0\n",
        "for samples, labels in val_loader:\n",
        "    outputs = net(samples)\n",
        "    predictions = outputs.argmax(dim=1)\n",
        "    total += samples.shape[0]\n",
        "    accuracy += (predictions.type(labels.dtype) == labels).float().sum()\n",
        "accuracy = accuracy / total\n",
        "\n",
        "print(f\"Accuracy: {accuracy}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Here, finally, let's report an improved training loop. Here we are also tracking the loss across the iterations and evaluating at each epoch. Moreover, we are also adding the instruction to load the data and model to a custom device (e.g., a GPU)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ota6WvslpSke",
        "outputId": "416e3514-b7e4-4218-cd2d-9e8cc722b0c8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "0 0.9156000018119812\n",
            "1 0.9298999905586243\n",
            "2 0.9426000118255615\n",
            "3 0.9469000101089478\n",
            "4 0.9513999819755554\n"
          ]
        }
      ],
      "source": [
        "learning_rate = 1e-2\n",
        "epochs = 5\n",
        "device = torch.device('cpu')\n",
        "net = MNISTModel()\n",
        "net.to(device)\n",
        "optimizer = torch.optim.SGD(net.parameters(), lr=learning_rate)\n",
        "scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer)\n",
        "loss_fn = torch.nn.CrossEntropyLoss()\n",
        "\n",
        "\n",
        "def train_epoch(model, train_loader, optimizer, scheduler, loss_fn):\n",
        "    train_loss = 0.0\n",
        "    total = 0\n",
        "    for samples, labels in train_loader:\n",
        "        samples, labels = samples.to(device), labels.to(device)\n",
        "        outputs = model(samples)\n",
        "        loss = loss_fn(outputs, labels)\n",
        "        optimizer.zero_grad()\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "        total += samples.shape[0]\n",
        "        train_loss += loss.sum().detach()\n",
        "    train_loss /= total\n",
        "    scheduler.step(train_loss)\n",
        "    return train_loss.item()\n",
        "\n",
        "def valid_epoch(model, val_loader, loss_fn):\n",
        "    accuracy = 0.0\n",
        "    validation_loss = 0.0\n",
        "    total = 0\n",
        "    for samples, labels in val_loader:\n",
        "        samples, labels = samples.to(device), labels.to(device)\n",
        "        outputs = net(samples)\n",
        "        loss = loss_fn(outputs, labels)\n",
        "        predictions = outputs.argmax(dim=1)\n",
        "        accuracy += (predictions.type(labels.dtype) == labels).float().sum()\n",
        "        total += samples.shape[0]\n",
        "        validation_loss += loss.sum()\n",
        "    validation_loss /= total\n",
        "    accuracy = accuracy / total\n",
        "    return validation_loss.item(), accuracy.item()\n",
        "\n",
        "train_losses, val_losses = [], []\n",
        "for epoch in range(epochs):\n",
        "    train_loss = train_epoch(net, train_loader, optimizer, scheduler, loss_fn)\n",
        "    val_loss, accuracy = valid_epoch(net, val_loader, loss_fn)\n",
        "    train_losses.append(train_loss)\n",
        "    val_losses.append(val_loss)\n",
        "    print(epoch, accuracy)\n",
        "\n",
        "# save the model\n",
        "torch.save(net.state_dict(), \"model.pth\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Since we tracked the loss, we can also visualize it in a plot. Let's plot the training and validation loss. \n",
        "Observe that if the validation loss starts increasing (while the training loss keeps becoming smaller), this can be indication of overfitting."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 488
        },
        "id": "1cGQOFG3r7aC",
        "outputId": "e7910d0a-058a-437c-de0c-8993a64d6e01"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<matplotlib.legend.Legend at 0x167ce2380>"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkgAAAGwCAYAAABSN5pGAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABhq0lEQVR4nO3deVhUZf8G8HtmWIZ9lR0FBXcFRUTQcomiMouyXFrU0ixTy9B81bfUll+Ybb6mqfX2qi2mGWmmRhmuKYKCmgouIAiow6KyCLLNnN8fAwODgIAMZwbuz3WdCznnmZnvYaC5e85znkciCIIAIiIiItKQil0AERERkb5hQCIiIiKqgwGJiIiIqA4GJCIiIqI6GJCIiIiI6mBAIiIiIqqDAYmIiIioDiOxCzBUKpUKV69ehZWVFSQSidjlEBERURMIgoCioiK4ublBKm24n4gBqYWuXr0KT09PscsgIiKiFsjMzISHh0eDxxmQWsjKygqA+gdsbW0tcjVERETUFIWFhfD09NR8jjeEAamFqi+rWVtbMyAREREZmLsNj+EgbSIiIqI6GJCIiIiI6mBAIiIiIqqDY5CIiEgvKJVKVFRUiF0GGThjY2PIZLJ7fh4GJCIiEpUgCFAoFMjPzxe7FGonbG1t4eLick/zFDIgERGRqKrDkZOTE8zNzTn5LrWYIAgoKSlBTk4OAMDV1bXFz8WAREREolEqlZpw5ODgIHY51A6YmZkBAHJycuDk5NTiy20cpE1ERKKpHnNkbm4uciXUnlT/Pt3LmDYGJCIiEh0vq1Frao3fJwYkIiIiojoYkIiIiIjqYEAiIiLSA15eXlixYoXoz0FqvItNz6hUAg6l5OF+X0dekyci0mMjRoyAv79/qwWSY8eOwcLColWei+4de5D0iEol4MkvD2Py/+Jx6GKe2OUQEdE9EgQBlZWVTWrbqVMn3s2nRxiQ9IhUKsEgL3sAwKq9KSJXQ0QkDkEQUFJe2eabIAhNrnHKlCk4cOAA/vOf/0AikUAikSA9PR379++HRCLB77//joCAAJiamuLvv/9GamoqnnjiCTg7O8PS0hKBgYH466+/tJ6z7uUxiUSC//73v3jyySdhbm4OX19f7Nixo1k/y4yMDDzxxBOwtLSEtbU1xo0bh+zsbM3xU6dOYeTIkbCysoK1tTUCAgJw/PhxAMDly5cxZswY2NnZwcLCAn369MHu3bub9fqGjJfY9Mz0+7viu9jLiE+/gbhL1xHUlROnEVHHcrtCid6L/2jz1016LwzmJk37WPzPf/6DCxcuoG/fvnjvvfcAqHuA0tPTAQALFizAJ598gq5du8LOzg6ZmZl49NFH8X//938wNTXFt99+izFjxuD8+fPo3Llzg6/z7rvvYvny5fj444/xxRdf4LnnnsPly5dhb29/1xpVKpUmHB04cACVlZWYOXMmxo8fj/379wMAnnvuOQwYMABr1qyBTCbDyZMnYWxsDACYOXMmysvLcfDgQVhYWCApKQmWlpZN+vm0BwxIesbZWo5nBnngh7gMrNqXwoBERKSHbGxsYGJiAnNzc7i4uNxx/L333sODDz6o+d7e3h5+fn6a799//31s27YNO3bswKxZsxp8nSlTpmDixIkAgA8//BArV65EfHw8Hn744bvWGBMTg9OnTyMtLQ2enp4AgG+//RZ9+vTBsWPHEBgYiIyMDLz11lvo2bMnAMDX11fz+IyMDIwdOxb9+vUDAHTt2vWur9meMCDpoVeHd8PmY5k4dDEPpzLz4edpK3ZJRERtxsxYhqT3wkR53dYyaNAgre9v3bqFpUuXYteuXbh27RoqKytx+/ZtZGRkNPo8/fv31/zbwsIC1tbWmnXG7iY5ORmenp6acAQAvXv3hq2tLZKTkxEYGIiIiAhMmzYN3333HUJDQ/HMM8+gW7duAIDXX38dM2bMwJ9//onQ0FCMHTtWq572jmOQ9JCnvTnC/d0BAKv2cSwSEXUsEokE5iZGbb615p3Dde9GmzdvHrZt24YPP/wQhw4dwsmTJ9GvXz+Ul5c3+jzVl7tq/2xUKlWr1bl06VKcPXsWo0ePxt69e9G7d29s27YNADBt2jRcunQJL7zwAk6fPo1Bgwbhiy++aLXX1ncMSHrqtZHdIJEAe5KycU5RKHY5RERUh4mJCZRKZZPaHj58GFOmTMGTTz6Jfv36wcXFRTNeSVd69eqFzMxMZGZmavYlJSUhPz8fvXv31uzr3r073nzzTfz555946qmnsH79es0xT09PvPrqq/jll18wd+5cfP311zqtWZ8wIOmpbp0s8Wg/VwDA6n2pIldDRER1eXl5IS4uDunp6cjLy2u0Z8fX1xe//PILTp48iVOnTuHZZ59t1Z6g+oSGhqJfv3547rnnkJiYiPj4eEyaNAnDhw/HoEGDcPv2bcyaNQv79+/H5cuXcfjwYRw7dgy9evUCAMyZMwd//PEH0tLSkJiYiH379mmOdQQMSHps5ggfAMDOf67iUu4tkashIqLa5s2bB5lMht69e6NTp06Njif67LPPYGdnh5CQEIwZMwZhYWEYOHCgTuuTSCT49ddfYWdnh/vvvx+hoaHo2rUrtmzZAgCQyWS4fv06Jk2ahO7du2PcuHF45JFH8O677wIAlEolZs6ciV69euHhhx9G9+7d8eWXX+q0Zn0iEZoz8QNpFBYWwsbGBgUFBbC2ttbZ60zbeAx/JefgmQAPfPyM390fQERkQEpLS5GWlgZvb2/I5XKxy6F2orHfq6Z+frMHSc/NHKnuRdp24goyb5SIXA0REVHHwICk5wZ0tsMwH0dUqgSsO8ixSERERG2BAckAzBql7kX66XgWsgtLRa6GiIio/WNAMgBB3vYY1MUO5ZUqfH3wktjlEBERtXsMSAZAIpFoepF+iMvAjeLGJxYjIiKie8OAZCCGd++Efu42uF2hxP/+ThO7HCIionaNAclASCQSzR1tG4+ko+B2hcgVERERtV8MSAbkod7O6O5siaKySnwXmy52OURERO0WA5IBkUprepG++TsNJeWVIldERET3wsvLCytWrNB8L5FIsH379gbbp6enQyKR4OTJk/f0uq31PHczZcoUhIeH6/Q1dIUBycCM7ucKLwdz3CypwKa4hqe1JyIiw3Pt2jU88sgjrfqc9YUUT09PXLt2DX379m3V12pPGJAMjJFMihkjugEA1h28hNKKpq0kTURE+s/FxQWmpqY6fx2ZTAYXFxcYGRnp/LUMFQOSAXpygAfcbOTILSrD1oQsscshIupwvvrqK7i5uUGlUmntf+KJJ/DSSy8BAFJTU/HEE0/A2dkZlpaWCAwMxF9//dXo89a9xBYfH48BAwZALpdj0KBBOHHihFZ7pVKJqVOnwtvbG2ZmZujRowf+85//aI4vXboUGzduxK+//gqJRAKJRIL9+/fXe4ntwIEDGDx4MExNTeHq6ooFCxagsrJmKMeIESPw+uuvY/78+bC3t4eLiwuWLl3arJ9bWVkZXn/9dTg5OUEul2PYsGE4duyY5vjNmzfx3HPPoVOnTjAzM4Ovry/Wr18PACgvL8esWbPg6uoKuVyOLl26IDIyslmv3xwMSAbIxEiKV4are5HW7k9FhVJ1l0cQERkQQQDKi9t+a8ba7c888wyuX7+Offv2afbduHED0dHReO655wAAt27dwqOPPoqYmBicOHECDz/8MMaMGYOMjKYNj7h16xYee+wx9O7dGwkJCVi6dCnmzZun1UalUsHDwwNbt25FUlISFi9ejEWLFuGnn34CAMybNw/jxo3Dww8/jGvXruHatWsICQm547WuXLmCRx99FIGBgTh16hTWrFmDb775Bh988IFWu40bN8LCwgJxcXFYvnw53nvvPezZs6fJP7f58+cjKioKGzduRGJiInx8fBAWFoYbN24AAN555x0kJSXh999/R3JyMtasWQNHR0cAwMqVK7Fjxw789NNPOH/+PH744Qd4eXk1+bWbi31rBmp8oCe+2JuCK/m3sf3EFTwzyFPskoiIWkdFCfChW9u/7qKrgIlFk5ra2dnhkUcewaZNm/DAAw8AAH7++Wc4Ojpi5MiRAAA/Pz/4+flpHvP+++9j27Zt2LFjB2bNmnXX19i0aRNUKhW++eYbyOVy9OnTB1lZWZgxY4amjbGxMd59913N997e3oiNjcVPP/2EcePGwdLSEmZmZigrK4OLi0uDr/Xll1/C09MTq1atgkQiQc+ePXH16lX861//wuLFiyGVqvtT+vfvjyVLlgAAfH19sWrVKsTExODBBx+86/kUFxdjzZo12LBhg2ac1ddff409e/bgm2++wVtvvYWMjAwMGDAAgwYNAgCtAJSRkQFfX18MGzYMEokEXbp0uetr3gv2IBkoubEML9/nDQD4cn8qlKqm/58PERHdu+eeew5RUVEoKysDAPzwww+YMGGCJkzcunUL8+bNQ69evWBrawtLS0skJyc3uQcpOTkZ/fv3h1wu1+wLDg6+o93q1asREBCATp06wdLSEl999VWTX6P2awUHB0MikWj2DR06FLdu3UJWVs1Qjv79+2s9ztXVFTk5OU16jdTUVFRUVGDo0KGafcbGxhg8eDCSk5MBADNmzMDmzZvh7++P+fPn48iRI5q2U6ZMwcmTJ9GjRw+8/vrr+PPPP5t1js3FHiQD9tyQLlhzIBVpecXYffoaxviJ8H9cREStzdhc3Zsjxus2w5gxYyAIAnbt2oXAwEAcOnQIn3/+ueb4vHnzsGfPHnzyySfw8fGBmZkZnn76aZSXt95yUZs3b8a8efPw6aefIjg4GFZWVvj4448RFxfXaq9Rm7Gxsdb3EonkjnFY9+KRRx7B5cuXsXv3buzZswcPPPAAZs6ciU8++QQDBw5EWloafv/9d/z1118YN24cQkND8fPPP7fa69fGHiQDZmlqhJeGqnuRVu9LgYq9SETUHkgk6ktdbb3V6j1pCrlcjqeeego//PADfvzxR/To0QMDBw7UHD98+DCmTJmCJ598Ev369YOLiwvS09Ob/Py9evXCP//8g9LSUs2+o0eParU5fPgwQkJC8Nprr2HAgAHw8fFBamqqVhsTExMolY3f8dyrVy/ExsZCqDUO6/Dhw7CysoKHh0eTa25Mt27dYGJigsOHD2v2VVRU4NixY+jdu7dmX6dOnTB58mR8//33WLFiBb766ivNMWtra4wfPx5ff/01tmzZgqioKM34pdbGgGTgJgd7wdLUCOcURYg517RuTiIiah3PPfccdu3ahf/973+awdnVfH198csvv+DkyZM4deoUnn322Wb1tjz77LOQSCR4+eWXkZSUhN27d+OTTz654zWOHz+OP/74AxcuXMA777yjdVcYoB7H888//+D8+fPIy8tDRcWdS1W99tpryMzMxOzZs3Hu3Dn8+uuvWLJkCSIiIjSXDO+VhYUFZsyYgbfeegvR0dFISkrCyy+/jJKSEkydOhUAsHjxYvz6669ISUnB2bNnsXPnTvTq1QsA8Nlnn+HHH3/EuXPncOHCBWzduhUuLi6wtbVtlfrqYkAycDbmxpgUrB6otmrvRa30T0REujVq1CjY29vj/PnzePbZZ7WOffbZZ7Czs0NISAjGjBmDsLAwrR6mu7G0tMRvv/2G06dPY8CAAfj3v/+Njz76SKvNK6+8gqeeegrjx49HUFAQrl+/jtdee02rzcsvv4wePXpg0KBB6NSpk1YPTjV3d3fs3r0b8fHx8PPzw6uvvoqpU6fi7bffbsZP4+6WLVuGsWPH4oUXXsDAgQORkpKCP/74A3Z2dgDUvV0LFy5E//79cf/990Mmk2Hz5s0AACsrKyxfvhyDBg1CYGAg0tPTsXv37lYLcHVJBH6itkhhYSFsbGxQUFAAa2trUWu5fqsMQz/ai9IKFb59aTDu795J1HqIiJqqtLQUaWlp8Pb21hqMTHQvGvu9aurnN3uQ2gEHS1M8O7iqF2lfisjVEBERGT4GpHZi+v1dYSKTIj7tBuLTdDNgjYiIqKNgQGonXGzkeHqQ+k4D9iIRERHdGwakdmTG8G6QSSU4eCEXpzLzxS6HiIjIYDEgtSOe9uZ4wl89WeRq9iIRkQHh/ULUmlrj90n0gLR69Wp4eXlBLpcjKCgI8fHxjbbfunUrevbsCblcjn79+mH37t1axwVBwOLFi+Hq6gozMzOEhobi4sWLWm0uXLiAJ554Ao6OjrC2tsawYcO0Fhw0ZK+N8IFEAvyZlI1zikKxyyEialT1zMwlJSUiV0LtSfXvU92Zv5tD1KVGtmzZgoiICKxduxZBQUFYsWIFwsLCcP78eTg5Od3R/siRI5g4cSIiIyPx2GOPYdOmTQgPD0diYiL69u0LAFi+fDlWrlyJjRs3wtvbG++88w7CwsKQlJSkudXvscceg6+vL/bu3QszMzOsWLECjz32GFJTUxtdzM8Q+DhZ4tG+rth1+hq+3JeKlRMHiF0SEVGDZDIZbG1tNet5mZuba60HRtQcgiCgpKQEOTk5sLW1hUwma/FziToPUlBQEAIDA7Fq1SoAgEqlgqenJ2bPno0FCxbc0X78+PEoLi7Gzp07NfuGDBkCf39/rF27FoIgwM3NDXPnzsW8efMAAAUFBXB2dsaGDRswYcIE5OXloVOnTjh48CDuu+8+AEBRURGsra2xZ88ehIaGNql2fZoHqa6kq4V4dOUhSCXAXxHD0bWTpdglERE1SBAEKBQK5Ofni10KtRO2trZwcXGpN2w39fNbtB6k8vJyJCQkYOHChZp9UqkUoaGhiI2NrfcxsbGxiIiI0NoXFhaG7du3AwDS0tKgUCi0Qo6NjQ2CgoIQGxuLCRMmwMHBAT169MC3336LgQMHwtTUFOvWrYOTkxMCAgIarLesrEyzYjOg/gHrq95u1nigpxNizuVgzf5UfPyMn9glERE1SCKRwNXVFU5OTvUug0HUHMbGxvfUc1RNtICUl5cHpVIJZ2dnrf3Ozs44d+5cvY9RKBT1tlcoFJrj1fsaaiORSPDXX38hPDwcVlZWkEqlcHJyQnR0tGaq8/pERkbi3Xffbd5JimjmKB/EnMvBthNX8EaoLzzsmrdKNRFRW5PJZK3ywUbUGkQfpN3WBEHAzJkz4eTkhEOHDiE+Ph7h4eEYM2YMrl271uDjFi5ciIKCAs2WmZnZhlU338DOdhjq44BKlYB1By6JXQ4REZFBES0gOTo6QiaTITs7W2t/dnZ2gwOlXVxcGm1f/bWxNnv37sXOnTuxefNmDB06FAMHDsSXX34JMzMzbNy4scF6TU1NYW1trbXpu1kjfQEAW45nIqewVORqiIiIDIdoAcnExAQBAQGIiYnR7FOpVIiJiUFwcHC9jwkODtZqDwB79uzRtPf29oaLi4tWm8LCQsTFxWnaVN/6V3f1X6lUCpVKde8npkeGdLVHQBc7lFeq8PUh9iIRERE1laiX2CIiIvD1119j48aNSE5OxowZM1BcXIwXX3wRADBp0iStQdxvvPEGoqOj8emnn+LcuXNYunQpjh8/jlmzZgFQjy+aM2cOPvjgA+zYsQOnT5/GpEmT4ObmhvDwcADqkGVnZ4fJkyfj1KlTuHDhAt566y2kpaVh9OjRbf4z0CWJRIJZo3wAAD/EZeBGcbnIFRERERkGUedBGj9+PHJzc7F48WIoFAr4+/sjOjpaM8g6IyNDq6cnJCQEmzZtwttvv41FixbB19cX27dv18yBBADz589HcXExpk+fjvz8fAwbNgzR0dGaOZAcHR0RHR2Nf//73xg1ahQqKirQp08f/Prrr/Dza393e43o3gl93a1x5koh1h9Ow9yHeohdEhERkd4TdR4kQ6bP8yDVFX3mGl79PhFWciMcXjAK1vKWzyxKRERkyJr6+d3h7mLriB7q7QJfJ0sUlVbiu9jLYpdDRESk9xiQOgCpVIKZI9Vjkf576BJKyitFroiIiEi/MSB1EI/1d0UXB3PcLKnAprgMscshIiLSawxIHYSRTIoZw7sBAL46eAmlFUqRKyIiItJfDEgdyFMDPeBqI0dOURl+TsgSuxwiIiK9xYDUgZgYSfHK/V0BAGv2p6JC2b4mxiQiImotDEgdzITBneFoaYIr+bfx68mrYpdDRESklxiQOhi5sQwv36fuRfpyXwqUKk6DRUREVBcDUgf03JAusDEzxqW8Yvx+5prY5RAREekdBqQOyNLUCC8N9QYArNqbAhV7kYiIiLQwIHVQU0K8YGlqhHOKIsScyxG7HCIiIr3CgNRB2Zgb44XgLgCAVftSwCX5iIiIajAgdWBTh3lDbizFqcx8/J2SJ3Y5REREeoMBqQNztDTFxMGdAajHIhEREZEaA1IHN/3+rjCRSRGXdgPH0m+IXQ4REZFeYEDq4FxtzDA2wAMAe5GIiIiqMSARZgzvBplUggMXcvFPVr7Y5RAREYmOAYnQ2cEcT/i5AQBW72MvEhEREQMSAQBeG9kNEgnwx9lsnFcUiV0OERGRqBiQCADg42SFR/q6AGAvEhEREQMSacwc6QMA2PnPVaTlFYtcDRERkXgYkEijj5sNRvV0gkoA1uxnLxIREXVcDEikpboX6ZfEK7iSf1vkaoiIiMTBgERaArrYYaiPAypVAtYdSBW7HCIiIlEwINEdqnuRNh/LRE5RqcjVEBERtT0GJLpDcFcHBHSxQ3mlCv89lCZ2OURERG2OAYnuIJFIMKuqF+n7o5dxs7hc5IqIiIjaFgMS1WtEj07o42aNknIl1h9mLxIREXUsDEhUr9q9SOuPpKOwtELkioiIiNoOAxI1KKyPC3ycLFFUWonvYi+LXQ4REVGbYUCiBkmlEswc2Q0A8M3faSgprxS5IiIiorbBgESNGtPfDZ3tzXGjuBw/xmeKXQ4REVGbYECiRhnJpHhthLoX6auDqSirVIpcERERke4xINFdPTXQA642cmQXluHnhCyxyyEiItI5BiS6KxMjKV65vysAYM3+VFQoVSJXREREpFsMSNQkEwZ3hqOlCbJu3saOk1fFLoeIiEinGJCoSeTGMky7T92LtHp/CpQqQeSKiIiIdIcBiZrs+SFdYGNmjEu5xfj9zDWxyyEiItIZBiRqMktTI7w41AsAsGpvCgSBvUhERNQ+MSBRs0wJ8YKFiQznFEWISc4RuxwiIiKdYECiZrE1N8ELwV4AgFX72ItERETtEwMSNdu0+7whN5biZGY+DqdcF7scIiKiVseARM3maGmKCYGdAQCr9l0UuRoiIqLWx4BELfLK8K4wlklw9NINHE+/IXY5RERErUovAtLq1avh5eUFuVyOoKAgxMfHN9p+69at6NmzJ+RyOfr164fdu3drHRcEAYsXL4arqyvMzMwQGhqKixdrejr2798PiURS73bs2DGdnGN742pjhqcDPACoxyIRERG1J6IHpC1btiAiIgJLlixBYmIi/Pz8EBYWhpyc+u+QOnLkCCZOnIipU6fixIkTCA8PR3h4OM6cOaNps3z5cqxcuRJr165FXFwcLCwsEBYWhtLSUgBASEgIrl27prVNmzYN3t7eGDRoUJucd3swY7gPZFIJ9p/PxemsArHLISIiajUSQeTbkIKCghAYGIhVq1YBAFQqFTw9PTF79mwsWLDgjvbjx49HcXExdu7cqdk3ZMgQ+Pv7Y+3atRAEAW5ubpg7dy7mzZsHACgoKICzszM2bNiACRMm3PGcFRUVcHd3x+zZs/HOO+80qe7CwkLY2NigoKAA1tbWLTn1duHNLSex7cQVhPVxxroXGC6JiEi/NfXzW9QepPLyciQkJCA0NFSzTyqVIjQ0FLGxsfU+JjY2Vqs9AISFhWnap6WlQaFQaLWxsbFBUFBQg8+5Y8cOXL9+HS+++GKDtZaVlaGwsFBrI+C1Ed0gkQB/nM3GhewiscshIiJqFaIGpLy8PCiVSjg7O2vtd3Z2hkKhqPcxCoWi0fbVX5vznN988w3CwsLg4eHRYK2RkZGwsbHRbJ6eno2fXAfh62yFR/q6AABWcywSERG1E6KPQRJbVlYW/vjjD0ydOrXRdgsXLkRBQYFmy8zMbKMK9d9rI3wAAL+duor0vGKRqyEiIrp3ogYkR0dHyGQyZGdna+3Pzs6Gi4tLvY9xcXFptH3116Y+5/r16+Hg4IDHH3+80VpNTU1hbW2ttZFaX3cbjOrpBJUArNmfKnY5RERE90zUgGRiYoKAgADExMRo9qlUKsTExCA4OLjexwQHB2u1B4A9e/Zo2nt7e8PFxUWrTWFhIeLi4u54TkEQsH79ekyaNAnGxsatdVod0syR6l6kX05k4Ur+bZGrISIiujeiX2KLiIjA119/jY0bNyI5ORkzZsxAcXGxZsD0pEmTsHDhQk37N954A9HR0fj0009x7tw5LF26FMePH8esWbMAABKJBHPmzMEHH3yAHTt24PTp05g0aRLc3NwQHh6u9dp79+5FWloapk2b1mbn214FdLFDSDcHVCgFfHWAvUhERGTYjMQuYPz48cjNzcXixYuhUCjg7++P6OhozSDrjIwMSKU1OS4kJASbNm3C22+/jUWLFsHX1xfbt29H3759NW3mz5+P4uJiTJ8+Hfn5+Rg2bBiio6Mhl8u1Xvubb75BSEgIevbs2TYn287NGumDI6nXsflYJmaO8oGTlfzuDyIiItJDos+DZKg4D9KdBEHA2DVHkJiRj1fu74qFj/YSuyQiIiItBjEPErUvEokEs0apxyJ9d/QybhaXi1wRERFRyzAgUasa2cMJvV2tUVKuxPoj6WKXQ0RE1CIMSNSqJBIJZlf1Im04nIai0gqRKyIiImo+BiRqdWF9XODjZInC0kp8d/Sy2OUQERE1GwMStTqpVIKZI7sBAL45lIbb5UqRKyIiImoeBiTSiTH93dDZ3hzXi8vxY3yG2OUQERE1CwMS6YSRTIoZI9S9SOsOpqKskr1IRERkOBiQSGeeGugOF2s5sgvLEJVwRexyiIiImowBiXTG1EiGV4Z3BQB8uT8FFUqVyBURERE1DQMS6dSEwM5wsDBB1s3b2HHyqtjlEBERNQkDEumUmYkM0+6r6UVSqriyDRER6T8GJNK554d0ho2ZMVJzixF9RiF2OURERHfFgEQ6ZyU3xpQQLwDAqn0p4PrIRESk7xiQqE28ONQLFiYyJF8rxN5zOWKXQ0RE1CgGJGoTtuYmeD64CwDgi73sRSIiIv3GgERtZtqwrjA1kuJkZj6OpF4XuxwiIqIGMSBRm+lkZYqJgzsDAFbtTRG5GiIiooYxIFGbmn5/VxjLJIi9dB3H02+IXQ4REVG9GJCoTbnZmmHsQA8A6jvaiIiI9BEDErW5GSO6QSoB9p/PxZkrBWKXQ0REdAcGJGpzXRws8IS/OwCORSIiIv3EgESieG1ENwBA9FkFLmQXiVwNERGRNgYkEoWvsxUe6esCAPiSY5GIiEjPMCCRaGaO9AEA7Dh1FZevF4tcDRERUQ0GJBJNX3cbjOzRCSoBWLM/VexyiIiINBiQSFSzRql7kaISs3A1/7bI1RAREakxIJGoArrYI7irAyqUAr46eEnscoiIiAAwIJEeqO5F+jE+A7lFZSJXQ0RExIBEeiCkmwMGdLZFWaUK//2bvUhERCQ+BiQSnUQiweyqXqTvYy8jv6Rc5IqIiKijY0AivTCyhxN6u1qjuFyJ9YfTxS6HiIg6OAYk0gsSiUQzFmn94TQUlVaIXBEREXVkDEikNx7u44JunSxQWFqJ749miF0OERF1YAxIpDekUolmdu3/HrqE2+VKkSsiIqKOigGJ9Mrjfm7wtDfD9eJybD7GXiQiIhIHAxLpFSOZFDOGq3uR1h24hLJK9iIREVHbY0AivTM2wB0u1nIoCksRlXBF7HKIiKgDYkAivWNqJMP0+7sCANYcSEGlUiVyRURE1NEwIJFemji4MxwsTJB54zZ2nLoqdjlERNTBMCCRXjIzkWHqfd4AgNX7UqBSCSJXREREHQkDEumtF4Z0gbXcCKm5xYg+qxC7HCIi6kAYkEhvWcmNMWWouhdp1d4UCAJ7kYiIqG0wIJFeezHECxYmMiRdK8S+8zlil0NERB0EAxLpNTsLEzw/pAsA4Av2IhERURsRPSCtXr0aXl5ekMvlCAoKQnx8fKPtt27dip49e0Iul6Nfv37YvXu31nFBELB48WK4urrCzMwMoaGhuHjx4h3Ps2vXLgQFBcHMzAx2dnYIDw9vzdOiVjT1Pm+YGklxIiMfsanXxS6HiIg6AFED0pYtWxAREYElS5YgMTERfn5+CAsLQ05O/ZdSjhw5gokTJ2Lq1Kk4ceIEwsPDER4ejjNnzmjaLF++HCtXrsTatWsRFxcHCwsLhIWFobS0VNMmKioKL7zwAl588UWcOnUKhw8fxrPPPqvz86WWcbKSY+LgzgDUvUhERES6JhFEvGYRFBSEwMBArFq1CgCgUqng6emJ2bNnY8GCBXe0Hz9+PIqLi7Fz507NviFDhsDf3x9r166FIAhwc3PD3LlzMW/ePABAQUEBnJ2dsWHDBkyYMAGVlZXw8vLCu+++i6lTp7a49sLCQtjY2KCgoADW1tYtfh5qmqv5tzH8432oUAqImhGMgC72YpdEREQGqKmf36L1IJWXlyMhIQGhoaE1xUilCA0NRWxsbL2PiY2N1WoPAGFhYZr2aWlpUCgUWm1sbGwQFBSkaZOYmIgrV65AKpViwIABcHV1xSOPPKLVC1WfsrIyFBYWam3UdtxszTB2oAcA9R1tREREuiRaQMrLy4NSqYSzs7PWfmdnZygU9c95o1AoGm1f/bWxNpcuXQIALF26FG+//TZ27twJOzs7jBgxAjdu3Giw3sjISNjY2Gg2T0/PZpwttYZXh3eDVALsO5+LM1cKxC6HiIjaMdEHabc1lUq9rte///1vjB07FgEBAVi/fj0kEgm2bt3a4OMWLlyIgoICzZaZmdlWJVMVL0cLPO7nBkA9uzYREZGuiBaQHB0dIZPJkJ2drbU/OzsbLi4u9T7GxcWl0fbVXxtr4+rqCgDo3bu35ripqSm6du2KjIyMBus1NTWFtbW11kZt77WRPgCA6LMKXMwuErkaIiJqr0QLSCYmJggICEBMTIxmn0qlQkxMDIKDg+t9THBwsFZ7ANizZ4+mvbe3N1xcXLTaFBYWIi4uTtMmICAApqamOH/+vKZNRUUF0tPT0aVLl1Y7P9KN7s5WeLiPCwQB+HJ/qtjlEBFROyXqJbaIiAh8/fXX2LhxI5KTkzFjxgwUFxfjxRdfBABMmjQJCxcu1LR/4403EB0djU8//RTnzp3D0qVLcfz4ccyaNQsAIJFIMGfOHHzwwQfYsWMHTp8+jUmTJsHNzU0zz5G1tTVeffVVLFmyBH/++SfOnz+PGTNmAACeeeaZtv0BUIvMrOpF2nHqKi5fLxa5GiIiao+MxHzx8ePHIzc3F4sXL4ZCoYC/vz+io6M1g6wzMjIgldZkuJCQEGzatAlvv/02Fi1aBF9fX2zfvh19+/bVtJk/fz6Ki4sxffp05OfnY9iwYYiOjoZcLte0+fjjj2FkZIQXXngBt2/fRlBQEPbu3Qs7O7u2O3lqsX4eNhjRoxP2n8/F2gOpiHyqv9glERFROyPqPEiGjPMgiSvh8g2MXRMLY5kEB94aCTdbM7FLIiIiA6D38yAR3YuALvYY0tUeFUoBXx28JHY5RETUzjAgkcGaPcoXAPBjfAZyi8pEroaIiNoTBiQyWCHdHODvaYuyShW++TtN7HKIiKgdYUAigyWRSDB7lPqOtu9i05FfUi5yRURE1F60KCBt3LgRu3bt0nw/f/582NraIiQkBJcvX2614ojuZlRPJ/RytUZxuRIbjqSLXQ4REbUTLQpIH374IczM1HcNxcbGYvXq1Vi+fDkcHR3x5ptvtmqBRI2RSCSYVTUv0vrD6bhVVilyRURE1B60KCBlZmbCx0f9obR9+3aMHTsW06dPR2RkJA4dOtSqBRLdzcN9XdC1kwUKblfg+6PswSQionvXooBkaWmJ69evAwD+/PNPPPjggwAAuVyO27dvt151RE0gk0owc4Q6sP/30CXcLleKXBERERm6FgWkBx98ENOmTcO0adNw4cIFPProowCAs2fPwsvLqzXrI2qSx/3d4Glvhrxb5dh8rOFFh4mIiJqiRQFp9erVCA4ORm5uLqKiouDg4AAASEhIwMSJE1u1QKKmMJZJ8erwbgCArw5eQlkle5GIiKjluNRIC3GpEf1TVqnE/cv3IbuwDJFP9cPEwZ3FLomIiPSMTpcaiY6Oxt9//635fvXq1fD398ezzz6LmzdvtuQpie6ZqZEM0+9X9yKt2Z+KSqVK5IqIiMhQtSggvfXWWygsLAQAnD59GnPnzsWjjz6KtLQ0REREtGqBRM0xcbAnHCxMkHGjBL/9c1XscoiIyEC1KCClpaWhd+/eAICoqCg89thj+PDDD7F69Wr8/vvvrVogUXOYmxjhpWHeAIDV+1KhUvEKMhERNV+LApKJiQlKSkoAAH/99RceeughAIC9vb2mZ4lILJOCu8BaboSUnFv446xC7HKIiMgAtSggDRs2DBEREXj//fcRHx+P0aNHAwAuXLgADw+PVi2QqLms5MaYMlTdi7RqXwp4HwIRETVXiwLSqlWrYGRkhJ9//hlr1qyBu7s7AOD333/Hww8/3KoFErXEiyFeMDeR4ezVQuw/nyt2OUREZGB4m38L8TZ//Re5OxnrDl7CwM62iJoRAolEInZJREQksqZ+fhu19AWUSiW2b9+O5ORkAECfPn3w+OOPQyaTtfQpiVrV1Pu8sf5IOhIz8hF76TpCujmKXRIRERmIFl1iS0lJQa9evTBp0iT88ssv+OWXX/D888+jT58+SE1Nbe0aiVrEyUqOiYGeAIBVe1NEroaIiAxJiwLS66+/jm7duiEzMxOJiYlITExERkYGvL298frrr7d2jUQtNn14NxhJJTiSeh0JlzmJKRERNU2LAtKBAwewfPly2Nvba/Y5ODhg2bJlOHDgQKsVR3Sv3G3NMHag+s7K1fvYi0RERE3TooBkamqKoqKiO/bfunULJiYm91wUUWuaMaIbpBJg77kcnLlSIHY5RERkAFoUkB577DFMnz4dcXFxEAQBgiDg6NGjePXVV/H444+3do1E98TL0QJj/NwAAF/uZy8SERHdXYsC0sqVK9GtWzcEBwdDLpdDLpcjJCQEPj4+WLFiRSuXSHTvZo70AQD8fkaBlJw7ez+JiIhqa9Ft/ra2tvj111+RkpKiuc2/V69e8PHxadXiiFpLd2crhPVxxh9ns/HlvlR8Nt5f7JKIiEiPNTkgRURENHp83759mn9/9tlnLa+ISEdmjfTFH2ez8eupq5gT2h2dHczFLomIiPRUkwPSiRMnmtSOsxWTvurnYYPh3TvhwIVcrDmQisin+oldEhER6akmB6TaPUREhmr2KB8cuJCLnxMy8foDPnC1MRO7JCIi0kMtGqRNZKgGedkjyNseFUoBXx28JHY5RESkpxiQqMOZPcoXAPBjfAbybpWJXA0REekjBiTqcIb6OMDP0xalFSp883ea2OUQEZEeYkCiDkcikWB21bxI38VeRkFJhcgVERGRvmFAog7pgV5O6OVqjVtlldhwJF3scoiISM8wIFGHJJFIMHNkNwDA/w6n4VZZpcgVERGRPmFAog7rkb6u6NrJAgW3K/D90ctil0NERHqEAYk6LJlUgtdGqMci/ffQJZRWKEWuiIiI9AUDEnVoT/i7wcPODHm3yrE5PkPscoiISE8wIFGHZiyT4tXh6rFI6w5eQnmlSuSKiIhIHzAgUYf3dIAHnK1Nca2gFL8kZoldDhER6QEGJOrw5MYyvHxfVwDAmgOpqFSyF4mIqKNjQCIC8GxQZ9hbmODy9RLs/Oea2OUQEZHIGJCIAJibGGHqMG8AwOp9KVCpBJErIiIiMTEgEVV5IbgLrORGuJhzC38mKcQuh4iIRMSARFTFWm6MF0O8AABf7E2BILAXiYioo9KLgLR69Wp4eXlBLpcjKCgI8fHxjbbfunUrevbsCblcjn79+mH37t1axwVBwOLFi+Hq6gozMzOEhobi4sWLWm28vLwgkUi0tmXLlrX6uZFheXGoN8xNZDh7tRD7L+SKXQ4REYlE9IC0ZcsWREREYMmSJUhMTISfnx/CwsKQk5NTb/sjR45g4sSJmDp1Kk6cOIHw8HCEh4fjzJkzmjbLly/HypUrsXbtWsTFxcHCwgJhYWEoLS3Veq733nsP165d02yzZ8/W6bmS/rOzMMHzQ7oAAFaxF4mIqMOSCCJ/AgQFBSEwMBCrVq0CAKhUKnh6emL27NlYsGDBHe3Hjx+P4uJi7Ny5U7NvyJAh8Pf3x9q1ayEIAtzc3DB37lzMmzcPAFBQUABnZ2ds2LABEyZMAKDuQZozZw7mzJnTpDrLyspQVlam+b6wsBCenp4oKCiAtbV1S0+f9FBOYSmGLd+H8koVfnx5CIK7OYhdEhERtZLCwkLY2Njc9fNb1B6k8vJyJCQkIDQ0VLNPKpUiNDQUsbGx9T4mNjZWqz0AhIWFadqnpaVBoVBotbGxsUFQUNAdz7ls2TI4ODhgwIAB+Pjjj1FZ2fCK7pGRkbCxsdFsnp6ezT5fMgxO1nJMCFS/v6v2XbxLayIiao9EDUh5eXlQKpVwdnbW2u/s7AyFov67iBQKRaPtq7/e7Tlff/11bN68Gfv27cMrr7yCDz/8EPPnz2+w1oULF6KgoECzZWZmNv1EyeC8MrwbjKQSHE65jsSMm2KXQ0REbcxI7ALEEhERofl3//79YWJigldeeQWRkZEwNTW9o72pqWm9+6l9crc1w1MD3fHT8Sys3puCb6YEil0SERG1IVF7kBwdHSGTyZCdna21Pzs7Gy4uLvU+xsXFpdH21V+b85yAeixUZWUl0tPTm3sa1E7NGOEDqQSIOZeDs1cLxC6HiIjakKgBycTEBAEBAYiJidHsU6lUiImJQXBwcL2PCQ4O1moPAHv27NG09/b2houLi1abwsJCxMXFNficAHDy5ElIpVI4OTndyylRO+LtaIHH+rsBAL7clypyNURE1JZEv8QWERGByZMnY9CgQRg8eDBWrFiB4uJivPjiiwCASZMmwd3dHZGRkQCAN954A8OHD8enn36K0aNHY/PmzTh+/Di++uorAIBEIsGcOXPwwQcfwNfXF97e3njnnXfg5uaG8PBwAOqB3nFxcRg5ciSsrKwQGxuLN998E88//zzs7OxE+TmQfpo50gc7Tl3F7jPXkJJTBB8nK7FLIiKiNiB6QBo/fjxyc3OxePFiKBQK+Pv7Izo6WjPIOiMjA1JpTUdXSEgINm3ahLfffhuLFi2Cr68vtm/fjr59+2razJ8/H8XFxZg+fTry8/MxbNgwREdHQy6XA1CPJ9q8eTOWLl2KsrIyeHt7480339Qal0QEAD1crPBQb2f8mZSNL/en4rNx/mKXREREbUD0eZAMVVPnUSDD909WPh5fdRgyqQT75o5AZwdzsUsiIqIWMoh5kIgMQX8PW9zfvROUKgFrD3IsEhFRR8CARNQEs0f5AAB+Pp4FRUHpXVoTEZGhY0AiaoJAL3sEedujXKnCVwcviV0OERHpGAMSURPNqupF2hR/GXm3yu7SmoiIDBkDkr4pUgC3csWuguoxzMcRfp62KK1Q4X9/p4ldDhER6RADkr7Zvwz4vA+wYzaQkyx2NVSLRCLBrJHqXqRvYy+joKRC5IqIiEhXGJD0iSAA11MAZRmQ+C3w5RDgu6eAlBj1MRLdAz2d0NPFCrfKKrHhSLrY5RARkY4wIOkTiQSY/Bvw0p9Ar8cBiRRIjQG+fwr4Mlgdmip4B5WYpFIJZlb1Iq0/koZbZZUiV0RERLrAgKRvJBKgcxAw/jvg9RPAkNcAE0sgN1l92e3zPurLcBynJJpH+7miq6MF8ksq8MPRy2KXQ0REOsCApM/svICHI4GIJOChDwAbT6AkD9gfqQ5Kv87iOCURyKQSzBjRDQDw9aE0lFYoRa6IiIhaGwOSIZDbACGzgddPAk+vB9wD1OOUTnxXa5zSXxyn1IbCB7jD3dYMebfKsOVYptjlEBFRK2NAMiQyI6DvU8C0GPU4pd5P1BqnNJbjlNqQsUyq6UVadyAV5ZUqkSsiIqLWxIBkiKrHKY37tuFxSvsiOU5Jx54O8ICTlSmuFpRi24ksscshIqJWxIBk6LTGKf1fzTilA8s4TknH5MYyTL+/KwBgzf5UVCrZi0RE1F4wILUXchsgZFatcUqD6oxTepLjlHTg2aDOsLcwQfr1Euw6fU3scoiIqJUwILU31eOUXo4Bpu6pNU5pb9U4pSFAwkaOU2ol5iZGmDrMGwCwam8KVCoGUCKi9oABqT3zHFxrnNJMwMQKyD0H/PZ6rXFKOWJXafBeCO4CK7kRLubcwp9J2WKXQ0RErYABqSOw8wIe/hCIOFvPOKW+wK8zgewksas0WNZyY0wJ8QIArNp3EQIvYxIRGTwGpI6k9jilZzbUGqf0PbAmWD1O6SLHKbXEi0O9YWYsw5krhThwgXcPEhEZOgakjkhmBPR5sv5xSj/UHqd0W+xKDYa9hQmeH9IZgHosEnuRiIgMGwNSR6cZp3SynnFKfYF9H3KcUhO9fF9XmBhJcfzyTcSl3RC7HCIiugcMSKRm16VqnFISEPYhYNO5apzSR1XzKXGc0t04WcsxfpAnAOCTP85DUcA7BYmIDJVE4LWAFiksLISNjQ0KCgpgbW0tdjmtT1kJnPsNiP0SyIqv2d91JBA8C/B5QD2jN2nJulmCER/vR6VKgFQCDPPthLED3RHWxwVyY5nY5RERdXhN/fxmQGqhdh+QasuMB2JXA8k7AKFqtuhOPYEhM4D+4wFjM3Hr0zMHLuRi9d4UxKfXXGazMjXC6P6uGBvggUFd7CBhuCQiEgUDko51qIBU7eZlIP4r9QDu8iL1PnMHIHCaerN0Erc+PXP5ejF+SbyCqMQsZN2sGfDexcEcTw3wwFMD3eFpby5ihUREHQ8Dko51yIBUrbRQvYTJ0bVAQYZ6n8wE6DcOCH4NcO4jbn16RqUSEJ9+A1EJWdh9+hqKy5WaY0He9ng6wAOP9HOFpamRiFUSEXUMDEg61qEDUjVlJXBup/ry2x3jlGYC3R4ApLwPoLaS8kr8cVaBqIQrOJyap5lyysxYhkf6umBsgAeCuzpAKuUlOCIiXWBA0jEGpDoyjwFHVwNJv9aMU3Lsoe5R4jilel3Nv41tJ64gKiELl/KKNfvdbOR4cqA7xg70QNdOliJWSETU/jAg6RgDUgMaGqc0aKp6nJKVs7j16SFBEHAiMx9RCVn47dRVFJZWao4N6GyLpwM88Fh/N9iYGYtYJRFR+8CApGMMSHdRWqhewiRuDZDPcUpNVVqhRExyDqISs3DgQi6UKvWfp4mRFA/2dsbTAz1wn68jjGS8dElE1BIMSDrGgNRE1eOUjn4JZMbV7O86Qj2fEscpNSinqBQ7Tl7FzwlZOKco0uzvZGWKcH83jA3wQE8X/u4RETUHA5KOMSC1AMcptYggCDh7tRBRiVnYcfIqrheXa471cbPG2IEeeMLfDQ6WpiJWSURkGBiQdIwB6R5Uj1NK/BYoK1Tv4zilJqlQqrD/fC6iErIQcy4bFUr1n6+RVIIRPZzwdIA7RvV0hokRe+WIiOrDgKRjDEitoMFxSs8AQ14DXPqKW5+eu1lcjt/+uYqohCycyirQ7LczN8bjfupLcP3cbThrNxFRLQxIOsaA1IqUlcD5Xer5lDhOqUUuZhfh58QsbD9xBdmFZZr9vk6WGBvggScHuMPZWi5ihURE+oEBSccYkHREM05pByBUzTjt2F3do+Q3geOU7kKpEvB3Sh6iErLwx1kFyirVY724cC4RkRoDko4xIOlYfgYQt66ecUovAYEvc5xSExSWVmD3P9cQlZiFY+k3NfutTI3wmJ8rxg70QAAXziWiDoYBSccYkNoIxym1ivS8YvySmIWoxCu4kl+zcK6XgzmeGqheONfDjgvnElH7x4CkYwxIbUylrFn3rfY4Je/h6nFKPqEcp9QEKpWAuLQbiEpUL5xbUmvh3CFd7TF2oAce7ecKCy6cS0TtFAOSjjEgiSjruDooJf1aZ5zSDKD/BMCEPSFNUVJeiegzCkQlZuFI6nXthXP7ueDpgR4YwoVziaidYUDSMQYkPZCfCcSvU6/7Vj1OycweCJzKcUrNdCX/NrZVXYJLq7VwrrutGZ4c4I6xAR7wdrQQsUIiotbBgKRjDEh6pKxIPU7p6Bog/7J6n8wE6Pu0epZul37i1mdABEFAYkY+ohLVC+cW1Vo4d2BnW4zlwrlEZOAYkHSMAUkPqZTAuer5lI7W7Oc4pRYprVDir+RsRCWoF86tWje3ZuHcAA/c58OFc4nIsDAg6RgDkp6rb5ySg2/Vum8cp9RcOUWl+PWEeuHc89naC+c+OcAdYwd6oIeLlYgVEhE1TVM/v/Xif/1Wr14NLy8vyOVyBAUFIT4+vtH2W7duRc+ePSGXy9GvXz/s3r1b67ggCFi8eDFcXV1hZmaG0NBQXLx4sd7nKisrg7+/PyQSCU6ePNlap0Ri8xgEPLMeeOMUEDIbMLUGrl8Edr4JfN4HiHkfKFKIXaXBcLKS4+X7uyJ6zn3YOXsYpoR4wd7CBLlFZfjq4CWErTiIx744hPWH03Cj1mK6RESGSvSAtGXLFkRERGDJkiVITEyEn58fwsLCkJOTU2/7I0eOYOLEiZg6dSpOnDiB8PBwhIeH48yZM5o2y5cvx8qVK7F27VrExcXBwsICYWFhKC0tveP55s+fDzc3N52dH4nM1hN46AMgIgl4+CPAtgtw+wZw6BPg877AthmA4rTYVRoMiUSCvu42WPp4Hxxd+AC+eiEAYX2cYSyT4MyVQrz7WxIG/99fePnb4/jjrALlVTN5ExEZGtEvsQUFBSEwMBCrVq0CAKhUKnh6emL27NlYsGDBHe3Hjx+P4uJi7Ny5U7NvyJAh8Pf3x9q1ayEIAtzc3DB37lzMmzcPAFBQUABnZ2ds2LABEyZM0Dzu999/R0REBKKiotCnTx+cOHEC/v7+9dZZVlaGsrKaNa4KCwvh6enJS2yGpsFxSvdXjVN6kOOUWuBGcTl+O6W+BHf6ivbCuU/4qy/B9XW35qzdRCQ6g7jEVl5ejoSEBISGhmr2SaVShIaGIjY2tt7HxMbGarUHgLCwME37tLQ0KBQKrTY2NjYICgrSes7s7Gy8/PLL+O6772BufvfxKJGRkbCxsdFsnp6ezTpX0hNSGdD7cWDqH8C0vUDfsYBEBqQdBDaNA1YPBo59A5SXiF2pQbG3MMHkEC/8NnsY/nzzfrxyf1c4WZniZkkFNhxJx5hVfyNsxUGsO5CKnMI7e3KJiPSNqAEpLy8PSqUSzs7a89U4OztDoah/fIhCoWi0ffXXxtoIgoApU6bg1VdfxaBBg5pU68KFC1FQUKDZMjMzm/Q40mMeAcDT/6s1TslGPU5pVwTweW+OU2qh7s5WWPhoLxxZMAobXgzEGD83mBpJcSH7FiJ/P4chkTGY/L947Dh1FaUVyrs/IRGRCDrkegJffPEFioqKsHDhwiY/xtTUFKampjqsikRTPU5p+L+AEz8AR79Uz6d06BPg8H+Afk+r131z7S92pQbFSCbFiB5OGNHDCQW3K7D79DVEJWTh+OWbOHAhFwcu5MJKboTH+nPhXCLSP6L2IDk6OkImkyE7O1trf3Z2NlxcXOp9jIuLS6Ptq7821mbv3r2IjY2FqakpjIyM4OPjAwAYNGgQJk+efO8nRobJ1AoY8irw+glg/PdA52BAVQGc+hFYdx+wcQxwPhpQceBxc9mYGWPi4M74eUYI9s0bgdmjfOBua4ai0kr8GJ+Jp9fGYtSnB/BFzEVk3eTlTSISn14M0h48eDC++OILAOpB2p07d8asWbMaHKRdUlKC3377TbMvJCQE/fv31xqkPW/ePMydOxeAekCWk5OTZpB2RkYGCgsLNY+/evUqwsLC8PPPPyMoKAgeHh53rZvzIHUQWQnA0dXA2e215lPyUfco+U3kfEr3QKUScDTtOqISruD3M9oL5wZ3dcDYAA880teFC+cSUasymIkit2zZgsmTJ2PdunUYPHgwVqxYgZ9++gnnzp2Ds7MzJk2aBHd3d0RGRgJQ3+Y/fPhwLFu2DKNHj8bmzZvx4YcfIjExEX379gUAfPTRR1i2bBk2btwIb29vvPPOO/jnn3+QlJQEuVx+Rw3p6enw9vZu9C62uhiQOpiCLCCuet23qru0zOyAQS+p132zdhW3PgNXXKa9cG41cxMZHu7LhXOJqPU09fNb9P81Gz9+PHJzc7F48WIoFAr4+/sjOjpaM8g6IyMD0lq3XYeEhGDTpk14++23sWjRIvj6+mL79u2acASo5zYqLi7G9OnTkZ+fj2HDhiE6OrrecETUJDYewEPvA8PnAyc3qccp3UwHDn0KHF7JcUr3yMLUCGMDPDA2wANZN0uwLfEKohKzkH69BL8kXsEviVfgbmuGpwa646mBXDiXiHRP9B4kQ8UepA5OpQTO71bPp5RRa0oKr/vU8yn5PsT5lO6ReuHcm/g54Qp2/qO9cG5AFzuMHeiB0f1duXAuETWLwVxiM1QMSKRxJQGI/RI4u61mnJKNp3px3G6jgK7DAbmNuDUauNIKJfYkZSMqMQsH6yyc+1BvZ4zlwrlE1EQMSDrGgER3KMgC4r8Cjm+oGacEqCei9AgEfB5QBya3AeoJK6lFsgtLsf2E+hLchexbmv1O1QvnBniguzMXziWi+jEg6RgDEjWovARI/xtIjQFSYtSTT9ZmZgd0HaEOS90eAGzcRSnT0AmCgDNXChGVmIVfT17BzZIKzbF+7jYYO9Adj/u7w97CRMQqiUjfMCDpGAMSNVl+BpC6Vx2WLh3Q7l0CgE49a8JSlxBOHdAC5ZUq7Dufg6iELOw9l4PKqmtwxjIJRvZwwtgAD4zs4QQTI16CI+roGJB0jAGJWkRZCVxNVIel1Bj1+CWh1sSTMlN1SOo2Sn1Jzqk3wNmlm+X6rTLsOHUVUYlZOHOlZr4zewsTPO7nhqcDPNDHjQvnEnVUDEg6xoBEreL2TXWvUmoMkLIXKMzSPm7pUhOWuo4ELBzEqdNAnVcUISoxC9tOXEFuUZlmfw9nK4wNcEe4vzucrDn9B1FHwoCkYwxI1OoEAci7WDN2Kf1voPJ2rQYSwNWvarD3A4DnYEDGW9ybolKpwqGUPEQlZOHPpGyUV6p77aQS4P7unTB2oAce7O0MuTEHzxO1dwxIOsaARDpXWaaeYyklBkjdB2Sf1j5uYgl431/Tw2TfVZw6DUzB7Qrs+ucafk7IRGJGvma/euFc9SW4gZ1teQmOqJ1iQNIxBiRqc0UKdVBK3aveSvK0j9t5qXuWfB5QT1gp5+/l3VzKvVU1U3cWrhaUavZ7O1ogrI8LhnS1xyAve1hyPTiidoMBSccYkEhUKhWg+Ed9OS51H5BxFFDV3OYOqRHgMRjwqbo7ztWfM3s3QqUScPTSdfycmIXfTytwu6Jm4VyZVIJ+7jYY0tWBgYmoHWBA0jEGJNIrZUVVcy9VTSdwI1X7uJk90G2kOix1G8XFdRtRXFaJPUnZOJySh6Np15F547bWcQYmIsPGgKRjDEik126m14SltINAWaH2cafeVXMvjVJPK2BsJkqZhiDrZgniLt3A0UvXGwxMfd1tMKSrPYZ0dUAgAxORXmNA0jEGJDIYygog63jV2KUY4EoigFp/9kZyoMvQmsHenXpy7qVGMDARGTYGJB1jQCKDVXIDuFQ12DtlL1B0Vfu4tXvV5bhR6rmXzO3FqdNAXMm/jbhL19WB6dINZNwo0TrOwESkXxiQdIwBidoFQQByz9Vcjrt8GKgsrdVAArgPrBm75BEIyPjh3pjmBqZBXexgJed8VkRthQFJxxiQqF2quF1r7qW9QE6S9nFTa+25l+y8RCnTkDAwEekXBiQdY0CiDqHwatXcS1XTCdy+oX3cvlvVzN6j1HMvmVqKU6cBuVtgkkpQ6y45BwzyYmAiak0MSDrGgEQdjkoJXDtVs25cVjygqqw5LjUGOg+pmU7ApT/nXmqCq/m3EZd2HUdTb+Bo2nVcvs7ARKRLDEg6xoBEHV5pIZB+qOpyXIx6aoHazB21516ychalTEPDwESkWwxIOsaARFTHjUs1Y5fSDgLlt7SPO/etGbvUORgwMhWnTgPDwETUuhiQdIwBiagRygogM75q7NJe4OpJaM+9ZAZ4Dasav/QA4OjLuZeaiIGJ6N4wIOkYAxJRMxTnAZf210wncEuhfdzGs+ZyXNfhgJmdKGUaomsFt2smrrx0Hen1BKa+dZZGsWZgog6MAUnHGJCIWkgQ1NMHVF+Ou3wEUJbVHJdIAfcAdVjyeQBwG8i5l5qBgYmocQxIOsaARNRKykvUIal6KZTcc9rH5TaA9/Ca6QRsO4tTp4FiYCLSxoCkYwxIRDpSkKU991JpvvZxB99acy8NA0wsRCnTUDEwUUfHgKRjDEhEbUClBK6eqBm7lHUMEJQ1x2UmVXMvVQUml34c7N1MTQlMfdxqrSXnzcBEho0BSccYkIhEcDtfe+6l/Azt4xZO6qBUvVl2EqVMQ6YoKFXfJVc103daXrHWcQYmMnQMSDrGgEQkMkGoNfdSDJB2CKjQ/jCHS/+auZc8hwBGJuLUasAYmKi9YUDSMQYkIj1TWVYz91JKDKD4R/u4sYX23EsO3Xg5rgWaG5gGednDxoyBifQHA5KOMSAR6blbOeq5l6qnEyjO0T5u27nqUlzV3EtyG1HKNHQMTGRoGJB0jAGJyICoVEDO2ZrLcRlHAWV5zXGJDPAYVGvupQGAVCZevQYsu7BUE5biLl3HpTqBSSIB+rhZY4i3g+aSHAMTtSUGJB1jQCIyYOXFQPrhmstx1y9qH5fbAh6BgHMf9Z1xzn0ABx9Axg/y5mJgIn3DgKRjDEhE7Uh+Rs1UApcOAGUFd7aRmQCdegDOVYHJuY96AV7eKdcsDEwkNgYkHWNAImqnlJXAtZPqLftszVZ+q/72Fk6AS9+awOTcB3DswTvmmoiBidoaA5KOMSARdSAqFZB/uVZgOqP+euMSgHr+Eyo1Uoek6p4ml75VvU3OvHPuLnIKS3E0rWbiyku5DEzUuhiQdIwBiYhQdku9dlx1YFJUfa3vEh0AmDtUhaZal+k69QSM5W1btwFpSmDq7WpdtTSKAwZ72cPGnIGJGsaApGMMSERUL0FQryeXfRbIPl3T63Q9BRBUd7aXyABHX+1xTc59AGt39jbVo6mBKcjbAf09bNDbzRpdHS1gJJOKVDHpGwYkHWNAIqJmqbit7m2q7mXKPqPebt+sv73ctiYsVV+m69QLMDFv07L13d0CEwCYGEnRw9kKvV2t0cvVCr3dbNDT1YozfndQDEg6xoBERPdMEICia9rjmhRngLwL2ovyakjUM4DXvUxn25m9TVVyCksRl3YDx9JvIOlqIc4pinCrrLLetp72ZlWhyRq9Xa3R280a7rZmkPBn2a4xIOkYAxIR6UxlGZB7XrunSXEGKMmrv72pNeDUW/tuOqdegKlV29ath1QqAZk3S5B8rRBJVwuRdK0QydeKcCX/dr3treVG6sDkVhOcfJ0tYWrEiUPbCwYkHWNAIqI2dyunJixVj23KPQeoKupvb+etPa7JpS9g6wVIOR4nv6RcE5aqg1NKThEqlHd+JBpJJfBxsqzpbaoKT/YWnMrBEDEg6RgDEhHpBWWF+pJc7ct02WfVl+7qY2wBOPeuFZz6qr/nWnQor1QhJeeWurepqscpWVGI/JL6A6iLtbwqLFmht6sNerlawcvBAlIpL9HpMwYkHWNAIiK9VpynPdFl9mkg5xygLKu/vU3nWnM2VYUn+64dfk06QRBwraC0ziW6QqRfL6m3vbmJDD1drDQ9Tb1drdHDxQrmJkZtXDk1hAFJxxiQiMjgKCuBG6l3XqYrzKq/vZGZeixT7ct0zn0Ac/u2rVsP3SqrxHlFTWhKulaEc9cKUVZ551QOEgng7WihdYmut6s1nKxMOSBcBAYVkFavXo2PP/4YCoUCfn5++OKLLzB48OAG22/duhXvvPMO0tPT4evri48++giPPvqo5rggCFiyZAm+/vpr5OfnY+jQoVizZg18fX01bR5//HGcPHkSOTk5sLOzQ2hoKD766CO4ubk1qWYGJCJqN27fBLKTagaEZ59Vf19Z/0BmWLvXmbepb9Vivh27l6RSqUL69WIkVY1rqr5Ul1tUf6+dg4WJ1mDwXq7W6NrJAsacs0mnDCYgbdmyBZMmTcLatWsRFBSEFStWYOvWrTh//jycnJzuaH/kyBHcf//9iIyMxGOPPYZNmzbho48+QmJiIvr27QsA+OijjxAZGYmNGzfC29sb77zzDk6fPo2kpCTI5eoZaz///HMEBwfD1dUVV65cwbx58zTP3xQMSETUrqmUwI007XFN2afVC/vWR2ZatZhvX+3LdBaObVu3HsotKtOEpepLdam5t6Cq59O3es4m9bgmdWjq5WbNOZtakcEEpKCgIAQGBmLVqlUAAJVKBU9PT8yePRsLFiy4o/348eNRXFyMnTt3avYNGTIE/v7+WLt2LQRBgJubG+bOnasJPQUFBXB2dsaGDRswYcKEeuvYsWMHwsPDUVZWBmPju/8iMiARUYdUWgDkJGvP25ST1PBivpbOtS7PVS/m273DL+ZbWqHEhewirXFNydcan7Opl0vN5blertbwsOOcTS3R1M9vUftDy8vLkZCQgIULF2r2SaVShIaGIjY2tt7HxMbGIiIiQmtfWFgYtm/fDgBIS0uDQqFAaGio5riNjQ2CgoIQGxtbb0C6ceMGfvjhB4SEhDQYjsrKylBWVtNNWlhY2OTzJCJqN+Q2QOch6q2a1mK+tS7T3UgDbmWrt9SYmvZS46repj61tn6ApVOHmfBSbixDfw9b9Pew1exTqQRk3byNpGsFVcGpCMnXCnEl/zYyb6i3P5OyNe2tqudsqjWuiXM2tR5RA1JeXh6USiWcnZ219js7O+PcuXP1PkahUNTbXqFQaI5X72uoTbV//etfWLVqFUpKSjBkyBCtXqm6IiMj8e677zbtxIiIOhKpFLD3Vm+9HqvZX3ZLu7epeisrqAlStZk71vQ0VV+mc+zRYRbzlUol6Oxgjs4O5ni4r6tmf35JuXq+plqX6C7mFKGotBLxaTcQn3ZD09ZIKkG3TpZaPU293ThnU0t06BF1b731FqZOnYrLly/j3XffxaRJk7Bz5856uywXLlyo1XNVWFgIT0/PtiyXiMiwmFoCnoHqrZogAAWZdy6vciNVPVN42gH1Vk2zmG+dy3TWbh2mt8nW3ATB3RwQ3M1Bs6+8UoXU3Ftag8GTrqnnbDqfXYTz2UXYduKKpr2LtbxqHbqaQeGcs6lxogYkR0dHyGQyZGdna+3Pzs6Gi4tLvY9xcXFptH311+zsbLi6umq18ff3v+P1HR0d0b17d/Tq1Quenp44evQogoOD73hdU1NTmJqaNvsciYioFolEvXacbWegxyM1+8tL1LOCawWn00Bpvnp/7jngzM817c3stKcecO7ToRbzNTGSqgdwu9aMoREEAYrCUvXluapJLpOuqudsUhSWQlFYin3nczXtzU1k6OFipblE18vVGj05Z5OGqD8FExMTBAQEICYmBuHh4QDUg7RjYmIwa9aseh8THByMmJgYzJkzR7Nvz549mlDj7e0NFxcXxMTEaAJRYWEh4uLiMGPGjAZrUanUc1fUHmdERERtxMQccB+o3qrVXsxXcbrmEl3eBfXUBOmH1Fs1iRSw71bT09SpO+Dgq57wsgNcppNIJHC1MYOrjRke6FUzzER7zib1pbrzikKUlCtxIiMfJzLyaz0H4O1ggV5Vl+iqw1NHnLNJ9LvYtmzZgsmTJ2PdunUYPHgwVqxYgZ9++gnnzp2Ds7MzJk2aBHd3d0RGRgJQ34Y/fPhwLFu2DKNHj8bmzZvx4Ycf3nGb/7Jly7Ru8//nn380t/nHxcXh2LFjGDZsGOzs7JCamop33nkH2dnZOHv2bJN6ingXGxGRSCrLavU2nb37Yr4SqbrHysFXfQedo0/NvzvQwPDalCoBaXnFWuOaGpuzyd7CpFZPk3ppFUOds8kg7mID1Lft5+bmYvHixVAoFPD390d0dLRmkHVGRgaktRZWDAkJwaZNm/D2229j0aJF8PX1xfbt2zXhCADmz5+P4uJiTJ8+Hfn5+Rg2bBiio6M1cyCZm5vjl19+wZIlS1BcXAxXV1c8/PDDePvtt3kZjYhI3xmZAq5+6q2aINQs5lsdnK5fBPJS1IPCb6art5Q92s9laq0e4+Tgqw5Ojh2j10lWtQCvj5MlHvermSC5es6m2uvRXcorxo3icvydkoe/U2pCqImRFN2day3i287mbBK9B8lQsQeJiMgAVAen6xeBvKrt+kX1Zbr8DEC4c2kQAOx1qqX2nE01E142PGeTh52Z1rim3no2Z5PBTBRpqBiQiIgMXGUZcOOSOixphaeqXqeGdNBep9q05myqtbTKlfz6l6fRmrOpKjz5OFlCbtz2czYxIOkYAxIRUTvFXqcWKyipqBnXVGvOpgrlnVFDJpXAp2rOpupxTb1creBgqduhLgxIOsaARETUAbHXqdmq52yqPRi8es6m+jhbm2rGNT05wB2+zlatWg8Dko4xIBERkQZ7nZql9pxNtcc1peUVa7VbPyUQI3veuXD9vWBA0jEGJCIiahL2OjWZZs6mqnFNbz7oCyer1j1XBiQdY0AiIqJ7otXrdEEdmFra61Qdntpxr1NrYUDSMQYkIiLSmYpSda/THZfsmtPr5Fvz73bc69RcBjNRJBEREdVhLAece6u32u7W61RWCFxJUG+1sdep2diD1ELsQSIiIr3CXqcmYQ8SERFRR8Jep1bFHqQWYg8SEREZvA7Y68QeJCIiImoce50axB6kFmIPEhERdUgG3uvEHiQiIiJqfR2k14k9SC3EHiQiIqImammv09hvgH5Pt2op7EEiIiIi/dDSXicHH3HqBQMSERERiUUiAayc1ZvXMO1jFaWAzFicusCARERERPpI5OkCpKK+OhEREZEeYkAiIiIiqoMBiYiIiKgOBiQiIiKiOhiQiIiIiOpgQCIiIiKqgwGJiIiIqA4GJCIiIqI6GJCIiIiI6mBAIiIiIqqDAYmIiIioDgYkIiIiojoYkIiIiIjqMBK7AEMlCAIAoLCwUORKiIiIqKmqP7erP8cbwoDUQkVFRQAAT09PkSshIiKi5ioqKoKNjU2DxyXC3SIU1UulUuHq1auwsrKCRCJptectLCyEp6cnMjMzYW1t3WrPq0/a+zny/Axfez/H9n5+QPs/R55fywmCgKKiIri5uUEqbXikEXuQWkgqlcLDw0Nnz29tbd0uf+lra+/nyPMzfO39HNv7+QHt/xx5fi3TWM9RNQ7SJiIiIqqDAYmIiIioDgYkPWNqaoolS5bA1NRU7FJ0pr2fI8/P8LX3c2zv5we0/3Pk+ekeB2kTERER1cEeJCIiIqI6GJCIiIiI6mBAIiIiIqqDAYmIiIioDgYkEaxevRpeXl6Qy+UICgpCfHx8o+23bt2Knj17Qi6Xo1+/fti9e3cbVdpyzTnHDRs2QCKRaG1yubwNq22egwcPYsyYMXBzc4NEIsH27dvv+pj9+/dj4MCBMDU1hY+PDzZs2KDzOluquee3f//+O94/iUQChULRNgU3U2RkJAIDA2FlZQUnJyeEh4fj/Pnzd32cofwdtuT8DO1vcM2aNejfv79mEsHg4GD8/vvvjT7GUN4/oPnnZ2jvX13Lli2DRCLBnDlzGm3X1u8hA1Ib27JlCyIiIrBkyRIkJibCz88PYWFhyMnJqbf9kSNHMHHiREydOhUnTpxAeHg4wsPDcebMmTauvOmae46AerbUa9euabbLly+3YcXNU1xcDD8/P6xevbpJ7dPS0jB69GiMHDkSJ0+exJw5czBt2jT88ccfOq60ZZp7ftXOnz+v9R46OTnpqMJ7c+DAAcycORNHjx7Fnj17UFFRgYceegjFxcUNPsaQ/g5bcn6AYf0Nenh4YNmyZUhISMDx48cxatQoPPHEEzh79my97Q3p/QOaf36AYb1/tR07dgzr1q1D//79G20nynsoUJsaPHiwMHPmTM33SqVScHNzEyIjI+ttP27cOGH06NFa+4KCgoRXXnlFp3Xei+ae4/r16wUbG5s2qq51ARC2bdvWaJv58+cLffr00do3fvx4ISwsTIeVtY6mnN++ffsEAMLNmzfbpKbWlpOTIwAQDhw40GAbQ/w7rNaU8zPkv8FqdnZ2wn//+996jxny+1etsfMz1PevqKhI8PX1Ffbs2SMMHz5ceOONNxpsK8Z7yB6kNlReXo6EhASEhoZq9kmlUoSGhiI2Nrbex8TGxmq1B4CwsLAG24utJecIALdu3UKXLl3g6el51/9TMjSG9h62lL+/P1xdXfHggw/i8OHDYpfTZAUFBQAAe3v7BtsY8nvYlPMDDPdvUKlUYvPmzSguLkZwcHC9bQz5/WvK+QGG+f7NnDkTo0ePvuO9qY8Y7yEDUhvKy8uDUqmEs7Oz1n5nZ+cGx2soFIpmtRdbS86xR48e+N///odff/0V33//PVQqFUJCQpCVldUWJetcQ+9hYWEhbt++LVJVrcfV1RVr165FVFQUoqKi4OnpiREjRiAxMVHs0u5KpVJhzpw5GDp0KPr27dtgO0P7O6zW1PMzxL/B06dPw9LSEqampnj11Vexbds29O7du962hvj+Nef8DPH927x5MxITExEZGdmk9mK8h0Y6e2aiJgoODtb6P6OQkBD06tUL69atw/vvvy9iZdQUPXr0QI8ePTTfh4SEIDU1FZ9//jm+++47ESu7u5kzZ+LMmTP4+++/xS5FJ5p6fob4N9ijRw+cPHkSBQUF+PnnnzF58mQcOHCgwRBhaJpzfob2/mVmZuKNN97Anj179HowOQNSG3J0dIRMJkN2drbW/uzsbLi4uNT7GBcXl2a1F1tLzrEuY2NjDBgwACkpKboosc019B5aW1vDzMxMpKp0a/DgwXofOmbNmoWdO3fi4MGD8PDwaLStof0dAs07v7oM4W/QxMQEPj4+AICAgAAcO3YM//nPf7Bu3bo72hri+9ec86tL39+/hIQE5OTkYODAgZp9SqUSBw8exKpVq1BWVgaZTKb1GDHeQ15ia0MmJiYICAhATEyMZp9KpUJMTEyD15aDg4O12gPAnj17Gr0WLaaWnGNdSqUSp0+fhqurq67KbFOG9h62hpMnT+rt+ycIAmbNmoVt27Zh79698Pb2vutjDOk9bMn51WWIf4MqlQplZWX1HjOk968hjZ1fXfr+/j3wwAM4ffo0Tp48qdkGDRqE5557DidPnrwjHAEivYc6G/5N9dq8ebNgamoqbNiwQUhKShKmT58u2NraCgqFQhAEQXjhhReEBQsWaNofPnxYMDIyEj755BMhOTlZWLJkiWBsbCycPn1arFO4q+ae47vvviv88ccfQmpqqpCQkCBMmDBBkMvlwtmzZ8U6hUYVFRUJJ06cEE6cOCEAED777DPhxIkTwuXLlwVBEIQFCxYIL7zwgqb9pUuXBHNzc+Gtt94SkpOThdWrVwsymUyIjo4W6xQa1dzz+/zzz4Xt27cLFy9eFE6fPi288cYbglQqFf766y+xTqFRM2bMEGxsbIT9+/cL165d02wlJSWaNob8d9iS8zO0v8EFCxYIBw4cENLS0oR//vlHWLBggSCRSIQ///xTEATDfv8EofnnZ2jvX33q3sWmD+8hA5IIvvjiC6Fz586CiYmJMHjwYOHo0aOaY8OHDxcmT56s1f6nn34SunfvLpiYmAh9+vQRdu3a1cYVN19zznHOnDmats7OzsKjjz4qJCYmilB101Tf1l53qz6nyZMnC8OHD7/jMf7+/oKJiYnQtWtXYf369W1ed1M19/w++ugjoVu3boJcLhfs7e2FESNGCHv37hWn+Cao79wAaL0nhvx32JLzM7S/wZdeekno0qWLYGJiInTq1El44IEHNOFBEAz7/ROE5p+fob1/9akbkPThPZQIgiDorn+KiIiIyPBwDBIRERFRHQxIRERERHUwIBERERHVwYBEREREVAcDEhEREVEdDEhEREREdTAgEREREdXBgERERERUBwMSEVEr2L9/PyQSCfLz88UuhYhaAQMSERERUR0MSERERER1MCARUbugUqkQGRkJb29vmJmZwc/PDz///DOAmstfu3btQv/+/SGXyzFkyBCcOXNG6zmioqLQp08fmJqawsvLC59++qnW8bKyMvzrX/+Cp6cnTE1N4ePjg2+++UarTUJCAgYNGgRzc3OEhITg/Pnzuj1xItIJBiQiahciIyPx7bffYu3atTh79izefPNNPP/88zhw4ICmzVtvvYVPP/0Ux44dQ6dOnTBmzBhUVFQAUAebcePGYcKECTh9+jSWLl2Kd955Bxs2bNA8ftKkSfjxxx+xcuVKJCcnY926dbC0tNSq49///jc+/fRTHD9+HEZGRnjppZfa5PyJqHVJBEEQxC6CiOhelJWVwd7eHn/99ReCg4M1+6dNm4aSkhJMnz4dI0eOxObNmzF+/HgAwI0bN+Dh4YENGzZg3LhxeO6555Cbm4s///xT8/j58+dj165dOHv2LC5cuIAePXpgz549CA0NvaOG/fv3Y+TIkfjrr7/wwAMPAAB2796N0aNH4/bt25DL5Tr+KRBRa2IPEhEZvJSUFJSUlODBBx+EpaWlZvv222+RmpqqaVc7PNnb26NHjx5ITk4GACQnJ2Po0KFazzt06FBcvHgRSqUSJ0+ehEwmw/DhwxutpX///pp/u7q6AgBycnLu+RyJqG0ZiV0AEdG9unXrFgBg165dcHd31zpmamqqFZJayszMrEntjI2NNf+WSCQA1OOjiMiwsAeJiAxe7969YWpqioyMDPj4+Ghtnp6emnZHjx7V/PvmzZu4cOECevXqBQDo1asXDh8+rPW8hw8fRvfu3SGTydCvXz+oVCqtMU1E1H6xB4mIDJ6VlRXmzZuHN998EyqVCsOGDUNBQQEOHz4Ma2trdOnSBQDw3nvvwcHBAc7Ozvj3v/8NR0dHhIeHAwDmzp2LwMBAvP/++xg/fjxiY2OxatUqfPnllwAALy8vTJ48GS+99BJWrlwJPz8/XL58GTk5ORg3bpxYp05EOsKARETtwvvvv49OnTohMjISly5dgq2tLQYOHIhFixZpLnEtW7YMb7zxBi5evAh/f3/89ttvMDExAQAMHDgQP/30ExYvXoz3338frq6ueO+99zBlyhTNa6xZswaLFi3Ca6+9huvXr6Nz585YtGiRGKdLRDrGu9iIqN2rvsPs5s2bsLW1FbscIjIAHINEREREVAcDEhEREVEdvMRGREREVAd7kIiIiIjqYEAiIiIiqoMBiYiIiKgOBiQiIiKiOhiQiIiIiOpgQCIiIiKqgwGJiIiIqA4GJCIiIqI6/h+osc8G2nktzQAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.plot(train_losses, label='train loss')\n",
        "plt.plot(val_losses, label='validation loss')\n",
        "plt.xlabel(\"epoch\")\n",
        "plt.ylabel(\"loss\")\n",
        "plt.legend()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's finally understand what are the limitations of the fully-connected models. These models learn a function by considering all correlations between each combination of pixels in the image. However, for us, locality is much more important. \n",
        "As we ravel the images, we loose for instance the 2D structure of the image. For the model, this information is passed as raw, thus it will be equally important to connect pixels that are neighbors and pixels that are far away from each other.\n",
        "\n",
        "Now we introduce a disturbance, where we shift the image and check performances of the model. \n",
        "\n",
        "The model is learning spurious correlations and also it is missing important information related to the locality of the pixels w.r.t. their neighborhood!\n",
        "\n",
        "However, while the performance of the model drops substantially, the image for us still looks unchanged and we are able to recognize the digit. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 489
        },
        "id": "_dKnRdSZOWZg",
        "outputId": "16d69726-ef19-4e14-f71c-25219ef9b091"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGzCAYAAABpdMNsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAkaUlEQVR4nO3dfXRU9Z3H8c8kkgElGQyQJ0xCAAHlyRUxZlVAEgmxUlE86+PZBDlQMbCLrGjhiFHcNi3tKliBrts2rHvABxCwKJtWMAlHCbhEEdmtKQlpQUkiD2YmBEnY5Ld/cJxlTABvmOGXhPfrnHtO5t7fd+4312s+/Gbu3HEZY4wAALjIwmw3AAC4NBFAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAQDu5XC49++yzQXu+/v37Kycnx3FdcXGxXC6X1q1bF7ReVq1aJZfLpb/85S9Be07guwggWPXZZ5/p3nvvVXJysrp3765+/frp9ttv169+9SvbreECuFyusy6333677fbQQVxmuwFcurZv367bbrtNSUlJmjFjhuLi4nTw4EHt2LFDy5Yt05w5c2y3iHb6j//4j1brdu3apWXLlmnixIkWOkJHRADBmp/85CfyeDz6r//6L/Xq1Stg21dffWWnKQTFww8/3Grdty8VPvDAAxY6QkfES3CwprKyUsOGDWsVPpIUExMT8LigoEATJkxQTEyM3G63rr32Wq1cubJVXf/+/XXnnXequLhYN9xwg3r06KERI0aouLhYkrR+/XqNGDFC3bt31+jRo/XJJ58E1Ofk5Khnz57av3+/MjMzdcUVVyghIUGLFy/W97lx/JdffqlHHnlEsbGxcrvdGjZsmH73u999/4NyhmPHjumJJ57QiBEj1LNnT0VFRSkrK0uffvppm+Obm5u1cOFCxcXF6YorrtAPf/hDHTx4sNW4nTt3atKkSfJ4PLr88ss1btw4ffjhh+ftx+v16vPPP5fX63X8uzQ2Nuqtt97SuHHjdNVVVzmuR9dEAMGa5ORklZWVae/evecdu3LlSiUnJ2vhwoX6l3/5FyUmJuqxxx7T8uXLW42tqKjQgw8+qMmTJys/P19ff/21Jk+erNWrV+vxxx/Xww8/rOeee06VlZX6u7/7O7W0tATUNzc3a9KkSYqNjdWSJUs0evRo5eXlKS8v75w91tbW6qabbtKWLVs0e/ZsLVu2TIMGDdL06dO1dOlSR8dGkvbv36+NGzfqzjvv1AsvvKD58+frs88+07hx43To0KFW43/yk5/o3Xff1VNPPaV/+Id/0HvvvaeMjAx98803/jHvv/++xo4dK5/Pp7y8PP30pz9VXV2dJkyYoI8++uic/WzYsEHXXHONNmzY4Ph32bx5s+rq6vTQQw85rkUXZgBL/vjHP5rw8HATHh5u0tLSzJNPPmn+8Ic/mKamplZjT5w40WpdZmamGTBgQMC65ORkI8ls377dv+4Pf/iDkWR69Ohh/vrXv/rX/+u//quRZIqKivzrsrOzjSQzZ84c/7qWlhbzgx/8wERERJjDhw/710syeXl5/sfTp0838fHx5siRIwE93X///cbj8bT5O3y39+zsbP/jkydPmubm5oAxVVVVxu12m8WLF/vXFRUVGUmmX79+xufz+de/+eabRpJZtmyZ//e4+uqrTWZmpmlpafGPO3HihElJSTG33367f11BQYGRZKqqqlqtKygoOOfv0ZapU6cat9ttvv76a8e16LqYAcGa22+/XaWlpfrhD3+oTz/9VEuWLFFmZqb69eun3//+9wFje/To4f/Z6/XqyJEjGjdunPbv39/qJaFrr71WaWlp/sepqamSpAkTJigpKanV+v3797fqbfbs2f6fXS6XZs+eraamJm3ZsqXN38UYo7feekuTJ0+WMUZHjhzxL5mZmfJ6vfr444+/76GRJLndboWFnf5ftLm5WUePHlXPnj01ZMiQNp/r7//+7xUZGel/fO+99yo+Pl6bN2+WJO3evVv79u3Tgw8+qKNHj/r7a2hoUHp6urZt29ZqNnimnJwcGWMcXyru8/n07rvv6o477mjz5VZcurgIAVaNGTNG69evV1NTkz799FNt2LBBL774ou69917t3r1b1157rSTpww8/VF5enkpLS3XixImA5/B6vfJ4PP7HZ4aMJP+2xMTENtd//fXXAevDwsI0YMCAgHWDBw+WpLN+Lubw4cOqq6vTK6+8oldeeaXNMU4vrGhpadGyZcu0YsUKVVVVqbm52b+td+/ercZfffXVAY9dLpcGDRrk73nfvn2SpOzs7LPu0+v16sorr3TU5/m89dZbOnnyJC+/oRUCCB1CRESExowZozFjxmjw4MGaNm2a1q5dq7y8PFVWVio9PV1Dhw7VCy+8oMTEREVERGjz5s168cUXW/2rPTw8vM19nG29CcK30n/bw8MPP3zWP/AjR4509Jw//elPtWjRIj3yyCN6/vnnFR0drbCwMM2dO/ecM5Xz9fiLX/xC1113XZtjevbs6fh5z2f16tXyeDy68847g/7c6NwIIHQ4N9xwgySpurpakrRp0yY1Njbq97//fcDspqioKCT7b2lp0f79+/2zHkn685//LOn0VXZt6du3ryIjI9Xc3KyMjIyg9LFu3Trddttt+u1vfxuwvq6uTn369Gk1/tsZzreMMaqoqPAH38CBAyVJUVFRQevxfKqrq1VUVKScnBy53e6Lsk90HrwHBGuKioranH18+57FkCFDJP3/zOXMsV6vVwUFBSHr7eWXX/b/bIzRyy+/rG7duik9Pb3N8eHh4Zo6dareeuutNq/qO3z4sOMewsPDWx2ftWvX6ssvv2xz/Kuvvqr6+nr/43Xr1qm6ulpZWVmSpNGjR2vgwIH65S9/qePHjzvusT2XYb/++utqaWnh5Te0iRkQrJkzZ45OnDihu+++W0OHDlVTU5O2b9+uN954Q/3799e0adMkSRMnTlRERIQmT56sH/3oRzp+/Lj+7d/+TTExMf5ZUjB1795dhYWFys7OVmpqqv7zP/9T7777rhYuXKi+ffuete5nP/uZioqKlJqaqhkzZujaa6/VsWPH9PHHH2vLli06duyYoz7uvPNOLV68WNOmTdPf/u3f6rPPPtPq1atbvT/1rejoaN1yyy2aNm2aamtrtXTpUg0aNEgzZsyQdPq9rd/85jfKysrSsGHDNG3aNPXr109ffvmlioqKFBUVpU2bNp21nw0bNmjatGkqKCj43hcirF69WgkJCRo/fryj3x2XBgII1vzyl7/U2rVrtXnzZr3yyitqampSUlKSHnvsMT399NP+K6aGDBmidevW6emnn9YTTzyhuLg4zZo1S3379tUjjzwS9L7Cw8NVWFioWbNmaf78+YqMjFReXp6eeeaZc9bFxsbqo48+0uLFi7V+/XqtWLFCvXv31rBhw/Tzn//ccR8LFy5UQ0OD1qxZozfeeEPXX3+93n33Xf34xz8+6/g9e/YoPz9f9fX1Sk9P14oVK3T55Zf7x4wfP16lpaV6/vnn9fLLL+v48eOKi4tTamqqfvSjHznu8VzKy8tVVlamefPm+a/mA87kMsF4BxboInJycrRu3bo2X6ICEFz8swQAYAUBBACwggACAFjBe0AAACuYAQEArCCAAABWdLjPAbW0tOjQoUOKjIyUy+Wy3Q4AwCFjjOrr65WQkHDOz4B1uAA6dOhQq7sWAwA6n4MHD57zG3A73EtwZ36fCQCg8zrf3/OQBdDy5cvVv39/de/eXampqef9ut9v8bIbAHQN5/t7HpIAeuONNzRv3jzl5eXp448/1qhRo5SZmen4C7kAAF1YKL7n+8YbbzS5ubn+x83NzSYhIcHk5+eft9br9RpJLCwsLCydfPF6vef8ex/0GVBTU5PKysoCvvAqLCxMGRkZKi0tbTW+sbFRPp8vYAEAdH1BD6AjR46oublZsbGxAetjY2NVU1PTanx+fr48Ho9/4Qo4ALg0WL8KbsGCBfJ6vf7l4MGDtlsCAFwEQf8cUJ8+fRQeHq7a2tqA9bW1tYqLi2s13u12813xAHAJCvoMKCIiQqNHj9bWrVv961paWrR161alpaUFe3cAgE4qJHdCmDdvnrKzs3XDDTfoxhtv1NKlS9XQ0KBp06aFYncAgE4oJAF033336fDhw3rmmWdUU1Oj6667ToWFha0uTAAAXLo63PcB+Xw+eTwe220AAC6Q1+tVVFTUWbdbvwoOAHBpIoAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALAi6AH07LPPyuVyBSxDhw4N9m4AAJ3cZaF40mHDhmnLli3/v5PLQrIbAEAnFpJkuOyyyxQXFxeKpwYAdBEheQ9o3759SkhI0IABA/TQQw/pwIEDZx3b2Ngon88XsAAAur6gB1BqaqpWrVqlwsJCrVy5UlVVVbr11ltVX1/f5vj8/Hx5PB7/kpiYGOyWAAAdkMsYY0K5g7q6OiUnJ+uFF17Q9OnTW21vbGxUY2Oj/7HP5yOEAKAL8Hq9ioqKOuv2kF8d0KtXLw0ePFgVFRVtbne73XK73aFuAwDQwYT8c0DHjx9XZWWl4uPjQ70rAEAnEvQAeuKJJ1RSUqK//OUv2r59u+6++26Fh4frgQceCPauAACdWNBfgvviiy/0wAMP6OjRo+rbt69uueUW7dixQ3379g32rgAAnVjIL0JwyufzyePx2G4DAHCBzncRAveCAwBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArQv6FdMCZ7r33Xsc1M2bMcFxz6NAhxzWSdPLkScc1q1evdlxTU1PjuOZsX+oIdFbMgAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGCFyxhjbDdxJp/PJ4/HY7sNhMj+/fsd1/Tv3z/4jVhWX1/vuOa///u/Q9AJgu2LL75wXLNkyRLHNbt27XJcc7F5vV5FRUWddTszIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACw4jLbDeDSMmPGDMc1I0eOdFzzpz/9yXGNJF1zzTWOa66//nrHNePHj3dcc9NNNzmuOXjwoOMaSUpMTGxX3cXwv//7v45rDh8+3K59xcfHt6vOqQMHDjiu6Qw3Iz0fZkAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAU3I8VFtXXr1otS016FhYUXZT9XXnml45rrrrvOcU1ZWZnjGkkaM2ZMu+ouhpMnTzqu+fOf/9yufbXnprbR0dGOayorKx3XdAXMgAAAVhBAAAArHAfQtm3bNHnyZCUkJMjlcmnjxo0B240xeuaZZxQfH68ePXooIyND+/btC1a/AIAuwnEANTQ0aNSoUVq+fHmb25csWaKXXnpJv/71r7Vz505dccUVyszMbNfrtgCArsvxRQhZWVnKyspqc5sxRkuXLtXTTz+tu+66S5L06quvKjY2Vhs3btT9999/Yd0CALqMoL4HVFVVpZqaGmVkZPjXeTwepaamqrS0tM2axsZG+Xy+gAUA0PUFNYBqamokSbGxsQHrY2Nj/du+Kz8/Xx6Px7905O+iBwAEj/Wr4BYsWCCv1+tfDh48aLslAMBFENQAiouLkyTV1tYGrK+trfVv+y63262oqKiABQDQ9QU1gFJSUhQXFxfwyXWfz6edO3cqLS0tmLsCAHRyjq+CO378uCoqKvyPq6qqtHv3bkVHRyspKUlz587VP//zP+vqq69WSkqKFi1apISEBE2ZMiWYfQMAOjnHAbRr1y7ddttt/sfz5s2TJGVnZ2vVqlV68skn1dDQoJkzZ6qurk633HKLCgsL1b179+B1DQDo9FzGGGO7iTP5fD55PB7bbQDoQqZOndquujfffNNxzd69ex3XnPmP+u/r2LFjjmsuNq/Xe8739a1fBQcAuDQRQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgheOvYwAAm2JiYhzXrFixol37Cgtz/m/0xYsXO67pDHe2DgVmQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBTcjBdCp5ObmOq7p27dvu/b19ddfO64pLy9v174uRcyAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKbkYKwJqbb77Zcc2Pf/zjEHTStilTpjiu2bt3b/Ab6aKYAQEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFdyMFIA1d9xxh+Oabt26Oa7ZunWr4xpJKi0tbVcdvh9mQAAAKwggAIAVjgNo27Ztmjx5shISEuRyubRx48aA7Tk5OXK5XAHLpEmTgtUvAKCLcBxADQ0NGjVqlJYvX37WMZMmTVJ1dbV/ee211y6oSQBA1+P4IoSsrCxlZWWdc4zb7VZcXFy7mwIAdH0heQ+ouLhYMTExGjJkiGbNmqWjR4+edWxjY6N8Pl/AAgDo+oIeQJMmTdKrr76qrVu36uc//7lKSkqUlZWl5ubmNsfn5+fL4/H4l8TExGC3BADogIL+OaD777/f//OIESM0cuRIDRw4UMXFxUpPT281fsGCBZo3b57/sc/nI4QA4BIQ8suwBwwYoD59+qiioqLN7W63W1FRUQELAKDrC3kAffHFFzp69Kji4+NDvSsAQCfi+CW448ePB8xmqqqqtHv3bkVHRys6OlrPPfecpk6dqri4OFVWVurJJ5/UoEGDlJmZGdTGAQCdm+MA2rVrl2677Tb/42/fv8nOztbKlSu1Z88e/fu//7vq6uqUkJCgiRMn6vnnn5fb7Q5e1wCATs9ljDG2mziTz+eTx+Ox3QYAh3r06OG45oMPPnBcM2zYMMc1EyZMcFwjSdu3b29XHU7zer3nfF+fe8EBAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADAiqB/JTeAS9P8+fMd1/zN3/yN45rCwkLHNdzVumNiBgQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVnAzUgCt/OAHP3Bcs2jRIsc1Pp/Pcc3ixYsd16BjYgYEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFZwM1KgC+vdu3e76l566SXHNeHh4Y5rNm/e7Lhmx44djmvQMTEDAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAAruBkp0Em052afhYWF7dpXSkqK45rKykrHNYsWLXJcg66DGRAAwAoCCABghaMAys/P15gxYxQZGamYmBhNmTJF5eXlAWNOnjyp3Nxc9e7dWz179tTUqVNVW1sb1KYBAJ2fowAqKSlRbm6uduzYoffee0+nTp3SxIkT1dDQ4B/z+OOPa9OmTVq7dq1KSkp06NAh3XPPPUFvHADQuTm6COG7b2iuWrVKMTExKisr09ixY+X1evXb3/5Wa9as0YQJEyRJBQUFuuaaa7Rjxw7ddNNNwescANCpXdB7QF6vV5IUHR0tSSorK9OpU6eUkZHhHzN06FAlJSWptLS0zedobGyUz+cLWAAAXV+7A6ilpUVz587VzTffrOHDh0uSampqFBERoV69egWMjY2NVU1NTZvPk5+fL4/H418SExPb2xIAoBNpdwDl5uZq7969ev311y+ogQULFsjr9fqXgwcPXtDzAQA6h3Z9EHX27Nl65513tG3bNl111VX+9XFxcWpqalJdXV3ALKi2tlZxcXFtPpfb7Zbb7W5PGwCATszRDMgYo9mzZ2vDhg16//33W31aevTo0erWrZu2bt3qX1deXq4DBw4oLS0tOB0DALoERzOg3NxcrVmzRm+//bYiIyP97+t4PB716NFDHo9H06dP17x58xQdHa2oqCjNmTNHaWlpXAEHAAjgKIBWrlwpSRo/fnzA+oKCAuXk5EiSXnzxRYWFhWnq1KlqbGxUZmamVqxYEZRmAQBdh8sYY2w3cSafzyePx2O7DaDDGTx4sOOazz//PASdtO2uu+5yXLNp06YQdIKOwuv1Kioq6qzbuRccAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArGjXN6ICuDDJycmOa/74xz+GoJO2zZ8/33HNO++8E4JO0JUxAwIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAK7gZKWDBzJkzHdckJSWFoJO2lZSUOK4xxoSgE3RlzIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwApuRgpcoFtuucVxzZw5c0LQCdC5MAMCAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACu4GSlwgW699VbHNT179gxBJ61VVla2q+748eNB7gRojRkQAMAKAggAYIWjAMrPz9eYMWMUGRmpmJgYTZkyReXl5QFjxo8fL5fLFbA8+uijQW0aAND5OQqgkpIS5ebmaseOHXrvvfd06tQpTZw4UQ0NDQHjZsyYoerqav+yZMmSoDYNAOj8HF2EUFhYGPB41apViomJUVlZmcaOHetff/nllysuLi44HQIAuqQLeg/I6/VKkqKjowPWr169Wn369NHw4cO1YMECnThx4qzP0djYKJ/PF7AAALq+dl+G3dLSorlz5+rmm2/W8OHD/esffPBBJScnKyEhQXv27NFTTz2l8vJyrV+/vs3nyc/P13PPPdfeNgAAnVS7Ayg3N1d79+7VBx98ELB+5syZ/p9HjBih+Ph4paenq7KyUgMHDmz1PAsWLNC8efP8j30+nxITE9vbFgCgk2hXAM2ePVvvvPOOtm3bpquuuuqcY1NTUyVJFRUVbQaQ2+2W2+1uTxsAgE7MUQAZYzRnzhxt2LBBxcXFSklJOW/N7t27JUnx8fHtahAA0DU5CqDc3FytWbNGb7/9tiIjI1VTUyNJ8ng86tGjhyorK7VmzRrdcccd6t27t/bs2aPHH39cY8eO1ciRI0PyCwAAOidHAbRy5UpJpz9seqaCggLl5OQoIiJCW7Zs0dKlS9XQ0KDExERNnTpVTz/9dNAaBgB0DY5fgjuXxMRElZSUXFBDAIBLA3fDBjqJTz/91HFNenp6u/Z17NixdtUBTnAzUgCAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwwmXOd4vri8zn88nj8dhuAwBwgbxer6Kios66nRkQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwosMFUAe7NR0AoJ3O9/e8wwVQfX297RYAAEFwvr/nHe5u2C0tLTp06JAiIyPlcrkCtvl8PiUmJurgwYPnvMNqV8dxOI3jcBrH4TSOw2kd4TgYY1RfX6+EhASFhZ19nnPZRezpewkLC9NVV111zjFRUVGX9An2LY7DaRyH0zgOp3EcTrN9HL7P1+p0uJfgAACXBgIIAGBFpwogt9utvLw8ud1u261YxXE4jeNwGsfhNI7DaZ3pOHS4ixAAAJeGTjUDAgB0HQQQAMAKAggAYAUBBACwggACAFjRaQJo+fLl6t+/v7p3767U1FR99NFHtlu66J599lm5XK6AZejQobbbCrlt27Zp8uTJSkhIkMvl0saNGwO2G2P0zDPPKD4+Xj169FBGRob27dtnp9kQOt9xyMnJaXV+TJo0yU6zIZKfn68xY8YoMjJSMTExmjJlisrLywPGnDx5Urm5uerdu7d69uypqVOnqra21lLHofF9jsP48eNbnQ+PPvqopY7b1ikC6I033tC8efOUl5enjz/+WKNGjVJmZqa++uor261ddMOGDVN1dbV/+eCDD2y3FHINDQ0aNWqUli9f3ub2JUuW6KWXXtKvf/1r7dy5U1dccYUyMzN18uTJi9xpaJ3vOEjSpEmTAs6P11577SJ2GHolJSXKzc3Vjh079N577+nUqVOaOHGiGhoa/GMef/xxbdq0SWvXrlVJSYkOHTqke+65x2LXwfd9joMkzZgxI+B8WLJkiaWOz8J0AjfeeKPJzc31P25ubjYJCQkmPz/fYlcXX15enhk1apTtNqySZDZs2OB/3NLSYuLi4swvfvEL/7q6ujrjdrvNa6+9ZqHDi+O7x8EYY7Kzs81dd91lpR9bvvrqKyPJlJSUGGNO/7fv1q2bWbt2rX/Mn/70JyPJlJaW2moz5L57HIwxZty4ceYf//Ef7TX1PXT4GVBTU5PKysqUkZHhXxcWFqaMjAyVlpZa7MyOffv2KSEhQQMGDNBDDz2kAwcO2G7JqqqqKtXU1AScHx6PR6mpqZfk+VFcXKyYmBgNGTJEs2bN0tGjR223FFJer1eSFB0dLUkqKyvTqVOnAs6HoUOHKikpqUufD989Dt9avXq1+vTpo+HDh2vBggU6ceKEjfbOqsPdDfu7jhw5oubmZsXGxgasj42N1eeff26pKztSU1O1atUqDRkyRNXV1Xruued06623au/evYqMjLTdnhU1NTWS1Ob58e22S8WkSZN0zz33KCUlRZWVlVq4cKGysrJUWlqq8PBw2+0FXUtLi+bOnaubb75Zw4cPl3T6fIiIiFCvXr0Cxnbl86Gt4yBJDz74oJKTk5WQkKA9e/boqaeeUnl5udavX2+x20AdPoDw/7Kysvw/jxw5UqmpqUpOTtabb76p6dOnW+wMHcH999/v/3nEiBEaOXKkBg4cqOLiYqWnp1vsLDRyc3O1d+/eS+J90HM523GYOXOm/+cRI0YoPj5e6enpqqys1MCBAy92m23q8C/B9enTR+Hh4a2uYqmtrVVcXJylrjqGXr16afDgwaqoqLDdijXfngOcH60NGDBAffr06ZLnx+zZs/XOO++oqKgo4PvD4uLi1NTUpLq6uoDxXfV8ONtxaEtqaqokdajzocMHUEREhEaPHq2tW7f617W0tGjr1q1KS0uz2Jl9x48fV2VlpeLj4223Yk1KSori4uICzg+fz6edO3de8ufHF198oaNHj3ap88MYo9mzZ2vDhg16//33lZKSErB99OjR6tatW8D5UF5ergMHDnSp8+F8x6Etu3fvlqSOdT7Yvgri+3j99deN2+02q1atMv/zP/9jZs6caXr16mVqampst3ZR/dM//ZMpLi42VVVV5sMPPzQZGRmmT58+5quvvrLdWkjV19ebTz75xHzyySdGknnhhRfMJ598Yv76178aY4z52c9+Znr16mXefvtts2fPHnPXXXeZlJQU880331juPLjOdRzq6+vNE088YUpLS01VVZXZsmWLuf76683VV19tTp48abv1oJk1a5bxeDymuLjYVFdX+5cTJ074xzz66KMmKSnJvP/++2bXrl0mLS3NpKWlWew6+M53HCoqKszixYvNrl27TFVVlXn77bfNgAEDzNixYy13HqhTBJAxxvzqV78ySUlJJiIiwtx4441mx44dtlu66O677z4THx9vIiIiTL9+/cx9991nKioqbLcVckVFRUZSqyU7O9sYc/pS7EWLFpnY2FjjdrtNenq6KS8vt9t0CJzrOJw4ccJMnDjR9O3b13Tr1s0kJyebGTNmdLl/pLX1+0syBQUF/jHffPONeeyxx8yVV15pLr/8cnP33Xeb6upqe02HwPmOw4EDB8zYsWNNdHS0cbvdZtCgQWb+/PnG6/Xabfw7+D4gAIAVHf49IABA10QAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFb8H4WZmF1Ue9WVAAAAAElFTkSuQmCC",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "accuracy:  0.9513999819755554 accuracy after augmentation: 0.49149999022483826\n"
          ]
        }
      ],
      "source": [
        "def shift_pixels(t):\n",
        "    shift = 3\n",
        "    return torch.roll(t, shift)\n",
        "\n",
        "# let's see how the network generalizes\n",
        "augmented_mnist_validation = datasets.MNIST(data_path, train=False,\n",
        "                                              download=False,\n",
        "                                              transform=transforms.Compose([\n",
        "                                              transforms.ToTensor(),\n",
        "                                              transforms.Lambda(shift_pixels),\n",
        "                                              normalize,]))\n",
        "augmented_val_loader = torch.utils.data.DataLoader(augmented_mnist_validation,\n",
        "                                         batch_size=64,\n",
        "                                         shuffle=False)\n",
        "\n",
        "images, labels = augmented_mnist_validation[0]\n",
        "display_image(images, labels)\n",
        "\n",
        "val_loss_augmented, accuracy_augmented = valid_epoch(net, augmented_val_loader, loss_fn)\n",
        "\n",
        "print(\"accuracy: \", accuracy, \"accuracy after augmentation:\", accuracy_augmented)"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.8"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}