03_pytorch_computer_vision.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c2dc16c4-c7f4-4945-ba91-6430a51e6f5a",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/mrdbourke/pytorch-deep-learning/blob/main/03_pytorch_computer_vision.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "\n",
    "[View Source Code](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/03_pytorch_computer_vision.ipynb) | [View Slides](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/slides/03_pytorch_computer_vision.pdf) | [Watch Video Walkthrough](https://youtu.be/Z_ikDlimN6A?t=50417) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08f47c6a-3318-4e3f-8bb3-c520e00e63dd",
   "metadata": {},
   "source": [
    "# 03. PyTorch Computer Vision\n",
    "\n",
    "[Computer vision](https://en.wikipedia.org/wiki/Computer_vision) is the art of teaching a computer to see.\n",
    "\n",
    "For example, it could involve building a model to classify whether a photo is of a cat or a dog ([binary classification](https://developers.google.com/machine-learning/glossary#binary-classification)).\n",
    "\n",
    "Or whether a photo is of a cat, dog or chicken ([multi-class classification](https://developers.google.com/machine-learning/glossary#multi-class-classification)).\n",
    "\n",
    "Or identifying where a car appears in a video frame ([object detection](https://en.wikipedia.org/wiki/Object_detection)).\n",
    "\n",
    "Or figuring out where different objects in an image can be separated ([panoptic segmentation](https://arxiv.org/abs/1801.00868)).\n",
    "\n",
    "![example computer vision problems](https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/03-computer-vision-problems.png)\n",
    "*Example computer vision problems for binary classification, multiclass classification, object detection and segmentation.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19179a39-0c6c-40f7-9891-09e17d107ecf",
   "metadata": {},
   "source": [
    "## Where does computer vision get used?\n",
    "\n",
    "If you use a smartphone, you've already used computer vision.\n",
    "\n",
    "Camera and photo apps use [computer vision to enhance](https://machinelearning.apple.com/research/panoptic-segmentation) and sort images.\n",
    "\n",
    "Modern cars use [computer vision](https://youtu.be/j0z4FweCy4M?t=2989) to avoid other cars and stay within lane lines.\n",
    "\n",
    "Manufacturers use computer vision to identify defects in various products.\n",
    "\n",
    "Security cameras use computer vision to detect potential intruders.\n",
    "\n",
    "In essence, anything that can described in a visual sense can be a potential computer vision problem."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "412e8bd1-0e6b-4ad6-8506-b28a8f669dc1",
   "metadata": {},
   "source": [
    "## What we're going to cover\n",
    "\n",
    "We're going to apply the PyTorch Workflow we've been learning in the past couple of sections to computer vision.\n",
    "\n",
    "![a PyTorch workflow with a computer vision focus](https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/03-pytorch-computer-vision-workflow.png)\n",
    "\n",
    "Specifically, we're going to cover:\n",
    "\n",
    "| **Topic** | **Contents** |\n",
    "| ----- | ----- |\n",
    "| **0. Computer vision libraries in PyTorch** | PyTorch has a bunch of built-in helpful computer vision libraries, let's check them out.  |\n",
    "| **1. Load data** | To practice computer vision, we'll start with some images of different pieces of clothing from [FashionMNIST](https://github.com/zalandoresearch/fashion-mnist). |\n",
    "| **2. Prepare data** | We've got some images, let's load them in with a [PyTorch `DataLoader`](https://pytorch.org/docs/stable/data.html) so we can use them with our training loop. |\n",
    "| **3. Model 0: Building a baseline model** | Here we'll create a multi-class classification model to learn patterns in the data, we'll also choose a **loss function**, **optimizer** and build a **training loop**. | \n",
    "| **4. Making predictions and evaluting model 0** | Let's make some predictions with our baseline model and evaluate them. |\n",
    "| **5. Setup device agnostic code for future models** | It's best practice to write device-agnostic code, so let's set it up. |\n",
    "| **6. Model 1: Adding non-linearity** | Experimenting is a large part of machine learning, let's try and improve upon our baseline model by adding non-linear layers. |\n",
    "| **7. Model 2: Convolutional Neural Network (CNN)** | Time to get computer vision specific and introduce the powerful convolutional neural network architecture. |\n",
    "| **8. Comparing our models** | We've built three different models, let's compare them. |\n",
    "| **9. Evaluating our best model** | Let's make some predictons on random images and evaluate our best model. |\n",
    "| **10. Making a confusion matrix** | A confusion matrix is a great way to evaluate a classification model, let's see how we can make one. |\n",
    "| **11. Saving and loading the best performing model** | Since we might want to use our model for later, let's save it and make sure it loads back in correctly. |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cddf62c3-f5e5-4f7e-852a-2ad6d38b7399",
   "metadata": {},
   "source": [
    "## Where can can you get help?\n",
    "\n",
    "All of the materials for this course [live on GitHub](https://github.com/mrdbourke/pytorch-deep-learning).\n",
    "\n",
    "If you run into trouble, you can ask a question on the course [GitHub Discussions page](https://github.com/mrdbourke/pytorch-deep-learning/discussions) there too.\n",
    "\n",
    "And of course, there's the [PyTorch documentation](https://pytorch.org/docs/stable/index.html) and [PyTorch developer forums](https://discuss.pytorch.org/), a very helpful place for all things PyTorch. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0bedcfc-e12a-4a81-9913-84c6a888742a",
   "metadata": {},
   "source": [
    "## 0. Computer vision libraries in PyTorch\n",
    "\n",
    "Before we get started writing code, let's talk about some PyTorch computer vision libraries you should be aware of.\n",
    "\n",
    "| PyTorch module | What does it do? |\n",
    "| ----- | ----- |\n",
    "| [`torchvision`](https://pytorch.org/vision/stable/index.html) | Contains datasets, model architectures and image transformations often used for computer vision problems. |\n",
    "| [`torchvision.datasets`](https://pytorch.org/vision/stable/datasets.html) | Here you'll find many example computer vision datasets for a range of problems from image classification, object detection, image captioning, video classification and more. It also contains [a series of base classes for making custom datasets](https://pytorch.org/vision/stable/datasets.html#base-classes-for-custom-datasets). |\n",
    "| [`torchvision.models`](https://pytorch.org/vision/stable/models.html) | This module contains well-performing and commonly used computer vision model architectures implemented in PyTorch, you can use these with your own problems. | \n",
    "| [`torchvision.transforms`](https://pytorch.org/vision/stable/transforms.html) | Often images need to be transformed (turned into numbers/processed/augmented) before being used with a model, common image transformations are found here. | \n",
    "| [`torch.utils.data.Dataset`](https://pytorch.org/docs/stable/data.html#torch.utils.data.Dataset) | Base dataset class for PyTorch.  | \n",
    "| [`torch.utils.data.DataLoader`](https://pytorch.org/docs/stable/data.html#module-torch.utils.data) | Creates a Python iteralbe over a dataset (created with `torch.utils.data.Dataset`). |\n",
    "\n",
    "> **Note:** The `torch.utils.data.Dataset` and `torch.utils.data.DataLoader` classes aren't only for computer vision in PyTorch, they are capable of dealing with many different types of data.\n",
    "\n",
    "Now we've covered some of the most important PyTorch computer vision libraries, let's import the relevant dependencies.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c263a60d-d788-482f-b9e7-9cab4f6b1f72",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PyTorch version: 1.12.1+cu113\n",
      "torchvision version: 0.13.1+cu113\n"
     ]
    }
   ],
   "source": [
    "# Import PyTorch\n",
    "import torch\n",
    "from torch import nn\n",
    "\n",
    "# Import torchvision \n",
    "import torchvision\n",
    "from torchvision import datasets\n",
    "from torchvision.transforms import ToTensor\n",
    "\n",
    "# Import matplotlib for visualization\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Check versions\n",
    "# Note: your PyTorch version shouldn't be lower than 1.10.0 and torchvision version shouldn't be lower than 0.11\n",
    "print(f\"PyTorch version: {torch.__version__}\\ntorchvision version: {torchvision.__version__}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48d6bfe7-91da-44eb-9ab6-7c41c1e9fa8e",
   "metadata": {},
   "source": [
    "## 1. Getting a dataset\n",
    "\n",
    "To begin working on a computer vision problem, let's get a computer vision dataset.\n",
    "\n",
    "We're going to start with FashionMNIST.\n",
    "\n",
    "MNIST stands for Modified National Institute of Standards and Technology.\n",
    "\n",
    "The [original MNIST dataset](https://en.wikipedia.org/wiki/MNIST_database) contains thousands of examples of handwritten digits (from 0 to 9) and was used to build computer vision models to identify numbers for postal services.\n",
    "\n",
    "[FashionMNIST](https://github.com/zalandoresearch/fashion-mnist), made by Zalando Research, is a similar setup. \n",
    "\n",
    "Except it contains grayscale images of 10 different kinds of clothing.\n",
    "\n",
    "![example image of FashionMNIST](https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/03-fashion-mnist-slide.png)\n",
    "*`torchvision.datasets` contains a lot of example datasets you can use to practice writing computer vision code on. FashionMNIST is one of those datasets. And since it has 10 different image classes (different types of clothing), it's a multi-class classification problem.*\n",
    "\n",
    "Later, we'll be building a computer vision neural network to identify the different styles of clothing in these images.\n",
    "\n",
    "PyTorch has a bunch of common computer vision datasets stored in `torchvision.datasets`.\n",
    "\n",
    "Including FashionMNIST in [`torchvision.datasets.FashionMNIST()`](https://pytorch.org/vision/main/generated/torchvision.datasets.FashionMNIST.html).\n",
    "\n",
    "To download it, we provide the following parameters:\n",
    "* `root: str` - which folder do you want to download the data to?\n",
    "* `train: Bool` - do you want the training or test split?\n",
    "* `download: Bool` - should the data be downloaded?\n",
    "* `transform: torchvision.transforms` - what transformations would you like to do on the data?\n",
    "* `target_transform` - you can transform the targets (labels) if you like too.\n",
    "\n",
    "Many other datasets in `torchvision` have these parameter options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "486f8377-6810-4367-859d-69dccc7aef95",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup training data\n",
    "train_data = datasets.FashionMNIST(\n",
    "    root=\"data\", # where to download data to?\n",
    "    train=True, # get training data\n",
    "    download=True, # download data if it doesn't exist on disk\n",
    "    transform=ToTensor(), # images come as PIL format, we want to turn into Torch tensors\n",
    "    target_transform=None # you can transform labels as well\n",
    ")\n",
    "\n",
    "# Setup testing data\n",
    "test_data = datasets.FashionMNIST(\n",
    "    root=\"data\",\n",
    "    train=False, # get test data\n",
    "    download=True,\n",
    "    transform=ToTensor()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a63246f6-3645-49de-88fe-ec18e78bfbaf",
   "metadata": {},
   "source": [
    "Let's check out the first sample of the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "43bfd3d9-a132-41e8-8ccd-5ae25a7da59a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000, 0.0039, 0.0000, 0.0000, 0.0510,\n",
       "           0.2863, 0.0000, 0.0000, 0.0039, 0.0157, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0039, 0.0039, 0.0000],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000, 0.0118, 0.0000, 0.1412, 0.5333,\n",
       "           0.4980, 0.2431, 0.2118, 0.0000, 0.0000, 0.0000, 0.0039, 0.0118,\n",
       "           0.0157, 0.0000, 0.0000, 0.0118],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000, 0.0235, 0.0000, 0.4000, 0.8000,\n",
       "           0.6902, 0.5255, 0.5647, 0.4824, 0.0902, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0471, 0.0392, 0.0000],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.6078, 0.9255,\n",
       "           0.8118, 0.6980, 0.4196, 0.6118, 0.6314, 0.4275, 0.2510, 0.0902,\n",
       "           0.3020, 0.5098, 0.2824, 0.0588],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0039, 0.0000, 0.2706, 0.8118, 0.8745,\n",
       "           0.8549, 0.8471, 0.8471, 0.6392, 0.4980, 0.4745, 0.4784, 0.5725,\n",
       "           0.5529, 0.3451, 0.6745, 0.2588],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0039, 0.0039, 0.0039, 0.0000, 0.7843, 0.9098, 0.9098,\n",
       "           0.9137, 0.8980, 0.8745, 0.8745, 0.8431, 0.8353, 0.6431, 0.4980,\n",
       "           0.4824, 0.7686, 0.8980, 0.0000],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.7176, 0.8824, 0.8471,\n",
       "           0.8745, 0.8941, 0.9216, 0.8902, 0.8784, 0.8706, 0.8784, 0.8667,\n",
       "           0.8745, 0.9608, 0.6784, 0.0000],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.7569, 0.8941, 0.8549,\n",
       "           0.8353, 0.7765, 0.7059, 0.8314, 0.8235, 0.8275, 0.8353, 0.8745,\n",
       "           0.8627, 0.9529, 0.7922, 0.0000],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0039, 0.0118, 0.0000, 0.0471, 0.8588, 0.8627, 0.8314,\n",
       "           0.8549, 0.7529, 0.6627, 0.8902, 0.8157, 0.8549, 0.8784, 0.8314,\n",
       "           0.8863, 0.7725, 0.8196, 0.2039],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0235, 0.0000, 0.3882, 0.9569, 0.8706, 0.8627,\n",
       "           0.8549, 0.7961, 0.7765, 0.8667, 0.8431, 0.8353, 0.8706, 0.8627,\n",
       "           0.9608, 0.4667, 0.6549, 0.2196],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0157, 0.0000, 0.0000, 0.2157, 0.9255, 0.8941, 0.9020,\n",
       "           0.8941, 0.9412, 0.9098, 0.8353, 0.8549, 0.8745, 0.9176, 0.8510,\n",
       "           0.8510, 0.8196, 0.3608, 0.0000],\n",
       "          [0.0000, 0.0000, 0.0039, 0.0157, 0.0235, 0.0275, 0.0078, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000, 0.9294, 0.8863, 0.8510, 0.8745,\n",
       "           0.8706, 0.8588, 0.8706, 0.8667, 0.8471, 0.8745, 0.8980, 0.8431,\n",
       "           0.8549, 1.0000, 0.3020, 0.0000],\n",
       "          [0.0000, 0.0118, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.2431, 0.5686, 0.8000, 0.8941, 0.8118, 0.8353, 0.8667,\n",
       "           0.8549, 0.8157, 0.8275, 0.8549, 0.8784, 0.8745, 0.8588, 0.8431,\n",
       "           0.8784, 0.9569, 0.6235, 0.0000],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 0.0706, 0.1725, 0.3216, 0.4196,\n",
       "           0.7412, 0.8941, 0.8627, 0.8706, 0.8510, 0.8863, 0.7843, 0.8039,\n",
       "           0.8275, 0.9020, 0.8784, 0.9176, 0.6902, 0.7373, 0.9804, 0.9725,\n",
       "           0.9137, 0.9333, 0.8431, 0.0000],\n",
       "          [0.0000, 0.2235, 0.7333, 0.8157, 0.8784, 0.8667, 0.8784, 0.8157,\n",
       "           0.8000, 0.8392, 0.8157, 0.8196, 0.7843, 0.6235, 0.9608, 0.7569,\n",
       "           0.8078, 0.8745, 1.0000, 1.0000, 0.8667, 0.9176, 0.8667, 0.8275,\n",
       "           0.8627, 0.9098, 0.9647, 0.0000],\n",
       "          [0.0118, 0.7922, 0.8941, 0.8784, 0.8667, 0.8275, 0.8275, 0.8392,\n",
       "           0.8039, 0.8039, 0.8039, 0.8627, 0.9412, 0.3137, 0.5882, 1.0000,\n",
       "           0.8980, 0.8667, 0.7373, 0.6039, 0.7490, 0.8235, 0.8000, 0.8196,\n",
       "           0.8706, 0.8941, 0.8824, 0.0000],\n",
       "          [0.3843, 0.9137, 0.7765, 0.8235, 0.8706, 0.8980, 0.8980, 0.9176,\n",
       "           0.9765, 0.8627, 0.7608, 0.8431, 0.8510, 0.9451, 0.2549, 0.2863,\n",
       "           0.4157, 0.4588, 0.6588, 0.8588, 0.8667, 0.8431, 0.8510, 0.8745,\n",
       "           0.8745, 0.8784, 0.8980, 0.1137],\n",
       "          [0.2941, 0.8000, 0.8314, 0.8000, 0.7569, 0.8039, 0.8275, 0.8824,\n",
       "           0.8471, 0.7255, 0.7725, 0.8078, 0.7765, 0.8353, 0.9412, 0.7647,\n",
       "           0.8902, 0.9608, 0.9373, 0.8745, 0.8549, 0.8314, 0.8196, 0.8706,\n",
       "           0.8627, 0.8667, 0.9020, 0.2627],\n",
       "          [0.1882, 0.7961, 0.7176, 0.7608, 0.8353, 0.7725, 0.7255, 0.7451,\n",
       "           0.7608, 0.7529, 0.7922, 0.8392, 0.8588, 0.8667, 0.8627, 0.9255,\n",
       "           0.8824, 0.8471, 0.7804, 0.8078, 0.7294, 0.7098, 0.6941, 0.6745,\n",
       "           0.7098, 0.8039, 0.8078, 0.4510],\n",
       "          [0.0000, 0.4784, 0.8588, 0.7569, 0.7020, 0.6706, 0.7176, 0.7686,\n",
       "           0.8000, 0.8235, 0.8353, 0.8118, 0.8275, 0.8235, 0.7843, 0.7686,\n",
       "           0.7608, 0.7490, 0.7647, 0.7490, 0.7765, 0.7529, 0.6902, 0.6118,\n",
       "           0.6549, 0.6941, 0.8235, 0.3608],\n",
       "          [0.0000, 0.0000, 0.2902, 0.7412, 0.8314, 0.7490, 0.6863, 0.6745,\n",
       "           0.6863, 0.7098, 0.7255, 0.7373, 0.7412, 0.7373, 0.7569, 0.7765,\n",
       "           0.8000, 0.8196, 0.8235, 0.8235, 0.8275, 0.7373, 0.7373, 0.7608,\n",
       "           0.7529, 0.8471, 0.6667, 0.0000],\n",
       "          [0.0078, 0.0000, 0.0000, 0.0000, 0.2588, 0.7843, 0.8706, 0.9294,\n",
       "           0.9373, 0.9490, 0.9647, 0.9529, 0.9569, 0.8667, 0.8627, 0.7569,\n",
       "           0.7490, 0.7020, 0.7137, 0.7137, 0.7098, 0.6902, 0.6510, 0.6588,\n",
       "           0.3882, 0.2275, 0.0000, 0.0000],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.1569,\n",
       "           0.2392, 0.1725, 0.2824, 0.1608, 0.1373, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "           0.0000, 0.0000, 0.0000, 0.0000]]]),\n",
       " 9)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# See first training sample\n",
    "image, label = train_data[0]\n",
    "image, label"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ad9d782-06cb-4591-ae3c-3a8b2389a1b2",
   "metadata": {},
   "source": [
    "### 1.1 Input and output shapes of a computer vision model\n",
    "\n",
    "We've got a big tensor of values (the image) leading to a single value for the target (the label).\n",
    "\n",
    "Let's see the image shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c2997d9f-b574-4d23-aa34-1a4df1751226",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 28, 28])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# What's the shape of the image?\n",
    "image.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5326a05-f807-448d-99a3-6d03fc8739f8",
   "metadata": {},
   "source": [
    "The shape of the image tensor is `[1, 28, 28]` or more specifically:\n",
    "\n",
    "```\n",
    "[color_channels=1, height=28, width=28]\n",
    "```\n",
    "\n",
    "Having `color_channels=1` means the image is grayscale.\n",
    "\n",
    "![example input and output shapes of the fashionMNIST problem](https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/03-computer-vision-input-and-output-shapes.png)\n",
    "*Various problems will have various input and output shapes. But the premise reamins: encode data into numbers, build a model to find patterns in those numbers, convert those patterns into something meaningful.*\n",
    "\n",
    "If `color_channels=3`, the image comes in pixel values for red, green and blue (this is also known a the [RGB color model](https://en.wikipedia.org/wiki/RGB_color_model)).\n",
    "\n",
    "The order of our current tensor is often referred to as `CHW` (Color Channels, Height, Width).\n",
    "\n",
    "There's debate on whether images should be represented as `CHW` (color channels first) or `HWC` (color channels last).\n",
    "\n",
    "> **Note:** You'll also see `NCHW` and `NHWC` formats where `N` stands for *number of images*. For example if you have a `batch_size=32`, your tensor shape may be `[32, 1, 28, 28]`. We'll cover batch sizes later.\n",
    "\n",
    "PyTorch generally accepts `NCHW` (channels first) as the default for many operators.\n",
    "\n",
    "However, PyTorch also explains that `NHWC` (channels last) performs better and is [considered best practice](https://pytorch.org/blog/tensor-memory-format-matters/#pytorch-best-practice). \n",
    "\n",
    "For now, since our dataset and models are relatively small, this won't make too much of a difference.\n",
    "\n",
    "But keep it in mind for when you're working on larger image datasets and using convolutional neural networks (we'll see these later).\n",
    "\n",
    "Let's check out more shapes of our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "fc4f768c-c3f6-454d-a633-673ad1d6eca0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 60000, 10000, 10000)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# How many samples are there? \n",
    "len(train_data.data), len(train_data.targets), len(test_data.data), len(test_data.targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e0267d5-946b-4c53-af69-61acd3527972",
   "metadata": {},
   "source": [
    "So we've got 60,000 training samples and 10,000 testing samples.\n",
    "\n",
    "What classes are there?\n",
    "\n",
    "We can find these via the `.classes` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e22849c6-d93f-4b38-8403-5ebf0deaf008",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['T-shirt/top',\n",
       " 'Trouser',\n",
       " 'Pullover',\n",
       " 'Dress',\n",
       " 'Coat',\n",
       " 'Sandal',\n",
       " 'Shirt',\n",
       " 'Sneaker',\n",
       " 'Bag',\n",
       " 'Ankle boot']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# See classes\n",
    "class_names = train_data.classes\n",
    "class_names"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abdd225c-5742-4d9c-8e8d-fb30a9c3cb6e",
   "metadata": {},
   "source": [
    "Sweet! It looks like we're dealing with 10 different kinds of clothes.\n",
    "\n",
    "Because we're working with 10 different classes, it means our problem is **multi-class classification**.\n",
    "\n",
    "Let's get visual."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb625d80-6a98-471e-a758-4de0ce0f3a64",
   "metadata": {},
   "source": [
    "### 1.2 Visualizing our data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b1df1f2c-28c9-43bf-aaef-cf996c9ae1c5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image shape: torch.Size([1, 28, 28])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAEICAYAAACZA4KlAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAU80lEQVR4nO3de2yc1ZkG8OeZ8fgSYydxEpwQXMK1DYUSWjehUAFtCoVotYFSEAixVEIbtEu72253BaKtyv6zQtXSCrW73abAEnYL3VYFQRGiQChlgZLGhJQEUgiXQEISxyFgO4kvM553//B06waf9zMz4/kGzvOTLNvz+niOZ+bxNzPnO+fQzCAiH3yZtDsgIrWhsItEQmEXiYTCLhIJhV0kEgq7SCQUdpFIKOwyKZKLST5Ksp/kyyQvTLtPUhmFXd6FZAOAewHcD6ADwCoA/03yhFQ7JhWhzqCTQ5E8CcDTANqs9AAh+RCAdWb2rVQ7J2XTkV0mw8BlJ9W6I1I9CrtM5g8A9gD4J5I5kucCOAvAjHS7JZXQ03iZFMmPAfg+xo/mPQD6AIyY2VWpdkzKprDLlJB8CsAaM/tR2n2R8uhpvEyK5MdINpOcQfIfASwAcHvK3ZIKKOwScgWAXRh/7b4cwDlmNpJul6QSehovEgkd2UUiobCLREJhF4mEwi4SiYZaXlkjm6wZrbW8SpGoDOMARm1kstOdKws7yfMA3AwgC+AWM7vR+/lmtGIZl1dylSLiWGdrg7Wyn8aTzAL4NwDnAzgRwGUkTyz394nI9KrkNftSAC+b2atmNgrgpwBWVqdbIlJtlYR9IYDtE77fUbrsz5BcRbKHZE8eOgFLJC2VhH2yNwHedTqema02s24z686hqYKrE5FKVBL2HQC6Jnx/JICdlXVHRKZLJWFfD+B4kkeTbARwKYD7qtMtEam2sofezKxA8ssAfoXxobfbzOz5qvVMRKqqonF2M3sAwANV6ouITCOdLisSCYVdJBIKu0gkFHaRSCjsIpFQ2EUiobCLREJhF4mEwi4SCYVdJBIKu0gkFHaRSCjsIpGo6VLSkgJOuqrwn1S41192Todbf/vzJwRr7Xc+XdF1J/1tbMgFa5Yfrey6K5V0v3jKvM90ZBeJhMIuEgmFXSQSCrtIJBR2kUgo7CKRUNhFIqFx9g84ZrNu3QoFt55Z4u/VueXqw/z2Q+Fa7sBSt23DUNGt5x7qcesVjaUnjeEn3K6gfxytpG9scGLr3J06sotEQmEXiYTCLhIJhV0kEgq7SCQUdpFIKOwikdA4+wecOyaL5HH27Z+f5dYv/9T/uvUn+44J1l5vmu+2tRa3jIbPfcqtn/DvbwZrhW1v+L88Yc540u2WJDt7drg4Nua2HRsYCBedblcUdpLbAAwCGANQMLPuSn6fiEyfahzZP2Nme6vwe0RkGuk1u0gkKg27AXiI5DMkV032AyRXkewh2ZPHSIVXJyLlqvRp/BlmtpPk4QAeJvkHM3t84g+Y2WoAqwGgnR2VrW4oImWr6MhuZjtLn/cAuAeAP41JRFJTdthJtpJs++PXAM4FsLlaHROR6qrkaXwngHs4Pu+3AcCdZvZgVXolVVMcHq6o/eip+936F2f6c8qbM/lg7TcZf776m492ufWxj/l9e/27bcFa8dnT3bZzNvtj3e3P7nLre89c6Nb7PhF+RduZsJz+7EdeCda4LxzpssNuZq8COKXc9iJSWxp6E4mEwi4SCYVdJBIKu0gkFHaRSNAq3LL3vWhnhy3j8ppdXzS8ZY8T7t/9l5zm1s//5mNufXHzTrc+WGwO1katshM4f/DiWW79wKszg7XMaMKWyQnlsU5/KWjL+8fR2RvCf3vLyl63LX88L1h7bu3N2L9v+6S915FdJBIKu0gkFHaRSCjsIpFQ2EUiobCLREJhF4mExtnrQcL2wBVJuH9Pesb/f/+F2f4U1iRZZ23jA9botn1nrLWi6+4rhKe45hPG+G/Z6k+B3e+M4QNApuDfp+d85tlg7aKO9W7b7xx7crC2ztZiwPZpnF0kZgq7SCQUdpFIKOwikVDYRSKhsItEQmEXiYS2bK4HNTzX4VBb9x/u1t9qP8yt7y7McutzsuHlntsyQ27bRTl/v9C+sfA4OgBkc+Glqkct67b954/+0q0PL8659Rz9pahPd9YBuPiFv3LbtuJVtx6iI7tIJBR2kUgo7CKRUNhFIqGwi0RCYReJhMIuEgmNs0duXpO/7XEzw1suA0AjC259Z352sLZ16MNu25cG/HMAzut83q3nnbF0b549kDxOfkTubbc+bP44vHerntHpj6NvdKthiUd2kreR3ENy84TLOkg+THJr6XP4HhWRujCVp/G3AzjvkMuuA7DWzI4HsLb0vYjUscSwm9njAPYdcvFKAGtKX68BcEF1uyUi1VbuG3SdZrYLAEqfgy+uSK4i2UOyJ4+RMq9ORCo17e/Gm9lqM+s2s+4cmqb76kQkoNyw95JcAAClz3uq1yURmQ7lhv0+AFeWvr4SwL3V6Y6ITJfEcXaSdwE4G8BckjsAfBvAjQB+RvIqAG8AuHg6O/mBl7BuPLP+3GsrhMe6s7P9UdGzZm1y631j7W79nbEZbn1W9mCwNlgI790OAPuG/N/9kaZdbn3DwUXB2rxGf5zc6zcAbBud69aPb9rt1r/TG94/oav50PfD/1xh+ZnBmq37bbCWGHYzuyxQ0m4PIu8jOl1WJBIKu0gkFHaRSCjsIpFQ2EUioSmu9SBhKWk2+HeTN/S2/arFbtvPzvCXTH5qeKFbn9cw6Na9aaYLmvrdtm2dw249adivoyE8fXdwrMVtOyPjn9qd9Hd/vNFfBvtrj3w8WGs76S23bXvOOUY7o7g6sotEQmEXiYTCLhIJhV0kEgq7SCQUdpFIKOwikdA4ex1grtGtF4f98WbP3E2jbn3vmL/k8ayMP9WzMWHJZW9r5NM7XnPb9iWMhW8YOtqtt2XDW0LPy/jj5F05f6x703CXW3/gwHFu/aq/eCRYu2v1OW7bxgefCtZo4ftLR3aRSCjsIpFQ2EUiobCLREJhF4mEwi4SCYVdJBLvr3F2Z8llNvjjxcwm/F/L+PXisDO/ueiPNSexvD8WXombf/QDt769MMut78779aQll8ecCdZPD8102zZn/O2i5zUMuPWBoj9O7xks+stce/P0geS+Xztna7B2d//n3Lbl0pFdJBIKu0gkFHaRSCjsIpFQ2EUiobCLREJhF4lEXY2zV7I+etJYtfnDnqkaWrnUrW+/wB/Hv/zU3wVruwttbttnnW2NAWCmMyccAFoT1lcftvD5DztH/e2kk8aqvXXhAeBwZxx+zPzj3Jt5v29Jks4/2FFw1rT/S3+u/aw7yupS8pGd5G0k95DcPOGyG0i+SXJj6WNFeVcvIrUylafxtwM4b5LLv2dmS0ofD1S3WyJSbYlhN7PHAeyrQV9EZBpV8gbdl0k+V3qaH3yBQ3IVyR6SPXn4r+9EZPqUG/YfAjgWwBIAuwDcFPpBM1ttZt1m1p1DU5lXJyKVKivsZtZrZmNmVgTwYwD+28kikrqywk5ywYRvLwSwOfSzIlIfEsfZSd4F4GwAc0nuAPBtAGeTXALAAGwDcHU1OuONo1eqYcF8t54/utOt71sc3gv84HxnU2wAS1Zscetf6vxPt9431u7Wc3T2Z8/PcdueOmObW3+0/0S3vrfhMLfujdOf3hqe0w0A7xT9/dePaHjbrV/78heDtc4Z/lj2LUf5A0x5K7r1F/P+S9b+Yng+/N+d+Gu37T2Y59ZDEsNuZpdNcvGtZV2biKRGp8uKREJhF4mEwi4SCYVdJBIKu0gk6mqK68j5n3Trh3/j1WBtSfsOt+2JLU+49eGivxS1N93yhaGFbtuDRX9L5q2j/rBgf8EfgsoyPAy0Z9Sf4nrTa/6yxWuX/odb/+bOyeZI/UmmxYK1t8b8YbuLDvOXigb8++zqDz0erB3TuMdte/+BBW59Z8IU2M5cv1tflOsL1r7Q9pLbttyhNx3ZRSKhsItEQmEXiYTCLhIJhV0kEgq7SCQUdpFI1Hacnf5y0cv+Zb3bfHnb88HaQfOnFCaNoyeNm3pmNvjLBo/k/Zt5T96fwprkhKbdwdqF7Rvdto//YJlb//TwV9z6K5/1p+euHQpP5ewr+H/3pa991q1veKPLrZ+26LVg7eS2N922Sec2tGWH3bo37RgADhTDj9enh/3zD8qlI7tIJBR2kUgo7CKRUNhFIqGwi0RCYReJhMIuEgmahecbV1vL/C479op/CNZXX/N9t/2d+04L1rqa/e3ojmrc69bnZP3tfz1tGX/M9cM5f8z1/gNHuvXH3vmIW/9E27ZgLUd/u+ezZ7zs1r/0ta+79UKzv4z2wKLw8aTQ6j/22k95y61/5bhH3Xqj87e/M+aPoyfdbklbMifx1iBoy/jbZN+04sJg7bfbbkf/0K5J7xQd2UUiobCLREJhF4mEwi4SCYVdJBIKu0gkFHaRSExly+YuAHcAmA+gCGC1md1MsgPA/wBYhPFtmy8xM3cP3UwemNEbHl+8f2CJ25djWsJrbe/N++uj/2r/yW79yBZ/+19v6+HjnPnkALBxeJZbf7Dvo279iBZ//fTe/Mxg7a18q9v2oDOvGgBu/d533fpNvf668xd2bAjWTmn0x9HfKfrHohcS1tsfLDYHa8Pmr2/QnzAO3+Y8HgAgb360ss6Wz7My/hj+wMnhbbjHesPXO5UjewHA181sMYDTAFxD8kQA1wFYa2bHA1hb+l5E6lRi2M1sl5ltKH09CGALgIUAVgJYU/qxNQAumKY+ikgVvKfX7CQXATgVwDoAnWa2Cxj/hwDg8Kr3TkSqZsphJ3kYgF8A+KqZJW3CNbHdKpI9JHsKIwfK6aOIVMGUwk4yh/Gg/8TM7i5d3EtyQam+AMCkO+WZ2Woz6zaz7oYm/80iEZk+iWEnSQC3AthiZhPfmr0PwJWlr68EcG/1uyci1TKVpaTPAHAFgE0kN5Yuux7AjQB+RvIqAG8AuDjpF2VHi2jbPhKsF82fLvno3vBUz87mQbftkrbtbv3Fg/4wzqahI4K1DQ0fctu2ZMPbPQPAzEZ/imxrQ/g2A4C5ufDffnSTvzWxNw0UANYP+3/b38x7zK2/UQgv0f3LAye4bV84GL7NAWB2whLemwbC7Q8W/G20R8b8aAwX/KHcmU3+ffrJjteDtRfhbxfdd4ozbfjJcLvEsJvZEwBCKVye1F5E6oPOoBOJhMIuEgmFXSQSCrtIJBR2kUgo7CKRqO2WzfuHkPnNs8Hyzx86w23+rZU/D9Z+k7Dc8v27/XHRgVF/que8GeFTfdudcW4A6Mj5pwknbfncnLD979uF8JmJIxl/KudYcFR13O6R8PRZAHiyeLxbzxfDWzaPODUg+fyEfaNz3foRLf3B2mAhPP0VALYNdrj1vf3+tsrDM/xoPTF2bLB23vzw1uQA0LInfJ9lnIeKjuwikVDYRSKhsItEQmEXiYTCLhIJhV0kEgq7SCRqumVzOztsGcufFdt/eXjL5mP+9kW37dJZr7n1DQP+vO03nHHXfMKSx7lMeNlgAJiRG3XrzQnjzY3Z8Jz0DPz7t5gwzt6a9fuWNNe+vSE8r7st68/5zjjbGk9F1vnbf9e/qKLf3ZbwdxfMf0x8auYrwdptr53utp25IrzN9jpbiwHbpy2bRWKmsItEQmEXiYTCLhIJhV0kEgq7SCQUdpFI1H6cPXtu+AeK/hrmlThw0TK3vuz69X69LTwu+pHGXrdtDv54cXPCeHJrxh8LH3buw6T/5k8Mdbn1sYTf8Ojbi9163hlv7j3Y7rbNOecPTIW3D8FQIWHL5iF/vns24+dm+DF/rv2cF8LnTjQ94D8WPRpnFxGFXSQWCrtIJBR2kUgo7CKRUNhFIqGwi0QicZydZBeAOwDMB1AEsNrMbiZ5A4C/BtBX+tHrzewB73dVOp+9XvGT/pr0Q/Nb3HrTW/7c6MGj/Pbtr4TXpc+M+GvOF3+/xa3L+4s3zj6VTSIKAL5uZhtItgF4huTDpdr3zOxfq9VREZk+iWE3s10AdpW+HiS5BcDC6e6YiFTXe3rNTnIRgFMBrCtd9GWSz5G8jeTsQJtVJHtI9uThP10Vkekz5bCTPAzALwB81cwGAPwQwLEAlmD8yH/TZO3MbLWZdZtZdw7+fmoiMn2mFHaSOYwH/SdmdjcAmFmvmY2ZWRHAjwEsnb5uikilEsNOkgBuBbDFzL474fIFE37sQgCbq989EamWqbwbfwaAKwBsIrmxdNn1AC4juQSAAdgG4Opp6N/7gq3f5Nb9yZLJ2p8qv21lizHLB8lU3o1/Aph0cXF3TF1E6ovOoBOJhMIuEgmFXSQSCrtIJBR2kUgo7CKRUNhFIqGwi0RCYReJhMIuEgmFXSQSCrtIJBR2kUgo7CKRqOmWzST7ALw+4aK5APbWrAPvTb32rV77Bahv5apm344ys3mTFWoa9nddOdljZt2pdcBRr32r134B6lu5atU3PY0XiYTCLhKJtMO+OuXr99Rr3+q1X4D6Vq6a9C3V1+wiUjtpH9lFpEYUdpFIpBJ2kueRfJHkyySvS6MPISS3kdxEciPJnpT7chvJPSQ3T7isg+TDJLeWPk+6x15KfbuB5Jul224jyRUp9a2L5K9JbiH5PMm/L12e6m3n9Ksmt1vNX7OTzAJ4CcA5AHYAWA/gMjN7oaYdCSC5DUC3maV+AgbJMwHsB3CHmZ1Uuuw7APaZ2Y2lf5SzzezaOunbDQD2p72Nd2m3ogUTtxkHcAGALyHF287p1yWowe2WxpF9KYCXzexVMxsF8FMAK1PoR90zs8cB7Dvk4pUA1pS+XoPxB0vNBfpWF8xsl5ltKH09COCP24ynets5/aqJNMK+EMD2Cd/vQH3t924AHiL5DMlVaXdmEp1mtgsYf/AAODzl/hwqcRvvWjpkm/G6ue3K2f68UmmEfbKtpOpp/O8MM/s4gPMBXFN6uipTM6VtvGtlkm3G60K5259XKo2w7wDQNeH7IwHsTKEfkzKznaXPewDcg/rbirr3jzvolj7vSbk//6+etvGebJtx1MFtl+b252mEfT2A40keTbIRwKUA7kuhH+9CsrX0xglItgI4F/W3FfV9AK4sfX0lgHtT7MufqZdtvEPbjCPl2y717c/NrOYfAFZg/B35VwB8I40+BPp1DIDflz6eT7tvAO7C+NO6PMafEV0FYA6AtQC2lj531FHf/gvAJgDPYTxYC1Lq26cx/tLwOQAbSx8r0r7tnH7V5HbT6bIikdAZdCKRUNhFIqGwi0RCYReJhMIuEgmFXSQSCrtIJP4PWrRbCyYGmKcAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "image, label = train_data[0]\n",
    "print(f\"Image shape: {image.shape}\")\n",
    "plt.imshow(image.squeeze()) # image shape is [1, 28, 28] (colour channels, height, width)\n",
    "plt.title(label);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adb19c5c-2f2b-4aaf-8300-256f3594e2db",
   "metadata": {},
   "source": [
    "We can turn the image into grayscale using the `cmap` parameter of `plt.imshow()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "92f09917-88f7-4446-b65f-baae586914c9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAEICAYAAACZA4KlAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAU7klEQVR4nO3de5CcVZnH8e/PcM3FQAghFyLhEta4uxjWGFi5FIgg8AegQJTa0lCCsVytXbe0SlathS3XhcULWruUWxFYcFFcq4QSi5ss5RZa4TZQMQkENQkBJgkTIFySEMjt2T/6jY7jvOdMunumOzm/T1XXdPczp/v02/PM+3Y/7zlHEYGZ7f3e1ukOmNnIcLKbFcLJblYIJ7tZIZzsZoVwspsVwsm+l5MUko7Z3VjmMS+V9KvWe2cjycm+h5D0f5JekbR/p/syXCSdJqm30/3YWznZ9wCSZgCnAAGc19ne2J7Kyb5n+DjwMHAzML9/QNLNkq6XdJekjZIekXT0YA8i6WRJz0s6fZDY/pK+Iek5SX2S/lPSgYk+SdK/S3pN0tOSzugXmCrpTkkbJK2Q9MkBz/NtSWury7er+8YA9wBTJW2qLlN3aytZkpN9z/Bx4AfV5YOSDhsQvwT4Z+BgYAXwtYEPIOmDwG3AhRHxi0Ge49+AY4HZwDHANOCfEn06AVgFTASuBG6XNKGK3Qb0AlOBi4B/7ffP4MvAidXzvBuYC3wlIjYD5wBrI2JsdVmbeH7bXRHhSxdfgJOBbcDE6vbTwD/0i98M3NDv9rnA0/1uB/CPwLPAXw547KCR2AI2A0f3i/018ExNny4F1gLqd9+jwMeA6cAOYFy/2NXAzdX1lcC5/WIfBFZX108Deju9zffWi/fs3W8+8POIeKm6/UMGHMoDL/S7/gYwdkD8c8CPI2JpzXMcCowGHpf0qqRXgXur++usiSpDK8/S2JNPBTZExMYBsWnV9anV7YHtbJjt0+kOWL3qM/M8YJSkXQm9P3CQpHdHxK+H+FAXAzdKWhMR3x4k/hKwBfjziFgzxMecJkn9Ev4dwJ009vgTJI3rl/DvAHY97lrgCODJfrFdh+segjmMvGfvbhfQOCR+F43PuLOBWcAvaXyOH6q1wBnA30n624HBiNgJfA+4TtIkAEnTqs/5dSZVj7evpIurft0dEc8Di4CrJR0g6TjgMhrfN0Dj8/xXJB0qaSKN7wVurWJ9wCGSxu/Ga7MhcrJ3t/nAf0XEcxHxwq4L8B/A30ga8pFZRDxHI+G/KOnyQX7lizS+3HtY0uvA/wJ/lnjIR4CZNI4KvgZcFBEvV7FLgBk0/sncAVwZEfdXsX8BeoAlwFLgieo+IuJpGv8MVlUfJ3x430b6449dZra38p7drBBOdrNCONnNCuFkNyvEiNbZJfnbQLNhFhEa7P6W9uySzpb0m2qwwxWtPJaZDa+mS2+SRgG/Bc6kMejhMeCSiHgq0cZ7drNhNhx79rnAiohYFRFbgR8B57fweGY2jFpJ9mnA8/1u9/KHwQ6/J2mBpB5JPS08l5m1qJUv6AY7VPiTw/SIWAgsBB/Gm3VSK3v2Xhpjl3c5nD+MXjKzLtNKsj8GzJR0pKT9gI/SGOJoZl2o6cP4iNgu6bPAfcAo4KaIeDLTzMw6ZERHvfkzu9nwG5aTasxsz+FkNyuEk92sEE52s0I42c0K4WQ3K4ST3awQTnazQjjZzQrhZDcrhJPdrBBOdrNCONnNCuElm/dy0qADoH6v1VGP48aNS8ZPPvnk2tg999zT0nPnXtuoUaNqY9u3b2/puVuV63tKs++Z9+xmhXCymxXCyW5WCCe7WSGc7GaFcLKbFcLJblYI19n3cm97W/r/+Y4dO5LxY445Jhm//PLLk/EtW7bUxjZv3pxs++abbybjjz76aDLeSi09VwfPbddc+1b6ljp/IPV+es9uVggnu1khnOxmhXCymxXCyW5WCCe7WSGc7GaFcJ19L5eqyUK+zv7+978/Gf/ABz6QjPf29tbG9t9//2Tb0aNHJ+NnnnlmMn7DDTfUxvr6+pJtc2PGc9stZ+zYsbWxnTt3Jtu+8cYbTT1nS8kuaTWwEdgBbI+IOa08npkNn3bs2U+PiJfa8DhmNoz8md2sEK0mewA/l/S4pAWD/YKkBZJ6JPW0+Fxm1oJWD+NPioi1kiYB90t6OiIe7P8LEbEQWAggqbXZDc2saS3t2SNibfVzPXAHMLcdnTKz9ms62SWNkTRu13XgLGBZuzpmZu3VymH8YcAd1bjdfYAfRsS9bemVtc3WrVtbav/e9743GZ8xY0Yynqrz58aE33fffcn48ccfn4xfe+21tbGenvRXSEuXLk3Gly9fnozPnZs+yE1t10WLFiXbPvTQQ7WxTZs21caaTvaIWAW8u9n2ZjayXHozK4ST3awQTnazQjjZzQrhZDcrhFpdsne3nsxn0A2L1LTFufc3N0w0Vb4COOigg5Lxbdu21cZyQzlzHnvssWR8xYoVtbFWS5JTpkxJxlOvG9J9v+iii5Jtr7/++tpYT08Pr7/++qB/EN6zmxXCyW5WCCe7WSGc7GaFcLKbFcLJblYIJ7tZIVxn7wK55X1bkXt/H3744WQ8N4Q1J/XacssWt1oLTy35nKvxP/HEE8l4qoYP+dd29tln18aOOuqoZNtp06Yl4xHhOrtZyZzsZoVwspsVwsluVggnu1khnOxmhXCymxXCSzZ3gZE812GgV155JRnPjdvesmVLMp5alnmffdJ/fqlljSFdRwc48MADa2O5Ovspp5ySjL/vfe9LxnPTZE+aNKk2du+9wzMju/fsZoVwspsVwsluVggnu1khnOxmhXCymxXCyW5WCNfZCzd69OhkPFcvzsXfeOON2thrr72WbPvyyy8n47mx9qnzF3JzCOReV2677dixIxlP1fmnT5+ebNus7J5d0k2S1kta1u++CZLul/S76ufBw9I7M2uboRzG3wwMnFbjCuCBiJgJPFDdNrMulk32iHgQ2DDg7vOBW6rrtwAXtLdbZtZuzX5mPywi1gFExDpJtSf6SloALGjyecysTYb9C7qIWAgsBE84adZJzZbe+iRNAah+rm9fl8xsODSb7HcC86vr84Gftqc7ZjZcsofxkm4DTgMmSuoFrgSuAX4s6TLgOeDi4ezk3q7Vmm+qppsbEz516tRk/K233mopnhrPnpsXPlWjh/za8Kk6fa5Ovt9++yXjGzduTMbHjx+fjC9ZsqQ2lnvP5syZUxt76qmnamPZZI+IS2pCZ+Tamln38OmyZoVwspsVwsluVggnu1khnOxmhfAQ1y6Qm0p61KhRyXiq9PaRj3wk2Xby5MnJ+IsvvpiMp6ZrhvRQzjFjxiTb5oZ65kp3qbLftm3bkm1z01znXvchhxySjF9//fW1sdmzZyfbpvqWKuN6z25WCCe7WSGc7GaFcLKbFcLJblYIJ7tZIZzsZoXQSC4X7JlqBper6W7fvr3pxz7hhBOS8bvuuisZzy3J3Mo5AOPGjUu2zS3JnJtqet99920qBvlzAHJLXeekXtvXv/71ZNtbb701GY+IQYvt3rObFcLJblYIJ7tZIZzsZoVwspsVwsluVggnu1kh9qjx7Kmxurl6b2465tx0zqnxz6kx20PRSh095+67707GN2/enIzn6uy5KZdT53Hkxsrn3tMDDjggGc+NWW+lbe49z/X9uOOOq43llrJulvfsZoVwspsVwsluVggnu1khnOxmhXCymxXCyW5WiK6qs7cyNno4a9XD7dRTT03GL7zwwmT8pJNOqo3llj3OjQnP1dFzY/FT71mub7m/h9S88JCuw+fmccj1LSe33TZt2lQb+/CHP5xs+7Of/aypPmX37JJukrRe0rJ+910laY2kxdXl3Kae3cxGzFAO428Gzh7k/usiYnZ1SZ+mZWYdl032iHgQ2DACfTGzYdTKF3SflbSkOsw/uO6XJC2Q1COpp4XnMrMWNZvs3wWOBmYD64Bv1v1iRCyMiDkRMafJ5zKzNmgq2SOiLyJ2RMRO4HvA3PZ2y8zaralklzSl380PAcvqftfMukN23nhJtwGnAROBPuDK6vZsIIDVwKciYl32yTo4b/yECROS8alTpybjM2fObLptrm567LHHJuNvvfVWMp4aq58bl51bZ3zt2rXJeG7+9VS9ObeGeW799dGjRyfjixYtqo2NHTs22TZ37kNuPHtuTHpqu/X19SXbzpo1Kxmvmzc+e1JNRFwyyN035tqZWXfx6bJmhXCymxXCyW5WCCe7WSGc7GaF6Kolm0888cRk+69+9au1sUMPPTTZ9qCDDkrGU0MxIT3c8tVXX022zQ2/zZWQciWo1DTYuamgly9fnozPmzcvGe/pSZ8FnVqW+eCDa8+yBmDGjBnJeM6qVatqY7nlojdu3JiM54bA5kqaqdLf29/+9mTb3N+Ll2w2K5yT3awQTnazQjjZzQrhZDcrhJPdrBBOdrNCjHidPVWvfuihh5Ltp0yZUhvL1clz8VamDs5NeZyrdbdq/PjxtbGJEycm21566aXJ+FlnnZWMf/rTn07GU0Nk33zzzWTbZ555JhlP1dEhPSy51eG1uaG9uTp+qn1u+OwRRxyRjLvOblY4J7tZIZzsZoVwspsVwsluVggnu1khnOxmhRjROvvEiRPjvPPOq41fc801yfYrV66sjeWmBs7Fc8v/puRqrqk6OMDzzz+fjOemc06N5U9NMw0wefLkZPyCCy5IxlPLIkN6THruPXnPe97TUjz12nN19Nx2yy3JnJOagyD395Sa9+GFF15g69atrrOblczJblYIJ7tZIZzsZoVwspsVwsluVggnu1khsqu4SpoOfB+YDOwEFkbEdyRNAP4HmEFj2eZ5EfFK6rG2b9/O+vXra+O5enNqjHBuWePcY+dqvqm6am6e7w0bNiTjzz77bDKe61tqvHxuzHhuTvs77rgjGV+6dGkynqqz55bRztXCc/P1p5arzr3u3JjyXC081z5VZ8/V8FNLfKe2yVD27NuBz0fELOBE4DOS3gVcATwQETOBB6rbZtalsskeEesi4onq+kZgOTANOB+4pfq1W4ALhqmPZtYGu/WZXdIM4HjgEeCwiFgHjX8IwKS2987M2mbIyS5pLPAT4HMR8fputFsgqUdST+4zmJkNnyElu6R9aST6DyLi9uruPklTqvgUYNBv3iJiYUTMiYg5rQ4eMLPmZZNdja8NbwSWR8S3+oXuBOZX1+cDP21/98ysXbKlN+Ak4GPAUkmLq/u+BFwD/FjSZcBzwMW5B9q6dStr1qypjeeG2/b29tbGxowZk2ybm1I5V8Z56aWXamMvvvhisu0++6Q3c254ba7MkxpmmpvSODeUM/W6AWbNmpWMb968uTaWK4e+8kqykpvdbqm+p8pykC/N5drnlmxODS1+7bXXkm1nz55dG1u2bFltLJvsEfEroK4oeEauvZl1B59BZ1YIJ7tZIZzsZoVwspsVwsluVggnu1khhlJnb5stW7awePHi2vjtt99eGwP4xCc+URvLTbecW943NxQ0Ncw0VwfP1VxzZxbmloRODe/NLVWdO7cht5T1unXrmn78XN9y5ye08p61Ony2leG1kK7jH3nkkcm2fX19TT2v9+xmhXCymxXCyW5WCCe7WSGc7GaFcLKbFcLJblaIEV2yWVJLT3bOOefUxr7whS8k206alJ4iLzduO1VXzdWLc3XyXJ09V29OPX5qymLI19lz5xDk4qnXlmub63tOqn2qVj0UufcsN5V0ajz7kiVLkm3nzZuXjEeEl2w2K5mT3awQTnazQjjZzQrhZDcrhJPdrBBOdrNCjHidPTVPea422YrTTz89Gb/66quT8VSdfvz48cm2ubnZc3X4XJ09V+dPSS2hDfk6fGodAEi/p5s2bUq2zW2XnFTfc+PNc+P4c+/p/fffn4wvX768NrZo0aJk2xzX2c0K52Q3K4ST3awQTnazQjjZzQrhZDcrhJPdrBDZOruk6cD3gcnATmBhRHxH0lXAJ4Fdi5N/KSLuzjzWyBX1R9A73/nOZLzVteEPP/zwZHz16tW1sVw9eeXKlcm47Xnq6uxDWSRiO/D5iHhC0jjgcUm7zhi4LiK+0a5OmtnwySZ7RKwD1lXXN0paDkwb7o6ZWXvt1md2STOA44FHqrs+K2mJpJskHVzTZoGkHkk9rXXVzFox5GSXNBb4CfC5iHgd+C5wNDCbxp7/m4O1i4iFETEnIua03l0za9aQkl3SvjQS/QcRcTtARPRFxI6I2Al8D5g7fN00s1Zlk12NKTpvBJZHxLf63T+l3699CFjW/u6ZWbsMpfR2MvBLYCmN0hvAl4BLaBzCB7Aa+FT1ZV7qsfbK0ptZN6krve1R88abWZ7Hs5sVzsluVggnu1khnOxmhXCymxXCyW5WCCe7WSGc7GaFcLKbFcLJblYIJ7tZIZzsZoVwspsVwsluVoihzC7bTi8Bz/a7PbG6rxt1a9+6tV/gvjWrnX07oi4wouPZ/+TJpZ5unZuuW/vWrf0C961ZI9U3H8abFcLJblaITif7wg4/f0q39q1b+wXuW7NGpG8d/cxuZiOn03t2MxshTnazQnQk2SWdLek3klZIuqITfagjabWkpZIWd3p9umoNvfWSlvW7b4Kk+yX9rvo56Bp7HerbVZLWVNtusaRzO9S36ZJ+IWm5pCcl/X11f0e3XaJfI7LdRvwzu6RRwG+BM4Fe4DHgkoh4akQ7UkPSamBORHT8BAxJpwKbgO9HxF9U910LbIiIa6p/lAdHxBe7pG9XAZs6vYx3tVrRlP7LjAMXAJfSwW2X6Nc8RmC7dWLPPhdYERGrImIr8CPg/A70o+tFxIPAhgF3nw/cUl2/hcYfy4ir6VtXiIh1EfFEdX0jsGuZ8Y5uu0S/RkQnkn0a8Hy/271013rvAfxc0uOSFnS6M4M4bNcyW9XPSR3uz0DZZbxH0oBlxrtm2zWz/HmrOpHsgy1N0031v5Mi4q+Ac4DPVIerNjRDWsZ7pAyyzHhXaHb581Z1Itl7gen9bh8OrO1APwYVEWurn+uBO+i+paj7dq2gW/1c3+H+/F43LeM92DLjdMG26+Ty551I9seAmZKOlLQf8FHgzg70409IGlN9cYKkMcBZdN9S1HcC86vr84GfdrAvf6RblvGuW2acDm+7ji9/HhEjfgHOpfGN/Ergy53oQ02/jgJ+XV2e7HTfgNtoHNZto3FEdBlwCPAA8Lvq54Qu6tt/01jaewmNxJrSob6dTOOj4RJgcXU5t9PbLtGvEdluPl3WrBA+g86sEE52s0I42c0K4WQ3K4ST3awQTnazQjjZzQrx/ybqlcORUfUKAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(image.squeeze(), cmap=\"gray\")\n",
    "plt.title(class_names[label]);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a09388a-d754-485f-aa26-4e7a0f782967",
   "metadata": {},
   "source": [
    "Beautiful, well as beautiful as a pixelated grayscale ankle boot can get.\n",
    "\n",
    "Let's view a few more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "7188ed7a-5959-48c4-ac7f-19129a2adc83",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgEAAAIHCAYAAAAGv498AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAB0fklEQVR4nO29d7wdVfX+/yyQnt57QioJLdQABgm9S/hQpBfpfkBFFEFBQUXKRwUBFfSHAirti4AIovQWIIghhIT03gvpIfT9++OcDHs9956Ze3LbuXee9+vFi72y58zMmb1nn333s9baFkKAEEIIIfLHJo19A0IIIYRoHDQJEEIIIXKKJgFCCCFETtEkQAghhMgpmgQIIYQQOUWTACGEECKnNMlJgJkFM+tfbl3GOc8ys1drf3ei0shqWzN7yszObMh7EkLUH/qNqDmNOgkwsxfNbIWZbdGY91GfmNkIM5vX2PeRB8xsuJm9ZmarzGy5mY0ysz2yPhdCODyEcE/KeZvly58HzGyWma03s7XFseZJM+vZ2PclaoZ+I+qfRpsEmFkfAPsCCAC+2lj3IZoHZtYKwBMAbgPQDkB3ANcC+KiW5/1S7e9ONDJHhxBaAOgKYDEKfURUOPqNaBgacyXgDABvALgbgFuKNbO7zew3xVn7GjMbbWb9qjtJ8a+/uWa2fzV1W5jZL8xsjpktNrM7zGyrlHsyM7ut+JfkJDM7MKroZmaPF//CnGZm59F1bjGzBcX/bin+2zYAngLQrfiXyFoz61bWUxI1ZSAAhBDuDyF8FkJYH0J4OoQwbsMBxb6wwsxmmtnh0b+/aGbnFstnFVcQbjaz5QAeBHAHgL2L7beyYb+WqCtCCB8CeBjAEAAwsyPN7G0zW10cQ66JjzezM8xstpm9b2ZXF1cVDmqEW88r+o1oABp7EvDX4n+Hmllnqj8Zhb/k2gKYBuA6PoGZHQrgfgDHhRBeqOYaN6Lw4zAUQH8U/jr8Uco9DQMwA0AHAD8G8IiZtSvW3Q9gHoBuAI4H8POoA/wQwF7F6+wMYE8AV4UQ1gE4HMCCEEKL4n8LUq4vNp4pAD4zs3vM7HAza0v1wwBMRqFtbwJwl5lZiXNt6AedAJwG4EIArxfbr0293L2od8xsawBfQ+GHBQDWoTAOtQFwJICLzGxk8dghAH4L4FQUVhBaozB+iIZDvxENQQihwf8DMBzAJwA6FO1JAC6N6u8G8P9F9hEAJkV2AHAlgNkAdqRzBxQa01B4yftFdXsDmFnins4CsACARf/2JoDTAfQE8BmAllHd9QDuLpanAzgiqjsUwKxieQSAeY3xnPP2H4DBxb4zD8CnAB4H0LnYttOi47Yu9pMuRftFAOdG/WBONX3j1cb+fvpvo/rELABrAaws9okFPGZEx94C4OZi+UcA7qc+8zGAgxr7O+XhP/1GNNx/jbUScCaAp0MIy4r2faDlHgCLovIHAFpQ/bcBPBRCeLfENTqi8OL+18xWFpdx/1X891LMD8VWKTIbhVldNwDLQwhrqG7DXwbdijZ/TjQgIYSJIYSzQgg9AOyAQhvcUqxeFB33QbHIfWoDc+vtJkVjMDIUVnC2AHAxgJfMrIuZDTOzF8xsqZmtQmHFp0PxM90Q9YNin3m/ge87z+g3ooFo8ElAUW85EcB+ZrbIzBYBuBTAzma2cxmnOgHASDP7don6ZQDWA9g+hNCm+F/rUHAQKkV3WiLuhcLMbwGAdmbWkurmF8sLAPSu5nNAYdYpGpgQwiQU/lrYYWM+nmGLJkgo+Io8gsJfbMNR+GF5HEDPEEJrFHw/Nrz/CwH02PDZ4rjVvmHvOJ/oN6JhaYyVgJEovIRDUNBHhqKwjPsKChpQTVkA4EAA3zSzb3BlCOFzAH8AcLOZdQIAM+te1IhK0al4vs3M7ITiff0zhDAXwGsArjezLc1sJwDnoKBVAQUt6Coz62hmHVBYSvxLsW4xgPZm1rqM7ybKxMy2M7PLzKxH0e6Jgmb4Rvona8RiAD3MbPM6OJdoJKzAMShoyBMBtEThr7cPzWxPAKdEhz8M4Ggz26fY7tfiiwmCqF9GQr8RDUZjTALOBPCnEMKcEMKiDf8BuB3AqVZGSFYIYQ4Kjfx9K3p3E99HwWHkDTNbDeBZAINSTjkawAAUZojXATg+hLBhCfBkAH1Q6FiPAvhxCOGZYt3PALwFYByAdwGMKf7bhr9I7wcwo7jkVBFLQM2QNSg47Yw2s3Uo/PiPB3BZHZz7eQATACwys2VZB4uK4x9mthbAahTe6zNDCBMAfAPAT8xsDQqD8kMbPlCsvwTAAyisCqwBsAS1DDkVNUK/EQ2IeXlDCCEEY2YtUHAuHBBCmNnItyNEndEk0wYLIUR9Y2ZHm9nWxVjuX6DwF9ysxr0rIeoWTQKEEKJ6jsEXTl8DAJwUtHQqmhmSA4QQQoicopUAIYQQIqdoEiCEEELklNRQCzOTVtBECCHUawxzpfQFTvfPctY222zj7JNOOsnZa9euTcorVqxwdV26dHH2mjVrnP3oo4+Wd7ONRF76gshGfUFsoFRf0EqAEEIIkVM0CRBCCCFySo0zLwnRUKQt+WdFsxx55JHObtvW7yi82WabJWVe/t9xxx2dPXjwYGfXpRyQJWsIIURDoJUAIYQQIqdoEiCEEELkFE0ChBBCiJySmjGwuYR/DBr0xaZQHTt2dHXr1693dqwZA8DHH3+cWv/ZZ58l5c8//9zVsc3n2mSTTVLtWDdu2bKlq3v77bedvXbt2lyEArVq1crZxx13nLN33313Z7/22mvO/v73v5+U2QdgwYIFzv7pT3/qbA4/nD17trOfeeaZpLxq1aoq995QKCxMbEB9oe5pTH+ebt38BoOxXxP/3owdO9bZChEUQgghhEOTACGEECKnNAs5gJfReVnkuuuuS8pdu3Z1dR999JGzOSyMl/+33nprZ8dL/ByOxnz44YfO/tKXfITmvHnznB23DX+nW2+91dn//Oc/m82y30477eTsXXbZJSnzM/7000+d3atXL2dz+8VLef369XN1Tz/9tLOnTp3q7P79+6eeO5Zsli5d6upiqQAApk2bhvqivpeAN9lkE9cXMsaQ1HMpNLJ+yYscUO4SPR8f2zzW8piz7bbbOnvy5MnOXrduXfrNRvTu3dvZnTt3djZLkEzr1q2TMkulDz/8MN+X5AAhhBBCfIEmAUIIIURO0SRACCGEyCnNIm1wlv4T6/6sy3OI4JQpU5y95ZZbOnvTTTd19vLly5Ny+/btXR37Kmy++eap5+Lv8cknnyTlLbbYwtVNnz4dzYUTTzzR2UOGDHH2zJkzkzLv/Bc/f8A/M8BrZoDX+5577jlXx/4FHTp0cHa8A2F114pDDNu0aePqzjzzTGf/7W9/czaH8zRl4n4dh9BWB7c1twET++DwudlHg98/ft9in5ws3wWG31W242tnfSfWoLfaaitn8+fj+kmTJrk67qOiesrxRYlDzIGqYeYffPCBs7lNYjikmWGfM257Hlfi9ubfF/6tK4VWAoQQQoicokmAEEIIkVM0CRBCCCFySrPwCcgiTUdkHYVt1gq5vnv37kmZtT3OA8CaZFZ+g7ie9c6a6j2VCG/hy7kZJkyY4Oy4DfgZsn7Kz4V9CGK/jdWrV7s61vpYj+O+w9eK8wRw3YwZM5w9dOhQZzcln4AsPTXND+CQQw5xNm/9PGvWLGcvW7bM2XGqZ9a/2b+H31XOCRL3q7SU3dXB9fz5tHTi3Gc5Dp2fAX+vOF/FqFGjXN1jjz1W+qabMbXNNxF/ntuWxwH2z2rRooWzO3Xq5OxzzjknKfM40K5dO2fzbwb3YfY/iPsSv3c1fSZaCRBCCCFyiiYBQgghRE7RJEAIIYTIKbnwCYj1WdaUeYte1mhYi2edJfY3SNMFq7s2by2ctm0xbyHJ2lBTYrvttnP2ypUrnZ0Wr89b9Ka1B1C1/eJrccwttxfrc6ztcg6JuP3XrFnj6rif8bUbc3vS+uTnP/+5s1mXZ12f8yfwM4/7/d577+3qOG9D1rsetzf3m6xtwZm0dzvLN4Fzz/Mz4PfhG9/4RlLeb7/9XN3zzz+fep8iG37v2WeD9wbgfsU+AfGeMJy3hM/F/TD2Oavu+LgPcx+sKVoJEEIIIXKKJgFCCCFETtEkQAghhMgpufAJiPdk5hhP1oFZy2U9iLXbOIY3a28A1va4nrXCWM9mLYjjp5sSrNMzafoq59bmGGqOz+f2jnVjbi/Wfbk9GP4e8b3wd2AfDtYGeZ+CpUuXpl67MeHvzVr8s88+m5R/85vfuDr2o+Bc6hdddJGzu3bt6ux4fwZun8MPP9zZnG+C+0Lsw8Ftz+NCuTHYcew4a8ycB4D3lN93331TrxXfC/fhAQMGpN6XKJDmg8M6/Lbbbutsfjd5zOF8I/H+Jpwjhcd1Hgfef/99Z3N7x32Y74P7XSm0EiCEEELkFE0ChBBCiJyiSYAQQgiRU3LhExDHcbL2x3obxy2zlsuaTawztmrVytWxDwDbrOFwjGistTblvQIYzrW9cOFCZ7MGN3v27KSctf8C62Bpuj5re/G+AtWdm23Wt2N/Eq6L/VKAqm3NunAl+wSwD8CIESOc3blz56T86KOPuroHH3zQ2dxe/fr1czY/tz59+iTlX/3qV66O378vf/nLzuZ4/HgsYM2f/Qey9grg+4z3rJg4caKr4+c3ZMgQZ7PumzaOsI8Mn1uUD49P7L/D7yb3FfZTiuGcAux/kPXe8+9Vmo8T+7OVQisBQgghRE7RJEAIIYTIKc1CDihna1NeTuZl27RtUKv7fLwcw0txvPTDy5WcLpeXFGMJgM/VlOFl9+nTpzt75513dna8RMYhnByulrUFbNx+vMyXFbLJS6/x0jQAvPLKK0l5p512Sj0Xtz2HwjUlzj77bGdfcsklJY/t1auXsxctWpR6bm7vON1yvEUrAPz0pz91NktonK6a3/0Yfpe5X82ZM8fZixcvdna8TTVvZ839htt+6tSpzubl5ni8GzhwoKtryunEG5K0EEFeVud+xOMGv8v8G9K3b9+kzCGCfG5OW8/tySGFMVmhu6XQSoAQQgiRUzQJEEIIIXKKJgFCCCFETmmSPgHlbrsaa26s+/LWtBy6xTr+q6++6uw4xInPxdof21tttZWzORVwvH3w22+/jeYCa+uxfgpUDRGMQ6yytnRl+Pi4/bPCCbme9buxY8c6Ow4tWrJkScnrAlV1R/YHqWQ49OirX/2qs88888ySn+U+z+8uP3N+bnHK3d12283VnXzyyc5+5513nM3b7O65555JecaMGa6OdXpO33r00Uc7e9SoUc6O33XWiPk+n3vuOWdzX+DxLu7TrEfzfYny4fc83goYqBrOy34t/G7HYefs28Xvw6RJk5ydleY+trNSXZdCKwFCCCFETtEkQAghhMgpmgQIIYQQOSUXPgGx5sx6DfsA8Lk4hSSn+IyZP3++szndJ8c8szbO/gZHHXVUUh4zZkzJ61Y6HFebleqXdbJY2+LPZuUF4OPjc7H2x32D4TjcNA2O+xVvL8paLn/nSuaMM85w9lNPPVXyWG4f1jiz4Pcxbk/OL3HggQc6m7Vazk8R5yz473//6+ruvfdeZ7OfA/t8cHvHqa7PPfdcV8e+JGn5CoCq70ccO859cIcddkg9lyiQ9psxd+5cZ7MPAL+77MPEun7s1/Laa6+5OvZZ4t8MHhfYXyQtbXBNxxStBAghhBA5RZMAIYQQIqdoEiCEEELklAbzCShXx6/La8VbO3KcJsc8s+bCGg3nDYh1mHgL1erOxbC2NGzYsJLHcj7xpgTHXPNWm5zjmp9xTJaGxnaaDwHnJ+jUqVPqtbJ8GeJr82dZC+ctq+Oc+JUOa6D33HNPyWP5PWe9lDVtzgvAxPo5P2N+n8477zxnP/PMM86eMmVKUuZx4Qc/+IGz+V3m9jv00EOdHecgeP31110d56PI8kXh7xn7pnBuefZNyAtZvy9cn3Y890H2C+PP8vjF73r8G8N+RQzvM8F7CfDWw9OmTUvKvK8A+yGVQisBQgghRE7RJEAIIYTIKZoECCGEEDmlwXwC6tIHICs2nONuY02HY/OzYr9ZB2Z9KI41L2fvZwDo2bNn6rljmvI+4Vn5r1nHZ40t9rXg2HDWU7P2A4j7Trw3A1BV5+W+kuUjENezxs/7iI8bN87Z/Iw2dm/whoB9JyZPnlzyWNZXs95dJk3b5fZgPZxzvg8fPtzZcRw1+4fE+1UAVePv+Rnwvh8vvvhiUs7aDyGtHwHpejafqz59rSqZrO+d5SMQ07dvX2fzGMJ5T9jHicez+FpZPjCs4/O9xPknAL+nBX+W76MUWgkQQgghcoomAUIIIUROaZJpg7OWfgYMGODseMmFl+h5WYiXl3mprpwQnKwlqDlz5jibl4Dj++Ywu6YEh1fxkhh/74ULFzo7XrbNavu0rYMB355ZoVpZ2xanbYPLy/ecopi/c9aSIy83Nya8vM3LoTEsuXB7ZL1Pae8Q1/Ez53c3bXtnPpbHEJYLeFmW+07cvlkSSFZYa20kkbxQbgh6Wthljx49XB2ne+dldw7j43DuuF/ydXlcZ9nwvffeczZLXLEsxRLkzJkzURO0EiCEEELkFE0ChBBCiJyiSYAQQgiRU5qET0BWCA1z8MEHOzvWdDg9KGuUWRob645xGBmH+HHYF4ecsbbKunHsvzB06FBX949//ANNhSytnZ8Ta6SxD0FWyFlWe8Y2P++slNHcD9Ns1vhZo+Rr8ffI2l62MeHnlqZ5Z21nys8ly48mbr+0tM1AdgriNN2Y/RzS/HWAdP8SHkOy+iz7zKTp3fwdmrNPQNpzyPK7yOpXcZpnbkse17l9xo8f72z+fOzPs/3227s6HvvefvvtKvce079/f2fH7yKnza5pX9BKgBBCCJFTNAkQQgghcoomAUIIIUROqZVPQJoOU9v0lbHexxoMc+yxxzqb4yXjz6fF81Z3rSydP9ZuWcdlbYg1S9ZWOTVwrDE3pa1mGe4naZp/dfWxH0dWimFuzzR/Etar+VjuK6yxsZYb2/yd2Oa+wt+L+1klwXHTaSmNs9ojK2dEmj7OY0xW7oU0vwv2AcjKQZCV2jduT/4OWemn2WbStqxmP4nmRNpvSpafGD/TgQMHOjt+1+NUvADQp08fZ7P2zuMAb7Ud5x3gYzkNOtfzGMRbDcd9nseMLJ+YDWglQAghhMgpmgQIIYQQOUWTACGEECKnpApIWfmYY7tcXYvPxZpO2jaI+++/v7N32mknZ3N+5TjGvm3btq6OdXjOI8DaLWuDsT7Eeg1/x6xtjPna8fPn/NRNiaxnmLW1atwX+FxpeQCqs2PNlK/LfYE1Nn4f+HvE507b/hXI3sa4pnpeY8Dv1/nnn+/s66+/Pinze5yl3WZt5Z22LWuW1s7EzzzrvWcWL17s7DS9upzcB0DV78U6f9zvarpdbH1RTs7+cmP5y6FVq1bO5veLdX2+VpyTZdCgQa6Ox2n2iWFfobTtgvmz/N7zM8ryUVu+fHlS5j1Xst61DWglQAghhMgpmgQIIYQQOUWTACGEECKn1MonIKam+kNNiWO4jzrqKFfH+ZNjXQQAOnTo4OxY78nSnXjfcI4Z5e+Zlseb9TrWjnr27OnsND2INa6mRJaOz5pnWhtx3CznWuDPsi4c6/yLFi1ydaz98bXYToth57bmfsXtyf4Idf0+1SXjxo1z9h133OHs2CeAv3fv3r2dzXnX2RciLTdDuXsDsA9HvBc75wlYsmRJ6n3wvu8cOx5rv1laeBbcz+L8B/wdOTdCQ1POPgdZ8LvL+2307ds3KXMeFdbeWS/n4+MxKv69ALL3AeF3tXPnzs6Oxxn+beJ+lOWLMmPGDGfH9/rQQw+5un79+qWeawNaCRBCCCFyiiYBQgghRE7RJEAIIYTIKak+Aax1tG/f3tmDBw/+4kQpsaxAVa2I9VfWUWLthI9lLTfW9oCqWvykSZOSMmv8Xbt2Tf0s55dPyzfP3zlrP3r2ZUjTpNPys1c6/By4r3D+hLlz5zo71suzcg5k6Y7x8Rwbnqa9AuXl8+e25vtmm3VIfiaVxJNPPunsd99919n77LNPUn7ttddcHbc99/msmPr4uZUbI8/Hx+PZqaee6urGjh2beq6rrrrK2YceeqizY5+CrP0QuD6L2BeCfWLYB6O+yXrf4vZjHZ5zn3Ts2NHZ/L6l+fewT02crx+o6vOxdOlSZ7du3Topr1q1ytWxPwh/do899nA2jyvxbxmPKTx+cXvyb0RaPh7uRzV9P7QSIIQQQuQUTQKEEEKInFLWvpMDBgxwdhzuw8uX5SyjA8CyZcucHS/fZKVW5Pq0JUcO5eHlGF6CYpki7dq8bMRSA6e2XLBggbP5GcZLR+UuGVYSWWFMnAqTl5fjZdt42Q6o2p7c9mkpWvmZZoWccV9JS2nLfYHDC1lqYzmgKbX35Zdf7uxLL700KbMc8MQTTzibU37z8mfalsvlht6lLSezBMX9iJdWedmdx414yZ+/Q1Yf5X7I105L1/7KK6+gMeG07PG7zd+LZZE5c+Y4m98hJv7ufG4eS/kZ8/sY/95w2/I4vtdeezmbf9u6d+/u7HjcnzlzpqvLCg3m78UhhfFY+frrr7s6/r0uhVYChBBCiJyiSYAQQgiRUzQJEEIIIXJKqk8Ah97tu+++zo6302TNhTVOTuPIKSA5fCTW2LLCwFhTZq0p1mRYn2PdiW3WbNj/IIbPnaaPAVV1KdaW4muxzsTpJyuZrDSb/JxeeuklZ++2225JmcNUszQ0ro/7UlZqXtZuWX/la8XvANex3smpr/leyglHbGj4XefUv7GfxuOPP+7q+N3k78n9Oi3MNmsL3qxQvNmzZyflQw45xNVNnDgx9T54/OJ0rvH3ykqTzfB4x/cd68LsS/Xqq6+mnru+GTJkiLPj9Nj8vThsj8dDHov5NyWu5/bgtuexldsgHrv5PnnM4fvicES+1oQJE5Iyhx9yOCH7PGWlvu7Vq1dS/s1vfuPq2B+nFFoJEEIIIXKKJgFCCCFETtEkQAghhMgpqeLUsGHDnM1b+r733ntJmWMrWTeJ/QcAYP78+c5mXSXWw1kHYW2ddZO0mOws3wS+D9ZosvShtPtgXZG1cNYO48+zxsX+GpUMPwf2nWCNmVM7x/o66/KsvzHcd+Jr8zPNSs3M+jUfH+c/4LaNdUEA2HvvvZ3Nbc/pQyuJrBTho0ePTsp77rmnq5s1a5azOWcE+wTwM4/ft7QUqkDVfsf9LB4LLr74YlfHWi3r16xPsx37IfF9Mew/wsezX9LLL7+clK+99trUczc0/F3i3wV+V7nPp/lbAVVztsRj4IoVK1Lvg8fp3Xff3dlxXgHO38J9kLcB53FkypQpzo7fF34GfF/8+5TllxRfO/YPqO5apdBKgBBCCJFTNAkQQgghcoomAUIIIUROSfUJ+Mc//uFs1thGjhyZlAcOHOjqWP9mjZO3A2YtJNaHWCPmuFnW9dNyELCuxLmY2beB7/vPf/6zs0888cSkzL4K/AxYC2fYtyHWtTgXNudVqGS4/bivsC7Gx6dtl8l6HPdR1tDiz3N7LVy40NmsCzOsO8Y264gcS88+Mex7MnXq1NRrNyZZ28fG7cfx8wzrwPzMuY3i+rS9APg+gPTc9aNGjXJ1/O7y+JS1z0TcnuxPwD4U3O84RwHvpZG2XXC5+ynUFn4fOZ9JPE7xFrzx3jNAVT8afsZpMfY8PmaNtUycb4G3OOb2498u9m0YPHiws+P24n6Ulb+A3zUeK9Nyk/DYWAqtBAghhBA5RZMAIYQQIqdoEiCEEELklPQk1sTf/va3kvZ2223n6k4++WRn9+3b19m81zFrOrG+wRoa615ZdpzLnOO1r776amezNpjFzTffXO11gKp6Z5Zmk5bnnjXKnj17lnWfjUlWDDzrZEys/bLmn6bLV0esFfK5svaoyOpn8X2yXs3fOWtPeX5f4pwcjQ33Rea5555Lys8//7yrYx2Y2yCOrweq+vfE40Ta3g01uc+5c+cmZfbZqGTS8g5k7YdR1wwaNMjZrJ/He2akjcuAbw+gat9g4neK+wnDvgvsVxHfG+eu4GfKfTRrbI51/7S8MkDVPs3jRtqYxJ/NynuyAa0ECCGEEDlFkwAhhBAip2gSIIQQQuSUsnwCWFeJNYdJkya5uh//+Mep52J9iLWlrl27JmXOJ856Ksdtcu7nyZMnp95LbbjwwguTMu+PwHt9s7bE2hPrabF+xBrWkiVLnP3AAw/U8I4bHvYX4Vhijudmxo4dm5SHDh3q6rgvsObGzzzWzfhYji3Oyj/O2my8rwTrnQyfm/ORZ32+qcAaJu8dIMqnoXX/NHgcOu6445wdv5883rGWnuUjxe9bXJ+VL4R9zjg+P67nfBMM3xePC7zXQ/w7mfUdeQ8KvpdycgFk5fPYgFYChBBCiJyiSYAQQgiRUyxtycDMaraeIBqdEEK95gutTV/gpTlOecxLYiyjxBx88MHOPvDAA53N6Xj5XHGaaL7uvHnznM0pa3k5klPcxsuA9957r6vLCg1ieaymS3nVUcl9QTQsDd0XWFKLZS5OC8yhdhzSxkv2bMdL4WlSNVBVekjbwpfHBX43+VosSXJ9OfDyPm9dn/Y9eDvlN99809ml+oJWAoQQQoicokmAEEIIkVM0CRBCCCFyinwCmgl51YFZM9thhx2czduCxjpkVmhPls7IoaccJttY5LUviKqoL4gNyCdACCGEEA5NAoQQQoicokmAEEIIkVNSfQKEEEII0XzRSoAQQgiRUzQJEEIIIXKKJgFCCCFETtEkQAghhMgpmgQIIYQQOUWTACFKYGbBzPrX4Lg+xWO/lHWsEKJp09zGhYqdBJjZ2ui/z81sfWSf2tj3JxoPMxtuZq+Z2SozW25mo8xsj8a+L1FZmNkpZvZWccxYaGZPmdnwWp7zRTM7t67uUdQdGhc2joqdoYQQks3czWwWgHNDCM/ycWb2pRDCp/zvDUkl3ENeMLNWAJ4AcBGAhwBsDmBfAB815n2JysLMvgPgCgAXAvg3gI8BHAbgGACvNuKtiXpA48LGU7ErAaUwsxFmNs/Mvm9miwD8ycy2MLNbzGxB8b9bzGyL4vFnmdmrdI5kOcfMjjCz98xsjZnNN7PvRscdZWZjzWxlcYa5U1Q3q3gP4wCsq/Qln2bEQAAIIdwfQvgshLA+hPB0CGGcmfUzs+fN7H0zW2ZmfzWzNhs+WGyz75rZuOJfCw+a2ZZR/feKfzEuMLOvxxc1syPN7G0zW21mc83smob6wqI8zKw1gJ8A+N8QwiMhhHUhhE9CCP8IIXwvY7xoa2ZPmNlSM1tRLPco1l2Hwg/L7cXVhdsb71sKQuPCRtLkJgFFugBoB6A3gPMB/BDAXgCGAtgZwJ4Arqrhue4CcEEIoSWAHQA8DwBmtiuAPwK4AEB7AHcCeHzDYFHkZABHAmijlYAGYwqAz8zsHjM73MzaRnUG4HoA3QAMBtATwDX0+RNR+ItwWwA7ATgLAMzsMADfBXAwgAEADqLPrQNwBoA2KLT5RWY2so6+k6hb9gawJYBHS9SnjRebAPgTCmNLLwDrAdwOACGEHwJ4BcDFIYQWIYSL6+n+RfloXNhImuok4HMAPw4hfBRCWA/gVAA/CSEsCSEsBXAtgNNreK5PAAwxs1YhhBUhhDHFfz8PwJ0hhNHFmeU9KCwt7RV99tYQwtziPYgGIISwGsBwAAHAHwAsNbPHzaxzCGFaCOGZYr9YCuBXAPajU9waQlgQQlgO4B8o/BAAhUHgTyGE8SGEdaBBIoTwYgjh3RDC5yGEcQDur+bcojJoD2BZysS85HgRQng/hPC3EMIHIYQ1AK6D2rni0biw8TTVScDSEMKHkd0NwOzInl38t5pwHIAjAMw2s5fMbO/iv/cGcFlRClhpZitRmEHG5527UXcvakUIYWII4awQQg8UVm+6AbjFzDqZ2QNFWWc1gL8A6EAfXxSVPwCwwfekG3x7xv0JZjbMzF4oLhOvQkFr5nOLyuB9AB1SJLqS44WZbW1md5rZ7GIfehlAGzPbtF7vWNQajQsbR1OdBPCuRwtQ+NHeQK/ivwGF5ZqtN1SYWRd3ohD+E0I4BkAnAI+h4FQCFBr+uhBCm+i/rUMI96fch2hgQgiTANyNwkt/PQptslMIoRWA01BYCqwJC1GY5G2gF9XfB+BxAD1DCK0B3FHGuUXD8jqADwGMLFGfNl5cBmAQgGHFPvSV4r9vaGu9800AjQs1p6lOApj7AVxlZh3NrAOAH6Ew2wOAdwBsb2ZDi84e12z4kJltbmanmlnrEMInAFYD+KxY/QcAFxZnemZm2xSdQFo22LcSVTCz7czssshZqycKvhlvAGgJYC2AlWbWHcD3yjj1QwDOMrMhZrY1gB9TfUsAy0MIH5rZngBOqe13EfVDCGEVCmPAb8xsZPGv+82KWvFNSB8vWqLgB7DSzNqhaj9YDKBvw3wTUVM0Lmw8zWUS8DMAbwEYB+BdAGOK/4YQwhQUPIWfBTAVVcODTgcwq7hMdCEKs0SEEN5CwS/gdgArAExD0VlENCprAAwDMNrM1qHwko9H4S+4awHsCmAVgCcBPFLTk4YQngJwCwqOodOK/4/5BoCfmNkaFH40HoKoWEIIvwLwHRQc/paisLJ3MQqrfSXHCxT6wFYAlqHQt/5Fp/41gOOLkQO31uuXEOWgcWEjsRC0uiWEEELkkeayEiCEEEKIMtEkQAghhMgpmgQIIYQQOUWTACGEECKnpOa7N7OK9Bp8//33nb1s2TJnf/75585u0SLZiwhTpkxxdW3btnX2Zptt5uy1a9c6u127ds4eO3ZsUv7a176Wctf1SwihXmNTK7UviKqoLxT4yle+4uwDDjjA2VtvnaQPwZZbbunqVq1a5ew5c+Y4+6677nI2jzmVgvqC2ECpvqCVACGEECKnaBIghBBC5BRNAoQQQoickposqCH1HjMvV8T3NWjQIFc3adIkZ8+bN8/Zm27q9/rYYosvdv9l7W7hwoUlj63OXrNmjbM//vjjpLzbbruhsZD2JzbQnPpC2rjAzJ8/39k8DvC7v8kmX/wNtM0227g69jvic/Xo0cPZw4cPd/aoUaNK3mdD0pz6gqgd8gkQQgghhEOTACGEECKnaBIghBBC5JTUPAENSZrW98c//tHZCxYscPbcuXOdzTpinCdg8803d3UffPCBs1n7izX/6u6TzyeEKA3n4fjkk09Sj+f366OPPkrKZ511lqtj/x3292HdPz7X7NmzXV08ZgBV8wjMnDnT2S+++KKz+XvGxL4IQOXmGBD5QCsBQgghRE7RJEAIIYTIKRUjB6Sxzz77OHvatGnO5lS+DC+/xfByIy/jffrpp6k2hwoJIUrDy/9ZIYDxkj3Tu3dvZ3Oq3zZt2ji7ZcuWzm7dunXJ665fv97ZWePAu+++W/I+GS3/i0pCKwFCCCFETtEkQAghhMgpmgQIIYQQOaVifQLiFLycwpP1ONYVOcwv1gY/++wzV8c2w/Vf+pJ/ZPG5OQRp3bp1qecWIu+khQYDVd+p2267LSkfffTRro5Dhbt16+bsrbbaytn33XdfUmZ/ghNOOMHZ7Hc0Y8YMZ3N44ksvvZSUf/CDH7i6rJTC5aRKFqK2aCVACCGEyCmaBAghhBA5RZMAIYQQIqdUrE/AnnvumZQ5dp9jh9u2betsjv2PdX3OGdCqVavU+8jKGxDDuqB8AoTwsL8O+9yw9s76eceOHZMypwXm923JkiUlPwv4LcnHjRvn6k4++WRnr1ixwtkffvihs3mc6N69e1J+/PHHXd3ZZ5/tbK7nc3HqciHqEq0ECCGEEDlFkwAhhBAip2gSIIQQQuSUivUJOOaYY5JyVl6A1atXO5vjgbfeeuuS1+G4f87rzTG6afsQ8H0IITxZeTnOOeccZ/MWvosXLy75WfbJYd0+9gEAgMMOOywpjxgxwtXxez9r1ixns26ftgX58uXLXd15553nbPYJkA+AaEi0EiCEEELkFE0ChBBCiJyiSYAQQgiRUyrWJ6Bnz55JmfcgT9Plgap6XZxHgHXDrH0J2GeA/RFijTIth4D4gqz2Yz22vnKn77rrrs5m35FXX3019fPcF2L4nrkfZfWVli1bOnvNmjWpxzcXOD5//fr1zo59CvgZss35QngcifclGDBggKtj3wNua85Nwr4Occ4CruvSpQvKgd8X9lsSojZoJUAIIYTIKZoECCGEEDlFkwAhhBAip1SsT0CfPn2SMu/1zdota30cWxzH3d5yyy2u7oorrnA270nOWiBf+6233oIoj7rUNLP2Xuf96L/+9a8nZdZm58yZ4+wdd9zR2XfddZez0+K5s3wA4tzyAHDrrbc6e+XKlc6eOnVqUn744Ydd3bRp00reR6WT1X5pew1k7c3B/j9r164tWT979uzU++B9B/ja7LPB/ggxrVu3djbnKHjxxRdLflY0PeI+Xq5/03PPPefse+65x9n33nvvxt9YEa0ECCGEEDlFkwAhhBAip1SMHJC27M5bgjK8xNKpUydnf+Mb30jKd955p6tjOSBrGZeXJydMmJB6byJ7yTerPo2sYz/44ANnx1IRh5Jyetf27ds7+9e//rWzf/aznzl7/vz5SZn7zXbbbZd6rs6dOzv7gQcecHa8xe6Xv/xlV9eU5YBBgwY5m1N+c/vGy+x8LL+bLDuxNBSfiyVE3q6cpR9OEc73GYd4sgzB/W748OHOlhzQtCknHPjAAw909qOPPursZcuWOTuWMwHgkUceScrcz/h9KIVWAoQQQoicokmAEEIIkVM0CRBCCCFySsX4BOy2224l6zgEkLW+bbfd1tmsqf3ud7/b6PvK0q/ffffdjT53XsjS7esrLTAAHHDAAc7+6le/mpRZlz/xxBOd/fLLLzubtb7rrrvO2XG/Gzt2rKv75je/6WwOR+RzDRw40NlxSCH7FzRlOAyTw/pYm+c2iOFxIuvdjVMSs57KsL7KqXzZju+Fr8vj13777eds9jVRmuCmRVr66quuusrVnXvuuc4eNWqUszk8/uCDD3b2jTfemJT/93//19VlpWdPjqvRUUIIIYRodmgSIIQQQuQUTQKEEEKInFIxPgG77757yTrWWHhrTk7Reeihh5Y8F6f9ZVhHYT3vww8/dPbrr7+eer7mSto2ulzHOi7HXHOseJs2bZwd+3ywzvvggw+m3idrcPG9nHnmma4u1teAqql9WRdeunSps/fcc8+kPGzYMFf3z3/+09kclz5y5Ehncz+N75vr0tqi0tljjz2czfo3v4/xu87x9myzjwBvSxw/Ux5DuJ/xmMPtx9eO7zur/7P/R17J8rNg4r5Sn34T5eYxYd+iONXv+PHjXR2nq+bcFzwWcuryyy+/vOR91HRre60ECCGEEDlFkwAhhBAip2gSIIQQQuSUivEJiLcOBrzGw3pcixYtnP3KK684O22LV84lz7D+w3aHDh2cPWnSpNTzNReynkusk7G+yn4U3NZ77723s3lb1vjz22+/vavbYYcdnN23b9+S9wUAN9xwQ1KO95QAgCuvvNLZnH9i8ODBzmZ9bvr06UmZtyk+5JBDnM3aH+v8K1ascHasb/OxcZ76pkbXrl2dzToma73xfg6cQ4DbOmvfj1hz5jGGYX2ar8V9PN6/hLeF5u/EviVNmTT/lHK2iQbqVufPar/YbyMr3wTz+OOPO5tzX8S/EbwFNY+VnDPiV7/6lbPTfAA2Fq0ECCGEEDlFkwAhhBAip2gSIIQQQuSUivEJ4Hjh999/PymzNsQxur///e83+rqsK2bFXGflGM8LaToZ66MM61oTJkxw9n/+8x9nx9oha+0nnHCCs1n7+7//+z9nd+zYMSnPmDHD1R155JHOfuKJJ5z97W9/29nsfxDnBuBz87F8n+wHwTn0Y92f491rmiO8EunWrZuz2Z+Hv9vChQuT8uLFi10d+3DwM03b553j/LP2HWC/pCVLljh7/vz5STnuc9XdF/t0dO7c2dn8PSuZcvYByfLD4PfxuOOOc3bcBr/85S9d3ejRo52d5V/AfgAxl156qbNZp+fYf+4LrVu3Tsr828X96H/+53+c/eijj5a8ryxq2hZNd/QQQgghRK3QJEAIIYTIKRUjB/Tq1cvZcSgFL43yknBtlkx4q0aGlxAXLVq00ddqymRthxovafJy57HHHutsTsfLbXD99dc7u23btkmZt4nmZcB4q+Dq7iVemv3Od77j6q6++mpnjxgxwtm8fLlgwQJnx8+EQxm5H/Hz69+/v7PjZW/Apx79+9//nnqupgS/97EMCAADBgxwdvyucxjlLrvs4uy5c+c6m/twLDVkyYBZaYM5zO+tt95Kyj/+8Y9d3bhx45zNYWIcNlnJckCWFBXXc8hmz549nf3b3/7W2SzV8vsWy44/+tGPXN3kyZOdzW3A6XiPP/74pMzbfvP4dPrppzub5YLevXs7O+7TvA04h0e/+eabaGi0EiCEEELkFE0ChBBCiJyiSYAQQgiRUyrGJ4BD79q1a5eU2SeAtaGsVMBpcDgb63OseU2cOHGjr9WUydKdWXuPYX2cNVEOEeT2jDW5l156qWQdUFXb5RSecejQD37wA1e31157OZu/c5Y2G+vEHHLG+ib36Z///OfOZt0/Jk3bbmrE/h5AVW2dQ6iWL1+elLnPpaUFBqr6ZaSRleI26/iXX345KXN7ZaUz5vdl7NixqdduTPh7s502brAPzjPPPOPsW2+91dnDhw93dhwyyKnI2cfmggsucDb3s5kzZyZlDjmfN2+es/ldfuGFF5zNPh377LNPUp42bZqri1ONA1XToHOK8AMPPNDZPXr0SMrsY3H22WejJjTd0UMIIYQQtUKTACGEECKnaBIghBBC5JSK8QlgbSTWNzhNKmtqtYHjkllXYT1vzpw5dXbtpgRrVRy//dxzzyXl1atXu7p33nnH2azXTZkyxdkPPPBAyfuIU3ACwG677eZsjg/mVKPbbLNNUmZ/At4SlHV7jmnneOC4n7KPC2ulnBo5zQcA8Po266zlpGqtNHhLZYZ1/WXLliVlzhfCsfxZ21/HzzHLn4CfOaeZZR04LQ1tVs6IoUOHOvuvf/1ryXM1Ntz3OEVy/L7NmjXL1b366qvOPvfcc5198MEHO3v33Xd3dpyz5eGHH3Z1rPmzjw77msR5TnisO+yww1LP9fbbb6fa/NsWw+mNeazk8SxOTc7njv0aAGDQoEElrxujlQAhhBAip2gSIIQQQuQUTQKEEEKInFIxPgGvv/66s/fdd9+knLWtZ21gDYv9DbK2fky7r6as1TIHHHCAs1mfO+mkk5JyrNsCVZ8ha/Hf+973nH3ttdc6e/DgwUmZ9VSO2eV9CXhL3ziml3Vc3ncg1jOruzb7PsR+APHeF9XB28WyXjpp0iRnx1vTcv7xn/zkJ6nXqiR462DuG/wOcR6PuE3YR2PlypXOZp2/Nu8j+wjE+QqAqv4JsR7LfkdZ98VbIlcy/fr1c/bFF1/s7DgXfpz7BaiqWfNeEOxX8dRTTzk71sDZf4DHBX7fuJ/FPgGs4fN9sQ8H+6LwM4l9i7jf8Hfi3x/2mXnllVdKXnvgwIGujseUUmglQAghhMgpmgQIIYQQOUWTACGEECKnVIxPAMffx5oN6zdZebzLgXMQtGrVKvX4jz/+uM6u3ZTgPN7MIYcckpQ5zvnYY491Nuu8rINdd911zo61vw4dOrg69gFgf4O+ffs6+1vf+lZSZt2Q83TzPhK85wFr0LGWy3kCGP4sx4LH+9EDwJIlS5Iya6tpcciVRpzrHKiqp7L2zvHeZ5xxRlLmtmYfjdr4BPB9se8C68LsAxJr46xt831w/+c+Xclw3g5u3/i7x35eQNV+y+8XjyM8Nsd5BXhPlzvvvDP1XLGPDeC1d96HgP09snwXuF/yOBOTlrsCqNqHeezcZZddkjLvwcJ77JRCKwFCCCFETtEkQAghhMgpmgQIIYQQOaVifAI4j3Sso7CGxjphbcjyAciKD84LO++8s7PjuFoAeOmll5Ly008/7epuuumm1HPzM27Tpo2z4/hXju3n/P58n2nXYq2OdUI+F8d7v/vuuyXrx48f7+pmz57t7HL7cKwdNuX8E5yXg78La56szceaM/tVsJ7KfaOcve65T7J2y/nj2Qck1so5vwSfi78j5z+oZMaMGePs888/39mxrxDv8zF8+HBns7bOvi+8x0i83wbr+HvssYezuT2POeaYkvU8LrDGz75C7NPRvn17Z8fty32O74uvzX2Y92yJ/WB+9rOfYWPQSoAQQgiRUzQJEEIIIXJKxcgBCxcudHa87MdLqZzOtTZw6A8vR2aliMwLvPS9//77OzsOiWLJhMMq//vf/zqbt8vkz7/xxhs1vs+0bYibMvFSdl2mzW5oOMQzK9U2L73GS7NZWypnhR+mhRpnfZaXbdOWiPk+s5aTOcVtU4LD5x588MFqy9XRqVMnZ3OKaU6nHG/5y/IaS4osyfC24bG8k7UNOIcSM3ztePxjqYAlRv7O3O8ee+wxZ99xxx0l76Om44RWAoQQQoicokmAEEIIkVM0CRBCCCFySsX4BDDTpk1LyqznsCbDGtrixYtrfJ2scKtywoqaM/y9n3vuuZI2a60cahdvDQwAF110kbM5JGfNmjVJmX02uK25vXhb4zglK5+Lt+1kTY21XK6Pw8Kytk/ma/G9cB+Pj587d66ry9JaKwn+XhwWlpXqN7a5PRjWU9ku57NZqcv5+FjnZ82fw77YZ4bPzWHMnB65uRCnxq7OHjt2rLMfffTR+r6lJk1NQ4m1EiCEEELkFE0ChBBCiJyiSYAQQgiRUyrWJyDW0Vj7Y5u3syzHJyBLg2QtkLVeURWOsebUomxL28sPLVq0cHaWj03a+8a+Qqy9Z+UgiPsp6/RZlLO9Oae/ZU0/S+PnbXBffvnlGtyhEDVDKwFCCCFETtEkQAghhMgpmgQIIYQQOaVifQLStgzN2tazHFhXzNrqkXVHIUTN4feN3y/eoprfx9hHgM/FsE5fjo7PsD8Bf5bzPPTu3Tspjx492tX169fP2ezjxD5NnFNfiLpEKwFCCCFETtEkQAghhMgpmgQIIYQQOaVifQJiP4CsPN61id2fOnWqszmml6/Feb6FEDWnbdu2zp4/f76zOefHk08+6exYe7/44otdHeeWZ5+BNN+hcvcI4VwYafn+DzroIFf32muvObtLly7O5vGO96AXoi7RSoAQQgiRUzQJEEIIIXKKJgFCCCFETqlYn4AVK1YkZdYROVa/a9euG30d1vyzSNuTPCtXuRB5Z8CAAc7m92mrrbZyNsffX3LJJUmZfQJ69uzp7PXr1zub/XnicYTvgzV+trfeemtnt2nTxtl33313tdcBgHfffdfZffr0QRp8LSHqEq0ECCGEEDlFkwAhhBAip1SsHBAvrXPYHqfZ3HHHHZ39xBNP1Pg6vMzHYUVZaU6FEDWHl+w51O6TTz5xNm87HcPL5Lfffruzv/KVrzibl91nzZqVlLPea77PRYsWOfuyyy5z9gMPPFDyXLfddpuzDzvsMGezJDJkyJDUexOiNugXTQghhMgpmgQIIYQQOUWTACGEECKnVKxPwH333ZeUd9llF1e3bNkyZz/zzDMbfZ1Vq1Y5m7XBNWvWOHv8+PElz6WQQCHS2X333Z3NYbVbbLGFszlEMIZDAM8555yNvi9OKcxbGvM4wT4C5cDpjTlsklMnL1y4cKOvJUQWWgkQQgghcoomAUIIIURO0SRACCGEyCkmHVsIIYTIJ1oJEEIIIXKKJgFCCCFETtEkQAghhMgpmgQIIYQQOaVZTQLM7CwzezWyg5n1b8x7Ek0TM5tlZgc19n2I2qNxQVQH94tq6p8yszMb8p4ag4qdBBQH4fVmttbMFpvZn8ysRWPfl2h4zGy4mb1mZqvMbLmZjTKzPRr7vkTDo3FBlMvGjh8hhMNDCPeknDd1EtFUqNhJQJGjQwgtAOwKYA8AVzXy/aRiZhWbhrmpYmatADwB4DYA7QB0B3AtgI8a875qgvpDvaFxQdSI+ho/mlObVvokAAAQQpgP4CkAOxSX8pIGMLMXzezcrHOYWWszu9fMlprZbDO7ysw2MbMtzGylme0QHdux+NdGp6J9lJmNLR73mpntFB07y8y+b2bjAKxrTp2jQhgIACGE+0MIn4UQ1ocQng4hjNswEzezX5jZCjObaWaHb/hgsc3vMrOFZjbfzH5mZpsW6/qZ2fNm9r6ZLTOzv5pZm+puwMy2K577pKKt/lABaFwQNaDk+LHhgJTxI+lDxbFmlJndbGbLATwI4A4AexdXpVY27NeqO5rEJMDMegI4AsCKWpzmNgCtAfQFsB+AMwCcHUL4CMAjAE6Ojj0RwEshhCVmtiuAPwK4AEB7AHcCeNzM4p1OTgZwJIA2IYRPa3GPoipTAHxmZveY2eFm1pbqhwGYDKADgJsA3GWW7EpzD4BPAfQHsAuAQwBs+GEwANcD6AZgMICeAK7hixfb/2kAl4QQHlB/qBw0LogaUJvxgxkGYAaATgBOA3AhgNdDCC1CCG3q5e4bgEqfBDxWnGG9CuAlAD/fmJMU//r7GoArQwhrQgizAPwSwOnFQ+6Df9lPKf4bAJwH4M4QwujiTPIeFJaS9oqOvzWEMDeE4Lc1E7UmhLAawHAAAcAfACw1s8fNrHPxkNkhhD+EED5D4Ue/K4DOxfrDAXw7hLAuhLAEwM0ATiqed1oI4ZkQwkchhKUAfoXCj0DMvgAeB3BmCOGJ4r+pPzQ+GhdEjdjY8aPE6RaEEG4LIXzanNq00peoRoYQnt1gmFmfjTxPBwCbA5gd/dtsFPQhAHgewFZmNgzAIgBDATxarOsN4EwzuyT67OYo/AW5gbkbeV+iBoQQJgI4CygszQP4C4BbAPwbhfbacNwHxUl8CxT0v80ALIwm9pug2FbFJd1bUfihb1ms478oL0ThL78Xon9Tf2h8NC6IGrOR40d1NMv2rPSVAGZd8f9bR//WpQafWwbgExRe3A30AjAfAEIInwN4CIVZ/ykAngghrCkeNxfAdSGENtF/W4cQ7o/OpQ0YGogQwiQAdwPYIePQuSj8ZdYhardWIYTti/XXo9BuO4UQWqGwvMfLgBcC6GVmN9N51R8qC40LokaUMX5U+/EMu0nSpCYBxWXb+QBOM7NNzezrAPrV4HOfofAyX2dmLc2sN4DvoDAj3MB9KCwNnoovlvyAwhLShWY2zApsY2ZHmlnLOvpaIoWiU95lZtajaPdEYVB+I+1zIYSFKGj5vzSzVkVnr35mtmHJvyWAtQBWmll3AN+r5jRrABwG4CtmdkPx39QfKgyNC6IUGzt+1JDFAHqY2eZ1cK5Go0lNAoqch8KA/T6A7QG8VsPPXYLCXwwzUNAS70PBsQcAEEIYXazvhoLH8YZ/f6t4zdtRWC6ehuLSkmgQ1qDgkDPazNah8PKOB3BZDT57BgpLtO+h0HYPo6D5AYUwoV0BrALwJApOYFUIIawEcDCAw83sp+oPFYvGBVEdtRk/sngewAQAi8xsWR2cr1HQVsJCCCFETmmKKwFCCCGEqAM0CRBCCCFyiiYBQgghRE7RJEAIIYTIKZoECCGEEDklNWOgmTVa6EDHjh2T8jHHHOPqVq1a5ey5c9MTOc2bNy8pf+lL/itvvrkP8WzRwieL2m8/n0n2pZdecvaYMWNSr91QhBBK5buuExqzL4jyyGtf6NWrl7Pnz5/v7M8++2yjz33cccc5+29/+9tGn6t0avrqqU0EV177QhaHHHKIs3v27JmUP/nkE1e34447OvsPf/iDs6dMmeJsbt9KicAr1Re0EiCEEELkFE0ChBBCiJySmiyoPpd6dtjBp24+8sgjnR0v2/MSPdubbrqps1es8PvAfPTRR0n5gw8+cHWtW7cued3qWLt2rbM322yzpDx58mRXd//996Oh0LKf2EAl94Vyl0qHDh3q7PXrv9i8rVu3bq7uwQcfdHYsKQLA//3f/zl76dKlSblfP59l+NRTT3X2Jpv4v5ceecQnmLzvvvucfcEFFyTlkSNHIg0ev/iZfP7556mfT6OS+0JDwrLu7bff7uwFCxYk5XhMB4D999/f2WPHjnX2LrvsUuP74H5Um7YtF8kBQgghhHBoEiCEEELkFE0ChBBCiJzSaD4BV155pbNjrQ8AZs+enZRZ++vdu7ez2Qcg1ncAYODAgSXrWOPfdtttnR37EwDAhAkTnL3NNtsk5c6dO7s69hF46qmnnF2X+pC0P7GBSu4L5fb5Zcv85mxTp04teS4+lnV+tuPP833E4w8AjBs3ztnt2rVz9tZbb13y3DwOsL8BU5chZpXcFxqS2267zdkjRoxwdqzzcz866aSTnP388887++mnn3b2Pffcs5F3Wb/IJ0AIIYQQDk0ChBBCiJzSYHJAjx49nP3tb3/b2XFWP8BnbeIlez5X27ZtnT1p0iRn8+djunTp4mzOOvbOO+84m+WBOOSwb9++rq5Vq1bO/slPflLyPmqLlv3EBppyX+BlWw7tSsv+ycQhgEDVJf8tt9wyKXPoMMPhhhxGxkvIcSbS7bff3tX96U9/cvaNN97obP5en376aeq9pdGU+0IWZ599dlLeddddXR1LsxxWztkjO3ToUPJYZtGiRc7eYostnL1y5UpnL1++PClffvnlrm7JkiXOrs8QQskBQgghhHBoEiCEEELkFE0ChBBCiJzSYD4BO++8s7Mvu+wyZ0+bNs3ZcSgf7xrIaTa7du3qbE4FPHPmzKTMeg+HG06cONHZaf4EfC9xKGJ1yCdANARNuS+89dZbzuZ3fc2aNUmZdfys3flYW1+3bl1SbtmyZep1WZuNPwsAW221lbNjn4E4jBioOl5xWDJTm5DBptwXmC9/+cvOvuaaa5Lyxx9/7OrYd4tDOFnHj30CYl8RoOougewfwr8R3F7xb86bb77p6v73f/8XDYV8AoQQQgjh0CRACCGEyCmaBAghhBA5JT3Qtg5hzS0rXW8cP8n6Dcd4zpo1y9kcI7rddtuVvC6nA+Vz87VZC+zfv39SZq2IU48KITzx+wNUTcfL/kDx+5mllbOOz8fH72uclwSo6j/AeQFiDRkAFi9e7OwBAwaUPHdaiuGa3Hdt0gg3ZY4//nhnz58/Pymzjs/PjP1H3n//fWen+Zpwn+ScEFntF+er6NOnj6sbPny4s1999VVnN0TbayVACCGEyCmaBAghhBA5RZMAIYQQIqc0mE8Ax8Zy/mWOAd1nn32S8v333+/qWP9hTYb1Hs7lHMN6Hes5nMeb9aLYl4GvK4RIh/N0sAbKvkTxluPs38M25wTheP0PP/wwKfMYwj4Aq1evdjbnIon9jvhe2M+I90lhf6jp06c7O68+AOwvwr5e7C8Sw78RvFU97zezcOHCpMw5B3gre/4NYD8x/s2I+xb7mvC20uwT0BBtr5UAIYQQIqdoEiCEEELkFE0ChBBCiJzSYD4BrMGwrjJp0iRnjxgxIin//ve/d3Wc15s1N9bz4uM57p9zfrN2FO87AFTVKGPdivcdYN2wPveKFqIpwvvAM+yz06lTp6TMPgCsxfK7zONG/O6zVss26/Zcz2NQfG72J9h8882d3a9fP2ezT0Be8wQcfPDBzmadP37m7PfFbc3jPv/+tGnTJinzvgMrVqxwdpYfGbdv7GPA/aRVq1ZobLQSIIQQQuQUTQKEEEKInKJJgBBCCJFTGswnIGs/53ivAADYaaedkvLIkSNdHcfjszbIGg77CKTVcWxqly5dnM16TxznzBpkrF8C8gmodDiOnPN877jjjs5+4IEHSp6r3LbOq+47ZMgQZ/P35ucYa72cv5/HkCyfnHgM4jGDxyceY9q3b+/sOD884H0IONc869WcY+Dpp592dl76ArPffvs5m8fXWE/nWH32EWBfLs4hEft4cD/J2leC322+z9jHgH3Q2FeO+wL7ytUHWgkQQgghcoomAUIIIUROaTA5gOElFLbHjx+flHnJhJfm+LMc4hEv+actLwJVQzZYLuBlpHgZkO+Dz81LiLz9qGhYrrjiCmdzCs+5c+c6e/vtt3c2t98LL7yQlLOW/7PCXJl4Sfiggw5ydc8991zqZysZTsnKS9+8fBrXx2FdQNXl/wULFjibpbx4mZdTxXI4GsuCvMTP9xlLFXPmzHF1fK2hQ4cijbzKATwWs+QSS8y8ZM+wXMB9Ia5nKSHeFhrw2w4DVaUjJv7N4XGBf4923313Z0sOEEIIIUS9oUmAEEIIkVM0CRBCCCFySoP5BLAezlsJs0bzzDPPJGXW41gbYlgfin0EWOPnVKOcspOP53ST8daP7LuQlc5Y1Iw4fK5cfbRjx47OHjVqVFJmre+qq65ydq9evZzNPgHPPvussx9//PGkfOmll7q6WbNmOTvLByDu/4DXKffYYw9X15R9AjhlLodfcfvF7zL7bHDIYN++fZ3NIYTxu89bGnNIGX+W+yH7eMQ+BWm+CEDVsLC8wto6vyOsn8c2j61Z4wSfOw4P5nBdPpZ/M5g0fyAOQ+Zzs0/AX/7yl9Rr1QVaCRBCCCFyiiYBQgghRE7RJEAIIYTIKQ3mE8BaCMfyDxs2zNk33HBDUr7oootcHesorMdxiuJYx2ddibUkzkHAqX/5WvPnz0/KrCNyemM+17x585AHslLiZtWn6XscS3zttdc6++yzz3b2L3/5y6R8/vnnu7oLL7yw5HWAqv4g3H5x/D5vQf388887m1MOn3zyyc7u2rVryWsdcMABri5+V5oa7EfDuRc4vnv58uVJmd9d9hXic/O4wPH6MewDkBXfzeeK+yz7Q/G4wH4ReYWfA7/3ablleFzmfsP1fO60WH/2U+G253r2GYjHtywfM/ZjaQi0EiCEEELkFE0ChBBCiJyiSYAQQgiRUxrMJ4BzcXOsLOv8cVw06zlsZ+ksMeyLwDohn5v1H84bHe9xcPrpp7u6iRMnOrtHjx7OHjNmTMn7bGqk5cfOitnNqo+39P3Tn/7k6kaMGOFs9h/h7X8XLlyYlNlfgHX42bNnO5t9PrhvxP4h3K9Yx2ebfVHicwHep2bnnXdGU2XXXXd1Nrc9v8vTpk1zdvxcOF8Cb+Ea+w8AVZ9xfG3Wcdnm+8zK8xBfm/sC3xePOZyzgPthc6Vdu3bOZq09bZz48MMPnc1+Rvyucu6GeCtp3lZ64MCBzs7yN+Dfn9g3hfsVn6tnz55oaLQSIIQQQuQUTQKEEEKInKJJgBBCCJFT6s0ngPMCsA8A6yYc4zto0KCk3KJFi9Rzsb6TRlpscHWwjshx6bF+NGHCBFfH8cCs9TUlsmL50/Jll8u3vvUtZ8c6P9/H1KlTnf3iiy86+5prrnH217/+9aT85ptvujre275z587OZq2QY4tjmzXKt99+29msBbK/Ae9PH/fbbbfd1tXF70qlw5ona+uc///JJ590dvzcvvKVr7g67oPcV1i7TYPHCT5X1n4AcV/h9579PWI/FSDbN6W5wu8Atye3X9qYw+M0j+PcvrFWz/fBfmR8H1zPfTruO1n+Utw3+HusXr0adY1WAoQQQoicokmAEEIIkVPqTQ7gkEBejuGlVA6Xi5diWTrgpTcOJWHi5ZusJUEO4eDlF17OjJd3ONUoLwmnpaasdHjZqn///s4+4YQTkjKHUe6yyy6p5+b0rrzF79VXX52UR44c6eo43S6HZXK/u+WWW5Iyb/f75z//2dmnnXaas3n76zRJJCv0jeUyJi0NLVPOMndjw2FgvDzK7x+PC3EYIId98bjA8DgRj1FZqaq5/fi+OQwwXsLnOrZZauDtZN944w3kAf7NKGeZneUzfqbcXtxXYgmA74P7Ar+baemMAWDVqlUlz83wu8yp5iUHCCGEEKLO0CRACCGEyCmaBAghhBA5pd58Anj7TIb1nmXLljk7To3Kum7r1q2dzVpR1rafMawNcTjiggULUq8Vh3DwdVn/6dixY8n7aGo8+uijzo59JX7zm9+4uji1MgDss88+zubQPN6i96ijjkrKQ4cOdXVZW8+y78J2222XlDn0ivsJ97s2bdo4m/W7WM8uV69m/5G0NLV8LD+DSobfXX4u/EzZvyROJ54VtpqmIVdnp9Vl6b483o0aNSopc5rgvffe29ncz7jP5oWsUHBuz3hs5lA67jdZ4YbxuMG/TezHwv2Mz8XjfuynxP4EPKbwuNGlSxdncxrtukArAUIIIURO0SRACCGEyCmaBAghhBA5pd58Ajhuk/UdjgdmrST+fJY2xHoPx+PHx3OsMOtxfN+s9/Dx8fk4TTBvHVyXqXUbGtb5+/Xr5+yxY8cm5ZNOOsnVsdbH7ccx82mpnTlWn30A+L6mT5/u7Fi7ZX2at6LlmFzuO7E+zWS1NZ+L49C538U6ZFb8dCXD35NhLZ5TuMY5JPgZ87lZu+VxIa7n6/L4xM88K+48vhfOH8KpkeM4cqCqvp0XWEvnd4TH5ngsYL8iHotZa2c7zkXDPhzcXlk+Adx+s2bNKnnsXnvt5WzuV9z/6wOtBAghhBA5RZMAIYQQIqdoEiCEEELklHrzCciKi2ZthLdpjTU11uFZu2X9Lk13ZP2U9Wc+N8dx8pbH8bVYn2Z4DwTWpSrZZ+CRRx5x9le/+lVnx/H7HLfOemrW3g+8TWuswbE+x+03adIkZ3PfiLdtnTx5sqvL2uaTSetnrDGn5aqo7ngm7recez7OqVHpZD1jfk5z5sxxdpxXn99V7gt8Lh5z4meelUOA25rHkbT8FNwns97zLL+J5gq/q1m+X7EvBecF4PbIykEQjyvso8TjV5Y/G+f7j6/FPk3sp8L3zb8/9YFWAoQQQoicokmAEEIIkVM0CRBCCCFySr35BHDMZ1aO8LSYUI75zMpBwD4EsYbKempWjDXnBGefgDivdFYOd9aWmhLPPfecs+O9AgDg8ssvT8qnnXaaq9txxx1rde00DZXbj3Vg1utiPa+5aK+HHXZYY99Cjcna14N9BPhd33bbbZMyv+dZfSEt90KWTwbrwPw9+N2O47u5/3McOvfDSvYNqk/YF4h/Ezhm/u233y5Zl5VrgZ9xPE5wP8ryB2FY54+ZOnWqszt37pxa3xD7zWglQAghhMgpmgQIIYQQOUWTACGEECKn1JtPAGvprJmxVss6S6zdx/tGA1X1HM4Bz7m6Y22Q9R3WDTmelO+bj4+1p3/961+ujuO3+b5Z72lK+8IzN910U7Xl6uA42iFDhjg7jgUHgK5duyblrLjZNB8AwLdBVo4Ibnu2P/zwQ2fHunLW/vNZvgusG8fxxXyuF154wdlXXHEFKpWsfd35ORx99NHOjt8Zfv5Zun3atdgngH0TsvYpYDs+d/fu3ZFGVl/IC6ylc3vxM4739sjyq+B3l33WYl8vzuey0047OXvZsmVV7j2G+1KXLl2SMue9yOp37CdRH2glQAghhMgpmgQIIYQQOaXB0gbz0g5vw8r18bLftGnTUq/FKYf53PGyIadl5OuyzUtUbMfLTrzkxEtU/EzSQkmaMyzXsP3iiy824N2IhoTfEV4O5b5w8MEHO3vKlClJOSukLCvsL152ZymBl+j5XPw90rYv5xBnlsN46ToOO84TvETPYzETt1laSmgge6vutm3bJmV+/hyKmrUtMf/GDBo0KCm/9tprro77UVYa7fpAKwFCCCFETtEkQAghhMgpmgQIIYQQOaXefAJYg2HNjMN7mHfffTcpsybDYWLdunVzNqe0ja/NGguHhbFuz/4GTJzWlMML+b65nlMYC9HcYc2T7axtpXv06JGUWbvld5XPzZpzfK2s9MUMhzRzmtr4fDymZG2Rm+XL0FzJCsNk4r7BYyuHWWb5X8VtlJUmOMuPjH0COFw+jSx/g/pAKwFCCCFETtEkQAghhMgpmgQIIYQQOaXefALStgYGqmrxrH2cdNJJSXnevHmubv78+c5mLYnTPsZ5A/g6rP9wKt/+/fs7m/0R4pjfm2++2dWlpRiu7r6FaO6wLxBr76yvchz1qFGjkvI222yTeixr62kpWPm+stJPZ8Whx2lov/zlL7u6Xr16OZt9h9inKS+sWrXK2azzcyrttC3K+feH+1majxrndRg4cKCzWfNn0lLg9+3b19VxXgzu/+zHUh9oJUAIIYTIKZoECCGEEDlFkwAhhBAip9SbTwBraOwDwPmz33jjDWefc845SZm19XhrxurOxVpSrNGw5sLb92Zt6cq61eTJk1EK1p3irS+ru08hmjuspbN2y+/MXXfd5ewbbrihfm6sHonHMgC48cYbnc1jUNZ22c0V3qKXfTzYh2r48OFJmcdl/s1gnZ79xlq2bJmUOX9LVk6brDwC8bWOOOIIV8c5JNjPpSHQSoAQQgiRUzQJEEIIIXKKJgFCCCFETqk3nwCOic/Ktc289dZbdX5PDQ37QXAOAt7zYMyYMfV+T0I0JrH2ClTN95+1L0hMufn+02Bfhaxz8fF8L7GvQ58+fVLPnZVzIC/E+VyAqs+Bc838/ve/T8qnnHKKq+N8/dw+7F/QunXrpMx5ADhWPyu3Bd937FPwz3/+09Xtt99+zmY/iNGjR6O+0UqAEEIIkVM0CRBCCCFyiiYBQgghRE6pN58A1r8Zjg9Og3MOlPPZuiZLW9rYY4XIA1nx2/yOpMVN1+X7VK4/QTnHx/sIAFW/M8eKz50719mxXg1UjYlvLsyePdvZWc/4iSeeqLYMAEOHDnX2Tjvt5Oy2bds6u2vXrkmZf284dwznEeD2fe6555zNOXBi9tprL2ez/wGfuz7QSoAQQgiRUzQJEEIIIXJKvckBDC9zcHrQNBpz+Z8pZwmSvyMvM3HIixDNnWHDhjmb5QEOt+Jl3EqFw/xi4qVmoOpyMkseHGJ20EEHOftvf/vbxtxixdOvXz9n85bLc+bMcXa8pM+hpmPHjk21KwVua+4r7dq1q/d70EqAEEIIkVM0CRBCCCFyiiYBQgghRE6pN58A3h6TtQ/Wd5ojU6ZMcfa2227rbE6TKURzZ9SoUc5mPZy3224qqbTTfALuuOMOZ/N3Zl+h6dOnO/vvf/97Le+uafDvf//b2YMGDXL2okWLnM1+ADENGZ7Nbc922rVfeOEFZ0+dOtXZr7zySi3vLhutBAghhBA5RZMAIYQQIqdoEiCEEELkFKvN9ptCCCGEaLpoJUAIIYTIKZoECCGEEDlFkwAhhBAip2gSIIQQQuQUTQKEEEKInNKsJgFmdpaZvRrZwcz6N+Y9ifolrY03tv25H4mmgfqCKIestjWzp8zszIa8p8agYicBZjbLzNab2VozW2xmfzKzFo19X6J+MLMXzWyFmdV8j+kmhpmNMLN5jX0flY76gqhLzGy4mb1mZqvMbLmZjTKzPbI+F0I4PIRwT8p5m8UEsWInAUWODiG0ALArgD0AXNXI95OKmdXbXgzNGTPrA2BfAAHAVxv3bkRjor4g6hIzawXgCQC3AWgHoDuAawF8VMvzNpuxvtInAQCAEMJ8AE8B2KG4rJc0QPGvhnOzzmFmrc3sXjNbamazzewqM9vEzLYws5VmtkN0bMfiKkSnon2UmY0tHveame0UHTvLzL5vZuMArGtOnaMBOQPAGwDuBuCW38zsbjP7jZk9aWZrzGy0mfWr7iTFGf9cM9u/mrotzOwXZjanuLJ0h5ltlXJPZma3Ff96mGRmB0YV3czs8eJfFdPM7Dy6zi1mtqD43y3Ff9sGhT7crbi6tdbMupX1lPKB+oKoSwYCQAjh/hDCZyGE9SGEp0MI4zYcUOwLK8xsppkdHv178ttS/Kt/lJndbGbLATwI4A4Aexfbb2XDfq26o0lMAsysJ4AjAJTeNiqb2wC0BtAXwH4oDDZnhxA+AvAIgJOjY08E8FIIYYmZ7QrgjwAuANAewJ0AHje/VHkygCMBtAkhfFqLe8wrZwD4a/G/Q82sM9WfjMLsvS2AaQCu4xOY2aEA7gdwXAjhBa4HcCMKA8JQAP1R+IvgRyn3NAzADAAdAPwYwCNm1q5Ydz+AeQC6ATgewM+jH4YfAtireJ2dAewJ4KoQwjoAhwNYEEJoUfxvQcr184r6gqhLpgD4zMzuMbPDzawt1Q8DMBmFtr0JwF1mJbeE3NAPOgE4DcCFAF4vtl+bern7hiCEUJH/AZgFYC2AlQBmA/gtgMEoLBN+KTruRQDnFstnAXg1qgsovOSborD8MySquwDAi8XyQQBmRHWjAJxRLP8OwE/p3iYD2C+6z6839vNqqv8BGA7gEwAdivYkAJdG9XcD+P8i+wgAk6iNryz2kR3p3Bva3wCsA9AvqtsbwMwS93QWgAUoptUu/tubAE4H0BPAZwBaRnXXA7i7WJ4O4Iio7lAAs4rlEQDmNfYzr9T/1Bf0Xz31q8HFvjMPwKcAHgfQudi206Ljti72ky5F+0X435Y51fSNVxviO9Tnf5W+EjAyhNAmhNA7hPANAOs38jwdAGyOwuCwgdko/AUAAM8D2MrMhplZbxRm7o8W63oDuKwoBawsLvv0RGHmv4G5G3lforDk+3QIYVnRvg+0DAwg3kj8AwDsIPptAA+FEN4tcY2OKLzg/43a8F/Ffy/F/FB804vMRqHNuwFYHkJYQ3Ub+lI3VO1nWuqtGeoLos4JIUwMIZwVQugBYAcU2uCWYvWi6LgPisVSDujNcpxvavr1uuL/twawuljuUoPPLUPhL4zeAN4r/lsvAPMBIITwuZk9hMJS42IAT0Qv9lwA14UQqiw7RmgXpo2gqMOeCGBTM9vwMm4BoI2Z7RxCeKeGpzoBhWW8+SGEW6qpX4bCBHL7UPAvqQndzcyiwb8XCn9BLADQzsxaRn0k6UvF+t4AJkR1G5Z61U9KoL4gGoIQwiQzuxuFleB/l/vxDLtJUukrAY4QwlIUXrDTzGxTM/s6gGodg+hznwF4CMB1Ztay+Nf+dwD8JTrsPgBfA3BqsbyBPwC4sLhKYGa2jZkdaWYt6+hr5ZmRKCynDkFh9WUoCkt3r6CgDdeUBQAOBPBNM/sGV4YQPkehHW+2L5w9uxe141J0Kp5vMzM7oXhf/wwhzAXwGoDrzWxLKziJnoOChg0UNOKrrOBc2gEFrXlDP1sMoL2ZtS7ju+WFkVBfEHWMmW1nZpeZWY+i3ROFP/beqIPTLwbQw8w2r4NzNRpNahJQ5DwA3wPwPoDtUXgJa8IlKKwkzADwKgo/9H/cUBlCGF2s74aC5+6Gf3+reM3bUXBMnIaCFiRqz5kA/hRCmBNCWLThPxSe9alWRqRFCGEOCoP/9636aJHvo9B2b5jZagDPAhiUcsrRAAag8JfjdQCODyG8X6w7GUAfFH5wHgXw4xDCM8W6nwF4C8A4AO8CGFP8N4QQJqHwwzCjuBStpeEvUF8Q9cEaFBz6RpvZOhR+/McDuKwOzv08Cqs8i8xsWdbBlYp5qUsIIYQQeaEprgQIIYQQog7QJEAIIYTIKZoECCGEEDlFkwAhhBAip6R63JqZvAabCCGEUqku64RK6Qubbrqpsz/77LOyPv+lL33R5QcOHOjqevbs6ewePXo4e9Ag70DetWtXZ2+zzTYl65Yt887DL730krN/+9vfOvuDDz7AxpKXviCyyWtf4Hf15JNPdvZ7773n7H322ScpT5482dXNmTPH2Xvs4TcgfPbZZ5396quVubFgqb6glQAhhBAip2gSIIQQQuQUTQKEEEKInJKaLKgh9Z7SuzcCWQmNWCf+f//v/zk71nQ222wzV7d+vd+T6KCDDnL2iSee6OwpU6aUvI9NNvFzKr7v+kzM1Jy1v/i5fv75565uyy23dPYVV1zh7J133tnZQ4cOTcrt2rVzda1atarNbWLhwoVJmfvkqlWrUu158+Y5+9hjj3V2/H5k9aPm3BdEeTSnvrD77rs7u3fv3kl5r732cnU8LvA7M2PGDGe3aPHFnkHvvuv3ntp6661T76t79+7Obt3aZ4J++eWXk/J//vMfV7dy5crUc9cl8gkQQgghhEOTACGEECKnaBIghBBC5JRG8wlg/Zy13jRYy+UY68039zs7xnpqy5Z+B+BPP/3U2R9++KGz16xZ4+xrr73W2dOmTavBHdc/zUn7Y7bYYouk/NFHH7m6k046ydl//vOfnc3tE7cv6/Lcb9q2bets7ivsQxD3Le43ixYtcnaXLl2c/f777zt71113xcbSnPtCcyTOXQFU7We1obH7Qjm+LKeeeqqz+R2J83AA/p2aPn26q2OfAM4nwueK331+/uyvxudmPzH+fJyzgH9/eAxavny5s//973+jrpBPgBBCCCEcmgQIIYQQOSU1bXB9krX8H4eDnHDCCa6uW7duzualHV76iVO28tLbihUrnM31LD3ceOONzo6Xgv7617+6uvHjx0PUnk8++aRkHS/NrV271tm8LB+37zvvvOPqWA7gtm/fvn3quePlTu7fHKrI9839jqWG1atXQ9QtsSRZ23Deww8/3NmxnHPYYYe5Og4p45TSV155pbMnTpzo7AULFpR1b41J2nM8/fTTnX3IIYc4++GHH3b2rFmznL3VVluVPDcvs/P7yKF58W8KL+fzu5cVCs73NXXq1JL3EYcmAsCIESOczTLhW2+9hbpGKwFCCCFETtEkQAghhMgpmgQIIYQQOaVi0gZz6F28zevHH3/s6tatW+dsDuFgbTcOuxgwYICrmzt3rrNZz4nD04Cq2m1cz6liebtK1vrqksYOBapP0tIG//rXv3b28ccf7+wJEyY4O94emNODdujQwdnc1rw98Pz5850dpxft3Lmzq+PQIPZj4b5z9dVXO/uGG25ATWnOfaEuSetXDKdxvvXWW53N207HOjC3LV+L+wKnNud+GY8rw4YNc3XsE1NJfYF9ao444ghnd+rUydm8pS+P+7Fezj42TDlhl+xTxmHjWSmJOc1wnJqe/Zv4t4rHHL4Wpzsuxz9EIYJCCCGEcGgSIIQQQuQUTQKEEEKInNJoPgGnnXaas1lzW7x48Uafm3WWWLfnmE/WYBj2CWBtKdb7OH6UNeSxY8c6+/LLL0+9djlUkvZXD9dOytxfH3jgAWfzdqOsoQ0ePDgpZ/kEcAwvX5vju2P9bsiQIa6OcwpwSuIsTTNtq22mOfeFuiT2/+Etxc8//3xns88S5xfhdz8eg1jj5xh1zgnB/Yy18Fhb521r2aepkvrCPvvs4+x+/fo5m58ht0m8JTzgx25+xqy987jNdnxt9kFj+F3lc/FvSuynkaX5s3/I7Nmznc1+E6+99lrqvcbIJ0AIIYQQDk0ChBBCiJyiSYAQQgiRUxpt7wCOb2XNNA3WR1lD4zjcePtZ1sxYO2LNhs/Fdnw830ddbg+bZ9L8Vjiml3XEtPzirK9xfC/rjB988IGz2fck/jwf26ZNG2f/4Ac/cPbNN9/s7DfeeMPZZ511VlK+++67IcqH3+24r+y1116u7qqrrnI26/jcrzgPRNz+nCeA/Yy4n7EmzXsNxDn0eWvtiy++GJVKz549nc16OG+jy74S7LMTx+9zLD8/0yxiXb9czT8rB0F8PJ+bxyCO++ffpyx/hY1BKwFCCCFETtEkQAghhMgpmgQIIYQQOaXRfAJY72FNJ9ZRsuJHWf9J05CztD32N8iyY82G75O1onJ1qryS5fMR06tXL2ez3sp2TFYeAO6TWe0Z+4uwdpe2x3h1cF71J598MinLJ6Du4bh/1tqz9hBhH5C4/Tn2m/sCXyvNjwXwPgYdO3ZMPVclwXlTVq1a5WzeW4CfOedLiH8H+BlzezL8fsbjeJaPRta5uG/EuRzYh4lt9i3hc3N93P5Lly5Nvc9SaCVACCGEyCmaBAghhBA5RZMAIYQQIqc0mk8Ax0tyPGSsfXCO9jSNGKiqzccaM2swrD9zHgC+Fp871o1Zn+PPsqbM+eOzdKy8wM8tbe+AWCsHgFNPPdXZrK/GfYm1O9YCua352tzesSaXFTt86KGHOvuJJ55wNsdM33vvvannE9nwux/De9dzbH9W7Dj3jfha/F5n9Q2+FuvCsVbO3+kPf/iDs3//+9+nXqu+6dy5c1Lm9y3tewFV8wrwb0Sch4VzsmTtFcD3kuY7lDbmV1fPdpzXYdttt3V17MvA34Pvk/PpxHuUvPTSS3zrNUIrAUIIIURO0SRACCGEyCmNJgfw0isvy3fp0iUpL1myJPVYXsLnpbx4yYxDE/lcWUvAfO54SYuX97O2Le7du7ezJQeUzw033JBq8zJvnL6XQ2o4zIvhtuf2fOGFF5LyQQcd5Oq4beM0wABwySWXpF77d7/7XWq9yCZLRoxhCZJDUTm8jceNGB6feAk4i3grWsAvXfPYWGnsvffeSXm//fZzdffdd5+z+/bt6+wjjzzS2b/4xS+cHS+z8zPNCutLay+uY3mGQ9RZuuO+E6cXZ1li//33d/Z///tfZ//rX/9y9m677ebseDyTHCCEEEKIstAkQAghhMgpmgQIIYQQOaXBfAI4RSvrq6zXxdo9f5a1dobPHWtyWT4AWaRtW8zbQnKaS742Hy+qpxwtl+EQwTiFZ1wGqrYPa3us7b755pvOjn1CeOtg9gno06ePs4866ihnc8hg3KfTQt1E3TBt2jRn77LLLs5euHChs3kb6lj7ZR2YfU+4X/GWuWnpdd97770q915JPPbYY0k51vAB4Mwzz3T2t7/9bWf/5z//cTaPp3GIblYoHfsM8LliuD2yUtxz2/PnY7jt+/Xr5+wzzjjD2bzl+OjRo53NIdIbg1YChBBCiJyiSYAQQgiRUzQJEEIIIXJKg/kEsDabFY8fw3oPa2xxHCZQNUY0zkmQlt6zOjsrJWSs1bKWN2fOHGdz2kvWkkQ2rNuz/wf3DY6jjv1LuF+x7wlvO815Bbi9+/fvn5S5n3BaUtYVOS1wu3btnC0/gIZlzJgxzj7hhBOcze2RpjFznDn3O04Vy+MZbw8ca+GcB6OSGTt2bKrN4+X06dOdfcoppzj7nnvuScppaX+B7G3AY62efQB4jOH2YdLyDLAvEGv+X/va15z961//OvVadYFWAoQQQoicokmAEEIIkVM0CRBCCCFySoP5BHCcZpZmGuuxc+fOdXXx9pRAVT0uLQcBa0NZ+QqYtG0lWX/OyhHOGrTIJmt7ZmbBggXOjrfy5JzfHHM9YcIEZ3NeAc7zEMdvsw7MfZTjhePPAlX3Hnj22Wch6pb4XebYbtbp2QcgK8dEDLct+4Ow/wePI5yrPmbixIkl6yqBtGfM3Hzzzan1xx9/vLMHDhyYlNl/h58ZX5vzCMQ+Atw+fGzXrl1Lfra64+Mxqnv37q7uoYcecvZbb72FNNL62cbmU9FKgBBCCJFTNAkQQgghcoomAUIIIUROaTSfANZy2Scg1jc4FnbAgAHO5r0EWI+N9TzWb1jjz8oTwDGiixcvTsrjx493dYMGDXI2a9Dl7lsgyictDwTXcdvusMMOzn766aedPWrUKGf/8Ic/TMqsQXKOCO6jrPVxvLB8AuqeNI2ax6usXCTcfnEuAK7jsS/ts1nE408lUpt9PxjW/dPquH041wK3QZzHhn0C+FjeU4T9DzgnTpwPhj9bLuzDVhf5Q/QrJIQQQuQUTQKEEEKInNJoWwnzMgYvj8bbZ86fP9/VsXTA50pbguJjeUkwa3mFl+7iZUJe6hk2bJizOVyNl3ZENiyhcPvxM+Ul/XhZkD/LYV8cxhenawWAwYMHOzsO9eIwL74vXvLl5UyllG5c9ttvP2dz+/DyMm8d3bJly6TMUhAvL2elLk+Dt6huzvB3jcdeflf5feMU3yzNxvC4ELclUPXd5d8ulhPiMYv7TRb8e8P3Vhdyi1YChBBCiJyiSYAQQgiRUzQJEEIIIXJKg/kEsG7CIRycCjjeRpI1FvYJYI0mLbUsH8uaS5bGwvWxzekiOVSEvzNr0CKbrNSjN954o7M5tW8cUsWhpdwXWMvdfvvtnb3zzjs7O27/rG2j582b52zWCssJExO1h0MCR4wY4Wz2S+LtZnl8i/Vq7mfc1qxfcz9M8w9h35NKoy5DBFnHj8fPrK26+X1M89Hhts0KQc9KXR7/DsS+bhtD1vi3MWglQAghhMgpmgQIIYQQOUWTACGEECKnNJrwyLoY615xzD2n+s3SmVhPZT0ohvU3vhZrR6w9xffNmhV/ltNLaivhuueoo45yNmtwccwu97mxY8c6m7eT3W677ZzNseOsJcawPwj3I04xfdNNN5U8lyiQ5c9Tjr/PT37yE2dnjSGcF4Dj1ONYf9aM+dz8WYb17Zh99tnH2WPGjEk9V1OGx9f4OaY9I6C8dLtZaYO5nn0EuK/E/Y5/XyoBrQQIIYQQOUWTACGEECKnaBIghBBC5JQG8wngmPksLT72CeDczayhcewkazixzceyPpeVz5/1njgeeNGiRa6ua9euqfel/PA1I+4rrOuOHDnS2d27d3c2x+PH/Yz71b/+9S9nT5kyxdnnnXees/faay9nx30ja4tq3odg5syZzn7ooYfQHCg3D0dtyNpXgrnkkkuS8ne+8x1X9/bbbzu7ffv2zk7LFwL4MYrHK85JkLX3CT/DeA+SAw880NXdfvvtyAtxe/Mz462fuT5N1+ffhKzfDCbt96cS0UqAEEIIkVM0CRBCCCFyiiYBQgghRE5pMJ+ALl26OJu1ddbzpk2blpRZu2WNhmOu03T7LJ0w7bNA1XjvWNdnDZmvxd+R40tF9aTpyI8++qizx40b52x+5j169Cj5WfYJ2HXXXUt+FqjqP8J9I4Z1XdYk58yZU/KzlQ5/tzS/miz9m+H2i69V7rmOPPJIZ3/rW99Kysccc4yru+uuu5zNe9mzzs9tH9ez/wc/H/6OnIOAx6B4jwrOXdGc4Rz+8X4z3bp1K1kHVM3pwbp+PI6n5RAAgB122CH1vti/LfYBKdcnpj59aDaglQAhhBAip2gSIIQQQuQUTQKEEEKInNJoPgGsdfBe7a+88kpSPuGEE1LPnaWxxfoPx2xmxRazvsPHt2vXLinHuQ2quxZre5w7Ia+UE9/N+cM5nvv+++939g033ODsN998MynzXg5nn322s/fbbz9nsxbIn4/7GX+nrH7373//G2nEx9fHnuK1gd/luoyLrs13Pfnkk519xRVXOHvnnXdOyt/97nddHe/rMX36dGfzeMY+AXE9+0yw3xH7PLHN+6zEz6Rt27ZorvBzO+2005z9zDPPJGXW+Pmz69atczb7kfXq1Ssp854ha9eudTb7h3C+F27f2MeA/YyeeuopZzfGu62VACGEECKnaBIghBBC5JRGSxuclcYxXpJJ26IVyA7FS6vLkhLStiEG/HIcb1sbh/IAVUMCeZmvucBLcVlpOLNCcm655ZakzMufsRwDAC+88IKzuT1vvfXWpMzLfF//+tedze3F9839Mg7zY+mHnwmHmD399NNoLnTo0CEp8/vD70ht4LTNV155pbO33XZbZ998883Ovu6665JyHC4IAEuXLnX2gAEDnM2SB6cCjvsO9zMeC3k5mcdG7odx3+F+1pTISinNchwTp9o+/PDDXd3cuXOdvWTJEmdzevH4XriP8nvOz5zHcf4ti2ViDjNmeeCtt95CQ6OVACGEECKnaBIghBBC5BRNAoQQQoic0mA+AayZcejd/PnzS362U6dOzmY9jjXlLI05hnVetlmnYo0z1vPYB4C3sc3aPrm5kPXMmFhDBqqGdl1wwQVJmUNqrr76amfvuOOOzuaUnueff35S5q2fObUoa4Gc/jUtlSz3dw4n5L7C6Y6ZSgsLjOHQylNOOSUpz5gxw9Vx6B2/qz179nQ2h8v17t07KXP7sF8Fh49ec801Ja/F7yqTFcbHvidx+2b5GS1cuNDZ7C+SRlNOPZ6VErd///7O5ucUP0fe6pl/T7gN+vTp4+w4xJP7KGv8TNwngar+JPG7z/4G7GsinwAhhBBCNBiaBAghhBA5RZMAIYQQIqc0mE8A69+sZfE2vDEcC876Ksf6s/Ye6z1ZaYGz4rv582nbFE+aNMnZvEUo69XNlaOPPtrZrVu3djZrfxwfPGbMmKQ8ZMgQV3fQQQc5e/Hixc6eOHGis2NdkfXoY4891tncRzn1KMd/x2mFOXaY/VoeeughNBe++c1vOjvW2vndZT8KTgMdtzVQNVVz/Nw4ZwTnDdhjjz2czVp7rBuzxs8+HdzW8VbnQNU49Fjv5nNlxZWzvs3jXVxfyb4itYW3B+ZnHPuGsV9RnBIaAF5//XVnd+3a1dmxDwePIfybwO05bNgwZ/NvWezvw+mmeTzjnCp1mYK7FFoJEEIIIXKKJgFCCCFETtEkQAghhMgpDeYTwPoq6/jvvPNOyc9y7DfriKyZcUxoWjwqa/5sZzFixIikzDnAWY9mzbK55gmYMGGCs3lrTtZyWZtnvS7Otc7aLcM5wXl758GDBydlju/lPsk6MLcX56aPtdoDDjjA1e20007O5ph1pjG0wY3ljTfecHYcvx8/b6CqLs/PeLvttnM2P4dDDjkkKbNWm3Utfj/jMYnfe353Oa8Dtz33y7jfpV0XqOo3wfkqOA9+7Afws5/9DJVM/Fyz8gLwc+I9FsaOHevs+Dnyu9uvXz9n81bPvJVw/BvC52K/C74vziuQNm5wn+W25jGJt7Bmynm+pdBKgBBCCJFTNAkQQgghcoomAUIIIUROaTCfANYrWGfh+OAYzhfPugrn4mZdONZ4svYGYDhfNetWf/nLX5Iya5CcN33fffd1dnOK8R06dGhS5rzc/BziY4Gq+bQ5j36sx7I+yvD+3IMGDXJ2HJOd1SfZV4HzPPz97393duyr8vDDD7s6trOoZB8A5qKLLnJ2/I6w78OFF17obM7/z+8Xx9THuj+PA/zM2J+A2zvWY9mPiP2QGG577ivxfU6ePNnVcfw7+0Gwj9Po0aOdHfvY3Hnnnan32djEzzzLz4Vj6Pk5cK6N+F1mv6NevXqlnov3qIhtzu/C/Yxt3lOE+yznMIhh/wH2h2KfAPZd2Vg/gBitBAghhBA5RZMAIYQQIqdoEiCEEELklAbzCeA4W9ZRZs6cWfKzP/rRj+rlnuob1nNYA8vap7opcdVVVyVl1tbZByDLL4M1ualTpyZljuHt0aOHs9k/hPW5OMaXdVy+L87hzjnFzz33XJSCdcOs78z3Uslk5dKIfWMuv/xyV8f2SSed5Ozvfve7zt5tt91KXoefYbk5PmI498gvf/lLZ990003O5jz2F1xwgbP/53/+Jyl37tzZ1XFegMcee8zZ7DPA+yfcdtttaIpk+blwLg3+zWD9PI795/ZjPwzO9895aeK+xGMM+zLwuWbNmuVs9iOLcxLwd+I+zP5sTF34ADBaCRBCCCFyiiYBQgghRE5pMDkgK9xq3rx5JT/L4TuVFFqXlrYxXsYGqi4Dpn3npsaoUaOS8qGHHurqePmMl205XOfiiy8ueR1+xrwdM4cC8bJ83Ab8WV6u5G1vDzzwQGdzSuIYTlPanMhakiwnlekDDzyQajNxmm6WCvj9Yji89MUXX0zKLEGVC4fqxamUeQmYQ4mz0s4uWrSoVvfWmJTTF7g9OXSS36k4tTOHBHL6XQ4v3H777Z0d/x7xOMC/Vbxkz2PMV77yFWfH58tKGc1jTkOglQAhhBAip2gSIIQQQuQUTQKEEEKInGIZ2+zWWTzCpZde6uwhQ4Y4+7zzziv52abqE8Dcc889zmafgB/+8IcbfR8hhI2Pj6oB5fQF1uN4C+X+/fun2rzVcOxTwCGADOuG69atc3asI3IIZ+zXAABz5sxJvVYa9ZHes6ZUUl8QjUtT7gucfpx9J8aPH5+UOVT4zDPPdPbPf/5zZ/P7GIcnzp4929XtvffezuZrxfcBVA0RjMMX2QeAx6c0P6PaUqovaCVACCGEyCmaBAghhBA5RZMAIYQQIqek+gQIIYQQovmilQAhhBAip2gSIIQQQuQUTQKEEEKInKJJgBBCCJFTNAkQQgghcoomAUIIIURO+f8BwbuM99hcfLUAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 648x648 with 16 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot more images\n",
    "torch.manual_seed(42)\n",
    "fig = plt.figure(figsize=(9, 9))\n",
    "rows, cols = 4, 4\n",
    "for i in range(1, rows * cols + 1):\n",
    "    random_idx = torch.randint(0, len(train_data), size=[1]).item()\n",
    "    img, label = train_data[random_idx]\n",
    "    fig.add_subplot(rows, cols, i)\n",
    "    plt.imshow(img.squeeze(), cmap=\"gray\")\n",
    "    plt.title(class_names[label])\n",
    "    plt.axis(False);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f356fbe9-95b1-4f81-a82d-dc15b3adc06a",
   "metadata": {},
   "source": [
    "Hmmm, this dataset doesn't look too aesthetic.\n",
    "\n",
    "But the principles we're going to learn on how to build a model for it will be similar across a wide range of computer vision problems.\n",
    "\n",
    "In essence, taking pixel values and building a model to find patterns in them to use on future pixel values.\n",
    "\n",
    "Plus, even for this small dataset (yes, even 60,000 images in deep learning is considered quite small), could you write a program to classify each one of them?\n",
    "\n",
    "You probably could.\n",
    "\n",
    "But I think coding a model in PyTorch would be faster.\n",
    "\n",
    "> **Question:** Do you think the above data can be model with only straight (linear) lines? Or do you think you'd also need non-straight (non-linear) lines?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43cdd23d-bd1f-4e8c-ba20-22d2b6ac14b1",
   "metadata": {},
   "source": [
    "## 2. Prepare DataLoader\n",
    "\n",
    "Now we've got a dataset ready to go.\n",
    "\n",
    "The next step is to prepare it with a [`torch.utils.data.DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.Dataset) or `DataLoader` for short.\n",
    "\n",
    "The `DataLoader` does what you think it might do.\n",
    "\n",
    "It helps load data into a model.\n",
    "\n",
    "For training and for inference.\n",
    "\n",
    "It turns a large `Dataset` into a Python iterable of smaller chunks.\n",
    "\n",
    "These smaller chunks are called **batches** or **mini-batches** and can be set by the `batch_size` parameter.\n",
    "\n",
    "Why do this?\n",
    "\n",
    "Because it's more computationally efficient.\n",
    "\n",
    "In an ideal world you could do the forward pass and backward pass across all of your data at once.\n",
    "\n",
    "But once you start using really large datasets, unless you've got infinite computing power, it's easier to break them up into batches.\n",
    "\n",
    "It also gives your model more opportunities to improve.\n",
    "\n",
    "With **mini-batches** (small portions of the data), gradient descent is performed more often per epoch (once per mini-batch rather than once per epoch).\n",
    "\n",
    "What's a good batch size?\n",
    "\n",
    "[32 is a good place to start](https://twitter.com/ylecun/status/989610208497360896?s=20&t=N96J_jotN--PYuJk2WcjMw) for a fair amount of problems.\n",
    "\n",
    "But since this is a value you can set (a **hyperparameter**) you can try all different kinds of values, though generally powers of 2 are used most often (e.g. 32, 64, 128, 256, 512).\n",
    "\n",
    "![an example of what a batched dataset looks like](https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/03-batching-fashionmnist.png)\n",
    "*Batching FashionMNIST with a batch size of 32 and shuffle turned on. A similar batching process will occur for other datasets but will differ depending on the batch size.*\n",
    "\n",
    "Let's create `DataLoader`'s for our training and test sets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "bb2dbf90-a326-43cb-b25b-71af142fafeb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataloaders: (<torch.utils.data.dataloader.DataLoader object at 0x7ff03be2c700>, <torch.utils.data.dataloader.DataLoader object at 0x7ff03be2c2e0>)\n",
      "Length of train dataloader: 1875 batches of 32\n",
      "Length of test dataloader: 313 batches of 32\n"
     ]
    }
   ],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "\n",
    "# Setup the batch size hyperparameter\n",
    "BATCH_SIZE = 32\n",
    "\n",
    "# Turn datasets into iterables (batches)\n",
    "train_dataloader = DataLoader(train_data, # dataset to turn into iterable\n",
    "    batch_size=BATCH_SIZE, # how many samples per batch? \n",
    "    shuffle=True # shuffle data every epoch?\n",
    ")\n",
    "\n",
    "test_dataloader = DataLoader(test_data,\n",
    "    batch_size=BATCH_SIZE,\n",
    "    shuffle=False # don't necessarily have to shuffle the testing data\n",
    ")\n",
    "\n",
    "# Let's check out what we've created\n",
    "print(f\"Dataloaders: {train_dataloader, test_dataloader}\") \n",
    "print(f\"Length of train dataloader: {len(train_dataloader)} batches of {BATCH_SIZE}\")\n",
    "print(f\"Length of test dataloader: {len(test_dataloader)} batches of {BATCH_SIZE}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "7a925ee7-484b-4149-be8f-3ad790172a5f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([32, 1, 28, 28]), torch.Size([32]))"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check out what's inside the training dataloader\n",
    "train_features_batch, train_labels_batch = next(iter(train_dataloader))\n",
    "train_features_batch.shape, train_labels_batch.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fee4cf8-ab73-4c81-8e5e-3c81691e799c",
   "metadata": {},
   "source": [
    "And we can see that the data remains unchanged by checking a single sample. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c863d66a-49be-43be-84dc-372a5d6fc2c2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image size: torch.Size([1, 28, 28])\n",
      "Label: 6, label size: torch.Size([])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAAD3CAYAAADmIkO7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAALIUlEQVR4nO3dzWuV6RnH8evyJWpeNIaMkpL4gq+giAzTEWE6iDsXbgQXQqGCXbQLV4M7/4JSaKGCxaFQV8UyzKIUAl2IA32ZuhihtNqItKijMzHRHE1iTGK8u9CWSfG5LuecHPM7+v2sdH7eT845mZ8P5PK+Hy+lGAA9Sxb7BQB4OcoJiKKcgCjKCYiinIAoygmIopwtyt2Pu/sfg3zQ3X/wOl8TFhblFOfuH7j7n939obs/cPc/uft3s3WllEOllPPBdcNyY/EtW+wXgGruvtrMfm9mPzaz35pZm5l9z8ymG7wu3/cWwJ1T23Yzs1LKb0opc6WUqVLKH0opf/vvH3D3n7r7mLv/290PfeO/X3L3H7749fEXd9yfufsDM7tgZr80s/3uPuHutdf7tvAqKKe262Y25+7n3f2Qu6/9v3yfmQ2ZWa+Z/cTMfuXuXnGtfWb2LzNbZ2bfN7MfmdlfSimdpZTuprx6NIRyCiulPDKzD8ysmNnHZjbi7r9z9/Uv/sjNUsrHpZQ5MztvZn1mtv7lV7O7pZRflFKellKmmv7i0TDKKa6Ucq2UcryU0m9mu83sO2b28xfx19/4c49f/LKz4lK3m/Yi0RSUs4WUUv5pZr+25yX91suT30MM5RTm7jvd/SN373/x+wEzO2Zmny/A5YfNrN/d2xbgWmgCyqlt3J7/IOev7j5pz0v5dzP7aAGufdHM/mFmX7v76AJcDwvM2WwNaOLOCYiinIAoygmIopyAqPAfQLs7Py16zTZu3BjmH374YZh/8cUXYf7OO++E+aVLl8I8Uv0vB5/jh48vV0p56QfHnRMQRTkBUZQTEEU5AVGUExBFOQFRlBMQFf7D9zd1zqk8jzt79myY79q1K8w/+eSTMD9y5EiYnzlzpu5rLybl72mGOSfQYignIIpyAqIoJyCKcgKiKCcginICot7KB9o0e+a1fn3VoevPHTx4sDIbHY0Pwmtvbw/zU6dOhXmtVgvz/fv3V2b3798P1w4NDYX53bt3w7wRynPMenHnBERRTkAU5QREUU5AFOUERFFOQNRbuWVs9+74CXp79+4N861btzb09Tdv3lyZdXV1hWu3bdsW5tl7y0Y1n39e/QCzNWvWhGsHBwfD/MmTJ2H+5ZdfVmaXL18O1968eTPMlbFlDGgxlBMQRTkBUZQTEEU5AVGUExBFOQFRb+ycc8+ePZXZ0aNHw7VXr14N86dPn4b5yMhImL/33nuVWXZ05fnz58P8xIkTYZ7NA/v7+yuzW7duhWvPnTsX5t3d3WHe29tbmfX09DT0tbPtbouJOSfQYignIIpyAqIoJyCKcgKiKCcginICot7YOefJkycrswcPHoRrs1lgZ2dnmC9bFp84eu/evcpsYmIiXLt69eowP3bsWJjfuXMnzD/77LPKbG5uLlybHQma7eeMHuPX19cXrp2ZmQnzCxcuhPliYs4JtBjKCYiinIAoygmIopyAKMoJiKKcgKg39hGAmzZtqszGxsbCtWvXrl3gVzPfunXrKrPs3Npnz56FebbX9Nq1a2G+fPnyymzDhg3h2myOuXLlyjCP5qhLlsT3ke3bt4d5K+LOCYiinIAoygmIopyAKMoJiKKcgKiWHaXs3LkzzKOtcNmj7LKRQDaumJqaCvNoa1S2LWvVqlVhno2JhoeHwzwa5WTvK9sql31u0fcl2yo3PT0d5jt27AjzoaGhMF8M3DkBUZQTEEU5AVGUExBFOQFRlBMQRTkBUS075zxw4ECYR3Ortra2cG22ZSw7vrJWq4X50qVLK7PZ2dlwbUdHR5h/9dVXYZ5tvYpMTk6GebQVzsxsxYoVYR4drZkd6Zl9z959990wZ84J4JVRTkAU5QREUU5AFOUERFFOQBTlBES17Jxz27ZtYX7lypXK7MaNG+Haffv2hXl3d3eYZ/saR0dHK7NsBhsdXWmWP94wu370eMNsz2S25zL73KL5cPZYxuxozGzGqog7JyCKcgKiKCcginICoignIIpyAqIoJyBKds6Zza2yPZXRnsnsbNjoXFmz/PzVbG/hzMxMZZadmRvNIc3y95ad2RvNKrNZYTZDzdZHn0t7e3u4dmRkJMyzz2VgYCDMb9++HebNwJ0TEEU5AVGUExBFOQFRlBMQRTkBUZQTECU75+zr6wvzbB4Yza2ieZqZWX9/f5hn+0Gz810j2fvKZomZbE9mND/OXlt07qxZPmuMZrjZPtZM9tr27t0b5sw5AfwP5QREUU5AFOUERFFOQBTlBER5KaU6dK8OxW3cuLEy6+rqCtcePnw4zLOtT9nj6qampiqz8fHxcG02Som+n2aNjaCyccazZ8/CvKenJ8y3bNlSmQ0ODoZrh4eHw/zq1asNrW+mUspL9yhy5wREUU5AFOUERFFOQBTlBERRTkAU5QREvbFzzkZkR1uePn06zK9fvx7mjx8/rsyyWWE2a8y2ZWXXj/LsWM4sz177p59+Wpll2/RaGXNOoMVQTkAU5QREUU5AFOUERFFOQBTlBETJHo2ZPYYvyyPZrC/b85jtmVy2LP5Yozw7tjObFWazxmzf4sqVKyuz7NGH2WvP1jdzlpn9/5J9TxcDd05AFOUERFFOQBTlBERRTkAU5QREUU5AlOycM5PNKhuZg0bnyr5K3sjj6rJZYPa1s3ldduZutD47M7ejoyPMszN5m0lxjpnhzgmIopyAKMoJiKKcgCjKCYiinIAoygmIkp1zNjqXauZca3p6Osyz/ZxLly6tzFatWhWujfZbmjW+DzabH0fa29vD/P79+3Vf+23EnRMQRTkBUZQTEEU5AVGUExBFOQFRsqMUZdm2rmwcMjExUZllY5psFJKNWh49ehTmS5ZU/33d6JimVquFOebjzgmIopyAKMoJiKKcgCjKCYiinIAoygmIYs5Zh2ye18iWsWxOmZmdnQ3zRracZe8rm+9m82HMx50TEEU5AVGUExBFOQFRlBMQRTkBUZQTEMWcsw5zc3NhHu2JNItnidna7DF72RxzcnIyzGdmZsI8Es1vzfLPDfNx5wREUU5AFOUERFFOQBTlBERRTkAU5QREMeesQ7ZnMnsUXrQv8smTJ+Ha5cuXh3m2ZzI7O7a7u7syy+aY2X5PfDvcOQFRlBMQRTkBUZQTEEU5AVGUExDFz76boJFH5WXjirGxsTDv7e0N82hU0qhsu1qjx36+bbhzAqIoJyCKcgKiKCcginICoignIIpyAqKYc9Yh2zKWbZ2KtnVlW8KyPHsMX3Y0ZrRlbcWKFeHaTPbaMR93TkAU5QREUU5AFOUERFFOQBTlBERRTkAUc846TE1NhXk2a4z2bE5MTIRrs72i2frx8fEwj471zB7hl+XZsZ2YjzsnIIpyAqIoJyCKcgKiKCcginICoignIIo5Zx2yWWNbW1vdeTYjzc6dzc6GzR4x2MjZstna0dHRuq+dfeallLqvrYo7JyCKcgKiKCcginICoignIIpyAqIoJyCKOWcdsnne9PR0mHd1dVVm2fM5Hz58WPe1zZr7jMzOzs4wz1475uPOCYiinIAoygmIopyAKMoJiKKcgKi3cpTS6Paj7HjJ999/P8wvXrxYmWWPycvGFR0dHWGebRmL3luj29lqtVqYYz7unIAoygmIopyAKMoJiKKcgCjKCYiinIAoj2Z67v7mnTf4GuzatSvMZ2dnK7OBgYFw7ebNm8O8p6cnzIeHh8M8mgFnW77u3LkT5pcvXw7zt1Up5aUfOndOQBTlBERRTkAU5QREUU5AFOUERFFOQFQ45wSweLhzAqIoJyCKcgKiKCcginICoignIOo/aknbdUQmEvMAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Show a sample\n",
    "torch.manual_seed(42)\n",
    "random_idx = torch.randint(0, len(train_features_batch), size=[1]).item()\n",
    "img, label = train_features_batch[random_idx], train_labels_batch[random_idx]\n",
    "plt.imshow(img.squeeze(), cmap=\"gray\")\n",
    "plt.title(class_names[label])\n",
    "plt.axis(\"Off\");\n",
    "print(f\"Image size: {img.shape}\")\n",
    "print(f\"Label: {label}, label size: {label.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db1695cf-f53d-4c7c-ad39-dfed76533125",
   "metadata": {},
   "source": [
    "## 3. Model 0: Build a baseline model\n",
    "\n",
    "Data loaded and prepared!\n",
    "\n",
    "Time to build a **baseline model** by subclassing `nn.Module`.\n",
    "\n",
    "A **baseline model** is one of the simplest models you can imagine.\n",
    "\n",
    "You use the baseline as a starting point and try to improve upon it with subsequent, more complicated models.\n",
    "\n",
    "Our baseline will consist of two [`nn.Linear()`](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) layers.\n",
    "\n",
    "We've done this in a previous section but there's going to one slight difference.\n",
    "\n",
    "Because we're working with image data, we're going to use a different layer to start things off.\n",
    "\n",
    "And that's the [`nn.Flatten()`](https://pytorch.org/docs/stable/generated/torch.nn.Flatten.html) layer.\n",
    "\n",
    "`nn.Flatten()` compresses the dimensions of a tensor into a single vector.\n",
    "\n",
    "This is easier to understand when you see it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "405319f1-f242-4bd9-90f5-3abdc50782ac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape before flattening: torch.Size([1, 28, 28]) -> [color_channels, height, width]\n",
      "Shape after flattening: torch.Size([1, 784]) -> [color_channels, height*width]\n"
     ]
    }
   ],
   "source": [
    "# Create a flatten layer\n",
    "flatten_model = nn.Flatten() # all nn modules function as a model (can do a forward pass)\n",
    "\n",
    "# Get a single sample\n",
    "x = train_features_batch[0]\n",
    "\n",
    "# Flatten the sample\n",
    "output = flatten_model(x) # perform forward pass\n",
    "\n",
    "# Print out what happened\n",
    "print(f\"Shape before flattening: {x.shape} -> [color_channels, height, width]\")\n",
    "print(f\"Shape after flattening: {output.shape} -> [color_channels, height*width]\")\n",
    "\n",
    "# Try uncommenting below and see what happens\n",
    "#print(x)\n",
    "#print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86bb7806-fca6-45af-8111-3e00e38f5be9",
   "metadata": {},
   "source": [
    "The `nn.Flatten()` layer took our shape from `[color_channels, height, width]` to `[color_channels, height*width]`.\n",
    "\n",
    "Why do this?\n",
    "\n",
    "Because we've now turned our pixel data from height and width dimensions into one long **feature vector**.\n",
    "\n",
    "And `nn.Linear()` layers like their inputs to be in the form of feature vectors.\n",
    "\n",
    "Let's create our first model using `nn.Flatten()` as the first layer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "1449f427-6859-41ae-8133-50b58ffbce72",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import nn\n",
    "class FashionMNISTModelV0(nn.Module):\n",
    "    def __init__(self, input_shape: int, hidden_units: int, output_shape: int):\n",
    "        super().__init__()\n",
    "        self.layer_stack = nn.Sequential(\n",
    "            nn.Flatten(), # neural networks like their inputs in vector form\n",
    "            nn.Linear(in_features=input_shape, out_features=hidden_units), # in_features = number of features in a data sample (784 pixels)\n",
    "            nn.Linear(in_features=hidden_units, out_features=output_shape)\n",
    "        )\n",
    "    \n",
    "    def forward(self, x):\n",
    "        return self.layer_stack(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d1b50bf-d00b-485c-be00-b3e4de156fab",
   "metadata": {},
   "source": [
    "Wonderful!\n",
    "\n",
    "We've got a baseline model class we can use, now let's instantiate a model.\n",
    "\n",
    "We'll need to set the following parameters:\n",
    "* `input_shape=784` - this is how many features you've got going in the model, in our case, it's one for every pixel in the target image (28 pixels high by 28 pixels wide = 784 features).\n",
    "* `hidden_units=10` - number of units/neurons in the hidden layer(s), this number could be whatever you want but to keep the model small we'll start with `10`.\n",
    "* `output_shape=len(class_names)` - since we're working with a multi-class classification problem, we need an output neuron per class in our dataset.\n",
    "\n",
    "Let's create an instance of our model and send to the CPU for now (we'll run a small test for running `model_0` on CPU vs. a similar model on GPU soon)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "dd18384a-76f9-4b5a-a013-fda077f16865",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FashionMNISTModelV0(\n",
       "  (layer_stack): Sequential(\n",
       "    (0): Flatten(start_dim=1, end_dim=-1)\n",
       "    (1): Linear(in_features=784, out_features=10, bias=True)\n",
       "    (2): Linear(in_features=10, out_features=10, bias=True)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.manual_seed(42)\n",
    "\n",
    "# Need to setup model with input parameters\n",
    "model_0 = FashionMNISTModelV0(input_shape=784, # one for every pixel (28x28)\n",
    "    hidden_units=10, # how many units in the hiden layer\n",
    "    output_shape=len(class_names) # one for every class\n",
    ")\n",
    "model_0.to(\"cpu\") # keep model on CPU to begin with "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03243179-1cdc-45d9-8b8c-82538ac02e9c",
   "metadata": {},
   "source": [
    "### 3.1 Setup loss, optimizer and evaluation metrics\n",
    "\n",
    "Since we're working on a classification problem, let's bring in our [`helper_functions.py` script](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/helper_functions.py) and subsequently the `accuracy_fn()` we defined in [notebook 02](https://www.learnpytorch.io/02_pytorch_classification/).\n",
    "\n",
    "> **Note:** Rather than importing and using our own accuracy function or evaluation metric(s), you could import various evaluation metrics from the [TorchMetrics package](https://torchmetrics.readthedocs.io/en/latest/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "31c91f17-d810-46a4-97c3-c734f93430b1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "helper_functions.py already exists, skipping download\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "from pathlib import Path \n",
    "\n",
    "# Download helper functions from Learn PyTorch repo (if not already downloaded)\n",
    "if Path(\"helper_functions.py\").is_file():\n",
    "  print(\"helper_functions.py already exists, skipping download\")\n",
    "else:\n",
    "  print(\"Downloading helper_functions.py\")\n",
    "  # Note: you need the \"raw\" GitHub URL for this to work\n",
    "  request = requests.get(\"https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/helper_functions.py\")\n",
    "  with open(\"helper_functions.py\", \"wb\") as f:\n",
    "    f.write(request.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "ce3d13b8-f018-4b44-8bba-375074dc4c5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import accuracy metric\n",
    "from helper_functions import accuracy_fn # Note: could also use torchmetrics.Accuracy()\n",
    "\n",
    "# Setup loss function and optimizer\n",
    "loss_fn = nn.CrossEntropyLoss() # this is also called \"criterion\"/\"cost function\" in some places\n",
    "optimizer = torch.optim.SGD(params=model_0.parameters(), lr=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4109f867-83f2-4394-a925-8acdc63ccffe",
   "metadata": {},
   "source": [
    "### 3.2 Creating a function to time our experiments\n",
    "\n",
    "Loss function and optimizer ready!\n",
    "\n",
    "It's time to start training a model.\n",
    "\n",
    "But how about we do a little experiment while we train.\n",
    "\n",
    "I mean, let's make a timing function to measure the time it takes our model to train on CPU versus using a GPU.\n",
    "\n",
    "We'll train this model on the CPU but the next one on the GPU and see what happens.\n",
    "\n",
    "Our timing function will import the [`timeit.default_timer()` function](https://docs.python.org/3/library/timeit.html#timeit.default_timer) from the Python [`timeit` module](https://docs.python.org/3/library/timeit.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "31adc3fe-ce90-4b4e-b0d4-3613abae5714",
   "metadata": {},
   "outputs": [],
   "source": [
    "from timeit import default_timer as timer \n",
    "def print_train_time(start: float, end: float, device: torch.device = None):\n",
    "    \"\"\"Prints difference between start and end time.\n",
    "\n",
    "    Args:\n",
    "        start (float): Start time of computation (preferred in timeit format). \n",
    "        end (float): End time of computation.\n",
    "        device ([type], optional): Device that compute is running on. Defaults to None.\n",
    "\n",
    "    Returns:\n",
    "        float: time between start and end in seconds (higher is longer).\n",
    "    \"\"\"\n",
    "    total_time = end - start\n",
    "    print(f\"Train time on {device}: {total_time:.3f} seconds\")\n",
    "    return total_time"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07b9560e-f5dc-45d6-b3b2-ddae17a71b34",
   "metadata": {},
   "source": [
    "### 3.3 Creating a training loop and training a model on batches of data\n",
    "\n",
    "Beautiful!\n",
    "\n",
    "Looks like we've got all of the pieces of the puzzle ready to go, a timer, a loss function, an optimizer, a model and most importantly, some data.\n",
    "\n",
    "Let's now create a training loop and a testing loop to train and evaluate our model.\n",
    "\n",
    "We'll be using the same steps as the previous notebook(s), though since our data is now in batch form, we'll add another loop to loop through our data batches.\n",
    "\n",
    "Our data batches are contained within our `DataLoader`s, `train_dataloader` and `test_dataloader` for the training and test data splits respectively.\n",
    "\n",
    "A batch is `BATCH_SIZE` samples of `X` (features) and `y` (labels), since we're using `BATCH_SIZE=32`, our batches have 32 samples of images and targets.\n",
    "\n",
    "And since we're computing on batches of data, our loss and evaluation metrics will be calculated **per batch** rather than across the whole dataset.\n",
    "\n",
    "This means we'll have to divide our loss and accuracy values by the number of batches in each dataset's respective dataloader. \n",
    "\n",
    "Let's step through it: \n",
    "1. Loop through epochs.\n",
    "2. Loop through training batches, perform training steps, calculate the train loss *per batch*.\n",
    "3. Loop through testing batches, perform testing steps, calculate the test loss *per batch*.\n",
    "4. Print out what's happening.\n",
    "5. Time it all (for fun).\n",
    "\n",
    "A fair few steps but...\n",
    "\n",
    "...if in doubt, code it out. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c07bbf10-81e3-47f0-990d-9a4a838276ab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "57e33888d3b64454bd78092be273c525",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0\n",
      "-------\n",
      "Looked at 0/60000 samples\n",
      "Looked at 12800/60000 samples\n",
      "Looked at 25600/60000 samples\n",
      "Looked at 38400/60000 samples\n",
      "Looked at 51200/60000 samples\n",
      "\n",
      "Train loss: 0.59039 | Test loss: 0.50954, Test acc: 82.04%\n",
      "\n",
      "Epoch: 1\n",
      "-------\n",
      "Looked at 0/60000 samples\n",
      "Looked at 12800/60000 samples\n",
      "Looked at 25600/60000 samples\n",
      "Looked at 38400/60000 samples\n",
      "Looked at 51200/60000 samples\n",
      "\n",
      "Train loss: 0.47633 | Test loss: 0.47989, Test acc: 83.20%\n",
      "\n",
      "Epoch: 2\n",
      "-------\n",
      "Looked at 0/60000 samples\n",
      "Looked at 12800/60000 samples\n",
      "Looked at 25600/60000 samples\n",
      "Looked at 38400/60000 samples\n",
      "Looked at 51200/60000 samples\n",
      "\n",
      "Train loss: 0.45503 | Test loss: 0.47664, Test acc: 83.43%\n",
      "\n",
      "Train time on cpu: 15.549 seconds\n"
     ]
    }
   ],
   "source": [
    "# Import tqdm for progress bar\n",
    "from tqdm.auto import tqdm\n",
    "\n",
    "# Set the seed and start the timer\n",
    "torch.manual_seed(42)\n",
    "train_time_start_on_cpu = timer()\n",
    "\n",
    "# Set the number of epochs (we'll keep this small for faster training times)\n",
    "epochs = 3\n",
    "\n",
    "# Create training and testing loop\n",
    "for epoch in tqdm(range(epochs)):\n",
    "    print(f\"Epoch: {epoch}\\n-------\")\n",
    "    ### Training\n",
    "    train_loss = 0\n",
    "    # Add a loop to loop through training batches\n",
    "    for batch, (X, y) in enumerate(train_dataloader):\n",
    "        model_0.train() \n",
    "        # 1. Forward pass\n",
    "        y_pred = model_0(X)\n",
    "\n",
    "        # 2. Calculate loss (per batch)\n",
    "        loss = loss_fn(y_pred, y)\n",
    "        train_loss += loss # accumulatively add up the loss per epoch \n",
    "\n",
    "        # 3. Optimizer zero grad\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        # 4. Loss backward\n",
    "        loss.backward()\n",
    "\n",
    "        # 5. Optimizer step\n",
    "        optimizer.step()\n",
    "\n",
    "        # Print out how many samples have been seen\n",
    "        if batch % 400 == 0:\n",
    "            print(f\"Looked at {batch * len(X)}/{len(train_dataloader.dataset)} samples\")\n",
    "\n",
    "    # Divide total train loss by length of train dataloader (average loss per batch per epoch)\n",
    "    train_loss /= len(train_dataloader)\n",
    "    \n",
    "    ### Testing\n",
    "    # Setup variables for accumulatively adding up loss and accuracy \n",
    "    test_loss, test_acc = 0, 0 \n",
    "    model_0.eval()\n",
    "    with torch.inference_mode():\n",
    "        for X, y in test_dataloader:\n",
    "            # 1. Forward pass\n",
    "            test_pred = model_0(X)\n",
    "           \n",
    "            # 2. Calculate loss (accumatively)\n",
    "            test_loss += loss_fn(test_pred, y) # accumulatively add up the loss per epoch\n",
    "\n",
    "            # 3. Calculate accuracy (preds need to be same as y_true)\n",
    "            test_acc += accuracy_fn(y_true=y, y_pred=test_pred.argmax(dim=1))\n",
    "        \n",
    "        # Calculations on test metrics need to happen inside torch.inference_mode()\n",
    "        # Divide total test loss by length of test dataloader (per batch)\n",
    "        test_loss /= len(test_dataloader)\n",
    "\n",
    "        # Divide total accuracy by length of test dataloader (per batch)\n",
    "        test_acc /= len(test_dataloader)\n",
    "\n",
    "    ## Print out what's happening\n",
    "    print(f\"\\nTrain loss: {train_loss:.5f} | Test loss: {test_loss:.5f}, Test acc: {test_acc:.2f}%\\n\")\n",
    "\n",
    "# Calculate training time      \n",
    "train_time_end_on_cpu = timer()\n",
    "total_train_time_model_0 = print_train_time(start=train_time_start_on_cpu, \n",
    "                                           end=train_time_end_on_cpu,\n",
    "                                           device=str(next(model_0.parameters()).device))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b02a939-a3a1-4a9d-bb9d-62928def2ded",
   "metadata": {},
   "source": [
    "Nice! Looks like our baseline model did fairly well.\n",
    "\n",
    "It didn't take too long to train either, even just on the CPU, I wonder if it'll speed up on the GPU?\n",
    "\n",
    "Let's write some code to evaluate our model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7442511b-bfe9-4ec7-9f5b-9c808f8e560b",
   "metadata": {},
   "source": [
    "## 4. Make predictions and get Model 0 results\n",
    "\n",
    "Since we're going to be building a few models, it's a good idea to write some code to evaluate them all in similar ways.\n",
    "\n",
    "Namely, let's create a function that takes in a trained model, a `DataLoader`, a loss function and an accuracy function.\n",
    "\n",
    "The function will use the model to make predictions on the data in the `DataLoader` and then we can evaluate those predictions using the loss function and accuracy function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "8317dd04-9de2-4fd7-97bd-1e202621397d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'model_name': 'FashionMNISTModelV0',\n",
       " 'model_loss': 0.47663894295692444,\n",
       " 'model_acc': 83.42651757188499}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.manual_seed(42)\n",
    "def eval_model(model: torch.nn.Module, \n",
    "               data_loader: torch.utils.data.DataLoader, \n",
    "               loss_fn: torch.nn.Module, \n",
    "               accuracy_fn):\n",
    "    \"\"\"Returns a dictionary containing the results of model predicting on data_loader.\n",
    "\n",
    "    Args:\n",
    "        model (torch.nn.Module): A PyTorch model capable of making predictions on data_loader.\n",
    "        data_loader (torch.utils.data.DataLoader): The target dataset to predict on.\n",
    "        loss_fn (torch.nn.Module): The loss function of model.\n",
    "        accuracy_fn: An accuracy function to compare the models predictions to the truth labels.\n",
    "\n",
    "    Returns:\n",
    "        (dict): Results of model making predictions on data_loader.\n",
    "    \"\"\"\n",
    "    loss, acc = 0, 0\n",
    "    model.eval()\n",
    "    with torch.inference_mode():\n",
    "        for X, y in data_loader:\n",
    "            # Make predictions with the model\n",
    "            y_pred = model(X)\n",
    "            \n",
    "            # Accumulate the loss and accuracy values per batch\n",
    "            loss += loss_fn(y_pred, y)\n",
    "            acc += accuracy_fn(y_true=y, \n",
    "                                y_pred=y_pred.argmax(dim=1)) # For accuracy, need the prediction labels (logits -> pred_prob -> pred_labels)\n",
    "        \n",
    "        # Scale loss and acc to find the average loss/acc per batch\n",
    "        loss /= len(data_loader)\n",
    "        acc /= len(data_loader)\n",
    "        \n",
    "    return {\"model_name\": model.__class__.__name__, # only works when model was created with a class\n",
    "            \"model_loss\": loss.item(),\n",
    "            \"model_acc\": acc}\n",
    "\n",
    "# Calculate model 0 results on test dataset\n",
    "model_0_results = eval_model(model=model_0, data_loader=test_dataloader,\n",
    "    loss_fn=loss_fn, accuracy_fn=accuracy_fn\n",
    ")\n",
    "model_0_results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a39c3042-1262-4d1f-b33e-c8e2ba6781d3",
   "metadata": {},
   "source": [
    "Looking good!\n",
    "\n",
    "We can use this dictionary to compare the baseline model results to other models later on."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b76784d-4cdb-43d2-a6da-8e4da9a812a9",
   "metadata": {},
   "source": [
    "## 5. Setup device agnostic-code (for using a GPU if there is one)\n",
    "We've seen how long it takes to train ma PyTorch model on 60,000 samples on CPU.\n",
    "\n",
    "> **Note:** Model training time is dependent on hardware used. Generally, more processors means faster training and smaller models on smaller datasets will often train faster than large models and large datasets.\n",
    "\n",
    "Now let's setup some [device-agnostic code](https://pytorch.org/docs/stable/notes/cuda.html#best-practices) for our models and data to run on GPU if it's available.\n",
    "\n",
    "If you're running this notebook on Google Colab, and you don't a GPU turned on yet, it's now time to turn one on via `Runtime -> Change runtime type -> Hardware accelerator -> GPU`. If you do this, your runtime will likely reset and you'll have to run all of the cells above by going `Runtime -> Run before`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "17b69fe9-f974-4538-922c-20c5cc8220cc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'cuda'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Setup device agnostic code\n",
    "import torch\n",
    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "device"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "514021a8-d6f2-47f3-ab50-55f844e42310",
   "metadata": {},
   "source": [
    "Beautiful!\n",
    "\n",
    "Let's build another model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7893907-5f82-4c5e-8fde-fa542a9f25af",
   "metadata": {},
   "source": [
    "## 6. Model 1: Building a better model with non-linearity\n",
    "\n",
    "We learned about [the power of non-linearity in notebook 02](https://www.learnpytorch.io/02_pytorch_classification/#6-the-missing-piece-non-linearity).\n",
    "\n",
    "Seeing the data we've been working with, do you think it needs non-linear functions?\n",
    "\n",
    "And remember, linear means straight and non-linear means non-straight.\n",
    "\n",
    "Let's find out.\n",
    "\n",
    "We'll do so by recreating a similar model to before, except this time we'll put non-linear functions (`nn.ReLU()`) in between each linear layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "2ccce5f2-b1e5-47a6-a7f3-6bc096b35ffb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a model with non-linear and linear layers\n",
    "class FashionMNISTModelV1(nn.Module):\n",
    "    def __init__(self, input_shape: int, hidden_units: int, output_shape: int):\n",
    "        super().__init__()\n",
    "        self.layer_stack = nn.Sequential(\n",
    "            nn.Flatten(), # flatten inputs into single vector\n",
    "            nn.Linear(in_features=input_shape, out_features=hidden_units),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(in_features=hidden_units, out_features=output_shape),\n",
    "            nn.ReLU()\n",
    "        )\n",
    "    \n",
    "    def forward(self, x: torch.Tensor):\n",
    "        return self.layer_stack(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b4b7a2f-4834-4aa1-a8e2-b6e3e2b49224",
   "metadata": {},
   "source": [
    "That looks good.\n",
    "\n",
    "Now let's instantiate it with the same settings we used before.\n",
    "\n",
    "We'll need `input_shape=784` (equal to the number of features of our image data), `hidden_units=10` (starting small and the same as our baseline model) and `output_shape=len(class_names)` (one output unit per class).\n",
    "\n",
    "> **Note:** Notice how we kept most of the settings of our model the same except for one change: adding non-linear layers. This is a standard practice for running a series of machine learning experiments, change one thing and see what happens, then do it again, again, again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "907091ec-7e46-470b-a305-788a3009b837",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cuda', index=0)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.manual_seed(42)\n",
    "model_1 = FashionMNISTModelV1(input_shape=784, # number of input features\n",
    "    hidden_units=10,\n",
    "    output_shape=len(class_names) # number of output classes desired\n",
    ").to(device) # send model to GPU if it's available\n",
    "next(model_1.parameters()).device # check model device"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b54a4e9d-a7ad-404c-920f-485fcff18a92",
   "metadata": {},
   "source": [
    "### 6.1 Setup loss, optimizer and evaluation metrics\n",
    "\n",
    "As usual, we'll setup a loss function, an optimizer and an evaluation metric (we could do multiple evaluation metrics but we'll stick with accuracy for now)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "fe7e463b-d46c-4f00-853c-fdf0a28d74c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from helper_functions import accuracy_fn\n",
    "loss_fn = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.SGD(params=model_1.parameters(), \n",
    "                            lr=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1eb30af6-a355-49a2-a59f-25169fd27a6e",
   "metadata": {},
   "source": [
    "### 6.2 Functionizing training and test loops\n",
    "\n",
    "So far we've been writing train and test loops over and over. \n",
    "\n",
    "Let's write them again but this time we'll put them in functions so they can be called again and again.\n",
    "\n",
    "And because we're using device-agnostic code now, we'll be sure to call `.to(device)` on our feature (`X`) and target (`y`) tensors.\n",
    "\n",
    "For the training loop we'll create a function called `train_step()` which takes in a model, a `DataLoader` a loss function and an optimizer.\n",
    "\n",
    "The testing loop will be similar but it'll be called `test_step()` and it'll take in a model, a `DataLoader`, a loss function and an evaluation function.\n",
    "\n",
    "> **Note:** Since these are functions, you can customize them in any way you like. What we're making here can be considered barebones training and testing functions for our specific classification use case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "3d239ed2-4028-4603-8db3-ffca2b727819",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_step(model: torch.nn.Module,\n",
    "               data_loader: torch.utils.data.DataLoader,\n",
    "               loss_fn: torch.nn.Module,\n",
    "               optimizer: torch.optim.Optimizer,\n",
    "               accuracy_fn,\n",
    "               device: torch.device = device):\n",
    "    train_loss, train_acc = 0, 0\n",
    "    for batch, (X, y) in enumerate(data_loader):\n",
    "        # Send data to GPU\n",
    "        X, y = X.to(device), y.to(device)\n",
    "\n",
    "        # 1. Forward pass\n",
    "        y_pred = model(X)\n",
    "\n",
    "        # 2. Calculate loss\n",
    "        loss = loss_fn(y_pred, y)\n",
    "        train_loss += loss\n",
    "        train_acc += accuracy_fn(y_true=y,\n",
    "                                 y_pred=y_pred.argmax(dim=1)) # Go from logits -> pred labels\n",
    "\n",
    "        # 3. Optimizer zero grad\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        # 4. Loss backward\n",
    "        loss.backward()\n",
    "\n",
    "        # 5. Optimizer step\n",
    "        optimizer.step()\n",
    "\n",
    "    # Calculate loss and accuracy per epoch and print out what's happening\n",
    "    train_loss /= len(data_loader)\n",
    "    train_acc /= len(data_loader)\n",
    "    print(f\"Train loss: {train_loss:.5f} | Train accuracy: {train_acc:.2f}%\")\n",
    "\n",
    "def test_step(data_loader: torch.utils.data.DataLoader,\n",
    "              model: torch.nn.Module,\n",
    "              loss_fn: torch.nn.Module,\n",
    "              accuracy_fn,\n",
    "              device: torch.device = device):\n",
    "    test_loss, test_acc = 0, 0\n",
    "    model.eval() # put model in eval mode\n",
    "    # Turn on inference context manager\n",
    "    with torch.inference_mode(): \n",
    "        for X, y in data_loader:\n",
    "            # Send data to GPU\n",
    "            X, y = X.to(device), y.to(device)\n",
    "            \n",
    "            # 1. Forward pass\n",
    "            test_pred = model(X)\n",
    "            \n",
    "            # 2. Calculate loss and accuracy\n",
    "            test_loss += loss_fn(test_pred, y)\n",
    "            test_acc += accuracy_fn(y_true=y,\n",
    "                y_pred=test_pred.argmax(dim=1) # Go from logits -> pred labels\n",
    "            )\n",
    "        \n",
    "        # Adjust metrics and print out\n",
    "        test_loss /= len(data_loader)\n",
    "        test_acc /= len(data_loader)\n",
    "        print(f\"Test loss: {test_loss:.5f} | Test accuracy: {test_acc:.2f}%\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e44121b6-c4be-4909-9175-dc9bd8dc6273",
   "metadata": {},
   "source": [
    "Woohoo!\n",
    "\n",
    "Now we've got some functions for training and testing our model, let's run them.\n",
    "\n",
    "We'll do so inside another loop for each epoch.\n",
    "\n",
    "That way for each epoch we're going a training and a testing step.\n",
    "\n",
    "> **Note:** You can customize how often you do a testing step. Sometimes people do them every five epochs or 10 epochs or in our case, every epoch.\n",
    "\n",
    "Let's also time things to see how long our code takes to run on the GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "2bb8094b-01a0-4b84-9526-ba8888d04901",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "09fbec22c8e3458ba22634fbbbfa17e4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0\n",
      "---------\n",
      "Train loss: 1.09199 | Train accuracy: 61.34%\n",
      "Test loss: 0.95636 | Test accuracy: 65.00%\n",
      "\n",
      "Epoch: 1\n",
      "---------\n",
      "Train loss: 0.78101 | Train accuracy: 71.93%\n",
      "Test loss: 0.72227 | Test accuracy: 73.91%\n",
      "\n",
      "Epoch: 2\n",
      "---------\n",
      "Train loss: 0.67027 | Train accuracy: 75.94%\n",
      "Test loss: 0.68500 | Test accuracy: 75.02%\n",
      "\n",
      "Train time on cuda: 17.356 seconds\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(42)\n",
    "\n",
    "# Measure time\n",
    "from timeit import default_timer as timer\n",
    "train_time_start_on_gpu = timer()\n",
    "\n",
    "epochs = 3\n",
    "for epoch in tqdm(range(epochs)):\n",
    "    print(f\"Epoch: {epoch}\\n---------\")\n",
    "    train_step(data_loader=train_dataloader, \n",
    "        model=model_1, \n",
    "        loss_fn=loss_fn,\n",
    "        optimizer=optimizer,\n",
    "        accuracy_fn=accuracy_fn\n",
    "    )\n",
    "    test_step(data_loader=test_dataloader,\n",
    "        model=model_1,\n",
    "        loss_fn=loss_fn,\n",
    "        accuracy_fn=accuracy_fn\n",
    "    )\n",
    "\n",
    "train_time_end_on_gpu = timer()\n",
    "total_train_time_model_1 = print_train_time(start=train_time_start_on_gpu,\n",
    "                                            end=train_time_end_on_gpu,\n",
    "                                            device=device)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "719b8eb9-9a7f-42ed-a49f-5eedc6fdd720",
   "metadata": {},
   "source": [
    "Excellent!\n",
    "\n",
    "Our model trained but the training time took longer?\n",
    "\n",
    "> **Note:** The training time on CUDA vs CPU will depend largely on the quality of the CPU/GPU you're using. Read on for a more explained answer.\n",
    "\n",
    "> **Question:** \"I used a a GPU but my model didn't train faster, why might that be?\"\n",
    ">\n",
    "> **Answer:** Well, one reason could be because your dataset and model are both so small (like the dataset and model we're working with) the benefits of using a GPU are outweighed by the time it actually takes to transfer the data there.\n",
    "> \n",
    "> There's a small bottleneck between copying data from the CPU memory (default) to the GPU memory.\n",
    ">\n",
    "> So for smaller models and datasets, the CPU might actually be the optimal place to compute on.\n",
    ">\n",
    "> But for larger datasets and models, the speed of computing the GPU can offer usually far outweighs the cost of getting the data there.\n",
    ">\n",
    "> However, this is largely dependant on the hardware you're using. With practice, you will get used to where the best place to train your models is. \n",
    "\n",
    "Let's evaluate our trained `model_1` using our `eval_model()` function and see how it went."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "32a544e3-9dbe-4aa1-b074-22e28b8f2f2a",
   "metadata": {},
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument mat1 in method wrapper_addmm)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[1;32m/home/daniel/code/pytorch/pytorch-course/pytorch-deep-learning/03_pytorch_computer_vision.ipynb Cell 62\u001b[0m in \u001b[0;36m<cell line: 4>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      <a href='vscode-notebook-cell://ssh-remote%2B192.168.1.25/home/daniel/code/pytorch/pytorch-course/pytorch-deep-learning/03_pytorch_computer_vision.ipynb#Y115sdnNjb2RlLXJlbW90ZQ%3D%3D?line=0'>1</a>\u001b[0m torch\u001b[39m.\u001b[39mmanual_seed(\u001b[39m42\u001b[39m)\n\u001b[1;32m      <a href='vscode-notebook-cell://ssh-remote%2B192.168.1.25/home/daniel/code/pytorch/pytorch-course/pytorch-deep-learning/03_pytorch_computer_vision.ipynb#Y115sdnNjb2RlLXJlbW90ZQ%3D%3D?line=2'>3</a>\u001b[0m \u001b[39m# Note: This will error due to `eval_model()` not using device agnostic code \u001b[39;00m\n\u001b[0;32m----> <a href='vscode-notebook-cell://ssh-remote%2B192.168.1.25/home/daniel/code/pytorch/pytorch-course/pytorch-deep-learning/03_pytorch_computer_vision.ipynb#Y115sdnNjb2RlLXJlbW90ZQ%3D%3D?line=3'>4</a>\u001b[0m model_1_results \u001b[39m=\u001b[39m eval_model(model\u001b[39m=\u001b[39;49mmodel_1, \n\u001b[1;32m      <a href='vscode-notebook-cell://ssh-remote%2B192.168.1.25/home/daniel/code/pytorch/pytorch-course/pytorch-deep-learning/03_pytorch_computer_vision.ipynb#Y115sdnNjb2RlLXJlbW90ZQ%3D%3D?line=4'>5</a>\u001b[0m     data_loader\u001b[39m=\u001b[39;49mtest_dataloader,\n\u001b[1;32m      <a href='vscode-notebook-cell://ssh-remote%2B192.168.1.25/home/daniel/code/pytorch/pytorch-course/pytorch-deep-learning/03_pytorch_computer_vision.ipynb#Y115sdnNjb2RlLXJlbW90ZQ%3D%3D?line=5'>6</a>\u001b[0m     loss_fn\u001b[39m=\u001b[39;49mloss_fn, \n\u001b[1;32m      <a href='vscode-notebook-cell://ssh-remote%2B192.168.1.25/home/daniel/code/pytorch/pytorch-course/pytorch-deep-learning/03_pytorch_computer_vision.ipynb#Y115sdnNjb2RlLXJlbW90ZQ%3D%3D?line=6'>7</a>\u001b[0m     accuracy_fn\u001b[39m=\u001b[39;49maccuracy_fn) \n\u001b[1;32m      <a href='vscode-notebook-cell://ssh-remote%2B192.168.1.25/home/daniel/code/pytorch/pytorch-course/pytorch-deep-learning/03_pytorch_computer_vision.ipynb#Y115sdnNjb2RlLXJlbW90ZQ%3D%3D?line=7'>8</a>\u001b[0m model_1_results\n",
      "\u001b[1;32m/home/daniel/code/pytorch/pytorch-course/pytorch-deep-learning/03_pytorch_computer_vision.ipynb Cell 62\u001b[0m in \u001b[0;36meval_model\u001b[0;34m(model, data_loader, loss_fn, accuracy_fn)\u001b[0m\n\u001b[1;32m     <a href='vscode-notebook-cell://ssh-remote%2B192.168.1.25/home/daniel/code/pytorch/pytorch-course/pytorch-deep-learning/03_pytorch_computer_vision.ipynb#Y115sdnNjb2RlLXJlbW90ZQ%3D%3D?line=18'>19</a>\u001b[0m \u001b[39mwith\u001b[39;00m torch\u001b[39m.\u001b[39minference_mode():\n\u001b[1;32m     <a href='vscode-notebook-cell://ssh-remote%2B192.168.1.25/home/daniel/code/pytorch/pytorch-course/pytorch-deep-learning/03_pytorch_computer_vision.ipynb#Y115sdnNjb2RlLXJlbW90ZQ%3D%3D?line=19'>20</a>\u001b[0m     \u001b[39mfor\u001b[39;00m X, y \u001b[39min\u001b[39;00m data_loader:\n\u001b[1;32m     <a href='vscode-notebook-cell://ssh-remote%2B192.168.1.25/home/daniel/code/pytorch/pytorch-course/pytorch-deep-learning/03_pytorch_computer_vision.ipynb#Y115sdnNjb2RlLXJlbW90ZQ%3D%3D?line=20'>21</a>\u001b[0m         \u001b[39m# Make predictions with the model\u001b[39;00m\n\u001b[0;32m---> <a href='vscode-notebook-cell://ssh-remote%2B192.168.1.25/home/daniel/code/pytorch/pytorch-course/pytorch-deep-learning/03_pytorch_computer_vision.ipynb#Y115sdnNjb2RlLXJlbW90ZQ%3D%3D?line=21'>22</a>\u001b[0m         y_pred \u001b[39m=\u001b[39m model(X)\n\u001b[1;32m     <a href='vscode-notebook-cell://ssh-remote%2B192.168.1.25/home/daniel/code/pytorch/pytorch-course/pytorch-deep-learning/03_pytorch_computer_vision.ipynb#Y115sdnNjb2RlLXJlbW90ZQ%3D%3D?line=23'>24</a>\u001b[0m         \u001b[39m# Accumulate the loss and accuracy values per batch\u001b[39;00m\n\u001b[1;32m     <a href='vscode-notebook-cell://ssh-remote%2B192.168.1.25/home/daniel/code/pytorch/pytorch-course/pytorch-deep-learning/03_pytorch_computer_vision.ipynb#Y115sdnNjb2RlLXJlbW90ZQ%3D%3D?line=24'>25</a>\u001b[0m         loss \u001b[39m+\u001b[39m\u001b[39m=\u001b[39m loss_fn(y_pred, y)\n",
      "File \u001b[0;32m~/code/pytorch/env/lib/python3.8/site-packages/torch/nn/modules/module.py:1130\u001b[0m, in \u001b[0;36mModule._call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1126\u001b[0m \u001b[39m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[1;32m   1127\u001b[0m \u001b[39m# this function, and just call forward.\u001b[39;00m\n\u001b[1;32m   1128\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mnot\u001b[39;00m (\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_backward_hooks \u001b[39mor\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_forward_hooks \u001b[39mor\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_forward_pre_hooks \u001b[39mor\u001b[39;00m _global_backward_hooks\n\u001b[1;32m   1129\u001b[0m         \u001b[39mor\u001b[39;00m _global_forward_hooks \u001b[39mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[0;32m-> 1130\u001b[0m     \u001b[39mreturn\u001b[39;00m forward_call(\u001b[39m*\u001b[39;49m\u001b[39minput\u001b[39;49m, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs)\n\u001b[1;32m   1131\u001b[0m \u001b[39m# Do not call functions when jit is used\u001b[39;00m\n\u001b[1;32m   1132\u001b[0m full_backward_hooks, non_full_backward_hooks \u001b[39m=\u001b[39m [], []\n",
      "\u001b[1;32m/home/daniel/code/pytorch/pytorch-course/pytorch-deep-learning/03_pytorch_computer_vision.ipynb Cell 62\u001b[0m in \u001b[0;36mFashionMNISTModelV1.forward\u001b[0;34m(self, x)\u001b[0m\n\u001b[1;32m     <a href='vscode-notebook-cell://ssh-remote%2B192.168.1.25/home/daniel/code/pytorch/pytorch-course/pytorch-deep-learning/03_pytorch_computer_vision.ipynb#Y115sdnNjb2RlLXJlbW90ZQ%3D%3D?line=12'>13</a>\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mforward\u001b[39m(\u001b[39mself\u001b[39m, x: torch\u001b[39m.\u001b[39mTensor):\n\u001b[0;32m---> <a href='vscode-notebook-cell://ssh-remote%2B192.168.1.25/home/daniel/code/pytorch/pytorch-course/pytorch-deep-learning/03_pytorch_computer_vision.ipynb#Y115sdnNjb2RlLXJlbW90ZQ%3D%3D?line=13'>14</a>\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mlayer_stack(x)\n",
      "File \u001b[0;32m~/code/pytorch/env/lib/python3.8/site-packages/torch/nn/modules/module.py:1130\u001b[0m, in \u001b[0;36mModule._call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1126\u001b[0m \u001b[39m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[1;32m   1127\u001b[0m \u001b[39m# this function, and just call forward.\u001b[39;00m\n\u001b[1;32m   1128\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mnot\u001b[39;00m (\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_backward_hooks \u001b[39mor\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_forward_hooks \u001b[39mor\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_forward_pre_hooks \u001b[39mor\u001b[39;00m _global_backward_hooks\n\u001b[1;32m   1129\u001b[0m         \u001b[39mor\u001b[39;00m _global_forward_hooks \u001b[39mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[0;32m-> 1130\u001b[0m     \u001b[39mreturn\u001b[39;00m forward_call(\u001b[39m*\u001b[39;49m\u001b[39minput\u001b[39;49m, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs)\n\u001b[1;32m   1131\u001b[0m \u001b[39m# Do not call functions when jit is used\u001b[39;00m\n\u001b[1;32m   1132\u001b[0m full_backward_hooks, non_full_backward_hooks \u001b[39m=\u001b[39m [], []\n",
      "File \u001b[0;32m~/code/pytorch/env/lib/python3.8/site-packages/torch/nn/modules/container.py:139\u001b[0m, in \u001b[0;36mSequential.forward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m    137\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mforward\u001b[39m(\u001b[39mself\u001b[39m, \u001b[39minput\u001b[39m):\n\u001b[1;32m    138\u001b[0m     \u001b[39mfor\u001b[39;00m module \u001b[39min\u001b[39;00m \u001b[39mself\u001b[39m:\n\u001b[0;32m--> 139\u001b[0m         \u001b[39minput\u001b[39m \u001b[39m=\u001b[39m module(\u001b[39minput\u001b[39;49m)\n\u001b[1;32m    140\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39minput\u001b[39m\n",
      "File \u001b[0;32m~/code/pytorch/env/lib/python3.8/site-packages/torch/nn/modules/module.py:1130\u001b[0m, in \u001b[0;36mModule._call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1126\u001b[0m \u001b[39m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[1;32m   1127\u001b[0m \u001b[39m# this function, and just call forward.\u001b[39;00m\n\u001b[1;32m   1128\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mnot\u001b[39;00m (\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_backward_hooks \u001b[39mor\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_forward_hooks \u001b[39mor\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_forward_pre_hooks \u001b[39mor\u001b[39;00m _global_backward_hooks\n\u001b[1;32m   1129\u001b[0m         \u001b[39mor\u001b[39;00m _global_forward_hooks \u001b[39mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[0;32m-> 1130\u001b[0m     \u001b[39mreturn\u001b[39;00m forward_call(\u001b[39m*\u001b[39;49m\u001b[39minput\u001b[39;49m, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs)\n\u001b[1;32m   1131\u001b[0m \u001b[39m# Do not call functions when jit is used\u001b[39;00m\n\u001b[1;32m   1132\u001b[0m full_backward_hooks, non_full_backward_hooks \u001b[39m=\u001b[39m [], []\n",
      "File \u001b[0;32m~/code/pytorch/env/lib/python3.8/site-packages/torch/nn/modules/linear.py:114\u001b[0m, in \u001b[0;36mLinear.forward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m    113\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mforward\u001b[39m(\u001b[39mself\u001b[39m, \u001b[39minput\u001b[39m: Tensor) \u001b[39m-\u001b[39m\u001b[39m>\u001b[39m Tensor:\n\u001b[0;32m--> 114\u001b[0m     \u001b[39mreturn\u001b[39;00m F\u001b[39m.\u001b[39;49mlinear(\u001b[39minput\u001b[39;49m, \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mweight, \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mbias)\n",
      "\u001b[0;31mRuntimeError\u001b[0m: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument mat1 in method wrapper_addmm)"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(42)\n",
    "\n",
    "# Note: This will error due to `eval_model()` not using device agnostic code \n",
    "model_1_results = eval_model(model=model_1, \n",
    "    data_loader=test_dataloader,\n",
    "    loss_fn=loss_fn, \n",
    "    accuracy_fn=accuracy_fn) \n",
    "model_1_results "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a3481a5-489d-4db9-ac95-c3ce385978b7",
   "metadata": {},
   "source": [
    "Oh no! \n",
    "\n",
    "It looks like our `eval_model()` function errors out with:\n",
    "\n",
    "> `RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument mat1 in method wrapper_addmm)`\n",
    "\n",
    "It's because we've setup our data and model to use device-agnostic code but not our evaluation function.\n",
    "\n",
    "How about we fix that by passing a target `device` parameter to our `eval_model()` function?\n",
    "\n",
    "Then we'll try calculating the results again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "f3665d99-1adc-4d9f-bfc6-e5601a80691c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'model_name': 'FashionMNISTModelV1',\n",
       " 'model_loss': 0.6850008964538574,\n",
       " 'model_acc': 75.01996805111821}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Move values to device\n",
    "torch.manual_seed(42)\n",
    "def eval_model(model: torch.nn.Module, \n",
    "               data_loader: torch.utils.data.DataLoader, \n",
    "               loss_fn: torch.nn.Module, \n",
    "               accuracy_fn, \n",
    "               device: torch.device = device):\n",
    "    \"\"\"Evaluates a given model on a given dataset.\n",
    "\n",
    "    Args:\n",
    "        model (torch.nn.Module): A PyTorch model capable of making predictions on data_loader.\n",
    "        data_loader (torch.utils.data.DataLoader): The target dataset to predict on.\n",
    "        loss_fn (torch.nn.Module): The loss function of model.\n",
    "        accuracy_fn: An accuracy function to compare the models predictions to the truth labels.\n",
    "        device (str, optional): Target device to compute on. Defaults to device.\n",
    "\n",
    "    Returns:\n",
    "        (dict): Results of model making predictions on data_loader.\n",
    "    \"\"\"\n",
    "    loss, acc = 0, 0\n",
    "    model.eval()\n",
    "    with torch.inference_mode():\n",
    "        for X, y in data_loader:\n",
    "            # Send data to the target device\n",
    "            X, y = X.to(device), y.to(device)\n",
    "            y_pred = model(X)\n",
    "            loss += loss_fn(y_pred, y)\n",
    "            acc += accuracy_fn(y_true=y, y_pred=y_pred.argmax(dim=1))\n",
    "        \n",
    "        # Scale loss and acc\n",
    "        loss /= len(data_loader)\n",
    "        acc /= len(data_loader)\n",
    "    return {\"model_name\": model.__class__.__name__, # only works when model was created with a class\n",
    "            \"model_loss\": loss.item(),\n",
    "            \"model_acc\": acc}\n",
    "\n",
    "# Calculate model 1 results with device-agnostic code \n",
    "model_1_results = eval_model(model=model_1, data_loader=test_dataloader,\n",
    "    loss_fn=loss_fn, accuracy_fn=accuracy_fn,\n",
    "    device=device\n",
    ")\n",
    "model_1_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "a9e916cf-f873-4481-a983-bac26ce4cac2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'model_name': 'FashionMNISTModelV0',\n",
       " 'model_loss': 0.47663894295692444,\n",
       " 'model_acc': 83.42651757188499}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check baseline results\n",
    "model_0_results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "340cbf14-e83f-4981-8a93-5fedb6b51418",
   "metadata": {},
   "source": [
    "Woah, in this case, it looks like adding non-linearities to our model made it perform worse than the baseline.\n",
    "\n",
    "That's a thing to note in machine learning, sometimes the thing you thought should work doesn't. \n",
    "\n",
    "And then the thing you thought might not work does.\n",
    "\n",
    "It's part science, part art.\n",
    "\n",
    "From the looks of things, it seems like our model is **overfitting** on the training data.\n",
    "\n",
    "Overfitting means our model is learning the training data well but those patterns aren't generalizing to the testing data.\n",
    "\n",
    "Two of the main to fix overfitting include:\n",
    "1. Using a smaller or different model (some models fit certain kinds of data better than others).\n",
    "2. Using a larger dataset (the more data, the more chance a model has to learn generalizable patterns).\n",
    "\n",
    "There are more, but I'm going to leave that as a challenge for you to explore.\n",
    "\n",
    "Try searching online, \"ways to prevent overfitting in machine learning\" and see what comes up.\n",
    "\n",
    "In the meantime, let's take a look at number 1: using a different model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac22d685-1b8d-4215-90de-c0476cb0fbdf",
   "metadata": {},
   "source": [
    "## 7. Model 2: Building a Convolutional Neural Network (CNN)\n",
    "\n",
    "Alright, time to step things up a notch.\n",
    "\n",
    "It's time to create a [Convolutional Neural Network](https://en.wikipedia.org/wiki/Convolutional_neural_network) (CNN or ConvNet).\n",
    "\n",
    "CNN's are known for their capabilities to find patterns in visual data.\n",
    "\n",
    "And since we're dealing with visual data, let's see if using a CNN model can improve upon our baseline.\n",
    "\n",
    "The CNN model we're going to be using is known as TinyVGG from the [CNN Explainer](https://poloclub.github.io/cnn-explainer/) website.\n",
    "\n",
    "It follows the typical structure of a convolutional neural network:\n",
    "\n",
    "`Input layer -> [Convolutional layer -> activation layer -> pooling layer] -> Output layer`\n",
    "\n",
    "Where the contents of `[Convolutional layer -> activation layer -> pooling layer]` can be upscaled and repeated multiple times, depending on requirements. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c358955-1d20-4903-b872-a239d2753d88",
   "metadata": {},
   "source": [
    "### What model should I use?\n",
    "\n",
    "> **Question:** Wait, you say CNN's are good for images, are there any other model types I should be aware of?\n",
    "\n",
    "Good question.\n",
    "\n",
    "This table is a good general guide for which model to use (though there are exceptions).\n",
    "\n",
    "| **Problem type** | **Model to use (generally)** | **Code example** |\n",
    "| ----- | ----- | ----- |\n",
    "| Structured data (Excel spreadsheets, row and column data) | Gradient boosted models, Random Forests, XGBoost | [`sklearn.ensemble`](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.ensemble), [XGBoost library](https://xgboost.readthedocs.io/en/stable/) |\n",
    "| Unstructured data (images, audio, language) | Convolutional Neural Networks, Transformers | [`torchvision.models`](https://pytorch.org/vision/stable/models.html), [HuggingFace Transformers](https://huggingface.co/docs/transformers/index) | \n",
    "\n",
    "> **Note:** The table above is only for reference, the model you end up using will be highly dependant on the problem you're working on and the constraints you have (amount of data, latency requirements).\n",
    "\n",
    "Enough talking about models, let's now build a CNN that replicates the model on the [CNN Explainer website](https://poloclub.github.io/cnn-explainer/).\n",
    "\n",
    "![TinyVGG architecture, as setup by CNN explainer website](https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/03-cnn-explainer-model.png)\n",
    "\n",
    "To do so, we'll leverage the [`nn.Conv2d()`](https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html) and [`nn.MaxPool2d()`](https://pytorch.org/docs/stable/generated/torch.nn.MaxPool2d.html) layers from `torch.nn`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "dce60214-63fd-46e2-89ba-125445ac76b7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FashionMNISTModelV2(\n",
       "  (block_1): Sequential(\n",
       "    (0): Conv2d(1, 10, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "    (1): ReLU()\n",
       "    (2): Conv2d(10, 10, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "    (3): ReLU()\n",
       "    (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
       "  )\n",
       "  (block_2): Sequential(\n",
       "    (0): Conv2d(10, 10, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "    (1): ReLU()\n",
       "    (2): Conv2d(10, 10, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "    (3): ReLU()\n",
       "    (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
       "  )\n",
       "  (classifier): Sequential(\n",
       "    (0): Flatten(start_dim=1, end_dim=-1)\n",
       "    (1): Linear(in_features=490, out_features=10, bias=True)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a convolutional neural network \n",
    "class FashionMNISTModelV2(nn.Module):\n",
    "    \"\"\"\n",
    "    Model architecture copying TinyVGG from: \n",
    "    https://poloclub.github.io/cnn-explainer/\n",
    "    \"\"\"\n",
    "    def __init__(self, input_shape: int, hidden_units: int, output_shape: int):\n",
    "        super().__init__()\n",
    "        self.block_1 = nn.Sequential(\n",
    "            nn.Conv2d(in_channels=input_shape, \n",
    "                      out_channels=hidden_units, \n",
    "                      kernel_size=3, # how big is the square that's going over the image?\n",
    "                      stride=1, # default\n",
    "                      padding=1),# options = \"valid\" (no padding) or \"same\" (output has same shape as input) or int for specific number \n",
    "            nn.ReLU(),\n",
    "            nn.Conv2d(in_channels=hidden_units, \n",
    "                      out_channels=hidden_units,\n",
    "                      kernel_size=3,\n",
    "                      stride=1,\n",
    "                      padding=1),\n",
    "            nn.ReLU(),\n",
    "            nn.MaxPool2d(kernel_size=2,\n",
    "                         stride=2) # default stride value is same as kernel_size\n",
    "        )\n",
    "        self.block_2 = nn.Sequential(\n",
    "            nn.Conv2d(hidden_units, hidden_units, 3, padding=1),\n",
    "            nn.ReLU(),\n",
    "            nn.Conv2d(hidden_units, hidden_units, 3, padding=1),\n",
    "            nn.ReLU(),\n",
    "            nn.MaxPool2d(2)\n",
    "        )\n",
    "        self.classifier = nn.Sequential(\n",
    "            nn.Flatten(),\n",
    "            # Where did this in_features shape come from? \n",
    "            # It's because each layer of our network compresses and changes the shape of our inputs data.\n",
    "            nn.Linear(in_features=hidden_units*7*7, \n",
    "                      out_features=output_shape)\n",
    "        )\n",
    "    \n",
    "    def forward(self, x: torch.Tensor):\n",
    "        x = self.block_1(x)\n",
    "        # print(x.shape)\n",
    "        x = self.block_2(x)\n",
    "        # print(x.shape)\n",
    "        x = self.classifier(x)\n",
    "        # print(x.shape)\n",
    "        return x\n",
    "\n",
    "torch.manual_seed(42)\n",
    "model_2 = FashionMNISTModelV2(input_shape=1, \n",
    "    hidden_units=10, \n",
    "    output_shape=len(class_names)).to(device)\n",
    "model_2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a20f25e-cc16-4f85-a69b-62008c01d0ed",
   "metadata": {},
   "source": [
    "Nice!\n",
    "\n",
    "Our biggest model yet!\n",
    "\n",
    "What we've done is a common practice in machine learning.\n",
    "\n",
    "Find a model architecture somewhere and replicate it with code. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6478cc5a-7b33-425d-9ab3-6d40168a1aee",
   "metadata": {},
   "source": [
    "### 7.1 Stepping through `nn.Conv2d()`\n",
    "\n",
    "We could start using our model above and see what happens but let's first step through the two new layers we've added:\n",
    "* [`nn.Conv2d()`](https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html), also known as a convolutional layer.\n",
    "* [`nn.MaxPool2d()`](https://pytorch.org/docs/stable/generated/torch.nn.MaxPool2d.html), also known as a max pooling layer.\n",
    "\n",
    "> **Question:** What does the \"2d\" in `nn.Conv2d()` stand for?\n",
    ">\n",
    "> The 2d is for 2-dimensional data. As in, our images have two dimensions: height and width. Yes, there's color channel dimension but each of the color channel dimensions have two dimensions too: height and width.\n",
    ">\n",
    "> For other dimensional data (such as 1D for text or 3D for 3D objects) there's also `nn.Conv1d()` and `nn.Conv3d()`. \n",
    "\n",
    "To test the layers out, let's create some toy data just like the data used on CNN Explainer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "058b01ac-3f6a-4472-bcbf-3377974e3254",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image batch shape: torch.Size([32, 3, 64, 64]) -> [batch_size, color_channels, height, width]\n",
      "Single image shape: torch.Size([3, 64, 64]) -> [color_channels, height, width]\n",
      "Single image pixel values:\n",
      "tensor([[[ 1.9269,  1.4873,  0.9007,  ...,  1.8446, -1.1845,  1.3835],\n",
      "         [ 1.4451,  0.8564,  2.2181,  ...,  0.3399,  0.7200,  0.4114],\n",
      "         [ 1.9312,  1.0119, -1.4364,  ..., -0.5558,  0.7043,  0.7099],\n",
      "         ...,\n",
      "         [-0.5610, -0.4830,  0.4770,  ..., -0.2713, -0.9537, -0.6737],\n",
      "         [ 0.3076, -0.1277,  0.0366,  ..., -2.0060,  0.2824, -0.8111],\n",
      "         [-1.5486,  0.0485, -0.7712,  ..., -0.1403,  0.9416, -0.0118]],\n",
      "\n",
      "        [[-0.5197,  1.8524,  1.8365,  ...,  0.8935, -1.5114, -0.8515],\n",
      "         [ 2.0818,  1.0677, -1.4277,  ...,  1.6612, -2.6223, -0.4319],\n",
      "         [-0.1010, -0.4388, -1.9775,  ...,  0.2106,  0.2536, -0.7318],\n",
      "         ...,\n",
      "         [ 0.2779,  0.7342, -0.3736,  ..., -0.4601,  0.1815,  0.1850],\n",
      "         [ 0.7205, -0.2833,  0.0937,  ..., -0.1002, -2.3609,  2.2465],\n",
      "         [-1.3242, -0.1973,  0.2920,  ...,  0.5409,  0.6940,  1.8563]],\n",
      "\n",
      "        [[-0.7978,  1.0261,  1.1465,  ...,  1.2134,  0.9354, -0.0780],\n",
      "         [-1.4647, -1.9571,  0.1017,  ..., -1.9986, -0.7409,  0.7011],\n",
      "         [-1.3938,  0.8466, -1.7191,  ..., -1.1867,  0.1320,  0.3407],\n",
      "         ...,\n",
      "         [ 0.8206, -0.3745,  1.2499,  ..., -0.0676,  0.0385,  0.6335],\n",
      "         [-0.5589, -0.3393,  0.2347,  ...,  2.1181,  2.4569,  1.3083],\n",
      "         [-0.4092,  1.5199,  0.2401,  ..., -0.2558,  0.7870,  0.9924]]])\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(42)\n",
    "\n",
    "# Create sample batch of random numbers with same size as image batch\n",
    "images = torch.randn(size=(32, 3, 64, 64)) # [batch_size, color_channels, height, width]\n",
    "test_image = images[0] # get a single image for testing\n",
    "print(f\"Image batch shape: {images.shape} -> [batch_size, color_channels, height, width]\")\n",
    "print(f\"Single image shape: {test_image.shape} -> [color_channels, height, width]\") \n",
    "print(f\"Single image pixel values:\\n{test_image}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd3291c2-854e-4d0c-97b9-8bf46085fc43",
   "metadata": {},
   "source": [
    "Let's create an example `nn.Conv2d()` with various parameters:\n",
    "* `in_channels` (int) - Number of channels in the input image.\n",
    "* `out_channels` (int) - Number of channels produced by the convolution.\n",
    "* `kernel_size` (int or tuple) - Size of the convolving kernel/filter.\n",
    "* `stride` (int or tuple, optional) - How big of a step the convolving kernel takes at a time. Default: 1.\n",
    "* `padding` (int, tuple, str) - Padding added to all four sides of input. Default: 0.\n",
    "\n",
    "![example of going through the different parameters of a Conv2d layer](https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/03-conv2d-layer.gif)\n",
    "\n",
    "*Example of what happens when you change the hyperparameters of a `nn.Conv2d()` layer.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "ebd39562-1dad-40e3-90f5-750a5dac24e2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 1.5396,  0.0516,  0.6454,  ..., -0.3673,  0.8711,  0.4256],\n",
       "         [ 0.3662,  1.0114, -0.5997,  ...,  0.8983,  0.2809, -0.2741],\n",
       "         [ 1.2664, -1.4054,  0.3727,  ..., -0.3409,  1.2191, -0.0463],\n",
       "         ...,\n",
       "         [-0.1541,  0.5132, -0.3624,  ..., -0.2360, -0.4609, -0.0035],\n",
       "         [ 0.2981, -0.2432,  1.5012,  ..., -0.6289, -0.7283, -0.5767],\n",
       "         [-0.0386, -0.0781, -0.0388,  ...,  0.2842,  0.4228, -0.1802]],\n",
       "\n",
       "        [[-0.2840, -0.0319, -0.4455,  ..., -0.7956,  1.5599, -1.2449],\n",
       "         [ 0.2753, -0.1262, -0.6541,  ..., -0.2211,  0.1999, -0.8856],\n",
       "         [-0.5404, -1.5489,  0.0249,  ..., -0.5932, -1.0913, -0.3849],\n",
       "         ...,\n",
       "         [ 0.3870, -0.4064, -0.8236,  ...,  0.1734, -0.4330, -0.4951],\n",
       "         [-0.1984, -0.6386,  1.0263,  ..., -0.9401, -0.0585, -0.7833],\n",
       "         [-0.6306, -0.2052, -0.3694,  ..., -1.3248,  0.2456, -0.7134]],\n",
       "\n",
       "        [[ 0.4414,  0.5100,  0.4846,  ..., -0.8484,  0.2638,  1.1258],\n",
       "         [ 0.8117,  0.3191, -0.0157,  ...,  1.2686,  0.2319,  0.5003],\n",
       "         [ 0.3212,  0.0485, -0.2581,  ...,  0.2258,  0.2587, -0.8804],\n",
       "         ...,\n",
       "         [-0.1144, -0.1869,  0.0160,  ..., -0.8346,  0.0974,  0.8421],\n",
       "         [ 0.2941,  0.4417,  0.5866,  ..., -0.1224,  0.4814, -0.4799],\n",
       "         [ 0.6059, -0.0415, -0.2028,  ...,  0.1170,  0.2521, -0.4372]],\n",
       "\n",
       "        ...,\n",
       "\n",
       "        [[-0.2560, -0.0477,  0.6380,  ...,  0.6436,  0.7553, -0.7055],\n",
       "         [ 1.5595, -0.2209, -0.9486,  ..., -0.4876,  0.7754,  0.0750],\n",
       "         [-0.0797,  0.2471,  1.1300,  ...,  0.1505,  0.2354,  0.9576],\n",
       "         ...,\n",
       "         [ 1.1065,  0.6839,  1.2183,  ...,  0.3015, -0.1910, -0.1902],\n",
       "         [-0.3486, -0.7173, -0.3582,  ...,  0.4917,  0.7219,  0.1513],\n",
       "         [ 0.0119,  0.1017,  0.7839,  ..., -0.3752, -0.8127, -0.1257]],\n",
       "\n",
       "        [[ 0.3841,  1.1322,  0.1620,  ...,  0.7010,  0.0109,  0.6058],\n",
       "         [ 0.1664,  0.1873,  1.5924,  ...,  0.3733,  0.9096, -0.5399],\n",
       "         [ 0.4094, -0.0861, -0.7935,  ..., -0.1285, -0.9932, -0.3013],\n",
       "         ...,\n",
       "         [ 0.2688, -0.5630, -1.1902,  ...,  0.4493,  0.5404, -0.0103],\n",
       "         [ 0.0535,  0.4411,  0.5313,  ...,  0.0148, -1.0056,  0.3759],\n",
       "         [ 0.3031, -0.1590, -0.1316,  ..., -0.5384, -0.4271, -0.4876]],\n",
       "\n",
       "        [[-1.1865, -0.7280, -1.2331,  ..., -0.9013, -0.0542, -1.5949],\n",
       "         [-0.6345, -0.5920,  0.5326,  ..., -1.0395, -0.7963, -0.0647],\n",
       "         [-0.1132,  0.5166,  0.2569,  ...,  0.5595, -1.6881,  0.9485],\n",
       "         ...,\n",
       "         [-0.0254, -0.2669,  0.1927,  ..., -0.2917,  0.1088, -0.4807],\n",
       "         [-0.2609, -0.2328,  0.1404,  ..., -0.1325, -0.8436, -0.7524],\n",
       "         [-1.1399, -0.1751, -0.8705,  ...,  0.1589,  0.3377,  0.3493]]],\n",
       "       grad_fn=<SqueezeBackward1>)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.manual_seed(42)\n",
    "\n",
    "# Create a convolutional layer with same dimensions as TinyVGG \n",
    "# (try changing any of the parameters and see what happens)\n",
    "conv_layer = nn.Conv2d(in_channels=3,\n",
    "                       out_channels=10,\n",
    "                       kernel_size=3,\n",
    "                       stride=1,\n",
    "                       padding=0) # also try using \"valid\" or \"same\" here \n",
    "\n",
    "# Pass the data through the convolutional layer\n",
    "conv_layer(test_image) # Note: If running PyTorch <1.11.0, this will error because of shape issues (nn.Conv.2d() expects a 4d tensor as input) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb0184ad-5c16-4e1c-bcfa-70ecf15377da",
   "metadata": {},
   "source": [
    "If we try to pass a single image in, we get a shape mismatch error:\n",
    "\n",
    "> `RuntimeError: Expected 4-dimensional input for 4-dimensional weight [10, 3, 3, 3], but got 3-dimensional input of size [3, 64, 64] instead`\n",
    ">\n",
    "> **Note:** If you're running PyTorch 1.11.0+, this error won't occur.\n",
    "\n",
    "This is because our `nn.Conv2d()` layer expects a 4-dimensional tensor as input with size `(N, C, H, W)` or `[batch_size, color_channels, height, width]`.\n",
    "\n",
    "Right now our single image `test_image` only has a shape of `[color_channels, height, width]` or `[3, 64, 64]`.\n",
    "\n",
    "We can fix this for a single image using `test_image.unsqueeze(dim=0)` to add an extra dimension for `N`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "abba741d-a1ed-44ed-ba53-41d589433a2c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 3, 64, 64])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add extra dimension to test image\n",
    "test_image.unsqueeze(dim=0).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "c7280a49-4ee0-452b-a514-61115b6a444c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 10, 62, 62])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Pass test image with extra dimension through conv_layer\n",
    "conv_layer(test_image.unsqueeze(dim=0)).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "181df81b-7c5a-46cc-b8d5-a592bf755a13",
   "metadata": {},
   "source": [
    "Hmm, notice what happens to our shape (the same shape as the first layer of TinyVGG on [CNN Explainer](https://poloclub.github.io/cnn-explainer/)), we get different channel sizes as well as different pixel sizes.\n",
    "\n",
    "What if we changed the values of `conv_layer`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "04445d45-cf2f-4c1d-b215-bc50865a207a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 10, 30, 30])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.manual_seed(42)\n",
    "# Create a new conv_layer with different values (try setting these to whatever you like)\n",
    "conv_layer_2 = nn.Conv2d(in_channels=3, # same number of color channels as our input image\n",
    "                         out_channels=10,\n",
    "                         kernel_size=(5, 5), # kernel is usually a square so a tuple also works\n",
    "                         stride=2,\n",
    "                         padding=0)\n",
    "\n",
    "# Pass single image through new conv_layer_2 (this calls nn.Conv2d()'s forward() method on the input)\n",
    "conv_layer_2(test_image.unsqueeze(dim=0)).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b27dbdbb-3e32-4ffa-803e-cf943d96c72b",
   "metadata": {},
   "source": [
    "Woah, we get another shape change.\n",
    "\n",
    "Now our image is of shape `[1, 10, 30, 30]` (it will be different if you use different values) or `[batch_size=1, color_channels=10, height=30, width=30]`.\n",
    "\n",
    "What's going on here?\n",
    "\n",
    "Behind the scenes, our `nn.Conv2d()` is compressing the information stored in the image.\n",
    "\n",
    "It does this by performing operations on the input (our test image) against its internal parameters.\n",
    "\n",
    "The goal of this is similar to all of the other neural networks we've been building.\n",
    "\n",
    "Data goes in and the layers try to update their internal parameters (patterns) to lower the loss function thanks to some help of the optimizer.\n",
    "\n",
    "The only difference is *how* the different layers calculate their parameter updates or in PyTorch terms, the operation present in the layer `forward()` method.\n",
    "\n",
    "If we check out our `conv_layer_2.state_dict()` we'll find a similar weight and bias setup as we've seen before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "46027ed1-c3a7-46bd-bab7-17f8c20e354b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OrderedDict([('weight', tensor([[[[ 0.0883,  0.0958, -0.0271,  0.1061, -0.0253],\n",
      "          [ 0.0233, -0.0562,  0.0678,  0.1018, -0.0847],\n",
      "          [ 0.1004,  0.0216,  0.0853,  0.0156,  0.0557],\n",
      "          [-0.0163,  0.0890,  0.0171, -0.0539,  0.0294],\n",
      "          [-0.0532, -0.0135, -0.0469,  0.0766, -0.0911]],\n",
      "\n",
      "         [[-0.0532, -0.0326, -0.0694,  0.0109, -0.1140],\n",
      "          [ 0.1043, -0.0981,  0.0891,  0.0192, -0.0375],\n",
      "          [ 0.0714,  0.0180,  0.0933,  0.0126, -0.0364],\n",
      "          [ 0.0310, -0.0313,  0.0486,  0.1031,  0.0667],\n",
      "          [-0.0505,  0.0667,  0.0207,  0.0586, -0.0704]],\n",
      "\n",
      "         [[-0.1143, -0.0446, -0.0886,  0.0947,  0.0333],\n",
      "          [ 0.0478,  0.0365, -0.0020,  0.0904, -0.0820],\n",
      "          [ 0.0073, -0.0788,  0.0356, -0.0398,  0.0354],\n",
      "          [-0.0241,  0.0958, -0.0684, -0.0689, -0.0689],\n",
      "          [ 0.1039,  0.0385,  0.1111, -0.0953, -0.1145]]],\n",
      "\n",
      "\n",
      "        [[[-0.0903, -0.0777,  0.0468,  0.0413,  0.0959],\n",
      "          [-0.0596, -0.0787,  0.0613, -0.0467,  0.0701],\n",
      "          [-0.0274,  0.0661, -0.0897, -0.0583,  0.0352],\n",
      "          [ 0.0244, -0.0294,  0.0688,  0.0785, -0.0837],\n",
      "          [-0.0616,  0.1057, -0.0390, -0.0409, -0.1117]],\n",
      "\n",
      "         [[-0.0661,  0.0288, -0.0152, -0.0838,  0.0027],\n",
      "          [-0.0789, -0.0980, -0.0636, -0.1011, -0.0735],\n",
      "          [ 0.1154,  0.0218,  0.0356, -0.1077, -0.0758],\n",
      "          [-0.0384,  0.0181, -0.1016, -0.0498, -0.0691],\n",
      "          [ 0.0003, -0.0430, -0.0080, -0.0782, -0.0793]],\n",
      "\n",
      "         [[-0.0674, -0.0395, -0.0911,  0.0968, -0.0229],\n",
      "          [ 0.0994,  0.0360, -0.0978,  0.0799, -0.0318],\n",
      "          [-0.0443, -0.0958, -0.1148,  0.0330, -0.0252],\n",
      "          [ 0.0450, -0.0948,  0.0857, -0.0848, -0.0199],\n",
      "          [ 0.0241,  0.0596,  0.0932,  0.1052, -0.0916]]],\n",
      "\n",
      "\n",
      "        [[[ 0.0291, -0.0497, -0.0127, -0.0864,  0.1052],\n",
      "          [-0.0847,  0.0617,  0.0406,  0.0375, -0.0624],\n",
      "          [ 0.1050,  0.0254,  0.0149, -0.1018,  0.0485],\n",
      "          [-0.0173, -0.0529,  0.0992,  0.0257, -0.0639],\n",
      "          [-0.0584, -0.0055,  0.0645, -0.0295, -0.0659]],\n",
      "\n",
      "         [[-0.0395, -0.0863,  0.0412,  0.0894, -0.1087],\n",
      "          [ 0.0268,  0.0597,  0.0209, -0.0411,  0.0603],\n",
      "          [ 0.0607,  0.0432, -0.0203, -0.0306,  0.0124],\n",
      "          [-0.0204, -0.0344,  0.0738,  0.0992, -0.0114],\n",
      "          [-0.0259,  0.0017, -0.0069,  0.0278,  0.0324]],\n",
      "\n",
      "         [[-0.1049, -0.0426,  0.0972,  0.0450, -0.0057],\n",
      "          [-0.0696, -0.0706, -0.1034, -0.0376,  0.0390],\n",
      "          [ 0.0736,  0.0533, -0.1021, -0.0694, -0.0182],\n",
      "          [ 0.1117,  0.0167, -0.0299,  0.0478, -0.0440],\n",
      "          [-0.0747,  0.0843, -0.0525, -0.0231, -0.1149]]],\n",
      "\n",
      "\n",
      "        [[[ 0.0773,  0.0875,  0.0421, -0.0805, -0.1140],\n",
      "          [-0.0938,  0.0861,  0.0554,  0.0972,  0.0605],\n",
      "          [ 0.0292, -0.0011, -0.0878, -0.0989, -0.1080],\n",
      "          [ 0.0473, -0.0567, -0.0232, -0.0665, -0.0210],\n",
      "          [-0.0813, -0.0754,  0.0383, -0.0343,  0.0713]],\n",
      "\n",
      "         [[-0.0370, -0.0847, -0.0204, -0.0560, -0.0353],\n",
      "          [-0.1099,  0.0646, -0.0804,  0.0580,  0.0524],\n",
      "          [ 0.0825, -0.0886,  0.0830, -0.0546,  0.0428],\n",
      "          [ 0.1084, -0.0163, -0.0009, -0.0266, -0.0964],\n",
      "          [ 0.0554, -0.1146,  0.0717,  0.0864,  0.1092]],\n",
      "\n",
      "         [[-0.0272, -0.0949,  0.0260,  0.0638, -0.1149],\n",
      "          [-0.0262, -0.0692, -0.0101, -0.0568, -0.0472],\n",
      "          [-0.0367, -0.1097,  0.0947,  0.0968, -0.0181],\n",
      "          [-0.0131, -0.0471, -0.1043, -0.1124,  0.0429],\n",
      "          [-0.0634, -0.0742, -0.0090, -0.0385, -0.0374]]],\n",
      "\n",
      "\n",
      "        [[[ 0.0037, -0.0245, -0.0398, -0.0553, -0.0940],\n",
      "          [ 0.0968, -0.0462,  0.0306, -0.0401,  0.0094],\n",
      "          [ 0.1077,  0.0532, -0.1001,  0.0458,  0.1096],\n",
      "          [ 0.0304,  0.0774,  0.1138, -0.0177,  0.0240],\n",
      "          [-0.0803, -0.0238,  0.0855,  0.0592, -0.0731]],\n",
      "\n",
      "         [[-0.0926, -0.0789, -0.1140, -0.0891, -0.0286],\n",
      "          [ 0.0779,  0.0193, -0.0878, -0.0926,  0.0574],\n",
      "          [-0.0859, -0.0142,  0.0554, -0.0534, -0.0126],\n",
      "          [-0.0101, -0.0273, -0.0585, -0.1029, -0.0933],\n",
      "          [-0.0618,  0.1115, -0.0558, -0.0775,  0.0280]],\n",
      "\n",
      "         [[ 0.0318,  0.0633,  0.0878,  0.0643, -0.1145],\n",
      "          [ 0.0102,  0.0699, -0.0107, -0.0680,  0.1101],\n",
      "          [-0.0432, -0.0657, -0.1041,  0.0052,  0.0512],\n",
      "          [ 0.0256,  0.0228, -0.0876, -0.1078,  0.0020],\n",
      "          [ 0.1053,  0.0666, -0.0672, -0.0150, -0.0851]]],\n",
      "\n",
      "\n",
      "        [[[-0.0557,  0.0209,  0.0629,  0.0957, -0.1060],\n",
      "          [ 0.0772, -0.0814,  0.0432,  0.0977,  0.0016],\n",
      "          [ 0.1051, -0.0984, -0.0441,  0.0673, -0.0252],\n",
      "          [-0.0236, -0.0481,  0.0796,  0.0566,  0.0370],\n",
      "          [-0.0649, -0.0937,  0.0125,  0.0342, -0.0533]],\n",
      "\n",
      "         [[-0.0323,  0.0780,  0.0092,  0.0052, -0.0284],\n",
      "          [-0.1046, -0.1086, -0.0552, -0.0587,  0.0360],\n",
      "          [-0.0336, -0.0452,  0.1101,  0.0402,  0.0823],\n",
      "          [-0.0559, -0.0472,  0.0424, -0.0769, -0.0755],\n",
      "          [-0.0056, -0.0422, -0.0866,  0.0685,  0.0929]],\n",
      "\n",
      "         [[ 0.0187, -0.0201, -0.1070, -0.0421,  0.0294],\n",
      "          [ 0.0544, -0.0146, -0.0457,  0.0643, -0.0920],\n",
      "          [ 0.0730, -0.0448,  0.0018, -0.0228,  0.0140],\n",
      "          [-0.0349,  0.0840, -0.0030,  0.0901,  0.1110],\n",
      "          [-0.0563, -0.0842,  0.0926,  0.0905, -0.0882]]],\n",
      "\n",
      "\n",
      "        [[[-0.0089, -0.1139, -0.0945,  0.0223,  0.0307],\n",
      "          [ 0.0245, -0.0314,  0.1065,  0.0165, -0.0681],\n",
      "          [-0.0065,  0.0277,  0.0404, -0.0816,  0.0433],\n",
      "          [-0.0590, -0.0959, -0.0631,  0.1114,  0.0987],\n",
      "          [ 0.1034,  0.0678,  0.0872, -0.0155, -0.0635]],\n",
      "\n",
      "         [[ 0.0577, -0.0598, -0.0779, -0.0369,  0.0242],\n",
      "          [ 0.0594, -0.0448, -0.0680,  0.0156, -0.0681],\n",
      "          [-0.0752,  0.0602, -0.0194,  0.1055,  0.1123],\n",
      "          [ 0.0345,  0.0397,  0.0266,  0.0018, -0.0084],\n",
      "          [ 0.0016,  0.0431,  0.1074, -0.0299, -0.0488]],\n",
      "\n",
      "         [[-0.0280, -0.0558,  0.0196,  0.0862,  0.0903],\n",
      "          [ 0.0530, -0.0850, -0.0620, -0.0254, -0.0213],\n",
      "          [ 0.0095, -0.1060,  0.0359, -0.0881, -0.0731],\n",
      "          [-0.0960,  0.1006, -0.1093,  0.0871, -0.0039],\n",
      "          [-0.0134,  0.0722, -0.0107,  0.0724,  0.0835]]],\n",
      "\n",
      "\n",
      "        [[[-0.1003,  0.0444,  0.0218,  0.0248,  0.0169],\n",
      "          [ 0.0316, -0.0555, -0.0148,  0.1097,  0.0776],\n",
      "          [-0.0043, -0.1086,  0.0051, -0.0786,  0.0939],\n",
      "          [-0.0701, -0.0083, -0.0256,  0.0205,  0.1087],\n",
      "          [ 0.0110,  0.0669,  0.0896,  0.0932, -0.0399]],\n",
      "\n",
      "         [[-0.0258,  0.0556, -0.0315,  0.0541, -0.0252],\n",
      "          [-0.0783,  0.0470,  0.0177,  0.0515,  0.1147],\n",
      "          [ 0.0788,  0.1095,  0.0062, -0.0993, -0.0810],\n",
      "          [-0.0717, -0.1018, -0.0579, -0.1063, -0.1065],\n",
      "          [-0.0690, -0.1138, -0.0709,  0.0440,  0.0963]],\n",
      "\n",
      "         [[-0.0343, -0.0336,  0.0617, -0.0570, -0.0546],\n",
      "          [ 0.0711, -0.1006,  0.0141,  0.1020,  0.0198],\n",
      "          [ 0.0314, -0.0672, -0.0016,  0.0063,  0.0283],\n",
      "          [ 0.0449,  0.1003, -0.0881,  0.0035, -0.0577],\n",
      "          [-0.0913, -0.0092, -0.1016,  0.0806,  0.0134]]],\n",
      "\n",
      "\n",
      "        [[[-0.0622,  0.0603, -0.1093, -0.0447, -0.0225],\n",
      "          [-0.0981, -0.0734, -0.0188,  0.0876,  0.1115],\n",
      "          [ 0.0735, -0.0689, -0.0755,  0.1008,  0.0408],\n",
      "          [ 0.0031,  0.0156, -0.0928, -0.0386,  0.1112],\n",
      "          [-0.0285, -0.0058, -0.0959, -0.0646, -0.0024]],\n",
      "\n",
      "         [[-0.0717, -0.0143,  0.0470, -0.1130,  0.0343],\n",
      "          [-0.0763, -0.0564,  0.0443,  0.0918, -0.0316],\n",
      "          [-0.0474, -0.1044, -0.0595, -0.1011, -0.0264],\n",
      "          [ 0.0236, -0.1082,  0.1008,  0.0724, -0.1130],\n",
      "          [-0.0552,  0.0377, -0.0237, -0.0126, -0.0521]],\n",
      "\n",
      "         [[ 0.0927, -0.0645,  0.0958,  0.0075,  0.0232],\n",
      "          [ 0.0901, -0.0190, -0.0657, -0.0187,  0.0937],\n",
      "          [-0.0857,  0.0262, -0.1135,  0.0605,  0.0427],\n",
      "          [ 0.0049,  0.0496,  0.0001,  0.0639, -0.0914],\n",
      "          [-0.0170,  0.0512,  0.1150,  0.0588, -0.0840]]],\n",
      "\n",
      "\n",
      "        [[[ 0.0888, -0.0257, -0.0247, -0.1050, -0.0182],\n",
      "          [ 0.0817,  0.0161, -0.0673,  0.0355, -0.0370],\n",
      "          [ 0.1054, -0.1002, -0.0365, -0.1115, -0.0455],\n",
      "          [ 0.0364,  0.1112,  0.0194,  0.1132,  0.0226],\n",
      "          [ 0.0667,  0.0926,  0.0965, -0.0646,  0.1062]],\n",
      "\n",
      "         [[ 0.0699, -0.0540, -0.0551, -0.0969,  0.0290],\n",
      "          [-0.0936,  0.0488,  0.0365, -0.1003,  0.0315],\n",
      "          [-0.0094,  0.0527,  0.0663, -0.1148,  0.1059],\n",
      "          [ 0.0968,  0.0459, -0.1055, -0.0412, -0.0335],\n",
      "          [-0.0297,  0.0651,  0.0420,  0.0915, -0.0432]],\n",
      "\n",
      "         [[ 0.0389,  0.0411, -0.0961, -0.1120, -0.0599],\n",
      "          [ 0.0790, -0.1087, -0.1005,  0.0647,  0.0623],\n",
      "          [ 0.0950, -0.0872, -0.0845,  0.0592,  0.1004],\n",
      "          [ 0.0691,  0.0181,  0.0381,  0.1096, -0.0745],\n",
      "          [-0.0524,  0.0808, -0.0790, -0.0637,  0.0843]]]])), ('bias', tensor([ 0.0364,  0.0373, -0.0489, -0.0016,  0.1057, -0.0693,  0.0009,  0.0549,\n",
      "        -0.0797,  0.1121]))])\n"
     ]
    }
   ],
   "source": [
    "# Check out the conv_layer_2 internal parameters\n",
    "print(conv_layer_2.state_dict())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b708eb6-ae46-4d8c-a8a4-1392827d3e37",
   "metadata": {},
   "source": [
    "Look at that! A bunch of random numbers for a weight and bias tensor.\n",
    "\n",
    "The shapes of these are manipulated by the inputs we passed to `nn.Conv2d()` when we set it up.\n",
    "\n",
    "Let's check them out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "e5518d61-c0b7-4351-b5ea-4d6b6144291a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "conv_layer_2 weight shape: \n",
      "torch.Size([10, 3, 5, 5]) -> [out_channels=10, in_channels=3, kernel_size=5, kernel_size=5]\n",
      "\n",
      "conv_layer_2 bias shape: \n",
      "torch.Size([10]) -> [out_channels=10]\n"
     ]
    }
   ],
   "source": [
    "# Get shapes of weight and bias tensors within conv_layer_2\n",
    "print(f\"conv_layer_2 weight shape: \\n{conv_layer_2.weight.shape} -> [out_channels=10, in_channels=3, kernel_size=5, kernel_size=5]\")\n",
    "print(f\"\\nconv_layer_2 bias shape: \\n{conv_layer_2.bias.shape} -> [out_channels=10]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0de23c7-4501-4156-80a4-ac889a636a42",
   "metadata": {},
   "source": [
    "> **Question:** What should we set the parameters of our `nn.Conv2d()` layers?\n",
    ">\n",
    "> That's a good one. But similar to many other things in machine learning, the values of these aren't set in stone (and recall, because these values are ones we can set ourselves, they're referred to as \"**hyperparameters**\"). \n",
    ">\n",
    "> The best way to find out is to try out different values and see how they effect your model's performance.\n",
    ">\n",
    "> Or even better, find a working example on a problem similar to yours (like we've done with TinyVGG) and copy it. \n",
    "\n",
    "We're working with a different of layer here to what we've seen before.\n",
    "\n",
    "But the premise remains the same: start with random numbers and update them to better represent the data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6370d45d-ca44-4fa0-a2d7-efaf0a207b91",
   "metadata": {},
   "source": [
    "### 7.2 Stepping through `nn.MaxPool2d()`\n",
    "Now let's check out what happens when we move data through `nn.MaxPool2d()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "1164c753-19d9-43b7-a04f-017d0f7188c3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test image original shape: torch.Size([3, 64, 64])\n",
      "Test image with unsqueezed dimension: torch.Size([1, 3, 64, 64])\n",
      "Shape after going through conv_layer(): torch.Size([1, 10, 62, 62])\n",
      "Shape after going through conv_layer() and max_pool_layer(): torch.Size([1, 10, 31, 31])\n"
     ]
    }
   ],
   "source": [
    "# Print out original image shape without and with unsqueezed dimension\n",
    "print(f\"Test image original shape: {test_image.shape}\")\n",
    "print(f\"Test image with unsqueezed dimension: {test_image.unsqueeze(dim=0).shape}\")\n",
    "\n",
    "# Create a sample nn.MaxPoo2d() layer\n",
    "max_pool_layer = nn.MaxPool2d(kernel_size=2)\n",
    "\n",
    "# Pass data through just the conv_layer\n",
    "test_image_through_conv = conv_layer(test_image.unsqueeze(dim=0))\n",
    "print(f\"Shape after going through conv_layer(): {test_image_through_conv.shape}\")\n",
    "\n",
    "# Pass data through the max pool layer\n",
    "test_image_through_conv_and_max_pool = max_pool_layer(test_image_through_conv)\n",
    "print(f\"Shape after going through conv_layer() and max_pool_layer(): {test_image_through_conv_and_max_pool.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de029abd-6674-4bfa-99ab-322f339f89f4",
   "metadata": {},
   "source": [
    "Notice the change in the shapes of what's happening in and out of a `nn.MaxPool2d()` layer.\n",
    "\n",
    "The `kernel_size` of the `nn.MaxPool2d()` layer will effects the size of the output shape.\n",
    "\n",
    "In our case, the shape halves from a `62x62` image to `31x31` image.\n",
    "\n",
    "Let's see this work with a smaller tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "e6a2b196-4845-4b40-9212-e75406e88875",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Random tensor:\n",
      "tensor([[[[0.3367, 0.1288],\n",
      "          [0.2345, 0.2303]]]])\n",
      "Random tensor shape: torch.Size([1, 1, 2, 2])\n",
      "\n",
      "Max pool tensor:\n",
      "tensor([[[[0.3367]]]]) <- this is the maximum value from random_tensor\n",
      "Max pool tensor shape: torch.Size([1, 1, 1, 1])\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(42)\n",
    "# Create a random tensor with a similiar number of dimensions to our images\n",
    "random_tensor = torch.randn(size=(1, 1, 2, 2))\n",
    "print(f\"Random tensor:\\n{random_tensor}\")\n",
    "print(f\"Random tensor shape: {random_tensor.shape}\")\n",
    "\n",
    "# Create a max pool layer\n",
    "max_pool_layer = nn.MaxPool2d(kernel_size=2) # see what happens when you change the kernel_size value \n",
    "\n",
    "# Pass the random tensor through the max pool layer\n",
    "max_pool_tensor = max_pool_layer(random_tensor)\n",
    "print(f\"\\nMax pool tensor:\\n{max_pool_tensor} <- this is the maximum value from random_tensor\")\n",
    "print(f\"Max pool tensor shape: {max_pool_tensor.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "002e586e-dcb3-40fe-a7dd-a1c18a3b8da0",
   "metadata": {},
   "source": [
    "Notice the final two dimensions between `random_tensor` and `max_pool_tensor`, they go from `[2, 2]` to `[1, 1]`.\n",
    "\n",
    "In essence, they get halved.\n",
    "\n",
    "And the change would be different for different values of `kernel_size` for `nn.MaxPool2d()`.\n",
    "\n",
    "Also notice the value leftover in `max_pool_tensor` is the **maximum** value from `random_tensor`.\n",
    "\n",
    "What's happening here?\n",
    "\n",
    "This is another important piece of the puzzle of neural networks.\n",
    "\n",
    "Essentially, **every layer in a neural network is trying to compress data from higher dimensional space to lower dimensional space**. \n",
    "\n",
    "In other words, take a lot of numbers (raw data) and learn patterns in those numbers, patterns that are predictive whilst also being *smaller* in size than the original values.\n",
    "\n",
    "From an artificial intelligence perspective, you could consider the whole goal of a neural network to *compress* information.\n",
    "\n",
    "![each layer of a neural network compresses the original input data into a smaller representation that is (hopefully) capable of making predictions on future input data](https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/03-conv-net-as-compression.png)\n",
    "\n",
    "This means, that from the point of view of a neural network, intelligence is compression.\n",
    "\n",
    "This is the idea of the use of a `nn.MaxPool2d()` layer: take the maximum value from a portion of a tensor and disregard the rest.\n",
    "\n",
    "In essence, lowering the dimensionality of a tensor whilst still retaining a (hopefully) significant portion of the information.\n",
    "\n",
    "It is the same story for a `nn.Conv2d()` layer.\n",
    "\n",
    "Except instead of just taking the maximum, the `nn.Conv2d()` performs a conovlutional operation on the data (see this in action on the [CNN Explainer webpage](https://poloclub.github.io/cnn-explainer/)).\n",
    "\n",
    "> **Exercise:** What do you think the [`nn.AvgPool2d()`](https://pytorch.org/docs/stable/generated/torch.nn.AvgPool2d.html) layer does? Try making a random tensor like we did above and passing it through. Check the input and output shapes as well as the input and output values.\n",
    "\n",
    "> **Extra-curriculum:** Lookup \"most common convolutional neural networks\", what architectures do you find? Are any of them contained within the [`torchvision.models`](https://pytorch.org/vision/stable/models.html) library? What do you think you could do with these?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39a3c646-52f0-4f4b-8527-2fc33d0dfb13",
   "metadata": {},
   "source": [
    "### 7.3 Setup a loss function and optimizer for `model_2`\n",
    "\n",
    "We've stepped through the layers in our first CNN enough.\n",
    "\n",
    "But remember, if something still isn't clear, try starting small.\n",
    "\n",
    "Pick a single layer of a model, pass some data through it and see what happens.\n",
    "\n",
    "Now it's time to move forward and get to training!\n",
    "\n",
    "Let's setup a loss function and an optimizer.\n",
    "\n",
    "We'll use the functions as before, `nn.CrossEntropyLoss()` as the loss function (since we're working with multi-class classification data).\n",
    "\n",
    "And `torch.optim.SGD()` as the optimizer to optimize `model_2.parameters()` with a learning rate of `0.1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "06a76a1b-5f6f-4018-bf7b-8388b385476f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup loss and optimizer\n",
    "loss_fn = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.SGD(params=model_2.parameters(), \n",
    "                             lr=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "758bc223-a244-4604-a07a-e2fc2f96c2f6",
   "metadata": {},
   "source": [
    "### 7.4 Training and testing `model_2` using our training and test functions\n",
    "\n",
    "Loss and optimizer ready!\n",
    "\n",
    "Time to train and test.\n",
    "\n",
    "We'll use our `train_step()` and `test_step()` functions we created before.\n",
    "\n",
    "We'll also measure the time to compare it to our other models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "861d126e-d876-40b3-9b7a-66cfc2f1bf05",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5cab6231275749b2b4d647850f3e361d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0\n",
      "---------\n",
      "Train loss: 0.59738 | Train accuracy: 78.40%\n",
      "Test loss: 0.40513 | Test accuracy: 85.51%\n",
      "\n",
      "Epoch: 1\n",
      "---------\n",
      "Train loss: 0.36556 | Train accuracy: 86.87%\n",
      "Test loss: 0.35803 | Test accuracy: 86.75%\n",
      "\n",
      "Epoch: 2\n",
      "---------\n",
      "Train loss: 0.32796 | Train accuracy: 88.16%\n",
      "Test loss: 0.32870 | Test accuracy: 88.19%\n",
      "\n",
      "Train time on cuda: 23.650 seconds\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(42)\n",
    "\n",
    "# Measure time\n",
    "from timeit import default_timer as timer\n",
    "train_time_start_model_2 = timer()\n",
    "\n",
    "# Train and test model \n",
    "epochs = 3\n",
    "for epoch in tqdm(range(epochs)):\n",
    "    print(f\"Epoch: {epoch}\\n---------\")\n",
    "    train_step(data_loader=train_dataloader, \n",
    "        model=model_2, \n",
    "        loss_fn=loss_fn,\n",
    "        optimizer=optimizer,\n",
    "        accuracy_fn=accuracy_fn,\n",
    "        device=device\n",
    "    )\n",
    "    test_step(data_loader=test_dataloader,\n",
    "        model=model_2,\n",
    "        loss_fn=loss_fn,\n",
    "        accuracy_fn=accuracy_fn,\n",
    "        device=device\n",
    "    )\n",
    "\n",
    "train_time_end_model_2 = timer()\n",
    "total_train_time_model_2 = print_train_time(start=train_time_start_model_2,\n",
    "                                           end=train_time_end_model_2,\n",
    "                                           device=device)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfec7b7e-4dba-4016-957a-a29c6c10fde0",
   "metadata": {},
   "source": [
    "Woah! Looks like the convolutional and max pooling layers helped improve performance a little.\n",
    "\n",
    "Let's evaluate `model_2`'s results with our `eval_model()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "c1bf8b89-1389-4395-a1c4-9c6e94d9e71c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'model_name': 'FashionMNISTModelV2',\n",
       " 'model_loss': 0.3286978006362915,\n",
       " 'model_acc': 88.18889776357827}"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get model_2 results \n",
    "model_2_results = eval_model(\n",
    "    model=model_2,\n",
    "    data_loader=test_dataloader,\n",
    "    loss_fn=loss_fn,\n",
    "    accuracy_fn=accuracy_fn\n",
    ")\n",
    "model_2_results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24c5ff68-b0bc-4b09-9da6-696736bc262d",
   "metadata": {},
   "source": [
    "## 8. Compare model results and training time\n",
    "\n",
    "We've trained three different models.\n",
    "\n",
    "1. `model_0` - our baseline model with two `nn.Linear()` layers.\n",
    "2. `model_1` - the same setup as our baseline model except with `nn.ReLU()` layers in between the `nn.Linear()` layers.\n",
    "3. `model_2` - our first CNN model that mimics the TinyVGG architecture on the CNN Explainer website.\n",
    "\n",
    "This is a regular practice in machine learning.\n",
    "\n",
    "Building multiple models and performing multiple training experiments to see which performs best.\n",
    "\n",
    "Let's combine our model results dictionaries into a DataFrame and find out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "52d84ee1-1ad4-4860-b147-f8912c1febc7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model_name</th>\n",
       "      <th>model_loss</th>\n",
       "      <th>model_acc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>FashionMNISTModelV0</td>\n",
       "      <td>0.476639</td>\n",
       "      <td>83.426518</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>FashionMNISTModelV1</td>\n",
       "      <td>0.685001</td>\n",
       "      <td>75.019968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>FashionMNISTModelV2</td>\n",
       "      <td>0.328698</td>\n",
       "      <td>88.188898</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            model_name  model_loss  model_acc\n",
       "0  FashionMNISTModelV0    0.476639  83.426518\n",
       "1  FashionMNISTModelV1    0.685001  75.019968\n",
       "2  FashionMNISTModelV2    0.328698  88.188898"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "compare_results = pd.DataFrame([model_0_results, model_1_results, model_2_results])\n",
    "compare_results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c67f3fb5-ce7b-40b8-86a0-2797492de0ef",
   "metadata": {},
   "source": [
    "Nice!\n",
    "\n",
    "We can add the training time values too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "297af38f-e69f-4c6f-9027-fcaf0482a55c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model_name</th>\n",
       "      <th>model_loss</th>\n",
       "      <th>model_acc</th>\n",
       "      <th>training_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>FashionMNISTModelV0</td>\n",
       "      <td>0.476639</td>\n",
       "      <td>83.426518</td>\n",
       "      <td>15.548849</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>FashionMNISTModelV1</td>\n",
       "      <td>0.685001</td>\n",
       "      <td>75.019968</td>\n",
       "      <td>17.355571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>FashionMNISTModelV2</td>\n",
       "      <td>0.328698</td>\n",
       "      <td>88.188898</td>\n",
       "      <td>23.650468</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            model_name  model_loss  model_acc  training_time\n",
       "0  FashionMNISTModelV0    0.476639  83.426518      15.548849\n",
       "1  FashionMNISTModelV1    0.685001  75.019968      17.355571\n",
       "2  FashionMNISTModelV2    0.328698  88.188898      23.650468"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add training times to results comparison\n",
    "compare_results[\"training_time\"] = [total_train_time_model_0,\n",
    "                                    total_train_time_model_1,\n",
    "                                    total_train_time_model_2]\n",
    "compare_results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbbe5832-1081-4c76-8d5b-06c7a06da7b9",
   "metadata": {},
   "source": [
    "It looks like our CNN (`FashionMNISTModelV2`) model performed the best (lowest loss, highest accuracy) but had the longest training time.\n",
    "\n",
    "And our baseline model (`FashionMNISTModelV0`) performed better than `model_1` (`FashionMNISTModelV1`).\n",
    "\n",
    "### Performance-speed tradeoff\n",
    "\n",
    "Something to be aware of in machine learning is the **performance-speed** tradeoff.\n",
    "\n",
    "Generally, you get better performance out of a larger, more complex model (like we did with `model_2`).\n",
    "\n",
    "However, this performance increase often comes at a sacrifice of training speed and inference speed.\n",
    "\n",
    "> **Note:** The training times you get will be very dependant on the hardware you use. \n",
    ">\n",
    "> Generally, the more CPU cores you have, the faster your models will train on CPU. And similar for GPUs.\n",
    "> \n",
    "> Newer hardware (in terms of age) will also often train models faster due to incorporating technology advances.\n",
    "\n",
    "How about we get visual?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "5eb0df60-9318-47d0-adce-f8788ed3999e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeIAAAEGCAYAAAC5PJY3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAbLklEQVR4nO3de7BmVX3m8e8DKBdRJgawmmvjCCJyh1ERNCG0CRFBSDRA0AEzUzrxmlFLcTROJt7AEsqhUCbECCjYgAoMhIsXgqUBEbu5dIMNxsJWEUYgIIE01+Y3f+zV9Mvh9Dnvge5eQH8/VafO+6699t5rrYJ+zlr7ffdOVSFJkvpYq3cDJElakxnEkiR1ZBBLktSRQSxJUkcGsSRJHa3TuwF65tl4441r9uzZvZshSc8Y8+fPv7OqNplsm0GsGZs9ezbz5s3r3QxJesZI8osVbXNpWpKkjgxiSZI6MoglSerIIJYkqSODWJKkjgxiSZI6MoglSerIIJYkqSODWJKkjgxiSZI6MoglSerIIJYkqSODWJKkjgxiSZI6MoglSerIIJYkqaN1ejdAzzwLf30Ps4++sHczJGm1WXzMAavs2M6IJUnqyCCWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKmjVRbESZYmuXbkZ/YM91+cZONJyg9KcvSTbNPsJJXkEyNlGyd5OMmJ7f3fJFmSZNOROvdNfJ1krSQnJLk+ycIkP06yTZIftf7+Mskdo/1vffrBhDZdm+T6Gfbje0n2HKdOklOTvGPCtoOTXJRkyySXJVmU5IYk75tJOyRJT92qfB7x/VW168o+aFWdD5z/FA5xM/AG4K/b+zcDN0yocyfwAeDDUxznUGAzYOeqejTJFsC/V9UrAZIcBexZVe9etkMSgOcn2bKqfpXkZU+hH+OaCxwN/N1I2WGt/BHgA1V1dZLnA/OTfKeqfrIa2iVJYjUuTSfZMMmlSa5uM8g3tvLnJbkwyXVtdnnoyG7vGam/fat/1Mjsdet2zAXt91at/NQ2W70iyc1J3jRyzPuBRSMzykOBsyc098vAoUleOEWXZgG3VdWjAFV1S1XdPcZQnN3OCXA4QyAuG6P1kpzS+ntNkn1b+fpJzmz9PAtYf2SfP0zywzZOX0+y4YTzfRfYPsmsVn8DYA5wXlXdVlVXt/bfCywCNh+jD5KklWRVBvH6I8uy5wIPAIdU1e7AvsBxGaaI+wO3VtUuVbUjcMnIMe5s9U8CPjjJOU4EvlJVOwNnACeMbJsF7MMw+z1mwn5nAoe1WexS4NYJ2+9jCOOplmrPBg5s/TsuyW5T1B31DeBP2usDgQtGtr0LoKp2Ygjp05KsB/wlsKT181PAHjAsqwMfA+a0cZoHvH/0ZFW1FDgH+LNWdBBwWQvex7RLB7sBPxqzH5KklWBVBvH9VbVr+zkECPDpJAsYZmmbAy8CFgJzkhyb5DVVdc/IMc5pv+cDsyc5x17A19rrrzIE7zLnVdWjbZn1RRP2uwR4HUPYnbWC9p8AHJnkBZNtrKpbgJcCHwEeBS5Nst8KjjXqLuDuJIcxzECXjGzbp/WDqroR+AWwHfBa4PRWvgBY0Oq/CtgBuDzJtcCRwNaTnHMuw3I0LF+WfkybRX8T+Kuq+rfJGp3k7UnmJZm3dMk9k1WRJD0Jq/Ia8URHAJsAe1TVw0kWA+tV1U+T7AG8HvhMkm9X1d+2fR5sv5eO2dYaef3gyOs8rlLVQ0nmM1wHfjnDzJQJdX6b5GvAO1d4sqoHgYuBi5P8BjgYuHSMdp4FfAE4akJ5nlh1+ekmKQvwnao6fJrzXQ7MSrIL8GqWhzJJnsMQwmdU1Tkr2J+qOhk4GWDdWdtO1hZJ0pOwOr++tBFwewvhfWkztySbMSy7ng58Dth9Bse8guWhcgTwzzPY9zjgw1X1r1PUOR54B5P8EZBk99Z2kqwF7Mwwgx3HucBngW9NKP8+Qz9Ish2wFXDThPId27kArgT2TvKStm2Dtt/jVFUxLKWfBlxUVQ+0+gH+AVhUVceP2XZJ0kq0OoP4DGDPJPMYQuXGVr4TcFVbWv0o8MkZHPO9wNvacvdbmfqa7uNU1Q1Vddo0de5kCM11J9m8KXBB++rRAoZPIJ845rnvrapjq+qhCZu+CKydZCHDrPmoNus+Cdiw9fNDwFXtOHcwzKrntm1XAtuv4LRzgV0Yro8vszfDuP3ByPX814/TB0nSypFhsiSNb91Z29asIz/fuxmStNosPuaAp7R/kvlVNen9H7yzliRJHRnEkiR1ZBBLktSRQSxJUkcGsSRJHRnEkiR1ZBBLktSRQSxJUkcGsSRJHRnEkiR1ZBBLktSRQSxJUkcGsSRJHT3hObvSdHbafCPmPcUnkUiSBs6IJUnqyCCWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKkjg1iSpI7W6d0APfMs/PU9zD76wt7NkPQMs/iYA3o34WnJGbEkSR0ZxJIkdWQQS5LUkUEsSVJHU35YK8kLp9peVXet3OZIkrRmme5T0/OBAjLJtgJevNJbJEnSGmTKIK6qbVZXQyRJWhONdY04g7ck+ev2fqskr1i1TZMk6dlv3A9rfRHYC/jz9v5e4AurpEWSJK1Bxr2z1iuravck1wBU1d1JnrsK2yVJ0hph3Bnxw0nWZviAFkk2AR5dZa2SJGkNMW4QnwCcC2ya5FPAPwOfXmWtkiRpDTHW0nRVnZFkPrAfw1eZDq6qRau0ZZIkrQFmckOP24G5o9u8oYckSU/NdEvT84F57fcdwE+Bf2mv50+1Y5KlSa4d+Zk9k4YlWZxk40nKD0py9EyONbLv7CSV5BMjZRsneTjJie393yRZkmTTkTr3TXydZK0kJyS5PsnCJD9Osk2SH7X+/jLJHaP9b336wYQ2XZvk+hn243tJ9hynTpJTk7xjwraDk1zUXn85ye0zbYMkaeWYMoirapuqejHwLeDAqtq4qn4XeANwzjTHvr+qdh35WbwyGlxV51fVMU/hEDcztH+ZNwM3TKhzJ/CBaY5zKLAZsHNV7QQcAvy2ql5ZVbsCHwfOmqT/z0+yJUCSlz2FfoxrLnDYhLLDWL66cSqw/2pohyRpEuN+WOs/VdVFy95U1cXA783kREk2THJpkqvbDPKNrfx5SS5Mcl2bXR46stt7Rupv3+ofNTJ73bodc0H7vVUrP7XNVq9IcnOSN40c835g0ciM8lDg7AnN/TJw6DT32p4F3FZVj7YxuaWq7h5jKM5u5wQ4nMcv96+X5JTW32uS7NvK109yZuvnWcD6I/v8YZIftnH6epINJ5zvu8D2SWa1+hsAc4DzWru/D3iJQZI6GTeI70zysba8unWSjwL/Os0+648sy54LPAAcUlW7A/sCxyUJw2zs1qrapap2BC4ZPW+rfxLwwUnOcSLwlaraGTiD4dPdy8wC9mGY/U6cQZ8JHJZkC2ApcOuE7fcxhPH7pujf2cCBrX/HJdltirqjvgH8SXt9IHDByLZ3AbQZ9uHAaUnWA/4SWNL6+SlgDxiW1YGPAXPaOM0D3j96sqpayrB68Wet6CDgsqq6d8z20s719iTzksxbuuSemewqSZrCuEF8OLAJw1eYzgM2bWVTGV2aPoTh09afTrKAYZa2OfAiYCEwJ8mxSV5TVaP/yi9b/p4PzJ7kHHsBX2uvv8oQvMucV1WPVtVP2nlGXQK8rvXhrBW0/wTgyCQvmGxjVd0CvBT4CMN3qi9Nst8KjjXqLuDuJIcBi4AlI9v2af2gqm4EfgFsB7wWOL2VLwAWtPqvAnYALk9yLXAksPUk5xxdnh5dlh5bVZ1cVXtW1Z5rb7DRTHeXJK3AuF9fugt4XwulR6vqvun2mcQRDGG+R1U9nGQxsF5V/TTJHsDrgc8k+XZV/W3b58H2e+mYba2R1w+OvH7c06Oq6qH2dawPAC9nmJkyoc5vk3wNeOcKT1b1IHAxcHGS3wAHA5eO0c6zGG4RetSE8smecvXY6SYpC/Cdqpruj6LLgVlJdgFezROvGUuSOhn3oQ87Zbi95ULghiTzk+w4w3NtBNzeQnhf2swtyWYMy66nA58Ddp/BMa9geagcwXCjkXEdB3y4qqZaYj8eeAeT/BGQZPfWdpKsBezMMIMdx7nAZxk+BDfq+wz9IMl2wFbATRPKd2znArgS2DvJS9q2Ddp+j1NVxbCUfhpwUVU9MGY7JUmr2LhL038HvL+qtq6qrRlmkifP8FxnAHsmmccQKje28p2Aq9rS6keBT87gmO8F3taWu9/K1Nd0H6eqbqiq06apcydDaK47yeZNgQva134WAI8wXLMe59z3VtWxVfXQhE1fBNZOspBh1nxUm3WfBGzY+vkh4Kp2nDsYZtVz27Yrge1XcNq5wC4M18cfk2Qu8EPgpUluSfJfxumDJGnlyDBZmqZScl1V7TJdmdYM687atmYd+fnezZD0DLP4mAN6N6GbJPOratL7P4z79KWbMzyL+Kvt/VuAn6+MxkmStCYbd2n6Lxg+aPVNhk8yb8wTP2gkSZJmaNwg/o/Alq3+cxge/vD9VdUoSZLWFOMuTZ/BcEON6/E5xJIkrTTjBvEdVXXB9NUkSdJMjBvE/zPJlxhuVvHYjTKqaroHP0iSpCmMG8RvY/h+6nNYvjRdTP8EJkmSNIVxg3iX9iACSZK0Eo37qekrk+ywSlsiSdIaaNwZ8T4MTyL6OcM14jDcwnjnqXeTJElTGTeI91+lrZAkaQ017mMQx32qkCRJmoFxZ8TSY3bafCPmrcE3b5eklWncD2tJkqRVwCCWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKkjg1iSpI7W6d0APfMs/PU9zD76wt7NkKSxLD7mgN5NmJIzYkmSOjKIJUnqyCCWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKkjg1iSpI4MYkmSOjKIJUnqaJUFcZKlSa4d+Zk9w/0XJ9l4kvKDkhz9JNs0O0kl+cRI2cZJHk5yYnv/N0mWJNl0pM59E18nWSvJCUmuT7IwyY+TbJPkR62/v0xyx2j/W59+MKFN1ya5fob9+F6SPcepk+TUJO+YsO3gJBe11/snuSnJz57suEqSnrxVOSO+v6p2HflZvDIOWlXnV9UxT+EQNwNvGHn/ZuCGCXXuBD4wzXEOBTYDdq6qnYBDgN9W1Suralfg48BZk/T/+Um2BEjysqfQj3HNBQ6bUHYYMDfJ2sAXgD8GdgAOT7LDamiTJKlZbUvTSTZMcmmSq9sM8o2t/HlJLkxyXZtdHjqy23tG6m/f6h81Mnvduh1zQfu9VSs/tc1Wr0hyc5I3jRzzfmDRyIzyUODsCc39MnBokhdO0aVZwG1V9ShAVd1SVXePMRRnt3MCHM4QlMvGaL0kp7T+XpNk31a+fpIzWz/PAtYf2ecPk/ywjdPXk2w44XzfBbZPMqvV3wCYA5wHvAL4WVXdXFUPAWcCbxyjD5KklWRVBvH6I8uy5wIPAIdU1e7AvsBxSQLsD9xaVbtU1Y7AJSPHuLPVPwn44CTnOBH4SlXtDJwBnDCybRawD8Psd+IM+kzgsCRbAEuBWydsv48hjN83Rf/OBg5s/TsuyW5T1B31DeBP2usDgQtGtr0LoM2wDwdOS7Ie8JfAktbPTwF7wLCsDnwMmNPGaR7w/tGTVdVS4Bzgz1rRQcBlVXUvsDnwq5Hqt7QySdJqsrqWpg8BAnw6yQKGWdrmwIuAhcCcJMcmeU1V3TNyjHPa7/nA7EnOsRfwtfb6qwzBu8x5VfVoVf2knWfUJcDrGMLurBW0/wTgyCQvmGxjVd0CvBT4CPAocGmS/VZwrFF3AXcnOQxYBCwZ2bZP6wdVdSPwC2A74LXA6a18AbCg1X8Vw5Ly5UmuBY4Etp7knKPL04exfBaeybo2WaOTvD3JvCTzli65Z7IqkqQnYZ3VeK4jgE2AParq4SSLgfWq6qdJ9gBeD3wmyber6m/bPg+230vHbOtoiDw48vpxgVNVDyWZz3Ad+OUMM1Mm1Pltkq8B71zhyaoeBC4GLk7yG+Bg4NIx2nkWw7XZoyaUTxaMj51ukrIA36mqw6c53+XArCS7AK9meSjfAmw5Um8Lnrg6MJy86mTgZIB1Z207aVhLkmZudX59aSPg9hbC+9Jmbkk2Y1h2PR34HLD7DI55BctD5Qjgn2ew73HAh6vqX6eoczzwDib5IyDJ7q3tJFkL2JlhBjuOc4HPAt+aUP59hn6QZDtgK+CmCeU7tnMBXAnsneQlbdsGbb/HqapiWEo/Dbioqh5om34MbNs+7f1chrE8f8w+SJJWgtUZxGcAeyaZxxAqN7bynYCr2tLqR4FPzuCY7wXe1pa738rU13Qfp6puqKrTpqlzJ0NorjvJ5k2BC9pXjxYAjzBcsx7n3PdW1bHtA1KjvgisnWQhw6z5qDbrPgnYsPXzQ8BV7Th3MMyq57ZtVwLbr+C0c4FdGK6PL2vHI8C7Gf4gWAScXVUTP0EuSVqFMkyWpPGtO2vbmnXk53s3Q5LGsviYA3o3gSTzq2rS+z94Zy1JkjoyiCVJ6sggliSpI4NYkqSODGJJkjoyiCVJ6sggliSpI4NYkqSODGJJkjoyiCVJ6sggliSpI4NYkqSODGJJkjp6wnN2penstPlGzHsaPM1Ekp4NnBFLktSRQSxJUkcGsSRJHRnEkiR1ZBBLktSRQSxJUkcGsSRJHRnEkiR1ZBBLktSRQSxJUkcGsSRJHRnEkiR1ZBBLktSRQSxJUkcGsSRJHRnEkiR1ZBBLktRRqqp3G/QMk+Re4Kbe7Xga2xi4s3cjnsYcn+k5RlN7Jo7P1lW1yWQb1lndLdGzwk1VtWfvRjxdJZnn+KyY4zM9x2hqz7bxcWlakqSODGJJkjoyiPVknNy7AU9zjs/UHJ/pOUZTe1aNjx/WkiSpI2fEkiR1ZBBLktSRQayxJdk/yU1Jfpbk6N7t6S3JlkkuS7IoyQ1J3tfKX5jkO0n+pf3+nd5t7SnJ2kmuSfKP7b3jMyLJf0jyjSQ3tv+W9nKMlkvy39v/X9cnmZtkvWfb+BjEGkuStYEvAH8M7AAcnmSHvq3q7hHgA1X1MuBVwLvamBwNXFpV2wKXtvdrsvcBi0beOz6P97+BS6pqe2AXhrFyjIAkmwPvBfasqh2BtYHDeJaNj0Gscb0C+FlV3VxVDwFnAm/s3Kauquq2qrq6vb6X4R/QzRnG5bRW7TTg4C4NfBpIsgVwAPClkWLHp0nyAuC1wD8AVNVDVfVbHKNR6wDrJ1kH2AC4lWfZ+BjEGtfmwK9G3t/SygQkmQ3sBvwIeFFV3QZDWAObdmxab58HPgQ8OlLm+Cz3YuAO4JS2fP+lJM/DMQKgqn4NfA74JXAbcE9VfZtn2fgYxBpXJinzu29Akg2BbwJ/VVX/1rs9TxdJ3gDcXlXze7flaWwdYHfgpKraDfh3nuHLrCtTu/b7RmAbYDPgeUne0rdVK59BrHHdAmw58n4LhiWiNVqS5zCE8BlVdU4r/k2SWW37LOD2Xu3rbG/goCSLGS5l/EGS03F8Rt0C3FJVP2rvv8EQzI7RYA7w86q6o6oeBs4BXs2zbHwMYo3rx8C2SbZJ8lyGD0yc37lNXSUJw7W9RVV1/Mim84Ej2+sjgf+7utv2dFBVH6mqLapqNsN/L/9UVW/B8XlMVf0/4FdJXtqK9gN+gmO0zC+BVyXZoP3/th/DZzGeVePjnbU0tiSvZ7jmtzbw5ar6VN8W9ZVkH+AHwEKWXwP9HwzXic8GtmL4h+TNVXVXl0Y+TST5feCDVfWGJL+L4/OYJLsyfJjtucDNwNsYJkmOEZDkfwGHMnxL4RrgvwIb8iwaH4NYkqSOXJqWJKkjg1iSpI4MYkmSOjKIJUnqyCCWJKkjg1iSRrQnIb04ybpJLmlP/XnnyPaTk+w28v7dSd7Wp7V6NjCIJT3jtAcArIrjvhxYu6puBv4ImA/sDLy9bd8FWKuqrhnZ7csMTwiSnhSDWNJKk+S8JPPb82PfPlK+f5Krk1yX5NJWtmGSU5IsTLIgyZ+28vtG9ntTklPb61OTHJ/kMuDYJK9IckV7WMIVy+5O1Z5//LmR474nyX5Jzh057uuSLLsl6agjWH6XpoeB9RnuB73MJ4CPj+5QVUuAxUle8WTHTWu2VfJXpaQ11l9U1V1J1gd+nOSbDH/w/z3w2qr6eZIXtrp/zfA0nZ3gsRv8T2c7YE5VLV32CMGqeiTJHODTwJ8yzF63AXZr214I3A18IckmVXUHw92rTpnk+HsDc9vr7wBvZbhT2meTHATMr6rJ7rE+D3gNcNUYfZAexyCWtDK9N8kh7fWWwLbAJsD3q+rnACO3IpzDcA9qWvndYxz/61W1tL3eCDgtybYMTwJ7zshx/09VPTJ6viRfBd6S5BRgL+A/T3L8WQyPJaTt/+dt3+cA32J4iMXxDLdW/EpVLbvf+u3A9mO0X3oCg1jSStHuJz0H2KuqliT5HrAewyM0J7uX7orKR8vWm7Dt30defwK4rKoOac+D/t40xz0FuAB4gCHQH5mkzv2TnBPgnQwPoN8LeIjh3sc/ZPmDT9Zr+0oz5jViSSvLRsDdLYS3B17Vyn8I/F6SbQBGlqa/Dbx72c4jS9O/SfKyJGsBy2bXKzrfr9vro0bKvw38t2Uf6Fp2vrakfCvwMeDUFRxzEfCS0YLWrjcAXwE2YHjAR/H4wN4OuH6KtkorZBBLWlkuAdZJsoBhtnolQLsm+3bgnCTXAWe1+p8Efqd9Peg6YN9WfjTwj8A/AbdNcb7PAp9JcjnDE8GW+RLDE3kWtOP++ci2M4BfVdVPVnDMC4Hfn1D2ceCTNTwh51vAngxP3Pr7kTp7A9+doq3SCvn0JUlrjCQnAtdU1T+sYPv6wGXA3iPXoqc75m7A+6vqrSuvpVqTGMSS1ghJ5jNcY35dVT04Rb0/AhZV1S/HPO7rgH+pqsUrpaFa4xjEkiR15DViSZI6MoglSerIIJYkqSODWJKkjgxiSZI6+v8EKQOo8X9IIQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualize our model results\n",
    "compare_results.set_index(\"model_name\")[\"model_acc\"].plot(kind=\"barh\")\n",
    "plt.xlabel(\"accuracy (%)\")\n",
    "plt.ylabel(\"model\");"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ba50d51-adb3-4e49-9b9a-85173e747352",
   "metadata": {},
   "source": [
    "## 9. Make and evaluate random predictions with best model\n",
    "\n",
    "Alright, we've compared our models to each other, let's further evaluate our best performing model, `model_2`.\n",
    "\n",
    "To do so, let's create a function `make_predictions()` where we can pass the model and some data for it to predict on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "d1d5d3e7-9601-4141-8bd7-9abbd016bf6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_predictions(model: torch.nn.Module, data: list, device: torch.device = device):\n",
    "    pred_probs = []\n",
    "    model.eval()\n",
    "    with torch.inference_mode():\n",
    "        for sample in data:\n",
    "            # Prepare sample\n",
    "            sample = torch.unsqueeze(sample, dim=0).to(device) # Add an extra dimension and send sample to device\n",
    "\n",
    "            # Forward pass (model outputs raw logit)\n",
    "            pred_logit = model(sample)\n",
    "\n",
    "            # Get prediction probability (logit -> prediction probability)\n",
    "            pred_prob = torch.softmax(pred_logit.squeeze(), dim=0)\n",
    "\n",
    "            # Get pred_prob off GPU for further calculations\n",
    "            pred_probs.append(pred_prob.cpu())\n",
    "            \n",
    "    # Stack the pred_probs to turn list into a tensor\n",
    "    return torch.stack(pred_probs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "420c7461-eaa9-4459-9e68-53574c758765",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test sample image shape: torch.Size([1, 28, 28])\n",
      "Test sample label: 5 (Sandal)\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "random.seed(42)\n",
    "test_samples = []\n",
    "test_labels = []\n",
    "for sample, label in random.sample(list(test_data), k=9):\n",
    "    test_samples.append(sample)\n",
    "    test_labels.append(label)\n",
    "\n",
    "# View the first test sample shape and label\n",
    "print(f\"Test sample image shape: {test_samples[0].shape}\\nTest sample label: {test_labels[0]} ({class_names[test_labels[0]]})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9f40dd9-7987-42a9-84cc-65dc912a6345",
   "metadata": {},
   "source": [
    "And now we can use our `make_predictions()` function to predict on `test_samples`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "79de2ac1-7d4b-4f81-ae8a-90099bca2a3d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[9.5605e-08, 1.9588e-08, 2.7693e-08, 2.5152e-08, 9.9790e-09, 9.9992e-01,\n",
       "         1.1720e-07, 8.6103e-06, 9.0887e-06, 5.7658e-05],\n",
       "        [8.2017e-02, 6.7443e-01, 8.9147e-04, 1.2562e-01, 2.5472e-02, 3.7753e-05,\n",
       "         9.0767e-02, 2.8317e-04, 3.7734e-04, 9.8759e-05]])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make predictions on test samples with model 2\n",
    "pred_probs= make_predictions(model=model_2, \n",
    "                             data=test_samples)\n",
    "\n",
    "# View first two prediction probabilities list\n",
    "pred_probs[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22d3c080-4eb6-4b5d-a5c4-2319e78228af",
   "metadata": {},
   "source": [
    "Excellent!\n",
    "\n",
    "And now we can go from prediction probabilities to prediction labels by taking the `torch.argmax()` of the output of the `torch.softmax()` activation function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "f9d97bcc-4310-4851-a1f8-6bcd757e9b26",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([5, 1, 7, 4, 3, 0, 4, 7, 1])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Turn the prediction probabilities into prediction labels by taking the argmax()\n",
    "pred_classes = pred_probs.argmax(dim=1)\n",
    "pred_classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "1141af97-0990-4920-83d4-c13cca3f9abc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([5, 1, 7, 4, 3, 0, 4, 7, 1], tensor([5, 1, 7, 4, 3, 0, 4, 7, 1]))"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Are our predictions in the same form as our test labels? \n",
    "test_labels, pred_classes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ea04387-c9ad-424f-8297-defd7b685683",
   "metadata": {},
   "source": [
    "Now our predicted classes are in the same format as our test labels, we can compare.\n",
    "\n",
    "Since we're dealing with image data, let's stay true to the data explorer's motto. \n",
    "\n",
    "\"Visualize, visualize, visualize!\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "679cb5f7-bb66-42dd-a4d6-400b27b7c019",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAg4AAAIFCAYAAAC6fHX6AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABca0lEQVR4nO3dd5xV1b3///dS6UMRQZoiYKWodDWxxx71atTYI9ZoitdfLDGxbLbeXEtMTLMkGjXJtcXE3ojfiBFFxUJRrIjYQJoUKSKS9ftj7wnHWZ89rDNzZpiB1/PxyCPymbPL2WftfdZZ533Wdt57AQAAxNhgbe8AAABoPug4AACAaHQcAABANDoOAAAgGh0HAAAQjY4DAACIRscBAABE26icB7vUrZL0ar7cG5JO8olfVpcNu9TdJulhn/i/1fKYbpL+KGlzSS0kzfCJP6gu26ux3j75tgfV5TEudTN84vvUqF0n6euSWkrqK+mt/E//U9tzLFl+lKR/+MTPrN6GpOE+8fPW/Iz+s46LJB0naZWkf0v6rk/8C7HL17LepySd5xP/Un0eszY1Ztt1qTtZ0n/n/xygrC2skvS4T/yFddlmpeWv1yif+BkltYskHZX/c3tlx0uSbvGJ/03EOg+T9LZP/Osl24huEy51L0hqJamzpDaSPs7/dFjpfq4v1pXrbbn7Uca6Rivbx9tKavtLuir/51bK2tBySVN84r8Tsc7Bknr6xD9aso0lPvHXlLFfO0v6tbK23ErS3T7xo2OXj9zGDJX5/lBJZXUcJC33iR8sSS51t0s6U9Ivq//oUrehT/yqyu2eLpP0hE/8r/P171DBdVeUT/z3pa90OAaX/j3i2IyS9JqkmXXZvkvdLpIOljTUJ36FS10XZZ0YZBqt7frE3yrp1ny9MyTtVfMEb4BzpZBL3UY+8V+u6XE+8T+T9LN8mSVGG3aSnE/8vwtWcZikhyW9Xpf99InfKd/OKGUXxR/U2H7U86iUxnyNCnC9zUW0PUmST/wYSWPyZZ6S0XFdw3EbLGm4pEfrsbt/kvRtn/jJLnUbStq2HuuqqEqdQ+V2HEqNk7SDS92ekhJJsyQNdqnbXtKVkvZU1tu6zif+9/kL/1tJe0t6T5KL2EYPSf+o/odP/BRJcqmrkvSApI2V9Ywv9ol/IH/TfkzSM5K+pqy3+V8+8ctd6oZJukXSsvzvytfVR9JfJLXLSz/wiR9f3qGwGcfmIJWMYrjUnSepSlmHYbik213qlkvaJV/FD13qDsmf41E+8W/Wsrkekub5xK+QpNI3Kpe6SyUdouxT3HhlIxE+P7FekLSXpE6STvWJH+dS10bZG98AZZ902pSs6wZJI/La33zikzodnLWrMdpuwKVuibIL//6SznWpGynplPzPN/vE/6rmSFd1G/GJH+1Sd7ayN48vJb3uE3+MS127fN+2V3Y+j87PhVGSvimptbK2vXcd97mPsnNqrLJ2eZhL3VSf+Kr870cq67D+QdKhkvZwqbtY0hH5Ko5yqbteJe2rzO2PltRTUh9J81zqfqLsPO4qaa6kk33iP6j5STbv+FS51PWQdLekDsqOz1l5G99PUqrsdX43X8+SvKN3i6T9JP1O0l3l7G8DWpvX2z0ljZY0T9IgSS9LOiG/hgxT1qar8r+P8omf5VJ3uqQzlH14mSbpxJqjJS51lysb3ThF0rmSvp0/h/t84hOr7Ul6P+J5BGq+ri51ZyrvVOQfsl6StI2yzlMbl7pdJV2RLz4gv1b2lvSriNG3TZW9Pso7KNUjcKPzdfSruS6XuhMkna3seL0g6Xs+8avWdL3Nr9X3Sfq7pDvUgNeCUnXKOLjUbSTpQK0eyhwp6SKf+AGSTpW0yCd+hLInfLpLXV9JhyvreW0v6XRlb+zV67vMpe5QY1PXSfqjS91Yl7qLXOp65vXPJR3uEz9U2ZveL/ITRZK2VnbyDJS0UKsvYLdKOtsnfhd91RxJ++brOlrSGodky1R6bEz5xe4lScf7xA/2iV+e/2levl83SDpPklzqhrvU3Wys5h+SNnepe9ul7nqXuj1K/vY7n/gR+ZtRG2UX+mob+cSPlHSOsguSJJ0laZlP/A7KPoEOK3n8RT7xwyXtoOxNosl8KonRiG3X0k7Sa/kn6+WSTpa0k6Sd820NWcPyF0oakr8uZ+a1iyQ9me/zXpJ+nncmpOxie5JPfH0vFNtK+rNP/BCfePPCnXe2H5R0ft6G383/FLQvl7qeLnXlfKIbpuwDwHHK3sz/nB+D27Xm8/U4SWPyT+47SpqUv1FcLGmf/Px6SdKPSpb53Cd+V5/4JtFpaALXW0kaouw1HKDsje/rLnUtlL1RHekTX/3B7Gf54+/Nrzk7KvvwcWqN53S1sjfYkyXto+y6PVLZJ/5hLnW75w9dY9srQ62vq0/8F5IuVfbVwmCf+LvzP22nrLM/UlKSP2+51D1a4xhVu1bSWy5197nUfdelrnXJ34J1udT1V/be8/W8na6SdHz++Nqut1WSHpJ0h0/8TWqca4Gk8kcc2rjUTcr/e5yy78O+JmmCT/x7eX0/ZT3jI/N/d1TWKHaXdGfeA5vpUvdk9Up94i+1NuYTP8alrp+kA5SdOBNd6gYp6xD8b964/i2pl6Ru+WLv+cRX7+PLkvq41HWU1Mkn/l95/S/5+qTs0/zvXPbd1iplvc5KKj025bo3//+XJX1LkvJht9NqPjD/tDRM0m7KGs3dLnUX+uz7v71c6i6Q1FbZ98dTlTW4mtvok//37sovyD7xU1zqppRs6tsudWcoazs9lF1ISv/eVDVq2y2wStknA0naVdknq6WS5FJ3r7LX7sFalp+ibFTqfkn3l+zzofnIhJR9quid//cTPvGflrF/Rd73iX++jssG7ctnOZ5yvjt/sKQzvYvyc0HZeXz1GpZ9UdIt+cX+fp/4SXmneoCkZ13qpOxT3nMly9wdrmataCrXW+Xb/EiS8n3qo+w6PEjSE/lx3FD5J21Jg1zq/kfZSFOV8q8PcpdIesEn/ox8ffvlz2Ni/veq/Dl8oPq1vZrq+ro+ko/krnCpm6PsveYjX5D/8Im/zGVfLe2nrON6rLIRoaJ1fUNZ5/jF/Di2UfaBVqr9evuApKt94m/P/90Y1wJJ9cg4VMuf6NLSkqQf5t81lT7uIEll31Erf7J3SLrDpe5hZSdEe2VDlcN84lfmw1DVvboVJYuvUvYiuFq2/f9Jmq3s08gGykYzKqn02Hypr47ytFbtqp/LKkW8VvlF4ilJT7nUvSrpJJe6uyRdr+w74w/z4bLS7RZtIzhe+SeZ8ySN8IlfkA8Pr+k5NBWN3nYNn/vV360WDR3X1ka+qaz9HyrpEpe6gfl6jvCJf6vkcXKp20lffW71UXM9pceiom04cvvWvvznuOWjjy0lySf+6fwDxjcl/cWl7ueSFii7kB5bh+01pqZyvZ2v8Lq6Ub7tqcYoriTdpizUOjkfKt+z5G8vKhtV6Jxvz0m6wif+9zWeQx9V9rUouhbHtmEp/lr8rqQbXOpukjTXpW6TWtblJP3JJ/4npeuIuN4+K+lAl7o7fOK9GudaIKlhfo45RtJZJcM52+TDJU9LOsalbkOXfe+415pW5FK3t0td2/y/20vaUlkvtKOkOXmnYS9JW9S2Hp/4hZIW5d9bSauHgZSva5bPQjcnKus1N5TZkjZ1qdvEpa6VvvqVwWfKOkR14lK3rUvd1iWlwcq+D6xuaPNclg05suayhqeVH6P8E0f18FgHZQ1wkcsS2AfaizdbFWu7EZ5Wlhdom2/jcGWfKs024lK3gaTNfeLHSrpAX/0k98Pqr+oivu6ohNkudf3zfTq8pF6vNhxpvKRj8v8+XqvzSjO0+iu1/1I2kiiXui2UXStuUvaJfaik55UNtW+VP6atS12lRxobS2Ncb4u8Jamry4LZyofdB+Z/ay9pVr5fx9dY7nFluYxH8u2MkXRKfn2SS10vl7pNY558PczQ6vZSek2sdxt2qfum++pX56uUjc4U+aekI6ufs0td57zdrul6e6myTt31+b8b7VrQEB2Hm5WFQV5xqXtN0u+V9aruk/SOsu/pbpBU/bVBbd+5DZP0Uj5U/pyyANmLyr7bHO5S95KyRllbaLDayZKuc6l7Ttn3y9WuV/bJ/HllX1M02KcNn/iVysI3LyhLn5fu922SbnSpm+SywIvJFWccqiT9yaXu9fx4DVAWjlko6SZlx/1+Zb39NblBUlW+ngskTcj3f7Ky4cSpyr7PfDZiXc1JJdturXziX1H2mk9Q1h5u9omfWEsb2VDS/+UjSRMlXZu/tpcre5Ocku/z5eXuSx1cmO/bk1o9NC1lQcLzXeomutRtWbSwKz/jUOpsSSfnbfNErf7Z603KvgOeoCw3Un0e76ks1zBRWd7p1z7xc5X9iunOfD3PK/vuuTlqjOutKc8EHCnpKpe6yZImaXWW4hJlbfgJGddnn/h7lL1mDyrrMN8h6bm8ff9NDd8BvUZZh2u8pC4l9bHKwpCTXOqOrm0FrjjjcKKyjMMkZV+nHe9r+fWLz36+fLGkf+TH/glJPSKvt+dIau2yzEijXQuc95UYgV2/OGMeB6A5ccY8DkBz4ox5HNA4mDkSAABEq888DuuzX63tHQDq6TbV/r0r0NQ9JdrwWsFXFQAAIBpfVQAAgGi1flXhnGM4AnXmva/T1MyV1FzbsHPhoSsaHayqqgpqBx0Uzk0ze/bsoLZqVRj23mCD8PPEv/8d3iJgww3tXy5b6+zWrVtQe/rpp4Pa3LlzzXWuLU2hDUvNtx2jaah0O2bEAQAARKPjAAAAotFxAAAA0fg5JtAEWTkDKzsgSZtuGs7Ou8ceewS1994L77Vm5SasPIP1uKLMxfLly4OatY9W5qKpZRwAhBhxAAAA0eg4AACAaHQcAABANDoOAAAgGuFIoAkqCkJapk+fHtSeeOKJoPbJJ58EtVatWgU1a/Ipq9axY0dzf6zHrly5MqhZIUwATR8jDgAAIBodBwAAEI2OAwAAiEbHAQAARCMcCaxlgwYNCmoXXHBBUPvwww/N5Xv06BHUunfvHtTmz58f1Fq3bh3UrLteWoHHW2+91dwfixWEbNeuXfTyAJoORhwAAEA0Og4AACAaHQcAABCNjgMAAIhGxwEAAETjVxXAWvatb30rqB155JFBbdGiRdHrrKqqCmrLli0LatYvKBYvXhzU+vbtG9Tuvfdec9uzZs0Kau3btw9qG23E5QdojhhxAAAA0eg4AACAaHQcAABANDoOAAAgGumkSNaUu1bNmlq3HBtsEPblWrZsGdQ+//zzem2npt13392sP/300xXdDkKdO3cOatb00AsXLoxe54IFC4LaqlWrgpr3Pqpmbbtbt27mtq2psa1t1/dcAbB2MOIAAACi0XEAAADR6DgAAIBodBwAAEA0wpEGK/QYGyKzZsOzgmFFs+atXLkyqNUnCHnggQcGtdNPPz2ojRgxwlz+1FNPDWr/+Mc/glqLFi3qsHeQpB49ekQ9rihM2LZt26C2fPnyoGa1OautW8ta27Zmg5Ts9mptp9IBX6ChtWrVKqitWLEialnrHKgva+bXL7/8suLbqYkRBwAAEI2OAwAAiEbHAQAARKPjAAAAoq2T4Uhr9sWiYJkVGIsNl3To0CGoWbcktlghSMkOnFm3NLb2++67745anzULoDXToCT94he/CGq77LJLUFuyZIm5PNYstr1aIVvJbktWYPKLL76I2h8rtGjNXmoFxcqxdOnSei0PlKuc9wbLQw89FNRuueWWoHbXXXcFNStMX1+NEYS0MOIAAACi0XEAAADR6DgAAIBodBwAAEC0dTIcaYVdimZqjA2XXHXVVUHtkEMOCWrHHntsUJs8eXJQO/nkk83tnHbaaUHNmtXx17/+dVCzQo/Tpk0LatZMhUWzAD766KNBjSBkZVmhKatdFoWrrNBj69atg5o1y5x1rrRp0yaoWcFMa4bJonVa2162bJm5PFCu2FkZywlCPvDAA0Fthx12CGrnnHNOUJswYUJQW7RoUVBr165dUBs4cKC5P4cddlhQs57PWWedZS5fSYw4AACAaHQcAABANDoOAAAgGh0HAAAQbZ0MR1rKmWHr7LPPDmojR44Mam+88UZQe+6554LatddeG9TOP/98c9vW8lZAcfPNNw9qr7/+elAbOnRoULNmTzv00EPN/Zk6dapZR+VYIUErzPvpp5+ay/fq1SuoWbM6WoHJ+fPnR+2PNctj7EyUkh3s/Oyzz6KXB2pTn1kZi5a9/vrrg9qsWbOC2j777BPUnn322aBmzfBqzdJa9F5lndP333+/+diGxogDAACIRscBAABEo+MAAACi0XEAAADR1slwZDm3Tj388MOD2nHHHRfUZsyYEdS6du0a1N55552g9p3vfCeoXX755eb+nH766UHtgw8+CGpWONJ63B133BHUrrjiCnPbWDtmzpwZ1KwglNWuJXvWzzFjxgS1XXfdNahZt8u2Znm0ZuYruoW8tZ/WOq2ZToFK2WqrrYKaFT63boEt2e27f//+Qc1q2/PmzQtq1uyrK1asiKoVLd+xY0fzsQ2NEQcAABCNjgMAAIhGxwEAAESj4wAAAKLRcQAAANGa/a8qrFS4NRWulbCV7OmgX3zxxaDWu3fvqP156623opb9xje+YS5/9913BzVryumxY8dG7Q+aPmsaW+tXFVZbL6pPnz49qG299dZBrU+fPkEtNhFuTU0t2Slz61dNRctj3Wb9QkeKnzbaal+vvPJKUGvRokVQs9r2//t//8/czje/+c2gduCBBwY169dsFmu/rVqbNm3M5a0pqzfeeOOobVcaIw4AACAaHQcAABCNjgMAAIhGxwEAAERrEuFIKyBiTVtrBbSsIKRl2rRpZv3GG28MameccUZQW7RoUVDr3r17UPvwww+DmnXP9SFDhpj787WvfS2odenSJajFhiMHDx4c1Pr27RvUrOmIJWngwIFBrV+/fkHNOmaIY7XNjTYKT80vv/zSXN4KlS1dujSoLVmyJKhZ517s1LjLly8398cKpVnrtGpoHqzrs9UOY2tFDjnkkKBmTaNvnUPWlNH33ntvUDvvvPPMbVvBQ+v6bj0fK8hovQ9Y50rR7RGs83fTTTc1H9vQGHEAAADR6DgAAIBodBwAAEA0Og4AACBao4cjrVnD6hOcateuXVCzgmFFrrzyyqBmhWIuuOCCoPbuu+8GtYMPPjioWcHKd955x9wfKwB3wgknBLW99947qFlBmdatWwc1K1BqzTRYtD9WoMcKXCKO1Y6s0GLbtm3N5a0g5cKFC4Pa7Nmzg5o166S1bStoZgUmJXvWSysshvqxrqXWa1c0U6MVwrOCftY66/N6WqFyyQ6qW9cVa5bI999/P6i9/PLLQW3AgAFBrej9wjoW1vuSdY203pc6duwY1KxZJ63zWbJfh5122imo7bfffubylcSIAwAAiEbHAQAARKPjAAAAotFxAAAA0WoNR8bO6GgpZ5Y7KzSy7777BjUrKDNq1Kig9uyzzwa1cmY2/PGPfxzUrFnyLrnkkqA2ceLEoGaFXYqCblaw7KWXXopafv78+UHNCv6UM2OfFYTs2bNnULNmVEMcKxxpnT9FM8pZdWuWus8++yyoWcE56xy32n/ROR47qyDqxzqmRa9JfRS1u5qsUJ51fT722GPN5Z977rmgNnPmzKBmXQ+7desW1E488cSgtuWWWwa1BQsWmPvz6aefBjXr+mwFIV999dWg9vrrrwe1I444IqhZgXZJ6tSpU1Czrs/HHHOMuXwlMeIAAACi0XEAAADR6DgAAIBodBwAAEC0WsORjXUrXGv2RivU99577wW1J598Mqj94Ac/CGr1ve3z6NGjg5o1w+TJJ58c1N54442gVjTz2ieffBLUrPCNNWufFXSzwjyxAVdJWrZsWVCzZhtkZsC6s84pK7BlhZWl+HCk9RpZ7chq1+PHjw9qVthSktq3b2/W0fCsa0VR2M6a0dYKV1ZVVQW1G264IahZM9yOHTs2qF199dXm/ljbtmr9+vULalaY3prtdNasWUHNmklXso+ldY21aptssklQ23nnnYPatttuG9SKrqXWtdg6z0eMGGEuX0mMOAAAgGh0HAAAQDQ6DgAAIBodBwAAEK3WcORmm20W1Kxgyttvvx3UrFv4SvZsZ1tttVVtu/Efu+22W1CLDWL17t3brFu3NbVYAc5TTjklqE2aNCmoWeG3Xr16mdsZNmxYULOO5dy5c4OaFZSxAnXW/pQzs1/s7WZRd9Ztsa1Z4iQ7xGW1DysEZrUPaztWMLlolkIrkNsQMxqu7/70pz8FNeuW1UW3jbZe+y222CKoWTMWWrM8/uhHPwpqVlsout537do1qFmzBXfo0CGoWbPmWqFh6zlb+1i0vFWzZlUdOnRoULPeq6zzvJxZl61gZmNcixlxAAAA0eg4AACAaHQcAABANDoOAAAgWq3hyG9961tB7dRTTw1q1sxkRaHDhx9+OKhZQcH+/fsHtY8++iioWWEXaxvWrF2SdPHFFwe1otu+1vTaa68FNWuWtcGDBwe16dOnm+s8/vjjg9q//vWvoGaFk6xZAK2gjRWoqa/YW+8ijjWrn3X7ecl+ja3bEVusW77HzlBXFOKy1mktj3iHH354ULOum1bYrm3btuY6rbCdNbPiyy+/HNSsoO12220X1KzAo1WT7PCgNUtt0YylMeobHIwNTFqB1Nhb2BfN9Gnte6WPTyxGHAAAQDQ6DgAAIBodBwAAEI2OAwAAiEbHAQAARKv1VxXW9MlW4tqqbbrppuY6//u//zuoffrpp0Ft4sSJQS02lWr9IuPGG28098da3ppC20qFW1OxWonYgw8+OKg98sgj5v7E6tKlS1CzfslipW4tRVNOW3WrxpTClWUdz6JEuJXqttL11i81rPZvsX4pUc6vKmgf9bNgwYKgZl1/rDS/NSWyZL9OVnvYfPPNg1qbNm2CmvUad+7cOahtvPHG5v5Y11jr+mXVrOdinRfWtaucX4RZy1u/MFm5cmVQs85f61eBRWLPc+v9a9ddd43eTgxGHAAAQDQ6DgAAIBodBwAAEI2OAwAAiFZrONIKxVhTKi9ZsiSoFQVyrLoVbLGmT7bum249ztrG+++/b+6PFWaMDfn885//DGpnn322uZ36sJ63FbSxjqMVKLWOmbWsZIelrOND+K2yrPBZESs0ZQUhralorXZkrc8KpFnnvWSfU0XXA8R56qmngtpLL70U1KxgeIcOHcx1xl63reuFFWi3rhVWSNCa1lqyr0FWu4udWr8o8F1TUejYCi7GBn9jw82x559kH8sWLVoEtYa4pUBNjDgAAIBodBwAAEA0Og4AACAaHQcAABCt1nDkmDFjgtqFF14Y1Hr06BHU5s6da67TChlaIR0rFGMFTqxAjhUOsbYr2UEU67HdunULarGzcVkBlqIwoRXoscJA1syaVnDVmtXMmtXTCjtJdiDnww8/DGqxMxAijjUDYFFg0mrvVuhq3rx5Qc06T61zL3Z2PElq27ZtULNmBUT9nHnmmVGPO/roo836qaeeGtQGDRoU1Kx2Z12zFy9eHNSsgKH1OMm+TlohXyt8a9Viw41Fszda50F9gofW+qxtF4U1rfcGKzj/4IMP1mHvysPVHgAARKPjAAAAotFxAAAA0eg4AACAaLWGI60w1fbbbx+14mOOOcasDx8+PKjtv//+Qc2awWzEiBFBzQp3WWEe6/ajRfWf/exnQc26DbYV3LECPlaIrGimRis0aYVqdt9996B2zz33BLWjjjoqqD300ENB7aCDDjL354033ghqVoCqXbt25vKoG+vcK7pVfWxgywrAWuFgK7BlzTpZFPC1liccufbcfffdZdVr2nnnnYOaNUNl7969g5o1a2VRyNcK1cbOHGmFNa3HWaHjorZphThjg8hWWNzatrWPRTNexs4G+8wzzwS1008/3VxnXTHiAAAAotFxAAAA0eg4AACAaHQcAABAtFrDkVbAo+iWnzXddddd0fXzzjsvap3WDJULFiwIaj179gxq06dPj9pGfRXNpldT7C1fJXvWvr322iuoWbfeHTZsWFB7+eWXg9qWW25pbrt9+/ZBzQqUzpgxw1wedfPmm28GNStYLBXPfFfTsmXL6rw/H3/8cVArmuHOYoXA0Dw8//zzUTWsPxhxAAAA0eg4AACAaHQcAABANDoOAAAgWq3hSCsIac2cZdViQ5TlmDVrVtTj6huEtEKhVq1o5ryaYmcbK4cVhLRYQUjLu+++W4+9QaVZs7+dcMIJ5mNff/31qHVaIVuL1dbfeuutoFY0++kWW2wR1JhZFFh3MOIAAACi0XEAAADR6DgAAIBodBwAAEC0WsORFmvGw3JmQWwOrGBnfcKe69rxQcOzZo4saoOx7WvChAlBzbqlsHVreOuW3J988om5nVtuuSWovf322zG7CKAZYMQBAABEo+MAAACi0XEAAADR6DgAAIBodBwAAEC0sn9VAaDhffTRR0Ft6dKl5mPr84ufZcuWBbWqqqqgFjtdtSRNmTKlzvsDoOljxAEAAESj4wAAAKLRcQAAANHoOAAAgGiEI4FmYuXKlWa9Q4cOdV6nFaxs3759UFu+fHn0OjfaKLysfPnll+XtGIAmixEHAAAQjY4DAACIRscBAABEo+MAAACiEY4EmolHH33UrPft27fO67zrrruC2nbbbRfUyglHrlq1qs77A6DpY8QBAABEo+MAAACi0XEAAADR6DgAAIBoznu/tvcBAAA0E4w4AACAaHQcAABANDoOAAAgGh0HAAAQjY4DAACIRscBAABEo+MAAACi0XEAAADR6DgAAIBodBwAAEA0Og4AACAaHQcAABCNjgMAAIhGxwEAAESj4wAAAKJt1FArdqlbJenVfBtvSDrJJ35ZHdd1m6SHfeL/tobHHSjpckntJLl8mfPqsL09JX3hEz/e+FsfSbf5xO9Zo/6CpFaSOktqI+nj/E+H+cTPiNjmT33i/7dkGw/7xA8qc79HSrpGUjdJXtIzks4u97i71A2W1NMn/tFylmvOGru9utSNkvRzSR9JqpI0XVJqtbmG4FL3lKRRpW3Tpe4iSUfl/9xe2fGQpFt84n8Tsc7DJL3tE/96yTbO84l/qYz9miHps/yfG0q6V9LlPvErYtfR1DVmW3OpO1nSf+f/HCDpLUmrJD3uE3/hGtY9WtISn/hrjL+N94n/mlHvJOk4n/jra9Qfl3Sqsvb1h7o+3xrr3FNZGx5VUttE0j/zf3ZX9lzn5v8e6RP/xRrW+ZX9z7dxnk/8wZH7tM4e71INOeKw3Cd+cP7m94WkM0v/6FK3YSU35lI3SNLvJJ3gE99f0iBlF+O62FNS8CLVxid+J5/4wZIulXR3/twHV1+YXerW1En7aR328z9c6rpJukfSj33it5XUX9LjktrXYXWDJR1Un/1phhq1vebu9okf4hO/taQrJd3rUte/5oMi2k5F+MT/rLrdavXxGFzdaXCpcy51tV0zDlN2sayvvXzit5c0UlI/SX+o+YAGej0aS6O1NZ/4W0te05nKju3gNb2JRazXehPbUFInSd+rUW8jqbNP/MeSzpHUtj7bXsN+zS95vjdKurakHX+R709t51Mn1dj/Mre/XhzvRrkgSRonaYe895ZImiVpsEvd9soumHsq+7R+nU/8713qnKTfStpb0nvKRg/W5AJJP/OJf1OSfOK/lFTda9xC0i2SuirrfZ7sE/+BS90hki6W1FLSfEnHKxstOFPSKpe6EyT90Cd+XF2edN6D7Cmpj6R5LnX/kDTcJ/4H+d8fVjZCcICkNi51kyRNlXSRpA1d6m5S1oH5WNJ/+cQvr2Vz35f0J5/45/Ln7yX9Ld9O5/z595O0TNIZPvFT8hGKX+XPebmkk5Ud78vy/dlV0hU+8XfX5fk3Y43RXr/CJ36sS90fJJ0h6f/LP62Pl/R1SQ/m//6lstGJeco+ac1yqTtbWXv9UtLrPvHHuNTtIenX1auWtLtP/Geqg3z06zFJYyXtIukwl7qpPvFV+d+PlHSwsjf3QyXt4VJ3saQj8lUc5VJ3vbIL3KnlnEs+8Utc6s6U9GHehndQ3OvRQ9Ldkjoou8adpexY/lHS8PyY3OITf21djkmFNXpbs1jtKP/TgLzt9Zb0q5JO5BKf+Kqa+y1piqQt82vZEz7x5+fP4al8Gz0ljXWpm+cTv5dL3bHKPjQ5SY/4xP+4ev2Sfi9pL0kLJB3jE189clCX53ebpE8lDZH0ikvdZyr5dO9S95qydnxl6f5LekRSlUvd35R9GH1Z2YdTX9d9ybfXrI93g2cc8t7dgVo97DlS0kU+8QOUDaUs8okfIWmEpNNd6vpKOlzStsqGS09Xyad/l7rLXOoONTZV/aJafifpzz7xO0i6XVL1sOszknb2iR8i6S5JF+QjBKU91Tp1GkoMU/amf1zRA/LeaPWnkOPz8tbKLhYDJS1UfiF2qTszv5jWVNvzTyVNzJ//TyX9Oa+/qexNZYiykZL/zXvlpaMm61WnoRHbq+UVSduV/LuTT/weytrrbyUd6RM/TFkn8Gf5Yy6UNCR/bavbxXmSvp9/6tlNWaewPrZVdv4M8Yl/33pA/hXLg5LOz9vNu/mfNvKJH6nsk08iSS51PV3qor4G84lfrOwNcuu8FPN6HCdpTP78d5Q0SdlFtpdP/KB8NOPWMp5/g1jLba0mqx1JWXvcP9+3xKWuhbFs6X5fKOndvA2cn//9QGVD9b/R6k/he7nU9ZR0lbJO0GBJI/Kvu6Ts6+ZXfOKHSvqX8rZTT9tI2scn/txaHmPt/xBl7XeAsg9fX5fW7+PdkCMO1Z+gpaxX/UdljXyCT/x7eX0/Zb3tI/N/d1R2gdhd0p0+8askzXSpe7J6pT7xl9ZhX3aR9K38v/8i6er8vzeTdHf+CaWlsgtUpT24hpGCIu/5xE/K//tlZaMW8om/sQ7r2lV5x8Mn/kmXuk1c6joq+0T2J5e6rZV9CrMa6fqiKbTXmp8eqztt2yrrGD7hUidl3/3Pyv82RdLtLnX3S7o/rz0r6ZcudbdLutcn/qMy9sHyvk/883Vc9t78/0vb8EyV91VY6XGJeT1elHRLftG93yd+kkvddEn9XOp+q+xT5D/q8mQqpCm0tZqsdiRln0pXSFrhUjdHWX6qZnsq3W/L15V1ZmsaIemp6k+2eXvdPd/+v7W6/f+fVrej+rgnP27lmlB9DuWvWx9Jz6zPx7shOw7L8x7/f+QXvaWlJWVfBYyp8biDlL2RlWOqsk/3kyMeW73u30r6pU/8g/kQ0Ogytxmj9Pl+qa+O8rSuZbnSMNgqZV8n1Kb6+T9g/M0azvTKgqRjfeIPz4ekn1rDNtZljd1eLUOUheWqVW/bSZrqE7+Lscw3lZ38h0q6xKVuoE/8lS51jyh7c37epW6f6q/w6mhpjX+XPtfa2rC0uh2vUh2uNy517ZVdqN9WNnqwxtcjX253ZcfmLy51P/eJ/7NL3Y7KPs19X9K3JZ1S7v5UyFpvay51typrbzN94g+S0Y7yh9a8DlmvYc32UbqdfpI+9HYosZyvWSpxflXqWlyXdrxOHe+1/XPMMZLOqh6OcanbxqWunaSnJR3jUrdhPhqwV8S6fi7ppy512+Tr2sCl7kf538ZLqv4O6XhlX1FIWS+++tcPJ5Ws6zPVLVS4JjOUfX+5gUvd5sqGnKqtLBiWivU7SSe51O1UXXCpO8Glrruy43l8XttT0rx8CLj0+Y8qWVdDPf/mrpLt9Stclks4Q9JNxp/fktTVpW6X/LEtXOoGuiyouLlP/FhlGZ9Oyr6P3dIn/lWf+KskvaSvfv1RCbNd6vrn2z+8pF7RduNSV6Usp3S/T/wC4yHm6+GyTNMcn/iblH2aH+pS10XSBj7xf5d0iaShldrPBtJgbU2SfOJPzoe3DypqR3Xc75pt4EBlIW3r7y8oy8R0cVnQ71hlw+RS9t5UPdpynFZfsytlhvI24FI3VFLfgv2viHXteK/tjsPNkl5XFlZ5TVk4YyNJ90l6R9l3fzdo9ZMr/F7JJ36Ksu+h7nSpe0PSa5J65H8+W9LJLnVTJJ2o1T+XGS3pHpe6ccoCZ9UeknS4S90kl7rdKvNUJWVDyO/lz+saZd9pV/uDpCn58FGhooyDT/xsZZ2ja1zq3sqPwW6SFit7nsPz53+lVneSrpZ0hUvds8qGv6uNVRbSmeRSd3T5T3OdVbH2mjs6P8ZvK8ueHOET/0bNB+WfHo6UdJVL3WRl39l/Tdlr9n8uda9Kmqgsl7NQ0jkuda/lj12uLNxYSRdKeljSk1r9lYmU5YTOd6mb6FK3ZdHCbs0Zh7H58Z0g6QNJ3y14XNHrsaekSS51E5V9RfdrSb2UBcYmSbpN0k/W8BzXtkq3tdoUtaOy+cTPl/Rs3v5+riz4XfpG9gdJj7nUjfWJn6XsdRirbKT4FZ/46hHTpZIGutS9rOw7+cvqsj+1+Lukznl7OEvZiJa1/4XW5+PtfP3CoesdVzCPA9CcOGMeB6CSXOpaSXrWJ354HZZd4vNf7tTymD1VYx6H9VlDH+9SjfVzTADAeiQP+ZX9Joa6aczjvba/qmiOFiob6gSas9uUtWWgyYn89DtDX/1FAuqonNEGia8qAABAGRhxAAAA0WrNODjnmvxwxIYbhtO6r1oVP8fHDTfcENQGDhwY1JYtC+8RYtUWL14c1BYuXBjUnnjiCXN/HnnkEbNek3PhT3Kb2uiR974i0+HWR3Now5YNNgj79P/+97/rtc5Ro0YFtZUrVwa1zz//PKj16dMnqP3iF7+I3rb1fCz1fY6V1hTasNR823E5jjjiiKC2zTbbBLXly+Pm09tll3Dak7Fjxwa1G2+sy5x6zUul2zEjDgAAIBodBwAAEI2OAwAAiEbHAQAARKv155jrUiDn+uuvN+vf/W44m60VerRCW61bh/dFsZatqgp/ImuFOiXprLPOCmq///3vgxrhyDjrUhsuxznnnBPU9thjj6A2c+bMoDZkyJCg1qJFeBuVX/7yl+a277zzzog9bB6aQhuWmkc73mGHHYLaYYcdFtT23Xdfc/kvv/wyqHXo0CGozZgxI6httdVWQc26Pr/3XnhTSWsbkh1U/9e//hXUnnmm0rfRqDzCkQAAYK2h4wAAAKLRcQAAANHoOAAAgGjrTTjy8ccfN+s777xzUHv33XeDWu/evYOaFXC0ZuKbPXt2UOvatau5P++8805Q23333c3H1hQbmGysYGVTCJY11zZshXatmfUkO2z25z//OahZwTBrNkkrMGmtb//99zf3Z6+99gpqv/3tb4Pab37zm6A2d+5cc51rS1Now1L92rF1vhfVrRD4lltuGdSSJAlqLVu2jKotXbrU3B/rGmRdY61rZ5cuXYLahAkTzO3U1L59e7NuBYKt4/PZZ58FtdNPPz1q242FcCQAAFhr6DgAAIBodBwAAEA0Og4AACBaswpHxt5Ce9CgQUHt6aefNtf5xRdfBLU5c+YENSuQs2DBgqC2ySabRK2ve/fu5v5YgZzhw4cHtWnTpgW1jTYK75JuzcZGOLJpuemmm4LaUUcdFdQWLVpkLm+9xrGz8Fm3u7ZmP23Xrl30/rRq1SqqtmLFiqB2wgknBLWic7cxNIU2LK3ddnz11VcHtf79+wc1K9htBQ+ta7YktW3bNqgtWbIkqFnXWOsaOW7cuKhtd+zY0dyfWbNmmfWatttuu6D26KOPBrVrr702an0NgXAkAABYa+g4AACAaHQcAABANDoOAAAgGh0HAAAQLYzhN2FFadyarJS6dW92yf6FwcCBA4Oa9csIa3rczTbbLKhZ94ovJ5H+4x//OKhZU5paSXpLQ/yCAnHatGkT1PbZZ5+gZiW6reluJfvXRlZt+fLlQc1qw59//nlQ69u3b1ArakfWlMLWL5CqqqqC2m233RbU+vXrZ24Hcaxfzkh2e+rVq1dQs177efPmBTXrVxGxbVOKb5/Wr+Y233zzoGa9Xzz11FNB7YMPPjD3x/q1xaabbhrUOnXqFNSGDBkS1Lp16xbUrNsRNAeMOAAAgGh0HAAAQDQ6DgAAIBodBwAAEK1JhCOt8I4V3LEe9+qrrwY1K8BihbMkeyrdxYsXm4+tyQojWoEcK4Bp3cNdssMyO++8c1B7++23g1qapkHt9ttvN7eDtWP06NFBzQqVWcFDa3r0IrHt0AooWtMEW229qA1by1vToVvTCbds2TKode7cOah9+umn5rYRKgrVWo477rigZoVlrddp/vz5Qc0KGBaFaq02Zm3Hepw1TbrV3q2p/rfffntzf6yAshWSt46vdf6dcsopQe2KK64wt93UMeIAAACi0XEAAADR6DgAAIBodBwAAEC0JhGOjA3v3H///UFtwIABQW3hwoVBzQpnSXaAxprBzHqcFWqznosVLrICZEXbscJEVqjNCto8//zzQe3dd981t42Gt+uuuwY16/W1QmVWmFCyg2HW7HxWqMx6nBXsatGiRVB77bXXzP3ZbrvtgprVXq3Qo/W8f/7znwe1U0891dw24lnXoG222SaoWSFYa4bJlStXBjXruls0w21sSN4KYVrXtBUrVgS1Dh06BLU99tjD3B/r3Jg4cWJQs2b7tUKU1gzCXbp0CWrWrJxNDSMOAAAgGh0HAAAQjY4DAACIRscBAABEaxLhSIsVErRuVWrNIFfOrVwtVpDLCoxZM/nFBt2s4I5kh3esoI31vK0g0vHHHx/ULrvsMnPbaHjXX399UNt7772D2kEHHRTUrICiZLc5K1RmPc6qWeeeFcwsCpVZ54UVIJs6dWpQe+CBB4LauHHjzO2gfo488sigZs3UaIUjreuUdct4qx2Wc3t4K/hrzQw8dOjQoGYFw++5556gNmXKFHN/rOupdS3/9re/HdSsa7b1/I4++uigdt1115n705Qw4gAAAKLRcQAAANHoOAAAgGh0HAAAQLQmG47cdtttg1qnTp2CmjXLo6UoHGnNVmbVrNnOevToEdSscFHsPhaJDVdajxs2bFi9to3Ksm5zHnvr86eeesqsDxw4MKjNnDkzqBXNnlpT7C25i25Vb82oai1/wAEHRO0PGsa+++4b1KxZGa1QrtVGrGuSdT202kLRY2Nnk7Susdb+fOc73wlqVhhesgPB1uyY1kyP1nO0QqbWbJLNASMOAAAgGh0HAAAQjY4DAACIRscBAABEa7LhyM022yyoWSEdK3BiBRmtmc4kOwxk3QbbCgNZgRyrZs2oVk5g0grpWLfEtcJvffr0id4OKqsoBFaTNXujZdq0aWbdChLHziZphc+sZa1wcdF+W+2wPrcKjg3IwTZq1Cizbl2Dim7dXpN13bVmB7VmEbWupZL9mnbt2jWoTZ48Oaq2dOnSoGZdi4v2x7rGWqxjZm3bOi969uwZ1C644AJzO1dffXXU/jQGRhwAAEA0Og4AACAaHQcAABCNjgMAAIjWZMOR1qxmVtjMCk5ZoR8rrCLZoa927doFNes21i+++GJQGz58eFAbNGhQULMCnEX7Y4WGrKCN9bx79eplbgcNLzb0aL3mVmDLCvIWiQ2qWe3IOqcs1ix6RescO3Zs1DotBCHrp3v37mbdCv9Z1z6r3VghQ6u9W23BmiFSsq/v1mtvtTtrWWum4cWLF0ctK9nP0QpC9uvXL6hZx9EK3S9atCioWe81TQ0jDgAAIBodBwAAEI2OAwAAiEbHAQAARKPjAAAAojXZX1VsscUWUY+zErZWenXOnDnm8j169Ihafuuttw5qVsrd+hXDsmXLglpRIt1K07dt2zZqnVY6vz5T/aJxxP764pNPPjHrRSn1mqxfS8TWLNYU8EXLP//881HrjP2FCeJdeeWV0Y/deeedg5p1LT766KODmvVLntjp8ouWt6a2ttqd9esL6xppLVv0izvr1xZdunQJalOnTg1qf/3rX4Pa9OnTg1o5tx5oShhxAAAA0eg4AACAaHQcAABANDoOAAAgWpMNR1qhxdgpp60wYVEY0ZpO1aoVBWhiHmcFd1asWGEuHztta1EwrSZr2tTYsCWaloULF5p1K1BonRexbcY6z6x2WTRVr/XYovMPTYsVYrVqVtj8ggsuCGqvv/56UCuaQtyaht+69lnhdSsgbJ0XltatW5t163yxzqvHHnssqFmByXUJIw4AACAaHQcAABCNjgMAAIhGxwEAAERrsuFI677pVnjGCiNaM8117tzZ3I41M5kV+rLWaQV3rBBYhw4dgpr1XCQ7OGQ9dsaMGUFtyy23DGrWbGy9e/cOam+++aa5P2h4sTNHfvTRR2bdah+xwTArCByraIZJqw3PmjWrzttB/RSFWC1WINBqI927dw9qc+fOjdq2Fc6W4mdRrKqqCmrWPlqBSStYWcQ6h6xzdbPNNotan/V+UfQ+0NQx4gAAAKLRcQAAANHoOAAAgGh0HAAAQLQmG460AjBWcOeVV14JalZI5/vf/765HStwZgUKrcCXtT9WGMh6XNFMelZw6OOPPw5q1113XVC79tpro7bdrVu3oEY4cu2JDUcOGzbMrFttqT63oo4N0xXtt7X8gAEDgtoLL7wQ1LiFduXFti8p/vhboT5rO7G3u5biQ+RWwNG6zsUG2otCmbHHIvb4rkttmxEHAAAQjY4DAACIRscBAABEo+MAAACiNdlwZLt27YKaFS6xbjU8efLkoFbO7GmxAUdrf6wwkDXDXtGtZS1W+Gb8+PFBLXa2wE6dOkVvG5VVn7Zw7rnnmnVr9lOrLVghSisIHHuL9aJzyjovzjzzzKB26623Rm0HTY81U6PVtmNn4S2qW0HI2BkYretmfa/F1raLZsKM2Z/mihEHAAAQjY4DAACIRscBAABEo+MAAACiNdlwpBXassJdn3zySVStiBXysQI91i1araCMtWw5t3KdN29eULPCbzNnzgxq7du3D2qLFi0KatYMbWgcsSFda6bFonZktWErsGWdP7FhMetx1jkqSStWrAhq1vOJVd9AGyrPeo0t1utUFOK22qcVSl+yZElQs2Yats4161wpascW632gY8eO0cuvKxhxAAAA0eg4AACAaHQcAABANDoOAAAgWpMNR1ohFIsV/rMChkWsgKM1w5d161UrfGPVrFnNikJyscHF2FnIrMeVEwbC2jFy5MigFjszaH1ZYURLUTuyZp60zskePXoEtVmzZgW1cmZ9ReOwXhPrWlPOaxc786R1zY69jlthy3JY+9OmTZt6rbM5YsQBAABEo+MAAACi0XEAAADR6DgAAIBodBwAAEC0JvuritatWwc1637t1vTSVprWms5UsqdEtX4FYU33a027aiWDrV9KFP1qxJpO1fpFR7du3YLap59+GtSsJH67du3MbaPhWW3YMmLEiKBWlFC3XuPGmJK56NcX1rat1Hvfvn2DmvWrithfEGHtstqn1UaK2o11PY29flnLWtfNLl26BLWi94aiek3r469+GHEAAADR6DgAAIBodBwAAEA0Og4AACBakw1HWmEqK3Rl1QYNGhTUZs6caW7Hmi7UCmNZgcnYqUatKXit8Kdkh+es/dlhhx2CmvUc+/TpE9TWx/vHNzfW62u1Qck+V6x2FDtFuhVei22Xkh1os9Zpnafjx4+P3g6aFus1tqZ4Lno927ZtG9QWLlwY1KzzwFrWarNWiLKcaaitfV8fw+aMOAAAgGh0HAAAQDQ6DgAAIBodBwAAEK3JhiOtgJUVbLGChwcddFDU4yQ7VGMFLmMDNNZ+WyGdohkErRkqq6qqgtrGG28ctaz1XNbH+8evDVZYLHZGx+233z6oWTODSlKrVq2CmtXerSBkbJDR2u+iNhy7/G677RbU/vCHPwQ1wpGNpz7HOrZtF23Dqltt23ofsNpi7HWuKKi+dOnSoGa1bes9ZF3HiAMAAIhGxwEAAESj4wAAAKLRcQAAANGaRDjSCpdYYUTrNqmdOnUKav379w9qRUEZK9Dz2WefRe2jddtVK2hjBWqKgm7WdqxwpHWr7tgQ5voY5mnKrIDi/Pnzg1pR+Cz2tr71uf2vFVwrWp917lrtcOjQoXXeH6xdLVu2DGqxt9Auul21tU4r8P3FF18ENesaaT2unBCl1Y6tc5CZIwEAAGpBxwEAAESj4wAAAKLRcQAAANGaRDiyffv2QS02VDN48OCg1qVLl6BWFCyzZtizwmrW8tbtjK39th5XTljTmsFsyy23jFrWCrVZ+4jKiw0j9u3bN6hZgavFixdHbye2Frs+S9E5ZbV361bIvXv3jtoOmh4ryGixrqVFrDCj1RatbVvXUytYaZ1XVnuV7PcbK1xZzm251xW8gwAAgGh0HAAAQDQ6DgAAIBodBwAAEK1JhCOtAExs0O+TTz4JauXcxjr2dsFWoNAK/lghMCtkUzR7mrW8Fb6xQp1WuMg6Zh07djS3jcqKvUXxHnvsEdSKAlv12U5sG7Zqsedo0Xasc8qaHXOrrbYKatOmTTO3g8qLbUudO3cOarG3bS+69lnLW/tjXeesIKS1rHVLbmsW3iJW216wYEH08jXFPuemhhEHAAAQjY4DAACIRscBAABEo+MAAACiNYlwpBVOiQ1YWbNOWoFJ6/bbkj0LWWwwzQqHxc7KWPQ4q26FZVq1ahXUpk+fHtR23HHHoLbFFlvE7CIaycCBA4Na7C2Ka6vXFBuErE9gsoj1WOsW9P379w9qhCMbT2wwz7qNdWz41rp2FT3WuhbHzv5oXdutbRQF52OfTzlB5nUFIw4AACAaHQcAABCNjgMAAIhGxwEAAESj4wAAAKI1iTjojBkzgtrixYuD2pw5c4LaK6+8EtQuvvjioGb90kKKv5e6lbC1pn221hf7SwnJnk7VShFvvvnmQe3mm28OaocffnhQmzdvnrltVFbRlMw1Wa9lUdLbYk23a7Uja53WPi5dujSoWe3Vmh69SOwU8tYvTB566KHo7aBxWG3Oal/WFM9F58WSJUui1hnbtq32Zf0Couhcs371YymaQntdxogDAACIRscBAABEo+MAAACi0XEAAADRmkQ4MjbE0rVr16D2xBNPBLWhQ4cGtW7dupnbjp2u19of6z7s1vqsqU+LLF++PKpmhUIvu+yyoNaxY8egFhv6QeMYMmRIULPajPVaSlKXLl2CWmww02Kdj1ataMppqz537tygZk0Xf+ihhwa1K6+80twO1p7u3bsHtX79+gW1WbNmBTXrOi7ZIUUreGiFIy3W9NDWebXxxhuby1v7Hjsl97qOEQcAABCNjgMAAIhGxwEAAESj4wAAAKI1iXCkFYD561//GvU4a5a7SZMmVWS/mptnnnkmqN17771B7c4772yM3UEkKxzZq1evoNanTx9zeWvmSSvwtckmmwS1Nm3aBDUr3GiFwhYtWmTujzWjqnWeWiHdF154wVwnmpZHHnkkqE2ePDmoWQHyDh06mOu0Zt3t3LlzULParBWYrE+tiHUOWW07VnMNWzLiAAAAotFxAAAA0eg4AACAaHQcAABANNdcwxkAAKDxMeIAAACi0XEAAADR6DgAAIBodBwAAEA0Og4AACAaHQcAABCNjgMAAIhGxwEAAESj4wAAAKLRcQAAANHoOAAAgGh0HAAAQDQ6DgAAIBodBwAAEI2OAwAAiLZRQ63YpW6VpFfzbbwh6SSf+GV1XNdtkh72if/bGh53oKTLJbWT5PJlzqvD9vaU9IVP/Hjjb30k3eYTv2eN+guSWknqLKmNpI/zPx3mEz8jYps/9Yn/35JtPOwTP6jM/R4p6RpJ3SR5Sc9IOrvc4+5SN1hST5/4R8tZrilp7PbnUtdN0h8lbS6phaQZPvEH1WV79dmPMtY1Wtk+3lZS21/SVfk/t1LWhpdLmuIT/52IdQ5WSbvJt7HEJ/6aMvZrZ0m/VnYutZJ0t0/86NjlI7cxQ9Jwn/h5lVxvQ2vMNu1Sd7Kk/87/OUDSW5JWSXrcJ/7Cumyz0lzqnpI0qvT66lJ3kaSj8n9ur+x4SdItPvG/iVjnYZLe9ol/vWQb5/nEvxS5T/V6H2guGnLEYblP/OD8ze8LSWeW/tGlbsNKbsylbpCk30k6wSe+v6RBkqbXcXV7SvpaOQv4xO/kEz9Y0qXKLnaD8//NyPdvTZ20n9ZhP/8jf+O6R9KPfeK3ldRf0uOS2tdhdYMlVfRNby1o1PYn6TJJT/jE7+gTP0BSk7i4SpJLnXOpW+O57hM/prrdSnpJ0vH5v//TaVjDcRus+rebP0k6I9+HQZL+Ws/1VUzEOdzQGq1N+8TfWtIWZkraK//3f9p1A5xDhWKPvU/8z0r2e3nJdfg3+XrWdC4cpqyjVCcVeB+oqIZ6jRrrSYyTtEP+ST6RNEvSYJe67SVdqeyNupWk63zif+9S5yT9VtLekt5TNnqwJhdI+plP/JuS5BP/paTrJcmlbgtJt0jqKmmupJN94j9wqTtE0sWSWkqaL+l4Zb3EMyWtcqk7QdIPfeLH1eVJ55+4ekrqI2meS90/lH3S+UH+94eVjRAcIKmNS90kSVMlXSRpQ5e6m5R1YD6W9F8+8ctr2dz3Jf3JJ/65/Pl7SX/Lt9M5f/79JC1TdmGeko9Q/Cp/zsslnazseF+W78+ukq7wib+7Ls+/CWmM9tdD0j+q/+ETP0X6z+jVaEnzlL0Rvqysc+td6oZJ+qWkqvzvo3ziZ7nUnS7pDGXtcpqkE2t+snSpu1zZ6MYpks6V9O38OdznE5/ko1aPSRoraRdlF8T3I55HIP+Efouk/ST9zqXuTOWfwlzquijrZGyjGu0mX3xA/qmtt6RfRXzq21TZ6yOf+FWSqj/5jc7X0a/muvLz9Gxlx+sFSd/ziV/lUneDpBHK2vfffOKTGs+rjaT7JP1d0h3KXvPtlV0XR/vEP+BSN0rSNyW1VjaSuXf0gWtYjdGmAy51S5S12f0lnZtfQ07J/3yzT/yvao6YutSdJ6nKJ360S93Zyq6vX0p63Sf+GJe6dmrAY2+dCy51U33iq/K/HynpYEl/kHSopD1c6i6WdES+iqNc6q6X1EnSqeW+HxjvAz+R/X50m0pGgVzqlvjEV7nU9ZB0t6QOyo7PWT7x41zq9pOUKnud383Xs6Tm+SrprnL2N0aDZxzyHtaBWj1kNFLSRfmnslMlLfKJH6HsBD/dpa6vpMMlbausIZ2ukk//LnWXudQdamyq+qJs+Z2kP/vE7yDpdknVF69nJO3sEz9E2cG9IO8Z3ijp2rynWKdOQ4lhyt70jyt6QN6Lr+4dH5+Xt1Z20g+UtFB5I3apOzO/cNdU2/NPJU3Mn/9PJf05r78paff8+V8q6X994r/QV3vLzbrT0Ijt7zpJf3SpG+tSd5FLXc+Svw2RdI6yTzL9JH3dpa6FsovlkT7xw5Sd6D/LH3+vT/wIn/gdlQ1Jn1rjOV2t7A32ZEn7KGsrI5V94h/mUrd7/tBtlbX7IT7xdeo0lPjcJ35Xn3jzIlRLu9lO2ZvMSElJ/rzlUvdojWNU7VpJb7nU3edS912XutYlfwvW5VLXX9LRkr6ef9JbpewDgJS9zsMl7aDszWCHknVVSXpI0h0+8Tcp66w/mbeFvST9PH9Dk7I3m5N84ptEp6ER27SlnaTXfOJ30uoPGztJ2jnf1pA1LH+hpCH5taj6OtYYx36N54LPvpp+UNL5eRt+N//TRj7xI5Wdw4kkudT1dKkr56vc0veBovejIsdJGpO37x0lTco77BdL2scnfqiyzvuPSpap9Xytr4Yccaj+BC1lveM/KmusE3zi38vr+ynrNR+Z/7ujsovg7pLuzD9xzHSpe7J6pT7xl9ZhX3aR9K38v/8i6er8vzeTdHfeo2uprCdeaQ+uYaSgyHs+8ZPy/35ZWW9VPvE31mFduyrvePjEP+lSt4lLXUdlPdg/udRtrSwT0aIO626qGrX9+cSPcanrp2z06EBJE/Ovz5Rv8yNJyvepj7LO4CBJT7jUSdKGyj9pSxrkUvc/yj7hVEkaU7KpSyS94BN/Rr6+/fLnMTH/e1X+HD6Q9L5P/PNrOE6x6tqBfMQnfoWkFS51c5Tlbz7yBfkPn/jLXOpuV/acjpN0rLJPz0Xr+oayi/KL+XFsI2lO/vhvu9Sdoew610NZx21K/rcHJF3tE397/u/9JB2afzqWsk+5vfP/fsIn/tM6Pv9KagrX1FXKRmik7Lpyn0/8UklyqbtX0m7K3nyLTJF0u0vd/ZLuL9nnhj729TkX7s3/v/Q6PFPlfS1X+j5Q9H5U5EVJt+Sd7vt94ie51O2hrD0/m7f7lpKeK1mmQT/wNWTHYXneQ/qP/AkuLS0p+ypgTI3HHaTsjawcU5VdQCZHPLZ63b+V9Euf+AdLhpQrrfT5fqmvjvK0VrEVJf+9StkFsTbVz/8B42/WsKRXFiQd6xN/eD6c99QattGcNHb7U36Bu0PSHfnXULsr+wqs5mu5Ub7tqT7xuxiruk1ZmGpyPly7Z8nfXlQ2qtA5355T9nXS72s8hz766nOtr6J2XFsbluznXqv8k94N+Vd1c13qNqllXU7ZV3Q/KV1H/in7PEkjfOIX5MPApfv6rKQDXeruyL/Wc5KO8Il/q8Z6dlJlj2N9NHqbNnyedz6qt2Wp7Tr3TWXnxaGSLnGpG6jGOfY111N6LGLbcFT7jdy+tS//OW75V0stJckn/ul8FPGbkv7iUvdzSQuUdaqOrcP26m1t/xxzjKSzSoYvt8mHqJ6WdIxL3Yb5aMBeEev6uaSfutRtk69rA5e66qGb8ZKOyf/7eGVfUUhZb7w69XpSybo+U91ChWsyQ9n3kBu41G2ubIix2srq41BHv5N0Un6yScq++3Wp667seB6f1/aUNM8nfrG++vxHlayroZ5/U1Ox9udSt7dLXdv8v9tL2lLZp/4ib0nq6lK3S75Mi/wiKmXHfla+X8fXWO5xZd9hP5JvZ4ykU1zqqr+v7eVSt2nMk6+HGco6qZJ0ZEm93u3Gpe6b+UVTyj4pr1I2OlPkn5KOrH7OLnWdXZZp6qDs4rnIZcHhA2ssd6myTt31+b/HSPph9bYjhtybqkpeU9fkaWV5gbb5Ng5XNhIyW9Km+chmK2X5AbkslLi5T/xYZZm0Tlo9otbYx362S13/fJ8OL6k3xrWv6P1ohlafV/+lfAQ4b89z8q/U/ihpqKTnlX3luVX+mLbV732NYW13HG5WFn56xaXuNUm/V9aju0/SO8q+w7tB0r+qFyj6Ps5nYbRzJN3pUveGpNeUDU9KWXDqZJe6KZJO1OqfGY2WdI9L3Thl4bRqD0k63KVukkvdbpV5qpKyTznv5c/rGkmvlPztD5Km5MO0hVxBxsEnfrayxniNS91b+THYTdJiZc9zeP78r9TqTtLVkq5wqXtW2VB5tbHKQm2TXOqOLv9pNhsVa3/KTviX8mP8nLKg2ItFG84zAUdKusqlbrKkSVr9vfMlykJ+TyjLodRc9h5JNykbEh6nbJTjOZe6V5UFYhv6wneNsjen8ZK6lNSj240rzjicqCzjMEnZMO7xJZ9wAz772dzFkv6RH/snJPXwiZ+s7OubqcryI88ai58jqbXLMiOXK7tQT8nbwuW17X8TVsk2XSuf+FeUjY5NUNZeb/aJn+gTv1JZUPYFSQ9rdRveUNL/5e10orIc2UKtnWN/Yb5vT2r1V4RSlnU736VuokvdlkULu/IzDqWK3o9uUpbFmaAsN1I9arCnslzDRGVfOf/aJ36usg97d+breV5ZBqhROO8rMXq1/nAF8zgAzYkz5nEAmhNnzOOAxrG2RxwAAEAzsrYnNGmOFiobngOas6dUe3YAaOpuE214reCrCgAAEI2vKgAAQLRav6pwzlV8OMK58Ke/1qhH7OPqq0+fPkHtO98J7+fz73//O6i1bds2qLVq1SqopWka1BYvXmzuT2M978bgva/TtLaV1BBtGOuPptCGpXWrHY8ePdqsL1y4MKgtWxbew8u6FltatAh/3b7hhuGtG/r27Wsuf+6550ZtpzmodDtmxAEAAESj4wAAAKLRcQAAANHoOAAAgGi1/hxzbYYjY5e1arHhGUmaOnVqUNtqq62CWsuWLYPaF198EdSs53L11eHNzy69tC43+VytOYQom0KwbF0KlaHxNYU2LK1b7bghrlOxAfsvv/wyqG20kf0bgQMOOCCojRkzxnhk00c4EgAArDV0HAAAQDQ6DgAAIBodBwAAEK3Rb3IVG+qzZviyQo+xQcizzz7brFdVVQW1adOmBTUrCLlgwYKg1r59+6B2zDHHBLXx48eb+/P444+b9RjWsbU0tRAlgHXTnnvuGdSKrj9vv/12nbcTG5y3ZqLs2bOnuc6RI0cGteYajqw0RhwAAEA0Og4AACAaHQcAABCNjgMAAIjWJGaOjJ390Xrc+eefH9ROO+20oFY0O9iKFSuCmjVLpBWEtB63dOnSqMdZt+SW7PDOVVddFdT++te/msvXtDZnmGwKs+6tSzPuofE1hTYsNd92/OMf/zioXXHFFeZj33nnnaBWdJ2syXq/iJ0lsk2bNuY6X3311aD2jW98I2p/mhpmjgQAAGsNHQcAABCNjgMAAIhGxwEAAESj4wAAAKI1+pTTVqLfqnXv3j2ovfzyy0Ft/vz5Qc36BcTixYvN/bF+ddCpU6eg1q1bt6A2Y8aMoGb9qmLlypVBreiXDRtvvHFQu+iii4La//zP/wS1/v37B7VVq1aZ2wGAhtauXbugVnTts34FYf3qLfZXYdbjrOvh8uXLzeWtfUeGEQcAABCNjgMAAIhGxwEAAESj4wAAAKI1ejgydgrkp59+OqjNmTMnqFnhyA033DCqJtlhGStI+eGHH5rL12RNX9q6deug1qpVq6j1SfZz7Nq1a1CzpqE+4ogjgtranIYawPrDus5Z00NL0gYbhJ9jrZp1PbVqVkjeCmAW4ZpYjBEHAAAQjY4DAACIRscBAABEo+MAAACiNYmZIy+99NKox82dOzeoWaEYa6bGokBO7Oxi1vIdO3YMalbw0FrfsmXLzP2x7hdvbdsKTG611VZBbccddwxqkydPNrcNAA2tKKhuXbdbtGgR1Kzw+syZM4Pa1ltvHfU465orSZ9//rlZByMOAACgDHQcAABANDoOAAAgGh0HAAAQrdHDkZbTTjstqFnhP+sWq1bw0Aq7lHN7aWv5li1bBrX6zMBYFMj54osvgtqSJUuCWq9evYKaNcvad7/73aD2ve99L2YXAaBe3nzzzejHWtfoTp06BbVx48YFtYsuuiioTZ06NahNnz49qHXr1s3cn3nz5pl1MOIAAADKQMcBAABEo+MAAACi0XEAAADRGj0cWVVVFdSsW6+2bds2qG2yySZBzZqBMXYGsqLHWqzbscYGIWNnopTsgGPsrbotW2yxRdTjEM8KxVqvW2wg15pJz1q2aMa9WNZ+W+160003DWqHHHJIUDvggAPM7UybNi2o/eQnPwlqsc8bzdeUKVOCmtUOi+pWzQqWF83EW5N13bXekyR735FhxAEAAESj4wAAAKLRcQAAANHoOAAAgGiNHo7cbLPNglrsrajrEzYrCjIWBXVqsgI5seFIa3+sUFrROq3lrVt6WzNrWjNeon6sIGRR2LWuYtulVPlA4WeffRbUbr755qBWFEg7+OCDo7ZT6f0uauvWbKxWAHTgwIEV3R+UN3OkFZK3rn1Wu4u9FluPK2o3r732WtQ610eMOAAAgGh0HAAAQDQ6DgAAIBodBwAAEK3Rw5F9+/YNalYYa/ny5UHNCjR9/PHHQa1z585BbdGiReb+WDMwWmEqK7hjLRsb+CoKv1nhHSuMZ+2Pte3YGSYRr9IzQlZ6u5I0dOjQoHbFFVcEtRNPPDGoWQHmO+64I6jNmDHD3PaPfvSjoHbkkUcGte9///vm8jGsY/v888+bj/3Vr34V1MaPHx/UCEdW3tKlS4Na0Wy91jXRus4tXrw4qFmz61qs9VnBd0l69dVXo9a5PmLEAQAARKPjAAAAotFxAAAA0eg4AACAaHQcAABAtEb/VUX37t2DmjWFqHWPdKtmJc2tJG/RtKLWLxas6aBbtGhR52XLma7aWt5KIVu/OrGmPbZ+iYLGEfsriNjHWYlwSbr88suD2vDhw4Oa1YZ/+MMfBrW///3vQc2acvqyyy4z9+dvf/tbUDvppJOC2osvvhjUrOmqrV9Q3HDDDUHtjTfeMPfn4YcfDmrf+973gtrChQvN5VFZs2fPNuvWddJqs3Pnzo1aNnYb1vuFJE2bNi1qnesjRhwAAEA0Og4AACAaHQcAABCNjgMAAIjW6OFIK+BlBQI7deoU1KxwpBVQtKaMLmKFJq19tKZDtYI7sdu2Al+S9Pnnn0ftjxWos0KUTDndtGy++eZBrV27dkHtqKOOCmpWcFCKb3PWtM9nn312UDv66KOD2i9+8YugNnbsWHM7o0aNCmpWCHPMmDFBbdy4cUFtxYoVQW3AgAFBbccddzT3x5rmvnfv3kFt/vz55vKorFmzZpl1KzhvXXetWxR06NAhatuxYUvUjhEHAAAQjY4DAACIRscBAABEo+MAAACiNXo40gqnWEHBtm3bBjUrBGaFGzt27BjUiu4BbwUzrXu7W7OLWQHFTTbZJKhZ4S4r4CPZx8LaH2sGNCtYae2jFSi1Zp1E/Vjt8Nhjjw1qRxxxRFCzXnOrHUnSs88+G9QeeuihoDZy5Mig9uCDDwa1nXbaKagdcMABQe3SSy8192f69OlBzQpmDhw4MKideuqpQe0HP/hBUDvrrLOCWlG48Rvf+EZQe+yxx4LaNttsYy6PyrLahyT16NEjannrGmkFjGOXnTFjRtSyWI0RBwAAEI2OAwAAiEbHAQAARKPjAAAAojV6ONIKOFqBSSscZs04Fntr66KZxT799NOofbTCmlZYzbpdtjXjZdFttRcvXhzUrODiokWLglps6NGaldM6DusqayZOK1wVG7iS7NCVNTuh1WamTp0a1Kzbplu3kpakfv36BbUlS5YENet132qrrYLaSy+9FNROO+20oNanTx9zfyZMmBDUtttuu6B27rnnBrV//vOfQa1z585BrVevXkFtxIgR5v5Y55R1zbFqqLyimRqt65c1c+QHH3wQ1GJnyLXWx+2zy8eIAwAAiEbHAQAARKPjAAAAotFxAAAA0Ro9HGmFtqxgizUz4rJly4KaFaixAmjWLH6SHZaxatZ2LNYMk1awsmvXruby8+bNC2rdunWL2vaCBQuCmhX4sm5fu66GI61ZEK0go9VmrHYg2WHX2HVar+8ZZ5wR1KzgoRVklOwZQ63HWrOnWueUtd/33XdfULNmk5SkXXfdNai99tprQc26xbj1ep144olBbeLEiUGtKNz4zjvvBDVrRtWi1xuVZYWBJftW8tb5Yl3nimZVrcl6jT/88MOoZbEaIw4AACAaHQcAABCNjgMAAIhGxwEAAERr9HBk7O2brbCLNSOkFayMnU2yaDvWrJXW7butZa1ZJ61QWtHMkVZoy1rnpptuGtRiZ44sCmaui2Jv1WvNSloUuLICvtZslNZrOXPmzKB2+OGHB7Vygnoff/xxULMCudZMmNYtjq3bU1szWT788MPm/uy///5Bbe+99w5q1vGx9tE6FtbxtmZTLWIdH+t1ReUVvU5WwNi6plnhyKJbdcesb10NhjckRhwAAEA0Og4AACAaHQcAABCNjgMAAIjW6OFIK9RkBfisMJY1g+Ls2bODWjm3Q7Zut22FsWJnJrOCh1aA0zoOkrTJJpsENWs2SitwaQU4rWMbOxPlusBqC9YxtmZftMJ7kt02rVkLrdfICrpas+NZs1MWhfesbVszQsbOSjpo0KCgZgWGi25l/NlnnwU1KxBnBVetGWNjZ4ncbLPNzP2x6tY5NW7cOHN5VJY1O6hkB8atW6pbt+W2Ar0WK4A5dOjQqGWxGiMOAAAgGh0HAAAQjY4DAACIRscBAABEo+MAAACiNfqvKqxfMVi/WLAeZ6Xc33jjjaA2bNiwoGalzKXiqahr6tSpU1CzUvNW4t76ZYOVPJfsNP2ECROC2uabbx7UrFSyleIvSsOvix599NGgtu+++wa1bbbZJqhZ6W1J+uSTT4Ka1YZjp023ftlgTYNr/RpEktq3bx/UrF8Gbb311kHNaofWeTZlypSgVvRLowEDBkSt0/q1hPXLD2t6aOvXS0XTdL/yyitBzUrhz5kzx1weldWzZ0+zHjvN+ltvvVXnbVvbWJ+uh5XCiAMAAIhGxwEAAESj4wAAAKLRcQAAANEaPRxZFPCqyQoUPvvss0HNCmhZAUNrilrJnoI0dsppa53W1LzWNLrWfeGL6m+++WZQO+yww4La4sWLo9ZXzpTczd2CBQuC2gMPPBDUrOmP99lnH3OdI0eODGrWdNALFy4MatZrZLGmJC9qM9b0yVYY8Zlnnglq7777blCzpnG3zqkdd9zR3B8rzGuFhq1zzwrzWsfRel2LpghH07Llllua9digen1Yba5o+n8UY8QBAABEo+MAAACi0XEAAADR6DgAAIBojR6OtMKDVqjJqr300ktB7ZprrglqAwcODGozZsyI3EM7LGPN5FcfRWFNazZKyyWXXBLUrJkjrfV17949ahvrKivoarWPm2++2VzeCpdaszJaITArKGvNfmq9lkWs2R+tEKYV4LTCYtZMelZocfr06eb+WG3OmqnR2p9YVoC5aOZB6zlaNcKVjcNqC5K02WabRS1vvYdY4VuLFTCOveZiNUYcAABANDoOAAAgGh0HAAAQjY4DAACI1ujhSCvE0rJly6BmBbQeeuihqG1MnTq1/B0rYc3EV2n1DeS88847Qc261bAVBLRm5UQ8q31MmjQpqtbUWIFA6/lZNWv2xsZSTnjUmpGwMWYphK3o2mcFVi3W+0VsONIK0BKOLB8jDgAAIBodBwAAEI2OAwAAiEbHAQAARGv0cGRsqMkKyrz11ltRy1qz8xUFAq2ZxKx9LCeMVZMVyClaX2xw0Zpt0ApHWtshHAlgbbGC70WsW6pb1/dY1vtKbLASqzHiAAAAotFxAAAA0eg4AACAaHQcAABAtEYPR8beurac22DHbKO+YcRYVhCynG3EBilffvnloLb77rsHNet2yMyaB2BtKSeoXumZHssJqqMYIw4AACAaHQcAABCNjgMAAIhGxwEAAERr9HDkkiVLgpoV1rNCLLHBFitkExvKLGc7DSF23ydPnlzn9S1evLj8HQOACrAC25J9rbLEhs3nzZsX1KxZKwlHlo8RBwAAEI2OAwAAiEbHAQAARKPjAAAAotFxAAAA0Rr9VxU77LBDUGvRokVQa9u2bVCL/cVBfX9VUZ+UbX0TurH7PmfOnKDWqlWroGaliIcPH17HvQOA+nn//ffN+m677RbUPv/886Bm/TLPYv16w7oezp49O2p9WI0RBwAAEI2OAwAAiEbHAQAARKPjAAAAojV6OHLChAlBrWXLlkHNCkfGBhxjpyRdm6xpraX4ff/444+D2qRJk4KaFfxZtGhR1DYAoNJeeOEFs37KKacEtaqqqjpvx7rGWuHzV199tc7bWF8x4gAAAKLRcQAAANHoOAAAgGh0HAAAQDRX20yHzrkmf6NyKwDTHO6vXhSObA77Hst7bz/JRtQc2jCarqbQhqV1qx3vuuuuZv2xxx4Lap9++mlQ22KLLaK28+GHH0at7+CDD45evrmqdDtmxAEAAESj4wAAAKLRcQAAANHoOAAAgGi1hiMBAABKMeIAAACi0XEAAADR6DgAAIBodBwAAEA0Og4AACAaHQcAABDt/we7/T0xPPye3QAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 648x648 with 9 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot predictions\n",
    "plt.figure(figsize=(9, 9))\n",
    "nrows = 3\n",
    "ncols = 3\n",
    "for i, sample in enumerate(test_samples):\n",
    "  # Create a subplot\n",
    "  plt.subplot(nrows, ncols, i+1)\n",
    "\n",
    "  # Plot the target image\n",
    "  plt.imshow(sample.squeeze(), cmap=\"gray\")\n",
    "\n",
    "  # Find the prediction label (in text form, e.g. \"Sandal\")\n",
    "  pred_label = class_names[pred_classes[i]]\n",
    "\n",
    "  # Get the truth label (in text form, e.g. \"T-shirt\")\n",
    "  truth_label = class_names[test_labels[i]] \n",
    "\n",
    "  # Create the title text of the plot\n",
    "  title_text = f\"Pred: {pred_label} | Truth: {truth_label}\"\n",
    "  \n",
    "  # Check for equality and change title colour accordingly\n",
    "  if pred_label == truth_label:\n",
    "      plt.title(title_text, fontsize=10, c=\"g\") # green text if correct\n",
    "  else:\n",
    "      plt.title(title_text, fontsize=10, c=\"r\") # red text if wrong\n",
    "  plt.axis(False);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ce6dc44-90a5-48c3-91a5-810fa084d98b",
   "metadata": {},
   "source": [
    "Well, well, well, doesn't that look good!\n",
    "\n",
    "Not bad for a couple dozen lines of PyTorch code!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab108078-6770-4cb9-ac62-a761ff159aba",
   "metadata": {},
   "source": [
    "## 10. Making a confusion matrix for further prediction evaluation\n",
    "\n",
    "There are many [different evaluation metrics](https://www.learnpytorch.io/02_pytorch_classification/#9-more-classification-evaluation-metrics) we can use for classification problems. \n",
    "\n",
    "One of the most visual is a [confusion matrix](https://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/).\n",
    "\n",
    "A confusion matrix shows you where your classification model got confused between predicitons and true labels.\n",
    "\n",
    "To make a confusion matrix, we'll go through three steps:\n",
    "1. Make predictions with our trained model, `model_2` (a confusion matrix compares predictions to true labels).\n",
    "2. Make a confusion matrix using [`torch.ConfusionMatrix`](https://torchmetrics.readthedocs.io/en/latest/references/modules.html?highlight=confusion#confusionmatrix).\n",
    "3. Plot the confusion matrix using [`mlxtend.plotting.plot_confusion_matrix()`](http://rasbt.github.io/mlxtend/user_guide/plotting/plot_confusion_matrix/).\n",
    "\n",
    "Let's start by making predictions with our trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "065b8090-c9c5-43df-b5c1-b45ba33af1be",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "66c3a00eb6f04026888eb9ad983e4a49",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Making predictions:   0%|          | 0/313 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Import tqdm for progress bar\n",
    "from tqdm.auto import tqdm\n",
    "\n",
    "# 1. Make predictions with trained model\n",
    "y_preds = []\n",
    "model_2.eval()\n",
    "with torch.inference_mode():\n",
    "  for X, y in tqdm(test_dataloader, desc=\"Making predictions\"):\n",
    "    # Send data and targets to target device\n",
    "    X, y = X.to(device), y.to(device)\n",
    "    # Do the forward pass\n",
    "    y_logit = model_2(X)\n",
    "    # Turn predictions from logits -> prediction probabilities -> predictions labels\n",
    "    y_pred = torch.softmax(y_logit, dim=1).argmax(dim=1)\n",
    "    # Put predictions on CPU for evaluation\n",
    "    y_preds.append(y_pred.cpu())\n",
    "# Concatenate list of predictions into a tensor\n",
    "y_pred_tensor = torch.cat(y_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "362002d9-ec41-4c74-a210-b5d4f53410c4",
   "metadata": {},
   "source": [
    "Wonderful!\n",
    "\n",
    "Now we've got predictions, let's go through steps 2 & 3:\n",
    "2. Make a confusion matrix using [`torchmetrics.ConfusionMatrix`](https://torchmetrics.readthedocs.io/en/latest/references/modules.html?highlight=confusion#confusionmatrix).\n",
    "3. Plot the confusion matrix using [`mlxtend.plotting.plot_confusion_matrix()`](http://rasbt.github.io/mlxtend/user_guide/plotting/plot_confusion_matrix/).\n",
    "\n",
    "First we'll need to make sure we've got `torchmetrics` and `mlxtend` installed (these two libraries will help us make and visual a confusion matrix).\n",
    "\n",
    "> **Note:** If you're using Google Colab, the default version of `mlxtend` installed is 0.14.0 (as of March 2022), however, for the parameters of the `plot_confusion_matrix()` function we'd like use, we need 0.19.0 or higher. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "e6c0a05d-d3e0-4b86-9ef7-ee6ea5629b07",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mlxtend version: 0.21.0\n"
     ]
    }
   ],
   "source": [
    "# See if torchmetrics exists, if not, install it\n",
    "try:\n",
    "    import torchmetrics, mlxtend\n",
    "    print(f\"mlxtend version: {mlxtend.__version__}\")\n",
    "    assert int(mlxtend.__version__.split(\".\")[1]) >= 19, \"mlxtend verison should be 0.19.0 or higher\"\n",
    "except:\n",
    "    !pip install -q torchmetrics -U mlxtend # <- Note: If you're using Google Colab, this may require restarting the runtime\n",
    "    import torchmetrics, mlxtend\n",
    "    print(f\"mlxtend version: {mlxtend.__version__}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5245ede6-fd7f-40ad-a0b3-ae678544b84a",
   "metadata": {},
   "source": [
    "To plot the confusion matrix, we need to make sure we've got and [`mlxtend`](http://rasbt.github.io/mlxtend/) version of 0.19.0 or higher."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "21383f88-a2dd-4678-94c6-479c592da0ab",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.21.0\n"
     ]
    }
   ],
   "source": [
    "# Import mlxtend upgraded version\n",
    "import mlxtend \n",
    "print(mlxtend.__version__)\n",
    "assert int(mlxtend.__version__.split(\".\")[1]) >= 19 # should be version 0.19.0 or higher"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c91b9346-e25f-48ab-967e-425649331dc6",
   "metadata": {},
   "source": [
    "`torchmetrics` and `mlxtend` installed, let's make a confusion matrix!\n",
    "\n",
    "First we'll create a `torchmetrics.ConfusionMatrix` instance telling it how many classes we're dealing with by setting `num_classes=len(class_names)`.\n",
    "\n",
    "Then we'll create a confusion matrix (in tensor format) by passing our instance our model's predictions (`preds=y_pred_tensor`) and targets (`target=test_data.targets`).\n",
    "\n",
    "Finally we can plot our confision matrix using the `plot_confusion_matrix()` function from `mlxtend.plotting`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "7aed6d76-ad1c-429e-b8e0-c80572e3ebf4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdUAAAHMCAYAAACDYDKkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAB5mklEQVR4nO3dd3gUZdfH8e8hAQTp0hN6C4QSUugdBJGqdAFBRMSOPrbH3kUQe/fVxy4oFooCIgrSmxQFCygghBoFpCfZnPePmcQlJhTd3dnA+VzXXtmdnd355d7ZPTP3NFFVjDHGGPPv5fM6gDHGGHOmsKJqjDHGBIgVVWOMMSZArKgaY4wxAWJF1RhjjAkQK6rGGGNMgER6HeBMIAWLqhQ+z+sYf9O4emmvI+QoXA/iCtejy0S8TpCzMI0FQHpGeH6YkfnCs9XCs7XCdx779tuVKapaJqfnrKgGgBQ+j3M63uN1jL9ZOHGE1xFylBGmP3ipvgyvI+SoYGR4dihJuFZ74I+DqV5HyFGpIgW8jpCjcD1fQbjOY4Xyy5bcngvPb6sxxhiTB1lRNcYYYwLEiqoxxhgTIFZUjTHGmACxomqMMcYEiBVVY4wxJkCsqBpjjDEBYkXVGGOMCRArqsYYY0yAWFE1xhhjAsSKqjHGGBMgVlSNMcaYALGi6oFru8ey4smLWP7ERbwxph0F80dwZ//GbHx5AEvG92LJ+F50aRwNQKkiBZlxX1d2vz2UJy5v5kneL2bNpGFsHWJjajJ+3FhPMgCMHjWCKtHlSGzcIGvYxx99SGJcfYqcE8G3K1d4kuvo0aN0bN2MVk3jaZ7QkEcfvA+AEUMH0bppAq2bJtAwpgatmyZ4ki9TTK1qJDVuSNPExrRsluRplkxbt26lS6f2xDWoS3yjWJ575mlP87zywtO0bx5Hh+aNufryoRw9epTRIwZzfuskzm+dRNOGtTm/tbdtFy7fx+x+/uknmiY2zrqVO684zz3zlNexQt5eQb9KjYicB8xxH5YHfMAe93ETVc3xchIiUhWYrqr1c3juAeAbVf0yh+eGA1+o6na/YYOA6sBCIFVVF/3jf+hfqliqMFd3rUf8jR9zNNXH2ze1p1/LagA8+9k6np76/XHjH03z8cDEb4mtXIJ6lUqGPK/P52PM9dfw2YzZREVH06pZEt2796RuvXohzzJk6HCuvOparhgxLGtYvXr1eW/SR1x/7eiQ58lUsGBBpsz4kiJFipCWlkbXjm3o1OUCXn/7/axx7rr9ZooVK+5ZxkwzZn9F6dLhc0nAyMhIxo6bQOP4eA4cOECLpgl07HS+J/PXju3JvP7y83y9ZA2FChXiyssuYcrHH/DS6+9mjXP/Xbd6+jmG0/cxu9p16rB0xSrAyVmjajQ9e13kaSYv2ivoa6qq+ruqxqlqHPAS8GTm49wK6im85z25FNQIYDhQMdtTFwAzgXZAi38yzUCKjBAKFYggIp9QuGAEO/YeznXcw8fSWfzjLo6m+kKY8C/Lly2jRo2aVKtenQIFCtBvwECmT5viSZZWrdtQqmSp44bF1K1L7Tp1PMmTSUQoUqQIAGlpaaSlpSN+V4JUVT75aDJ9+g/0KmLYqlChAo3j4wEoWrQoMTF12b492bM86ek+jh49Qnp6OkcOH6Z8+QpZz6kq0z75iF59+nuWL5y+jyfy9VdzqF69BpWrVPE0hxftFRbdvyISKyLLRGS1iKwVkVruUxEi8qqIrBORL0SkkDv+GyLS172/WUTuEZEFwCAgEXjXfa9C4lyQLw74AxgN3Og+11pEqojIHHeac0Skst/7vyQi80XkZxHpHqj/dfsfh3lq6vf89OIAfn11IPsPpzFnjbNSPfqCuiyd0JuXrm5FiXPD47qL27cnEx1dKetxVFQ0ycne/eiFK5/PR+umCdSuUoF2HTuS2KRp1nOLFs6nbNly1KhZ6wTvEHwiQo8Lu9CiaSKv/d8rnmbJyZbNm1m9ehVJfm0XShUqRjH6ujE0aVCTxjFVKFasOG07nJ/1/NJFCyhTtizVa3j3OeaV7+OHH0yk3wDvFyK9aK+wKKo4xe5pd202EdjmDq8FPK+qscA+oE8urz+qqq1U9R1gBTDYXRM+AjQG1qjqJo5fU54PPAe8paoNgXeBZ/zesyrQFugGvCQi5wTiHy1xbgG6J1Wm3jUfUmPURM4tGMnA1jV4ddYPxF47mWY3f8rOvUcYO6xJICb3r+V08eJwvXCwlyIiIpi/dCXrNmzh2xXLWb/ur278jz6YRJ/+AzxM55gzdwGLl63k02mf88qLL7Bg/jdeR8py8OBBBvXvw/gJT1GsWDFPMuzbt5dZn09nyeqf+PaHzRw+fIiPJr2X9fynH03ydC0V8sb3MTU1lc+nT+PiPv28juJJe4VLUV0M3CEitwFV3GIIsElVV7v3V+IUupxMOsF7XwDMyOW55kDmt+ZtoJXfcx+oaoaqbgB+BWL8Xygio0RkhYis4NjBE0z+eO0bVmTL7oOk/HmUdJ8yZekWmtUpy+79R8nIUFTh9S9/IqFmmVN+z2CKiopm27atWY+Tk7dRsWL23nWTqXiJErRq3ZY5s2cBkJ6ezvSpn3CRxz/GQNbnVrZsWXr06s2K5cs8TuRIS0tjUP8+DBg0mN4XXexZjvlzv6JylaqcV7oM+fPnp2uP3qxYthhwPscZ06fQ8yJvC0Ve+D7OmjmDuMbxlCtXzusonrSXJ0VVRC5yu2BXi0iiqr4H9ASOALNEpIM76jG/l/nIfceqQyeYXGfgi1OMprnc/9tjVX1FVRNVNZGCRU7x7WFbyiGSapehUIEIANo1qMCPyfsoX6JQ1jg9m1Zh/da9p/yewZSYlMTGjRvYvGkTqampfDhpIt269/Q6VlhJ2bOH/fv2AXDkyBHmfj2HWrWd7bxzv/qSWrXrEBUd7WFCOHToEAcOHMi6P+fL2dSL/ds+gCGnqoy+4nLqxNTlhhtv8jRLVHQlvl2xlCOHD6OqLJj3NbXqOMvS8+fOoWatOlSM8vZzzAvfxw8nhUfXL3jTXkHf+zcnqvoJ8EnmYxGpDvyqqs+49xvirB3+EweAou77FgciVfV3v+f8+5YWAQNx1lIHAwv8nusnIm8C1XD2HP7pH+Y5zvINe/h08WYWje9Fuk9Zs+l3Xp/9Ey9c1YqGVUuhwG+7D3LdywuzXvPDC/0oWqgABSLz0aNJFXo8OIsft+0LRJyTioyM5Mmnn6NHty74fD6GDR9BvdjYkEw7u2FDL2H+N3P5PSWFWtUrcdfd91GyVCn+c+P1pOzZw8W9u9OwYRxTP5sZ0lw7d+7g6itG4MvwkZGRwUUX9+WCC53N8B9P/oA+/bz/gdm9axcD+zlrgenp6fQfOIjOXS7wOBUsWriQ9959m/r1G9A0IQ6A+x96hAu6XhjyLPGJTejW82K6tGtKZEQksQ3jGDxsJABTPv7Q865fCK/vY04OHz7MV3Nm8+wLL3kdBfCmvSSnPuegTUzkPuCgqj6ebfh/gSFAGrATuASn+GUdUiMiNwNFVPU+EXnDfW6yiGwGElU1xR2vD/AIzlrvBKCGqt7nPlcbmAxkANcBW4HXgdI4h/lcpqq/ue+/F2f7bjngJlWdntv/la9kVT2n4z3/qm2C4Y+JI7yOkKOMjNDNc6cj1ZfhdYQcFYwMl600xwu3bXn+/jj4jw4sCLpSRcJjB8TsQlkHTke4zmOF8stKVU3M6bmQFtVQE5H/A/5PVZec5uvewC3apzK+FdXTY0X19FhRPX1WVE9PuNaBcJ3HTlRUPen+DRVVHel1BmOMMWePM7qo/lOqOtzrDMYYY/Ke8OxXMsYYY/IgK6rGGGNMgFhRNcYYYwLEiqoxxhgTIFZUjTHGmACxomqMMcYEiBVVY4wxJkCsqBpjjDEBYkXVGGOMCRArqsYYY0yAWFE1xhhjAsTO/RsAjauXZmEYXhGmZHNvL/qcm72Ln/A6Qo4iMsLzihjm9IXr1WDCVbheDSYvsjVVY4wxJkCsqBpjjDEBYkXVGGOMCRArqsYYY0yAWFE1xhhjAsSKqjHGGBMgVlSNMcaYALGiaowxxgSIFVVjjDEmQKyoGmOMMQFiRdUYY4wJECuqxhhjTIBYUTXGGGMCxIqqMcYYEyBWVMPE1q1b6dKpPXEN6hLfKJbnnnk65BmuGdiaFRNvYeWkW7l2UJus4Vf1b8WaybezctKtPHxddwAqVyjJH/MfY8m7/2HJu//hmdv7hjzvF7Nm0jC2DrExNRk/bmzIp59p29atdOvSkcS4WJrEN+CF55457vlnnpxAsUIR/J6S4lFCR0ytaiQ1bkjTxMa0bJbkaRZ/4fI5+jt69CitmjehSXwj4hvF8uD993odKUs4thfAlSNHULliWRLi6nsd5Tihbq+wvZ6qiJwHzHEflgd8wB73cRNVTfUkWJBERkYydtwEGsfHc+DAAVo0TaBjp/OpW69eSKZfr0Z5LuvdjNbDniI13cfUZ0YxY8F6osqWoHvb+iQNGk9qmo8yJYtkvebX5BSaDZ4QknzZ+Xw+xlx/DZ/NmE1UdDStmiXRvXvPkLWXv8jISB4eO564xs5n16ZFEh06diKmbj22bd3KV1/NplKlyiHPlZMZs7+idOnSXsfIEk6fo7+CBQsyc/ZXFClShLS0NDq0bUXnLl1p2qyZp7nCtb0Ahg4bzuirr2XkiEu9jpLFi/YK2zVVVf1dVeNUNQ54CXgy87GqpopISBcIRCQimO9foUIFGsfHA1C0aFFiYuqyfXtyMCd5nJiq5Vj23RaOHEvD58tg/re/0KtdA0b1acHjb84hNc0HwJ69B0OW6USWL1tGjRo1qVa9OgUKFKDfgIFMnzbFkyzlK1QgrvFfn12dmJisz+6/t97Egw8/ZheBzkU4fY7+RIQiRZwFyLS0NNLT0sLiMwzX9gJo1boNpUqV8jrGcbxor7AtqjkRkTdE5AkR+Rp4TETiRGSJiKwVkU9EpKQ73lwRSXTvlxaRze79WBFZJiKr3dfUcocP8Rv+cmYBFZGDIvKAiCwFmofq/9yyeTOrV68iqUnTUE2Sdb/soFXj6pQqXphCBfNzQYu6RJcrQc0qZWgZV51v/ncDX7x8DQn1KmW9pmrFUix+5ya+ePkaWsZVC1lWgO3bk4mO/itLVFQ0ycmhWwjJzZYtm1m7ejWJSU35fPpUKlSMokHDRl7HApxC0ePCLrRomshr//eK13GA8P0cwVnLaZoQR+WKZenQ6XyaNA3d9zE34dxe4ciL9grb7t8TqA10UlWfiKwFrlPVeSLyAHAvMOYErx0NPK2q74pIASBCROoCA4CWqpomIi8Ag4G3gHOB71X1nmD+Q/4OHjzIoP59GD/hKYoVKxaqyfLT5t1MeOtrpj83mkOHj7F2w3bSfRlERuSjZNHCtLnsaRLrVeadRy6lbu+H2ZnyJ7V7PMgf+w/TOCaaDx6/jPgB4zhw6FhI8qrq34Z5vSZx8OBBhg7qx9jxTxAZGcn4xx7l0+kzPc3kb87cBVSsWJHdu3fTo2tn6tSJoVXrNid/YRCF4+eYKSIigqUrV7Nv3z4G9L2Idd9/T2x9b7cXhnN7hSMv2itPram6PnQLanGghKrOc4e/CZzsF2IxcIeI3AZUUdUjQEcgAVguIqvdx9Xd8X3ARzm9kYiMEpEVIrJiT8qenEY5bWlpaQzq34cBgwbT+6KLA/Kep+PNqUtpMfQJzr/yefb+eZiNW/eQvHs/n369FoAV638jQ5XSJc4lNc3HH/sPA7Dqx238uu13alUuE7KsUVHRbNu2NetxcvI2KlasGLLpZ5eWlsaQQX3pP+ASeva+mE2//sKWLZto2aQx9etUJzl5G62bJ7Jr507PMma2T9myZenRqzcrli/zLEumcPscc1KiRAnatG3HF194v4CUF9ornHjRXnmxqB46hXHS+et/OydzoKq+B/QEjgCzRKQDIMCbfttr66jqfe5LjqqqL6cJqOorqpqoqollSv/7YqKqjL7icurE1OWGG2/61+/3T2TuhFSpXAl6tW/AB7NWMW3ud7RLqgVAzcplKJA/gpR9hyhd4lzy5XOW+KpGlaJmpTJsSv4jZFkTk5LYuHEDmzdtIjU1lQ8nTaRb954hm74/VeWa0SOpU6cu195wIwCx9Rvw6287+f6nX/n+p1+Jiopm/uIVlCtf3pOMhw4d4sCBA1n353w5m3qx3u+lGU6fo789e/awb98+AI4cOcJXc76kTp0Yb0MRvu0Vrrxor7zY/QuAqu4Xkb0i0lpV5wNDgcy11s04a5/LgKxjPUSkOvCrqj7j3m8IfAFMEZEnVXW3iJQCiqrqllD+P4sWLuS9d9+mfv0GNE2IA+D+hx7hgq4XhizD+48Np1TxwqSlZzBm3MfsO3CEN6cu4+V7BrJi4i2kpvkYed/7ALRqXIO7R19AenoGvowMrhv7IXv/PByyrJGRkTz59HP06NYFn8/HsOEjqBcbG7Lp+1uyaCET33uH2PoNaNnU2WHpnvsfossFofvsTmb3rl0M7Of0fqSnp9N/4CA6d7nA41Th9Tn627ljB1eMGIbP5yNDM+jTtz8XduvudaywbS+AS4cMYv68uaSkpFCjajR333M/w0dc7mkmL9pLcupzDjcich9wEKgPTFfVye7wOJw9gwsDvwKXqepeEYkBPnBf8xUwRFWrish/gSFAGrATuERV/xCRAcB/cdZu04BrVHWJiBxU1SKcREJCoi5cuiKg/3MglGzuzRrvyexd/ITXEXKUlp7hdYQcRUaE5zYz25ZnzlaF8stKVU3M6bk8UVTDnRXV02NF9fRYUTUmvJyoqObFbarGGGNMWLKiaowxxgSIFVVjjDEmQKyoGmOMMQFiRdUYY4wJECuqxhhjTIBYUTXGGGMCxIqqMcYYEyBWVI0xxpgAsaJqjDHGBIgVVWOMMSZArKgaY4wxAWJF1RhjjAmQPHs91XCigC8j/K72E65Xg6l61WSvI+To1+f7eB0hR3sOpHodIUdlixX0OkKu/jgYnm1WqkgBryOYILM1VWOMMSZArKgaY4wxAWJF1RhjjAkQK6rGGGNMgFhRNcYYYwLEiqoxxhgTIFZUjTHGmACxomqMMcYEiBVVY4wxJkCsqBpjjDEBYkXVGGOMCRArqsYYY0yAWFE1xhhjAsSKqjHGGBMgVlQ9dNWoEVSNLkdS4wZZw+68/RYaN6hL04RGDOx3Mfv27fMuoOuLWTNpGFuH2JiajB83NqTTrlGuCF/e0ynrtuGZXlzRsSaxlYrz2X/b8+U9nZh1ZwcaVy0JQKXzCrPp+Yuyxn9sSOOgZxw9agRVosuR6Pc5/vHHH3Tv2pmG9WrTvWtn9u7dG/QcADdfN4r4OpU4v2V81rB9e/9g8MUX0jYplsEXX8j+fU6Wrb9tpnZUCbq2bULXtk244z/XhiRjdl7OX9m98sLTtG8eR4fmjbn68qEcPXoUgNdfeZ7WSfVp3zyOh+75r6cZw6m9Mm3dupUundoT16Au8Y1iee6Zp72OlCXU7RVWRVVEfCKyWkS+F5EPRaTwScafKyKJ7v3NIlI6NEkDY/DQ4Xw6bcZxwzp0PJ/lq75j6co11KpViwnjHvUoncPn8zHm+muYMm0Gq9au58OJ7/PD+vUhm/4vuw7S6YEv6fTAl3R+8EuOpPqYsWo7d/dpyIRpP9DpgS8ZN2U9d/dtmPWaLXv+es1t76wKesYhOXyOE8aPpV2HDqxd/zPtOnRgwvjQ/Pj1GzSUNz+YetywF55+nJZt2jNv+TpatmnPC089nvVclarVmTFvGTPmLeORCc+FJKM/r+cvfzu2J/P6y8/z+VeL+WrxKnwZPqZ8/AEL589l1ufT+HLBSr5evJrR193oST4Ir/byFxkZydhxE1j93Q/MW7CEl196PixyedFeYVVUgSOqGqeq9YFUYLTXgQDEEfC2atW6DSVLljpuWMfzOxMZ6Vw7PqlpM5KTkwM92dOyfNkyatSoSbXq1SlQoAD9Bgxk+rQpnmRpXbccm/ccZNsfh1GUouc47VS0cH527jviSSZwPsdS2T7Hz6ZNZfCQYQAMHjKM6VND02ZNW7SmRMmSxw2b/fk0+gwcAkCfgUP44vOpOb3UE+E0fwGkp/s4evQI6enpHDl8mPLlK/DW669wzZhbKFjQuSh76TJlPcsXbu2VqUKFCjSOd3pHihYtSkxMXbZv9/a3C7xpr3Arqv7mAzVFpJ2ITM8cKCLPicjwE71QRG5y13a/F5Ex7rDHRORqv3HuE5H/uPdvEZHlIrJWRO53h1UVkR9E5AXgW6BSwP/Dk3j7jf/RucsFoZ7scbZvTyY6+q9/PSoq2rNC3zspmk+XbQXgnolruLtvQ1Y+diH39m3IIx9/nzVe5dLnMvvujnxyc1ua1vKm82L37l1UqFABcH5w9uzZ7UkOgJQ9uylX3slSrnwFUlL2ZD239bfNdG3XlP49OrFs8YKQZwun+atCxShGXzeGJg1q0jimCsWKFadth/P5deMGli1eSPdOrejTrROrv13hST4Ir/bKzZbNm1m9ehVJTZp6HcWT9grLoioikUBX4Lt/8NoE4DKgKdAMuEJEGgMTgQF+o/YHPhSRzkAtoAkQBySISBt3nDrAW6raWFW3/MN/5x8ZN/ZhIiIjGTBocCgn+zeq+rdhIhLyHPkjhM6NKjJ1xTYAhrWrzr0frCHhts+594M1PDEsAYBd+4+ScNvnnP/gHO79YA0vjGxCEXeN1hyvbLkKLF6zgRlzl3L3g+O4ftQwDvz5Z0gzhMv8BbBv315mfT6dJat/4tsfNnP48CE+mvQevvR09u/by7TZ87nrgUcZfdklOeYOhXBqr5wcPHiQQf37MH7CUxQrVszrOJ60V7gV1UIishpYAfwGvPYP3qMV8ImqHlLVg8DHQGtVXQWUFZGKItII2KuqvwGd3dsqnDXSGJwiC7BFVZfkNBERGSUiK0Rkhf+SfyC8+/abzPz8M15/8x3PvzBRUdFs27Y163Fy8jYqVqwY8hwd6pfnu9/2kXLgGAD9m1fls2+dJc6pK7bRuJrT/ZqansHeQ6kArP1tH1v2HKJGuaIhz1u2bDl27NgBwI4dOyjjYZdh6TJl2bXTybJr5w5Kly4DQMGCBSlZ6jwAGsTFU6VadTb9siGk2cJl/gKYP/crKlepynmly5A/f3669ujNimWLqRAVRdcevRERGickkS9fPv74PcWTjOHUXtmlpaUxqH8fBgwaTO+LLvY6DuBNe4VbUc3cphqnqtepaiqQzvE5zznJe5yoCk0G+uKssU70G/9Rv+nWVNXMYn4otzdS1VdUNVFVEzN/pAJh9qyZPPH4OCZ9NIXChU+4n1ZIJCYlsXHjBjZv2kRqaiofTppIt+49Q57joiaV+XTZb1mPd+4/QovaTru3iinLr7sPAnBekQLkc+eAyqXPpVrZImzZczDkeS/s3oN333kTgHffeZNuPULfZpk6de3ORxPfAeCjie9w/oU9APg9ZQ8+nw+A3zb/yqZffqFy1WohzRYu8xdAVHQlvl2xlCOHD6OqLJj3NbXqxNDlwp4s/GYuAL9s/JnU1DRKnefNZoVwai9/qsroKy6nTkxdbrjxJq/jZPGivfJCv9gWoJ6IFMQpqB2BE238+QZ4Q0TG4hTMi4Ch7nMTgVeB0kBbd9gs4EEReVdVD4pIFJAW+H/j74YPvYT538zl95QUalevxJ1338eEcWM5lnqMnhd2BiCpSVOeef6lUMTJUWRkJE8+/Rw9unXB5/MxbPgI6sXGhjRDoQIRtKlXllveWZk17Oa3VvLgwDgi8wnH0jK45S3nuWa1y3Brr3qk+xRfhnLrO9+y73BwP85hfp9jreqVuOvu+/jPLbcz9JIBvPW/14muVJl33v8gqBkyXXfFUBYvnM/e31NoWr8GN95+F1ffcDNXjxjMpHffoGJUJV7833sALF20gCfGPkBkZCT5IiJ4ZMKzlMi2w1WwhcP8lSk+sQndel5Ml3ZNiYyIJLZhHIOHjURE+M+1o+jQvDH5CxTgqRf/z7MepHBqL3+LFi7kvXffpn79BjRNiAPg/oce4YKuF3qay4v2Eq+2DeRERA6qapEcho8DegEbcPYKnqqqb4jIXOBmVV0hIpuBRFVNEZGbgBHuy/9PVZ/ye6/vgBRVbe837AZgpPvwIDAE8AHT3T2RTyg+IVHnL15+2v9vsEXkC59tLf6qXjXZ6wg5+vX5Pl5HyFHKwVSvI+SobLGCXkfI1R9h2malihTwOoIJgEL5ZaWqJub0XFitqeZUUN3htwK35jC8nd/9qn73nwCeyOW9GuQw7Gkgp6OVT1pQjTHGmEzhtk3VGGOMybOsqBpjjDEBYkXVGGOMCRArqsYYY0yAWFE1xhhjAsSKqjHGGBMgVlSNMcaYALGiaowxxgSIFVVjjDEmQKyoGmOMMQFiRdUYY4wJECuqxhhjTIBYUTXGGGMCJKyuUpOXhdMl9P4Snpd+C9dLrJ13waNeR8jRnhm3ex0hzzm3YITXEcxZytZUjTHGmACxomqMMcYEiBVVY4wxJkCsqBpjjDEBYkXVGGOMCRArqsYYY0yAWFE1xhhjAsSKqjHGGBMgVlSNMcaYALGiaowxxgSIFVVjjDEmQKyoGmOMMQFiRdUYY4wJECuqxhhjTIBYUfXQtq1bubBzRxIaxZLUuAEvPPcMAH/88Qc9L+xMXGwdel7Ymb1793qW8cqRI6hcsSwJcfU9y5CbZ59+ksS4+iQ2bsCwoZdw9OjRkE7/mouTWPHaFax8/Qqu7ZMEwMVtY1j5+hUc+vK/xNcunzXuwI6xLHnl8qzboS//S8MaZYOa76pRl1OtUnmaxDfMGvbd2jV0aNuSpgmN6HdxT/7888+gZjgVX8yaScPYOsTG1GT8uLGe5Th69CgdWjejZdN4miU05JEH7wPgofvvoUWTxrRqmsBFPS5gx/btnmWE8Gkvf1u3bqVLp/bENahLfKNYnnvmaa8jZQl1e+XpoioiPhFZLSLrRGSNiNwkInnmf4qMjOSRx8azcs06vvpmEa+89AI//rCeJx5/jLbtO7J63U+0bd+RJx5/zLOMQ4cNZ8r0mZ5NPzfbk5N58flnmb94OStWfUeGz8eHH0wM2fTrVS3DZd3iaH31/2gy8v/o2qwmNaJKsm7THgbe+xEL1v523PgT56yj2ajXaDbqNS5/dCpbdu5j7S+7g5px8NBhfDL18+OGXXvVKB548BGWrlxDj569efqJx4Oa4WR8Ph9jrr+GKdNmsGrtej6c+D4/rF/vSZaCBQsydcaXLFz6LfOXrGTO7FksX7aE62+8mUXLVrFg6Uq6dO3GuEcf8iQfhFd7+YuMjGTsuAms/u4H5i1YwssvPR8WubxorzxTgHJxRFXjVDUWOB+4ELg3+0giEpYXYy9foQJxjeMBKFq0KHViYtienMxn06YyeMilAAwecinTp07xLGOr1m0oVaqUZ9M/kXRfOkeOHCE9PZ3Dhw9ToULFkE07psp5LFufzJFj6fgylPlrfqNXqzr89NvvbNj6xwlf279DPT74Kvg/OK1at6FkyeM/uw0//0TL1m0A6NDxfKZ8+nHQc5zI8mXLqFGjJtWqV6dAgQL0GzCQ6dO8md9FhCJFigCQlpZGWlo6glCsWLGscQ4fOoSIeJIPwqu9/FWoUIHG8X/9lsXE1GX79mSPU3nTXnm9qGZR1d3AKOBacQwXkQ9FZBrwhYicKyKvi8hyEVklIr0ARCRWRJa5a7xrRaSWO+5n7trv9yIyINj5t2zezNrVq0ls0pQ9u3dRvkIFwCm8KXuCu0aTF1WMiuKGMf8hpmYValSpSLHixel0fueQTX/dpj20aliJUsUKUahgJBc0rUF02WInfyHQt31oimpO6sbW57PpUwH45OPJJG/b6kmOTNu3JxMdXSnrcVRUNMnJ3v0Y+3w+WjVNoFaVCrTv2JHEJk0BePDeu4itVZUPJ73PHXff51m+cGuvnGzZvJnVq1eR5Ladl7xorzOmqAKo6q84/1PmxqrmwDBV7QDcCXylqklAe2C8iJwLjAaeVtU4IBHYBlwAbFfVRqpaHwhq/+fBgwcZMqgfYx9/4rilYpO7vXv3Mn36VNb99CsbNydz+NAh3n/vnZBN/6fffmfCxCVMHz+IqY8NZO0vu0n3ZZz0dUkxFTl8NI31m/eEIOXfvfDy//HqSy/QunkSBw8cIH+BAp7kyKSqfxvm5ZpgREQEC5auZN2GLaxcsZz1674H4O77H2Ldhs30GzCIV1563rN84dZe2R08eJBB/fswfsJTYfFb5kV7nVFF1eXfYrNVNbMvrjNwu4isBuYC5wCVgcXAHSJyG1BFVY8A3wGdROQxEWmtqvv/NhGRUSKyQkRWpOz55z+QaWlpDBnYl/4DL6FX74sBKFO2HDt37ABg544dlC4T3B1a8qKvv/qSqlWrUqZMGfLnz0/P3hexdPGikGZ4c8YaWlz5OuePeYe9B46wcduJu30B+oWo6zc3derEMOWzWcxfvJy+AwZSvXoNz7KAs+awzW9tOTl5GxUrhq4bPzclSpSgVeu2zJk967jhfQcMYtqUTzxKFb7tBc5v2aD+fRgwaDC9L7rY6ziAN+11RhVVEakO+IDM/tJD/k8DfdxtsHGqWllVf1DV94CewBFgloh0UNWfgQSc4vqoiNyTfVqq+oqqJqpqYukyZf5RXlXlmitHUiemLtfdcGPW8Au79+Ddd94C4N133qJbj57/6P3PZJUqVWb50qUcPnwYVWXu119RJ6ZuSDOUKVHYyVK2GL1ax5y0WIo4ewd/+LV3RXXPbuerkZGRwfhHH2bEyFGeZQFITEpi48YNbN60idTUVD6cNJFu3b2Z31P27GHfvn0AHDlyhHlfz6FW7Tr8snFD1jgzPptGrdp1PMkH4dVe/lSV0VdcTp2Yutxw401ex8niRXuF5Q48/4SIlAFeAp5TVc1hFX8WcJ2IXOc+31hVV7mF+FdVfca931BEfgT+UNV3ROQgMDwYmRcvWsj7771DbP0GtGjibOS/94GHuOnm2xg2eCBvv/E60ZUq89Z7k4Ix+VNy6ZBBzJ83l5SUFGpUjebue+5n+IjLPcuTKalJU3pf3IeWTROIiIykUVzjkBeI9+/rQ6lihUjz+Rjz9Cz2HTxKz1a1eeK6zpQuXpiPHxnA2l920fM2Z6/kVg0rk7znAJt37AtJvsuGXsL8+fP4PSWFOjUqc8dd93Lo0CFeeekFAHr2voihwy4LSZbcREZG8uTTz9GjWxd8Ph/Dho+gXmysJ1l27tzBVVeMwJfhQzMy6H1xXy64sDtDB/Vj44afkXz5qFSpMk8+84In+SC82svfooULee/dt6lfvwFNE+IAuP+hR7ig64We5vKivSSnPue8QkR8OGuT+YF04G3gCVXNEJHhQKKqXuuOWwh4CmiBs9a6WVW7i8h/gSFAGrATuARIAsYDGe7wq1R1RW454hMS9ZtFy4LyP/4bkRHh2RGRkRGe89x5FzzqdYQc7Zlxu9cRchSu8xfAsTSf1xFyVDB/hNcRTAAUyi8rVTUxp+fy9JqqquY6h6rqG8Abfo+PAFfmMN6jQPZf01nuzRhjjDll4buoaYwxxuQxVlSNMcaYALGiaowxxgSIFVVjjDEmQKyoGmOMMQFiRdUYY4wJECuqxhhjTIBYUTXGGGMCxIqqMcYYEyBWVI0xxpgAsaJqjDHGBIgVVWOMMSZArKgaY4wxAZKnr1ITLoTwvgxWuDl4LN3rCDnaMf02ryPkqMoo766neyLJrw3yOkKujqTapd+MN6wSGGOMMQFiRdUYY4wJECuqxhhjTIBYUTXGGGMCxIqqMcYYEyBWVI0xxpgAsaJqjDHGBIgVVWOMMSZArKgaY4wxAWJF1RhjjAkQK6rGGGNMgFhRNcYYYwLEiqoxxhgTIFZUjTHGmACxohpGvpg1k4axdYiNqcn4cWO9jgPAlSNHULliWRLi6nsdBYD9+/Zx+dABtEyoT6vEBixfuoQrhl9Ch5aJdGiZSGL9WnRomehJNp/PR5vmiQzo0xOAu++4lSaNY2nZpDFDBvZh/759IckxuksdFj5yIQse7sorV7WgYP58lDi3AB/d0p5lj3Xno1vaU7xwfgDyR+Tj2ZFNmf9QV+Y9eAEtY8qGJKO/cJrvX37+ado2i6Nd88ZcdflQjh49yt69fzCgd1daxNdjQO+u7Nu319OM4dRe/sLttyJTqNvrjCyqIlJeRCaKyC8isl5EPheR2qf5HiVE5OpgZczO5/Mx5vprmDJtBqvWrufDie/zw/r1oZp8roYOG86U6TO9jpHlrttuon2nLixc+T1fLVpJ7ToxvPrGe3y1cAVfLVxBt54X0a1Hb0+yvfT8M9SuE5P1uH2HTixavoaFy1ZRo2Ytnng8+F/oCiULMer82nS8dxat7pxBRD7h4qZVuKFbPb5Zv5Mmt03nm/U7GdO9HgCXtqsBQOu7ZtBn3Nc8MKgxIkGPmSWc5vsd25N57eXnmfn1YuYuXoXP52PKRx/w3JPjadW2A4u+XU+rth147snxnuSD8Gqv7MLttwK8aa9ci6qIHBCRP93bAb/HB0Tkz6Cm+hdERIBPgLmqWkNV6wF3AOVO861KACErqsuXLaNGjZpUq16dAgUK0G/AQKZPmxKqyeeqVes2lCpVyusYABz4808WL1rA4EsvA6BAgQIUL1Ei63lVZeonk7mo74CQZ0tO3sYXMz/n0uEjsoZ16NSZyMhIAJKaNGN7cnJIskTmE84pEEFEPqFQgQh27DvChfFRTFywCYCJCzZxYXw0AHUqFuOb9bsASDlwjD8PpdK4Wug+73Cb730+H0ePHiE9PZ0jRw5TrkIFZn0+jf6DhgDQf9AQZn421bN84dZe/sLptyKTF+2Va1FV1aKqWsy9FfV7XFRViwU11b/THkhT1ZcyB6jqamCBiIwXke9F5DsRGQAgIkVEZI6IfOsO7+W+bCxQQ0RWi0jQF023b08mOrpS1uOoqGiSQ/QjnFds2fwr551XmhuuGknHVknceO2VHDp0KOv5JYsWUKZsWarXrBXybHfcehP3PzyWfPly/kq989b/6NT5gqDn2LH3CM/N+JE1T/Rk/dO9+fNwGnO/30mZYuewa/9RAHbtP0rpYucA8P3WfXRtHEVEPqFy6XNpVLUUUaUKBz1npnCa7ytUjGL0tWNIrF+TRnWqULRYcdp1OJ89u3dTrnwFAMqVr0DKnj2e5IPwaq+8wIv2OqXuXxFpJSKXufdLi0i1oKb6d+oDK3MYfjEQBzQCOgHjRaQCcBS4SFXjcQryBHdt93bgF1WNU9Vbgh1aVf82TELZD5cHpKf7+G7NKoZdfiVzFiyncOFzefaJcVnPfzJ5kidrqTNnTKd0mbLENU7I8fnHxz1CZGQk/QdeEvQsxQvn58L4aOJvnkbsmE85t2Ak/VpUzXX8d7/5le17DzPnvi48MjieZRtTSPf9fV4MlnCa7/ft28usz6ezdM1PrP5xM4cPHWLypPc8yZKbcGqvvMCL9oo82Qgici+QCNQB/gcUAN4BWgY1WeC1At5XVR+wS0TmAUnADOAREWkDZABRnEJXsYiMAkYBVKpc+V+Hi4qKZtu2rVmPk5O3UbFixX/9vmeSilFRVIyKJiGpCQA9el/Ms084nQjp6el8NvVTZn+zJOS5li5exMzPpjF71gyOHT3KgQN/MmrEpbzy+lu8/85bfDHjMz79bHZIfvzaxpZny56D/H7gGADTV26lSc3S7PnzKOWKO2ur5YqfQ8qfzlqrL0O5671VWa+fcVcnft11IOg5M4XTfD9/7ldUrlKV0qXLAHBhj96sWLaYMmXLsmvnDsqVr8CunTsoXaaMJ/kgvNorL/CivU5lTfUioCdwCEBVtwNFgxnqX1oH5LTKkNsv2mCgDJCgqnHALuCck01EVV9R1URVTSxT+t9/yRKTkti4cQObN20iNTWVDydNpFv3nv/6fc8kZcuVp2JUNBs3/AQ4P4K1Y+oC8M3Xc6hVuw4Vo6JDnuveBx5h3YYtrP3hF157811at23PK6+/xZdfzOTpJ8fz3gefUrhwaLpUk38/TGLN0hQqEAFAm3rl+Xn7fmasSmZgK6eDaWCranz+rdMFVqhABIXdcdvFlic9Q/lpe+h2mQin+T4quhIrVyzl8OHDqCoL5n1NrdoxdO7anQ/efweAD95/hy4X9vAkH4RXe+UFXrTXSddUgVRVVRFRABE5N6iJ/r2vcNY8r1DVVwFEJAnYCwwQkTeBUkAb4BZgALBbVdNEpD1QxX2fA4Rw4SEyMpInn36OHt264PP5GDZ8BPViY0M1+VxdOmQQ8+fNJSUlhRpVo7n7nvsZPuJyz/I8Mv5Jrh45jNTUVKpUrcbTL/wfAJ9+9IEnXb8ncut/buDYsWNc1MPZlprYpClPPvNCUKe58tffmbr8N76+/wLSMzL4bste3pz7C+cWjOT1a1oyuE0Nkn8/xGXPLwSgdLFzmHxzOzJU2bH3CFe9vDio+bILp/k+PrEJ3XteTOe2TYmMjKR+gziGDB/JoUMHuXL4Jbz/9v+Iiq7EK2++70k+CK/2yi7cfivAm/aSnPqcjxtB5GagFnA+8CgwAnhPVZ8NarJ/QUQqAk/hrLEeBTYDY3C6a7sCCjykqpNEpDQwDcgPrMbp1u6qqptF5D2gITDjRNtVExISdeHSFcH6d844fx5J8zpCjgpEhOcRZjWu+sDrCDlKfm2Q1xFyte9QqtcRclTi3AJeRzABUCi/rFTVHA+IP+maqqo+LiLnA38CtYF7VHV2gDMGlNtF3T+Hp25xb/7jpgDNc3mf4O9ZYowx5oxxKt2/AN8BhXDW8L4LXhxjjDEm7zppf5eIjASW4RyS0hdYIiIjTvwqY4wx5uxzKmuqtwCNVfV3ABE5D1gEvB7MYMYYY0xecyp7ZmzD2RM20wFgay7jGmOMMWetXNdUReQm924ysFREpuBsU+2F0x1sjDHGGD8n6v7NPEbzF/eWKTzO3myMMcaEmVyLqqreH8ogxhhjTF53Kuf+LQPcCsTid/o+Ve0QxFzGGGNMnnMqOyq9C/wIVAPuxzk70fIgZjLGGGPypFMpquep6ms41yidp6ojgGZBzmWMMcbkOadynGrmiVp3iEg3YDsQ+kuBGGOMMWHuVIrqQyJSHPgP8CxQDLgxqKmMMcaYPOhUTqg/3b27H2gf3DjGGGNM3nWikz88i3Oyhxyp6vVBSWTOeMUK5fc6Qp4SrpdYK5l0rdcRcrV3+XNeR8iRL+PEl9r0SkQ+8TrCGeNEa6p2gVBjjDHmNJzo5A9vhjKIMcYYk9edyiE1xhhjjDkFVlSNMcaYALGiaowxxgTISYuqiNQWkTki8r37uKGI3BX8aMYYY0zeciprqq8C/8U9s5KqrgUGBjOUMcYYkxedSlEtrKrZL0qeHowwxhhjTF52KkU1RURq4J4IQkT6AjuCmsoYY4zJg07l3L/XAK8AMSKSDGwChgQ1lTHGGJMHncq5f38FOonIuUA+VT0Q/FjGGGNM3nPSoioi92R7DICqPhCkTMYYY0yedCrdv4f87p8DdAd+CE4cY4wxJu86le7fCf6PReRxYGrQEhljjDF51D85o1JhoHqggxj4YtZMGsbWITamJuPHjfU6ThbLdXquHDmCyhXLkhBX3+sox/G6va4Z1I4VH97Bysl3cu0l7QBoWDuKeW/+hyUTb2fBu7eSGFsFgMTYKiyZeDtLJt7O0km307N9w5Dn9bq9/F01agRVo8uR1LjB3557+onHKVIwHykpKR4k+0s4tZe/rVu30qVTe+Ia1CW+USzPPfN0UKd3KmdU+k5E1rq3dcBPQHBT5ZzjThFZ5+ZYLSJNA/Cec0Uk8d+OEwg+n48x11/DlGkzWLV2PR9OfJ8f1q8P9mQtVxAMHTacKdNneh3jOF63V70aFbjs4ha0HjqeJgMepWub+tSoXIaHx/Tm4Vdm0GzgWB58cToPj+kNwLpfttNy8DiaDRxLr2te4Nm7BhEREbqzqnrdXtkNHjqcT6fN+NvwbVu38tWcL6lUubIHqf4Sbu3lLzIykrHjJrD6ux+Yt2AJL7/0fFCzncpc2h3o4d46AxVVNaRXABaR5m6OeFVtCHQCtoYyQ7AtX7aMGjVqUq16dQoUKEC/AQOZPm2K17Es1z/QqnUbSpUq5XWM43jdXjHVyrPsu80cOZqGz5fB/JUb6dW+EapQ7NxzAChepBA79uwHyBoPoGCB/KiG9uLeXrdXdq1at6Fkyb/PU7fdchMPPfpY1g6kXgm39vJXoUIFGsfHA1C0aFFiYuqyfXty0KZ3wqIqIvmAz1R1i3tLVlUvzqZUAUhR1WMAqpqiqttF5B4RWS4i34vIK+LOWe7a5WMiskxEfhaR1u7wQiIy0V3bnQQU8vtfXxSRFe7a8P2h/ge3b08mOrpS1uOoqGiSk4P3wZ8qy3Vm8Lq91v2ynVbxNSlV/FwKnZOfC1rFEl2+JLc8PplHxvRmw4wHefTGi7jn2b9+iJPqV2Hl5DtZ8eEdXP/wxKwiGwpet9ep+GzaVCpWrEiDho28jpIn2gtgy+bNrF69iqQm/7qjM1cnLKqqmgGsERFv+xbgC6CSWyBfEJG27vDnVDVJVevjFMjufq+JVNUmwBjgXnfYVcBhd233YSDBb/w7VTURaAi0FZGQbsTJaUnc66VPsFxnCq/b66dNu5jwxmymv3gtU5+/hrU/J5Oe7mNUv9bcOuFjanW9m1sf/4gX7x2c9Zrl328hoe/DtBoyjltGdKZggVM5WCEwvG6vkzl8+DDjH3uEu+4NjyMbw729AA4ePMig/n0YP+EpihUrFrTpnEr3bwVgnXulmqmZt6AlyoGqHsQpgKOAPcAkERkOtBeRpSLyHdABiPV72cfu35VAVfd+G+Ad9z3XAmv9xu8vIt8Cq9z3qXeiTCIyyl2zXbEnZc+/+O8cUVHRbNv2V492cvI2Klas+K/f99+yXGeGcGivNz9dTItLHuP8y59i7/5DbPxtD4O7N+XTOasB+Gj2qqwdlfz9tGkXh46kElszdHnDob1O5Ndff2Hz5k00T4qjXu1qJG/bRqtmCezaudOTPOHeXmlpaQzq34cBgwbT+6KLgzqtUymq9+OsAT4ATPC7hZSq+lR1rqreC1wLDAZeAPqqagOcq+mc4/eSY+5fH8cfOvS3RSoRqQbcDHR012I/y/ZeOeV5RVUTVTWxTOky//TfypKYlMTGjRvYvGkTqampfDhpIt269/zX72u5DIRHe5UpWQSASuVL0qtDIz6YuYIde/bTOqEWAO2a1Gbjb84CapWK52XtmFS5QklqVy3Hlu2/hyxrOLTXidSv34DN23ax/udNrP95E1HR0SxYspJy5ct7kiec20tVGX3F5dSJqcsNN94U9OmdSn/Khap6m/8AEXkMmBecSH8nInWADFXd4A6Kw9kLuSHOCf+LAH2BySd5q29wivHXIlLffT1AMZyTXOwXkXJAV2BuIP+Hk4mMjOTJp5+jR7cu+Hw+hg0fQb3Y2JO/0HKFVS6AS4cMYv68uaSkpFCjajR333M/w0dc7mmmcGiv9x8fSakS55KW7mPM2A/Yd+AI1zz4HuNv6UtkZD6OHUvn2ofeB6BF4+rcfFln0tJ9ZGQoNzwyid/3HTrJFAInHNrL3/ChlzD/m7n8npJC7eqVuPPu+xh2mbfzlL9way9/ixYu5L1336Z+/QY0TYgD4P6HHuGCrhcGZXpysr3qRORbVY3PNmytu0YXEiKSADwLlMC57NxGnK7gMTjXdt2MszfwFlW9T0TmAjer6goRKQ2sUNWqIlII+B9O1+5qoCZwvTveG0BT4FectdypqvqG/3vlli8hIVEXLs31aWPOSCWTrvU6Qq72Lg/pAQqnzJcR2r2YT1VEvvDa/hnuCuWXle4+OH+T65qqiFwFXA1UFxH/bY9FgYWBjXhiqroSaJHDU3e5t+zjt/O7n4K7TVVVj5DLBdZVdXguw9vlNNwYY4zJ7kTdv+8BM4BHgdv9hh9Q1T+CmsoYY4zJg3Itqqq6H9gPDApdHGOMMSbvCt15v4wxxpgznBVVY4wxJkCsqBpjjDEBYkXVGGOMCRArqsYYY0yAWFE1xhhjAsSKqjHGGBMgVlSNMcaYALGiaowxxgSIFVVjjDEmQKyoGmOMMQFiRdUYY4wJkFO5SLk5iWPpGWxJOex1jL+pUrqw1xFytP9wmtcRclQwf3guYx5Ly/A6Qo7C9ZqlAO0nzPM6Qo6+/k9bryPkKN0XnvNYZER4fidPJO8lNsYYY8KUFVVjjDEmQKyoGmOMMQFiRdUYY4wJECuqxhhjTIBYUTXGGGMCxIqqMcYYEyBWVI0xxpgAsaJqjDHGBIgVVWOMMSZArKgaY4wxAWJF1RhjjAkQK6rGGGNMgFhRNcYYYwLEimqI3XHjaFrUr0KPdolZw559/GHaNK5J707N6N2pGfPmzARg4bw5XNy5JT3aJ3Fx55YsWTA35HmvHDmCyhXLkhBXP+TTzm7jhp/o2Cox61Yz+jxeeeEZpn4ymTZNG1GhREFWf7sy5LmOHj1Kx9bNaNU0nuYJDXn0wfsAGDF0EK2bJtC6aQINY2rQumlCSHPl1l57//iD/r260rxxPfr36sq+vXtDmiu7L2bNpGFsHWJjajJ+3NiQT79IwQge7l2PiSOTeH9kIvUrFuPyllWYenUz3hyewJvDE2hevRQASVVL8r9h8bwzIoH/DYsnoXKJkOf1ur38XTXqcqpVKk+T+IZZw4YNGUiLJvG0aBJPbO3qtGgS72HC0LeXqGrQJxJqInIncAngAzKAK4FJQKKqpmQbtydQT1X/1toi0g5IVdVFJ5pe/Ubx+tGsBaeUbfniBRQ+91xuv/4Kps1dAThFtfC553L5VWOOG3f9d6s5r0w5ypWvwM8/rmPkoF58s2rjKU0HAnM91QXzv+Hcc4swcsSlrFz9/b9+PwjM9VR9Ph9xMVX5fM4Cjhw5TL58+bhlzDXc++BjxMX/s+L1T6+nqqocOnSIIkWKkJaWRteObXj08SdJatIsa5y7br+ZYsWKc+sdd5/2+wfieqr+7fW/V1+kZMlSXHfTrTz7xDj27dvL3Q88etrvWbxw/oDkalCvNp/NmE1UdDStmiXx5jvvU7devX/1vqdzPdW7L6zD6m37mbZ2J5H5hHPy52NAYjRH0ny8t2zbcePWLluEPw6nknIwleqlC/NU/4b0fGHJKU/r315PNVjt9U+vp7pg/jcUKVKEUZcPZ9m3a//2/H9vu5nixYpz+52nP9/Dv7+earDaq1B+WamqiTk9d8atqYpIc6A7EK+qDYFOwNbcxlfVqbkU1EigHdAikPmSmreieMlSpzRuvQZxlCtfAYBadepx7NgxUo8dC2Sck2rVug2lSp1a3lCaP/crqlarTqXKVahdpy41a9XxLIuIUKRIEQDS0tJIS0tHkKznVZVPPppMn/4DvYp4XHvN+nwa/S8ZCkD/S4Yy87OpnuVavmwZNWrUpFr16hQoUIB+AwYyfdqUkE2/cIEI4ioVZ9ranQCkZygHj/lyHf/n3QdJOZgKwK8phykQmY/8EZLr+IHmdXtl16p1G0rm8numqnwy+UP6DvBuvveivc64ogpUAFJU9RiAqqao6nb3uetE5FsR+U5EYgBEZLiIPOfef0NEnhCRr3HWbEcDN4rIahFpHczQ777+Mj07NOGOG0ezf9/fu+NmffYp9eo3pEDBgsGMkWd8+vEH9O47wOsYWXw+H62bJlC7SgXadexIYpOmWc8tWjifsmXLUaNmLc/y+bfXnj27sxbWypWvQMqePZ7l2r49mejoSlmPo6KiSU5ODtn0o0qcw77Dadx1YR3eHB7Pfy+ozTluj0Xf+CjeviyBO7vWpmjByL+9tn2d0vy86yBpvtD19nndXqdj4YL5lC1XjpoezvdetNeZWFS/ACqJyM8i8oKI+Pe3pKhqPPAicHMur68NdFLVPsBLwJOqGqeq8/1HEpFRIrJCRFbs/T0lxzc6VYOGjWT2ku/59MsllClbnsfu/+9xz2/4aT0THrqb+8c9+6+mc6ZITU3li8+n07N3H6+jZImIiGD+0pWs27CFb1csZ/26v7rKP/pgEn36e7cAEI7tlSmnzU8ioVvzi8gn1C5flI9XbWfYG99yJM3Hpc0q8/Gq7fR9eSmX/m8lKQdTub5D9eNeV610Ya5uW53HZv0csqzgfXudjskfTKSvh70z4E17nXFFVVUPAgnAKGAPMElEhrtPf+z+XQlUzeUtPlTV3Pt//prOK6qaqKqJJc8r/a8yly5TjoiICPLly0e/IZfx3aoVWc/t3J7MtSMG8dgzr1K5avUTvMvZ46vZM2nQqDFlypbzOsrfFC9Rglat2zJn9iwA0tPTmT71Ey7q09+zTNnbq0yZsuzauQOAXTt3ULpMGc+yRUVFs23bX1tnkpO3UbFixZBNf/eBY+w5cIz1Ow4A8PVPKdQuV4S9h9PIUFBgypod1K1QLOs1ZYoWYOxFsTz42Y8k7zsasqzgfXudqvT0dKZO+YQ+fb2b78Gb9jrjiiqAqvpUda6q3gtcC2QuomdukPQBf+/PcRwKdr7sdu/akXX/y8+nUismFoA/9+/jyqEXc9N/7ye+SfNQxwpbn0yeFFZdvyl79rB/3z4Ajhw5wtyv51CrtrONd+5XX1Krdh2ioqM9y5e9vTp37cEH770NwAfvvU2XC3t4FY3EpCQ2btzA5k2bSE1N5cNJE+nWvWfIpv/HoTR2/XmMyqUKOXmqlGBzymHOO7dA1jjtapfm1xTnZ6FIwQgm9G3Ai/M2sTb5z5DlzOR1e52qr7/6ktq1Yzyd78Gb9sqtsORZIlIHyFDVDe6gOGAL0OAfvN0BoNhJxzoNN101jOWL5rP3j99pG1+L626+i2WLvuGHdWsREaIqVeH+cc8AznbW3zb9yotPjeXFp5x9qV6bOJXzSpcNZKQTunTIIObPm0tKSgo1qkZz9z33M3zE5SGbfnaHDx/mm6/nMP6pF7KGfT7tU+689UZ+T9nDkP69qN+gERM/+SxkmXbu3MHVV4zAl+EjIyODiy7uywUXdgfg48kf0Kefd11gObXXdTfdwqhhl/De228QFV2JV99837N8kZGRPPn0c/To1gWfz8ew4SOoFxsb0gxPfLmB+7rXJX+EkLzvKA9//hM3dqpJ7XLnogo79h/lsVnOz0nf+CiiSxTishZVuKxFFQDGfLCWvQHYo/1UhEN7+bts6CXMnz+P31NSqFOjMnfcdS/DLrucyR9Mot8A7xd8vWivM+6QGhFJAJ4FSgDpwEacruAVuIfUiEgi8LiqtnO7hhNV9VoReQOYrqqT3feqDUzGOSznuuzbVTOdziE1oRSIQ2qCIRCH1ATDPz2kJtgCcUhNMATikJpgOZ1DakLp3x5SEyz/9JCaYPu3h9QEy4kOqTnj1lRVdSU5HwZT1W+cFTiHy6CqbwBvuPeHZ3uvn4GGGGOMMacgPBcDjDHGmDzIiqoxxhgTIFZUjTHGmACxomqMMcYEiBVVY4wxJkCsqBpjjDEBYkXVGGOMCRArqsYYY0yAWFE1xhhjAsSKqjHGGBMgVlSNMcaYALGiaowxxgSIFVVjjDEmQM64q9R4oWBkvrC9zFo4CudLhoWjc/JHeB0hR6np4Xm5MAjfS6yVbPNfryPkaO83j3odIUdpYTyP5cbWVI0xxpgAsaJqjDHGBIgVVWOMMSZArKgaY4wxAWJF1RhjjAkQK6rGGGNMgFhRNcYYYwLEiqoxxhgTIFZUjTHGmACxomqMMcYEiBVVY4wxJkCsqBpjjDEBYkXVGGOMCRArqsYYY0yAWFENI1/MmknD2DrExtRk/LixXsfJYrlOj+U6sW1bt9K9S0eS4mJpGt+AF597BoDv1q6hU9uWNE9sxIA+Pfnzzz89ywjet9c1/Vuw4p0bWPnOGK7t3/K458YMas2RRY9yXnHnkpMDO8ex5I3rsm6HFjxMw1oVQprX6/bKtG3rVrp16UhiXCxN4hvwgjt/ZXrmyQkUKxTB7ykpQZl+2BVVEblTRNaJyFoRWS0iTQP43u1EZHqg3i+QfD4fY66/hinTZrBq7Xo+nPg+P6xf73Usy2W5Ai4yMpKHxo5n+ep1fDlvEa++/AI//rCe664axX0PPcLiFWvo3rM3zzz5uCf5wPv2qle9HJf1TKL15S/QZNgzdG0ZQ43o8wCILlucDk1q8tvOvVnjT/xiNc2GP0uz4c9y+QMfsGXHPtZu2BGyvF63l7/IyEgeHjueFavXMcdv/gKn4H711WwqVaoctOmHVVEVkeZAdyBeVRsCnYCt3qZyiEhQL+i+fNkyatSoSbXq1SlQoAD9Bgxk+rQpwZyk5bJcnmQpX6ECcY3jAShatCh1YmLYvj2ZjRt+omWrNgC073A+Uz/92JN84H17xVQpw7Lvt3LkWBo+XwbzV22iV9tYAMbd0I07n5+Bas6v7X9+Iz74ck3IsoL37eUvt/kL4L+33sSDDz+GiARt+mFVVIEKQIqqHgNQ1RRV3S4im0XkfhH5VkS+E5EYABE5V0ReF5HlIrJKRHq5w6uKyHx3/G9FpEX2CYlIkvua6iKSICLzRGSliMwSkQruOHNF5BERmQfcEMx/fPv2ZKKjK2U9joqKJjk5OZiTPCWW6/RYrtOzZctm1q5eTWJSU+rWq8/n06cC8OnHk0ne5t3ytNftte7XXbSKq0apYoUpVDA/F7SoQ3TZ4nRrVZfte/7ku407c31t304N+WB2aIuq1+2VG//56/PpU6lQMYoGDRsFdZrhVlS/ACqJyM8i8oKItPV7LkVV44EXgZvdYXcCX6lqEtAeGC8i5wK7gfPd8QcAx3Wqu0X2JaAXzprws0BfVU0AXgce9hu9hKq2VdUJ2d5jlIisEJEVe1L2/Ot/XHNY7Azm0tSpslynx3KduoMHDzJ0UD8eHf8ExYoV4/mX/49XX36BNi2SOHjwAPkLFPAsm9ft9dOWPUx4Zx7Tnx7B1CcvY+2GHaT7MrhtWHseeHV2rq9LqleJw0fTWP/rrpBlBe/bKyeZ89fY8U8QGRnJ+Mce5c577g/6dIPapXm6VPWgiCQArXGK5CQRud19OrMvaCVwsXu/M9BTRDKL7DlAZWA78JyIxAE+oLbfZOoCrwCd3bXg+kB9YLY7E0QA/hsjJuWS9RX3fUhISMylI+bURUVFs81vyTw5eRsVK1b8t2/7r1mu02O5Tk1aWhpDB/Wl/4BL6Nnb+TrXrhPDp9NnAbBxw8/MmvG5Z/nCob3enL6CN6evAOD+Kzuze+9BBnSJY9lbTqdZVJliLP7fdbQe+Ty7/jgIQD8P1lIhPNrLX1paGkP85q9133/Hli2baNmkcVa+1s0T+Xr+EsqVLx/QaYfbmiqq6lPVuap6L3At0Md96pj718dfCwMC9FHVOPdWWVV/AG4EdgGNgETAf5F3B3AUaOz3Huv83qOBqnb2G/9QoP/HnCQmJbFx4wY2b9pEamoqH06aSLfuPUMxactluUJKVbl29Ejq1KnLtTfcmDV8z+7dAGRkZDB+7MOMuGKUJ/kgPNqrTMlzAahUrji92sXy7oxvqdLtYWL6jCOmzziS9/xJ88uezSqoIsLFHRrwYYi3p0J4tFcmVeWabPNXbP0G/PrbTr7/6Ve+/+lXoqKimb94RcALKoTZmqqI1AEyVHWDOygO2AI0yOUls4DrROQ6VVURaayqq4DiwDZVzRCRYThrn5n2AZcDX4jIIWARUEZEmqvqYhHJD9RW1XUB/wdPIDIykieffo4e3brg8/kYNnwE9WJjQxnBclmukFiyaCET33uH2PoNaNXU2aHknvsf4peNG3n15RcA6NHrIoZcepkn+SA82uv9hwdTqnhh0tIzGPP4VPYdOHrC8VvFVSV59342b997wvGCIRzaK5P//NXSb/7qcsGFIZm+5NQX7hW36/dZoASQDmwERgErgERVTRGRROBxVW0nIoWAp4AWOGucm1W1u4jUAj4CDgNfA9epahERaQfc7I5TGZgBjMBZC34GpxhHAk+p6qsiMtcdf8WJcickJOrCpSccxZgzTmp6htcRclUgMuw64QAo2ea/XkfI0d5vHvU6Qo7SwnQeK1YoYqWqJub0XFgV1bzKiqo5G1lRPX1WVE9PXiyq4TnnGWOMMXmQFVVjjDEmQKyoGmOMMQFiRdUYY4wJECuqxhhjTIBYUTXGGGMCxIqqMcYYEyBWVI0xxpgAsaJqjDHGBIgVVWOMMSZArKgaY4wxAWJF1RhjjAkQK6rGGGNMgITV9VTzKsW5MG64ERGvI+QoHNsqnIXr55g/IjxzAWRkhOc89se8R7yOkKOSzW/yOkKO9i5+wusIp83WVI0xxpgAsaJqjDHGBIgVVWOMMSZArKgaY4wxAWJF1RhjjAkQK6rGGGNMgFhRNcYYYwLEiqoxxhgTIFZUjTHGmACxomqMMcYEiBVVY4wxJkCsqBpjjDEBYkXVGGOMCRArqsYYY0yAWFENIzG1qpHUuCFNExvTslmS13EAuHLkCCpXLEtCXH2vo+TI5/PRLCmei3v38DpKlnD8HLdu3UqXTu2Ja1CX+EaxPPfM015HOk44fo7PPv0kiXH1SWzcgGFDL+Ho0aNeRwK8n7+uGdiaFRNvYeWkW7l2UJus4Vf1b8WaybezctKtPHxd96zh9WtWYO5r17Ny0q0sf/8WChYI7RVHv5g1k4axdYiNqcn4cWODPr0z/nqqIuIDvgME8AHXquoib1PlbsbsryhdurTXMbIMHTac0Vdfy8gRl3odJUfPP/s0MTF1+fPAn15HOU64fY6RkZGMHTeBxvHxHDhwgBZNE+jY6Xzq1qvndTQg/D7H7cnJvPj8s6xcs45ChQox9JIBfPjBRIZeOtzraIB381e9GuW5rHczWg97itR0H1OfGcWMBeuJKluC7m3rkzRoPKlpPsqULAJAREQ+Xn9gMJff+x7fbdhOqeKFSUv3hSyvz+djzPXX8NmM2URFR9OqWRLdu/cM6nx/NqypHlHVOFVtBPwXeNTrQHlJq9ZtKFWqlNcxcrRt2zZmzvic4SMu9zpK2KtQoQKN4+MBKFq0KDExddm+PdnjVI5w/RzTfekcOXKE9PR0Dh8+TIUKFb2O5LmYquVY9t0WjhxLw+fLYP63v9CrXQNG9WnB42/OITXNKZh79h4EoFPTOny/cQffbdgOwB/7D4f0AvLLly2jRo2aVKtenQIFCtBvwECmT5sS1GmeDUXVXzFgL4CIFBGROSLyrYh8JyK9MkcSkbtF5EcRmS0i74vIzaEIJyL0uLALLZom8tr/vRKKSeZpt/7nRh569DHy5Quv2TjcP8ctmzezevUqkpo09ToKEJ6fY8WoKG4Y8x9ialahRpWKFCtenE7nd/Y6FuDt/LXulx20alydUsULU6hgfi5oUZfociWoWaUMLeOq883/buCLl68hoV4lAGpVKYOqMvWZUSx6+yZuGto+pHm3b08mOrpS1uOoqGiSk4O7MHnGd/8ChURkNXAOUAHo4A4/Clykqn+KSGlgiYhMBRKAPkBjnPb5FliZ/U1FZBQwCqBS5coBCTpn7gIqVqzI7t276dG1M3XqxNCqdZuTv/As9Pln0ylTtgzx8Ql8M2+u13GOE86f48GDBxnUvw/jJzxFsWLFvI4Ttp/j3r17mT59Kut++pUSJUowZFB/3n/vHQZdMsTraJ7OXz9t3s2Et75m+nOjOXT4GGs3bCfdl0FkRD5KFi1Mm8ueJrFeZd555FLq9n6YyIh8tGhUjVbDnuLw0VRmvHAV3/64jbnLN4Qkr+rf14pFJKjTDJ9Fw+DJ7P6NAS4A3hKnVQV4RETWAl8CUUA5oBUwRVWPqOoBYFpOb6qqr6hqoqomli5dJiBBK1Z0upfKli1Lj169WbF8WUDe90y0ZNFCPps+jZha1bh0yCDmff0VI4YN9ToWEL6fY1paGoP692HAoMH0vuhir+MA4fs5fv3Vl1StWpUyZcqQP39+eva+iKWLw2NXDK/nrzenLqXF0Cc4/8rn2fvnYTZu3UPy7v18+vVaAFas/40MVUqXOJfkXfuYv+oXft9/iCPH0pi56Aca14kOWdaoqGi2bdua9Tg5eVtW+wXL2VBUs6jqYqA0UAYY7P5NUNU4YBfO2mxwF2NycejQIQ4cOJB1f86Xs6kXG5573IaDBx5+lI2btvLjhk289c77tG3fgdfffNvrWGH7Oaoqo6+4nDoxdbnhxpu8jpMlXD/HSpUqs3zpUg4fPoyqMvfrr6gTU9frWGExf2XuhFSpXAl6tW/AB7NWMW3ud7RLqgVAzcplKJA/gpR9h5i95Cfq16xIoYL5iYjIR+v4GvywaWfIsiYmJbFx4wY2b9pEamoqH06aSLfuPYM6zbOh+zeLiMQAEcDvQHFgt6qmiUh7oIo72gLgZRF5FKd9ugGvBjvb7l27GNjPWXtIT0+n/8BBdO5yQbAne1KXDhnE/HlzSUlJoUbVaO6+5/6w26EknITr57ho4ULee/dt6tdvQNOEOADuf+gRLuh6obfBwlRSk6b0vrgPLZsmEBEZSaO4xowYOcrrWGExf73/2HB3L94Mxoz7mH0HjvDm1GW8fM9AVky8hdQ0HyPvex+AfQeO8Mx781jw1o2oKrMW/sDMhT+ELGtkZCRPPv0cPbp1wefzMWz4COrFxgZ1mpJTn/OZxO+QGnDWQu9Q1c/c7ajTgPzAaqAl0FVVN4vIfcAgYAuwB5irqrkW1viERF24ZHnw/ol/KNjbDv6pM32eCzT7HE9fuEYL04+SUi3+43WEHO1d/ITXEXJUKL+sVNXEnJ4749dUVTUil+EpQPNcXva4qt4nIoWBb4AJwcpnjDHmzHHGF9V/6BURqYezjfVNVf3W60DGGGPCnxXVHKjqJV5nMMYYk/ecVXv/GmOMMcFkRdUYY4wJECuqxhhjTIBYUTXGGGMCxIqqMcYYEyBWVI0xxpgAsaJqjDHGBIgVVWOMMSZArKgaY4wxAWJF1RhjjAkQK6rGGGNMgNi5fwNACN/Lc4Uja6szQ0aYXl4NICJfeM5jR1J9XkfIUbheYq1k2zu9jnDabE3VGGOMCRArqsYYY0yAWFE1xhhjAsSKqjHGGBMgVlSNMcaYALGiaowxxgSIFVVjjDEmQKyoGmOMMQFiRdUYY4wJECuqxhhjTIBYUTXGGGMCxIqqMcYYEyBWVI0xxpgAsaJqjDHGBIgV1TDyxayZNIytQ2xMTcaPG+t1nCyW6/RYrhO7atQIqkaXI6lxg6xhd95+C40b1KVpQiMG9ruYffv2eZYv05UjR1C5YlkS4up7HQUAn89H2+aJDOzT87jhzz41gVLnRvJ7SopHyRxet9c1/Zqz4u3rWfnO9Vzbv8Vxz40Z1IojCx/mvOKFAeiQVIOFr13N8reuY+FrV9M2vnrAcgS1qIrIRSKiIhJziuNvFpHSOQw/eJrTPa3xT/A+w0WkYiDe62R8Ph9jrr+GKdNmsGrtej6c+D4/rF8fiklbLssVUoOHDufTaTOOG9ah4/ksX/UdS1euoVatWkwY96gn2fwNHTacKdNneh0jy0vPP0PtOsf/lG7btpW5X31JdKXKHqX6i5ftVa9aWS7rmUTrkS/SZNhzdG1RhxrR5wEQXbY4HZJq8tvOvVnj/77vMH1ve5ukS5/liocm8/o9/QKWJdhrqoOABcDAIE8nWIYDISmqy5cto0aNmlSrXp0CBQrQb8BApk+bEopJWy7LFVKtWrehZMlSxw3reH5nIiMjAUhq2ozk5GQvoh2nVes2lCpV6uQjhkBy8jZmz/ycocNHHDf8ztv+w/0PjUXE+4uye9leMVXLsmzdVo4cS8Pny2D+6s30alMPgHHXX8idL8xE9a/x12zYwY6UAwCs37SbggUiKZA/IiBZglZURaQI0BK4HL+iKiLtRGSuiEwWkR9F5F3JNkeISCERmSkiV+TwvreIyHIRWSsi959g+hNE5FsRmSMiZdxhcSKyxH3tJyJSMrfhItIXSATeFZHVIlIoIA2Ti+3bk4mOrpT1OCoqOix+WCzX6bFc/97bb/yPzl0u8DpGWLnj1pu47+Gx5Mv310/2jM+mUaFCFPUbNvIwWXhY9+suWjWqSqlihShUMD8XNK9NdLnidGsVw/Y9f/Ldxp25vvaidrGs+Xk7qWm+gGQJ5ppqb2Cmqv4M/CEi8X7PNQbGAPWA6jjFN1MRYBrwnqq+6v+GItIZqAU0AeKABBFpk8O0zwW+VdV4YB5wrzv8LeA2VW0IfHei4ao6GVgBDFbVOFU9ki3LKBFZISIr9qTsOcUmyZ36L0b9NY1//b7/luU6PZbr3xk39mEiIiMZMGiw11HCxqwZ0ylTpixxjROyhh0+fJgJ4x7hjrvv8y5YGPlpyx4mvPsN058awdQnhrF2407SfRncdmk7Hvi/L3N9Xd1qZXno6i5cOz5wvTbBLKqDgInu/Ynu40zLVHWbqmYAq4Gqfs9NAf6nqm/l8J6d3dsq4FsgBqfIZpcBTHLvvwO0EpHiQAlVnecOfxNok9vwk/1zqvqKqiaqamKZ0mVONvpJRUVFs23b1qzHycnbqFgxJD3PJ2S5To/l+ufefftNZn7+Ga+/+U5YFnyvLF28iBmfTaNR3RqMHDaY+fO+ZvTIYfy2eTOtm8XTqG4Ntidvo13LJHbtzH2N7Ez35vSVtBjxPOdf83/s/fMwW3bspUrFkix78zp+nHwzUWWKsfj1ayhXqggAUWWKMemRwYx8cDKbkv8IWI6gFFUROQ/oAPyfiGwGbgEG+HXzHvMb3QdE+j1eCHTN3iWc+dbAo+6aY5yq1lTV104h0t8X08NMYlISGzduYPOmTaSmpvLhpIl0697z5C+0XJYrD+fKNHvWTJ54fByTPppC4cKFvY4TVu554BHWbdjCmh9+4f/efJfWbdvz1nsf8vOWHaz54RfW/PALFaOimbtwOeXKl/c6rmfKlDgXgErlitOrbSzvzlxFle6PEtP3cWL6Pk7ynj9pPuJ5dv1xkOJFzuHj8Zdyz8tfsPi73wKaI1hrqn2Bt1S1iqpWVdVKwCag1Sm89h7gd+CFHJ6bBYxwt9ciIlEiUjaH8fK5GQAuARao6n5gr4i0docPBeblNty9fwAoegqZ/7XIyEiefPo5enTrQlyDuvTp1596sbGhmLTlslwhNXzoJXRo24INP/9E7eqVePN/r/GfMddx8OABel7YmeZJjbn+mtGeZPN36ZBBtGvdnJ9/+okaVaN54/VTWX4/e3ndXu8/cgnfvnMDkx8bypgJU9l34Giu447u04wa0edx+/D2LHnjWpa8cW1WUf63JKdtLf/6TUXmAmNVdabfsOuBujjdsjerand3+HPAClV9w12rTcQpqq8De1T1VhE5qKqZhfQGYKT7tgeBIar6S7bpHwSeBC4E9gMDVHWPiMQBLwGFgV+By1R17wmG9wEeAY4AzbNvV82UkJCoC5eu+MftZUxe5MsI3w6giHzh2X18JDUwO8MEWqECgdnzNdBKtr3T6wg5OrrokZWqmpjTc0EpqmcbK6rmbGRF9fRZUT09ebGo2hmVjDHGmACxomqMMcYEiBVVY4wxJkCsqBpjjDEBYkXVGGOMCRArqsYYY0yAWFE1xhhjAsSKqjHGGBMgVlSNMcaYALGiaowxxgSIFVVjjDEmQKyoGmOMMQFiRdUYY4wJELtKTQCIyB5gS4DerjSQEqD3CrRwzWa5To/lOj2W6/ScDbmqqGqZnJ6wohpmRGRFbpcU8lq4ZrNcp8dynR7LdXrO9lzW/WuMMcYEiBVVY4wxJkCsqIafV7wOcALhms1ynR7LdXos1+k5q3PZNlVjjDEmQGxN1RhjjAkQK6rGGGNMgFhRNSckIuJ1BmOM+adEJJ/7NyS/ZVZUTa5ERNTd6C4i54tIlNeZ/IVjwQ/HTOFORIqEW7uJSKTXGXLi304iUsDLLLkJp89SREoCxdyHcaGYphXVMJE5I4pIIREp7HUeAL+C2hK4DTjgbaK/ZCv4PUSkhMeRsmfqICL1vM6USURK+d2v42UWfyJSC3ibEP3gnQoRKQ4kuffPD5fPMdv8dSlwSeZaWLgQkdrApWFU8FsDt4nI/cDboViAC6sP5GymqioivYCpwJciMlREzvU6l4j0Bd4DnlbVP8Ply+K/Bg38B4jwNtFxmcYAjwFHPA3kcn94O4jIMyIyGudHptjJXhcKqroB2ATcLiINvc7jKge0EZEpwLME7hSk/4rf/JUI9AQmq2qGt6mOWyFoDTwCXAv0FpGCngYDVHUq0AS4AbhaVQ9qkA95saIaJkQkBrgGuBN4GBgKDHafC1l3Sg7TmgL8CYwBUNVUEfG8gEHWGvT/gCdU9fdw+BKLSAecz62Vqm4SkcYi0tHLTKqaoaqTcZbaHwHudReQ8nuVSRz53Hw34RTWe70srJnzvqr+DETh/Bh/ABzzKlN2IpIA/B9Or1FYLLS5KwStgReB14DVQBtggBcL4Tn8hj2D8zkOcX9ngzt9O07VeyJSE3gU5/Po6w5rDkwCBqnqwhDl8O9e6gL4gO+BPThflG9VdZj7fISq+kKRK6d87uNzcdqooqrGe5Erh0y1gNuBfUAG0BQ4DLyhqhNDlSt7NneB42GgNs5C0nBVTQ9lnlxynaeqv7v3HwTq4xT9tR5mGgZ0Ar4CagF/AO+p6nYRKQ38Huy1nZxy+Q27AhgG3AisCFWWExGRW4EiqnqPO69dDlyEswDwiaqmhiiH/+fYB4gEDqnqdBEZD1TAabeLgaOq+magM9iaqkeyLU1tBr4FSrrbcAqr6mJgIs6VFULCb2a8GacwXICzvasWEA/UF5FP3XE9K6gi0klEugMlgSHAahGZkllQQ7UmnS3TVSJyMU4BnQNUBD7E6ab7BufLHTLZstUFSqvqzaraE6er/H33uXbuAlTI+OW6FnhKRB4RkRqqejewBrhbROJDlUdEivllaoYz31+lqv8DlgLRQF8RuQO4BwhZj4hfrpEicrf73XwbZ966B0gMk+2qPwJNRaSeqh5T1Rfc4c2AkG3D92uv64FbgUbACBF5Q1VvAXYDTwK3AKuCFcJuIb7xVw9BK2AQ0NN9fDPOqbTuADrgbMtpFeJsNXGWLMFZs5kCFHQf5wfm4xQM8ajt/gPMA54D5gLNgSLAq+7jfB5kuhZYDtTM4bnB7pc3xqP2uhGnqM8BXsZZkC4KfIyzILcKqOFBrivcXNHARpzuudbuc+NxCkfBEOSoAfwXOAdnIe0TYAXQwm+cbsDdwGKgoQdtdQPwpfubsBa4yR1+m/t9jA9xnszfrwSgI1AZKICzaeFOnG7z2sAM4CPg7hDnO8f9HGu6jwsA7wK3uY9jgLJBm36oZxC7ZX3wXYDv3MI1B/jYHX49sAx4GjjfHRa0QpG9OAJVgLfc6U/3K6gXe1GwsmWr7ddOt7lf2nzurRjwPFApxJlKuz94sW5xHwTcBLTF6fqdAzTwqL2GAPPd+48Ah4DX/Z7vC1QNUZYkoBfOWl4Rt3CWd+f3L4GxwCy/wlo6RLmqASVw9j6u6Tf/34TfwoY7j50bokz5sk33SUBwFro/c9swwn3+2lDO834FtSvwM3AXsAtIdIvsbcASnDX8GKAfMC4zbzAz+T0ujLPg3c1vWFecfS+C30ah+jDsdtyHns/94g7wGzYDeN69fy/OxvWWQP4g5hC/+wOAujhLdc/hrMVEu8+NBFYSxKW7U8haGmcN6xngdeDzzLbBKfjFsn+5gt1m7uNIt70m4SyVv4bTNXc9zpp9iRC2UfZsDXHWIq7B6XEoDvyCs5YaGeLP7xKchcVu7uMCOGuJs/zG+QVn34LCoWwrd7562P3squJ0V76Ns5ZfK5TtlC1jF/czm4yz4DEZKOA+Nxp3oduDXLG4PRxuxl043b8d3efL4Kz1dwHWA/VD9Dk2x1nwLobThf8Lbo8DTs/IFJy12KD+Tngys5xtN/cHpI57v6o70z0D9Pcbpwrwpns/PzABZwkvFD8w1+CsNVd3H5+Ps+bwNc4azvdArIftF+fmKeW223zctQjgMpwusfIhyOH/Be6Fs1Zax83XF6jmPjfa/QEMWeHKlq04UMy9n7kA19V9/ADOmkSZEOXyX+t6EFgA9HMfV3Z/jDO3Y34ERIWyrfyGVcLpunzJ/Y7WBj51vxsh+Rxx9lvIXFMvCixx73d0C0Rmuw0Dfsj8voYgVw2cBddefsNq4yz0r3Af3wakAh3cx8VwNjeE5HcDuA5nE8ybOJuBOgMXAsnACzjFvV5IsoRiImfzDTgPZ/fyK3COXVwNnItzyMyOzJkOaO/+4JRxH0cG64cv2w9wA5y10Cj3cVucbTeZ3XV98WCbW7a8SThrzjE4Ox68jLPN5HlgXai+uH55rgIWAVcCaUDLzHbFKfLfheoLnEO2m91isBpne240zk4Zz+P0gHyM2wMR4lxX4+wc9S6wFbjIHT4aWOjOg0H/HP0LEc5hYv/D2eGnHE5vyD1uW1XH6Q6uGKL2iXTnq6/85qdlbnE6x/0e/gq8Eaq2cjPUxlmonuDO86P9nhsJvODeb+tmb+b3fNB62bJlrOW2SXmcrt9O7u9FLZyF3nqEsos8VBM6G284a6jXAn1wugiP4RwukPn8DThLnM/gLElldo2FbNslToF/FmcJ/SWctcBP8VuL9rD9yme2hfsDONH98anufnH64a4dBjmH/0JIWbd9igMjcLaZRvg9N4EQFlSc7VhNcLYLdvD7Ie6Ps/AxHGfb7g3AtFD9GPu3Hc6azorMHzacNfzlQF+/dgv6pgWcBdyfcLYDNnHn9SE4C7tTcQ63OA+nV+QJQt9Ffh4wyv2cOgKPusMz569aODsJhmQzjFuMVgE93MdDcAp/nPu4Nc4OZk+7n2eTzM88yLny+U/H/T34LNs49wEjQ/n5Zd7sONUgE5FrcLotP8DptlGcNZmPVDXNPYHBnzhf4FU5HZcWpFxdgDtUta2I9MDpevpEVde6p/Q6R1VvC1WeHPI1xVkSVpy1r2I4XTwvq+qvIczhf2jKIJyi1RdnLaYSzg9Omoj8B2c73B8aouM/RaQbTvf8EzjFvSXQXVWHus93xvnB66aqv4pIflVNC0Gu7MfuFsDplnsBWKyq6SLyAM42y36qOjMEmbrjbAtcjtNeu4Bn1Tl+sQxON2+c+/cIzg93SghyZW+rYjjFazjOzj/v4JyIYifOCR+u0RAdziYirYBvVDXzhPRrcbpTK+CsCV6B08PWCqerOuifo5tDVFVFpDKwU50T0nwK7Ne/jqN/GOcIm7tCkek4XlTys+mGs6Q+A+e4N3DWGJ7C6e9vhdP9FfQ1U/6+E0s+nA33H2YbfgkeHALC8WuD+XG6vOrjrPl9gdN9uAZ4zqPPsRfOziI1cNbov+ev7Zb93GxVQ5inLc6hKE39hsXhFC//Ya/jHpaVfR4IwedYlb82K4zDOSylqvv4QpwFzVBsQ+3ufj6Za8ZROGvOr/qNcx7OHskTCeKeqidoq444C0Xl3MdX4WwbHIHThV8Vbw596orT7fwlcI87rADOnr//ye3/CVKWjkAn9/51OJs43nF/I0rgbJOf5c5na3D3Ywl5m3kx0TP55haDou79yu7fmjhdTGVx1rhuxDml1278dvsOUb7auIcG4JwEYBLwufu4rvtD58khIG6Gq9wvyqtAW3dYB+BSYANON3nJEGdKxFkTzFwwKuv+yLzh/givIIh7OOaS6SbgBvd+pPu3OM6a62M4a/fDcXZwCck21GxF4iac7d3T3CyF3PZ6253nVhCCHW1wNiF8DSS5jzPn/dY4p0a81m/cUoToUJ5sGa/F6QG5253HM383RuN0UbcMdaZs+ToC6Ry/09nl2YtqCHIMwjlL2fU4vR4xOAver+HsjJcPZz+Hy/DouHBVDe1ZXs4S7YHaIrIX+K+ItME5ZZ0PZw1imoi8jLN331Oq+lMouljds65UwtlZ5DkRmaSqh91Tsn0tItNVtbuIjFDVg8HMcoKMF+GskQ7E6QLrKSKVVPUd9/mlQKqq7g1yjlo4ay6FcXaASMYpTgNEZJGqrnFPgdYAp1j8pKq/BTOTX7bMeaUasN8d7BORfKq63z0V2zU4PziCc2KRbaHIljkPu1338ThriAVxdk6KUNXhItIY54dwkYamG/8Yzs5kR0XkHOAWEWmH0/27FefiAmVU9V5V/SMEeY4jIp1wikV7nM+tILBWROJV9SUR8QEhmbdyo6pzRKQnztppTXFOq3oLTnELCXe+f19EjuEsdH+kqj+Kc4m+G3F6ZBJU9eVQZcqVl0tAZ+INZ2npa5ztH339hrfD2Z6TEMIsOR060ApnretS/lpqvwPnYOmgd8Vly9IG91AP9/GtwJ3u/QicNa2PCcFhRX4ZuuFsL/oEZ210M04RiAbux9mpK6Rrpbnk7ODmS/Cb7zLXWG/E6ZEI+hmJss9vOHtnr8X5kcvcwaYK7glNPGgnwTkL1yxgG87a8kicrtaxOGvRWXvdhyJPtselcLZRDgO+dIe9jXN4SkhPZHIK2S/AOQ3nOuACDz7HzH2A+uCssbb1e/494GKv20hV7dy/QVAX54v7DdBYRKq4O4jMxemW6y4huqSbZs6NIpeJyKsi8hB/XXFmGDBaRO7DOWSlr6omhyKXn3LAWyJygft4Hc75Qxuqqk9V38A5+06NUIRxc9wN3KiqF6lqJ5xDLqbj9Cw8D6QAN4v31yRdilMMBohIgjpXokkXkYE4h2sdUdWgX13F/xzW6liDs+BRE+ezzK+qW3BOLtJQRMplO+91ULnfgZdx9ga9CbhSVf9PnYtUVMTZ9tZaVfcEO0u2nd7qiEhdVf1DVXfg9CJ94o46F2c/As+vuuRPnR2RegB3aYh2SvKbdma7iap+hDOPfykid7lr0bE4O4B6zvb+DSARqY5zSrErcbaXvo/TzfRfnO1yLXC+yI+q6vYgZcjcMy7z7xVunvHutEfg7La/H+dk742AR1Q1ZDOkOJevKqKq89yT0D+O0+27EOcg8gicY+Ly4Rxb2SXYP3riXMQ7Bae7dLqInKOqR93n7sfZgashzmENXYC3VXVnMDOdjIhE4Wzb6ojTC3IUZ8/kvqr6fYiz9MVZ61qqTvf4lThrFA+4w9JEJFI9ujJOdiLSD+eiEQNUdWOIp30TTo9IGs5C7miczR31cT7DOJxD2jydv04k2Juscnt/dzOWur9t/XC2z7+E8xsWks0cJ+X1qnJevuHsBJGE8+NfE2cvucf9ni+Cs+PP/wHbcX6M2wY5U9Vs078BuNBvWG+cvX4zu369OAH9TThrWZlnj+mHs82yDc62wtE4e0x/CDQKYa5uOEu757mPC/o9NxdonH241zecbbqtcNbErgRqh2i6hf3uj8HZoeZenE0fV7rDR+KsUTf3up38slZw867Dg258nLOVzXDvP4R7mkacBZJBOL1Znm9e8Pgz8t/hrSvOMdd1gEKZz/PXCuGFeLSXb675vQ6Ql2/uj4b/h/0QzskcKvuNUwhnDTXOf4Yhh+2dAcjTHWdngnNw1kYfwen6+tRvnHI42x9KeNBe/l+Wa4DZQBv3cX+3sGaeP7QQIdyW6perq5ujpPs48/zCU/DgCiXheHMXPp7COTSlKTDJHX4zzvboV3DPvIOz7b6yV1lzyF7Izf+3KwqFaPrxOL1FD+Fs582cv8JmwcPDzyb79uabcDajjXUXzjr5jxuM39CA/B9eB8jrN5zDK14H2ruPH3NngBxPbxasGYG/Tl5dxy2cH/s9Nxdn22AEzp61C/H25PiZZ9YZna2w9sHZU7qzx59p9sJ6qfuZetZm4XLjr2M+e7uPC7nFtZs7n0XirDWvw11jPVtvORQJwdmEsAznghCZC9gjcdb0S4ZroQhRe5V2/2b2/E10H48BZrrD84d7G9khNacpcycLdT9tVd0tItuAQSKSps5ZiB4BZotIZ82280/m6wKcqTPOcVoLcPa0fBQoIyKxqroO54fwE3ecmsDlqro70DlOMWtD4FYRmazOIQOCc1HqB1T1IxFJw+lG94yqzhDnAtrfiMgLODtFeNZm4UJEyuPsSTtSVZeLSCGcvTAFZwe9L9TZWepXnDWMT3J/tzNf5nddRG7AKaabcU5J+gzOAuXVIlIOZ5PMIA3yoWLhyv0NKANsEpFBqjpVRP4A9ojIezgLG91VNUNE+uNsYtjlYeQTsqJ6GkSkoLp7VIpIC5wZYbGq3iMiN+NcYV5V9Q4RyY9zFpSg7lErIh1xzit8I8423nic7YJVgSQR2a/OBvzz3b2OI1T1z2Bm8st23AKIe3+tiKwCuohIhqq+KCIKTBCRMao6NRTZTsYtrBE4h/Q0dhdOznbZj/m8DWd7bjrONsFE9xjGFjg7fJ2VCyEiUlhVD7v3W+HsPPYyTmF9A6fnYzfOjkkRODuW/exN2vDgrpyMAP4nIsPVOZ7/ENAYGOUurA3D2cQw39OwJ2F7/54iESmBcyL10TjdEFNxThP3GzDdXbq6CWfHpRdV9ZsQ5UrC2S6zSETq4nTvHsE5bdd5OGuvc9U5rCGk/Pf2FOdcw2VV9W338fU4ezl+oKoz3S/UlxqikyicKv8fyLOdu5B0E85ltWJxjpNdgLMfQeZltg4DCzTEe9SGC3HOx3w+zmkZ2+Ds1T5O/zrH8HU4xXRMuM3r4cA9rO0DnNOC/oCzKSEK2IOz/b5/uC/gWlE9DSJyG39dguledc7ocSXOmXW+cAvrrcBMVV0b4mz53O6ROjiHfxzCOW1dDZwFgEkaohNxu3nOx9khYw3OzlPgnID7dVX90B3ndZyl97tVdUaospl/TkSK4MzvlYApfj03bwJT1TmG8Kwkzkn7H8Y5R+4UEakEfIZz9qjR7jjn4azhV8P5nvpUNcOrzF7K/M1y718CVFDVCe7hWa/jHLWwwO0VPA9YkxcWROzkD6fArxvzMZxjonrgXG4InMM+1gK9ROQiVR0X6oLqZstw//6Es3fvOThddT8AX4e4oF6A8+OyCOfSchcAB3G+KEPd7SLgHOD+I86pAE0eoKoHVXWxqn7gV1D74ax9rfY0nIeybW+eIiLnqupWnDXVLu42elT1d5y9Wa9S1bSzuKA2Aj6Tv06EEwX8DqCqk3HOpjbF/U1dpKrT8kJBBdumelJ+J1FohXNpoddEpDTwmIjsUtWVIjIZZ9tIWGwXUed8wpNxTu7wqvtFDgn3JAqfA73c7SKVcbrCCgKT3dEeEJFeOF2Ifc/WbW95nYhUwDlT0hU4J1H4xeNIXgrrcwyHG3VOEJIOTBLnnN8lcLp4M5//2F2ZeUZEZgOH88oCiHX/ngIR6YpzirphqjrfHXY1TvfmNaq6NJzOFpNJQnT9zBym2w2nkDZX1T9F5F2c6zK+7D4fi3OVkNln+Q9xnubu/dsB54ICZ+U21Ewn2d7cA+dsXb2BizQEp0QMV2475cvsORORj3DOc/yL+/cHnAUUcA79O6yqR7zI+k9ZUT0JcU4F9zlwhaouc7stiuFcoulinO6dFsDBvLIkFQrugsgzOAe4VwQGq+qRzDV/b9MZE3gn2N78Fs5J8r88m+d9/+++iERlHm4ozlW7rsDZQzpzX5BCwB15pcvXnxXVbESkHs71RCe5j4vjnGT9MM4hNLVwjst7X1X/T0SqqeomzwKHMXEua/UFUN7dZT7rfLrGnA3Ew3MMh5NsBfVaYDDOvhQvqOp6EXke56QwPd1xCqhqqneJ/znbUcmPiNTGufZj1lVkVHU/zpmKCuCc7q8TzrbBBPd5K6i5UNUvcc6087WIlLWCas4WIlJBRMbgHBIy7GwuqHDciTB642wyuBZn5eRKEWmuqtcA+UQkc1tqWG1KOx22o5LLPRRlOjBZVV93hxVy+/PfBjJU1SfOBZivxNkt3pyEexKFAsBMEUnEvcKE17mMCbJ9OJuIep3tBTWTiMTgXLXoPXcHz19xLvgxUEQiVLW7iFR0fx/y7G+EramS1eX7Ds5pxPaLSEsAdxtgdeBVoLw4p9i7Eec4tFmZh9qYE1PVKTjn982wgmrOBqp6RFU/O5sLqjinYPT3J84Zyga7a6d7gQk4Oyj1dDcPBeWSmKF01m9Tdfdg/Bzn8mzTcY41K4Bz9qQVOFcnWaSqD7vjV1DVHbbDjTHG5MxdK12PczWjH1T1VXf4OThnpeuIcw3Uxe4OXueoaopXeQPprC+q4By4re4Fgd1u4ME4XeMrgJ9V9Xv/s38YY4zJnXs2qYk4Z3PrgHO87gc4J6I5JCLX4BzjfKuqLvEuaeBZ9y/gV1DzuWckehtnQ3kjnIOSsYJqjDGnxj2b1DKcC3x0A2bgHDbzuYgk4Jy+9DmCfMERL1hR9eN3qr8NOIX1HJy+/pKeBjPGmDzCb1+T23B2OCoN7MA5hvdH4E6cC3/McovvGcW6f09ARGpBVpE1xhhzCtzCWgDnGP/qOGust6vqp+6hi3v0DL1+rBVVY4wxQeHuozIfeFZVH/Q6TyhY968xxpigcPdRuQ2IEJHCXucJBSuqxhhjgmkx7hnozgbW/WuMMSaoRKSwqh72OkcoWFE1xhhjAsS6f40xxpgAsaJqjDHGBIgVVWOMMSZArKgacxYSkXYiMt2931NEbj/BuCVE5Op/MI37ROTmUx2ebZw3RKTvaUyrqoh8f7oZjQk0K6rGnEFEJOJ0X6OqU1V17AlGKQGcdlE15mxkRdWYPMBdE/tRRN4UkbUiMjnzYHoR2Swi94jIAqCfiHQWkcUi8q2IfOheWgsRucB9jwXAxX7vPVxEnnPvlxORT0RkjXtrAYwFaojIahEZ7453i4gsd7Pc7/ded4rITyLyJVDnFP6vK9z3WSMiH2U7QUAnEZkvIj+LSHd3/AgRGe837Sv/bdsaE0hWVI3JO+oAr6hqQ5wLPvuvPR5V1VbAl8BdQCdVjce5fOFN7nUsXwV6AK2B8rlM4xlgnqo2wjlf6zrgduAXVY1T1VtEpDNQC2gCxAEJItLGvfrIQKAxTtFOOoX/6WNVTXKn9wNwud9zVYG2OFc5ecn9Hy4H9qtqkvv+V4hItVOYjjEhEel1AGPMKduqqgvd++8A1wOPu48nuX+bAfWAhe7FQgrgnNEmBtiUeXEIEXkHGJXDNDoAlwKoqg/Yn8NVmjq7t1Xu4yI4RbYo8EnmQf4iMvUU/qf6IvIQThdzEWCW33MfuFeO2iAiv7r/Q2egod/21uLutH8+hWkZE3RWVI3JO7KfqcX/8SH3rwCzVXWQ/4giEpfD6/8pAR5V1ZezTWPMP5jGG0BvVV0jIsOBdn7P5fT/CnCdqvoXX0Sk6mlO15igsO5fY/KOyiLS3L0/CFiQwzhLgJYiUhOc08O5l9r6EagmIjX8Xp+TOcBV7msjRKQYcABnLTTTLGCE37baKBEpC3wDXCQihUSkKE5X88kUBXaISH5gcLbn+olIPjdzdeAnd9pXueMjIrVF5NxTmI4xIWFF1Zi84wdgmIisBUoBL2YfQVX3AMOB993xlgAxqnoUp7v3M3dHpS25TOMGoL2IfAesBGJV9Xec7uTvRWS8qn4BvAcsdsebDBRV1W9xuqFXAx/hXPLrZO4GlgKzcQq/v5+AecAMYLT7P/wfsB741j2E5mWsx82EETv3rzF5gNu9OV1V63udxRiTO1tTNcYYYwLE1lSNMcaYALE1VWOMMSZArKgaY4wxAWJF1RhjjAkQK6rGGGNMgFhRNcYYYwLEiqoxxhgTIP8POYMglac0oWEAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 720x504 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from torchmetrics import ConfusionMatrix\n",
    "from mlxtend.plotting import plot_confusion_matrix\n",
    "\n",
    "# 2. Setup confusion matrix instance and compare predictions to targets\n",
    "confmat = ConfusionMatrix(num_classes=len(class_names), task='multiclass')\n",
    "confmat_tensor = confmat(preds=y_pred_tensor,\n",
    "                         target=test_data.targets)\n",
    "\n",
    "# 3. Plot the confusion matrix\n",
    "fig, ax = plot_confusion_matrix(\n",
    "    conf_mat=confmat_tensor.numpy(), # matplotlib likes working with NumPy \n",
    "    class_names=class_names, # turn the row and column labels into class names\n",
    "    figsize=(10, 7)\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "381c1c93-df30-451c-b65e-5d4c1680dc30",
   "metadata": {},
   "source": [
    "Woah! Doesn't that look good?\n",
    "\n",
    "We can see our model does fairly well since most of the dark squares are down the diagonal from top left to bottom right (and ideal model will have only values in these squares and 0 everywhere else).\n",
    "\n",
    "The model gets most \"confused\" on classes that are similar, for example predicting \"Pullover\" for images that are actually labelled \"Shirt\".\n",
    "\n",
    "And the same for predicting \"Shirt\" for classes that are actually labelled \"T-shirt/top\".\n",
    "\n",
    "This kind of information is often more helpful than a single accuracy metric because it tells use *where* a model is getting things wrong.\n",
    "\n",
    "It also hints at *why* the model may be getting certain things wrong.\n",
    "\n",
    "It's understandable the model sometimes predicts \"Shirt\" for images labelled \"T-shirt/top\".\n",
    "\n",
    "We can use this kind of information to further inspect our models and data to see how it could be improved.\n",
    "\n",
    "> **Exercise:** Use the trained `model_2` to make predictions on the test FashionMNIST dataset. Then plot some predictions where the model was wrong alongside what the label of the image should've been. After visualing these predictions do you think it's more of a modelling error or a data error? As in, could the model do better or are the labels of the data too close to each other (e.g. a \"Shirt\" label is too close to \"T-shirt/top\")?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25818e83-89de-496d-8b56-af4fc9f2acc5",
   "metadata": {},
   "source": [
    "## 11. Save and load best performing model\n",
    "\n",
    "Let's finish this section off by saving and loading in our best performing model.\n",
    "\n",
    "Recall from [notebook 01](https://www.learnpytorch.io/01_pytorch_workflow/#5-saving-and-loading-a-pytorch-model) we can save and load a PyTorch model using a combination of:\n",
    "* `torch.save` - a function to save a whole PyTorch model or a model's `state_dict()`. \n",
    "* `torch.load` - a function to load in a saved PyTorch object.\n",
    "* `torch.nn.Module.load_state_dict()` - a function to load a saved `state_dict()` into an existing model instance.\n",
    "\n",
    "You can see more of these three in the [PyTorch saving and loading models documentation](https://pytorch.org/tutorials/beginner/saving_loading_models.html).\n",
    "\n",
    "For now, let's save our `model_2`'s `state_dict()` then load it back in and evaluate it to make sure the save and load went correctly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "d058e8fa-560f-4350-a154-49593ff403c9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving model to: models/03_pytorch_computer_vision_model_2.pth\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "# Create models directory (if it doesn't already exist), see: https://docs.python.org/3/library/pathlib.html#pathlib.Path.mkdir\n",
    "MODEL_PATH = Path(\"models\")\n",
    "MODEL_PATH.mkdir(parents=True, # create parent directories if needed\n",
    "                 exist_ok=True # if models directory already exists, don't error\n",
    ")\n",
    "\n",
    "# Create model save path\n",
    "MODEL_NAME = \"03_pytorch_computer_vision_model_2.pth\"\n",
    "MODEL_SAVE_PATH = MODEL_PATH / MODEL_NAME\n",
    "\n",
    "# Save the model state dict\n",
    "print(f\"Saving model to: {MODEL_SAVE_PATH}\")\n",
    "torch.save(obj=model_2.state_dict(), # only saving the state_dict() only saves the learned parameters\n",
    "           f=MODEL_SAVE_PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1542284-8132-42ba-b00d-57e9b9037e4e",
   "metadata": {},
   "source": [
    "Now we've got a saved model `state_dict()` we can load it back in using a combination of `load_state_dict()` and `torch.load()`.\n",
    "\n",
    "Since we're using `load_state_dict()`, we'll need to create a new instance of `FashionMNISTModelV2()` with the same input parameters as our saved model `state_dict()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "634a8f7a-3013-4b45-b365-49b286d3c478",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a new instance of FashionMNISTModelV2 (the same class as our saved state_dict())\n",
    "# Note: loading model will error if the shapes here aren't the same as the saved version\n",
    "loaded_model_2 = FashionMNISTModelV2(input_shape=1, \n",
    "                                    hidden_units=10, # try changing this to 128 and seeing what happens \n",
    "                                    output_shape=10) \n",
    "\n",
    "# Load in the saved state_dict()\n",
    "loaded_model_2.load_state_dict(torch.load(f=MODEL_SAVE_PATH))\n",
    "\n",
    "# Send model to GPU\n",
    "loaded_model_2 = loaded_model_2.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "feeaebf4-6040-4fa5-852d-5eb8d2bbb94c",
   "metadata": {},
   "source": [
    "And now we've got a loaded model we can evaluate it with `eval_model()` to make sure its parameters work similarly to `model_2` prior to saving. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "3e3bcd06-d99b-47bc-8828-9e3903285599",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'model_name': 'FashionMNISTModelV2',\n",
       " 'model_loss': 0.3286978006362915,\n",
       " 'model_acc': 88.18889776357827}"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Evaluate loaded model\n",
    "torch.manual_seed(42)\n",
    "\n",
    "loaded_model_2_results = eval_model(\n",
    "    model=loaded_model_2,\n",
    "    data_loader=test_dataloader,\n",
    "    loss_fn=loss_fn, \n",
    "    accuracy_fn=accuracy_fn\n",
    ")\n",
    "\n",
    "loaded_model_2_results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2b37855-c0da-4834-a2d4-a0faa8410b65",
   "metadata": {},
   "source": [
    "Do these results look the same as `model_2_results`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "68544254-c99a-47ec-a32f-9816c21a993e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'model_name': 'FashionMNISTModelV2',\n",
       " 'model_loss': 0.3286978006362915,\n",
       " 'model_acc': 88.18889776357827}"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_2_results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ee07f93-4344-4c7a-8b1d-92a56034e7b2",
   "metadata": {},
   "source": [
    "We can find out if two tensors are close to each other using `torch.isclose()` and passing in a tolerance level of closeness via the parameters `atol` (absolute tolerance) and `rtol` (relative tolerance).\n",
    "\n",
    "If our model's results are close, the output of `torch.isclose()` should be true."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "48dcf0ba-7e00-4406-8aaa-41918856361a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(True)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check to see if results are close to each other (if they are very far away, there may be an error)\n",
    "torch.isclose(torch.tensor(model_2_results[\"model_loss\"]), \n",
    "              torch.tensor(loaded_model_2_results[\"model_loss\"]),\n",
    "              atol=1e-08, # absolute tolerance\n",
    "              rtol=0.0001) # relative tolerance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3969b7d-9955-4b6f-abf8-fe8eedf233a9",
   "metadata": {},
   "source": [
    "## Exercises\n",
    "\n",
    "All of the exercises are focused on practicing the code in the sections above.\n",
    "\n",
    "You should be able to complete them by referencing each section or by following the resource(s) linked.\n",
    "\n",
    "All exercises should be completed using [device-agnostic code](https://pytorch.org/docs/stable/notes/cuda.html#device-agnostic-code).\n",
    "\n",
    "**Resources:**\n",
    "* [Exercise template notebook for 03](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/extras/exercises/03_pytorch_computer_vision_exercises.ipynb)\n",
    "* [Example solutions notebook for 03](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/extras/solutions/03_pytorch_computer_vision_exercise_solutions.ipynb) (try the exercises *before* looking at this)\n",
    "\n",
    "1. What are 3 areas in industry where computer vision is currently being used?\n",
    "2. Search \"what is overfitting in machine learning\" and write down a sentence about what you find. \n",
    "3. Search \"ways to prevent overfitting in machine learning\", write down 3 of the things you find and a sentence about each. **Note:** there are lots of these, so don't worry too much about all of them, just pick 3 and start with those.\n",
    "4. Spend 20-minutes reading and clicking through the [CNN Explainer website](https://poloclub.github.io/cnn-explainer/).\n",
    "    * Upload your own example image using the \"upload\" button and see what happens in each layer of a CNN as your image passes through it.\n",
    "5. Load the [`torchvision.datasets.MNIST()`](https://pytorch.org/vision/stable/generated/torchvision.datasets.MNIST.html#torchvision.datasets.MNIST) train and test datasets.\n",
    "6. Visualize at least 5 different samples of the MNIST training dataset.\n",
    "7. Turn the MNIST train and test datasets into dataloaders using `torch.utils.data.DataLoader`, set the `batch_size=32`.\n",
    "8. Recreate `model_2` used in this notebook (the same model from the [CNN Explainer website](https://poloclub.github.io/cnn-explainer/), also known as TinyVGG) capable of fitting on the MNIST dataset.\n",
    "9. Train the model you built in exercise 8. on CPU and GPU and see how long it takes on each.\n",
    "10. Make predictions using your trained model and visualize at least 5 of them comparing the prediciton to the target label.\n",
    "11. Plot a confusion matrix comparing your model's predictions to the truth labels.\n",
    "12. Create a random tensor of shape `[1, 3, 64, 64]` and pass it through a `nn.Conv2d()` layer with various hyperparameter settings (these can be any settings you choose), what do you notice if the `kernel_size` parameter goes up and down?\n",
    "13. Use a model similar to the trained `model_2` from this notebook to make predictions on the test [`torchvision.datasets.FashionMNIST`](https://pytorch.org/vision/main/generated/torchvision.datasets.FashionMNIST.html) dataset. \n",
    "    * Then plot some predictions where the model was wrong alongside what the label of the image should've been. \n",
    "    * After visualing these predictions do you think it's more of a modelling error or a data error? \n",
    "    * As in, could the model do better or are the labels of the data too close to each other (e.g. a \"Shirt\" label is too close to \"T-shirt/top\")?\n",
    "\n",
    "## Extra-curriculum\n",
    "* **Watch:** [MIT's Introduction to Deep Computer Vision](https://www.youtube.com/watch?v=iaSUYvmCekI&list=PLtBw6njQRU-rwp5__7C0oIVt26ZgjG9NI&index=3) lecture. This will give you a great intuition behind convolutional neural networks.\n",
    "* Spend 10-minutes clicking thorugh the different options of the [PyTorch vision library](https://pytorch.org/vision/stable/index.html), what different modules are available?\n",
    "* Lookup \"most common convolutional neural networks\", what architectures do you find? Are any of them contained within the [`torchvision.models`](https://pytorch.org/vision/stable/models.html) library? What do you think you could do with these?\n",
    "* For a large number of pretrained PyTorch computer vision models as well as many different extensions to PyTorch's computer vision functionalities check out the [PyTorch Image Models library `timm`](https://github.com/rwightman/pytorch-image-models/) (Torch Image Models) by Ross Wightman."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.13",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "3fbe1355223f7b2ffc113ba3ade6a2b520cadace5d5ec3e828c83ce02eb221bf"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}