shlomo_dl_0003_pytorch_ffn_mnist.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "shlomo_dl_0003_pytorch_ffn_mnist.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/deeponcology/PyTorchMedicalAI/blob/master/shlomo_dl_0003_pytorch_ffn_mnist.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "-SmJrhTsGo4K",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Deep Learning in Medical AI 2018/2019 using PyTorch & Google Collab.\n",
        "\n",
        "\n",
        "<table align=\"left\"><td>\n",
        "<a target=\"_blank\" href=\"https://colab.research.google.com/drive/1JEIeD_445sFvcjSrITB5Z_oW8VHRS_kA\">\n",
        "    <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>  \n",
        "</td><td>\n",
        "<a target=\"_blank\" href=\"https://github.com/deeponcology/PyTorchMedicalAI/blob/master/shlomo_dl_0001_cuda_collab_pytorch.ipynb\"><img width=32px src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a></td></table>\n",
        "\n",
        "<img src=\"https://github.com/deeponcology/PyTorchMedicalAI/raw/master/assets/tumor_visdom.jpg\" align=\"center\" width=30%>\n",
        "\n",
        "### Author: \n",
        "***Shlomo Kashani***, Head of AI at www.DeepOncology.AI, shlomo@deeponcology.ai \n",
        "\n",
        "<img src=\"https://github.com/deeponcology/PyTorchMedicalAI/raw/master/assets/line-up-small.png\" align=\"center\" width=30%>\n",
        "\n",
        "\n",
        "### Synopsys:\n",
        "This is the hands-on deep learning tutorial series for the 2018/2019 Medical AI course. The series will guide you through the most basic building blocks such as installing CUDA to training advanced CNN's such as SeNet. \n",
        "\n",
        "### DataSets:\n",
        "We foster the use of Medical Data Sets (https://grand-challenge.org/All_Challenges/) and predominantly those available (but not only) via Kaggle.\n",
        "\n",
        "### About PyTorch:\n",
        "\n",
        "PyTorch is an open source library for numerical computation using  computation graphs. Nodes in the graph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) communicated between them. \n",
        "\n",
        "\n",
        "Similar to python programming, we can add and execute a node to the computation graph immediately. This property makes it easy to debug the code and inspect the values in the network.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "cqAEW3UrfWBA",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "iqwkKJcGf4oR",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "tCz9v0DaoXno",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Collab notebook: 0003 PyTorch MNIST FFN\n",
        "\n",
        "This Jupyter notebook explains various approaches for implementing neural networks that recognize digits on MNIST dataset.\n"
      ]
    },
    {
      "metadata": {
        "id": "jUdrKNcTHaqe",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Install PyTorch"
      ]
    },
    {
      "metadata": {
        "id": "ed-8FUn2GqQ4",
        "colab_type": "code",
        "outputId": "e0dd35c6-139b-4a13-8f58-64c093a6c0f8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 462
        }
      },
      "cell_type": "code",
      "source": [
        "import sys\n",
        "sys.version\n",
        "\n",
        "%reset -f\n",
        "import os\n",
        "os.environ['PATH'] += ':/usr/local/cuda/bin'\n",
        "import sys\n",
        "sys.version\n",
        "\n",
        "# !pip3 install torch==0.4\n",
        "# !pip3 install torchvision\n",
        "\n",
        "!pip3 install 'torch==0.4.0'\n",
        "!pip3 install 'torchvision==0.2.1'\n",
        "!pip3 install --no-cache-dir -I 'pillow==5.1.0'\n",
        "\n",
        "# Restart Kernel\n",
        "# This workaround is needed to properly upgrade PIL on Google Colab.\n",
        "import os\n",
        "os._exit(0)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting torch==0.4.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/69/43/380514bd9663f1bf708abeb359b8b48d3fabb1c8e95bb3427a980a064c57/torch-0.4.0-cp36-cp36m-manylinux1_x86_64.whl (484.0MB)\n",
            "\u001b[K    100% |████████████████████████████████| 484.0MB 35kB/s \n",
            "tcmalloc: large alloc 1073750016 bytes == 0x5b242000 @  0x7f39c522c2a4 0x591a07 0x5b5d56 0x502e9a 0x506859 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x504c28 0x502540 0x502f3d 0x507641\n",
            "\u001b[?25hInstalling collected packages: torch\n",
            "Successfully installed torch-0.4.0\n",
            "Collecting torchvision==0.2.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ca/0d/f00b2885711e08bd71242ebe7b96561e6f6d01fdb4b9dcf4d37e2e13c5e1/torchvision-0.2.1-py2.py3-none-any.whl (54kB)\n",
            "\u001b[K    100% |████████████████████████████████| 61kB 2.9MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision==0.2.1) (1.14.6)\n",
            "Collecting pillow>=4.1.1 (from torchvision==0.2.1)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/62/94/5430ebaa83f91cc7a9f687ff5238e26164a779cca2ef9903232268b0a318/Pillow-5.3.0-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n",
            "\u001b[K    100% |████████████████████████████████| 2.0MB 11.8MB/s \n",
            "\u001b[?25hRequirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (from torchvision==0.2.1) (0.4.0)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision==0.2.1) (1.11.0)\n",
            "Installing collected packages: pillow, torchvision\n",
            "  Found existing installation: Pillow 4.0.0\n",
            "    Uninstalling Pillow-4.0.0:\n",
            "      Successfully uninstalled Pillow-4.0.0\n",
            "Successfully installed pillow-5.3.0 torchvision-0.2.1\n",
            "Collecting pillow==5.1.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/5f/4b/8b54ab9d37b93998c81b364557dff9f61972c0f650efa0ceaf470b392740/Pillow-5.1.0-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n",
            "\u001b[K    100% |████████████████████████████████| 2.0MB 60.8MB/s \n",
            "\u001b[?25hInstalling collected packages: pillow\n",
            "Successfully installed pillow-5.3.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "W-LqHiVsHGrZ",
        "colab_type": "code",
        "outputId": "b94ba5f8-cb13-4fbf-ad11-8f2585d65f75",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "cell_type": "code",
      "source": [
        "!pip3 install torchvision"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: torchvision in /usr/local/lib/python3.6/dist-packages (0.2.1)\n",
            "Requirement already satisfied: pillow>=4.1.1 in /usr/local/lib/python3.6/dist-packages (from torchvision) (5.3.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n",
            "Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (from torchvision) (0.4.0)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "vxgCzP8pHqYL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Import PyTorch"
      ]
    },
    {
      "metadata": {
        "id": "tqgh6bI9HgiH",
        "colab_type": "code",
        "outputId": "2ecc2633-a290-4044-fe81-24c4a1f26eea",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 289
        }
      },
      "cell_type": "code",
      "source": [
        "import sys\n",
        "print('__Python VERSION:', sys.version)\n",
        "print('__pyTorch VERSION:', torch.__version__)\n",
        "print('__CUDA VERSION')\n",
        "from subprocess import call\n",
        "# call([\"nvcc\", \"--version\"]) does not work\n",
        "! nvcc --version\n",
        "print('__CUDNN VERSION:', torch.backends.cudnn.version())\n",
        "print('__Number CUDA Devices:', torch.cuda.device_count())\n",
        "print('__Devices')\n",
        "# call([\"nvidia-smi\", \"--format=csv\", \"--query-gpu=index,name,driver_version,memory.total,memory.used,memory.free\"])\n",
        "print('Active CUDA Device: GPU', torch.cuda.current_device())\n",
        "\n",
        "print ('Available devices ', torch.cuda.device_count())\n",
        "print ('Current cuda device ', torch.cuda.current_device())\n",
        "\n",
        "use_cuda = torch.cuda.is_available()\n",
        "# use_cuda = False\n",
        "import random \n",
        "import numpy as np \n",
        "\n",
        "print(\"USE CUDA=\" + str (use_cuda))\n",
        "FloatTensor = torch.cuda.FloatTensor if use_cuda else torch.FloatTensor\n",
        "LongTensor = torch.cuda.LongTensor if use_cuda else torch.LongTensor\n",
        "Tensor = FloatTensor\n",
        "\n",
        "manualSeed = 2222\n",
        "def fixSeed(seed):\n",
        "    random.seed(seed)\n",
        "    np.random.seed(seed)\n",
        "    torch.manual_seed(seed)\n",
        "    if use_cuda:\n",
        "        torch.cuda.manual_seed(seed)\n",
        "        torch.cuda.manual_seed_all(seed)\n",
        "\n",
        "\n",
        "if manualSeed is None:\n",
        "        manualSeed = 999\n",
        "fixSeed(manualSeed)\n",
        "\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torchvision.datasets as dsets\n",
        "import torchvision.transforms as transforms\n",
        "from torch.autograd import Variable\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import torch\n",
        "import torch.nn.functional as F \n",
        "print(torch.__version__)\n",
        "import torchvision.transforms as transforms\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "__Python VERSION: 3.6.7 (default, Oct 22 2018, 11:32:17) \n",
            "[GCC 8.2.0]\n",
            "__pyTorch VERSION: 0.4.0\n",
            "__CUDA VERSION\n",
            "nvcc: NVIDIA (R) Cuda compiler driver\n",
            "Copyright (c) 2005-2018 NVIDIA Corporation\n",
            "Built on Tue_Jun_12_23:07:04_CDT_2018\n",
            "Cuda compilation tools, release 9.2, V9.2.148\n",
            "__CUDNN VERSION: 7102\n",
            "__Number CUDA Devices: 1\n",
            "__Devices\n",
            "Active CUDA Device: GPU 0\n",
            "Available devices  1\n",
            "Current cuda device  0\n",
            "USE CUDA=True\n",
            "0.4.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "CgFsm9fPHv-W",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Initialize Hyper-parameters"
      ]
    },
    {
      "metadata": {
        "id": "-YeEfyh5Hpq5",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "input_size    = 784   # The image size = 28 x 28 = 784\n",
        "hidden_size   = 500   # The number of nodes at the hidden layer\n",
        "num_classes   = 10    # The number of output classes. In this case, from 0 to 9\n",
        "num_epochs    = 5     # The number of times entire dataset is trained\n",
        "batch_size    = 100   # The size of input data took for one iteration\n",
        "learning_rate = 1e-3  # The speed of convergence\n",
        "\n",
        "from torch.utils.data import DataLoader\n",
        "from torchvision.datasets import MNIST\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "XsLqpXjxILFn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Download MNIST Dataset\n",
        "\n",
        "MNIST is a huge database of handwritten digits (i.e. 0 to 9) that is often used in image classification."
      ]
    },
    {
      "metadata": {
        "id": "sRuFHe7GIDdq",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "train_dataset = dsets.MNIST(root='./data',\n",
        "                           train=True,\n",
        "                           transform=transforms.ToTensor(),\n",
        "                           download=True)\n",
        "\n",
        "test_dataset = dsets.MNIST(root='./data',\n",
        "                           train=False,\n",
        "                           transform=transforms.ToTensor())\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "7rEhTJBjg0TX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# View the data (EDA ...) "
      ]
    },
    {
      "metadata": {
        "id": "PbxTtKTqg4po",
        "colab_type": "code",
        "outputId": "e797b7e4-9584-4f37-d071-c07d0a0d7a7a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 415
        }
      },
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Also an option \n",
        "from torch.utils.data import DataLoader\n",
        "from torchvision.datasets import MNIST\n",
        "\n",
        "mnist_data = MNIST('./data', train=True, download=True, transform=transforms.ToTensor())\n",
        "data_loader = DataLoader(mnist_data,\n",
        "                         batch_size=4,\n",
        "                         shuffle=False)\n",
        "\n",
        "data_iter = iter(data_loader)\n",
        "print (data_iter)\n",
        "\n",
        "images, labels = data_iter.next()\n",
        "\n",
        "npimg = images[0].numpy()\n",
        "print (npimg.shape)\n",
        "npimg = npimg.reshape((28, 28)) # Why do we do this ... ? \n",
        "print (npimg.shape)\n",
        "\n",
        "plt.imshow(npimg, cmap='gray')\n",
        "print('Label:', labels[0].item())\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<torch.utils.data.dataloader._DataLoaderIter object at 0x7f863251d7b8>\n",
            "(1, 28, 28)\n",
            "(28, 28)\n",
            "Label: 5\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUsAAAFKCAYAAACU6307AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAFNNJREFUeJzt3X1sU+X7x/FPt7lABb5jk00xPhLU\nyUaMCehQ0AFqZjQyNMHNgUaiGB0BiZplAj4QeRiIcWLCQCEqgTRZTESj2UR8io4aUAnDxKF/mEnm\nLDhwuKFs9PeHsT8H3Xq169qe4/uVLLF3r97nvjzbh7an59QTDAaDAgAMKC3ZCwAAJyAsAcCAsAQA\nA8ISAAwISwAwICwBwCKYAJLC/hw4cKDf+5z648ae3NoXPTnnJ1F9DcSTiM9ZejyesOPBYLDf+5zK\njT1J7uyLnpwjUX0NFIcZsU66cuVK7d+/Xx6PR9XV1Zo4cWKsUwFAyospLL/66iv99NNP8vl8+vHH\nH1VdXS2fzxfvtQFAyojpAE9TU5NmzpwpSRo3bpyOHz+uEydOxHVhAJBKYnpmeeTIEU2YMCF0Ozs7\nW4FAQCNGjAhbf+DAARUUFIS9LwFvmSacG3uS3NkXPTlHsvuK+T3Lf4vURGFhYb+Pc9ub0W7sSXJn\nX/TkHKlwgCeml+G5ubk6cuRI6Pavv/6qMWPGxDIVADhCTGF5ww03qKGhQZJ08OBB5ebm9vsSHADc\nIKaX4ddee60mTJige++9Vx6PR88880y81wUAKYUPpceZG3uS3NkXPTmHY9+zBID/GsISAAwISwAw\nICwBwICwBAADwhIADAhLADAgLAHAgLAEAAPCEgAMCEsAMCAsAcCAsAQAA8ISAAwISwAwICwBwICw\nBAADwhIADAhLADAgLAHAgLAEAAPCEgAMCEsAMCAsAcCAsAQAA8ISAAwISwAwICwBwICwBAADwhIA\nDAhLADAgLAHAgLAEAAPCEgAMCEsAMCAsAcCAsAQAA8ISAAwISwAwICwBwICwBAADwhIADDKSvQC4\nX3p6urn2f//73xCu5GzZ2dl9bldWVpoe5/V6zdu48sorzbWPPfaYuXbdunVhx7dv397ndllZmXnO\nkydPmmtXr15tqnvuuefMc6YynlkCgEFMzyz9fr8WLVqk8ePHS5KuuOIKLVu2LK4LA4BUEvPL8MmT\nJ6u2tjaeawGAlMXLcAAwiDksf/jhBz3yyCMqKyvTF198Ec81AUDK8QSDwWC0D2pvb9e+fftUUlKi\n1tZWzZs3T42NjcrMzAxb39zcrIKCgkEvFgCSJaawPNM999yjl156SRdddFH4jXg8YceDwWC/9zmV\nG3uSBtdXqn506OjRo8rJyekz5vSPDpWVlWnHjh1njVml6keHEvV3NVAcxvQyfOfOnXr99dclSYFA\nQEePHlVeXl5sqwMAB4jpaPj06dP1xBNP6KOPPtKpU6f07LPP9vsSHADcIKawHDFihDZu3BjvtQBA\nyuJ0R4e6+OKLzbXRPOufMmVKv/fNmzcv9N833nijec6srCxz7d13322ujYdAIDDk2/j555/NtdF8\ndrm0tDTs+Jw5c/rc7uzsNM+5f/9+c+2nn35qrnUDPmcJAAaEJQAYEJYAYEBYAoABYQkABoQlABgQ\nlgBgQFgCgAFhCQAGhCUAGMTlEm0RN8Il2syuueYaU93u3bvNc8bjsmdpaWk6ffr0oOdJJYPpKZrH\nPfjgg+baEydOxLKckLfffluzZ8/uM9bW1mZ+fEdHh7n2+++/N9cOlmMv0QYA/zWEJQAYEJYAYEBY\nAoABYQkABoQlABgQlgBgQFgCgAFhCQAGnMETZ4PtKTs721Tn9/vNc15++eWxLick1c7giab/Y8eO\nhR0vKSnRBx980GesuLjYNOdff/1l3n48zqCycuPflMQZPADgGIQlABgQlgBgQFgCgAFhCQAGhCUA\nGBCWAGBAWAKAAWEJAAaEJQAYcLpjnCWqp1mzZplr77jjDnPtN998E3Z8w4YNqqysDN2ura01zxmN\nb7/91lQ3bdo085x//PFH2PFw+2rChAmmORctWmTe/sMPP2yuHSw3/k1JnO4IAI5BWAKAAWEJAAaE\nJQAYEJYAYEBYAoABYQkABoQlABgQlgBgQFgCgAGnO8ZZKvY0atQoc21nZ2fY8dOnTyst7f//ba2r\nqzPPOX/+fHNtRUWFqW7Hjh3mOfuTivtqsNzYk+Sg0x1bWlo0c+ZMbdu2TZLU1tamuXPnqry8XIsW\nLYrqa0EBwIkihmVXV5dWrFihoqKi0Fhtba3Ky8u1fft2XXLJJaqvrx/SRQJAskUMy8zMTG3evFm5\nubmhMb/frxkzZkj6+0vpm5qahm6FAJACMiIWZGQoI6NvWXd3tzIzMyVJOTk5CgQCQ7M6AEgREcMy\nEsvxoQMHDqigoCDmxzuNG3uS/j7IM9S2b98e17pI3Liv3NiTlPy+YgpLr9erkydPatiwYWpvb+/z\nEj2cwsLCsONuPHKXij1xNDy8VNxXg+XGniQHHQ0/05QpU9TQ0CBJamxs1NSpU2NbGQA4RMRnls3N\nzVqzZo0OHz6sjIwMNTQ0aN26daqqqpLP59PYsWOj+ooDAHCiiGFZUFCgt95666zxrVu3DsmCACAV\nDfoAD1Lf77//Hpd5/v1+zvHjx+My55keeughU53P5zPPmYgDU3A/zg0HAAPCEgAMCEsAMCAsAcCA\nsAQAA8ISAAwISwAwICwBwICwBAADwhIADPjCsjhzY0/S2X2de+655se+++675tqbbrrJVFdSUmKe\ns7GxMey4G/eVG3uSHHyJNgD4ryEsAcCAsAQAA8ISAAwISwAwICwBwICwBAADwhIADAhLADAgLAHA\ngNMd48yNPUmD62vcuHHm2q+//tpUd+zYMfOcH3/8cdjx+++/X2+88Uafsb1795rmfPXVV83bT8Cf\nWJ9t8fs3uO30h2eWAGBAWAKAAWEJAAaEJQAYEJYAYEBYAoABYQkABoQlABgQlgBgwBk8cebGnqTE\n9VVaWmqq27p1q3nOkSNHhh1PS0vT6dOnzfP8W3V1tbn2zTffNNe2tbXFspwQfv8Gv53+8MwSAAwI\nSwAwICwBwICwBAADwhIADAhLADAgLAHAgLAEAAPCEgAMCEsAMOB0xzhzY09S6vVVUFBgrl2/fn3Y\n8VtuuUUffvhhn7EZM2YMal3h1NXVmWtfeOEFc+3hw4fPGku1/RQvnO4IAA5hCsuWlhbNnDlT27Zt\nkyRVVVXpzjvv1Ny5czV37lx98sknQ7lGAEi6jEgFXV1dWrFihYqKivqML1myRMXFxUO2MABIJRGf\nWWZmZmrz5s3Kzc1NxHoAICWZD/C88sorGj16tCoqKlRVVaVAIKBTp04pJydHy5YtU3Z2dr+PbW5u\njuoNeQBINRFfhodz1113KSsrS/n5+dq0aZM2bNig5cuX91tfWFgYdtyNR+7c2JOUen1xNJyj4UO1\nnf7EdDS8qKhI+fn5kqTp06erpaUltpUBgEPEFJYLFy5Ua2urJMnv92v8+PFxXRQApJqIL8Obm5u1\nZs0aHT58WBkZGWpoaFBFRYUWL16s4cOHy+v1atWqVYlYKwAkTcSwLCgo0FtvvXXW+G233TYkCwKA\nVMTpjnHmxp4kZ/eVlZUVdryjo0OjR4/uM3bnnXea5ozm2yWj+f+2e/duc+0tt9xy1piT99NAHHuA\nBwD+awhLADAgLAHAgLAEAAPCEgAMCEsAMCAsAcCAsAQAA8ISAAwISwAw4HTHOHNjT5I7+xpMT3/+\n+ae5NiPDftnYnp4ec2246zN8/PHHZ33dixu+I4vTHQHAIQhLADAgLAHAgLAEAAPCEgAMCEsAMCAs\nAcCAsAQAA8ISAAzspxYAKWTixInm2nvuuaff+55//vk+tydNmmSaM5qzcqLx3XffmWs/++yzqMYx\nODyzBAADwhIADAhLADAgLAHAgLAEAAPCEgAMCEsAMCAsAcCAsAQAA8ISAAw43RFD7sorrzTXVlZW\nmupmz55tnvP888/v976nn37aPE+sent7zbVtbW3m2tOnT0c1jsHhmSUAGBCWAGBAWAKAAWEJAAaE\nJQAYEJYAYEBYAoABYQkABoQlABgQlgBgwOmO6GOgUwP/fV9ZWZl5TuspjJJ06aWXmmuTae/eveba\nF154wVy7c+fOWJaDBDCFZU1Njfbt26eenh4tWLBAhYWFeuqpp9Tb26sxY8Zo7dq1yszMHOq1AkDS\nRAzLPXv26NChQ/L5fOro6FBpaamKiopUXl6ukpISrV+/XvX19SovL0/EegEgKSK+Zzlp0iS9/PLL\nkqRRo0apu7tbfr9fM2bMkCQVFxerqalpaFcJAEkWMSzT09Pl9XolSfX19Zo2bZq6u7tDL7tzcnIU\nCASGdpUAkGTmAzy7du1SfX29tmzZoltvvTU0HgwGIz72wIEDKigoCHuf5fFO48aepOiutegUaWmx\nfSBk8uTJ5tp33nknpm3Eyq2/f8nuyxSWn3/+uTZu3KjXXntNI0eOlNfr1cmTJzVs2DC1t7crNzd3\nwMcXFhaGHQ8Gg/J4PNGvOoU5vaf+joa3tbXpggsuCN12w9HwtLS0mC+Um6pHw53++9efRPU1UCBH\n/Ge1s7NTNTU1qqurU1ZWliRpypQpamhokCQ1NjZq6tSpcVoqAKSmiM8s33//fXV0dGjx4sWhsdWr\nV2vp0qXy+XwaO3asZs2aNaSLBIBkixiWc+bM0Zw5c84a37p165AsCABSEWfwOFReXp659uqrrzbX\nbtiwod/7Pvroo9B/X3XVVeY5k83v94cdLyoqOuu+tWvXmuaM5qANXyDmDpwbDgAGhCUAGBCWAGBA\nWAKAAWEJAAaEJQAYEJYAYEBYAoABYQkABoQlABh4ggm4SFx/l1Zy4+WkwvWUnZ1tfnxdXZ2p7ppr\nrjHPefnll5tr+zOYy5lF48svvzTVvfjii+Y5/7lC1pm6urpCF7b+R3d3t3neVOTGvynJIZdoAwAQ\nlgBgQlgCgAFhCQAGhCUAGBCWAGBAWAKAAWEJAAaEJQAYEJYAYMC3O57huuuuM9U9+eST/d5XX1/f\n5/bkyZPN27/wwgvNtcnU1dVlrq2trTXXrly50lT3xx9/mOcciNNPb0Ti8MwSAAwISwAwICwBwICw\nBAADwhIADAhLADAgLAHAgLAEAAPCEgAMOIPnDKWlpYOus84xGN9995259r333jPX9vT0hB1funRp\nn7NrovnCsGPHjplrgVTFM0sAMCAsAcCAsAQAA8ISAAwISwAwICwBwICwBAADwhIADAhLADAgLAHA\nwBMMBoNDvhGPJ+x4MBjs9z6ncmNPkjv7oifnSFRfA8Wh6dzwmpoa7du3Tz09PVqwYIF2796tgwcP\nKisrS5I0f/583XzzzXFZLACkoohhuWfPHh06dEg+n08dHR0qLS3V9ddfryVLlqi4uDgRawSApIsY\nlpMmTdLEiRMlSaNGjVJ3d7d6e3uHfGEAkEqies/S5/Np7969Sk9PVyAQ0KlTp5STk6Nly5YpOzu7\n/43wnqXjubEvenKOVHjP0hyWu3btUl1dnbZs2aLm5mZlZWUpPz9fmzZt0i+//KLly5f3+9jm5mYV\nFBREv3IASBVBg88++yx49913Bzs6Os6679ChQ8H77rtvwMdLCvsz0H1O/XFjT27ti56c85OovgYS\n8XOWnZ2dqqmpUV1dXejo98KFC9Xa2ipJ8vv9Gj9+fKRpAMDRIh7gef/999XR0aHFixeHxmbPnq3F\nixdr+PDh8nq9WrVq1ZAuEgCSjQ+lx5kbe5Lc2Rc9OUei+hooDjndEQAMCEsAMCAsAcCAsAQAA8IS\nAAwISwAwICwBwICwBAADwhIADAhLADAgLAHAgLAEAAPCEgAMCEsAMCAsAcCAsAQAA8ISAAwISwAw\nICwBwICwBAADwhIADAhLADBIyFfhAoDT8cwSAAwISwAwICwBwICwBAADwhIADAhLADDISMZGV65c\nqf3798vj8ai6uloTJ05MxjLiyu/3a9GiRRo/frwk6YorrtCyZcuSvKrYtbS06NFHH9UDDzygiooK\ntbW16amnnlJvb6/GjBmjtWvXKjMzM9nLjMqZPVVVVengwYPKysqSJM2fP18333xzchcZpZqaGu3b\nt089PT1asGCBCgsLHb+fpLP72r17d9L3VcLD8quvvtJPP/0kn8+nH3/8UdXV1fL5fIlexpCYPHmy\namtrk72MQevq6tKKFStUVFQUGqutrVV5eblKSkq0fv161dfXq7y8PImrjE64niRpyZIlKi4uTtKq\nBmfPnj06dOiQfD6fOjo6VFpaqqKiIkfvJyl8X9dff33S91XCX4Y3NTVp5syZkqRx48bp+PHjOnHi\nRKKXgQFkZmZq8+bNys3NDY35/X7NmDFDklRcXKympqZkLS8m4XpyukmTJunll1+WJI0aNUrd3d2O\n309S+L56e3uTvKokhOWRI0c0evTo0O3s7GwFAoFEL2NI/PDDD3rkkUdUVlamL774ItnLiVlGRoaG\nDRvWZ6y7uzv0ci4nJ8dx+yxcT5K0bds2zZs3T48//rh+++23JKwsdunp6fJ6vZKk+vp6TZs2zfH7\nSQrfV3p6etL3VVLes/w3t5xteemll6qyslIlJSVqbW3VvHnz1NjY6Mj3iyJxyz676667lJWVpfz8\nfG3atEkbNmzQ8uXLk72sqO3atUv19fXasmWLbr311tC40/fTv/tqbm5O+r5K+DPL3NxcHTlyJHT7\n119/1ZgxYxK9jLjLy8vT7bffLo/Ho4svvljnnXee2tvbk72suPF6vTp58qQkqb293RUvZ4uKipSf\nny9Jmj59ulpaWpK8ouh9/vnn2rhxozZv3qyRI0e6Zj+d2Vcq7KuEh+UNN9yghoYGSdLBgweVm5ur\nESNGJHoZcbdz5069/vrrkqRAIKCjR48qLy8vyauKnylTpoT2W2Njo6ZOnZrkFQ3ewoUL1draKunv\n92T/+SSDU3R2dqqmpkZ1dXWho8Ru2E/h+kqFfZWUqw6tW7dOe/fulcfj0TPPPKOrrroq0UuIuxMn\nTuiJJ57Q77//rlOnTqmyslI33XRTspcVk+bmZq1Zs0aHDx9WRkaG8vLytG7dOlVVVenPP//U2LFj\ntWrVKp1zzjnJXqpZuJ4qKiq0adMmDR8+XF6vV6tWrVJOTk6yl2rm8/n0yiuv6LLLLguNrV69WkuX\nLnXsfpLC9zV79mxt27YtqfuKS7QBgAFn8ACAAWEJAAaEJQAYEJYAYEBYAoABYQkABoQlABgQlgBg\n8H/LFmKD6IYI7AAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f8634d85320>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "8oureGhUIT5V",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Load the Dataset\n",
        "\n",
        "**Note**: We shuffle the loading process of `train_dataset` to make the learning process independent of data order, but the order of `test_loader` remains so as to examine whether we can handle unspecified bias order of inputs.\n"
      ]
    },
    {
      "metadata": {
        "id": "lFaQ0dNUIU6C",
        "colab_type": "code",
        "outputId": "5dc10f96-2d15-4e76-b21d-7e9ea0fb0f27",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "train_loader = torch.utils.data.DataLoader(dataset=train_dataset,\n",
        "                                          batch_size=batch_size,\n",
        "                                          shuffle=True)\n",
        "\n",
        "test_loader = torch.utils.data.DataLoader(dataset=test_dataset,\n",
        "                                          batch_size=batch_size,\n",
        "                                          shuffle=False)\n",
        "\n",
        "dataset_sizes = {\n",
        "    'train': len(train_loader.dataset), \n",
        "    'test': len(test_loader.dataset)\n",
        "}\n",
        "\n",
        "\n",
        "print (dataset_sizes)\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{'train': 60000, 'test': 10000}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "4hgCecJDIi8o",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Build two different Feedforward Neural Networks\n",
        "\n",
        "### Feedforward Neural Network Model Structure\n",
        "\n",
        "### NetA\n",
        "The FNN includes two fully-connected layers (i.e. fc1 & fc2) and a non-linear ReLU layer in between. Normally we call this structure **1-hidden layer FNN**, without counting the output layer (fc2) in.\n",
        "\n",
        "By running the forward pass, the input images (x) can go through the neural network and generate a output (out) demonstrating how are the likabilities it belongs to each of the 10 classes. _For example, a cat image can have 0.8 likability to a dog class and a 0.3 likability to a airplane class._\n",
        "\n",
        "### NetB"
      ]
    },
    {
      "metadata": {
        "id": "_rnAhgUlIr5t",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from torch.autograd import Variable\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "\n",
        "\n",
        "class NetA(nn.Module):\n",
        "    def __init__(self, input_size, hidden_size, num_classes):\n",
        "        super(NetA, self).__init__()                    # Inherited from the parent class nn.Module\n",
        "        self.fc1 = nn.Linear(input_size, hidden_size)  # 1st Full-Connected Layer: 784 (input data) -> 500 (hidden node)\n",
        "        self.relu = nn.ReLU()                          # Non-Linear ReLU Layer: max(0,x)\n",
        "        self.fc2 = nn.Linear(hidden_size, num_classes) # 2nd Full-Connected Layer: 500 (hidden node) -> 10 (output class)\n",
        "    \n",
        "    def forward(self, x):                              # Forward pass: stacking each layer together\n",
        "        out = self.fc1(x)\n",
        "        out = self.relu(out)\n",
        "        out = self.fc2(out)\n",
        "        return out\n",
        "      \n",
        "\n",
        "\n",
        "class NetB(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(NetB, self).__init__()\n",
        "        self.l1 = nn.Linear(784, 128)\n",
        "        self.l2 = nn.Linear(128, 32)\n",
        "        self.l3 = nn.Linear(32, 10)\n",
        "        \n",
        "    def forward(self, x):\n",
        "        x = x.view(-1, 28 * 28)\n",
        "        x = torch.relu(self.l1(x))\n",
        "        x = torch.relu(self.l2(x))\n",
        "        x = self.l3(x)\n",
        "        x = F.log_softmax(x, dim=1)\n",
        "        return x\n",
        "    "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3qgBYVkxIxym",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Instantiate the FNN\n",
        "\n",
        "We now create a real FNN based on our structure."
      ]
    },
    {
      "metadata": {
        "id": "xvLScxMxI0_Y",
        "colab_type": "code",
        "outputId": "666e5a96-ce77-432c-d511-02b9fc0969d5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "cell_type": "code",
      "source": [
        "neta = NetA(input_size, hidden_size, num_classes)\n",
        "print (neta)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "NetA(\n",
            "  (fc1): Linear(in_features=784, out_features=500, bias=True)\n",
            "  (relu): ReLU()\n",
            "  (fc2): Linear(in_features=500, out_features=10, bias=True)\n",
            ")\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "Q1dAfU8sjuAn",
        "colab_type": "code",
        "outputId": "b369b260-7d96-4847-88e9-8e351a709f7e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "cell_type": "code",
      "source": [
        "netb = NetB()\n",
        "print (netb)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "NetB(\n",
            "  (l1): Linear(in_features=784, out_features=128, bias=True)\n",
            "  (l2): Linear(in_features=128, out_features=32, bias=True)\n",
            "  (l3): Linear(in_features=32, out_features=10, bias=True)\n",
            ")\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "JNb7cbEVlQ_a",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# When using several GPU's"
      ]
    },
    {
      "metadata": {
        "id": "NLrI_hs6kpcf",
        "colab_type": "code",
        "outputId": "a061280e-feaf-4057-cf99-0ce5d778a9ad",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        }
      },
      "cell_type": "code",
      "source": [
        "if use_cuda:\n",
        "    model = neta.cuda()\n",
        "model_name = (type(model).__name__) # remember the real name\n",
        "print(model)\n",
        "\n",
        "# for using more than one GPU\n",
        "model = torch.nn.DataParallel(model, device_ids=list(range(1)))\n",
        "print(model)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "NetA(\n",
            "  (fc1): Linear(in_features=784, out_features=500, bias=True)\n",
            "  (relu): ReLU()\n",
            "  (fc2): Linear(in_features=500, out_features=10, bias=True)\n",
            ")\n",
            "DataParallel(\n",
            "  (module): NetA(\n",
            "    (fc1): Linear(in_features=784, out_features=500, bias=True)\n",
            "    (relu): ReLU()\n",
            "    (fc2): Linear(in_features=500, out_features=10, bias=True)\n",
            "  )\n",
            ")\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "UYBesVm1I4yf",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Enable GPU\n",
        "\n",
        "_**Note**: You could enable this line to run the codes on GPU_"
      ]
    },
    {
      "metadata": {
        "id": "I5qmjLJlMyNY",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "use_cuda = True"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "oQ6isf-kI2HD",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "if use_cuda and torch.cuda.is_available():\n",
        "    neta.cuda()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "J5pOMFEPJEtd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Choose the Loss Function and Optimizer\n",
        "\n",
        "Loss function (**criterion**) decides how the output can be compared to a class, which determines how good or bad the neural network performs. And the **optimizer** chooses a way to update the weight in order to converge to find the best weights in this neural network."
      ]
    },
    {
      "metadata": {
        "id": "Jo60XGznI_Ul",
        "colab_type": "code",
        "outputId": "68f3aace-9461-4a04-a5e9-362bf4ed532d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 170
        }
      },
      "cell_type": "code",
      "source": [
        "criterion = nn.CrossEntropyLoss()\n",
        "optimizer = torch.optim.Adam(neta.parameters(), lr=learning_rate)\n",
        "\n",
        "print (criterion)\n",
        "print (optimizer)\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "CrossEntropyLoss()\n",
            "Adam (\n",
            "Parameter Group 0\n",
            "    amsgrad: False\n",
            "    betas: (0.9, 0.999)\n",
            "    eps: 1e-08\n",
            "    lr: 0.001\n",
            "    weight_decay: 0\n",
            ")\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "VLUaX6tuJMQi",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Training the FNN Model\n",
        "\n",
        "This process might take around 3 to 5 minutes depending on your machine. The detailed explanations are listed as comments (#) in the following codes."
      ]
    },
    {
      "metadata": {
        "id": "Avh7XbTflwma",
        "colab_type": "code",
        "outputId": "a365d814-ebb6-4031-b0ba-53c50dfa9ba8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "cell_type": "code",
      "source": [
        "neta.train() # set training mode"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "NetA(\n",
              "  (fc1): Linear(in_features=784, out_features=500, bias=True)\n",
              "  (relu): ReLU()\n",
              "  (fc2): Linear(in_features=500, out_features=10, bias=True)\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        }
      ]
    },
    {
      "metadata": {
        "id": "qBrGa7qMJKcB",
        "colab_type": "code",
        "outputId": "74acfd86-546c-4d0b-bfd7-f1ab43308b4e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 544
        }
      },
      "cell_type": "code",
      "source": [
        "from livelossplot import PlotLosses\n",
        "\n",
        "liveloss = PlotLosses()\n",
        "\n",
        "num_epochs=5\n",
        "\n",
        "for epoch in range(num_epochs):\n",
        "#     correct = 0\n",
        "#     total = 0\n",
        "\n",
        "    for i, (images, labels) in enumerate(train_loader):   # Load a batch of images with its (index, data, class)\n",
        "        images = Variable(images.view(-1, 28*28))         # Convert torch tensor to Variable: change image from a vector of size 784 to a matrix of 28 x 28\n",
        "        labels = Variable(labels)\n",
        "        \n",
        "        if use_cuda and torch.cuda.is_available():\n",
        "            images = images.cuda()\n",
        "            labels = labels.cuda()\n",
        "        \n",
        "        optimizer.zero_grad()                             # Intialize the hidden weight to all zeros\n",
        "        outputs = neta(images)                             # Forward pass: compute the output class given a image\n",
        "        loss = criterion(outputs, labels)                 # Compute the loss: difference between the output class and the pre-given label\n",
        "        loss.backward()                                   # Backward pass: compute the weight\n",
        "        optimizer.step()                                  # Optimizer: update the weights of hidden nodes\n",
        "        \n",
        "                \n",
        "#         _, predicted = torch.max(outputs.data, 1)  # Choose the best class from the output: The class with the best score\n",
        "#         total += labels.size(0)                    # Increment the total count\n",
        "#         correct += (predicted == labels).sum()     # Increment the correct count\n",
        "            \n",
        "\n",
        "        if (i+1) % 100 == 0:                              # Logging\n",
        "            print('Epoch [%d/%d], Step [%d/%d], Loss: %.4f'\n",
        "                 %(epoch+1, num_epochs, i+1, len(train_dataset)//batch_size, loss.data[0]))\n",
        "            \n",
        "#             # Visualize the loss and accuracy values.\n",
        "#             liveloss.update({\n",
        "#                 'log loss': loss.data[0],\n",
        "#                 #'val_log loss': test_loss,\n",
        "#                 #'accuracy': (100 * correct / total),\n",
        "#                 #'val_accuracy': test_correct,\n",
        "#             })\n",
        "#             liveloss.draw()\n",
        "\n"
      ],
      "execution_count": 69,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:33: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch [1/5], Step [100/600], Loss: 0.0171\n",
            "Epoch [1/5], Step [200/600], Loss: 0.0037\n",
            "Epoch [1/5], Step [300/600], Loss: 0.0004\n",
            "Epoch [1/5], Step [400/600], Loss: 0.0008\n",
            "Epoch [1/5], Step [500/600], Loss: 0.0097\n",
            "Epoch [1/5], Step [600/600], Loss: 0.0001\n",
            "Epoch [2/5], Step [100/600], Loss: 0.0001\n",
            "Epoch [2/5], Step [200/600], Loss: 0.0003\n",
            "Epoch [2/5], Step [300/600], Loss: 0.0026\n",
            "Epoch [2/5], Step [400/600], Loss: 0.0000\n",
            "Epoch [2/5], Step [500/600], Loss: 0.0010\n",
            "Epoch [2/5], Step [600/600], Loss: 0.0000\n",
            "Epoch [3/5], Step [100/600], Loss: 0.0000\n",
            "Epoch [3/5], Step [200/600], Loss: 0.0000\n",
            "Epoch [3/5], Step [300/600], Loss: 0.0000\n",
            "Epoch [3/5], Step [400/600], Loss: 0.0000\n",
            "Epoch [3/5], Step [500/600], Loss: 0.0000\n",
            "Epoch [3/5], Step [600/600], Loss: 0.0032\n",
            "Epoch [4/5], Step [100/600], Loss: 0.0000\n",
            "Epoch [4/5], Step [200/600], Loss: 0.0000\n",
            "Epoch [4/5], Step [300/600], Loss: 0.0000\n",
            "Epoch [4/5], Step [400/600], Loss: 0.0000\n",
            "Epoch [4/5], Step [500/600], Loss: 0.0000\n",
            "Epoch [4/5], Step [600/600], Loss: 0.0000\n",
            "Epoch [5/5], Step [100/600], Loss: 0.0001\n",
            "Epoch [5/5], Step [200/600], Loss: 0.0000\n",
            "Epoch [5/5], Step [300/600], Loss: 0.0000\n",
            "Epoch [5/5], Step [400/600], Loss: 0.0000\n",
            "Epoch [5/5], Step [500/600], Loss: 0.0000\n",
            "Epoch [5/5], Step [600/600], Loss: 0.0000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "lhOeVK6RKz1-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Testing the FNN Model\n",
        "\n",
        "Similar to training the neural network, we also need to load batches of test images and collect the outputs. The differences are that:\n",
        "\n",
        "1. No loss & weights calculation\n",
        "2. No wights update\n",
        "3. Has correct prediction calculation\n"
      ]
    },
    {
      "metadata": {
        "id": "ZV3yG9ZsovA5",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "4bcf77f1-27f1-47ed-baef-edf1fa7ae0e6"
      },
      "cell_type": "code",
      "source": [
        "neta.eval() # Inference mode"
      ],
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "NetA(\n",
              "  (fc1): Linear(in_features=784, out_features=500, bias=True)\n",
              "  (relu): ReLU()\n",
              "  (fc2): Linear(in_features=500, out_features=10, bias=True)\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 55
        }
      ]
    },
    {
      "metadata": {
        "id": "0ue19srtK4C1",
        "colab_type": "code",
        "outputId": "474d966d-359b-4bae-cf34-40abc8f80825",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "correct = 0\n",
        "total = 0\n",
        "for images, labels in test_loader:\n",
        "    images = Variable(images.view(-1, 28*28))\n",
        "    \n",
        "    if use_cuda and torch.cuda.is_available():\n",
        "        images = images.cuda()\n",
        "        labels = labels.cuda()\n",
        "    \n",
        "    \n",
        "    outputs = neta(images)\n",
        "    _, predicted = torch.max(outputs.data, 1)  # Choose the best class from the output: The class with the best score\n",
        "    total += labels.size(0)                    # Increment the total count\n",
        "    correct += (predicted == labels).sum()     # Increment the correct count\n",
        "    \n",
        "print('Accuracy of the network on the 10K test images: %d %%' % (100 * correct / total))"
      ],
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy of the network on the 10K test images: 98 %\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "EM7EnBoyK8nR",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Save the trained FNN Model for future use\n",
        "\n",
        "We save the trained model as a pickle that can be loaded and used later."
      ]
    },
    {
      "metadata": {
        "id": "uof2bcfIK5n7",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "torch.save(neta.state_dict(), 'fnn_model.pkl')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "usLTAaYZLFml",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Congrats\n",
        "\n",
        "You have done building your first Feedforward Neural Network!"
      ]
    },
    {
      "metadata": {
        "id": "HKRe1dAYsPwY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 3 layer FFN"
      ]
    },
    {
      "metadata": {
        "id": "U3yuII9mnv6e",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 323
        },
        "outputId": "f96a2f13-998e-462b-a61d-2e983ac918e6"
      },
      "cell_type": "code",
      "source": [
        "model = nn.Sequential()\n",
        "model.add_module('fc1',      nn.Linear(784, 256))\n",
        "model.add_module('relu1',    nn.ReLU())\n",
        "model.add_module('dropout1', nn.Dropout())\n",
        "model.add_module('fc2',      nn.Linear(256, 256))\n",
        "model.add_module('relu2',    nn.ReLU())\n",
        "model.add_module('dropout2', nn.Dropout())\n",
        "model.add_module('fc3',      nn.Linear(256, 10))\n",
        "\n",
        "print(model)\n",
        "model.cuda()"
      ],
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sequential(\n",
            "  (fc1): Linear(in_features=784, out_features=256, bias=True)\n",
            "  (relu1): ReLU()\n",
            "  (dropout1): Dropout(p=0.5)\n",
            "  (fc2): Linear(in_features=256, out_features=256, bias=True)\n",
            "  (relu2): ReLU()\n",
            "  (dropout2): Dropout(p=0.5)\n",
            "  (fc3): Linear(in_features=256, out_features=10, bias=True)\n",
            ")\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Sequential(\n",
              "  (fc1): Linear(in_features=784, out_features=256, bias=True)\n",
              "  (relu1): ReLU()\n",
              "  (dropout1): Dropout(p=0.5)\n",
              "  (fc2): Linear(in_features=256, out_features=256, bias=True)\n",
              "  (relu2): ReLU()\n",
              "  (dropout2): Dropout(p=0.5)\n",
              "  (fc3): Linear(in_features=256, out_features=10, bias=True)\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 67
        }
      ]
    },
    {
      "metadata": {
        "id": "E3uhmMTysWa_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 544
        },
        "outputId": "23542306-e84b-4df5-f5a5-a74c7a553e4e"
      },
      "cell_type": "code",
      "source": [
        "\n",
        "num_epochs=5\n",
        "for epoch in range(num_epochs):\n",
        "    for i, (images, labels) in enumerate(train_loader):   # Load a batch of images with its (index, data, class)\n",
        "        images = Variable(images.view(-1, 28*28))         # Convert torch tensor to Variable: change image from a vector of size 784 to a matrix of 28 x 28\n",
        "        labels = Variable(labels)\n",
        "        \n",
        "        if use_cuda and torch.cuda.is_available():\n",
        "            images = images.cuda()\n",
        "            labels = labels.cuda()\n",
        "        \n",
        "        optimizer.zero_grad()                             # Intialize the hidden weight to all zeros\n",
        "        outputs = model(images)                             # Forward pass: compute the output class given a image\n",
        "        loss = criterion(outputs, labels)                 # Compute the loss: difference between the output class and the pre-given label\n",
        "        loss.backward()                                   # Backward pass: compute the weight\n",
        "        optimizer.step()                                  # Optimizer: update the weights of hidden nodes\n",
        "        \n",
        "                \n",
        "#         _, predicted = torch.max(outputs.data, 1)  # Choose the best class from the output: The class with the best score\n",
        "#         total += labels.size(0)                    # Increment the total count\n",
        "#         correct += (predicted == labels).sum()     # Increment the correct count\n",
        "            \n",
        "\n",
        "        if (i+1) % 100 == 0:                              # Logging\n",
        "            print('Epoch [%d/%d], Step [%d/%d], Loss: %.4f'\n",
        "                 %(epoch+1, num_epochs, i+1, len(train_dataset)//batch_size, loss.data[0]))\n",
        "                        \n"
      ],
      "execution_count": 71,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:30: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch [1/5], Step [100/600], Loss: 2.3019\n",
            "Epoch [1/5], Step [200/600], Loss: 2.2906\n",
            "Epoch [1/5], Step [300/600], Loss: 2.3014\n",
            "Epoch [1/5], Step [400/600], Loss: 2.3103\n",
            "Epoch [1/5], Step [500/600], Loss: 2.2980\n",
            "Epoch [1/5], Step [600/600], Loss: 2.3078\n",
            "Epoch [2/5], Step [100/600], Loss: 2.3022\n",
            "Epoch [2/5], Step [200/600], Loss: 2.3053\n",
            "Epoch [2/5], Step [300/600], Loss: 2.3067\n",
            "Epoch [2/5], Step [400/600], Loss: 2.3039\n",
            "Epoch [2/5], Step [500/600], Loss: 2.2966\n",
            "Epoch [2/5], Step [600/600], Loss: 2.3005\n",
            "Epoch [3/5], Step [100/600], Loss: 2.2972\n",
            "Epoch [3/5], Step [200/600], Loss: 2.2960\n",
            "Epoch [3/5], Step [300/600], Loss: 2.3131\n",
            "Epoch [3/5], Step [400/600], Loss: 2.2995\n",
            "Epoch [3/5], Step [500/600], Loss: 2.3048\n",
            "Epoch [3/5], Step [600/600], Loss: 2.3081\n",
            "Epoch [4/5], Step [100/600], Loss: 2.3030\n",
            "Epoch [4/5], Step [200/600], Loss: 2.3052\n",
            "Epoch [4/5], Step [300/600], Loss: 2.2996\n",
            "Epoch [4/5], Step [400/600], Loss: 2.3014\n",
            "Epoch [4/5], Step [500/600], Loss: 2.2988\n",
            "Epoch [4/5], Step [600/600], Loss: 2.3017\n",
            "Epoch [5/5], Step [100/600], Loss: 2.3205\n",
            "Epoch [5/5], Step [200/600], Loss: 2.3152\n",
            "Epoch [5/5], Step [300/600], Loss: 2.3026\n",
            "Epoch [5/5], Step [400/600], Loss: 2.2980\n",
            "Epoch [5/5], Step [500/600], Loss: 2.2961\n",
            "Epoch [5/5], Step [600/600], Loss: 2.3022\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "WyrdOlATsxZY",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "q-ZvW2sJtg-I",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Why it does not work ?! can you trace the error? \n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "- criterion = nn.CrossEntropyLoss()\n",
        "- optimizer = torch.optim.Adam(**neta**.parameters(), lr=learning_rate)"
      ]
    },
    {
      "metadata": {
        "id": "xG4hVyKYtojA",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 544
        },
        "outputId": "22e35d61-907e-46b8-fd70-72f6724ff8da"
      },
      "cell_type": "code",
      "source": [
        "criterion = nn.CrossEntropyLoss()\n",
        "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n",
        "\n",
        "\n",
        "num_epochs=5\n",
        "for epoch in range(num_epochs):\n",
        "    for i, (images, labels) in enumerate(train_loader):   # Load a batch of images with its (index, data, class)\n",
        "        images = Variable(images.view(-1, 28*28))         # Convert torch tensor to Variable: change image from a vector of size 784 to a matrix of 28 x 28\n",
        "        labels = Variable(labels)\n",
        "        \n",
        "        if use_cuda and torch.cuda.is_available():\n",
        "            images = images.cuda()\n",
        "            labels = labels.cuda()\n",
        "        \n",
        "        optimizer.zero_grad()                             # Intialize the hidden weight to all zeros\n",
        "        outputs = model(images)                             # Forward pass: compute the output class given a image\n",
        "        loss = criterion(outputs, labels)                 # Compute the loss: difference between the output class and the pre-given label\n",
        "        loss.backward()                                   # Backward pass: compute the weight\n",
        "        optimizer.step()                                  # Optimizer: update the weights of hidden nodes\n",
        "        \n",
        "                \n",
        "#         _, predicted = torch.max(outputs.data, 1)  # Choose the best class from the output: The class with the best score\n",
        "#         total += labels.size(0)                    # Increment the total count\n",
        "#         correct += (predicted == labels).sum()     # Increment the correct count\n",
        "            \n",
        "\n",
        "        if (i+1) % 100 == 0:                              # Logging\n",
        "            print('Epoch [%d/%d], Step [%d/%d], Loss: %.4f'\n",
        "                 %(epoch+1, num_epochs, i+1, len(train_dataset)//batch_size, loss.data[0]))\n",
        "                        \n"
      ],
      "execution_count": 72,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:29: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch [1/5], Step [100/600], Loss: 0.3614\n",
            "Epoch [1/5], Step [200/600], Loss: 0.4324\n",
            "Epoch [1/5], Step [300/600], Loss: 0.2259\n",
            "Epoch [1/5], Step [400/600], Loss: 0.1309\n",
            "Epoch [1/5], Step [500/600], Loss: 0.2889\n",
            "Epoch [1/5], Step [600/600], Loss: 0.2968\n",
            "Epoch [2/5], Step [100/600], Loss: 0.2008\n",
            "Epoch [2/5], Step [200/600], Loss: 0.1542\n",
            "Epoch [2/5], Step [300/600], Loss: 0.1963\n",
            "Epoch [2/5], Step [400/600], Loss: 0.0969\n",
            "Epoch [2/5], Step [500/600], Loss: 0.0908\n",
            "Epoch [2/5], Step [600/600], Loss: 0.1436\n",
            "Epoch [3/5], Step [100/600], Loss: 0.0924\n",
            "Epoch [3/5], Step [200/600], Loss: 0.0682\n",
            "Epoch [3/5], Step [300/600], Loss: 0.1377\n",
            "Epoch [3/5], Step [400/600], Loss: 0.1065\n",
            "Epoch [3/5], Step [500/600], Loss: 0.1059\n",
            "Epoch [3/5], Step [600/600], Loss: 0.2072\n",
            "Epoch [4/5], Step [100/600], Loss: 0.1572\n",
            "Epoch [4/5], Step [200/600], Loss: 0.1179\n",
            "Epoch [4/5], Step [300/600], Loss: 0.1904\n",
            "Epoch [4/5], Step [400/600], Loss: 0.0973\n",
            "Epoch [4/5], Step [500/600], Loss: 0.0969\n",
            "Epoch [4/5], Step [600/600], Loss: 0.1013\n",
            "Epoch [5/5], Step [100/600], Loss: 0.1378\n",
            "Epoch [5/5], Step [200/600], Loss: 0.1349\n",
            "Epoch [5/5], Step [300/600], Loss: 0.0261\n",
            "Epoch [5/5], Step [400/600], Loss: 0.0430\n",
            "Epoch [5/5], Step [500/600], Loss: 0.1706\n",
            "Epoch [5/5], Step [600/600], Loss: 0.0563\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "qrW1UnHdtvbu",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}