Pytorch_CustomNN.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Pytorch-CustomNN.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyPBW4PQr9fHB137OwCNyfyG",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/manthanthakker/DeepNeuralNetworks/blob/master/Pytorch_CustomNN.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bO1uZHCgk9Py",
        "colab_type": "text"
      },
      "source": [
        "#**Pytorch Custom DNN**\n",
        "\n",
        "Goal: Implement a custom Neural Network to play around with a sample dataset from sci-kit learn. We have selected a sample dataset breast cancer detection which is the most basic classification dataset we could get hold off.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sgsnXp1qlyN7",
        "colab_type": "text"
      },
      "source": [
        "First of all import the necessary modules\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KH86Z7tAl45n",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.datasets import load_breast_cancer\n",
        "from sklearn import preprocessing\n",
        "import numpy as np\n",
        "from sklearn.model_selection import train_test_split\n",
        "from torch.utils.data import DataLoader, Dataset\n",
        "import torch\n",
        "import torchvision.transforms as transforms\n",
        "import torch\n",
        "import torch.nn as nn"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wXKoG-nqmJax",
        "colab_type": "text"
      },
      "source": [
        "Now lets load the breast cancer dataset for classification task, which is readily available from scikit learn package. \n",
        "\n",
        "\n",
        "> Preprocessing Steps\n",
        "\n",
        "\n",
        "1.   Normalize the datasets\n",
        "2.   Split into training and testing\n",
        "3.   Transform the dataset to Pytorch tensors\n",
        "\n",
        "You can read more information about the data set by simply reading data.DESCR attribute.\n",
        "\n",
        "Lets do this\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZiE_QO0Yl1OT",
        "colab_type": "code",
        "outputId": "749b54fa-4f0d-4471-b987-c65e6f7150da",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "data = load_breast_cancer()\n",
        "X,y=data.data,data.target\n",
        "\n",
        "## Normalizing\n",
        "scalerX = preprocessing.StandardScaler().fit(X)\n",
        "X=scalerX.transform(X)\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)\n",
        "\n",
        "print(data.DESCR)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            ".. _breast_cancer_dataset:\n",
            "\n",
            "Breast cancer wisconsin (diagnostic) dataset\n",
            "--------------------------------------------\n",
            "\n",
            "**Data Set Characteristics:**\n",
            "\n",
            "    :Number of Instances: 569\n",
            "\n",
            "    :Number of Attributes: 30 numeric, predictive attributes and the class\n",
            "\n",
            "    :Attribute Information:\n",
            "        - radius (mean of distances from center to points on the perimeter)\n",
            "        - texture (standard deviation of gray-scale values)\n",
            "        - perimeter\n",
            "        - area\n",
            "        - smoothness (local variation in radius lengths)\n",
            "        - compactness (perimeter^2 / area - 1.0)\n",
            "        - concavity (severity of concave portions of the contour)\n",
            "        - concave points (number of concave portions of the contour)\n",
            "        - symmetry \n",
            "        - fractal dimension (\"coastline approximation\" - 1)\n",
            "\n",
            "        The mean, standard error, and \"worst\" or largest (mean of the three\n",
            "        largest values) of these features were computed for each image,\n",
            "        resulting in 30 features.  For instance, field 3 is Mean Radius, field\n",
            "        13 is Radius SE, field 23 is Worst Radius.\n",
            "\n",
            "        - class:\n",
            "                - WDBC-Malignant\n",
            "                - WDBC-Benign\n",
            "\n",
            "    :Summary Statistics:\n",
            "\n",
            "    ===================================== ====== ======\n",
            "                                           Min    Max\n",
            "    ===================================== ====== ======\n",
            "    radius (mean):                        6.981  28.11\n",
            "    texture (mean):                       9.71   39.28\n",
            "    perimeter (mean):                     43.79  188.5\n",
            "    area (mean):                          143.5  2501.0\n",
            "    smoothness (mean):                    0.053  0.163\n",
            "    compactness (mean):                   0.019  0.345\n",
            "    concavity (mean):                     0.0    0.427\n",
            "    concave points (mean):                0.0    0.201\n",
            "    symmetry (mean):                      0.106  0.304\n",
            "    fractal dimension (mean):             0.05   0.097\n",
            "    radius (standard error):              0.112  2.873\n",
            "    texture (standard error):             0.36   4.885\n",
            "    perimeter (standard error):           0.757  21.98\n",
            "    area (standard error):                6.802  542.2\n",
            "    smoothness (standard error):          0.002  0.031\n",
            "    compactness (standard error):         0.002  0.135\n",
            "    concavity (standard error):           0.0    0.396\n",
            "    concave points (standard error):      0.0    0.053\n",
            "    symmetry (standard error):            0.008  0.079\n",
            "    fractal dimension (standard error):   0.001  0.03\n",
            "    radius (worst):                       7.93   36.04\n",
            "    texture (worst):                      12.02  49.54\n",
            "    perimeter (worst):                    50.41  251.2\n",
            "    area (worst):                         185.2  4254.0\n",
            "    smoothness (worst):                   0.071  0.223\n",
            "    compactness (worst):                  0.027  1.058\n",
            "    concavity (worst):                    0.0    1.252\n",
            "    concave points (worst):               0.0    0.291\n",
            "    symmetry (worst):                     0.156  0.664\n",
            "    fractal dimension (worst):            0.055  0.208\n",
            "    ===================================== ====== ======\n",
            "\n",
            "    :Missing Attribute Values: None\n",
            "\n",
            "    :Class Distribution: 212 - Malignant, 357 - Benign\n",
            "\n",
            "    :Creator:  Dr. William H. Wolberg, W. Nick Street, Olvi L. Mangasarian\n",
            "\n",
            "    :Donor: Nick Street\n",
            "\n",
            "    :Date: November, 1995\n",
            "\n",
            "This is a copy of UCI ML Breast Cancer Wisconsin (Diagnostic) datasets.\n",
            "https://goo.gl/U2Uwz2\n",
            "\n",
            "Features are computed from a digitized image of a fine needle\n",
            "aspirate (FNA) of a breast mass.  They describe\n",
            "characteristics of the cell nuclei present in the image.\n",
            "\n",
            "Separating plane described above was obtained using\n",
            "Multisurface Method-Tree (MSM-T) [K. P. Bennett, \"Decision Tree\n",
            "Construction Via Linear Programming.\" Proceedings of the 4th\n",
            "Midwest Artificial Intelligence and Cognitive Science Society,\n",
            "pp. 97-101, 1992], a classification method which uses linear\n",
            "programming to construct a decision tree.  Relevant features\n",
            "were selected using an exhaustive search in the space of 1-4\n",
            "features and 1-3 separating planes.\n",
            "\n",
            "The actual linear program used to obtain the separating plane\n",
            "in the 3-dimensional space is that described in:\n",
            "[K. P. Bennett and O. L. Mangasarian: \"Robust Linear\n",
            "Programming Discrimination of Two Linearly Inseparable Sets\",\n",
            "Optimization Methods and Software 1, 1992, 23-34].\n",
            "\n",
            "This database is also available through the UW CS ftp server:\n",
            "\n",
            "ftp ftp.cs.wisc.edu\n",
            "cd math-prog/cpo-dataset/machine-learn/WDBC/\n",
            "\n",
            ".. topic:: References\n",
            "\n",
            "   - W.N. Street, W.H. Wolberg and O.L. Mangasarian. Nuclear feature extraction \n",
            "     for breast tumor diagnosis. IS&T/SPIE 1993 International Symposium on \n",
            "     Electronic Imaging: Science and Technology, volume 1905, pages 861-870,\n",
            "     San Jose, CA, 1993.\n",
            "   - O.L. Mangasarian, W.N. Street and W.H. Wolberg. Breast cancer diagnosis and \n",
            "     prognosis via linear programming. Operations Research, 43(4), pages 570-577, \n",
            "     July-August 1995.\n",
            "   - W.H. Wolberg, W.N. Street, and O.L. Mangasarian. Machine learning techniques\n",
            "     to diagnose breast cancer from fine-needle aspirates. Cancer Letters 77 (1994) \n",
            "     163-171.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UZLdochPnJU7",
        "colab_type": "text"
      },
      "source": [
        "So this data set has 30 attributes and a label that the given record has a breast cancer or not. Its a typical Classification problem, so we will be using cross entropy loss function and gradient descent as our training algorithm. \n",
        "We will be directly jumping to desinging the neural network as thats what we are trying to learn, one can also explore, random forest, SVM etc. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C2VKaP5Nn9Th",
        "colab_type": "text"
      },
      "source": [
        "This is a typical Neural Network setup, pytorch exposes it with defining classes as interface. All you need to do is extend the nnModule class and leverage all the goodness from the base class to construct your custom Neural Networks. You can also use most of the activation functions and they are part of the nn Module in Pytorch. \n",
        "\n",
        "We will be creating a single hidden layer Neural network and would parameterize the hidden neurons, so that we can fine tune it according our experimentation.\n",
        "\n",
        "Sigmoid activation function is fine. One can use relu too. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oOkS4i3BmLJy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class FeedforwardNeuralNetModel(nn.Module):\n",
        "    def __init__(self, input_dim, hidden_dim, output_dim):\n",
        "        super(FeedforwardNeuralNetModel, self).__init__()\n",
        "        # Linear function\n",
        "        self.fc1 = nn.Linear(input_dim, hidden_dim) \n",
        "        # Non-linearity\n",
        "        self.sigmoid = nn.Sigmoid()\n",
        "        # Linear function (readout)\n",
        "        self.fc2 = nn.Linear(hidden_dim, output_dim)  \n",
        "    \n",
        "    def forward(self, x):\n",
        "        # Linear function  # LINEAR\n",
        "        out = self.fc1(x)\n",
        "        # Non-linearity  # NON-LINEAR\n",
        "        out = self.sigmoid(out)\n",
        "        # Linear function (readout)  # LINEAR\n",
        "        out = self.fc2(out)\n",
        "        return out"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IQhRWI7Xo6Ey",
        "colab_type": "code",
        "outputId": "7af48883-5fde-47a0-a946-8115651eecc9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "source": [
        "print(model.parameters())\n",
        "\n",
        "print(len(list(model.parameters())))\n",
        "\n",
        "# FC 1 Parameters \n",
        "print(list(model.parameters())[0].size())\n",
        "\n",
        "# FC 1 Bias Parameters\n",
        "print(list(model.parameters())[1].size())\n",
        "\n",
        "# FC 2 Parameters\n",
        "print(list(model.parameters())[2].size())\n",
        "\n",
        "# FC 2 Bias Parameters\n",
        "print(list(model.parameters())[3].size())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<generator object Module.parameters at 0x7fb176883938>\n",
            "4\n",
            "torch.Size([8, 30])\n",
            "torch.Size([8])\n",
            "torch.Size([2, 8])\n",
            "torch.Size([2])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L-QnWMQoRLST",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z_HYu7Vxo2zJ",
        "colab_type": "text"
      },
      "source": [
        "Lets train the Neural Network now, it will follow a typical process in any Machine learning algo:\n",
        "\n",
        "\n",
        "\n",
        "1.   Loading data from numpy as tensors\n",
        "2.   Clearning the gradients from previous epoch\n",
        "3.   output=apply the model to the input\n",
        "4.   compute the loss from ouput and target values\n",
        "5.   compute the gradients\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-Oqroxt4o-fi",
        "colab_type": "code",
        "outputId": "22fa6e3d-dcc1-4546-b7ef-3b354fe13b0e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 952
        }
      },
      "source": [
        "input_dim = 30\n",
        "hidden_dim = 8\n",
        "output_dim = 2\n",
        "\n",
        "## Create an instance of NN\n",
        "model = FeedforwardNeuralNetModel(input_dim, hidden_dim, output_dim)\n",
        "\n",
        "## Select a Loss Function\n",
        "criterion = nn.CrossEntropyLoss()\n",
        "learning_rate = 0.15\n",
        "\n",
        "## Select an optimizer\n",
        "optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate,momentum=0.09)  \n",
        "\n",
        "\n",
        "num_epochs=55000\n",
        "iter = 0\n",
        "\n",
        "## Training...\n",
        "for epoch in range(num_epochs):\n",
        "\n",
        "        # Load images with gradient accumulation capabilities\n",
        "        images = torch.from_numpy(X_train).requires_grad_()\n",
        "        labels=torch.from_numpy(y_train)\n",
        "\n",
        "        # Clear gradients w.r.t. parameters\n",
        "        optimizer.zero_grad()\n",
        "        \n",
        "        # Forward pass to get output/logits\n",
        "        outputs = model(images.float())\n",
        "        \n",
        "        # Calculate Loss: softmax --> cross entropy loss\n",
        "        loss = criterion(outputs, labels)\n",
        "      \n",
        "        # Getting gradients w.r.t. parameters\n",
        "        loss.backward()\n",
        "       \n",
        "        # Updating parameters\n",
        "        optimizer.step()\n",
        "\n",
        "        iter += 1\n",
        "        \n",
        "        if iter % 1000 == 0:\n",
        "              # Print Loss\n",
        "            print('Iteration: {}. Loss: {}'.format(iter, loss.item()))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Iteration: 1000. Loss: 0.05288170650601387\n",
            "Iteration: 2000. Loss: 0.04305471479892731\n",
            "Iteration: 3000. Loss: 0.036482714116573334\n",
            "Iteration: 4000. Loss: 0.030957181006669998\n",
            "Iteration: 5000. Loss: 0.025832319632172585\n",
            "Iteration: 6000. Loss: 0.020235512405633926\n",
            "Iteration: 7000. Loss: 0.01587308757007122\n",
            "Iteration: 8000. Loss: 0.012684253975749016\n",
            "Iteration: 9000. Loss: 0.010386420413851738\n",
            "Iteration: 10000. Loss: 0.008670968934893608\n",
            "Iteration: 11000. Loss: 0.007362376432865858\n",
            "Iteration: 12000. Loss: 0.006345524452626705\n",
            "Iteration: 13000. Loss: 0.005540094804018736\n",
            "Iteration: 14000. Loss: 0.004890663083642721\n",
            "Iteration: 15000. Loss: 0.004358766600489616\n",
            "Iteration: 16000. Loss: 0.003917139023542404\n",
            "Iteration: 17000. Loss: 0.0035460004583001137\n",
            "Iteration: 18000. Loss: 0.0032307635992765427\n",
            "Iteration: 19000. Loss: 0.0029604274313896894\n",
            "Iteration: 20000. Loss: 0.002726608654484153\n",
            "Iteration: 21000. Loss: 0.002522808965295553\n",
            "Iteration: 22000. Loss: 0.00234392611309886\n",
            "Iteration: 23000. Loss: 0.0021859488915652037\n",
            "Iteration: 24000. Loss: 0.0020455988124012947\n",
            "Iteration: 25000. Loss: 0.0019202867988497019\n",
            "Iteration: 26000. Loss: 0.0018078390276059508\n",
            "Iteration: 27000. Loss: 0.0017064936691895127\n",
            "Iteration: 28000. Loss: 0.0016147841233760118\n",
            "Iteration: 29000. Loss: 0.0015314616030082107\n",
            "Iteration: 30000. Loss: 0.0014554993249475956\n",
            "Iteration: 31000. Loss: 0.001386013231240213\n",
            "Iteration: 32000. Loss: 0.0013222538400441408\n",
            "Iteration: 33000. Loss: 0.0012635792372748256\n",
            "Iteration: 34000. Loss: 0.0012094468111172318\n",
            "Iteration: 35000. Loss: 0.0011593480594456196\n",
            "Iteration: 36000. Loss: 0.0011129010235890746\n",
            "Iteration: 37000. Loss: 0.0010697157122194767\n",
            "Iteration: 38000. Loss: 0.001029488630592823\n",
            "Iteration: 39000. Loss: 0.000991932232864201\n",
            "Iteration: 40000. Loss: 0.0009568036184646189\n",
            "Iteration: 41000. Loss: 0.0009238815982826054\n",
            "Iteration: 42000. Loss: 0.0008929838659241796\n",
            "Iteration: 43000. Loss: 0.0008639313746243715\n",
            "Iteration: 44000. Loss: 0.000836574996355921\n",
            "Iteration: 45000. Loss: 0.0008107548346742988\n",
            "Iteration: 46000. Loss: 0.0007863720529712737\n",
            "Iteration: 47000. Loss: 0.0007633143104612827\n",
            "Iteration: 48000. Loss: 0.0007414728752337396\n",
            "Iteration: 49000. Loss: 0.0007207577582448721\n",
            "Iteration: 50000. Loss: 0.0007010852568782866\n",
            "Iteration: 51000. Loss: 0.0006823879084549844\n",
            "Iteration: 52000. Loss: 0.000664597493596375\n",
            "Iteration: 53000. Loss: 0.0006476510898210108\n",
            "Iteration: 54000. Loss: 0.0006314850761555135\n",
            "Iteration: 55000. Loss: 0.0006160429329611361\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CEpyKbPQpzMr",
        "colab_type": "text"
      },
      "source": [
        "As you can see the accuracy is 96% with just 55000 iterations and it takes hardly 2 minutes to train.. The best model has accuracy of 98% for this model which is also a Neural Network. We can fine tune the network and compare the type1 or type2 errors but is not the scope of this notebook."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7Fh2KKOxh2uO",
        "colab_type": "code",
        "outputId": "5c7eb9ac-1b39-4025-e7d8-9859df873940",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "with torch.no_grad():\n",
        "          # Calculate Accuracy   \n",
        "          \n",
        "          correct = 0\n",
        "          total = 0\n",
        "          # Iterate through test dataset\n",
        "          test_images = torch.from_numpy(X_test)\n",
        "          test_target=torch.from_numpy(y_test)\n",
        "\n",
        "          # Forward pass only to get logits/output\n",
        "          outputs = model(test_images.float())\n",
        "          \n",
        "          #print(outputs.data)\n",
        "          # # Get predictions from the maximum value\n",
        "          _, predicted = torch.max(outputs.data, 1)\n",
        "          \n",
        "\n",
        "          # Total number of labels\n",
        "          total += test_target.size(0)\n",
        "          \n",
        "          # Total correct predictions\n",
        "          correct += (predicted == test_target).sum()\n",
        "          \n",
        "          accuracy = 100 * correct / total\n",
        "          print('Iteration: {}. Loss: {}. Accuracy: {}'.format(iter, loss.item(), accuracy))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Iteration: 55000. Loss: 0.0006160429329611361. Accuracy: 96\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2XjlBUQMsDvR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}