4.Pytorch_Neural_Net.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Pytorch](images/pytorch_logo.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning in PyTorch with all the bells and whistles\n",
    "Let's reimplement our **Deep Learning&trade;** network, but this time we use all the toys!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from sklearn.datasets import load_boston\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "# Some new imports\n",
    "from torch import nn\n",
    "import torch.nn.functional as F\n",
    "from torch import optim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set seed\n",
    "seed = 42\n",
    "torch.manual_seed(seed);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data\n",
    "\n",
    "We are using the built-in sklearn dataset Boston House Prices.\n",
    "\n",
    "Our goal is to predict the median price of a home in a given town from a number of features, such as Crime Rate, Property Tax Rate, amount of Industry etc.\n",
    "\n",
    "It's generally a good idea to scale our data, so we use Sklearn's MinMax scaler to scale our values between 0 and 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load our dataset\n",
    "boston = load_boston()\n",
    "train_x, test_x, train_y, test_y = train_test_split(boston.data, boston.target, random_state=seed)\n",
    "scaler = MinMaxScaler()\n",
    "\n",
    "train_x = torch.tensor(scaler.fit_transform(train_x), dtype=torch.float)\n",
    "test_x = torch.tensor(scaler.transform(test_x), dtype=torch.float)\n",
    "train_y = torch.tensor(train_y, dtype=torch.float).view(-1, 1)\n",
    "test_y = torch.tensor(test_y, dtype=torch.float).view(-1, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup parameters\n",
    "\n",
    "We have some hyperparameters to set, as well as some numbers we need to know upfront.\n",
    "\n",
    "`layer_size` --> We need to know how many input variables there are, so we can create an equivalent number of weights\n",
    "\n",
    "`lr` --> Aka learning rate.\n",
    "When we take a step in our gradient descent, we multiply by this factor, so we don't take too big or too large a step. \n",
    "\n",
    "`epochs` --> How many times should we keep stepping?\n",
    "\n",
    "**new param** `hidden_size` --> How many nodes in the 2nd layer?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set some parameters\n",
    "layer_size = train_x.shape[1]\n",
    "lr = 0.01\n",
    "epochs = 1000\n",
    "hidden_size = 16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Defining the Model\n",
    "\n",
    "We get to play with some new toys here - `nn.Module` makes it very easy to define a model and the `nn` module contains plenty of premade functionality such as the `nn.Linear` layer and the `ReLU` built right in. It's regression in a box!\n",
    "\n",
    "Now we see how useful our new toys are - look how easy it is for me to add 2 additional layers!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Model(nn.Module):\n",
    "    def __init__(self, layer_size, hidden_size):\n",
    "        super().__init__()\n",
    "        self.l1 = nn.Linear(layer_size, hidden_size)\n",
    "        self.l2 = nn.Linear(hidden_size, hidden_size)\n",
    "        self.l3 = nn.Linear(hidden_size, 1)\n",
    "        self.relu = nn.ReLU()\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = self.relu(self.l1(x))\n",
    "        x = self.relu(self.l2(x))\n",
    "        return self.l3(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Define loss function and optimizer\n",
    "Pytorch predefines a number of loss functions for me, so I can just use those directly, such as `nn.MSELoss()`\n",
    "\n",
    "Another new feature is the `optim.SGD` optimizer. Until now we have been updating the weights \"by hand\" in a fairly naive fashion. There are many ways of updating our weights and Pytorch provides implementations for most of these. \n",
    "\n",
    "SGD *(Stochastic Gradient Descent)* is the closest to what we've been doing so far, so we use that to handle our weight updates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Model(layer_size, hidden_size)\n",
    "opt = optim.SGD(model.parameters(), lr=lr)\n",
    "loss_func = nn.MSELoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training Loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0 Train Loss: 606.623779296875 Validation Loss: 495.802978515625\n",
      "Epoch: 10 Train Loss: 324.54266357421875 Validation Loss: 171.4607391357422\n",
      "Epoch: 20 Train Loss: 67.16031646728516 Validation Loss: 52.76789093017578\n",
      "Epoch: 30 Train Loss: 58.69449234008789 Validation Loss: 43.758201599121094\n",
      "Epoch: 40 Train Loss: 37.83702850341797 Validation Loss: 30.774845123291016\n",
      "Epoch: 50 Train Loss: 48.848297119140625 Validation Loss: 47.153846740722656\n",
      "Epoch: 60 Train Loss: 37.94041442871094 Validation Loss: 28.001794815063477\n",
      "Epoch: 70 Train Loss: 62.90554428100586 Validation Loss: 39.652427673339844\n",
      "Epoch: 80 Train Loss: 47.16319274902344 Validation Loss: 31.31838607788086\n",
      "Epoch: 90 Train Loss: 47.825740814208984 Validation Loss: 29.859413146972656\n",
      "Epoch: 100 Train Loss: 43.41510772705078 Validation Loss: 28.221847534179688\n",
      "Epoch: 110 Train Loss: 41.27399444580078 Validation Loss: 27.049591064453125\n",
      "Epoch: 120 Train Loss: 38.60835266113281 Validation Loss: 25.96039581298828\n",
      "Epoch: 130 Train Loss: 36.89274215698242 Validation Loss: 24.821922302246094\n",
      "Epoch: 140 Train Loss: 36.654300689697266 Validation Loss: 24.63176727294922\n",
      "Epoch: 150 Train Loss: 36.02035140991211 Validation Loss: 24.26666831970215\n",
      "Epoch: 160 Train Loss: 33.71978759765625 Validation Loss: 23.978553771972656\n",
      "Epoch: 170 Train Loss: 32.328060150146484 Validation Loss: 23.199262619018555\n",
      "Epoch: 180 Train Loss: 30.872652053833008 Validation Loss: 22.602283477783203\n",
      "Epoch: 190 Train Loss: 29.886734008789062 Validation Loss: 22.087142944335938\n",
      "Epoch: 200 Train Loss: 28.213571548461914 Validation Loss: 21.20842742919922\n",
      "Epoch: 210 Train Loss: 28.417978286743164 Validation Loss: 21.483497619628906\n",
      "Epoch: 220 Train Loss: 28.024433135986328 Validation Loss: 20.819482803344727\n",
      "Epoch: 230 Train Loss: 26.707000732421875 Validation Loss: 20.48577308654785\n",
      "Epoch: 240 Train Loss: 25.996248245239258 Validation Loss: 20.18963050842285\n",
      "Epoch: 250 Train Loss: 25.43320655822754 Validation Loss: 19.99430274963379\n",
      "Epoch: 260 Train Loss: 24.76863670349121 Validation Loss: 19.71442985534668\n",
      "Epoch: 270 Train Loss: 24.223432540893555 Validation Loss: 19.464570999145508\n",
      "Epoch: 280 Train Loss: 23.92273712158203 Validation Loss: 19.28449058532715\n",
      "Epoch: 290 Train Loss: 23.115415573120117 Validation Loss: 18.9387264251709\n",
      "Epoch: 300 Train Loss: 21.929758071899414 Validation Loss: 18.560012817382812\n",
      "Epoch: 310 Train Loss: 22.263671875 Validation Loss: 18.437442779541016\n",
      "Epoch: 320 Train Loss: 21.320392608642578 Validation Loss: 18.240509033203125\n",
      "Epoch: 330 Train Loss: 21.37239646911621 Validation Loss: 18.087486267089844\n",
      "Epoch: 340 Train Loss: 21.20035171508789 Validation Loss: 17.976940155029297\n",
      "Epoch: 350 Train Loss: 20.688928604125977 Validation Loss: 17.805341720581055\n",
      "Epoch: 360 Train Loss: 20.774845123291016 Validation Loss: 17.79633903503418\n",
      "Epoch: 370 Train Loss: 20.314537048339844 Validation Loss: 17.57132339477539\n",
      "Epoch: 380 Train Loss: 19.60828971862793 Validation Loss: 17.59961700439453\n",
      "Epoch: 390 Train Loss: 19.392578125 Validation Loss: 17.46255111694336\n",
      "Epoch: 400 Train Loss: 19.1945858001709 Validation Loss: 17.325511932373047\n",
      "Epoch: 410 Train Loss: 19.61870765686035 Validation Loss: 17.309499740600586\n",
      "Epoch: 420 Train Loss: 19.073156356811523 Validation Loss: 17.257726669311523\n",
      "Epoch: 430 Train Loss: 18.837738037109375 Validation Loss: 17.115537643432617\n",
      "Epoch: 440 Train Loss: 18.25390625 Validation Loss: 16.986480712890625\n",
      "Epoch: 450 Train Loss: 18.254199981689453 Validation Loss: 16.959779739379883\n",
      "Epoch: 460 Train Loss: 17.325395584106445 Validation Loss: 16.741464614868164\n",
      "Epoch: 470 Train Loss: 17.8006534576416 Validation Loss: 16.780845642089844\n",
      "Epoch: 480 Train Loss: 16.75773811340332 Validation Loss: 16.458721160888672\n",
      "Epoch: 490 Train Loss: 17.044368743896484 Validation Loss: 16.661821365356445\n",
      "Epoch: 500 Train Loss: 16.26368522644043 Validation Loss: 16.30971336364746\n",
      "Epoch: 510 Train Loss: 16.485017776489258 Validation Loss: 16.32636070251465\n",
      "Epoch: 520 Train Loss: 15.534064292907715 Validation Loss: 16.041208267211914\n",
      "Epoch: 530 Train Loss: 15.534432411193848 Validation Loss: 16.067731857299805\n",
      "Epoch: 540 Train Loss: 16.006481170654297 Validation Loss: 16.276714324951172\n",
      "Epoch: 550 Train Loss: 15.563882827758789 Validation Loss: 16.20657730102539\n",
      "Epoch: 560 Train Loss: 15.485568046569824 Validation Loss: 16.218095779418945\n",
      "Epoch: 570 Train Loss: 15.713105201721191 Validation Loss: 16.347061157226562\n",
      "Epoch: 580 Train Loss: 15.644527435302734 Validation Loss: 16.39573097229004\n",
      "Epoch: 590 Train Loss: 15.661449432373047 Validation Loss: 16.362720489501953\n",
      "Epoch: 600 Train Loss: 15.08491325378418 Validation Loss: 15.959749221801758\n",
      "Epoch: 610 Train Loss: 14.937925338745117 Validation Loss: 15.851583480834961\n",
      "Epoch: 620 Train Loss: 14.601248741149902 Validation Loss: 15.701205253601074\n",
      "Epoch: 630 Train Loss: 14.301410675048828 Validation Loss: 15.54996395111084\n",
      "Epoch: 640 Train Loss: 14.480409622192383 Validation Loss: 15.460403442382812\n",
      "Epoch: 650 Train Loss: 13.59145450592041 Validation Loss: 15.286134719848633\n",
      "Epoch: 660 Train Loss: 14.127201080322266 Validation Loss: 15.360635757446289\n",
      "Epoch: 670 Train Loss: 13.473465919494629 Validation Loss: 15.181802749633789\n",
      "Epoch: 680 Train Loss: 13.207205772399902 Validation Loss: 15.108858108520508\n",
      "Epoch: 690 Train Loss: 13.267156600952148 Validation Loss: 15.165528297424316\n",
      "Epoch: 700 Train Loss: 13.181891441345215 Validation Loss: 14.933782577514648\n",
      "Epoch: 710 Train Loss: 12.854429244995117 Validation Loss: 15.029191970825195\n",
      "Epoch: 720 Train Loss: 13.203179359436035 Validation Loss: 14.974419593811035\n",
      "Epoch: 730 Train Loss: 13.076308250427246 Validation Loss: 15.161513328552246\n",
      "Epoch: 740 Train Loss: 12.867422103881836 Validation Loss: 14.839799880981445\n",
      "Epoch: 750 Train Loss: 12.555878639221191 Validation Loss: 15.074254035949707\n",
      "Epoch: 760 Train Loss: 12.760419845581055 Validation Loss: 14.881545066833496\n",
      "Epoch: 770 Train Loss: 12.222660064697266 Validation Loss: 14.96712875366211\n",
      "Epoch: 780 Train Loss: 12.330111503601074 Validation Loss: 14.728710174560547\n",
      "Epoch: 790 Train Loss: 11.946552276611328 Validation Loss: 14.85890007019043\n",
      "Epoch: 800 Train Loss: 12.144166946411133 Validation Loss: 14.694925308227539\n",
      "Epoch: 810 Train Loss: 11.804481506347656 Validation Loss: 14.834359169006348\n",
      "Epoch: 820 Train Loss: 12.012144088745117 Validation Loss: 14.664719581604004\n",
      "Epoch: 830 Train Loss: 11.292776107788086 Validation Loss: 15.010109901428223\n",
      "Epoch: 840 Train Loss: 12.508331298828125 Validation Loss: 14.858146667480469\n",
      "Epoch: 850 Train Loss: 11.747228622436523 Validation Loss: 15.267419815063477\n",
      "Epoch: 860 Train Loss: 12.047210693359375 Validation Loss: 14.707488059997559\n",
      "Epoch: 870 Train Loss: 11.217974662780762 Validation Loss: 14.638837814331055\n",
      "Epoch: 880 Train Loss: 11.55931282043457 Validation Loss: 14.540738105773926\n",
      "Epoch: 890 Train Loss: 11.233378410339355 Validation Loss: 15.055017471313477\n",
      "Epoch: 900 Train Loss: 11.973969459533691 Validation Loss: 14.64835262298584\n",
      "Epoch: 910 Train Loss: 10.881865501403809 Validation Loss: 14.727633476257324\n",
      "Epoch: 920 Train Loss: 11.431143760681152 Validation Loss: 14.502288818359375\n",
      "Epoch: 930 Train Loss: 10.634356498718262 Validation Loss: 14.76683235168457\n",
      "Epoch: 940 Train Loss: 11.253118515014648 Validation Loss: 14.47608470916748\n",
      "Epoch: 950 Train Loss: 10.56429672241211 Validation Loss: 14.632436752319336\n",
      "Epoch: 960 Train Loss: 11.031052589416504 Validation Loss: 14.648893356323242\n",
      "Epoch: 970 Train Loss: 10.6365966796875 Validation Loss: 14.413620948791504\n",
      "Epoch: 980 Train Loss: 11.157353401184082 Validation Loss: 14.473560333251953\n",
      "Epoch: 990 Train Loss: 10.754892349243164 Validation Loss: 14.620599746704102\n",
      "Epoch: 999 Train Loss: 9.659093856811523 Validation Loss: 14.620599746704102\n"
     ]
    }
   ],
   "source": [
    "# Training loop\n",
    "for epoch in range(epochs):\n",
    "    # Forward pass\n",
    "    pred = model(train_x) # Our model acts like a function!\n",
    "    loss = loss_func(pred, train_y)\n",
    "    \n",
    "    # Backpropagation    \n",
    "    loss.backward() # The magic bit from previously\n",
    "    opt.step() # A new magic bit\n",
    "    opt.zero_grad() # Gotta reset the gradients to zero, so they don't accumulate\n",
    "    \n",
    "    if epoch % 10 == 0:\n",
    "        val_pred = model(test_x)\n",
    "        val_loss = loss_func(val_pred, test_y) # Calculate validation loss\n",
    "        print(f\"Epoch: {epoch} Train Loss: {loss.item()} Validation Loss: {val_loss.item()}\")\n",
    "print(f\"Epoch: {epoch} Train Loss: {loss.item()} Validation Loss: {val_loss.item()}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}