3.Plain_Pytorch_Neural_Net.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Pytorch](images/pytorch_logo.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning in Pure Pytorch\n",
    "Let's take our Regression and add a hidden layer to it, so we have **Deep Learning&trade;**!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.datasets import load_boston"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set seed\n",
    "seed = 42\n",
    "torch.manual_seed(seed);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data\n",
    "\n",
    "We are using the built-in sklearn dataset Boston House Prices.\n",
    "\n",
    "Our goal is to predict the median price of a home in a given town from a number of features, such as Crime Rate, Property Tax Rate, amount of Industry etc.\n",
    "\n",
    "It's generally a good idea to scale our data, so we use Sklearn's MinMax scaler to scale our values between 0 and 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load our dataset\n",
    "boston = load_boston()\n",
    "train_x, test_x, train_y, test_y = train_test_split(boston.data, boston.target, random_state=seed)\n",
    "scaler = MinMaxScaler()\n",
    "\n",
    "train_x = torch.tensor(scaler.fit_transform(train_x), dtype=torch.float)\n",
    "test_x = torch.tensor(scaler.transform(test_x), dtype=torch.float)\n",
    "train_y = torch.tensor(train_y, dtype=torch.float).view(-1, 1)\n",
    "test_y = torch.tensor(test_y, dtype=torch.float).view(-1, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup parameters\n",
    "\n",
    "We have some hyperparameters to set, as well as some numbers we need to know upfront.\n",
    "\n",
    "`layer_size` --> We need to know how many input variables there are, so we can create an equivalent number of weights\n",
    "\n",
    "`lr` --> Aka learning rate.\n",
    "When we take a step in our gradient descent, we multiply by this factor, so we don't take too big or too large a step. \n",
    "\n",
    "`epochs` --> How many times should we keep stepping?\n",
    "\n",
    "**new param** `hidden_size` --> How many nodes in the 2nd layer?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set some parameters\n",
    "layer_size = train_x.shape[1]\n",
    "lr = 0.01\n",
    "epochs = 1000\n",
    "hidden_size = 16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize weights and bias\n",
    "\n",
    "We need one weight to multiply each feature with - we are learning what these should be, so we start them as a random number.\n",
    "Since we have two layers now, we need two sets of weights and biases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initializing weights\n",
    "w_1 = torch.randn(layer_size, hidden_size, requires_grad=True, dtype=torch.float)\n",
    "w_2 = torch.randn(hidden_size, 1, requires_grad=True, dtype=torch.float)\n",
    "b_1 = torch.zeros(1, requires_grad=True, dtype=torch.float)\n",
    "b_2 = torch.zeros(1, requires_grad=True, dtype=torch.float)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define Loss Function\n",
    "\n",
    "Just like before, we want to use mean squared error to say how bad or good our line is.\n",
    "\n",
    "In addition we define our non-linear function to apply to the output of each layer - in this case the **Re**ctified __L__inear **U**nit. It just means return 0 if input is negative, else return the input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define loss function\n",
    "def mean_squared_error(y_hat, y):\n",
    "    return ((y_hat - y) ** 2).mean()\n",
    "\n",
    "# Define non-linear function\n",
    "def relu(x):\n",
    "    return torch.max(torch.tensor(0, dtype=torch.float), x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0 Train Loss: 363.3018493652344 Test Loss: 109.4739761352539\n",
      "Epoch: 10 Train Loss: 102.40554809570312 Test Loss: 88.30647277832031\n",
      "Epoch: 20 Train Loss: 96.2284927368164 Test Loss: 78.51646423339844\n",
      "Epoch: 30 Train Loss: 77.7370834350586 Test Loss: 66.46686553955078\n",
      "Epoch: 40 Train Loss: 68.07283020019531 Test Loss: 58.34284973144531\n",
      "Epoch: 50 Train Loss: 60.89570236206055 Test Loss: 52.54545974731445\n",
      "Epoch: 60 Train Loss: 56.42441177368164 Test Loss: 48.4997673034668\n",
      "Epoch: 70 Train Loss: 52.62266159057617 Test Loss: 45.205509185791016\n",
      "Epoch: 80 Train Loss: 49.424903869628906 Test Loss: 42.511512756347656\n",
      "Epoch: 90 Train Loss: 46.830299377441406 Test Loss: 40.26917266845703\n",
      "Epoch: 100 Train Loss: 43.64454650878906 Test Loss: 37.8774299621582\n",
      "Epoch: 110 Train Loss: 40.93580627441406 Test Loss: 35.834739685058594\n",
      "Epoch: 120 Train Loss: 38.56975173950195 Test Loss: 34.152000427246094\n",
      "Epoch: 130 Train Loss: 36.780113220214844 Test Loss: 32.845672607421875\n",
      "Epoch: 140 Train Loss: 35.17841339111328 Test Loss: 31.687761306762695\n",
      "Epoch: 150 Train Loss: 33.84559631347656 Test Loss: 30.76373291015625\n",
      "Epoch: 160 Train Loss: 32.826560974121094 Test Loss: 30.016803741455078\n",
      "Epoch: 170 Train Loss: 31.69351577758789 Test Loss: 29.267698287963867\n",
      "Epoch: 180 Train Loss: 30.82904815673828 Test Loss: 28.667911529541016\n",
      "Epoch: 190 Train Loss: 29.978971481323242 Test Loss: 28.093921661376953\n",
      "Epoch: 200 Train Loss: 29.256900787353516 Test Loss: 27.603544235229492\n",
      "Epoch: 210 Train Loss: 29.065488815307617 Test Loss: 27.593475341796875\n",
      "Epoch: 220 Train Loss: 28.84413719177246 Test Loss: 27.442293167114258\n",
      "Epoch: 230 Train Loss: 28.662202835083008 Test Loss: 27.384967803955078\n",
      "Epoch: 240 Train Loss: 27.875490188598633 Test Loss: 26.880525588989258\n",
      "Epoch: 250 Train Loss: 26.9995059967041 Test Loss: 26.362287521362305\n",
      "Epoch: 260 Train Loss: 26.589601516723633 Test Loss: 26.103130340576172\n",
      "Epoch: 270 Train Loss: 26.325149536132812 Test Loss: 25.903573989868164\n",
      "Epoch: 280 Train Loss: 26.060739517211914 Test Loss: 25.727924346923828\n",
      "Epoch: 290 Train Loss: 25.871591567993164 Test Loss: 25.621883392333984\n",
      "Epoch: 300 Train Loss: 25.796733856201172 Test Loss: 25.57125473022461\n",
      "Epoch: 310 Train Loss: 25.678855895996094 Test Loss: 25.479358673095703\n",
      "Epoch: 320 Train Loss: 25.30638885498047 Test Loss: 25.247852325439453\n",
      "Epoch: 330 Train Loss: 24.969507217407227 Test Loss: 25.05238151550293\n",
      "Epoch: 340 Train Loss: 24.892810821533203 Test Loss: 24.980064392089844\n",
      "Epoch: 350 Train Loss: 24.73666763305664 Test Loss: 24.863740921020508\n",
      "Epoch: 360 Train Loss: 24.495418548583984 Test Loss: 24.69251823425293\n",
      "Epoch: 370 Train Loss: 24.207332611083984 Test Loss: 24.51780891418457\n",
      "Epoch: 380 Train Loss: 24.072649002075195 Test Loss: 24.431392669677734\n",
      "Epoch: 390 Train Loss: 23.906435012817383 Test Loss: 24.30495262145996\n",
      "Epoch: 400 Train Loss: 23.860177993774414 Test Loss: 24.290456771850586\n",
      "Epoch: 410 Train Loss: 23.911039352416992 Test Loss: 24.35488510131836\n",
      "Epoch: 420 Train Loss: 23.76475715637207 Test Loss: 24.231407165527344\n",
      "Epoch: 430 Train Loss: 23.444129943847656 Test Loss: 24.035245895385742\n",
      "Epoch: 440 Train Loss: 23.173131942749023 Test Loss: 23.876659393310547\n",
      "Epoch: 450 Train Loss: 22.9772891998291 Test Loss: 23.744770050048828\n",
      "Epoch: 460 Train Loss: 22.895767211914062 Test Loss: 23.672088623046875\n",
      "Epoch: 470 Train Loss: 22.74297523498535 Test Loss: 23.57851791381836\n",
      "Epoch: 480 Train Loss: 22.287948608398438 Test Loss: 23.28433609008789\n",
      "Epoch: 490 Train Loss: 22.07915496826172 Test Loss: 23.162593841552734\n",
      "Epoch: 500 Train Loss: 22.054304122924805 Test Loss: 23.14552879333496\n",
      "Epoch: 510 Train Loss: 22.125492095947266 Test Loss: 23.221349716186523\n",
      "Epoch: 520 Train Loss: 22.01447296142578 Test Loss: 23.127216339111328\n",
      "Epoch: 530 Train Loss: 21.72856903076172 Test Loss: 22.962892532348633\n",
      "Epoch: 540 Train Loss: 21.60595703125 Test Loss: 22.912384033203125\n",
      "Epoch: 550 Train Loss: 21.63629913330078 Test Loss: 22.938608169555664\n",
      "Epoch: 560 Train Loss: 21.633121490478516 Test Loss: 22.93720054626465\n",
      "Epoch: 570 Train Loss: 21.614107131958008 Test Loss: 22.930950164794922\n",
      "Epoch: 580 Train Loss: 21.46636390686035 Test Loss: 22.8277645111084\n",
      "Epoch: 590 Train Loss: 21.210954666137695 Test Loss: 22.697616577148438\n",
      "Epoch: 600 Train Loss: 21.071453094482422 Test Loss: 22.64063262939453\n",
      "Epoch: 610 Train Loss: 20.945186614990234 Test Loss: 22.592098236083984\n",
      "Epoch: 620 Train Loss: 21.048330307006836 Test Loss: 22.638601303100586\n",
      "Epoch: 630 Train Loss: 20.913076400756836 Test Loss: 22.56533432006836\n",
      "Epoch: 640 Train Loss: 20.775800704956055 Test Loss: 22.502582550048828\n",
      "Epoch: 650 Train Loss: 20.58298683166504 Test Loss: 22.38600730895996\n",
      "Epoch: 660 Train Loss: 20.498600006103516 Test Loss: 22.345373153686523\n",
      "Epoch: 670 Train Loss: 20.58557891845703 Test Loss: 22.370494842529297\n",
      "Epoch: 680 Train Loss: 20.624696731567383 Test Loss: 22.39405632019043\n",
      "Epoch: 690 Train Loss: 20.595529556274414 Test Loss: 22.37335205078125\n",
      "Epoch: 700 Train Loss: 20.41478157043457 Test Loss: 22.257892608642578\n",
      "Epoch: 710 Train Loss: 20.209867477416992 Test Loss: 22.16459846496582\n",
      "Epoch: 720 Train Loss: 20.172218322753906 Test Loss: 22.13526153564453\n",
      "Epoch: 730 Train Loss: 20.097911834716797 Test Loss: 22.085285186767578\n",
      "Epoch: 740 Train Loss: 20.135141372680664 Test Loss: 22.08552360534668\n",
      "Epoch: 750 Train Loss: 20.044178009033203 Test Loss: 22.01833724975586\n",
      "Epoch: 760 Train Loss: 19.970651626586914 Test Loss: 22.01441764831543\n",
      "Epoch: 770 Train Loss: 19.967533111572266 Test Loss: 21.975326538085938\n",
      "Epoch: 780 Train Loss: 19.761592864990234 Test Loss: 21.881065368652344\n",
      "Epoch: 790 Train Loss: 19.572477340698242 Test Loss: 21.757991790771484\n",
      "Epoch: 800 Train Loss: 19.58597183227539 Test Loss: 21.743284225463867\n",
      "Epoch: 810 Train Loss: 19.45720672607422 Test Loss: 21.669126510620117\n",
      "Epoch: 820 Train Loss: 19.279123306274414 Test Loss: 21.55274772644043\n",
      "Epoch: 830 Train Loss: 19.18411636352539 Test Loss: 21.50772476196289\n",
      "Epoch: 840 Train Loss: 19.142024993896484 Test Loss: 21.468326568603516\n",
      "Epoch: 850 Train Loss: 19.068071365356445 Test Loss: 21.410484313964844\n",
      "Epoch: 860 Train Loss: 18.932937622070312 Test Loss: 21.322290420532227\n",
      "Epoch: 870 Train Loss: 18.87670135498047 Test Loss: 21.270397186279297\n",
      "Epoch: 880 Train Loss: 18.769412994384766 Test Loss: 21.188982009887695\n",
      "Epoch: 890 Train Loss: 18.72982406616211 Test Loss: 21.162565231323242\n",
      "Epoch: 900 Train Loss: 18.66480255126953 Test Loss: 21.093984603881836\n",
      "Epoch: 910 Train Loss: 18.587627410888672 Test Loss: 21.00994300842285\n",
      "Epoch: 920 Train Loss: 18.40464210510254 Test Loss: 20.915178298950195\n",
      "Epoch: 930 Train Loss: 18.292844772338867 Test Loss: 20.82303237915039\n",
      "Epoch: 940 Train Loss: 18.240833282470703 Test Loss: 20.77931785583496\n",
      "Epoch: 950 Train Loss: 18.116436004638672 Test Loss: 20.69620704650879\n",
      "Epoch: 960 Train Loss: 18.02684211730957 Test Loss: 20.623424530029297\n",
      "Epoch: 970 Train Loss: 17.988386154174805 Test Loss: 20.572057723999023\n",
      "Epoch: 980 Train Loss: 17.919919967651367 Test Loss: 20.51207733154297\n",
      "Epoch: 990 Train Loss: 17.874494552612305 Test Loss: 20.447816848754883\n"
     ]
    }
   ],
   "source": [
    "# Training loop\n",
    "for epoch in range(epochs):\n",
    "    layer_1 = relu(train_x @ w_1 + b_1) # Result of first layer...\n",
    "    pred = layer_1 @ w_2 + b_2 # ...is passed through the second layer\n",
    "    \n",
    "    loss = mean_squared_error(pred, train_y)\n",
    "    \n",
    "    # Backpropagation\n",
    "    loss.backward() # The magic bit!\n",
    "    with torch.no_grad():\n",
    "        w_1 -= w_1.grad * lr\n",
    "        w_2 -= w_2.grad * lr\n",
    "        b_1 -= b_1.grad * lr\n",
    "        b_2 -= b_2.grad * lr\n",
    "        w_1.grad.zero_()\n",
    "        w_2.grad.zero_()\n",
    "        b_1.grad.zero_()\n",
    "        b_2.grad.zero_()\n",
    "        \n",
    "        if epoch % 10 == 0:\n",
    "            val_pred = relu(test_x @ w_1 + b_1) @ w_2 + b_2\n",
    "            val_loss = mean_squared_error(val_pred, test_y)\n",
    "            print(f\"Epoch: {epoch} Train Loss: {loss.item()} Test Loss: {val_loss.item()}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}