2B_MultiLayer_Perceptron_Assignment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyTorch Assignment: Multi-Layer Perceptron (MLP)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[Duke Community Standard](http://integrity.duke.edu/standard.html): By typing your name below, you are certifying that you have adhered to the Duke Community Standard in completing this assignment.**\n",
    "\n",
    "Name: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multi-Layer Perceptrons\n",
    "\n",
    "The simple logistic regression example we went over in the previous notebook is essentially a one-layer neural network, projecting straight from the input to the output predictions.\n",
    "While this can be effective for linearly separable data, occasionally a little more complexity is necessary.\n",
    "Neural networks with additional layers are typically able to learn more complex functions, leading to better performance.\n",
    "These additional layers (called \"hidden\" layers) transform the input into one or more intermediate representations before making a final prediction.\n",
    "\n",
    "In the logistic regression example, the way we performed the transformation was with a fully-connected layer, which consisted of a linear transform (matrix multiply plus a bias).\n",
    "A neural network consisting of multiple successive fully-connected layers is commonly called a Multi-Layer Perceptron (MLP). \n",
    "In the simple MLP below, a 4-d input is projected to a 5-d hidden representation, which is then projected to a single output that is used to make the final prediction.\n",
    "\n",
    "<img src=\"Figures/MLP.png\" width=\"300\"/>\n",
    "\n",
    "For the assignment, you will be building a MLP for MNIST.\n",
    "Mechanically, this is done very similary to our logistic regression example, but instead of going straight to a 10-d vector representing our output predictions, we might first transform to a 500-d vector with a \"hidden\" layer, then to the output of dimension 10.\n",
    "Before you do so, however, there's one more important thing to consider.\n",
    "\n",
    "### Nonlinearities\n",
    "\n",
    "We typically include nonlinearities between layers of a neural network.\n",
    "There's a number of reasons to do so.\n",
    "For one, without anything nonlinear between them, successive linear transforms (fully connected layers) collapse into a single linear transform, which means the model isn't any more expressive than a single layer.\n",
    "On the other hand, intermediate nonlinearities prevent this collapse, allowing neural networks to approximate more complex functions.\n",
    "\n",
    "There are a number of nonlinearities commonly used in neural networks, but one of the most popular is the [rectified linear unit (ReLU)](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)):\n",
    "\n",
    "\\begin{align}\n",
    "x = \\max(0,x)\n",
    "\\end{align}\n",
    "\n",
    "There are a number of ways to implement this in PyTorch.\n",
    "We could do it with elementary PyTorch operations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: tensor([[ 0.3996,  0.8827,  0.6843],\n",
      "        [ 0.6203,  0.3854,  0.4527],\n",
      "        [-0.3724,  0.9542, -0.3486],\n",
      "        [-0.2337,  0.5612, -0.1787],\n",
      "        [ 0.3337,  0.3465,  0.6031]])\n",
      "x after ReLU with max: tensor([[0.3996, 0.8827, 0.6843],\n",
      "        [0.6203, 0.3854, 0.4527],\n",
      "        [0.0000, 0.9542, 0.0000],\n",
      "        [0.0000, 0.5612, 0.0000],\n",
      "        [0.3337, 0.3465, 0.6031]])\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "x = torch.rand(5, 3)*2 - 1\n",
    "x_relu_max = torch.max(torch.zeros_like(x),x)\n",
    "\n",
    "print(\"x: {}\".format(x))\n",
    "print(\"x after ReLU with max: {}\".format(x_relu_max))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, PyTorch also has the ReLU implemented, for example in `torch.nn.functional`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x after ReLU with nn.functional: tensor([[0.3996, 0.8827, 0.6843],\n",
      "        [0.6203, 0.3854, 0.4527],\n",
      "        [0.0000, 0.9542, 0.0000],\n",
      "        [0.0000, 0.5612, 0.0000],\n",
      "        [0.3337, 0.3465, 0.6031]])\n"
     ]
    }
   ],
   "source": [
    "import torch.nn.functional as F\n",
    "\n",
    "x_relu_F = F.relu(x)\n",
    "\n",
    "print(\"x after ReLU with nn.functional: {}\".format(x_relu_F))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assignment\n",
    "\n",
    "Build a 2-layer MLP for MNIST digit classfication. Feel free to play around with the model architecture and see how the training time/performance changes, but to begin, try the following:\n",
    "\n",
    "Image (784 dimensions) ->  \n",
    "fully connected layer (500 hidden units) -> nonlinearity (ReLU) ->  \n",
    "fully connected (10 hidden units) -> softmax\n",
    "\n",
    "Try building the model both with basic PyTorch operations, and then again with more object-oriented higher-level APIs. \n",
    "You should get similar results!\n",
    "\n",
    "\n",
    "*Some hints*:\n",
    "- Even as we add additional layers, we still only require a single optimizer to learn the parameters.\n",
    "Just make sure to pass all parameters to it!\n",
    "- As you'll calculate in the Short Answer, this MLP model has many more parameters than the logisitic regression example, which makes it more challenging to learn.\n",
    "To get the best performance, you may want to play with the learning rate and increase the number of training epochs.\n",
    "- Be careful using `torch.nn.CrossEntropyLoss()`. \n",
    "If you look at the [PyTorch documentation](https://pytorch.org/docs/stable/nn.html#crossentropyloss): you'll see that `torch.nn.CrossEntropyLoss()` combines the softmax operation with the cross-entropy.\n",
    "This means you need to pass in the logits (predictions pre-softmax) to this loss.\n",
    "Computing the softmax separately and feeding the result into `torch.nn.CrossEntropyLoss()` will significantly degrade your model's performance!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a02bd3af3eed449f92d8d7c5df496663",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/600 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6b8ebe77570748cb98316f6a612407f7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/100 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.9115999937057495\n"
     ]
    }
   ],
   "source": [
    "### YOUR CODE HERE\n",
    "\n",
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn.functional as F\n",
    "from torchvision import datasets, transforms\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "# Load the data\n",
    "mnist_train = datasets.MNIST(root=\"./datasets\", train=True, transform=transforms.ToTensor(), download=True)\n",
    "mnist_test = datasets.MNIST(root=\"./datasets\", train=False, transform=transforms.ToTensor(), download=True)\n",
    "train_loader = torch.utils.data.DataLoader(mnist_train, batch_size=100, shuffle=True)\n",
    "test_loader = torch.utils.data.DataLoader(mnist_test, batch_size=100, shuffle=False)\n",
    "\n",
    "## Training\n",
    "# Initialize parameters\n",
    "W_0 = torch.randn(784, 500)/np.sqrt(784)\n",
    "W_0.requires_grad_()\n",
    "b_0 = torch.zeros(500, requires_grad=True)\n",
    "\n",
    "W_1 = torch.randn(500, 10)/np.sqrt(500)\n",
    "W_1.requires_grad_()\n",
    "b_1 = torch.zeros(10, requires_grad=True)\n",
    "\n",
    "# Optimizer\n",
    "optimizer = torch.optim.SGD([W_0,b_0, W_1, b_1], lr=0.1)\n",
    "\n",
    "# Iterate through train set minibatchs \n",
    "for images, labels in tqdm(train_loader):\n",
    "    # Zero out the gradients\n",
    "    optimizer.zero_grad()\n",
    "    \n",
    "    # Forward pass\n",
    "    x = images.view(-1, 28*28)\n",
    "    y = torch.matmul(x, W_0) + b_0\n",
    "    y_relu = F.relu(y)\n",
    "    \n",
    "    z = torch.matmul(y, W_1) + b_1\n",
    "    cross_entropy = F.cross_entropy(z, labels)\n",
    "    # Backward pass\n",
    "    cross_entropy.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "## Testing\n",
    "correct = 0\n",
    "total = len(mnist_test)\n",
    "\n",
    "with torch.no_grad():\n",
    "    # Iterate through test set minibatchs \n",
    "    for images, labels in tqdm(test_loader):\n",
    "        # Forward pass\n",
    "        x = images.view(-1, 28*28)\n",
    "        y = torch.matmul(x, W_0) + b_0\n",
    "        y_relu = F.relu(y)\n",
    "        z = torch.matmul(y, W_1) + b_1\n",
    "        \n",
    "        predictions = torch.argmax(z, dim=1)\n",
    "        correct += torch.sum((predictions == labels).float())\n",
    "    \n",
    "print('Test accuracy: {}'.format(correct/total))\n",
    "# Make sure to print out your accuracy on the test set at the end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "175519304b434e9cbc4954a8e59e4f49",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/600 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "64ca56b3025c4f08a01882b0af2ad8f2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/600 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "904024494d6d485699600b693603867b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/100 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.9363999962806702\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torchvision import datasets, transforms\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "dim = 784\n",
    "layer1 = 500\n",
    "layer2 = 10\n",
    "LR = 0.1\n",
    "\n",
    "class MNIST_MLP(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.lin1 = nn.Linear(dim, layer1)\n",
    "        self.relu = nn.ReLU()\n",
    "        self.lin2 = nn.Linear(layer1, layer2)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        y_prime = self.lin1(x)\n",
    "        y_prime_relu = self.relu(y_prime)\n",
    "        return self.lin2(y_prime_relu)\n",
    "    \n",
    "# Load the data\n",
    "mnist_train = datasets.MNIST(root=\"./datasets\", train=True, transform=transforms.ToTensor(), download=True)\n",
    "mnist_test = datasets.MNIST(root=\"./datasets\", train=False, transform=transforms.ToTensor(), download=True)\n",
    "train_loader = torch.utils.data.DataLoader(mnist_train, batch_size=100, shuffle=True)\n",
    "test_loader = torch.utils.data.DataLoader(mnist_test, batch_size=100, shuffle=False)\n",
    "\n",
    "## Training\n",
    "# Instantiate model\n",
    "model = MNIST_MLP()\n",
    "\n",
    "# Loss and Optimizer\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=LR)\n",
    "\n",
    "for images, labels in tqdm(train_loader):\n",
    "    optimizer.zero_grad()\n",
    "    \n",
    "    x = images.view(-1, 28*28)\n",
    "    y = model(x)\n",
    "    loss = criterion(y, labels)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "for images, labels in tqdm(train_loader): #2nd epoch (accuracy from 92 to 93 %)\n",
    "    optimizer.zero_grad()\n",
    "    \n",
    "    x = images.view(-1, 28*28)\n",
    "    y = model(x)\n",
    "    loss = criterion(y, labels)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "## Testing\n",
    "correct = 0\n",
    "total = len(mnist_test)\n",
    "\n",
    "with torch.no_grad():\n",
    "    # Iterate through test set minibatchs \n",
    "    for images, labels in tqdm(test_loader):\n",
    "        # Forward pass\n",
    "        x = images.view(-1, 28*28)\n",
    "        y = model(x)\n",
    "        \n",
    "        predictions = torch.argmax(y, dim=1)\n",
    "        correct += torch.sum((predictions == labels).float())\n",
    "    \n",
    "print('Test accuracy: {}'.format(correct/total))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Short answer\n",
    "How many trainable parameters does your model have? \n",
    "How does this compare to the logisitic regression example?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`[Your answer here]`\n",
    "\n",
    "MLP:500x784 + 500 + 10x500 + 10\n",
    "\n",
    "LR: 10x784+10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.12 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "23574fd47338620f5fa9034c1801f73296c9ae358dd5ea70f96243aff333c802"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}