src/neural_networks/pytorch_introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "8b34af0b-a325-45c2-8f12-091fea83e194",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "# PyTorch\n",
    "\n",
    "* Developed at Facebook AI in 2016\n",
    "* Preferable to quickly prototype, and by researchers\n",
    "* More Pythonic, and easy to tinker with low level details than Tensorflow, according to most users\n",
    "* [Building the Same Model with PyTorch & TensorFlow](https://www.youtube.com/watch?v=ay1E1f8VqP8)\n",
    "* [Crash Course Notebook](https://colab.research.google.com/drive/1eiUBpmQ4m7Lbxqi2xth1jBaL61XTKdxp?usp=sharing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "a0754757-5cb3-4dbd-8363-072616fefb25",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "### Components"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "2fb9a6d3-2c09-4a53-aae6-a5e712a46be1",
     "showTitle": true,
     "title": "torch.NN import"
    }
   },
   "outputs": [],
   "source": [
    "from torch import nn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "bb1b5c6c-9f76-42fe-96f3-3d1e62360644",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "* Every model created in Pytorch inherits from `pytorch.nn` called `nn.module`\n",
    "* We define the inputs and outputs of the model along with the definitions of the layers of the network within this class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "d4b76e3b-60ef-4b4f-818e-dc4eeb58f7ab",
     "showTitle": true,
     "title": "Extending NN class"
    }
   },
   "outputs": [],
   "source": [
    "class SimpleNeuralNet(nn.Module):\n",
    "    ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "f0fd62be-dca1-4072-a29c-a1bb8b9329bc",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "* Every Pytorch inherited neural net needs to implement the `__init__` method and `forward` method.\n",
    "* The `__init__` can define the layers of the model, and `forward` defines how data is passed from one layer to the next. \n",
    "> `__init__` is like the static declaration, while `forward` is like `main()` or where execution comes forth across the static components"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "c3685e64-af46-4693-9a32-fe3bcc9636c0",
     "showTitle": true,
     "title": "Defining required methods"
    }
   },
   "outputs": [],
   "source": [
    "class SimpleNeuralNet(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "    \n",
    "    def forward(self, x):\n",
    "        ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "5a2b357a-b528-4e4c-96b2-959f82352be9",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Declare a GPU torch device and pass the model object to the GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "cca9dbdf-27de-40c7-afc6-0c6638afc556",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "print(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "c4ffcbf3-159d-413e-91b6-6a57347c1d34",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "simple_neural_net_model = SimpleNeuralNet().to(device=device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "fad42bd2-db60-426e-b423-4b5856d36d4b",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "### Tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "7357c875-41c4-4623-a7a5-50e1b8132600",
     "showTitle": true,
     "title": "Tensor"
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([3])\nTrue\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([1., 2., 3.])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t = torch.Tensor([1,2,3])\n",
    "print(t.shape)\n",
    "print(t.shape == t.size())\n",
    "t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "83dbd7ee-c6a1-4b0d-9827-1f7d93bd7d76",
     "showTitle": true,
     "title": "Argmax function"
    }
   },
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor(2)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.argmax(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "3e5dadfd-c385-48eb-814f-da92c8dc3354",
     "showTitle": true,
     "title": "Multi-Dim Tensor"
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([3, 3])\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[1., 2., 3.],\n",
       "        [1., 2., 4.],\n",
       "        [2., 5., 7.]])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t2 = torch.Tensor([[1, 2, 3], [1, 2, 4], [2, 5, 7]])\n",
    "print(t2.shape)\n",
    "t2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "1a2b7720-d7ee-4ca9-9ba1-5e9c7d8f1485",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor(8)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Note different method of getting argmax\n",
    "t2.argmax()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "2f524423-f618-4655-a0bd-f1b981612e2e",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "When you call `t2.argmax()`, it treats the tensor as if it's a flat array (because no dimension is specified) and returns the index of the maximum value in this flat array. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "46fa76f9-fb00-4e56-a8c8-7a88e348c4e9",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([2, 2, 2])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t2.argmax(dim=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "e9ca79e3-35d3-4f46-9240-3dee37756f4a",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[ 1.4165e-31,  4.5904e-41,  2.6298e-16],\n         [ 3.0838e-41,  1.5414e-44,  0.0000e+00],\n         [ 2.6087e-16,  3.0838e-41,  2.2161e-16]],\n\n        [[ 3.0838e-41,  4.2039e-45,  0.0000e+00],\n         [ 0.0000e+00,  0.0000e+00,         nan],\n         [        nan, -1.0326e+15,  4.5902e-41]],\n\n        [[ 0.0000e+00,  4.5904e-41,  1.3565e-42],\n         [ 0.0000e+00,  7.0065e-45,  0.0000e+00],\n         [ 2.6087e-16,  3.0838e-41,  2.2161e-16]]])\n"
     ]
    }
   ],
   "source": [
    "print(torch.empty(3, 3, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "3359d153-d7e5-4024-a6b8-18a2e7a279b6",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[0.8454, 0.2887, 0.4848],\n         [0.9340, 0.7250, 0.1804],\n         [0.2271, 0.1709, 0.4196]],\n\n        [[0.1896, 0.2917, 0.7042],\n         [0.5166, 0.3489, 0.2211],\n         [0.5743, 0.7340, 0.5540]],\n\n        [[0.4630, 0.8700, 0.8625],\n         [0.0443, 0.8961, 0.4219],\n         [0.6832, 0.0181, 0.4393]]])\n"
     ]
    }
   ],
   "source": [
    "print(torch.rand(3, 3, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "aba847b4-9324-4ea3-8ce5-7f9b698151d1",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[0., 0., 0.],\n         [0., 0., 0.],\n         [0., 0., 0.]],\n\n        [[0., 0., 0.],\n         [0., 0., 0.],\n         [0., 0., 0.]],\n\n        [[0., 0., 0.],\n         [0., 0., 0.],\n         [0., 0., 0.]]])\n"
     ]
    }
   ],
   "source": [
    "print(torch.zeros(3, 3, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "30615f37-4f80-4e74-89a2-6490d0b989c5",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[1., 1., 1.],\n         [1., 1., 1.],\n         [1., 1., 1.]],\n\n        [[1., 1., 1.],\n         [1., 1., 1.],\n         [1., 1., 1.]],\n\n        [[1., 1., 1.],\n         [1., 1., 1.],\n         [1., 1., 1.]]])\n"
     ]
    }
   ],
   "source": [
    "print(torch.ones(3, 3, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "099d5724-ee37-4e60-a682-353ef976406c",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "application/vnd.databricks.v1+notebook": {
   "dashboards": [],
   "language": "python",
   "notebookMetadata": {
    "pythonIndentUnit": 4
   },
   "notebookName": "pytorch_introduction",
   "widgets": {}
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}