C2_W1_Lab01_Neurons_and_Layers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Optional Lab - Neurons and Layers\n",
    "In this lab we will explore the inner workings of neurons/units and layers. In particular, the lab will draw parallels to the models you have mastered in Course 1, the regression/linear model and the logistic model. The lab will introduce Tensorflow and demonstrate how these models are implemented in that framework.\n",
    "<figure>\n",
    "   <img src=\"./images/C2_W1_NeuronsAndLayers.png\"  style=\"width:540px;height:200px;\" >\n",
    "</figure>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Packages\n",
    "**Tensorflow and Keras**  \n",
    "Tensorflow is a machine learning package developed by Google. In 2019, Google integrated Keras into Tensorflow and released Tensorflow 2.0. Keras is a framework developed independently by François Chollet that creates a simple, layer-centric interface to Tensorflow. This course will be using the Keras interface. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.layers import Dense, Input\n",
    "from tensorflow.keras import Sequential\n",
    "from tensorflow.keras.losses import MeanSquaredError, BinaryCrossentropy\n",
    "from tensorflow.keras.activations import sigmoid\n",
    "from lab_utils_common import dlc\n",
    "from lab_neurons_utils import plt_prob_1d, sigmoidnp, plt_linear, plt_logistic\n",
    "plt.style.use('./deeplearning.mplstyle')\n",
    "import logging\n",
    "logging.getLogger(\"tensorflow\").setLevel(logging.ERROR)\n",
    "tf.autograph.set_verbosity(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neuron without activation - Regression/Linear Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### DataSet\n",
    "We'll use an example from Course 1, linear regression on house prices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "X_train = np.array([[1.0], [2.0]], dtype=np.float32)           #(size in 1000 square feet)\n",
    "Y_train = np.array([[300.0], [500.0]], dtype=np.float32)       #(price in 1000s of dollars)\n",
    "\n",
    "fig, ax = plt.subplots(1,1)\n",
    "ax.scatter(X_train, Y_train, marker='x', c='r', label=\"Data Points\")\n",
    "ax.legend( fontsize='xx-large')\n",
    "ax.set_ylabel('Price (in 1000s of dollars)', fontsize='xx-large')\n",
    "ax.set_xlabel('Size (1000 sqft)', fontsize='xx-large')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regression/Linear Model \n",
    "The function implemented by a neuron with no activation is the same as in Course 1, linear regression:\n",
    "$$ f_{\\mathbf{w},b}(x^{(i)}) = \\mathbf{w}\\cdot x^{(i)} + b \\tag{1}$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can define a layer with one neuron or unit and compare it to the familiar linear regression function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_layer = tf.keras.layers.Dense(units=1, activation = 'linear', )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's examine the weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_layer.get_weights()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are no weights as the weights are not yet instantiated. Let's try the model on one example in `X_train`. This will trigger the instantiation of the weights. Note, the input to the layer must be 2-D, so we'll reshape it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a1 = linear_layer(X_train[0].reshape(1,1))\n",
    "print(a1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is a tensor (another name for an array) with a shape of (1,1) or one entry.   \n",
    "Now let's look at the weights and bias. These weights are randomly initialized to small numbers and the bias defaults to being initialized to zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w, b= linear_layer.get_weights()\n",
    "print(f\"w = {w}, b={b}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A linear regression model (1) with a single input feature will have a single weight and bias. This matches the dimensions of our `linear_layer` above.   \n",
    "\n",
    "The weights are initialized to random values so let's set them to some known values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set_w = np.array([[200]])\n",
    "set_b = np.array([100])\n",
    "\n",
    "# set_weights takes a list of numpy arrays\n",
    "linear_layer.set_weights([set_w, set_b])\n",
    "print(linear_layer.get_weights())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compare equation (1) to the layer output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a1 = linear_layer(X_train[0].reshape(1,1))\n",
    "print(a1)\n",
    "alin = np.dot(set_w,X_train[0].reshape(1,1)) + set_b\n",
    "print(alin)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "They produce the same values!\n",
    "Now, we can use our linear layer to make predictions on our training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction_tf = linear_layer(X_train)\n",
    "prediction_np = np.dot( X_train, set_w) + set_b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt_linear(X_train, Y_train, prediction_tf, prediction_np)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neuron with Sigmoid activation\n",
    "The function implemented by a neuron/unit with a sigmoid activation is the same as in Course 1, logistic  regression:\n",
    "$$ f_{\\mathbf{w},b}(x^{(i)}) = g(\\mathbf{w}x^{(i)} + b) \\tag{2}$$\n",
    "where $$g(x) = sigmoid(x)$$ \n",
    "\n",
    "Let's set $w$ and $b$ to some known values and check the model.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### DataSet\n",
    "We'll use an example from Course 1, logistic regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = np.array([0., 1, 2, 3, 4, 5], dtype=np.float32).reshape(-1,1)  # 2-D Matrix\n",
    "Y_train = np.array([0,  0, 0, 1, 1, 1], dtype=np.float32).reshape(-1,1)  # 2-D Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pos = Y_train == 1\n",
    "neg = Y_train == 0\n",
    "\n",
    "fig,ax = plt.subplots(1,1,figsize=(4,3))\n",
    "ax.scatter(X_train[pos], Y_train[pos], marker='x', s=80, c = 'red', label=\"y=1\")\n",
    "ax.scatter(X_train[neg], Y_train[neg], marker='o', s=100, label=\"y=0\", facecolors='none', \n",
    "              edgecolors=dlc[\"dlblue\"],lw=3)\n",
    "\n",
    "ax.set_ylim(-0.08,1.1)\n",
    "ax.set_ylabel('y', fontsize=12)\n",
    "ax.set_xlabel('x', fontsize=12)\n",
    "ax.set_title('one variable plot')\n",
    "ax.legend(fontsize=12)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic Neuron\n",
    "We can implement a 'logistic neuron' by adding a sigmoid activation. The function of the neuron is then described by (2) above.   \n",
    "This section will create a Tensorflow Model that contains our logistic layer to demonstrate an alternate method of creating models. Tensorflow is most often used to create multi-layer models. The [Sequential](https://keras.io/guides/sequential_model/) model is a convenient means of constructing these models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model = Sequential(\n",
    "    [\n",
    "        tf.keras.layers.Dense(1, input_dim=1,  activation = 'sigmoid', name='L1')\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`model.summary()` shows the layers and number of parameters in the model. There is only one layer in this model and that layer has only one unit. The unit has two parameters, $w$ and $b$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logistic_layer = model.get_layer('L1')\n",
    "w,b = logistic_layer.get_weights()\n",
    "print(w,b)\n",
    "print(w.shape,b.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's set the weight and bias to some known values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set_w = np.array([[2]])\n",
    "set_b = np.array([-4.5])\n",
    "# set_weights takes a list of numpy arrays\n",
    "logistic_layer.set_weights([set_w, set_b])\n",
    "print(logistic_layer.get_weights())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compare equation (2) to the layer output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a1 = model.predict(X_train[0].reshape(1,1))\n",
    "print(a1)\n",
    "alog = sigmoidnp(np.dot(set_w,X_train[0].reshape(1,1)) + set_b)\n",
    "print(alog)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "They produce the same values!\n",
    "Now, we can use our logistic layer and NumPy model to make predictions on our training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt_logistic(X_train, Y_train, model, set_w, set_b, pos, neg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The shading above reflects the output of the sigmoid which varies from 0 to 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Congratulations!\n",
    "You built a very simple neural network and have explored the similarities of a neuron to the linear and logistic regression from Course 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}