C2_W1_Lab03_CoffeeRoasting_Numpy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Optional Lab - Simple Neural Network\n",
    "In this lab, we will build a small neural network using Numpy. It will be the same \"coffee roasting\" network you implemented in Tensorflow.\n",
    "   <center> <img  src=\"./images/C2_W1_CoffeeRoasting.png\" width=\"400\" />   <center/>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('./deeplearning.mplstyle')\n",
    "import tensorflow as tf\n",
    "from lab_utils_common import dlc, sigmoid\n",
    "from lab_coffee_utils import load_coffee_data, plt_roast, plt_prob, plt_layer, plt_network, plt_output_unit\n",
    "import logging\n",
    "logging.getLogger(\"tensorflow\").setLevel(logging.ERROR)\n",
    "tf.autograph.set_verbosity(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DataSet\n",
    "This is the same data set as the previous lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X,Y = load_coffee_data();\n",
    "print(X.shape, Y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the coffee roasting data below. The two features are Temperature in Celsius and Duration in minutes. [Coffee Roasting at Home](https://www.merchantsofgreencoffee.com/how-to-roast-green-coffee-in-your-oven/) suggests that the duration is best kept between 12 and 15 minutes while the temp should be between 175 and 260 degrees Celsius. Of course, as the temperature rises, the duration should shrink. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt_roast(X,Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalize Data\n",
    "To match the previous lab, we'll normalize the data. Refer to that lab for more details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Temperature Max, Min pre normalization: {np.max(X[:,0]):0.2f}, {np.min(X[:,0]):0.2f}\")\n",
    "print(f\"Duration    Max, Min pre normalization: {np.max(X[:,1]):0.2f}, {np.min(X[:,1]):0.2f}\")\n",
    "norm_l = tf.keras.layers.Normalization(axis=-1)\n",
    "norm_l.adapt(X)  # learns mean, variance\n",
    "Xn = norm_l(X)\n",
    "print(f\"Temperature Max, Min post normalization: {np.max(Xn[:,0]):0.2f}, {np.min(Xn[:,0]):0.2f}\")\n",
    "print(f\"Duration    Max, Min post normalization: {np.max(Xn[:,1]):0.2f}, {np.min(Xn[:,1]):0.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Numpy Model (Forward Prop in NumPy)\n",
    "<center> <img  src=\"./images/C2_W1_RoastingNetwork.PNG\" width=\"200\" />   <center/>  \n",
    "Let's build the \"Coffee Roasting Network\" described in lecture. There are two layers with sigmoid activations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "As described in lecture, it is possible to build your own dense layer using NumPy. This can then be utilized to build a multi-layer neural network. \n",
    "\n",
    "<img src=\"images/C2_W1_dense3.png\" width=\"600\" height=\"450\">\n",
    "\n",
    "In the first optional lab, you constructed a neuron in NumPy and in Tensorflow and noted their similarity. A layer simply contains multiple neurons/units. As described in lecture, one can utilize a for loop to visit each unit (`j`) in the layer and perform the dot product of the weights for that unit (`W[:,j]`) and sum the bias for the unit (`b[j]`) to form `z`. An activation function `g(z)` can then be applied to that result. Let's try that below to build a \"dense layer\" subroutine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, you will define the activation function `g()`. You will use the `sigmoid()` function which is already implemented for you in the `lab_utils_common.py` file outside this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the activation function\n",
    "g = sigmoid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, you will define the `my_dense()` function which computes the activations of a dense layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def my_dense(a_in, W, b):\n",
    "    \"\"\"\n",
    "    Computes dense layer\n",
    "    Args:\n",
    "      a_in (ndarray (n, )) : Data, 1 example \n",
    "      W    (ndarray (n,j)) : Weight matrix, n features per unit, j units\n",
    "      b    (ndarray (j, )) : bias vector, j units  \n",
    "    Returns\n",
    "      a_out (ndarray (j,))  : j units|\n",
    "    \"\"\"\n",
    "    units = W.shape[1]\n",
    "    a_out = np.zeros(units)\n",
    "    for j in range(units):               \n",
    "        w = W[:,j]                                    \n",
    "        z = np.dot(w, a_in) + b[j]         \n",
    "        a_out[j] = g(z)               \n",
    "    return(a_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note: You can also implement the function above to accept `g` as an additional parameter (e.g. `my_dense(a_in, W, b, g)`). In this notebook though, you will only use one type of activation function (i.e. sigmoid) so it's okay to make it constant and define it outside the function. That's what you did in the code above and it makes the function calls in the next code cells simpler. Just keep in mind that passing it as a parameter is also an acceptable implementation. You will see that in this week's assignment.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell builds a two-layer neural network utilizing the `my_dense` subroutine above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_sequential(x, W1, b1, W2, b2):\n",
    "    a1 = my_dense(x,  W1, b1)\n",
    "    a2 = my_dense(a1, W2, b2)\n",
    "    return(a2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can copy trained weights and biases from the previous lab in Tensorflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "W1_tmp = np.array( [[-8.93,  0.29, 12.9 ], [-0.1,  -7.32, 10.81]] )\n",
    "b1_tmp = np.array( [-9.82, -9.28,  0.96] )\n",
    "W2_tmp = np.array( [[-31.18], [-27.59], [-32.56]] )\n",
    "b2_tmp = np.array( [15.41] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predictions\n",
    "<img align=\"left\" src=\"./images/C2_W1_RoastingDecision.PNG\"     style=\" width:380px; padding: 10px 20px; \" >\n",
    "\n",
    "Once you have a trained model, you can then use it to make predictions. Recall that the output of our model is a probability. In this case, the probability of a good roast. To make a decision, one must apply the probability to a threshold. In this case, we will use 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by writing a routine similar to Tensorflow's `model.predict()`. This will take a matrix $X$ with all $m$ examples in the rows and make a prediction by running the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_predict(X, W1, b1, W2, b2):\n",
    "    m = X.shape[0]\n",
    "    p = np.zeros((m,1))\n",
    "    for i in range(m):\n",
    "        p[i,0] = my_sequential(X[i], W1, b1, W2, b2)\n",
    "    return(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can try this routine on two examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_tst = np.array([\n",
    "    [200,13.9],  # postive example\n",
    "    [200,17]])   # negative example\n",
    "X_tstn = norm_l(X_tst)  # remember to normalize\n",
    "predictions = my_predict(X_tstn, W1_tmp, b1_tmp, W2_tmp, b2_tmp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To convert the probabilities to a decision, we apply a threshold:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "yhat = np.zeros_like(predictions)\n",
    "for i in range(len(predictions)):\n",
    "    if predictions[i] >= 0.5:\n",
    "        yhat[i] = 1\n",
    "    else:\n",
    "        yhat[i] = 0\n",
    "print(f\"decisions = \\n{yhat}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can be accomplished more succinctly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "yhat = (predictions >= 0.5).astype(int)\n",
    "print(f\"decisions = \\n{yhat}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Network function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This graph shows the operation of the whole network and is identical to the Tensorflow result from the previous lab.\n",
    "The left graph is the raw output of the final layer represented by the blue shading. This is overlaid on the training data represented by the X's and O's.   \n",
    "The right graph is the output of the network after a decision threshold. The X's and O's here correspond to decisions made by the network.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "netf= lambda x : my_predict(norm_l(x),W1_tmp, b1_tmp, W2_tmp, b2_tmp)\n",
    "plt_network(X,Y,netf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congratulations!\n",
    "You have built a small neural network in NumPy. \n",
    "Hopefully this lab revealed the fairly simple and familiar functions which make up a layer in a neural network. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}