week1_session5_deep_learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Akn_mosVUh85"
   },
   "source": [
    "# AI in Medicine: Data Science - Basics IV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "l5_jtm5BUh88"
   },
   "source": [
    "## Python Programming: Deep Learning using *Keras*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MulWXbThUh88"
   },
   "source": [
    "* **Instructor**: Moritz Seiler, AG Ritter, Charité - Universitätsmedizin Berlin (moritz.seiler@charite.de) <br>\n",
    "* **Target audience**:  Medical students from Charité<br>\n",
    "* **Course date**:  January 2022"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rhSxBhNcUh8-"
   },
   "source": [
    "**First, please make sure, that you're using the Python 3.6 kernel!**<br> You can check that in the top right corner of your browser window. If you're using a different Version, go to the Tab *Kernel* --> *Change Kernel* and select *Python 3.6*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "LeBKwxQtjPYA",
    "outputId": "094f63b1-5789-4d0e-ed98-e714f709180d"
   },
   "outputs": [],
   "source": [
    "!python --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ub-T-KMxjN84"
   },
   "source": [
    "**TensorFlow Update:**<br>\n",
    "The pre-installed version of the tensorflow in this environment is 1.1.0, but the following notebook requires version 2.0.0. Please update the tensorflow library by executing the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 36
    },
    "id": "dIaqZZDtivmz",
    "outputId": "404cd42b-2707-46cf-9683-2f4ef9810a92"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "tf.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u_pJOsotUh9J"
   },
   "source": [
    "## 1. Aims of this session"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "O1nsdV1IUh9L"
   },
   "source": [
    "This tutorial introduces **deep learning** in Python with ***Keras***. It is meant to be an **introduction to the *Keras* library** and **<ins>not</ins> a deep dive into the theory** surrounding deep learning. The goal of this session is that you will be able to implement your own neural network in Python.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Z5gY7MmcUh9M"
   },
   "source": [
    "## 2. Learning goals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "muMORQO_Uh9N"
   },
   "source": [
    "### Theory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DlJh1x8WUh9N"
   },
   "source": [
    "Deep Learning\n",
    "- Definition\n",
    "- Fully-connected Neural Networks\n",
    "- Activation functions\n",
    "- Forward pass\n",
    "- Backward pass\n",
    "- Convolutional Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y4A0vJJXUh9P"
   },
   "source": [
    "### Practical"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_R6reocbUh9Q"
   },
   "source": [
    "1. Deep Learning Playground\n",
    "2. Data set\n",
    "3. Keras\n",
    "    - Introduction\n",
    "    - Workflow\n",
    "        - Prepare data\n",
    "        - Build model\n",
    "        - Compile model\n",
    "        - Fit model\n",
    "        - Predict\n",
    "        - Evaluate model\n",
    "        - Save model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cosraY4LUh9Q"
   },
   "source": [
    "## 3. References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NeV_-HNfUh9Y"
   },
   "source": [
    "- Deep Learning<br>\n",
    "https://www.deeplearningbook.org/<br>\n",
    "https://cs231n.github.io/\n",
    "\n",
    "- Keras<br>\n",
    "https://keras.io/guides/<br>\n",
    "https://www.tensorflow.org/tutorials/keras/classification\n",
    "\n",
    "- Data<br>\n",
    "https://www.kaggle.com/uciml/pima-indians-diabetes-database"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FyXVV-5hUh9Y"
   },
   "source": [
    "## 4. Theory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DzloYLXvUh9Z"
   },
   "source": [
    "See accompanying slide deck."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-IKKFtrEUh9Z"
   },
   "source": [
    "## 5. Practical"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LxDD36XgUh9Z"
   },
   "source": [
    "### 5.1 Playground excersises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AtlCJ6UeUh9Z"
   },
   "source": [
    "The *Tensorflow* Playground is an interactive visualisation of neural networks in your browser. In this playground, you can choose different hyper-parameters of your neural network and visually observe the training process. This allows you to get some intuition on neural network workings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qIHWjpWYUh9b"
   },
   "source": [
    "https://playground.tensorflow.org"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-WP0QGVMUh9b"
   },
   "source": [
    "### Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FnNgJVqSUh9b"
   },
   "source": [
    "**1.** Play around and get some intution!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LpjHqPCWUh9c"
   },
   "source": [
    "**2.** Let's try a different dataset. Investigate the effects of the learning rate on the training results. <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4YNYpk80Uh9f"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4YIwIAzfUh9h"
   },
   "source": [
    "**3.** We now select the spiral dataset and add noise to the training data (noise=50). What do you observe using the following network architecture?<br>\n",
    "https://playground.tensorflow.org/#activation=sigmoid&batchSize=10&dataset=spiral&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=50&networkShape=2&seed=0.00906&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=true&ySquared=true&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "YR--lsyqUh9i"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Bwo7O4_gUh9k"
   },
   "source": [
    "### 5.2 Data set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YIPH1UIRUh9l"
   },
   "source": [
    "The data we are using in this tutorial is the Diabetes data set that you have already prepared in the previous tutorial.\n",
    "\n",
    "<ins>Source:</ins><br>\n",
    "*Smith, J.W., Everhart, J.E., Dickson, W.C., Knowler, W.C., & Johannes, R.S. (1988). Using the ADAP learning algorithm to forecast the onset of diabetes mellitus. In Proceedings of the Symposium on Computer Applications and Medical Care (pp. 261--265). IEEE Computer Society Press.*\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "SbCC38ffUh9m"
   },
   "outputs": [],
   "source": [
    "# Import the Pandas and NumPy library\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "# Read the Diabetes data set \n",
    "diabetes = pd.read_csv('https://raw.githubusercontent.com/volkamerlab/ai_in_medicine/master/data/diabetes.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "pTGfru68Uh9m",
    "outputId": "2fe33c3e-4c77-4926-97c8-0ae212b826cf"
   },
   "outputs": [],
   "source": [
    "diabetes.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4WgXZmQlVlQS"
   },
   "source": [
    "Remember yesterday's tutorial, we observed that some patients had 0 entries for the variables `Glucose`,`BMI`, and `BloodPressure`. We are going to remove these entries from the dataset since those unrealistic entries might have an influence on our predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "diabetes_clean = diabetes[(diabetes['Glucose']!=0) & \n",
    "                          (diabetes['BMI']!=0) & \n",
    "                          (diabetes['BloodPressure']!=0)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the dimensions of our final dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "diabetes_clean.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7NZLHzTmUh9o"
   },
   "source": [
    "### 5.3 Keras\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "acsny_xVUh9p"
   },
   "source": [
    "Keras is a high-level neural networks API, capable of running on top of Tensorflow, Theano, and CNTK. It enables fast experimentation through a high level, user-friendly, modular and extensible API\n",
    "\n",
    "<img src=\"https://keras.io/img/logo.png\" width=\"40%\">\n",
    "\n",
    "<ins>Source:</ins> https://keras.io/img/logo.png"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j_YMY3nWUh9p"
   },
   "source": [
    "### Installation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "__XnytcRUh9p"
   },
   "source": [
    "For an installation of the *Keras* library, you have to enter the following command in the terminal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j3KOYchaUh9q"
   },
   "source": [
    "```python\n",
    "!pip install tensorflow\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_ql4nQgRUh9q"
   },
   "source": [
    "Keras comes packaged with *TensorFlow 2.0* as `tensorflow.keras`. To start using *Keras*, simply install *TensorFlow 2.0*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mEIhPfo5Uh9r"
   },
   "source": [
    "*Keras* is a model-level library, providing high-level building blocks for developing deep learning models. It doesn't handle itself low-level operations such as tensor products, convolutions and so on. Instead, it relies on a specialized, well-optimized tensor manipulation library to do so, serving as the \"backend engine\" of Keras. \n",
    "\n",
    "Rather than picking one single tensor library and making the implementation of *Keras* tied to that library, *Keras* handles the problem in a modular way, and several different backend engines can be plugged seamlessly into *Keras*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rtgIJJ1GUh9r"
   },
   "source": [
    "### Importing the library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "a9U5V2EEUh9t"
   },
   "source": [
    "We aren't going to import the whole *Keras* library but only specific classes that we need for our implementation. This will later be done using the following command:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Sev4kbIjUh9u"
   },
   "source": [
    "```python\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.xyz import ...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ThgpIYjAUh9u"
   },
   "source": [
    "### Workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EeLZsFYCUh9v"
   },
   "source": [
    "Training your first simple neural network with *Keras* doesn't require a lot of code, but we're going to start slow, taking it step-by-step, ensuring you understand the process of how to train a network. The workflow in *Keras* consits of the following steps:\n",
    "1. **Prepare data**\n",
    "2. **Build model**\n",
    "3. **Compile model**\n",
    "4. **Fit model**\n",
    "5. **Predict**\n",
    "6. **Evaluate model**\n",
    "7. **Save model**\n",
    "\n",
    "We're now going to introduce all these steps in more detail in the following part of the notebook. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X3Iej-MNUh9v"
   },
   "source": [
    "#### 1. Data preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oWnAxNECUh9v"
   },
   "source": [
    "Before we start building our neural network, we have to prepare our data. A first step should always be data inspection, which you have already done in the last tutorial. Here, we will split our data in different subsets to train our model and will standardise the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JPrH05GZUh9x"
   },
   "source": [
    "**Data splitting**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9HV0QMqcUh9x"
   },
   "source": [
    "As you have learnt in the last tutorial, we have to split our dataset in subsets since we do not know the (hidden) data generating distribution to calculate the true error of our model. For this, we split the diabetes dataset into a training set, validation set and test set using the following split 60-20-20 to approximate the true error. We use the `train_test_split()`function from them `sklearn.model_selection` module to perform the splitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SXzeWl3RUh9x"
   },
   "source": [
    "Here, we split the dataset only in a training and test set. Later in the training step, we use *Keras*' functionality to create a validation set from the training set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "URVclPL_Uh9x"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(diabetes_clean[['Glucose','BMI','Age']], diabetes_clean['Outcome'], \n",
    "                                                    test_size=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QkPdmBBHUh9x"
   },
   "source": [
    "Let's check how many observations are in each set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4AMXv4ODUh9y",
    "outputId": "48d5d796-7b24-4244-99f3-91289269e965"
   },
   "outputs": [],
   "source": [
    "print(\"Number of training samples: %i\" % len(X_train))\n",
    "print(\"Number of test samples: %i\" % len(X_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y1DBHAb7Uh9z"
   },
   "source": [
    "**Tranformations**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wYXY_WmJUh90"
   },
   "source": [
    "Convergence of neural networks is ususally faster if the **average of each input variable**  over the training set is **close to zero**. In general, any shift of the average input away from zero will bias the updates in a particular direction and thus slow down learning. Additionally, **scaling the inputs** - to have **about the same covariance** - speeds up learning because it helps to balance out the rate at which the weights connected to the input nodes learn. Therefore, we will *standardize* our data \n",
    "\n",
    "$$z=\\frac{x-\\mu}{\\sigma} $$\n",
    "\n",
    "with mean $\\mu$ and standard deviation $\\sigma$. This transformation will speed up and stabilise the training process. We can use the `StandardScaler()` function from the `sklearn.preprocessing` module to perform the standardisation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "I5Hj349qUh92"
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# Instatiate scaler\n",
    "scaler = StandardScaler()\n",
    "\n",
    "# Train scaler on the data set\n",
    "scaler.fit(X_train)\n",
    "\n",
    "# Transform datasets\n",
    "X_train = scaler.transform(X_train)\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IFA4d1z7Uh92"
   },
   "source": [
    "Let us have a look at our transformed training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "GBRjFuFOUh92",
    "outputId": "001a9ec3-bbb0-4495-a4a5-a0fbf412d61a"
   },
   "outputs": [],
   "source": [
    "X_train[:10,:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZSyq9fCZUh93"
   },
   "source": [
    "Let us also check the mean and standard deviation of our standardised training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "_r_zsDIHUh94",
    "outputId": "cf580001-ddba-45e6-9a0d-e5611f91913d"
   },
   "outputs": [],
   "source": [
    "print(np.mean(X_train,0))\n",
    "print(np.std(X_train,0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ztbd-o7HUh94"
   },
   "source": [
    "#### 2. Model Building"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2DCl3nGqUh94"
   },
   "source": [
    "The core data structures of *Keras* are **layers** and **models**. There are three ways to create Keras models:\n",
    "\n",
    "- The Sequential model is simply a linear stack of layers. *(for simple implementations)*\n",
    "- The Functional API fully-featured API that supports arbitrary model architectures. *(for more complex implementations)*\n",
    "- Model subclassing, where you implement everything from scratch on your own. *(for more complex implementations)*\n",
    "\n",
    "We are going to use the **Sequential model** class in this tutorial. You simply need to import the model from the *Keras* library and initialise a model object using the following commands:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "a7gc80ZfUh95"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras.models import Sequential\n",
    "\n",
    "model = Sequential()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IpmpOjfxUh95"
   },
   "source": [
    "Then, we can stack layers and add those to our model object. Before we do so, the implementation of the layer types  \n",
    "- dense layer and \n",
    "- convolutional layer \n",
    "\n",
    "is introduced, which you are going to use later in this notebook. A complete list of supported layer types is given in the *Keras* documentation https://keras.io/api/layers/. First, we have to import these layers from the *Keras* layer module using the following command:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w72LP4ipUh95"
   },
   "source": [
    "```python\n",
    "from tensorflow.keras.layers import Dense, Conv2D\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "R52VX13DUh95"
   },
   "source": [
    "**Dense layer**<br>\n",
    "A dense layer/fully-connected layer is a layer in which each input neuron is connected to each output neuron, the parameters units just tells you the dimensionality of your output. \n",
    "\n",
    "```python\n",
    "Dense(units='dimensionality of the output space',\n",
    "      activation='activation function to use',\n",
    "      ...)\n",
    "```\n",
    "<ins>Example:</ins>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CMincqCAUh95"
   },
   "source": [
    "```python\n",
    "Dense(units=6, activation='relu',...)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "S3JgPKH1Uh95"
   },
   "source": [
    "**Convolutional layer**<br>\n",
    "The convolutional layer’s parameters consist of a set of learnable filters. During the forward pass, we convolve each filter across the width and height of the input and compute dot products between the entries of the filter and the input at any position. By doing so, we produce an activation map that gives the responses of that filter at every spatial position.\n",
    "\n",
    "```python\n",
    "Conv2D(filters='the dimensionality of the output space (i.e. the number of output filters in convolution)', \n",
    "       kernel_size='specifying the height and width of the 2D convolutional kernel', \n",
    "       strides=' specifying the strides of the convolution along the height and width', \n",
    "       activation='activation function to use',\n",
    "       ...)\n",
    "```\n",
    "<ins>Example:</ins>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8E14gK7wUh96"
   },
   "source": [
    "```python\n",
    "Conv2D(filters=16, kernel_size=(3,3), strides=(1,1), activation='relu',...)\n",
    "# if width and height are equal you can write in short\n",
    "Conv2D(filters=16, kernel_size=3, strides=1, activation='relu',...)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NiIPEAy_Uh96"
   },
   "source": [
    "We can simply add these layers to the Sequential model using the `add()` method.\n",
    "Here, we add  \n",
    "- an dense input layer with 3 units, \n",
    "- a dense sigmoid output layer with 1 unit\n",
    "\n",
    "to create a **fully-connected neural network**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cwi8JcVZUh96"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras import Input\n",
    "from tensorflow.keras.layers import Dense\n",
    "\n",
    "model.add(Input(shape=(3,)))\n",
    "model.add(Dense(1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MewTFbooUh97"
   },
   "source": [
    "We can display the content of our neural network by calling the `summary()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "eDek3IM-Uh97",
    "outputId": "d5d92226-850e-4e6b-e7d7-6db3efdb882d"
   },
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Z5RVIWapUh98"
   },
   "source": [
    "*Why does our network have 4 parameters?*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EWzeB6VJUh98"
   },
   "source": [
    "#### 3. Model compilation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bBBy-ENyUh99"
   },
   "source": [
    "In order to train our neural network, we need to compile our model. This requires us to define a loss function and an optimiser to do so. A list already implemented losses and optimisers can be found in the documentary:\n",
    "\n",
    "- losses: https://keras.io/api/losses/,\n",
    "- optimisers: https://keras.io/api/optimizers/. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jupgR4smUh99"
   },
   "source": [
    "**Loss function**\n",
    "\n",
    "The purpose of loss functions is to compute the quantity that a model should seek to minimize during training. \n",
    "This function will essentially calculate how poorly our model is performing by comparing what the model is predicting with the actual value it is supposed to output. Therefore, the selection of the loss function depends on the the task.\n",
    "\n",
    "<ins>Examples:</ins><br>\n",
    "\n",
    "*Classification:*\n",
    "- `BinaryCrossentropy()`<br> Computes the cross-entropy loss between true labels and predicted labels *(binary)*.\n",
    "- `CategoricalCrossentropy()`<br> Computes the cross-entropy loss between true labels and predicted labels *(multi-class)*.\n",
    "- ...\n",
    "\n",
    "*Regression:*\n",
    "- `MeanSquaredError()`<br> Computes the mean of squares of errors between labels and predictions.\n",
    "- ...\n",
    "\n",
    "You can import the loss functions from the *Keras* losses module by entering the following command:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UDP-4PpbUh9-"
   },
   "source": [
    "```python\n",
    "from tensorflow.keras.losses import BinaryCrossentropy, CategoricalCrossentropy, MeanSquaredError\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-EIWQ_pZUh9-"
   },
   "source": [
    "**Optimiser**\n",
    "\n",
    "We have already dealt with the loss function, which is a mathematical way of measuring how wrong your predictions are. In other words, the loss function lets us quantify the quality of any particular set of model parameters. The goal of optimisation is to find those parameteres that minimize the loss function. For this, we can compute the best direction along which we should change our parameters that is mathematically guaranteed to be the direction of the steepest descend. This direction will be related to the gradient of the loss function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BAb-sovUUh9-"
   },
   "source": [
    "<ins>Examples:</ins>\n",
    "- `SGD()`<br> Stochastic gradient descent is an implementation of gradient descent - that you know from the lecture - on batches of observations or single random examples on each pass.\n",
    "\n",
    "- `Adam()`<br> Adam stands for adaptive moment estimation, and is another way of using past gradients to calculate current gradients. Adam also utilizes the concept of momentum by adding fractions of previous gradients to the current one.\n",
    "\n",
    "You can import the optimiser from the `tf.keras.optimizers` module by entering the following command:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TqRtHMErUh9-"
   },
   "source": [
    "```python\n",
    "from tensorflow.keras.optimizers import SGD, Adam\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "v2e4xBarUh9-"
   },
   "source": [
    "We instatiate the loss function and the optimiser before we pass them to the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fxlquR4jUh9-"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.losses import BinaryCrossentropy\n",
    "from tensorflow.keras.optimizers import SGD\n",
    "\n",
    "loss = BinaryCrossentropy()\n",
    "\n",
    "lr =5e-2 #lr = 0.05\n",
    "optimiser = SGD(learning_rate=lr)\n",
    "\n",
    "model.compile(loss=loss,\n",
    "              optimizer=optimiser, \n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3IIzAO07Uh-A"
   },
   "source": [
    "After compiling our model, we're now ready to start the training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qpXJdIUxUh-A"
   },
   "source": [
    "#### 4. Model training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Xydm8CZcUh-B"
   },
   "source": [
    "To train our model we can simply use the `fit()` method. Here, we just have to provide the *number of epochs*, the *batch size* and the *training dataset* (features and targerts). Instead of providing a separate validation dataset, we use the `validation_split` functionality, which uses the last x% observations of the training dataset before shuffling to create a validation set.\n",
    "\n",
    "At the end of each epoch, the model will iterate over the validation dataset and compute the validation loss and validation metric.\n",
    "\n",
    "The `fit()` method outputs a `History` object; its `History.history` attribute is a record of loss and metrics values (training + validation)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-rfGPuS6Uh-B"
   },
   "source": [
    "In our example, we use 50 *epochs* and a *batch size* of 64 to train our model. Additionally, we use a validation split of 0.25 such that our overall data split in training, validation and test set is 60-20-20."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rfmSlXFeUh-B",
    "outputId": "5fa547c8-005a-4db6-a368-240c5067d2c2"
   },
   "outputs": [],
   "source": [
    "epochs = 50\n",
    "batch_size = 64\n",
    "\n",
    "\n",
    "history = model.fit(x=X_train, y=y_train,\n",
    "                validation_split=0.25,\n",
    "                epochs=epochs, \n",
    "                batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HM6xNwsDUh-C"
   },
   "source": [
    "The `fit()` methods prints the loss and metric values after every epochs, but we can also use the `History` object to visualise those values to get intuition about the performance of our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 918
    },
    "id": "bjzzBU5nUh-C",
    "outputId": "6d80c6d7-670d-4f92-8dc8-be4dd8199b3c"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "n_epochs = np.arange(0,epochs)\n",
    "\n",
    "fig, (ax1,ax2) = plt.subplots(2,1,figsize=(8,16))\n",
    "ax1.plot(n_epochs, history.history['loss'], label='training loss')\n",
    "ax1.plot(n_epochs, history.history['val_loss'], label='validation loss')\n",
    "ax1.set_ylim(-0.05,1.05)\n",
    "ax1.legend()\n",
    "\n",
    "ax2.plot(n_epochs, history.history['accuracy'], label='training accuracy')\n",
    "ax2.plot(n_epochs, history.history['val_accuracy'], label='validation accuracy')\n",
    "ax2.set_ylim(-0.05,1.05)\n",
    "ax2.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LoVD-RmuUh-C"
   },
   "source": [
    "#### 5. Prediciton"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DNpO23_cUh-D"
   },
   "source": [
    "After training, we can use our model to predict the target variable using the `predict()` method, which returns the probabilities $P(Outcome=1|X,w)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "igTgfUfzUh-D"
   },
   "outputs": [],
   "source": [
    "y_pred = model.predict(X_test).ravel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9o_Rg_1SUh-E",
    "outputId": "085a028e-a425-4b72-d2c6-f085498d922b"
   },
   "outputs": [],
   "source": [
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-gAVC9_eUh-E"
   },
   "source": [
    "If you're not interested in the probability $P(Outcome=1|X,w)$ but simply want to predict the class label, you can use the `predict_classes()` method. This method uses a threshold of 0.5 to classify examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bJKBfZaoUh-E",
    "outputId": "2971e304-f65c-4365-ad70-9c6f32dcfc98"
   },
   "outputs": [],
   "source": [
    "(model.predict(X_test) > 0.5).astype('int32').ravel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "o3OF4CXlUh-G"
   },
   "source": [
    "#### 6. Model evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K6flm6tPUh-G"
   },
   "source": [
    "To see how good/bad those predictions are, we've to evaluate them using the built-in `evaluate()` method. This returns the loss and metrics values for the model in test mode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "36Nbr-opUh-G",
    "outputId": "1d6e0309-d934-4cd3-e6ec-3316795dcd85"
   },
   "outputs": [],
   "source": [
    "loss_and_metrics = model.evaluate(X_test, y_test, \n",
    "                                  batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "466tUYg2Uh-H"
   },
   "source": [
    "In addition to loss and metric values, we're going to use the receiver operating characteristics (ROC) and the area under the curve (AUC) to evaluate our model. We can use the `roc_curve()` and `auc` function from the `sklearn.metrics` module to calculate those values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6d_7aYgXUh-I"
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_curve, auc\n",
    "fpr, tpr, thresholds_keras = roc_curve(y_test, y_pred)\n",
    "auc_keras = auc(fpr, tpr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5rbMl597Uh-I"
   },
   "source": [
    "After calculating all values, we can use *matplotllib* to plot the ROC curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 513
    },
    "id": "fRgkYOVTUh-I",
    "outputId": "db9c7d98-9430-46a7-c369-d5488b5d763d"
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1,1,figsize=(12,8))\n",
    "ax.plot([0, 1], [0, 1], 'k--')\n",
    "ax.plot(fpr, tpr, label='Keras (area = {:.3f})'.format(auc_keras))\n",
    "ax.set_xlabel('False positive rate')\n",
    "ax.set_ylabel('True positive rate')\n",
    "ax.set_title('ROC curve')\n",
    "ax.legend(loc='best')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i7VjoafyUh-K"
   },
   "source": [
    "#### 7. Model saving"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5YeHs3dsUh-L"
   },
   "source": [
    "A Keras model consists of multiple components:\n",
    "\n",
    "- An architecture, or configuration, which specifyies what layers the model contains, and how they're connected.\n",
    "- A set of weights values (the \"state of the model\").\n",
    "- An optimizer (defined by compiling the model).\n",
    "- A set of losses and metrics (defined by compiling the model).\n",
    "\n",
    "We can save these pieces to disk at once into a single archive in the *Tensorflow* `SavedModel`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qcoS90GJUh-L"
   },
   "source": [
    "```python\n",
    "model.save('path/to/location')\n",
    "```\n",
    "\n",
    "We can save our neural network to the working directory by entering the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pAlEyHd4Uh-M",
    "outputId": "ff29ca90-e45f-42b7-ab6e-801f8d280d24"
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "Path(\"results/week1_session5_deep-learning\").mkdir(parents=True, exist_ok=True)\n",
    "model.save('results/week1_session5_deep-learning/my_nn')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xidD1YzCUh-M"
   },
   "source": [
    "This can then simply be loaded using the following command:\n",
    "\n",
    "```python\n",
    "model = tf.keras.models.load_model('my_nn')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WJEjehtYUh-M"
   },
   "source": [
    "**Put everything together**<br>\n",
    "After having itroduced all the steps separately in more detail, everything is put together in one cell, so you don't have to scroll too much."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "cfZhhlzrUh-M",
    "outputId": "7ced0f04-dcfe-48e9-bae4-b761ae4329d6",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras import Input\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras.losses import BinaryCrossentropy\n",
    "from tensorflow.keras.optimizers import SGD\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.metrics import roc_curve, auc\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "## Data preparation\n",
    "# Data Splitting\n",
    "X_train, X_test, y_train, y_test = train_test_split(diabetes_clean[['Glucose','BMI','Age']], \n",
    "                                                    diabetes_clean['Outcome'], \n",
    "                                                    test_size=0.2)\n",
    "\n",
    "# Data transformation       \n",
    "scaler = StandardScaler()\n",
    "scaler.fit(X_train)\n",
    "X_train = scaler.transform(X_train)\n",
    "X_test = scaler.transform(X_test)\n",
    "\n",
    "\n",
    "\n",
    "## Model building\n",
    "model = Sequential()\n",
    "model.add(Input(shape=(3,)))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "## Model compilation\n",
    "# Loss function\n",
    "loss = BinaryCrossentropy()\n",
    "\n",
    "# Optimiser\n",
    "lr = 4e-2 \n",
    "optimiser = SGD(learning_rate=lr)\n",
    "\n",
    "model.compile(loss=loss,\n",
    "              optimizer=optimiser, \n",
    "              metrics=['accuracy'])\n",
    "\n",
    "## Model training\n",
    "epochs = 50\n",
    "batch_size = 16\n",
    "\n",
    "history = model.fit(x=X_train, y=y_train, \n",
    "                    validation_split=0.25,\n",
    "                    epochs=epochs,\n",
    "                    batch_size=batch_size)\n",
    "\n",
    "\n",
    "n_epochs = np.arange(0,epochs)\n",
    "\n",
    "fig, (ax1,ax2) = plt.subplots(2,1,figsize=(8,16))\n",
    "ax1.plot(n_epochs, history.history['loss'], label='training loss')\n",
    "ax1.plot(n_epochs, history.history['val_loss'], label='validation loss')\n",
    "ax1.set_ylim(-0.05,1.05)\n",
    "ax1.legend()\n",
    "\n",
    "ax2.plot(n_epochs, history.history['accuracy'], label='training accuracy')\n",
    "ax2.plot(n_epochs, history.history['val_accuracy'], label='validation accuracy')\n",
    "ax2.set_ylim(-0.05,1.05)\n",
    "ax2.legend()\n",
    "plt.show()\n",
    "\n",
    "## Prediciton\n",
    "y_pred = model.predict(X_test).ravel()\n",
    "\n",
    "## Model evaluation\n",
    "loss_and_metrics = model.evaluate(X_test, y_test, \n",
    "                                  batch_size=batch_size)\n",
    "\n",
    "fpr, tpr, thresholds_keras = roc_curve(y_test, y_pred)\n",
    "auc_keras = auc(fpr, tpr)\n",
    "\n",
    "fig, ax = plt.subplots(1,1,figsize=(12,8))\n",
    "ax.plot([0, 1], [0, 1], 'k--')\n",
    "ax.plot(fpr, tpr, label='Keras (area = {:.3f})'.format(auc_keras))\n",
    "ax.set_xlabel('False positive rate')\n",
    "ax.set_ylabel('True positive rate')\n",
    "ax.set_title('ROC curve')\n",
    "ax.legend(loc='best')\n",
    "\n",
    "plt.show()\n",
    "\n",
    "## Save model\n",
    "model.save('results/week1_session5_deep-learning/my_nn')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XXnhc-0AUh-N"
   },
   "source": [
    "## 6. Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3aF_LeRsUh-N"
   },
   "source": [
    "In this notebook, the implementation of a neural network using the *Keras* library was introduced.\n",
    "The workflow to build and train a neural network was presented using a step-by-step introduction.\n",
    "- **Data Preparation:** We splitted the dataset and standardised the input features.\n",
    "- **Model Building:** We created a neural network using the Sequential model class and introduced dense and convolutional layers.\n",
    "- **Model compilaton:** We compiled a neural network and introduced different loss functions and optimisers.\n",
    "- **Model training:** We trained a neural network and visualised the training process.\n",
    "- **Model prediction and evaluation:** We used the trained network to predict the target variable and evaluated the network using different metrics.\n",
    "- **Save model:** We saved all model components in a single archive `SavedModel`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "H1S--I93Uh-N"
   },
   "source": [
    "## 7. Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "v3vgvCFrUh-N"
   },
   "source": [
    "Now that you know the basics to implement a neural network in *Keras*, it's time for you to get your hands dirty and get started.\n",
    "\n",
    "*Tip:* The following resources might be helpful:\n",
    "- www.stackoverflow.com\n",
    "- Keras documentation: https://keras.io/api/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "a-k7cUjsUh-O"
   },
   "source": [
    "### 7.1 Fully-connected neural network (Basic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Jny6Au3PUh-P"
   },
   "source": [
    "<ins>**Task:**</ins> Please **implement your own classifier to predict the outcome classes** in the diabetes dataset. <br>\n",
    "<ins>**Model:**</ins> Use a **fully-connected neural network** with at least 1 hidden layer.<br> (*Note:* Since this is still a binary classification task, please use 1 unit and the sigmoid activation in your output layer.)<br>\n",
    "<ins>**Data:**</ins> Use all available data features to predict the outcome class. Split the data into training, validation and test set using a split of 60-20-20.\n",
    "\n",
    "*Do your results look different from the ones in the example above?*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "czwiTTuZUh-P"
   },
   "source": [
    "#### Data preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "phQ7DUzTUh-P"
   },
   "outputs": [],
   "source": [
    "### Data preparation ###\n",
    "# TO DO: Split and transform dataset\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "g_ySb_OPUh-Q"
   },
   "source": [
    "#### Model building"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "DZziPuBQUh-Q"
   },
   "outputs": [],
   "source": [
    "### Model building ###\n",
    "model = Sequential()\n",
    "# TO DO: Create a neural network with at least 1 hidden layer\n",
    "\n",
    "\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YicHVCCFUh-Q"
   },
   "source": [
    "#### Model compilation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ujSj8JNGUh-Q"
   },
   "outputs": [],
   "source": [
    "# TO DO: Define loss function and optimiser\n",
    "\n",
    "\n",
    "# TO DO: compile model\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3DS2NSSRUh-Q"
   },
   "source": [
    "#### Model training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "j7u9YkQ0Uh-Q"
   },
   "outputs": [],
   "source": [
    "# TO DO: Choose the training length and batch size\n",
    "\n",
    "history = # TO DO: Train your model (Don't forget validation_split)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "92pv2JUzUh-R"
   },
   "outputs": [],
   "source": [
    "n_epochs = np.arange(0,epochs)\n",
    "\n",
    "fig, (ax1,ax2) = plt.subplots(2,1,figsize=(8,16))\n",
    "ax1.plot(n_epochs, history.history['loss'], label='training loss')\n",
    "ax1.plot(n_epochs, history.history['val_loss'], label='validation loss')\n",
    "ax1.set_ylim(-0.05,1.05)\n",
    "ax1.legend()\n",
    "\n",
    "ax2.plot(n_epochs, history.history['accuracy'], label='training accuracy')\n",
    "ax2.plot(n_epochs, history.history['val_accuracy'], label='validation accuracy')\n",
    "ax2.set_ylim(-0.05,1.05)\n",
    "ax2.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ub7ZLspqUh-R"
   },
   "source": [
    "#### Model evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4Rz2Wg58Uh-R"
   },
   "outputs": [],
   "source": [
    "# TO DO: Calculate test loss and accurcy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5oL2ZXgbUh-R"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Jhdgx4uvUh-R"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_bOg7qVCUh-S"
   },
   "source": [
    "### 7.2 Convolutional neural network (Advanced)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0a707cYtUh-S"
   },
   "source": [
    "The second task is to implement a multi-class image classifier for the famous MNIST dataset. This dataset contains 70,000 (28*28 pixel) images of digits 0-9.\n",
    "\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/2/27/MnistExamples.png\" width=\"40%\">\n",
    "\n",
    "*Source: https://upload.wikimedia.org/wikipedia/commons/2/27/MnistExamples.png*\n",
    "\n",
    "\n",
    "<ins>**Task:**</ins> Please **implement your own multi-class image classifier to predict the outcome classes** in the MNIST dataset. Can you get a **model accuracy of $> 0.98$?** <br>\n",
    "<ins>**Model:**</ins> Use a **convolutional neural network** with at least 1 hidden convolutional layer.<br> (*Note:* Since this is still a multi-class classification task, please use 10 units and the softmax activation in your output layer.)<br>\n",
    "<ins>**Data:**</ins> Use the 28*28 pixel images to predict the digit class.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0fr0snQswRkR"
   },
   "source": [
    "#### Hardware acceleration (Colab users only!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kZ7bt-EzwdIn"
   },
   "source": [
    "When using high-dimensional data (e.g., images), training your network on a CPU is very slow. Hardware accelarators like GPUs or TPUs decrease the training time by parallelising the your computations. Colab allows us to use a TPU to train our neural network. To use such kind of hardware acceleration, select the device: Runtime > change runtime type > Hardware accelarator > GPU/TPU. After changing the runtime type, all the data in memory is lost; so, you have to import all libraries again.\n",
    "\n",
    "For this tutorial, please use a TPU for your training and run the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FujHii7Bcmt1",
    "outputId": "8db9d990-0fec-44ad-e76c-e575f79b3fc9"
   },
   "outputs": [],
   "source": [
    "print(\"Tensorflow version \" + tf.__version__)\n",
    "\n",
    "try:\n",
    "  tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection\n",
    "  print('Running on TPU ', tpu.cluster_spec().as_dict()['worker'])\n",
    "except ValueError:\n",
    "  raise BaseException('ERROR: Not connected to a TPU runtime; please see the previous cell in this notebook for instructions!')\n",
    "\n",
    "tf.config.experimental_connect_to_cluster(tpu)\n",
    "tf.tpu.experimental.initialize_tpu_system(tpu)\n",
    "tpu_strategy = tf.distribute.experimental.TPUStrategy(tpu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "U6BbEM5LUh-S"
   },
   "source": [
    "#### Data preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "V6URfze6Uh-S"
   },
   "source": [
    "Before we can use the MNIST data to predict the class labels, we have to **split** the dataset, **normalise** (transform range from 0-255 to 0-1) and reshape the data and **one-hot encode** the target. The data preparation step is already done for you, just execute the following cell and check the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zDmzkfsoUh-S"
   },
   "outputs": [],
   "source": [
    "# Import the MNIST dataset\n",
    "from tensorflow.keras.datasets import mnist\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "\n",
    "# Load MNIST images and labels and normalise (range 0 to 1) image data\n",
    "(X_train, y_train), (X_test, y_test) = mnist.load_data() # 60,000 training + 10,000 test images/labels\n",
    "X_train = X_train / 255.0\n",
    "X_test  = X_test / 255.0\n",
    "\n",
    "# Reshape dataset to have a single channel \n",
    "X_train = X_train.reshape((X_train.shape[0], 28, 28, 1)) # The CNN requires this layout (batch_size, height, width, n_channels)\n",
    "X_test = X_test.reshape((X_test.shape[0], 28, 28, 1)) # The CNN requires this layout (batch_size, height, width, n_channels)\n",
    "\n",
    "# One-hot encode target values\n",
    "y_train = to_categorical(y_train)\n",
    "y_test = to_categorical(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uvMb_jKuUh-T"
   },
   "source": [
    "Let's have a look at the image data..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 195
    },
    "id": "5uYuYEvtUh-T",
    "outputId": "110a2128-3a46-47ae-eee2-4d6e4d3609c5",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "fig, (ax1,ax2,ax3,ax4) = plt.subplots(1,4, figsize=(12,4))\n",
    "ax1.imshow(X_train[0,:,:,0])\n",
    "ax2.imshow(X_train[1,:,:,0])\n",
    "ax3.imshow(X_train[2,:,:,0])\n",
    "ax4.imshow(X_train[3,:,:,0]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UlXwmgIiUh-T"
   },
   "source": [
    "and the one-hot encoded target variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "BTOe7H8eUh-T",
    "outputId": "bde5a4e9-f1a5-4cbf-b266-291adc876f42"
   },
   "outputs": [],
   "source": [
    "y_train[:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "M3uYMIOtUh-U"
   },
   "source": [
    "Instead of the categorical class values, we now have a vector of length equal to the number of categories. The i-th component is equal to 1 if target is assigned to the i-th category and all other components are 0. By using this transformation, we transform the categorical variable into a numerical form."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZzPEZojLUh-U"
   },
   "source": [
    "#### Model building"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dYBab8vqUh-U"
   },
   "source": [
    "Now, it's time to implement your convolutional neural network. The last layer of the network is a fully-connected output layer with 10 units and softmax activation function, since we have a classification task with 10 classes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Zr1HsRSw3RY_"
   },
   "source": [
    "**Colab users only!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "luDQB20wUh-U",
    "outputId": "e87b2d9a-f80b-43a8-efdd-355ec833f671"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Flatten, # TO DO: Import convolutional layer\n",
    "\n",
    "with tpu_strategy.scope():\n",
    "  model = Sequential()\n",
    "  model.add(Input(shape=(28,28,1)))\n",
    "\n",
    "  # TO DO: Add at least 1 convolutional layer\n",
    "\n",
    "  model.add(Flatten()) # This flattens your image with width and height into a vector of length widht*height. \n",
    "  model.add(Dense(10, activation='softmax')) \n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZzpKEcyT3r9E"
   },
   "source": [
    "**Non-Colab users only!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "uRWYC2IN3cOC"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Flatten, # TO DO: Import convolutional layer\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Input(shape=(28,28,1)))\n",
    "\n",
    "# TO DO: Add at least 1 convolutional layer\n",
    "\n",
    "model.add(Flatten()) # This flattens your image with width and height into a vector of length widht*height. \n",
    "model.add(Dense(10, activation='softmax')) \n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c73CAtOHUh-V"
   },
   "source": [
    "#### Model compilation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5uMpREwKyLl6"
   },
   "source": [
    "Here, we have a multi-class problem; therefore, you can't use the binary cross-entropy loss. Do you know a loss function which you can use in this setting?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "F-loiCMoUh-V"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.losses import # TO DO: Import loss function\n",
    "from tensorflow.keras.optimizers import # TO DO: Import optimiser\n",
    "\n",
    "loss = # TO DO: Define loss function\n",
    "optimiser = # TO DO: Define optimiser\n",
    "\n",
    "# TO DO: Compile your model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "H07mRdelUh-V"
   },
   "source": [
    "#### Model training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Hwj4ti0oUh-V"
   },
   "source": [
    "Please train your model choosing an adequate number of epochs and batch size. To validate your model, use `validation_split = 0.1`.\n",
    "\n",
    "*Recommendation:* For non-Colab users, please use $\\leq 2$ epochs to train your network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lGF5gg9kUh-W"
   },
   "outputs": [],
   "source": [
    "# Define number of epochs and batch size\n",
    "epochs = # TO DO: Choose number of epochs\n",
    "batch_size = # TO DO: Choose batch size\n",
    "\n",
    "history = # TO DO: Train your model (Don't forget validation_split!)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7a_dlAQZUh-W"
   },
   "outputs": [],
   "source": [
    "n_epochs = np.arange(0,20)\n",
    "\n",
    "fig, (ax1,ax2) = plt.subplots(2,1,figsize=(8,16))\n",
    "ax1.plot(n_epochs, history.history['loss'], label='training loss')\n",
    "ax1.plot(n_epochs, history.history['val_loss'], label='validation loss')\n",
    "ax1.set_ylim(-0.05,1.05)\n",
    "ax1.legend()\n",
    "\n",
    "ax2.plot(n_epochs, history.history['accuracy'], label='training accuracy')\n",
    "ax2.plot(n_epochs, history.history['val_accuracy'], label='validation accuracy')\n",
    "ax2.set_ylim(-0.05,1.05)\n",
    "ax2.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-Fdb9yHdUh-X"
   },
   "source": [
    "#### Model evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1UHhheIBUh-X"
   },
   "outputs": [],
   "source": [
    "# TO DO: Calculate test loss and accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "nehuxiYjUh-X"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Iz1gBS3VUh-X"
   },
   "source": [
    "## 8. Solutions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "olNYzb3eUh-X"
   },
   "source": [
    "### 1) Fully-connected neural network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "W-PeqRuxUh-X"
   },
   "source": [
    "<ins>**Task:**</ins> Please **implement your own classifier to predict the outcome classes** in the diabetes dataset. <br>\n",
    "<ins>**Model:**</ins> Use a **fully-connected neural network** with at least 1 hidden layer.<br> (*Note:* Since this is still a binary classification task, please use 1 unit and the sigmoid activation in your output layer.)<br>\n",
    "<ins>**Data:**</ins> Use all available data features to predict the outcome class. Split the data into training, validation and test set using a split of 60-20-20.\n",
    "\n",
    "The following solution consider a fully-connected neural network with the following architecture:\n",
    "- <ins>input layer</ins> with 8 units,\n",
    "- <ins>1st hidden layer</ins> with **6 units** and **ReLu** activations,\n",
    "- <ins>2nd hidden layer</ins> with **6 units** and **tanh** activations,\n",
    "- <ins>3rd hidden layer</ins> with **4 units** and **tanh** activations,\n",
    "- <ins>output layer</ins> with **1 unit** and **sigmoid** activation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0XECT_D5Uh-Y"
   },
   "source": [
    "**Data preparation**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QXvxUBjaUh-Y"
   },
   "outputs": [],
   "source": [
    "# Data Splitting\n",
    "X_train, X_test, y_train, y_test = train_test_split(diabetes.iloc[:,:-1], \n",
    "                                                    diabetes['Outcome'], \n",
    "                                                    test_size=0.2)\n",
    "\n",
    "# Data transformation       \n",
    "scaler = StandardScaler()\n",
    "scaler.fit(X_train)\n",
    "X_train = scaler.transform(X_train)\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "85RUvcV3Uh-Y"
   },
   "source": [
    "**Model building**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "jtAqS2TQUh-Y",
    "outputId": "f58b69bc-ab4e-4aa8-dfb7-40c1a5c523ba"
   },
   "outputs": [],
   "source": [
    "## Model building\n",
    "model = Sequential()\n",
    "model.add(Input(shape=(8,)))\n",
    "model.add(Dense(6, activation='relu'))\n",
    "model.add(Dense(6, activation='tanh'))\n",
    "model.add(Dense(6, activation='tanh'))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UvG9OjLdUh-Y"
   },
   "source": [
    "**Model compilation**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rHqe1wjNUh-Y"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.optimizers import Adam\n",
    "\n",
    "# Loss function\n",
    "loss = BinaryCrossentropy()\n",
    "\n",
    "# Optimiser\n",
    "lr = 1e-3\n",
    "#optimiser = SGD(learning_rate=lr)\n",
    "optimiser = SGD(learning_rate=lr)\n",
    "\n",
    "model.compile(loss=loss,\n",
    "              optimizer=optimiser, \n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9JBmuGEQUh-Z"
   },
   "source": [
    "**Model training**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "C5sQs8jKUh-Z",
    "outputId": "c11edd74-c533-4471-9a1b-eba10dce191f"
   },
   "outputs": [],
   "source": [
    "## Model training\n",
    "epochs = 50\n",
    "batch_size = 16\n",
    "\n",
    "history = model.fit(x=X_train, y=y_train, \n",
    "                    validation_split=0.25,\n",
    "                    epochs=epochs,\n",
    "                    batch_size=batch_size)\n",
    "\n",
    "\n",
    "n_epochs = np.arange(0,epochs)\n",
    "\n",
    "fig, (ax1,ax2) = plt.subplots(2,1,figsize=(8,16))\n",
    "ax1.plot(n_epochs, history.history['loss'], label='training loss')\n",
    "ax1.plot(n_epochs, history.history['val_loss'], label='validation loss')\n",
    "ax1.set_ylim(-0.05,1.05)\n",
    "ax1.legend()\n",
    "\n",
    "ax2.plot(n_epochs, history.history['accuracy'], label='training accuracy')\n",
    "ax2.plot(n_epochs, history.history['val_accuracy'], label='validation accuracy')\n",
    "ax2.set_ylim(-0.05,1.05)\n",
    "ax2.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BENpofJbUh-Z"
   },
   "source": [
    "**Model evaluation**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ugjjIBXSUh-a",
    "outputId": "39cd1121-539f-4053-f276-aad87fbb1cc9"
   },
   "outputs": [],
   "source": [
    "loss_and_metrics = model.evaluate(X_test, y_test, \n",
    "                                  batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LWSbmOPGUh-a"
   },
   "source": [
    "### 2) Convolutional Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0SZE57X7Uh-a"
   },
   "source": [
    "<ins>**Task:**</ins> Please **implement your own multi-class image classifier to predict the outcome classes** in the diabetes dataset. <br>\n",
    "<ins>**Model:**</ins> Use a **convolutional neural network** with at least 1 hidden convolutional layer.<br> (*Note:* Since this is still a multi-class classification task, please use 10 units and the softmax activation in your output layer.)<br>\n",
    "<ins>**Data:**</ins> Use the 28*28 pixel images to predict the digit class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XCl6xpkYUh-a"
   },
   "source": [
    "The following solution consider a convolutional neural network with the following architecture:\n",
    "- <ins>input</ins> shape $28 \\times 28 \\times 1$,\n",
    "- <ins>1st hidden (convolutional) layer</ins> with **32 filters**, **kernel size (3,3)**, **stride (1,1)** and **ReLu** activations,\n",
    "- <ins>2nd hidden (convolutional) layer</ins> with **64 filters**, **kernel size (3,3)**, **stride (1,1)** and **ReLu** activations,\n",
    "- <ins>3rd hidden (convolutional) layer</ins> with **256 filters**, **kernel size (3,3)**, **stride (1,1)** and **tanh** activations,\n",
    "- <ins>output layer</ins> with **10 units** and **softmax** activation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vSqvLQGTUh-b"
   },
   "source": [
    "**Data Preparation**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2c-WB7USUh-b"
   },
   "outputs": [],
   "source": [
    "# Import the MNIST dataset\n",
    "from tensorflow.keras.datasets import mnist\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "\n",
    "# Load MNIST images and labels and normalise (range 0 to 1) image data\n",
    "(X_train, y_train), (X_test, y_test) = mnist.load_data() # 60,000 training + 10,000 test images/labels\n",
    "X_train = X_train / 255.0\n",
    "X_test  = X_test / 255.0\n",
    "\n",
    "# Reshape dataset to have a single channel \n",
    "X_train = X_train.reshape((X_train.shape[0], 28, 28, 1)) # The CNN requires this layout (batch_size, height, width, n_channels)\n",
    "X_test = X_test.reshape((X_test.shape[0], 28, 28, 1)) # The CNN requires this layout (batch_size, height, width, n_channels)\n",
    "\n",
    "# One-hot encode target values\n",
    "y_train = to_categorical(y_train)\n",
    "y_test = to_categorical(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ek8NxwWtUh-b"
   },
   "source": [
    "**Model building**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For Colab-users:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "2pJdleZDUh-c",
    "outputId": "1a9ed8fe-f461-45e0-e71f-8a43277f278d"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Input, Conv2D, Dense, Flatten\n",
    "with tpu_strategy.scope():\n",
    "  model = Sequential()\n",
    "  model.add(Input(shape=(28,28,1)))\n",
    "  model.add(Conv2D(32, kernel_size=(3,3), strides=(1,1), activation='relu'))\n",
    "  model.add(Conv2D(64, kernel_size=(3,3), strides=(1,1), activation='relu'))\n",
    "  model.add(Conv2D(256, kernel_size=(3,3), strides=(1,1), activation='tanh'))\n",
    "  model.add(Flatten()) # This flattens your image with width and height into a vector of length widht*height. \n",
    "  model.add(Dense(10, activation='softmax')) \n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qSsr9bcCbLm6"
   },
   "source": [
    "For non-Colab-users:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Input, Conv2D, Dense, Flatten\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Input(shape=(28,28,1)))\n",
    "model.add(Conv2D(32, kernel_size=(3,3), strides=(1,1), activation='relu'))\n",
    "model.add(Conv2D(64, kernel_size=(3,3), strides=(1,1), activation='relu'))\n",
    "model.add(Conv2D(256, kernel_size=(3,3), strides=(1,1), activation='tanh'))\n",
    "model.add(Flatten()) # This flattens your image with width and height into a vector of length widht*height. \n",
    "model.add(Dense(10, activation='softmax')) \n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1uKhX5rBUh-c"
   },
   "source": [
    "**Model compilation**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oFWio_qOUh-c"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.losses import CategoricalCrossentropy\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "\n",
    "# Loss function\n",
    "loss = CategoricalCrossentropy()\n",
    "\n",
    "# Optimiser\n",
    "lr = 1e-3\n",
    "\n",
    "optimiser = Adam(learning_rate=lr)\n",
    "\n",
    "model.compile(loss=loss,\n",
    "              optimizer=optimiser, \n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "YhVeA-QFZdSE"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RvHWyEp_Uh-c"
   },
   "source": [
    "**Model training**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "0-F3al1VUh-c",
    "outputId": "79a1d53e-8945-4c66-c38a-79db2d2acc50"
   },
   "outputs": [],
   "source": [
    "# Define number of epochs and batch size\n",
    "epochs = 50\n",
    "batch_size = 512\n",
    "\n",
    "\n",
    "history = model.fit(x=X_train, y=y_train, \n",
    "                    validation_split=0.1,\n",
    "                    epochs=epochs,\n",
    "                    batch_size=batch_size)\n",
    "\n",
    "\n",
    "n_epochs = np.arange(0,epochs)\n",
    "\n",
    "fig, (ax1,ax2) = plt.subplots(2,1,figsize=(8,16))\n",
    "ax1.plot(n_epochs, history.history['loss'], label='training loss')\n",
    "ax1.plot(n_epochs, history.history['val_loss'], label='validation loss')\n",
    "ax1.set_ylim(-0.05,1.05)\n",
    "ax1.legend()\n",
    "\n",
    "ax2.plot(n_epochs, history.history['accuracy'], label='training accuracy')\n",
    "ax2.plot(n_epochs, history.history['val_accuracy'], label='validation accuracy')\n",
    "ax2.set_ylim(-0.05,1.05)\n",
    "ax2.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Xa3TPeflUh-c"
   },
   "source": [
    "**Model evaluation**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YDONIU6yUh-d",
    "outputId": "30a546ff-d255-4635-8d6a-5c7dfd6fb9e9"
   },
   "outputs": [],
   "source": [
    "loss_and_metrics = model.evaluate(X_test, y_test, \n",
    "                                  batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4uMtcpK_drVz"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "week1_session5_deep_learning.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}