Ex6.strategies-against-overfitting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://www.th-koeln.de/img/logo.svg\" style=\"float: right;\" width=\"200\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6th exercise: <font color=\"#C70039\">Strategies against overfitting</font>\n",
    "* Course: DIS21a.1\n",
    "* Lecturer: <a href=\"https://www.gernotheisenberg.de/\">Gernot Heisenberg</a>\n",
    "* Author of notebook modifications and adaptations: <a href=\"https://www.gernotheisenberg.de/\">Gernot Heisenberg</a>\n",
    "* Date:   09.09.2024\n",
    "\n",
    "<img src=\"https://docs.aws.amazon.com/machine-learning/latest/dg/images/mlconcepts_image5.png\" style=\"float: center;\" width=\"600\">\n",
    "\n",
    "---------------------------------\n",
    "**GENERAL NOTE 1**: \n",
    "Please make sure you are reading the entire notebook, since it contains a lot of information about your tasks (e.g. regarding the set of certain paramaters or specific computational tricks, etc.), and the written mark downs as well as comments contain a lot of information on how things work together as a whole. \n",
    "\n",
    "**GENERAL NOTE 2**: \n",
    "* Please, when commenting source code, just use English language only. \n",
    "* When describing an observation (for instance, after you have run through your test plan) you may use German language.\n",
    "This applies to all exercises in DIS 21a.1.  \n",
    "\n",
    "---------------------\n",
    "\n",
    "### <font color=\"ce33ff\">DESCRIPTION</font>:\n",
    "\n",
    "Until now, in all the examples  (movie review sentiment prediction, topic classification, and house price regression) \n",
    "we noticed that the model performance on the simple hold-out validation data always peaked after a few epochs and then started degrading, i.e. the model quickly started to _overfit_ to the training data. Overfitting happens in every single machine learning problem. Learning how to deal with overfitting is essential to mastering machine learning and becoming a good data scientist.\n",
    "\n",
    "This notebook allows you for learning how you effectively can fight overfitting by preventing a model from learning misleading or irrelevant patterns found in the training data. As you have learned in my lecture already, the best solution is of course to get more training data. A model trained on more data will naturally generalize better. When that is not possible, the next best solution is to modulate the quantity of information that your model is allowed to store, or to add constraints on what information it is allowed to store. If a network can only afford to memorize a small number of patterns, the optimization process will force it to focus on the most prominent patterns, which have a better chance of generalizing well.\n",
    "\n",
    "The process of fighting overfitting this way is called _regularization_. Let us review some of the most common regularization techniques and apply them in practice. For doing so, the IMDB movie classification model from the previous exercise is chosen again.\n",
    "\n",
    "-------------------------------------------------------------------------------------------------------------\n",
    "\n",
    "### <font color=\"FFC300\">TASKS</font>:\n",
    "Within this notebook, the tasks that you need to work on are always listed as bullet points below. \n",
    "If a task is more challenging and consists of several steps, this is indicated as well. \n",
    "Make sure you have worked down the task list and commented your doings. \n",
    "This should be done using markdown.<br> \n",
    "<font color=red>Make sure you don't forget to specify your name and your matriculation number in the notebook before submitting it.</font>\n",
    "\n",
    "**YOUR TASKS in this exercise are as follows**:\n",
    "1. import the notebook to Google Colab.\n",
    "2. make sure you specified you name and your matriculation number in the header below my name and date. \n",
    "    * set the date too and remove mine.\n",
    "3. read the entire notebook carefully. \n",
    "    * for better understanding, add comments whereever you feel it necessary \n",
    "    * run the notebook for the first time and note the result in a markdown table. \n",
    "        * I have provided you with an example of a table in markdown (see below). Make sure you adapt your table accordingly. \n",
    "        * Put the table at the end of the notebook. \n",
    "        * This type of table will be needed in the other exercises as well. Always put it at the end.\n",
    "    \n",
    "| type of method | loss function | optimizer | accuracy |\n",
    "| :-: | :-: | :-: | :-: |\n",
    "| classification | categorical_crossentropy | bamm !|.666\n",
    "\n",
    "4. in section 1 (Reducing the network's size) add your own code whereever it is indicated in the notebook and get it running.\n",
    "5. add networks that are slightly bigger and smaller than the one that is optimal (i.e., in addition to the ones that are largely smaller or bigger).\n",
    "    * visualize all their performance curves in the same two charts: the one for the acc and the one for the loss.\n",
    "    * (`little hint`: since the batch_size and the no_of_epochs stays the same always, you should consider to build yourself a (ANN_build_compile_function) which you can instantiate everytime you want to build a new ANN of different size. Store the history object in a list and visualize them as indicated above.\n",
    "    * use a table similar to the one from above to store the min_no_of_epochs for every model. You might consider to call a python function to compute the minimum rather than doing it visually.\n",
    "    \n",
    "6. write a test plan for testing the so-called dropout regularization. Start with dropout of 10% and then increase stepwise to end up with 90% and see what happens to the ANN. Store the values in the table (as the one from above) and try find the best dropout-rate.\n",
    "\n",
    "-----------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## START OF THE NOTEBOOK CODE\n",
    "----------------------------------------------------------------------------------------------------------------------\n",
    "### necessary imports\n",
    "others are going to be included as soon as they are needed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import tensorflow\n",
    "from tensorflow.keras.datasets import imdb\n",
    "import numpy as np\n",
    "tensorflow.keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### loading and preparing the IMDB movie review data set\n",
    "as done before"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)\n",
    "\n",
    "def vectorize_sequences(sequences, dimension=10000):\n",
    "    # Create an all-zero matrix of shape (len(sequences), dimension)\n",
    "    results = np.zeros((len(sequences), dimension))\n",
    "    for i, sequence in enumerate(sequences):\n",
    "        results[i, sequence] = 1.  # set specific indices of results[i] to 1s\n",
    "    return results\n",
    "\n",
    "# Our vectorized training data\n",
    "x_train = vectorize_sequences(train_data)\n",
    "# Our vectorized test data\n",
    "x_test = vectorize_sequences(test_data)\n",
    "# Our vectorized labels\n",
    "y_train = np.asarray(train_labels).astype('float32')\n",
    "y_test = np.asarray(test_labels).astype('float32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fighting overfitting\n",
    "\n",
    "#### 1. Reducing the network's size\n",
    "\n",
    "The simplest way to prevent overfitting is to reduce the size of the model, i.e. the number of learnable parameters in the model (which is determined by the number of layers and the number of units per layer). In deep learning, the number of learnable parameters in a model is often referred to as the model's \"capacity\". Intuitively, a model with more parameters will have more \"memorization capacity\" and therefore will be able to easily learn a perfect dictionary-like mapping between training samples and their targets, a mapping without any generalization power. \n",
    "\n",
    "<font color=\"C70039\">Always keep this in mind:</font> <font color=\"ce33ff\">machine learning, and especially deep learning models tend to be good at fitting to the training data, but the real challenge is generalization, not fitting.</font>\n",
    "\n",
    "However, if the network has limited memorization resources, it will not be able to learn this mapping as easily, and thus, in order to minimize its loss, it will have to resort to learning compressed representations that have predictive power regarding the targets, in other words, the type of representations that we data scientists are interested in.\n",
    "\n",
    "<font color=\"C70039\">NOTE:</font><font color=\"ce33ff\">Always keep in mind that you should be using models that have enough parameters that they will not be underfitting: your model should not be starved for memorization resources. </font>\n",
    "    \n",
    "<font color=\"C70039\">There is a compromise to be found between `too much capacity` and `not enough capacity`.</font>\n",
    "\n",
    "Unfortunately, there is no magical formula to determine what the right number of layers is or what the right size for each layer is. A data scientist needs to evaluate an array of different architectures in order to find the right model size. The general workflow to find an appropriate model size is to start with relatively few layers and parameters and start increasing the size of the layers or adding new layers until diminishing returns with regard to the validation loss. \n",
    "\n",
    "If you are interested to deepen this topic on how to compute the right size of training data, you should query the web for _learning curves_ such as the following:\n",
    "\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/2/24/Learning_Curves_%28Naive_Bayes%29.png\" style=\"float: center;\" width=\"500\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, let us try to determine the right size stepwise on the IMDB movie review classifications ANN, who's architecture was: (compare lecture slides)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras import models\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "original_model = models.Sequential()\n",
    "original_model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
    "original_model.add(layers.Dense(16, activation='relu'))\n",
    "original_model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "original_model.compile(optimizer='rmsprop',\n",
    "                       loss='binary_crossentropy',\n",
    "                       metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try to replace that ANN of (16,16,1) with a smaller network of (4,4,1), as shown on lecture slides p.210."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "smaller_model = models.Sequential()\n",
    "\n",
    "'''ADD THE MISSING CODE HERE'''\n",
    "\n",
    "# your code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Now, let's compare the validation losses of the original network and the smaller network. The dots are the validation loss values of the smaller network, and the crosses are the initial network (remember: a lower validation loss signals a better model)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "original_hist = original_model.fit(x_train, y_train,\n",
    "                                   epochs=20,\n",
    "                                   batch_size=512,\n",
    "                                   validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "smaller_model_hist = smaller_model.fit(x_train, y_train,\n",
    "                                       epochs=20,\n",
    "                                       batch_size=512,\n",
    "                                       validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "epochs = range(1, 21) # always add one\n",
    "original_val_loss = original_hist.history['val_loss']\n",
    "smaller_model_val_loss = smaller_model_hist.history['val_loss']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# b+ is for \"blue cross\"\n",
    "plt.plot(epochs, original_val_loss, 'b+', label='Original model')\n",
    "# \"bo\" is for \"blue dot\"\n",
    "plt.plot(epochs, smaller_model_val_loss, 'bo', label='Smaller model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the smaller network starts overfitting later than the reference one and its performance \n",
    "degrades much more slowly once it starts overfitting.\n",
    "\n",
    "Now, let's add to this benchmark a network that has much more capacity, far more than the problem would need (again see lectures slide p.210)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "bigger_model = models.Sequential()\n",
    "bigger_model.add(layers.Dense(512, activation='relu', input_shape=(10000,)))\n",
    "bigger_model.add(layers.Dense(512, activation='relu'))\n",
    "bigger_model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "bigger_model.compile(optimizer='rmsprop',\n",
    "                     loss='binary_crossentropy',\n",
    "                     metrics=['acc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "bigger_model_hist = bigger_model.fit(x_train, y_train,\n",
    "                                     epochs=20,\n",
    "                                     batch_size=512,\n",
    "                                     validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how the bigger ANN performes compared to the reference one. The dots are the validation loss values of the bigger network, and the crosses are the initial network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "bigger_model_val_loss = bigger_model_hist.history['val_loss']\n",
    "\n",
    "plt.plot(epochs, original_val_loss, 'b+', label='Original model')\n",
    "plt.plot(epochs, bigger_model_val_loss, 'bo', label='Bigger model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The bigger network starts overfitting almost right away, after just one epoch and overfits much more severely. Its validation loss is also more noisy.\n",
    "\n",
    "Meanwhile, here are the training losses for our two networks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "original_train_loss = original_hist.history['loss']\n",
    "bigger_model_train_loss = bigger_model_hist.history['loss']\n",
    "\n",
    "plt.plot(epochs, original_train_loss, 'b+', label='Original model')\n",
    "plt.plot(epochs, bigger_model_train_loss, 'bo', label='Bigger model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Training loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the bigger network gets its training loss near zero very quickly. The more capacity the network has, the quicker it will be \n",
    "able to model the training data (resulting in a low training loss), but the more susceptible it is to overfitting (resulting in a large \n",
    "difference between the training and validation loss)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Adding weight regularization\n",
    "\n",
    "According to Occam's Razor, given some training data and a network architecture, there are multiple sets of weights values (multiple _models_) that could explain the data, and simpler models are less likely to overfit than complex ones.\n",
    "\n",
    "A `simple model` in this context is a model where the distribution of parameter values has got less entropy or a model with fewer parameters altogether, as seen in the first section above. Thus, a common way to fight overfitting is to put constraints on the complexity of a network by forcing its weights to only take small values, which makes the distribution of weight values more \"regular\". This is called \"weight regularization\", and it is done by adding to the loss function of the network a kind of _cost_ associated with having large weights. In the lecture you have learned two ways of regularization:\n",
    "\n",
    "* L1 regularization, where the cost added is proportional to the _absolute value of the weight coefficients_ (i.e. to what is called the \"L1 norm\" of the weights).\n",
    "* L2 regularization, where the cost added is proportional to the _square of the value of the weight coefficients_ (i.e. to what is called the \"L2 norm\" of the weights). L2 regularization is also called _weight decay_ in the context of neural networks. As often in machine learning, do not let the different names confuse you: weight decay is mathematically the exact same as L2 regularization.\n",
    "\n",
    "In Keras, weight regularization is quite easy by super simply passing _weight regularizer instances_ to layers as keyword arguments. Let us add L2 weight regularization to the ANN from above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras import regularizers\n",
    "\n",
    "l2_model = models.Sequential()\n",
    "l2_model.add(layers.Dense(16, kernel_regularizer=regularizers.l2(0.001),\n",
    "                          activation='relu', input_shape=(10000,)))\n",
    "l2_model.add(layers.Dense(16, kernel_regularizer=regularizers.l2(0.001),\n",
    "                          activation='relu'))\n",
    "l2_model.add(layers.Dense(1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "l2_model.compile(optimizer='rmsprop',\n",
    "                 loss='binary_crossentropy',\n",
    "                 metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`l2(0.001)` means that every coefficient in the weight matrix of the layer will add `0.001 * weight_coefficient_value` to the total loss of \n",
    "the network. Note that because this penalty is _only added during training time_, the loss for this network will be much higher at training than at test time.\n",
    "\n",
    "Here is the impact of our L2 regularization penalty:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "l2_model_hist = l2_model.fit(x_train, y_train,\n",
    "                             epochs=20,\n",
    "                             batch_size=512,\n",
    "                             validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "l2_model_val_loss = l2_model_hist.history['val_loss']\n",
    "\n",
    "plt.plot(epochs, original_val_loss, 'b+', label='Original model')\n",
    "plt.plot(epochs, l2_model_val_loss, 'bo', label='L2-regularized model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "As you can see, the model with L2 regularization (dots) has become much more resistant to overfitting than the reference model (crosses), \n",
    "even though both models have the same number of parameters.\n",
    "\n",
    "As alternatives to L2 regularization, you could use one of the following Keras weight regularizers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import regularizers\n",
    "\n",
    "# L1 regularization\n",
    "regularizers.l1(0.001)\n",
    "\n",
    "# L1 and L2 regularization at the same time\n",
    "regularizers.l1_l2(l1=0.001, l2=0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Adding dropout\n",
    "\n",
    "Dropout is one of the most effective and most commonly used regularization technique for neural networks, developed by Hinton and team. Dropout, applied to a layer, consists of randomly \"dropping out\" (which simply means 'setting to zero') a number of \n",
    "output features of the layer during training. \n",
    "\n",
    "The \"dropout rate\" is the fraction of the features that are being zero'ed-out; it is usually set between 0.2 and 0.5. \n",
    "\n",
    "In Keras, dropout can be added via the so-called `Dropout` layer, which gets applied to the output of layer right before it, e.g.: \n",
    "\n",
    "model.add(layers.Dropout(0.5))\n",
    "\n",
    "Let us add two `Dropout` layers in our IMDB network to see how effective they are for reducing overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dpt_model = models.Sequential()\n",
    "dpt_model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
    "dpt_model.add(layers.Dropout(0.5))\n",
    "dpt_model.add(layers.Dense(16, activation='relu'))\n",
    "dpt_model.add(layers.Dropout(0.5))\n",
    "dpt_model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "dpt_model.compile(optimizer='rmsprop',\n",
    "                  loss='binary_crossentropy',\n",
    "                  metrics=['acc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dpt_model_hist = dpt_model.fit(x_train, y_train,\n",
    "                               epochs=20,\n",
    "                               batch_size=512,\n",
    "                               validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dpt_model_val_loss = dpt_model_hist.history['val_loss']\n",
    "\n",
    "plt.plot(epochs, original_val_loss, 'b+', label='Original model')\n",
    "plt.plot(epochs, dpt_model_val_loss, 'bo', label='Dropout-regularized model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <font color=\"#C70039\">Include your dropout result table here and reflect a good test plan (see task list)</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}