Ex8.Using-CNNs-with-small-datasets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://www.th-koeln.de/img/logo.svg\" style=\"float: right;\" width=\"200\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8th exercise: <font color=\"#C70039\">Fight overfitting in CNNs using data augmentation</font>\n",
    "* Course: DIS21a.1\n",
    "* Lecturer: <a href=\"https://www.gernotheisenberg.de/\">Gernot Heisenberg</a>\n",
    "* Author of notebook modifications and adaptations: <a href=\"https://www.gernotheisenberg.de/\">Gernot Heisenberg</a>\n",
    "* Date:   13.07.2021\n",
    "\n",
    "<img src=\"https://miro.medium.com/max/700/1*bqNylp7FcqIBWg0DrcimUw.png\" style=\"float: center;\" width=\"600\">\n",
    "\n",
    "---------------------------------\n",
    "**GENERAL NOTE 1**: \n",
    "Please make sure you are reading the entire notebook, since it contains a lot of information about your tasks (e.g. regarding the set of certain paramaters or specific computational tricks, etc.), and the written mark downs as well as comments contain a lot of information on how things work together as a whole. \n",
    "\n",
    "**GENERAL NOTE 2**: \n",
    "* Please, when commenting source code, just use English language only. \n",
    "* When describing an observation (for instance, after you have run through your test plan) you may use German language.\n",
    "This applies to all exercises in DIS 21a.1.  \n",
    "\n",
    "---------------------\n",
    "\n",
    "### <font color=\"ce33ff\">DESCRIPTION</font>:\n",
    "\n",
    "This notebook allows for learning how you effectively fight overfitting when using convolutional neural networks (CNN) with small data sets. Training an image classification model using only very little data is a very common situation whenever working in computer vision within a professional context.\n",
    "\n",
    "It is not possible to train a CNN to solve a complex problem with just a few tens of samples, but a few hundreds can \n",
    "potentially suffice if the model is small and well-regularized and if the task is simple. \n",
    "\n",
    "Because CNNs learn local, translation-invariant features, they are very data-efficient on perceptual problems. \n",
    "Training a CNN from scratch on a very small image dataset will still yield reasonable results despite a relative lack of data, without the need for any custom feature engineering. You will see this in action in this exercise.\n",
    "\n",
    "This exercise focuses on classifying images as \"dogs\" or \"cats\", in a dataset containing 4000 pictures of cats and dogs (2000 cats, 2000 dogs). 2000 pictures are used for training, 1000 for validation, and finally 1000 for testing.\n",
    "\n",
    "For the start, a small CNN is going to be trained on 2000 training samples, without any regularization for getting a performance baseline (classification accuracy). It will become obvious quite quickly that the main issue is overfitting. \n",
    "Then, *data augmentation* is going to be used, as you have learned it to be a powerful technique for fighting overfitting in computer vision.\n",
    "\n",
    "-------------------------------------------------------------------------------------------------------------\n",
    "\n",
    "### <font color=\"FFC300\">TASKS</font>:\n",
    "Within this notebook, the tasks that you need to work on are always listed as bullet points below. \n",
    "If a task is more challenging and consists of several steps, this is indicated as well. \n",
    "Make sure you have worked down the task list and commented your doings. \n",
    "This should be done using markdown.<br> \n",
    "<font color=red>Make sure you don't forget to specify your name and your matriculation number in the notebook before submitting it.</font>\n",
    "\n",
    "**YOUR TASKS in this exercise are as follows**:\n",
    "1. import the notebook to Google Colab.\n",
    "2. make sure you specified you name and your matriculation number in the header below my name and date. \n",
    "    * set the date too and remove mine.\n",
    "3. read the entire notebook carefully. \n",
    "    * for better understanding, add comments whereever you feel it necessary.\n",
    "    * run the notebook for the first time and note the result in a markdown table. \n",
    "        * I have provided you with an example of a table in markdown (see below). Make sure you adapt your table accordingly. \n",
    "        * Put the table at the end of the notebook. \n",
    "        * This type of table will be needed in the other exercises as well. Always put it at the end.\n",
    "    \n",
    "| type of method | loss function | optimizer | accuracy |\n",
    "| :-: | :-: | :-: | :-: |\n",
    "| classification | categorical_crossentropy | bamm !|.666\n",
    "\n",
    "4. given: batch_size=20 and size of training and validation data set.<br>\n",
    "   Your task: Calculate \n",
    "    * steps_per_epoch=?\n",
    "    * validation_steps=?\n",
    "    \n",
    "5. given: batch_size=32 and size of training and validation data set.<br>\n",
    "    Your task: Calculate\n",
    "    * steps_per_epoch=?\n",
    "    * validation_steps=?\n",
    "\n",
    "5. the original data set has got a total of 25.000 images (12.500 cats and 12.500 dogs, respectively).<br>\n",
    "    Your task is to develop a test plan of the following kind:\n",
    "    * take a randomly chosen stack of 4000 training, 2000 validation and 1000 testing images. Make sure that no image is used \n",
    "      twice in training and validation or testing. Change the hyperparameters accordingly.\n",
    "      * <font color=\"red\">NOTE:</font> make sure you save your model under different names every time you create a new one.\n",
    "    * take a randomly chosen stack of 6000 training, 3000 validation and 1000 testing images. Make sure that no image is used \n",
    "      twice in training and validation or testing. Change the hyperparameters accordingly.\n",
    "\n",
    "8. Do a hyperparameter optimization. Change the CNN's architecture according to your personal test plan. Store the performance values in a table as the one shown above, at the very end of the notebook. \n",
    "\n",
    "-----------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## START OF THE NOTEBOOK CODE\n",
    "----------------------------------------------------------------------------------------------------------------------\n",
    "### necessary imports\n",
    "others are going to be included as soon as they are needed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow\n",
    "tensorflow.keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### loading the cats and dogs data set\n",
    "\n",
    "The cats and dogs data set is not packaged with Keras. It was made available by Kaggle.com as part of a computer vision competition in late 2013, back when CNNs were not quite mainstream yet. \n",
    "\n",
    "You can download the original data set at: \n",
    "`https://www.kaggle.com/c/dogs-vs-cats/data`\n",
    "\n",
    "All pictures are color images using the jpg format (see example):\n",
    "<img src=\"https://s3.amazonaws.com/book.keras.io/img/ch5/cats_vs_dogs_samples.jpg\" style=\"float: center;\" width=\"300\">\n",
    "\n",
    "The original data set contains 25.000 images of dogs and cats (12.500 of each class) and is, even though compressed, 543MB large. After downloading and uncompressing it, we will create a new data set containing three subsets: \n",
    "* a training set with 1000 samples of each class\n",
    "* a validation set with 500 samples of each class\n",
    "* a test set with 500 samples of each class.\n",
    "\n",
    "Here is the code to do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os, shutil"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=\"#C70039\">NOTE:</font> Make sure you set the pathes according to your programming environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The path to the directory where the original\n",
    "# dataset is uncompressed\n",
    "original_dataset_dir = '../data/kaggle_original_data'\n",
    "\n",
    "# The directory where to\n",
    "# store the smaller dataset\n",
    "base_dir = '../data/cats_and_dogs_small'\n",
    "if not os.path.exists(base_dir):\n",
    "    os.mkdir(base_dir)\n",
    "\n",
    "# Create the directories for the \n",
    "# training partition\n",
    "train_dir = os.path.join(base_dir, 'train')\n",
    "if not os.path.exists(train_dir):\n",
    "    os.mkdir(train_dir)\n",
    "    \n",
    "# validation partition\n",
    "validation_dir = os.path.join(base_dir, 'validation')\n",
    "if not os.path.exists(validation_dir):\n",
    "    os.mkdir(validation_dir)\n",
    "\n",
    "# test partition\n",
    "test_dir = os.path.join(base_dir, 'test')\n",
    "if not os.path.exists(test_dir):\n",
    "    os.mkdir(test_dir)\n",
    "\n",
    "print(\"train_dir=\",train_dir)\n",
    "print(\"validation_dir=\",validation_dir)\n",
    "print(\"test_dir=\",test_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now create cat and dog subfolders within each data partition folder. Make sure to setting up the following folder structure.\n",
    "\n",
    "<img src=\"https://miro.medium.com/max/700/1*HpvpA9pBJXKxaPCl5tKnLg.jpeg\" style=\"float: center;\" width=\"600\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Directory with the training cat pictures\n",
    "train_cats_dir = os.path.join(train_dir, 'cats')\n",
    "if not os.path.exists(train_cats_dir):\n",
    "    os.mkdir(train_cats_dir)\n",
    "\n",
    "# Directory with the training dog pictures\n",
    "train_dogs_dir = os.path.join(train_dir, 'dogs')\n",
    "if not os.path.exists(train_dogs_dir):\n",
    "    os.mkdir(train_dogs_dir)\n",
    "\n",
    "# Directory with the validation cat pictures\n",
    "validation_cats_dir = os.path.join(validation_dir, 'cats')\n",
    "if not os.path.exists(validation_cats_dir):\n",
    "    os.mkdir(validation_cats_dir)\n",
    "\n",
    "# Directory with the validation dog pictures\n",
    "validation_dogs_dir = os.path.join(validation_dir, 'dogs')\n",
    "if not os.path.exists(validation_dogs_dir):\n",
    "    os.mkdir(validation_dogs_dir)\n",
    "\n",
    "# Directory with the test cat pictures\n",
    "test_cats_dir = os.path.join(test_dir, 'cats')\n",
    "if not os.path.exists(test_cats_dir):\n",
    "    os.mkdir(test_cats_dir)\n",
    "\n",
    "# Directory with the test dog pictures\n",
    "test_dogs_dir = os.path.join(test_dir, 'dogs')\n",
    "if not os.path.exists(test_dogs_dir):\n",
    "    os.mkdir(test_dogs_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now from the original data set, copy \n",
    "\n",
    "* 1000 cats and 1000 dogs into the training partition folder\n",
    "* 500 cats and 500 dogs into the validation partition folder\n",
    "* 500 cats and 500 dogs into the testing partition folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copy first 1000 cat images to train_cats_dir\n",
    "fnames = ['cat.{}.jpg'.format(i) for i in range(1000)]\n",
    "for fname in fnames:\n",
    "    src = os.path.join(original_dataset_dir, fname)\n",
    "    dst = os.path.join(train_cats_dir, fname)\n",
    "    shutil.copyfile(src, dst)\n",
    "\n",
    "# Copy NEXT 500 cat images to validation_cats_dir\n",
    "fnames = ['cat.{}.jpg'.format(i) for i in range(1000, 1500)]\n",
    "for fname in fnames:\n",
    "    src = os.path.join(original_dataset_dir, fname)\n",
    "    dst = os.path.join(validation_cats_dir, fname)\n",
    "    shutil.copyfile(src, dst)\n",
    "    \n",
    "# Copy NEXT 500 cat images to test_cats_dir\n",
    "fnames = ['cat.{}.jpg'.format(i) for i in range(1500, 2000)]\n",
    "for fname in fnames:\n",
    "    src = os.path.join(original_dataset_dir, fname)\n",
    "    dst = os.path.join(test_cats_dir, fname)\n",
    "    shutil.copyfile(src, dst)\n",
    "    \n",
    "# Copy first 1000 dog images to train_dogs_dir\n",
    "fnames = ['dog.{}.jpg'.format(i) for i in range(1000)]\n",
    "for fname in fnames:\n",
    "    src = os.path.join(original_dataset_dir, fname)\n",
    "    dst = os.path.join(train_dogs_dir, fname)\n",
    "    shutil.copyfile(src, dst)\n",
    "    \n",
    "# Copy next 500 dog images to validation_dogs_dir\n",
    "fnames = ['dog.{}.jpg'.format(i) for i in range(1000, 1500)]\n",
    "for fname in fnames:\n",
    "    src = os.path.join(original_dataset_dir, fname)\n",
    "    dst = os.path.join(validation_dogs_dir, fname)\n",
    "    shutil.copyfile(src, dst)\n",
    "    \n",
    "# Copy next 500 dog images to test_dogs_dir\n",
    "fnames = ['dog.{}.jpg'.format(i) for i in range(1500, 2000)]\n",
    "for fname in fnames:\n",
    "    src = os.path.join(original_dataset_dir, fname)\n",
    "    dst = os.path.join(test_dogs_dir, fname)\n",
    "    shutil.copyfile(src, dst)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=\"#C70039\">Check</font>, whether all copying worked correctly.<br>\n",
    "So, count how many pictures are in each cat and dog split of every partition (training/validation/testing)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('total training cat images:', len(os.listdir(train_cats_dir)))\n",
    "print('total training dog images:', len(os.listdir(train_dogs_dir)))\n",
    "print('--------------------------')\n",
    "print('total validation cat images:', len(os.listdir(validation_cats_dir)))\n",
    "print('total validation dog images:', len(os.listdir(validation_dogs_dir)))\n",
    "print('--------------------------')\n",
    "print('total testing cat images:', len(os.listdir(test_cats_dir)))\n",
    "print('total testing dog images:', len(os.listdir(test_dogs_dir)))\n",
    "print('--------------------------')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PART I\n",
    "### building the CNN for baseline computation\n",
    "\n",
    "From the last exercise the general structure of the CNN is reused, hence the CNN will be a stack of alternating `Conv2D` (with `relu` activation) and `MaxPooling2D` layers.\n",
    "\n",
    "However, since this exercise is dealing with bigger images and a more complex problem, the CNN architecture should be made accordingly larger and should have one more `Conv2D` + `MaxPooling2D` stage. This serves both, to augment the capacity of the network and to further reduce the size of the feature maps, so that they are not by far to large when the `Flatten` layer is reached. Here, since the starting image size is 150x150 (an arbitrary choice btw), the feature maps can end with a size of 7x7 right before the `Flatten` layer.\n",
    "\n",
    "Note that the depth of the feature maps is progressively increasing in the network (from 32 to 128), while the size of the feature maps is decreasing (from 148x148 to 7x7). This is a pattern that is typical for almost all CNNs.\n",
    "\n",
    "Since the problem at hand is a binary classification problem, the network ends with a single unit (a `Dense` layer of size 1) and a `sigmoid` activation. This unit will encode the probability that the network is looking at one class or the other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import layers\n",
    "from tensorflow.keras import models\n",
    "\n",
    "model = models.Sequential()\n",
    "model.add(layers.Conv2D(32, (3, 3), activation='relu',input_shape=(150, 150, 3)))\n",
    "\n",
    "model.add(layers.MaxPooling2D((2, 2)))\n",
    "model.add(layers.Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(layers.MaxPooling2D((2, 2)))\n",
    "model.add(layers.Conv2D(128, (3, 3), activation='relu'))\n",
    "model.add(layers.MaxPooling2D((2, 2)))\n",
    "model.add(layers.Conv2D(128, (3, 3), activation='relu'))\n",
    "model.add(layers.MaxPooling2D((2, 2)))\n",
    "\n",
    "model.add(layers.Flatten())\n",
    "model.add(layers.Dense(512, activation='relu'))\n",
    "model.add(layers.Dense(1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at how the dimensions of the feature maps change with every successive layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=\"#C70039\">NOTE:</font> There are 2000 training images, and then 1000 validation images and 1000 test images. In each split, there is the same number of samples from each class (cats and dogs): since this is a balanced binary classification problem, it means that classification `accuracy` will be an appropriate measure of performance. <br>\n",
    "\n",
    "For the compilation step take the `RMSprop` optimizer as usual. \n",
    "Since the network ends with a single sigmoid unit, use binary crossentropy as loss function \n",
    "(<font color=\"#C70039\">CHECK:</font>The last page (p.216) of file `DIS21a.1-8.Tools.And.Strategies.For.Better.Results.pdf` can be used as a cheatsheet on what last layer activation and what loss function to use in the various situations)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import optimizers\n",
    "\n",
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer=optimizers.RMSprop(lr=1e-4),\n",
    "              metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### data preparation: generating image tensors from image files by using  `ImageDataGenerator()`\n",
    "\n",
    "In general, data must be transformed into appropriately pre-processed floating point tensors before being fed into a neural network. \n",
    "Until now, the image data is packed into jpeg files, so the algorithm for transforming them is as follows:\n",
    "\n",
    "1. Read the picture files.\n",
    "2. Decode the JPEG content to RBG grids of pixels.\n",
    "3. Convert these into floating point tensors.\n",
    "4. Rescale the pixel values (between 0 and 255) to the [0,1] interval (min/max-standardization).\n",
    "\n",
    "Luckily, Keras has got utilities to take care of these steps automatically. \n",
    "Keras has got a module with image processing helper tools, located at `keras.preprocessing.image`. \n",
    "In particular, it contains the class `ImageDataGenerator()` which allows to quickly set up Python generators that can automatically turn image files on disk into batches of pre-processed tensors. \n",
    "\n",
    "Let's have a look. The ImageDataGenerator class has three methods `flow()`, `flow_from_directory()` and `flow_from_dataframe()` to read images from a numpy array or from folders containing images.\n",
    "\n",
    "If you are interested in diving deeper into the topic, this article is very recommendable.<br>\n",
    "https://medium.com/@vijayabhaskar96/tutorial-image-classification-with-keras-flow-from-directory-and-generators-95f75ebe5720 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
    "\n",
    "# All images will be [0,1] standardized\n",
    "train_datagen = ImageDataGenerator(rescale=1.0/255)\n",
    "valid_datagen = ImageDataGenerator(rescale=1.0/255)\n",
    "\n",
    "training_generator = train_datagen.flow_from_directory(\n",
    "        # This is the target directory\n",
    "        train_dir,\n",
    "        # All images will be resized to 150x150\n",
    "        target_size=(150, 150),\n",
    "        batch_size=20,\n",
    "        # Since binary_crossentropy loss is used, binary labels are needed\n",
    "        class_mode='binary')\n",
    "\n",
    "validation_generator = valid_datagen.flow_from_directory(\n",
    "        validation_dir,\n",
    "        target_size=(150, 150),\n",
    "        batch_size=20,\n",
    "        class_mode='binary')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the output of, e.g. the training generator:<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for data_batch, labels_batch in training_generator:\n",
    "    print('This is the shape of the training data batch:', data_batch.shape)\n",
    "    print('This is the shape of the training label batch:', labels_batch.shape)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It yields batches of 150x150 RGB images (shape `(20, 150, 150, 3)`) and binary labels (shape `(20,)`). \n",
    "20 is the number of samples in each batch (batch size=20). \n",
    "\n",
    "<font color=\"red\">Note</font> that the generator works as an iterator and outputs these batches in an infinite loop over the images present in the destination folder. For this reason, the iteration loop must be interrupted at some point.\n",
    "\n",
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us fit our model to the data using the generator. We do it using the `fit` method as always. It expects as first argument a Python data generator that will yield batches of inputs and targets indefinitely, like the one from above does. \n",
    "Because the data is being generated endlessly, the generator needs to know how many samples to draw from the generator before declaring one epoch to be complete. \n",
    "This is the role of the `steps_per_epoch` argument: after having drawn `steps_per_epoch` batches from the generator, i.e. after having run for `steps_per_epoch` gradient descent steps, the fitting process will go to the next epoch. \n",
    "\n",
    "In this case here, batches are 20-sample large, so it will take 100 batches until we see our target of 2000 samples. Keep this in mind for being able to solve some of the tasks from the task list.\n",
    "\n",
    "When using `fit`, a `validation_data` argument can be passed as well and like done many times before already. \n",
    "Importantly, also this argument is allowed to be a data generator itself, but it could be a tuple of Numpy arrays as well. \n",
    "If a generator is passed as `validation_data`, then this generator is expected to yield batches of validation data endlessly, and thus, again, the `validation_steps` argument needs to be specified, which tells the process how many batches to draw from the validation generator for evaluation. With this information the model can be trained now."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### training and validation in one pass using the history object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "history = model.fit(\n",
    "            training_generator,\n",
    "            steps_per_epoch=10,\n",
    "            epochs=30,\n",
    "            validation_data=validation_generator,\n",
    "            validation_steps=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### save the baseline classification model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "It is good practice to always save your models after training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save('cats_dogs_classification_CNN_baseline.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ### plot the history object\n",
    "Plot the loss and accuracy of the model over the training and validation epochs using pyplot again. It is pretty straight forward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(len(acc))\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These plots are characteristic for overfitting and because there are relatively few training samples (total is 2000), this  is going to be the number-one-concern here. \n",
    "\n",
    "The training accuracy increases linearly over time, while the validation accuracy stalls at a lower percentage. The validation loss reaches its minimum after only a few epochs and then stalls, while the training loss keeps decreasing linearly until the bitter end :-).\n",
    "\n",
    "A number of techniques that can help to fight overfitting are known already. These are dropout and L2 regularization. \n",
    "However, a new technique you have learned during the lectures is called `data augmentation` and is used almost universally when processing images with deep learning models.\n",
    "\n",
    "So let us dive into the second part and see what it can do for us. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PART II\n",
    "### building the CNN using data augmentation\n",
    "\n",
    "<font color=\"#C70039\">General thougts:</font>\n",
    "Since overfitting is caused by having too few samples to learn from (leaving a model unable to generalize to new data), hypothetically, given infinite data, a model would be exposed to every possible aspect of that data distribution and in consequence it  would never overfit. \n",
    "\n",
    "Data augmentation takes this idea and generates more training data from existing training samples, by \"augmenting\" the samples via a number of random transformations that yield *believable*-looking images. \n",
    "The goal is that at training time, the model would never see the exact same picture twice. This helps the model get exposed to more aspects of the data and generalize much better. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Keras, configuring a number of random transformations on training images can easily be performed by using the `ImageDataGenerator` instance, that was introduced earlier already. Here an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_aug_gen = ImageDataGenerator(\n",
    "      rotation_range=40,\n",
    "      width_shift_range=0.2,\n",
    "      height_shift_range=0.2,\n",
    "      shear_range=0.2,\n",
    "      zoom_range=0.2,\n",
    "      horizontal_flip=True,\n",
    "      fill_mode='nearest')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are just a few of the options available (for more information, see the Keras documentation or external resources such as: https://machinelearningmastery.com/how-to-configure-image-data-augmentation-when-training-deep-learning-neural-networks/ ). Let's look at them in more detail:\n",
    "\n",
    "* `rotation_range` is a value in degrees (0-180), a range within which to randomly rotate pictures.\n",
    "* `width_shift` and `height_shift` are ranges (as a fraction of total width or height) within which to randomly translate pictures vertically or horizontally.\n",
    "* `shear_range` is for randomly applying shearing transformations.\n",
    "* `zoom_range` is for randomly zooming inside pictures.\n",
    "* `horizontal_flip` is for randomly flipping half of the images horizontally -- relevant when there are no assumptions of horizontal asymmetry (e.g. real-world pictures).\n",
    "* `fill_mode` is the strategy used for filling in newly created pixels, which can appear after a rotation or a width/height shift.\n",
    "-----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the augmented images by picking one and plotting its transformations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This module contains some image preprocessing utilities\n",
    "from tensorflow.keras.preprocessing import image\n",
    "\n",
    "all_filenames = [os.path.join(train_cats_dir, fname) for fname in os.listdir(train_cats_dir)]\n",
    "\n",
    "# Pick one image to \"augment\": \n",
    "'''PLay around and take some others, maybe a dog also.'''\n",
    "image_picked = all_filenames[666]\n",
    "\n",
    "# Load that image and resize it\n",
    "img = image.load_img(image_picked, target_size=(150, 150))\n",
    "\n",
    "# Convert it to a Numpy array with shape (150, 150, 3), \n",
    "# so all three color channels are separated now.\n",
    "x = image.img_to_array(img)\n",
    "\n",
    "# Reshape it to (1, 150, 150, 3)\n",
    "x = x.reshape((1,) + x.shape)\n",
    "\n",
    "# The .flow() command generates batches of randomly transformed images.\n",
    "# It will loop indefinitely, so make sure to `break` the loop at some point!\n",
    "i = 0\n",
    "for batch in data_aug_gen.flow(x, batch_size=1):\n",
    "    plt.figure(i) # every image has got its own figure\n",
    "    imgplot = plt.imshow(image.array_to_img(batch[0]))\n",
    "    i += 1\n",
    "    if i % 4 == 0:\n",
    "        break\n",
    "\n",
    "# finally show all transformations\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If a new network using data augmentation is trained, that network will never see twice the same input. \n",
    "However, the inputs that it sees are still heavily intercorrelated (compare the transformed images above), since they come from a small number of original images.  \n",
    "It is nearly impossible to produce completely new information, only remixing existing information is an alternative. \n",
    "As such, this might not be quite enough to completely get rid of overfitting. \n",
    "To further fight overfitting, adding a Dropout layer right before the densely-connected classifier seems to be the right choice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import losses\n",
    "from tensorflow.keras import metrics\n",
    "\n",
    "model = models.Sequential()\n",
    "\n",
    "model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(150, 150, 3)))\n",
    "model.add(layers.MaxPooling2D((2, 2)))\n",
    "\n",
    "model.add(layers.Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(layers.MaxPooling2D((2, 2)))\n",
    "\n",
    "model.add(layers.Conv2D(128, (3, 3), activation='relu'))\n",
    "model.add(layers.MaxPooling2D((2, 2)))\n",
    "\n",
    "model.add(layers.Conv2D(128, (3, 3), activation='relu'))\n",
    "model.add(layers.MaxPooling2D((2, 2)))\n",
    "\n",
    "model.add(layers.Flatten())\n",
    "model.add(layers.Dropout(0.5)) # NOTE: dropout is used as well before the FC-layer starts\n",
    "model.add(layers.Dense(512, activation='relu'))\n",
    "model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer=optimizers.RMSprop(lr=1e-4),\n",
    "              metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train the network using data augmentation and dropout:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_datagen = ImageDataGenerator(\n",
    "    rescale=1./255,\n",
    "    rotation_range=40,\n",
    "    width_shift_range=0.2,\n",
    "    height_shift_range=0.2,\n",
    "    shear_range=0.2,\n",
    "    zoom_range=0.2,\n",
    "    horizontal_flip=True,)\n",
    "\n",
    "# Note that the validation data should not be augmented!\n",
    "test_datagen = ImageDataGenerator(rescale=1./255)\n",
    "\n",
    "train_generator = train_datagen.flow_from_directory(\n",
    "        # This is the target directory\n",
    "        train_dir,\n",
    "        # All images will be resized to 150x150\n",
    "        target_size=(150, 150),\n",
    "        batch_size=20,\n",
    "        # Since binary_crossentropy loss is used, binary labels are needed\n",
    "        class_mode='binary')\n",
    "\n",
    "validation_generator = test_datagen.flow_from_directory(\n",
    "        validation_dir,\n",
    "        target_size=(150, 150),\n",
    "        batch_size=20,\n",
    "        class_mode='binary')\n",
    "\n",
    "print(\"train_dir=\",train_dir)\n",
    "print(\"validation_dir=\",validation_dir)\n",
    "\n",
    "history = model.fit(\n",
    "      train_generator,\n",
    "      steps_per_epoch=100, \n",
    "      epochs=100,\n",
    "      validation_data=validation_generator,\n",
    "      validation_steps=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for data_batch, labels_batch in train_generator:\n",
    "    print('data batch shape:', data_batch.shape)\n",
    "    print('labels batch shape:', labels_batch.shape)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's save the model again. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save('cats_dogs_classification_CNN_with_data_augmentation.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history_dict = history.history\n",
    "history_dict.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the results again using the history object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(len(acc))\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.ylim((0.5,0.9))\n",
    "plt.title('Training and validation accuracy using data augmentation')\n",
    "plt.legend()\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.ylim((0.2,0.7))\n",
    "plt.title('Training and validation loss using data augmentation')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thanks to data augmentation and dropout, overfitting is reduced almost completely. \n",
    "The training curves are rather closely tracking the validation curves. \n",
    "With these methods it was possible to reach a much higer accuracy and this a high relative improvement compared to the non-regularized model.\n",
    "\n",
    "By leveraging regularization techniques even further and by tuning the network's hyperparameters (such as the number of filters per convolution layer, or the number of layers in the network), it will be possible to get an even better accuracy, likely up to 86-87%. \n",
    "\n",
    "For this purpose, you may consider using \"weights and biases\" (https://wandb.ai/site/sweeps), which allows for automatic, scalable, customizable hyperparameter tuning in a few lines of python code. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <font color=\"#C70039\">Include the result table of your test plan here (see task list)</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}