02_Segmentation_TF2_train.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Content**: [Olivier Bernard](https://www.creatis.insa-lyon.fr/~bernard/), [Thomas Grenier](https://www.creatis.insa-lyon.fr/~grenier/) from the [CREATIS](https://www.creatis.insa-lyon.fr/site7/en) Laboratory, France<br>\n",
    "\n",
    "\n",
    "***\n",
    "\n",
    "Hands-on: Echocardiographic image segmentation with deep learning (part 2)\n",
    "===\n",
    "\n",
    "# <span style=\"color:brown\"> **Preamble**\n",
    "\n",
    "The second part of this hands-on will allow you to study in depth the structure and performance of the popular deep learning U-Net architecture, in the context of echocardiographic image segmentation. You will still use the open access CAMUS dataset. In particular, you will use only a fraction of this dataset, *i.e.* apical four-chamber views (4CH) at End-Diastole (ED) and End-Systole (ES) for 500 patients.\n",
    "    \n",
    "# <span style=\"color:brown\"> **Objectives**\n",
    "\n",
    "* Study how to code a deep learning segmentation method with the tensorflow python library,\n",
    "* Compare the performance of a U-Net model based on the choice of key parameters.\n",
    "    \n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:brown\"> **Reminder on the U-Net architecture**\n",
    "\n",
    "U-Net is based on a two-stage convolutional network architecture. The first part, known as the encoder, is similar to conventional CNNs and extracts high-level information. The second part is the decoder, which uses information from the encoder and applies a set of convolutions and upsampling operations to gradually transform feature maps with the purpose of reconstructing segmentation maps at the resolution of the imput image. U-Net architecture also integrates skip connections between the encoder and decoder parts with the goal of retrieving details that were potentially lost during the downsampling while also stabilizing the learning procedure. An illustration of the network architecture is given below (please refer to Chapter 4 of the book for further explanation, in particular Figure 4.4).\n",
    "    \n",
    "<br>\n",
    "        \n",
    "<img src=\"figures/illustration_UNet_US.png\" alt=\"UNet\" style=\"width: 85%;\"/>\n",
    "\n",
    "<br>\n",
    "\n",
    "The U-Net architecture can be defined through the following main parameters:\n",
    "- the number of feature maps at the first level\n",
    "- the number of levels\n",
    "- the use of the batch normalizations at each level\n",
    "- the type of activation functions\n",
    "- the use of dropout operations\n",
    "- the use of data augmentation\n",
    "\n",
    "The performance of deep learning model also depends on the optimization conditions that were used during the learning process, the main ones being:\n",
    "- the optimization algorithm (*ADAM* and *RMSprop* being among the most popular)\n",
    "- the learning rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Questions\n",
    "\n",
    "- ### What were the main parameters of the U-Net architecture you used in the previous notebook ? (in terms of number of feature maps at the first level, number of levels, batch normalization, dropout)\n",
    "- ### How many parameters does this network have ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">TODO: Your answer here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "***\n",
    "\n",
    "# <span style=\"color:brown\"> **Before starting: system setting**\n",
    " \n",
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to download pre-trained models that you will use during this session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's download pre-trained AI models\n",
    "!rm -r trained_model\n",
    "!wget https://www.creatis.insa-lyon.fr/~bernard/jupyter/trained_model.zip\n",
    "!unzip -q trained_model.zip\n",
    "!rm trained_model.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "***\n",
    "\n",
    "# <span style=\"color:brown\"> **Before starting: system setting**\n",
    " \n",
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to import all the packages that you will need during this session.\n",
    "\n",
    "- [tensorflow](https://keras.io/) is a Python library for fast numerical computing created and released by Google that can be used to create deep learning models directly or by using wrapper libraries that simplify the process built on top of TensorFlow such as keras.\n",
    "\n",
    "- [numpy](www.numpy.org) is the fundamental package for scientific computing with Python.\n",
    "- [matplotlib](http://matplotlib.org) is a famous library to plot graphs in Python.\n",
    "- [medpy](http://loli.github.io/medpy/) is a library dedicated to medical imaging in Python.\n",
    "- [os](https://docs.python.org/3/library/os.html) is a python module that provides functions for creating and removing a directory (folder), fetching its contents, changing and identifying the current directory, etc...\n",
    "- [sys](https://docs.python.org/3/library/sys.html) is a python module that provides various functions and variables that are used to manipulate different parts of the Python runtime environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib import cm\n",
    "from matplotlib.colors import ListedColormap\n",
    "import sys\n",
    "import os\n",
    "!pip install medpy\n",
    "!pip install tqdm\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams[\"figure.figsize\"] = (15,15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to import our own modules that you will need during this session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from keras_unet.losses import dice_loss\n",
    "# from keras_unet.metrics import dice_coef, iou, iou_thresholded\n",
    "from keras_unet.visualization import plot_overlay_segmentation\n",
    "\n",
    "# Add our own python modules stored in the folders present at the root level of this project\n",
    "if \"../\" in sys.path: \n",
    "    print(sys.path)\n",
    "else: \n",
    "    sys.path.append(\"../\")\n",
    "    print(sys.path)\n",
    "\n",
    "if \"model\" in locals(): \n",
    "    print(\"deleting model\")\n",
    "    del model    \n",
    "    \n",
    "# Select the CPU device to run on\n",
    "num_CPU = 1\n",
    "num_cores = 4\n",
    "tf.config.threading.set_intra_op_parallelism_threads(num_CPU)\n",
    "tf.config.threading.set_inter_op_parallelism_threads(num_cores)\n",
    "os.environ[\"TF_CPP_MIN_LOG_LEVEL\"]=\"3\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "***\n",
    "\n",
    "# <span style=\"color:brown\"> **1) Load and prepare dataset**\n",
    "    \n",
    "As for the previous part of the hands-on, you will use a the CAMUS dataset restricted to apical four-chamber views at End-Diastole and End-Systole time instants from 500 patients. This dataset was divided into three folds:\n",
    "\n",
    "- ***dataset/segmentation/train*** representing 80% of the dataset. This subset will be used to train the network by updating its parameters to get the best results on the corresponding data.\n",
    "- ***dataset/segmentation/valid*** representating 10% of the dataset. This subset will be used to select the network parameters which produce the best results on the training dataset.\n",
    "- ***dataset/segmentation/test*** representing 10% of the dataset. This subset will be used to assess the scores that can reach the trained network on unseen data.    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:brown\"> 1.1) Load data\n",
    "    \n",
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to load the train / validation / test datasets.\n",
    "    \n",
    "**Remark:** The training/validation/testing sets contain data from 80/10/10 patients, but you'll see that it consists of 100 samples: i.e. ED and ES time instants for each patient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras_unet.utils import load_CAMUS_dataset\n",
    "\n",
    "# Parameters\n",
    "IMG_SIZE = 128  # All the images will be resized to IMG_SIZE x IMG_SIZE to speed up the process. \n",
    "\n",
    "dataset_train_path = \"dataset/segmentation/train/\"\n",
    "dataset_valid_path = \"dataset/segmentation/valid/\"\n",
    "dataset_test_path = \"dataset/segmentation/test/\"\n",
    "\n",
    "# Load data\n",
    "[X_train, y_train] = load_CAMUS_dataset(dataset_train_path, IMG_SIZE)\n",
    "[X_valid, y_valid] = load_CAMUS_dataset(dataset_valid_path, IMG_SIZE)\n",
    "[X_test, y_test] = load_CAMUS_dataset(dataset_test_path, IMG_SIZE)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to see the size of the loaded images and the corresponding reference masks stored in the `X_train`, `X_valid`, `X_test` and `y_train`, `y_valid`, `y_test` variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display the matrix size and pixel type for the return variables \n",
    "print(\" Dimensions of X_train : \", X_train.shape)\n",
    "print(\" Dimensions of y_train : \", y_train.shape)\n",
    "print(\" \")\n",
    "print(\" Dimensions of X_valid : \", X_valid.shape)\n",
    "print(\" Dimensions of y_valid : \", y_valid.shape)\n",
    "print(\" \")\n",
    "print(\" Dimensions of X_test : \", X_test.shape)\n",
    "print(\" Dimensions of y_test : \", y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Questions: </span>\n",
    "\n",
    "- ### How many images are there in the train / valid and test datasets ?\n",
    "- ### What does each dimension correspond to ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">TODO: Your answer here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:brown\"> 1.2) Plot images + masks + overlay (mask over original)\n",
    "\n",
    "Let's visualize the dataset for different samples by executing the following cell. Feel free re-run the cell multiple times to see other images.\n",
    "\n",
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to display the images, manual annotations and an overlay for easy viewing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parameter to play with\n",
    "dataset_type = 2    # select 0, 1 or 2 to see images from the train, valid or test dataset, respectively\n",
    "nb_imgs = 2         # number of images that will be displayed\n",
    "\n",
    "# Select random values\n",
    "start_ind = (np.random.randint(0, X_test.shape[0]-nb_imgs, size=1))[0]\n",
    "end_ind = start_ind+nb_imgs\n",
    "\n",
    "# Display the random images with the corresponding reference mask\n",
    "if dataset_type==0:\n",
    "    plot_overlay_segmentation(X_train[start_ind:end_ind], y_train[start_ind:end_ind])\n",
    "if dataset_type==1:\n",
    "    plot_overlay_segmentation(X_valid[start_ind:end_ind], y_valid[start_ind:end_ind])\n",
    "if dataset_type==2:\n",
    "    plot_overlay_segmentation(X_test[start_ind:end_ind], y_test[start_ind:end_ind])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "# <span style=\"color:brown\"> **2) Prepare the U-Net architecture**\n",
    "    \n",
    "## <span style=\"color:brown\"> 2.1) Define important parameters\n",
    "    \n",
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to select the values of the main parameters that define a U-Net architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Network architecture related values \n",
    "NBFILTERS_L1_UNET = 32  # Number of feature maps for the first level\n",
    "NBLAYERS_UNET = 4       # Number of levels\n",
    "DROPOUT_RATE = 0.1      # Dropout action used at each level (value between 0.0 and 1.0, 0.0 meaning no dropout)\n",
    "BATCHNORM_ON = True     # Use of batch normalisation after each convolutional layer (value = True or False)\n",
    "\n",
    "# Training parameters\n",
    "NBEPOCHS = 100          # Number of epochs\n",
    "BATCH_SIZE = 32         # Number of samples in each batch (usually the value is a multiple of 2)\n",
    "NBSTEPS_PER_EPOCH = 50  # Number of batches per epoch (1  to ...) (used for data augmentation)\n",
    "NBPATIENCE_EPOCHS = 50  # Number of epoch after a minimum detection before stopping (early stop)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:brown\"> 2.2) Prepare a generator for data augmentation\n",
    "    \n",
    "During the learning phase of a deep learning model, it is generally recommended to use a data augmentation strategy to improve the generalization capability of the algorithm. Data augmentation consists of applying some pre-defined transformation operations (*e.g.* rotation, translation or pixel intensity clipping, etc...) on the input data during the filling of the batches. By doing so, the algorithm will never see the exact same input images, which will force it to be less sensitive to the changes applied.\n",
    "    \n",
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to initialize a generator (tensorflow object) dedicated to data augmentation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras_unet.utils import get_augmented\n",
    "\n",
    "# Creation of a tensorflow generator object to manage data augmentation.\n",
    "# The different data augmentation operations are defined as input parameter of the function\n",
    "train_gen = get_augmented(\n",
    "    X_train, y_train, batch_size=BATCH_SIZE,  \n",
    "    data_gen_args = dict(\n",
    "        rotation_range=5.,\n",
    "        width_shift_range=0.05,\n",
    "        height_shift_range=0.05,\n",
    "        shear_range=10,\n",
    "        zoom_range=0.2,\n",
    "        horizontal_flip=False,\n",
    "        vertical_flip=False,\n",
    "        fill_mode='constant',\n",
    "        cval = 0\n",
    "    ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Questions: </span>\n",
    "\n",
    "- ### How many transformations are applied on each input data thanks to the data augmentation strategy ?\n",
    "- ### What are the different transformations ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">TODO: Your answer here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to visualize the impact of the data augmentation strategy on the input images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute data augmentation on input images\n",
    "XX, yy = next(train_gen)\n",
    "\n",
    "# Display the corresponding transformed images \n",
    "plot_overlay_segmentation(XX[0:3,:,:,:], yy[0:3,:,:,:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "# <span style=\"color:brown\"> **3) Network and training**\n",
    "    \n",
    "## <span style=\"color:brown\"> 3.1) Initialize the network    \n",
    "    \n",
    "You are now ready to create a U-Net network from the parameters defined earlier.     \n",
    "    \n",
    "    \n",
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to create a U-Net model defined through the tensorflow library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras_unet.models import custom_unet\n",
    "from tensorflow.keras import models\n",
    "from tensorflow.keras import regularizers\n",
    "\n",
    "# Define the size of the input data\n",
    "input_shape = (IMG_SIZE, IMG_SIZE, 1)\n",
    "\n",
    "# This instruction is needed to avoid any confusion with the instantiation of multiple models\n",
    "if 'model' in locals(): \n",
    "    print(\"deleting existing model\")\n",
    "    del model\n",
    "    \n",
    "# Define a U-Net model from the tensorflow library.\n",
    "model = custom_unet(\n",
    "    input_shape,\n",
    "    use_batch_norm=BATCHNORM_ON,  \n",
    "    num_classes=y_train[0].shape[-1],\n",
    "    filters=NBFILTERS_L1_UNET, \n",
    "    dropout=DROPOUT_RATE,\n",
    "    num_layers=NBLAYERS_UNET,\n",
    "    output_activation='softmax',\n",
    "    kernel_regularizer=None      # You can apply some regularization on the model parameters: .l1(0.001)\n",
    "    )                            # used kernel_regularizer=0.001 instead of the \"None\" value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "**Remark**: We coded our own function named `custom_unet` to build the U-Net model from the tensorflow library. For those who are interested in how to define a deep learning model from tensorflow, feel free to analyze the code of this function defined in the file `custom_unet.py` present in the folder named `keras_unet/models`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below and analyze the corresponding architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# display the network architecture\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Questions: </span>\n",
    "\n",
    "- ### Try to find the parameters you have previously selected (*i.e.* number of feature maps of the first layer, number of levels, use of dropout, use of batch normalization)\n",
    "- ### How many parameters need to be trained ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">TODO: Your answer here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Instruction: </span> Change some parameters of the model (*e.g.* `NBFILTERS_L1_UNET=16` instead of `32`) and re-execute the corresponding cell and the following ones until you reach this cell. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Questions: </span>\n",
    "\n",
    "- ### Do you observe any changes in the above description of the architecture ?\n",
    "- ### How many parameters need to be trained now ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">TODO: Your answer here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## <span style=\"color:brown\"> 3.2) Create callbacks for interactions during training\n",
    "\n",
    "It is possible with tensorflow to define and call some functions (named callback functions) to interact with the learning process. For instance, the callback functions defined above allow the following actions:\n",
    "- `ModelCheckpoint`: This function indicates to store at the end of the learning process the network parameters that obtained the best score on some metrics passed as input (in our case on the Dice coefficient computed on the validation set) \n",
    "- `EarlyStopping`: This function indicates that the learning process should stop if some conditions passed as input are reached (in our case if there is no improvement of the Dice score computed from the validation set after `NBPATIENCE_EPOCHS` epochs)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to create the callback functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping\n",
    "\n",
    "# Create the name for the U-Net model that will be saved\n",
    "model_path = 'trained_model/'\n",
    "model_name = 'trained_Unet_f' + str(NBFILTERS_L1_UNET) + '_b' + str(BATCH_SIZE) + '_l'+ str(NBLAYERS_UNET) + '_do' + str(DROPOUT_RATE) +'_Std'\n",
    "if BATCHNORM_ON == True:\n",
    "    model_name = model_name + '_BN'\n",
    "model_filename = model_path + model_name + '_input' +'.h5'\n",
    "print(\" -> model_study : \", model_name)\n",
    "print(\" -> model_filename : \", model_filename)\n",
    "\n",
    "# Create first callback function\n",
    "callback_checkpoint = ModelCheckpoint(\n",
    "    model_filename, \n",
    "    verbose=1, \n",
    "    monitor='val_dice_coef', #'val_loss'\n",
    "    mode ='max', # use 'min' or 'auto' if val_loss\n",
    "    save_best_only=True,\n",
    ")\n",
    "\n",
    "\n",
    "# Create second callback function\n",
    "callback_earlystopping = EarlyStopping(\n",
    "    monitor='val_dice_coef', #'val_loss'\n",
    "    mode ='max', # use 'min' or 'auto' if val_loss\n",
    "    patience=NBPATIENCE_EPOCHS,\n",
    "    restore_best_weights=True  # at the end of fitting, restore best model weights \n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:brown\"> 3.3) Compile the model\n",
    "\n",
    "In tensorflow, the `compile` function allows the definition of the optimization strategy applied during the learning process of the network model.\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to define the optimization strategy that will be used during the learning phase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras_unet.losses import dice_loss\n",
    "from keras_unet.metrics import dice_coef\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "\n",
    "model.compile(  \n",
    "    loss=dice_loss,\n",
    "    optimizer=Adam(learning_rate=0.001),\n",
    "    metrics=[dice_coef]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## <span style=\"color:brown\"> 3.4) It's time to train your first model !\n",
    "\n",
    "It's now time to start training your first model. By default, the training will be performed on the CPU card of your computer. As you will see, it will take around 10 minutes to perform only 3 epochs. This is the reason why the training of deep learning models is generally performed on GPU cards, which correspond to hardware systems dedicated to intensive computing.  \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to train your U-Net model. If this step takes too long, you can stop it at any time by clicking the *Interrupt the kernel* button at the top of this page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history = model.fit(\n",
    "    train_gen,\n",
    "    steps_per_epoch=NBSTEPS_PER_EPOCH,\n",
    "    batch_size=BATCH_SIZE,\n",
    "    epochs=3,\n",
    "    validation_data=(X_valid, y_valid),\n",
    "    shuffle=True,\n",
    "    verbose=1,\n",
    "    callbacks=[ callback_earlystopping, callback_checkpoint]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:brown\"> 3.5) Plot training and validation history\n",
    "    \n",
    "After the learning of a network, it is interesting to analyse the behavior of the training and validation loss curves to see if the training went well (*i.e.* decrease of the curves since the goal of the training is to minimize the loss function across the epochs).  \n",
    "\n",
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to display the loss and metric curves corresponding to the learning process you just executed.    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras_unet.visualization import plot_segm_history\n",
    "\n",
    "plot_segm_history(history,  metrics=['dice_coef', 'val_dice_coef'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Questions: </span>\n",
    "\n",
    "- ### What information can you retrieve from these curves ? \n",
    "- ### Why is it important to use a validation set ? \n",
    "- ### Why don't we use directly the testing dataset instead ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">TODO: Your answer here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:brown\"> 3.6) Select which model to play with\n",
    "\n",
    "As you can see, the number of epochs is not sufficient to achieve convergence. This means that the model was still learning at the end of the training process. Since you are working on the CPU card of your computer by default, restarting a training with more epochs would take too much time. So we have run 8 different trainings for you. For each of these training runs we learned a model over 400 epochs and saved the best learned parameters and the associated loss and metric curves. We have used an early stopping callback function, so some training ended before the 100 epochs. ***The corresponding data were stored on a remote server. To be able to use these models, you have to first download the archive [(download link)](https://www.creatis.insa-lyon.fr/~bernard/handsonsegmentation) and extract it at the root of this notebook.*** You will afterwards be able to load each of these models and analyze their performance. This will allow you to analyze the influence of the main parameters to be defined when learning a U-Net model. The different models that have been learned are the following:\n",
    "- ***model_1***: This is the reference model that you have used during the first hands on session. The default settings provided in the cell in section 3-1 were used.\n",
    "- ***model_2***: This model uses the same parameters as *model_1* except the number of feature maps at the first level. This parameter was set to: *NBFILTERS_L1_UNET = 16*.\n",
    "- ***model_3***: This model uses the same parameters as *model_1* except the number of level. This parameter was set to: *NBLAYERS_UNET = 3*.\n",
    "- ***model_4***: This model uses the same parameters as *model_1* except the number of level. This parameter was set to: *NBLAYERS_UNET = 5*.\n",
    "- ***model_5***: This model uses the same parameters as *model_1* except that there was no dropout (DROPOUT_RATE=0.0).\n",
    "- ***model_6***: This model uses the same parameters as *model_1* except that there was no data augmentation.\n",
    "- ***model_7***: This model uses the same parameters as *model_1* except that the learning rate used during the optimisation process was set to *0.1* instead of *0.001*.\n",
    "- ***model_8***: This model uses the same parameters as *model_1* except that the network optimization was performed using the RMSprop algorithm instead of the classical ADAM's algorithm.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to select which model you want ot work with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras_unet.utils import select_model\n",
    "\n",
    "# Parameters: play with this value (between 1 and 8) to select the associated model\n",
    "model_type = 1\n",
    "\n",
    "# Get the number of class to segment\n",
    "num_classes = y_train[0].shape[-1]\n",
    "\n",
    "# Select which model you want to work with. \n",
    "# The input argument of the function corresponds to model number of interest (*i.e.* a value between 1 and 6)\n",
    "model = select_model(model_type, num_classes, IMG_SIZE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:brown\"> 3.7) Visual assessment of the quality of the predictions\n",
    "    \n",
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to apply the U-Net model on the full test dataset, predict the different segmentation masks and compute the metrics defined above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras_unet.evaluation import  evaluate_segmentation, evaluate_set\n",
    "import pandas as pd\n",
    "from IPython.display import display, HTML \n",
    "\n",
    "# Predict segmentations on the full test dataset\n",
    "y_pred = model.predict(X_test, batch_size=1, verbose=1)\n",
    "\n",
    "# Compute score on the full test dataset\n",
    "dice_all, hausdorff_all, assd_all, valid_all = evaluate_set(y_test, y_pred)\n",
    "\n",
    "# Put the results into a matrix and graft it into a pandas data frame (object from the panda library)\n",
    "overall_results = np.column_stack((dice_all, hausdorff_all, assd_all))\n",
    "overall_results_df = pd.DataFrame(data=overall_results, index = [\"All\", \"Right Ventricle\", \"Myocardium\", \"Left Ventricle\"], \n",
    "                                  columns=[\"Dice\", \"Hausdorff (px)\", \"ASSD (px)\"]) \n",
    "\n",
    "# Display the data as HTML tables and graphs\n",
    "display(HTML(overall_results_df.to_html(float_format=lambda x: '%.2f' % x)))\n",
    "overall_results_df.plot(kind='bar', figsize=(10,6)).legend() #bbox_to_anchor=(1.6,0.9))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to display some prediction results. Don't hesitate to re-run the cell several times to see more cases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras_unet.visualization import plot_compare_segmentation\n",
    "\n",
    "# Select random values\n",
    "nb_imgs = 3\n",
    "start_ind = (np.random.randint(0, X_test.shape[0]-nb_imgs, size=1))[0]\n",
    "end_ind = start_ind+nb_imgs\n",
    "\n",
    "# Display the random images with the corresponding reference and predicted mask\n",
    "plot_compare_segmentation(X_test[start_ind:end_ind], y_test[start_ind:end_ind], y_pred[start_ind:end_ind], \" \", spacing=(1,1), step=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Questions\n",
    "\n",
    "- ### Compare the performance of the different models\n",
    "- ### Are the results relatively stable with respect to parameter changes ?\n",
    "- ### Which parameter changes have particularly degraded the results ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">TODO: Your answer here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}