01_Segmentation_TF2_test.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Content**: [Olivier Bernard](https://www.creatis.insa-lyon.fr/~bernard/), [Thomas Grenier](https://www.creatis.insa-lyon.fr/~grenier/) from the [CREATIS](https://www.creatis.insa-lyon.fr/site7/en) Laboratory, France<br>\n",
    "\n",
    "\n",
    "\n",
    "***\n",
    "\n",
    "Hands-on: Echocardiographic image segmentation with deep learning (part 1)\n",
    "===\n",
    "\n",
    "# <span style=\"color:brown\"> **Preamble**\n",
    "\n",
    "In this hands-on, you will study in depth and apply the very popular deep learning U-Net architecture to perform the segmentation of 2D echocardiographic images. This is a 30-years old problem which has taken an important step forward thanks to deep learning techniques. \n",
    "    \n",
    "The material is based on a study recently published in *IEEE Transactions on Medical Imaging* (DOI: 10.1109/TMI.2019.2900516). To provide answers to the problem of echocardiographic image segmentation by deep learning methods, an open access dataset, named CAMUS, composed of 500 patients acquired in both apical two and four chamber views was set up. If you want to go further on this topic, don't hesitate to go on the corresponding website, you will find additional information: https://www.creatis.insa-lyon.fr/Challenge/camus/\n",
    "    \n",
    "# <span style=\"color:brown\"> **Objectives**\n",
    "\n",
    "* Consolidate the knowledge you've gained on deep learning\n",
    "* Use a pre-trained model based on the classical U-Net architecture to segment echocardiographic images and assess its performance (part 1 of this hands-on),\n",
    "* Train your own U-Net (part 2 of this hands-on).\n",
    "    \n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:brown\"> **CAMUS dataset**\n",
    "\n",
    "The data for this hands-on comes from the open access CAMUS dataset. In particular, you will focus on the segmentation of three adjacent cardiac structures: the left ventricle (LV), the myocardium (MYO), and the left atrium (LA). An illustration of the reference annotation of a typical case is given below:\n",
    "    \n",
    "<br>\n",
    "        \n",
    "<img src=\"figures/illustration_annotations_US.png\" alt=\"UNet\" style=\"width: 70%;\"/>\n",
    "    \n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "# <span style=\"color:brown\"> **Segmentation network under investigation: the U-Net architecture**\n",
    "\n",
    "U-Net is one of the most famous deep learning architectures used for image segmentation in medical imaging. It has been created by Olaf Ronneberger, Philipp Fischer and Thomas Brox in 2015 and published in the paper \"UNet: Convolutional Networks for Biomedical Image Segmentation\" (DOI: 10.1007/978-3-319-24574-4_28). An illustration of the network architecture is given below.\n",
    "    \n",
    "<br>\n",
    "        \n",
    "<img src=\"figures/illustration_UNet_US.png\" alt=\"UNet\" style=\"width: 70%;\"/>    \n",
    "    \n",
    "This network is based on a two-stage convolutional neural network (CNN) architecture. The first part, known as the encoder, is similar to conventional CNNs and extracts high-level information. The second part is the decoder, which uses information from the encoder and applies a set of convolutions and upsampling operations to gradually transform feature maps with the purpose of reconstructing segmentation maps at the resolution of the input image. U-Net architecture also integrates skip connections between the encoder and decoder parts with the goal of retrieving details that were potentially lost during the downsampling while also stabilizing the learning procedure.\n",
    "    \n",
    "**Remark:** The number of filters on the last convolution layer (in red, 4 filters) is directly linked to the amount of structures to segment (4: LV, MYO, LA, and background).\n",
    "    \n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:brown\"> **Before starting: system setting**\n",
    " \n",
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to download the CAMUS dataset and a pre-trained model that you will use during this session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's download the CAMUS dataset\n",
    "!rm -r dataset\n",
    "!wget https://www.creatis.insa-lyon.fr/~bernard/jupyter/dataset.zip\n",
    "!unzip -q dataset.zip\n",
    "!rm dataset.zip\n",
    "\n",
    "# and a pre-trained AI model\n",
    "!rm -r backup_model\n",
    "!wget https://www.creatis.insa-lyon.fr/~bernard/jupyter/backup_model.zip\n",
    "!unzip -q backup_model.zip\n",
    "!rm backup_model.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to import all the packages that you will need during this session.\n",
    "\n",
    "- [tensorflow](https://keras.io/) is a Python library for fast numerical computing created and released by Google that can be used to create deep learning models directly or by using wrapper libraries that simplify the process built on top of TensorFlow such as keras.\n",
    "\n",
    "- [numpy](www.numpy.org) is the fundamental package for scientific computing with Python.\n",
    "- [matplotlib](http://matplotlib.org) is a famous library to plot graphs in Python.\n",
    "- [medpy](http://loli.github.io/medpy/) is a library dedicated to medical imaging in Python.\n",
    "- [cv2](https://pypi.org/project/opencv-python/) is a library bindings designed to solve computer vision problems in Python.\n",
    "- [os](https://docs.python.org/3/library/os.html) is a Python module that provides functions for creating and removing a directory (folder), fetching its contents, changing and identifying the current directory, etc...\n",
    "- [sys](https://docs.python.org/3/library/sys.html) is a Python module that provides various functions and variables that are used to manipulate different parts of the Python runtime environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import cv2\n",
    "!pip install medpy\n",
    "!pip install tqdm\n",
    "\n",
    "import os\n",
    "import sys\n",
    "from datetime import datetime\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams[\"figure.figsize\"] = (15,15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "In order to make the hands-on session more fluent, we have coded our own Python modules. Some of these modules contain functions that call others tensorflow functions. So, for those who are not comfortable with python programming, we invite you to focus on running the cells in the notebook and analyzing the results generated. For the others, we invite you to regularly take a look at the functions present in our modules (loadable from the file explorer) in order to analyze how we developed the different functionalities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to import our own modules that you will need during this session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-11T09:30:08.991851Z",
     "start_time": "2019-06-11T09:30:08.985860Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from keras_unet.utils import load_CAMUS_dataset\n",
    "from keras_unet.visualization import plot_overlay_segmentation, plot_compare_segmentation\n",
    "\n",
    "# Add our own python modules stored in the folders present at the root level of this project\n",
    "if \"../\" in sys.path: \n",
    "    print(sys.path)\n",
    "else: \n",
    "    sys.path.append(\"../\")\n",
    "    print(sys.path)\n",
    "\n",
    "if \"model\" in locals(): \n",
    "    print(\"deleting model\")\n",
    "    del model    \n",
    "    \n",
    "# Select the CPU device to run on\n",
    "num_CPU = 1\n",
    "num_cores = 4\n",
    "tf.config.threading.set_intra_op_parallelism_threads(num_CPU)\n",
    "tf.config.threading.set_inter_op_parallelism_threads(num_cores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "# <span style=\"color:brown\"> **1) Load and prepare dataset**\n",
    "    \n",
    "As presented in the introduction of this hands-on session, you will use a the CAMUS dataset restricted to apical four-chamber views at End-Diastole and End-Systole time instants from 500 patients. This dataset was divided into three folds:\n",
    "\n",
    "- ***dataset/segmentation/train*** representing 80% of the dataset. This subset will be used to train the network by updating its parameters to get the best results on the corresponding data.\n",
    "- ***dataset/segmentation/valid*** representating 10% of the dataset. This subset will be used to select the network parameters which produce the best results on the training dataset.\n",
    "- ***dataset/segmentation/test*** representing 10% of the dataset. This subset will be used to assess the scores that can reach the trained network on unseen data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## <span style=\"color:brown\"> 1.1) Load data\n",
    "    \n",
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to load the test dataset and to get more familiar with it.\n",
    "    \n",
    "**Remark:** The testing set contains data from 50 patients, but you'll see that it consists of 100 samples: i.e. ED and ES time instants for each patient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-11T09:30:09.006881Z",
     "start_time": "2019-06-11T09:30:08.994853Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Parameters\n",
    "IMG_SIZE = 128  # All the images will be resized to IMG_SIZE x IMG_SIZE to speed up the process.                \n",
    "dataset_test_path = \"dataset/segmentation/test/\"\n",
    "\n",
    "# Load data\n",
    "[X_test, y_test] = load_CAMUS_dataset(dataset_test_path, IMG_SIZE)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Exercise: </span> The following lines allow you to see the size of the loaded images and reference masks stored in the `X_test` and `y_test` variables. Execute it and try to understand what each dimension corresponds to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Display the matrix size and pixel type for the return variables X_test and y_test \n",
    "print(\" Dimensions of X_test : \", X_test.shape)\n",
    "print(\" Dimensions of y_test : \", y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "**Remark**: The last dimension of `y_test` is 4. Indeed, in tensorflow implementation, it is necessary to provide one binary channel per class to segment. Since we want to segment three different structures in this project (left ventricle, myocardium and left atrium), we set 4 channels, one for each structure plus one for the background. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Exercice: </span> Execute the following code and try to better assess this notion of channel by playing with the *ind_image* parameter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ind_image = 2    # indice of the selected image from the test dataset\n",
    "\n",
    "# Display selected image along with the corresponding manual annotation in overlay\n",
    "plot_overlay_segmentation(X_test[ind_image:ind_image+1], y_test[ind_image:ind_image+1])\n",
    "\n",
    "# Display the different channel of y_test\n",
    "f, axarr = plt.subplots(1, 4, figsize=(15,3))\n",
    "for i in range(4):\n",
    "    axarr[i].imshow(y_test[ind_image,:,:,i], 'gray', interpolation='none')\n",
    "    axarr[i].set_title(\"y_test: channel[{}]\".format(i))\n",
    "    axarr[i].axis(\"off\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:brown\"> 1.2) Plot images + masks + overlay (mask over original)\n",
    "\n",
    "Let's visualize the dataset for different samples by executing the following cell. Feel free to re-run the cell multiple times to see other images. For those interested with Python programming, don't hesitate to analyze the code of the `plot_overlay_segmentation` function defined in the file `visualization.py` present in the folder named `keras_unet`.\n",
    "\n",
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to display the images, manual annotations and an overlay for easy viewing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Select random values\n",
    "nb_imgs = 4\n",
    "start_ind = (np.random.randint(0, X_test.shape[0]-nb_imgs, size=1))[0]\n",
    "end_ind = start_ind+nb_imgs\n",
    "\n",
    "# Display the random images with the corresponding reference mask\n",
    "plot_overlay_segmentation(X_test[start_ind:end_ind], y_test[start_ind:end_ind])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "# <span style=\"color:brown\"> **2) Load the network and its weights**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We already trained a U-Net architecture to perform the segmentation of cardiac structures on ultrasound images thanks to the CAMUS dataset (you will do this task in the next hands-on session, be patient :) ). The corresponding network parameters and structure have been saved in the file `backup_model/Unet_f32_b32_l4_do0.1_Std_BN_input128.h5`. In this part of the session, you will:\n",
    "\n",
    "- Load the pre-trained U-Net model.\n",
    "- Process the test dataset with this U-Net network.\n",
    "- Assess the quality of the obtained results thanks to the corresponding references.\n",
    "\n",
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to load the pre-trained U-Net model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras_unet.metrics import iou, iou_thresholded\n",
    "from keras_unet.losses import dice_loss, dice_coef, adaptive_loss\n",
    "from tensorflow.keras import models\n",
    "\n",
    "# Filename of the pre-trained U-Net model\n",
    "model_filename = \"./backup_model/Unet_f32_b32_l4_do0.1_Std_BN_input128.h5\"\n",
    "\n",
    "tf.keras.backend.clear_session()\n",
    "\n",
    "# Load the network with its custom functions\n",
    "loaded_model = models.load_model(model_filename, custom_objects={'dice_coef': dice_coef, 'adaptive_loss': adaptive_loss, 'dice_loss': dice_loss})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "By running the next cell, you will see a summary of the U-Net architecture that was used during the training and whose best parameters were saved.\n",
    "\n",
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below and try to analyze the corresponding architecture.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# display the network\n",
    "loaded_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-11T09:30:09.224878Z",
     "start_time": "2019-06-11T09:30:09.009844Z"
    },
    "tags": []
   },
   "source": [
    "### <span style=\"color:red\"> Questions: </span>\n",
    "\n",
    "- ### How many parameters need to be trained ?\n",
    "- ### What can you say about this value in view of the difficulty of the segmentation task to be performed ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">TODO: Your answer here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "# <span style=\"color:brown\"> **3) Predict segmentations on the whole test set using the network**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:brown\"> 3.1) Visual assessment of the quality of the predictions\n",
    "    \n",
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to apply the U-Net model on the full test dataset and predict the different segmentation masks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-25T14:24:19.609395Z",
     "start_time": "2019-04-25T14:24:03.016351Z"
    }
   },
   "outputs": [],
   "source": [
    "# Predict segmentations on the full test dataset\n",
    "y_pred = loaded_model.predict(X_test, batch_size=1, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Instruction: </span> Run the cell below to display some prediction results. Don't hesitate to re-run the cell several times to see more cases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Select random values\n",
    "nb_imgs = 10\n",
    "start_ind = (np.random.randint(0, X_test.shape[0]-nb_imgs, size=1))[0]\n",
    "end_ind = start_ind+nb_imgs\n",
    "\n",
    "# Display the random images with the corresponding reference and predicted mask\n",
    "plot_compare_segmentation(X_test[start_ind:end_ind], y_test[start_ind:end_ind], y_pred[start_ind:end_ind], \" \", spacing=(1,1), step=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Questions: </span>\n",
    "\n",
    "- ### How do you rate visually the quality of the predictions obtained ?\n",
    "- ### Does the quality of the results vary according to the basal, mid and apex level ?\n",
    "- ### If so, do you have any suggestions about the reason ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">TODO: Your answer here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:brown\"> 3.2) Evaluation of the segmentation quality using standard metrics\n",
    "    \n",
    "In the segmentation domain, the standard metrics to assess segmentation quality are the Dice similarity index, the Hausdorff distance (HD) and the average symmetric surface distance (ASSD) whose definitions are given below:\n",
    "\n",
    "- *Dice similarity index*: defined as $D=2\\left(\\left|S_{user}\\cap S_{ref}\\right|\\right)/\\left(\\left|S_{user}\\right|+\\left|S_{ref}\\right|\\right)$. It is a measure of overlap between the segmented surface $S_{user}$ extracted from a method and the corresponding reference surface $S_{ref}$. The Dice index gives a value between 0 (no overlap) and 1 (full overlap).\n",
    "    \n",
    "- *Hausdorff distance*: measures the local maximum distance between two surfaces.\n",
    "    \n",
    "- *Average Symmetric Surface Distance (ASSD)*: corresponds to the average distance between two surfaces.\n",
    "    \n",
    "<br>\n",
    "\n",
    "**Remark**: for simplicity, the Hausdorff distance and the average symmetric surface distance (ASSD) will be expressed in pixels - *px* - and not in *mm*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\"> Instruction: </span> Run the cells below to compute the metrics defined above on the test dataset and using the pre-trained U-Net model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras_unet.evaluation import  evaluate_segmentation, evaluate_set\n",
    "import pandas as pd\n",
    "from IPython.display import display, HTML \n",
    "\n",
    "# Compute score on the full test dataset\n",
    "dice_all, hausdorff_all, assd_all, valid_all = evaluate_set(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Put the results into a matrix and graft it into a pandas data frame (object from the panda library)\n",
    "overall_results = np.column_stack((dice_all, hausdorff_all, assd_all))\n",
    "overall_results_df = pd.DataFrame(data=overall_results, index = [\"All\", \"Left atrium\", \"Myocardium\", \"Left Ventricle\"], \n",
    "                                  columns=[\"Dice\", \"Hausdorff (px)\", \"ASSD (px)\"]) \n",
    "\n",
    "# Display the data as HTML tables and graphs\n",
    "display(HTML(overall_results_df.to_html(float_format=lambda x: '%.2f' % x)))\n",
    "overall_results_df.plot(kind='bar', figsize=(10,6)).legend() #bbox_to_anchor=(1.6,0.9))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# <span style=\"color:brown\"> **You are now ready to go to part 2 of this hands-on to train your own U-Net!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}