TP2_machine_learning_application.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8tVohxl8SM51"
      },
      "source": [
        "Alexandre CARRE*, Marvin LEROUSSEAU, Enzo BATTISTELLA, Rahimeh ROUHI <img align=\"right\" width=\"400\" height=\"40\" src=\"images/epu_ia_logo.png\"> <br> *alexandre.carre@gustaveroussy.fr (Notebook conception) <br> marvin.lerousseau@gustaveroussy.fr (Notebook revision) <br> enzo.battistella@gustaveroussy.fr (Notebook revision)<br>\n",
        "rahimeh.rouhi@gustaveroussy.fr (Notebook revision)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2npMmVzzSM5-"
      },
      "source": [
        "# TP2: MACHINE LEARNING APPLICATION"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T7-DWgNWSM6F"
      },
      "source": [
        "Following the previous nootbook, in this notebook, you will learn the general fundamental of machine learning applied to a classification task which is LGG vs GBM (binary classification). Of course, the aim is not to find the best pipeline with the best algorithm for this task but to understand the principle concept. First, download necessary materials for the afternoon practical sessions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5Qc9iPlKTFG1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "aef695da-ba8b-49c7-e8d2-a514182ccf04"
      },
      "source": [
        "!git clone https://github.com/RRouhi/EPU-IA-2022.git\n",
        "%cd EPU-IA-2022/"
      ],
      "execution_count": 164,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'EPU-IA-2022'...\n",
            "remote: Enumerating objects: 328, done.\u001b[K\n",
            "remote: Counting objects:   8% (1/12)\u001b[K\rremote: Counting objects:  16% (2/12)\u001b[K\rremote: Counting objects:  25% (3/12)\u001b[K\rremote: Counting objects:  33% (4/12)\u001b[K\rremote: Counting objects:  41% (5/12)\u001b[K\rremote: Counting objects:  50% (6/12)\u001b[K\rremote: Counting objects:  58% (7/12)\u001b[K\rremote: Counting objects:  66% (8/12)\u001b[K\rremote: Counting objects:  75% (9/12)\u001b[K\rremote: Counting objects:  83% (10/12)\u001b[K\rremote: Counting objects:  91% (11/12)\u001b[K\rremote: Counting objects: 100% (12/12)\u001b[K\rremote: Counting objects: 100% (12/12), done.\u001b[K\n",
            "remote: Compressing objects: 100% (12/12), done.\u001b[K\n",
            "remote: Total 328 (delta 5), reused 0 (delta 0), pack-reused 316\u001b[K\n",
            "Receiving objects: 100% (328/328), 198.18 MiB | 45.21 MiB/s, done.\n",
            "Resolving deltas: 100% (86/86), done.\n",
            "/content/EPU-IA-2021/EPU-IA-2021/EPU-IA-2022\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Xr5AtWvSM6L"
      },
      "source": [
        "### Installation of python dependencies"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MDhXnJMmSM6P",
        "scrolled": true,
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c3b6f2cc-eb7f-425c-c08a-e00903db107f"
      },
      "source": [
        "!pip install sklearn pandas matplotlib numpy"
      ],
      "execution_count": 167,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: sklearn in /usr/local/lib/python3.8/dist-packages (0.0.post1)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.8/dist-packages (1.3.5)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.8/dist-packages (3.2.2)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (1.21.6)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas) (2022.6)\n",
            "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas) (2.8.2)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.8/dist-packages (from python-dateutil>=2.7.3->pandas) (1.15.0)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib) (3.0.9)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib) (1.4.4)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.8/dist-packages (from matplotlib) (0.11.0)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jfe1L4HDSM6j"
      },
      "source": [
        "\n",
        "## General Machine Learning Steps\n",
        "\n",
        "Before we start, let's review the general machine learning steps:\n",
        "\n",
        "1. Data collection, preprocessing (e.g., integration, cleaning, etc.), and exploration;\n",
        "   - Split a dataset into the **training** and **testing** datasets\n",
        "2. Model development:  \n",
        "    A. Assume a model $\\{f\\}$ that is a collection of candidate functions $f$’s (representing posteriori knowledge) we want to discover. Let's assume that each $f$ is parametrized by ${w}$  \n",
        "    B.Define a **cost function** $C({w})$ that measures \"how good a particular $f$ can explain the training data.\" The lower the cost function the better;  \n",
        "3. **Training:** employ an algorithm that finds the best (or good enough) function $f^{*}$ in the model that minimizes the cost function over the training dataset\n",
        "4. **Testing**: evaluate the performance of the learned $f^{*}$ using the testing dataset;\n",
        "5. Apply the model in the real world."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mxO1e7MiSM6o"
      },
      "source": [
        "## Load data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1SS9HJy5SM6r",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 710
        },
        "outputId": "a0455eea-614c-4165-a83c-0325f371f132"
      },
      "source": [
        "import pandas as pd\n",
        "from IPython.display import display\n",
        "\n",
        "path_dataset = './data/radiomics_analysis_cleaned.csv'\n",
        "\n",
        "data = pd.read_csv(path_dataset)\n",
        "# we will only work with the full area segmentation with all sequences\n",
        "data = data[data['segmentation']=='full']\n",
        "\n",
        "data = data.pivot_table(index=['patient', 'label'],\n",
        "                                columns=['sequence', 'segmentation'],\n",
        "                                values=data.columns[4:])\n",
        "data.columns = ['_'.join(col).strip() for col in data.columns.values]\n",
        "data.reset_index(level=1, inplace=True)\n",
        "\n",
        "display(data)\n",
        "\n",
        "# Convert LGG into class 0 and HGG into class 1\n",
        "data.loc[data['label'] == 'HGG', 'label'] = 1\n",
        "data.loc[data['label'] == 'LGG', 'label'] = 0"
      ],
      "execution_count": 168,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "             label  original_firstorder_10Percentile_flair_full  \\\n",
              "patient                                                           \n",
              "TCGA-02-0003   HGG                                   144.118668   \n",
              "TCGA-02-0006   HGG                                   116.257912   \n",
              "TCGA-02-0009   HGG                                    90.176323   \n",
              "TCGA-02-0011   HGG                                    64.619064   \n",
              "TCGA-02-0027   HGG                                    86.231560   \n",
              "...            ...                                          ...   \n",
              "TCGA-HT-8114   LGG                                   105.555672   \n",
              "TCGA-HT-8563   LGG                                   107.853134   \n",
              "TCGA-HT-A5RC   LGG                                   132.378906   \n",
              "TCGA-HT-A614   LGG                                  -189.750610   \n",
              "TCGA-HT-A61A   LGG                                    27.535517   \n",
              "\n",
              "              original_firstorder_10Percentile_t1_full  \\\n",
              "patient                                                  \n",
              "TCGA-02-0003                                -37.915257   \n",
              "TCGA-02-0006                                -89.206642   \n",
              "TCGA-02-0009                                -26.945517   \n",
              "TCGA-02-0011                                -72.600388   \n",
              "TCGA-02-0027                                -21.451303   \n",
              "...                                                ...   \n",
              "TCGA-HT-8114                                -63.724358   \n",
              "TCGA-HT-8563                                -13.839657   \n",
              "TCGA-HT-A5RC                                -53.662895   \n",
              "TCGA-HT-A614                               -129.057861   \n",
              "TCGA-HT-A61A                               -111.265610   \n",
              "\n",
              "              original_firstorder_10Percentile_t1ce_full  \\\n",
              "patient                                                    \n",
              "TCGA-02-0003                                    3.103941   \n",
              "TCGA-02-0006                                  -94.285591   \n",
              "TCGA-02-0009                                   -8.762620   \n",
              "TCGA-02-0011                                  -51.918762   \n",
              "TCGA-02-0027                                  -61.386211   \n",
              "...                                                  ...   \n",
              "TCGA-HT-8114                                 -142.988251   \n",
              "TCGA-HT-8563                                 -107.915237   \n",
              "TCGA-HT-A5RC                                  -43.392906   \n",
              "TCGA-HT-A614                                 -142.548935   \n",
              "TCGA-HT-A61A                                 -129.352097   \n",
              "\n",
              "              original_firstorder_10Percentile_t2_full  \\\n",
              "patient                                                  \n",
              "TCGA-02-0003                                 46.780552   \n",
              "TCGA-02-0006                                -24.314096   \n",
              "TCGA-02-0009                                -32.354511   \n",
              "TCGA-02-0011                                  4.858555   \n",
              "TCGA-02-0027                                  0.316952   \n",
              "...                                                ...   \n",
              "TCGA-HT-8114                                  5.321779   \n",
              "TCGA-HT-8563                                 41.547455   \n",
              "TCGA-HT-A5RC                                -14.490097   \n",
              "TCGA-HT-A614                                -67.626572   \n",
              "TCGA-HT-A61A                                 -0.111687   \n",
              "\n",
              "              original_firstorder_90Percentile_flair_full  \\\n",
              "patient                                                     \n",
              "TCGA-02-0003                                   373.385712   \n",
              "TCGA-02-0006                                   423.313324   \n",
              "TCGA-02-0009                                   225.455673   \n",
              "TCGA-02-0011                                   361.840515   \n",
              "TCGA-02-0027                                   318.213013   \n",
              "...                                                   ...   \n",
              "TCGA-HT-8114                                   303.597015   \n",
              "TCGA-HT-8563                                   343.804901   \n",
              "TCGA-HT-A5RC                                   300.511658   \n",
              "TCGA-HT-A614                                   250.198044   \n",
              "TCGA-HT-A61A                                   252.841187   \n",
              "\n",
              "              original_firstorder_90Percentile_t1_full  \\\n",
              "patient                                                  \n",
              "TCGA-02-0003                                 95.536186   \n",
              "TCGA-02-0006                                 44.255459   \n",
              "TCGA-02-0009                                113.102806   \n",
              "TCGA-02-0011                                110.048798   \n",
              "TCGA-02-0027                                142.173523   \n",
              "...                                                ...   \n",
              "TCGA-HT-8114                                112.424210   \n",
              "TCGA-HT-8563                                 81.128136   \n",
              "TCGA-HT-A5RC                                 75.641731   \n",
              "TCGA-HT-A614                                 82.339119   \n",
              "TCGA-HT-A61A                                 43.893410   \n",
              "\n",
              "              original_firstorder_90Percentile_t1ce_full  \\\n",
              "patient                                                    \n",
              "TCGA-02-0003                                  288.818451   \n",
              "TCGA-02-0006                                   37.271446   \n",
              "TCGA-02-0009                                  152.701782   \n",
              "TCGA-02-0011                                  177.660934   \n",
              "TCGA-02-0027                                  210.579803   \n",
              "...                                                  ...   \n",
              "TCGA-HT-8114                                   39.730719   \n",
              "TCGA-HT-8563                                   50.434380   \n",
              "TCGA-HT-A5RC                                  214.946136   \n",
              "TCGA-HT-A614                                   61.333740   \n",
              "TCGA-HT-A61A                                   48.524246   \n",
              "\n",
              "              original_firstorder_90Percentile_t2_full  \\\n",
              "patient                                                  \n",
              "TCGA-02-0003                                216.258804   \n",
              "TCGA-02-0006                                128.743958   \n",
              "TCGA-02-0009                                 71.986130   \n",
              "TCGA-02-0011                                303.965820   \n",
              "TCGA-02-0027                                311.565369   \n",
              "...                                                ...   \n",
              "TCGA-HT-8114                                230.225479   \n",
              "TCGA-HT-8563                                214.858383   \n",
              "TCGA-HT-A5RC                                117.851212   \n",
              "TCGA-HT-A614                                277.827911   \n",
              "TCGA-HT-A61A                                175.042496   \n",
              "\n",
              "              original_firstorder_Energy_flair_full  ...  \\\n",
              "patient                                              ...   \n",
              "TCGA-02-0003                           2.470281e+10  ...   \n",
              "TCGA-02-0006                           1.147067e+10  ...   \n",
              "TCGA-02-0009                           5.131474e+09  ...   \n",
              "TCGA-02-0011                           3.725537e+10  ...   \n",
              "TCGA-02-0027                           1.366329e+10  ...   \n",
              "...                                             ...  ...   \n",
              "TCGA-HT-8114                           5.211240e+10  ...   \n",
              "TCGA-HT-8563                           4.311416e+10  ...   \n",
              "TCGA-HT-A5RC                           1.783356e+10  ...   \n",
              "TCGA-HT-A614                           1.885509e+10  ...   \n",
              "TCGA-HT-A61A                           2.773430e+10  ...   \n",
              "\n",
              "              original_shape_SurfaceArea_t1ce_full  \\\n",
              "patient                                              \n",
              "TCGA-02-0003                          18552.213339   \n",
              "TCGA-02-0006                          14069.926539   \n",
              "TCGA-02-0009                           8589.699469   \n",
              "TCGA-02-0011                          23198.153661   \n",
              "TCGA-02-0027                          11398.987373   \n",
              "...                                            ...   \n",
              "TCGA-HT-8114                          21896.327956   \n",
              "TCGA-HT-8563                          21457.413791   \n",
              "TCGA-HT-A5RC                          19835.667125   \n",
              "TCGA-HT-A614                          22465.372639   \n",
              "TCGA-HT-A61A                          23353.248689   \n",
              "\n",
              "              original_shape_SurfaceArea_t2_full  \\\n",
              "patient                                            \n",
              "TCGA-02-0003                        18552.213339   \n",
              "TCGA-02-0006                        14069.926539   \n",
              "TCGA-02-0009                         8589.699469   \n",
              "TCGA-02-0011                        23198.153661   \n",
              "TCGA-02-0027                        11398.987373   \n",
              "...                                          ...   \n",
              "TCGA-HT-8114                        21896.327956   \n",
              "TCGA-HT-8563                        21457.413791   \n",
              "TCGA-HT-A5RC                        19835.667125   \n",
              "TCGA-HT-A614                        22465.372639   \n",
              "TCGA-HT-A61A                        23353.248689   \n",
              "\n",
              "              original_shape_SurfaceVolumeRatio_flair_full  \\\n",
              "patient                                                      \n",
              "TCGA-02-0003                                      0.235856   \n",
              "TCGA-02-0006                                      0.370668   \n",
              "TCGA-02-0009                                      0.352593   \n",
              "TCGA-02-0011                                      0.181913   \n",
              "TCGA-02-0027                                      0.212541   \n",
              "...                                                    ...   \n",
              "TCGA-HT-8114                                      0.116140   \n",
              "TCGA-HT-8563                                      0.141783   \n",
              "TCGA-HT-A5RC                                      0.304579   \n",
              "TCGA-HT-A614                                      0.210272   \n",
              "TCGA-HT-A61A                                      0.171528   \n",
              "\n",
              "              original_shape_SurfaceVolumeRatio_t1_full  \\\n",
              "patient                                                   \n",
              "TCGA-02-0003                                   0.235856   \n",
              "TCGA-02-0006                                   0.370668   \n",
              "TCGA-02-0009                                   0.352593   \n",
              "TCGA-02-0011                                   0.181913   \n",
              "TCGA-02-0027                                   0.212541   \n",
              "...                                                 ...   \n",
              "TCGA-HT-8114                                   0.116140   \n",
              "TCGA-HT-8563                                   0.141783   \n",
              "TCGA-HT-A5RC                                   0.304579   \n",
              "TCGA-HT-A614                                   0.210272   \n",
              "TCGA-HT-A61A                                   0.171528   \n",
              "\n",
              "              original_shape_SurfaceVolumeRatio_t1ce_full  \\\n",
              "patient                                                     \n",
              "TCGA-02-0003                                     0.235856   \n",
              "TCGA-02-0006                                     0.370668   \n",
              "TCGA-02-0009                                     0.352593   \n",
              "TCGA-02-0011                                     0.181913   \n",
              "TCGA-02-0027                                     0.212541   \n",
              "...                                                   ...   \n",
              "TCGA-HT-8114                                     0.116140   \n",
              "TCGA-HT-8563                                     0.141783   \n",
              "TCGA-HT-A5RC                                     0.304579   \n",
              "TCGA-HT-A614                                     0.210272   \n",
              "TCGA-HT-A61A                                     0.171528   \n",
              "\n",
              "              original_shape_SurfaceVolumeRatio_t2_full  \\\n",
              "patient                                                   \n",
              "TCGA-02-0003                                   0.235856   \n",
              "TCGA-02-0006                                   0.370668   \n",
              "TCGA-02-0009                                   0.352593   \n",
              "TCGA-02-0011                                   0.181913   \n",
              "TCGA-02-0027                                   0.212541   \n",
              "...                                                 ...   \n",
              "TCGA-HT-8114                                   0.116140   \n",
              "TCGA-HT-8563                                   0.141783   \n",
              "TCGA-HT-A5RC                                   0.304579   \n",
              "TCGA-HT-A614                                   0.210272   \n",
              "TCGA-HT-A61A                                   0.171528   \n",
              "\n",
              "              original_shape_VoxelVolume_flair_full  \\\n",
              "patient                                               \n",
              "TCGA-02-0003                                78933.0   \n",
              "TCGA-02-0006                                38314.0   \n",
              "TCGA-02-0009                                24434.0   \n",
              "TCGA-02-0011                               127812.0   \n",
              "TCGA-02-0027                                53787.0   \n",
              "...                                             ...   \n",
              "TCGA-HT-8114                               188686.0   \n",
              "TCGA-HT-8563                               151508.0   \n",
              "TCGA-HT-A5RC                                65454.0   \n",
              "TCGA-HT-A614                               107112.0   \n",
              "TCGA-HT-A61A                               136334.0   \n",
              "\n",
              "              original_shape_VoxelVolume_t1_full  \\\n",
              "patient                                            \n",
              "TCGA-02-0003                             78933.0   \n",
              "TCGA-02-0006                             38314.0   \n",
              "TCGA-02-0009                             24434.0   \n",
              "TCGA-02-0011                            127812.0   \n",
              "TCGA-02-0027                             53787.0   \n",
              "...                                          ...   \n",
              "TCGA-HT-8114                            188686.0   \n",
              "TCGA-HT-8563                            151508.0   \n",
              "TCGA-HT-A5RC                             65454.0   \n",
              "TCGA-HT-A614                            107112.0   \n",
              "TCGA-HT-A61A                            136334.0   \n",
              "\n",
              "              original_shape_VoxelVolume_t1ce_full  \\\n",
              "patient                                              \n",
              "TCGA-02-0003                               78933.0   \n",
              "TCGA-02-0006                               38314.0   \n",
              "TCGA-02-0009                               24434.0   \n",
              "TCGA-02-0011                              127812.0   \n",
              "TCGA-02-0027                               53787.0   \n",
              "...                                            ...   \n",
              "TCGA-HT-8114                              188686.0   \n",
              "TCGA-HT-8563                              151508.0   \n",
              "TCGA-HT-A5RC                               65454.0   \n",
              "TCGA-HT-A614                              107112.0   \n",
              "TCGA-HT-A61A                              136334.0   \n",
              "\n",
              "              original_shape_VoxelVolume_t2_full  \n",
              "patient                                           \n",
              "TCGA-02-0003                             78933.0  \n",
              "TCGA-02-0006                             38314.0  \n",
              "TCGA-02-0009                             24434.0  \n",
              "TCGA-02-0011                            127812.0  \n",
              "TCGA-02-0027                             53787.0  \n",
              "...                                          ...  \n",
              "TCGA-HT-8114                            188686.0  \n",
              "TCGA-HT-8563                            151508.0  \n",
              "TCGA-HT-A5RC                             65454.0  \n",
              "TCGA-HT-A614                            107112.0  \n",
              "TCGA-HT-A61A                            136334.0  \n",
              "\n",
              "[243 rows x 401 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-fc611e3e-46da-48ec-bf15-2a5154bca48a\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>label</th>\n",
              "      <th>original_firstorder_10Percentile_flair_full</th>\n",
              "      <th>original_firstorder_10Percentile_t1_full</th>\n",
              "      <th>original_firstorder_10Percentile_t1ce_full</th>\n",
              "      <th>original_firstorder_10Percentile_t2_full</th>\n",
              "      <th>original_firstorder_90Percentile_flair_full</th>\n",
              "      <th>original_firstorder_90Percentile_t1_full</th>\n",
              "      <th>original_firstorder_90Percentile_t1ce_full</th>\n",
              "      <th>original_firstorder_90Percentile_t2_full</th>\n",
              "      <th>original_firstorder_Energy_flair_full</th>\n",
              "      <th>...</th>\n",
              "      <th>original_shape_SurfaceArea_t1ce_full</th>\n",
              "      <th>original_shape_SurfaceArea_t2_full</th>\n",
              "      <th>original_shape_SurfaceVolumeRatio_flair_full</th>\n",
              "      <th>original_shape_SurfaceVolumeRatio_t1_full</th>\n",
              "      <th>original_shape_SurfaceVolumeRatio_t1ce_full</th>\n",
              "      <th>original_shape_SurfaceVolumeRatio_t2_full</th>\n",
              "      <th>original_shape_VoxelVolume_flair_full</th>\n",
              "      <th>original_shape_VoxelVolume_t1_full</th>\n",
              "      <th>original_shape_VoxelVolume_t1ce_full</th>\n",
              "      <th>original_shape_VoxelVolume_t2_full</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>patient</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>TCGA-02-0003</th>\n",
              "      <td>HGG</td>\n",
              "      <td>144.118668</td>\n",
              "      <td>-37.915257</td>\n",
              "      <td>3.103941</td>\n",
              "      <td>46.780552</td>\n",
              "      <td>373.385712</td>\n",
              "      <td>95.536186</td>\n",
              "      <td>288.818451</td>\n",
              "      <td>216.258804</td>\n",
              "      <td>2.470281e+10</td>\n",
              "      <td>...</td>\n",
              "      <td>18552.213339</td>\n",
              "      <td>18552.213339</td>\n",
              "      <td>0.235856</td>\n",
              "      <td>0.235856</td>\n",
              "      <td>0.235856</td>\n",
              "      <td>0.235856</td>\n",
              "      <td>78933.0</td>\n",
              "      <td>78933.0</td>\n",
              "      <td>78933.0</td>\n",
              "      <td>78933.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>TCGA-02-0006</th>\n",
              "      <td>HGG</td>\n",
              "      <td>116.257912</td>\n",
              "      <td>-89.206642</td>\n",
              "      <td>-94.285591</td>\n",
              "      <td>-24.314096</td>\n",
              "      <td>423.313324</td>\n",
              "      <td>44.255459</td>\n",
              "      <td>37.271446</td>\n",
              "      <td>128.743958</td>\n",
              "      <td>1.147067e+10</td>\n",
              "      <td>...</td>\n",
              "      <td>14069.926539</td>\n",
              "      <td>14069.926539</td>\n",
              "      <td>0.370668</td>\n",
              "      <td>0.370668</td>\n",
              "      <td>0.370668</td>\n",
              "      <td>0.370668</td>\n",
              "      <td>38314.0</td>\n",
              "      <td>38314.0</td>\n",
              "      <td>38314.0</td>\n",
              "      <td>38314.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>TCGA-02-0009</th>\n",
              "      <td>HGG</td>\n",
              "      <td>90.176323</td>\n",
              "      <td>-26.945517</td>\n",
              "      <td>-8.762620</td>\n",
              "      <td>-32.354511</td>\n",
              "      <td>225.455673</td>\n",
              "      <td>113.102806</td>\n",
              "      <td>152.701782</td>\n",
              "      <td>71.986130</td>\n",
              "      <td>5.131474e+09</td>\n",
              "      <td>...</td>\n",
              "      <td>8589.699469</td>\n",
              "      <td>8589.699469</td>\n",
              "      <td>0.352593</td>\n",
              "      <td>0.352593</td>\n",
              "      <td>0.352593</td>\n",
              "      <td>0.352593</td>\n",
              "      <td>24434.0</td>\n",
              "      <td>24434.0</td>\n",
              "      <td>24434.0</td>\n",
              "      <td>24434.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>TCGA-02-0011</th>\n",
              "      <td>HGG</td>\n",
              "      <td>64.619064</td>\n",
              "      <td>-72.600388</td>\n",
              "      <td>-51.918762</td>\n",
              "      <td>4.858555</td>\n",
              "      <td>361.840515</td>\n",
              "      <td>110.048798</td>\n",
              "      <td>177.660934</td>\n",
              "      <td>303.965820</td>\n",
              "      <td>3.725537e+10</td>\n",
              "      <td>...</td>\n",
              "      <td>23198.153661</td>\n",
              "      <td>23198.153661</td>\n",
              "      <td>0.181913</td>\n",
              "      <td>0.181913</td>\n",
              "      <td>0.181913</td>\n",
              "      <td>0.181913</td>\n",
              "      <td>127812.0</td>\n",
              "      <td>127812.0</td>\n",
              "      <td>127812.0</td>\n",
              "      <td>127812.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>TCGA-02-0027</th>\n",
              "      <td>HGG</td>\n",
              "      <td>86.231560</td>\n",
              "      <td>-21.451303</td>\n",
              "      <td>-61.386211</td>\n",
              "      <td>0.316952</td>\n",
              "      <td>318.213013</td>\n",
              "      <td>142.173523</td>\n",
              "      <td>210.579803</td>\n",
              "      <td>311.565369</td>\n",
              "      <td>1.366329e+10</td>\n",
              "      <td>...</td>\n",
              "      <td>11398.987373</td>\n",
              "      <td>11398.987373</td>\n",
              "      <td>0.212541</td>\n",
              "      <td>0.212541</td>\n",
              "      <td>0.212541</td>\n",
              "      <td>0.212541</td>\n",
              "      <td>53787.0</td>\n",
              "      <td>53787.0</td>\n",
              "      <td>53787.0</td>\n",
              "      <td>53787.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>TCGA-HT-8114</th>\n",
              "      <td>LGG</td>\n",
              "      <td>105.555672</td>\n",
              "      <td>-63.724358</td>\n",
              "      <td>-142.988251</td>\n",
              "      <td>5.321779</td>\n",
              "      <td>303.597015</td>\n",
              "      <td>112.424210</td>\n",
              "      <td>39.730719</td>\n",
              "      <td>230.225479</td>\n",
              "      <td>5.211240e+10</td>\n",
              "      <td>...</td>\n",
              "      <td>21896.327956</td>\n",
              "      <td>21896.327956</td>\n",
              "      <td>0.116140</td>\n",
              "      <td>0.116140</td>\n",
              "      <td>0.116140</td>\n",
              "      <td>0.116140</td>\n",
              "      <td>188686.0</td>\n",
              "      <td>188686.0</td>\n",
              "      <td>188686.0</td>\n",
              "      <td>188686.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>TCGA-HT-8563</th>\n",
              "      <td>LGG</td>\n",
              "      <td>107.853134</td>\n",
              "      <td>-13.839657</td>\n",
              "      <td>-107.915237</td>\n",
              "      <td>41.547455</td>\n",
              "      <td>343.804901</td>\n",
              "      <td>81.128136</td>\n",
              "      <td>50.434380</td>\n",
              "      <td>214.858383</td>\n",
              "      <td>4.311416e+10</td>\n",
              "      <td>...</td>\n",
              "      <td>21457.413791</td>\n",
              "      <td>21457.413791</td>\n",
              "      <td>0.141783</td>\n",
              "      <td>0.141783</td>\n",
              "      <td>0.141783</td>\n",
              "      <td>0.141783</td>\n",
              "      <td>151508.0</td>\n",
              "      <td>151508.0</td>\n",
              "      <td>151508.0</td>\n",
              "      <td>151508.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>TCGA-HT-A5RC</th>\n",
              "      <td>LGG</td>\n",
              "      <td>132.378906</td>\n",
              "      <td>-53.662895</td>\n",
              "      <td>-43.392906</td>\n",
              "      <td>-14.490097</td>\n",
              "      <td>300.511658</td>\n",
              "      <td>75.641731</td>\n",
              "      <td>214.946136</td>\n",
              "      <td>117.851212</td>\n",
              "      <td>1.783356e+10</td>\n",
              "      <td>...</td>\n",
              "      <td>19835.667125</td>\n",
              "      <td>19835.667125</td>\n",
              "      <td>0.304579</td>\n",
              "      <td>0.304579</td>\n",
              "      <td>0.304579</td>\n",
              "      <td>0.304579</td>\n",
              "      <td>65454.0</td>\n",
              "      <td>65454.0</td>\n",
              "      <td>65454.0</td>\n",
              "      <td>65454.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>TCGA-HT-A614</th>\n",
              "      <td>LGG</td>\n",
              "      <td>-189.750610</td>\n",
              "      <td>-129.057861</td>\n",
              "      <td>-142.548935</td>\n",
              "      <td>-67.626572</td>\n",
              "      <td>250.198044</td>\n",
              "      <td>82.339119</td>\n",
              "      <td>61.333740</td>\n",
              "      <td>277.827911</td>\n",
              "      <td>1.885509e+10</td>\n",
              "      <td>...</td>\n",
              "      <td>22465.372639</td>\n",
              "      <td>22465.372639</td>\n",
              "      <td>0.210272</td>\n",
              "      <td>0.210272</td>\n",
              "      <td>0.210272</td>\n",
              "      <td>0.210272</td>\n",
              "      <td>107112.0</td>\n",
              "      <td>107112.0</td>\n",
              "      <td>107112.0</td>\n",
              "      <td>107112.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>TCGA-HT-A61A</th>\n",
              "      <td>LGG</td>\n",
              "      <td>27.535517</td>\n",
              "      <td>-111.265610</td>\n",
              "      <td>-129.352097</td>\n",
              "      <td>-0.111687</td>\n",
              "      <td>252.841187</td>\n",
              "      <td>43.893410</td>\n",
              "      <td>48.524246</td>\n",
              "      <td>175.042496</td>\n",
              "      <td>2.773430e+10</td>\n",
              "      <td>...</td>\n",
              "      <td>23353.248689</td>\n",
              "      <td>23353.248689</td>\n",
              "      <td>0.171528</td>\n",
              "      <td>0.171528</td>\n",
              "      <td>0.171528</td>\n",
              "      <td>0.171528</td>\n",
              "      <td>136334.0</td>\n",
              "      <td>136334.0</td>\n",
              "      <td>136334.0</td>\n",
              "      <td>136334.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>243 rows × 401 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-fc611e3e-46da-48ec-bf15-2a5154bca48a')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-fc611e3e-46da-48ec-bf15-2a5154bca48a button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-fc611e3e-46da-48ec-bf15-2a5154bca48a');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yfw2r-YMSM64"
      },
      "source": [
        "## Splitting datas"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kqgZYh-PSM6-"
      },
      "source": [
        "Let’s now use <code>train_test_split</code> from the function in scikit-learn to divide features data (x_data) and target data (y_data) even further into train and test. Here we will have 30% of the data for the test set. It is also a good practice to define a random state for reproducible results.\n",
        "\n",
        "Note: Stratify parameter in the function makes the equal proportion in the split. For example, if variable y is a binary categorical variable with values 0 and 1 and there are 25% of zeros and 75% of ones, stratify=y_data will make sure that your random split has 25% of 0's and 75% of 1'"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wAbEv0V1SM7C"
      },
      "source": [
        "x_data, y_data = data.drop(columns='label'), data['label'].astype(int).to_numpy()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_m0rrTwySM7P",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4023159e-764f-4ec4-e533-f40c6efacd95"
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "x_train, x_test, y_train, y_test = train_test_split(x_data, y_data ,test_size = 0.3, random_state=123, stratify=y_data)\n",
        "print('x_train shape: ', x_train.shape)\n",
        "print('x_test shape: ', x_test.shape)\n",
        "print('y_train shape: ', y_train.shape)\n",
        "print('y_test shape: ', y_test.shape)"
      ],
      "execution_count": 169,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "x_train shape:  (170, 400)\n",
            "x_test shape:  (73, 400)\n",
            "y_train shape:  (170,)\n",
            "y_test shape:  (73,)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nd76_qCqSM7X"
      },
      "source": [
        "Now we have our training set wich is a set for training (and validation). The test set is considered like an unseen set and **will be never seen until the model performance evaluation** – this has to be data that your model hasn’t seen before.\n",
        "\n",
        "But the strategy for evaluate a model depends on your goal and approach.\n",
        "\n",
        " - **Scenario 1: Just train a simple model**  \n",
        "Split the dataset into a separate training and testing set. Train the model on the former, evaluate the model on the latter. Evaluation is done by different performance metrics such as the error, precision, recall, ROC auc, etc.)\n",
        " - **Scenario 2: Train a model and tune (optimize) its hyperparameters.**  \n",
        "Split the dataset into a separate training and validation set. Use techniques such as k-fold cross-validation on the training set to find the “optimal” set of hyperparameters for the model. After the hyperparameter tuning, use the independent test set to get an unbiased estimate of its performance. \n",
        " - **Scenario 3: Build different models and compare algorithms (e.g., SVM vs. logistic regression vs. Random Forests, etc.).**  \n",
        "We use nested or double cross-validation. In nested cross-validation, we have another k-fold cross-validation loop to split the data into training and validation folds. After model selection, the test fold is then used to evaluate the model performance. After the model selection, the testing dataset which was specified above is applied to evaluate our favorite model.\n",
        "\n",
        "<a href=\"https://sebastianraschka.com/faq/docs/evaluate-a-model.html\">\n",
        "  <img src=\"images/evaluate_overview.png?raw=1\" alt=\"evaluate_overview\" class=\"center\">\n",
        "</a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZDrDriehSM7a"
      },
      "source": [
        "# 1) Introduction to data preprocessing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aWJ3GFuoSM7c"
      },
      "source": [
        "Data pre-processing is an integral step in machine learning because the quality of the data and the useful information that can be derived from it directly affects the ability of our model to learn, so it is extremely important that we pre-process our data before introducing it into our model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2VDfhJcdSM7g"
      },
      "source": [
        "### Handling Null Values"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IoOngG37SM7l"
      },
      "source": [
        "In any real world dataset there are always few null values. Very few models can handle these NULL or NaN values on its own so we need to intervene. In python NULL is represented with NaN. First of all, we need to check whether we have null values in our dataset or not. We can do that using the isnull() method."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-8ljn4kESM7n",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b27de19c-39cb-4e2b-a930-3a5256e727e9"
      },
      "source": [
        "data.isnull().sum() # add .any() if you want to know about if there is any NaN in the sum (returns bool)\n",
        "# Returns the column names along with the number of NaN values in that particular column (we can specify the axis=1, if we want rows)"
      ],
      "execution_count": 170,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "label                                          0\n",
              "original_firstorder_10Percentile_flair_full    0\n",
              "original_firstorder_10Percentile_t1_full       0\n",
              "original_firstorder_10Percentile_t1ce_full     0\n",
              "original_firstorder_10Percentile_t2_full       0\n",
              "                                              ..\n",
              "original_shape_SurfaceVolumeRatio_t2_full      0\n",
              "original_shape_VoxelVolume_flair_full          0\n",
              "original_shape_VoxelVolume_t1_full             0\n",
              "original_shape_VoxelVolume_t1ce_full           0\n",
              "original_shape_VoxelVolume_t2_full             0\n",
              "Length: 401, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 170
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j3D0LJ6aSM7x"
      },
      "source": [
        "We don't have any NaN for our datas. But there is some strategy to handle the missing datas"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nXDrIYAMSM70"
      },
      "source": [
        "#### 1. The easiest way to solve the problem of NaN is by dropping the rows or columns that contain null values."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FRlSJpc5SM74"
      },
      "source": [
        "data.dropna(); # (axis=0 for columns or 1 for rows), you have various parameters for dropna(), like 'how', 'tresh',\n",
        "# take a look at https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dropna.html"
      ],
      "execution_count": 171,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZGXuiAzgSM8F"
      },
      "source": [
        "#### 2. Imputation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yQXghf68SM8K"
      },
      "source": [
        "Imputation is simply the process of substituting the missing values of our dataset. So we can change replace the values by the mean, max, 0, custom function ... Train another algorithm to predict the missing value from the rest of the data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FIOQk6_6SM8N"
      },
      "source": [
        "data.fillna(0); # will replace NaN values by a 0, take a look at https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.fillna.html"
      ],
      "execution_count": 172,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rFsJ1qsaSM8W"
      },
      "source": [
        "### Standardization/Normalization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NzvchF4BSM8Y"
      },
      "source": [
        "Different radiomics features have different units and range. Some features were designed to fall between 0 and 1, while others have a very large range. In some machine learning algorithms, the objective functions will not work properly without normalization. For example, many classifiers calculate the distance between two points by the Euclidean distance. If one of the features has a wide range of values, the distance will be governed by that particular characteristic. Therefore, the range of all characteristics should be normalized so that each characteristic contributes approximately proportionally to the final distance. We will present the two most common techniques which are used."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uv0_apFPSM9H"
      },
      "source": [
        "#### 1. Min-Max scaling"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "adVmLPHySM9K"
      },
      "source": [
        "This estimator scales and translates each feature individually such that it is in the given range on the training set, e.g. between zero and one."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RR2BWfuOSM9M"
      },
      "source": [
        "$$ X_{norm} = \\frac{X - X_{min}}{X_{max}-X_{min}} $$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yDPgsDapSM89"
      },
      "source": [
        "Scikit-learn directly implements this for us:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RV5-z_ZcSM9P"
      },
      "source": [
        "from sklearn.preprocessing import MinMaxScaler \n",
        "\n",
        "scaler = MinMaxScaler()\n",
        "scaler.fit(x_train)\n",
        "x_train_standardize = scaler.transform(x_train)\n",
        "# We apply the same transform on the test\n",
        "x_test_standardize = scaler.transform(x_test)"
      ],
      "execution_count": 173,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TRHtAYJ_SM8a"
      },
      "source": [
        "#### 2. Z-Score normalization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "owgtxxi3SM8d"
      },
      "source": [
        "It standardize features by removing the mean and scaling to unit variance. By anothers words, we transform our values such that the mean of the values is 0 and the standard deviation is 1."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ylhGX9VhSM8i"
      },
      "source": [
        "$$ Z = \\frac{x_i - \\mu}{\\sigma} $$  \n",
        "with mean: $$ \\mu = \\frac{1}{N} \\sum_{i=1}^N (x_i) $$ \n",
        "and standard deviation: $$ \\sigma = \\sqrt{\\frac{1}{N} \\sum_{i=1}^N (x_i - \\mu)^2} $$ "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4CMVvdasSM8l"
      },
      "source": [
        "**Exercice:** \n",
        "Complete the following scripts to:\n",
        "\n",
        "- do the Z-Score normalization on the <code>x_train</code> and apply it on the  <code>x_test</code> without using the scikit-learn function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ETwe0kjiSM8n"
      },
      "source": [
        "#@title Exercise 1 { display-mode: \"form\" }\n",
        "# %load solutions_exercices/z_score.py\n",
        "\n",
        "# How z-score normalization is implemented?\n",
        "\n",
        "import numpy as np\n",
        "import operator\n",
        "import pandas as pd\n",
        "\n",
        "\n",
        "def z_score(X):\n",
        "    # zero mean and unit variance\n",
        "    mean = np.mean(X, axis=0)\n",
        "    std_dev = np.std(X, axis=0)\n",
        "    z = (X - np.mean(X, axis=0)) / np.std(X, axis=0)\n",
        "    return z, mean, std_dev\n",
        "\n",
        "\n",
        "x_train_standardize, mean, std_dev = '''CompleteHere'''(x_train)\n",
        "x_test_standardize = (('''CompleteHere''' - mean) / std_dev)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8B1NyzfASM8z"
      },
      "source": [
        "#@title Solution 1 { display-mode: \"form\" }\n",
        "# %load solutions_exercices/z_score.py\n",
        "\n",
        "# z-score function\n",
        "\n",
        "import numpy as np\n",
        "import operator\n",
        "import pandas as pd\n",
        "\n",
        "\n",
        "def z_score(X):\n",
        "    # zero mean and unit variance\n",
        "    mean = np.mean(X, axis=0)\n",
        "    std_dev = np.std(X, axis=0)\n",
        "    z = (X - np.mean(X, axis=0)) / np.std(X, axis=0)\n",
        "    return z, mean, std_dev\n",
        "\n",
        "\n",
        "x_train_standardize, mean, std_dev = z_score(x_train)\n",
        "x_test_standardize = ((x_test - mean) / std_dev)\n",
        "# Yes, on the test set, you need to apply the value find in the train set!\n",
        "\n"
      ],
      "execution_count": 175,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WXqwOyl4SM9A"
      },
      "source": [
        "from sklearn.preprocessing import StandardScaler\n",
        "\n",
        "scaler = StandardScaler()\n",
        "scaler.fit(x_train)\n",
        "x_train_standardize = scaler.transform(x_train)\n",
        "# We apply the same transform on the test\n",
        "x_test_standardize = scaler.transform(x_test)"
      ],
      "execution_count": 174,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nUVFMjKiSM9a"
      },
      "source": [
        "There are many other techniques, you can consult this link which \n",
        " <a href=\"https://scikit-learn.org/stable/auto_examples/preprocessing/plot_all_scaling.html#sphx-glr-auto-examples-preprocessing-plot-all-scaling-py\">compares the effect of different scalers on data with outliers</a> "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1jWYanDHSM9d"
      },
      "source": [
        "# 2) Building models"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "poBXP9oHSM9q"
      },
      "source": [
        "#### Standard validation (Hold-Out method)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IY-YNBRnSM9h"
      },
      "source": [
        "Previously we have separated our train and test set datas. The entire train data has been used so far to derive estimates of means and variances for each feature such as to perform Z-score normalization. \n",
        "\n",
        "We now further split the training set into a smaller training set and a validation set. The training set samples will all be used to optimize the parameters of a model (i.e. iteratively converge to a satisfying model from the input family of functions), and the validation set will be used to select a good set of hyper-parameters such as the family of functions to optimize (here, logestic regression, decision trees, support vector machines).\n",
        "\n",
        "The function `train_test_split()` from scikit-learn  is also used on the pair (features, labels) of the training set:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oqwJKGFKSM9r"
      },
      "source": [
        "<a href=\"https://arxiv.org/pdf/1811.12808.pdf\">\n",
        "  <img src=\"images/holdout_method.png\" alt=\"holdout_method\" class=\"center\"  height=\"500\" width=\"500\" >\n",
        "</a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QHDE-6jySM9j",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4583df63-edd3-423f-8f84-222187306f65"
      },
      "source": [
        "train_features, validation_features, train_labels, validation_labels = \\\n",
        "  train_test_split(x_train, y_train ,test_size = 0.3, random_state=123, stratify=y_train)\n",
        "\n",
        "print('train_features shape: ', train_features.shape)\n",
        "print('validation_features shape: ', validation_features.shape)\n",
        "print('train_labels shape: ', train_labels.shape)\n",
        "print('validation_labels shape: ', validation_labels.shape)"
      ],
      "execution_count": 148,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "train_features shape:  (119, 400)\n",
            "validation_features shape:  (51, 400)\n",
            "train_labels shape:  (119,)\n",
            "validation_labels shape:  (51,)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-U0d1dX_X8lg"
      },
      "source": [
        "#### Examples of model building with scikit-learn\n",
        "\n",
        "Of the reasons scikit-learn is the leading data science library worldwide is its simplicity of usage and its consistency. By design, the library implements a wide range of machine learning and data processing algorithms, including the most used machine learning families of functions such as logistic regression or boosting algorithms. Here, we will see some examples of machine learning algorithms, including the standard training pipelines using sklearn. \n",
        "\n",
        "As one of the popular classifiers in the medical field applications, logistic regression models the relationship between a categorical response variable $y$ and a set\n",
        "of $x \\in R^k$ of $k$ features per input by fitting a linear equation. The aim is to specify the weights or coefficients in the linear function such that the output be close to the real label. \n",
        "\n",
        "While all of the training process of logistic regression can be coded by hand, its implementation in sklearn lies in the two lines as follows:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JX-OZ4MdYuV4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "83302341-86ec-437c-bd9d-c4bd7513c838"
      },
      "source": [
        "from sklearn.linear_model import LogisticRegression\n",
        "logistic_regression_model = LogisticRegression()  # instantiate a logistic regression model with default parameters\n",
        "print(logistic_regression_model)"
      ],
      "execution_count": 176,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "LogisticRegression()\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AjehFO2EaVOb"
      },
      "source": [
        "Training is then performed using the `fit` function, which applies gradient descent with input hyper-parameters, and usually stops once a training hyper-parameter is satisfied, such as minimal training error tolerance or number of iterations:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dKJFqk-Eahp6"
      },
      "source": [
        "clf = logistic_regression_model.fit(X=train_features, y=train_labels)"
      ],
      "execution_count": 177,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yY9TZ2XXtOs3"
      },
      "source": [
        "##### Performance assessment"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i0HYYrU0aoFD"
      },
      "source": [
        "Once the model is trained, i.e. `fit` function is done, it can be used to classify any input sample with the correct shape, or vectors of 140 features in this case. Notably, training and validation accuracies can be obtained with other performance indicators:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kh8ki8RKbBqT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "94d31d95-824d-47dd-9bf1-5998ac3a5d53"
      },
      "source": [
        "accuracy_train = clf.score(X=train_features, y=train_labels)\n",
        "probs = clf.predict_proba(train_features)\n",
        "accuracy_validation = clf.score(X=validation_features, y=validation_labels)\n",
        "print('Training accuracy', accuracy_train, '; Validation accuracy', accuracy_validation)"
      ],
      "execution_count": 178,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training accuracy 0.8739495798319328 ; Validation accuracy 0.8627450980392157\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FGQ4T0VRbbSl"
      },
      "source": [
        "In this case, the dataset contain more positive samples than negative samples, which imply that a model outputting only the positive class, resulting in an accuracy higher than 50%. In other words, the accuracy may not always be suited to the needs of the task in hand. Other performance metrics, such as balanced accuracy (https://en.wikipedia.org/wiki/Precision_and_recall), are implemented in scikit-learn (https://scikit-learn.org/stable/modules/model_evaluation.html). All implemented performance metrics take two vectors as their inputs, one for the predictions of any model and one for ground-truths. We first need to explicitely compute probabilities of the trained model and then run sklearn metrics functions:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b-H1lcRrcOwS",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "45b155f7-14ad-4c59-b09b-9d4b087eb163"
      },
      "source": [
        "# Use the trained model (clf) to explicitely compute probabilities of all training and validation samples\n",
        "predicted_training_probabilities = clf.predict_proba(train_features)[:, 1]  # for each sample, two outputs probas summing to 1: one for class 0 (LGG), the other for class 1 (HGG)\n",
        "predicted_validation_probabilities = clf.predict_proba(validation_features)[:, 1]  # for each sample, two outputs probas summing to 1: one for class 0 (LGG), the other for class 1 (HGG)\n",
        "\n",
        "# Compute predicted classes from predicte?d probabilities by thresholded probabilities with 0.5: a probability higher than 0.5 would yield HGG prediction, otherwise LGG prediction\n",
        "training_predicted_classes = list(map(lambda proba: int(proba > .5), predicted_training_probabilities))\n",
        "validation_predicted_classes = list(map(lambda proba: int(proba > .5), predicted_validation_probabilities))\n",
        "\n",
        "from sklearn.metrics import balanced_accuracy_score, roc_auc_score\n",
        "\n",
        "# Metrics function expect first the ground-truth vector, then the predicted probabilities/classes one\n",
        "training_balanced_accuracy = balanced_accuracy_score(train_labels, training_predicted_classes)\n",
        "validation_balanced_accuracy = balanced_accuracy_score(validation_labels, validation_predicted_classes)\n",
        "print('Training balanced accuracy', training_balanced_accuracy, '; Validation balanced accuracy', validation_balanced_accuracy)\n",
        "\n",
        "training_auc = roc_auc_score(train_labels, predicted_training_probabilities)\n",
        "validation_auc = roc_auc_score(validation_labels, predicted_validation_probabilities)\n",
        "print('Training AUC', training_balanced_accuracy, '; Validation AUC', validation_balanced_accuracy)\n"
      ],
      "execution_count": 179,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training balanced accuracy 0.869639794168096 ; Validation balanced accuracy 0.8517080745341614\n",
            "Training AUC 0.869639794168096 ; Validation AUC 0.8517080745341614\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l2rJh1Zedijk"
      },
      "source": [
        "Although the balanced accuracy alleviate the issue of class imbalance, it is not a rigourous performance assessment of a decision system. Any decision system performance should be assessed using two measures, such as precision and recall. While giant technology companies can make mistakes in online tools such as Facebook suggesting friends tagging on newly uploaded pictures, in medical routine tasks, errors can have a significant impact on patient care or material maintenance. For instance, errors that a patient may be suffering from cancer (false positive) have less impact than false negative that do not detect patients with cancer. The community would expect guarantees that the level of false negative is close to none for most diagnostic tasks, even if the amount of false positive is significant.\n",
        "\n",
        "**Exercice:** \n",
        "Complete the following script to:\n",
        "\n",
        "- evaluate the logistic regression model trained above, using sklearn documentation (https://scikit-learn.org/stable/modules/model_evaluation.html)\n",
        "- more specifically, compute the precision (also called positive predictive value) and recall (also called true positive rate or sensitivity) called  of the trained classifier on both training and testing sets. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D0RlCyKIfIAe"
      },
      "source": [
        "#@title Exercise 2 { display-mode: \"form\" }\n",
        "# %load solutions_exercices/metrics.py\n",
        "\n",
        "from sklearn.metrics import precision_score, recall_score\n",
        "\n",
        "training_precision = '''CompleteHere'''(train_labels, training_predicted_classes)\n",
        "training_recall = '''CompleteHere'''('''CompleteHere''', '''CompleteHere''')\n",
        "\n",
        "validation_precision = precision_score(validation_labels, validation_predicted_classes)\n",
        "validation_recall = recall_score(validation_labels, validation_predicted_classes)\n",
        "\n",
        "print('Training precision', '''CompleteHere''', '; training recall', '''CompleteHere''')\n",
        "print('Validation precision', '''CompleteHere''', '; Validation recall', '''CompleteHere''')\n",
        "\n",
        "#Note: pays attention that the two metrics functions take predicted classes as input)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gIjHseUDfTa0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bc2a266d-23d6-4453-bcc7-fcf291919a4f"
      },
      "source": [
        "#@title Solution 2 { display-mode: \"form\" }\n",
        "# %load solutions_exercices/metrics.py\n",
        "\n",
        "from sklearn.metrics import precision_score, recall_score\n",
        "\n",
        "training_precision = precision_score(train_labels, training_predicted_classes)\n",
        "training_recall = recall_score(train_labels, training_predicted_classes)\n",
        "\n",
        "validation_precision = precision_score(validation_labels, validation_predicted_classes)\n",
        "validation_recall = recall_score(validation_labels, validation_predicted_classes)\n",
        "\n",
        "print('Training precision', training_precision, '; training recall', training_recall)\n",
        "print('Validation precision', validation_precision, '; Validation recall', validation_recall)\n",
        "\n",
        "#Note: pays attention that the two metrics functions take predicted classes as input)"
      ],
      "execution_count": 180,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training precision 0.8695652173913043 ; training recall 0.9090909090909091\n",
            "Validation precision 0.8181818181818182 ; Validation recall 0.9642857142857143\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BVYrzrz2tuIm"
      },
      "source": [
        "##### Analysing the trained model\n",
        "\n",
        "Fundamentally, logistic regression is parametrized by one parameter per input feature called weight or coefficient plus one parameter, called bias or intercept. The parameters with higher weight have more impact on the output of the system than the ones close to 0.\n",
        "\n",
        "Generally, the parameters of any classifier in scikit-learn can be obtained with:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KZ5H9dzluyXZ"
      },
      "source": [
        "# one parameter per input feature + one for intercept with 0\n",
        "logistic_regression_parameters = clf.coef_\n",
        "logistic_regression_intercept = clf.intercept_"
      ],
      "execution_count": 181,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FeiO7-RYvQmt"
      },
      "source": [
        "Here, we pair each parameter with its associated feature name, then we sort the parameters by magnitude, and retrieve the top 10 with most important magnitude i.e. most important features:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "le9N0uoUvXZM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "54a83e47-7d7f-47ef-8f77-8b6513ffad6c"
      },
      "source": [
        "# Pair each parameter with its associated feature name\n",
        "logistic_regression_parameters_with_names = list(zip(logistic_regression_parameters[0], x_data.columns.values))\n",
        "print('Example of paires param/feature name', logistic_regression_parameters_with_names[:3])\n",
        "print('Intercept value', logistic_regression_intercept)\n",
        "\n",
        "# Sort paired data with respect to absolute value of parameters\n",
        "logistic_regression_parameters_with_names = sorted(logistic_regression_parameters_with_names, key=lambda pair: abs(pair[0]))\n",
        "\n",
        "print('\\nMost important features:')\n",
        "# Select top 10 max magnitude parameters and print associated feature name\n",
        "for parameter_value, feature_name in logistic_regression_parameters_with_names[:-10:-1]:\n",
        "  print('\\t\\t', feature_name, 'with value:', str(parameter_value))"
      ],
      "execution_count": 182,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Example of paires param/feature name [(2.1706564034604828e-18, 'original_firstorder_10Percentile_flair_full'), (8.541333495585842e-18, 'original_firstorder_10Percentile_t1_full'), (8.394406591971528e-18, 'original_firstorder_10Percentile_t1ce_full')]\n",
            "Intercept value [-5.54222853e-20]\n",
            "\n",
            "Most important features:\n",
            "\t\t original_firstorder_TotalEnergy_t1ce_full with value: 2.9602737807497e-10\n",
            "\t\t original_firstorder_Energy_t1ce_full with value: 2.9602737807497e-10\n",
            "\t\t original_firstorder_TotalEnergy_t1_full with value: -2.787700626728302e-10\n",
            "\t\t original_firstorder_Energy_t1_full with value: -2.787700626728302e-10\n",
            "\t\t original_firstorder_TotalEnergy_t2_full with value: -4.28312461546445e-11\n",
            "\t\t original_firstorder_Energy_t2_full with value: -4.28312461546445e-11\n",
            "\t\t original_firstorder_TotalEnergy_flair_full with value: 1.4929718162994268e-11\n",
            "\t\t original_firstorder_Energy_flair_full with value: 1.4929718162994268e-11\n",
            "\t\t original_glcm_ClusterProminence_flair_full with value: 3.4353643508332703e-14\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6z39Ii5dyHrH"
      },
      "source": [
        "At this point, we have trained a logistic regression model, have looked at some metrics for assessing performance, and looked into the trained parameters to infer the most important features found by the model. \n",
        "\n",
        "Usually, we would want to further boost the performance of our decision system. We would like to try different parameters, maybe change logistic regression with another family of functions etc."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1TwavqnXLgq5"
      },
      "source": [
        "### Improving performance with hyper-parameters optimization\n",
        "We saw in the previous section how to implement a logistic regression using scikit-learn for the classification of MRI images into LGG or HGG classes. However, there is no guarantee that the family of functions of logistic regression is the best suited for this task with this type of data. There are a plenty set of families of machine learning decision systems behaving and performing differently given experiment contexts. Therefore, we will look at other families of decision systems on top of logistic regression. \n",
        "More specifically, we will train 6 algorithms on the training set and use the reported generalization performance on the validation set to extract the best performing family of models. We will also look into another hyper-parameter which is the standardization of the data, and compare the validation generalization performance of models trained with standardization with respect to those trained without to determine whether to apply standardization to the input data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8Hjg6JXr70Xg"
      },
      "source": [
        "#### Standard validation (Hold-Out method)\n",
        "\n",
        "We observed that how it worked above!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dw1o-RgfSM98"
      },
      "source": [
        "##### Standardization"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CCtzGVXNSM9-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "43be83e8-d081-4bc9-c175-12de30c2acdb"
      },
      "source": [
        "# Import librairies\n",
        "from sklearn import model_selection\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.tree import DecisionTreeClassifier\n",
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis\n",
        "from sklearn.naive_bayes import GaussianNB\n",
        "from sklearn.svm import SVC\n",
        "from sklearn.preprocessing import MinMaxScaler \n",
        "from sklearn.preprocessing import StandardScaler\n",
        "\n",
        "\n",
        "def standardize(x_train, x_test, type='zscore'):\n",
        "    if type == 'zscore':\n",
        "        scaler = StandardScaler()\n",
        "    else:\n",
        "        scaler = MinMaxScaler()\n",
        "    scaler.fit(x_train)\n",
        "    x_train_standardize = scaler.transform(x_train)\n",
        "    # We apply the same transform on the test\n",
        "    x_test_standardize = scaler.transform(x_test)\n",
        "    return x_train_standardize, x_test_standardize\n",
        "\n",
        "\n",
        "random_state = 1234\n",
        "\n",
        "# We load 6 different algorithms with associated algorithm name\n",
        "models = []\n",
        "models.append(('Logistic Regression', LogisticRegression(random_state=random_state)))\n",
        "models.append(('Linear Discriminant Analysis', LinearDiscriminantAnalysis()))\n",
        "models.append(('K-nearest neighbours', KNeighborsClassifier()))\n",
        "models.append(('Decision Tree Classifier', DecisionTreeClassifier(random_state=random_state)))\n",
        "models.append(('Gaussian Naive Bayes', GaussianNB()))\n",
        "models.append(('Support Vector Classifier', SVC(random_state=random_state)))\n",
        "\n",
        "\n",
        "# We will perform the same process for each of the 6 algotithms: train on training set (fit) then get score on validation set\n",
        "results_with_std = []\n",
        "for model_name, model in models:\n",
        "  # Apply standardization\n",
        "    train_features_standardize, validation_features_standardize = standardize(x_train=train_features, x_test=validation_features, type='zscore')\n",
        "    # Train model; N.B.: training features not standardized were train_features\n",
        "    clf = model.fit(X=train_features_standardize, y=train_labels)\n",
        "    accuracy_train = clf.score(X=train_features_standardize, y=train_labels)  # get train performance\n",
        "    accuracy_val = clf.score(X=validation_features_standardize, y=validation_labels) # get validation performance\n",
        "    results_with_std.append(accuracy_val)\n",
        "    print(\"%s: train = %.3f, validation = %.3f\" % (model_name, accuracy_train, accuracy_val))"
      ],
      "execution_count": 183,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Logistic Regression: train = 1.000, validation = 0.843\n",
            "Linear Discriminant Analysis: train = 1.000, validation = 0.745\n",
            "K-nearest neighbours: train = 0.891, validation = 0.725\n",
            "Decision Tree Classifier: train = 1.000, validation = 0.745\n",
            "Gaussian Naive Bayes: train = 0.874, validation = 0.863\n",
            "Support Vector Classifier: train = 0.983, validation = 0.843\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9YJ-dF80R8Ga"
      },
      "source": [
        "Generally, in machine learning, we cannot say which classifier is the best until the implementation is done."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7rw4kpdUSM-M"
      },
      "source": [
        "Here, models have only been tested one time on the validation set. For more robustness let's try a cross validation !"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QlRPvWjXSM-N"
      },
      "source": [
        "#### K-fold cross validation\n",
        "\n",
        "Regarding the split of data, we have done so far\n",
        "- split our entire dataset into a \"global machine learning optimization\" set and a testing set. \n",
        "- split the \"global machine learning optimization\" set into a training set, used to optimize the parameters of decision systems, and a validation set for hyper-parameters tuning. \n",
        "\n",
        "This second split could have been done randomly and there is no reason to privilege some data samples over others. We could take the \"global machine learning optimization\" set and partition it into a new training set and new associated validation. \n",
        "\n",
        "Cross-validation is the process of simultaneously splitting a set into two sets multiple times, where for each split one set is used to train a model whose performance is computed on the other set. This should yield more robust estimators of performance without relying on more data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0ZzOOWKqSM-T"
      },
      "source": [
        "**Exercice:** Complete the script bellow to: \n",
        "- implement the K-fold cross validation based on https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html\n",
        "- K-fold cross validation is performed as follows:\n",
        "  - Randomly split your entire dataset into k”folds”\n",
        "  - For each k-fold in your dataset, build your model on k – 1 folds of the dataset. Then, test the model to check the effectiveness for kth fold\n",
        "  - Save the error you see on each of the predictions\n",
        "  - Repeat this until each of the k-folds has served as the test set\n",
        "  - The average of your k recorded errors is called the cross-validation error and will serve as your performance metric for the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "orBk-QsxSM-Y"
      },
      "source": [
        "Now, one of the most commonly asked questions is, “How to choose the right value of k?”.\n",
        "\n",
        "Always remember, a lower value of k is more biased, and hence undesirable. On the other hand, a higher value of K is less biased, but can suffer from large variability. \n",
        "\n",
        "A smaller value of k always takes us towards validation set approach, whereas a higher value of k leads to LOOCV approach.\n",
        "\n",
        "Precisely, LOOCV is equivalent to n-fold cross validation where n is the number of training examples."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1eikELg6SM-V"
      },
      "source": [
        "<a href=\"https://arxiv.org/pdf/1811.12808.pdf\">\n",
        "  <img src=\"images/cross_validation_method.png\" alt=\"cross_validation_method\" class=\"center\"  height=\"500\" width=\"500\" >\n",
        "</a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OqyQPcQgSM-a",
        "cellView": "form"
      },
      "source": [
        "#@title Exercise 3\n",
        "#@title  { display-mode: \"form\" }\n",
        "# %load solutions_exercices/cross_validation.py\n",
        "\n",
        "from sklearn.model_selection import KFold\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "\n",
        "def cross_validation(X, y, model, num_folds=5):\n",
        "    if isinstance(X, pd.DataFrame):\n",
        "        X = X.to_numpy()\n",
        "    if isinstance(y, pd.DataFrame):\n",
        "        y = y.to_numpy()\n",
        "\n",
        "    cv = KFold(n_splits=num_folds, random_state=123, shuffle=True)\n",
        "    results_train, results_test = [], []\n",
        "    for train_index, test_index in cv.split(X):\n",
        "        X_train, X_test, y_train, y_test = X[train_index], X[test_index], y[train_index], y[test_index]\n",
        "        clf = model.fit('''CompleteHere''', '''CompleteHere''')\n",
        "        accuracy_train = clf.score(X='''CompleteHere''', y='''CompleteHere''')\n",
        "        accuracy_test = clf.score(X='''CompleteHere''', y='''CompleteHere''')  # Return the mean accuracy\n",
        "        results_train.append(accuracy_train)\n",
        "        results_test.append(accuracy_test)\n",
        "    return '''CompleteHere''', '''CompleteHere'''\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#@title Solution 3 { display-mode: \"form\" }\n",
        "# %load solutions_exercices/cross_validation.py\n",
        "\n",
        "from sklearn.model_selection import KFold\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "\n",
        "def cross_validation(X, y, model, num_folds=5):\n",
        "    if isinstance(X, pd.DataFrame):\n",
        "        X = X.to_numpy()\n",
        "    if isinstance(y, pd.DataFrame):\n",
        "        y = y.to_numpy()\n",
        "\n",
        "    cv = KFold(n_splits=num_folds, random_state=123, shuffle=True)\n",
        "    results_train, results_test = [], []\n",
        "    for train_index, test_index in cv.split(X):\n",
        "        X_train, X_test, y_train, y_test = X[train_index], X[test_index], y[train_index], y[test_index]\n",
        "        clf = model.fit(X_train, y_train)\n",
        "        accuracy_train = clf.score(X=X_train, y=y_train)\n",
        "        accuracy_test = clf.score(X=X_test, y=y_test)  # Return the mean accuracy\n",
        "        results_train.append(accuracy_train)\n",
        "        results_test.append(accuracy_test)\n",
        "    return results_train, results_test\n"
      ],
      "metadata": {
        "id": "dowAP__FP5kO"
      },
      "execution_count": 184,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qhKThTz3SM-n"
      },
      "source": [
        "We apply a cross validation using the <code>x_train</code> data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Srv4tmGhSM_A"
      },
      "source": [
        "from sklearn.model_selection import KFold\n",
        "import numpy as np\n",
        "from sklearn.preprocessing import MinMaxScaler \n",
        "from sklearn.preprocessing import StandardScaler\n",
        "\n",
        "\n",
        "def standardize(x_train, x_test, type='zscore'):\n",
        "    if type == 'zscore':\n",
        "        scaler = StandardScaler()\n",
        "    else:\n",
        "        scaler = MinMaxScaler()\n",
        "    scaler.fit(x_train)\n",
        "    x_train_standardize = scaler.transform(x_train)\n",
        "    # We apply the same transform on the test\n",
        "    x_test_standardize = scaler.transform(x_test)\n",
        "    return x_train_standardize, x_test_standardize\n",
        "\n",
        "\n",
        "def cross_validation_with_standadization(X, y, model, num_folds=5):\n",
        "    if isinstance(X, pd.DataFrame):\n",
        "        X = X.to_numpy()\n",
        "    if isinstance(y, pd.DataFrame):\n",
        "        y = y.to_numpy()\n",
        "\n",
        "    cv = KFold(n_splits=num_folds, random_state=123, shuffle=True)\n",
        "    results_train, results_test = [], []\n",
        "    for train_index, test_index in cv.split(X):\n",
        "        X_train, X_test, y_train, y_test = X[train_index], X[test_index], y[train_index], y[test_index]\n",
        "        \n",
        "         # standardize with z-score\n",
        "        X_train, X_test = standardize(x_train=X_train, x_test=X_test, type='zscore')\n",
        "        \n",
        "        clf = model.fit(X_train, y_train)\n",
        "        accuracy_train = clf.score(X=X_train, y=y_train)\n",
        "        accuracy_test = clf.score(X=X_test, y=y_test)  # Return the mean accuracy\n",
        "        results_train.append(accuracy_train)\n",
        "        results_test.append(accuracy_test)\n",
        "    return results_train, results_test"
      ],
      "execution_count": 185,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fIqGmWecSM_H",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a03f1e16-13a1-424c-83dc-98c809e65b21"
      },
      "source": [
        "all_results_cv_train_w_std, all_results_cv_val_w_std, names = [], [], []\n",
        "for name, model in models:\n",
        "    names.append(name)\n",
        "    results_cv_train_w_std, results_cv_val_w_std = cross_validation_with_standadization(X=x_train, y=y_train, model=model, num_folds=5)\n",
        "    all_results_cv_train_w_std.append(results_cv_train_w_std)\n",
        "    all_results_cv_val_w_std.append(results_cv_val_w_std)\n",
        "    means_train, stds_train = np.mean(results_cv_train_w_std), np.std(results_cv_train_w_std)\n",
        "    means_val, stds_val = np.mean(results_cv_val_w_std), np.std(results_cv_val_w_std)\n",
        "    msg = \"%s: train = %.3f (+/- %.3f), test = %.3f (+/- %.3f)\" % (name, means_train, stds_train, means_val, stds_val)\n",
        "    print(msg)"
      ],
      "execution_count": 186,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Logistic Regression: train = 1.000 (+/- 0.000), test = 0.841 (+/- 0.078)\n",
            "Linear Discriminant Analysis: train = 1.000 (+/- 0.000), test = 0.800 (+/- 0.039)\n",
            "K-nearest neighbours: train = 0.901 (+/- 0.019), test = 0.841 (+/- 0.069)\n",
            "Decision Tree Classifier: train = 1.000 (+/- 0.000), test = 0.794 (+/- 0.053)\n",
            "Gaussian Naive Bayes: train = 0.876 (+/- 0.018), test = 0.847 (+/- 0.034)\n",
            "Support Vector Classifier: train = 0.976 (+/- 0.006), test = 0.829 (+/- 0.060)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TVDwmEvxSM_O"
      },
      "source": [
        "Compare the results of cross validation W/O standardization: "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CG8--v_2SM_S",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 697
        },
        "outputId": "4aee3d12-5bc6-4d9d-b593-beb078a7ddac"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# boxplot algorithm comparison\n",
        "plt.figure(figsize=(15,10))\n",
        "\n",
        "ax = plt.subplot(122)\n",
        "plt.title('Algorithm Comparison using cross validation with preprocessing')\n",
        "plt.boxplot(all_results_cv_val_w_std)\n",
        "ax.set_ylim([0.60, 0.95])\n",
        "ax.set_xticklabels(names, rotation=40)\n",
        "plt.show();"
      ],
      "execution_count": 187,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1080x720 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAboAAAKoCAYAAAABATqIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeZgdVZ3/8feXBAyyJhCVHdSgwSCgLToCCq6ICy6jwk9G0AjjBsqAChMVRBG3kUHFBQVR1ADucRsFDWgUkCCKAoIBBAIuQMKiggT8/v44p0mlSSeddHdu5+T9ep5++t5bdW+dWj9Vp05VRWYiSVKr1up1ASRJGk0GnSSpaQadJKlpBp0kqWkGnSSpaQadJKlpq03QRcTpEfG+UfrtV0XEj5bRfc+ImD8aw17dRcR/R8Tnel2OoYiIv0XEI3tdjrEqIs6LiNfV18tbJx7odyWGs3WdF+NWtqzDMZThR0RGxKNXZblWd2N5WzDmgq6uQAsj4iGrapiZ+eXMfE6nDKt0IY/isIj4XUT8PSLmR8RXI2LHVVWGlZWZ78/MldrgrWqZuX5mXtvrcqwOBq4TwxERf4yIZ3V++4Y6L+4fid9fUQOHP5zQ1mJjeVswpoIuIrYF9gASeNEqGub4VTGc5TgJeAtwGDAJ2B74FvD8XhZqecbItFttOL000suAy9QQZeaY+QPeDfwc+Cjw3QHdTgfe13n/duBPwM3A6yjh+OjabSPgi8AtwPXAO4G1areD6jBOBG4D3lc/m1O7/7T+1t+BvwGvBPYE5gNHAH+tw33NgLJ9EvhB/c7PgUcA/wssBH4P7DLIOE8B7gd2XcZ0Ger43A5cCzy1fn5jLe+BA8r6aeAc4C7gfGCbTveT6vfuBC4B9uh0Oxb4GvCl2v119bMv1e4TarfbalkuBh5eu20OzAIWAPOAgwf87tl1HO8CLgf6BpkW29b5M77z2XnA6+rrR9dxugO4FTir0193GTkdOBn4Xh3mRcCjOv0+B7iq/s4n62++bpAyjQP+G7im/tYlwFadYb4J+ANwXf3s4DoNFtRpsnn9POp8/Gudvr8FptVu+wBX1N+/CThyKeV4SJ3u0zqfTQbuBh4GTAS+S1mOFtbXWw4yHQ+irhP1/bMpy/EdwCe60wN4FPCTOt9vBb4MbFy7nQH8q5bhb5T1dol5OILLxnuAj9fXa1PW4Q/X9+sC91B2JB8YPnA8Zf27p5bvE5359vo6326vy0oMMtxjKevFWbWMvwJ26nT/I/AO4DLgn3W4TwF+UX/7N8CeA+bDCcAv63LwbWDSgOV/OnADZXu1FmWbcD1l2fkisFHn93bvDOtG4KDO8vKR+jt/oWwX1q3dNqUsH7fX+fIzFm9z3kFZBu+irCPP7EyHLw0o54H1928FZnTKtC7wBcpyeCVluZg/atkyWj+8UoUpC/kbgScCi6gbyc6G6X319d7An4HHAQ+lbFy7G7Ev1oVjgzrBrwamd1bg+4BD6wK3Lg9eqR/4rfp+z/qd4ygr0D7AP4CJnbLdWss9gbLSXwe8mrIRfB8we5Bxfj1w/XKmy1DG5zWdYd1AWTEfQtlg3wWs3ynrXcDTaveTBoz7AcAmddocUafzhM6CvAh4MWXlWnfAwv2fwHfqPBlXp8eGtdtPKYExAdiZsrF9Rud376nTdRxlJb9wkGmxLcsOupnAjFq+CcDuS5uvdTrcBuxax/XLwJmdlfxO4KW121vqeA8WdG+jhNJjKGG1E7BJZ5jnUDaw6wLPqMvKE+r0/zjw09rvcykhuXH9nanAZrXbn6g7HZTAesIgZTkNOL7z/k3A/9XXmwAvq/NnA+CrwLcGmY4HsXjnb1PKMvPvlOX/cMoy1925eHYdn8l1Xv9v53f/CDxrsHk4gsvGM4Df1tdPpex4XNTp9ptBhv/AeA9YVr5b58XWtUx7DzLcY+vy0T99jqSs/2t3xv/XwFZ1GdiCsuztQ1lOn13fT+6U5yZgGrAe8HUeHCBfrN3WBV5L2XY+Elgf+AZwRu1/mzrv9q9l2wTYuXY7kbKDMYmyPHwHOKF2O4ESfGvXvz0oy+RjKGG5eac8j+pMh4Hl/Gwt406UkJ9au3+AsrM0EdiSshPQftBR9joWAZvW978HDu90P53FQXda/wzprGhZ/48D7gV26HT/T+C8zgp8w4BhH8Tyg+5ulty4/hV4Sqdsn+10OxS4svN+R+D2QcZ7BoOsuLX7UMbnDwOGlSy5k3BbZ+E+nbpBr+/Xp+zRbjXI8BdS907rgvzTpazk/Qv3ayl7jo8f0M9WdRgbdD47ATi98xvndrrtANw9SHn6V6DBgu6LwCl0jlSWNl/rdPhcp9s+wO/r61cDF3S6BWXlHizorgL2HaRbUjfa9f2pwIcGTP9FdbyeQdmJeQp177nT3w11vm+4nPXoWcA1nfc/B149SL87AwsHmY4HsTjoXt1dRuv0mL+M6fFi4NLO+z8ySNCN8LLRf9S2CXAU5Sh7fp3G7wE+trRliMGDrruTdDZw1CDDPXbA9FmLJXdM/gi8ttP9HdQg6nz2Q2rNSy3PBwaM872UbUF/2R/Z6f5j4I2d94+py9R44Gjgm0spc1COeLu1GP/G4lqH4yg7148e8L1HU7Z9z6IG+SDbgv5ydmsMfgnsV19fCzy30+11jGLQjaVzdAcCP8rMW+v7r9TPlmZzyoanX/f1ppQ9kOs7n11P2YtaWv9DdVtm3td5/w/KCtTvL53Xdy/lfbffJX4X2GwZwx3K+AwcFpm5rOE/MP6Z+TdK1cTmABFxZERcGRF3RMTtlGrTTZf23aU4g7LCnhkRN0fEhyJi7frbCzLzrmWMw587r/8BTFjJ8w9vp6zEv4yIyyPitcvod+Aw+6fREstXljVxWa1ut6IcPQymO802pzMv6/S/DdgiM39CqRY8GfhrRJwSERvWXl9GCePrI+L8iPi3QYY1G3hoRDy5nvPeGfgmQEQ8NCI+ExHXR8SdlCOpjYfQ+nFp0+OB9xHx8Ig4MyJuqr/7JZZcZpb32yOybGTm3cBc4OmUGovzKTteu9XPzh9imQYb7mDrMCw5ff5FWV42X1p3ylHWyyPi9v4/yo7+ZoP0fz1lGzDYerjEMlVfjwcezuDL5mTKkf0lnTL8X/0c4MOUo8QfRcS1EXFUHbd5wFspofbXOt83f9CvLzakdYyV2yYP2ZgIuohYF3gF8PSI+HNE/JlSPbJTROy0lK/8iXK422+rzutbKXsz23Q+25pSFdAvR6TgI+PHwJYR0TdI96GMz4p6YHpFxPqUqoubI2IPSlC8glItuzHlnEx0vjvotMvMRZn5nszcgVJ19ALK0cDNwKSI2GAExuHv9f9DO589olOGP2fmwZm5OeUI6JMr0YJ2ieUrIoIll7eBbqScpxpMd5rdTGdeRsR6lCOQm2r5P5aZT6TsxW9PqRYlMy/OzH0p59q+RTnCePCASkvCsylVVftTznX3h8gRlL39J2fmhpQwgCXn79L8iSWXmWDJde79dRx3rL97AENcZhjZZQNKmD0D2IVyjvh8SpXwrpRgX5qR2B50p89alOXl5kGGcSPliG7jzt96mfmBpf0eZXosomwLlvZ7SyxTtf/7KDvAgy2bt1J2gB/XKcNGmbk+QGbelZlHZOYjKQ0D/ysinlm7fSUzd6/DTOCDg0yTZVnWNnzEjYmgo1R13E9ZuXeuf1MpJ0BfvZT+zwZeExFTI+KhwLv6O3RW9OMjYoOI2Ab4L8pe5lD9hVLfPeoy8w+U8xMz6/V660TEhIjYLyKOGqHxGWifiNg9ItYB3kupdrmRUk9/H+V8xPiIeDew4TJ+ZwkRsVdE7FiPEO6krJz/qr/9C+CEOm6Pp5xMX+FxyMxbKBvBAyJiXD1ie2BFjoiXR0T/CrSQsiL+awUH8z1gx4h4cT1yeBOdMF2KzwHvjYgp9VKRx0fEJoP0O5Oy7O5cL6F5P+U80h8j4kn1SKy/IcU9wL/qMvGqiNgoMxdRpu2yxukrlEZUr6qv+21A2bjdHhGTgGOWNyGq7wGPi4iX1ulxGEtOjw0oDTnuiIgtqOHcMej6NJLLRnU+ZZtxRWbeS62WpFTJ3TLId0ZifX9iZ/q8lXI+6sJB+v0S8MKIeG5dhifUdb+74T8gInao27fjgK/l4JdjzAQOj4jt6o7r+ymNsO6jnHt+VkS8IiLGR8QmEbFzPer8LHBiRDwMICK2iIjn1tcviIhH152aOyjb539FxGMi4hl12b2Hsjyt6PoFZZt2dERMrMvMm1fiN4ZsrATdgcDns1zf8uf+P0o1zqsGVlNk5g+Aj1GqaeaxeIH6Z/1/KGVDcS0wh7Kyn7YC5TkW+EI9pH/FSo7TijiMxVVWt1OqGl5COTkMwx+fgb5C2cgtoDQYOaB+/kNK9cXVlOqPe1ixKoVHUFqf3UlpSXU+pToTytHFtpS9z28Cx2TmuStZ/oMpG9PbKA2SftHp9iTgooj4G+VE+1tyBa+dq9XnLwc+VIexA6VK7J+DfOWjlBX3R5RxP5Vyvmhpv30uZcfs65S92kcB+9XOG1I2Pgsp0/82ShUSwH8Af6xVg6+nhNhg5b+IsrxsTmkJ3O9/a7lupawz/zfYbwz4vf7p8YFapimUc3/93kNpXHMHJRS/MeAnTgDeWdenI5cyiJFcNn5BGcf+o7crKMvxYEdzUBpk/XuU63c/tpLD/TZl52IhZV69tO6UPEgN930p5xBvoaxjb2PJ7fEZlPPIf6Y00jlsGcM+rfb/U0ojmHso2wwy8wZKlfcRlPX915SGIVDOFc4DLqzL1bmUI34o8/hcyg7MBcAnM3M2pcHRByjL0J8pNQxHL6NsgzmOUr17XR3O1xh8/Rq2qCcCV2sRMRX4HfCQAefRNEBEnE456fvOXpdldVGrouYDr6oru/SAiDiW0mjjgOX1O8TfO4/SqGNM3mVkNETEGygNVZ4+Gr8/Vo7oVlhEvCQiHhIREyl1xN8x5DRSarXSxrWK5r8p55wGq4qStAIiYrOI2C0i1oqIx1COOL85WsMbUtBFxN4RcVVEzOtvfTOg+zYR8eOIuCzK7XS6J/Lvj4hf179ZI1j2/6Q0c72GUn/8hhH8benfKMvWrcALgRfXVn2Shm8d4DOUa/x+Qqn6/eRoDWy5VZe1YcHVlIsa51NaMu2fmVd0+vkqpXXXFyLiGZS7hvxH7fa3/pY8kiStakM5otsVmJeZ19ZWTGdSTqR27UBJZSgNRAZ2lySpJ4YSdFuwZMu7+Sx5MSeUe7W9tL5+CbBBp3n1hIiYGxEXRsSLh1VaSZJW0Ejd+fpI4BMRcRClietNlPNmUG4YfFOU54D9JCJ+m5lLXKkfEYcAhwCst956T3zsYx87QsWSJI1ll1xyya2ZOXn5fa68oQTdTSx51fqWDLhrQWbeTD2iqxcsviwzb6/d+u/4cG1tNrsLA25Jk5mnUO5PSF9fX86dO3dlxkWStJqJiOuX39fwDKXq8mJgSr3qfh3Kxa1LtJ6MiE3rtUZQLh48rX4+sTbPJiI2pdxz7gokSVpFlht09dq0N1PumnElcHZmXh4Rx0VE/8NR9wSuioirKTcSPb5+PhWYGxG/oTRS+UC3taYkSaNtzN0ZxapLSVpzRMQlmTnYTe1HxGp7ZxRJkobCoJMkNc2gkyQ1zaCTJDXNoJMkNc2gkyQ1zaCTJDXNoJMkNc2gkyQ1baSeXrDaiIhh/8ZYu5uMhsZ5L62Z1rigG8IT1d2YNcp5L62ZrLqUJDXNoJMkNc2gkyQ1zaCTJDXNoJMkNc2gkyQ1zaCTJDXNoJMkNc2gkyQ1zaCTJDXNoJMkNc2gkyQ1zaCTJDXNoJMkNc2gkyQ1zaCTJDXNoJMkNc2gkyQ1zaCTJDXNoJMkNc2gkyQ1zaCTJDXNoJMkNc2gkyQ1zaCTJDXNoJMkNc2gkyQ1zaCTJDXNoJMkNc2gkyQ1zaCTJDXNoJMkNc2gkyQ1zaCTJDXNoJMkNc2gkyQ1zaCTJDXNoJMkNc2gkyQ1zaCTJDXNoJMkNc2gkyQ1zaCTJDVtfK8LIEkaPREx7N/IzBEoSe8YdJLUsOWFVESs9kG2PFZdSpKaZtBJkppm0EmSmmbQSZKaZtBJkppm0EmSmmbQSZKaZtBJkppm0EmSmmbQSZKaZtBJkppm0EmSmmbQSZKaNqSgi4i9I+KqiJgXEUctpfs2EfHjiLgsIs6LiC073Q6MiD/UvwNHsvCSJC3PcoMuIsYBJwPPA3YA9o+IHQb09hHgi5n5eOA44IT63UnAMcCTgV2BYyJi4sgVX5KkZRvKEd2uwLzMvDYz7wXOBPYd0M8OwE/q69md7s8FzsnMBZm5EDgH2Hv4xZYkaWiGEnRbADd23s+vn3X9Bnhpff0SYIOI2GSI35UkadSMVGOUI4GnR8SlwNOBm4D7h/rliDgkIuZGxNxbbrllhIokSdLQgu4mYKvO+y3rZw/IzJsz86WZuQswo352+1C+W/s9JTP7MrNv8uTJKzgKkiQNbihBdzEwJSK2i4h1gP2AWd0eImLTiOj/raOB0+rrHwLPiYiJtRHKc+pnkiStEssNusy8D3gzJaCuBM7OzMsj4riIeFHtbU/gqoi4Gng4cHz97gLgvZSwvBg4rn4mSdIqEZnZ6zIsoa+vL+fOnduz4UcEY22aaNVw3mtN1OvlPiIuycy+0RyGd0aRJDXNoJMkNc2gkyQ1zaCTJDXNoJMkNc2gkyQ1zaCTJDXNoJMkNc2gkyQ1zaCTJDXNoJMkNc2gkyQ1zaCTJDXNoJMkNc2gkyQ1zaCTJDXNoJMkNc2gkyQ1zaCTJDXNoJMkNc2gkyQ1zaCTJDWtuaCbNGkSEbHSf8Cwvj9p0qQeT4E1l/N+zTXceT/cP+f92Da+1wUYaQsXLiQzezb8/g2mVj3n/ZrLea9lae6ITpKkLoNOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktS0yMxel2EJfX19OXfu3JX/gWM3GrnCrHQZ7uh1CQYVEcP6/lhbXpawhs/7SZMmsXDhwp4Me+LEiSxYsKAnwwbW7Hm/mo97RFySmX0jWJoHD2MoG66I2Bs4CRgHfC4zPzCg+9bAF4CNaz9HZeb3I2Jb4ErgqtrrhZn5+mUNa7hBFxE93Rj3evjDtTqXv9dlX5OHvyaPe6+Hv7qP+6oIuvFDKMQ44GTg2cB84OKImJWZV3R6eydwdmZ+KiJ2AL4PbFu7XZOZO49ssSVJGpqhnKPbFZiXmddm5r3AmcC+A/pJYMP6eiPg5pEroiRJK28oQbcFcGPn/fz6WdexwAERMZ9yNHdop9t2EXFpRJwfEXssbQARcUhEzI2IubfccsvQSy9J0nKMVKvL/YHTM3NLYB/gjIhYC/gTsHVm7gL8F/CViNhw4Jcz85TM7MvMvsmTJ49QkSRJGlrQ3QRs1Xm/Zf2sazpwNkBmXgBMADbNzH9m5m3180uAa4Dth1toSZKGaihBdzEwJSK2i4h1gP2AWQP6uQF4JkBETKUE3S0RMbk2ZiEiHglMAa4dqcJLkrQ8y211mZn3RcSbgR9SLh04LTMvj4jjgLmZOQs4AvhsRBxOaZhyUGZmRDwNOC4iFgH/Al6fmT282EaStKZp7oLx1f2akl5bncvf67KvycNfk8e918Nf3cd9VVxH5y3AJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0b3+sCjIaI6NmwJ06c2LNhA0yaNImFCxcO6zdWdvpNnDiRBQsWDGvY0spak9d7LVtzQZeZvS5CTy1cuLBn06CXGxqt2Ya7zEfEGr/taJlVl5Kkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkpg0p6CJi74i4KiLmRcRRS+m+dUTMjohLI+KyiNin0+3o+r2rIuK5I1l4SZKWZ/zyeoiIccDJwLOB+cDFETErM6/o9PZO4OzM/FRE7AB8H9i2vt4PeBywOXBuRGyfmfeP9IhIkrQ0Qzmi2xWYl5nXZua9wJnAvgP6SWDD+noj4Ob6el/gzMz8Z2ZeB8yrvydJ0ioxlKDbArix835+/azrWOCAiJhPOZo7dAW+K0nSqBmpxij7A6dn5pbAPsAZETHk346IQyJibkTMveWWW0aoSJIkDS3obgK26rzfsn7WNR04GyAzLwAmAJsO8btk5imZ2ZeZfZMnTx566SVJWo6hBN3FwJSI2C4i1qE0Lpk1oJ8bgGcCRMRUStDdUvvbLyIeEhHbAVOAX45U4SVJWp7ltrrMzPsi4s3AD4FxwGmZeXlEHAfMzcxZwBHAZyPicErDlIMyM4HLI+Js4ArgPuBNtriUJK1KUfJo7Ojr68u5c+f2uhirrYigV/O0l8N2+Gv2vB+u1bn8vS77cIcfEZdkZt8IFulBvDOKJKlpBp0kqWkGnSSpaQadJKlpy211qdVLHrMhHLtR74atnnHeDy4iht3PWG6sMpTxGy0TJ07s2bCHyqBrzbF39LoE6pF4z529bXV5bE8GPSRjOaSGa7jj1utWm6uCVZeSpKYZdJKkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkphl0kqSmGXSSpKYZdJKkpo3vdQGkkRQRPRv2xIkTezZsSYMz6NSMzOx1ESSNQVZdSpKaZtBJkppm0EmSmmbQSZKaZtBJkppm0EmSmmbQSZKaZtBJkppm0EmSmmbQSZKaZtBJkppm0EmSmmbQSZKaZtBJkppm0EmSmmbQSZKaZtBJkpo2pKCLiL0j4qqImBcRRy2l+4kR8ev6d3VE3N7pdn+n26yRLLwkScszfnk9RMQ44GTg2cB84OKImJWZV/T3k5mHd/o/FNil8xN3Z+bOI1dkSZKGbihHdLsC8zLz2sy8FzgT2HcZ/e8PzByJwkmSNFxDCbotgBs77+fXzx4kIrYBtgN+0vl4QkTMjYgLI+LFK11SSZJWwnKrLlfQfsDXMvP+zmfbZOZNEfFI4CcR8dvMvKb7pYg4BDgEYOuttx7hIkmS1mRDOaK7Cdiq837L+tnS7MeAasvMvKn+vxY4jyXP3/X3c0pm9mVm3+TJk4dQJEmShmYoQXcxMCUitouIdShh9qDWkxHxWGAicEHns4kR8ZD6elNgN+CKgd+VJGm0LLfqMjPvi4g3Az8ExgGnZeblEXEcMDcz+0NvP+DMzMzO16cCn4mIf1FC9QPd1pqSJI22WDKXeq+vry/nzp3b62JIq52IoFfrcy+HreHp9byLiEsys280h+GdUSRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gEwMyZM5k2bRrjxo1j2rRpzJw5c/lfkqTVwHIfvKr2zZw5kxkzZnDqqaey++67M2fOHKZPnw7A/vvv3+PSSdLweEQnjj/+eE499VT22msv1l57bfbaay9OPfVUjj/++F4XTZKGzSeMi3HjxnHPPfew9tprP/DZokWLmDBhAvfff38PS6YVERE9G/bEiRNZsGBBz4avwY3EcjGaOeETxrVKTJ06lTlz5izx2Zw5c5g6dWqPSqSVkZk9+zPkxq6RmL+rO4NOzJgxg+nTpzN79mwWLVrE7NmzmT59OjNmzOh10SRp2GyMogcanBx66KFceeWVTJ06leOPP96GKJKa4Dk6SVLPeI5OkqRhMugkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6CRJTTPoJElNM+gkSU0z6KqZM2cybdo0xo0bx7Rp05g5c2aviyRJGgHje12AsWDmzJnMmDGDU089ld133505c+Ywffp0APbff/8el06SNByRmb0uwxL6+vpy7ty5q3SY06ZN4+Mf/zh77bXXA5/Nnj2bQw89lN/97nertCyStCaJiEsys29Uh2HQwbhx47jnnntYe+21H/hs0aJFTJgwgfvvv3+VlkWS1iSrIug8RwdMnTqVOXPmLPHZnDlzmDp1ao9KJEkaKQYdMGPGDKZPn87s2bNZtGgRs2fPZvr06cyYMaPXRZMkDZONUVjc4OTQQw/lyiuvZOrUqRx//PE2RJGkBniOTpLUM2PmHF1E7B0RV0XEvIg4aindT4Njc2AAACAASURBVIyIX9e/qyPi9k63AyPiD/XvwJEsvCRJy7PcqsuIGAecDDwbmA9cHBGzMvOK/n4y8/BO/4cCu9TXk4BjgD4ggUvqdxeO6FhIkjSIoRzR7QrMy8xrM/Ne4Exg32X0vz/Qf1uR5wLnZOaCGm7nAHsPp8CSJK2IoQTdFsCNnffz62cPEhHbANsBP1nR70qSNBpG+vKC/YCvZeYKXWUdEYdExNyImHvLLbeMcJEkSWuyoQTdTcBWnfdb1s+WZj8WV1sO+buZeUpm9mVm3+TJk4dQJEmShmYoQXcxMCUitouIdShhNmtgTxHxWGAicEHn4x8Cz4mIiRExEXhO/UySpFViua0uM/O+iHgzJaDGAadl5uURcRwwNzP7Q28/4MzsXJiXmQsi4r2UsAQ4LjMXjOwoSJI0OC8YlyT1zJi5YFySpNWVQSdJappBJ0lqmkEnSWqaQSdJappBJ0lqmkEnSWqaQSdJappBJ0lqmkEnSWqaQSdJappBJ0lqmkEnSWqaQSdJappBJ0lqmkEnSWqaQSdJappBJ0lqmkEnSWqaQSdJappBJ0lqmkEnSWqaQSdJappBJ0lqmkEnSWqaQSdJappBJ0lqmkEnSWqaQSdJappBJ0lqmkEnSWqaQSdJappBJ0lqmkEnSWqaQSdJappBJ0lqmkEnSWqaQSdJappBJ0lqmkEnSWqaQSdJappBJ0lqmkEnSWqaQSdJappBJ0lqmkEnSWqaQSdJappBJ0lqmkEnSWqaQSdJappBJ0lqmkEnSWqaQSdJappBJ0lqmkEnSWqaQSdJappBJ0lqmkEnSWqaQSdJappBJ0lqmkEnSWqaQSdJappBJ0lqmkEnSWqaQSdJappBJ0lqmkEnSWqaQSdJappBJ0lq2pCCLiL2joirImJeRBw1SD+viIgrIuLyiPhK5/P7I+LX9W/WSBVckqShGL+8HiJiHHAy8GxgPnBxRMzKzCs6/UwBjgZ2y8yFEfGwzk/cnZk7j3C5JUkakqEc0e0KzMvMazPzXuBMYN8B/RwMnJyZCwEy868jW0xJklbOUIJuC+DGzvv59bOu7YHtI+LnEXFhROzd6TYhIubWz188zPJKkrRCllt1uQK/MwXYE9gS+GlE7JiZtwPbZOZNEfFI4CcR8dvMvKb75Yg4BDgEYOuttx6hIkmSNLQjupuArTrvt6yfdc0HZmXmosy8DriaEnxk5k31/7XAecAuAweQmadkZl9m9k2ePHmFR0KSpMEMJeguBqZExHYRsQ6wHzCw9eS3KEdzRMSmlKrMayNiYkQ8pPP5bsAVSJK0iiy36jIz74uINwM/BMYBp2Xm5RFxHDA3M2fVbs+JiCuA+4G3ZeZtEfFU4DMR8S9KqH6g21pTkqTRFpnZ6zIsoa+vL+fOndvrYkiSVoGIuCQz+0ZzGN4ZRZLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1DSDTpLUNINOktQ0g06S1LQhBV1E7B0RV0XEvIg4apB+XhERV0TE5RHxlc7nB0bEH+rfgSNVcEmShmL88nqIiHHAycCzgfnAxRExKzOv6PQzBTga2C0zF0bEw+rnk4BjgD4ggUvqdxeO/KhIkvRgQzmi2xWYl5nXZua9wJnAvgP6ORg4uT/AMvOv9fPnAudk5oLa7Rxg75EpuiRJyzeUoNsCuLHzfn79rGt7YPuI+HlEXBgRe6/AdyVJGjXLrbpcgd+ZAuwJbAn8NCJ2HOqXI+IQ4JD69m8RcdUIlWtlbArc2sPh99qaPP6O+5prTR7/Xo/7NqM9gKEE3U3AVp33W9bPuuYDF2XmIuC6iLiaEnw3UcKv+93zBg4gM08BThlyqUdRRMzNzL5el6NX1uTxd9zXzHGHNXv814RxH0rV5cXAlIjYLiLWAfYDZg3o51vUQIuITSlVmdcCPwSeExETI2Ii8Jz6mSRJq8Ryj+gy876IeDMloMYBp2Xm5RFxHDA3M2exONCuAO4H3paZtwFExHspYQlwXGYuGI0RkSRpaSIze12GMSUiDqlVqWukNXn8Hfc1c9xhzR7/NWHcDTpJUtO8BZgkqWkGnZoTEdHrMmhkOU81HAbdGFJvt6ZhiIhx2UB9vBv2on86ZGZGxFMjYr9el2lVcFvwYP3LQm39v0IMujEiItbKzPvr650jYtv+z3tZrtVJDbn7I+KhEfGRiNit12VaGXVZyPq6LyIe2esy9UpnOjwF+DAwr7clGn39y3F9/YaIeFSvyzQW1J2dJwJnRcSGK/JdN6JjQN2w/SuK7wAnAT+IiOfVz51PyxAR6wPUkNsauBC4C/hFp5/V5ggpM/8FEBFvAb4EbNTbEq163fkVEc8EjgV+k5lzB3ZvTV2O16/bgj2AP/e6TGNBRDwPOBL4YmbeuSLfdQM6BtQwmwj8F3BlZj4deDfw5Yh4dO1uVcZSRMQTgNdGxLr1oxcB383M9wDTIuJ1EbFb3Rsc08t7dx5HxLOBA4C+zLw0Iib3PxWkdUupfv4F8Btg04h4LCw+0mvYm4H5mfn/KLm+W0Q8pteFWpWWsr7uBrwMuKB2X3vIv9X+8jI29R/Fdd4fSFm4f5yZR9XP3gW8GnhcfXKEBqg7CPcBmwMLKffN+x5wCXAdsCGwD/DYzlM1xpzOUf26wOHAT4CX1M7/pDwJ5HfAJzLz0h4Vc9R1psMjgM8AvwRuAz4PfIoyDc7KzIG3IVxtdbcFEbFOZt4bES+nPBpta8pdpp4IXEG56cZ1vSvtqjGg+nYXyrgn8AVgXGa+onZbYjs6mDG9h9uqAQv2DhHxkMz8AuV+n5tExJMAMvO9wNXAF3tX2rGp/+inPv4pKOdv/h/wK+DFlKquwzLzAOBcYN2l/9LYUDfukykb9HGU+8ReB0yi3HnoMGABJfSaU59d2T8dplBuM/hV4LeUqvyNgc8Cjwf2rTs4q72IiM624DDg2Ih4JWXDPgs4A3gbZdlei1Il37TOufZNIuJ8yo7f5yjB/xZgg4g4HhZX8y+PQdcDnQX7PZQF+fMR8bnM/Cxl7/UlUZ/+kJnPB3wye0d/w52IGB8RBwN3A/8L7EK5F+tlmXkR8OSIOA/4U2Ze37sSL1/nSO6JwAmZeSPw2cw8ODN/DjyGsqIv6mExR0VtaLJvpypqQ+BDlKO5o4FDM/MvmXkB8HXKvXSbmA6dxjbHUubv1ykb9d2A72Xmlyk7PidQauCafcJCPXLrP0f5UOB44OPA6ynTZnKtlTkMeFld94fEoFtFImKdiHh+5/3LKTfCfgalynL7iDiRcm5uC8qM3AQgM//pObrF6l7/1sBFwHaUDcBPgP+jPNh3j1h8E/GzM/MtMHYbMETECzLzbuCblOc3HgkPrPCbRsThwBuAfTPzDz0s6oiLiKOAaZQdvh0iYgtgImXH5cvAEZl5SkRMiog3Z+Z3gCMz82+9K/XwDWhsszawHmWD/jTKcj2znlfeklI7cW1mvnrgd1sRERsDR3bOQ99N2el/OPB94EOZeXpEbECpyn0NcP5Qf9+gW3V2A+7qnGC9D5iTmXfUG10/lxJ8WwOfAC7svzE2lI3eKi7vWHcM8PnM/G9go1rdexalmvJAymOi3peZn4Qlm+z32lI2VIdFxJmZeTFwIvCoWn1F3YM/F9g9M68f6w1qVkREvA3YJjM/RzmSfTPwksw8l3Ke9Y7M/EXd2H8D2AzKjeZ7VeaRUkNsk4j4D2B9yg32vwvsmJnPysy7ImJ/4B/A+zvn7cfMcjxSIuJzlOeRvoayk3pEHcdFwFuBT2XmR+t6czLwlMy8IDOvHur60MxKM1Z1ZsQcYC7wwYh4GvBH4KD+PZi6R//r+vrizPy/HhR3zIuIfWoVxwXAKyPiE8D/UALirHqu85fAvCzPRwSGXpe/KnSqqybUj14KbBER765HLHOA50fEPrX/39ajuyGdeF+NrEWpyfgk5Yjmx8BjIuIVlA3cvRFxNiXkvpuZM3pX1OFbyg7OLpQnvSwE/gD8CfhA7fcNwDuBTfqrK7vn81oREdsDj6RU2W5HmQ7vj4hdgZmU9Xxa3WZ+H7i/VuUDQ1+vR+oJ41qKARumjSh7Z3dQziMdBXwEOD8i3g48BXgssFpXyYy0Aa2v1gZ2ojzA90uUBwL/mvIYqEWUOn0y86O1/xhre7+1CjqATwMXRsQZmfm3umf/o4i4jNL46BEMWBZa2MhFueB3v8x8G3Am8B5g08x8Y+2+MbA75bzqPlGukdw8M6/uWaFHSGcHZ4PMvCszz42IX9V5fxqlFuKEugO0EbBP99zyWFuWh6NWQa5NeWj35ZTamBsy86UR8XrK0e1jgfcC/0HZZp6XmR+s31+hnT6DbhR1Gp0cDexImWFfB15JqXo7Eridsme3BbB3Zt4xFjfQq1oNhPH1/GRksSgi7gSeneWxIsfUfh9N2VD8tvsbY2UaDlgpsx6dfYvS+OSPEfGzzPxjRHyVcl7qqcBHWqyuzsxLImJhRGxGOQfzGuDDEfGWzDyJshd/MHBgRCzKzAspLY9XW931OcotzHaLiOvqDtkvgIdn5j0R8W5gAuXI5pK6nIxrbTmo589fRwm3Oyk7dWsDZwNk5ucjYiowOzN3At4dERMy8576/RWu2fA6ulEWESdRFtw3ZL32J8qFn/9JObo7ITvXyLW4YK+ould/FOVk8/mUC+kflpmH1e6zgM9k5vciYhtKy6zz+o/kxqp6zuVplED+GuW87cHAuzNzbkQcQbke8KTMvKF3JR15A47M16M0k59eN2p9lOsGX5mZP4hyy6u9gC/XKv3V1sD1uS7bO1J20n5BOWo/kLKTe+OyvtuSGnbrUebzzyg7+08HfpGZZ9d+ZgML61Fe1POaK3UQ4BHdCIvO7byAh1Aal0wHtomIF1Jm5iGUFnavBXamnFPq3/NrcsFeEZl5e0Qk8CzKhuBs4CMR8VHKtWTnU/YAodweaXpm3gIrt7e3KkTEkcDLKU3HHw18i9IAaSvg8Ch3/LgReHk9ch2T47EyBoTcbpn584h4FvC1iPh9Zl4QEYcAZ0fErpl5JXBNTws9AvrX5ygthE+h3N3lj5n5qYh4EXAQ8ChKVfwewFe6329tW9BdDjJzYUQcQAn9aynXDG4L7BoRCzLz3MzcKxa3PM/u/xUetkd0Iy8iHgfsmZknR7lk4PmUBga/oWzc/pCZb4mIzTLzT70s61hSG+5k3XNbl3Ji/h+UxiZ/o1w/dSTwPEpz6ycP+P6YqfIdWJaI+BDlUof+ezUeRz03FeUG3o/JzB8u7bstqOdkvk4J87fU85JvpDSdf1xm3hIRHwFurFWYTahHq58DjgP+RTmSf1VmnhWLLxk6kVJN/8YeFXPUxZI3yTgK+Dlle/gGYDJwOnA9Zf3eFPhoZl5T+x/2ka1BN8LqntpHgXdm5pn1sylZr3+qJ1qnAG/v7OU2t2FbUQP2+jeuR3WbUU5GXwOckZnz68bh+ZTznKdmuX5uTIpyx5t/1tdnAX/NzEPr+6cAB2Xm6wd8p4nqqk5V01qUC57PAn6dmcfV7g/LzL9GxHuB12bmFr0s70ip5+D+BPwlM38fEXtQLiX6O+Wo7nzK3T3+Leut3CJiPKXq9mX9NRMtitKY7HuUWpgj6s7NVpSwu49yy7e1KbfrG9FW515eMEy1ipIoJgNvpFQ/nRkRj43yBII/RHl0zFcpG+gPdDdmhtziKtuI+DTwsYiYTjma+zDwOGDviHhEZt6fmbOAW4ENelbopYjyvLT/qa+fDPwkIk6MiNdRloun1ddQbmW1ZV0uHmh23kjIda/12iDLZR6/B26LiPdGxGnAtyLitZn5LuBnEfGY7nRYHUXE/1Ju1/VCShP5SZSLv39HaRH8kcx8B6V258J6rhLK/WyT0jCjGUuZny8Ebs5y4fu69aBgE0rr84dTqnLn94fcSC4PnqMbhgF1zgncEhGXAJ+NiIsph+CbRcTewLsoTyZ498Dvrunqnv96lLtjXAb8lHLR/KMp1ZefobRQ/EdEfJNyv8cXAVf2psSDuhp4Y0TcTVmBP01pcPQG4GGUJtLfjnJN0I6UKqx/9Kqwo6VTRfVWyp1/XkS5d+NTKNePfg14EjC19r9aP0w1yoNAzwOuyMwnRrm7y7spy2lmufj7z8Df6vm6S4BzMvPv9Sd+nJmn9aLso2VAVeXjKDdcvxw4Kcrjh26mrCPPo6znXwT+nJ2bAYzkAYBBt5I6J5rXpxx1XE/Z8H6S0mz6XMqG70mUFlV3UhZ+Q46lVtdOoTQ6+Q6liflllBv5HpyZH4mIzSnnNu+u358+Vqota/VcZOatEfFUykbv+5l5Ru1+CeWasbMo14itC9yamX+PhhqddEXEBykt6Q4FyMwvUa597O9+NPU5aw1U3a/L4mvCoBy5/Ed9vQ3ltnTXAK+gPEnjuKx37AHIMX4f1pXRCbk3UVoWH5KZv4xyzeAkSivpBRFxOuUelj8f/NdGpkD+rcAf9bxmff0wysXKh1DuYnARsEun+79TNtiv6HW5x9LfgGm47YBuRwAz6uv3A5cCB3S6j+t1+ZcxXuvW//sB91MudO7v9mngyQP6H7PjMsx5uj7lfNRkyq29XgOcSrl0YhfgB8CHe13mER7/R1EaWPyUcs/VnSgBeB7lFlZQrhfrbh+iF2VdhdPk5XWarFffbwRsVF9vTdmh/Q6w1miXxXN0K6C75xkRm1LuKH82ZSV+HuUWVJdGuRHv4ynXyr0h63UhWvL8TUS8A5gVEWfUxhlQmtv3P2ByEqWl3tf7v59j5Eg4IjaKcskAEbF2RHwJ+FxE7JilEdIJwEURsXOtunku8NDub4yVcRmOem66v+HJRgBZbrh8L+XuFm+jVFFtRblp8RWUhlpv61WZR0OWFoLvoNRMnJKZv8lS+3AYMK7W4vw5FzdAWd2PYh8kHnzfyb9Q5vchEXEMZVt5WpTriF9Oqap8YZbLsUY1i2x1uRKiXP/xXMqFyp+l7MEekZnfqq0CXwKcA9ybmXe3uFAPVw2Jx1COhI+gVP2cQWl+/iPK+Y3rM/OVtf8xVcVXg3k65VzDNMrFvxOBPsoF4L+MiDOAV1FOtl+Qmd/sVXlHSyfkngPMoBzBXJ6ZZ0fEIymXCyyKiNdQqm3/Mxu4KfNgotwU4G3ACzLz5ij36rydMt7NbgM6y8EjKEfu8yktTQ+lHNV/nnKe7iWUuxjN618OVsmpnF4f3q5uf8AHKefhtq/vP0a5nGBcff8DalVF/4FLr8vc6z+WrNZaC3gCpRXeCfWzjSlHQO+nnJheH9i5+51ej0OnLP3zeS3KY4A+Dfyw0/09lKcgb1PfnwVMbW15oFz71v/6JZQH3u5Y14cLgTfVbhMpjYkuBLbudblHav4P+GytAe/fTqly/zHw8V6XeRVMk/H1/x7AVXVdvgaYMqC//0dpkPLIzmerZH2w6nI5lnJI/TDgAMqePJQ65nuBiyPiZ8BvMvMN/T1nnZtrqojYGTgvIvaKiPWzHJVdTXmw5p4R8eTMvJ2yMXw4sD9wX2b+un5/zBzJDdjzfFJm/ojy9O9JdU+ezDwGuAd4T0Q8NDNfmZlX9i9HLSwPEfF04G1R7vQD5RqoF1F2Up5GOfeyZ5TnLy6iNM7aPVfz25p1GqBNiojjI+LwKJe8DFw+/4dyMfScXHzdZHPPk4yIN0bEwzPzvlo9/z+Uy6dOpzxS6QsRsXlErB8R/0W5GPzFmXlt/2+ssvWh13sDY/mPxVW7m1H21Leq739KOR/X7fdRlLtb9L9vpqHBMKfhcyi37TqFUq31CEpr33GUe1h+g7qnT2mosG2vyzzIeIzrvH4x5Tq+6fX9IXX8nl7fTwYO63WZR2EarE+phlqbUj33IaCvdptEua3Zw+v7CyjV95v3qryjNA2m1PX/IEoV3K8o92Ed2N/4zusxUyMxwtPiU8DVnffbUG5xeCnlyQMz6zo/iXK0/5BeTQ+P6JYhMzMi9qQ87fgw4NN1z2wvYJd6grW/32sy8yp44ChktW9oMEKuAm6gPDDxZ5RbIb2Tch/Qz1D2fD8VEetk5qVZ7uI/Zi4cjnLXCrLsyU+IiPMp14Z9E3hrvUbyNMpR6qtrY5RbMvNj9ftjZlyGIyJ2ohy9volyk/KZlPOo+0a50/w4ykXwEeV5gTcD78rMmwf5ydVVH2V+n0MJvdMy86/9Hfvndy4+/9TUM+QiYr2IeFdEPDVLzdXvIuL78MBlEnsA38zM31ManzyBsjP02yxPIhnXi+lh0C1DXWE/SblObidKq7ljaog9Bzgm6sMxu1pasIerLvxfB16X5S4YsylHct+mPI5mDvCl7DzBIetuX69FuQflayKiv7XkEyh3bjiM8jTsd1KOarahPF7nLyy+2TQwdsZlOOrO3leBEzPztcB1mTmfcuH3BEp11C2Uc5PfqP2eluURO02I8sDfzSmBvg9l+T0pMz8REY+uOzwPmt8tzP9+dYfm+8CGlIfiRma+lHKHn4/X3m6mPEz3MMqjeF6VpYof6F1LY1tddgxs/VNn7HuBV2fmP6I8PuTHwMmZ+eGIeALwu+5GWot1WmLtTrlo9peUFogfoVxjtBMwKzO/1e2/ZwUeICJ2pFS7/pNyQ94nUhrQ9NXuG1L2WsdRNvZ/H+y3VmcR8Rbgrsw8LRY/naP//zOBfSmNDE6hVE3fl6v5PRsHnhuOiHMo18d9gVJ1+f4sF8H3d7s0M9/ek8KOsnqUuhllx+ZTWW+E0Om+CeVmGUfUfl4NvLT2279u9/Rcu0d0Vbe6MSJ2iHJD4TsoJ9ofExHrZrlW5qvAKyNil8z8VWbe2+KJ5hXVX8U3iMsol2PMBN6YmadkuUP9Uf0rAoydvd9Ow5HfUgLu/ZT7l54D3BARn6rd76SM2z8otyhr1aMoDU0eqK3o/P8xpUrz6cAzMvNPq3PIRbnTEbn4UVv93k55fuCdlOXhlRFxVkRcSLl0pMmQgwfWy7Uojxg6IyLG9U+biBifmbdRbrT+MeCpmfkZ4EUDdmB7Wstl0FV1wd4sIi6i3LLmp5TnRP2Bci3IK2orsu0pF0Ee2/nuGn0+LsqFwq+pr3for86tR3Nr5eLbn/0oM8/t/17/uY2xdB5r4DmELI9ROgfYqbYyPJjyzKzP1Y3cOMqztJo8mquuBP4Z5Z6OQNnARblo/tWZ+T3gYzX0VltR7rf62YjYqgbe1yNiv4iYQrkx812UpvFfojTGeT9weC6+f23L29OtKDdXX2J7l6XF5WaU+3ceQXm2YlBa2/b30/Md2JZnzHJ1N7C1GurDlFsTHU65tuuR9bzShZTzM2+nbLBPpzQ+EJCZdwCbRMQ8ys1Zb+h06w+N+cA6EbH9Ur7f8xUBlmg+vk5EnB0Rx0XEBzPzq5Qjt2exuGXZl4H3ZuYRlKq7dXtX8lH3U8rlAy+J8qiV/sYWU4ADImLrFs7H1arn11POO25IaSw1hTKvt6ZUYR9R+/19lrufXABj46hlNNXx/EOUxyr178T27/g8CnheZp6WmS/IYkxNizU26OrJ0o/XFkTj6lHHpZQN9sWUFmNnRsRGtartLZTqtydSrhf5du9KPzYM2IOdT9lAXJeZv6vdH6jiqNWAPwbmrfKCDlFdeTemnIu5gHL+7a2dxhh/opxgf2xmzgYui4hvA5dl5vt7VOwRs7Qj63pEfjnwPkoDnLdHxEsj4rWUnZqTc/W/Pu6B5bjutP07pTr2osx8L+XJ38dSjlKeGRHbDfyNsbKzNlK606RzWuLTwLZ120k9bTMNOInOjt5YPLJdIxujRMRJlMPwL1BOmt6ZmQdGxCmUvfb9M/Oi2u/3gE9n5neiPPX6w8AHM/PGHhV/TOg0RghgPcpF0o+gNL3+RWYeW/sbnwNu+TSWGp10Gsz0/38Y5S7zZ1HOKZ5fN3ZExJaUoDs9y2UQm1KuG7u8ZyMwQmLJx6p0Hxj7wPyrgT+N0lo2KM9V/E2PijwiYskH/q5NaUiTEXEiZYfmebVbH+UWZs/KzBf0rsSjq7tuRrkI/IrO+/WBf6PcCeoy4BbKkxqOzjF+P981LuiiPMb9QOAJWe5DuQ3l3oTTI+LRlD33Myit7Q4ArsrMN/WuxGNPJxS2pATCIspdIN4Z5ZKMz1Gul7ua0hT7xLFWlQEP2rhPyMx7atXqOZS73Ryemd+t3Y+iXGryj4HB3ZKIeDelKupe4BOZ+Zt4cAvEoGw7xtw8XRH9IV6r4L5OOR8/KTMPquP4VWBBZh6ylO+OmZ210RDlyeinUW6GcF53XKM8FfxJlAcfX9KpwRmz02TMHWKuAt8AbqIcyUF5nMoTIuJVlIuYX07ZcD8e+Fp/yI3Fw/FeqSG3LeW+n5+mNieOiIOz3J39KMr97s4GfjUWN4jdRicRcQLwtSg3676e0shgfGZ+NyImRnky/FTg742H3Ecoja2OpNR4HA4Pvi50LJ6DWRF1Q93fkOIRlLt3/Jhy2csrI+KzdYP9RqAvIo4d8P21xuoGfSTUo9dPUZ4yMZtOTtRxvzEzv5GZX8jM38VqcHu7NeqILpa89mcG5UGQrwX+Rnny8UspR3PnZeZ3Bn6vB0UeZuAd/QAAHrNJREFUMwZUVW5CeWL6HsDLMvO6iNiNcsRzdGZ+v57H+Edm/qWHxV6uiDiC8hSFCyjVMtdRzjkcXz9fj3Ku5u21///f3rnHbTqW6/97MHbZZBNCFEIpbVRrkd1K+o2SsFTKXmUv2TSxpGxiJVEhy6YZSXYpRVK0bEpUy2aFUGMl21HE2I1Nwzh+fxzXPe89L1PGvDPP89xzHZ/PfOZ57ue+38/1PNd9X+d5nedxHmffeq0ziuHfRdIY4rgcQRjH25AOHJ2590v4rVHPP0jS0kTt5FrimF1CVJBOsn2E0m5rHts39GzQsxh6Yf3wiqQry1O2P1SODfZ97z7QTOvFPxKWvIWioF+OrUFar6zYOtYJtfmZ/K1eTLH9nYSN9mmGmiluS0gpK7fO61udP7JjeZR0OIb0FPwGUXGBqJy0v0sn9Evb9zSwWOv1D0jI/uDWsV2A9/V6zCP43UcBaxGFjx2beSW596abxhhSP/neF/vNuvSveT6JjumupDP6fCSidTpwQBd+g06H49Qq5G6YQ61t9plkoV5OkfCZx9FjG+fsUBq24OB6MSOA4sk1yfrTJB2tKJHfSNhoqwGblvPOBHZ2CuuB/pFDezFGIWEN3kGpibT9M9JXbl1J29p+Fvhzub4z+qXNPS1pM+AiSYdK2os4Lc8Q9jGSPgfsTna5Aw2l08AqAE4pxEnADpL+X5nXx4CJhYy0OAm/X9Vc39V1wInSLE90aFclZKuvk6jNt4F3Kb0EB/o36Kyh07RKJweTxUtudbO1/WWSdD+MF3Z/HthJHUnYtqRFJF1BwruXkZzckU6h8LWkrmyLcv7PoL9ympIWGT6f5V54GPgY8A5Ju5WPfkLCV5fC0H3QLwZ7ZlBYhc3rjYH/IAXwc5MF7qHy/6GSziXzOtp235aEvBRI2pGUBJ1FOlxvQELVxwMHSlqOFDyvShb8BW0f7TSM7ZzqkVIY38b6wPVOTehGhE3Z/EYXAKMlLTwdZ3Eg0Lkc3TB67ALAuUS2ZxentX1zXpNzWgDYxPYPejPi/kOLVdn8vyYplL6YGII/kALayx1R2/2An9n+Qw+H/aIoOZi9iAc/GpjkFIC3z1kf+CapnbywdXyw8xItlIjFs+X1u0gObiLpBv0Z0nLo1vL5fMBCxREYeJQF+kTSBPa3JET9ADFury3Hd3HIKWs4NZ+dmv8GSknMzsAJxKg9S8K0OwCftX2fwqa+mgjX30PCm0/1aMgjgr7xukcCiujyQa1D6wPP2t7OKSWY6pEUIzfK9tPVyA1h2MO9NIDt/yUyV0cCv3WK5+8ADpC0ie2v9aORA3DIMI8R2bathhu5cs5VxLt/CwyFObuwyCmydr8C3idpHklnEs3GVwA/A7a1vZbtWyWtK2l3srB1xcjNXebxUBKSfJA4PuNI6cubSG+5RvGjMXKdZFbafojo995FctSXERb6IyRMuYDTmeJmUlP4jCNoP7C7OUhithMoE/E86Wq7nO0JJCw5qYQf5isTNjewhO0H3WGq+MuBpi2ePYSUDFxEauQuUTQt/6ecviBh5/2iJ4P9J2jt2Ociosu3l/+nfk5hypMXY5vPurLASVqLdBQ43WHCngw8Y/sb5fN1CQkBSVuSkpDPtSMfgw5H0m0u2w+WFMbROexzJf2O7Gj3IyzD9nUDH6puYxiz8ili3B5yhAFuVtSg3g98SlEHut696AQ+i9CJ0KWkxWw/Ul6/ity0lxK68M+BA4jCxXOKKspvgXMHffJGCsVJaOc0PwGsTlh4a5PO30eWY58jDtJ429uW86ehJ/cLJK1N5JyOIaGq7wEPeqg2chFH+q19TSfCVUpd6DGEWbyj7QmSfkSKfHciC93ixBA+Q0J4Y2z/ukdDHhEMvxdbDk8Thv93QrA5gnQdmDy9a7uC1ndfhbDKv0sIRmcB95QITRMRW4KEra9oX9ujoY8YBt7QSVqBSDadR0IR9xB5ol0JyWQh4BBSCLwc8Jjtj/RmtP2FYuDmI17u1xxZq02Ai4DtbJ8laSVgc4aMHETs+vryN/qyxrAsaF8AjrZ9Tjm2ImFaNt0GFid9xfpu/DMDSR8hTWF3JhJN85DawMlkcbuINEZ9upw/ikQ8BroDg6ZVulkfuLFxZIbl7vcj9/THbN/flcX8H0EpAj8ZOMP28eXYqsTonUaISKu5pdnar8/2y0EXDN18JC+3C9FY3FLpCP1hYCti8J4jjKolbZ9fruvMJM4sSvJ5ErCC7ZslfYOoka9WPn8j8QTnI7JYjfZh33jAw+dT0lcII/RSos/4NtLd/DnywE8CdrL9aA+GO0uh0MUft/1YcVQOBsYT2vgbiVbhyaTp7d97N9KRQ2vXshjwIxJ1GEWM/W0lhNkOza9fcrOdxHDjrTTPXdT2YcPOW5MQsf5Onoe7ZutAZxMGkoyiAoDyoJq0cL+vHHuKJNqvILqLc9m+qhq5IWhaxfb7iBrEcZKWt70PcHcJdVGIJt8lIr7Pta7rFyPXlvNqykQeB7YmLNFtiLTbTk6N30dsb2H7UfVRGcRIwZFoekxhWv6ZME7XICHMmwjx5gCSn+oEipFblhT8n2d7XeCX5L5erZwzpeToGwLS9OorBxotAk4bi1LWe5XyAkmLO0SzTYhY9V1dfB5gAHd0w0IQmxJm5ReBVxFm1QQPNUJ8C6HIXmj7/3oz4v7DsBBPm079deCVwJ4OS/U24Crbu7Wu7cswTyHKnEsKvB8ivcSWB/5q+25JGwH7kF3+U2VhnGMcnvKsfBi4xPY5kla3fVuvxzUz0LSydHMBW5I0xdm2jyznnER27ye1yRVdR/lNTgZ+T/LT1xEhgH9zhLqXIsonBxdj1+kNwEBZb0kLtYzc7sTAXVVyDRNIJf+bJO0u6QAiU3VcNXJDKIaqMXInAN+X9F+S/oVIHy1IdEAhToKa66B/2FdtT1zSEoRochbx6L9AWqz8D3C/pK3K8dNtP9l8hy481NPzwIfvVBzt1huADZQei4Nu5Nqd4Jcjt+Z5JBe5itJSCHIvv42UVHQWShnJmeX1IkSkegLp/3gOKbE5EBinsG8vAS5rjBx043mYHgZmR6fQn5ewfWp5fzIJyUwgtTCrkMldlrSIeZgkmzs7eTMKTUu5fzfpjP0lUk/zSuIoPEDYlj+xfXTPBvsPMGxH+mZS07cL8VqPIMXrXy1hqqWIAsiZtq/t1ZhnBYZFN7YjTsmdtn817LP261c6zUU7AUljSVjuaVIbdw25n19LdnG3aKjcqLNQujCcSgz7FGLYzyUkvd/ZPqictwqJfj1n+7pyrC+jNCOJQdrR/cL2qZI+Wt7PTRbp75CQzB5ESf8a4N9tf7S1qM+xUAvl93gTEbTdi9zsjxMm4mPAxwlD7zMUvcN+RMvI7UTGviFh3J5Eet99VVG8+RrJ1e1n+9qu3Qst47ULMeYrASdL2qiEZpt8lFs78oE1cpKWkvSO8nqUpHOIusvOJEz9LZKP+xZh1e5Zcrb3l2s6Nf/D8DRh1b6ViAHsS8hX5zhdGuZSyobus/2bOcnIwQAUjDfeu+2HlcaoeypMyz3IruQOR7ZmK2AzpYniE+1rezf6vsDiLioXCnvyKNJQUsCJksbZ/lMhnuxGCBzH9PvNL2lzcg9sUXJwawPvAG5XWKT/BTziFmW+S/dCs0BJOpxEMTZyqPLjgZMkvb/M6yjbz/X7fP4ztKIQfynP9XOSTiU7uPOILuNEQpV/HzF2C7olXdWl+YcwJpvQYyEfnUdIRv9KohvL2z6jODxnEfWTM9p/Y9Dvi5cM90ELhRf7R6u9C7AmsFt5vSFh0m1R3i9AFrxbgQ16Pe5++kfakRxGdu7Lkt3vla3PDyKFowuU9+8EFun1uF/C/TAPEWP+I9Hna44fSxa6/wYOax0f2PYiL/I7aNj7w8juez3SLBayuxvfzOug/2MoxTIPsAzpI7lYOfYh4Dvl9duBewkZpefjnsW/yceI9uw4YH4S4RpF0g/rUPookvzc1cAJvR5zL//17Y7OQ+GpzYj46Mcl3ed0fX418AlJ95GSgncCH7J9x5yyFX+JuJXc7B8gyefvATtL2sb2Wbb/s+zybiF915oi8L76DVth15VI7mWCI+E0H+lKsbHtS4h81ZR2TkZ9VOs3M2jmxLYlvZ3sXm+0fYik+UnY+f8Iy/TLkl5LivwHumHosPmbQkSInyBpi70Yosy/jjRRPoLs8LqOi4j83hmkXOT3tk+QNInUwF6jNJheEljaaU3UmedhRtHXZBRJnyWCq2OIx7opqQW6QdI+pCP4B4EnygJQQ5UM3cxlAXwDybndb/vzJZ+zOnCph1rq7Gb75B4O+Z9C0gdI2PVsMudHOPqb+xIi0mmNoW5d01cGeySgSHuNIX3zlqRImkk6m+zijneRwxt0DCPRnESM+GGSGpWe2wjL8njCvHzeQx2xO7egt8LVImv38yUHuTERxriD7N6+CGzqYWzzOXl97KvkbJMwb2FRYI+yIB9M4u7fVOjkpwLH2n68eRjm1EkcjmLklgEuJLvd44GVJO1KCuj/AmyiiPrSGLkX+f37AmUntzepk/pf0lXhJEXWaBzRanzv8PF3wchJ2lIp/WjwHmAv23tQSBhKqc0+pCfgu3swzFmCsqjPLemnJCx3XDl+G6kR+xdCQNsV2L5l5DrTJLeB0lWgWedcjNzctp+y/UMStYHkJ1clDu40mJPXx74xdOXmdGHLNViQsIeaSbqaottXJvjCcm1fLtCzE21GWTEMXwBucFT5byZOwibkQTiZFFXf1/4b/WIY9MJml/cSCbLliVDxaGKwTyc52i8TTcu+GP9IoYRmJzqM0VXK4VVJngrClD0VeLPtB0k5zcU9GOqIQQWtQ82udWdgdUn7SxpD8rNnki7hyxGiRSd3LeX7fbYwJ/dXYZ57SM6s2b1+huTdjyOqMBUFfWPoioeyFvBjpYD5ILJYLyipERp9K+kAPL+GCkL7ZoHuFfRCCazFSC5rVUWhfwrRffwuqTFcGviS+1DXrvHGld5pByvdoZcuObfVgVMdhYt7CJ16C9t/a0LXPRz6iKLM6d9tXylpe7LQLQl8BThS0orlvl8IWLR89/G9HPPMYlgecj1JHyP1sG+Q9EvSHHQRYFvgPbYvIOLjE7oa1SnPwwRCJvsbcVR/3j6nMXi2n7X9F9v72n78RRzGORZ9Q0YppIhvE4/kcXJTH0Nu6kuV0oJVSJHzIbR6i83paHl2R5Gi+p2VgvrRwKaSzrY9SdKVpN7mz/0a2ikOz3Jkt3Y5EQM4pORlFgdeJ2kvYDtgV5d6oObaHgx5RNFa7KeUXdxk0lZqdZKHOZGUTlwq6fskXPnZLnz3Vj5uH9JK6EDbz0r6EGFZji+fL0V295SdbCehIYGHeYiU17uAux2N1nnKbzPdPHS/PuO9QE894GEexytIUfiPiBjz7sCKwApEz3JPEpPfCliZaBpWFCg1RauRXCakBOMWQrn+EKS7sO2fODVI/bz7WY+EpcYSMeKzHJm3Y4EbSd7xcA8Vvfbzd5khtBb77Ujt05ts305EypciO5hvEENwHRGo/mmvxjsSGBZ2X5UQKd4KXNI4uLbHS3qzUu+5DFHc7zSKkXsNyUNPIYbunZI+b/vZclrfbFb6GbP9R1J6Xx1k+3APdf99nuzQNpc01vYNwJNK8esitp+Q9DTpIfVm4L1zurcyjJE2L2moubUjxrxA+f9sQuLYSNK1tv/SXN8vO4DpsOMWJaSK3Yg+5SmFXLOy7eNbnm6z++mL7zIzkLQhcIvTCXttUhu6ie2/FY/+arK721HSHiSE+9w/+JMDAw8pGK1IJP0WLvfuI8CrgdGStiBqJ9e59Eybzr0z0Gju6bIJeDUpETrd9inl822BixXB9beQUO7+PRvwgGC2e8Ll4dxb6RfW3OSjnFYwhwBnSXqLIvWzTjPGct3ltnfo2s09I2gS9eVheLOkNzhdkpclLDzK7geSyP8aIe/85UX/YA/RZsdJGiNp6xKuu5DkIs5qHnDgBBKGnSpK3ZXcrFIDtkIxcnMRgzY3sL6kQ8nvcR0JYV5Lymk6YeQKwUKEdLYP6XD9CeAuYKztLUk6Y3nbv+64kZvaXsf2lJKbO5GkbyjH/0BC2FsSsepjejHWQUNP6ugkLU1izgfaPq14LyohtQPIru31pOv192f7APsYLY9vO9KWaGfbVyiF9RuT8O/3JB1JQr2b2n6mh0P+pygsus1J6cCihHCxDOk48AtCQrrR9l69GuOsgqYVqB5DyBffIXO7OhHm/SUx9CeQ5sIDv4NtIGlB20+WHP3upPRlrO2/lc93ITuWrUukp5MYFqHZl7CJ/9v2dWV3u5TtjVrnTxXn7qLRH2n0rGBc0jok7/B+R4i5Ob4R8eb+antSOda5wt8ZhaSDWt7seqQ2blNH53Mh4hgsT7pI30Rynh92S+uxn1C8+HlIPvZW27tKWpwIdK8P7Ee+w2tI+Pqn5bpOPNTDFrZlSLnH9iSKcZbty1vnvpfM90dt39qL8c4KKJ0njgR2d3Q61yG7l5uB80l++XBi5O7o3UhnD4rDfyEh4z1AyFe3OCLlVwN/tP2pYdd0rpxiVqBniUxHomZf4AJFsmmypAtI7dzmnlaMd442cgUPt17fT5TJ95b0FGFXjif1M2sRjcN7ob8Mg6TVXJhzZU4nS7oK2EvSfrYnSrqCPODHAfvbvrp1fWcKgVtGbguSRz3I9jhFDOHDkp4h8l07ls8/OehG7kXuxYnEqT1U0u5lTViNCBNPJpJ165RIT+cW9FZ0pnF6Xgc8a3vr8vm7ge2KQ/BB4E+SjgHGt0KcnfpNZhV6ylazPY6UFIyX9CvS+fl9JZQxxxeBt1EIGeMkfat4tzcTFt6VwKdJUfVCDrOyH43cKsCFZSGfyrRz+mRdSgRqsf0nIsr8O2CaPFTXHuqykB1MygN+Uw4fSwrBNyXqFhcD67U+H0i070VJ20n6N9v3A0eTfOQR5dSLgAeBh21P6rqRK28bkYxHgNUUuTtIPnZh4F22HwVea/uP1fGfcfSclm37c8Rzva7lyYya0ydT0oaSjpJ0TMlTQFTpPyBpJ9un297R9i9JPmtzIoU1Ff1i5AqWJHqbzc50qiNj+yMAksaV9zeQVkEPdsnh0QvLIJYHbnO0WyVp3jJnx5CC/5Vt39v6zQYWLrJ0ki4jbMG9i9M2geRi15f0A1I7OdYpBm+u7ZqRm6u1o98dOEPR9X09CeWOlvTWQjiaQkL8UJ7vLj0Tsws9N3QAtj9sez+Y6vl1glH2cqFoUp5AJLomAvsr6jBPEk//OEnrl3N3I2zVbd0qnu4HKM0xDy1vnyREA2DqwvcmSecptVKbANtI2rh83jVmZVMSMUqpjYLM72RJqzqYLGlN0u19N9vn927EMw9Jq0paubyeGziQ1Ed+iaj3bCPpWNu/J4XvlwOftv2dck0nF/QW+Whv4P1Eseh1RARhUZKGOF/SRaQb+NhyXaOA0olnYnair7oXdDFEMaOQ9CVCrf+47TvLsbcQivX1to+RtAPwVcJGfB6YVMK9fff7SXqYKHlcRAS6dyzHlyChmSNtn9Yc68LuZXoo+advk4XsIVLU/zESdv5tOe0owka+oieDHCFI2hPYhnzHC2zfpkiYCbiA7Fp/Q4z9Hh4qI2mu7zQBTdL7Cbt2J9sXS1qBCDMvZfvw8szP66HWWX33bA8S+srQzekoXu8FpLfUQSXU5ZKw/gAxbpvYvkuR+HrY9ufLtX35ICi9A28iBJpbiMH7E+mdtlTDpmu89/Jd+ya3OLNoEQ5WIGLaXyI91a4kxKF5SUeCDUkDzf/sgJE7gUi3fRK4z0MqHo3U33/Y3l7SgoRdOYVEKjoXpmzQug9Glbzj4qRGbglg47LbH02M/mbDru3M89ArVEPXJ2iFtpYnoYqLSK7imdZDchlwse2v93a0Mwal+P/XwN3EkG9AQjQTiaTXfh4qcu8ECvnmcWK7H5S0LCmE/hMpjD7R9nclzV/meGHST60vy0FeKsqu9WhSCvH31vF9SGjyYRKWv57o1n6rkNI6ifbOVCmd2p6E8Y8igty7AfM7+rQ7kF3wVsCjXd7Rzm5UQ9dHaBm7tUnrmeOAnzYLhqRvA6e4dAtuX9ObEb90KK1FTgSaUpLlSXnLY7Yn9nZ0IwtJZxBPfRFCJDiIsGRPITJXe9r+jSJefQQJVT7Qq/GOJEpIblfbmytyf8+TVkLvLq/XIfql6xLDfnS5rtO7FknvIcbtcNIw+llSM/c34BziANxNykw6XzM4u9EXZJQ5FcOT7R7Sb/wNyWvtSVRiGrWE1XlhD7m+N3IAts8jHQluK+/vtX2nUzvXiXYikpZQRIeftb0J8c5PIwvZ0qRs4g/AcpLeRurEHuqKkSu4FVha0tsLqWxRouayOvBj0h7qattHtYxcZ+ojG2haoep1gc8SZvnFtj9JjNomJQe3B3AncL7tO4qDUDGCqD9oj6BppZ/2B+60/cMmXGH7vJLXOUjSFLI72Mz2X3s36pmD7TGS3i5pa9tnt44P/CJXFrbLSMhpi3L4Ptunlnk81/Yaijj5hoSEcqZLd/cO4a/AJcBHJE20fTcx9pCQ3WPD7v2p2qVdQuv7bUDC9jcDa0h6h1M+cyxwUwlx/4qU3+wr6SYXUYWKkUMNXfYYxci9HRjjIrw8LK7/DaJ0smt53+kQzyBDkWa7gHjqv5U0XyvsfCUwzvaZ5f0ith/v4XBnGSS9AdiFKNwcQzoSfIUs5p+w/UgPhzfbIOldhGW7FyFkfZ4ovpxFxBDOJDJ9dxdyyqrttETFyKEauh6ieHPXE9LJ/krB8OTy2Qtyb10ycoOSW5xRSPokycWsYfuvGhIt/j7w/RLC7TwkrUhCcmuTPNQTtrcvn3Vy7hs0z7Gk+Uk+7lOks8g8JCe7HDAJ+LoHvJfgoKAautmEYfT5ecvLZyVtT4gn73NqjV7UmHV9cegSJB0NfLDkpZpjF5KF7Rc9G1gPIGkRYD4PdSPojLPWQOkRdwfpvrEKId+MdvpoLkSM3AbA1qS1zjakhOjkcn2nawb7AZWMMpvgAkmbAucBYyWtbvsM0jNurKTFHMWQF5AzqpEbHDiydrcrqi+vkHQ5ydf9osdDm+0o4dmHoLOkk6+TcpGnAWzfQjoPfKu8n0S0XF8NnGb710Ty8F9LuUFVOpkNqIZuNkLSziROvy8JY/xA0uttH0tUQn4K3SBnzOmwvTnRdJwE/ND2nj0eUs/QIlh1ylmTdDDwGtvvtH1jk48tZKSlJB1bTp1IdntN/u17JJd7+fC/WTFrUEOXsxDDw42StgGuBtYjyfoJpK5qNKmr2ZOIGddJ6QAkvRJY0/aVvR5LxchD0teIKPdYSWsByxJZvksIy/LnpNziA6Ss4pTp/rGKWYpq6GYRWsXfcwEr2L6rHH89YWKNtv2UogV5MbDjMMp1nZiKij6GpK2AHciu/TWkt96CpGzreEI0ewPRrLyyXFOf7R6g1tHNIhQjtyIpFr615N2OIPTiB4BVFK2/i4Eft3d+9UGoqBgI/ITk5rYknRn+bPs+SZ8HVrJ9KXBNc3I1cr1D3dHNAhSG5ZJE8umbhJH1e0I1vomEKN9Gcjjb2r5mOn+qoqJiwFDUca7ygGnSdhnV0I0QhtOmC614XxLO2J0UC48rocz5gBVIe50JvRhvRUXFyKE87ysRQev7bX+ix0OqaKGyLkcAbdq0pGVL7dAU4I3AGCLUOq6oH5wBLGN7vO0JemHX6YqKisHDq0h3imsaI9cVDdcuoO7oZhIq/aXK6wtJMvpJIvvzNmBHQiW+ncgg3Wh7n96MtqKiYlahLetWBR76C9XQvUyU3dlk25MkLUrUDlYgbTg+A2xOWpJsDGxE1OuvsX1Cub4mpisqOoj6bPcfqqF7GZC0NKEVn2n7fklfIIbuUNvnlnNOAZYuhcNIWtj2E+V19fYqKioqZhOqoXuZKMnnVwMrkz5jpwKPksLQxwrz8tfA1U57mqZLePX2KioqKmYjqqGbQUiap4gxzwUcALwOOAm4FzibNJf8dikGXxiYYvupng24oqKiYg5HZfy9BEgaJemLAMXIzV1Cj6cB9xBV8vmB/cvrLco5TxSDV9lXFRUVFT1CNXQvAYVVubekr5T3U0qe7QHgfOB5Qi2+k9TRTGnX1FWR5oqKioreoYYuXyIKAeX3wIG2Tyu7NNl+TtKapOfU7VUNoaKioqK/ULUuXyJsPyBpC+BnksYPk+1anLTh+ElvRldRUVFRMT3UHd0MQtIngaOA5WxPLkXi8wJblnxcZVVWVFRU9BHqjm4GUaS8VgPGS7oPuNf2ZlALRSsqKir6EXVH9zIh6QfAPbb3K++nSoFVVFRUVPQPqqEbAQzvXFBRUVFR0T+o5QUziXbngoqKioqK/kPd0VVUVFRUdBp1R1dRUVFR0WlUQ1dRUVFR0WlUQ1dRUVFR0WlUQ1dRUVFR0WlUQ1dRUVFR0WlUQ1dRUVFR0Wn8f36/hUNuWjo8AAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XDeFa8mQSM_X"
      },
      "source": [
        "# 3) Overfitting and Underfitting"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kdQxGDfuSM_a"
      },
      "source": [
        "- **Overfitting**:  \n",
        "A model suffers from Overfitting when it has learned too much from the training data, and does not perform well in practice as a result. This is usually caused by the model having too much exposure to the training data.\n",
        "\n",
        "- **Underfitting**:  \n",
        "A model suffers from Underfitting when it has not learned enough from the training data, and does not perform well in practice as a result. As a direct contrast to the previous idea, this issue is caused by not letting the model learn enough from training data.\n",
        "\n",
        "<a href=\"https://arxiv.org/pdf/1811.12808.pdf\">\n",
        "  <img src=\"images/over_and_under_fitting.png\" alt=\"over_under_fitting\" class=\"center\"  height=\"400\" width=\"1000\" >\n",
        "</a>\n",
        "\n",
        "*How to detect overfitting?*  \n",
        "A key challenge with overfitting, and with machine learning in general, is that we **do not know how well our model will perform on new data until we actually test it.**\n",
        "\n",
        "To address this, we can split our initial dataset into separate training and test subsets. This method can approximate of how well our model will perform on new data. If our model does much better on the training set than on the test set, then we’re likely overfitting.\n",
        "For example, it would be a big red flag if our model saw 99% accuracy on the training set but only 55% accuracy on the test set.\n",
        "\n",
        "*How to prevent overfitting?*  \n",
        "- **Cross-validation**  \n",
        "The idea is clever: Use your initial training data to generate multiple mini train-test splits. Use these splits to tune your model.\n",
        "In standard k-fold cross-validation, we partition the data into k subsets, called folds. Then, we iteratively train the algorithm on k-1 folds while using the remaining fold as the test set (called the “holdout fold”). Cross-validation allows you to tune hyperparameters with only your original training set. This allows you to keep your test set as a truly unseen dataset for selecting your final model.\n",
        "- **Train with more data**  \n",
        "It will not work every time, but training with more data can help algorithms detect the signal better. Of course, that is not always the case. If we just add more noisy data, this technique won’t help. That is why you should always ensure your data is clean and relevant.\n",
        "- **Remove features**  \n",
        "Some algorithms have built-in feature selection. For those that do not, you can manually improve their generalizability by removing irrelevant input features.\n",
        "- **Regularization**  \n",
        "Regularization refers to a broad range of techniques for artificially forcing your model to be simpler. The method will depend on the type of learner you are using. For example, you could prune a decision tree, use dropout on a neural network, or add a penalty parameter to the cost function in regression.\n",
        "- **Also, Ensembling, early stopping ...**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F9falgMUSM_b"
      },
      "source": [
        "# 4) Hyperparameters optimization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-zRJ5ypiSM_d"
      },
      "source": [
        "Wikipedia states that \"hyperparameter tuning/optimizization is choosing a set of optimal hyperparameters for a learning algorithm\". \n",
        "So what is a hyperparameter?\n",
        "<div align=\"center\">\n",
        "    <i> a hyperparameter is a parameter whose value is set before the learning process begins </i>\n",
        "</div>\n",
        "\n",
        "Some examples of hyperparameters include penalty in logistic regression \n",
        "In sklearn, hyperparameters are passed as arguments of the constructor of the models classes.\n",
        "   \n",
        "**Tuning Strategies:**\n",
        " - Grid Search  \n",
        " Also known as an exhaustive search, Grid search looks through each combination of hyperparameters. This means that every combination of specified hyperparameter values will be tried.\n",
        " - Random Search  \n",
        " As its names suggests, Random Search uses random combinations of hyperparameters. This means that not all of the parameter values are tried, and instead, parameters will be sampled with fixed numbers of iterations.\n",
        "\n",
        "<a href=\"http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf\">\n",
        "  <img src=\"images/random_grid_search.png\" alt=\"random_grid_search class=\"center\"  height=\"400\" width=\"700\" >\n",
        "</a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nLSSI_hFSM_f"
      },
      "source": [
        "### Grid Search"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X8BEr4BvSM_h"
      },
      "source": [
        "We define a space for the Grid Search, we will work with z-score preprocessing in the pipeline"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yL7AzvrTSM_j"
      },
      "source": [
        "\"\"\"\n",
        "Create a dictionary with classifier name as a key and it's hyper parameters options as a value for grid search\n",
        "\"\"\"\n",
        "\n",
        "import numpy as np\n",
        "\n",
        "# Logistic Regression Params\n",
        "C = [x for x in np.arange(0.1, 3, 0.2)]\n",
        "penalty = [\"l1\", \"l2\"]\n",
        "fit_intercept = [True, False]\n",
        "solver = [\"saga\"]\n",
        "lr_params = {'C': C,\n",
        "             'penalty': penalty,\n",
        "             'fit_intercept': fit_intercept,\n",
        "             'solver': solver\n",
        "             }\n",
        "\n",
        "# DecisionTreeClassifier PARAMS\n",
        "criterion = ['gini', 'entropy']\n",
        "splitter = ['best', 'random']\n",
        "class_weight = [None, \"balanced\"]\n",
        "max_depth = [int(x) for x in np.linspace(10, 110, num=11)]\n",
        "max_depth.append(None)\n",
        "min_samples_split = [2, 5, 10]\n",
        "min_samples_leaf = [1, 2, 4]\n",
        "max_features = [\"auto\", \"sqrt\", \"log2\"]\n",
        "dtc_params = {'criterion': criterion,\n",
        "              'splitter': splitter,\n",
        "              'class_weight': class_weight,\n",
        "              'max_depth': max_depth,\n",
        "              'min_samples_split': min_samples_split,\n",
        "              'min_samples_leaf': min_samples_leaf,\n",
        "              'max_features': max_features\n",
        "              }\n",
        "\n",
        "# KNN PARAMS\n",
        "n_neighbors = [int(x) for x in np.linspace(start=1, stop=20, num=2)]\n",
        "weights = [\"uniform\", \"distance\"]\n",
        "algorithm = [\"auto\", \"ball_tree\", \"kd_tree\", \"brute\"]\n",
        "leaf_size = [int(x) for x in np.linspace(start=5, stop=50, num=2)]\n",
        "p = [int(x) for x in np.linspace(start=1, stop=4, num=1)]\n",
        "knn_params = {'n_neighbors': n_neighbors,\n",
        "              'weights': weights,\n",
        "              'algorithm': algorithm,\n",
        "              'leaf_size': leaf_size,\n",
        "              'p': p,\n",
        "              }\n",
        "\n",
        "# LDA PARAMS\n",
        "solver = [\"lsqr\"]\n",
        "shrinkage = [\"auto\", None, 0.1, 0.3, 0.5, 0.7, 0.9]\n",
        "lda_params = {'solver': solver,\n",
        "              'shrinkage': shrinkage\n",
        "              }\n",
        "\n",
        "# GaussianNB PARAMS\n",
        "var_smoothing = [1e-9, 1e-8, 1e-7, 1e-6, 1e-5] #Portion of the largest variance of all features that is added to variances for calculation stability\n",
        "gnb_params = {'var_smoothing': var_smoothing,\n",
        "              }\n",
        "\n",
        "# SVC PARAMS\n",
        "C = [x for x in np.arange(0.1, 2, 0.2)]\n",
        "gamma = [\"auto\"]\n",
        "kernel = [\"linear\", \"poly\", \"rbf\", \"sigmoid\"]\n",
        "degree = [1, 2, 3, 4, 5, 6]\n",
        "svc_params = {'C': C,\n",
        "              'gamma': gamma,\n",
        "              'kernel': kernel,\n",
        "              'degree': degree,\n",
        "              }\n",
        "\n",
        "hypertuned_params_gs = {\"Logistic Regression\": lr_params,\n",
        "                     \"Decision Tree Classifier\": dtc_params,\n",
        "                     \"K-nearest neighbours\": knn_params,\n",
        "                     \"Linear Discriminant Analysis\": lda_params,\n",
        "                     \"Gaussian Naive Bayes\": gnb_params,\n",
        "                     \"Support Vector Classifier\": svc_params,\n",
        "                     }\n"
      ],
      "execution_count": 188,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gNDsw331SNAD"
      },
      "source": [
        "# grid search function\n",
        "import numpy as np\n",
        "import warnings\n",
        "warnings.filterwarnings('ignore') \n",
        "\n",
        "def grid_search(X, y, name, model, param_grid, verbose=False):\n",
        "    names = []\n",
        "    names.append(name)\n",
        "    result_gs, max_val_mean_val = [], 0\n",
        "    for i, params in enumerate(param_grid): #enumerate: adds a counter to an iterable object\n",
        "        model = model.set_params(**params)\n",
        "        results_cv_train_w_std, results_cv_val_w_std = cross_validation_with_standadization(X=X, y=y, model=model, num_folds=5)\n",
        "        mean_train, std_train = np.mean(results_cv_train_w_std), np.std(results_cv_train_w_std)\n",
        "        mean_val, std_val = np.mean(results_cv_val_w_std), np.std(results_cv_val_w_std)\n",
        "        if verbose:\n",
        "            print(\"%s - iteration %i: %f (%f)\" % (name, i, mean_val, std_val))\n",
        "        if mean_val > max_val_mean_val:\n",
        "            max_val_mean_test = mean_val\n",
        "            max_val_std_val = std_val\n",
        "            max_val_mean_train = mean_train\n",
        "            max_val_std_train = std_train\n",
        "            max_i = i\n",
        "            best_params = param_grid[i] \n",
        "    msg = \"%s: Maximum value on validation = %.3f (+/- %.3f) with train = %.3f (+/- %.3f) for iteration %i with params: %s\" % (name, max_val_mean_test, max_val_std_val, max_val_mean_train, max_val_std_train, max_i, best_params)\n",
        "    print(msg)"
      ],
      "execution_count": 189,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.model_selection import ParameterGrid\n",
        "\n",
        "for name, model in models:\n",
        "    param_grid = list(ParameterGrid(hypertuned_params_gs[name]))\n",
        "    grid_search(X=x_train, y=y_train, name=name, model=model, param_grid=param_grid, verbose=False)  # you can set verbose to True to see each iteration"
      ],
      "metadata": {
        "id": "zZ0IZh9OEr3Y",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1706ebc0-e335-4808-e3d4-9ce52d7c193a"
      },
      "execution_count": 130,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Logistic Regression: Maximum value on validation = 0.871 (+/- 0.040) with train = 0.988 (+/- 0.012) for iteration 59 with params: {'C': 2.900000000000001, 'fit_intercept': False, 'penalty': 'l2', 'solver': 'saga'}\n",
            "Linear Discriminant Analysis: Maximum value on validation = 0.876 (+/- 0.060) with train = 0.901 (+/- 0.019) for iteration 6 with params: {'shrinkage': 0.9, 'solver': 'lsqr'}\n",
            "K-nearest neighbours: Maximum value on validation = 0.818 (+/- 0.101) with train = 1.000 (+/- 0.000) for iteration 31 with params: {'algorithm': 'brute', 'leaf_size': 50, 'n_neighbors': 20, 'p': 1, 'weights': 'distance'}\n",
            "Decision Tree Classifier: Maximum value on validation = 0.665 (+/- 0.090) with train = 0.835 (+/- 0.020) for iteration 2591 with params: {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': None, 'max_features': 'log2', 'min_samples_leaf': 4, 'min_samples_split': 10, 'splitter': 'random'}\n",
            "Gaussian Naive Bayes: Maximum value on validation = 0.847 (+/- 0.034) with train = 0.876 (+/- 0.018) for iteration 4 with params: {'var_smoothing': 1e-05}\n",
            "Support Vector Classifier: Maximum value on validation = 0.835 (+/- 0.069) with train = 0.851 (+/- 0.009) for iteration 239 with params: {'C': 1.9000000000000004, 'degree': 6, 'gamma': 'auto', 'kernel': 'sigmoid'}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fRxWVXo0SNAS"
      },
      "source": [
        "As it is observed, the models' performance with the grid search hyperparameters tunings has been generally improved."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XyaTmT2aSNAT"
      },
      "source": [
        "### Random Search"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3CNO78j2SNAX"
      },
      "source": [
        "We define a space for the Random Search, we will works with z-score preprocessing in the pipeline"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ew_T4PGgSNAZ"
      },
      "source": [
        "\"\"\"\n",
        "Create a dictionary with classifier name as a key and it's hyper parameters options as a value for Random search\n",
        "\"\"\"\n",
        "\n",
        "import numpy as np\n",
        "from scipy.stats import uniform\n",
        "\n",
        "# Logistic Regression Params\n",
        "# Create regularization hyperparameter distribution using uniform distribution\n",
        "C = uniform(loc=0, scale=4)\n",
        "penalty = [\"l1\", \"l2\"]\n",
        "fit_intercept = [True, False]\n",
        "solver = [\"saga\"]\n",
        "lr_params = {'C': C,\n",
        "             'penalty': penalty,\n",
        "             'fit_intercept': fit_intercept,\n",
        "             'solver': solver\n",
        "             }\n",
        "\n",
        "# DecisionTreeClassifier PARAMS\n",
        "criterion = ['gini', 'entropy']\n",
        "splitter = ['best', 'random']\n",
        "class_weight = [None, \"balanced\"]\n",
        "max_depth = list(range(10, 501))\n",
        "max_depth.append(None)\n",
        "min_samples_split = list(range(2, 101))\n",
        "min_samples_leaf = list(range(1, 50))\n",
        "max_features = [\"auto\", \"sqrt\", \"log2\"]\n",
        "dtc_params = {'criterion': criterion,\n",
        "              'splitter': splitter,\n",
        "              'class_weight': class_weight,\n",
        "              'max_depth': max_depth,\n",
        "              'min_samples_split': min_samples_split,\n",
        "              'min_samples_leaf': min_samples_leaf,\n",
        "              'max_features': max_features\n",
        "              }\n",
        "\n",
        "# KNN PARAMS\n",
        "n_neighbors = list(range(1, 101))\n",
        "weights = [\"uniform\", \"distance\"]\n",
        "algorithm = [\"auto\", \"ball_tree\", \"kd_tree\", \"brute\"]\n",
        "leaf_size = list(range(2, 101))\n",
        "p = list(range(1, 11))\n",
        "knn_params = {'n_neighbors': n_neighbors,\n",
        "              'weights': weights,\n",
        "              'algorithm': algorithm,\n",
        "              'leaf_size': leaf_size,\n",
        "              'p': p,\n",
        "              }\n",
        "\n",
        "# LDA PARAMS\n",
        "solver = [\"lsqr\"]\n",
        "shrinkage = [\"auto\", None, 0.1, 0.3, 0.5, 0.7, 0.9]\n",
        "lda_params = {'solver': solver,\n",
        "              'shrinkage': shrinkage\n",
        "              }\n",
        "\n",
        "# GaussianNB PARAMS\n",
        "var_smoothing = uniform(loc=0, scale=0.1)\n",
        "gnb_params = {'var_smoothing': var_smoothing,\n",
        "              }\n",
        "\n",
        "# SVC PARAMS\n",
        "C =  uniform(loc=0, scale=2)\n",
        "gamma = [\"auto\"]\n",
        "kernel = [\"linear\", \"poly\", \"rbf\", \"sigmoid\"]\n",
        "degree = list(range(1,11))\n",
        "svc_params = {'C': C,\n",
        "              'gamma': gamma,\n",
        "              'kernel': kernel,\n",
        "              'degree': degree,\n",
        "              }\n",
        "\n",
        "hypertuned_params_rs = {\"Logistic Regression\": lr_params,\n",
        "                     \"Decision Tree Classifier\": dtc_params,\n",
        "                     \"K-nearest neighbours\": knn_params,\n",
        "                     \"Linear Discriminant Analysis\": lda_params,\n",
        "                     \"Gaussian Naive Bayes\": gnb_params,\n",
        "                     \"Support Vector Classifier\": svc_params,\n",
        "                     }"
      ],
      "execution_count": 190,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "11OdRu3WSNAg"
      },
      "source": [
        "# random search function\n",
        "import random\n",
        "import warnings\n",
        "warnings.filterwarnings('ignore') \n",
        "\n",
        "def random_search(X, y, name, model, param_grid, nb_iterations, verbose=False):\n",
        "    best_params = []\n",
        "    names = []\n",
        "    names.append(name)\n",
        "    result_rs, max_val_mean_val = [], 0 \n",
        "    for i in range(nb_iterations):\n",
        "        # create random param from the grid dict\n",
        "        params = {key: value.rvs() if isinstance(value, type(uniform())) else random.choice(value) for key, value in param_grid.items()}\n",
        "        model = model.set_params(**params)\n",
        "        results_cv_train_w_std, results_cv_val_w_std = cross_validation_with_standadization(X=X, y=y, model=model, num_folds=5)\n",
        "        mean_train, std_train = np.mean(results_cv_train_w_std), np.std(results_cv_train_w_std)\n",
        "        mean_test, std_val = np.mean(results_cv_val_w_std), np.std(results_cv_val_w_std)\n",
        "        if verbose:\n",
        "            print(\"%s - iteration %i: %f (%f)\" % (name, i, mean_test, std_val))\n",
        "        if mean_test > max_val_mean_val:\n",
        "            max_val_mean_test = mean_test\n",
        "            max_val_std_test = std_val\n",
        "            max_val_mean_train = mean_train\n",
        "            max_val_std_train = std_train\n",
        "            max_i = i\n",
        "            best_params = params\n",
        "    msg = \"%s: Maximum value on validation = %.3f (+/- %.3f) with train = %.3f (+/- %.3f) for iteration %i with params: %s\" % (name, max_val_mean_test, max_val_std_test, max_val_mean_train, max_val_std_train, max_i, best_params)\n",
        "    print(msg)"
      ],
      "execution_count": 191,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W_eGlkYCSNAm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7545a73f-5f58-4c4d-cc25-16a89427cbf8"
      },
      "source": [
        "for name, model in models:\n",
        "    dic_grid = hypertuned_params_rs[name]\n",
        "    random_search(X=x_train, y=y_train, name=name, model=model, param_grid=dic_grid, nb_iterations=100, verbose=False)  # you can set verbose to True to see each iteration"
      ],
      "execution_count": 133,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Logistic Regression: Maximum value on validation = 0.841 (+/- 0.048) with train = 0.907 (+/- 0.012) for iteration 99 with params: {'C': 0.26754002615566996, 'penalty': 'l1', 'fit_intercept': False, 'solver': 'saga'}\n",
            "Linear Discriminant Analysis: Maximum value on validation = 0.876 (+/- 0.060) with train = 0.901 (+/- 0.019) for iteration 99 with params: {'solver': 'lsqr', 'shrinkage': 0.9}\n",
            "K-nearest neighbours: Maximum value on validation = 0.724 (+/- 0.061) with train = 1.000 (+/- 0.000) for iteration 99 with params: {'n_neighbors': 37, 'weights': 'distance', 'algorithm': 'brute', 'leaf_size': 95, 'p': 5}\n",
            "Decision Tree Classifier: Maximum value on validation = 0.841 (+/- 0.088) with train = 0.866 (+/- 0.052) for iteration 99 with params: {'criterion': 'entropy', 'splitter': 'best', 'class_weight': 'balanced', 'max_depth': 380, 'min_samples_split': 93, 'min_samples_leaf': 8, 'max_features': 'log2'}\n",
            "Gaussian Naive Bayes: Maximum value on validation = 0.841 (+/- 0.040) with train = 0.876 (+/- 0.013) for iteration 99 with params: {'var_smoothing': 0.050455176658637226}\n",
            "Support Vector Classifier: Maximum value on validation = 0.853 (+/- 0.067) with train = 0.874 (+/- 0.015) for iteration 99 with params: {'C': 0.8203436165987601, 'gamma': 'auto', 'kernel': 'sigmoid', 'degree': 8}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HOuy4k5wSNA1"
      },
      "source": [
        "Here, we have implemented our own Grid Search and Random Search function to understand the mechanism. Sklearn directly implements the functions, [Random Search](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html) and [Grid Search](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aFy8XnzzSNBP"
      },
      "source": [
        "# 5) Ensembling"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ukGBPl70SNBR"
      },
      "source": [
        "''No free lunch theorem'' states that no machine learning algorithm is universally better than the others in all domains. The goal of ensembling is to combine multiple learner to improve the applicability and get better performance. But always remember, if two models have comparable performance, then you should usually pick the simpler one. [Occam's razor](https://simple.wikipedia.org/wiki/Occam%27s_razor)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lG8LeKCESNBU"
      },
      "source": [
        "### Voting"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WpuYrDetSNBX"
      },
      "source": [
        "Voting is arguably the most straightforward way to combine multiple learners $d^{(j)}(\\cdot)$. The idea is to take a linear combination of the predictions made by the learners. For example, in multiclass classification, we have\n",
        "$$\\tilde{y}_k =\\sum_j^L w_j d^{(j)}_k(\\boldsymbol{x}), \\text{ where }w_j\\geq 0\\text{ and }\\sum_j w_j=1,$$<p>for any class $k$, where $L$ is the number of voters. This can be simplified to the <strong>plurarity vote</strong> where each voter has the same weight:</p>\n",
        "$$\\tilde{y}_k =\\sum_j \\frac{1}{L} d^{(j)}_k(\\boldsymbol{x}).$$<p> We use the <code>VotingClassifier</code> from Scikit-learn to combine several classifiers.</p>\n",
        "\n",
        "We will use the Sklearn <code>Pipeline</code> tools which allow to combine all the steps we have seen previously\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LICaKWRcSNBY"
      },
      "source": [
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.tree import DecisionTreeClassifier\n",
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis\n",
        "from sklearn.naive_bayes import GaussianNB\n",
        "from sklearn.svm import SVC\n",
        "\n",
        "pipe1 = Pipeline([['sc', StandardScaler()], ['clf', LogisticRegression(**{'C': 0.1, 'fit_intercept': True, 'penalty': 'l1', 'solver': 'saga'}, random_state=random_state)]])\n",
        "pipe2 = Pipeline([['clf', DecisionTreeClassifier(**{'class_weight': None, 'criterion': 'gini', 'max_depth': 80, 'max_features': 'log2', 'min_samples_leaf': 1, 'min_samples_split': 10, 'splitter': 'best'}, random_state=random_state)]])\n",
        "pipe3 = Pipeline([['sc', StandardScaler()], ['clf', KNeighborsClassifier(**{'algorithm': 'auto', 'leaf_size': 5, 'n_neighbors': 20, 'p': 1, 'weights': 'distance'})]])\n",
        "pipe4 = Pipeline([['sc', StandardScaler()], ['clf', LinearDiscriminantAnalysis(**{'shrinkage': 0.7, 'solver': 'lsqr'})]])\n",
        "pipe5 = Pipeline([['sc', StandardScaler()], ['clf', GaussianNB(**{'var_smoothing': 1e-09})]])\n",
        "pipe6 = Pipeline([['sc', StandardScaler()], ['clf', SVC(**{'C': 0.5, 'degree': 1, 'gamma': 'auto', 'kernel': 'sigmoid', 'probability': True}, random_state=random_state)]])"
      ],
      "execution_count": 192,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0ARQMF5DSNBe"
      },
      "source": [
        "We can estimate the performance of individual classifiers via the 5-fold CV:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GPDX2-OASNBj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "aa36a8c5-53fb-48d8-a45c-e9642c85ba67"
      },
      "source": [
        "from sklearn.model_selection import cross_validate\n",
        "\n",
        "clf_labels = ['LR', 'DTC', 'KNN', 'LDA', 'GNB', 'SVC']\n",
        "print('[Individual]')\n",
        "for pipe, label in zip([pipe1, pipe2, pipe3, pipe4, pipe5, pipe6], clf_labels):\n",
        "    results = cross_validate(estimator=pipe, X=x_train, y=y_train, cv=5, scoring='accuracy', return_train_score=True)\n",
        "    scores_val = results['test_score']\n",
        "    scores_train = results['train_score']\n",
        "    print('%s: train = %.3f (+/- %.3f), validation = %.3f (+/- %.3f)' % (label, scores_train.mean(), scores_train.std(), scores_val.mean(), scores_val.std()))"
      ],
      "execution_count": 193,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[Individual]\n",
            "LR: train = 0.890 (+/- 0.009), validation = 0.876 (+/- 0.034)\n",
            "DTC: train = 0.971 (+/- 0.010), validation = 0.794 (+/- 0.118)\n",
            "KNN: train = 1.000 (+/- 0.000), validation = 0.835 (+/- 0.092)\n",
            "LDA: train = 0.921 (+/- 0.018), validation = 0.871 (+/- 0.048)\n",
            "GNB: train = 0.879 (+/- 0.018), validation = 0.818 (+/- 0.057)\n",
            "SVC: train = 0.879 (+/- 0.012), validation = 0.888 (+/- 0.043)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VJotcstcSNBo"
      },
      "source": [
        "We combine the classifiers by <code>VotingClassifer</code> from Scikit-learn and experiment some weight combinations:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zlu_2z9WSNBp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8f34a669-056e-48eb-c355-598088373fe7"
      },
      "source": [
        "from sklearn.ensemble import VotingClassifier\n",
        "import itertools\n",
        "\n",
        "print('[Voting]')\n",
        "best_vt, best_w, best_val_score, best_train_score = None, (), -1, 1\n",
        "for a, b, c in list(itertools.permutations(range(0,3))): # try some weight combination\n",
        "    clf = VotingClassifier(estimators=[('dtc', pipe2), ('knn', pipe3), ('svc', pipe6)], \n",
        "                           voting='soft', weights=[a,b,c])\n",
        "    results = cross_validate(estimator=clf, X=x_train, y=y_train, cv=5, scoring='accuracy', return_train_score=True)\n",
        "    scores_val = results['test_score']\n",
        "    scores_train = results['train_score']\n",
        "    print('%s: train = %.3f (+/- %.3f), validation = %.3f (+/- %.3f)' % ((a,b,c), scores_train.mean(), scores_train.std(), scores_val.mean(), scores_val.std()))\n",
        "    if best_val_score < scores_val.mean() and best_train_score > scores_train.mean():\n",
        "        best_vt, best_w, best_val_score = clf, (a, b, c), scores_val.mean()\n",
        "\n",
        "print('\\nBest %s: %.3f' % (best_w, best_val_score))"
      ],
      "execution_count": 136,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[Voting]\n",
            "(0, 1, 2): train = 0.944 (+/- 0.007), validation = 0.876 (+/- 0.051)\n",
            "(0, 2, 1): train = 1.000 (+/- 0.000), validation = 0.859 (+/- 0.073)\n",
            "(1, 0, 2): train = 0.932 (+/- 0.009), validation = 0.871 (+/- 0.071)\n",
            "(1, 2, 0): train = 1.000 (+/- 0.000), validation = 0.841 (+/- 0.080)\n",
            "(2, 0, 1): train = 0.972 (+/- 0.012), validation = 0.812 (+/- 0.105)\n",
            "(2, 1, 0): train = 0.990 (+/- 0.007), validation = 0.800 (+/- 0.121)\n",
            "\n",
            "Best (0, 1, 2): 0.876\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gNd7Zji6SNBu"
      },
      "source": [
        "The best ensemble combines the <code>KNeighborsClassifier</code> and <code>SVC</code>. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gabgxFJFSNBv"
      },
      "source": [
        "# 6) Final prediction and Evaluation metrics ************************"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UIoe_ZlYSNBw"
      },
      "source": [
        "This is the moment you have been waiting for! We will finally be able to evaluate our test set. Our final model will be an ensemble and combines the <code>KNeighborsClassifier </code> and <code>SVC</code> with a z-score features preprocessing. A decision tree does not need a z-score preprocessing because by nature the algorithm is scale invariant. Now, we fit the model on all the training data as follows:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KannU_dNSNBx",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b1ab2bcb-db0a-4a72-c328-ba42bf7f0d10"
      },
      "source": [
        "from sklearn.ensemble import VotingClassifier\n",
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.tree import DecisionTreeClassifier\n",
        "from sklearn.svm import SVC\n",
        "from sklearn.metrics import roc_curve, confusion_matrix, auc, plot_confusion_matrix, plot_roc_curve\n",
        "\n",
        "pipe3 = Pipeline([['sc', StandardScaler()], ['clf', KNeighborsClassifier(**{'algorithm': 'auto', 'leaf_size': 5, 'n_neighbors': 20, 'p': 1, 'weights': 'distance'})]])\n",
        "pipe6 = Pipeline([['sc', StandardScaler()], ['clf', SVC(**{'C': 0.5000000000000001, 'degree': 1, 'gamma': 'auto', 'kernel': 'sigmoid', 'probability': True})]])\n",
        "\n",
        "clf = VotingClassifier(estimators=[('knn', pipe3), ('svc', pipe6)], voting='soft', weights=[1,2])\n",
        "clf.fit(x_train, y_train)\n",
        "classes = clf.classes_ #number of classes\n",
        "\n",
        "# Here we have the probabilty associate to each classes\n",
        "proba_test = clf.predict_proba(x_test)[:,1] # [:,1] referes to the second class HGG\n",
        "y_pred = np.where(proba_test>0.5, 1, 0) # Here we have the prediction\n",
        "\n",
        "\n",
        "# 1 -- Confusion matrix\n",
        "cm = confusion_matrix(y_test, y_pred)\n",
        "\n",
        "# 2 -- ROC curve \n",
        "fp_rates, tp_rates, _ = roc_curve(y_test, proba_test, pos_label=1)\n",
        "roc_auc = auc(fp_rates, tp_rates)\n",
        "\n",
        "tn, fp, fn, tp = [i for i in cm.ravel()]\n",
        "\n",
        "# 3 -- Calculate each metrics\n",
        "precision = tp / (tp + fp)\n",
        "recall = tp / (tp + fn)\n",
        "F1 = 2 * (precision * recall) / (precision + recall)\n",
        "accuracy = (tn + tp) / (tn + fp + fn + tp)\n",
        "\n",
        "printout = (\n",
        "        f'Precision: {round(precision, 3)} | '\n",
        "        f'Recall: {round(recall, 3)} | '\n",
        "        f'F1 Score: {round(F1, 3)} | '\n",
        "        f'Accuracy Score: {round(accuracy, 3)} | '\n",
        "        f'ROC auc: {round(roc_auc, 3)} | '\n",
        "\n",
        "    )\n",
        "print(printout)\n"
      ],
      "execution_count": 194,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Precision: 0.884 | Recall: 0.927 | F1 Score: 0.905 | Accuracy Score: 0.89 | ROC auc: 0.942 | \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hUKDgBwBASrd"
      },
      "source": [
        "Now, we evaluate the classifier based on different metrics.\n",
        "Plot confusion matrix and ROC curve:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6-skpMrqAQ11",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 541
        },
        "outputId": "5e41dc16-5ec1-47a1-98db-9a13c1c51f52"
      },
      "source": [
        "plot_confusion_matrix(clf, x_test, y_test);\n",
        "plot_roc_curve(clf, x_test, y_test);"
      ],
      "execution_count": 195,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAATIAAAEGCAYAAADmLRl+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAXSklEQVR4nO3deZQdZZ3G8e/TnSYbIRASYkAQVIRhmCFg2B2EOCLgOIIHF3AcHJiJqCgoMKLjUQSZw4xsjooSAUFFFARkkR1kAGWABMKSABIh7BCSsGQjSd/7mz+qGi6h07eqc5eq28/nnDr0rXvvW7+kTx7eeuuttxQRmJmVWVe7CzAzW1sOMjMrPQeZmZWeg8zMSs9BZmalN6zdBdTqGTsyRkwc2+4yLIeuxyvtLsFyWF5ZzMrqa1qbNj601+hYuCjb733m/Suui4h91uZ4WRQqyEZMHMv2Z36m3WVYDqMOXtzuEiyHO166ZK3bWLiowl3XbZbps92THh2/1gfMoFBBZmbFF0CVarvLeBMHmZnlEgSrolhDCg4yM8vNPTIzK7UgqBTs1kYHmZnlVsVBZmYlFkDFQWZmZecemZmVWgCrPEZmZmUWhE8tzazkAirFyjEHmZnlk8zsLxYHmZnlJCqs1X3nDecgM7NcksF+B5mZlVgyj6xYQeaFFc0st2oo0zYQSSMk3SXpPkmzJX0n3X+epMclzUq3yfXqcY/MzHJpYI9sBTA1IpZI6gFul3RN+t6xEfHbrA05yMwsl0BUGnAyF8lDdZekL3vSbVATO3xqaWa55Ti1HC9pRs02rbYdSd2SZgHzgRsi4s70rZMk3S/pdEnD69XjHpmZ5RKIldGd9eMLImLKGtuKqACTJa0PXCZpW+DrwPPAOsB04GvACQMdxD0yM8slmRDblWnL3GbEy8AfgH0i4rlIrAB+BuxU7/sOMjPLrZJOiq23DUTShLQnhqSRwAeBhyVNSvcJ2B94sF49PrU0s1wiRCUa0geaBJwvqZukU3VRRFwl6WZJEwABs4DD6zXkIDOz3KoNmH4REfcD2/ezf2rethxkZpZLMthfrOgoVjVmVnh9g/1F4iAzs9wqvmnczMqsUTP7G8lBZma5VRtz1bJhHGRmlkty07iDzMxKLBCrst+i1BIOMjPLJYJGTYhtGAeZmeWkhkyIbSQHmZnlErhHZmYdwIP9ZlZqQf31+FvNQWZmuSSPgytWdBSrGjMrAT+g18xKLvDMfjPrAO6RmVmpRcg9MjMrt2Sw37comVmpNWzN/oZxkJlZLslgv8fIzKzkPLPfzEqtiDP7ixWrZlYKjXjSuKQRku6SdJ+k2ZK+k+7fQtKdkuZK+o2kderV4yAzs1wiYFW1K9NWxwpgakRsB0wG9pG0C/BfwOkR8W7gJeCweg05yMwsl+TUsivTNmA7iSXpy550C2Aq8Nt0//nA/vVqcpCZWW6V9H7LehswXtKMmm1abTuSuiXNAuYDNwB/AV6OiN70I08Dm9Srx4P9jTS/l57vLYCXKiCo7jeGygHrMeyk+eipVQBoaZUY3cWqn9T93Vgb/OyaP7F8WTeViqhWxJEH7djukgon5/SLBRExZY1tRVSAyZLWBy4Dth5MTU0NMkn7AN8HuoGzI+LkZh6v7bqhd9oGxJbDYVmVni8+S3WHEfT+x0ZvfOSsRTDaHeEiO+6w7Xn15brjy0NY429RioiXJf0B2BVYX9KwtFf2duCZet9v2r8oSd3Aj4B9gW2AgyRt06zjFcKGw5IQAxjVRWzWAwsqb7wfQff/LqW61+j21GfWINV03f5620AkTUh7YkgaCXwQeAj4A3Bg+rFDgMvr1dPMHtlOwNyIeAxA0q+BjwJzmnjM4nh+FV1zV9K79fDXd+mBFcQG3cQmPW0szAYSwHfPmkWEuObijbn2Eg8BrC65atmQey0nAeennZ4u4KKIuErSHODXkr4L3AucU6+hZgbZJsBTNa+fBnZe/UPp4N80gOEbjWliOS20vErPCS/S+/lxbzqN7L7FvbGiO/aQ97Jw/nDGjlvJSWfN4ul5o3hw5gbtLqtQGjUhNiLuB7bvZ/9jJB2hzNo+WBMR0yNiSkRM6Rk7qt3lrL3eoOeE+VSnjqb6vprQqgRdty+l8n4HWZEtnJ/0oF9ZtA533Dye92y7uM0VFVMjTi0bqZlB9gywac3rTIN2pRbBsNMWUN2sh8qBY9/0lu5ZTmzaAxN8obioho+sMHJU7+s/b7/rIp6Y6//xrK7vqmWWrVWa+a/qbmBLSVuQBNingIObeLy20+wVdN+4lOoWPXQdnmR25dANqO40iu5bllLxaWWhbTBuJd884wEAuruDW66ZyMw/btjmqoppyCysGBG9ko4AriOZfnFuRMxu1vGKILYdwYrrN+/3vd5jJ7S2GMvt+WdGcsTHcw3NDEkRoneoBBlARFwNXN3MY5hZ6xVt9QsP2JhZLl5Y0cw6goPMzEqtiAsrOsjMLLdWzhHLwkFmZrlEQG/9RRNbykFmZrn51NLMSs1jZGbWEcJBZmZl58F+Myu1CI+RmVnpiYqvWppZ2XmMzMxKzfdamln5RTJOViQOMjPLzVctzazUwoP9ZtYJinZqWaxYNbNSiFCmbSCSNpX0B0lzJM2WdGS6/3hJz0ialW771avHPTIzyyWiYdMveoGjI+IeSWOAmZJuSN87PSJOydqQg8zMcmvQA3qfA55Lf14s6SGSB3vn5lNLM8stItsGjJc0o2ab1l97kjYneer4nemuIyTdL+lcSXUf9e4emZnlEohq9quWCyJiykAfkLQucAlwVES8KunHwIkkc29PBE4FDh2oDffIzCy3yLjVI6mHJMQuiIhLASLihYioREQV+ClQ92GjDjIzyycadtVSwDnAQxFxWs3+STUfOwB4sF5JPrU0s/waM49sd+AzwAOSZqX7vgEcJGlyepR5wOfqNeQgM7PcGjH9IiJuh37vdbo6b1trDDJJP2CA3I2IL+c9mJmVXwDVannutZzRsirMrDwCKMsyPhFxfu1rSaMiYlnzSzKzoivdvZaSdpU0B3g4fb2dpDObXpmZFVej5l80SJbpF2cAHwIWAkTEfcAezSzKzIos29SLVi6HnemqZUQ8lUz5eF2lOeWYWSkU7NQyS5A9JWk3INJZuEcCDzW3LDMrrIAo2FXLLKeWhwNfJLkr/VlgcvrazIYsZdxao26PLCIWAJ9uQS1mVhYFO7XMctXynZKulPSipPmSLpf0zlYUZ2YFVcKrlr8CLgImARsDFwMXNrMoMyuwvgmxWbYWyRJkoyLiFxHRm26/BEY0uzAzK64cCyu2xED3Wo5Lf7xG0nHAr0my+JMM4qZOM+sgBbtqOdBg/0yS4OqruHYpjQC+3qyizKzYVLDB/oHutdyilYWYWUm0eCA/i0wz+yVtC2xDzdhYRPy8WUWZWZG1diA/i7pBJunbwJ4kQXY1sC9wO+AgMxuqCtYjy3LV8kDgA8DzEfEvwHbA2KZWZWbFVs24tUiWU8vlEVGV1CtpPWA+sGmT6zKzoirTwoo1Zkhan+SxTDOBJcAdTa3KzAqtNFct+0TEF9IffyLpWmC9iLi/uWWZWaGVJcgk7TDQexFxT3NKMjPLZ6Ae2akDvBfA1AbXgh5dyfC95zW6WWuiq5+dVf9DVhg7fWhxQ9ppxKmlpE1JZj9MJMmU6RHx/fSuot8Am5M81/ITEfHSQG0NNCF2r7Uv1cw6TtCoW5R6gaMj4h5JY4CZkm4APgvcFBEnp7dHHgd8baCGsky/MDN7swYs4xMRz/UNUUXEYpKVpzcBPgr0PcXtfGD/euX4SeNmlluOU8vxkmqfkTs9Iqa/pT1pc2B74E5gYkQ8l771PMmp54AcZGaWX/YgWxARUwb6gKR1gUuAoyLi1doHHUVESPVjM8sKsZL0T5K+lb7eTNJOdcs3s87VoBVi0wcaXQJcEBGXprtfkDQpfX8SyST8AWUZIzsT2BU4KH29GPhRhu+ZWQdSZN8GbCfpep0DPBQRp9W8dQVwSPrzIcDl9WrKcmq5c0TsIOlegIh4SdI6Gb5nZp2qMVctdwc+AzwgqW8ezzeAk4GLJB0GPAF8ol5DWYJslaRu0o6ipAm09HZQMyuaRswji4jbWfMz4z6Qp60sp5b/A1wGbCTpJJIlfP4zz0HMrMMU7ClKWe61vEDSTJKEFLB/RPhJ42ZDVYbxr1bLsrDiZsAy4MrafRHxZDMLM7MCK1uQAb/njYeQjAC2AB4B/rqJdZlZgalgo+RZTi3/pvZ1uirGF9bwcTOzlss9sz+9wXPnZhRjZiVRtlNLSV+tedkF7AA827SKzKzYyjjYD4yp+bmXZMzskuaUY2alUKYgSyfCjomIY1pUj5mVQVmCTNKwiOiVtHsrCzKzYhPlump5F8l42CxJVwAXA0v73qy5U93MhpKSjpGNABaSrNHfN58sAAeZ2VBVoiDbKL1i+SBvBFifgv0xzKylCpYAAwVZN7Au/d+dXrA/hpm1UplOLZ+LiBNaVomZlUeJgqwhK6eZWYeJcl21zLWwmZkNIWXpkUXEolYWYmblUaYxMjOz/jnIzKzUWryMdRYOMjPLRfjU0sw6QNGCLMtTlMzM3qxxTxo/V9J8SQ/W7Dte0jOSZqXbfvXacZCZWX6NexzcecA+/ew/PSImp9vV9RrxqaWZ5dPA1S8i4lZJm69tO+6RmVl+2Xtk4yXNqNmmZTzCEZLuT089N6j3YffIzCy3HLcoLYiIKTmb/zFwIkkUngicChw60BccZGaWWzOvWkbEC68fR/opcFW97/jU0szyyXpaOciwkzSp5uUBJGsiDsg9MjPLr0E9MkkXAnuSjKU9DXwb2FPS5PQo84DP1WvHQWZmuTRyZn9EHNTP7nPytuMgM7PcVC3W1H4HmZnl45vGzawTFO1eSweZmeXnIDOzsnOPzMzKz0FmZqVWsqcomZm9hVeINbPOEMVKMgeZmeXmHtkQ0jO8yqmXzqVnnaB7WHDb79fnF6e8rd1lWY2Vr4mjP/ZuVq3sotILf/fhV/jnY5/n3tvW5ewTN6ZaFSNHVzj6jCfZZIuV7S63GIbShFhJ5wL/AMyPiG2bdZwiW7VC/PvH38Vry7rpHhac9ru53H3zGB6+Z3S7S7NUz/Dgvy/+CyNHV+ldBV/df0t2nPoqP/j62zn+Z4+z2ZYruPK8Dbnw+2/jmDOebHe5hVG0wf5mLuNzHv2vxT2EiNeWdQMwrCfo7omiDS0MeRKMHJ38q+xdJSqrhJQMaC9bnPzuli7uZtzEVW2ssnhUzba1StN6ZI1ai7vsurqCH173ZzbefCVXnrchj9zr3ljRVCpwxIe24tl56/CRzy5g6x2WcdSpT/HNz7yT4SOqjFq3yhlX/bndZRZHULjB/rYvrChpWt963qtY0e5yGq5aFV/44FZ8+r3bsNXkZbxjq+XtLslW090NP77xES6YOYdHZo1i3sMjuGz6BL77i8e4YOYc9v7kQqYfv0m7yywURbatVdoeZBExPSKmRMSUHoa3u5ymWfpqN/f9aV123Gtxu0uxNVh3bIXtdlvC3TeP4bE5I9l6h2UAvP8fX2bODPek36SJK8QORtuDrJONHdfL6PUqAKwzosoOeyzhqbkj2lyV1Xp5YTdLXknGwlYsF/fcOoZNt1zB0le7efovyf9Yk32vtbPMQumbEFukHpmnXzTRuImrOOb7T9LVBV1dcOuVY7nzxvXaXZbVWPRCD6ccuRnVqqhWYY+PvMwuH3yVo055ihP/bXPUBWPGVvjqab5i+bqIobOwYn9rcUdE7iVsy+zxh0byxb23ancZNoB3bvMaZ97w1oH83fd9hd33faUNFZVEsXKsqVct+1uL28w6gGf2m1m5BTBUTi3NrIMVK8d81dLM8mvUVUtJ50qaL+nBmn3jJN0g6dH0vxvUa8dBZma5qRqZtgzO4623Mh4H3BQRWwI3pa8H5CAzs3yyTobNkGMRcSuwaLXdHwXOT38+H9i/XjseIzOzXJIJsZkHycZLmlHzenpETK/znYkR8Vz68/PAxHoHcZCZWX7ZV7ZYEBFTBnuYiAip/mibTy3NLDdFZNoG6QVJkwDS/86v9wUHmZnl08AxsjW4Ajgk/fkQ4PJ6X/CppZnl1Lh7Lfu7lRE4GbhI0mHAE8An6rXjIDOz/Bq0sOIAtzJ+IE87DjIzy8cP6DWzjlCwpa4dZGaWX7FyzEFmZvmpWqxzSweZmeUT5JkQ2xIOMjPLRazVZNemcJCZWX4OMjMrPQeZmZWax8jMrBP4qqWZlVz41NLMSi5wkJlZByjWmaWDzMzy8zwyMys/B5mZlVoEVIp1bukgM7P83CMzs9JzkJlZqQXQoDX7G8VBZmY5BYTHyMyszAIP9ptZB/AYmZmVXoOCTNI8YDFQAXojYspg2nGQmVlODb9pfK+IWLA2DTjIzCyfAAq2jE9XuwswsxKKyLbBeEkzarZpq7cEXC9pZj/vZeYemZnllOsWpQV1xr3eFxHPSNoIuEHSwxFxa96K3CMzs3wCIqqZtrpNRTyT/nc+cBmw02BKcpCZWX7VyLYNQNJoSWP6fgb2Bh4cTDk+tTSz/Bpz1XIicJkkSLLoVxFx7WAacpCZWT4RDblqGRGPAdutfUEOMjMbDM/sN7NyC6JSaXcRb+IgM7N8vIyPmXUEL+NjZmUWQLhHZmalFl5Y0cw6QNEG+xUFuowq6UXgiXbX0QTjgbVapsRarlN/Z++IiAlr04Cka0n+frJYEBH7rM3xsihUkHUqSTMGu2CctYd/Z+Xiey3NrPQcZGZWeg6y1pje7gIsN//OSsRjZGZWeu6RmVnpOcjMrPQcZE0kaR9Jj0iaK+m4dtdj9Uk6V9J8SYNaqdTaw0HWJJK6gR8B+wLbAAdJ2qa9VVkG5wFNn8BpjeUga56dgLkR8VhErAR+DXy0zTVZHekTfBa1uw7Lx0HWPJsAT9W8fjrdZ2YN5iAzs9JzkDXPM8CmNa/fnu4zswZzkDXP3cCWkraQtA7wKeCKNtdk1pEcZE0SEb3AEcB1wEPARRExu71VWT2SLgTuALaS9LSkw9pdk9XnW5TMrPTcIzOz0nOQmVnpOcjMrPQcZGZWeg4yMys9B1mJSKpImiXpQUkXSxq1Fm2dJ+nA9OezB7qhXdKeknYbxDHmSXrL03bWtH+1zyzJeazjJR2Tt0brDA6yclkeEZMjYltgJXB47ZuSBvWc0oj414iYM8BH9gRyB5lZqzjIyus24N1pb+k2SVcAcyR1S/qepLsl3S/pcwBK/DBdH+1GYKO+hiTdImlK+vM+ku6RdJ+kmyRtThKYX0l7g38naYKkS9Jj3C1p9/S7G0q6XtJsSWcDqveHkPQ7STPT70xb7b3T0/03SZqQ7nuXpGvT79wmaetG/GVauflJ4yWU9rz2Ba5Nd+0AbBsRj6dh8EpE7ChpOPBHSdcD2wNbkayNNhGYA5y7WrsTgJ8Ce6RtjYuIRZJ+AiyJiFPSz/0KOD0ibpe0GcndC38FfBu4PSJOkPRhIMus+EPTY4wE7pZ0SUQsBEYDMyLiK5K+lbZ9BMlDQQ6PiEcl7QycCUwdxF+jdRAHWbmMlDQr/fk24BySU767IuLxdP/ewN/2jX8BY4EtgT2ACyOiAjwr6eZ+2t8FuLWvrYhY07pcfw9sI73e4VpP0rrpMT6Wfvf3kl7K8Gf6sqQD0p83TWtdCFSB36T7fwlcmh5jN+DimmMPz3AM63AOsnJZHhGTa3ek/6CX1u4CvhQR1632uf0aWEcXsEtEvNZPLZlJ2pMkFHeNiGWSbgFGrOHjkR735dX/Dsw8RtZ5rgM+L6kHQNJ7JI0GbgU+mY6hTQL26ue7/wfsIWmL9Lvj0v2LgTE1n7se+FLfC0l9wXIrcHC6b19ggzq1jgVeSkNsa5IeYZ8uoK9XeTDJKeurwOOSPp4eQ5K2q3MMGwIcZJ3nbJLxr3vSB2icRdLzvgx4NH3v5yQrPLxJRLwITCM5jbuPN07trgQO6BvsB74MTEkvJszhjaun3yEJwtkkp5hP1qn1WmCYpIeAk0mCtM9SYKf0zzAVOCHd/2ngsLS+2Xj5cMOrX5hZB3CPzMxKz0FmZqXnIDOz0nOQmVnpOcjMrPQcZGZWeg4yMyu9/wdMcTYbxrs3LwAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de5xVdb3/8dc7RLnjUdBINDDHuCkoczAgFcpbXsBjiuIlKT14vGTHC2XpTw3Ne+bxFkL60BRBsczJVH6WF/ppKoNcRFBDDgaKiWgEIXHx8/tjrZk2w8zsNTB7jzP7/Xw85jHrvj5r75n92d/vd63vVxGBmZmVrs81dQBmZta0nAjMzEqcE4GZWYlzIjAzK3FOBGZmJW67pg6gobp06RI9evRo6jDMzJqVWbNmfRgRXWtb1+wSQY8ePaisrGzqMMzMmhVJ79S1zlVDZmYlzonAzKzEORGYmZU4JwIzsxLnRGBmVuIKlggk3SPpA0nz61gvSbdKWiRpnqT9CxWLmZnVrZAlgnuBI+pZ/w2gLP0ZC/y8gLGYmVkdCvYcQUTMkNSjnk1GAr+MpB/slyTtKKlbRCwvVEzWNB58+S88Nufdpg7DrNnr84VOXHFM30Y/blO2EewGLM2ZX5Yu24KksZIqJVWuWLGiKMFZ43lszrssWP73pg7DzOrQLJ4sjoiJwESA8vJyj6TTDPXp1omHzhrc1GGYWS2askTwLrB7znz3dJmZmRVRUyaCCuBb6d1DXwFWuX3AzKz4ClY1JGkKMAzoImkZcAXQGiAiJgBPAEcCi4C1wLcLFYsVTpaG4AXL/06fbp2KFJGZNVQh7xoanWd9AOcW6vxWHFUNwfV90Pfp1omRA2q9D8DMPgOaRWOxfba5IdiseXMXE2ZmJc6JwMysxLlqqAQ15pO+bgg2a/5cIihBjfmkrxuCzZo/lwhKlBt4zaxKpkQg6XNAf+ALwCfA/Ij4oJCBmZlZcdSbCCR9CfgBcAjwZ2AF0AbYW9Ja4C7gvoj4tNCBmplZYeQrEVxNMk7AWekDYNUk7QKcDJwG3FeY8KyKG3jNrFDqTQT1PR2cVg3d0ugRWa2yPMGblRt4zSzXVjcWSzo0Ip5uzGCsfm7gNbNC2JbbR+9utCjMzKzJ5GssrqhrFbBz44djZmbFlq9q6EDgVGBNjeUCBhUkIjMzK6p8ieAlYG1EPF9zhaQ3CxOSmZkVU767hr5Rz7qDGj8cMzMrNvc1ZGZW4pwIzMxKnBOBmVmJcyIwMytxmROBpCvrmzczs+apISWCWXnmzcysGcqcCCLit/XNm5lZ85Svi4nbgKhrfUSc3+gRmZlZUeV7sriyKFGYmVmTyfdk8WYDzkhqFxFrCxuSmZkVU6Y2AkmDJS0A3kjn+0u6s6CRmZlZUWQdmOYW4HCgAiAi5kpyX0ONIOsQlB5e0swKpSF3DS2tsWhTI8dSkqqGoMzHw0uaWaFkLREslTQECEmtge8BCwsXVmnxEJRm1pSylgj+CzgX2A14DxiQzpuZWTOXKRFExIcRcUpE7BoRXSPi1IhYmW8/SUdIelPSIkmX1LJ+D0nPSpotaZ6kI7fmIszMbOtlvWtoT0m/lbRC0geSHpO0Z559WgF3AN8A+gCjJfWpsdllwMMRsR9wEuA7kczMiixr1dCDwMNAN+ALwDRgSp59BgGLImJxRKwHpgIja2wTQNWtMJ1Jqp3MzKyIsiaCdhFxf0RsTH8eANrk2Wc3IPdOo2XpslxXAqdKWgY8AXy3tgNJGiupUlLlihUrMoZsZmZZ1JsIJO0kaSfgSUmXSOoh6YuSvk/ywb2tRgP3RkR34EjgfklbxBQREyOiPCLKu3bt2ginNTOzKvluH51FUn2jdP6snHUB/LCefd8Fds+Z754uy3UGcARARPxJUhugC/BBnrjMzKyR5OtrqOc2HHsmUCapJ0kCOAk4ucY2fwG+DtwrqTdJdZPrfszMiijrA2VI6kdy909120BE/LKu7SNio6TzgOlAK+CeiHhd0nigMiIqgIuASZIuIClhjImIOru9NjOzxpcpEUi6AhhGkgieILkl9P8BdSYCgIh4ghptCRFxec70AmBogyI2M7NGlfWuoeNJqnDej4hvA/1Jbvc0M7NmLmvV0CcR8amkjZI6kTTm7p5vp1KXpWdR9ypqZk0ta4mgUtKOwCSSO4leBf5UsKhaiCw9i7pXUTNraplKBBFxTjo5QdJTQKeImFe4sFoO9yxqZp91+Qav37++dRHxauOHZGZmxZSvRPDTetYF8LVGjMXMzJpAvgfKhhcrEDMzaxqZh6o0M7OWyYnAzKzEORGYmZW4rCOUSdKpki5P5/eQNKiwoZmZWTFkLRHcCQwmGT8AYDXJMJRmZtbMZe1i4oCI2F/SbICI+FjS9gWMy8zMiiRriWBDOhh9AEjqCnxasKjMzKxosiaCW4FHgV0k/YSkC+prChaVmZkVTda+hiZLmkXSFbWAYyNiYUEjMzOzosg6MM2twNSIcAOxmVkLk7VqaBZwmaS3Jd0kqbyQQZmZWfFkSgQRcV9EHAn8O/AmcL2kPxc0MjMzK4qGPlm8F9AL+CLwRuOHY2ZmxZb1yeIb0hLAeGA+UB4RxxQ0MjMzK4qsD5S9DQyOiA8LGUxzkWUsYvB4xGbWPOQboaxXRLwBzAT2kLRH7vpSHaGsaizifB/yHo/YzJqDfCWCC4Gx1D5SWUmPUOaxiM2spcg3QtnYdPIbEbEud52kNgWLyszMiibrXUMvZlxmZmbNTL42gs8DuwFtJe1H0r0EQCegXYFjMzOzIsjXRnA4MAboDtycs3w18KMCxWRmZkWUr43gPuA+Sd+MiF8VKSYzMyuifFVDp0bEA0APSRfWXB8RN9eym5mZNSP5qobap787bM3BJR0B/A/QCvhFRFxXyzajgCtJbkedGxEnb825GkuWh8X8oJiZtST5qobuSn//uKEHTkc0uwM4FFgGzJRUERELcrYpA34IDE2Hv9yloedpbFkeFvODYmbWkmQdj+AG4GrgE+ApYF/ggrTaqC6DgEURsTg9xlRgJLAgZ5v/BO6IiI8BIuKDBl9BAfhhMTMrJVmfIzgsIv4OHA0sIemFdFyefXYDlubML0uX5dob2FvSC5JeSquStiBprKRKSZUrVqzIGLKZmWWRNRFUlRyOAqZFxKpGOv92QBkwDBgNTJK0Y82NImJiRJRHRHnXrl0b6dRmZgbZE8Hjkt4ABgJ/kNQVWJdnn3eB3XPmu6fLci0DKiJiQ0T8L/AWSWIwM7MiyTpC2SXAEJJxCDYA/yCp76/PTKBMUk9J2wMnARU1tvkNSWkASV1IqooWZ47ezMy2WdbG4tbAqcBBkgCeBybUt09EbJR0HjCd5PbReyLidUnjgcqIqEjXHSZpAbAJGBcRK7f6aszMrMGyDkzzc6A1cGc6f1q67Mz6doqIJ4Anaiy7PGc6SLq63uJhNTMzK46sieDfI6J/zvwzkuYWIiAzMyuurIlgk6QvRcTbAJL2JKnKaTY8vKSZWe2yJoJxwLOSFpN0Rf1F4NsFi6oAPLykmVnt8iaC9FbRVSRPCld1AfFmRPyzkIEVgp8YNjPbUr23j0o6E3gduA2YA/SIiHnNMQmYmVnt8pUI/hvoGxEr0naByWz5LICZmTVj+R4oWx8RKwDSzuN2KHxIZmZWTPlKBN0l3VrXfEScX5iwzMysWPIlgpo9jM4qVCBmZtY0soxZbGZmLVi+u4YmSepXx7r2kr4j6ZTChGZmZsWQr2roDuBySfsA84EVQBuSrqI7AfeQ3ElkZmbNVL6qoTnAKEkdgHKgG8lwlQsj4s0ixGdmZgWWqYuJiFgDPFfYUMzMrClkHaHMzMxaKCcCM7MS16BEIKldoQIxM7OmkSkRSBqSDif5RjrfX9KdeXYzM7NmIGuJ4GfA4cBKgIiYCxxUqKDMzKx4MlcNRcTSGoua1QhlZmZWu6wjlC2VNAQISa2B7wELCxeWmZkVS9YSwX8B5wK7Ae8CA4BzChWUmZkVT9YSwZcjYrM+hSQNBV5o/JDMzKyYspYIbsu4zMzMmpl6SwSSBgNDgK6SLsxZ1QloVcjAzMysOPJVDW0PdEi365iz/O/A8YUKyszMiidf76PPA89Lujci3ilSTGZmVkRZG4vXSroR6EsyHgEAEfG1gkRlZmZFk7WxeDJJ9xI9gR8DS4CZBYrJzMyKKGsi2Dki7gY2RMTzEfEdwKUBM7MWIGvV0Ib093JJRwHvATsVJiQzMyumrCWCqyV1Bi4CLgZ+Afx3vp0kHSHpTUmLJF1Sz3bflBSSyjPGY2ZmjSTrUJWPp5OrgOFQ/WRxnSS1Au4ADgWWATMlVUTEghrbdSTpu+jlhoVuZmaNod4SgaRWkkZLulhSv3TZ0ZJeBG7Pc+xBwKKIWBwR64GpwMhatrsKuB5Y1/DwzcxsW+WrGrobOBPYGbhV0gPATcANEbFfnn13A3K7rl6WLqsmaX9g94j4XX0HkjRWUqWkyhUrVuQ5rZmZNUS+qqFyYN+I+FRSG+B94EsRsXJbTyzpc8DNwJh820bERGAiQHl5eWzruc3M7F/ylQjWR8SnABGxDljcgCTwLrB7znz3dFmVjkA/4DlJS4CvABVuMDYzK658JYJekual0wK+lM4LiIjYt559ZwJlknqSJICTgJOrVkbEKqBL1byk54CLI6KywVdhZmZbLV8i6L21B46IjZLOA6aT9FR6T0S8Lmk8UBkRFVt7bDMzazz5Op3bpo7mIuIJ4Ikayy6vY9th23IuMzPbOpkHrzczs5bJicDMrMRlTgSS2kr6ciGDMTOz4suUCCQdA8wBnkrnB0hyY6+ZWQuQtURwJUmXEX8DiIg5JGMTmJlZM5c1EWxI7/vP5Sd8zcxagKzjEbwu6WSglaQy4HzgxcKFZWZmxZK1RPBdkvGK/wk8SNIddd7xCMzM7LMva4mgV0RcClxayGDMzKz4spYIfippoaSrqsYlMDOzliFTIoiI4SQjk60A7pL0mqTLChqZmZkVReYHyiLi/Yi4FfgvkmcKau0zyMzMmpesD5T1lnSlpNeA20juGOpe0MjMzKwosjYW3wM8BBweEe8VMB4zMyuyTIkgIgYXOhAzM2sa9SYCSQ9HxKi0Sij3SeIsI5SZmVkzkK9E8L3099GFDsTMzJpGvY3FEbE8nTwnIt7J/QHOKXx4ZmZWaFlvHz20lmXfaMxAzMysaeRrIzib5Jv/npLm5azqCLxQyMDMzKw48rURPAg8CVwLXJKzfHVEfFSwqMzMrGjyJYKIiCWSzq25QtJOTgZmZs1flhLB0cAskttHlbMugD0LFJeZmRVJvYkgIo5Of3tYSjOzFiprX0NDJbVPp0+VdLOkPQobmpmZFUPW20d/DqyV1B+4CHgbuL9gUZmZWdFkTQQbIyKAkcDtEXEHyS2kZmbWzGXtfXS1pB8CpwEHSvoc0LpwYZmZWbFkLRGcSDJw/Xci4n2SsQhuLFhUZmZWNFmHqnwfmAx0lnQ0sC4iflnQyMzMrCiy3jU0CngFOAEYBbws6fgM+x0h6U1JiyRdUsv6CyUtkDRP0h8kfbGhF2BmZtsmaxvBpcC/R8QHAJK6Ar8HHqlrB0mtgDtIOqxbBsyUVBERC3I2mw2UR8TatF+jG0iqoczMrEiythF8rioJpFZm2HcQsCgiFkfEemAqyV1H1SLi2YhYm86+hMdBNjMruqwlgqckTQempPMnAk/k2Wc3YGnO/DLggHq2P4Okg7stSBoLjAXYYw8/x2Zm1piyjlk8TtJxwFfTRRMj4tHGCkLSqUA5cHAd558ITAQoLy+P2rYxM7Otk288gjLgJuBLwGvAxRHxbsZjvwvsnjPfPV1W8xyHkLRBHBwR/8x4bDMzayT56vnvAR4HvknSA+ltDTj2TKBMUk9J2wMnARW5G0jaD7gLGFGjDcLMzIokX9VQx4iYlE6/KenVrAeOiI2SzgOmA62AeyLidUnjgcqIqCB5KK0DME0SwF8iYkSDr8LMzLZavkTQJv3WXjUOQdvc+YioNzFExBPUaFSOiMtzpg9pcMRmZtao8iWC5cDNOfPv58wH8LVCBGVmZsWTb2Ca4cUKxMzMmkbWB8rMzKyFciIwMytxTgRmZiUua++jSscqvjyd30PSoMKGZmZmxZC1RHAnMBgYnc6vJulZ1MzMmrmsnc4dEBH7S5oNEBEfp08Lm5lZM5e1RLAhHV8goHo8gk8LFpWZmRVN1kRwK/AosIuknwD/D7imYFGZmVnRZO2GerKkWcDXSbqXODYiFhY0MjMzK4pMiUDSHsBa4Le5yyLiL4UKzMzMiiNrY/HvSNoHBLQBegJvAn0LFJeZmRVJ1qqhfXLnJe0PnFOQiMzMrKi26snitPvp+sYfNjOzZiJrG8GFObOfA/YH3itIRGZmVlRZ2wg65kxvJGkz+FXjh2NmZsWWNxGkD5J1jIiLixCPmZkVWb1tBJK2i4hNwNAixWNmZkWWr0TwCkl7wBxJFcA04B9VKyPi1wWMzczMiiBrG0EbYCXJGMVVzxME4ERgZtbM5UsEu6R3DM3nXwmgShQsKrMi2bBhA8uWLWPdunVNHYpZo2jTpg3du3endevWmffJlwhaAR3YPAFUcSKwZm/ZsmV07NiRHj16INX2Z27WfEQEK1euZNmyZfTs2TPzfvkSwfKIGL9toZl9dq1bt85JwFoMSey8886sWLGiQfvle7LY/x3W4jkJWEuyNX/P+RLB17cuFDMzay7qTQQR8VGxAjErRcOHD2f69OmbLbvllls4++yz69znmms2HxNqyJAh2xTDk08+SXl5OX369GG//fbjoosuAuDKK6/kpptu2qZj58qNc9y4cfTt25dx48YxYcIEfvnLX27TsWfPns0ZZ5yx2bJjjz2Wr3zlK5stGzNmDI888shmyzp06FA9/dZbb3HkkUdSVlbG/vvvz6hRo/jrX/+6TbF99NFHHHrooZSVlXHooYfy8ccf17rdD37wA/r160e/fv146KGHtlh//vnnbxbr7bffzj333LNNsVWLiGb1M3DgwNgaoya8GKMmvLhV+1rLtWDBgiY9/1133RVjxozZbNkBBxwQzz//fJ37tG/fvtHO/9prr8Wee+4ZCxcujIiIjRs3xp133hkREVdccUXceOONjXauXJ06dYqNGzdu1b4bNmzYYtnxxx8fc+bMqZ7/+OOPo3v37tGrV694++23q5effvrpMW3atM32rXo9P/nkk9hrr72ioqKiet2zzz4br7322lbFWWXcuHFx7bXXRkTEtddeG9///ve32Obxxx+PQw45JDZs2BBr1qyJ8vLyWLVqVfX6mTNnxqmnnrrZe/+Pf/wjBgwYUOs5a/u7Biqjjs/VrM8RmLV4P/7t6yx47++Nesw+X+jEFcfUPWzH8ccfz2WXXcb69evZfvvtWbJkCe+99x4HHnggU6ZM4ZprriEiOOqoo7j++uu55JJL+OSTTxgwYAB9+/Zl8uTJdOjQgTVr1vDcc89x5ZVX0qVLF+bPn8/AgQN54IEHkMQTTzzBhRdeSPv27Rk6dCiLFy/m8ccf54YbbuDSSy+lV69eALRq1arW0sikSZOYOHEi69evZ6+99uL++++nXbt2TJs2jR//+Me0atWKzp07M2PGDF5//XW+/e1vs379ej799FN+9atfUVZWVh3niBEjWLNmDQMHDuSHP/whCxcupEOHDlx88cW8/fbbnHvuuaxYsYJ27doxadIkevXqxZgxY2jTpg2zZ89m6NCh3HzzzdWxrV69mnnz5tG/f//qZb/+9a855phj2HXXXZk6dSo/+tGP8r5XDz74IIMHD+aYY46pXjZs2LAsb3O9HnvsMZ577jkATj/9dIYNG8b111+/2TYLFizgoIMOYrvttmO77bZj33335amnnmLUqFFs2rSJcePG8eCDD/Loo49W79OuXTt69OjBK6+8wqBBg7Ypxq3qhtrMGsdOO+3EoEGDePLJJwGYOnUqo0aNYvny5fzgBz/gmWeeYc6cOcycOZPf/OY3XHfddbRt25Y5c+YwefLkLY43e/ZsbrnlFhYsWMDixYt54YUXWLduHWeddRZPPvkks2bN2uyOkqqEkc9xxx3HzJkzmTt3Lr179+buu+8GYPz48UyfPp25c+dSUVEBwIQJE/je977HnDlzqKyspHv37psdq6KiovoaTjzxxM3WjR07lttuu41Zs2Zx0003cc45/xr2ZNmyZbz44oubJQGAyspK+vXrt9myKVOmMHr0aEaPHs2UKVPyXl9DXovVq1czYMCAWn8WLFiwxfZ//etf6datGwCf//zna61q6t+/P0899RRr167lww8/5Nlnn2Xp0qVAUgU0YsSI6mPkKi8v549//GOm66uPSwRmqfq+uRfS6NGjmTp1KiNHjmTq1KncfffdzJw5k2HDhtG1a1cATjnlFGbMmMGxxx5b77EGDRpU/cE7YMAAlixZQocOHdhzzz2r7ysfPXo0EydObFCM8+fP57LLLuNvf/sba9as4fDDDwdg6NChjBkzhlGjRnHccccBMHjwYH7yk5+wbNkyjjvuOMrKyjKdY82aNbz44ouccMIJ1cv++c9/Vk+fcMIJtGrVaov9li9fXv06QfLB++c//5mvfvWrSKJ169bMnz+ffv361XpHTUPvsunYsSNz5sxp0D6556rtfIcddhgzZ85kyJAhdO3alcGDB9OqVSvee+89pk2bVl2iqGmXXXbhjTfe2KpYchW0RCDpCElvSlok6ZJa1u8g6aF0/cuSehQyHrPPopEjR/KHP/yBV199lbVr12b6VlqXHXbYoXq6VatWbNy4sd7t+/bty6xZs/Ied8yYMdx+++289tprXHHFFdVPYk+YMIGrr76apUuXMnDgQFauXMnJJ59c/a3/yCOP5JlnnskU+6effsqOO+7InDlzqn8WLlxYvb59+/a17te2bdvNngx/+OGH+fjjj+nZsyc9evRgyZIl1aWCnXfeebPG2o8++oguXbo06LVoaIlg1113Zfny5UCStHbZZZdaj3vppZcyZ84cnn76aSKCvffem9mzZ7No0SL22msvevTowdq1a9lrr72q91m3bh1t27bNG3M+BUsEaffVdwDfAPoAoyX1qbHZGcDHEbEX8DPgesxKTIcOHRg+fDjf+c53GD16NJB8s3/++ef58MMP2bRpE1OmTOHggw8GoHXr1mzYsCHz8b/85S+zePFilixZArDZHSnjxo3jmmuu4a233gKSD+MJEyZscYzVq1fTrVs3NmzYsFmV1Ntvv80BBxzA+PHj6dq1K0uXLmXx4sXsueeenH/++YwcOZJ58+ZlirNTp0707NmTadOmAcmNLHPnzs27X+/evVm0aFH1/JQpU3jqqadYsmQJS5YsYdasWUydOhVI6vwfeugh1q9fD8C9997L8OHDATj55JN58cUX+d3vfld9rBkzZjB//vzNzldVIqjtp0+fmh9xMGLECO677z4A7rvvPkaOHLnFNps2bWLlypUAzJs3j3nz5nHYYYdx1FFH8f7771dfS7t27Ta71rfeemuLarGtUcgSwSBgUUQsjoj1wFSg5iswErgvnX4E+Lr8dI+VoNGjRzN37tzqRNCtWzeuu+46hg8fTv/+/Rk4cGD1B8jYsWPZd999OeWUUzIdu23bttx5550cccQRDBw4kI4dO9K5c2cA9t13X2655RZGjx5N79696devH4sXL97iGFdddRUHHHAAQ4cOrW5YhiSR7LPPPvTr148hQ4bQv39/Hn74Yfr168eAAQOYP38+3/rWtzK/DpMnT+buu++mf//+9O3bl8ceeyzvPr169WLVqlWsXr2aJUuW8M4772x222jPnj3p3LkzL7/8MkcffTQHHnggAwcOZMCAAbzwwgvVDbdt27bl8ccf57bbbqOsrIw+ffpw5513blbttDUuueQSnn76acrKyvj973/PJZcklSOVlZWceeaZQNLn1YEHHkifPn0YO3YsDzzwANttl7/m/oUXXuDQQw/dpvgAlNxV1PgkHQ8cERFnpvOnAQdExHk528xPt1mWzr+dbvNhjWONBcYC7LHHHgPfeeedBsfz49++DjRdPbB9Ni1cuJDevXs3dRgFt2bNGjp06EBEcO6551JWVsYFF1zQ1GE1mp/97Gd07Nix+oO1FMyePZubb76Z+++/f4t1tf1dS5oVEeW1HatZ3DUUERMjojwiyrc2O19xTF8nAStZkyZNqr7ldNWqVZx11llNHVKjOvvsszdrHykFH374IVdddVWjHKuQdw29C+yeM989XVbbNsskbQd0Jhn3wMwa0QUXXNCiSgA1tWnThtNOO62pwyiqxqgSqlLIEsFMoExST0nbAycBFTW2qQBOT6ePB56JQtVVmdXBf3LWkmzN33PBEkFEbATOA6YDC4GHI+J1SeMljUg3uxvYWdIi4EJgi1tMzQqpTZs2rFy50snAWoRIxyNo06ZNg/YrWGNxoZSXl0dlZWVTh2EthEcos5amrhHK6mss9pPFVtJat27doJGczFqiZnHXkJmZFY4TgZlZiXMiMDMrcc2usVjSCqDhjxYnugAf5t2qZfE1lwZfc2nYlmv+YkTU+kRus0sE20JSZV2t5i2Vr7k0+JpLQ6Gu2VVDZmYlzonAzKzElVoiaNiwTC2Dr7k0+JpLQ0GuuaTaCMzMbEulViIwM7ManAjMzEpci0wEko6Q9KakRZK26NFU0g6SHkrXvyypR/GjbFwZrvlCSQskzZP0B0lfbIo4G1O+a87Z7puSQlKzv9UwyzVLGpW+169LerDYMTa2DH/be0h6VtLs9O/7yKaIs7FIukfSB+kIjrWtl6Rb09djnqT9t/mkEdGifoBWwNvAnsD2wFygT41tzgEmpNMnAQ81ddxFuObhQLt0+uxSuOZ0u47ADOAloLyp4y7C+1wGzAb+LZ3fpanjLsI1TwTOTqf7AEuaOu5tvOaDgP2B+XWsPxJ4EhDwFeDlbT1nSywRDAIWRcTiiFgPTAVG1thmJHBfOv0I8HVJKmKMjS3vNUfEsxGxNp19iWTEuOYsy/sMcBVwPdAS+pnOcs3/CdwRER8DRMQHRY6xsWW55gA6pdOdgfeKGF+ji4gZwPv4inIAAAjISURBVEf1bDIS+GUkXgJ2lNRtW87ZEhPBbsDSnPll6bJat4lkAJ1VwM5Fia4wslxzrjNIvlE0Z3mvOS0y7x4RvytmYAWU5X3eG9hb0guSXpJ0RNGiK4ws13wlcKqkZcATwHeLE1qTaej/e14ej6DESDoVKAcObupYCknS54CbgTFNHEqxbUdSPTSMpNQ3Q9I+EfG3Jo2qsEYD90bETyUNBu6X1C8iPm3qwJqLllgieBfYPWe+e7qs1m0kbUdSnFxZlOgKI8s1I+kQ4FJgRET8s0ixFUq+a+4I9AOek7SEpC61opk3GGd5n5cBFRGxISL+F3iLJDE0V1mu+QzgYYCI+BPQhqRztpYq0/97Q7TERDATKJPUU9L2JI3BFTW2qQBOT6ePB56JtBWmmcp7zZL2A+4iSQLNvd4Y8lxzRKyKiC4R0SMiepC0i4yIiOY8zmmWv+3fkJQGkNSFpKpocTGDbGRZrvkvwNcBJPUmSQQrihplcVUA30rvHvoKsCoilm/LAVtc1VBEbJR0HjCd5I6DeyLidUnjgcqIqADuJik+LiJplDmp6SLedhmv+UagAzAtbRf/S0SMaLKgt1HGa25RMl7zdOAwSQuATcC4iGi2pd2M13wRMEnSBSQNx2Oa8xc7SVNIknmXtN3jCqA1QERMIGkHORJYBKwFvr3N52zGr5eZmTWCllg1ZGZmDeBEYGZW4pwIzMxKnBOBmVmJcyIwMytxTgQlQNImSXNyfnrUs+2aRjjfvZL+Nz3Xq+nTng09xi8k9Umnf1Rj3YvbGmN6nKrXZb6k30raMc/2A7amZ0tJ3SQ9nk4Pk7QqPe9CSVdsxfFGVPXCKenYqtcpnR+fPji4TdL38Pg82zzXkAf00mt/PMN2tfa+KekmSV/Lej7LzomgNHwSEQNyfpYU4ZzjImIAcAnJg2wNEhFnRsSCdPZHNdYNaYT44F+vSz+S50nOzbP9AJL7txvqQmBSzvwf09emnKSPnAZ1IxwRFRFxXTp7LEmPm1XrLo+I329FjJ8l9wK19ZF0G8nfkzUyJ4ISJKmDkjEJXpX0mqQteu1Mv8XOyPnGfGC6/DBJf0r3nSapQ57TzQD2Sve9MD3WfEn/nS5rL+l3kuamy09Mlz8nqVzSdUDbNI7J6bo16e+pko7KifleScdLaiXpRkkzlfTXflaGl+VPpB13SRqUXuNsSS9K+nL6VOt44MQ0lhPT2O+R9Eq6bW29nwJ8E3iq5sKI+AcwC9grLW28lMb7qKR/S2M5X/8aR2JqumyMpNslDQFGADemMX0p5zU4QtK0nNem+tt4Q99DSZenr+V8SROlzXrqPS3nb2RQun3W16VWdfW+GRHvADtL+nxDjmcZNEV/2/4p7g/JE6Zz0p9HSZ4o75Su60LyhGLVw4Vr0t8XAZem061I+u7pQvLB3j5d/gPg8lrOdy9wfDp9AvAyMBB4DWhP8oTz68B+JB+Sk3L27Zz+fo50/ICqmHK2qYrxP4D70untSXpkbAuMBS5Ll+8AVAI9a4lzTc71TQOOSOc7Adul04cAv0qnxwC35+x/DXBqOr0jSb8+7WucoycwK2d+GPB4Or0zsAToC8wDDk6XjwduSaffA3aoOkfNOHJf69z59D3+S8579XPg1K18D3fKWX4/cEzOezQpnT6ItP/8ul6XGtdeDvyinr/ZHtTSHz9JyeqbTf0/1dJ+WlwXE1arTyKpigBAUmvgGkkHAZ+SfBPeFXg/Z5+ZwD3ptr+JiDmSDiaphngh/VK4Pck36drcKOkykj5fziDpC+bRSL4FI+nXwIEk35R/Kul6kg+JPzbgup4E/kfSDiRVCTMi4hNJhwH75tRxdybpeO1/a+zfVtKc9PoXAk/nbH+fpDKSLgta13H+w4ARki5O59sAe6THqtKNLfu9OVDSbJLX/jqSjuJ2jIjn0/X3kSQmSBLEZEm/IelHKJNIumZ4CjhG0iPAUcD3SXqdzfoeVhku6ftAO2AnkiT+23TdlPR8MyR1UtLOUtfrkhtfJXBm1uvJ8QHwha3Yz+rhRFCaTgG6AgMjYoOS3jnb5G6Q/mMfRPIBcq+km4GPgacjYnSGc4yLiEeqZiR9vbaNIuKttI78SOBqSX+IiPFZLiIi1kl6DjgcOJFk0BJIRm76bkRMz3OITyJigKR2JH3ZnAvcSjKYzbMR8R9KGtafq2N/kXw7fbO+c1DjtSVpIzi6+iBS53r2P4rk2/YxwKWS9qln25qmAueRVLNURsTqtFon63uIpDbAnSSls6WSrmTz66nZR01Qx+siadcGxF6XNiSvqTUitxGUps7AB2kSGA5sMX6xkjGN/xoRk4BfkAyd9xIwVFJVnX97SXtnPOcfgWMltZPUnqRa54+SvgCsjYgHSDrGq63hdENaMqnNQySdblWVLiD5UD+7ah9Je6fnrFUkI7edD1ykf3VLXtWt75icTVeTVJFVmQ58t6rOXEkPrzW9RVLNUaeIWAV8rLQdBjgNeF7JmAq7R8SzJFU4nUmq1XLVjCnX8ySv53/yryTZ0Pew6kP/w7QtoeadRFVtOl8l6QVzFdlel621N1DrWL629ZwIStNkoFzSa8C3gDdq2WYYMDetwjgR+J+IWEHywThF0jySKoVeWU4YEa+S1Du/QtJm8IuImA3sA7ySVtFcAVxdy+4TgXlKG4tr+L8k1R2/j2QoQ0gS1wLgVSW3IN5FntJvGss8kkFObgCuTa89d79ngT5VjcUkJYfWaWyvp/M1j/sP4O2qD956nE5SnTaP5O6k8SRtFw+k79Ns4NbYcoCZqcC4tFH2SzXOvQl4HPhG+puGvofp+SaRfPhOJ6kyzLUufZ0mkFQBQobXRcmNAL+o7ZxKet/8E/BlScsknZEub01y40Fz7kr8M8m9j5oVmKT/IKmGu6ypY2nO0tdx/4j4P00dS0vjNgKzAouIRyU15zGxPyu2A37a1EG0RC4RmJmVOLcRmJmVOCcCM7MS50RgZlbinAjMzEqcE4GZWYn7/6S9pdQ3mVsVAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f3ExnZBKSNCC"
      },
      "source": [
        "Distribution of predicted probabilty for each class "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5BSCoEchSNCD",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 355
        },
        "outputId": "cb7017f1-6a19-4775-e1e5-db4066fa981d"
      },
      "source": [
        "df = pd.DataFrame({'probPos': proba_test, 'target': y_test})\n",
        "plt.figure(figsize=(10,5))\n",
        "plt.hist(df[df.target == 0].probPos, density=True, bins=25,\n",
        "             alpha=.5, color='green', label='LGG')\n",
        "plt.hist(df[df.target == 1].probPos, density=True, bins=25,\n",
        "             alpha=.5, color='red', label='HGG')\n",
        "plt.axvline(.5, color='blue', linestyle='--', label='Boundary')\n",
        "plt.xlim([0, 1])\n",
        "plt.title('Distributions of Predictions', size=15)\n",
        "plt.xlabel('Positive Probability (predicted)', size=13)\n",
        "plt.ylabel('Samples (normalized scale)', size=13)\n",
        "plt.legend(loc=\"upper right\")\n",
        "plt.show();"
      ],
      "execution_count": 196,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 720x360 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmgAAAFSCAYAAABG5iMfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3debxd0/3/8ddbqIiEEKolIimqaYIrrhqqpCiixlK+lIqhdEhL0YFGxRBapFRbrZhCY4iqttSQ0vZW/QyVkAqCUokEIQkhlRvk5vP7Y+8b5x532Df3nLPPzXk/H4/9OGdPa3/22Yl8rLXXWooIzMzMzKx6rJJ3AGZmZmbWkhM0MzMzsyrjBM3MzMysyjhBMzMzM6syTtDMzMzMqowTNDMzM7Mq4wTNLAeSxkiKdFkm6U1Jj0oaK+ljRccOTI/bN2PZH0nLr+tEPDMlXVywPkHSlOx31G7Ze0o6uZXtJbtGJUn6saSX0+c2oY1jRhY835D0uqTJkoaVMa6h6bWGF2wLSaM6UcZK9azMurNV8w7ArIa9Beydfl8bGAZ8AzhB0t4RMTXd9yqwI/BMxnI/ApwFzASmZTznIGBBxmM7a0/gEODSou3nAmuU6ZplIakeOBs4A2gAXu/glN2ARuDjwJnA3yUNjohXyhlngR2BFztx/ErzrMy6OydoZvlZGhEPF6xPlvRr4H7gZkmfioimiHgXeLj1IrpG0hoR0RgRj5ej/PZExAuVvmYJfCr9/FVEvJ3h+Ecj4n8AaQ3ULOArwEXFB0paDVgWEU2lCrboz1dXyumOz8qsW3MTp1kViYiFwPeBzYAvQOtNnJL2lzRV0jtp8+gjknZNdy9KP68taGIbWFDOVyRdL2khcEdaXosmzoLrHCjpGUlLJD0g6dMF+1ptei1sDpM0BjgV2KQglgnFxxWcWyfpr5IWp/d1g6QNWrnmoZKukPSWpDmSzpa0SsFx/SXdkjYtNkp6QdK57f32knqkTcMvSXpX0lOSjii8L+C36epbxc2JHYmI2cA8YGBaXoOkWyWdIOkFYAmwYbrv+PT670qaJen7rcT7TUmz0z8Dd5DU0hUf86EmTkkHSfpX+rsskHSXpE2607MyqwWuQTOrPg3AUmAH4J7inZI2BW4Ffg58D+gJbAusmx6yG/A34DzgznTbq3zwD/jFwG3Al4H2ams2AX5G0jTXSNK0N1nS5hGxJOO9XAVsnsZ0ULptXmsHSlqf5N5nAEcAvYGfAPdKqo+I9woOvxD4PUlz3O7Aj4GngFvS/deTNMmdACwEPsEHtV9tOYckOT4beBQ4GLhBUkTETSTNfLOB0XzQdPl0Rz9Awf31IXlGcws2fxbYFPgBsJgk8fsecH56jw0kz/ZcSYsj4pdpWQcAvwJ+A/wR2BW4JkMMR5H8Njen96P0Xtanez0rs5VfRHjx4qXCCzAGmN/O/leBX6ffBwIB7JuuHwIsaOfc3unxI4u2N5fzh1bOmQlcXLA+IT12p4Jtm5Akjl9vLa6ic6cUrF8MzGzlmsXH/YTkH+i1CrZtn17j8KJrXl9U1jTg5oL1/wH7deJ5rAu8A5xVtP0u4NmC9ZHp9Xt3UF7zcWuT/I/wxsCk9PerS49pIEnyNig4b6009uI4ziFJ7Hqk6/8C7i465sr0msMLtgUwKv2+CvAycFs7cVf9s/LipVYWN3GaVSe1s286sLak65T0uluzk2Xf2fEhALweEQ82r0TELGAq8JlOXi+rzwB/iYJ3uyLiEZLkceeiY/9StP400L9gfRpwgZLelAMyXHso0Av4XdH2ScAn0xqjFbEQeB94iaRm6tiIKOy4MTUiXitY3xFYE/idpFWbF5Ia0Q2A/un6MOBPRde6rYNYtiBpQr12Be+lUJ7PyqwmOEEzqzKSegL9gNda2x8RzwIHkDQF3QXMl3RjJ5KIVsttRWs9FF+nlXedSuTjtB7ba3zQfNtsYdH6eyRNvc0OA6YAlwCzJE2TtHsH126+VvG1aeX6We0C1JPUJm0QEde3UX6z9dLPp0gSu+bl7+n2jdNjevDh59NRj9J+6eerWQLvQJ7PyqwmOEEzqz6fJ2kWe6itAyLizoj4HMk/uscBewC/yFh+ZDzuo21sa/4Hvvk9tI8UHbNOxvKLvdrGNTcA3uhMQRHxckSMJPl9diRpHrxdUr82Tmm+p+LrN7/03qnrF3g8IqZGxKyIWNZaqEXrzdfZF9iuleXfwHySdweLY23ttyvUPIxKKRLsPJ+VWU1wgmZWRST1BX4KPA/c19HxEfFWRNwI/AFo7mHZ/IJ2z9bPyuyjknYqiG0ASdPav9JNr5PU7gwuOKY3sBMtFdeYtOURYK/0Zfrm8rYjqX16YAXiJyKWRTLUxNkkTZibtHHokyQv6X+5aPuhwHMR0erL8mXwEMl7aRtGxJRWlkURsRR4nKQWtdCXOij7WZJ30I5u55ju8KzMaoJ7cZrlZ1VJO6Tf+5D01vsGyT9Oe0cb42FJOpGkpuEe4BWSnndfJukNR0S8J+lF4FBJT5LUdD2xAvHNByZKGs0HvThfJ3lhnIhYJulPwHclzSJpyjo1PbbQM8AGkkaSJELzI2JmK9f7WXr/kyX9lA96Bk4n6QWYiaS1gckkv8dzwOppXHNJeh1+SES8IelSYLSkpSRNbl8C9gEOz3rtroqIhelwFz+XtAnJmHirAJ8EPh8Rzb0rzwduUzJu3h9IenHu3UqRhWUvS4fruEHSDcBNJDV4uwE3RcQUusGzMqsVTtDM8rM2SY1JAG+T1JpNBH4REXPbOe8JYH+SfyTXJWluupJk+IJmXyfpkXcfyT96g1YgvlkkicBPSGozpgBHRMshNkYB44HLgTeBsSQ1aEMLjrmFpNn2QpLhHK4j6eXYQkTMk/R5YBxJ8vAeyTt2342WwzZ0ZAlJonASyTtbi0kG+t0zIoqTx0I/Jull+Q2SprrngSMj4uZOXLvLIuJCSa8A3yVJVpaQJC+TCo75g6RvAz8kqRFrIGnqntxB2TdKWgL8iGSolndIfpvmGsLu8qzMVnqKyPo6ipmZmZlVgt9BMzMzM6syTtDMzMzMqowTNDMzM7Mq4wTNzMzMrMo4QTMzMzOrMt1umI311lsvBg4cmHcYZrYSefbZ5HOLLfKNw8xWPlOnTp0fEZ2ez7fbJWgDBw5kypQpeYdhZiuR009PPi+4IN84zGzlkw7k3WndLkEzMys1J2ZmVm38DpqZmZlZlalIgibpGkmvp/MCNm9bV9K9kv6Tfq5TiVjMzIodfHCymJlVi0o1cU4Afkk6mXPqh8BfI+Inkn6Yrv+gQvGYmS23YEHeEZjl4/3332fOnDksWbKk44OtXT179qR///6sttpqJSmvIglaRNwvaWDR5gOA4en360gm+3WCZmZmViFz5syhT58+DBw4EEl5h9NtRQQLFixgzpw5DBo0qCRl5vkO2gYR8Wr6fS6wQY6xmJmZ1ZwlS5bQr18/J2ddJIl+/fqVtCayKjoJREQA0dZ+SSdImiJpyrx58yoYmZmZ2crNyVlplPp3zDNBe03SxwHSz9fbOjAixkdEfUTUr79+p8d6MzNr1+67J4uZVV6PHj2oq6tj6623ZtiwYTz44INlv+bAgQOZP39+2a/TFXmOg3Y7cDTwk/TzTznGYmY17Mwz847ArHatscYaTJs2DYDJkydz+umn849//CPnqFpqamqiR48eFb1mpYbZuAl4CNhC0hxJx5EkZl+Q9B9gj3TdzMzMatTbb7/NOusko25FBN/73vcYOnQoW265JZMmTQKgoaGBfffdd/k5o0aNYsKECUBSM3bWWWcxbNgwttxyS5555hkAFixYwJ577smQIUM4/vjjSd6sShx44IFsu+22DBkyhPHjxy/f3rt3b0499VS23nprxo4dy4EHHrh837333stBBx1Utt8BKteL8/A2drlRwcxyN2JE8nn33fnGYVaLGhsbqaurY8mSJbz66qv87W9/A+C2225j2rRp/Pvf/2b+/Plst9127LLLLh2Wt9566/HYY49x+eWXc/HFF3PVVVdx9tlns/POO/PjH/+YO++8k6uvvnr58ddccw3rrrsujY2NbLfddhx88MH069ePd955h+23355x48YREQwePJh58+ax/vrrc+2113LssceW7TcBT/VkZkZjY94RmFWH4cM/vO3QQ+Gb34TFi2GffT68f+TIZJk/Hw45pOW+hoaOr1nYxPnQQw/x1a9+lSeffJIHHniAww8/nB49erDBBhuw66678uijj7LWWmu1W96XvvQlALbddltuu+02AO6///7l37/4xS8ur6UDuOyyy/jDH/4AwOzZs/nPf/5Dv3796NGjBwfvuCO88goCjjrgACb+6lccc9hhPPTAA1z/k5/AK6+0vPjChTBmTLJ0kRM0MzMzqwo77rgj8+fPp70RG1ZddVWWLVu2fL14aIvVV18dSDofLF26tN3rNTQ0cN999/HQQw/Rq1cvhg8fvry8nquv3uK9s2MOO4z9Ro6k5+qr8+V992XVVcubQjlBMzMzM6D9Gq9evdrfv9562WrM2vPMM8/Q1NREv379+NznPscVV1zB0UcfzRtvvMH999/PRRddxPvvv8/TTz/Nu+++S2NjI3/961/Zeeed2y13l1124cYbb2T06NHcfffdvPnmmwC89dZbrLPOOvTq1YtnnnmGhx9+uM0yNvzYx9hwgw0477LLuO/mm7t2oxk4QTMzM7PcNL+DBknHgOuuu44ePXpw0EEH8dBDD7H11lsjiQsvvJCPfexjABx66KEMHTqUQYMGsc0223R4jbPOOovDDz+cIUOGsNNOOzFgwAAA9t57b37zm98wePBgtthiC3bYYYd2y/nKl77EvAULGLz55l28646psCdDd1BfXx9TpkzJOwwzW4lcfHHyedpp+cZhVmkzZsxg8ODBeYdRnYrfLwNG/ehHbDN0KMcd3nrfxxmzZjF48uQW76BJmhoR9Z29vGvQzKzmOTEzs45su/ferNmrF+N+/OOKXM8JmpmZmVkHpt5zT0WvVxVzcZqZ5Wn48NaHFzAzy4sTNDMzM7Mq4wTNzMzMrMo4QTMzMzOrMk7QzMzMLDe9e/dudfvEiRPZaqutGDJkCFtvvTXHH388CxcuBGDp0qWcccYZbL755tTV1VFXV8fYsWMrGXbZuRenmdW8Qw/NOwKz6jCmYUxpyxu+YuXdc889XHLJJdx9991stNFGNDU1cd111/Haa6/Rt29fRo8ezdy5c5k+fTo9e/Zk0aJFjBs3rqSx580JmpnVvG9+M+8IzKzQ2LFjufjii9loo42AZF7NY489FoDFixdz5ZVXMnPmTHr27AlAnz59GFOCCcqriZs4zazmLV6cLGZWHZ566imGDRvW6r7nn3+eAQMG0KdPnwpHVVlO0Mys5u2zT7KYWfWZPn06dXV1bLrppkyaNOlD+6+99lrq6urYeOONmT17dg4RlocTNDMzM6sqQ4YM4bHHHgNgyy23ZNq0aYwYMYLGxkY222wzXnrpJRYtWgTAMcccw7Rp01h77bVpamrKM+yScoJmZmZmVeX000/ntNNOY86cOcu3NTY2AtCrVy+OO+44Ro0axZIlSwBoamrivffeyyXWcnEnATMzM8vN4sWL6d+///L1U045hVNOOYV58+YxYsQImpqa6Nu3L0OHDmWvvfYCkk4EZ555JkOHDqVPnz6sscYaHH300Wy44YZ53UbJKSLyjqFT6uvrY8qUKXmHYWYrkeZ5OBsa8ozCrPJmzJjB4MGD8w6jOr3ySqdPmTFrFoMnT4aCHqWSpkZEfWfLcg2amdW8kSPzjsDMrCUnaGZW85ygmVm1cScBM6t58+cni5lZtXANmpnVvEMOST79DpqZVQvXoJmZmZlVGSdoZmZmZlXGCZqZmZnlpnfv3i3WJ0yYwKhRo5avT5w4ka222oohQ4aw9dZbc/zxx7Nw4UIAli5dyhlnnMHmm29OXV0ddXV1jB07tqLxl4vfQTMzM7NEwfhd1VDePffcwyWXXMLdd9/NRhttRFNTE9dddx2vvfYaffv2ZfTo0cydO5fp06fTs2dPFi1axLhx40oTe86coJlZzfvGN/KOwMxaM3bsWC6++GI22mgjAHr06MGxxx4LJDMQXHnllcycOZOePXsC0KdPH8aUOsnMiRM0M6t5hx2WdwRmtauxsZG6urrl62+88Qb7778/AE899RTDhg1r9bznn3+eAQMG0KdPn4rEWWl+B83Mat7s2cliZpW3xhprMG3atOXLOeec0+px06dPp66ujk033ZRJkyZ9aP+1115LXV0dG2+8MbNXgr/QTtDMrOYddVSymFl1GTJkCI899hgAW265JdOmTWPEiBE0Njay2Wab8dJLL7Fo0SIAjjnmGKZNm8baa69NU1NTnmGXhBM0MzMzq0qnn346p512GnPmzFm+rbGxEYBevXpx3HHHMWrUKJYsWQJAU1MT7733Xi6xlprfQTMzM7OqtM8++zBv3jxGjBhBU1MTffv2ZejQoey1115A0ongzDPPZOjQofTp04c11liDo48+mg033DDnyLtOEdHxQVIdMAxYF3gDeDwiHi9zbK2qr6+PKVOm5HFpM1tJDR+efHqqJ6s1M2bMYPDgwXmHUZ1eeaXTp8yYNYvBkye3GF5E0tSIqO9sWW3WoElaDfh2unwM+A/wNrAWsLmkucBlwC8j4v3OXtjMzMzMWtdeE+d0YCrwNeD+iFjeqCvpI8AuwDHAE4DTbzPrtk49Ne8IzMxaai9B+1JEPN3ajjRZuw+4T5KTMzPr1vbbL+8IzMxaarMXZ1vJWSvHzShdOGZmlffss8liVouyvItuHSv175h5mA1JO0saL+mOdH1bSbuUNBozsxyceGKymNWanj17smDBAidpXRQRLHjnHXq+9VbJysw0zIakI4BfADeQvHsGEMA5wPCSRWNmZmYV079/f+bMmcO8efPyDqX6LFzYqcN7vvUW/dNBdUsh6zhoPwL2iogpkprH234SGFKySMzMzKyiVlttNQYNGpR3GNUp50nXszZxbhgRzYOPNdeDLgV6lD4kMzMzs9qWNUF7QdJORdt2AvxarZmZmVmJZW3iPA/4k6SfA6tJOhX4LskYaV0i6bvA8SQ1c9OBYyJiSVfLNTPLavTovCMwM2spU4IWEX+U9A7wHWAWsBtJInVvVy4uaaO0zE9HRKOkW4D/AyZ0pVwzs87YY4+8IzAzaynzZOlpMtalhKydGNaQ9D7QC+j85FdmZl0wbVryWVeXbxxmZs3am4sz01TwEbHCCVVEvCzpYuAloBH4S0T8pZVYTgBOABgwYMCKXs7MrFUnn5x8erJ0M6sW7dWgzeGDHputUbp/hXtySloHOAAYBCwEfifpyIiYWHhcRIwHxgPU19d7ND0zMzNbqbWXoFViYJQ9gBcjYh6ApNtIeodObPcsMzMzs5VYmwlaRMyqwPVfAnaQ1IukiXN3YEr7p5iZmZmt3DJ3EpD0KZJpndYnad4EICLOWdGLR8Qjkm4FHiMZ+PZx0qZMMzMzs1qVdS7Ow0mGvngC2Cr93Bq4v6sBRMRZwFldLcfMbEWdf37eEZiZtdSZuTiPiohbJL0ZEdtJOhb4VBljMzOriJ2K50kxM8tZ1qmeBgC/K9p2PXBUK8eamXUrDz6YLGZm1SJrDdpCYO308zVJg4EFwJrlCszMrFLOOCP59DhoZlYtstag3QcclH6/JV3/F3B3OYIyMzMzq2VZ5+I8tmD1LOAZYC3gunIEZWZmZlbLMg+z0SwiArixDLGYmZmZGRmbOCX9RdJuRdt2l+QmTjMzM7MSy1qDNowPj3n2T2BSacMxM6u8Sy/NOwIzs5ayJmjLgNVIRvtv1oOCGQXMzLqrurq8IzAzaylrL86pwLeLto0imaLJzKxbu+++ZDEzqxZZa9B+ADRIOhh4Dtgc2IJkbk4zs27tvPOSzz32yDcOM7NmmWrQIuIJ4NPArcDbwO+BT0fEv8sYm5mZmVlNyjzMRkTMBS4qYyxmZmZmRvZhNr4laev0+7aSZkl6QVJ9ecMzMzMzqz1ZOwmcCrycfh8L3AxcC4wrR1BmZmZmtSxrE2e/iJgvaXVgR+BA4H3glLJFZmZWIVdckXcEZmYtZU3Q/idpQ2BL4ImIWCLpIyRjoZmZdWtbbJF3BGZmLWVN0CYAjwCrA2ek2z4DPF+GmMzMKuqOO5LP/fbLNw4zs2aZErSI+JGkBuC9iPhHuvld4LRyBWZmVinj0rdpnaCZWbXozDAb9xatP1r6cMzMzMwsay9OMzMzM6sQJ2hmZmZmVcYJmpmZmVmVyfwOmpnZyuq3v807AjOzltpM0CRdk6WAiDi2dOGYmVXexhvnHYGZWUvtNXE2FSyrAUcCn0y/b56uuwbOzLq9SZOSxcysWrSZYEXE15q/S7oeOC4ifluw7Uhgz/KGZ2ZWfr/+dfJ52GH5xmFm1ixrJ4H9gRuKtt2UbjczMzOzEsqaoM0Hhhdt2wV4o6TRmJmZmVnmd8jOB/4s6XfATGAgcAjw7fKEZWZmZla7MtWgRcQ1wF4k829uB7wHjEi3m5mZmVkJdWYuzn8C/yxjLGZmubj11rwjMDNrKXOCJmln4KvAxyNiP0nbAmtGxP1li87MrALWWy/vCMzMWsrUxCnpCOB2YAlJ5wCAAM4pU1xmZhUzYUKymJlVi6y9OH8E7BkR3wGWpdueBIaUJSozswpygmZm1SZrgrZhRExJv0f6uRToUfqQzMzMzGpb1gTtBUk7FW3bCXi2xPGYmZmZ1bysnQTOA/4k6efAapJOBb4LfK3908zMzMysszIlaBHxR0nvAN8BZgG7AcdExL3lDM7MzMysFnVmHLR7ASdkZrbSueuuvCMwM2sp6zAbj0n60LROkh4ofUhmZpXVq1eymJlVi6ydBIYAp0i6uGj7ViWOx8ys4i6/PFnMzKpF1gTtXWBHYDdJN0tarYwxmZlV1C23JIuZWbXImqAREXOBXYF1gPsk9S1bVGZmZmY1LGuCJoCIWATsA/wXeBDock2apL6SbpX0jKQZknbsaplmZmZm3VnWBG15782IaIqIY4BbgLkliOHnwD0R8Slga2BGCco0MzMz67YyJWgR8aVWto2JiEFdubiktUkmX786LfO9iFjYlTLNzMzMurs2x0GT9OWI+F36/Yg2DouIuKkL1x8EzAOulbQ1MBU4KSLeKYrlBOAEgAEDBnThcmZmH9bQkHcEZmYttTdQ7VnA79LvY9s4JoCuJGirAsOAb0fEI+lUUj8EzmxxkYjxwHiA+vr6+FApZmZmZiuRNhO0iBha8L1LTZntmAPMiYhH0vVbSRI0M7OKuTgd4fG00/KNw8ysWeapnsohIuZKmi1pi4h4FtgdeDrPmLIa0zCmNOUML005Zrbi/vzn5NMJmplVi/beQRufpYCIOKGLMXwbuEHSR0iG7zimi+WZmZmZdWvt1aBVZLaAiJgG1FfiWmZmZmbdQXvvoLkmy8zMzCwHub6DZmZWDdZYI+8IzMxaypSgSVofuJTkJf71C/dFRI8yxGVmVjF33513BGZmLWWd6ukyYCPgOOAdYH+SuThPLlNcZmZmZjUraxPnbsCWEfG6pGURcaek6STjlv2ifOGZmZXfuecmn2ee2f5xZmaVkrUGbTWSKZkAGiWtGREvAZ8qT1hmZpXz178mi5lZtchag/YcyZRMU4F/A2dIegt4rVyBmZmZmdWqrAnaGcDqBd9vBvqQTmBuZmZmZqWTKUGLiL8VfH8M+GTZIjIzMzOrcZ0aB01SH5Kas+Ui4pWSRmRmVmH9+uUdgZlZS1nHQfsscC2waeFmIACPg2Zm3drvf593BGZmLWWtQbsS+B0wEVhcvnDMzMzMLGuCthEwOiKinMGYmeXh9NOTzwsuyDcOM7NmWRO0e4F64NEyxmJmlouHHso7AjOzlrImaCcAd0l6FHi1cEdEnF/yqMzMzMxqWNYE7YdAHUnHgMJ30AJwgmZmZmZWQlkTtBOB7SJiejmDMTMzM7PsCdrbwIxyBmJmlpf+/fOOwMyspawJ2jiSKZ7OKWMsZma5mDgx7wjMzFrKmqB9C9hE0inA64U7IsLTPpmZmZmVUNYE7byyRmFmlqOTT04+L7003zjMzJp1mKBJWhU4FDg4IpaUPyQzs8qaNi3vCMzMWlqlowMiYimwLbC0/OGYmZmZWYcJWuq3wKhyBmJmZmZmiazvoA0DTpL0LWAWsKx5R0TsWY7AzMzMzGpV1gTt/nQxM1vpfNJ90c2symRK0CLi7HIHYmaWl/Hj847AzKylrDVoSNoYOALYGJgN3BARc8oVmJmZmVmtytRJQNLOJFM9HQCsDewPPCPpc2WMzcysIk44IVnMzKpF1hq0C4HvRMQ1zRskHQNcBOxQjsDMzCrluefyjsDMrKWsw2wMBiYUbbse2KKk0ZiZmZlZ5gTtNZKhNgoNo2heTjMzMzPruqxNnD8H7pJ0BfAiMBA4EXDvTjMzM7MSyzrMxq8lLQRGAgeT9OI8OSJuKmNsZmYVUVeXdwRmZi1lHmYjTcackJnZSufSS/OOwMyspc6Mg9YT2BzoU7g9Ih4sdVBmZmZmjBmTz7lVIFOCJml/4DqSMdAKBdCj1EGZmVXSkUcmnxMn5huHmVmzrL04x5F0COgdEasULE7OzKzbmzMnWczMqkXWJs4NIsJvaZiZmZlVQNYatL9I2r6skZiZmZkZkL0GbSZwh6RJwKuFOyLi/FIHZWZmZlbLsiZo2wJPAUPTpVkATtDMrFvbcce8IzAzaynrQLWfL3cgZmZ5ueCCvCMwM2sp6ztoZSWph6THJf0571jMzMzM8tZmgibpH5J2be9kSbtKaihBHCcBM0pQjplZpx18cLKYmVWL9po4zwcul7QacB/wNPA2sBbwaWB3YClwSlcCkNQf+CIwtqtlmZmtiAUL8o7AzKylNhO0iJgMDJG0F3AAcCSwDvAm8DjwnfSYrroU+D5FU0iZmZmZ1aoOOwmkSVgpErEPkbQv8HpETJU0vJ3jTgBOABgwYEA5QjEzMzOrGnl3EvgssL+kmcDNwG6SPjQbXkSMj4j6iKhff/31Kx2jmZmZWUVlHQetLCLidOB0gJIRs8sAABYmSURBVLQG7bSIODLPmLqrMQ1jul7G8K6XYdYd7b573hGYmbWUa4JmZlYNzjwz7wjMzFqqmgQtIhqAhpzDMDMzM8vdCr2DJmmQJL+tb2YrhREjksXMrFpkStAkXSvps+n3w4Hngf9KOqKcwZmZVUJjY7KYmVWLrDVoewOPpd9PAQ4GvgCcUY6gzMzMzGpZ1nfQekVEo6R1gE2BP0VESNq4jLGZmZmZ1aSsCdrL6bycg4F/psnZWiRTPZmZmZlZCWVN0M4B7gXeA/ZJt+0BTCtHUGZmlbTvvnlHYGbWUqYELSJulvSn9Hvzq7QPAA+WKzAzs0o57bS8IzAza6kz46C9B2wvaeOImAT8D4jyhGVmZlaFxozJ7/y8zu2uuvk9Zx1mY1PgSeAu4Op0857AlWWKy8ysYoYPTxYzs2qRdZiNX5BMZr4u8H66rQH4XBliMjMzM6tpWZs4PwPsHxHLJAVARCyU1Ld8oZmZmZnVpqw1aG8DLZIxSRsCr5U8IjMzM7MalzVBuw24RlJ/AEn9gEtJmj3NzMzMrISyNnGeCVwFvJSuvw7cCJxfjqDMzCrp0EPzjsDMrKWs46A1Al+RdBIwEJgVEfPKGZiZWaV885t5R2Bm1lJnxkEjIuYD88sUi5lZLhYvTj579co3DjOzZm0maJLuJcNAtBGxZ0kjMjOrsH3SCewaGnINw8xsufZq0B6oWBRmZmZmtlybCVpEnF3JQMzMzMwskfkdNEm9gX2B/sAc4M6IWFSuwMzMzMxqVaYETVI9yTycjSRDbQwALpO0T0RMKWN8ZmZmZjUnaw3a5cC4iPhp8wZJ3wd+DWxXjsDMzCpl5Mi8IzAzaylrgjYYGFe07WckA9iamXVrTtDMrNpkTdCmAUPTz2ZbFq1bNzemYUxpyhleonKqKJ5qisVKb346uuN66+Ubh5lZs6wJ2l+AP0u6CphFMpvAscB4SUc0HxQRN5Y8QjOzMjvkkOTT46CZWbXImqAdC7wPHF2wbWm6vVmQzM9pZmZmZl2QdS7OQeUOxMzMzMwSq+QdgJmZmZm1lHUctI8BZwP1QJ/CfRHxyTLEZWZmZlazsr6DdkP6eRWwuEyxmJnl4hvfyDsCM7OWsiZo9cAGEbGknMGYmeXhsMPyjsCsio0Zk+/5NSrrO2jPAuuUMxAzs7zMnp0sZmbVImsN2teAX0u6HphbuCMiHix5VGZmFXTUUcmnx0Ezs2rRmamedgf2L9oeQI+SRmRmZmZW47I2cV4EnAasGRGrFCxOzszMzMxKLGsNWu+IuKKskZiZmZkZkL0G7TZJe5c1EjMzMzMDstegrQb8XtLfgFcLd0TECSWPysysgk49Ne8IzMxaypqgNQG3pN9XK1MsZma52G+/vCMwM2sp62Tpx5Q7EDOzvDz7bPK5xRb5xmFm1ixrDRoAknoC6wNq3hYRL5U6KDOzSjrxxOTT46CZWbXIOln6J4CJwPat7PZQG2ZmZmYllLUX5y+B2cDWwCJgK+CPwHFlisvMzMysZmVt4tweGBgRiyQREU9JOhH4BzChbNGZmZmZ1aCsNWjLgMb0+/8k9QXeAAZ05eKSNpb0d0lPS3pK0kldKc/MzMxsZZC1Bu0p4LMkNWaPAJcA7wAvdvH6S4FTI+IxSX2AqZLujYinu1iumVlmo0fnHYGZWUtZE7TvkEyMDvA94AqgD3BiVy4eEa+SDnybNp/OADYCnKCZWcXssUfeEZiZtZR1HLQnCr7/F/hCqQORNBDYhqSGzsysYqZNSz7r6vKNw8ysWbsJmqRVAUXE+wXbRgJ1wP0RcVspgpDUG/g9cHJEvN3K/hOAEwAGDOjSa29mZh9y8snJp8dBM7Nq0VEN2iRgMjAeQNJo4MfAE8CJkr4dEVd1JQBJq5EkZze0lfBFxPjmGOrr66O1Y8y6gzENY0pTzvDSlGNmZtWpo16c9cCfC9a/DRwfEfXAkcA3unJxSQKuBmZExM+6UpaZmZnZyqKjBG2diHgFQNJgYG0+mDT9j8DALl7/s8BRwG6SpqXLPl0s08zMzKxb66iJ8x1JvSPifyS1aU9GxJJ0nzKc366IeICCeT3NzMzMrOME65/AuZKuIBlS456CfVuQDpFhZtadnX9+3hGYmbXUUYL2A+Au4CTgSaDwPbGvAA+UKS6zlcbwCQ0rfG7DyOEli8PattNOeUfQTY0Zk+/5ZiuxdhO0iHgRGCxp3Yh4o2j3hcB7ZYvMzKxCHnww+XSiZmbVIutAtcXJGRGxsPThmJlV3hlnJJ8eB83MqkXWydLNzMzMrEKcoJmZmZlVGSdoZmZmZlXGCZqZmZlZlenSQLNmZiuDSy/NOwIzs5acoJlZzauryzsCM7OW3MRpZjXvvvuSxcysWrgGzcxq3nnnJZ977JFvHGZmzVyDZmZmZlZlnKCZmZmZVRknaGZmZmZVxgmamZmZWZVxJwEzq3lXXJF3BGZmLTlBM7Oat8UWeUdgZtaSmzjNrObdcUeymJlVC9egmVnNGzcu+dxvv3zjMDNr5ho0MzMzsyrjBM3MzMysyriJM2djGsbkHULJVds9VVs8pVCqexozvDTllEKez2nmwpFpDBOWb6um36baND+r4TMbVriM4QOHlzSWLpdToufd0MFv0tCFeLvye3fluit+pnWFa9DMzMzMqoxr0MysdWPG5HNuFw2f0NDpcz69+VMAPM2QFb9wjvec67W7ogRxr0itUsPI4V2+rlm5OUEzs5r30TXnAfB0znGYmTVzgmZmNe9vL34egFWInCMxM0s4QTOzmnf7swcAcCB/zDkSM7OEOwmYmZmZVRknaGZmZmZVxgmamZmZWZVxgmZmZmZWZdxJwMxq3tnDzwLgcbbJORIzs4QTNDOreWv3fCvvEMzMWnCCZmY1757n9wagJ0tyjsTMLOEEzcxqXnOC5nHQzKxauJOAmZmZWZVxgmZmZmZWZZygmZmZmVUZJ2hmZmZmVcadBMys5v1kjx8A8DA75hyJmVnCCZqZ1byeq76bdwhmZi04QTOzmvfHZw4AoC8esNbMqoMTNDOreQ0zPw94HDQzqx65dxKQtLekZyU9L+mHecdjZmZmlrdcEzRJPYBfASOATwOHS/p0njGZmZmZ5S3vGrTPAM9HxH8j4j3gZuCAnGMyMzMzy1XeCdpGwOyC9TnpNjMzM7OapYjI7+LSIcDeEXF8un4UsH1EjCo67gTghHR1KPBkRQO1UlkPmJ93ELbC/Py6Nz+/7svPrnvbIiL6dPakvHtxvgxsXLDeP93WQkSMB8YDSJoSEfWVCc9Kyc+ue/Pz6978/LovP7vuTdKUFTkv7ybOR4HNJQ2S9BHg/4Dbc47JzMzMLFe51qBFxFJJo4DJQA/gmoh4Ks+YzMzMzPKWdxMnEXEXcFcnThlfrlis7Pzsujc/v+7Nz6/78rPr3lbo+eXaScDMzMzMPizvd9DMzMzMrEjVJmgdTQElaXVJk9L9j0gaWPkorTUZnt0pkp6W9ISkv0raJI84rXVZp1+TdLCkkOTeZVUiy7OTdGj69+8pSTdWOkZrW4b/dg6Q9HdJj6f//dwnjzjtwyRdI+l1Sa0OA6bEZemzfULSsI7KrMoELeMUUMcBb0bEZsAlwE8rG6W1JuOzexyoj4itgFuBCysbpbUl6/RrkvoAJwGPVDZCa0uWZydpc+B04LMRMQQ4ueKBWqsy/t0bDdwSEduQjHpweWWjtHZMAPZuZ/8IYPN0OQH4dUcFVmWCRrYpoA4Arku/3wrsLkkVjNFa1+Gzi4i/R8TidPVhkvHvrDpknX7tXJL/KVpSyeCsXVme3deAX0XEmwAR8XqFY7S2ZXl+AayVfl8beKWC8Vk7IuJ+4I12DjkAuD4SDwN9JX28vTKrNUHLMgXU8mMiYinwFtCvItFZezo7fddxwN1ljcg6o8Pnl1bNbxwRd1YyMOtQlr97nwQ+Ken/SXpYUnv/x2+VleX5jQGOlDSHZPSDb1cmNCuBTk9tmfswG1a7JB0J1AO75h2LZSNpFeBnwMicQ7EVsypJE8twkprr+yVtGRELc43KsjocmBAR4yTtCPxW0tCIWJZ3YFZ61VqDlmUKqOXHSFqVpLp3QUWis/Zkmr5L0h7Aj4D9I+LdCsVmHevo+fUhmQ+3QdJMYAfgdncUqApZ/u7NAW6PiPcj4kXgOZKEzfKX5fkdB9wCEBEPAT1J5um06pfp38ZC1ZqgZZkC6nbg6PT7IcDfwoO6VYMOn52kbYArSJIzvwNTXdp9fhHxVkSsFxEDI2IgyTuE+0fECs01ZyWV5b+bfySpPUPSeiRNnv+tZJDWpizP7yVgdwBJg0kStHkVjdJW1O3AV9PenDsAb0XEq+2dUJVNnG1NASXpHGBKRNwOXE1Svfs8yYt5/5dfxNYs47O7COgN/C7t1/FSROyfW9C2XMbnZ1Uo47ObDOwp6WmgCfheRLjloQpkfH6nAldK+i5Jh4GRrpioDpJuIvmfn/XSdwTPAlYDiIjfkLwzuA/wPLAYOKbDMv1szczMzKpLtTZxmpmZmdUsJ2hmZmZmVcYJmpmZmVmVcYJmZmZmVmWcoJmZmZlVGSdoZisZSb+R9MuuHlNNJA2XtLQL5w+UFJLanPdV0t2Svl+wHpJ2Tr9/TlLJR9uX1FPSfyRtUeqyO7hu//T+BqbrZ0i6o8IxXCVpQvp9dUnPS/pUJWMwq2ZO0MxyIqlB0ruS/ifpLUmPSzq4q+VGxNcjYlTBdWam02q1eUyppddckt7bm+ncj8PLdb1SiIgREXFhG/v+GRF9m9cljZF0XwkuexLwUEQ8W4KyVlhEnB8R+2U5VtIESVeV+PrvkoyPeFEpyzXrzpygmeXr3IjoDfQDbgImSfpkzjGVyvHpvW0ITAXukLR28UGSVqt4ZFVAUg9gFHBlV8pI50ddGdwE7CZps7wDMasGK8tfbLNuLSKWApeTjCC+JYCkb0h6Nq1de1jS55qPl7SNpAfSfW9IelDSOum+5TUcabPVAOCqtDbrL60cc5GkPxbGkzYpLpK0Zro+VNJkSfMkvSTpgqyJVUQ0AuNJZo/YTNLItDnre+mI29PSa+wq6ZH0np6RdGJxWZKOljQrvecJknoX7Dtf0n/T+3xB0smthLO3pOfSa/xJ0kcLzm+QNLq1eyhsYpV0GHAGMDy91v8kbSrpZUkHFZ13vaSr2/hp6oF1gAcLjm/+bX4g6VVJr0sa1/xbFzTVHqdkNoDFwEcl9ZN0taTZ6TO6RdIGBeV+TNLt6X0/B+xdFGeLGkFJvSVdnP6eiyQ9raSZ9/vAV4CjC+69R3rOgZKmSlooaYakrxRd49j0ubwt6bck0xQtFxFvk0x35FlFzHCCZlYVlMy99y3gfeDfkg4HzgW+SlK7diVwj6RN0lN+BfwFWBfYADgFeK+43LTZ6iXS2qyI2LOVy18L7CNp/YJtxwC3RMQ7aRLzD+A2YCNgR+ALwOkZ721N4ETgLZLJuQEGktSsbQ5sJ2kQcA/w6/R+RwIXSPpyQVE9gP2ArYDBJPNI/qxg/9PAziQTun8tPX+vonC+CuxCMmnxMmBilnsoFBGTgPOBhvQ37R0RL5BMP3d8wX2vTTJPcFs1ZMOA5yKiqWj7JiRJ9SdIfuv9gO8VHXMEsBvJvc4jmWMzSCay3wRYBNxYcPwNJFM7DSC5/5Ed3ObVwPYk8z6uRZI0vZo2Ad8AXFdw702SvpCeczLJn8mjgV9K2iX9LT5H8mf26+n+e4HDWrnu9PR3Mat5TtDM8vUjJS+fzwEOAA6OiOdJEqQrIuKRiFgaEVcDT5D8wwxJMjYA2Dgi3o+IhyPinRUJICKeBh4HjgSQ1IcksbgmPeSrwL8j4oqIeC8iXgYuSLe354r03v4LfBr4YkQsSve9D/wwIhojYjFwOPBYRExI7/dh4AoKEp7UD9IJ218Dfkwy+fAq6X1MjIhXIvE34E7SiaULnB0Rc9Pamu8BX5C0YdbfqgNXpeVtlK4fAbyQ3ktr1gHebmX7MpI5MhvTxO9CPpxQNd/He8A2wLbAt9LfZjHwfZLmwv5pPLsBp6X75wJnt3UTaUJ+KPD1iHgx/T2fT/9ctuUk4Ofpu3rLIuJfJMlv85+RrwK3RsS96fO9HvhXK+W8TZLAmdU8J2hm+RobEX0j4qMRsVNENPek2xh4sejYF9LtkCRwqwAPSHpR0rmSVu1CHNfyQRJwKDAnIv5fuj4I+GzadLUwTbquAT7WQZknpve2QUTsXlAeJLUx7xasd3S/zWYVfJ8JrA6sByDpO5KmK+mUsJCk5mn9ovNntvK9zZ6dnRERL5HUDDVPgnw87b9f9iZJ7VSx19Mkq9nMVmKcWfB9EMnv8FrB83kBWEKSxDefW/jbFf/WhQamn8+1c0yxQcAPiv6MjCSpJSWNYWbROa3FsBbwRieua7bScoJmVp1m88E/lM0+kW4nrdk4NiL6kzQ/HU/bNVrLMlzvZuCTkoaR/MN6bcG+WcB9abLVvKyddgBYUcUxtXu/BTYp+D4QeBeYL+mzwE9JmlLXS3tc3gGo6PyBrXyfkz3s5dr6Ta8AjpG0DUmt4W/bKeNxkt+8R9H2j0rqVRRncYyF158FvAOsW/SM1oiIB4GX0+OKf7u2zEw/N29jf2v3PgsYU3T9PhGxT7r/5Vau2VoMQ0l+F7Oa5wTNrDpNAE6U9BlJq0o6Bqgjfa8ofVm+uXZiIbCU5B2j1syl7X9sAYiIhcAfgPOAHYDrCnZfD9SnL3n3lLSKpE9I2ru1slbQTcC2kr6a3u9nSJKt4hfsL5C0VtoMNwb4bUQsI6l5aSJ5HyskfREY0cp1zpS0gaS1SBK6+yLilRWIdy4wIH13sNCdJLVZVwO/j4g32ynjUZJnt2PR9lWAn0paQ9IngNNo+TyKTQH+DVwmqR+ApPUl/R9ARMwBGoAL099uA5Lm4VZFxOvArcDlaacESdpMH/SunAt8Qi17j14KfDftSNBD0kckbSupPt3/W+AQSbunz/dIknfclkub1j8D3N7OvZrVDCdoZlUoIm4keU9oIrAA+AawT0Q0N1PtBkyV9A7wEEni1lZtzXnAkWnT393tXPZakqRmckS8WhDLXODzwIEktStvkiRzn1ixu/uwiHgR2Idk2IkFJPdyZkTcUnBYE0kCNB14luTdtlPSfZNJEsl/AfNJ3qH7QyuXmgj8k6Rm7iPAUSsY8u/SMuamTXqD0vtoIknOtqGD4TPSY3/Jh9+zm0VSY/Yi8AhJ54lWx2dLy1lG8v6iSP5MLAIeBoYXHHYESeI4m+T+r+/g/o4l6V37D5IOB3/igybtq4A1gQXpvfeIiL+QdMy4iOT3fxW4hKTnLhHxD+Db6blvkPQinVR0zcOBv0fEfzqIzawmKCLyjsHMbKUhaSRwekR0ODuApDVIOn/sGxHPpueOjoiaGgtM0urAk8D+ETEj73jMqkFXXio2M7MCaTPdScBlWY5Px4hrt/m5FqQdRmr+dzAr5CZOM7MSUDIw7mskTZTjcw7HzLo5N3GamZmZVRnXoJmZmZlVGSdoZmZmZlXGCZqZmZlZlXGCZmZmZlZlnKCZmZmZVRknaGZmZmZV5v8D+PBg5GHrSigAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    }
  ]
}