1-supervised.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7d1633d6-3e47-4d79-86d5-24cff79bc20e",
   "metadata": {},
   "source": [
    "# Lab 1: Benchmarking data transformations and machine learning algorithms for cell type identification from single-cell RNAseq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "74731cb0-18a5-44a1-ba0a-ca9bbbf5172d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# import base modules\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1907f2bf-ff6a-44e8-807b-c991abb82e76",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sns.set(style=\"ticks\", font_scale=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfe07258-6d72-4df6-8534-f5ffe1aa47da",
   "metadata": {},
   "source": [
    "seaborn (works on top of matplotlib) figure styles: [https://seaborn.pydata.org/tutorial/aesthetics.html#seaborn-figure-styles](https://seaborn.pydata.org/tutorial/aesthetics.html#seaborn-figure-styles)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e184ff73-84c7-4ec7-bcf1-2cb662a89579",
   "metadata": {},
   "source": [
    "# 1. Problem definition "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd97746b-f977-459f-97e8-973707793cd8",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 1.1 Data\n",
    "\n",
    "We shall use the single cell gene expression of human peripheral blood mononuclear cells (PBMC) as prepared in Luecken et al., 2021 ([Benchmarking atlas-level data integration in single-cell genomics](https://www.nature.com/articles/s41592-021-01336-8)).\n",
    "\n",
    "Data comes from 2 studies:\n",
    "\n",
    "Freytag et al., 2018: [Comparison of clustering tools in R for medium-sized 10x Genomics single-cell RNA-sequencing data](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6124389/)\n",
    "\n",
    "Sun et al., 2019: [A Bayesian mixture model for clustering droplet-based single-cell transcriptomic data from population studies](https://www.nature.com/articles/s41467-019-09639-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a205d95-272a-44d2-8df7-1b03ba8000f8",
   "metadata": {},
   "source": [
    "## 1.2 Task\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d644719-0443-45d2-a2ba-efd922c5f3b6",
   "metadata": {
    "tags": []
   },
   "source": [
    "In this tutorial, we'll first build a model for cell type classification using datasets within Sun et al., 2019 (\"sun\") and later also predict it on Freytag et al., 2019 (\"freytag\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb4f67e1-b512-48d8-8188-b9678f496c3b",
   "metadata": {},
   "source": [
    "## 1.3 Evaluation metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba59901f-a40a-47a7-b424-c7981bd43527",
   "metadata": {
    "tags": []
   },
   "source": [
    "# overall accuracy\n",
    "\n",
    "$acc = m/n$,\n",
    "where,\n",
    "\n",
    "$m$: Number of predictions where a label matches exactly with the ground truth, and\n",
    "\n",
    "$n$: Number of total samples\n",
    "\n",
    "Between 0 and 1. 1 is the best. By chance is 0.5 (in binary case)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b72c182-2fa7-4e2c-a95d-71ea4f8a9181",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Balanced accuracy\n",
    "\n",
    "balanced_acc is the average of recall per class.\n",
    "\n",
    "$\\text{Recall} = \\frac{\\text{TP}}{\\text{TP}+\\text{FN}},$ where\n",
    "\n",
    "TP: True positive\n",
    "TN: True negative\n",
    "FP: False positive\n",
    "FN: False negative\n",
    "\n",
    "Between 0 and 1. 1 is the best. By chance: $\\frac{1}{\\text{number of classes}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03e53041-18bb-469d-8be0-60f83daaa2cd",
   "metadata": {},
   "source": [
    "# Read in the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2defce74-ce12-49e9-abe5-2fa127caa2e1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['.gitignore',\n",
       " 'freytag_counts.pqt',\n",
       " 'freytag_meta.csv',\n",
       " 'sun_counts.pqt',\n",
       " 'sun_meta.csv']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(\"../data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27a1b3c8-0da8-4121-83fb-5008094f8091",
   "metadata": {
    "tags": []
   },
   "source": [
    "The count data is large and we saved it as parquet files which provides an efficient way to strore and read large data matrices among other things. \n",
    "\n",
    "\"Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides high performance compression and encoding schemes to handle complex data in bulk and is supported in many programming language and analytics tools.\" From [https://parquet.apache.org/](https://parquet.apache.org/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5c52af7e-1bee-46fa-8807-82213fa13a1d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Sun\n",
    "\n",
    "df = pd.read_parquet(os.path.join(\"../data\", \"sun_counts.pqt\"))\n",
    "df_meta = pd.read_csv(os.path.join(\"../data\", \"sun_meta.csv\"), index_col=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "fb028a23-1b6f-41da-bb4a-4259b6873004",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "count data size: (8829, 10851)\n",
      "metadata size: (8829, 14)\n"
     ]
    }
   ],
   "source": [
    "print(f\"count data size: {df.shape}\")\n",
    "print(f\"metadata size: {df_meta.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03f068de-2aab-49d3-95f6-daffaffcde09",
   "metadata": {},
   "source": [
    "Gene expression data consists of \"counts\" and the associated metadata. Below is a protocol on how this data is generated for a specific technology (10x Chromium)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edc3ae6e-eda6-4bfc-a12e-2df7f3015fef",
   "metadata": {
    "tags": []
   },
   "source": [
    "![Single cell RNA sequencing](images/10xschematic.jpg \"Title\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "689f9b1a-86f9-4d9f-a95b-801714352d8a",
   "metadata": {
    "tags": []
   },
   "source": [
    "The result is a count matrix (a matrix of integer where each value indicates the copies of transcripts for each genes.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c342ca00-a92d-42eb-b2b5-4db7a981a8a4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>index</th>\n",
       "      <th>LINC00115</th>\n",
       "      <th>NOC2L</th>\n",
       "      <th>KLHL17</th>\n",
       "      <th>PLEKHN1</th>\n",
       "      <th>HES4</th>\n",
       "      <th>ISG15</th>\n",
       "      <th>AGRN</th>\n",
       "      <th>TNFRSF18</th>\n",
       "      <th>TNFRSF4</th>\n",
       "      <th>SDF4</th>\n",
       "      <th>...</th>\n",
       "      <th>COL6A1</th>\n",
       "      <th>COL6A2</th>\n",
       "      <th>LSS</th>\n",
       "      <th>MCM3AP-AS1</th>\n",
       "      <th>MCM3AP</th>\n",
       "      <th>YBEY</th>\n",
       "      <th>PCNT</th>\n",
       "      <th>DIP2A</th>\n",
       "      <th>S100B</th>\n",
       "      <th>PRMT2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AAACCTGCACAGACAG-1-Sun_sample1_CS</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AAACGGGCAGTAGAGC-1-Sun_sample1_CS</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AAACGGGCATGGTAGG-1-Sun_sample1_CS</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AAACGGGGTTCCGTCT-1-Sun_sample1_CS</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AAACGGGTCATCGCTC-1-Sun_sample1_CS</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 10851 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "index                              LINC00115  NOC2L  KLHL17  PLEKHN1  HES4  \\\n",
       "index                                                                        \n",
       "AAACCTGCACAGACAG-1-Sun_sample1_CS        0.0    0.0     0.0      0.0   0.0   \n",
       "AAACGGGCAGTAGAGC-1-Sun_sample1_CS        0.0    0.0     0.0      0.0   0.0   \n",
       "AAACGGGCATGGTAGG-1-Sun_sample1_CS        0.0    1.0     0.0      0.0   0.0   \n",
       "AAACGGGGTTCCGTCT-1-Sun_sample1_CS        0.0    0.0     0.0      0.0   0.0   \n",
       "AAACGGGTCATCGCTC-1-Sun_sample1_CS        0.0    0.0     0.0      0.0   0.0   \n",
       "\n",
       "index                              ISG15  AGRN  TNFRSF18  TNFRSF4  SDF4  ...  \\\n",
       "index                                                                    ...   \n",
       "AAACCTGCACAGACAG-1-Sun_sample1_CS    0.0   0.0       0.0      0.0   0.0  ...   \n",
       "AAACGGGCAGTAGAGC-1-Sun_sample1_CS    1.0   0.0       0.0      0.0   0.0  ...   \n",
       "AAACGGGCATGGTAGG-1-Sun_sample1_CS    0.0   0.0       0.0      0.0   0.0  ...   \n",
       "AAACGGGGTTCCGTCT-1-Sun_sample1_CS    0.0   0.0       0.0      0.0   0.0  ...   \n",
       "AAACGGGTCATCGCTC-1-Sun_sample1_CS    0.0   0.0       0.0      0.0   0.0  ...   \n",
       "\n",
       "index                              COL6A1  COL6A2  LSS  MCM3AP-AS1  MCM3AP  \\\n",
       "index                                                                        \n",
       "AAACCTGCACAGACAG-1-Sun_sample1_CS     0.0     0.0  0.0         0.0     0.0   \n",
       "AAACGGGCAGTAGAGC-1-Sun_sample1_CS     0.0     0.0  0.0         0.0     0.0   \n",
       "AAACGGGCATGGTAGG-1-Sun_sample1_CS     0.0     0.0  0.0         0.0     0.0   \n",
       "AAACGGGGTTCCGTCT-1-Sun_sample1_CS     0.0     0.0  0.0         0.0     0.0   \n",
       "AAACGGGTCATCGCTC-1-Sun_sample1_CS     0.0     0.0  0.0         1.0     0.0   \n",
       "\n",
       "index                              YBEY  PCNT  DIP2A  S100B  PRMT2  \n",
       "index                                                               \n",
       "AAACCTGCACAGACAG-1-Sun_sample1_CS   0.0   1.0    0.0    0.0    0.0  \n",
       "AAACGGGCAGTAGAGC-1-Sun_sample1_CS   0.0   0.0    0.0    0.0    0.0  \n",
       "AAACGGGCATGGTAGG-1-Sun_sample1_CS   0.0   0.0    0.0    0.0    3.0  \n",
       "AAACGGGGTTCCGTCT-1-Sun_sample1_CS   1.0   0.0    1.0    0.0    0.0  \n",
       "AAACGGGTCATCGCTC-1-Sun_sample1_CS   1.0   0.0    0.0    0.0    2.0  \n",
       "\n",
       "[5 rows x 10851 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print head of data\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aff9d452-3fa2-4352-9e10-a526ff725ae9",
   "metadata": {
    "tags": []
   },
   "source": [
    "The dataset has 8,829 cells and 10,851 genes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ba77327b-9184-4c2d-b519-e37a24ef3355",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>batch</th>\n",
       "      <th>chemistry</th>\n",
       "      <th>data_type</th>\n",
       "      <th>dpt_pseudotime</th>\n",
       "      <th>final_annotation</th>\n",
       "      <th>mt_frac</th>\n",
       "      <th>n_counts</th>\n",
       "      <th>n_genes</th>\n",
       "      <th>sample_ID</th>\n",
       "      <th>size_factors</th>\n",
       "      <th>species</th>\n",
       "      <th>study</th>\n",
       "      <th>tissue</th>\n",
       "      <th>cell_type</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AAACCTGCACAGACAG-1-Sun_sample1_CS</th>\n",
       "      <td>Sun_sample1_CS</td>\n",
       "      <td>10X</td>\n",
       "      <td>UMI</td>\n",
       "      <td>NaN</td>\n",
       "      <td>CD4+ T cells</td>\n",
       "      <td>0.023411</td>\n",
       "      <td>5083.0</td>\n",
       "      <td>1269.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1.050655</td>\n",
       "      <td>Human</td>\n",
       "      <td>Sun</td>\n",
       "      <td>PBMCs</td>\n",
       "      <td>CD4+ T cells</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AAACGGGCAGTAGAGC-1-Sun_sample1_CS</th>\n",
       "      <td>Sun_sample1_CS</td>\n",
       "      <td>10X</td>\n",
       "      <td>UMI</td>\n",
       "      <td>NaN</td>\n",
       "      <td>CD4+ T cells</td>\n",
       "      <td>0.033540</td>\n",
       "      <td>4025.0</td>\n",
       "      <td>1314.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1.179662</td>\n",
       "      <td>Human</td>\n",
       "      <td>Sun</td>\n",
       "      <td>PBMCs</td>\n",
       "      <td>CD4+ T cells</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AAACGGGCATGGTAGG-1-Sun_sample1_CS</th>\n",
       "      <td>Sun_sample1_CS</td>\n",
       "      <td>10X</td>\n",
       "      <td>UMI</td>\n",
       "      <td>NaN</td>\n",
       "      <td>CD4+ T cells</td>\n",
       "      <td>0.020093</td>\n",
       "      <td>5574.0</td>\n",
       "      <td>1246.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1.050509</td>\n",
       "      <td>Human</td>\n",
       "      <td>Sun</td>\n",
       "      <td>PBMCs</td>\n",
       "      <td>CD4+ T cells</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AAACGGGGTTCCGTCT-1-Sun_sample1_CS</th>\n",
       "      <td>Sun_sample1_CS</td>\n",
       "      <td>10X</td>\n",
       "      <td>UMI</td>\n",
       "      <td>NaN</td>\n",
       "      <td>CD4+ T cells</td>\n",
       "      <td>0.021680</td>\n",
       "      <td>6227.0</td>\n",
       "      <td>1381.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1.266189</td>\n",
       "      <td>Human</td>\n",
       "      <td>Sun</td>\n",
       "      <td>PBMCs</td>\n",
       "      <td>CD4+ T cells</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AAACGGGTCATCGCTC-1-Sun_sample1_CS</th>\n",
       "      <td>Sun_sample1_CS</td>\n",
       "      <td>10X</td>\n",
       "      <td>UMI</td>\n",
       "      <td>NaN</td>\n",
       "      <td>CD4+ T cells</td>\n",
       "      <td>0.068300</td>\n",
       "      <td>8287.0</td>\n",
       "      <td>1892.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1.758033</td>\n",
       "      <td>Human</td>\n",
       "      <td>Sun</td>\n",
       "      <td>PBMCs</td>\n",
       "      <td>CD4+ T cells</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            batch chemistry data_type  \\\n",
       "index                                                                   \n",
       "AAACCTGCACAGACAG-1-Sun_sample1_CS  Sun_sample1_CS       10X       UMI   \n",
       "AAACGGGCAGTAGAGC-1-Sun_sample1_CS  Sun_sample1_CS       10X       UMI   \n",
       "AAACGGGCATGGTAGG-1-Sun_sample1_CS  Sun_sample1_CS       10X       UMI   \n",
       "AAACGGGGTTCCGTCT-1-Sun_sample1_CS  Sun_sample1_CS       10X       UMI   \n",
       "AAACGGGTCATCGCTC-1-Sun_sample1_CS  Sun_sample1_CS       10X       UMI   \n",
       "\n",
       "                                   dpt_pseudotime final_annotation   mt_frac  \\\n",
       "index                                                                          \n",
       "AAACCTGCACAGACAG-1-Sun_sample1_CS             NaN     CD4+ T cells  0.023411   \n",
       "AAACGGGCAGTAGAGC-1-Sun_sample1_CS             NaN     CD4+ T cells  0.033540   \n",
       "AAACGGGCATGGTAGG-1-Sun_sample1_CS             NaN     CD4+ T cells  0.020093   \n",
       "AAACGGGGTTCCGTCT-1-Sun_sample1_CS             NaN     CD4+ T cells  0.021680   \n",
       "AAACGGGTCATCGCTC-1-Sun_sample1_CS             NaN     CD4+ T cells  0.068300   \n",
       "\n",
       "                                   n_counts  n_genes  sample_ID  size_factors  \\\n",
       "index                                                                           \n",
       "AAACCTGCACAGACAG-1-Sun_sample1_CS    5083.0   1269.0          3      1.050655   \n",
       "AAACGGGCAGTAGAGC-1-Sun_sample1_CS    4025.0   1314.0          3      1.179662   \n",
       "AAACGGGCATGGTAGG-1-Sun_sample1_CS    5574.0   1246.0          3      1.050509   \n",
       "AAACGGGGTTCCGTCT-1-Sun_sample1_CS    6227.0   1381.0          3      1.266189   \n",
       "AAACGGGTCATCGCTC-1-Sun_sample1_CS    8287.0   1892.0          3      1.758033   \n",
       "\n",
       "                                  species study tissue     cell_type  \n",
       "index                                                                 \n",
       "AAACCTGCACAGACAG-1-Sun_sample1_CS   Human   Sun  PBMCs  CD4+ T cells  \n",
       "AAACGGGCAGTAGAGC-1-Sun_sample1_CS   Human   Sun  PBMCs  CD4+ T cells  \n",
       "AAACGGGCATGGTAGG-1-Sun_sample1_CS   Human   Sun  PBMCs  CD4+ T cells  \n",
       "AAACGGGGTTCCGTCT-1-Sun_sample1_CS   Human   Sun  PBMCs  CD4+ T cells  \n",
       "AAACGGGTCATCGCTC-1-Sun_sample1_CS   Human   Sun  PBMCs  CD4+ T cells  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print head of metadata\n",
    "df_meta.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21648701-1228-45f6-885a-efeaaa65ba84",
   "metadata": {},
   "source": [
    "Our column of interest is \"cell_type\" which we will train models on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a2f7bc65-d2cb-4526-bd23-3b31f906e429",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# prepare X and y\n",
    "\n",
    "X = df.copy() # changes made in X won't affect df. For dictionaries, use deepcopy()\n",
    "y = df_meta[\"cell_type\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c3e03a0e-eb1b-4b80-a27a-1ceb15801649",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(8829, 10851) (8829,)\n"
     ]
    }
   ],
   "source": [
    "# updated sizes\n",
    "print(X.shape, y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af25cbc4-22a2-45be-92f3-0d04b268aea5",
   "metadata": {},
   "source": [
    "# Data split"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8bd0449-113b-40f7-8215-7b4aa0837024",
   "metadata": {
    "tags": []
   },
   "source": [
    "Create a training set and a test set.\n",
    "\n",
    "Two rules apply:\n",
    "1. There should be no leakage between the train and test set. If there are more samples from one patient, both samples should remain in the same split.\n",
    "2. Test set should be sufficiently large to assess the performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "136fa510-ce1b-48ed-a71f-0ceee8a5ee18",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "91cbb17d-591d-4199-9d1c-ab79919ed7b8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# We'll do a 80-20 split\n",
    "train_X, test_X, train_y, test_y = train_test_split(X, y, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c46e8bd4-c777-47db-8d8e-d70c78b30da4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(7063, 10851)\n",
      "(1766, 10851)\n",
      "(7063,)\n",
      "(1766,)\n"
     ]
    }
   ],
   "source": [
    "for i in train_X, test_X, train_y, test_y:\n",
    "    print(i.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ccf8320-97dc-49d1-b410-30cf506f3851",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Preprocessing and QC"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45fd8b46-7f19-4dcf-9afb-e933952196c3",
   "metadata": {},
   "source": [
    "Before we proceed with building a classifiers, there are some things to keep in mind:\n",
    "    \n",
    "1. There may be poor quality samples that we want to avoid learning from,\n",
    "2. There may be some features that might be redundant, and \n",
    "3. In cases like the one in hand, we have more features than samples (10,851 vs 7,063). Such high dimensional data may be difficult for learners. This refers to \"curse of dimensionality\", especially when the data can not be represented appropriates in 3D physical spaces. For gene expression data, this is the case, however due to biology, several genes are cell type-specific i.e. they are expressed higher in specific cell types. **Domain knowledge helps**!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea9bfb4d-dbd4-4b79-9044-8b73adc33bb1",
   "metadata": {},
   "source": [
    "### Removing atypical samples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b6f9d6a-b6d1-415e-ad6a-8b3c06c35693",
   "metadata": {},
   "source": [
    "First we will compute total genes expressed per sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "87928063-8b8a-4dfc-abf5-006134b79133",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def count_genes(df):\n",
    "    df_bool = df.astype(bool)\n",
    "    exp_genes = df_bool.sum(axis=1)\n",
    "    return exp_genes\n",
    "\n",
    "def count_total_count(df):\n",
    "    sums = df.sum(1)\n",
    "    return sums"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f3a1200f-ccfb-46aa-859b-2105fdcfa591",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gene_counts, total_counts = count_genes(train_X), count_total_count(train_X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "15e4366b-2583-41db-8ef9-ae8ce6d8d408",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0UAAAErCAYAAAAR5ii2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABRzUlEQVR4nO3deVxU9f4/8BcgA4mMiAuoqCU46AQuqIC5oGDlAlfNa9cMxQRcMDVvrl1yu2aZC+GCG1hqaMrNrEjaNLFFrcRKLbuK/nK5hLjAsMiMwPn94XemmWEGZnCG2V7Px6OHzZnPnPmcM4f5zPt83ud9nARBEEBEREREROSgnC3dASIiIiIiIktiUERERERERA6NQRERERERETk0BkVEREREROTQGBQREREREZFDY1BEREREREQOjUERERERERE5NAZFRERERETk0BgUERERERGRQ2NQ9BAOHjyIwMBABAcH48aNG7WenzhxIqKjoy3QM+DUqVMIDAzEp59+apH3N9b169cxdepUhIaGIjAwEK+99pqlu2TXAgMDsXHjRkt3g0xI+Td/6tQp1bKNGzciMDDQgr0ispzAwECD/lP/m9Fn69at+PLLLx+6P/bwvZuZmYmDBw9auhtWbeLEiZg4caLGMnv5/O1ZE0t3wB4oFAq89dZbWLNmjaW7YrNef/11/Pzzz1i1ahVatWqF1q1bW7pLRERkw/bv36/xOC0tDadOncKuXbs0lgcEBNS7rm3btuHpp5/G0KFDTdpHW7Rv3z60aNECzzzzjKW7QmRSDIpMYODAgcjOzkZ8fDy6du1q6e40qsrKSri5ucHJyemh1nPx4kV0797d5gace/fu4ZFHHrF0N8iKVVZWwt3d3dLdIHI4PXv21Hjs7e0NZ2fnWsvJMZjq9wrZL6bPmUBCQgK8vLzqnSm6fv06AgMDdU47a0+rKtNeLly4gNmzZ6N3794IDQ3F66+/jqqqKly+fBnx8fHo1asXIiMjsWPHDp3vKZfL8frrr6N///7o3r07YmNj8euvv9Zqd/bsWUyfPh2hoaEIDg7G6NGjcfjwYY02ynTBb775BosXL0Z4eDh69OgBhUKhd5v/97//Yd68eejXrx+CgoIwfPhw7Ny5EzU1NQD+Svn5448/cPz4cVU6w/Xr1/WuUxAEZGZmYtSoUejevTv69u2L2bNn49q1a6o2n3zyCQIDA/Huu+9qvHbDhg3o1q0bvv32WwB/fSY7duzAli1bMHjwYAQHB+OZZ57BiRMnNF6r/EzOnz+P2bNno2/fvnjyyScN7hMA/Prrr5g2bZpqfwwYMABTp07Fn3/+qWqTk5ODcePGoXfv3ujRoweioqKwePFijfWUlZVh9erViIyMRFBQEAYOHIjXXnsNFRUVtdolJycjLCwMvXr1Qnx8PK5cuaJ332q7ePEipkyZgh49eiA8PBzLly/HsWPHdKacfPfdd4iLi0NISAh69OiB8ePH692HFy9exD//+U/07t0bTzzxBBYvXozS0lKNtqbcp7oo01t//PFHPPvss+jevTsGDhyIt956C9XV1RptFQoF0tLSMGzYMAQFBSE8PByLFy/GnTt3NNpFRkZi2rRp+PzzzzF69GgEBwdj06ZNdfbj+PHjiIuLU33ew4cPx7Zt2zTaGPL3SUTGKy4uxrJlyzBw4EAEBQUhKioKKSkpGuNaYGAgKioq8MEHH6jGKGVq1J07d7Bs2TKMGDECvXr1Qr9+/TBp0iT8+OOPDe6TQqHApk2bMHz4cAQHByMsLAwTJ05EXl6eqo1cLse6des0xoDly5dDJpNprEtfylZkZCQWLVqkeqwc30+ePImlS5ciLCwMYWFhePHFF1FYWKjxuosXL+L7779X7YvIyEgAQE1NDdLS0vD000+je/fu6NOnD2JiYmrNzGlT/g748MMPLfp7RSaT4Y033kBUVBSCgoLQr18/JCYmIj8/X9XG0LGAbBNnikzAw8MDM2bMwGuvvYYTJ06gX79+Jlv3Sy+9hL/97W8YP348vv32W6Snp6OqqgrfffcdJkyYgPj4eHz88cdYu3YtOnXqhKeeekrj9SkpKZBKpVi5ciVKS0uxadMmTJw4EYcOHUKHDh0AACdPnkRCQgJ69OiBZcuWwdPTE4cPH8bcuXNRWVlZa4r8lVdeweDBg/Hmm2/i3r17aNJE92F0584djB8/Hvfv38ecOXPQvn17HDt2DKtXr8bVq1exbNkyPP7449i/fz9efPFFdOjQAQsXLgQAtGnTRu8+WbJkCT744ANMnDgR8+bNQ0lJCTZv3ozx48fjww8/RKtWrTBy5Eh8//33eOONN9CjRw8EBwfjxIkT2LJlC6ZNm4b+/ftrrDMzMxPt2rXDK6+8gpqaGqSnpyMxMRF79uxBr169NNrOmjULI0aMwPjx41VBiCF9qqiowAsvvAA/Pz8sWbIErVq1QlFREU6dOoXy8nIAwJkzZzB37lyMGDECL774Itzc3PC///0PJ0+eVL3/vXv3EBsbiz///BPTp09XBRkbNmzAf//7X7zzzjtwcnKCIAhISkrCmTNnMHPmTAQHByMvLw+JiYl69626mzdvIjY2Fk2bNsWyZcvg7e2NTz75BCtWrKjV9sMPP8TChQsRFRWF1atXo0mTJti/fz/i4+ORkZFR629CuQ///ve/47///S/WrVsH4EEapTGfsyH7tC5FRUWYO3cupk6ditmzZ+PYsWPYsmULZDIZlixZAuDBQJ+UlITTp08jPj4eISEhuHHjBjZu3IhffvkF77//vsZM0Pnz55Gfn48ZM2bAz8+vzpnErKwsvPrqq+jbty+WL1+Oli1b4sqVK7h48aKqjbF/n0RkGLlcjkmTJuHatWuYNWsWAgMD8eOPP2L79u347bffsH37dgAP0vDi4uIQFhaGpKQkAECzZs0APAiqAODFF19UfSd98cUXmDhxIt555x2EhYUZ1aeqqiokJCTg9OnTmDRpEsLDw1FdXY2ff/4ZBQUFAKD6bj958iSmTp2KPn364Pfff8fGjRvx008/Yf/+/RCJRA3aJ8nJyRg8eDDWrVuHgoICrFmzBvPnz8fu3bsBAJs2bcLs2bPh6emJpUuXAoDqvdLT07Fp0ybMmDEDffr0UZ3A1T7hpY8lf6+UlZVhwoQJuHHjhmr9FRUV+OGHH1BUVAR/f3+jxwKyQQI12Pvvvy9IJBLhl19+EeRyuRAVFSU888wzQk1NjSAIghAbGyuMHDlS1f7atWuCRCIR3n///VrrkkgkwoYNG1SPN2zYIEgkEmHnzp0a7UaNGiVIJBLh888/Vy27f/++EB4eLrz44ouqZSdPnhQkEokwZswYVX8EQRCuX78uPP7448K//vUv1bJhw4YJo0ePFu7fv6/xXtOmTRP69+8vVFdXa2zvggULDNo/a9euFSQSifDzzz9rLF+6dKkQGBgoXL58WbVsyJAhwtSpU+td55kzZ3Tul4KCAqF79+7Cm2++qVoml8uF0aNHC5GRkcKlS5eEJ554QoiNjRWqqqpUbZSfyYABA4TKykrV8tLSUiE0NFSYPHmyapnyM0lNTW1Qn86ePStIJBLhiy++0Lt9GRkZgkQiEWQymd4227ZtE7p27Sr88ssvGss//fRTQSKRCMeOHRMEQRByc3MFiUQi7Nq1S6Pdli1bah1vuqxevVoIDAwULl68qLF8ypQpgkQiEU6ePCkIgiBUVFQIoaGhwrRp0zTaVVdXC3/729+Ev//976plyn24Y8cOjbbLli0TgoODVceqKfepPrGxsYJEIhG+/PJLjeXJyclC165dhRs3bgiCIAjZ2dmCRCIRPvvsM412v/zyiyCRSITMzEzVsiFDhgjdunXTOLb1KSsrE0JCQoTnnntO429Um6F/n8q/eeXnIgh/7W8iEoSFCxcKPXv2VD3et2+fIJFIhMOHD2u02759uyCRSIRvvvlGtaxnz57CwoUL632Pqqoq4f79+0JcXJwwc+ZMjecM+d794IMPBIlEIhw4cEBvm+PHj+v8Hv3kk08EiUQi7N+/v973HDJkiMb2KMf3ZcuWabTbsWOHIJFIhJs3b6qWjRw5UoiNja21zmnTpgmjRo2qc/t0sYbfK5s2bRIkEonw7bff6m1jzFgQGxtbax8Z8vmTZTF9zkREIhFeeuklnDt3Djk5OSZb7+DBgzUe+/v7w8nJCYMGDVIta9KkCTp16qSzAl50dLRG/mz79u3Rq1cvVerTH3/8gcuXLyMmJgbAg7NUyv8GDRqEoqKiWulW2rNR+pw8eRIBAQHo3r27xvJnnnkGgiBozH4Y6quvvoKTkxP+9re/afS1VatW6Nq1K77//ntVW5FIhLfeegvFxcUYM2YMBEHAunXr4OLiUmu9Tz31FNzc3FSPmzVrhiFDhuCHH36olUqlvf2G9qlTp05o3rw51q5di3379uHSpUu1+hEcHAzgwQzh4cOHNdIW1N+vS5cu6Natm8b7DRgwAE5OTqr3U37Gys9WydCKiD/88AO6dOlS6yJk7defOXNGtY/V+1NTU4OBAwfi7NmztdL6lOkWSoGBgZDL5bh9+7ZqG021T+vi4eGBqKioWttXU1ODH374QdUXsViMIUOGaPSlW7duaN26tcYxp9yWxx57rN73PnPmjOrspL4c94b8fRKRYU6ePImmTZti2LBhGsuVsw3a6b/67Nu3D2PGjEFwcDCkUikef/xxnDhxQiPtylBff/013NzcMHbs2Dr7rd5PpeHDh6Np06YG91sXXd/NwINU+PoEBwfjwoULWLZsGb7++muUlZUZ9d6W/L3y9ddf49FHH8UTTzyht42xYwHZHqbPmdDIkSOxc+dOpKSkqK41eVjNmzfXeOzq6opHHnlE4we8crmuL6BWrVrpXHbhwgUAwK1btwAAq1evxurVq3X24e7duxqPDa0MV1xcjPbt29darkyNU6YdGOP27dsQBEHvF5dyil2pU6dO6NOnD44dO4bnnntOb1qevv10//59VFRUwNPTs1b/je2Tp6cn9uzZg61btyIlJQUlJSVo3bo1nn32WcyYMQOurq7o27cvNm/ejD179mDhwoVQKBTo0qULpk+frgpGbt++jT/++AOPP/64zvdTfl7FxcVo0qQJWrRoofG8MZ+fn59freUtW7bUeKw8hmbPnq13XSUlJWjatKnqsZeXl8bzyvSLyspKAKbdp3XR97kDfx2ft2/fhkwmQ1BQkM51NPTvQ5mD7uvrq7dNQ/4+icgwxcXFaNWqVa2TEi1btkSTJk0MGqPefvttvPHGGxg/fjzmzJmDFi1awNnZGampqbh8+bLRfbpz5w7atGkDZ2f956yV3+3e3t4ay52cnNCqVasGja1K9X0312XatGlo2rQpPvroI7z33ntwcXFBnz59MG/ePNUJv7pY8vfKnTt30LZt2zrbGDsWkO1hUGRCTk5OmDdvHl544QUcOHCg1vPKQEb7Qj9z/iEpv0S0lym/+JQ/mKdNm6Y3kNM+621o5RYvLy8UFRXVWn7z5k2N9zZGixYt4OTkhMzMTJ0509rLsrKycOzYMXTv3h2ZmZkYMWIEevToUet1+vaTq6urxo/5h+1TYGAgUlJSIAgCfv/9dxw8eBCbN2+Gu7s7pk6dCgAYOnQohg4dCoVCgZ9++gnbtm3Dyy+/rDpr1qJFC7i5uWHVqlV6+wM82P9VVVW4e/euxr7W9Zno4uXlpXe/6Hq/V199Vee+BWoHUvUx9T7Vp67tU/8b8fLyQnp6us51eHh4aDw29O9D+YOmroIQDfn7JCLDeHl54eeff4YgCBp/t7dv30ZVVZVBY9RHH32E0NBQLF++XGO5Idc06uLt7Y3Tp0+jpqZGb2Ck/G6/c+eORmAkCAJu3bqlEYCIRCKdxQXM8bujSZMmeOGFF/DCCy9AJpPhu+++Q0pKChISEnDs2LF6K7Va8veKt7d3vcV5jB0LyPYwfc7EnnjiCfTv3x+bN2+u9aXYqlUruLm54ffff9dYfuTIEbP1Jzs7G4IgqB7fuHEDZ86cQWhoKACgc+fOePTRR3HhwgUEBwfr/E95Qamx+vXrh0uXLuH8+fMayw8dOgQnJyejL0AFHqQTCoKAwsJCnX1Vv1Hl77//jpUrV2L06NHIzMyERCLB3LlzUVJSUmu9n3/+OeRyuepxWVkZvvrqK/Tp00dnul1D+6Tk5OSErl274pVXXoFYLK61j4AHg1loaCjmz58PAKoqPIMHD8a1a9fg5eWl8/2UszvK/fvxxx9rrDc7O7vO7VHq27cvLl68WCsl7ZNPPtF4HBISArFYjEuXLuk9hoy96Ndc+1RbeXl5rb+/7OxsODs7o2/fvqq+FBcXo6amRmdfOnfubNS2KfXq1Quenp547733NP5G1Znz75PI0fXr1w8VFRW1bsp66NAh1fNKIpFI52yJk5NTre+3Cxcu4KeffmpQnwYOHAi5XF7nzVGV/froo480ln/22WeoqKjQ6Hf79u1r/eY4ceJErZRmY+jbF+rEYjGGDRuGCRMmoLi4WGd6vzZL/l4ZOHAg/t//+391ph6aaywg68GZIjOYN28ennnmGdy+fRtdunRRLVdeI/H++++jY8eO6Nq1K3755ReDf6Q2xJ07dzBz5kw8++yzKC0txcaNGyESiTBt2jRVm+XLlyMxMRHx8fEYM2YMfHx8UFJSgvz8fJw/fx4bNmxo0HtPnjwZhw4dwrRp0zB79my0a9cOx44dw969e/Hcc8816Ax379698Y9//AOvvPIKzp07h759++KRRx5BUVERTp8+DYlEggkTJqCiogIvvfQS/Pz8sHTpUohEIqSmpmLMmDFYvHgx0tLSNNbr4uKiOsNVU1ODHTt2oKysDLNmzTJZn7766ivs3bsXQ4cORYcOHSAIAj7//HPIZDJVNbzU1FT8+eef6NevH3x9fSGTybB79264urqqBoa4uDh8/vnniI2NxeTJkxEYGIiamhoUFBTgm2++UZXQHjBgAPr27Ys1a9bg3r17CAoKQl5eHj788EOD9nVcXBzef/99JCYmYvbs2WjZsiWys7NVOdvKs5geHh5ITk7GokWLUFJSgqeffhotW7bEnTt3cOHCBdy5c6fWWdTG3Kd18fLywrJly1BQUIBHH30Uubm5OHDgAJ577jm0a9cOwIO02I8//hhTp07FxIkT0b17d7i6uuLPP//EqVOnEBUV1aB0WQ8PDyxcuBDJycmYPHkynn32WbRs2RJXr17FhQsXVNXvzPX3SeTolCfMFi5ciBs3bkAikeD06dPYtm0bIiIiNNJ3JRIJvv/+exw9ehStW7eGh4cHOnfujMGDByMtLQ0bNmxA3759ceXKFaSlpcHPz6/W9aiGiI6OxsGDB7Fs2TJcuXIFYWFhEAQBP//8M/z9/TFy5Ej0798fAwYMwNq1a1FWVoaQkBD8/vvv2LBhA6RSKUaNGqVa36hRo5CamorU1FSEhobi0qVLePfddzVSwo0lkUjwySef4PDhw/Dz84ObmxsCAwMxffp0dOnSBUFBQfD29saNGzewa9cutG/fHp06dap3vZb8vRIXF4ecnBwkJSVh6tSp6N69OyorK/HDDz9g8ODBCA8PN9tYQNaDQZEZSKVSjBw5Umewo7wvQHp6OioqKhAWFoatW7fWurjRVObOnYuzZ89i8eLFKCsrQ/fu3bF+/Xp07NhR1SY8PBxZWVnYunUrVq1aBZlMBi8vL/j7+2P48OENfm9vb2+89957WLduHdatW4fy8nL4+flh/vz5eOGFFxq83hUrVqBHjx7Yv38/9u3bh5qaGrRp0wYhISGqog5Lly5FQUEB/vOf/6jS3zp06ICVK1dizpw5eOeddzB58mTVOp9//nnI5XKsXLlSFcxu27YNvXv3NlmfOnXqBLFYjPT0dNy8eROurq547LHH8MYbb2DMmDEAgB49euDcuXNYu3Yt7ty5A7FYjKCgILzzzjuqALtp06bIzMzE9u3bsX//fly/fh3u7u5o27YtnnjiCdV1XM7OztiyZQtef/11pKen4/79+wgJCcH27dsN+lx9fHzw7rvvYtWqVVi6dCkeeeQRDB06FLNnz8bChQs1BtVRo0ahXbt2SE9Px9KlS1FeXg5vb29069ZNtW3GMtU+rUvr1q2xZMkSrF69Gv/973/RvHlzTJ8+XSMYdnFxwZYtW7B79258+OGH2L59O1xcXODr64u+fftCIpE0aPsAYNy4cWjTpg3S09ORnJwMQRDQvn17jB49WtXGXH+fRI7Ozc0Nu3fvRkpKCtLT03H37l34+PhgypQpePHFFzXa/utf/8Ly5cvxz3/+E/fu3UNoaCj27NmD6dOn4969e/jPf/6D9PR0BAQEYNmyZfjyyy8bdOF9kyZNsGPHDmzbtg2ffPIJdu3aBQ8PD3Tt2hUDBw4E8OAEa1paGjZu3IiDBw9i69at8PLywqhRo/DPf/5TY+YqPj4eZWVl+OCDD7Bz5050794dqampqtLiDTFr1iwUFRUhOTkZ5eXlaN++PY4ePYqwsDB89tlnyMrKQllZGVq3bo0nnngCSUlJ9V7fCVj290qzZs2wd+9ebNy4EQcOHMDmzZshFosRHByMZ599FoB5xwKyDk6CvrwNIgdw/fp1REVFYcGCBYiPj7d0d2zCq6++iuzsbJw6darB98KwBhMnTsTdu3fNOlNLRER1O3XqFCZNmoTU1NRalQCJGhNniohIr02bNsHHxwd+fn6oqKjAsWPHkJWVhRkzZth0QERERESkjkEREenl6uqK9PR0FBYWoqqqCo8++igWLVqEuLg4S3eNiIiIyGSYPkdERERERA6NJbmJiIiIiMihMSgiIiIiIiKHxqCIiIiIiIgcml0VWujTpw8UCgVat25t6a4QETmcoqIiiEQi/Pjjj5builXh2EREZBnGjEtGzRR9/fXXiI2NRXh4OIKCghAVFYXXX38dpaWlGu1yc3MxevRoBAcH48knn0RmZqbO9WVkZCAyMhLBwcEYO3YsTp06ZUx3apHL5aiqqnqodRARUcNUVVVBLpdbuhtWh2MTEZFlGDMuGTVTVFJSgl69eiEuLg5isRgXL17Exo0bcfHiRezcuRMAcObMGSQlJWHUqFFYtGgR8vLysHLlSohEIowbN061royMDKSkpGDu3LmQSqXIyspCYmIisrKyEBgYaEy3VNq0aQMAOHLkSINeT0REDRcVFWXpLlgljk1ERJZhzLhkVFAUHR2N6Oho1eOwsDCIRCK8+uqrKCwshI+PDzZv3gypVIpVq1YBAMLDw1FQUIDU1FSMHTsWzs7OUCgU2LJlCyZNmoT4+HgAQGhoKGJiYrB161akpKQY0y0iIiIiIqIGe+hCC15eXgAeTE8pFAqcPHkSI0eO1GgTExODoqIi/PrrrwCAvLw8lJaWagRYLi4uGDFiBHJzc8FbJxERERERUWNpUFBUXV0NuVyO8+fPY/PmzRgyZAjat2+Pq1ev4v79++jcubNG+4CAAABAfn6+xr/a7fz9/VFeXo7CwsKGdIuIiIiIiMhoDao+N2TIEFXgMnDgQKxfvx7Ag2uOAEAsFmu0Vz5WPi+TySASieDu7q7Rrnnz5gCA4uJi+Pr66nzvunIDCwoK0LZtW2M3h4iIiIiIHFiDgqLt27ejoqICly5dQlpaGqZPn463335b9byTk5PO16kv19VGmTan7/VERERERESm1qCgqGvXrgCAkJAQSKVSjB07Fl988YUqTU45I6Qkk8kA/DVjJBaLIZfLIZfL4ebmVqudcsZIl7qq97DyERERERERGeuhCy1069YNLi4uuHr1Kjp27AhXV1dcvnxZo82lS5cAPLhmSP1f5bVFSvn5+fDw8ICPj8/DdouIiIiIiMggDx0UnTlzBtXV1fDz84NIJEJ4eDhycnI02mRnZ6N169aQSqUAHswweXp64vDhw6o21dXVyMnJQUREBNPniIiIiIio0RiVPvfiiy8iKCgIgYGBcHd3x4ULF5Ceno7AwEAMHToUADBz5kzExsYiOTkZMTExyMvLQ1ZWFlasWAFn5wcxmEgkwowZM5CSkgJvb2/VzVuvXbumKtpAtuPm3QrIyhWqx2IPEdq0aGrBHhERkT1RH2c4xhCRORgVFHXv3h2HDx/G9u3bIQgC2rdvj2effRbx8fEQiUQAgF69eiEtLQ3r16/HoUOH4Ovri+TkZIwbN05jXVOmTIEgCNizZw9u3boFiUSC7du3IzAw0HRbR2Z3824Fkt48CrmiWrXMTeSCtAWRHLSIiKgWY0+kaY8zHGOIyByMCoqmTp2KqVOn1tsuIiICERERdbZxcnJCQkICEhISjOkCWRlZuQJyRTVenhACPx9PXC8sxbq9eZCVKzhgERGRhoacSFMfZwBwjCEis2hQ9TkibX4+ngjw86qzDdMfiIgc28OcSPPz8WykXhKRI2JQRI2C6Q9ERKRkyIk0IqLGxKCIGkQ563O9sNSg9kx/ICIiIiJrxaCIjKZr1kfsITLotUx/ICIiIiJrw6CIjKadE87rg4iIiIjIljEoogZjTjgRERER2QNnS3eAiIiIiIjIkjhTRBajLNLA9DsiIjKGepEfjiFEZAoMiqjRiT1EcBO5YN3ePAAsz01ERIbRHj8AjiFEZBoMiqjRtWnRFGkLIlUlvVmem4iIDKE+fgDgGEJEJsOgiCyiTYumHMCIiMhoHD+IyBxYaIGIiIiIiBwagyIiIiIiInJoTJ8js7p5t0J17RAREZGhOH4QUWNiUERmcb2wFLIyBVbt+h5yRTWABxWCxB4iC/eMiIis3c27FUh68yjHDyJqNAyKyKR0ldtentgP4mYi3kuCiIjqpD47JFdU4+UJIfDz8eT4QURmx6CITEq7XCoHMiIiMoSu2SFp55YcQ4ioUTAoIoMZmt/NcqlERGQsWbmCs0NEZDEMisggzO8mImuVk5ODjz/+GOfPn0dJSQk6dOiA5557DuPHj4ez84Miq4sWLcIHH3xQ67U7duzAoEGDNJZlZGQgMzMTRUVFkEgkWLBgAcLCwhplWwjw8/FEgJ+XpbtBRA6GQREZhGfwiMhavf3222jXrh0WLFiAli1b4tSpU3jttddw7do1LFy4UNWuQ4cOWLt2rcZr/f39NR5nZGQgJSUFc+fOhVQqRVZWFhITE5GVlYXAwMBG2R5HxApzRGRpDIrIKDyDR0TWZuvWrfD29lY9Dg8PR0VFBTIzMzF37lyIRA9mtd3d3dGzZ0+961EoFNiyZQsmTZqE+Ph4AEBoaChiYmKwdetWpKSkmHU7HJGu4jzMQiAiS2BQRERENk09IFLq1q0b5HI5iouL0aZNG4PWk5eXh9LSUkRHR6uWubi4YMSIEdi5cycEQYCTk5PJ+k0szkNE1oNBERER2Z3Tp0/Dy8sLLVu2VC27evUq+vTpg8rKSkgkEiQlJWHo0KGq5/Pz8wEAnTt31liXv78/ysvLUVhYCF9fX53vFxUVpbcvBQUFaNu27cNsjl1jcR4isgYMioiIyK6cPXsWBw8exMyZM+Hi4gLgwcxRcHAwAgICUFpain379mHmzJlITU3FsGHDAAAymQwikQju7u4a62vevDkAoLi4WG9QREREto1BEVkF9YtsmT5BRA1VVFSE2bNnIzg4GImJiarlcXFxGu0iIyMxfvx4bNiwQRUUAdCZHicIgt7nlI4cOaL3ubpmkYiIyDoYFRSx7CmZmvZFtsCDC21fiQuFuJmIARIRGay0tBSJiYlwd3fHli1b4Orqqrets7MznnrqKaxZswaVlZVwd3eHWCyGXC6HXC6Hm5ubqq1MJgPw14wRERHZH6OCIpY9JVPTvshWVqbAql3fY+mOEwAeBEhpCyIZGBFRneRyOWbMmIFbt25h//79aNGiRb2vUc4AKSnHqfz8fEilUtXy/Px8eHh4wMfHx7SdJiIiq2FUUMSyp2QO2hfZKoOk64WlWLc3D7JyBYMiItKrqqoKc+bMwYULF/Duu++iffv29b6mpqYGn332Gbp06aK6higkJASenp44fPiwKiiqrq5GTk4OIiIiWHmOiMiOGRUUsewpNQZWIiIiY6xYsQJfffUV5s+fj8rKSvz000+q5wICAlBSUoJFixYhOjoaHTt2RElJCfbt24dz585h48aNqrYikQgzZsxASkoKvL29VVkM165dw/r16y2wZURE1FgeutACy54SEZElffPNNwCANWvW1Hpu9+7dCAwMRLNmzbB582bcuXMHrq6uCAoKwo4dOzBw4ECN9lOmTIEgCNizZw9u3boFiUSC7du3M63bBt28W8H7HxGRwR4qKGLZUyIisrSjR4/W22bLli0GrcvJyQkJCQlISEh42G6RBd28W4GkN49CrqgGwOtTiah+DQ6KWPaUiIiIrJGsXAG5ohovTwgBAF6fSkT1cm7IixpS9jQ/Px+VlZUAoFH2VB3LnhIREdmfm3crcOl6scY96RqDn48n/Hw8G/U9icg2GT1TxLKnREREZChdqWxiD5GFe0VEpMmooIhlTx0DL04lIiJTUU9l8/PxNOu4ohy/GntGiohsn1FBEcue2j99F6cSERE9DD8fTwT4eZlt/fpmpJQn+YiI6mJUUMSyp/ZP38WpRERE1kzfjBTHMCIyhFFBEcueOg5emEpERLbI3DNSRGSfGlR9joiIiIiIyF4wKCIiIiIiIofGoIiIiIiIiBwagyIiIiIiInJoRt+8lYiIiMhWqd+LD+D9+IjoAQZFREREZHLWdiPV64WlkJUpsGrX96p7GQF/3Y+PgRGRY2NQRERERCal70aqliD2EMFN5IJ1e/NUfVme2A/iZiJcLyxV3Y+PQRGRY2NQRERERCal70aqltCmRVOkLYhUpcwxXY6IdGFQRERERGZhLTdSbdOiKQMhIqoTq88REREREZFD40wR1ctaLpIlIiIiIjIHBkWkl66LUy1xoawyKGMeOBERERGZA4Mi0svSF6fqCspYNpWIiIiITI1BEdXJkhenqgdlLJtKRERERObCoIisGisGEREREZG5sfocERERERE5NM4UkU1h0QUiIsu6ebdCda0pwO9jIrIPDIrIJrDoAhGR5d28W4GkN49CrqhWLeP3MRHZAwZFZBNYdIGIyPJk5QrIFdV4eUII/Hw87e77WH0WjDNgRI6FQRHZDBZdICJdcnJy8PHHH+P8+fMoKSlBhw4d8Nxzz2H8+PFwdv7r0tnc3FykpKQgPz8fvr6+mDx5Mp5//vla68vIyEBmZiaKioogkUiwYMEChIWFNeYmWT0/H08E+HnVWq4MKmzxpt/as2CcASNyLAyKiIjIpr399tto164dFixYgJYtW+LUqVN47bXXcO3aNSxcuBAAcObMGSQlJWHUqFFYtGgR8vLysHLlSohEIowbN061royMDKSkpGDu3LmQSqXIyspCYmIisrKyEBgYaKlNtAm6ggpL3PC7odRnwQDY1QwYEdWPQREREdm0rVu3wtvbW/U4PDwcFRUVyMzMxNy5cyESibB582ZIpVKsWrVK1aagoACpqakYO3YsnJ2doVAosGXLFkyaNAnx8fEAgNDQUMTExGDr1q1ISUmxyPbZCu3UusZMPzPlzJSfj6fJ1kVEtoMluYmIyKapB0RK3bp1g1wuR3FxMRQKBU6ePImRI0dqtImJiUFRURF+/fVXAEBeXh5KS0sRHR2tauPi4oIRI0YgNzcXgiCYd0PshDK1rjECIvUiPOv25tnc7BQRWQ/OFBERkd05ffo0vLy80LJlS1y5cgX3799H586dNdoEBAQAAPLz8xEUFIT8/HwAqNXO398f5eXlKCwshK+vr873i4qK0tuXgoICtG3b9mE2h/RQL8IDsDgCETWcUTNFOTk5SEpKQkREBHr27ImYmBjs3bsXNTU1Gu1yc3MxevRoBAcH48knn0RmZqbO9WVkZCAyMhLBwcEYO3YsTp061fAtIaPcvFuBS9eLcel6MW7erbB0d4iITObs2bM4ePAg4uLi4OLigpKSEgCAWCzWaKd8rHxeJpNBJBLB3d1do13z5s0BAMXFxWbuOTVEmxZNEeDn1WizU0Rkn4yaKeLFrPbBXirs8EauRKStqKgIs2fPRnBwMBITEzWec3Jy0vka9eW62ijT5vS9HgCOHDmi97m6ZpGIiMg6GBUU8WJW+2DrFXZ4I1ci0qW0tBSJiYlwd3fHli1b4OrqCuCvmR7ljJCSTCYD8NeMkVgshlwuh1wuh5ubW612yvUQEZH9MSp9jhez2hc/H0+brLKjzCFPmRuBlyeEQK6oVuWTE5FjksvlmDFjBm7duoX09HS0aNFC9VzHjh3h6uqKy5cva7zm0qVLAB5cM6T+r/LaIqX8/Hx4eHjAx8fHnJtAREQW9NDV59QvZr169Wq9F7Oq/1vXxaxEdVHmkNtiUEdEplVVVYU5c+bgwoULSE9PR/v27TWeF4lECA8PR05Ojsby7OxstG7dGlKpFAAQEhICT09PHD58WNWmuroaOTk5iIiIqDN9joiIbNtDVZ9TXsw6c+ZMk1/Mygo/jUt5fY4t3oWciBzbihUr8NVXX2H+/PmorKzETz/9pHouICAAzZo1w8yZMxEbG4vk5GTExMQgLy8PWVlZWLFiBZydH5wfFIlEmDFjBlJSUuDt7a263vXatWtYv369hbaOiIgaQ4ODIktdzEqmpX19DmB7dyEnIsf2zTffAADWrFlT67ndu3cjLCwMvXr1QlpaGtavX49Dhw7B19cXycnJGgWAAGDKlCkQBAF79uzBrVu3IJFIsH37dhYAIiKycw0Kiix5MSsr/JiW9j0eAFZzIyLbcvToUYPaRUREICIios42Tk5OSEhIQEJCgim6RkRENsLoa4p4Mav9Ub/HA+/zQERERESOxqigiBezEhERkb25XljKa2qJHJxR6XO8mJWIiIjsha773ok9RLzNA5EDMioo4sWsREREZC+0r6tVXlPLoIjI8RgVFPFiViIiIrInbVo05bW0RPTwN28lIiIiIiKyZQyKiIiIiIjIoTEoIiIiIiIih9agm7cSERERKbGcNRHZOgZFZBfUB2Rl9SAiIjIvfSWtiYhsDYMismnaAzLwYFBOWxDJwIiIyMz0lbQmIrI1DIrIpmkPyNcLS7Fubx5k5QoOzEREjcCeS1orsxAY7BHZPwZFZPPseUAmIqLGpystkBkIRPaNQRHZtZt3K5jWQURERlHPQmAGApFjYFBEduvm3QokvXkUckU1AJ7pIyIiwzELgcixMCgiuyUrV0CuqMbLE0IAgGf6iIjooalnIADMQiCyFwyKyO75+XhaugtERGQHtDMQAGYhENkLBkVEREREBlDPQPDz8eT1RkR2hEERERERkRH8fDwR4Odl6W4QkQk5W7oDRERERERElsSgiIiIiIiIHBqDIiIiIiIicmgMioiIiIiIyKExKCIiIiIiIofG6nNkl64Xllq6C0REZEc4rhDZNwZFZFfEHiK4iVywbm8egAc31RN7iDTuPk5ERMa5ebcCsnKFQwYG+sYVIrIvDIrIrrRp0RRpCyJVQZDYQ4Q2LZoyKCKyY3/88QcyMjLw888/4+LFi+jcuTOys7M12ixatAgffPBBrdfu2LEDgwYN0liWkZGBzMxMFBUVQSKRYMGCBQgLCzPrNlizm3crkPTmUcgV1QAcLyjQN64QkX1hUER2p02LphywiBzIxYsXkZubix49eqCmpgaCIOhs16FDB6xdu1Zjmb+/v8bjjIwMpKSkYO7cuZBKpcjKykJiYiKysrIQGBhotm2wZrJyBeSKarw8IQR+Pp4OGRRwXCGyfwyKiIjIpkVGRmLo0KEAHswInTt3Tmc7d3d39OzZU+96FAoFtmzZgkmTJiE+Ph4AEBoaipiYGGzduhUpKSkm77st8fPxRICfl6W7QURkFqw+R0RENs3Z2TRDWV5eHkpLSxEdHa1a5uLighEjRiA3N1fvDBQREdk+o2eKmLtNRES26OrVq+jTpw8qKyshkUiQlJSkmmECgPz8fABA586dNV7n7++P8vJyFBYWwtfXV+e6o6Ki9L5vQUEB2rZta4ItICIiczE6KGLuNhER2Zpu3bohODgYAQEBKC0txb59+zBz5kykpqZi2LBhAACZTAaRSAR3d3eN1zZv3hwAUFxcrDcoIiIi22Z0UMTcbdvlyCVVicixxcXFaTyOjIzE+PHjsWHDBlVQBABOTk61Xqs8+afrOaUjR47ofa6uWSQiIrIORgdFjZG7vXPnTgiCUOcARIZRBkKyMgVW7freYUuqEhGpc3Z2xlNPPYU1a9agsrIS7u7uEIvFkMvlkMvlcHNzU7WVyWQA/poxIiIi+2O26nPmyt1m3rbhdN1bYnliP4ibiRyypCoRkTrt9G9lind+fj6kUqlqeX5+Pjw8PODj49Oo/SMiosZjlqCIudvWgfeWICLSraamBp999hm6dOmiGodCQkLg6emJw4cPq4Ki6upq5OTkICIigtkLRER2zCxBkTlzt5m3bTzeW4KI7Nm9e/eQm5sLALhx4wbKysrw6aefAnhwreq9e/ewaNEiREdHo2PHjigpKcG+fftw7tw5bNy4UbUekUiEGTNmICUlBd7e3qoCQNeuXcP69estsm1kG5TX6vLkI5HtapSbtzJ3m4iIzOX27duYM2eOxjLl4927dyMwMBDNmjXD5s2bcefOHbi6uiIoKAg7duzAwIEDNV43ZcoUCIKAPXv24NatW5BIJNi+fTsropJOYg8R3EQuWLc3D8CDNPW0BZEMjIhsUKMERQBzt4mIyDz8/Pzw+++/19lmy5YtBq3LyckJCQkJSEhIMEXXyM61adEUaQsiVZVd1+3Ng6xcwaCIyAaZppRcPerL3VZi7jYRERHZkjYtmiLAzwt+Pp6W7goRPQSjZ4qYu01ERERERPbE6KCIudtERERERGRPjA6KmLtNRERERET2pNEKLRAREZHtuHm3QlVAgIyn3H8AS3UT2QIGRURERKTh5t0KJL15FHJFNYAHpabFHiIL98p26Np/LNVNZN0YFJFD4Q32iIjqJytXQK6oxssTQuDn48nvTCOp7z8ALNVNZAMYFJFD4A32iIiM5+fjiQA/L0t3w2axTDeR7WBQRA6BN9gjIqLGwGuwiGwTgyJyGG1aNGUQREREZqErI0HsIVIVWyAi68agyM6oV7vh2SoiIqLGoZ6RAPx17SqDIiLbwKDIjmhXuwFYMYiIiKix1JWRwEI/RNaNQZEd0a4WBPDLl4iIyJJY6IfINjAoskOsFkRERGQdWOiHyDYwKCIiIiIyIxb6IbJ+zpbuABERERERkSVxpoiIiIgA/FXBlNVLicjRMCgiIiKiWhVMWb2UiBwJgyIiIiKqVcGU1UuJyJEwKCIiInJg2ilzrGBKRI6IQREREZGDYsocEdEDDIqIiIgcFFPmiIgeYFBERETk4JgyR0SOjkEROSxl/jzPjBIRERE5NgZF5HDEHiK4iVywbm8egAc59GkLIhkYERERETkoBkXkcNq0aIq0BZGqakvr9uZBVq5gUERERETkoJwt3QEiS2jToikC/Lzg5+Np6a4Q0UP6448/sGTJEowaNQpSqRTR0dE62+Xm5mL06NEIDg7Gk08+iczMTJ3tMjIyEBkZieDgYIwdOxanTp0yZ/eJiMgKMCgiwoPriy5dL8al68W4ebfC0t0hIiNcvHgRubm56NSpE/z9/XW2OXPmDJKSkiCVSrFjxw6MGTMGK1euRFZWlka7jIwMpKSk4Pnnn8f27dvRqVMnJCYm4vfff2+MTSEHwTGHyPoYnT73xx9/ICMjAz///DMuXryIzp07Izs7u1a73NxcpKSkID8/H76+vpg8eTKef/75Wu0yMjKQmZmJoqIiSCQSLFiwAGFhYQ3bGiIjaV9fBPAaIyJbExkZiaFDhwIAFi1ahHPnztVqs3nzZkilUqxatQoAEB4ejoKCAqSmpmLs2LFwdnaGQqHAli1bMGnSJMTHxwMAQkNDERMTg61btyIlJaXxNorsEsccIutl9EwRz8iRPVFeX5QyNwIpcyPw8oQQyBXVkJUrLN01IjKQs3PdQ5lCocDJkycxcuRIjeUxMTEoKirCr7/+CgDIy8tDaWmpRvqdi4sLRowYgdzcXAiCYPrOk0PhmENkvYyeKeIZObI3bVo05Rk6Ijt29epV3L9/H507d9ZYHhAQAADIz89HUFAQ8vPzAaBWO39/f5SXl6OwsBC+vr463yMqKkrv+xcUFKBt27YPswlkRzjmEFkno2eKeEaOiIhsSUlJCQBALBZrLFc+Vj4vk8kgEong7u6u0a558+YAgOLiYjP3lIiILMXkJbkb44wcabp5t0JVXpqIiHRzcnKqd7muNsqTdPpeDwBHjhzR+1xds0hERGQdTB4UmfKMnK6giCkKmm7erUDSm0chV1QDeHDBpthDZOFeERFZD+W4ohx/lGQyGYC/xiexWAy5XA65XA43N7da7ZTrISIi+2O2m7ea84wc/UVWroBcUY2XJ4TAz8cTYg8Rc5WJiNR07NgRrq6uuHz5MgYNGqRafunSJQBQFQ1S/pufnw+pVKpql5+fDw8PD/j4+DRir4mIqDGZPCgy9xk5pijo5ufjiQA/L0t3g4jI6ohEIoSHhyMnJweTJ09WLc/Ozkbr1q1VAVBISAg8PT1x+PBh1bLq6mrk5OQgIiKCJ+uIiOyYyYMinpEjIqLGdO/ePeTm5gIAbty4gbKyMnz66acAHlQ19fb2xsyZMxEbG4vk5GTExMQgLy8PWVlZWLFihaqAkEgkwowZM5CSkgJvb29IpVJkZWXh2rVrWL9+vcW2j4iIzM/kQRHPyBERUWO6ffs25syZo7FM+Xj37t0ICwtDr169kJaWhvXr1+PQoUPw9fVFcnIyxo0bp/G6KVOmQBAE7NmzB7du3YJEIsH27dsRGBjYaNtDRESNz+igiGfkiIjImvj5+Rl00++IiAhERETU2cbJyQkJCQlISEgwVfesjrJiKQBWLSUi+j9GB0U8I0dERGSbtCuWAqxaag2UwSmLJRFZjtFBEc/IERER2SbtiqUAf4hbkthDBDeRC9btzQPwIEBNWxDJz4PIAsxWkpuIiIisEyuWWoc2LZoibUGk6gbs6/bmQVauYFBEZAEMioiIiIgspE2LpgyCiKyAs6U7QEREREREZEkMioiIiIiIyKExKCIiIiIiIofGa4psiPq9JVgtiIiItOkbJ5TLeV8iIiLdGBTZCO17SyjLdhIREQG6x4lX4kIBAKt2fa+xnPclIiLSxKDIyqmf3VPeWwIA1u3Nw6+Xb1u4d0REZC3U70Ek9nDDql3fY+mOEwAeBELLE/tB3EzETAMiIh0YFFkxXWf9pJ1bqv5f/WZvPOtHRETAX/cgUt7/BmDKNRFRfRgUWTHtO4+rD2oc7IiIqC68/w0RkeEYFNkAXXce52BHRERkf5TFMHjCk6hxMSgi0oGDEhERNSaxh6hWanzagkiOQUSNhEERkRpdg9IrcaG8OJmIiMyqTYumqtT464WlWLc3D7JyBccdokbCoMhCeM8h66Q+KMnKFLWqN/GsHRERmQtT44ksh0GRBei75xC/CK2D+qCkfdbu18u3IfNRMJAlIiIisiMMiixAvaocAE6RWzFlgMRcbyIiIiL7xaDIgvx8PC3dBTIQc72JiIiI7BeDIiIDMdebiIgak7ISqjqmbxOZB4MiIiIiIiuinbKtjunbRObBoIiIiIjIiqinbKtj+jaR+TAoshK8WSgRERlC/ZYOAMcNe8WUbaLGxaDIwljVjIiIDKV9Swfgr5tMy8rlFuwZEZFtY1BkYaxqRkREhlK/pYOfj6fOm0yLPUQW7iURke1hUGQFOEVORGR+Bw8exOLFi2stT0xMxLx581SPc3NzkZKSgvz8fPj6+mLy5Ml4/vnnG7Or9fLz8USAnxcAaFx7wlQ6IqKGYVBEREQOJT09HZ6ef90nzsfHR/X/Z86cQVJSEkaNGoVFixYhLy8PK1euhEgkwrhx4yzR3XrxxBoR0cMzS1BkT2fjiPRhcQwi2/T444/D29tb53ObN2+GVCrFqlWrAADh4eEoKChAamoqxo4dC2dn58bsKhERNRKzzhTZ29k4IoDFMYjslUKhwMmTJzVO3gFATEwMDhw4gF9//RVBQUEW6h0REZmTWYMino1rGOUMhK47WZPlsTgGkW2Ljo7G3bt30a5dOzz77LNISEiAi4sLrl69ivv376Nz584a7QMCAgAA+fn5DIqIiOyURa4p4tk43XTdwZqVhKyTdg6/egCrnk6nfj8RptkRWVbr1q0xa9Ys9OjRA05OTjh69CjeeustFBYWYsmSJSgpKQEAiMVijdcpHyuf1yUqKkrvcwUFBWjbtq0JtoCIiMzFrEGROc7G2fPAo+sO1vwhbd30BbJpCyIBQON+IkyzI7KsgQMHYuDAgarHAwYMgJubG3bt2oXp06erljs5Oel8vb7lRERk+8wSFJnzbJy9YxUh26IdyKqn0wFQ3U8EANPsiKzQ8OHDsXPnTvz2229o3749gNpjkEwmA1B7zFJ35MgRvc/VdTKPiIisg1mCInOejePAQ9amvkDWz+evYiOsWEdkvTp27AhXV1dcvnwZgwYNUi2/dOkSAMDf399SXSMiIjNrtGoGw4cPR3V1NX777Tc0b94cQMPOxhHZIvU0u7kpuUh68yhu3q2wdLeIHN7hw4fh4uICqVQKkUiE8PBw5OTkaLTJzs5G69atIZVKLdRLIiIyN4sUWuDZOHI0rFhHZHnx8fEIDw+HRCIB8CDz4MCBA5g0aRJat24NAJg5cyZiY2ORnJyMmJgY5OXlISsrCytWrHDoqqhERPau0YIifWfjJk+erGpjj2fjWH2MlHi9GJFlPfbYY/jPf/6DP//8EzU1NXj00UfxyiuvYOLEiao2vXr1QlpaGtavX49Dhw7B19cXycnJvH8e2QT+5iBqOLMERTwb98DNuxW1qo+9EhcKWbncwj0jInI8ycnJBrWLiIhARESEmXtDZFq6fnOw4imR4cwSFPFs3AOycoWq+pjYww2rdn2PpTtOAOD9h4iIiKhhdN0bT/03B8CKp0TGMktQxLNxmvx8PBHg56VRupnT2vZNfcAiIiIyhfrujQdoVjwlIsNZpNCCPVPP59X+YcxrSuyf9oDFGUEiIjKV+u6NR0QNx6DIhLTzeQH+KHY02gMWZwSJiMiUdJ1gZXYC0cNjUGRC6vm8yulr/ih2PMbOCKrPLgI8ZoiIyDD6shM4c0RkPAZFJqD8Uas8U6O8hoioPvpmF1kxiIiI6qMvO4FBEZHxGBQ9JF0lMJkuR4bSnl3kjV2JiMgYvF6ZyDQYFD0k7R+1TH0iQ6nngHN2kYiIiMhyGBSZCH/UkqEMqVCn66JZ9YBb+zokfe2IiIiIqH4MiogaWV0V6nTdg0JJ/V4U2tch6WrHwIiIiIjIMAyKiCxAXw64dsCkpH0vCu0qh7raMSgiIiIiMgyDIiIrY+hFs0zZJCKiuihTsZlWTVQ/BkVEdogDIRGR49J17SrTqonqxqCIyI5wICQiIvVUbKZVExmGQZERdFX80lUljMhSGjIQah/XnF0isj7aNwknqo++VGxDq5eqt+O4QI6AQZGBtG/Sqo43bKXGYsgPImNu5KfruObsEpF14U3CyVTq+i0D6K9yynGBHAGDIgNp36RVHc+gkLkZcm+j+ug666d9XDPNgsj68CbhZCp1/ZbRV+UUAMcFcggMiozEil9kCXXd28gQus40K88GAjyuiWwB/07JVOo6ltQzErQDJyJ7xqCIyEYYkxanTf3sIACNs4FERET6MhI4VpCjYFBE5EAa46wfL84lIrI9+jISGhoUcSwgW8OgqB6s+EO27mGOXWMHNX1peuYcDFk9j4jINB4mI0GdJcYCoofFoKgOrPhDtqy+VIj6gqW6BjV9wZKuNL1fL9+GzEfRoGClvqCM1fOIiAzzsCd39b1e13ezvpRtXeW+61sXUWNhUFQHVvwhW1ZXcQZDKtnVdR1SfWcAlX8v+m4ka8gMlCFnGlk9j4iobg9bvVT79drcRC54JS4U4mZ/rVMZQOlK2a7vFifKdfE3FzU2BkUGYMUfslW6UiEMqWRXV/Wh+s4A6nof7VKvhqRV1PU+2mmt/BslItLtYauXar9enaxMgVW7vsfSHSdqPacvO+F6YanOsuDa66pr1p/XK5E5MCjSov6HxuuIyF7pyxs3JuVOfTBTH+zqex9Dgypd7wMwrZWIyFgPe61QXa/XFzDpy05QPpZ2bllrnbpOpNWXNs2UaTIVBkVq9F2fwB9c5CgaknKna7Az5G9GV1Cl/Z7alO2Y1kpEZB3qC7h0zTTp+942JHir69pVdervwYI8ZAiLBkVXrlzBypUrcfr0aTzyyCMYOXIk5s2bB3d3d4v0R9ednvmHQ47G2JQ7Qwc7oPZMkq5cdWVOuaxcrredvrOM2vSlWNSVesG0DLK2sYnI1jV0pko5Zuj6LtZ17ao65VgCAKt2fc+CPFQviwVFMpkMcXFxaNeuHTZs2IA7d+7g9ddfR3FxMdauXWupbgHg9QlEuugb1AwZ7PSl5WkHW7pyynW1qy9YuV5YqlqXdooFoP+aJl1pGdoXEBvy/mS7rHlsInIUdRXqUafveiddY8nyxH4QNxOZtSCPISfitJ8j62GxoOi9996DTCbDoUOH4O3tDQBwcXHBvHnzMGPGDPj7+1uqa0RkYnUFNdpBlaHtdNE1kC5P7AdZuVyj0IO+a5rUZ4vFHm51XkBsyFlGzjrZHo5NRJanr1CPMSfmDD2Rpq88uLq6MgqUdJ2Iq2umypgqe6YOqjg26WaxoOj48ePo16+fatABgKeffhqvvPIKcnNzOfAQ2RlD0yce5oJgfcHXpevFAAwvFKGcLdZ1BlI5QOvKYVenb6ZKuyS5trrOLhrSxtA0QXUcIP/CsYnIOmiPBfoK+hj6el3qKg+urq6MAu12yxP7AYDemSpds1h1ZSTou95d12t0vV65ndoZGcYUqmhIUGaL44rFgqL8/HyMHTtWY5lIJELHjh2Rn59voV4Rka3TNRDWdZ+OugpFGLKuuuibqaprEK7r7KJ6G12pgHW9Xt/Ax0pOmjg2EVkXfdeePmwRLF2Fe/S1Uz8Rpq+kuLKvuq631Q4KlM/VV9JceX2t+vvV9Rpdrwd0z1Rpj0367iFY303SjZk1UwZxhpy8s0RQ5SQIgmD2d9Hh8ccfx5w5czB16lSN5c899xxatmyJTZs26XxdVFSU3nVev34dLi4uaNu2bYP6dL+6BndKKuHd3B2uLs4NWgcRWafqGgE1//d15+zkBBdnp1rLtZ8zZF11Ua5L+d3S/P8G8ZJyBZp7iODSRPN7RqgRcLdUrrGshacbnNT6U11Vo3q99rr0vb6mRtD7nrrW19DvwIKCAri4uODs2bNGv9ZacGwisj4N+Z6ua123iu9pLGvl9Yje9elqX99rjO2P9nii67tc/f3qG4PqG0t0jU3Ozk4GjR/qY4au12i/HoDONvqe07e8ofvbmHHJ6kpyC4IAJ6eGHWROTk5o0qThm+Tq4gwfb9NFogUFBQDQ4IHQ3nB/1MZ9osmc+8PF2QkuqP3dom95Q9alj/Z3i7ubnu8pF2i0KygowG255v5wdXHWeL3GurRer07fe9a5PiM1adIEIpF93sLAnsYmU3O07zFub+NryPd0Xeuq6+9Je3vra2+K/tTatjq+y/W+xojXK7m6OKNGXoK7/zfOGDJ+aI8ZhrxPXW30PWeqfW7MuGSxoEgsFkMmk9VaXlpaWmfO9pEjR8zZLZNSnjm0pT6bE/dHbdwnmrg/NHF/ND5HGJtMzdGOU26vfeP2Oi6LzcP7+/vXys9WKBS4evUqL2QlIiKL4NhEROSYLBYUDRo0CCdPnsTdu3dVy7744gsoFApERERYqltEROTAODYRETkmiwVF48ePh6enJ5KSkvD111/j0KFD+Pe//42YmBiejSMiIovg2ERE5Jgsek3Rrl27sHLlSsyaNQvu7u6Ijo7GvHnzLNUlIiJycBybiIgck0Wrzz322GPIyMiwZBeIiIg0cGwiInI8vOEBERERERE5NIvdvJWIiIiIiMgacKaIiIiIiIgcGoMiIiIiIiJyaAyKiIiIiIjIoTEoIiIiIiIih8agiIiIiIiIHBqDIiMcPHgQgYGBtf5bu3atRrvc3FyMHj0awcHBePLJJ5GZmalzfRkZGYiMjERwcDDGjh2LU6dONcZmNNgff/yBJUuWYNSoUZBKpYiOjtbZzpTbX1ZWhiVLliAsLAy9evXC9OnTcePGDZNuV0MZsj8WLVqk85g5fvx4rba2vj9ycnKQlJSEiIgI9OzZEzExMdi7dy9qamo02jnK8WHI/nCk44OsjyMeo5YYxy25zRMnTtS5vYGBgfjkk08A2PZnbM2/S65cuYL4+Hj07NkT/fr1w8qVK1FZWWnW7a2ursaOHTsQGxuL8PBw9O3bF88//zxOnDhRa12RkZE6P3e5XG4129voBDLY+++/L0gkEuH48ePCmTNnVP/973//U7XJy8sTpFKpsHjxYuHEiRPC5s2bha5duwoHDhzQWFd6errw+OOPC+np6cJ3330nzJ07VwgODhYuXLjQ2JtlsC+++EIYNGiQMGvWLCE6OloYOXJkrTam3v6pU6cK/fv3Fz7++GPhq6++EsaMGSM8+eSTwr1798y6rYYwZH8sXLhQiIqK0jhezpw5I8hkMo129rA/xo0bJ8yZM0fIzs4WTpw4Ibz11luCVCoV3njjDVUbRzo+DNkfjnR8kPVxxGPUEuO4Jbf54sWLtT67l156SZBKpcLt27cFQbDtz9haf5eUlJQIAwcOFP7xj38Iubm5wgcffCCEhoYKL7/8slm3t6ysTOjdu7fw73//Wzh69Khw/PhxYe7cuUJgYKBw9OhRjbZDhgwRZs2aVetzr6mpsZrtbWwMioyg/DJVfpHoEh8fL/z973/XWJacnCz0799fqK6uFgRBEORyudC7d29h9erVqjZVVVXC8OHDhZdeesk8nTcBZf8F4cGXqK4vH1Nu/08//SRIJBLh2LFjqmU3btwQpFKpsHfvXpNtV0MZsj/0LVdnL/tD19/FqlWrhODgYEEulwuC4FjHhyH7w5GOD7I+jniMNvY4bg3brC0yMlJITExUPbblz9haf5ds27ZN6NGjh8Zx9tFHHwkSiUS4dOmS2ba3qqpKKC4u1lhWU1MjjBkzRoiNjdVYPmTIEGH58uV1vp+lt7exMX3OhBQKBU6ePImRI0dqLI+JiUFRURF+/fVXAEBeXh5KS0s1pj1dXFwwYsQI5ObmQrDS++k6O9d9uJh6+3NzcyEWizFo0CBVu3bt2iEkJAS5ubmm2qwGq29/GMpe9oe3t3etZd26dYNcLkdxcbHDHR/17Q9D2cv+IOvDY7Q2e/+eysvLw/Xr1xETE2P066xxe631d8nx48fRr18/jb+xp59+GiKR6KH2Q33b6+LigubNm2ssc3JyQteuXXHz5k2j38/S29vYGBQ1QHR0NLp164aoqChs27YN1dXVAICrV6/i/v376Ny5s0b7gIAAAEB+fr7Gv9rt/P39UV5ejsLCQnNvglmYevvz8/Px2GOPwcnJqdb6lOuwBVevXkWfPn0QFBSEZ555Bl9++aXG8/a8P06fPg0vLy+0bNmSxwc094eSIx8fZH0c5RhtrHHcmrYZALKzs/HII48gKipKY7k9fsaA5T7P/Px8+Pv7a7QRiUTo2LFjo++HmpoanDlzplZ/AODjjz9GUFAQevXqhcTERPz+++8az9vi9j6MJpbugC1p3bo1Zs2ahR49esDJyQlHjx7FW2+9hcLCQixZsgQlJSUAALFYrPE65WPl8zKZDCKRCO7u7hrtlNF9cXExfH19zb05Jmfq7ZfJZPD09Kz1PmKxWLUua9etWzcEBwcjICAApaWl2LdvH2bOnInU1FQMGzYMgP3uj7Nnz+LgwYOYOXMmXFxcHP740N4fgGMfH2R9HOEYbexx3Bq2WamqqgqffvopoqKi0LRpU9Vye/uM1Vnq85TJZLXeU1e7xrBnzx5cuXIFK1as0FgeGRmJ7t27o127drh27Rq2bt2KCRMm4NChQ+jQoQMA2OT2PgwGRUYYOHAgBg4cqHo8YMAAuLm5YdeuXZg+fbpquXZErWu5rjbK6Vl9r7cVptx+Q9ZlzeLi4jQeR0ZGYvz48diwYYNqsAHsb38UFRVh9uzZCA4ORmJiosZzjnh86Nsfjnp8kPVxlGPUEuO4pbdZ6dtvv8Xt27drVSyzt8/YmPdu7M9TEIRG3Q/ff/891qxZgylTpqBv374azyUnJ6v+v0+fPujfvz+GDx+OjIwMLFu2TPWcLW3vw2L63EMaPnw4qqur8dtvv6nOKGhHxTKZDMBfZybEYjHkcnmtsofKdtr5oLbC1NsvFotVy7Tb6TojYQucnZ3x1FNPIT8/X1Wq0t72R2lpKRITE+Hu7o4tW7bA1dUVgOMeH/r2hy6OcHyQ9XH0Y9Sc47g1bXN2dja8vLwwYMCAOtvZ02dsqc9TX7vS0tJG2w8XLlxAUlIShg4divnz59fbvk2bNujduzfOnz+vWmZL22sKDIpMqGPHjnB1dcXly5c1ll+6dAkAVPmWyn+18yzz8/Ph4eEBHx+fRuit6Zl6+/39/XHlypVahScuXbqkMzfWVmhvjz3tD7lcjhkzZuDWrVtIT09HixYtVM854vFR1/7Qx56PD7I+PEY12ev3VGVlJY4cOYJhw4bVGfQq2ctnbKnP09/fv9a6FAoFrl692ij74erVq0hISIBUKsWbb75p8GyNrs/dFrbXVBgUPaTDhw/DxcUFUqkUIpEI4eHhyMnJ0WiTnZ2N1q1bQyqVAgBCQkLg6emJw4cPq9pUV1cjJycHERERNjXVqM7U2x8REQGZTIavv/5a1a6goAB5eXmIiIhohC0yvZqaGnz22Wfo0qWLKnfZXvZHVVUV5syZgwsXLiA9PR3t27fXeN7Rjo/69ocu9nx8kPXhMfqAOcdxa9nmo0ePory83KCqc/b0GVvq8xw0aBBOnjyJu3fvqpZ98cUXUCgUZt8PRUVFmDJlClq1aoW0tDSIRCKDXldYWIi8vDwEBwerltnC9poSrykyQnx8PMLDwyGRSAAAR44cwYEDBzBp0iS0bt0aADBz5kzExsYiOTkZMTExyMvLQ1ZWFlasWKEqpSgSiTBjxgykpKTA29sbUqkUWVlZuHbtGtavX2+x7avPvXv3VKUVb9y4gbKyMnz66acAgNDQUHh7e5t0+3v06IHBgwfjX//6FxYtWoRmzZohNTUV7du3x5gxYxp/B2ipb3/cu3cPixYtQnR0NDp27IiSkhLs27cP586dw8aNG1XrsZf9sWLFCnz11VeYP38+Kisr8dNPP6meCwgIQLNmzRzq+Khvf5SUlDjU8UHWxxGP0cYex61hm4EHVcbatWuH3r17ayy/ceOGTX/G1vq7ZPz48Xj33XeRlJSEpKQk3L59G2+88QZiYmIeauakvu1t2rQpEhIScPv2bSxatEg1I6bUs2dPAA+CwmPHjmHQoEFo06YNrl27hu3bt8PFxQUvvPCC1WxvY3MSrPWmOFZo5cqV+Prrr/Hnn3+ipqYGjz76KMaNG4eJEydqzO7k5uZi/fr1yM/Ph6+vL1544QU8//zzGusSBAEZGRnIzMzErVu3IJFIMH/+fISHhzf2Zhns+vXrtcp4Ku3evRthYWEATLv9ZWVlWL16NT777DPcv38fYWFhePXVVw06o2lu9e2PwMBALF68GOfPn8edO3fg6uqKoKAgTJ06VeNCX8A+9kdkZCRu3Lih8zlHPD7q2x+OdnyQ9XHEY9QS47ilt7mkpAT9+/dHXFxcrWtLiouLbfoztubfJVeuXMHKlStx+vRpuLu7Izo6GvPmzatV4c6U29u+fXu9zwNQldz+6aefsG7dOly8eBGlpaXw9PREeHg4Zs+eXassuSW3t7ExKCIiIiIiIofGa4qIiIiIiMihMSgiIiIiIiKHxqCIiIiIiIgcGoMiIiIiIiJyaAyKiIiIiIjIoTEoIiIiIiIih8agiIiIiIiIHBqDIiIiIiIicmgMioiIiIiIyKExKCIiIiIiIofGoIiIiIiIiBza/wddntDA4GG+gAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 1000x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# We can plot these to see if there are some outliers\n",
    "\n",
    "plt.figure(figsize=(10,3))\n",
    "ax = plt.subplot(1,2,1)\n",
    "ax.hist(gene_counts, histtype=\"step\", bins=100)\n",
    "ax.set_title(\"Number of expressed genes per cell\")\n",
    "\n",
    "# We can plot these to see if there are some outliers\n",
    "ax = plt.subplot(1,2,2)\n",
    "plt.hist(total_counts, histtype=\"step\", bins=100)\n",
    "plt.title(\"Total counts per cell\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31b0b6e5-6da5-4c45-b06c-775b9dacbf4c",
   "metadata": {},
   "source": [
    "<ins>Note</ins>: The choice of how to select outliers is mostly domain specific. In clinical data, it maybe \"exceptionally atypical measurements\", in financial data, it may be a specific month (April saw a reducing in US financial debt, so can't rely observations during that time, unless of course if that is your target. In cases, where such information is not possible, dedicated outlier detection methods can be utilized. [https://scikit-learn.org/stable/modules/outlier_detection.html](https://scikit-learn.org/stable/modules/outlier_detection.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4a47b30-2672-4406-90d9-11bae107eb2b",
   "metadata": {},
   "source": [
    "From our training data, we will keep cells with total expressed genes between 500 and 2000 and remove cells with total genes over 7500."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "aea9a469-79ac-40bf-871b-5c41416fcd13",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gene_min, gene_max = 500, 2000\n",
    "count_max = 7500"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "88c9db71-0d45-42a8-b4f9-148919dd3927",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "idxs_rm = []\n",
    "idxs_rm = gene_counts[(gene_counts<gene_min)&(gene_counts>gene_max)].index.tolist()\n",
    "idxs_rm = idxs_rm + total_counts[total_counts>count_max].index.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "d14485f7-087b-4a98-9a2f-2fa43a4da4f4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "223\n"
     ]
    }
   ],
   "source": [
    "print(len(idxs_rm))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "089d1c95-6f20-4283-8efe-98ac8cbdbfb9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_X, train_y = train_X.drop(idxs_rm), train_y.drop(idxs_rm)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "644d75e4-8e5a-4a38-b061-dcf4974b5dd7",
   "metadata": {},
   "source": [
    "## Normalization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5415b54-e750-40b3-a63a-0dfe8825502f",
   "metadata": {},
   "source": [
    "Domain knowledge also helps in deciding on preprocessing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b4a242b5-0b40-4d6a-ba49-5a5a02cd2165",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# domain specific (Make sure each cell has same total RNA)\n",
    "train_X = train_X.div(train_X.sum(1), 0)  # Divide each value in a row by the row sum\n",
    "train_X = train_X*1e6 \n",
    "train_X = np.log2(train_X+1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "7e8d77da-5bdf-4070-b706-8e74b57eb7af",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.0, 18.968884563660744)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_X.min().min(), train_X.max().max()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f1b32e9-2210-450a-9c57-da04dd800dd2",
   "metadata": {},
   "source": [
    "This normalizes counts such that each cell's total count sums to 1 million and take log2 of it. Computing log2 makes really high gene expressions to be comparable to others. It's a lightweight way of mitigating gene expression disparity. \\\n",
    "Further, it \"approximates\" a normal distribution (a nice property of have). \\\n",
    "In single-cell RNA data, \"approximates\" doesn't usually occur due to high sparsity i.e. many features are 0's (check with (train_X==0).sum()). This is not a settled issue. Several different approaches are possible and the science is not settled. Some normalizations are discussed in this review: [https://link.springer.com/article/10.1186/s12864-024-10364-5](https://link.springer.com/article/10.1186/s12864-024-10364-5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "46bb23c8-8b7e-4a7d-8d7c-a3a592bc3944",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "index\n",
       "LINC00115    6758\n",
       "NOC2L        5885\n",
       "KLHL17       6816\n",
       "PLEKHN1      6817\n",
       "HES4         6654\n",
       "             ... \n",
       "YBEY         6358\n",
       "PCNT         6654\n",
       "DIP2A        6145\n",
       "S100B        6631\n",
       "PRMT2        3962\n",
       "Length: 10851, dtype: int64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(train_X==0).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c72169fb-1fe8-43a4-b54c-096a64623573",
   "metadata": {
    "tags": []
   },
   "source": [
    "Here, we want the model to weigh each feature appropriately, we need to make the data such that each feature has same impact in theory.\n",
    "\n",
    "After this, we have two standard routes to take\n",
    "\n",
    "1. Min-Max normalization: Scale all values between 0 and 1: Appropriate when you expect uniformly distributed features\n",
    "2. Mean-var or standard scaling: Zero center data: Appropriate when you expect \"somewhat\" normally distributed features\n",
    "\n",
    "These can be sensitive to outliers. There are other \"exotic\" ways to normalize like based on median etc.\n",
    "\n",
    "<ins>Note</ins>: Some algorithms like Random Forest are less sensitive to preprocessing and can handle data as it is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "b9f101d3-70b0-4449-b234-8d4cf124e23e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA08AAAErCAYAAAAGxLh/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAAA+8klEQVR4nO3dcXzMV77/8XdERpZkaLoIUZbYpIYU2SXR7TWE7N5FWmmqrVIrJH5ES1VpusJdotRaUtUkGk11qd5to9itFfe2VKp3y24btcqyTdgGm42wjYmoTCTz+8MjU2OCCUmGyev5eHikc76fOd/zTdv5+Mw53/P1stlsNgEAAAAArquFuwcAAAAAAHcCiicAAAAAcAHFEwAAAAC4gOIJAAAAAFxA8QQAAAAALqB4AgAAAAAXUDwBAAAAgAsongAAAADABRRPAAAAAOACiic0S5s3b1ZoaKgOHjxY5/H/9//+n6Kiouyvo6KiFBoaqgULFjjF7tu3T6GhodqxY4dT/7V/TCaTHnjgAc2aNUv/+Mc/nPp48sknFRoaqsmTJzsdO3nypEJDQ5Wdne10zto/vXr10v33368ZM2aosLDQqY+dO3cqNDRUERERslqtdV7zRx99pLlz5yomJka9e/dWaGhonXF1Xe/VTp8+rbS0ND322GOKiIhQeHi4Hn74Yb3zzjuqrq6+5vsAAA2rrnw0ePBgvfDCCyopKZFUd06JjIzU1KlT68yTycnJDvH9+vVTVFSUpk6dqvfee88pz/z73//W/fffryeeeEI1NTUOx6xWqx588EFFRUXp/PnzDsc+++wzJSYmasCAAbrvvvv005/+VOnp6Q38GwLqp6W7BwDcSd577z1NnDhRPXr0cCl+6dKl6tGjhyorK5Wfn681a9Zo3759ys3NVdu2bZ3iP/nkE3366acaNGiQS/0/++yzioiIUFVVlb788kulp6dr7969ev/999WxY0d73KZNmyRJZWVl+vDDDzVixAinvj744AMdOHBAvXr1ko+Pjw4dOuTSGOpy6NAh/f73v9dDDz2kpKQktWzZUh9//LF+9atf6YsvvtDSpUtvum8AQP3V5qOLFy/qs88+02uvvaY///nPev/99+0xtTnl0qVLOnz4sNLT0/Xkk09q69at+sEPfuDQn6+vr377299Kki5evKh//etf+vjjjzV//nytW7dOr7/+ugIDAyVJAQEBWrRokaZPn64333xTkyZNsvezevVq/f3vf9ebb74pPz8/e/v777+vuXPn6uc//7mWLVum1q1b68SJE/aCD3AXiifARf3791dBQYHS0tK0evVql97zwx/+UGFhYZKkiIgIVVdXa/Xq1frwww8VFxfnEPuDH/xA1dXVWr58ud577z15eXndsP9u3bqpX79+kqQBAwbI399f8+bN0+bNmzVt2jRJUmlpqT7++GNFRkZq//792rRpU53F0+LFi9WixeXJ6EWLFt1S8RQeHq4PPvhAPj4+9raf/OQnqqqq0saNGzVjxgx16tTppvsHANTPlfkoMjJS1dXVysjI0Icffmj/su3KnPLjH/9YRqNRzz//vP7whz9oxowZDv21aNHCHltr9OjRevjhhzV16lTNmDFD7777rv3Y8OHD9eCDD+rll1+W2WxWcHCw9u/fr+zsbI0fP16RkZH22JKSEi1YsECPPfaYfvWrX9nbr4wB3IVle4CL2rZtqylTpuh///d/9cUXX9xUH7WJ6+zZs07HfHx8NGvWLB06dEjbt2+/qf5rE9k///lPe9uWLVt06dIlTZw4UdHR0fr000916tQpp/fWFk4NoW3btg6FU6377rtPkvSvf/2rwc4FAKi/uvLF1fr06SNJOnPmjMv9PvDAAxozZowOHDigv/zlLw7HUlJS1LZtWz3//PM6f/68kpOTdc8992j27NkOcTk5Obpw4YISExNdPi/QVCie0KzV1NTo0qVLTn9sNlud8RMmTFDHjh21fPnymzrfyZMnJclp+UOtESNGqHfv3nr55ZdVVVVV7/6//vprSZeXSNR677331L59ew0ePFiPPPKIampqtGXLlvoPvgHs3btXLVu2vOb1AwCaRl354mq1Oat79+716rv2nuGri6e2bdtq8eLFOnjwoGJjY1VUVKSlS5fqe9/7nkPcX/7yF7Vr107Hjh3TQw89JJPJpEGDBmnBggVO90UBTY3iCc3ao48+qt69ezv9ycvLqzPe19dXTz/9tD777DN99NFHN+y/tjirqKjQnj17lJmZqQEDBjhsRnElLy8vzZkzR0VFRXrnnXdc7r92DfuyZcvk7e2tn//855Iu32z7j3/8Q7GxsfL29lZkZKS6dOmizZs3X7NAbCyffPKJfv/732v8+PG66667mvTcANDcXZmPdu/erczMTLVp08YhH12ZU/Lz8/XSSy+pZ8+eTsvMb6Rz586SLm8edDWz2awHHnhARUVFeuKJJxQeHu4UU1JSom+//VYzZ87UiBEjtG7dOk2ePFm///3vlZiY2OT5C7gS9zyhWVu2bJmCg4Od2pcuXXrNpWUPP/yw3nzzTa1YsUJms/m6/T/66KMOr4ODg5WRkaGWLa/9v96gQYP0wAMPKD09XaNHj75u/7NmzXJ43aVLF73yyiu69957JX23UURt4vPy8lJsbKxWr16tTz/9VPfff/91+28ohw4d0jPPPKO+ffs6Lc8AADS+q/NRSEiIfvWrX+n73/++fZfWq3NK+/bt9bvf/U5Go7HBxnHkyBHt27dPLVq00J///GdZrVYZDAaHGJvNpsrKSj311FOaMmWKpMv3Dfv4+GjJkiVNmr+AqzHzhGYtODhYYWFhTn/8/f2v+R5vb2/NmjVLX3311Q2Xvy1btkybNm3Sb3/7Wz322GMqLCzUs88+e8NxPffcc/rmm2/0xhtv3DBu06ZN2rJli3bv3q2dO3dq+PDhkqTz589rx44duu+++xQQECCLxSKLxaLo6Gh5eXnZC6vGdvjwYU2aNEndunXT2rVrnZIkAKDx1eajrVu3as+ePXr//ff1ox/9yCGmNqe89dZbmjp1qs6ePavp06df8xEX11J7X22HDh0c2quqqvT888+rQ4cOWr16tb766itlZGQ4vb9du3aSLt8/daXBgwdL0i1taATcKmaegJswfPhwhYeHa/Xq1Vq0aNE142qLM+nyLkE1NTXKycnRjh079J//+Z/XfF+vXr00cuRIrVu37rqzW/fcc4+9/6v98Y9/1Lfffqu//vWvGjBggNPxDz74QOfOnatzy/SGcvjwYcXHx6tz58564403rluUAgAaz5X56FquzCkDBgyQr6+vXn75ZW3YsKHO5xBey65duyRJAwcOdGhPT0/X0aNHtW7dOg0aNEiPP/641q5dq+joaPXu3dseFxoaWufGTLXL9RpygyOgvvivD7hJzz33nIqLi7VhwwaX3zNnzhy1bdtWr7zyitODAq/2zDPPqKqqSq+++upNjW/Tpk1q06aN3nzzTa1fv97hz9y5c2W1Wh2e79HQ/va3vyk+Pl4dO3bUG2+80ahFGgCg4SUkJKhbt27KyspyeaOG//u//9OmTZvUv39//fjHP7a3Hzx4UGvXrtUTTzxhf5bhnDlz1LFjRyUnJzvMbv30pz+VJH388ccOfde+7tu37y1dF3ArmHkCbtKPfvQjDRs2TDt37nT5PbXbnS9fvlzvv/++HnrooWvG3nPPPRo7dqzWr19f77H9/e9/11//+leNHTu2zgfuhoeHa926ddq0aZPGjx8v6fIyi9onyRcVFUmSduzYIUkKCgpy+sbywIEDdZ574MCBKisr08SJEyVdXkP/9ddf23d2kqSuXbted4cnAID71T5C45lnntH69euVlJRkP1ZTU2OfHbJarfrnP/+pPXv2KDc3V8HBwVq1apU91mq1Kjk5WZ07d9Zzzz1nb2/Tpo2WLFmiiRMnKiMjQ88884yky8v1hg4dqvT0dNXU1Khfv3768ssv9eqrr2ro0KEORRnQ1CiegFswe/Zs7d69W9XV1S6/58knn9TGjRuVkZGhUaNGydvb+5qx06ZN0+bNm+u9NWvt/UyPP/54ncd9fHwUGxurrKwsHTp0SL1799a+ffv0wgsvOMTNnDlTkhQbG6uXXnrJ4di17sdav369Tp06pbKyMknS1KlTnWKWLl2qhx9+uF7XBABoej//+c+1bt06vfnmm3ryySfty68vXryoxx57TNLlnWgDAgIUGhqq1NRUxcTEONzf+vLLL+vYsWPasGGDWrdu7dB/ZGSkxo4d67R87+WXX9arr76qd999V+np6erQoYMmTpyop556qomuHKibl439HgEAAADghrjnCQAAAABcQPEEAAAAAC6geAIAAAAAF1A8AQAAAIALKJ4AAAAAwAUUTwAAAADggmb5nKcf//jHslqtat++vbuHAgDNSmlpqQwGgz777DN3D+W2Q24CAPeoT25qlsVTZWVlvR5qCgBoGJcuXRKPF6wbuQkA3KM+ualZFk8dOnSQJO3cudPNIwGA5mXYsGHuHsJti9wEAO5Rn9zEPU8AAAAA4AKKJwAAAABwAcUTAAAAALiA4gkAAAAAXEDxBAAAAAAuoHgCAAAAABdQPAEAAACACyieAAAAAMAFzfIhuQ3h9DcXZKmwuu38xjYGdbirtdvODwC4/ZCbAKBxUTzdhNPfXFDSr3ep0lrttjG0MngrY24USQoAIIncBABNgeLpJlgqrKq0Vmv2E+Hq0tG/yc9/sqRcK97Ol6XCSoICAEgiNwFAU6B4ugVdOvqrZ5d27h4GAAB25CYAaDxsGAEAAAAALqB4AgAAAAAXUDwBAAAAgAsongAAAADABRRPAAAAAOACiicAAAAAcAHFEwAAAAC4gOIJAAAAAFxA8QQAAAAALqB4AgAAAAAXUDwBAAAAgAsongAAAADABRRPAAAAAOACiicAAAAAcAHFEwAAAAC4gOIJAAAAAFxA8QQAAAAALqB4AgAAAAAXUDwBAAAAgAsongAAHqOiokKDBw9WaGioDh486HAsLy9Po0ePVlhYmKKjo7Vx48Y6+8jOzlZUVJTCwsIUFxenffv2OcWcP39eCxYsUEREhPr376+pU6fq1KlTjXJNAIDbB8UTAMBjZGRkqLq62ql9//79SkpKkslk0tq1axUbG6vFixcrJyfHIS47O1tpaWkaN26csrKy1K1bNyUmJuro0aMOcbNnz9auXbs0f/58paWl6fTp04qPj9fFixcb9foAAO5F8QQA8AiFhYV6++239fTTTzsdS09Pl8lk0pIlSxQZGamkpCQ98sgjWrVqlWpqaiRJVqtVmZmZmjBhgiZPnqxBgwZp+fLl6tKli9asWWPv68CBA9q9e7defPFFjRo1SkOGDNGrr76qU6dOacuWLU12vQCApkfxBADwCC+++KIef/xxde/e3aHdarVq7969GjlypEN7TEyMSktLdfjwYUlSfn6+ysvLNWrUKHuMt7e3RowYoby8PNlsNkmXl/8ZjUYNHjzYHte5c2eFh4crLy+vsS4PAHAbaOnuAQAAcKt27NihI0eO6JVXXtGhQ4ccjhUVFamqqko9evRwaO/Zs6ekyzNWffr0UWFhoSQ5xQUHB6uiokIlJSUKDAxUYWGhunfvLi8vL6f+Pvnkk+uOc9iwYdc8VlxcrE6dOl3/QgEAbsXMEwDgjvbtt9/qpZde0rPPPis/Pz+n4+fOnZMkGY1Gh/ba17XHLRaLDAaDfH19HeLatm0rSSorK7PH+fv7O53HaDTa+wIAeCZmngAAd7TMzEzdfffdevjhh68bd/VMUV3tdcXULte7Udz12mvt3LnzmseuNysFALg9MPMEALhjnTp1Sm+88YZmzJih8+fPy2Kx6MKFC5KkCxcuqKKiwj5zdPWskMVikfTdDJTRaFRlZaUqKyvrjKvtx2g02tuujrt6dgsA4FmYeQIA3LFOnjypqqoqTZkyxenYhAkT1LdvX7311lvy8fHRsWPHHDZ5KCgokHT5nqYrfxYWFspkMtnjCgsL1aZNG3Xs2NEe96c//Uk2m81hpqmgoMDeBwDAMzHzBAC4Y/Xq1Uvr1693+PPCCy9IkhYuXKj/+q//ksFgUGRkpHJzcx3eu23bNrVv395eKIWHh8vf31/bt2+3x1RXVys3N1dms9leKJnNZlksFu3Zs8ceV1xcrPz8fJnN5sa+ZACAGzHzBAC4YxmNRkVERNR5rHfv3urdu7ckafr06Ro/frxSUlIUExOj/Px85eTkaNGiRWrR4vL3iAaDQdOmTVNaWpoCAgJkMpmUk5OjEydOaOXKlfZ++/btqyFDhmjevHlKTk6Wn5+fVq1apaCgIMXGxjb+RQMA3IbiCQDg8fr376+MjAytXLlSW7duVWBgoFJSUjRmzBiHuEmTJslms2nDhg06c+aMQkJClJWVpdDQUIe4FStWaNmyZVq4cKGqqqoUERGh1atXO+3UBwDwLBRPAACPEhERoaNHjzq1m83mGy6r8/LyUkJCghISEq4b5+fnp9TUVKWmpt7SWAEAdxbueQIAAAAAF1A8AQAAAIALKJ4AAAAAwAUUTwAAAADgAoonAAAAAHBBvYqnPXv2aPz48YqMjFSfPn00bNgwLV26VOXl5Q5xeXl5Gj16tMLCwhQdHa2NGzfW2V92draioqIUFhamuLg47du3zynm/PnzWrBggSIiItS/f39NnTpVp06dqs+wAQAAAOCW1at4OnfunPr376/U1FRlZ2crPj5eW7du1cyZM+0x+/fvV1JSkkwmk9auXavY2FgtXrxYOTk5Dn1lZ2crLS1N48aNU1ZWlrp166bExESn7WVnz56tXbt2af78+UpLS9Pp06cVHx+vixcv3sJlAwAAAED91Os5T6NGjdKoUaPsryMiImQwGDR//nyVlJSoY8eOSk9Pl8lk0pIlSyRJkZGRKi4u1qpVqxQXF6cWLVrIarUqMzNTEyZM0OTJkyVJAwcOVExMjNasWaO0tDRJ0oEDB7R7925lZWXZn80REhKi6OhobdmyRWPHjm2QXwIAAAAA3Mgt3/PUrl07SdKlS5dktVq1d+9ejRw50iEmJiZGpaWlOnz4sCQpPz9f5eXlDoWYt7e3RowYoby8PNlsNkmXl/8ZjUYNHjzYHte5c2eFh4crLy/vVocOAAAAAC67qeKpurpalZWVOnTokNLT0zV06FAFBQWpqKhIVVVV6tGjh0N8z549JUmFhYUOP6+OCw4OVkVFhUpKSuxx3bt3l5eXl1N/tX0AAAAAQFOo17K9WkOHDrUXOP/xH/+hlStXSrp8T5QkGY1Gh/ja17XHLRaLDAaDfH19HeLatm0rSSorK1NgYKAsFov8/f2dzm80Gu19XcuwYcOueay4uFidOnW67vsBAAAA4Eo3VTxlZWXpwoULKigoUEZGhqZOnap169bZj189U1RXe10xtcv1bhR3vXYAAAAAaAw3VTzde++9kqTw8HCZTCbFxcXpgw8+sC/Pu3pWyGKxSPpuBspoNKqyslKVlZVq1aqVU1ztDJTRaFRxcbHT+S0Wi9Ps1tV27tx5zWPXm5UCAAAAgLrc8oYRvXr1kre3t4qKitS1a1f5+Pjo2LFjDjEFBQWSLt/TdOXPq+9bKiwsVJs2bdSxY0d73PHjx+0zUlf2V9sHAAAAADSFWy6e9u/fr+rqanXp0kUGg0GRkZHKzc11iNm2bZvat28vk8kk6fKMlb+/v7Zv326Pqa6uVm5ursxms31JntlslsVi0Z49e+xxxcXFys/Pt29dDgAAAABNoV7L9p566in16dNHoaGh8vX11ZEjR/T6668rNDRUw4cPlyRNnz5d48ePV0pKimJiYpSfn6+cnBwtWrRILVpcrtUMBoOmTZumtLQ0BQQEyGQyKScnRydOnLBvPiFJffv21ZAhQzRv3jwlJyfLz89Pq1atUlBQkGJjYxvw1wAAAAAA11ev4um+++7T9u3blZWVJZvNpqCgID366KOaPHmyDAaDJKl///7KyMjQypUrtXXrVgUGBiolJUVjxoxx6GvSpEmy2WzasGGDzpw5o5CQEGVlZSk0NNQhbsWKFVq2bJkWLlyoqqoqRUREaPXq1U479QEAAABAY6pX8TRlyhRNmTLlhnFms/mGy+q8vLyUkJCghISE68b5+fkpNTVVqamp9RkqAAAAADSoW77nCQAAAACaA4onAAAAAHABxRMAAAAAuIDiCQAAAABcQPEEAAAAAC6geAIAAAAAF1A8AQAAAIALKJ4AAAAAwAUUTwAAAADgAoonAAAAAHABxRMAAAAAuIDiCQAAAABcQPEEAAAAAC6geAIAAAAAF1A8AQAAAIALKJ4AAAAAwAUUTwAAAADgAoonAMAdbc+ePRo/frwiIyPVp08fDRs2TEuXLlV5eblDXF5enkaPHq2wsDBFR0dr48aNdfaXnZ2tqKgohYWFKS4uTvv27XOKOX/+vBYsWKCIiAj1799fU6dO1alTpxrl+gAAtw+KJwDAHe3cuXPq37+/UlNTlZ2drfj4eG3dulUzZ860x+zfv19JSUkymUxau3atYmNjtXjxYuXk5Dj0lZ2drbS0NI0bN05ZWVnq1q2bEhMTdfToUYe42bNna9euXZo/f77S0tJ0+vRpxcfH6+LFi01yzQAA92jp7gEAAHArRo0apVGjRtlfR0REyGAwaP78+SopKVHHjh2Vnp4uk8mkJUuWSJIiIyNVXFysVatWKS4uTi1atJDValVmZqYmTJigyZMnS5IGDhyomJgYrVmzRmlpaZKkAwcOaPfu3crKypLZbJYkhYSEKDo6Wlu2bNHYsWOb+DcAAGgqzDwBADxOu3btJEmXLl2S1WrV3r17NXLkSIeYmJgYlZaW6vDhw5Kk/Px8lZeXOxRi3t7eGjFihPLy8mSz2SRdXv5nNBo1ePBge1znzp0VHh6uvLy8Rr4yAIA7MfMEAPAI1dXVunTpkgoKCpSenq6hQ4cqKChIBQUFqqqqUo8ePRzie/bsKUkqLCxUnz59VFhYKElOccHBwaqoqFBJSYkCAwNVWFio7t27y8vLy6m/Tz755LpjHDZs2DWPFRcXq1OnTi5fLwCg6VE8AQA8wtChQ1VSUiJJ+o//+A+tXLlS0uV7oiTJaDQ6xNe+rj1usVhkMBjk6+vrENe2bVtJUllZmQIDA2WxWOTv7+90fqPRaO8LAOCZKJ4AAB4hKytLFy5cUEFBgTIyMjR16lStW7fOfvzqmaK62uuKqV2ud6O467XX2rlz5zWPXW9WCgBwe6B4AgB4hHvvvVeSFB4eLpPJpLi4OH3wwQf25XlXzwpZLBZJ381AGY1GVVZWqrKyUq1atXKKq52BMhqNKi4udjq/xWJxmt0CAHgWNowAAHicXr16ydvbW0VFReratat8fHx07Ngxh5iCggJJl+9puvJn7b1PtQoLC9WmTRt17NjRHnf8+HH7jNSV/dX2AQDwTBRPAACPs3//flVXV6tLly4yGAyKjIxUbm6uQ8y2bdvUvn17mUwmSZdnrPz9/bV9+3Z7THV1tXJzc2U2m+1L8sxmsywWi/bs2WOPKy4uVn5+vn3rcgCAZ2LZHgDgjvbUU0+pT58+Cg0Nla+vr44cOaLXX39doaGhGj58uCRp+vTpGj9+vFJSUhQTE6P8/Hzl5ORo0aJFatHi8veIBoNB06ZNU1pamgICAmQymZSTk6MTJ07YN5+QpL59+2rIkCGaN2+ekpOT5efnp1WrVikoKEixsbFu+R0AAJoGxRMA4I523333afv27crKypLNZlNQUJAeffRRTZ48WQaDQZLUv39/ZWRkaOXKldq6dasCAwOVkpKiMWPGOPQ1adIk2Ww2bdiwQWfOnFFISIiysrIUGhrqELdixQotW7ZMCxcuVFVVlSIiIrR69WqnnfoAAJ6F4gkAcEebMmWKpkyZcsM4s9l8w2V1Xl5eSkhIUEJCwnXj/Pz8lJqaqtTU1HqNFQBwZ+OeJwAAAABwAcUTAAAAALiA4gkAAAAAXEDxBAAAAAAuoHgCAAAAABdQPAEAAACACyieAAAAAMAFFE8AAAAA4AKKJwAAAABwAcUTAAAAALiA4gkAAAAAXEDxBAAAAAAuoHgCAAAAABdQPAEAAACACyieAAAAAMAF9SqecnNzlZSUJLPZrH79+ikmJkZvv/22ampqHOLy8vI0evRohYWFKTo6Whs3bqyzv+zsbEVFRSksLExxcXHat2+fU8z58+e1YMECRUREqH///po6dapOnTpVn2EDAAAAwC2rV/G0bt06GQwGzZ07V2vWrNHw4cP14osvavny5faY/fv3KykpSSaTSWvXrlVsbKwWL16snJwch76ys7OVlpamcePGKSsrS926dVNiYqKOHj3qEDd79mzt2rVL8+fPV1pamk6fPq34+HhdvHjxFi4bAAAAAOqnZX2C16xZo4CAAPvryMhIXbhwQRs3btSsWbNkMBiUnp4uk8mkJUuW2GOKi4u1atUqxcXFqUWLFrJarcrMzNSECRM0efJkSdLAgQMVExOjNWvWKC0tTZJ04MAB7d69W1lZWTKbzZKkkJAQRUdHa8uWLRo7dmyD/BIAAAAA4EbqNfN0ZeFUq1evXqqsrFRZWZmsVqv27t2rkSNHOsTExMSotLRUhw8fliTl5+ervLxco0aNssd4e3trxIgRysvLk81mk3R5+Z/RaNTgwYPtcZ07d1Z4eLjy8vLqM3QAAAAAuCW3vGHE559/rnbt2unuu+9WUVGRqqqq1KNHD4eYnj17SpIKCwsdfl4dFxwcrIqKCpWUlNjjunfvLi8vL6f+avsAAAAAgKZQr2V7Vzt48KA2b96s6dOny9vbW+fOnZMkGY1Gh7ja17XHLRaLDAaDfH19HeLatm0rSSorK1NgYKAsFov8/f2dzms0Gu19XcuwYcOueay4uFidOnW6wdUBAAAAwHdueuaptLRUM2bMUFhYmBITEx2OXT1TVFd7XTG1y/VuFHe9dgAAAABoDDc181ReXq7ExET5+voqMzNTPj4+kr6bObp6VshisUj6bgbKaDSqsrJSlZWVatWqlVNcbT9Go1HFxcVO57dYLE6zW1fbuXPnNY9db1YKAAAAAOpS75mnyspKTZs2TWfOnNHrr7+uu+66y36sa9eu8vHx0bFjxxzeU1BQIOnyPU1X/rz6vqXCwkK1adNGHTt2tMcdP37cPiN1ZX+1fQAAAABAU6hX8XTp0iXNnDlTR44c0euvv66goCCH4waDQZGRkcrNzXVo37Ztm9q3by+TySRJCg8Pl7+/v7Zv326Pqa6uVm5ursxms31JntlslsVi0Z49e+xxxcXFys/Pt29dDgAAAABNoV7L9hYtWqSPPvpIc+bM0cWLF/XFF1/Yj/Xs2VN+fn6aPn26xo8fr5SUFMXExCg/P185OTlatGiRWrS4XKsZDAZNmzZNaWlpCggIkMlkUk5Ojk6cOKGVK1fa++zbt6+GDBmiefPmKTk5WX5+flq1apWCgoIUGxvbML8BAAAAAHBBvYqnTz75RJK0fPlyp2Pr169XRESE+vfvr4yMDK1cuVJbt25VYGCgUlJSNGbMGIf4SZMmyWazacOGDTpz5oxCQkKUlZWl0NBQh7gVK1Zo2bJlWrhwoaqqqhQREaHVq1c77dQHAAAAAI2pXsXTrl27XIozm803XFbn5eWlhIQEJSQkXDfOz89PqampSk1NdXmcAAAAANDQbvkhuQAAAADQHFA8AQAAAIALKJ4AAAAAwAUUTwAAAADgAoonAAAAAHABxRMAAAAAuIDiCQAAAABcQPEEALij5ebmKikpSWazWf369VNMTIzefvtt1dTUOMTl5eVp9OjRCgsLU3R0tDZu3Fhnf9nZ2YqKilJYWJji4uK0b98+p5jz589rwYIF9ofDT506VadOnWqU6wMA3D4ongAAd7R169bJYDBo7ty5WrNmjYYPH64XX3xRy5cvt8fs379fSUlJMplMWrt2rWJjY7V48WLl5OQ49JWdna20tDSNGzdOWVlZ6tatmxITE3X06FGHuNmzZ2vXrl2aP3++0tLSdPr0acXHx+vixYtNcs0AAPdo6e4BAABwK9asWaOAgAD768jISF24cEEbN27UrFmzZDAYlJ6eLpPJpCVLlthjiouLtWrVKsXFxalFixayWq3KzMzUhAkTNHnyZEnSwIEDFRMTozVr1igtLU2SdODAAe3evVtZWVkym82SpJCQEEVHR2vLli0aO3ZsE/8GAABNhZknAMAd7crCqVavXr1UWVmpsrIyWa1W7d27VyNHjnSIiYmJUWlpqQ4fPixJys/PV3l5uUaNGmWP8fb21ogRI5SXlyebzSbp8vI/o9GowYMH2+M6d+6s8PBw5eXlNcYlAgBuE8w8AbjjnP7mgiwVVred39jGoA53tXbb+XFjn3/+udq1a6e7775bx48fV1VVlXr06OEQ07NnT0lSYWGh+vTpo8LCQklyigsODlZFRYVKSkoUGBiowsJCde/eXV5eXk79ffLJJ9cd17Bhw655rLi4WJ06dXL5GgEATY/iCcAd5fQ3F5T0612qtFa7bQytDN7KmBtFAXWbOnjwoDZv3qzp06fL29tb586dkyQZjUaHuNrXtcctFosMBoN8fX0d4tq2bStJKisrU2BgoCwWi/z9/Z3OazQa7X0BADwTxROAO4qlwqpKa7VmPxGuLh2d/wLb2E6WlGvF2/myVFgpnm5DpaWlmjFjhsLCwpSYmOhw7OqZorra64qpXa53o7jrtdfauXPnNY9db1YKAHB7oHgCcEfq0tFfPbu0c/cwcBspLy9XYmKifH19lZmZKR8fH0nfzRxdPStksVgkfTcDZTQaVVlZqcrKSrVq1coprrYfo9Go4uJip/NbLBan2S0AgGdhwwgAwB2vsrJS06ZN05kzZ/T666/rrrvush/r2rWrfHx8dOzYMYf3FBQUSLp8T9OVP2vvfapVWFioNm3aqGPHjva448eP22ekruyvtg8AgGeieAIA3NEuXbqkmTNn6siRI3r99dcVFBTkcNxgMCgyMlK5ubkO7du2bVP79u1lMpkkSeHh4fL399f27dvtMdXV1crNzZXZbLYvyTObzbJYLNqzZ489rri4WPn5+fatywEAnollewCAO9qiRYv00Ucfac6cObp48aK++OIL+7GePXvKz89P06dP1/jx45WSkqKYmBjl5+crJydHixYtUosWl79HNBgMmjZtmtLS0hQQECCTyaScnBydOHFCK1eutPfZt29fDRkyRPPmzVNycrL8/Py0atUqBQUFKTY2tqkvHwDQhCieAAB3tNrtwZcvX+50bP369YqIiFD//v2VkZGhlStXauvWrQoMDFRKSorGjBnjED9p0iTZbDZt2LBBZ86cUUhIiLKyshQaGuoQt2LFCi1btkwLFy5UVVWVIiIitHr1aqed+gAAnoXiCQBwR9u1a5dLcWaz+YbL6ry8vJSQkKCEhITrxvn5+Sk1NVWpqakujxMAcOfjnicAAAAAcAHFEwAAAAC4gOIJAAAAAFxA8QQAAAAALqB4AgAAAAAXUDwBAAAAgAsongAAAADABRRPAAAAAOACiicAAAAAcAHFEwAAAAC4gOIJAAAAAFxA8QQAAAAALqB4AgAAAAAXUDwBAAAAgAsongAAAADABRRPAAAAAOACiicAAAAAcAHFEwAAAAC4gOIJAAAAAFxA8QQAAAAALqB4AgAAAAAXtHT3AAAAAIBbdfqbC7JUWN12fmMbgzrc1dpt50fToHgCAADAHe30NxeU9OtdqrRWu20MrQzeypgbRQHl4SieAAAAcEezVFhVaa3W7CfC1aWjf5Of/2RJuVa8nS9LhZXiycNRPAEAAMAjdOnor55d2rl7GPBg9d4w4uuvv9aCBQv00EMPyWQyadSoUXXG5eXlafTo0QoLC1N0dLQ2btxYZ1x2draioqIUFhamuLg47du3zynm/PnzWrBggSIiItS/f39NnTpVp06dqu/QAQAAAOCm1bt4+uqrr5SXl6du3bopODi4zpj9+/crKSlJJpNJa9euVWxsrBYvXqycnByHuOzsbKWlpWncuHHKyspSt27dlJiYqKNHjzrEzZ49W7t27dL8+fOVlpam06dPKz4+XhcvXqzv8AEAAADgptR72V5UVJSGDx8uSUpOTtaXX37pFJOeni6TyaQlS5ZIkiIjI1VcXKxVq1YpLi5OLVq0kNVqVWZmpiZMmKDJkydLkgYOHKiYmBitWbNGaWlpkqQDBw5o9+7dysrKktlsliSFhIQoOjpaW7Zs0dixY2/uygEAAACgHuo989SixfXfYrVatXfvXo0cOdKhPSYmRqWlpTp8+LAkKT8/X+Xl5Q7L/ry9vTVixAjl5eXJZrNJurz8z2g0avDgwfa4zp07Kzw8XHl5efUdPgAAAADclAZ/SG5RUZGqqqrUo0cPh/aePXtKkgoLCx1+Xh0XHBysiooKlZSU2OO6d+8uLy8vp/5q+wAAAACAxtbgu+2dO3dOkmQ0Gh3aa1/XHrdYLDIYDPL19XWIa9u2rSSprKxMgYGBslgs8vd33nLSaDTa+6rLsGHDrnmsuLhYnTp1cuFqAAAAAOCyBp95qnX1TFFd7XXF1C7Xu1Hc9doBAAAAoKE1+MxT7czR1bNCFotF0nczUEajUZWVlaqsrFSrVq2c4mr7MRqNKi4udjqPxWJxmt260s6dO6957HqzUgAAAABQlwafeeratat8fHx07Ngxh/aCggJJsm9vXvvz6vuWCgsL1aZNG3Xs2NEed/z4cfuM1JX9XWurdAAAAABoaA1ePBkMBkVGRio3N9ehfdu2bWrfvr1MJpMkKTw8XP7+/tq+fbs9prq6Wrm5uTKbzfYleWazWRaLRXv27LHHFRcXKz8/3751OQAAAAA0tnov2/v222/tW4SfOnVK58+f144dOyRdfk5TQECApk+frvHjxyslJUUxMTHKz89XTk6OFi1aZN/q3GAwaNq0aUpLS1NAQIBMJpNycnJ04sQJrVy50n6+vn37asiQIZo3b56Sk5Pl5+enVatWKSgoSLGxsQ3xOwAA3MG+/vprZWdn68CBA/rqq6/Uo0cPbdu2zSkuLy9PaWlpKiwsVGBgoCZOnKhx48Y5xWVnZ2vjxo0qLS1VSEiI5s6dq4iICIeY8+fP69e//rX+53/+R1arVREREZo/f76CgoIa7ToBAO5X7+Lp7NmzmjlzpkNb7ev169crIiJC/fv3V0ZGhlauXKmtW7cqMDBQKSkpGjNmjMP7Jk2aJJvNpg0bNujMmTMKCQlRVlaWQkNDHeJWrFihZcuWaeHChaqqqlJERIRWr17ttFMfAKD5+eqrr5SXl6e+ffuqpqbGaZm3JO3fv19JSUl66KGHlJycrPz8fC1evFgGg8EhN2VnZystLU2zZs2yf6mXmJionJwch9w0e/ZsHTp0SPPnz5efn59eeeUVxcfH6w9/+AO5CQA8WL2Lpy5duujo0aM3jDObzTdcVufl5aWEhAQlJCRcN87Pz0+pqalKTU2t11gBAJ4vKipKw4cPlyQlJyfryy+/dIpJT0+XyWTSkiVLJEmRkZEqLi7WqlWrFBcXpxYtWshqtSozM1MTJkzQ5MmTJV1eURETE6M1a9YoLS1NknTgwAHt3r1bWVlZ9jwXEhKi6OhobdmyRWPHjm2KywYAuEGjbVUOAEBTqF0Ofi1Wq1V79+7VyJEjHdpjYmJUWlqqw4cPS5Ly8/NVXl6uUaNG2WO8vb01YsQI5eXl2We08vLyZDQaNXjwYHtc586dFR4ebl/WDgDwTBRPAACPVlRUpKqqKvXo0cOhvWfPnpK+2/W19ufVccHBwaqoqFBJSYk9rnv37k7PGuzZs6fTDrIAAM/S4M95AgDgdlL73MGrnw1Y+7r2uMVikcFgcLpnqfa5g2VlZQoMDJTFYpG/v7/TeYxGo9MzDq92vecMFhcXq1OnTje4GgCAO1E8AcBNOFlS7rZzG9sY1OGu1m47/53q6pmiutrriqldrnejuOu1AwA8A8UTANSDsY1BrQzeWvF2vtvG0MrgrYy5URRQLqqdObp6VshisUj6bgbKaDSqsrJSlZWVatWqlVNcbT9Go1HFxcVO57FYLE6zW1fbuXPnNY9db1YKAHB7oHgCgHrocFdrZcyNkqXC6pbznywp14q382WpsFI8uahr167y8fHRsWPHHDZ5KCgokHT5nqYrfxYWFtof6F77uk2bNurYsaM97k9/+pNsNpvDTFNBQYG9DwCAZ6J4AoB66nBXawqXO4jBYFBkZKRyc3M1ceJEe/u2bdvUvn17e6EUHh4uf39/bd++3d5WXV2t3Nxcmc1me6FkNpuVnp6uPXv22Iux4uJi5efnKyUlpWkvDgDQpCieAAB3tG+//da+RfipU6d0/vx57dixQ9Ll5zQFBARo+vTpGj9+vFJSUhQTE6P8/Hzl5ORo0aJF9q3ODQaDpk2bprS0NAUEBNgfknvixAmtXLnSfr6+fftqyJAhmjdvnpKTk+Xn56dVq1YpKChIsbGxTf8LAAA0GYonAMAd7ezZs5o5c6ZDW+3r9evXKyIiQv3791dGRoZWrlyprVu3KjAwUCkpKRozZozD+yZNmiSbzaYNGzbozJkzCgkJUVZWlkJDQx3iVqxYoWXLlmnhwoWqqqpSRESEVq9e7bRTHwDAs1A8AQDuaF26dNHRo0dvGGc2m2U2m68b4+XlpYSEBCUkJFw3zs/PT6mpqUpNTa3XWAEAdzYekgsAAAAALqB4AgAAAAAXsGwPAAAAaAA8QN3zUTwBAAAAt4AHqDcfFE8AAADALeAB6s0HxRMAAABwi3iAevPAhhEAAAAA4AKKJwAAAABwAcUTAAAAALiA4gkAAAAAXEDxBAAAAAAuYLc9APV2+psLbt2OFQAAwB0ongDUy+lvLijp17tUaa122xhaGbxlbGNw2/kBAEDzRPEEoF4sFVZVWqs1+4lwdeno75YxGNsYeJYGAABochRPAG5Kl47+6tmlnbuHAQAA0GTYMAIAAAAAXEDxBAAAAAAuoHgCAAAAABdQPAEAAACACyieAAAAAMAFFE8AAAAA4AKKJwAAAABwAcUTAAAAALiA4gkAAAAAXEDxBAAAAAAuoHgCAAAAABdQPAEAAACACyieAAAAAMAFFE8AAAAA4AKKJwAAAABwAcUTAAAAALiA4gkAAAAAXEDxBAAAAAAuoHgCAAAAABdQPAEAAACAC1q6ewCuOH78uBYvXqzPP/9c3/ve9zRy5Eg999xz8vX1dffQAMAtTpaUu+3cxjYGdbirtdvOf7sgNwFA83PbF08Wi0W/+MUv1LlzZ73yyiv697//raVLl6qsrEy/+c1v3D08AGhSxjYGtTJ4a8Xb+W4bQyuDtzLmRjXrAorcBOB2xBdrje+2L55+97vfyWKxaOvWrQoICJAkeXt767nnntO0adMUHBzs5hECQNPpcFdrZcyNkqXC6pbznywp14q382WpsDaLJHkt5CYAtxO+WGs6t33x9PHHH2vQoEH25CRJP/vZz/TLX/5SeXl5JCgAzU6Hu1p7fHK63ZGbANxO+GKt6dz2xVNhYaHi4uIc2gwGg7p27arCwkI3jQoA0JyRmwBnp7+54Na/vDd3fLHWNG774slischoNDq1G41GnTt37prvGzZs2DWPnTx5Ut7e3teNuZ6q6hr9+9xFTfr8Zfl4N/2GhbXn/8W+lfJuyYaJaFrVl2p0rsLqtv/+4V63+vlXXFwsb2/vRhhZ0yI3Xfv85KbmyVZj0zflle4ehhLyV8m7hZe7h9Hs3A7//3t7eanFTf67r09uuu2Lp2ux2Wzy8rq5X5CXl5datrz5Sz9zukSS5OPtnurex7uFOgY0zbmLi4slSZ06dWqS87kL1+k6H+8W8m11+3908O+0cdzq50/Lli1lMBgacES3F3JTw5/bE/9f9sRrkrdUU3n5iwNPui6P/Helhr+upvy76fXc7HXVJzfd9n8DMhqNslgsTu3l5eXXXVO+c+fORhtT7beCjXmO20VzuVau0/M0l2ttLtd5uyE3NR1PvC5PvCbJM6/LE69J4rpuxW0/rx4cHOy0ftxqtaqoqIgbcgEAbkFuAoDm6bYvngYPHqy9e/fqm2++sbd98MEHslqtMpvNbhwZAKC5IjcBQPN02xdPjz/+uPz9/ZWUlKQ9e/Zo69atSk1NVUxMDN/uAQDcgtwEAM3THXHP029/+1stXrxYTz/9tHx9fTVq1Cg999xz7h4aAKCZIjcBQPN02xdPktS9e3dlZ2e7exgAANiRmwCg+bntl+0BAAAAwO3Ay2az2dw9CAAAAAC43THzBAAAAAAuoHgCAAAAABdQPAEAAACACyieAAAAAMAFFE8AAAAA4AKKp3o4fvy4Jk+erH79+mnQoEFavHixLl686O5hNbjc3FwlJSXJbDarX79+iomJ0dtvv62amhp3D61RVVRUaPDgwQoNDdXBgwfdPZxGkZOTowcffFBhYWEaNGiQpk6d6u4hNbgPP/xQY8aMUXh4uO6//3499dRTOnbsmLuHdUu+/vprLViwQA899JBMJpNGjRpVZ1xeXp5Gjx6tsLAwRUdHa+PGjU08UriDJ+am5pCHPC3neFp+8YRc4qm540bXVV1drbVr12r8+PGKjIzUgAEDNG7cOH366acNcv474iG5twOLxaJf/OIX6ty5s1555RX9+9//1tKlS1VWVqbf/OY37h5eg1q3bp06d+6suXPn6u6779a+ffv04osv6sSJE3r++efdPbxGk5GRoerqancPo9GsXr1ab775pqZOnaq+ffvq3Llz2rNnj7uH1aD+9Kc/6amnntKDDz6oZ555RhaLRa+++qri4+P1xz/+UX5+fu4e4k356quvlJeXp759+6qmpkZ1PWFi//79SkpK0kMPPaTk5GTl5+dr8eLFMhgMGjNmjBtGjabgqbmpOeQhT8o5npZfPCWXeGruuNF1Xbx4Ua+99ppGjx6tyZMnq2XLltqyZYvi4+OVmZmpoUOH3toAbHDJa6+9Zuvbt6/t7Nmz9rY//OEPtpCQEFtBQYEbR9bwrrzGWkuWLLGFhYXZKisr3TCixldQUGDr16+f7b//+79tISEhtr/+9a/uHlKDKigosPXq1cu2Z88edw+lUf3yl7+0DR061FZTU2NvO3DggC0kJMS2e/duN47s1lRXV9v/+fnnn7eNHDnSKWby5Mm2Rx55xKEtJSXF9pOf/MTh/fAsnpqbPD0PeVLO8cT84im5xFNzx42u69KlS7aysjKHtpqaGltsbKxt/Pjxt3x+lu256OOPP9agQYMUEBBgb/vZz34mg8GgvLw8N46s4V15jbV69eqlyspKlZWVNf2AmsCLL76oxx9/XN27d3f3UBrF5s2bdc899+iBBx5w91Aa1aVLl9SmTRt5eXnZ2/z9/d04oobRosX1P6qtVqv27t2rkSNHOrTHxMSotLRUhw8fbszhwY08NTd5eh7ypJzjifnFU3KJp+aOG12Xt7e32rZt69Dm5eWle++9V6dPn771899yD81EYWGhgoODHdoMBoO6du2qwsJCN42q6Xz++edq166d7r77bncPpcHt2LFDR44c0fTp0909lEZz4MABhYSEKD09XYMGDVKfPn00fvx4/e1vf3P30BrUI488omPHjmnDhg2yWCw6efKkli1bpuDgYA0aNMjdw2s0RUVFqqqqUo8ePRzae/bsKUnN4jOquWpOuclT8pCn5RxPzC/NJZc0p9xRU1Oj/fv3O31e3gyKJxdZLBYZjUandqPRqHPnzrlhRE3n4MGD2rx5s37xi1/I29vb3cNpUN9++61eeuklPfvss3fMGuabUVpaqk8++UTvv/++Fi5cqNWrV+vbb79VfHy8LBaLu4fXYAYMGKBXX31VaWlpGjBggIYNG6YTJ07ojTfekMFgcPfwGk3tZ9DVn1G1rz39M6o5ay65yVPykCfmHE/ML80llzSn3LFhwwYdP35c8fHxt9wXxdMtstlsDtO6nqa0tFQzZsxQWFiYEhMT3T2cBpeZmam7775bDz/8sLuH0qhsNpsuXLig1atX66c//amGDh2qzMxMVVRU6J133nH38BpMfn6+5syZo7i4OL355pt69dVX5evrq8TERJ0/f97dw2t01/os8uTPKNTNk3KTJ+UhT8w5nphfmlsu8fTc8ec//1nLly/XpEmTNGDAgFvuj932XGQ0Guv8BqW8vLxBpgBvR+Xl5UpMTJSvr68yMzPl4+Pj7iE1qFOnTumNN95Qenq6/cPwwoUL9p8VFRVq06aNO4fYYNq2bavvf//7+uEPf2hv69Chg3r06KGCggI3jqxhLV68WJGRkZo3b5697Uc/+pEGDx6snJycBvnG6XZUu7b76m8Jaz+z6pqZgGfw9NzkSXnIU3OOJ+aX5pJLmkPuOHLkiJKSkjR8+HDNmTOnQfqkeHJRcHCw09pPq9WqoqIixcXFuWlUjaeyslLTpk3TmTNn9M477+iuu+5y95Aa3MmTJ1VVVaUpU6Y4HZswYYL69u2rd9991w0ja3jBwcH65z//6dRus9lueOPlnaSwsFBRUVEObQEBAerQoYOKiorcNKrG17VrV/n4+OjYsWMaPHiwvb32Ly6e8Jdo1M2Tc5On5SFPzTmemF+aSy7x9NxRVFSkhIQEmUwm/frXv26wmTSKJxcNHjxYmZmZ+uabb+wf4B988IGsVqvMZrObR9ewLl26pJkzZ+rIkSN66623FBQU5O4hNYpevXpp/fr1Dm1/+9vftHTpUi1cuFBhYWFuGlnDGzJkiLZs2aK///3vCgkJkSSVlJTo2LFjHrV8pHPnzjp06JBDW2lpqU6fPu2x/x1LlzcIiIyMVG5uriZOnGhv37Ztm9q3by+TyeS+waFReWpu8sQ85Kk5xxPzS3PJJZ6cO0pLSzVp0iR9//vfV0ZGRoPeq0bx5KLHH39cb731lpKSkpSUlKSzZ8/qpZdeUkxMzB1fmV9t0aJF+uijjzRnzhxdvHhRX3zxhf1Yz549PeYmV6PRqIiIiDqP9e7dW717927iETWe6Oho9e7dW08//bRmzpwpg8Gg9PR0BQQE6NFHH3X38BrMuHHjlJqaqkWLFmnYsGGyWCx67bXX1Lp1az344IPuHt5N+/bbb+3bTp86dUrnz5/Xjh07JEkDBw5UQECApk+frvHjxyslJUUxMTHKz89XTk6OFi1adMd++4sb89Tc5Il5yFNzjifmF0/JJZ6aO250Xa1bt1ZCQoLOnj2r5ORkp+Wj/fr1u6Xze9lsdTxuGHU6fvy4Fi9erM8//1y+vr4aNWqUnnvuOfn6+rp7aA0qKipKp06dqvPY+vXrr/nh7wn27dunCRMmaNOmTXfst4DXcvbsWS1ZskR5eXm6dOmSBgwYoBdeeMFpi9I7mc1m07vvvqu3335bRUVFat26tcLCwjRr1iyFhoa6e3g37eTJkxo2bFidx678fzIvL08rV65UYWGhAgMDFR8fr3HjxjXlUOEGnpibmkse8pSc42n5xVNyiafmjhtdV1BQ0DWPS9LRo0dv6fwUTwAAAADggttzPg4AAAAAbjMUTwAAAADgAoonAAAAAHABxRMAAAAAuIDiCQAAAABcQPEEAAAAAC6geAIAAAAAF1A8AQAAAIALKJ4AAAAAwAUUTwAAAADgAoonAAAAAHDB/wersUwtEjQH+gAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 1000x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's see the distribution of two genes and how they can differ\n",
    "\n",
    "# set seed for numpy random number generators\n",
    "np.random.seed(42)\n",
    "genes = np.random.choice(train_X.columns, 2)\n",
    "plt.figure(figsize=(10,3))\n",
    "ax = plt.subplot(1,2,1)\n",
    "ax.hist(train_X[genes[0]], histtype=\"step\", bins=10)\n",
    "ax.set_title(genes[0])\n",
    "\n",
    "# We can plot these to see if there are some outliers\n",
    "ax = plt.subplot(1,2,2)\n",
    "plt.hist(train_X[genes[1]], histtype=\"step\", bins=10)\n",
    "plt.title(genes[1])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "711da0b0-ad38-427f-97aa-3503c4377178",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Here we'll generate three scalings and store data \n",
    "\n",
    "from sklearn.preprocessing import MinMaxScaler, StandardScaler\n",
    "minmaxscaler = MinMaxScaler() # This will store the computed parameters for the scaling\n",
    "meanvarscaler = StandardScaler()\n",
    "\n",
    "minmaxscaler.fit(train_X)\n",
    "meanvarscaler.fit(train_X)\n",
    "\n",
    "train_X_minmax = minmaxscaler.transform(train_X) # transform converts pandas dataframe to numpy array\n",
    "train_X_meanvar = meanvarscaler.transform(train_X) # transform converts pandas dataframe to numpy array\n",
    "\n",
    "train_X_minmax = pd.DataFrame(train_X_minmax, columns=train_X.columns, index=train_X.index) # reattach indices and columns\n",
    "train_X_meanvar = pd.DataFrame(train_X_meanvar, columns=train_X.columns, index=train_X.index) # reattach indices and columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "02bb4304-47f9-432d-8f7a-01291549baec",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# transform test set\n",
    "test_X_minmax = minmaxscaler.transform(test_X) \n",
    "test_X_meanvar = meanvarscaler.transform(test_X) \n",
    "\n",
    "test_X_minmax = pd.DataFrame(test_X_minmax, columns=test_X.columns, index=test_X.index) # reattach indices and columns\n",
    "test_X_meanvar = pd.DataFrame(test_X_meanvar, columns=test_X.columns, index=test_X.index) # reattach indices and columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "2f44fa22-a264-400c-b784-433648d36ebd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Min Max scaler\n",
      "Min: [0. 0. 0. ... 0. 0. 0.]\n",
      "Max: [ 8.97154933 10.24218023  9.55086927 ... 10.92318296 12.57041028\n",
      " 11.06187894]\n",
      "Shape of Min and Max: 10851\n",
      "-----\n",
      "Mean: [0.09520964 1.13363265 0.02843088 ... 0.83848201 0.27343459 3.54945278]\n",
      "Var: [ 0.74918476  7.96709673  0.23065692 ...  6.25453479  2.40668545\n",
      " 17.57992874]\n",
      "Shape of Mean and Var: 10851\n"
     ]
    }
   ],
   "source": [
    "# parameters can be checked\n",
    "print(\"Min Max scaler\")\n",
    "print(f\"Min: {minmaxscaler.data_min_}\")\n",
    "print(f\"Max: {minmaxscaler.data_max_}\")\n",
    "print(f\"Shape of Min and Max: {minmaxscaler.data_min_.shape[0]}\")\n",
    "\n",
    "print(\"-----\")\n",
    "# parameters can be checked\n",
    "print(f\"Mean: {meanvarscaler.mean_}\")\n",
    "print(f\"Var: {meanvarscaler.var_}\")\n",
    "print(f\"Shape of Mean and Var: {meanvarscaler.mean_.shape[0]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95adff7f-573e-4701-ac25-7b077d67826b",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Further since we may have so-called \"curse of dimensionality\", we can take following approaches:\n",
    "\n",
    "1. Do nothing: Use data as it is\n",
    "2. Select highly variable genes, as they impact more the change in classes\n",
    "3. Supervised feature selection\n",
    "4. Do regularization while learning -> Not all methods can do that. Support vector machines, multilayered perceptrons can. RF can't.\n",
    "3. Perform a dimensionality reduction technique: PCA, autoencoder (Autoencoders will be covered tomorrow)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "4ce8ca18-94bd-4431-9b12-366dafc7f92d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def select_khvg(X, y, k=500):\n",
    "    \"\"\"\n",
    "    Per cell type, select k most variable features (by standard deviation)\n",
    "    \"\"\"\n",
    "    feats = []\n",
    "    for e in y.unique():\n",
    "        idxs_e = y[y==e].index\n",
    "        sub_df = X.loc[idxs_e]\n",
    "        stds = sub_df.std() # compute standard deviation\n",
    "        stds.sort_values(ascending=False) # sort by decreasing order\n",
    "        feats = feats + stds.index[:k].tolist()\n",
    "    \n",
    "    feats = list(set(feats)) # remove duplicated features\n",
    "    return X[feats]\n",
    "\n",
    "def do_pca():\n",
    "    \"\"\"\n",
    "    Perform PCA with n components. See from here: https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html\n",
    "    Remember to save the PCA class so that you can apply it on the test set. Similar to minmaxscaler\n",
    "    Bonus: make informed choice about n.\n",
    "    \"\"\"\n",
    "    pass\n",
    "\n",
    "def do_sup():\n",
    "    \"\"\"\n",
    "    Perform suprvised feature selection using method of your choice (Model based or univariate). \n",
    "    See from here https://scikit-learn.org/stable/modules/feature_selection.html#univariate-feature-selection\n",
    "    \"\"\"\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "6b0f4ea6-dba1-44b9-a12f-572b75bc1547",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_X_meanvar_khvg = select_khvg(train_X_meanvar, train_y)\n",
    "\n",
    "# train_X_pca = do_pca(...)\n",
    "# train_X_sup = do_sup(...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "84df5963-0a64-4b0c-95ba-cfd531cb1874",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Apply it on the test data\n",
    "\n",
    "test_X_meanvar_khvg = test_X_meanvar[train_X_meanvar_khvg.columns]\n",
    "\n",
    "# test_X_pca = ... # Hint: Apply PCA transform\n",
    "# test_X_sup = ... # Hint: Will vary depending on the method of choice\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa1c8dcd-89cb-453a-ad10-a1560fddcf5c",
   "metadata": {},
   "source": [
    "## Build general optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "d3a6a334-3726-43b2-a3f7-d664667dcd6c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "def optimize_clf(X, y, estimator, params, n_iter=10, cv=None, verbose=3, scoring=\"balanced_accuracy\"):\n",
    "    \"\"\"\n",
    "    Wraps around https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html and randomly searches over\n",
    "        parameter distributions and scores each iteration\n",
    "    \n",
    "    X, y: train_X, train_y\n",
    "    estimator: the classifier e.g. forest = RandomForestClassifier(random_state=42)\n",
    "    params: dictionary containing parameter distributions\n",
    "    n_iter: Number of iterations\n",
    "    cv: k for k-fold cross validation, default 5\n",
    "    verbose: Controls the verbosity: the higher, the more messages.\n",
    "        >1 : the computation time for each fold and parameter candidate is displayed;\n",
    "        >2 : the score is also displayed;\n",
    "        >3 : the fold and candidate parameter indexes are also displayed together with the starting time of the computation.\n",
    "    scoring: a string specifying method to use for scoring each run or callable function\n",
    "    \"\"\"\n",
    "    cv = RandomizedSearchCV(estimator=estimator, param_distributions=params, n_iter=n_iter, cv=cv, verbose=verbose, scoring=scoring)\n",
    "    cv.fit(X, y)\n",
    "    return cv.best_params_, cv.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2135b2a0-f758-4a1d-98ca-5b2bce01f535",
   "metadata": {},
   "source": [
    "## Build classifiers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "5a41efd6-e32c-49b2-9fd0-fed7a30f78b6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## Below, we call RandomForestClassifier class and optimize the hyperparameters on data with all genes and khvg\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "forest1 = RandomForestClassifier(random_state=42)\n",
    "forest2 = RandomForestClassifier(random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "1cdfa796-dd50-4b0c-bafe-42c1c633bb48",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bootstrap': True,\n",
       " 'ccp_alpha': 0.0,\n",
       " 'class_weight': None,\n",
       " 'criterion': 'gini',\n",
       " 'max_depth': None,\n",
       " 'max_features': 'sqrt',\n",
       " 'max_leaf_nodes': None,\n",
       " 'max_samples': None,\n",
       " 'min_impurity_decrease': 0.0,\n",
       " 'min_samples_leaf': 1,\n",
       " 'min_samples_split': 2,\n",
       " 'min_weight_fraction_leaf': 0.0,\n",
       " 'n_estimators': 100,\n",
       " 'n_jobs': None,\n",
       " 'oob_score': False,\n",
       " 'random_state': None,\n",
       " 'verbose': 0,\n",
       " 'warm_start': False}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get params (https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html)\n",
    "RandomForestClassifier().get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "bbd8079f-3457-4073-b964-6b7d15f53c47",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# We'll optimize two hyperparameters\n",
    "params = {'max_depth': [5, 10, 15, 20],\n",
    "          'min_samples_split': [5, 10, 20]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "a5ae5794-7aa1-416c-a192-c41987836974",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 10 candidates, totalling 50 fits\n",
      "[CV 1/5] END .max_depth=20, min_samples_split=5;, score=0.734 total time=  10.2s\n",
      "[CV 2/5] END .max_depth=20, min_samples_split=5;, score=0.659 total time=  10.3s\n",
      "[CV 3/5] END .max_depth=20, min_samples_split=5;, score=0.685 total time=  10.1s\n",
      "[CV 4/5] END .max_depth=20, min_samples_split=5;, score=0.718 total time=  10.2s\n",
      "[CV 5/5] END .max_depth=20, min_samples_split=5;, score=0.673 total time=  10.2s\n",
      "[CV 1/5] END max_depth=20, min_samples_split=20;, score=0.714 total time=   9.8s\n",
      "[CV 2/5] END max_depth=20, min_samples_split=20;, score=0.662 total time=   9.8s\n",
      "[CV 3/5] END max_depth=20, min_samples_split=20;, score=0.689 total time=   9.7s\n",
      "[CV 4/5] END max_depth=20, min_samples_split=20;, score=0.723 total time=   9.7s\n",
      "[CV 5/5] END max_depth=20, min_samples_split=20;, score=0.678 total time=   9.7s\n",
      "[CV 1/5] END ..max_depth=5, min_samples_split=5;, score=0.413 total time=   3.5s\n",
      "[CV 2/5] END ..max_depth=5, min_samples_split=5;, score=0.395 total time=   3.5s\n",
      "[CV 3/5] END ..max_depth=5, min_samples_split=5;, score=0.377 total time=   3.5s\n",
      "[CV 4/5] END ..max_depth=5, min_samples_split=5;, score=0.398 total time=   3.5s\n",
      "[CV 5/5] END ..max_depth=5, min_samples_split=5;, score=0.394 total time=   3.5s\n",
      "[CV 1/5] END .max_depth=10, min_samples_split=5;, score=0.720 total time=   6.5s\n",
      "[CV 2/5] END .max_depth=10, min_samples_split=5;, score=0.649 total time=   6.5s\n",
      "[CV 3/5] END .max_depth=10, min_samples_split=5;, score=0.678 total time=   6.5s\n",
      "[CV 4/5] END .max_depth=10, min_samples_split=5;, score=0.708 total time=   6.5s\n",
      "[CV 5/5] END .max_depth=10, min_samples_split=5;, score=0.666 total time=   6.5s\n",
      "[CV 1/5] END max_depth=10, min_samples_split=20;, score=0.629 total time=   6.3s\n",
      "[CV 2/5] END max_depth=10, min_samples_split=20;, score=0.642 total time=   6.3s\n",
      "[CV 3/5] END max_depth=10, min_samples_split=20;, score=0.645 total time=   6.3s\n",
      "[CV 4/5] END max_depth=10, min_samples_split=20;, score=0.616 total time=   6.3s\n",
      "[CV 5/5] END max_depth=10, min_samples_split=20;, score=0.662 total time=   6.3s\n",
      "[CV 1/5] END max_depth=15, min_samples_split=20;, score=0.724 total time=   8.3s\n",
      "[CV 2/5] END max_depth=15, min_samples_split=20;, score=0.655 total time=   8.3s\n",
      "[CV 3/5] END max_depth=15, min_samples_split=20;, score=0.679 total time=   8.3s\n",
      "[CV 4/5] END max_depth=15, min_samples_split=20;, score=0.707 total time=   8.3s\n",
      "[CV 5/5] END max_depth=15, min_samples_split=20;, score=0.675 total time=   8.3s\n",
      "[CV 1/5] END .max_depth=5, min_samples_split=20;, score=0.406 total time=   3.5s\n",
      "[CV 2/5] END .max_depth=5, min_samples_split=20;, score=0.391 total time=   3.5s\n",
      "[CV 3/5] END .max_depth=5, min_samples_split=20;, score=0.382 total time=   3.5s\n",
      "[CV 4/5] END .max_depth=5, min_samples_split=20;, score=0.404 total time=   3.5s\n",
      "[CV 5/5] END .max_depth=5, min_samples_split=20;, score=0.397 total time=   3.5s\n",
      "[CV 1/5] END .max_depth=5, min_samples_split=10;, score=0.419 total time=   3.5s\n",
      "[CV 2/5] END .max_depth=5, min_samples_split=10;, score=0.380 total time=   3.5s\n",
      "[CV 3/5] END .max_depth=5, min_samples_split=10;, score=0.385 total time=   3.5s\n",
      "[CV 4/5] END .max_depth=5, min_samples_split=10;, score=0.402 total time=   3.5s\n",
      "[CV 5/5] END .max_depth=5, min_samples_split=10;, score=0.392 total time=   3.5s\n",
      "[CV 1/5] END .max_depth=15, min_samples_split=5;, score=0.711 total time=   8.7s\n",
      "[CV 2/5] END .max_depth=15, min_samples_split=5;, score=0.660 total time=   8.7s\n",
      "[CV 3/5] END .max_depth=15, min_samples_split=5;, score=0.681 total time=   8.6s\n",
      "[CV 4/5] END .max_depth=15, min_samples_split=5;, score=0.699 total time=   8.7s\n",
      "[CV 5/5] END .max_depth=15, min_samples_split=5;, score=0.671 total time=   8.7s\n",
      "[CV 1/5] END max_depth=10, min_samples_split=10;, score=0.695 total time=   6.4s\n",
      "[CV 2/5] END max_depth=10, min_samples_split=10;, score=0.649 total time=   6.4s\n",
      "[CV 3/5] END max_depth=10, min_samples_split=10;, score=0.671 total time=   6.4s\n",
      "[CV 4/5] END max_depth=10, min_samples_split=10;, score=0.671 total time=   6.4s\n",
      "[CV 5/5] END max_depth=10, min_samples_split=10;, score=0.658 total time=   6.4s\n",
      "Fitting 5 folds for each of 10 candidates, totalling 50 fits\n",
      "[CV 1/5] END max_depth=20, min_samples_split=10;, score=0.396 total time=   2.3s\n",
      "[CV 2/5] END max_depth=20, min_samples_split=10;, score=0.347 total time=   2.3s\n",
      "[CV 3/5] END max_depth=20, min_samples_split=10;, score=0.403 total time=   2.3s\n",
      "[CV 4/5] END max_depth=20, min_samples_split=10;, score=0.342 total time=   2.3s\n",
      "[CV 5/5] END max_depth=20, min_samples_split=10;, score=0.340 total time=   2.3s\n",
      "[CV 1/5] END .max_depth=20, min_samples_split=5;, score=0.385 total time=   2.3s\n",
      "[CV 2/5] END .max_depth=20, min_samples_split=5;, score=0.389 total time=   2.4s\n",
      "[CV 3/5] END .max_depth=20, min_samples_split=5;, score=0.417 total time=   2.4s\n",
      "[CV 4/5] END .max_depth=20, min_samples_split=5;, score=0.375 total time=   2.4s\n",
      "[CV 5/5] END .max_depth=20, min_samples_split=5;, score=0.385 total time=   2.4s\n",
      "[CV 1/5] END ..max_depth=5, min_samples_split=5;, score=0.239 total time=   0.8s\n",
      "[CV 2/5] END ..max_depth=5, min_samples_split=5;, score=0.236 total time=   0.8s\n",
      "[CV 3/5] END ..max_depth=5, min_samples_split=5;, score=0.233 total time=   0.8s\n",
      "[CV 4/5] END ..max_depth=5, min_samples_split=5;, score=0.243 total time=   0.8s\n",
      "[CV 5/5] END ..max_depth=5, min_samples_split=5;, score=0.239 total time=   0.8s\n",
      "[CV 1/5] END max_depth=15, min_samples_split=20;, score=0.313 total time=   1.9s\n",
      "[CV 2/5] END max_depth=15, min_samples_split=20;, score=0.314 total time=   1.9s\n",
      "[CV 3/5] END max_depth=15, min_samples_split=20;, score=0.331 total time=   1.9s\n",
      "[CV 4/5] END max_depth=15, min_samples_split=20;, score=0.322 total time=   1.9s\n",
      "[CV 5/5] END max_depth=15, min_samples_split=20;, score=0.311 total time=   1.9s\n",
      "[CV 1/5] END .max_depth=15, min_samples_split=5;, score=0.330 total time=   2.0s\n",
      "[CV 2/5] END .max_depth=15, min_samples_split=5;, score=0.372 total time=   2.0s\n",
      "[CV 3/5] END .max_depth=15, min_samples_split=5;, score=0.385 total time=   2.0s\n",
      "[CV 4/5] END .max_depth=15, min_samples_split=5;, score=0.329 total time=   2.0s\n",
      "[CV 5/5] END .max_depth=15, min_samples_split=5;, score=0.321 total time=   2.0s\n",
      "[CV 1/5] END .max_depth=10, min_samples_split=5;, score=0.287 total time=   1.5s\n",
      "[CV 2/5] END .max_depth=10, min_samples_split=5;, score=0.298 total time=   1.5s\n",
      "[CV 3/5] END .max_depth=10, min_samples_split=5;, score=0.307 total time=   1.5s\n",
      "[CV 4/5] END .max_depth=10, min_samples_split=5;, score=0.303 total time=   1.5s\n",
      "[CV 5/5] END .max_depth=10, min_samples_split=5;, score=0.294 total time=   1.5s\n",
      "[CV 1/5] END .max_depth=5, min_samples_split=10;, score=0.240 total time=   0.8s\n",
      "[CV 2/5] END .max_depth=5, min_samples_split=10;, score=0.235 total time=   0.8s\n",
      "[CV 3/5] END .max_depth=5, min_samples_split=10;, score=0.234 total time=   0.8s\n",
      "[CV 4/5] END .max_depth=5, min_samples_split=10;, score=0.243 total time=   0.8s\n",
      "[CV 5/5] END .max_depth=5, min_samples_split=10;, score=0.240 total time=   0.8s\n",
      "[CV 1/5] END max_depth=10, min_samples_split=10;, score=0.294 total time=   1.4s\n",
      "[CV 2/5] END max_depth=10, min_samples_split=10;, score=0.305 total time=   1.5s\n",
      "[CV 3/5] END max_depth=10, min_samples_split=10;, score=0.303 total time=   1.4s\n",
      "[CV 4/5] END max_depth=10, min_samples_split=10;, score=0.299 total time=   1.4s\n",
      "[CV 5/5] END max_depth=10, min_samples_split=10;, score=0.290 total time=   1.4s\n",
      "[CV 1/5] END max_depth=10, min_samples_split=20;, score=0.291 total time=   1.4s\n",
      "[CV 2/5] END max_depth=10, min_samples_split=20;, score=0.304 total time=   1.4s\n",
      "[CV 3/5] END max_depth=10, min_samples_split=20;, score=0.311 total time=   1.4s\n",
      "[CV 4/5] END max_depth=10, min_samples_split=20;, score=0.298 total time=   1.4s\n",
      "[CV 5/5] END max_depth=10, min_samples_split=20;, score=0.285 total time=   1.4s\n",
      "[CV 1/5] END .max_depth=5, min_samples_split=20;, score=0.240 total time=   0.8s\n",
      "[CV 2/5] END .max_depth=5, min_samples_split=20;, score=0.234 total time=   0.8s\n",
      "[CV 3/5] END .max_depth=5, min_samples_split=20;, score=0.234 total time=   0.8s\n",
      "[CV 4/5] END .max_depth=5, min_samples_split=20;, score=0.245 total time=   0.8s\n",
      "[CV 5/5] END .max_depth=5, min_samples_split=20;, score=0.239 total time=   0.8s\n"
     ]
    }
   ],
   "source": [
    "best_params_meanvar_all, best_estimator_meanvar_all = optimize_clf(train_X_meanvar, train_y, forest1, params, verbose=3)\n",
    "best_params_meanvar_khvg, best_estimator_meanvar_khvg = optimize_clf(train_X_meanvar_khvg, train_y, forest2, params, verbose=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "12ec4f27-8996-4285-9fcb-237854791c1b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best params\n",
      "All genes: {'min_samples_split': 5, 'max_depth': 20}\n",
      "khvg: {'min_samples_split': 5, 'max_depth': 20}\n"
     ]
    }
   ],
   "source": [
    "print(\"Best params\")\n",
    "print(f\"All genes: {best_params_meanvar_all}\")\n",
    "print(f\"khvg: {best_params_meanvar_khvg}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e141be16-c4cc-43f2-a070-0cd98d5c5373",
   "metadata": {},
   "source": [
    "Q: Number of samples required to split a node is less for khvg. Why do you think that could be?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2496232-2dfe-453d-a8c8-0bf18dc3df18",
   "metadata": {},
   "source": [
    "## Save models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "a08a4bed-8f8a-422e-b02a-4c9be7959dd9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_dir = \"models\"\n",
    "\n",
    "if not os.path.exists(model_dir):\n",
    "    os.mkdir(model_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "bb5729ff-625b-4fe6-89a5-53f1537bbea8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['models/meanvar_rf_best_khvg.pkl']"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import joblib\n",
    "joblib.dump(best_params_meanvar_all, os.path.join(model_dir, \"meanvar_rf_best_all.pkl\"))\n",
    "joblib.dump(best_params_meanvar_khvg, os.path.join(model_dir, \"meanvar_rf_best_khvg.pkl\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "501c45b7-a70d-40d6-a853-6392d890807f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.3.0\n"
     ]
    }
   ],
   "source": [
    "# Models are now \"portable\", always keep in mind the sklearn version, else could be issue with reproducibility in another machine\n",
    "import sklearn\n",
    "print(sklearn.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "3e84acc4-1bad-4537-85c2-2cfbcbd96543",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Make predictions\n",
    "\n",
    "preds_X_meanvar_all = best_estimator_meanvar_all.predict(test_X_meanvar)\n",
    "preds_X_meanvar_khvg = best_estimator_meanvar_khvg.predict(test_X_meanvar_khvg)\n",
    "# .predict() method returns numpy arrays, but dataframes are human readble (tables).\n",
    "\n",
    "preds_X_meanvar_all = pd.DataFrame(preds_X_meanvar_all, index=test_X.index, columns=[\"cell_type\"])\n",
    "preds_X_meanvar_khvg = pd.DataFrame(preds_X_meanvar_khvg, index=test_X.index, columns=[\"cell_type\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3baa89ba-2fe8-45ba-be4b-6c2e7a058b40",
   "metadata": {
    "tags": []
   },
   "source": [
    "## save predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "30c470da-5510-4731-913e-f374e8b4c13e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "preds_dir = \"models\"\n",
    "\n",
    "if not os.path.exists(preds_dir):\n",
    "    os.mkdir(preds_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "435957c2-9b7f-4f49-8376-44d4f4a521c1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "preds_X_meanvar_all.to_csv(os.path.join(preds_dir, \"preds_meanvar_rf_best_all.csv\"))\n",
    "preds_X_meanvar_khvg.to_csv(os.path.join(preds_dir, \"preds_meanvar_rf_best_khvg.csv\"))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9802bc25-5f44-4db7-8025-36e6f6d84056",
   "metadata": {},
   "source": [
    "## compute metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "1bb142e6-36a7-4c3d-b085-c456e707768e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import balanced_accuracy_score\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.metrics import confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "62b2c0e3-f891-40b6-bb62-3b5c29b2d600",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# create a metrics dir\n",
    "\n",
    "metrics_dir = \"metrics\"\n",
    "if not os.path.exists(metrics_dir):\n",
    "    os.mkdir(metrics_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "8b71ce67-8abd-4463-bffd-b85a259d427c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_scores(preds, true, classes, run_name=\"rf\", metrics_dir=\"metrics\"):\n",
    "    \"\"\"\n",
    "    Compute accuracy, balanced accuracy and confusion matrices\n",
    "    preds: model output\n",
    "    true: ground truth classes\n",
    "    classes: class labels (accessible from .classes_ method from the classifier)\n",
    "    run_name: Attach a readable run name in index of the accuracy dataframe for later use\n",
    "    \"\"\"\n",
    "    acc = accuracy_score(preds, true)\n",
    "    print(f\"Accuracy: {acc}\")\n",
    "    balanced_acc = balanced_accuracy_score(preds, true)\n",
    "    print(f\"Balanced accuracy: {balanced_acc}\")\n",
    "    print(pd.DataFrame([acc, balanced_acc]))\n",
    "    df_acc = pd.DataFrame([acc, balanced_acc], columns=[run_name], index=[\"accuracy\", \"balanced accuracy\"])\n",
    "    cm = confusion_matrix(preds, true)\n",
    "    df_cm = pd.DataFrame(cm, index=classes, columns=classes)\n",
    "    \n",
    "    df_acc.to_csv(os.path.join(metrics_dir, f\"scores_{run_name}.csv\"))\n",
    "    df_cm.to_csv(os.path.join(metrics_dir, f\"cm_{run_name}.csv\"))\n",
    "    return df_acc, df_cm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "b655af77-48f9-458d-89ed-0ce317e2fc35",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.5962627406568517\n",
      "Balanced accuracy: 0.8384054975340876\n",
      "          0\n",
      "0  0.596263\n",
      "1  0.838405\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.10/site-packages/sklearn/metrics/_classification.py:2394: UserWarning: y_pred contains classes not in y_true\n",
      "  warnings.warn(\"y_pred contains classes not in y_true\")\n"
     ]
    }
   ],
   "source": [
    "df_acc, df_cm = get_scores(preds_X_meanvar_all[\"cell_type\"], test_y, best_estimator_all.classes_, \"meanvar_rf_best_all\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "76cffe66-1030-4f7a-903d-c28344bee580",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.5062287655719139\n",
      "Balanced accuracy: 0.7212633695215118\n",
      "          0\n",
      "0  0.506229\n",
      "1  0.721263\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.10/site-packages/sklearn/metrics/_classification.py:2394: UserWarning: y_pred contains classes not in y_true\n",
      "  warnings.warn(\"y_pred contains classes not in y_true\")\n"
     ]
    }
   ],
   "source": [
    "_, _ = get_scores(preds_X_meanvar_khvg[\"cell_type\"], test_y, best_estimator_all.classes_, \"meanvar_rf_best_khvg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01ff2465-61af-4a7a-8ea6-302df0035da0",
   "metadata": {},
   "source": [
    "Q: why the user warning?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "dcc12b00-5b59-464c-8eed-c7bac5f5f6d1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## Plot the metrics\n",
    "\n",
    "dfs = []\n",
    "for run_name in [\"meanvar_rf_best_all\", \"meanvar_rf_best_khvg\"]:\n",
    "    dfs.append(pd.read_csv(os.path.join(metrics_dir, f\"scores_{run_name}.csv\"), index_col=0))\n",
    "df = pd.concat(dfs, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "8b9c0b37-7102-406c-abbe-1b84597acc73",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: >"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhcAAAGiCAYAAABUNuQTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABOm0lEQVR4nO3deVxUVf8H8M+wjIAwIqAoKooYaKig5oIiuKSoYJpbpiFtLoDm8uASmpryiBouqeW+ZRlqKuZWbolLLqUmZq4o7oqsgygMMPf3hz/naWLAO8OlGeTz7jWvV5xzz7nfmUb7crYrEwRBABEREZFEzIwdABEREb1amFwQERGRpJhcEBERkaSYXBAREZGkmFwQERGRpJhcEBERkaSYXBAREZGkmFwQERGRpCyMHUBxwusNMHYIRCZpxf3jxg6ByOQUqO6V+T3yU29I0o+lU31J+jFlJptcEBERmRR1obEjKDc4LUJERESS4sgFERGRGILa2BGUG0wuiIiIxFAzuRCL0yJEREQkKY5cEBERiSBwWkQ0JhdERERicFpENE6LEBERkaQ4ckFERCQGp0VEY3JBREQkBg/REo3TIkRERCQpjlwQERGJwWkR0ZhcEBERicHdIqIxuSAiIhKB51yIxzUXREREJCmOXBAREYnBaRHRmFwQERGJwWkR0TgtQkRERJLiyAUREZEYPERLNCYXREREYnBaRDROixAREZGkOHJBREQkBneLiMbkgoiISAxOi4jGaREiIiKSFEcuiIiIxOC0iGhMLoiIiEQQBG5FFYvJBRERkRhccyEa11wQERGRpDhyQUREJAbXXIjG5IKIiEgMTouIxmkRIiIikhRHLoiIiMTgg8tEY3JBREQkBqdFROO0CBEREUmKIxdERERicLeIaEwuiIiIxOC0iGicFiEiIiJJMbkgIiISQ62W5lUKN2/exEcffQQfHx/4+voiOjoaubm5L2339OlTxMbG4s0334S3tze6du2KxYsXQ6VSaV2Xn5+PefPmwc/PD97e3ggJCcHly5f1jpPTIkRERGIYec2FUqlEaGgoXFxcsGjRIqSnpyMmJgaZmZmIjY0tse306dNx4MABjB07Fq+99hoSExOxaNEiZGVlYcqUKZrrYmJiEB8fj0mTJqFWrVpYtWoV3n//fezcuRPVqlUTHSuTCyIiIhGM/VTUuLg4KJVKxMfHw8HBAQBgbm6OyMhIhIWFwd3dXWe7goIC/PTTT/j4448REhICAGjTpg3u37+PPXv2aJKLR48eIS4uDpMnT8aAAQMAAN7e3ujcuTPWr1+PyMhI0bFyWoSIiKgcOHLkCHx9fTWJBQAEBgZCLpcjISGh2HaCIKCwsBB2dnZa5QqFAoIgaH4+duwYCgsLERQUpCmztbVFp06dSuxfF45cEBERiSHRtEjnzp1LrD948KDO8qSkJPTt21erTC6Xw9XVFUlJScX2Z2lpiT59+mDDhg1o3rw5GjRogAsXLmDz5s147733tPp3cnKCvb29Vnt3d3fs3LkTarUaZmbixiSYXBAREYlh5K2oSqUSCoWiSLlCoUBWVlaJbadPn45p06ZppjsAICQkBCNHjtTq/5+jGwBQpUoV5Ofn4+nTp7C1tRUVK5MLIiKif1FxIxOGEgQBMpmsxGtiY2Nx+PBhzJw5E25ubrh48SIWLVoEhUKBTz75RHOdrn7+PnUiFpMLIiIiMYy8W0ShUECpVBYpz87OLnYxJwBcvXoVa9aswddff62ZkmnZsiVkMhnmzp2LwYMHw9HRsdj+lUolLC0tYWNjIzpWLugkIiISQ1BL8zKQu7t7kbUVKpUKt2/fLjG5uH79OgCgUaNGWuWNGjVCQUEB7t27p+k/LS0NmZmZWtclJSXBzc1N9HoLgMkFERFRueDv74+TJ08iIyNDU7Z//36oVCoEBAQU265WrVoAgIsXL2qV//nnnwCA2rVrAwD8/PxgZmaGvXv3aq7JycnBoUOHSuxfF06LEBERiWHkaZGBAwfi22+/RXh4OMLDw5GWlobZs2ejZ8+eWiMXUVFRiI+Px19//QUAaNy4MZo2bYpp06YhNTUVbm5uuHDhAr7++mv06NFDs7XV2dkZAwcORGxsLCwsLODi4oI1a9YAAEJDQ/WKlckFERGRGEbeLaJQKLB+/XpER0dj1KhRsLKyQnBwcJHDrdRqNQoL/3fgl7m5OZYtW4Yvv/wSK1euRGpqKmrWrIn33nsPI0aM0Go7adIk2NjYYOHChcjOzoa3tzfWr1+v1+mcACATDFkG+i8Irzfg5RcRVUAr7h83dghEJqdAda/M7/Hs5yWS9GMdOPLlF5VzHLkgIiISw8jTIuUJkwsiIiIxmFyIxuSCiIhIDCOvuShPuBWViIiIJMWRCyIiIjE4LSIakwsiIiIxOC0iGqdFiIiISFIcuSAiIhKD0yKiMbkgIiISg9MionFahIiIiCRlUHLxySef4MiRIzDRk8OJiIikp1ZL86oADJoWSUlJwbBhw+Ds7Iw+ffqgT58+qFOnjtSxERERmY4KkhhIwaCRi7i4OOzZswdBQUHYsmULAgMDERoail27dkGlUkkdIxEREZUjBq+5qF+/PiZMmICEhAQsWrQIlStXxqRJk+Dn54eZM2fi0qVLUsZJRERkXIIgzasCKPVuEXNzc3Tq1AkAkJGRgXPnzmHbtm3YuHEjWrRogZkzZ8LNza3UgRIRERkVp0VEK9VukRs3buCLL76Av78/xowZA0dHRyxfvhxnzpzBmjVr8PTpU4wfP16qWImIiIyHCzpFM2jk4ocffsAPP/yA8+fPo3bt2hgyZAj69OkDJycnzTW+vr749NNPERoaKlmwREREZPoMSi4+//xzdOnSBaNHj4avr2+x19WtWxfh4eEGB0dERGQyeIiWaAYlF0eOHEHVqlVfel316tUxcuRIQ25BRERkWirIlIYUDFpzkZubi4sXL+qsu3jxIh4+fFiqoIiIiKj8MmjkYvr06ahbty68vLyK1O3atQvJyclYunRpqYMjIiIyGRVkG6kUDBq5OH/+PNq0aaOzrnXr1vjjjz9KExMREZHp4W4R0QxKLp4+fQoLC92DHjKZDDk5OaUKioiIiMovg5ILd3d3HDhwQGfdwYMHeWgWERG9ejhyIZpBay5CQ0MxadIkmJmZoW/fvqhevTpSUlKwbds2bNmyBbNmzZI6TiIiIuPiVlTRDEouevfujdTUVHz11VfYtGmTptzKygr/+c9/8Pbbb0sWIBEREZUvBj9b5OOPP8bAgQNx7tw5ZGZmwt7eHs2aNYOtra2U8REREZkEQc3dImKV6sFltra2aN++vVSxEBERma4Ksl5CCqVKLm7duoXk5GTk5eUVqevatWtpuiYiIjItXHMhmkHJxZMnTzBy5EicOnUKACD8/8EiMplMc82lS5ckCI+IiIjKG4O2on7xxRd4/PgxvvvuOwiCgCVLlmDDhg3o168fateurbXIk4iI6JWgFqR5VQAGJRdHjx7FiBEj4O3tDeD5A8patmyJmTNn4s0338TatWslDZKIiMjoeM6FaAYlF+np6ahZsybMzc1hbW2NzMxMTZ2/vz+OHj0qVXxERERUzhiUXNSoUQMZGRkAgHr16uHQoUOaurNnz6JSpUrSREdERGQqOHIhmkELOtu1a4dff/0VXbp00ZzWmZiYCEtLSyQmJuKDDz6QOk4iIiLj4lNRRTMouYiMjMSzZ88APD+ts3Llyvjpp5+Ql5eHzz77DAMHDpQ0SCqdSjaV0DNyIJoH+aKyvS0eJt3HvqXxOLPzV1Htm3Z5A50/Dkbt1+vBzNwMaXcf45e1e3D8+4Na13m2a4Ke/3kHtRvVhepZHi4cOovtMd/iSZqyLN4WUalUrmyDGZ9PRP9+wXBwsMflK0mY+8USbN78Y4nthoQMwJrVC3TW1arjg0ePHmt+DurxJvr16wkfHy809GwAS0tLWMhrSfo+iEyR3smFSqXCkSNH0KhRIzg4OAAAunTpgi5dukgeHElj2LJI1PV2R/ycjUi5+QBvvOWHjxaPgUwmw+8/Hi+xbdewXngr8l0c/W4/fv56OwrzC+HsXgsWltpfnddaN8LIdZ/iz1/OYdnQubBzrILekwZj9HdTMeetSShQFZTlWyTS2w+bV+GNN7wRNTkG167dwMCBvbHx26UwMzNDXFz8S9t/+NFYXLlyXassLS1D6+devbqhdevm+OOPP5GXp8IbLbylfAv0b6sgUxpS0Du5kMvliIyMxKpVq+Dq6loWMZGEvDo0QyN/b6z55EtNInH1xEU41nJCn6gQnNn1a7FH2tZp7Ia3It/FjrkbsX/5/36bu/Lrn0WuffvTEDy6+QArw+ZBXfj8D2DanRREbouG74COOPrt/jJ4d0SG6d6tE7p0CcDgkHBs2rQDAHA44VfUda2NOTFTsHnzj1C/5H8kFy9expmziSVeM3zEeM05QF8ujGZyUd5VkG2kUjBoQWf9+vXx4MEDqWOhMuAd2Aq5T57h7O4TWuUnfjgM+xoOcPN5rdi2HUK7oUCVj8Pr9pZ4jyrOVVHPpwFObz+iSSwA4MbZq3iUdB8+XVuV7k0QSaxXr27Izn6CH37YpVW+7ptNqFWrJlq3ai7JfQTO0VMFZVByMW7cOCxduhR//ln0N1gyLS6edfDw+j2t/+kDwL1LtwAANT3rFNu2QatGeHj9Hpp1b4NpBxdiSVIc/ntiKXpNHARzS/O/3cP1//u8XaSPe5dvlXgPImNo7NUQly9fQ2FhoVb5hQvPTxb28vJ8aR874tcj79ltpDz8E1s2rxTVhso5QS3NqwIwaEFnbGwsMjMz0b9/f1StWhWOjo5a9TKZDD/+WPKiqM6dO5dY7wnHEutJnMr2tki9k1KkPCfzCQDAtqpdsW3tazjA1kGB/tM/wM55m/Dw+l14tm2MrmG9UbWmI9aNWay5BwDkZD0pep+sJ6hsX/w9iIzBwbEqbt64VaQ8PT0TAODoWLXYto8epWBWzJc4deoslMpsNG7cEBPGj8Txozvh36E3EhP/Kquwydg4LSKaQcmFl5cXGjduLHUsVFZKGJotadhWJjODtZ0NVo9aqNlZcvXERchtrND5oyDsXrAZj289EnEf/oEk01PSd7+kup/3HcbP+w5rfj567BT27D2IP84exPRpkejT90MpwyQTInBBp2gGJRezZ88u9Y0PHjxYYn14vQGlvgc9H6HQNXKgGW3ILDra8L+22ahSvSr+SjivVf7X4XPo/FEQ6jSuj8e3Hmn60HmfKrYl3oPIGNLTMnSOTjg42D+v//8RDLFu3bqL48dPS7ZWg6g4N2/eRHR0NM6cOQNra2sEBQUhMjISVlZWxba5e/dusbMFlpaWWkscPD2LTu85OTnh+PGSdxb+U6keuU6m7/7l23jjrXYwMzfTWnfh0vD5OokHV+4U2/be5VuoUl3H8PD/P/32RRZ//8ptTZ8XD5/TutSloWuJ9yAyhgt/XsLAd3rD3Nxca91F48YNAQAXL17Ru0+ZTPbSHSZUzhl5WkSpVCI0NBQuLi5YtGgR0tPTERMTg8zMTMTGxhbbrnr16kUeKCoIAoYOHYrWrVsXuT4kJATBwcGany0tLfWO1aDk4tNPP33pNTExMYZ0TRL7Y99p+A16E826t8aZXf/bMdKmbwAyH6bj5h/Xim17bu8pvO7vA68OPlrnYTTu2AzqQjVuJSYBALIeZeDmH9fQqnd7HFjxo2Zra71mr6GGey38smZPGb07IsPs2PEThn78Hvr0CcKWLf9bHzbkvf64d+8BTp0+q1d/9erVQdu2LXHwEJ+r9Eoz8mLMuLg4KJVKxMfHa86ZMjc3R2RkJMLCwuDu7q6znVwuh4+Pj1bZqVOnkJ2drZVEvFCzZs0i1+vLoORC1y6RzMxMpKamwt7eHk5OTqUKiqTz1+E/cOnIeQyMHgorWxs8vvUQb/RsB68OzbB29CJNIvDenBFo3TcA0wJGIf1eKgDgxJbDaD+oCwZGfwxbBwUeXLuLhn5N4B8SiCMbftZcBwDxs7/DJxumYOjX45CwYR/sHBXoPXEw7l2+jRNbfjHKeycqzk8//4L9+xPw1eJZUNjZIikpGe+80wvdunVCSOhIzQjEiuWxGBLSHx4N2+L27XsAgJ/3xuHosZNIvHAJ2conaNy4ISL/EwZBEDBt+hda93F1rYU33vABALjXrwsA6NMnCABwK/nOS8/JIPq7I0eOwNfXV5NYAEBgYCCioqKQkJBQbHKhy65du2Bra4tOnTqVRaiGJRc7d+7UWX716lWMHz8eUVFRpQqKpLViRCzeGv8ugscNgE0VWzy6cU9rkSYAyMzNYG5hrpnyAAB1QSEWvTcTvSYMQmDE26hcxRZpd1KwY85GHFylfT7AtZN/4asPYhA87h2ErZ4I1bM8/HnoLLbN2sDTOckk9RvwMWbOmIjp0yI1x38Pei9M6/hvc3NzWFhYQPa3Pxd//nkZ/fu/hXFjR8Da2gopKan45fCv+O+shbh27YbWPToEtCtyVPjmuBUAgPXfbMZHH48tw3dIkpNoWuRluyWLW5OYlJSEvn37apXJ5XK4uroiKSlJ9P3z8/Oxb98+dOnSReeDRlesWIH58+fD2toafn5+mDBhAlxcXET3DwAyQeJTXnbt2oWVK1dix44dpeqHCzqJdFtxX7+FVUQVQYHqXpnfI2f6u5L089bRoscD/F1xyYWXlxdGjx6NYcOGaZW/++67cHR0xJIlS0Td/+DBgwgPD8fq1avh5+enVTdx4kR06NABTk5OuHr1KpYuXQoLCwvs2LEDVapUEdU/UAYLOu3s7HD7dtHDlIiIiOjluyX1JQiC1ujay+zcuRNOTk7w9fUtUjdnzhzNv7ds2RItWrRAnz59sHnzZgwdOlT0PQxKLjIzM4uU5efnIykpCfPnz8drrxV/pDQREVG5ZOTdIgqFAkpl0adMZ2dni15vkZOTg8OHD6Nfv34wNzd/6fUNGzaEm5sbLl68qFesBiUXbdq00ZklCYKAmjVr4quvvjKkWyIiItNl5N0i7u7uRdZWqFQq3L59u8hajOLs378fz549Q8+ePUXf15DVEwYlF7NmzSqSXFSqVAnOzs7w9vaGhQWPzyAiIpKSv78/li5dioyMDFSt+vwMov3790OlUiEgIEBUH7t27YKrqyu8vcU9offSpUtITk4Wnby8YFAW0KdPH0OaERERlV9GnhYZOHAgvv32W4SHhyM8PBxpaWmYPXs2evbsqTUtEhUVhfj4ePz1l/ZzbtLT03HixIli106sXr0ad+7cQatWreDg4IBr165h2bJlqFGjBvr3769XrAYlF5cvX8ajR490ZkoJCQlwdnZGw4YNDemaiIjIJBn72SIKhQLr169HdHQ0Ro0aBSsrKwQHByMyMlLrOrVaXeSJvwCwd+9eFBQUFDsl4ubmhn379mHPnj3IyclB1apVERAQgDFjxkChUOgVq0FbUYcMGYLmzZtjzJgxReoWL16Ms2fPYu3atfp2q4VbUYl041ZUoqL+ja2oTyZKM2pvO2ebJP2YMjNDGl2+fBnNm+t+QI+Pj0+RoRgiIiKqOAyaFlGpVMjPzy+2Li8vr1RBERERmRwjr7koTwwauWjUqFGxJ3Du2LGD6y2IiOjVI6ileVUABo1cDB8+HGFhYRg2bBj69OmD6tWrIyUlBdu2bcOxY8fw9ddfSx0nERERlRMGJRcdOnTAvHnzMHfuXIwZMwYymQyCIKBGjRqIjY1Fhw4dJA6TiIjIyDgtIprBp1316NEDPXr0wI0bN5CZmQl7e3vUr19fytiIiIhMhsDkQrRSH6XJhIKIiIj+zqAFnQsWLMDUqVN11k2dOhVffvllqYIiIiIyOWpBmlcFYFBysWvXrmLPuWjRogV2795dqqCIiIhMjlotzasCMCi5SElJQc2aNXXW1ahRAw8fPixVUERERFR+GZRcvHigiS7Xrl1DlSpVShUUERGRyeG0iGgGJRdvvvkmFi9ejMTERK3yxMREfPXVV+jSpYskwREREZkMJheiGbRbZMyYMTh79izeeecduLu7aw7RSkpKQqNGjTB27Fip4yQiIjIqA57zWWEZlFzY2dlh06ZNiI+Px8mTJ5GZmQkPDw+EhoaiV69ekMvlUsdJRERE5YTB51zI5XIMGDAAAwbw0ehERFQBVJApDSkYnFzs2LEDcXFxSE5O1vkU1LNnz5YqMCIiIpPC5EI0gxZ07tixA1OmTMFrr72GjIwMdO/eHYGBgbC0tISjoyM+/PBDqeMkIiKicsKgkYu1a9ciPDwcw4YNw+bNmzFo0CB4eXnhyZMn+Oijj1C5cmWp4yQiIjIqPltEPINGLm7duoXmzZvD3Nwc5ubmePLkCQDA1tYWQ4cOxYYNGyQNkoiIyOi4FVU0g5ILW1tbqFQqAICzszOuX7+uqSssLERGRoY00REREVG5Y9C0SOPGjXHlyhW0b98enTp1wldffQVBEGBhYYEVK1bA29tb6jiJiIiMq2I8FkQSBiUXw4cPx/379wEAn3zyCe7du4eYmBgUFhaiSZMmmDlzpqRBEhERGRvXXIhnUHLh4+MDHx8fAIBCocDSpUuhUqmgUqlga2srZXxERESmgcmFaAafc/FPcrmcJ3MSERGRdMkFERHRK41rLkRjckFERCQC11yIZ9BWVCIiIqLicOSCiIhIDE6LiMbkgoiISAROi4jHaREiIiKSFEcuiIiIxOC0iGhMLoiIiEQQmFyIxmkRIiIikhRHLoiIiMTgyIVoTC6IiIhE4LSIeEwuiIiIxGByIRrXXBAREZGkOHJBREQkAqdFxGNyQUREJAKTC/E4LUJERESS4sgFERGRCBy5EI/JBRERkRiCzNgRlBucFiEiIiJJceSCiIhIBFOYFrl58yaio6Nx5swZWFtbIygoCJGRkbCysiq2zd27d9G5c2eddZaWlvjzzz81P+fn52PRokXYvn07srOz0bRpU0yePBkNGzbUK04mF0RERCIIauNOiyiVSoSGhsLFxQWLFi1Ceno6YmJikJmZidjY2GLbVa9eHZs2bdIqEwQBQ4cORevWrbXKY2JiEB8fj0mTJqFWrVpYtWoV3n//fezcuRPVqlUTHSuTCyIionIgLi4OSqUS8fHxcHBwAACYm5sjMjISYWFhcHd319lOLpfDx8dHq+zUqVPIzs5GcHCwpuzRo0eIi4vD5MmTMWDAAACAt7c3OnfujPXr1yMyMlJ0rFxzQUREJIKgluZlqCNHjsDX11eTWABAYGAg5HI5EhIS9Opr165dsLW1RadOnTRlx44dQ2FhIYKCgjRlL67Rt3+OXBAREYkgSLRbpLj1Dy8cPHhQZ3lSUhL69u2rVSaXy+Hq6oqkpCTR98/Pz8e+ffvQpUsXVKpUSat/Jycn2Nvba13v7u6OnTt3Qq1Ww8xM3JgEkwsiIiIRjL2gU6lUQqFQFClXKBTIysoS3c+RI0eQmZmpNSXyon87O7si11epUgX5+fl4+vQpbG1tRd2DyQUREdG/qLiRCUMJggCZTPyoys6dO+Hk5ARfX98idbr6EQRB75i45oKIiEgEQS2T5GUohUIBpVJZpDw7O1vniIYuOTk5OHz4MLp37w5zc3NR/SuVSlhaWsLGxkZ0rEwuiIiIRBAEaV6Gcnd3L7K2QqVS4fbt28XuFPmn/fv349mzZ+jZs6fO/tPS0pCZmalVnpSUBDc3N9HrLQAmF0REROWCv78/Tp48iYyMDE3Z/v37oVKpEBAQIKqPXbt2wdXVFd7e3kXq/Pz8YGZmhr1792rKcnJycOjQIdH9v8DkgoiISARjT4sMHDgQdnZ2CA8Px9GjRxEfH4+ZM2eiZ8+eWiMXUVFReP3114u0T09Px4kTJ7S2mv6ds7MzBg4ciNjYWGzZsgXHjx/HJ598AgAIDQ3VK1Yu6CQiIhLB2Cd0KhQKrF+/HtHR0Rg1ahSsrKwQHBxc5HArtVqNwsLCIu337t2LgoICnVMiL0yaNAk2NjZYuHAhsrOz4e3tjfXr1+t1OicAyARDloH+C8LrDTB2CEQmacX948YOgcjkFKjulfk9kn26SNJPvT/2S9KPKePIBRERkQim+au4aWJyQUREJIKxp0XKEy7oJCIiIklx5IKIiEgEqZ4tUhEwuSAiIhLB2M8WKU+YXBAREYmg5siFaFxzQURERJLiyAUREZEIXHMhHpMLIiIiEbgVVTxOixAREZGkOHJBREQkAk/oFI/JBRERkQicFhGP0yJEREQkKY5cEBERicBzLsRjckFERCQCt6KKx2kRIiIikhRHLoiIiETgbhHxmFwQERGJwDUX4jG5ICIiEoFrLsTjmgsiIiKSFEcuiIiIROCaC/GYXBAREYnANRficVqEiIiIJGWyIxdf/j7b2CEQmaRZER8ZOwSiCokLOsUz2eSCiIjIlHBaRDxOixAREZGkOHJBREQkAjeLiMfkgoiISAROi4jHaREiIiKSFEcuiIiIROBuEfGYXBAREYmgNnYA5QiTCyIiIhEEcORCLK65ICIiIklx5IKIiEgENfeiisbkgoiISAQ1p0VE47QIERERSYojF0RERCJwQad4TC6IiIhE4FZU8TgtQkRERJLiyAUREZEInBYRj8kFERGRCJwWEY/TIkRERCQpjlwQERGJwJEL8ZhcEBERiWAKay5u3ryJ6OhonDlzBtbW1ggKCkJkZCSsrKxe2jYzMxMLFy7EgQMHkJWVBRcXF3zwwQcYOHCg5hpPT88i7ZycnHD8+HG94mRyQUREJILayLmFUqlEaGgoXFxcsGjRIqSnpyMmJgaZmZmIjY0tsW1OTg5CQkJQqVIlREVFwdHREbdu3UJ+fn6Ra0NCQhAcHKz52dLSUu9Y9U4u+vTpg/79+yM4OBh2dnZ635CIiIj0FxcXB6VSifj4eDg4OAAAzM3NERkZibCwMLi7uxfbdvny5cjNzcWWLVs0oxytW7fWeW3NmjXh4+NTqlj1XtDp5uaG2bNno3379pgwYQJOnz5dqgCIiIjKAzVkkrwMdeTIEfj6+moSCwAIDAyEXC5HQkJCiW23bt2Kfv36iZo+kYLeIxfz5s1DdnY2fvzxR2zbtg1DhgxBnTp10K9fP/Tu3RvOzs5lEScREZFRSfVQ1M6dO5dYf/DgQZ3lSUlJ6Nu3r1aZXC6Hq6srkpKSiu3vzp07SE1NhUKhwPDhw3H8+HFUrlwZPXr0wMSJE4skHCtWrMD8+fNhbW0NPz8/TJgwAS4uLiLf3XMGbUW1s7PD4MGDsXXrVsTHx6NDhw5Yu3YtOnXqhBEjRuDAgQNQq7muloiISCpKpRIKhaJIuUKhQFZWVrHtUlNTAQBz586Fg4MDVq5ciZEjRyI+Ph7R0dFa1/bu3RvTp0/HunXrMHbsWPz+++8YNGhQif3rUuoFnTVq1EDt2rXh5OSEzMxMJCcnY9SoUahVqxZiY2NLPW9DRERkCqT6lbm4kQlDCYIAmaz46ZYXv+y7u7sjJiYGAODr64uCggLMnTsXo0ePRrVq1QAAc+bM0bRr2bIlWrRogT59+mDz5s0YOnSo6JgMPkTr6NGjGDNmDPz9/bFs2TK0a9cOu3fvxk8//YSff/4ZdevWRVRUlKHdExERmRS1TCbJy1AKhQJKpbJIeXZ2ts4RjRfs7e0BAG3atNEqb9OmDdRqdYlTKg0bNoSbmxsuXryoV6x6j1x8+eWX2L59Ox49eoRWrVohJiYGXbp0gVwu11zj6uqKkSNHYtCgQfp2T0RERDq4u7sXSQRUKhVu375dZC3G39WpU0fndlJBeL6KxMys5HGGF9fpQ+/kYsuWLXj77bfRv39/uLq6Fnudm5sbZs2apXdAREREpkiqBZ2G8vf3x9KlS5GRkYGqVasCAPbv3w+VSoWAgIBi28nlcrRr1w4nTpzQKj9x4gQsLCzQoEGDYtteunQJycnJJSYvuuidXBw+fBgWFi9vZm9vj7ffflvf7omIiEySsbcpDBw4EN9++y3Cw8MRHh6OtLQ0zJ49Gz179tQ64yIqKgrx8fH466+/NGUREREYNGgQJkyYgLfeegvXr1/H4sWLMXjwYM3W1tWrV+POnTto1aoVHBwccO3aNSxbtgw1atRA//799YpV7+Ti+vXrePTokc4sKSEhAc7OzmjYsKG+3RIREVEJFAoF1q9fj+joaIwaNQpWVlYIDg5GZGSk1nVqtRqFhYVaZU2bNsXy5csxb948jBgxAvb29njvvfcwevRozTVubm7Yt28f9uzZg5ycHFStWhUBAQEYM2ZMiWs6dJEJek6mDBkyBM2bN8eYMWOK1C1evBhnz57F2rVr9QpCl/zUG6Xug+hVlBPxkbFDIDI59pt+KfN7fO8yWJJ+3r3/nST9mDK9d4tcvnwZzZs311nn4+OjNQxDRET0qjD2CZ3lid7TIiqVSueDTl7U5eXllTooIiIiU2PsBZ3lid4jF40aNcKOHTt01u3YsYPrLYiIiCo4vUcuhg8fjrCwMAwbNgx9+vRB9erVkZKSgm3btuHYsWP4+uuvyyJOIiIiozL2I9fLE72Tiw4dOmDevHmYO3cuxowZA5lMBkEQUKNGDcTGxqJDhw5lECYREZFxGXsranli0LNFevTogR49euDGjRvIzMyEvb096tevL3VsREREVA6V6sFlTCiIiKii4IJO8QxKLtRqNU6ePImbN29CpVJp1clkMrz//vtSxEZERGQyuOZCPL2Ti8ePHyMkJATJycma9RYAtB73yuSCiIio4tJ7K+rs2bNhb2+PhIQECIKAzZs349ChQxg9ejTq1q2Ln3/+uSziJCIiMiq1RK+KQO/k4rfffsOHH36IatWqacpcXFwwYsQI9OrVCzNmzJA0QCIiIlPA5EI8vZOL7OxsODg4wMzMDLa2tkhLS9PU+fj44MyZM5IGSEREROWL3slF7dq1kZKSAgBo0KCB1mmdBw4cgL29vWTBERERmQpBJs2rItB7QWdAQACOHz+OHj16ICwsDBEREfD19YWFhQVSU1OLPPqViIjoVVBRpjSkoHdy8ffkISAgAN9//z0OHDiA3NxctG3bFgEBAZIGSEREZAqYXIinV3KhUqnw7bffol27dvD09AQANGnSBE2aNCmT4IiIiKj80WvNhVwux5dffgmlUllW8RAREZkkQaJXRWDQI9evX79eFrEQERGZLLVMmldFoHdyERUVhXXr1mHfvn3Izc0ti5iIiIioHNN7QWdoaCjy8/MxevRoAICVlZXW0d8ymYxnXRAR0SuHCzrF0zu5+PDDD7WSCSIiooqAyYV4eicXo0aNKos4iIiI6BVh0CPXiYiIKpqKstNDCnonF0OGDHnpNd98841BwRAREZmqirLTQwp6JxfW1tZF1lxkZmbiypUrUCgUaNSokWTBERERUfmjd3KxfPlyneVpaWkICwtDr169Sh0UERGRqeGCTvH0PueiOI6Ojhg6dCgWLlwoVZdEREQmgyd0iifpgk61Wo3Hjx9L2SUREZFJUFeY1KD09E4uLl68WKQsPz8fSUlJ+Oqrr9C0aVNJAiMiIqLySe/kom/fvkUWdArC82zO29sbM2fOlCYyIiIiE8I1F+LpnVzo2mZaqVIl1KhRA87OzpIERUREZGo4KSKe3slFq1atyiIOIiIiekXovVvkxIkT2Lp1q866bdu24eTJk6UOioiIyNSoJXpVBHonFwsXLkRaWprOuvT0dG5FJSKiV5JaJs2rItA7ubh27RoaN26ss87LywvXr18vdVBERERUfum95kImkyE7O1tnXVZWFgoLC0sdFBERkanhORfi6T1y4e3tje+++06z/fQFQRCwceNGeHt7SxYcERGRqeAJneLpPXIxatQoDBkyBG+99RbefvttVKtWDSkpKYiPj0dycjI2bNhQFnESERFROaF3ctGsWTOsW7cOX3zxBWJjY6FWq2FmZgYfHx+sW7cOPj4+ZRAmERGRcVWUnR5SMOjZIi1atEBcXBxyc3ORlZUFhUIBa2trqWMjIiIyGVxzIV6pHlxmZWUFKysrqWIhIiIyWUwtxNN7QWdUVBRGjx6ts27cuHH47LPPSh0UERERlV96JxfHjx9HYGCgzrouXbrg2LFjpQ6KiIjI1JjCCZ03b97ERx99BB8fH/j6+iI6Ohq5ubmi2mZmZmL69Onw8/NDkyZNEBgYiLi4OK1r8vPzMW/ePPj5+cHb2xshISG4fPmy3nHqPS2Snp6OqlWr6qyzt7dHamqq3kEQERGZOmOvuVAqlQgNDYWLiwsWLVqE9PR0xMTEIDMzE7GxsSW2zcnJQUhICCpVqoSoqCg4Ojri1q1byM/P17ouJiYG8fHxmDRpEmrVqoVVq1bh/fffx86dO1GtWjXRseqdXDg7OyMxMRG+vr5F6hITE/W6OREREYkTFxcHpVKJ+Ph4ODg4AADMzc0RGRmJsLAwuLu7F9t2+fLlyM3NxZYtWzRrJVu3bq11zaNHjxAXF4fJkydjwIABAJ6fbdW5c2esX78ekZGRomPVe1okKCgIy5Ytw549e7TK9+7di2XLlqFnz576dklERGTyjH2I1pEjR+Dr66tJLAAgMDAQcrkcCQkJJbbdunUr+vXrV+ImjGPHjqGwsBBBQUGaMltbW3Tq1Oml/f+T3iMXERERuHz5MsaNG4fJkyejevXqSElJQW5uLvz9/REREaFvl0RERCZPqnMuOnfuXGL9wYMHdZYnJSWhb9++WmVyuRyurq5ISkoqtr87d+4gNTUVCoUCw4cPx/Hjx1G5cmX06NEDEydO1CQcSUlJcHJygr29vVZ7d3d37Ny5U3OulRh6JxdyuRzLly/H8ePHcfLkSWRmZsLe3h5t27bVOVVCREREpadUKqFQKIqUKxQKZGVlFdvuxVrIuXPnolu3bli5ciWuX7+O+fPnIz8/H9HR0Zr+7ezsirSvUqUK8vPz8fTpU9ja2oqK1eBzLtq1a4d27doZ2pyIiKhcESRa0FncyIShBEGATFb8s9zV6udjLu7u7oiJiQEA+Pr6oqCgAHPnzsXo0aM16yV19fPPZ4mJUapDtJ49e4a8vLwi5f8cUiEiIirvjH38t0KhgFKpLFKenZ1d4mLOF/9PbtOmjVZ5mzZtoFarkZSUhGrVqhXbv1KphKWlJWxsbETHqndyIQgCli5diri4ODx+/FjnNZcuXdK3WyIiIiqBu7t7kbUVKpUKt2/fLrIW4+/q1KkDS0vLIuUvRiRerKNwd3dHWlqaZrnDC0lJSXBzcxO93gIwYLfIunXrsHbtWgwePBiCIGDEiBGIiIhAvXr1UKtWLcycOVPfLomIiEyeGoIkL0P5+/vj5MmTyMjI0JTt378fKpUKAQEBxbaTy+Vo164dTpw4oVV+4sQJWFhYoEGDBgAAPz8/mJmZYe/evZprcnJycOjQoRL710Xv5OKHH37AqFGj8PHHHwMA3nzzTYwcORK7d++Gu7s7bt++rW+XREREJs/YW1EHDhwIOzs7hIeH4+jRo4iPj8fMmTPRs2dPrWmRqKgovP7661ptIyIicOXKFUyYMAHHjh3DunXrsHjxYgwePFiztdXZ2RkDBw5EbGwstmzZguPHj+OTTz4BAISGhuoVq97TIvfu3UOjRo1gbm4OCwsLzfyMmZkZBg0ahMmTJ2PcuHH6dktl6OnTZ1i0Yj1+PnQUWdnZcHOtg49C+qPHmx1e2vb0mfNY8c0mXL1+A8/y8lDbpQb69uyGd/sEw9zcXGeb3Lw89A2NwK079/CfiI/wwaB+Er8jIglUsoL1wI9g2aYDZLYKqO/fRu6Ojcj/9ZcSm9lOXQALL59i67OG9YGQlQFY26BSt7dh2eQNmNVyhczKGuqUB1Ad3Y+8vVuBf5yMSKbP2Cd0KhQKrF+/HtHR0Rg1ahSsrKwQHBxc5HArtVqNwsJCrbKmTZti+fLlmDdvHkaMGAF7e3u89957RZ4VNmnSJNjY2GDhwoXIzs6Gt7c31q9fr/cBmXonF/b29nj69CkAwMXFBX/99ZdmC2pGRoboM87p3zM6aiYuXr6GMSM+QL06tbB7/2FMmDYHglpAUNeOxbY78ds5DB83BS28G2P6pNGwtrLCL8dOYvbCZbhz7wE+HTNCZ7vFK7/Bs2f8HpBpq/yfGTB3b4jc71dC/eAOLNt1RuXRU5EjM0P+8eJX8z9dvRCyfyxsk8mtUDlqDgpvXH2eWAAwc6qOSj36QXVkPwp2bwFyn8G8URNY9X8fFk3fQE60+NMOiV5wc3PD6tWrS7xm9uzZmD17dpFyMbs85XI5IiMj9TqNUxe9k4vmzZvjwoULCAgIQHBwMJYsWYLU1FRYWFhg8+bNPOvCxBz59TRO/HYOc6dPRI8uHQAArVp44/7DR5j31Wp06+xf7AhE/J79sLAwx1dffA4b6+eHrPi2bIbk23exY89+ncnFhb+uYOMPP2LOtAkYN2VWmb0votKw8GkNS++WyPlyJvJ/PQQAKLj4B8ycnGH93vDnoxeC7r0B6nu3ipRZ+gdCZmEJ1aH/nVysTnkI5ch3gbz/JdoFF88BubmwDgmDuWdjFF75U+J3RmXJ2LtFyhO911yMHDkSLVq0AACMGDECffv2xa5du7Bp0yb4+vpi+vTpUsdIpXDwyAnYWFuja8f2WuVvB3VFSmoaEv+6UmxbSwsLWFpYwKqSXKvcztYWcrm8yPX5+fn4bNYCvNunJ7waekjzBojKgGUrPwjPniL/5GGtctXhn2DmUA3mrzXSq79KnbpDePYUqv9PVAA8Tyryio7gFSQ9f8KkmWN1veMm4xIk+qci0Du5qF+/vmZ0Qi6XY8qUKTh69ChOnz6NhQsXwtHRUfIgyXDXbySjfr06sLDQHp3wcHfT1BdnQO8eyC8oQMyCZUh5nAZl9hP8+NNBHDzyKz4cXHQdxdK1G/EsNxcjhw6R9D0QSc28jhsK790C1Nq/ixbevqGpF8usRi1YNPJ+nljoSCb+ydKr2fN73U0WHzBROVOqQ7RK42Vnq/+0aeW/FMmrLVOZjdouNYqUV1E8P+I1Myu72LZNvRpi9aLZ+M+UWfh+204AgLm5GUYP/wDvv6u9p/ry1SSs+e4HfDV3OmysrZCRWfxRtETGJrNVQJ3yoEi58OT5AnWZXdEjlosj79gDAKD6Zc9LrgTMXOuj0lsDoTp9BOr/T2So/OC0iHhGSy7o31PSsbAl1V28fA2jP52Jpq97YuqEUbCxssKpM+exeOV6qFQqjPhgEACgoKAQn8UsQLfO/mjXuoXk8ROViZKONBZ73LGZGeQBgSi8cxOF10o+PNCsmjNsJ86COi0Fz5bF6hEomYqKMqUhBaMlFy87Wz0/lVm9FOwVdsjMKnqca5by+YhFFUXxD6H577yv4Ohgjy9jPtMs+mzVwhsyMxm+XvMdgrp2RJ1aNbFh83bcvf8Q82ZGQZn9BADwJOf5jiKVKh/K7CeobGNd7MJRon+b8ESpc3RCZqv4//riR/T+zqJZG5hVdcSzH78v8TqZkzMqT10AobAQT2b+B0KOuP6Jyiu911xQ+fKaez3cSL6DggLtPc9Xb9wEADSoX6/Ytpev34CX52tFkoLGjTygVqtx49YdAMD1G7eQ/SQHPd75CG279Ufbbv3RNzQcwPNtqW279ce1pGTp3hRRKRXevgnzWnWBfxxnbO76fK1F4Z2bovqp1LEHhHwVVEf2F3uNzMkZttMWQAYZnswYByE91fDAyajUEr0qAk6LvOI6+7fFDz/+hP2Hj6H7m/87vvXHvQdR3ckRTV/3LLZtdSdHXLx8DYWFhVoJxvk/nw//OldzAgB8FDIAvXp00Wqbmp6OCdPmYEDvHujWOQCutV2kfFtEpZL/21FUejMYlq0DkH/if4dmyf0DoU5//NIpDgCQVakKi2atkX/6qGatRpFrHKs/TyzMzPDk87EQUh9J9h7o36c24OmgFRWTi1dce9+W8G3ZDDNjlyDn6VO41nLBngOHcezk75g9dbwmafgsZgF+3HsAezevgUsNZwBAyIDeiFm4DCMnTEf/3j1gVakSTp35A+u/34Y2bzRDw9fqAwDq162D+nXraN333oPnf4nWqVUTrZo3/RffMdHLFfxxGvnnf4P1x2Mgs7aB+tE9WLbtDMtmrZGz+L+aMy6sh4+HPCAQyk8GF0kM5AGBkFlYQHVot857yBT2sJ06H2b2jni6/AvIqtjDvIq9pl6d9pijGPTKEpVcdOrUqcSFf/8k9bPqqXS+nPUZvlyxHktWbUCWMhtudetg7ucTtY7/VheqUVio1lrHNrh/L1Sv5oQNm7Zj2uyFyM1ToVYNZ4R9OBgh77z9778RIgnlzJsK64Efw2rAB5DZ2kF9/w5yvpyhdfy3zMwMMnNzyGRFnwkh79gdhSkPUHDhjM7+zWvXhXmNWgCAyqMmF6nP3bIOuT+sl+z9UNnjuIV4MkF4+TjPjBkztJKLAwcOQKlUok2bNnByckJqaipOnjyJKlWq4M0338SUKVNKHRgXdBLplhPxkbFDIDI59ptKfiaMFAbVleaXqo23tkvSjykTNXIxdepUzb+vXr0aNWrUwM6dO6FQ/G+1dVZWFoYNGwZnZ2fpoyQiIjIybkUVT+/dIt988w2GDx+ulVgAQJUqVTBs2DBs2LBBsuCIiIio/NF7QWdWVhays3Xv0c7OztY8gp2IiOhVUlG2kUpB75GLNm3aIDY2FqdPn9YqP3XqFObNm4c2bdpIFhwREZGpUEOQ5FUR6D1yMWPGDISFhSE0NBR2dnaoWrUqMjIykJ2djUaNGuHzzz8viziJiIionNA7uahevTq2bt2KI0eOIDExEY8fP0a1atXQtGlT+Pv7l0WMRERERscFneIZfIiWv78/kwkiIqowuOZCPIOTiyNHjuDChQt4+PAhwsLC4OLigt9++w2urq7cjkpERFSB6Z1cpKenIzw8HOfPn0e1atXw+PFjDBw4EC4uLti6dSusra0xbdq0soiViIjIaEScOUn/T+/dIv/973+RkZGBnTt34tChQ1oftq+vL06cOCFpgERERKaAu0XE0zu5SEhIwJgxY9CgQYMizxupWbMmHj3iU/+IiIgqMr2nRQoLC2FjY6OzTqlUwtLSstRBERERmRou6BRP75GLpk2bYuvWrTrrdu/ejebNm5c6KCIiIlMjSPRPRaD3yMWYMWMwZMgQDB48GIGBgZDJZDhw4ACWL1+OhIQEbNy4sSziJCIiMqqKsl5CCnqPXDRr1gzffPMNZDIZ5syZA0EQsGzZMjx+/Bjr1q2Dl5dXWcRJRERE5YRB51w0a9YM3377LXJzc5GVlQWFQgFra2upYyMiIjIZ3IoqnsGHaAGAlZUVrKyspIqFiIjIZHFBp3h6T4tERUVh9OjROuvGjRuHzz77rNRBERERUfmld3Jx/PhxBAYG6qzr0qULjh07VuqgiIiITA13i4hn0PHfVatW1Vlnb2+P1NTUUgdFRERkarhbRDy9Ry6cnZ2RmJiosy4xMRHVqlUrdVBERERUfumdXAQFBWHZsmXYs2ePVvnevXuxbNky9OzZU7LgiIiITIUgCJK8KgK9p0UiIiJw+fJljBs3DpMnT0b16tWRkpKC3Nxc+Pv7IyIioiziJCIiMipOi4ind3Ihl8uxfPlyHD9+HCdPnkRmZibs7e3Rtm1b+Pr6lkWMREREVI4YfM5Fu3bt0K5dOyljISIiMlkVZaeHFAxOLh49eoRHjx4hLy+vSF3Lli1LFRQREZGpUVeQ9RJS0Du5uHPnDsaPH4/z588DKHocqkwmw6VLl6SJjoiIyEQwtRBP7+RiypQpePDgAWbMmIEGDRrA0tKyLOIiIiKickrv5CIxMRFz5sxB165dyyIeIiIik8TdIuLpnVw4OzvDzEzv4zGIiIjKNSYX4umdJYwZMwYrV65EZmZmGYRDRERE5Z3eIxfx8fF4+PAhOnXqhEaNGsHOzk6rXiaTYenSpZIFSEREZAoqyumaUtA7ucjJyYGrq6vWz0RERK86U5gWuXnzJqKjo3HmzBlYW1sjKCgIkZGRsLKyKrFdSEgITp8+XaR8z549cHd31/zs6elZ5BonJyccP35crzj1Ti42bNigbxMiIiIqJaVSidDQULi4uGDRokVIT09HTEwMMjMzERsb+9L2zZs3x8SJE7XKateuXeS6kJAQBAcHa342ZFeowYdoERERVSTGPqEzLi4OSqUS8fHxcHBwAACYm5sjMjISYWFhWiMQuigUCvj4+Lz0PjVr1hR1XUkMSi7UajVOnjyJmzdvQqVSFan/4IMPShUUERGRqTH2mosjR47A19dXk1gAQGBgIKKiopCQkPDS5OLfpHdy8fjxY4SEhCA5ORkymUzzYctkMs01TC6IiIh069y5c4n1Bw8e1FmelJSEvn37apXJ5XK4uroiKSnppfc9ffo0fHx8UFhYCG9vb4wePVrn4zpWrFiB+fPnw9raGn5+fpgwYQJcXFxe2v/f6Z1czJ49G/b29khISEBAQAA2b94MJycn/Pjjj4iPj8eKFSv07ZKIiMjkGXtBp1KphEKhKFKuUCiQlZVVYtuWLVuiV69eqFevHlJSUrB69Wp88MEH2LBhA5o1a6a5rnfv3ujQoQOcnJxw9epVLF26FIMGDcKOHTtQpUoV0bHqnVz89ttvmDJlCqpVq6Ypc3FxwYgRIyAIAmbMmIFVq1bp2y0REZFJk2papLiRCUMJgqA1e6DLJ598ovVzhw4dEBwcjK+//horV67UlM+ZM0fz7y1btkSLFi3Qp08fbN68GUOHDhUdk96HaGVnZ8PBwQFmZmawtbVFWlqaps7HxwdnzpzRt0siIiKTp4YgyctQCoUCSqWySHl2drbOEY2S2NjYICAgABcvXizxuoYNG8LNze2l1/2T3slF7dq1kZKSAgBo0KABduzYoak7cOAA7O3t9e2SiIiIXsLd3b3I2gqVSoXbt28btJhT7EiMISM2eicXHTp00BymERYWhgMHDsDX1xft27fHxo0b8d577+kdBBERkakTJPrHUP7+/jh58iQyMjI0Zfv374dKpUJAQIBefT19+hQJCQlo0qRJidddunQJycnJL73un2RCKSeRLly4gAMHDiA3Nxdt27bV+w0WJz/1hiT9EL1qciI+MnYIRCbHftMvZX6Pxs5tJOnnz0cnDWqnVCoRHByMWrVqITw8HGlpaZg9ezb8/Py0DtGKiopCfHw8/vrrLwDA77//jtWrV6NLly5wcXFBSkoK1q5di2vXrmHjxo1o2rQpAGD16tW4c+cOWrVqBQcHB1y7dg3Lli1DpUqVEB8fr9fUS6kP0WrSpIneGQ0RERHpR6FQYP369YiOjsaoUaNgZWWF4OBgREZGal2nVqtRWFio+blatWpQqVSYP38+MjMzYW1tjWbNmuHzzz/XJBYA4Obmhn379mHPnj3IyclB1apVERAQgDFjxui9pkPUyIW+T0CVYt0FRy6IdOPIBVFR/8bIhZdza0n6ufjolCT9mDJRIxdt2rR56TaXv7t06ZLBAREREZkiNZ+KKpqo5GLWrFl6JRdERERUcYlKLvr06VPWcRAREZk0Yz+4rDzhU1GJiIhE4LSIeAYlF7du3cK2bduQnJyMvLy8IvXLli0rdWBERERUPumdXCQmJiIkJAQuLi5ITk6Gp6cnsrOzce/ePdSoUQOurq5lEScREZFRcVpEPL1P6Pziiy/QrVs37Nq1C4Ig4L///S8OHjyIjRs3wszMTK8HmxAREZUXakGQ5FUR6J1cXLlyBcHBwTAze970xbRI8+bNERERgXnz5kkbIRERkQkw9vHf5YneyYVMJoOlpSVkMhkcHR1x//59TV2NGjWQnJwsZXxERERUzuidXLi7u+POnTsAnj9ifc2aNbh69Spu3LiBFStWoE6dOpIHSUREZGyCoJbkVRHovaBzwIABmtGKcePG4cMPP0SvXr0AANbW1li0aJG0ERIREZkAdQWZ0pCC3slF7969Nf/u7u6OPXv24Ny5c8jLy4OPjw8cHR2ljI+IiIjKmVIfolW5cmX4+flJEQsREZHJEvGcT/p/BiUX6enpWL9+Pc6fP4/Hjx+jWrVq8Pb2RmhoKBwcHKSOkYiIyOg4LSKe3gs6z58/j8DAQGzYsAE2NjZo3rw5bGxssGHDBnTp0gXnz58viziJiIionNB75OLzzz9HgwYNsGLFCtjZ2WnKs7OzMXToUMyYMQNbt26VNEgiIiJj47SIeHqPXFy/fh3Dhg3TSiwAwM7ODkOHDsW1a9ckC46IiMhU8IRO8fROLurWrQulUqmzLjs7m+dcEBERVXB6Jxfjx4/H4sWLcfr0aa3yU6dOYcmSJZgwYYJkwREREZkKHv8tnqg1Fz179tT6OTs7G6GhobCzs0PVqlWRkZGB7OxsKBQKxMbGIiAgoEyCJSIiMhauuRBPVHLh5eUFmUxW1rEQERGZLG5FFU9UcjF79uyyjoOIiIheEaU+oZOIiKgi4LSIeEwuiIiIRKgo20iloPduESIiIqKScOSCiIhIBE6LiMfkgoiISATuFhGP0yJEREQkKY5cEBERicBpEfGYXBAREYnA3SLiMbkgIiISoaI8F0QKXHNBREREkuLIBRERkQicFhGPyQUREZEIXNApHqdFiIiISFIcuSAiIhKBCzrFY3JBREQkAqdFxOO0CBEREUmKIxdEREQicORCPCYXREREIjC1EI/TIkRERCQpmcBxHipB586dAQAHDx40ciREpoV/NoiKx5ELIiIikhSTCyIiIpIUkwsiIiKSFJMLIiIikhSTCyIiIpIUkwsiIiKSFJMLIiIikhTPuSAiIiJJceSCiIiIJMXkgoiIiCTF5IKIiIgkxeSCiIiIJMXkgoiIiCTF5IJKZefOnejatSu8vLzQq1evl16/ePFiNGvW7F+IDLh79y4WL16MR48elUnfnp6e+OmnnzRlISEhGD58uOT3ItPyKn7nJ02ahODg4BKv4feb9MHkggz25MkTREVFoUWLFvjmm28wd+5cY4ek5d69e1iyZAlSUlKMHQq9IvidJxLHwtgBkPGoVCpYWFjAzEy/HPNFuzt37kClUuGtt95CixYtyihKIunwO0/076iwIxcvhgGPHj2Knj17omnTphg0aBDu3LmDzMxMjBkzBs2bN8ebb76JPXv2aLU9fPgw+vfvj6ZNm6JNmzaYNm0anj59qql/+vQpZsyYgcDAQHh7e6NTp06YOnUqsrOztfrp1KkTZsyYgW+//RYdO3ZEixYtEB4ejvT0dE0/zZo1w5o1a4rEP2rUKPTr18+g+61atQodO3aEt7c3MjMzX/pZ6Wo3c+ZM9O7dGwDw/vvvw9PTE4sXL35pXy8kJiaiX79+aNKkCbp3745ffvmlyDUv+5zz8/MxZ84cdOzYEY0bN4afnx9GjBiB7OxsnDp1CkOGDAEA9OvXD56envD09BQV27lz5zBixAj4+fnBx8cHvXr1Qnx8vOj3Zqr4ned3Xiy1Wo2pU6eiVatWSExM1Krbu3cvAgMD0axZMwwZMgS3b9/W1HXu3BkzZ84s0t/s2bPh5+eHwsJCAMDDhw8xfPhwNG3aFO3bt8eqVaswY8YMdOrUyaB4yfRU6JGLx48fIzY2FmFhYbCwsEB0dDTGjx8PGxsbtGjRAv3798fmzZsxfvx4eHt7o1atWvjpp58wduxY9OnTB6NGjcLjx48xb948KJVKLFiwAACQm5uLwsJCjB07Fg4ODnjw4AGWLVuGiIgIfPPNN1oxHDp0CLdu3cLUqVORkZGBWbNmYebMmViwYAFsbGzQqVMn7Nq1Cx9++KGmzZMnT5CQkIBx48bpfb99+/ahXr16mDx5MszMzGBlZSXqs/pnO09PTzRp0gSffvoppk6dCi8vL9SoUUNUX/n5+Rg7diw+/PBD1K5dG99//z1GjhyJ7du3w8PDAwBEfc7Lly9HXFwcIiMj8dprryEjIwPHjx+HSqWCl5cXpk6dihkzZiAmJgb169cXFRsA3L9/H82bN8e7774LuVyOs2fPYsqUKQCg+Z9LecXvPL/zL1NQUICJEyfi5MmT2LBhg1aCcunSJaSnpyMyMhKFhYWYNWsWxo8fj02bNgEAgoKC8MMPPyAqKgrm5uYAAEEQsHfvXnTr1g3m5uYQBAHh4eFITU3FjBkzYGdnh1WrVuH+/fuaNvQKECqoiRMnCp6ensL169c1ZRs2bBA8PDyEL774QlOWlZUlNGrUSFi3bp2gVquFjh07CuPGjdPq6/Dhw4Knp6dw9epVnffKz88Xfv/9d8HDw0O4ceOGprxjx46Cv7+/kJeXpymbP3++4OXlJRQWFgqCIAgHDx4UPDw8hJs3b2qu2b59u9CwYUPh4cOHet+vdevWwtOnT0V8Qv9TXLvExETBw8NDOHnypOi+Fi1aJHh4eAhbtmzRlBUUFGh9rmI/52HDhgkjR44s9l4nT54UPDw8hMTERNHx/ZNarRby8/OFzz77THjnnXc05Xfu3BE8PDyEvXv3asree+89YdiwYQbfq6zxOy9eRfvOT5w4UQgKChLy8vKEsLAwISAgQOvzF4Tn328fHx8hLS1NU7Z582bBw8NDePDggSAIgnD58mXBw8NDOHbsmOaa06dPCx4eHsK5c+c078nDw0P47bffNNdkZ2cLzZo1Ezp27KhX3GS6Kuy0CABUr14d7u7ump/r1asHAGjbtq2mTKFQwMHBAQ8fPsTNmzdx7949dO/eHQUFBZpXy5YtIZPJ8Oeff2raxcfHo3fv3mjWrBm8vLwwaNAgAEBycrJWDC1btoRcLtf83KBBA+Tn5yMtLQ0A0L59e9jb22P37t2aa3bv3o2WLVvC2dlZ7/u1atUK1tbWen9WhrYrTpcuXTT/bm5ujk6dOuGPP/4AANGf8+uvv46EhAQsXrwYiYmJUKvVksSWlZWF6OhodOzYEV5eXvDy8sKmTZtw8+ZNSfo3Jn7nxatI33ng+WjQ8OHDcf36dWzcuFHz3fi7hg0bwsHBQfPzi+/Sw4cPAQCenp7w8PAo8t+udu3a8PHxAQBcuHABCoUCb7zxhuYaW1tbtG7dWrL3QsZXoadFFAqF1s+WlpYAADs7O61yuVyOvLw8ZGRkAAAiIiJ09vfgwQMAwP79+zFx4kS88847GDt2LOzt7fH48WNEREQgLy9PVAwvrrO0tETXrl2xe/duREREICMjA7/++iumTZumaaPP/RwdHV/+wehgaDtdLC0tUaVKlSL9P378GABEf85hYWEwMzPD9u3bsWTJEjg4OGDw4MGIiIiATCYzOL5Jkybh3LlziIiIQIMGDWBra4vvv/8ee/fuNbhPU8HvvHgV6TsPAOnp6Xj48CEGDRoEFxcXnde87L8d8HxqZPXq1Zg+fTrMzMzw888/Y8CAAZr6lJQUrQTlBSk/bzK+Cp1c6Mve3h4AMHXqVDRt2rRIffXq1QE8nztt1KgRZsyYoak7ffq0wfcNCgrC5s2bcfnyZfzxxx+QyWTo2rWrpl6f+xn6F1Bp/+L6u/z8fGRlZWn9ZZuWloZq1aoBEP85y+VyjBo1CqNGjcKtW7ewdetWLF68GLVr1zZ4bUReXh4SEhIwceJEhISEaMo3btxoUH/lHb/z0jDl7/wLLi4uGDlyJMaNG4eqVasiLCzMoH6CgoKwYMECHD16FHK5HOnp6QgKCtJ6Ly8W8P7di5ErejUwudBD/fr1UaNGDdy5cweDBw8u9rrc3FxNRv/Czp07Db5vq1atUL16dezevRt//PEH/Pz8NH8ZlcX9/g379+/XrPwvLCzEoUOHNMOmYj/nv6tbty7GjRuHTZs24caNGwB0/1b1MiqVCoWFhVqf55MnT3Do0CHRfbxK+J2Xjql+5/+uW7duUKlUmDhxIqysrPDBBx/o3UedOnXg7e2N3bt3Qy6Xw8PDQ7NoFQCaNGkCpVKJ3377DS1btgTw/M/YqVOntP4bU/nG5EIPMpkMkyZNQmRkJJ4+fYoOHTrA2toa9+/fR0JCAsaOHQs3Nze0bdsWM2bMwJIlS9C8eXMcOXIEJ06cMPi+ZmZm6NGjB7Zv3460tDR88cUXWvVS36+sWVpaYunSpcjLy9OsnH/06BGGDRsGQPznHB4eDi8vL7z++uuwtrbGL7/8gszMTLRp0wbA8/UE5ubm2Lp1K8zNzWFhYYEmTZqUGJudnR2aNGmClStXwsHBARYWFlixYgVsbW11/rb1quN3Xhqm/J3/p7feegt5eXn47LPPIJfLRSc7fxccHIwFCxbA3NwcQ4cO1arz9/eHl5cX/vOf/2DcuHFQKBRYuXIl7OzsJB0tIuNicqGn7t27Q6FQYNmyZZrflGrVqoX27dvDyckJADBw4EDcvXsX3333HdasWQM/Pz/MmzdPa95RX0FBQVi3bh2sra2L7AUvi/uVJUtLS8yfPx+ff/45rl69itq1a2PRokVo2LCh5hoxn3Pz5s2xd+9erF27FoWFhXBzc8O8efM0ixMdHBwwdepUrFq1Cj/++CMKCgpw5cqVl8Y3b948fPbZZ5g0aRLs7e0REhKCp0+f6jx7oSLgd770TP07/0/9+/dHXl4eZs6cCSsrK/Tt21ev9t27d8fs2bNRWFiIHj16aNXJZDJ8/fXXmDp1KqZOnQqFQoEhQ4bg2rVruHbtmt6xkmmSCYIgGDsIIiKquFQqFbp3745WrVohJibG2OGQBDhyQURE/6pNmzZBrVbDzc0NSqUS33//PR48eKDZTkzlH5OLCq6goKDYOplMpteJeWq1usR99+bm5kafUy0sLERJg3UWFvwj8arjd16bMb7zlSpVwsqVK3H37l0Az8/PWL58ud7rQ8h0cVqkgivp2QO1atXSa5fEpEmTsH379mLrv/nmG6MflBMSElLiFsmDBw+idu3a/2JE9G/jd14bv/NUFphcVHAXLlwotk4ul+v14KO7d+9qDgPSxc3NDba2tnrFJ7UbN24gJyen2HpPT0+t0yPp1cPvvDZ+56ksMLkgIiIiSVXoZ4sQERGR9JhcEBERkaSYXBAREZGkmFwQERGRpJhcEBERkaSYXBAREZGkmFwQERGRpP4PEO8Bix3gyPkAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.heatmap(df, fmt=\".2f\", annot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "231bed48-45d9-43cc-9f95-0c7bbf7900f6",
   "metadata": {},
   "source": [
    "# Task: \n",
    "\n",
    "1. Add f1 score to our metrics function, re-run and save the metrics again after adding f1 score to the saved dataframe\n",
    "(https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html) \\\n",
    "F-1 score = ...\n",
    "\n",
    "<ins>Hint:</ins> LaTeX works with jupyter markdown in general. You can use $$ to write LaTeX."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7d378e5-b76d-47df-8707-72aa8b908d2e",
   "metadata": {},
   "source": [
    "2. For random forest, we can assess the most important features. Create a dataframe with genes ranked by their feature importances\n",
    "\n",
    "<ins>Hint</ins>: .feature_importances_ attribute from the random forest classifier will return an array of importances for each feature. The order of features remains same as in train_X."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c072d64-f27c-44ee-a1ea-77a0f9725941",
   "metadata": {
    "tags": []
   },
   "source": [
    "Q: How would you compute feature importance per cell type?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98aa21e1-fb6e-41e5-ab6c-a4f74bf6ccea",
   "metadata": {},
   "source": [
    "3. Do the same as above for the following classifiers: \n",
    "\n",
    "Classifiers:\n",
    "\n",
    "a. Support vector classifier: https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC \\\n",
    "b. Linear support vector classifier with l1 regularization: https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html#sklearn.svm.LinearSVC \\\n",
    "c. Multi-layered perceptron: https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html \\\n",
    "\n",
    "\n",
    "Subtasks:\n",
    "\n",
    "3.1. Optimize the classifier on train set using cross-validation \\\n",
    "3.2. Save the model \\\n",
    "3.2. Predict on test set \\\n",
    "3.3. save results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab31e99f-b1af-405a-9bfa-61fd2dd23f87",
   "metadata": {},
   "source": [
    "# Bonus #1: Lab"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28a7783b-5ebe-4715-802e-6194e27da1cf",
   "metadata": {},
   "source": [
    "Choose the best classifier, build it on Sun again and predict on freytag."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "015dc262-881b-4155-a87f-0791eeec5890",
   "metadata": {},
   "source": [
    "<ins>Hint</ins>: You will need to subset both the dataset so that they contain same gene."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31660094-b9eb-4f93-aa17-3270cc9c571f",
   "metadata": {},
   "source": [
    "# Bonus #2: Lab"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6997cd1b-c747-4c96-b202-28045adca9ee",
   "metadata": {},
   "source": [
    "Write a function to compute metrics per class.\n",
    "\n",
    "<ins>Hint</ins>: You will need to run the functions per class, easiest to do in a lopp.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "422c8866-db23-4b69-85b7-42d5275ecf7d",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Bonus #3: Reading and lab"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "958ce818-91c2-45b9-b92e-f7be77adbaa0",
   "metadata": {},
   "source": [
    "RandomForestClassifier provides feature importances at a global level (i.e. not per sample), while many other algorithms like MLPs do not provide any inherent interpretability. To compute exact or approximated importances or interpretability, SHAP [https://shap.readthedocs.io/en/latest/](https://shap.readthedocs.io/en/latest/) provides a nice framework that is model-agnostic. \n",
    "\n",
    "For tree based models, SHAP provides exact computation of Shapley values [https://en.wikipedia.org/wiki/Shapley_value](https://en.wikipedia.org/wiki/Shapley_value). Paper for the second part: Lundberg et al., 2020 [https://www.nature.com/articles/s42256-019-0138-9](https://www.nature.com/articles/s42256-019-0138-9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0450ecb-e8db-443d-b58f-8ccecd862052",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}