13-EX-SVM-v2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creative Commons CC BY 4.0 Lynd Bacon & Associates, Ltd. Not warranted to be suitable for any particular purpose. (You're on your own!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Support Vector Machines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "from IPython.core.display import HTML "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Support Vector Machine Classifier\n",
    "\n",
    "* A support vector machine [SVM](https://en.wikipedia.org/wiki/Support-vector_machine) classifier is a non-probability classifier.\n",
    "* Attempts to separate classes of observations by finding the hyperplane in the space of features that best divides the two classes, that separates them by the largest \"margin\".\n",
    "    Hence, SVMs are \"margin classifiers.\"\n",
    "* Can also be used for _unsupervised_ learning, i.e., for clustering.\n",
    "\n",
    "Here's an illustration borrowed from [Data Camp](https://www.datacamp.com/):\n",
    "\n",
    "![SVM example](./images/svm-datacamp-ex.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SVMs can be trained, tweaked, and evaluated like binary logistic regression machine learners, and other kinds of machine learners, can.\n",
    "\n",
    "In the following we're going to apply SVMs to a [cervical cancer dataset](https://archive.ics.uci.edu/ml/datasets/Cervical+cancer+%28Risk+Factors%29) available on the [UCI ML repository](https://archive.ics.uci.edu/ml/index.php)\n",
    "\n",
    "The indicated citation: Kelwin Fernandes, Jaime S. Cardoso, and Jessica Fernandes. 'Transfer Learning with Partial Observability Applied to Cervical Cancer Screening.' Iberian Conference on Pattern Recognition and Image Analysis. Springer International Publishing, 2017. ([paper link](http://inescporto.pt/~jsc/publications/conferences/2017KelwinIBPRIA.pdf))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Essentials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\"\n",
    "from IPython.display import display, Markdown\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import scikitplot as skplt\n",
    "from sklearn import linear_model  \n",
    "from sklearn.model_selection import cross_val_predict, cross_validate\n",
    "from sklearn import preprocessing\n",
    "from sklearn.metrics import accuracy_score, confusion_matrix, roc_auc_score\n",
    "from sklearn.metrics import roc_curve, precision_score, recall_score\n",
    "from sklearn.svm import SVC  # support vector classifier\n",
    "import os\n",
    "import shelve\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Data\n",
    "\n",
    "The data are in a headered csv file that may be in the pwd, or in a data subirectory. We'll input it into a Pandas DataFrame, examine the data, select some variables to use, and then we'll extract from our DataFrame the numpy arrays we need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "cervical=pd.read_csv('DATA/ML/risk_factors_cervical_cancer.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(858, 36)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "Index(['Age', 'Number of sexual partners', 'First sexual intercourse',\n",
       "       'Num of pregnancies', 'Smokes', 'Smokes (years)', 'Smokes (packs/year)',\n",
       "       'Hormonal Contraceptives', 'Hormonal Contraceptives (years)', 'IUD',\n",
       "       'IUD (years)', 'STDs', 'STDs (number)', 'STDs:condylomatosis',\n",
       "       'STDs:cervical condylomatosis', 'STDs:vaginal condylomatosis',\n",
       "       'STDs:vulvo-perineal condylomatosis', 'STDs:syphilis',\n",
       "       'STDs:pelvic inflammatory disease', 'STDs:genital herpes',\n",
       "       'STDs:molluscum contagiosum', 'STDs:AIDS', 'STDs:HIV',\n",
       "       'STDs:Hepatitis B', 'STDs:HPV', 'STDs: Number of diagnosis',\n",
       "       'STDs: Time since first diagnosis', 'STDs: Time since last diagnosis',\n",
       "       'Dx:Cancer', 'Dx:CIN', 'Dx:HPV', 'Dx', 'Hinselmann', 'Schiller',\n",
       "       'Citology', 'Biopsy'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "Age                                    int64\n",
       "Number of sexual partners             object\n",
       "First sexual intercourse              object\n",
       "Num of pregnancies                    object\n",
       "Smokes                                object\n",
       "Smokes (years)                        object\n",
       "Smokes (packs/year)                   object\n",
       "Hormonal Contraceptives               object\n",
       "Hormonal Contraceptives (years)       object\n",
       "IUD                                   object\n",
       "IUD (years)                           object\n",
       "STDs                                  object\n",
       "STDs (number)                         object\n",
       "STDs:condylomatosis                   object\n",
       "STDs:cervical condylomatosis          object\n",
       "STDs:vaginal condylomatosis           object\n",
       "STDs:vulvo-perineal condylomatosis    object\n",
       "STDs:syphilis                         object\n",
       "STDs:pelvic inflammatory disease      object\n",
       "STDs:genital herpes                   object\n",
       "STDs:molluscum contagiosum            object\n",
       "STDs:AIDS                             object\n",
       "STDs:HIV                              object\n",
       "STDs:Hepatitis B                      object\n",
       "STDs:HPV                              object\n",
       "STDs: Number of diagnosis              int64\n",
       "STDs: Time since first diagnosis      object\n",
       "STDs: Time since last diagnosis       object\n",
       "Dx:Cancer                              int64\n",
       "Dx:CIN                                 int64\n",
       "Dx:HPV                                 int64\n",
       "Dx                                     int64\n",
       "Hinselmann                             int64\n",
       "Schiller                               int64\n",
       "Citology                               int64\n",
       "Biopsy                                 int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>STDs: Number of diagnosis</th>\n",
       "      <th>Dx:Cancer</th>\n",
       "      <th>Dx:CIN</th>\n",
       "      <th>Dx:HPV</th>\n",
       "      <th>Dx</th>\n",
       "      <th>Hinselmann</th>\n",
       "      <th>Schiller</th>\n",
       "      <th>Citology</th>\n",
       "      <th>Biopsy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>858.000000</td>\n",
       "      <td>858.000000</td>\n",
       "      <td>858.000000</td>\n",
       "      <td>858.000000</td>\n",
       "      <td>858.000000</td>\n",
       "      <td>858.000000</td>\n",
       "      <td>858.000000</td>\n",
       "      <td>858.000000</td>\n",
       "      <td>858.000000</td>\n",
       "      <td>858.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>26.820513</td>\n",
       "      <td>0.087413</td>\n",
       "      <td>0.020979</td>\n",
       "      <td>0.010490</td>\n",
       "      <td>0.020979</td>\n",
       "      <td>0.027972</td>\n",
       "      <td>0.040793</td>\n",
       "      <td>0.086247</td>\n",
       "      <td>0.051282</td>\n",
       "      <td>0.064103</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>8.497948</td>\n",
       "      <td>0.302545</td>\n",
       "      <td>0.143398</td>\n",
       "      <td>0.101939</td>\n",
       "      <td>0.143398</td>\n",
       "      <td>0.164989</td>\n",
       "      <td>0.197925</td>\n",
       "      <td>0.280892</td>\n",
       "      <td>0.220701</td>\n",
       "      <td>0.245078</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>13.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>20.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>25.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>32.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>84.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Age  STDs: Number of diagnosis   Dx:Cancer      Dx:CIN  \\\n",
       "count  858.000000                 858.000000  858.000000  858.000000   \n",
       "mean    26.820513                   0.087413    0.020979    0.010490   \n",
       "std      8.497948                   0.302545    0.143398    0.101939   \n",
       "min     13.000000                   0.000000    0.000000    0.000000   \n",
       "25%     20.000000                   0.000000    0.000000    0.000000   \n",
       "50%     25.000000                   0.000000    0.000000    0.000000   \n",
       "75%     32.000000                   0.000000    0.000000    0.000000   \n",
       "max     84.000000                   3.000000    1.000000    1.000000   \n",
       "\n",
       "           Dx:HPV          Dx  Hinselmann    Schiller    Citology      Biopsy  \n",
       "count  858.000000  858.000000  858.000000  858.000000  858.000000  858.000000  \n",
       "mean     0.020979    0.027972    0.040793    0.086247    0.051282    0.064103  \n",
       "std      0.143398    0.164989    0.197925    0.280892    0.220701    0.245078  \n",
       "min      0.000000    0.000000    0.000000    0.000000    0.000000    0.000000  \n",
       "25%      0.000000    0.000000    0.000000    0.000000    0.000000    0.000000  \n",
       "50%      0.000000    0.000000    0.000000    0.000000    0.000000    0.000000  \n",
       "75%      0.000000    0.000000    0.000000    0.000000    0.000000    0.000000  \n",
       "max      1.000000    1.000000    1.000000    1.000000    1.000000    1.000000  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get a look at what we've got\n",
    "cervical.shape\n",
    "cervical.columns\n",
    "cervical.dtypes\n",
    "cervical.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It would appear the missing values in the data are codes with a question mark, ?.  That would explain the variables that are type \"object.\"  This type is essentially just a char type, and it's what Pandas defaults to when it can't figure out what numeric type an input variable might be.\n",
    "\n",
    "To set the ? to np.nan's, numpy missing value codes, we can read the csv file again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "cervical=pd.read_csv('DATA/ML/risk_factors_cervical_cancer.csv',na_values='?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "float64    26\n",
       "int64      10\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cervical.dtypes.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like the ? are now gone. Let's check for missings values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Age                                     0\n",
       "Number of sexual partners              26\n",
       "First sexual intercourse                7\n",
       "Num of pregnancies                     56\n",
       "Smokes                                 13\n",
       "Smokes (years)                         13\n",
       "Smokes (packs/year)                    13\n",
       "Hormonal Contraceptives               108\n",
       "Hormonal Contraceptives (years)       108\n",
       "IUD                                   117\n",
       "IUD (years)                           117\n",
       "STDs                                  105\n",
       "STDs (number)                         105\n",
       "STDs:condylomatosis                   105\n",
       "STDs:cervical condylomatosis          105\n",
       "STDs:vaginal condylomatosis           105\n",
       "STDs:vulvo-perineal condylomatosis    105\n",
       "STDs:syphilis                         105\n",
       "STDs:pelvic inflammatory disease      105\n",
       "STDs:genital herpes                   105\n",
       "STDs:molluscum contagiosum            105\n",
       "STDs:AIDS                             105\n",
       "STDs:HIV                              105\n",
       "STDs:Hepatitis B                      105\n",
       "STDs:HPV                              105\n",
       "STDs: Number of diagnosis               0\n",
       "STDs: Time since first diagnosis      787\n",
       "STDs: Time since last diagnosis       787\n",
       "Dx:Cancer                               0\n",
       "Dx:CIN                                  0\n",
       "Dx:HPV                                  0\n",
       "Dx                                      0\n",
       "Hinselmann                              0\n",
       "Schiller                                0\n",
       "Citology                                0\n",
       "Biopsy                                  0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cervical.isnull().sum(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Missing values\n",
    "\n",
    "Missing values can be dealt with in various ways.  Values can be predicted, or \"imputed,\" for them using various methods, including:\n",
    "\n",
    "* replacing them with a measure of central tendency, e.g., the mean computed from non-missing values;\n",
    "* imputing them using a supervised learning algorithm like k-nearest neighbors (\"KNN\") or RandomForest\n",
    "* removing observations with missing values.\n",
    "\n",
    "For our present purposes, we're just going to remove cases with missing values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting Features\n",
    "\n",
    "Let's use the following as the features we'll use for prediction:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "cervicalFeats=cervical.loc[:,'Age':'Hormonal Contraceptives (years)'].copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to the available documentation, there are four \"target\" variables.  Here are their relative frequencies in this dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Hinselmann    0.040793\n",
       "Schiller      0.086247\n",
       "Citology      0.051282\n",
       "Biopsy        0.064103\n",
       "dtype: float64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cervical.loc[:,\"Hinselmann\":\"Biopsy\"].apply(np.sum,axis=0)/len(cervical)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that all these putative results are infrequent in this data.  Classifiers tend to struggle when trying to predict rare events.  \n",
    "\n",
    "In what follows we'll work with `Biopsy` as the target to be predicted. Let's put this target variable together with the features we'll use to predict it, in a new DataFrame that we'll drop rows with any missing values from."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Biopsy', 'Age', 'Number of sexual partners',\n",
       "       'First sexual intercourse', 'Num of pregnancies', 'Smokes',\n",
       "       'Smokes (years)', 'Smokes (packs/year)', 'Hormonal Contraceptives',\n",
       "       'Hormonal Contraceptives (years)'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(676, 10)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cervical2=pd.concat([cervical.Biopsy,cervicalFeats],axis=1).dropna(axis=0)\n",
    "cervical2.columns\n",
    "cervical2.shape\n",
    "# cervical2.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks good enough for government work.  Let's create the numpy arrays we'll need for training our SVM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(676, 9)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(676,)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X=cervical2.iloc[:,1:].to_numpy()  # features\n",
    "y=cervical2.iloc[:,0].to_numpy()  # target\n",
    "X.shape\n",
    "y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Support Vector Classifier\n",
    "\n",
    "Here we'll do simple k-fold cross-validation of our classifier.  We'll then get predictions for our cross-validated model, and we'll use them for plotting ROC curves and doing precision-recall plots."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cross-Validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "svclf=SVC(probability=True)   #instance of the support vector classifier\n",
    "svcRes=cross_validate(svclf,X,y,cv=20,n_jobs=-1,return_train_score=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>20.000000</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>20.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.052203</td>\n",
       "      <td>0.000631</td>\n",
       "      <td>0.932218</td>\n",
       "      <td>0.941375</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.004389</td>\n",
       "      <td>0.000125</td>\n",
       "      <td>0.012064</td>\n",
       "      <td>0.001568</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.041598</td>\n",
       "      <td>0.000414</td>\n",
       "      <td>0.914286</td>\n",
       "      <td>0.937792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.051720</td>\n",
       "      <td>0.000496</td>\n",
       "      <td>0.914286</td>\n",
       "      <td>0.940810</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.053520</td>\n",
       "      <td>0.000670</td>\n",
       "      <td>0.939394</td>\n",
       "      <td>0.940902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>0.055257</td>\n",
       "      <td>0.000717</td>\n",
       "      <td>0.939394</td>\n",
       "      <td>0.942390</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>0.057179</td>\n",
       "      <td>0.000800</td>\n",
       "      <td>0.941176</td>\n",
       "      <td>0.943838</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        fit_time  score_time  test_score  train_score\n",
       "count  20.000000   20.000000   20.000000    20.000000\n",
       "mean    0.052203    0.000631    0.932218     0.941375\n",
       "std     0.004389    0.000125    0.012064     0.001568\n",
       "min     0.041598    0.000414    0.914286     0.937792\n",
       "25%     0.051720    0.000496    0.914286     0.940810\n",
       "50%     0.053520    0.000670    0.939394     0.940902\n",
       "75%     0.055257    0.000717    0.939394     0.942390\n",
       "max     0.057179    0.000800    0.941176     0.943838"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's take a look at performance\n",
    "pd.DataFrame(svcRes).describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## But, Why?  \n",
    "\n",
    "Looks like a really accurate classifier, for sure.  But, the positive class of the `Biopsy` target variable is rare, i.e., there are very few cases for which `Biopsy` = 1. \n",
    "\n",
    "Let's take a look at the confusion matrix, an ROC curve, AUC, and a precision-recall plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, get predicted probabilites for the target labels\n",
    "\n",
    "yPredProb=cross_val_predict(svclf,X,y,n_jobs=-1,cv=20,\n",
    "                           method='predict_proba')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.93928309, 0.06071691],\n",
       "       [0.93767837, 0.06232163],\n",
       "       [0.92875601, 0.07124399],\n",
       "       [0.92808085, 0.07191915],\n",
       "       [0.932451  , 0.067549  ],\n",
       "       [0.92875631, 0.07124369],\n",
       "       [0.93405588, 0.06594412],\n",
       "       [0.93075004, 0.06924996]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# What's returned is an N X 2 numpy array containing predicted label probabilities.\n",
    "# Here's the first 8 rows.  The second col is the probability that Biopsy = 1:\n",
    "\n",
    "yPredProb[:8,:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWsAAADtCAYAAAB52FQJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAeGklEQVR4nO3de7xVVb338c93bwQ0Ne9EgIKKGPR4QaVSK8sexbTwnNfR8GhR+uRTadbpCl3N4mj6OpUdtZNpSVoiph7JGxo+VnZU8JohKiheEBTQUlMTwd/zxxxbF9u915ob1tprzTW/b1/ztdeaa6wxx95ufnus3xwXRQRmZtbaOprdADMzq83B2sysAByszcwKwMHazKwAHKzNzApgQLMbYGbWDJ2b7xCx5qVcZeOllXMiYmKDm1SVg7WZlVKseYlBY47MVfYfd5+9TYObU5ODtZmVlEDFyQQ7WJtZOQno6Gx2K3JzsDaz8pKa3YLcHKzNrKScBjEzKwb3rM3MWpwoVM+6OC01M6srZT3rPEee2qQtJP1G0v2SFkp6l6StJN0gaVH6umVF+WmSFkt6QNLBtep3sDaz8urozHfkcyZwXUTsCuwOLASmAnMjYjQwNz1H0lhgMjAOmAicI6nqhRyszayk0g3GPEetmqTNgfcA5wNExOqI+BswCZiRis0ADk+PJwEzI+LliFgCLAYmVLuGg7WZlZPoSxpkG0m3VxzHd6ttR2Al8AtJd0k6T9KbgCERsRwgfd0ulR8GPF7x/qXpXK98g9HMyiv/DcZVEbF3ldcHAOOBz0bEbZLOJKU8ertyD+eqbtvlnrWZlVT90iBkPeOlEXFbev4bsuD9lKShAOnrioryIyrePxxYVu0CDtZmVk4COjvzHTVExJPA45LGpFMHAvcBs4Ep6dwU4Mr0eDYwWdIgSaOA0cC8atdwGsTMyqu+k2I+C/xK0kDgYeATZB3iWZKOAx4DjgCIiAWSZpEF9DXACRGxtlrlDtZmVlL1nW4eEXcDPeW1D+yl/HRget76HazNrLw83dzMrAAKNN3cwdrMyqkPU8lbgYO1mZWXNx8wM2t1Xs/azKwYnAYxM2txBVvP2sHazErKaRAzs2LwDUYzswIoUM66OJ8BrKVI2ljSbyU9K+nSDajnaEnX17NtzSDpWklTape0lqG6rrrXcK3RCmsYSf+aFkv/u6TlKajsX4eq/wUYAmwdEUesbyUR8auIOKgO7VmHpAMkhaTLu53fPZ2/KWc9J0u6qFa5iDgkImbUKmctpo57MDaag3Ubk/QF4EfAv5MF1u2Bc8i2FNpQOwAPRsSaOtTVKCuBfSVtXXFuCvBgvS6gjP8dFZSkXEcr8C9Zm5L0ZuAUsqUXL4+IFyLilYj4bUR8OZUZJOlHkpal40eSBqXXDpC0VNIXJa1IvfJPpNe+A3wL+EjqsR/XvQcqaWTqwQ5Izz8u6WFJz0taIunoivM3V7xvX0nzU3plvqR9K167SdJ3Jf0p1XO9pG2q/BhWA/9NtjEpaUPSI4FfdftZnSnpcUnPSbpD0rvT+YnA1yq+z3sq2jFd0p+AF4Ed07n/k17/iaTfVNT/fUlz1Sr/6g3o2tXLwdqa713AYOCKKmW+DrwT2INsN+YJwDcqXn8L8GayveGOA86WtGVEfJust35JRGwaEedXa0jai+7HwCERsRmwL3B3D+W2Aq5OZbcGfgBc3a1n/K9k6wRvBwwEvlTt2sAvgY+lxwcDC3jjjhzzyX4GWwG/Bi6VNDgiruv2fe5e8Z6PAscDmwGPdqvvi8Bu6Q/Ru8l+dlMiouq2TdbPJNSR72gFDtbta2uyfeOqpSmOBk6JiBURsRL4DlkQ6vJKev2ViLgG+Dswpod68ngVeLukjSNieUQs6KHMocCiiLgwItZExMXA/cCHKsr8IiIejIiXgFlkQbZXEfE/wFZpB4+PkQXv7mUuioin0zX/AxhE7e/zgohYkN7zSrf6XgSOIftjcxHZvnxLa9RnTeCetbWCp8l2ZK42PPOtrNsrfDSde62ObsH+RWDTvjYkIl4APgJ8Clgu6WpJu+ZoT1ebKnd9fnI92nMhcCLwPnr4pJFSPQtT6uVvZJ8mqqVXYN2dqd8gIuaR7RYisj8q1oIcrK0V3AL8Azi8SpllZDcKu2xPjU07q3gB2KTi+VsqX4yIORHxv4GhZL3ln+VoT1ebnljPNnW5EPgMcE3q9b4mpSm+SpbL3jIitgCe5fXdp3tLXVRNaUg6gayHvgz4yvo33RrJwdqaLiKeJbsJeLakwyVtImkjSYdIOj0Vuxj4hqRt0426b5F9bF8fdwPvkbR9urk5resFSUMkfTjlrl8mS6f0tN/cNcAuyoYbDpD0EWAscNV6tgmAiFgCvJcsR9/dZmR74K0EBkj6FrB5xetPASPVhxEfknYBvkeWCvko8BVJVdM11gTqw9ECHKzbWET8APgC2U3DlWQf3U8kGyEBWUC5HfgzcC9wZzq3Pte6Abgk1XUH6wbYDrKbbsuAZ8gC52d6qONp4LBU9mmyHulhEbFqfdrUre6bI6KnTw1zgGvJhvM9SvZppDLF0TXh52lJd9a6Tko7XQR8PyLuiYhFZCNKLlQaaWOtQeTrVbdKz1q+QW1mZTRg6x1j8w/m65v89aKj74iInjbD7TfuWTfPFsBvyPK3C8mG2n2XrGd6N3A9697smwYsBh4gG4JmJSJpoqQHJC2WNLXZ7WkXRepZNzRY+xesqjOB64BdycY4LwTOAHYjG452FVkOGbK87WRgHDCRbBZicZYLsw2SJvOcDRxC9rtwlKSxzW1VG3DOOuNfsKo2B94DdE0mWQ38DXiuosybeH3EwSRgJtnNuSVkPewJ/dJSawUTgMUR8XBErCb7XajHkgGl5551xr9gvduR7IbfL4C7gPPIgjPAdLIbXEfzes96GOve9FrKumOPrb35/38DFO0GYyPXs+7pF+wd3QtJOp5s2i5owF4avGUDm9Qa9hq/O7fcdC37H3jYPvPm38mPzvjeHs89//xx3zrl+6+Vmfqlk4YPHjx4xcnfO52zfngat9x2+wm/mvmbCwHO+8kPuea6ucdefuUGjWgrjD3ftn2zm9BUo0btyHPPPctee+19HMAOO4zkxRdfYK+99j6h2W1rlkcffYRVq1ZtcBSt51RySY8Az5MNS10TEXunJRQuAUYCjwBHRsRfU/lpZEsRrAVOiog51epvZLDu6afwhqEnEXEucC5AxybbxaAxRzawSa1h5eDNeGLFs9zz3M4MGrMzs+94gS9+4mAGjVnyWpnLbn+Zy398LKde+gjLV2/DyD0nMuiu7IPQiNHjWXXjkwwas0lvl2grf7rtrGY3oaluveUWpn/3ZH57TfZv+YzvnwrAl786rdrb2tp+76jDwAzRiF7z+7oNNZ0KzI2I09J9u6nAV1NKuOs+1FuB30naJSJ6mn8ANDYNshQYUfF8OOs/O66tPPX08yx98q+M3mE7AA6YMIb7H36Snbbf9rUyh753Nx585CkArr7pzxxx8HgGbjSAHd66NTtvvy3z//JIM5puTbD3PvuwePEiHlmyhNWrV3PpJTM59LAPN7tZbaEf0iCTgK51zmfw+oziScDMiHg5TdqqeR+qkT3r+cBoSaPIpgtPJlsxzYAvfP9SfvHvH2fggE4eeWIVx3/7In7y7aMZvcN2vPpq8NjyZzhp+kwAFj78JJddfxd3XfZ11qx9lc+fNotXX/X4+LIYMGAAPzzzLD506MGsXbuWKR8/lrHjxjW7WW2hD4F4G0m3Vzw/N2UFKgVwvaQAfppeHxIRywEiYrmk7VLZYcCtFe+teR+iYcE6ItZIOpFshlgn8PNeVlorpT8/+AT7H336OueO+tJ5vZY//fw5nH5+1ZSWtbGJh3yQiYd8sNnNaCtdNxhzWpVjUsx+EbEsBeQbJN1f9fJvVLUH1tANc9Oymtc08hpmZuutjinrruUMImKFpCvI0hpPSRqaetVDgRWpeJ/TxJ7BaGblJOjo6Mh11KxKepOkzboeAwcBfwFmk20lR/p6ZXo8G5isbLemUcBoYF61azS0Z21m1srqOBpkCHBFqm8A8OuIuE7SfGCWpOOAx4AjACJigaRZwH1kqz6eUG0kSFelZmblVKdYHREPky0b0f3808CBvbxnOtkkuFwcrM2stFpldmIeDtZmVkqtNJU8DwdrMystB2szswKo59ogjeZgbWal5Z61mVmra8xCTg3jYG1mpSSgQLHawdrMysqjQczMCqHDNxjNzFqcnAYxM2t5wj1rM7NCcM/azKwAfIPRzKzVOWdtZtb6hHJtLNAqHKzNrLTcszYzKwDnrM3MWp1z1mZmrS9bG6Q40drB2sxKq0Cx2sHazMrLMxjNzFqd17M2M2t9Xs/azKwQvJ61mVkhFChWO1ibWUmpWDcYizMx3sysjrrGWec5ctUndUq6S9JV6flWkm6QtCh93bKi7DRJiyU9IOngPPU7WJtZadUzWAOfAxZWPJ8KzI2I0cDc9BxJY4HJwDhgInCOpM5alTtYm1lpSfmO2vVoOHAocF7F6UnAjPR4BnB4xfmZEfFyRCwBFgMTal3DwdrMSqsPPettJN1ecRzfraofAV8BXq04NyQilgOkr9ul88OAxyvKLU3nqvINRjMrp74t5LQqIvbusRrpMGBFRNwh6YB8V36DqPUmB2szK6Vs84G6jAbZD/iwpA8Cg4HNJV0EPCVpaEQslzQUWJHKLwVGVLx/OLCs1kWcBjGz0uqQch3VRMS0iBgeESPJbhzeGBHHALOBKanYFODK9Hg2MFnSIEmjgNHAvFptdc/azEqrwZNiTgNmSToOeAw4AiAiFkiaBdwHrAFOiIi1tSpzsDazUlIDFnKKiJuAm9Ljp4EDeyk3HZjel7p7DdaSNq/RqOf6ciEzs1ZToAmMVXvWC8juUFZ+O13PA9i+ge0yM2u4Ik037zVYR8SI3l4zMys6kY0IKYpco0EkTZb0tfR4uKS9GtssM7PG61C+oxXUDNaSzgLeB3w0nXoR+K9GNsrMrOFyzl5slTWv84wG2Tcixku6CyAinpE0sMHtMjNruBaJw7nkCdavSOogTYeUtDXrzn83MyscQc0JL60kT7A+G7gM2FbSd4Ajge80tFVmZv2gLUaDdImIX0q6A/hAOnVERPylsc0yM2usvMuftoq8Mxg7gVfIUiFeT8TM2kKR0iB5RoN8HbgYeCvZ6lC/ljSt0Q0zM2s05TxaQZ6e9THAXhHxIoCk6cAdwKmNbJiZWaO1yrC8PPIE60e7lRsAPNyY5piZ9Y9sNEizW5FftYWcfkiWo34RWCBpTnp+EHBz/zTPzKxBVLfNB/pFtZ5114iPBcDVFedvbVxzzMz6T1ukQSLi/P5siJlZf2qbNEgXSTuRLZI9lmx/MQAiYpcGtsvMrOGK1LPOM2b6AuAXZH+IDgFmATMb2CYzs35RpKF7eYL1JhExByAiHoqIb5CtwmdmVlgSdHYo19EK8gzde1nZZ4WHJH0KeALYrrHNMjNrvCKlQfIE638DNgVOIstdvxk4tpGNMjPrDwWK1bkWcrotPXye1zcgMDMrNKFCrQ1SbVLMFaQ1rHsSEf/ckBaZmfWHNlp176x+a0Xyv8aM4Prf/7C/L2tmJdUWOeuImNufDTEz608COgsUrL02tZmVVr12N5c0WNI8SfdIWpB21ULSVpJukLQofd2y4j3TJC2W9ICkg2u2dUO+UTOzIqtXsAZeBt4fEbsDewATJb0TmArMjYjRwNz0HEljgcnAOGAicI6kzqptzftNSRqUt6yZWavLtvVSrqOWyPw9Pd0oHQFMAmak8zOAw9PjScDMiHg5IpYAi4EJ1a6RZ6eYCZLuBRal57tL+s+arTcza3F96FlvI+n2iuP47nVJ6pR0N7ACuCENex4SEcsB0teuCYXDgMcr3r40netVnkkxPwYOA/47XfAeSZ5ubmaF14f7i6siYu9qBSJiLbCHpC2AKyS9vdqle6qiWv15gnVHRDza7aPA2hzvMzNrWQIGNGA0SET8TdJNZLnopyQNjYjlkoaS9boh60mPqHjbcGBZtXrz5KwflzQBiNTN/zzwYJ+/AzOzFiPlO2rXo21TjxpJGwMfAO4HZgNTUrEpwJXp8WxgsqRBkkYBo4F51a6Rp2f9abJUyPbAU8Dv0jkzs8KS6jrdfCgwI43o6ABmRcRVkm4BZkk6DngMOAIgIhZImgXcB6wBTkhplF7lWRtkBdkQEzOztlKvWB0Rfwb27OH808CBvbxnOtnieLnk2SnmZ/SQ+I6IN9wNNTMrkhZZqjqXPGmQ31U8Hgz8E+sOOTEzKxxBy2wskEeeNMgllc8lXQjc0LAWmZn1h/yzE1tCnp51d6OAHerdEDOz/qaW2WGxtjw567/yes66A3iGNL/dzKyoRBv1rNPei7uT7bsI8GpEVJ1lY2ZWFEUK1lUnxaTAfEVErE2HA7WZtY16LeTUH/LMYJwnaXzDW2Jm1o8k6OzId7SCanswDoiINcD+wCclPQS8QJbqiYhwADezQmuLDXPJ5qmP5/X1V83M2kY73WAUQEQ81E9tMTPrVwXqWFcN1ttK+kJvL0bEDxrQHjOzfiI62mScdSewKT0vkm1mVmiifXrWyyPilH5riZlZfxIMKFDSumbO2sysHbVTz7rHNVjNzNpFWwzdi4hn+rMhZmb9rUCxer1W3TMzKzyRbwp3q3CwNrNyUpukQczM2lk2g9HB2sys5RUnVDtYm1mJFahj7WBtZmXVOmtV5+FgbWal5NEgZmYF4RuMZmatThQqDVKkTwFmZnXTlQbJc9SsSxoh6f9JWihpgaTPpfNbSbpB0qL0dcuK90yTtFjSA5IOrnUNB2szK606bpi7BvhiRLwNeCdwgqSxwFRgbkSMBuam56TXJgPjgInAOZI6q13AwdrMSks5j1oiYnlE3JkePw8sBIYBk4AZqdgMXt8mcRIwMyJejoglwGJgQrVrOGdtZqUkoDN/znobSbdXPD83Is7tsV5pJLAncBswJCKWQxbQJW2Xig0Dbq1429J0rlcO1mZWWn24v7gqIvauXZ82BS4DPh8Rz1VJofT0QlSr22kQMysp5f4vV23SRmSB+lcRcXk6/ZSkoen1ocCKdH4pMKLi7cOBZdXqd7A2s9KS8h2165GA84GF3TYTnw1MSY+nAFdWnJ8saZCkUcBoYF61azgNYmallA3dq9s46/2AjwL3Sro7nfsacBowS9JxwGPAEQARsUDSLOA+spEkJ0TE2moXcLA2s3LK2WvOIyJupveBIz1ukRgR04Hpea/hYG1mpeXp5mZmLS7bfKDZrcjPwdrMSivvSI9W4GBtZqVVoCyIg7WZlZd71mZmLc45azOzIpA8GsTMrAiKE6odrM2spLI0SHHCtYO1mZVWcUK1g7WZlVmBorWDtZmVltMgZmYFUJxQ7WBtZmVWoGjtYG1mpZRthlucaO1gbWblVMf1rPuDg7WZlVaBYrX3YGwVa9eu5QP778MxRx7+2rnzfno2++01jve8Y3dO+ebUJrbOmu36Odex27gxjNt1Z844/bRmN6dNCCnf0Qoa1rOW9HPgMGBFRLy9UddpFz/7yX8yesyuPP/88wDc/IebmHP1b7nxf+5k0KBBrFy5okYN1q7Wrl3L5086gauvvYFhw4ez/zv34bDDPszbxo5tdtMKr0XicC6N7FlfAExsYP1tY9kTS/ndnGs5+mPHvnZuxvk/5bP/9mUGDRoEwLbbbtes5lmTzZ83j5122plRO+7IwIEDOeIjk7nqt1fWfqNVpT4craBhwToi/gA806j628k3p36Rb55yKup4/X/Hww8t4tZbbuaQ9+/H4R88kLvuuL2JLbRmWrbsCYYPH/Ha82HDhvPEE080sUVtpEDRWhHRuMqlkcBV1dIgko4Hjk9PxwAPNKxBrenN6XgM2AwYAiwGdgP+CjwObALsBNzbpDZac20JbA68AKwCtgLeRPa7UVY7RMS2G1LBuN3Gx8VX/yFX2d233+yOiNh7Q663oZo+GiQizgXObXY7mkXSqcBHgW2AwcBA4H5gNPCRiLgplXsIOCQiVjapqdYkkt4FnAxsHRF7S5oGEBGnNrVhbcA5a8stIqZFxPCIGAlMBm6MiGOAvwHvB5C0C1kQX9W0hlozzSf74z1Q0kCy35PZzW1SG0jjrPMcraDpPWvr1SpgR0l/AVYDU6KROStrWRGxRtKJwOXAQuDnEbGgyc1qC0WawdiwnrWki4FbgDGSlko6rlHXahcRcVNEHJae/jQijomIt0fE+Ii4samNs6aKiGuAkyJip4iY3uz2tANRv561pJ9LWpE6V13ntpJ0g6RF6euWFa9Nk7RY0gOSDs7T3kaOBjkqIoZGxEbpY/75jbpWO0q5fLPX+Hei/uo4GOQC3jhUeSowNyJGA3PTcySNJUtljUvvOUdSZ60LOGdtZuVVp2jdy1DlScCM9HgGcHjF+ZkR8XJELCEb/TWh1jWcszaz0urD5gPbSKqc7HBujk86QyJiOUBELJfUNbNtGHBrRbml6VxVDtYtRtJE4EygEzgvIrwQRMl56YbG6cPtxVV1HGfd02VrDh5wGqSFpLzV2cAhwFjgqJTfsnK7AC/d0BiNncH4lKShAOlr1wI/S4ERFeWGA8tqVeZg3VomAIsj4uGIWA3MJMtvWYl56YbG6Np8IM9/62k2MCU9ngJcWXF+sqRBkkaRjaGfV6syp0FayzDWnUK8FHhHk9pi1t7qOOElDVU+gCy3vRT4NnAaMCsNW34MOAIgIhZImgXcB6wBToiItbWu4WDdWtYrl2Vm66deU2Ii4qheXjqwl/LTgT6Nl3ewbi3rlcsys/XROhsL5OGcdWuZD4yWNMprQJg1XpHWBnGwbiERsQY4EZhDtgbELK8BYV66oTGKtvmA0yAtJq0BcU2z22Gto0o+1DZUq0TiHByszay0irTqnoO1mZVWq+Sj83CwNrNyEnQ4WJuZFUFxorWDtZmVUtfmA0XhoXu2DklrJd0t6S+SLpW0yQbUdYCkq9LjD0uaWqXsFpI+sx7XOFnSl/Ke71bmAkn/0odrjazcCcSKr0hD9xysrbuXImKPtBTnauBTlS8q0+ffm4iYXWO51y2APgdrsw3hSTHWLv4I7Jx6lAslnQPcCYyQdJCkWyTdmXrgm0K2Hrek+yXdDPxzV0WSPi7prPR4iKQrJN2Tjn3JFr3ZKfXqz0jlvixpvqQ/S/pORV1fT3vX/Q4YU+ubkPTJVM89ki7r9mnhA5L+KOlBSYel8p2Szqi49v/d0B+ktSZJuY5W4GBtPZI0gGxd7XvTqTHALyNiT+AF4BvAByJiPHA78AVJg4GfAR8C3g28pZfqfwz8PiJ2B8YDC8j2p3so9eq/LOkgsqUjJwB7AHtJeo+kvcim4e9J9sdgnxzfzuURsU+63kKgcgbgSOC9wKHAf6Xv4Tjg2YjYJ9X/ybSUpbWZIqVBfIPRuttY0t3p8R+B84G3Ao9GRNdWRO8k2xzhT6nXMZBsOvSuwJKIWAQg6SLg+B6u8X7gYwBpachnK3d+Tg5Kx13p+aZkwXsz4IqIeDFdI8/aKW+X9D2yVMumZNP5u8yKiFeBRZIeTt/DQcBuFfnsN6drP5jjWlYQrZTiyMPB2rp7KSL2qDyRAvILlaeAG7pPg5a0B/Vb0lXAqRHx027X+Px6XOMC4PCIuEfSx8nWHe7Sva5I1/5sRFQGdSSN7ON1rcUVaQaj0yC2Pm4F9pO0M4CkTSTtAtwPjJK0UyrX25oWc4FPp/d2StoceJ6s19xlDnBsRS58WNpw9A/AP0naWNJmZCmXWjYDlkvaCDi622tHSOpIbd4ReCBd+9OpPJJ2kfSmHNexoilQHsQ9a+uziFiZeqgXSxqUTn8jIh6UdDxwtaRVwM1ATxu8fg44N60etxb4dETcIulPaWjctSlv/TbgltSz/ztwTETcKekS4G7gUbJUTS3fBG5L5e9l3T8KDwC/B4YAn4qIf0g6jyyXfaeyi68EDs/307EiaZE4nIsivBGJmZXPHuP3jhv/eFuusltvOuCOOu5uvl7cszazUvIMRjMzqzv3rM2stIrUs3awNrPSKtLQPQdrMysnT4oxM2t9RbvB6GBtZqXlNIiZWQEUqWftoXtmVlr1nG2elgd+QNLiahttrC8HazMrrzpFa0mdwNlkywqPBY6SNLaeTXWwNrNSEtAh5TpymAAsjoiHI2I1MBOYVM/2OmdtZqV05513zNl4I22Ts/hgSbdXPD83Is6teD4MeLzi+VLgHRvaxkoO1mZWShExsY7V9dT9rusqeU6DmJltuKXAiIrnw4Fl9byAg7WZ2YabD4yWNErSQLJ9QvNsOZeb0yBmZhsoItZIOpFsl6FO4OcRsaCe1/DmA2ZmBeA0iJlZAThYm5kVgIO1mVkBOFibmRWAg7WZWQE4WJuZFYCDtZlZAfx/wWI5ewAth1YAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "skplt.metrics.plot_confusion_matrix(y, yPredProb[:,1]>0.5,\n",
    "                                    normalize=False);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ugh!  Here's a different version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[630,   0],\n",
       "       [ 46,   0]])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_matrix(y, yPredProb[:,1]>0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, our model is correct almost all the time by just predicting `Biopsy` = 0.  Let's take a look at the ROC. AUC will be in the legend of this plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcIAAAG5CAYAAAAd/TRHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeZxP5f//8cdrFlv2yJosKZQikxYpYqbsSyEhIZLSquVT+tS3TeqnlfKRFJGSLR+GsSZb1g8hyb6TfTdjZq7fH+8xzZjFe8y85z3L8367za0551znvF/jRs+5zrmuc5lzDhERkdwqwN8FiIiI+JOCUEREcjUFoYiI5GoKQhERydUUhCIikqspCEVEJFdTEIqISK6mIBRJAzN72MxWmNkpM9tnZtPN7C4z62hm283MLmofZGZ/m1nzZK6Vx8wGmdnuuOttM7OP445FmNlbyZzTysz2m1lQ3HZdMws3s2NmdsTMlplZtxRqf9TMYuI+64SZrbm4LjPLa2YDzGynmZ01s01m9mIyP9d9ZvarmZ00s4NmNt/MWqb9T1TE/xSEIl4ys+eBT4D3gFJABeALoBUwCSgK3HPRafcDDpiRzCX/BYQAdYFCQEPgf3HHvgW6XBxAQBdgjHMu2szuAOYC84FrgSuBJ4AmqfwYS5xzBeNq/QL4wcyKJjj+E9AIaBpXUxegF/Bpgj+HB+PajQLKx/1Z/BtokcrnimRZpjfLiFyamRUB9gDdnHM/pdBmGBDknOueYN84YLdz7vlk2k8FZjvnPknmWH5gP9DCOfdr3L5iwD7gNufcGjNbCKxxzj3p5c/wKPCYc+6uuO0CwGmgrnNuuZk1AqYBVZ1zuxKcdxuwGLge2ALsAD53zn3ozeeKZHXqEYp45w4gH56eX0pGAg/GhdiF8GyBp+eUnN+A582sj5nVTNj7c86dBcYBjyRo3x74My4EC8TVNP5yfhgzCwS6AefxBBtAKLA0YQjG1bIU2I2np3g9cPXlfq5IVqQgFPHOlcAh51x0Sg2cc4uAA0CbuF3tgb+cc6tTOGUAMBDoBKwA9phZ1wTHRwLtLgQrnlAcGfd9MTz/fvel8ee43cyOAeeA/wd0ds79HXesRCrX2xd3/MoE2yI5goJQxDuHgRIXBqmkYhT/9OK68E9wJeGci3HODXHO1cPzzO5dYISZVY87vhA4CLQys8rArcD3cacfBWKBMmn8OX5zzhXFE6RTgPoJjh1K5Xpl4o4fTrAtkiMoCEW8swRPL6r1JdqNAhrFDWS5nX+CK1XOubPOuSF4Aq7GRdd7BE+oznTOHYhrfyaupgfS8kMk+LxTQB88A3Jqx+2eDdxmZlcnbGtmdfHcDp0LbAR2Xe7nimRFCkIRLzjnjuMZGTnEzFqbWQEzCzazJmb2QYJ2O4CFwFhglnNuf0rXNLNnzayBmeWPm2bRFc9Izf8laDYKaAz0JGnv8iXg0bjpDVfGXfNmM/vBy5/pMDA87ufCOTcbmANMMLMbzCzQzG4HxgBfOuc2Oc/ouueB182sm5kVNrOAuCkkw7z5XJGsRkEo4iXn3Ed4QqA/nluWu4CngMkXNR0JXEPKg2QuOAsMwjM69BDwJPCAc25rgs/cjmfE5hV4bmUmrGcxcG/c11YzOwIMA8LT8GN9AjQ1s5vith8A5uGZ7nEKGA18DfRN8LnjgQ5Ad2Avnuei7wA/p+FzRbIMTZ8QEZFcTT1CERHJ1RSEIiKSqykIRUQkV1MQiohIrnapycFZTokSJVzFihX9XYaIiGQhK1euPOScK3k552a7IKxYsSIrVqzwdxkiIpKFmNmOS7dKnm6NiohIrqYgFBGRXE1BKCIiuZqCUEREcjUFoYiI5GoKQhERydUUhCIikqspCEVEJFdTEIqISK6mIBQRkVxNQSgiIrmaglBERHI1BaGIiORqPgtCMxthZn+b2boUjpuZfWZmm83sdzO7xVe1iIiIpMSXPcJvgftTOd4EqBr31Qv40oe1iIiIJMtnQeic+xU4kkqTVsAo5/EbUNTMyviqHhERyXnuX3Y/Nt7SdQ1/LsxbDtiVYHt33L59Fzc0s154eo1UqFAhU4oTEZGs7eTJk0T0joCj6buOPwfLJBfhLrmGzrlhzrkQ51xIyZIlfVyWiIhkB126dIH/AcfTdx1/BuFu4OoE2+WBvX6qRUREspm3337bM8rkk/Rdx59BOAV4JG706O3AcedcktuiIiIiFxw//k/3r2bNmvAFnodq6eDL6RNjgSXA9Wa228x6mFlvM+sd1yQc2ApsBr4C+viqFhERyf5Wr17N9ddfz/Dhw//Zmb5xMoAPB8s45zpe4rgDnvTV54uISM6xcOFCmjdvzvHjx5kwYQI9evRg0KAl8HMHKHwG+O9lX1tvlhERkSwtPDycsLAwjh8/zoMPPsjkyZMxM+bM2QYLq0N4nXRdX0EoIiJZUrPVzbBXjWYtm3H27FloAuN7jiffwnzYHGPG5gUZ8jkKQhERyZLCvw+H94EYoAPwPBCYoMGnX8M1f6f7cxSEIiKSNVUHCsP777+P+8HhGjtcowRfTc/zw8An0v0xCkIREckyPOMo41QAHrqZxYuvSbw/gQcfrMHAgY3T9ZkKQhERyRKio6Pp1q0bn3/+uWfH+NvhP22YMmUjERFbkj0nMDCAl16ql67PVRCKiIjfnTt3jgcffJCRI0fy6quvsm7dNhjZMP74m2/+kmKvML0UhCIi4lcnTpygadOm/PzzzxQrVoyZM2fyzTd/wdm88W2WLt3D5s2pLWh0+fy5+oSIiORyhw4dokmTJqxYsYIyZcowc+ZMbrzxRm68MZKPDgyE8XdCZDCDBoVRteqVPqlBQSgiIn6xa9cuwsLC+PPPP6lcuTKzZs2icuXKADy0pS10mwctVvDiygk89VRdn9WhIBQREb+Iiori2LFj1KxZk4iICMqU+Wdt9vDD4QA0vb4+H3QI9WkdCkIREfGLKlWqMHfuXEqXLk2xYsWSbTOt1jSf16HBMiIikml+/fVXPvvss/jt6tWrU6xYMZxz/PLLdr/UpCAUEZFMMXXqVO677z6eeeYZZs+eHb9/376TtGz5Aw0bjmTChD8yvS4FoYiI+NyYMWNo3bo1586do1evXjRs6JkjOH36Jm688UumTv0LgN69p/H336cztTYFoYiI+NTgwYPp3LkzMTEx/Otf/2Lo0KEEBnrenl2uXGFOnoyMb3vo0BmefXZGptanIBQREZ9wzvHWW2/Rt29fAD788EPee+89zP5ZVv6mm0rxxhv3xG/fcUd5/u//GmRqnRo1KiIiPnH06FG++uorAgICGDZsGD169Ig/1mx1s/gpEtwaADW7wu0bWfLgEq7b+Vim1qkgFBERnyhevDizZs1iw4YNtGnTJtGx+BAECIyFQd9CQOJ3iTa9smkmVKkgFBGRDHT27Fn++9//0r59ewCqVatGtWrVUmzvGvnmRdppoSAUEZEMceLECVq2bMn8+fM5duwYvXr18ndJXlEQiohIuh08eJD777+fVatWUbZsWerVS98agZlJQSgiIumyc+dOwsLC2LhxI9deey2zZs2iYsWKKbYfP/4P+KQDXLeXmTFbuPXWshQrlj/zCr6Ipk+IiMhl+/PPP6lXrx4bN27k5ptvZuHChamGIMDcudtgUXX4phH33Teazz5bmjnFpkBBKCIil8U5x6OPPsru3bupV68ev/zyC6VKlbrkeStW7E20HRJS1lclekW3RkVE5LKYGWPGjOGdd95hX699FFuZ/AoSiZwPhNX/ImH8KAhFRCRb2bBhA9WrVwc8Syl988032By7xFlxAmLhs69hY1mu3nEbNc7cQalSBX1Y7aUpCEVExGujRo2ie/fuvP/++/Tr1y/J8awwLzCt9IxQRES88umnn9K1a1diYmI4fvy4V+c454iJifVxZemjIBQRkVQ55/j3v//Ns88+C8BHH33E22+/neo5R46cZdasLdx77yg+/dS/o0IvRbdGRUQkRbGxsTz99NMMGTKEgIAAvv76ax599NFLnte16+T4NQaXLdtD27bVqVixqI+rvTzqEYqISIpee+01hgwZQp48eZgwYYJXIQhQoULh+O/PnDnPk0+G41zWfH6oIBQRkRT17t2b6tWrM336dFq3bu31eRUqFEm0HR0dy+nT5zO6vAyhW6MiIpLI/b/dT8SpCLgwI+ITaOQawZyLGm4qDdfu/6ddAlWrXkn16iWoUqU4jzxyEw8+WCPRgrxZiYJQRETiHThwgIjOEVAf6By3M/CiRn+Uh+m1YXodaLEcngqHQJdo/cC2bavTtm31zCo7XRSEIiICwI4dOwgNDYUtQCSc/PwkBQsmneze84cpDJ/+P8/Gf2/lwbxdGT26DXnzZs9I0TNCERHhjz/+oF69emzatAnKBUPoLcmGIED9+tck2p44cQOLF+/KjDJ9QkEoIpLLLVu2jLvvvps9e/ZQtOj1sOcFGNWM48fPJdu+fv0K8d8XKBDMiBEtadiwUmaVm+GyZz9WREQyxMKFC2nSpAmnTp3C7DqOHXsQCIYYmDFjMx063JjknIoVi/LMM7dRq1Zp7ruvCmXKFMr8wjOQglBEJBerXLkyJUqUoGXLlhQs2J5hw1bHH5sy5a9kg9DM+OST+zOzTJ9SEIqI5CJnzpwnNtZRsGAeAMqWLcuSJUu46qqr2L//NKNGrePcuWi4aTtdu3a+xNVyBgWhiEgO02x1M8IPh/+zY0M5+LUGrL0GNpWBO76FCruge4KT1sf9t1cIlDsCt2wlrPE3mVi1/ygIRURymEQhCPB7RfipHuCAubAwboRnfaDqRSe3XAGQaE5gTqcgFBHJoS6sDbikwC7u/Go4EA6sAAL49tsRdO3a1Z/lZRkKQhGRHK5mzRIEBEwiNnYtEEjp0t1o2bK9v8vKMjSPUEQkBztz5gzt2z9AbOxa8ue/gvHjp7Bv31cUK5bf36VlGeoRiojkYKdPn2bLli2UKFGCGTNmUKdOHX+XlOUoCEVEcpDvv18Ls+tDUAyfrP2N1q2rMWvWLM6cOUO1atX8XV6WpCAUEclB/vOflfDrLcA6nmMGNWqUJCysir/LytIUhCIiOcj69euAr4FTQAEqVuzr54qyPgWhiEgOMWvWfA4fHgKcA64hKKgmlSsX83dZWZ6CUEQkB5g9ezZt2rQGzkGFYnD9jbQvWYegIE0OuBT9CYmIZHMTJkygWbNmnD59mi5dusCwo/DyNMaMaevv0rIFBaGISDYWHR3NW2+9RVRUFE8//TTffvut7vWlkf64RESysaCgIMLDwxk/fjxPP/00ZubvkrId9QhFRLIZ5xyTJ0/GOc+7RMuVK8czzzyjELxMCkIRkWwkJiaG3r1706ZNG/79738D8MMP6zh9Oopmq5thcxSGaaVboyIi2URUVBSdO3fmp59+Il++fNx2223MmbOVjh0ncM01Rdjx+F9Q19M2Ny2jlF4KQhGRbOD06dO0bduWmTNnUrhwYf773/9y6613cNNNQwHYseM4vNoZHv4VN2aOn6vNXhSEIiJZ3JEjR2jevDlLliyhZMmSREREULt2bYYNW8nmzUf+aWgObv/Lf4VmU3pGKCKSxT399NMsWbKEChUqsHDhQmrXrg3AY4/dwttvNyR+jEyL5VBjt/8KzaYUhCIiWdygQYNo2bIlixYt4rrrrovfHxBg9O9/N+HhnahduzR01y3Ry2EXht9mFyEhIW7FihX+LkNExKd27txJ+fLlCQjwrr8SG+sInOdp6xplr/+vZwQzW+mcC7mcc9UjFBHJYpYsWUKtWrV44YUX8LazEhCgaROXy6dBaGb3m9lGM9tsZq8kc7yImf3XzNaY2Xoz6+bLekREsrqZM2fSuHFjjh49yrZt24iOjk61/YW5g5o/ePl8FoRmFggMAZoANYCOZlbjomZPAn84524GGgCDzCyPr2oSEcnKfvrpJ5o3b86ZM2d49NFHGT9+PMHBwameE344PNG25g+mnS+nT9QFNjvntgKY2Q9AK+CPBG0cUMg87wUqCBwBUv/1R0QkB/rqq694/PHHcc7x3HPP8f/+3/9L8nxwz54TlC//MY8/Xofy5QtTqtQVUAmw3PlcMKP4MgjLAbsSbO8GbruozWBgCrAXKAR0cM7FXnwhM+sF9AKoUKGCT4oVEfGX7777jl69egHwzjvv8Oqrryb73tD77hsNwH/+sxKAwoXzwsTMqzOn8uUzwuRuWF/8K8t9wGqgLFALGGxmhZOc5Nww51yIcy6kZMmSGV+piIgfNW3alJo1a/LII//ioYeeSPHl2b17Jx4UWbVq8cwoL8fzZRDuBq5OsF0eT88voW7AROexGdgGVPNhTSIiWUJMTAwxMTEAXHnllbz11veMHp2PW24Zxk8/rU/2nHbtasSPDs2fP4j+/e/OtHpzMl8G4XKgqplVihsA8xCe26AJ7QQaAZhZKeB6YKsPaxIR8bvIyEg6dOjAU089hXOOMWN+p127ScTGOk6ciKR9+/H07z83yXmlShWkT58QPvigMRs3PkXr1uo3ZASfPSN0zkWb2VNABBAIjHDOrTez3nHHhwJvA9+a2Vo8t1Jfds4d8lVNIiL+durUKdq0acPs2bMpXLgwL730EkWK5EsyX7BevauTPf/zzzUqNKP59KXbzrlwIPyifUMTfL8XCPNlDSIiWcWRI0do2rQpS5cu5aqrriIiIoJKlSpRqRIMH96Sbt1+xgwGD25KkyZVU7xOs9XNkkybkMun1SdERDLB3r17CQsLY/369VxzzTXMmjWLqlX/CbtHH63FoUNnqFChCO3b35DqtTR3MGMpCEVEfGz79u00bNiQ7du3U6NGDWbOnEm5cuWStOvX7840XVdzBzOG3jUqIuJjxYoVo2jRotStW5f58+cnG4LiPwpCEREfK1KkCBEREfz44xSaNZvM/Pnb/V2SJKAgFBHxgRkzZtCnTx9iYz0vy7rqqqt47rlfWLZsD40ajWLQoMVerywhvqVnhCIiGeyHH36gS5cuREdH06BBA9q3b8/OnceZPPlPAGJiHP36zaJ48fx061bbz9WKeoQiIhlo6NChPPzww0RHR9OvXz/atWsHwKJFOxO1q1y5GJ063eSPEuUi6hGKiGQA5xwDBgzgtddeA2DAgAG8/PLL8e8NrVGjJC++eCeLF+9ixYq9tG9fgzx5AhNdQ/MD/UNBKCKSTs45+vXrx0cffYSZ8eWXX/L4448nanPzzaW5+ebSAERFxXD27Pkk10lLCGruYMZREIqIpNPZs2dZuHAhwcHBjBo1ikKFbmHp0t3cdlv5ZNvnyROYpDeYkOYHZi4FoYhIOhUoUIDw8HC+/TaCjz46yfLlY6lfvwLz5z+a4pJKknVosIyIyGU4efIkAwcOjF9KaeXKY7z44iaWL/esNrdgwU7mzt3mzxLFSwpCEZE0Onz4MI0aNeKVV17h9ddfB2DUqDVcPC3wzTfn+6E6SSvdGhURSYPdu3cTFhbGhg0bqFSpEj169ADgrbca0qLFdaxZc4CxY9cRHBzA55838XO14g3Lbm82CAkJcStWrPB3GSKSC23atInQ0FB27NjBjTfeSEREBGXLlk3S7sSJSAoXzpvm69scz/NEDZZJOzNb6ZwLuZxz1SMUEfHC8uUruffeUE6dOkpQUAWmT59N2bKlkm3rbQhq3mDWoGeEIiKX4JyjZctenDp1FKhCdHRnJk3anu7rJheCmh+Y+dQjFBG5hF9/3cH+/aFAAaABEMSHHy7m8cdDUp0P6C3dCvUv9QhFRFLwyy+/EB0dzd13X8OsWb249dZHuNB/aNOmGufORfu3QMkQ6hGKiCRjyJAh9O3bl65duzJixAgaN65Mo0aVGD/+D8qWLUS9ehX8XaJkEAWhiEgCzjnefffd+PmB1apVi387jJnRrt0N/ixPfEBBKCISJzY2lhdeeIFPPvmEgIAAhg4dSs+ePf1dlviYglBEBIiOjuaxxx5j5MiRBAQE0b372wrBXEKDZURE8KwfOHLkSMzyEBvbkTFjYvjzz0MZ+hnNVjfD5lj8l2QNCkIREaBUqUbAtTjXBajC2bPRdOo0kaiomAz7DM0bzJp0a1REcq3Dhw9TqFAh8uTJw4IFByhdujf795+KP37uXDT795+iQoUiGfq5mjeYtSgIRSRX2rVrF2FhYdSqVYvRo0fz3XdtADhy5Cxjx65l48bDDBzYmPz5g/1cqfiaglBEcp2NGzcSGhrKrl27CA4O5vjx4xQvXhyA4sXz8+CDNShVqqCfq5TMomeEIpKrrFq1ivr167Nr1y7uuOMO5s+fHx+CFygEcxcFoYjkGvPnz6dBgwYcPHiQsLAwunYdxOHDel6X2ykIRSRXWLx4Mffffz8nT56kfv2m7N7dnN69Z/Lqq3P8XZr4mZ4RikiuUKtWLUJCQihcuDzh4dcBRwD46ac/WLZsD3Xrlkv3Z2h9wexJPUIRydFiYjzzAAsUKEBERARTpozhllsSh17//nMz5LO8CUHNG8x61CMUkRzJOcdbb73FmjVrGDduHEFBQRQoUACAgQMbExr6HWbQqdNNvP12w4z9bM0TzFYUhCKS48TGxvLcc8/x2WefERAQwOLFi7n77rvjjzduXJnXX7+btm2rU6tWaT9WKlmBglBEcpTz58/TvXt3Ro8eTZ48eRg7dmyiELzgrbcythco2ZeCUERyjLNnz9K+fXumTp3KFVdcQffuA2nVqrW/y5IsToNlRCRHOHHiBPfffz9Tp06lePHiNGnyNp9/foj69b9h48aMXUVCchYFoYjkCHny5CEoKIiyZcvy4YdjGT/+BABLluymVq3/MG/eNj9XKFmVbo2KSI6QL18+Jk+ezOHDh3nggZmJjpUvXzhd8wQ1PzBnU49QRLKtP//8k8cee4yoqCgAChUqRMWKFZk6tSMtWlwHgBl8800rrrgiz2V/TlpCUPMEsx/1CEUkW1qxYgVNmjTh0KFDVKxYkf79+8cfK1OmED///BCjR//O5s1HuOuuChnymZofmDMpCEUk25k3bx4tW7bk1KlTNG3alOeffz5JGzOjS5eb/VCdZDe6NSoi2crPP/9MkyZNOHXqFA8//DCTJ0+Of2OMyOVQEIpItjFq1CgeeOABIiMj6dOnD9999x3BwVpBXtJHQSgi2YJzjnHjxhETE0P//v0ZPHgwAQEBbN58hM6dJ7J79wl/lyjZlJ4Riki2YGaMGzeOn3/+mY4dO+Kc46uvVvL88zM5dSqKqVP/4pNP7qdr15sxs8v6DE2TyJ3UIxSRLCs2NpbBgwdz7tw5wLOUUseOHQE4eTKKN9+cz6lTnqkTx49H0q3bz3z77erL/rzUQlDTInIu9QhFJEs6f/48Xbt2ZezYsSxatIixY8cmOl64cF6GD29B06bfx+9r0eI6Hn64Zro/W9Mkchf1CEUkyzlz5gytW7dm7NixFCxYkF69eiXbrkmTqvTseQtXXBHMl1824+efHyJvXv1+L2nj1d8YM8sDVHDObfZxPSKSyx07dowWLVqwcOFCrrzySqZPn06NGinPBxw0KIyXX65HlSrFM7FKyUku2SM0s2bAWmBW3HYtM5vk68JEJPc5cOAADRs2ZOHChZQrV44FCxYwefIJQkO/IyYmNtlzChXKqxCUdPHm1uhbwG3AMQDn3GrgWl8WJSK504ABA1i9ejVVq1Zl0aJFzJt3ivfeW8iSJbsZNGiJv8uTHMqbIDzvnDt20T49SRaRDPf+++/z1FNPsXDhQiZN2seTT/4zivP11+exbt3ffqxOcipvnhFuMLP2QICZVQKeAX7zbVkiklusWbOGqlWrUqBAAfLly8fnn38OwEMPFeDDDxezd+9JAKKiz1Nz2p1wYIs/y5UcyJse4VNAHSAWmAicwxOGIiLpMnv2bOrVq0e7du04f/58omOlSxdkwoT2EBwN+aLgtfEQ4vsQ1HzB3MebHuF9zrmXgZcv7DCztnhCUUTkskycOJGOHTsSFRVFsWLFkm1z++3l4aVJcM1BXM8DmVyh5Bbe9Aj7J7PvtYwuRERyjxEjRtCuXTuioqLo27cvo0aNSvnl2Q3XQ2U9GxTfSbFHaGb3AfcD5czsowSHCuO5TSoikmaDBg2iX79+ALzxxhu89trrzJ+/g4YNK/m5MsmtUusR/g2sw/NMcH2Cr5lAE9+XJiI5zbhx4+JD8NNPP+XNN9/kjTd+4d57R/HCCxGcPx/j5wolNzLnUp8JYWb5nHPnMqmeSwoJCXErVqzwdxkichmioqJo164dDz74IF26dGHmzC3cd9/o+ON33nk1P/3UjrJlC8XvszmelST0/k9JjZmtdM6FXM653gyWKWdm7wI1gHwXdjrnrrucDxSR3CUqKoqoqCgKFixInjx5mDx5MmbGuXPRdO/+c6K227YdJTDw8pZQErlc3gyW+Rb4BjA8t0THAT/4sCYRySHOnDlDq1ataN26NZGRkQDxawXmyxfEiBGtKFfO0/sLCDAqvLWc0usKYXMs/kvE17wJwgLOuQgA59wW51x/oKE3Fzez+81so5ltNrNXUmjTwMxWm9l6M5vvfekikpUdPXqU0NBQZsyYwZo1a9i+fXuSNmFhVVi79gk6darJyy/XY2mlH5O9lub2iS95c2s00jy/wm0xs97AHuCqS51kZoHAECAU2A0sN7Mpzrk/ErQpCnwB3O+c22lml7yuiGR9+/fv57777uP333/n6quvZtasWVx//fXJti1WLD+jR7clNtYxYJ5nn54HSmbypkf4HFAQeBqoB/QEuntxXl1gs3Nuq3MuCs/t1FYXtXkYmOic2wngnNNkIZFsbtu2bdx11138/vvvVKtWjUWLFqUYggkFBOg2qPjHJXuEzrmlcd+eBLoAmFl5L65dDtiVYHs3nlUsEroOCDazX4BCwKfOuVEXX8jMegG9ACpUqODFR4uIP+zYsYN69eqxb98+6tSpw/Tp0ylZsqS/yxJJVao9QjO71cxam1mJuO0bzGwU3r10O7lf7y6+3xGE5z2mzYD7gNfNLMloVOfcMOdciHMuRP+oRLKucuXKUbduXRo0aMDcuXMTheCWLUd4//2FREfrfRyStaT2ZpkBwAPAGqB/3GK8zwADgd5eXOdCz74AACAASURBVHs3cHWC7fLA3mTaHHLOnQZOm9mvwM3AX17/BCLid845zIygoCB++MEzqDxfvvjZVsTExPLII5NZvHgXP/+8kVGjWlO16pX+KlckkdR6hK2Am51z7YAw4HWgvnNukHPujBfXXg5UNbNKZpYHeAiYclGbn4H6ZhZkZgXw3DrdkOafQkT8ZsKECYSGhnLmjOd/C/ny5UsUggAffriYxYs9T0p++203tWr9hw0bDmZ6rSLJSe0Z4Tnn3FkA59wRM/vTObfR2ws756LN7CkgAggERjjn1seNPMU5N9Q5t8HMZgC/43l/6XDn3LrL/mlEJFMNHz6cxx9/nNjYWL7//nsee+yxJG02bTrMv/89L9G+MzXXU2PPVUnvEYn4QWpBWNnMLiy1ZEDFBNs459pe6uLOuXAg/KJ9Qy/a/hD40OuKRSRL+OCDD3j5Zc/qbG+99RY9evRIse2//30Pr8/4Av4qC5HB8MKU5EcRoDmDkvlSfNeomTVK7UTn3ByfVHQJeteoiH855/jXv/7FwIEDARg8eDBPPvnkJc+78JaY/TeepFSpgj6tUXIfn7xr1F9BJyJZV0xMDE888QRfffUVQUFBjBw5kocffjhN11AISlbjzZtlRETinThxgnz58jF+/HiaNWvm73JE0k1BKCJeCwwMZNSoUaxfv57atWv7uxyRDOHNK9YAMLO8vixERLKmI0eO8MQTT3Dy5EkA8uTJoxCUHOWSQWhmdc1sLbApbvtmM/vc55WJiN/t3buXe+65h6FDh9K3b980nRsVFcP3369l167jPqpOJGN4c2v0M6A5MBnAObfGzLxahklEsq8tW7YQGhrKtm3bqFatGu+8847X50ZFxXDVVR9y/HgkefMGctNNpaBUM+ihMXiS9XhzazTAObfjon0xvihGRLKGtWvXctddd7Ft2zZCQkJYsGAB5ct78659jzx5Aqle3fOe0cjIGJYv3wsRtSHfeV+VLHLZvAnCXWZWF3BmFmhmz6J3gYrkWIsXL+buu+9m//793HvvvcydO5cSJUokaTdjxmZOn45K8Tr33HNN4h1NV0GwfoeWrMebIHwCeB6oABwAbo/bJyI50OjRozl27BitW7dm2rRpFCpUKEmbv/46TPPm31O16ud89dXKZFeUuOeeayhbthAdO97ImDFt4cnpmVG+SJql+GaZ+AZmxZ1zRzKpnkvSm2VEfCs6Opphw4bRq1cvgoKSH0bQrt1PjB//R/x206ZVmTYt8cT62FiHGZh53ihz4c0yWn1efCE9b5bxpke43MzCzayrmSX91VBEsr1x48Zx/LhndGdQUBB9+vRJMQSXLt2dKAQBHnnkpiTtAgIsPgRFsrJLBqFzrgrwDp4FdNea2WQze8jnlYmIzznneP/99+nQoQMtWrQgOjr6kudcf30JXn65HvnyeYKyTp0ytGt3g69LFfEZrybUO+cWO+eeBm4BTgBjfFqViPicc46XXnqJf/3rX5gZDz/8cIq9wISKFs3H++83ZtOmvnTvXosPPgglIEA9P8m+Lvm33swK4lmk9yGgOp7FdO/0cV0i4kMxMTE8/vjjfP311wQFBfHdd9/x0EOXvtHTbHUzwg8nWFntYRjhAE0PlGzMmwn164D/Ah845xb4uB4R8bHIyEg6derEhAkTyJ8/PxMmTKBJkyZenZsoBC+D1hqUrMibIKzsnEs6NlpEsqVhw4YxYcIEihQpwrRp06hXr16ar6GRn5KTpBiEZjbIOfcCMMHMkvyt92aFehHJevr06cPGjRvp2bMnN9988yXbnz17nnffXcCrr9bPhOpEMl9qPcIf4/47ODMKERHf2bt3L3nz5uXKK68kMDCQwYO9+2d9/nwM7duPZ+rUv/j11x3wQl4oGOnjakUyV4qjRp1zy+K+re6cm5PwC8+gGRHJBjZv3ky9evVo1qwZp06d8vq82FhHt24/M3Wq542KCxbshH6PwtErfFSpiH94M32iezL7emR0ISKS8dasWcNdd93F9u3bcc4RFZXyu0Evdvz4OX7//UDinTEBEKT3hUrOkmIQmlkHM5sEVDKziQm+ZgHHMq9EEbkcixYt4p577uHAgQM0atSIOXPmULx4ca/PL1YsP/PnP8odd3hWnahcuRi8/x0UOuerkkX8IrVnhMuAw0B5YEiC/SeB//myKBFJn+nTp/PAAw9w9uxZ2rZty/fff0/evHkveV6SeYIArwbDZ83Y2mU+XOn9rVWR7CLFIHTObQO2AbMzrxwRSa+VK1fSsmVLoqOj6dGjB0OHDk31jTETJ27g998P8OabDZKfJ5j/PLw8OX5TcwElp0lt+sR859w9ZnYUSDh9wgDnnPP+HouIZJratWvTqVMnrrrqKgYOHJjii6+dcwwYsJDXXpsLQKVKRT33f9A8QcldUlyGycwCnHOxZhaY3HHnnF+emGsZJpGknHOcPn2aggULAhAbG0tAQMpj4WJiYunefQqjRq2J3xccHMD5D4ZDzZ0KQsl2fLIMU4K3yVwNBMYF3x3A44DGT4tkEc45+vXrR/369eOXUkotBAG+/35tohD0XAfYV8xXZYpkWd5Mn5gMODOrAozCM4fwe59WJSJeufAc8KOPPmL9+vUsW7bs0icBnTrdxK5dz/Hrr48ycmRr7rijPDNndoawNZc+WSSH8eZdo7HOufNm1hb4xDn3mZlp1KiIn507d46OHTsyefJkChQowMSJEwkNDfXq3IAAo3z5wpQvX5hbby3H3XdfQ8WKRbWKhORK3gRhtJm1A7oAreP2BfuuJBG5lJMnT9K6dWvmzp1L0aJFmTZtGnfeeXmro+XLF+QJQZFcypsg7A70wbMM01YzqwSM9W1ZIpKSU6dO0ahRI5YvX07p0qWJiIjgpptuSvN1kp0zKJILXTIInXPrzOxp4FozqwZsds696/vSRCQ5V1xxBbfccguHDx9m1qxZVK5cOc3XcM6lGIKaJyi5jTcr1NcHvgP24JlDWNrMujjnFvm6OBFJyswYMmQIR48epUSJEmk69/z5GL75ZjVffbUK3giG/Oc1VUJyPW9GjX4MNHXO1XPO3Qk0Az71bVkiktD//vc/7r33Xg4fPgxAYGBgmkNw3Lj1VKs2hMcfn8qKFXvh57q+KFUk2/EmCPM45/64sOGc2wDk8V1JIpLQggULaNCgAfPmzWPAgAGXfZ01a/azdevRf3b8eBecuvT7R0VyOm+CcJWZ/cfM7or7+hK9dFskU0ybNo2wsDBOnDhBu3btePfdy38836/fnRQpkiD4zgXDH1dnQJUi2Zs3Qdgb2AK8BLwMbMXzdhkR8aExY8bQunVrzp07R8+ePRk7dqxXK0ikpFix/Lzwwh0EBQXQu3cdGPUZ1N2cgRWLZE+pDpYxs5pAFWCSc+6DzClJRAYPHkzfvn0BeOWVV3jvvfdSfHn2xVKdFlErD3xTgKFltKSoyAWpLcz7Kp7Xq3UCZplZcivVi4gPbN26FYCBAwcyYMAAr0MQIHxfRMoHC0RBghDUVAmR1FefWA/Udc6dNrOSQLhz7tZMrS4ZWn1CcoPY2Fh+/fVXGjRokKbznHMEVOwLZY4y8eX+NG9+HcHByS4gI5Kj+GT1CSDSOXcawDl38BJtRSQdoqOjefXVV/n7778Bz+oRaQ1BgIULd8LOkrD0Otq2HUeVKp8RFeWXFdNEso3UnhFWNrOJcd8bUCXBNs65tj6tTCSXOHfuHB06dGDKlCksXLiQ+fPnp+lWaEIjRyZePeL228uTJ496hCKpSS0IH7hoe7AvCxHJjU6cOEGrVq345ZdfKFasGB9++OFlh6BzjqVL9yTa17177YwoUyRHSzEInXNakEXEhw4ePEiTJk1YuXIlZcuWZebMmdxwww1enx8ZGU10dCz58gURGBiAmbF69eMEDa4Ii6pTZ+sDNGpUyXc/gEgOoed+In6wc+dO6tevz8qVK6lSpQoLFy70OgRXrdpH48ajuOKK9yhYcACrVu2LPxYYGAA37oLHZ7JiRS8NlBHxgjfLMIlIBhs7diwbN27kpptuIiIigtKlSydpk+x8wFWV4OVHwP1z+7TusrpwYq+vSxbJsbwOQjPL65yL9GUxIrnFSy+9RHBwMN26daNYsWLJtkl2UvxNO+Cag7D9qlSvr/mBIt7zZhmmusDXQBGggpndDDzmnOvr6+JEcpKFCxdSuXJlypYti5nx/PPPe3XexcskzRuxjXvvHRW/veL25dSpUzZDaxXJTbx5RvgZ0Bw4DOCcWwM09GVRIjnNlClTaNy4MWFhYRw/fjzVtpGR0ezefSLF4w0bVuKll+4kPPxhnHtDISiSTt7cGg1wzu24aEi3ZuiKeOm7776jW7duxMTEUL9+fQoWLJhq+wULdhIa+h1c/RTcspU5bKVRo8Sr0A8cGOrLkkVyFW96hLvibo86Mws0s2eBv3xcl0iO8Omnn/LII48QExPDa6+9xhdffEFgYOojOWfN2uL5ZlcJ+LkuP/ywLhMqFcm9vAnCJ4DngQrAAeD2uH0ikgLnHG+++SbPPvssAIMGDeKdd9655GR55xwzZmxJtK9x48optBaRjHDJIHTO/e2ce8g5VyLu6yHn3KHMKE4ku4qIiOD//u//CAgIYMSIEV4PjAF46aU7KV36n9un996rSfEivuTNqNGvgCRLVDjnevmkIpEc4L777uPFF1/kjjvuoE2bNl6f13xNc8JLh8N/8sLoe2B3cUqWvMKHlYqIN4NlZif4Ph/QBtjlm3JEsq+zZ89y9OjR+OkRH3yQ9rWs4+cOXhEJj8+kSTHNBxTxtUsGoXPux4TbZvYdMMtnFYlkQ8ePH6dFixYcOHCABQsWcNVVqU94Tyg21hEQkPjZ4cVzB0XEdy7nXaOVgGsyuhCR7OrAgQM0aNCABQsWcPr0aY4cOeLVeTt3HqdNmx95++35Pq5QRFLjzTPCo/zzjDAAOAK84suiRLKLHTt2EBoayqZNm7j22muZNWsWFStWTPWcqKgYPv54CW+99Stnzpxn+vRNdOp0E9deWzxzihaRRFLtEZpnrPfNQMm4r2LOucrOuXGZUZxIVrZhwwbuuusuNm3aRK1atVi4cOElQxDgzJnzfPTRb5w5cx6AyMgYnnoqHOd0O1TEH1INQuf5lznJORcT96V/qSJ4bofWr1+f3bt3c9dddzFv3jxKlSrl1blFi+Zj0KCwRPuOH4/k+HG9017EH7x5RrjMzG7xeSUi2UipUqXo2bMnTZs2JSIigqJFi6bp/E6datKgQUWKFcvHsGHNWbSoO0WL5vNRtSKSGkupk2dmQc65aDNbC1QHtgCnAcPTWfRLOIaEhLgVK1b446NFiIyMJG/evIDnLTAxMTG0Wtcq6ZJJBwvDqAYQthpq7kz+YgeKQN7zUPRMkkMaNSqSNma20jkXcjnnpjZYZhlwC9D6sqoSyWG+/fZb3nvvPX755Zf4uYJBQUGJQ/BkPvjhLph0G0QFw/aS8NnXnl8fL1Yq+VUotJagSOZKLQgNwDm3JZU2IrnCxx9/HP+atMmTJ9OnT58kbVwjx6JFO7nrx2/+2bnhaiae/IM2bapnVqkikkapPSMsaWbPp/TlzcXN7H4z22hmm80sxSkXZnarmcWY2YNp/glEfMg5R//+/eND8OOPP042BC+oV68CrVpdn2jft9+u8WmNIpI+qQVhIFAQKJTCV6rMLBAYAjQBagAdzaxGCu0GAhFpLV7El2JjY3nyySd59913CQwMZOTIkfGrSaTmvfcaERBglClTkGHDmjNhQvtMqFZELldqt0b3OefeSse16wKbnXNbAczsB6AV8MdF7foCE4Bb0/FZIhkqNjaWzp07M3bsWPLmzcu4ceNo2bKlV+fWqFGSKVMeomHDShQoEOzjSkUkvVLrEaa+cNqllSPxy7l3x+375wPMyuF5iffQ1C5kZr3MbIWZrTh48GA6yxK5tICAAKpVq0ahQoWYMWNGkhCcPn0TBw+eTvH8Zs2uUwiKZBOpBWGjdF47uSC9eEz4J8DLzrmY1C7knBvmnAtxzoWULFkynWWJeOf1119n3bp1NGjQINH+lSv30qbNj9SpM4zly/f4pzgRyTApBqFzzrs3B6dsN3B1gu3ywN6L2oQAP5jZduBB4Asz03QN8YsDBw7QokULdu70zPszMypUqJCozZkz53nggXFERsawa9cJ6tYbCjNq+aNcEckg3qxHeLmWA1XNrBKwB3gIeDhhA+dc/NLbZvYtMNU5N9mHNYkka/v27YSGhrJ582bMjClTpiTbbty49ezYkWD+3/kgCI7R3D+RbOxylmHyinMuGngKz2jQDcA459x6M+ttZr199bkiafXHH39Qr149Nm/eTO3atRk+fHiKbfPmDaRWrdLx21263IR773em1ZqWGaWKiA+k+Iq1rEqvWJOMtGzZMpo0acKRI0e4++67mTJlCkWKFLnkeTauEKyswp/dZ3P99SUyoVIRSU16XrHmsx6hSFY3e/Zs7r33Xo4cOULz5s2ZMWOGVyEIwJWnIGyNQlAkB1AQSq61Zs0aTp8+TefOnZk4cSL58+dP0iYyMprjx8/5oToRySwKQsm1XnjhBaZMmcLIkSMJDk485y821vH992upXn0Ir7wy208VikhmUBBKrvLFF1+wZcs/75Fv0aIFAQGJ/xns2HGMkJBhdOo0kW3bjvHVV6vYuPFQZpcqIpnEl9MnRPyu2epmnmWSHPAN8D1QFhgO5EnhpPOBcOBJoDgAMTGOao89CW+Oy4ySRSSTqUcoOVr44XCIAT7FE4IBwCNAVD5Ydm3yJwXHQLc5/2wHxkCJkxCT+GVJmjsokjOoRyg523k8a5v8Annz5qN16/7snF+apR/sITbWsW3bM1SsWDTJabENHbfPHE7lysV45517ufba4plduYhkEgWh5FinT5+G14EVUKhQYaKjO/Djj9F43v7nMWvWFnr2rJPk3IAAY/78R8mfXy/OFsnpdGtUcqxp06bBCqAozJw5iylTXuHZZ2+jatV/enezZm1N8XyFoEjuoDfLSI5mTxrcCu7Rf/6e799/ijvv/JoCBYJ58MEavPlmA/8VKCIZIj1vltGtUclRtm3bRlRUFNdff71nR9ukbXbsOMaCBd0oV65w5hYnIlmSglByjHXr1hEWFkZQUBCLFi3i6quvTrbdbbeVz+TKRCQr0zNCyRF+++037r77bvbt20eVKlXosaMHNie5taFFRBJTj1CyvVmzZtG6dWvOnDlDy5Yt+fHHH8m/6J/3hmq+n4ikRj1CydbGjx9Ps2bNOHPmDF27dmXChAnky5cv/vjyIntoNLc/GzceIrsNDBORzKEglGzrr7/+okOHDpw/f55nn32WESNGEBSU+CbH6NG/88ILM6lWbQhVq37ODz+s81O1IpJV6daoZFvXXXcd77//PpGRkbz22muYJX0mOG3apvjvt2w5St68gZlZoohkAwpCyVacc+zbt4+yZcsC8OKLL6bceHdxNm8+Er8ZHBxA48aVfV2iiGQzujUq2UZMTAy9evWiTp06bN2a8hth4hU6xyef3EdoaGXy5AnknnsqUqhQXt8XKiLZinqEki1ERkbSuXNnxo8fT758+diyZQuVK1+id1fkDM+0vZ1nnrmdU6ei+Pvv05lTrIhkKwpCyfJOnTpFpdBKHPrtEBSAc++eIywgDOZc+twLChbMQ8GCKS1AKCK5mW6NSpZ25MgRQkNDPSFYFBgE1PTuXM0fFBFvqEcoWVZkZCQNGjRg7dq1UAoYCK6r5gKKSMZSj1CyrLx589KzZ0+qV68OnwBeviJUE+dFJC0UhJLlREdHx3/ft29fVq5cCSW9Ozc21nHHHV/zySe/cfp0lI8qFJGcREEoWcrixYupUaMGf/75Z/y+/Pnzp3JGYsuW7WHp0j0891wEFSp8wocfLvJFmSKSgygIJcuYMWMGjRs3ZtOmTXz++eeXdY1JkzbEf3/kyFlWrNiXUeWJSA6lIJQs4ccff6Rly5acPXuWbt268emnn6b5Gs45Jk36M9G+Nm2qZVSJIpJDWXYbWBASEuJWrFjh7zJyvWarmxF+ODxjLjYV+BRwQDugF5DCUoJn651n7txtTJmykTfeuIcyZQrFH3POsXTpHiZN2sCkSX+yffsxDh58kSJF8iV/MRHJMcxspXMu5LLOVRDK5ciwRW/HAl/Hfd8d6EiKIVjl+17s++Eazpw5D8CwYc3p2bNOsm2dc+zadYIKFYpkTJ0ikqWlJwg1j1DSxTVK3y9SX239it7f9GbIkCH07t071bbvLPmV18/Mi9+eMuWvFIPQzBSCIuIVPSMUv+rZsyfr16+/ZAgCtGx5faLt2bO3xvcORUQul4JQMlVkZCQ9evTwvC0mTrVq3g1oqVnzKq65pghlyxaid+86TJzYnuBg/RUWkfTRrVHJNCdPnqRNmzbMmTOHJUuWsHbtWgIDvV8o18xYtKg7ZcsWSnYRXhGRy6EglExx+PBhmjZtyrJlyyhVqhRjx45NUwheUK5cYR9UJyK5mYJQEsnQaRFx9uzZQ1hYGH/88QcVK1Zk1qxZXHvttameExvrmDRpA0uX7uGDD0IztB4RkYQUhJJIWkLQm2WONm3aRGhoKDt27OCGG24gIiKCcuXKpdjeOcf06Zvp338u//vffgA6dLiBOnXKel2XiEhaKAglWemdFnHBqlWr2LFjB7fddhvh4eEUL178kue88cYv8SEI8Prr8wgP75Qh9YiIXExD7sSnOnTowKRJk5g9e7ZXIWhmvPNOw0T7IiK2sGnTYV+VKCK5nIJQMtz06dM9SyfFad26NQULFkzU5uzZ85w7F33xqQCEhVWhfv0KALRrV4N1656gatUrfVewiORqujUqGWrs2LE88sgjFC1alNWrVyd5Hrh//ym++GI5X365gjfeuIennqqb5BpmxmefNcE5R+3aZTKrdBHJpRSEkmG+/PJLnnzySZxzdOvWjbJlEw9wmTDhDzp2nMD587EAfPrpUvr0uZWAgKRzAmvVKp0pNYuI6NaopJtzjnfffZc+ffrgnOP999/ngw8+SDLp/c47r060vXnzEaZN+yszSxURSUI9wlzEF3MEY2Nj6devHx9//DFmRu/eb3L99a04fTqKK67Ik6htmTKF6NixJqNGrQGgSJG8HDhwOkPrERFJKy3DlIt4u3RS0yubMq3WNK/aLlu2jNtvvx3nAoA2wI0AbN7clypVko4S/d//9tG+/XieeeY2Hn20FgUL5knSRkQkrbQMk6RJRs0RBKhbty7Dhw/ntdeWs3//pZ/r1a5dho0bn0r2uaCIiD/oGaGk2cmTJ1mzZk38dvfu3SlQoLrX5ysERSQrUY9QvHbuXDSnTh2jSZMmbNmyhfnz51OzZk0AQkMrs3//qfi2BQoE+6tMEZE0URDKJTnnGD58Ff37T6ZQoR/ZsmUTlStX5oorrohvM3Rocz9WKCJy+RSEkqq9e0/y2GNTmD79N+A7/v77ODfeeCMzZ86kTBlNdheR7E9BKClyztG580TmzVsMjAbOAOV56KH/pxAUkRxDg2VygWarm3k9dSIhM2PEiPsIChqDJwSrUKBAD0qVKpnhNYqI+IuCMBdIOInemzUEE6pYsTRDh35OuXJ3UK/eS/z++9M89tgtGV2iiIjfaEJ9LnChN5iW+YOHDh2iRIkS8dvR0TGYGYGB+t1JRLKe9Eyo1//VJInPP/+ca6+9llWrVsXvCwoKVAiKSI6kwTICQGRkNP36zSRPnoV89NEAAJYsWcItt+g2qIjkbApC4fjxc7RuPZZffvkPsJSAgACGDRtGjx49/F2aiIjPKQhzuSNHztKw4Qh+/30Y8DsQyHXX9ebhhx/xd2kiIplCD31yucKF83Do0Hd4QjAYeBjnqnH06Dk/VyYikjkUhDnIhfmCF3+lJigokA8+eJbg4EJAV26//R4WLepO2bKFMqdoERE/UxDmIKktutvItUq0nXDaTKdOD7F582aeeeYBZs/uwpVXFvBZjSIiWY2CMAdyjVyir41XH2Jxy1t54ompHDt2jp07dxISEsKSJUviz6lQ4So++eT+JKvKi4jkdArCHM45R+/eUzl7NpqhQ1dSteob1KlzO6tWreLVV18lu71QQUQko/k0CM3sfjPbaGabzeyVZI53MrPf474Wm9nNvqwnNxo1ag3z5m2P29rLoUODOXRoH/Xq1WPSpEmYaZFcEcndfBaEZhYIDAGaADWAjmZW46Jm24B7nHM3AW8Dw3xVT25VqVIxqlYtDmwHvgXO0KRJE2bOnEnRokX9WpuISFbgyx5hXWCzc26rcy4K+AFINGLDObfYOXc0bvM3oLwP68mV7r77GgYMuJrAwDFAFM2bt2Xy5MkUKKABMSIi4NsgLAfsSrC9O25fSnoA05M7YGa9zGyFma04ePBgBpaYOwQHBwCx9OjRi8mTx5EnjwbEiIhc4Ms3yyT38CnZkRlm1hBPEN6V3HHn3DDibpuGhIRodEecZqubpThlwjkX//yvZcuWLF++nFq1aumZoIjIRXzZI9wNXJ1guzyw9+JGZnYTMBxo5Zw77MN6cpxEIXgqL/w3hHIfP8W1135Gly7P8uuvv8Yfrl27tkJQRCQZvuwRLgeqmlklYA/wEPBwwgZmVgGYCHRxzv3lw1pytNUl9tGkyxj27Tv1/9u79/ic6/6B46/PNmyLkFNqaUTGTk4r50OY5VQhcijc90jqFv0cKlK/qFTuu5NT7jtREVHDL7odolqynBrmLJSxDGM2jB3evz+uy7fNrtllbbt2eD8fj++D7/U9va/Pru19fb6Hz5sTZAAL+fXXLaxe/SlHjx6lYsWKrg5RKaWKrAJLhCKSZox5BlgDuAPzRGSPMWaEffkcYDJQBZhl762k5bWwYmnm5VWGypW9iItLBJYDuzHGg3nz5mkSVEqpXGiF+mIsc+X506cTCQ5+gLi4HUBZvL2fyBH0AQAAIABJREFU4MKFD7WYrlKqVNAK9aVcYmIivXv3IC5uB97etzJnzhfEx8/UJKiUUk7QeoQlwM6dO4mKiuLOO+9k7dq1NGx4/bgFSimlcqKJsARo27YtX375JYGBgfj6+ro6HKWUKlb03Fkx1C26G2aegV/+fK1Hjx6aBJVSKg+0R1gMrd60Gp43kCq0/MjhGARKKaWcpD3CYmbVqjXwrDskC1ypx0+D2lKhwhvs3PmHq0NTSqliSRNhMbJ8+XJ69eoJ6elAILYxCsqSnHyVSpU8XRydUkoVT5oIi4mPP/6Y3r17c/XqVehUDsr2wDZOATRv7sPdd2tJJaWUygtNhMXAyZMnGTlyJBkZGUyePBkmXIEnvuf228vzwgutWbt2kKtDVEqpYktHliliDhw4w3//e5hDhxJo2fIuBgwIBGDFihUcO3aMZ5991jaiTLobVztcpUwZdxdHrJRSrvdXRpbRu0aLkJkzt/DMM9dKMmbw6adr6dz5NapVu4WHHnoo68ruGZoElVIqH+ip0SKkZ8/6eNa4AqQBX3Hhwkyqdw3DfGuyTEoppfKPJsIi5K67KpLy8lxwXwTEAG6wozb8kf1GmK5VuhZ6fEopVRLpqdEi5Pz58zDzLKSfxcurIi+8MIe//70bd9zxiatDU0qpEksTYRHxxx9/EBYWBnuAarDjhyj8/PxcHZZSSpV4emq0CMjIyCAsLIydO3eCD/AumgSVUqqQaCJ0kfPnUzh79hIAbm5uvPnmmzRv3hzeAWq4NjallCpNNBG6yIIF0VStOpVq1d6mbduP+eOPGmzatAkquzoypZQqXTQRusj69euBdzlzZjeRkb8TH38RNzf9cSilVGHTv7wu8OWXX7Jq1f8Cl4G9AIy/8jd9RlAppVxAE2Eh++ijj+jbty8iaXh5tQK6gRFoEGuto88IKqVU4dHHJwrR9OnTGTduHACvvPIKkya9hMecu+BwTaT3RRdHp5RSpZMmwkIyZcoUW+UI4P333+cf//iHbYHfSduklFLKJTQRFpI2bdpQvnx5Zs+ezaBBWjZJKaWKCk2EBUhEMMZ2A0z79u05evQoVatWdXFUSimlMtObZQrIxYsX6dGjB6tWrbJe0ySolFJFj/YIC0BCQgLdu3dn8+bNxMTE0KlTJ8qVK0dk5G+kpmbQoYMvxhi6RXdzdahKKVXqaSLMZ3FxcYSGhhITE0OtWrVYu3Yt5cqVA2DChPVs3hyLv381/vGP+1h95zrw0scllFLKlfTUaD46cuQIrVu3JiYmBj8/P3788UfuvfdeALZuPcHmzbZnBffsOc2IEavgdEUAVjValeM+lVJKFSztEeaT3bt306VLF+Li4mjWrBnffPNNlmuCH3ywJcv6Xbrcw5paZwo7THWd1NRUYmNjSUlJcXUoSikneHp64uPjQ5kyZfJtn5oI80lSUhLnz5+nQ4cOrFixggoVKmRZPnx4Uy5fTiMiYh/p6cKoUfezxkWxqj/FxsZSoUIFfH19rTt8lVJFk4hw9uxZYmNjqV27dr7tVxNhPmnZsiXff/89gYGBeHp6ZlveunUtWreuxfHjiSxcuJuwsLqw0QWBqixSUlI0CSpVTBhjqFKlCqdPn87X/Woi/AuWLVuGMYbevXsDEBISkus2d91Vkeefb13QoamboElQqeKjIH5fNRHm0b///W9GjBiBh4cHu3bton79+q4OSSmlVB7oXaN58OabbzJ8+HAyMjKYPHmydWdobrpFd8N8a6xJKQB3d3caNWpEQEAAPXr04Pz589ayPXv28MADD3DvvfdSr149pkyZgohYy7/55huaNWtGgwYN8PPzY+zYsa54C3nSv39/goKCeOedd5xav3z58gUSx/79+2nRogXlypVj+vTpOa4nIjzwwANcuHChQOLID9u3bycwMJC6desyatSoLJ+V6/3++++UL1/ees9JSUk0atTImqpWrcro0aMBmDFjBh9//HGhvAeXEJFiNTVt2lRcJSMjQ8aPHy+AGGNk1qxZOa577txlGTDgSzlx4oL1GuvJNnX9pWthhK5ysHfvXleHILfccov1/yeeeEKmTp0qIiKXLl2SOnXqyJo1a0RE5OLFixIWFiYzZswQEZHdu3dLnTp1ZN++fSIikpqaKjNnzszX2FJTU/N1f9fExcVJrVq1bmqbzO2Un06dOiVbtmyRF198Ud5+++0c1/v6669l9OjRN7XvtLS0vxreTQkJCZGffvpJMjIyJCwsTFavXp3jur169ZI+ffrk+J6bNGki33//vYjYPnuNGjUqkJjzwtHvLbBN8phXtEfopPT0dIYPH85bb72Fh4cHCxcu5KmnnnK4bkLCZTp1+oRFi3bTseMnnDqVnGW5dBRr0mcIi47MvfX8nG5GixYtOHHiBACLFi2iVatWhIaGAuDt7c2MGTOYNm0aAG+99RYTJ07Ez88PAA8PD0aOHJltn8nJyQwdOpTAwECCgoL48ssvgaw9rGXLljFkyBAAhgwZwnPPPUeHDh0YN24cvr6+WXqpdevW5dSpU5w+fZrevXsTEhJCSEgImzZtynbslJQU69iNGzdm40bbHWKhoaHEx8fTqFEjIiMjs2xz6tQpHnnkEYKDgwkODuann37K9n46duxIkyZNCAwMZMWKFYBtWMNu3boRHBxMQEAAS5YsAeD555+nYcOGBAUFOewxV69enZCQkFxvx1+4cCEPPfSQNf/www/TtGlT/P39mTt3rvV6+fLlmTx5Mvfffz+bN29m+/bttGvXjqZNm1qPWIHt8kpISAjBwcH07t2bS5cu3fD4uYmLi+PChQu0aNECYwxPPPEEy5cvd7ju8uXLqVOnDv7+/g6XHzp0iPj4eNq0aQPYPnu+vr5s2bLF4frFnV4jdNLhw4dZvHgxXl5eLFu2jK5dHY8Gc/VqOh06LGDXrlMA7N9/hk6dPmXjxsGFGa4qhtLT0/n222/5+9//DthOizZt2jTLOvfccw/JyclcuHCBmJgY/ud//ifX/U6ZMoWKFSuye/duAM6dO5frNgcPHmT9+vW4u7uTkZFBREQEQ4cO5eeff8bX15caNWowYMAAxowZQ+vWrfn999/p0qUL+/bty7KfmTNnArbnbPfv309oaCgHDx5k5cqVdO/enejo6GzHHjVqFO3atSMiIoL09HSSk7N+kfT09CQiIoJbb72VM2fO0Lx5c3r27Ml///tf7rjjDmt838TERBISEoiIiGD//v0YY7Ik9Ju1adMmPvzwQ2t+3rx53HbbbVy+fJmQkBB69+5NlSpVuHjxIgEBAbz66qukpqbSrl07VqxYQbVq1ViyZAkTJ05k3rx59OrVi2HDhgEwadIkPvrooz/Ls9lt3LiRMWPGZIvF29s72xeEEydO4OPjY837+PhYX6oyu3jxIm+++Sbr1q3L8VTw559/Tr9+/bLcmNKsWTMiIyO57777nGit4kUToZPq16/PypUrKVOmDK1b53zXZ9my7jz8cH0rEQJUrFiOsmXdCyNM9RdIx5yvpxSky5cv06hRI44dO0bTpk3p3LmzLZ5M1UuudzN3zq1fv57Fixdb85UrV851m0cffRR3d9tntl+/frz66qsMHTqUxYsX069fP2u/e/futba5cOECSUlJWZ6h/fHHH60/7n5+ftx9990cPHiQW2+9Ncdjb9iwgU8++QSwXT+tWLFiluUiwosvvsgPP/yAm5sbJ06c4NSpUwQGBjJ27FgmTJhA9+7dadOmDWlpaXh6ehIeHk63bt3o3r17ru89JwkJCVne2/vvv09ERAQAx48f59ChQ1SpUgV3d3frTvIDBw4QExNj/UzT09OpWbMmADExMUyaNInz58+TnJxMly5dsh2zQ4cODr8sOCIOrgc6+py8/PLLjBkz5obXXBcvXsynn36a5bXq1auzf/9+p2IpbjQR3kBCQgJRUVFW769Dhw5ObffSS+1Yu/YIUVGxtG/vy//9X3/Kly9bkKGqYszLy4vo6GgSExPp3r07M2fOZNSoUfj7+/PDDz9kWffIkSOUL1+eChUq4O/vz/bt2wkODr7h/nNKqJlfu35knVtuucX6f4sWLTh8+DCnT59m+fLlTJo0CYCMjAw2b96Ml5fXDY+d3xYuXMjp06fZvn07ZcqUwdfXl5SUFO699162b9/O6tWreeGFFwgNDWXy5Mls2bKFb7/9lsWLFzNjxgw2bNiQp+N6eHiQkZGBm5sb3333HevXr2fz5s14e3vTvn17qw09PT2tLxEigr+/P5s3b862vyFDhrB8+XKCg4OZP38+3333XbZ1bqZH6OPjQ2xsrDUfGxvLHXfckW3bn3/+mWXLljF+/HjOnz+Pm5sbnp6ePPPMMwDs3LmTtLS0bGcjUlJSbvizLs70GmEOTp48Sbt27ejZsydr1tzcGDAeHm589tkjhIc3ZtWqAZoElVMqVqzI+++/z/Tp00lNTWXgwIH8+OOPrF+/HrD1HEeNGsX48eMBGDduHK+//joHDx4EbInpX//6V7b9hoaGMmPGDGv+2qnRGjVqsG/fPuvUZ06MMTzyyCM899xzNGjQgCpVqjjcr6OeS9u2bVm4cCFgO936+++/5/qoUceOHZk9ezZg60Fdf5dmYmIi1atXp0yZMmzcuJHffvsNsP3Oent7M2jQIMaOHcuOHTtITk4mMTGRrl278u677zrdu3Kkfv36HDlyxIqhcuXKeHt7s3//fqKionLc5vTp01YiTE1NZc+ePYDtLs2aNWuSmppqtdH1rvUIr5+uT4IANWvWpEKFCkRFRSEifPLJJ1muaV4TGRnJsWPHOHbsGKNHj+bFF1+0kiDYTov2798/23YHDx4kICAgl1YqnjQROnD48GFr8Oz69es7/OFfvpzKihU5nya4557b+Pe/e+LtnX/j4amSr3HjxgQHB1vXo1esWMHUqVOpX78+gYGBhISEWH+0goKCePfdd+nfvz8NGjQgICDAuhEjs0mTJnHu3DkCAgIIDg62bliZNm0a3bt354EHHrBO1+WkX79+fPbZZ9ZpUbCdGty2bRtBQUE0bNiQOXPmZNtu5MiRpKenExgYSL9+/Zg/f75VjSUn7733Hhs3biQwMJCmTZtaieOagQMHsm3bNpo1a8bChQutm4V2797NfffdR6NGjXjttdeYNGkSSUlJdO/enaCgINq1a+fwUY0//vgDHx8f/vWvfzF16lR8fHwcPiLRrVs3q9cWFhZGWloaQUFBvPTSSzRv3tzheylbtizLli1jwoQJBAcH06hRIyuJTZkyhfvvv5/OnTtb7+Gvmj17NuHh4dStW5d77rmHBx98EICVK1cyefJkp/bxxRdfOEyEmzZtolOnTvkSZ5GT19tNXTUV9OMTO3fulBo1aggg9913n5w5cybbOr/8EicNGswQY16RDRuOOLXfrr90tR6ZUEVHUXh8QhUPJ0+elE6dOrk6DJfYsWOHDBo0yNVhWPTxiQK0adMm2rZty6lTp+jYsSPr16+3TgNds2BBNPff/x/27TuDCDz+eAQJCZdz3ffqs6sBrT2oVHFVs2ZNhg0bVqQfqC8oZ86cYcqUKa4Oo8BoIrRLSUmhb9++JCYm0qtXL1atWpWtggRAkyY1SU/PsOZPnEjilVe+c/o4+tygUsVX3759b3jHa0nVuXNnfH19XR1GgdFEaOfp6ckXX3zByJEjWbJkSY7XMQIDazB69J/XA4YMacRrrz1QWGEqpZTKZ6X+8YkDBw5Yd7G1atWKVq1a5brNyy+347vvjjFhQisefdTxyAxKKaWKh1LbIxQRXn/9dfz9/fnqq6+yLIuM/I0uXT4jKemKw20rVCjH1q3DNAkqpVQJUCoToYgwbtw4Jk6cSEZGBmfOnLFef+65NbRtO5+1a3+94bU/rWGnlFIlQ6lLhGlpaYSHh/PPf/4TDw8PPv/8c4YPHw7A3LnbeeedPx+Mfe+9n7MMlaaUKlpKegmrX375hfDwcFeHcUNvvPEGdevWpX79+rkOPjJ9+nSMMVbnIzU1lcGDBxMYGEiDBg144403rHU7derk1Li4+aFUXSNMSUlhwIABRERE4OXlxVdffUVYWBhgqxW4OvoceHSCNNvwSOnpQvCM3tAv+6j6qmQy5n+zzIu87HC9uXO38+STX1vzw4Y1Ye7cHgUaW16lp6dbQ36VtONfG54OYPDgwcycOZOJEydy+fJlevbsyezZswkNDeXSpUv07t2bWbNm8fTTTxMTE8MzzzzDqlWr8PPzIy0tLUsFifyQlpaGh8df+xP7+uuvW0PaFdYxb8bevXtZvHgxe/bs4eTJk3Tq1ImDBw86/HkfP36cdevWUatWLeu1pUuXcuXKFXbv3s2lS5do2LAh/fv3x9fXl8cff5xZs2YxceLEAn8fpapHOGTIECIiIqhUqRLr1q2zkiDYn/PrsxlmzoU6f0DZVHjzk3xNgvoMobresWPH8PPzIzw8nICAAAYOHMj69etp1aoV9erVs8rebNmyhZYtW9K4cWNatmzJgQMHAFuSGTt2rFVi6YMPPgDA19eXV199ldatW7N06VKio6Np3rw5QUFBPPLIIzl+03ZUWmj27NnWsG4A8+fPtwbS/uyzz6zRXJ588knS09OB7KWIXn31VUJCQggICGD48OFWz2zr1q0EBQXRokULxo0bZ43ilJ6ezrhx4wgJCSEoKChL1YeclLQSVklJSezatcsaSzanz8D8+fN59NFH6dGjh/V+3377bavtXn75zy9zOZWOyqsVK1bw2GOPUa5cOWrXrk3dunVzLNU0ZswY3nrrrSyXlYwxXLx4kbS0NC5fvkzZsmWtx1N69uzJ559//pdjdEpen8R31fRXRpbZunWr1KtXT3bu3JltWeZRX65cSZMtW2LzfBxVfFw/QgW8kmXKyYcfbsuy3rBhK/N0/KNHj4q7u7vs2rVL0tPTpUmTJjJ06FDJyMiQ5cuXy0MPPSQiIomJiVaR3HXr1kmvXr1ERGTWrFnSq1cva9nZs2dFROTuu++WN9980zpOYGCgfPfddyIi8tJLL8mzzz7rMJ5r21+6dEn8/f3lzJkzEh8fL/fcc4+1TlhYmERGRsrevXule/fucvXqVREReeqpp2TBggUiIgLIkiVLsu1XRGTQoEGycqWtvfz9/WXTpk0iIjJhwgTx9/cXEZEPP/xQpkyZIiIiKSkp0rRpUzlyJPsoTteK9aalpUmfPn3km2++ERGRMWPGyLvvvptt/UqVKkliYqI0btxYoqOjHbZBZuPHj8/SVgkJCVmOKyKydOlSGTx4sIiIDB48WLp162YV5B01apTMmzdPRESioqKkY8eOIiLSv39/iYyMFBGR3377Tfz8/LIde8OGDdbPWSTnz8DHH38sd955p9XGa9askWHDhklGRoakp6dLt27drAK7jn6+1xs9erQEBwdnm954441s6z799NPy6aefWvN/+9vfZOnSpdnWW7FihYwaNUpEbJ/N06dPi4jI1atXpV+/flK1alXx9vaWDz/8MMt2devWdRhjfo8sU+JPjaakpODp6QnY6mnt3bs311MHZcu6ExJyZ2GEpxS1a9cmMDAQAH9/fzp27IgxhsDAQI4dOwbYBnkePHgwhw4dwhhDamoqYCuFNGLECOszfdttt1n7vTYuaGJiIufPn6ddu3aA7RTio48+6jAWR6WFmjdvTp06dYiKiqJevXocOHCAVq1aMXPmTLZv305ISAhgGxS8evXqAFlKEYGtisJbb73FpUuXSEhIwN/fnzZt2pCUlETLli0BGDBgAF9/bTvdvHbtWnbt2sWyZcus93Do0CFq166dJd6SXMIqLi6OatWqWfM5fQbA9sD7tZ/92rVrWbt2LY0bNwZsvdpDhw7Rtm3bHEtHZeZoPNaciORe+unSpUu89tprrF27Ntu6W7Zswd3dnZMnT3Lu3DnatGlDp06dqFOnDmAr/XTy5MlsMea3Ep0IDx06RJcuXZg6dSoDBgwArpVSEWbM2EL79r4EBdVwcZSqKMnpmuD1hg9vyvDhTXNf0QmZB29wc3Oz5t3c3EhLSwPgpZdeokOHDkRERHDs2DHat29vjzfnP/iZSyk5cvz4cXr0sF3XHDFiBH5+fjmWFurXrx9ffPEFfn5+PPLIIxhjEBEGDx6c5QaHazKXIkpJSWHkyJFs27aNu+66i1deeYWUlBSHf0SvERE++OADhzX6MivJJay8vLyy7Dunz8D1xxQRXnjhBZ588sks+7tR6ajMxowZYw3Mntljjz3G888/n+U1Hx8fjh8/bs07Kv3066+/cvToUautY2NjadKkCVu2bGHRokWEhYVRpkwZqlevTqtWrdi2bZuVCAur9FOJvUYYHR1N69atOXr0KHPmzCEjwzYsWlxcEl27LuTZZ//LwIFfkZKS5uJIlcpdYmIid95pO0sxf/586/XQ0FDmzJljJcyEhIRs21asWJHKlSsTGRkJwKeffkq7du246667rLI+I0aMuGFpoV69erF8+XKrcjnYyiUtW7aM+Ph469jXSiJldu2PbdWqVUlOTrZ6eZUrV7bKBgFZel5dunRh9uzZVq/n4MGDXLx4Mcf2KYklrBo0aMDhw4et+Zw+A9fr0qUL8+bNIzk5GbBVro+Pj3e6dNQ777zjsPTT9UkQbNfxFi9ezJUrVzh69CiHDh3KVsE+MDCQ+Ph4q/STj48PO3bs4Pbbb6dWrVps2LABEeHixYtERUVZ121FhD/++KNQhnYrkYkwMjKSdu3aER8fT+fOnVm9ejVubm6cOpVMSMi/WbPmVwBiYuJ5/vn1Lo5WqdyNHz+eF154gVatWlk3pACEh4dTq1YtgoKCCA4OZtGiRQ63X7BgAePGjSMoKIjo6GiHJXluVFqocuXKNGzYkN9++836Q9ewYUOmTp1KaGgoQUFBdO7c2WEZqEqVKjFs2DACAwN5+OGHrVOpAB999BHDhw+nRYsWiIhVjT48PJyGDRvSpEkTAgICePLJJ61kn5OSVsLKz8+PxMREkpKSgJw/A9cLDQ1lwIABtGjRgsDAQPr06UNSUpLTpaNuhr+/P3379qVhw4aEhYUxc+ZM60xAeHg427Ztu+H2Tz/9NMnJyQQEBBASEsLQoUMJCgoCYPv27TRv3rxQ7oI1Nzo9URQ1a9ZMbtS4q1atok+fPqSkpNCnTx8+++wz61TTnDnbeOqprINeV6rkycGDz1B9l+0uMOlYvNpD/TX79u2jQYMGrg6j1EpOTrbuwJw2bRpxcXG89957Lo6q6HjnnXeoUKFCkX+WsCA8++yz9OzZk44dO2Zb5uj31hizXUSa5eVYJapHuGzZMh5++GFSUlIIDw9n8eLFWa6/NG/uw/Tpnan9RBx02A2VLnJ+5htWElRKFa5Vq1ZZD8RHRkbe1DNzpcFTTz2VayHjkiogIMBhEiwIJapHuGvXLtq2bcuTTz7JtGnTEIHERNv1icqV/7zgar41cMUDPNLB/c/337VKVy2TVMpoj1Cp4ie/e4Ql6q7RoKAgYmJi2LDhLFWqvMX58ymIwIgRTZk9u3vWlcul6WlQBdz4zkulVNFSEJ23Yn1qNCMjg+eee4558+ZZr/n4+ODh4ca5c7YkCHDuXPZbhJUC223+Z8+eLZBfLqVU/hIRzp49az0bnl+KbY/w2uDZCxYswMvLi65du3L77bcDULly1kZKSLjsihBVMeDj40NsbCynT592dShKKSd4enri4+OTr/sslokwOvo43bv35sSJrXh7exMREWElQch6PbBChbKUK1cs36YqBGXKlMk2WolSqnQp0JtljDFhwHuAO/AfEZl23XJjX94VuAQMEZEdN9pntWr3yJkzAhylXLnybNy4lhYtWmRZ5+rVdC5cuEKlSp54eGQ/+2u+tV0P0muESilVMhTJm2WMMe7ATKAzEAtsNcasFJG9mVZ7EKhnn+4HZtv/zVFCQixwFShPWtpgqla9N9s6Zcu6U7Wqd768D6WUUiVbQZ4zvA84LCJHAIwxi4GHgMyJ8CHgE/vI4VHGmErGmJoikn2IB7uMjFSgMvA46em3ce8rfeFvGwruXSillCrRCjIR3gkczzQfS/benqN17gSyJEJjzHBguH32CpyLgfdtc4vsUx4YStUt81WBM64OohjSdss7bbu80XbLm/p53bAgE6GjLHP9RTln1kFE5gJzAYwx2/J6Hrg003bLG223vNO2yxttt7wxxtx4YNMbKMjnCGOBuzLN+wAn87COUkopVWAKMhFuBeoZY2obY8oCjwErr1tnJfCEsWkOJN7o+qBSSimV3wrs1KiIpBljngHWYHt8Yp6I7DHGjLAvnwOsxvboxGFsj08MdWLXcwso5JJO2y1vtN3yTtsub7Td8ibP7VbsBt1WSiml8lOxHmtUKaWU+qs0ESqllCrVimwiNMaEGWMOGGMOG2Oed7DcGGPety/fZYxp4oo4ixon2m2gvb12GWN+MsYEuyLOoia3dsu0XogxJt0Y06cw4yuqnGk3Y0x7Y0y0MWaPMeb7wo6xKHLi97SiMeb/jDE77e3mzP0TJZ4xZp4xJt4YE5PD8rzlBREpchO2m2t+BeoAZYGdQMPr1ukKfIPtWcTmwM+ujtvVk5Pt1hKobP//g9puzrVbpvU2YLvJq4+r43b15OTnrRK20aRq2eeruzpuV09OttuLwJv2/1cDEoCyro7d1RPQFmgCxOSwPE95oaj2CK3h2UTkKnBteLbMrOHZRCQKqGSMqVnYgRYxubabiPwkIufss1HYnt0s7Zz5vAH8A/gSiC/M4IowZ9ptAPCViPwOICLads61mwAV7IUJymNLhGmFG2bRIyI/YGuLnOQpLxTVRJjT0Gs3u05pc7Nt8nds355Ku1zbzRhzJ/AIMKcQ4yrqnPm83QtUNsZ8Z4zZbox5otCiK7qcabcZQANsA4zsBp4VkYzCCa9Yy1NeKKqF+vJteLZSxuk2McZ0wJYIWxdoRMWDM+1I5cIzAAAFPElEQVT2LjBBRNJtX9IVzrWbB9AU6Ah4AZuNMVEicrCggyvCnGm3LkA08ABwD7DOGBMpIhcKOrhiLk95oagmQh2eLW+cahNjTBDwH+BBETlbSLEVZc60WzNgsT0JVgW6GmPSRGR54YRYJDn7e3pGRC4CF40xPwDBQGlOhM6021BgmtgufB02xhwF/IAthRNisZWnvFBUT43q8Gx5k2u7GWNqAV8Bj5fyb+WZ5dpuIlJbRHxFxBdYBows5UkQnPs9XQG0McZ4GGO8sVWg2VfIcRY1zrTb79h60RhjamCrrHCkUKMsnvKUF4pkj1AKbni2Es3JdpsMVAFm2Xs3aVLKR7p3st3UdZxpNxHZZ4z5L7ALyAD+IyIOb30vLZz8vE0B5htjdmM73TdBREp9aSZjzOdAe6CqMSYWeBkoA38tL+gQa0oppUq1onpqVCmllCoUmgiVUkqVapoIlVJKlWqaCJVSSpVqmgiVUkqVapoIValnryYRnWnyvcG6vjmNfH+Tx/zOXn1gpzFmkzGmfh72MeLakGXGmCHGmDsyLfuPMaZhPse51RjTyIltRtufGVSqWNBEqBRcFpFGmaZjhXTcgSISDCwA3r7Zje3P6X1inx0C3JFpWbiI7M2XKP+McxbOxTka0ESoig1NhEo5YO/5RRpjdtinlg7W8TfGbLH3IncZY+rZXx+U6fUPjTHuuRzuB6CufduOxphfjDG77bXXytlfn2aM2Ws/znT7a68YY8YaW23EZsBC+zG97D25ZsaYp4wxb2WKeYgx5oM8xrmZTAMYG2NmG2O2GVu9vP+1vzYKW0LeaIzZaH8t1Biz2d6OS40x5XM5jlKFShOhUuCV6bRohP21eKCziDQB+gHvO9huBPCeiDTClohijTEN7Ou3sr+eDgzM5fg9gN3GGE9gPtBPRAKxjfz0lDHmNmyVL/xFJAiYmnljEVkGbMPWc2skIpczLV4G9Mo03w9Yksc4w4DMw8pNtI9KFAS0M8YEicj72MZ27CAiHYwxVYFJQCd7W24DnsvlOEoVqiI5xJpSheyyPRlkVgaYYb8mlo6tnND1NgMTjTE+2GruHTLGdMRWbWGrfQg7L3KuX7jQGHMZOIat1mF94GimMWAXAE9jK8mTAvzHGLMK+NrZNyYip40xR+zjLh6yH2OTfb83E+ct2IYDy1zxu68xZji2vyM1gYbYhlLLrLn99U3245TF1m5KFRmaCJVybAxwClulBDdsiSgLEVlkjPkZ6AasMcaEYxsXcoGIvODEMQaKyLZrM8aYKo5Wso9NeR+2QZgfA57BVp7HWUuAvsB+IEJExNiyktNxYquiPg2YCfQyxtQGxgIhInLOGDMf8HSwrQHWiUj/m4hXqUKlp0aVcqwiEGcvhvo4tt5QFsaYOsAR++nAldhOEX4L9DHGVLevc5sx5m4nj7kf8DXG1LXPPw58b7+mVlFEVmO7EcXRnZtJQIUc9vsV8DDQH1tS5GbjFJFUbKc4m9tPq94KXAQSja06woM5xBIFtLr2nowx3sYYR71rpVxGE6FSjs0CBhtjorCdFr3oYJ1+QIwxJhpbrbhP7HdqTgLWGmN2AeuwnTbMlYikYBstf6m96kAGMAdbUvnavr/vsfVWrzcfmHPtZpnr9nsO2AvcLSJb7K/ddJz2a4//BMaKyE7gF2APMA/b6dZr5gLfGGM2ishpbHe0fm4/ThS2tlKqyNDqE0oppUo17REqpZQq1TQRKqWUKtU0ESqllCrVNBEqpZQq1TQRKqWUKtU0ESqllCrVNBEqpZQq1f4fujBXTmYUjtkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 504x504 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "skplt.metrics.plot_roc(y, yPredProb, title=\"CV SVC ROC\",\n",
    "                      figsize=(7,7),plot_micro=False,\n",
    "                      classes_to_plot=1)\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Precision and Recall\n",
    "\n",
    "You'll recall that:  \n",
    "\n",
    "**Precision** = proportion of cases classified as positive that are actually positive  \n",
    "**Recall** = proportion of positive cases that are classified as positive\n",
    "\n",
    "Here are the precision and recall estimates for the current results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision: 0.000 Recall: 0.000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/lynd/anaconda3/lib/python3.6/site-packages/sklearn/metrics/classification.py:1437: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    }
   ],
   "source": [
    "print('Precision: {:5.3f} Recall: {:5.3f}'.format(\n",
    "        precision_score(y,yPredProb[:,1]>0.5),\n",
    "        recall_score(y,yPredProb[:,1]>0.5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pretty sucky."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unbalanced Class Label Tweak\n",
    "\n",
    "One way of trying to get a better classifier model for an \"unbalanced\" target is to weight observations based on target class frequencies.  This is akin to using class prior probabilities, or penalizing a likelihood function. In this case, the weights are actually applied to a shrinkage parameter (see \"C\" in the [Support Vector Classifier](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html) documentation.  C defaults to 1.0.  We'll give this tweak a try here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here's our SVC with target classes weighted by frequency of occurrance.\n",
    "svclf2=SVC(probability=True,class_weight='balanced')   #instance of the support vector classifier\n",
    "svcRes2=cross_validate(svclf2,X,y,cv=20,n_jobs=-1,return_train_score=True)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# covert to code cell to get DataFrame of results\n",
    "\n",
    "pd.DataFrame(svcRes2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "yPredProb2=cross_val_predict(svclf2,X,y,n_jobs=-1,cv=20,\n",
    "                           method='predict_proba')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWsAAADtCAYAAAB52FQJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAeGklEQVR4nO3de7xVVb338c93bwQ0Ne9EgIKKGPR4QaVSK8sexbTwnNfR8GhR+uRTadbpCl3N4mj6OpUdtZNpSVoiph7JGxo+VnZU8JohKiheEBTQUlMTwd/zxxxbF9u915ob1tprzTW/b1/ztdeaa6wxx95ufnus3xwXRQRmZtbaOprdADMzq83B2sysAByszcwKwMHazKwAHKzNzApgQLMbYGbWDJ2b7xCx5qVcZeOllXMiYmKDm1SVg7WZlVKseYlBY47MVfYfd5+9TYObU5ODtZmVlEDFyQQ7WJtZOQno6Gx2K3JzsDaz8pKa3YLcHKzNrKScBjEzKwb3rM3MWpwoVM+6OC01M6srZT3rPEee2qQtJP1G0v2SFkp6l6StJN0gaVH6umVF+WmSFkt6QNLBtep3sDaz8urozHfkcyZwXUTsCuwOLASmAnMjYjQwNz1H0lhgMjAOmAicI6nqhRyszayk0g3GPEetmqTNgfcA5wNExOqI+BswCZiRis0ADk+PJwEzI+LliFgCLAYmVLuGg7WZlZPoSxpkG0m3VxzHd6ttR2Al8AtJd0k6T9KbgCERsRwgfd0ulR8GPF7x/qXpXK98g9HMyiv/DcZVEbF3ldcHAOOBz0bEbZLOJKU8ertyD+eqbtvlnrWZlVT90iBkPeOlEXFbev4bsuD9lKShAOnrioryIyrePxxYVu0CDtZmVk4COjvzHTVExJPA45LGpFMHAvcBs4Ep6dwU4Mr0eDYwWdIgSaOA0cC8atdwGsTMyqu+k2I+C/xK0kDgYeATZB3iWZKOAx4DjgCIiAWSZpEF9DXACRGxtlrlDtZmVlL1nW4eEXcDPeW1D+yl/HRget76HazNrLw83dzMrAAKNN3cwdrMyqkPU8lbgYO1mZWXNx8wM2t1Xs/azKwYnAYxM2txBVvP2sHazErKaRAzs2LwDUYzswIoUM66OJ8BrKVI2ljSbyU9K+nSDajnaEnX17NtzSDpWklTape0lqG6rrrXcK3RCmsYSf+aFkv/u6TlKajsX4eq/wUYAmwdEUesbyUR8auIOKgO7VmHpAMkhaTLu53fPZ2/KWc9J0u6qFa5iDgkImbUKmctpo57MDaag3Ubk/QF4EfAv5MF1u2Bc8i2FNpQOwAPRsSaOtTVKCuBfSVtXXFuCvBgvS6gjP8dFZSkXEcr8C9Zm5L0ZuAUsqUXL4+IFyLilYj4bUR8OZUZJOlHkpal40eSBqXXDpC0VNIXJa1IvfJPpNe+A3wL+EjqsR/XvQcqaWTqwQ5Izz8u6WFJz0taIunoivM3V7xvX0nzU3plvqR9K167SdJ3Jf0p1XO9pG2q/BhWA/9NtjEpaUPSI4FfdftZnSnpcUnPSbpD0rvT+YnA1yq+z3sq2jFd0p+AF4Ed07n/k17/iaTfVNT/fUlz1Sr/6g3o2tXLwdqa713AYOCKKmW+DrwT2INsN+YJwDcqXn8L8GayveGOA86WtGVEfJust35JRGwaEedXa0jai+7HwCERsRmwL3B3D+W2Aq5OZbcGfgBc3a1n/K9k6wRvBwwEvlTt2sAvgY+lxwcDC3jjjhzzyX4GWwG/Bi6VNDgiruv2fe5e8Z6PAscDmwGPdqvvi8Bu6Q/Ru8l+dlMiouq2TdbPJNSR72gFDtbta2uyfeOqpSmOBk6JiBURsRL4DlkQ6vJKev2ViLgG+Dswpod68ngVeLukjSNieUQs6KHMocCiiLgwItZExMXA/cCHKsr8IiIejIiXgFlkQbZXEfE/wFZpB4+PkQXv7mUuioin0zX/AxhE7e/zgohYkN7zSrf6XgSOIftjcxHZvnxLa9RnTeCetbWCp8l2ZK42PPOtrNsrfDSde62ObsH+RWDTvjYkIl4APgJ8Clgu6WpJu+ZoT1ebKnd9fnI92nMhcCLwPnr4pJFSPQtT6uVvZJ8mqqVXYN2dqd8gIuaR7RYisj8q1oIcrK0V3AL8Azi8SpllZDcKu2xPjU07q3gB2KTi+VsqX4yIORHxv4GhZL3ln+VoT1ebnljPNnW5EPgMcE3q9b4mpSm+SpbL3jIitgCe5fXdp3tLXVRNaUg6gayHvgz4yvo33RrJwdqaLiKeJbsJeLakwyVtImkjSYdIOj0Vuxj4hqRt0426b5F9bF8fdwPvkbR9urk5resFSUMkfTjlrl8mS6f0tN/cNcAuyoYbDpD0EWAscNV6tgmAiFgCvJcsR9/dZmR74K0EBkj6FrB5xetPASPVhxEfknYBvkeWCvko8BVJVdM11gTqw9ECHKzbWET8APgC2U3DlWQf3U8kGyEBWUC5HfgzcC9wZzq3Pte6Abgk1XUH6wbYDrKbbsuAZ8gC52d6qONp4LBU9mmyHulhEbFqfdrUre6bI6KnTw1zgGvJhvM9SvZppDLF0TXh52lJd9a6Tko7XQR8PyLuiYhFZCNKLlQaaWOtQeTrVbdKz1q+QW1mZTRg6x1j8w/m65v89aKj74iInjbD7TfuWTfPFsBvyPK3C8mG2n2XrGd6N3A9697smwYsBh4gG4JmJSJpoqQHJC2WNLXZ7WkXRepZNzRY+xesqjOB64BdycY4LwTOAHYjG452FVkOGbK87WRgHDCRbBZicZYLsw2SJvOcDRxC9rtwlKSxzW1VG3DOOuNfsKo2B94DdE0mWQ38DXiuosybeH3EwSRgJtnNuSVkPewJ/dJSawUTgMUR8XBErCb7XajHkgGl5551xr9gvduR7IbfL4C7gPPIgjPAdLIbXEfzes96GOve9FrKumOPrb35/38DFO0GYyPXs+7pF+wd3QtJOp5s2i5owF4avGUDm9Qa9hq/O7fcdC37H3jYPvPm38mPzvjeHs89//xx3zrl+6+Vmfqlk4YPHjx4xcnfO52zfngat9x2+wm/mvmbCwHO+8kPuea6ucdefuUGjWgrjD3ftn2zm9BUo0btyHPPPctee+19HMAOO4zkxRdfYK+99j6h2W1rlkcffYRVq1ZtcBSt51RySY8Az5MNS10TEXunJRQuAUYCjwBHRsRfU/lpZEsRrAVOiog51epvZLDu6afwhqEnEXEucC5AxybbxaAxRzawSa1h5eDNeGLFs9zz3M4MGrMzs+94gS9+4mAGjVnyWpnLbn+Zy398LKde+gjLV2/DyD0nMuiu7IPQiNHjWXXjkwwas0lvl2grf7rtrGY3oaluveUWpn/3ZH57TfZv+YzvnwrAl786rdrb2tp+76jDwAzRiF7z+7oNNZ0KzI2I09J9u6nAV1NKuOs+1FuB30naJSJ6mn8ANDYNshQYUfF8OOs/O66tPPX08yx98q+M3mE7AA6YMIb7H36Snbbf9rUyh753Nx585CkArr7pzxxx8HgGbjSAHd66NTtvvy3z//JIM5puTbD3PvuwePEiHlmyhNWrV3PpJTM59LAPN7tZbaEf0iCTgK51zmfw+oziScDMiHg5TdqqeR+qkT3r+cBoSaPIpgtPJlsxzYAvfP9SfvHvH2fggE4eeWIVx3/7In7y7aMZvcN2vPpq8NjyZzhp+kwAFj78JJddfxd3XfZ11qx9lc+fNotXX/X4+LIYMGAAPzzzLD506MGsXbuWKR8/lrHjxjW7WW2hD4F4G0m3Vzw/N2UFKgVwvaQAfppeHxIRywEiYrmk7VLZYcCtFe+teR+iYcE6ItZIOpFshlgn8PNeVlorpT8/+AT7H336OueO+tJ5vZY//fw5nH5+1ZSWtbGJh3yQiYd8sNnNaCtdNxhzWpVjUsx+EbEsBeQbJN1f9fJvVLUH1tANc9Oymtc08hpmZuutjinrruUMImKFpCvI0hpPSRqaetVDgRWpeJ/TxJ7BaGblJOjo6Mh11KxKepOkzboeAwcBfwFmk20lR/p6ZXo8G5isbLemUcBoYF61azS0Z21m1srqOBpkCHBFqm8A8OuIuE7SfGCWpOOAx4AjACJigaRZwH1kqz6eUG0kSFelZmblVKdYHREPky0b0f3808CBvbxnOtkkuFwcrM2stFpldmIeDtZmVkqtNJU8DwdrMystB2szswKo59ogjeZgbWal5Z61mVmra8xCTg3jYG1mpSSgQLHawdrMysqjQczMCqHDNxjNzFqcnAYxM2t5wj1rM7NCcM/azKwAfIPRzKzVOWdtZtb6hHJtLNAqHKzNrLTcszYzKwDnrM3MWp1z1mZmrS9bG6Q40drB2sxKq0Cx2sHazMrLMxjNzFqd17M2M2t9Xs/azKwQvJ61mVkhFChWO1ibWUmpWDcYizMx3sysjrrGWec5ctUndUq6S9JV6flWkm6QtCh93bKi7DRJiyU9IOngPPU7WJtZadUzWAOfAxZWPJ8KzI2I0cDc9BxJY4HJwDhgInCOpM5alTtYm1lpSfmO2vVoOHAocF7F6UnAjPR4BnB4xfmZEfFyRCwBFgMTal3DwdrMSqsPPettJN1ecRzfraofAV8BXq04NyQilgOkr9ul88OAxyvKLU3nqvINRjMrp74t5LQqIvbusRrpMGBFRNwh6YB8V36DqPUmB2szK6Vs84G6jAbZD/iwpA8Cg4HNJV0EPCVpaEQslzQUWJHKLwVGVLx/OLCs1kWcBjGz0uqQch3VRMS0iBgeESPJbhzeGBHHALOBKanYFODK9Hg2MFnSIEmjgNHAvFptdc/azEqrwZNiTgNmSToOeAw4AiAiFkiaBdwHrAFOiIi1tSpzsDazUlIDFnKKiJuAm9Ljp4EDeyk3HZjel7p7DdaSNq/RqOf6ciEzs1ZToAmMVXvWC8juUFZ+O13PA9i+ge0yM2u4Ik037zVYR8SI3l4zMys6kY0IKYpco0EkTZb0tfR4uKS9GtssM7PG61C+oxXUDNaSzgLeB3w0nXoR+K9GNsrMrOFyzl5slTWv84wG2Tcixku6CyAinpE0sMHtMjNruBaJw7nkCdavSOogTYeUtDXrzn83MyscQc0JL60kT7A+G7gM2FbSd4Ajge80tFVmZv2gLUaDdImIX0q6A/hAOnVERPylsc0yM2usvMuftoq8Mxg7gVfIUiFeT8TM2kKR0iB5RoN8HbgYeCvZ6lC/ljSt0Q0zM2s05TxaQZ6e9THAXhHxIoCk6cAdwKmNbJiZWaO1yrC8PPIE60e7lRsAPNyY5piZ9Y9sNEizW5FftYWcfkiWo34RWCBpTnp+EHBz/zTPzKxBVLfNB/pFtZ5114iPBcDVFedvbVxzzMz6T1ukQSLi/P5siJlZf2qbNEgXSTuRLZI9lmx/MQAiYpcGtsvMrOGK1LPOM2b6AuAXZH+IDgFmATMb2CYzs35RpKF7eYL1JhExByAiHoqIb5CtwmdmVlgSdHYo19EK8gzde1nZZ4WHJH0KeALYrrHNMjNrvCKlQfIE638DNgVOIstdvxk4tpGNMjPrDwWK1bkWcrotPXye1zcgMDMrNKFCrQ1SbVLMFaQ1rHsSEf/ckBaZmfWHNlp176x+a0Xyv8aM4Prf/7C/L2tmJdUWOeuImNufDTEz608COgsUrL02tZmVVr12N5c0WNI8SfdIWpB21ULSVpJukLQofd2y4j3TJC2W9ICkg2u2dUO+UTOzIqtXsAZeBt4fEbsDewATJb0TmArMjYjRwNz0HEljgcnAOGAicI6kzqptzftNSRqUt6yZWavLtvVSrqOWyPw9Pd0oHQFMAmak8zOAw9PjScDMiHg5IpYAi4EJ1a6RZ6eYCZLuBRal57tL+s+arTcza3F96FlvI+n2iuP47nVJ6pR0N7ACuCENex4SEcsB0teuCYXDgMcr3r40netVnkkxPwYOA/47XfAeSZ5ubmaF14f7i6siYu9qBSJiLbCHpC2AKyS9vdqle6qiWv15gnVHRDza7aPA2hzvMzNrWQIGNGA0SET8TdJNZLnopyQNjYjlkoaS9boh60mPqHjbcGBZtXrz5KwflzQBiNTN/zzwYJ+/AzOzFiPlO2rXo21TjxpJGwMfAO4HZgNTUrEpwJXp8WxgsqRBkkYBo4F51a6Rp2f9abJUyPbAU8Dv0jkzs8KS6jrdfCgwI43o6ABmRcRVkm4BZkk6DngMOAIgIhZImgXcB6wBTkhplF7lWRtkBdkQEzOztlKvWB0Rfwb27OH808CBvbxnOtnieLnk2SnmZ/SQ+I6IN9wNNTMrkhZZqjqXPGmQ31U8Hgz8E+sOOTEzKxxBy2wskEeeNMgllc8lXQjc0LAWmZn1h/yzE1tCnp51d6OAHerdEDOz/qaW2WGxtjw567/yes66A3iGNL/dzKyoRBv1rNPei7uT7bsI8GpEVJ1lY2ZWFEUK1lUnxaTAfEVErE2HA7WZtY16LeTUH/LMYJwnaXzDW2Jm1o8k6OzId7SCanswDoiINcD+wCclPQS8QJbqiYhwADezQmuLDXPJ5qmP5/X1V83M2kY73WAUQEQ81E9tMTPrVwXqWFcN1ttK+kJvL0bEDxrQHjOzfiI62mScdSewKT0vkm1mVmiifXrWyyPilH5riZlZfxIMKFDSumbO2sysHbVTz7rHNVjNzNpFWwzdi4hn+rMhZmb9rUCxer1W3TMzKzyRbwp3q3CwNrNyUpukQczM2lk2g9HB2sys5RUnVDtYm1mJFahj7WBtZmXVOmtV5+FgbWal5NEgZmYF4RuMZmatThQqDVKkTwFmZnXTlQbJc9SsSxoh6f9JWihpgaTPpfNbSbpB0qL0dcuK90yTtFjSA5IOrnUNB2szK606bpi7BvhiRLwNeCdwgqSxwFRgbkSMBuam56TXJgPjgInAOZI6q13AwdrMSks5j1oiYnlE3JkePw8sBIYBk4AZqdgMXt8mcRIwMyJejoglwGJgQrVrOGdtZqUkoDN/znobSbdXPD83Is7tsV5pJLAncBswJCKWQxbQJW2Xig0Dbq1429J0rlcO1mZWWn24v7gqIvauXZ82BS4DPh8Rz1VJofT0QlSr22kQMysp5f4vV23SRmSB+lcRcXk6/ZSkoen1ocCKdH4pMKLi7cOBZdXqd7A2s9KS8h2165GA84GF3TYTnw1MSY+nAFdWnJ8saZCkUcBoYF61azgNYmallA3dq9s46/2AjwL3Sro7nfsacBowS9JxwGPAEQARsUDSLOA+spEkJ0TE2moXcLA2s3LK2WvOIyJupveBIz1ukRgR04Hpea/hYG1mpeXp5mZmLS7bfKDZrcjPwdrMSivvSI9W4GBtZqVVoCyIg7WZlZd71mZmLc45azOzIpA8GsTMrAiKE6odrM2spLI0SHHCtYO1mZVWcUK1g7WZlVmBorWDtZmVltMgZmYFUJxQ7WBtZmVWoGjtYG1mpZRthlucaO1gbWblVMf1rPuDg7WZlVaBYrX3YGwVa9eu5QP778MxRx7+2rnzfno2++01jve8Y3dO+ebUJrbOmu36Odex27gxjNt1Z844/bRmN6dNCCnf0Qoa1rOW9HPgMGBFRLy9UddpFz/7yX8yesyuPP/88wDc/IebmHP1b7nxf+5k0KBBrFy5okYN1q7Wrl3L5086gauvvYFhw4ez/zv34bDDPszbxo5tdtMKr0XicC6N7FlfAExsYP1tY9kTS/ndnGs5+mPHvnZuxvk/5bP/9mUGDRoEwLbbbtes5lmTzZ83j5122plRO+7IwIEDOeIjk7nqt1fWfqNVpT4craBhwToi/gA806j628k3p36Rb55yKup4/X/Hww8t4tZbbuaQ9+/H4R88kLvuuL2JLbRmWrbsCYYPH/Ha82HDhvPEE080sUVtpEDRWhHRuMqlkcBV1dIgko4Hjk9PxwAPNKxBrenN6XgM2AwYAiwGdgP+CjwObALsBNzbpDZac20JbA68AKwCtgLeRPa7UVY7RMS2G1LBuN3Gx8VX/yFX2d233+yOiNh7Q663oZo+GiQizgXObXY7mkXSqcBHgW2AwcBA4H5gNPCRiLgplXsIOCQiVjapqdYkkt4FnAxsHRF7S5oGEBGnNrVhbcA5a8stIqZFxPCIGAlMBm6MiGOAvwHvB5C0C1kQX9W0hlozzSf74z1Q0kCy35PZzW1SG0jjrPMcraDpPWvr1SpgR0l/AVYDU6KROStrWRGxRtKJwOXAQuDnEbGgyc1qC0WawdiwnrWki4FbgDGSlko6rlHXahcRcVNEHJae/jQijomIt0fE+Ii4samNs6aKiGuAkyJip4iY3uz2tANRv561pJ9LWpE6V13ntpJ0g6RF6euWFa9Nk7RY0gOSDs7T3kaOBjkqIoZGxEbpY/75jbpWO0q5fLPX+Hei/uo4GOQC3jhUeSowNyJGA3PTcySNJUtljUvvOUdSZ60LOGdtZuVVp2jdy1DlScCM9HgGcHjF+ZkR8XJELCEb/TWh1jWcszaz0urD5gPbSKqc7HBujk86QyJiOUBELJfUNbNtGHBrRbml6VxVDtYtRtJE4EygEzgvIrwQRMl56YbG6cPtxVV1HGfd02VrDh5wGqSFpLzV2cAhwFjgqJTfsnK7AC/d0BiNncH4lKShAOlr1wI/S4ERFeWGA8tqVeZg3VomAIsj4uGIWA3MJMtvWYl56YbG6Np8IM9/62k2MCU9ngJcWXF+sqRBkkaRjaGfV6syp0FayzDWnUK8FHhHk9pi1t7qOOElDVU+gCy3vRT4NnAaMCsNW34MOAIgIhZImgXcB6wBToiItbWu4WDdWtYrl2Vm66deU2Ii4qheXjqwl/LTgT6Nl3ewbi3rlcsys/XROhsL5OGcdWuZD4yWNMprQJg1XpHWBnGwbiERsQY4EZhDtgbELK8BYV66oTGKtvmA0yAtJq0BcU2z22Gto0o+1DZUq0TiHByszay0irTqnoO1mZVWq+Sj83CwNrNyEnQ4WJuZFUFxorWDtZmVUtfmA0XhoXu2DklrJd0t6S+SLpW0yQbUdYCkq9LjD0uaWqXsFpI+sx7XOFnSl/Ke71bmAkn/0odrjazcCcSKr0hD9xysrbuXImKPtBTnauBTlS8q0+ffm4iYXWO51y2APgdrsw3hSTHWLv4I7Jx6lAslnQPcCYyQdJCkWyTdmXrgm0K2Hrek+yXdDPxzV0WSPi7prPR4iKQrJN2Tjn3JFr3ZKfXqz0jlvixpvqQ/S/pORV1fT3vX/Q4YU+ubkPTJVM89ki7r9mnhA5L+KOlBSYel8p2Szqi49v/d0B+ktSZJuY5W4GBtPZI0gGxd7XvTqTHALyNiT+AF4BvAByJiPHA78AVJg4GfAR8C3g28pZfqfwz8PiJ2B8YDC8j2p3so9eq/LOkgsqUjJwB7AHtJeo+kvcim4e9J9sdgnxzfzuURsU+63kKgcgbgSOC9wKHAf6Xv4Tjg2YjYJ9X/ybSUpbWZIqVBfIPRuttY0t3p8R+B84G3Ao9GRNdWRO8k2xzhT6nXMZBsOvSuwJKIWAQg6SLg+B6u8X7gYwBpachnK3d+Tg5Kx13p+aZkwXsz4IqIeDFdI8/aKW+X9D2yVMumZNP5u8yKiFeBRZIeTt/DQcBuFfnsN6drP5jjWlYQrZTiyMPB2rp7KSL2qDyRAvILlaeAG7pPg5a0B/Vb0lXAqRHx027X+Px6XOMC4PCIuEfSx8nWHe7Sva5I1/5sRFQGdSSN7ON1rcUVaQaj0yC2Pm4F9pO0M4CkTSTtAtwPjJK0UyrX25oWc4FPp/d2StoceJ6s19xlDnBsRS58WNpw9A/AP0naWNJmZCmXWjYDlkvaCDi622tHSOpIbd4ReCBd+9OpPJJ2kfSmHNexoilQHsQ9a+uziFiZeqgXSxqUTn8jIh6UdDxwtaRVwM1ATxu8fg44N60etxb4dETcIulPaWjctSlv/TbgltSz/ztwTETcKekS4G7gUbJUTS3fBG5L5e9l3T8KDwC/B4YAn4qIf0g6jyyXfaeyi68EDs/307EiaZE4nIsivBGJmZXPHuP3jhv/eFuusltvOuCOOu5uvl7cszazUvIMRjMzqzv3rM2stIrUs3awNrPSKtLQPQdrMysnT4oxM2t9RbvB6GBtZqXlNIiZWQEUqWftoXtmVlr1nG2elgd+QNLiahttrC8HazMrrzpFa0mdwNlkywqPBY6SNLaeTXWwNrNSEtAh5TpymAAsjoiHI2I1MBOYVM/2OmdtZqV05513zNl4I22Ts/hgSbdXPD83Is6teD4MeLzi+VLgHRvaxkoO1mZWShExsY7V9dT9rusqeU6DmJltuKXAiIrnw4Fl9byAg7WZ2YabD4yWNErSQLJ9QvNsOZeb0yBmZhsoItZIOpFsl6FO4OcRsaCe1/DmA2ZmBeA0iJlZAThYm5kVgIO1mVkBOFibmRWAg7WZWQE4WJuZFYCDtZlZAfx/wWI5ewAth1YAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "skplt.metrics.plot_confusion_matrix(y, yPredProb2[:,1]>0.5,\n",
    "                                    normalize=False);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[630,   0],\n",
       "       [ 46,   0]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_matrix(y, yPredProb2[:,1]>0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not really any better. Not an uncommon result when doing machine learning, unfortunately. And all ML classifiers struggle with very unbalanced distributions of target variable labels.  Also, note the difference model performance between training and test CV folds. (There are other ways to do this, btw):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    20.000000\n",
       "mean      0.074472\n",
       "std       0.069270\n",
       "min      -0.029596\n",
       "25%       0.029262\n",
       "50%       0.040052\n",
       "75%       0.127751\n",
       "max       0.191856\n",
       "dtype: float64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def difCols(x):\n",
    "    return(x[0]-x[1])\n",
    "\n",
    "pd.DataFrame(svcRes2)[['train_score','test_score']].apply(\n",
    "    difCols,axis=1).describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When a model's accuracy decreases when it is applied to \"new\" data, data it hasn't \"seen\" before, it can be symptomatic of _over-fitting_ the training data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A UDU 4 U: Something to Try\n",
    "\n",
    "A source of variation might result from the way that CV was done, here.  Try using _stratified_ K fold cross-validation here to see what difference it might make.\n",
    "\n",
    "A QUESTION:  Do you think that adding additional features to your SVM model would improve its performance?\n",
    "\n",
    "ANOTHER QUESTION: Would _increasing_ the amount of regulatization result in any improvement?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What, Now?\n",
    "\n",
    "Various tactics can be used to deal with extremely unbalanced target label classes.  They include \"data augmentation\" (adding additional data to the rare class), synthesizing cases for the rare class (AKA \"making up data,\" but perhaps more defensible), downsampling (reducing the prevalent class), and various specialized algorithms. \n",
    "\n",
    "A frequently used technique:\n",
    "\n",
    "[SMOTE: Synthetic Minority Over-sampling Technique](https://arxiv.org/pdf/1106.1813.pdf)\n",
    "\n",
    "See also:\n",
    "\n",
    "[SMOTE for high-dimensional class-imbalanced\n",
    "data](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3648438/pdf/1471-2105-14-106.pdf)\n",
    "\n",
    "Here's a Python package that includes many such tools: [imbalanced-learn](https://imbalanced-learn.readthedocs.io/en/stable/user_guide.html#)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Shelving Cervical Data\n",
    "\n",
    "We'll serialize some of the data we've worked with here for subsequent use.  If we can't access this shelve database for some reason, we can just read the csv file again. (Less fun, though.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "cervicaldb='cervical'\n",
    "with shelve.open(cervicaldb) as sdb:\n",
    "    sdb['cervical']=cervical\n",
    "    sdb['cervical2']=cervical2\n",
    "    sdb['X']=X\n",
    "    sdb['y']=y\n",
    "sdb.close()"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": false,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}