16-EX-Trees-v1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creative Commons CC BY 4.0 Lynd Bacon & Associates, Ltd. Not warranted to be suitable for any particular purpose. (You're on your own!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification and Regression Trees\n",
    "\n",
    "There are many different algorithms that use tree-structured partitionings of features to predict target variable values.  They include:  \n",
    "\n",
    "* CART\n",
    "* C 5.0\n",
    "* Random Forest (an _ensemble_ method)\n",
    "* CHAID ($\\chi^2$ interaction detection)\n",
    "* Various boosting implmentations\n",
    "\n",
    "## CART \n",
    "\n",
    "CART \"trains\" by recursively partitioning, or splitting the data in a sequence of steps that each attempts to best explain, or discriminate between, target variable labels.  It can be used for regression and for classification problems. In the case of the latter, CART can be used to predict k labels, where k $\\ge$ 2. \n",
    "\n",
    "It is a _greedy algorithm_: it attempts to optimize predictive accuracy binary split by split, rather than on an overall basis.  As a result, it may not find the best, or optimal, tree.  Classifier tree splits are determined based on the resulting quality of the results of the split based on a measure of homogeneity, [_Gini Impurity_](https://en.wikipedia.org/wiki/Decision_tree_learning#Gini_impurity), or in terms of [entropy](https://en.wikipedia.org/wiki/Entropy#Information_theory). The utility of regression tree splits are measured in terms of mean squared predictor error, and variations on it.\n",
    "\n",
    "\n",
    "CART requires making few assumptions about the data.  It tends to overfit.  Overfitting can be mitigated by regularization using hyperparameters that limit tree size, minimum size of the result of a split, the number of features used for splitting, and the final number of distinct partitions of the data.\n",
    "\n",
    "A CART model can be _examined_:  it can be represented as a hierarchy of \"if then\" rules.  It's not \"black box.\"\n",
    "\n",
    "CART and other tree models have some specific terminology:\n",
    "\n",
    "* A partition is called a _node_;\n",
    "* Splits result in _branches_ that lead to new, smaller nodes that descend from \"parent\" nodes;\n",
    "* The final nodes are sometimes called _leaves_.\n",
    "* The pre-split data is usually called the _root_ node.\n",
    "\n",
    "This terminology becomes more clear when looking \n",
    "\n",
    "The CART algorithm was originally described by [Breiman, Freedman, Stone & Olshen (1984)](https://www.amazon.com/Classification-Regression-Wadsworth-Statistics-Probability/dp/0412048418). \n",
    "\n",
    "Let's give using CART on the cervical CA data a try."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\"\n",
    "from IPython.display import display, Markdown\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import scikitplot as skplt\n",
    "from sklearn import linear_model\n",
    "from sklearn.model_selection import KFold, StratifiedShuffleSplit\n",
    "from sklearn.model_selection import cross_val_predict, train_test_split\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn import preprocessing\n",
    "from sklearn.metrics import accuracy_score, confusion_matrix, roc_auc_score\n",
    "from sklearn.metrics import roc_curve, precision_score, recall_score\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "import pickle\n",
    "import os\n",
    "import shelve\n",
    "from sklearn.tree import DecisionTreeClassifier, export_graphviz \n",
    "%matplotlib inline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting the Data\n",
    "\n",
    "Let's get the data in that `shelve` database. \n",
    "\n",
    "Assuming they are in the data subdirectory of the current pwd: (Adjust as necessary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "with shelve.open('DATA/ML/cervical') as sdb:\n",
    "    cervical=sdb['cervical']  # this should be aPandas DataFrame\n",
    "    cervical2=sdb['cervical2']  # this should be, too"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "X=cervical2.iloc[:,1:].to_numpy()\n",
    "y=cervical2.iloc[:,0].to_numpy()\n",
    "feat_nams=cervical2.columns.to_list()[1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CART Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we're using the [scikit-learn DecisionTreeClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "cTree=DecisionTreeClassifier(max_depth=2,\n",
    "                             min_samples_leaf=20,\n",
    "                             min_samples_split=50)\n",
    "tree_clf1=cTree.fit(X,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export to dot file, use the dot program to create a png file.\n",
    "export_graphviz(\n",
    "    tree_clf1,\n",
    "     out_file='cTree4.dot',  #doesn't seem to overwrite by default\n",
    "    feature_names=feat_nams,\n",
    "    rounded=True,\n",
    "    filled=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After exporting the dot file, it is converted to a png file using the dot program from the command prompt. Then the png is inserted into this Notebook:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# dot program command; > is the terminal command prompt:\n",
    "\n",
    "> dot -Tpng cTree.dot -o cTree.png\n",
    "\n",
    "Then in a markdown cell, do something like the following:\n",
    "\n",
    "![cTree](cTree.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![cTree](cTree4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CART Predictions, etc.\n",
    "\n",
    "Predictions can be obtained from either type of CART model.  Here we'll examine the predictions the classifier above produces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "cervPred=tree_clf1.predict(X)\n",
    "cervPredProb=tree_clf1.predict_proba(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accuracy, AUC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CART accuracy: 0.932 AUC: 0.600\n"
     ]
    }
   ],
   "source": [
    "print(\"CART accuracy: {:5.3f} AUC: {:5.3f}\".format(\n",
    "        accuracy_score(y,cervPred),\n",
    "        roc_auc_score(y,cervPredProb[:,1]))\n",
    "     )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Precision and Recall"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "print(\"CART precision: {:5.3f} Recall: {:5.3f}\".format(\n",
    "        precision_score(y,cervPredProb),\n",
    "        recall_score(y,cervPredProb))\n",
    "     )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcIAAAG5CAYAAAAd/TRHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU9b3H/9cn+77IorIoiCgCgQABQVmrdWldW73a0lqXW2uttrf7cn+t3HvtbW+1Vq223v6sP6q1arW1y61trTVsIkqAhE1BQIEAypaV7Mn398dMDpmQZRIyme39fDzmYc6c75z5zCHOO9/v+Z5zzDmHiIhIvEoIdwEiIiLhpCAUEZG4piAUEZG4piAUEZG4piAUEZG4piAUEZG4piAUOUlmdrOZrQrh9v9qZp/psHyvmR02s/fN7AwzqzWzxFC9v0isUxBKRDKzT5pZif9L/oA/DOZ2anOzmTkz+5dOzy80szb/a2vMbJuZ3eJf1x4c7Q9nZsc6LM/rpp5LzWyFf3uHzGy5mV0Vuj1wnHPucufcr/x1jAa+Ckx0zp3mnNvjnMtyzrUOxHuZ2RIza/bvi0ozW21mc/zrbjazVv+6ajMrM7Mretlejpk9aGZ7/K/b4V8eOhD1igwEBaFEHDP7CvAg8N/AqcAZwM+Aqzs1/Qxw1P/fzvY757KAHODLwP9rZud2CI4s/3qAqR2eW9lFPdcBzwNPAqP8NX0PuPJkP2s/nAkccc4dPNkNmVlSN6ue8++bYcAq4PdmZv51r/vX5eH7N3nWzPK62X4K8E9gEnAZvn+LC4AjwKwBrFfk5Djn9NAjYh5ALlALXN9LuzOBNuDjQAtwaod1C4HyTu0PdrVNwAFn9/A+BuwBvt5Dm5uBVR2WHwL2AtXAOmBeh3WzgBL/ug+AB/zPpwG/xhcSlcDa9s8ELAP+FbgYqPd/7lpgKTDG/xmSOuy/XwIHgH3AvUBihzpfA36C7w+Ie7v4LEuAX3dYnuTf/tAuPmeGf93MbvbLv/o/Y1YP+y5g//s/070d/x2BbwLvA08BbwFXdGifBBwGpvuXZwOr/fuwDFgY7t9pPSL/oR6hRJo5+ELhxV7a3QSUOOd+h+/LcXFXjcwswT+EORTY0Y96zgVGAy/04TVrgULgFOA3wPNmluZf9xDwkHMuBxgH/Nb//GfwhdhoYAhwB77Q8zjnXgEux9/bdc7d3MV7/wrfHwZnA9OAS/AFUrvzgV3AcOD7PX0IM0vFF37lzrnDndYlArcAzcDubjZxMfA351xtT+/Ti9Pw7cczgduBZ4BPdFh/KXDYObfezEYCf8EX/qcAXwN+Z2bDTuL9JQ4oCCXSDMH3xdbSS7ub8IUM/v92Hh4dYWaV+MLkReArzrkN/awHfD2soDjnfu2cO+Kca3HO/RhIxReo4AuOs81sqHOu1jm3psPzQ/D1jlqdc+ucc9V9KdTMTsUXlP/mnDvmfMOnPwFu7NBsv3Pup/7a6rvcEPyLf9/tBWYA13RYN9u/rgG4H/iU636Ydgh92G/daAPucc41+uv9DXCVmWX413+S478HnwJecs695Jxrc879A1/v+yMnWYPEOAWhRJojwNCejgeZ2YXAWOBZ/1O/AQrMrLBDs/3OuTx8x6UeBj50EvUAnB7sC8zsq2b2lplV+UMjF1+PFOA24BzgbTNb22GyyVPA3/Edc9tvZj8ys+Q+1nomkAwc8E90qQT+F1/vr93eILbzW+dcnnNuuHPuQ865dR3WrfHv13zgT0CXk4v8jtCH/daNQ865hvYF59wOfCMAV/rD8CqOB+GZwPXtn93/+ecOQA0S4xSEEmlex9fbuKaHNp/Bd+yu1MzeB97wP39T54bOuUZ8x5gKzKynbXZnG77w+Hgwjf2zTr8J/AuQ7w+NKn+9OOfecc59Al84/Q/wgpllOueanXP/4ZybiG9CyRVdfZ5e7AUagaH+IMtzzuU45yZ1aDMgt5vxD3feCXzazKZ10+wV4FIzy+xhU3X4jjW2O63zW3Xxmvbh0auBrf5wBN/nf6rDZ89zzmU6537Y2+eR+KYglIjinKvCNyPzUTO7xswyzCzZzC7395LS8IXM7fiOw7U/7gYWd9WTdM41AT/2b7ev9TjgK8B3zewW/+kACWY218x+0cVLsvEdozsEJJnZ9/D1SgEws0+Z2TDnXBu+CR0ArWa2yMwK/MfeqvENlfbplAjn3AHgZeDHHeocZ2YL+vq5g3y/I8DjdL9fn8IXTr8zswn+eoaY2XfMrH24shT4pJklmtllQDC1Povv2OfnOd4bBN9koyv9p7okmlma/1SaUf35fBI/FIQScZxzD+ALn/8HX6DsBe4C/oCvp1gPPOmce7/9gW+mZCK+afpdeQI4w8z6fMqDc+4F4AbgVmA/vpmQ9wJ/7KL534G/AtvxTSJpIHA48jJgi5nV4ps4c6N/6O80fBNyqvEN/S3H98XeVzcBKcBWoMK/zVAODT4IfMTMpnRe4e+NXwy8DfwD32d7E98wcXsv/kv4TkOpxDfh6Q+9vaE/8F/H13N+rsPze/H1Er/D8d+br6PvOemF+f7gFRERiU/6S0lEROKaglBEROKaglBEROKaglBEROJa1F3EdujQoW7MmDHhLkNERCLIunXrDjvn+nU5vagLwjFjxlBSUhLuMkREJIKYWXfXvO2VhkZFRCSuKQhFRCSuKQhFRCSuRd0xQokdzc3NlJeX09DQ0HtjEREgLS2NUaNGkZzc15uzdE9BKGFTXl5OdnY2Y8aMwczCXY6IRDjnHEeOHKG8vJyxY8cO2HY1NCph09DQwJAhQxSCIhIUM2PIkCEDPoqkIJSwUgiKSF+E4jtDQSgiInFNQSgiInFNQShxLTExkcLCQiZPnsz1119PXV3dSW+zpKSEL37xi92u379/P9ddd91Jv08oLV26lLvuuguAJUuWcP/994e5ouA0NjZy8cUXU1hYyHPPPddr+/fee4/JkyeHpJYVK1Ywffp0kpKSeOGFF7ptV19fz4IFC2htbQ1JHQPhb3/7G+eeey5nn302P/zhD7ts8/bbbzNnzhxSU1NP+H3p7vVLlixh5MiRFBYWUlhYyEsvvQTApk2buPnmm0P2eTpTEEpcS09Pp7S0lM2bN5OSksJjjz0WsN45R1tbW5+2WVRUxMMPP9zt+hEjRvT4xdhfLS0tA77NUAlVrRs2bKC5uZnS0lJuuOGGkLxHsM444wyWLl3KJz/5yR7bPfHEE3zsYx8jMTExqO3253fyZLS2tvKFL3yBv/71r2zdupVnnnmGrVu3ntDulFNO4eGHH+ZrX/tan17/5S9/mdLSUkpLS/nIRz4CQEFBAeXl5ezZsye0H85PQSgRwf5pIXn0xbx589ixYwfvvfce5513HnfeeSfTp09n7969vPzyy8yZM4fp06dz/fXXU1tbC8DatWu54IILmDp1KrNmzaKmpoZly5ZxxRVXALB8+XLvr91p06ZRU1MT0AtpaGjglltuoaCggGnTplFcXAz4emQf+9jHuOyyyxg/fjzf+MY3uqx56dKlXH/99Vx55ZVccsklANx3333MnDmTKVOmcM8993htn3zySaZMmcLUqVP59Kc/DcCf//xnzj//fKZNm8bFF1/MBx98EPT++uCDD7j22muZOnUqU6dOZfXq1Sf0sO6//36WLFkCwMKFC/nOd77DggUL+P73v8+YMWO8L/S6ujpGjx5Nc3MzO3fu5LLLLmPGjBnMmzePt99++4T3Pnr0KNdccw1Tpkxh9uzZbNy4kYMHD/KpT32K0tJSCgsL2blzZ8BrduzYwcUXX8zUqVOZPn36Cevfe+895s2bx/Tp05k+fTqrV68G4MCBA8yfP98bOVi5ciWtra3cfPPNTJ48mYKCAn7yk5+cUOOYMWOYMmUKCQk9f80+/fTTXH311QDU1tZy0UUXMX36dAoKCvjjH//o1Rbs7+R//ud/MnPmTCZPnsztt9+Oc67H9+/Nm2++ydlnn81ZZ51FSkoKN954o1dXR8OHD2fmzJknnN8X7Os7u/LKK3n22WdPqvZgKQhF8PVQ/vrXv1JQUADAtm3buOmmm9iwYQOZmZnce++9vPLKK6xfv56ioiIeeOABmpqauOGGG3jooYcoKyvjlVdeIT09PWC7999/P48++iilpaWsXLnyhPWPPvoo4BsKeuaZZ/jMZz7jTQ0vLS3lueeeY9OmTTz33HPs3bu3y9pff/11fvWrX/Hqq6/y8ssv88477/Dmm29SWlrKunXrWLFiBVu2bOH73/8+r776KmVlZTz00EMAzJ07lzVr1rBhwwZuvPFGfvSjHwW9z774xS+yYMECysrKWL9+PZMmTer1NZWVlSxfvpx77rmHqVOnsnz5csAXyJdeeinJycncfvvt/PSnP2XdunXcf//93HnnnSds55577mHatGls3LiR//7v/+amm25i+PDhPP7448ybN4/S0lLGjRsX8JrFixfzhS98gbKyMlavXs3pp58esH748OH84x//YP369Tz33HPe8PZvfvMbLr30UkpLSykrK6OwsJDS0lL27dvH5s2b2bRpE7fcckvQ+62jpqYmdu3aRfsdddLS0njxxRdZv349xcXFfPWrX/WCLJjfSYC77rqLtWvXsnnzZurr6/m///u/E9736aef9v5A6/joash+3759jB492lseNWoU+/btC/oz9vb6Rx55hClTpnDrrbdSUVHhPV9UVMTKlSuDfp+TEbIT6s3sCeAK4KBz7oRBePPNgX0I+AhQB9zsnFsfqnoksrmLTu6v1v6qr6+nsLAQ8PUIb7vtNvbv38+ZZ57J7NmzAVizZg1bt27lwgsvBHxfXnPmzGHbtm2cfvrpzJw5E4CcnJwTtn/hhRfyla98hcWLF/Oxj32MUaNGBaxftWoVd999NwATJkzgzDPPZPv27QBcdNFF5ObmAjBx4kR2794d8IXS7sMf/jCnnHIKAC+//DIvv/wy06ZNA3w9jHfeeYeysjKuu+46hg4dCuC1Ly8v54YbbuDAgQM0NTX16STlV199lSeffBLwHWvNzc0N+CLrSsfhyhtuuIHnnnuORYsW8eyzz3LnnXdSW1vL6tWruf766712jY2NJ2xn1apV/O53vwPgQx/6EEeOHKGqqqrb962pqWHfvn1ce+21gC9wOmtubuauu+6itLSUxMRE799h5syZ3HrrrTQ3N3PNNddQWFjIWWedxa5du7j77rv56Ec/6vXG++rw4cPk5eV5y845vvOd77BixQoSEhLYt2+f10sP5ncSoLi4mB/96EfU1dVx9OhRJk2axJVXXhnwvosXL2bx4sVB1dhVj7IvpzD09PrPf/7zfPe738XM+O53v8tXv/pVnnjiCcD3h8n+/fuDfp+TEcoryywFHgGe7Gb95cB4/+N84Of+/4oMmvZjhJ1lZmZ6Pzvn+PCHP8wzzzwT0Gbjxo29fiF861vf4qMf/SgvvfQSs2fP5pVXXgn4Eu5p2Co1NdX7OTExkZaWFl588UX+4z/+A4DHH3+8y1q//e1v87nPfS5gWw8//HCXtd5999185Stf4aqrrmLZsmXeMGZ/JSUlBRy/6nzic8dar7rqKr797W9z9OhR1q1bx4c+9CGOHTtGXl5el/8mHfX1yzmY4cGf/OQnnHrqqZSVldHW1ub9O82fP58VK1bwl7/8hU9/+tN8/etf56abbqKsrIy///3vPProo/z2t7/1vsD7Ij09PWAfPf300xw6dIh169aRnJzMmDFjvPXB/E42NDRw5513UlJSwujRo1myZEmXJ58//fTT3HfffSc8f/bZZ59w/HrUqFEBoxHl5eWMGDEi6M/Y0+tPPfVU7/nPfvaz3iGF9s/SeQQlVEI2NOqcWwEc7aHJ1cCTzmcNkGdmp/fQHoCd9TupaakZqDJFejV79mxee+01duzYAfiOZ23fvp0JEyawf/9+1q5dC/h6HZ0ngezcuZOCggK++c1vUlRUdMLxrvnz5/P0008DsH37dvbs2cO5557bbS3XXnutN7GgqKjohPWXXnopTzzxhHe8aN++fRw8eJCLLrqI3/72txw5cgTwHWMDqKqqYuTIkQD86le/6tN+ueiii/j5z38O+CZEVFdXc+qpp3Lw4EGOHDlCY2Njl8Ny7bKyspg1axZf+tKXuOKKK0hMTCQnJ4exY8fy/PPPA74v/LKyshNe23G/LVu2jKFDh3bZI2+Xk5PDqFGj+MMf/gD4epmdZwhXVVVx+umnk5CQwFNPPeXN4ty9ezfDhw/ns5/9LLfddhvr16/n8OHDtLW18fGPf5z/+q//Yv36/g1m5efn09ra6oVVVVUVw4cPJzk5meLiYnbv7voWe939TrZvZ+jQodTW1nY7KWvx4sXe71HHR1ftZ86cyTvvvMO7775LU1MTzz77LFdddVXQn7Gn1x84cMBr9+KLLwYcX96+fXvIZvR2Fs5rjY4EOh70KPc/d6BzQzO7HbgdgHOgtKaUefnzBqNGEYYNG8bSpUv5xCc+4Q3T3XvvvZxzzjk899xz3H333dTX15Oens4rr7wS8NoHH3yQ4uJiEhMTmThxIpdffnnA//x33nknd9xxBwUFBSQlJbF06dKAnmBfXXLJJbz11lveMFlWVha//vWvmTRpEv/+7//OggULSExMZNq0aSxdupQlS5Zw/fXXM3LkSGbPns27774b9Hs99NBD3H777fzyl78kMTGRn//858yZM4fvfe97nH/++YwdO5YJEyb0uI0bbriB66+/nmXLlnnPPf3003z+85/n3nvvpbm5mRtvvJGpU6cGvG7JkiXccsstTJkyhYyMjKBC/KmnnuJzn/sc3/ve90hOTub5558PmMhy55138vGPf5znn3+eRYsWeT2wZcuWcd9995GcnExWVhZPPvkk+/bt45ZbbvF6vz/4wQ9OeL+1a9dy7bXXUlFRwZ///GfuuecetmzZckK7Sy65hFWrVnHxxRezePFirrzySoqKiigsLOx2//X0O/nZz36WgoICxowZ4w3bn4ykpCQeeeQRLr30UlpbW7n11lu948Hts6zvuOMO3n//fYqKiqiuriYhIYEHH3yQrVu3kpOT0+3rv/GNb1BaWoqZMWbMGP73f//Xe9/i4mI++tGPnnT9wbCTnVHU48bNxgD/180xwr8AP3DOrfIv/xP4hnNuXY/bPNfcA/94gC+f8eUQVCyD6a233uK8884LdxkiYbVhwwYeeOABnnrqqXCXEjEaGxtZsGABq1atIinpxP5aV98dZrbOOXfiMEkQwjlrtBzoeOR/FBDUkdF1NT1mpYhI1Jg2bRqLFi2K6BPqB9uePXv44Q9/2GUIhkI4g/BPwE3mMxuocs6dMCzalZLqktBWJoMmlCMSItHi1ltvDfqE+ngwfvx4Fi5c2OW6UHxnhPL0iWeAhcBQMysH7gGSAZxzjwEv4Tt1Yge+0yeCOhHHMLbXbae6pZqcpO4PjkvkS0tL48iRI7oVk4gEpf1+hF2d/nIyQhaEzrlP9LLeAV/o63bTE9Kpo44NNRtYkL+g3/VJ+I0aNYry8nIOHToU7lJEJEq036F+IEXdHeozEjOoo451NesUhFEuOTl5QO8yLSLSH1F3ibWMxAwA1lVrwoyIiJy8qAvCzATfuT0lNZowIyIiJy/qgjA9MZ0US/EmzIiIiJyMqAtCwyjI8t0hYH2NrtEtIiInJ+qCEKAox3fxAB0nFBGRkxWVQTgjewagK8yIiMjJi8ogbO8R6gozIiJysqIyCCdlTiLFUnin/h2qWrq/GaeIiEhvojIIUxJSmJrtuy3L+mpNmBERkf6LyiAEHScUEZGBoSAUEZG4FrVBqAkzIiIyEKI2CCdlTiI1IZUd9TuobK4MdzkiIhKlojYIkxOSmZrlnzCjK8yIiEg/RW0Qgo4TiojIyYvuIMzxBaGOE4qISH9FdRAWZfuvOaoeoYiI9FNUB+HEzImkJqSys34nFc0V4S5HRESiUFQHYXJCMoVZhYAmzIiISP9EdRCCJsyIiMjJifog1In1IiJyMqI+CNUjFBGRkxH1QTgxcyJpCWnsqt+lCTMiItJnUR+ESQlJ3oQZ9QpFRKSvoj4I4fiJ9euqFYQiItI3MRGE7SfWl9RowoyIiPRNTASheoQiItJfMRGE52WcR3pCOu82vMvR5qPhLkdERKJITARhUkIShdn+CTPqFYqISB/ERBCCzicUEZH+iZkg1BVmRESkP2ImCNUjFBGR/oiZIJyQMYGMhAzea3iPI81Hwl2OiIhEiZgJQk2YERGR/oiZIAQNj4qISN/FVBBqwoyIiPRVTAWheoQiItJXMRWEEzJ9E2Z2N+zmcNPhcJcjIiJRIKaCMNESmZY9DVCvUEREghNTQQgaHhURkb6JuSDUhBkREemLmAtC9QhFRKQvYi4Iz808l8zETPY07OFQ06FwlyMiIhEu5oIw0RKZlqUJMyIiEpyYC0I4fsd6HScUEZHexGQQFmX7JsyoRygiIr2JySBs7xHq4tsiItKbmAzCczLOISsxi72NeznYdDDc5YiISASLySAMuMKMeoUiItKDmAxCOH6csKRGE2ZERKR7MRuEOk4oIiLBiN0g1BVmREQkCDEbhO0TZsoby/mg8YNwlyMiIhEqZoMwwRKYnj0dUK9QRES6F7NBCLoThYiI9C6mg1DHCUVEpDcKQhERiWsxHYTjM8aTnZjNvsZ9vN/4frjLERGRCBTTQagJMyIi0puYDkLQhBkREelZzAehjhOKiEhPYj4I1SMUEZGexHwQjksfR05iDgeaDnCg8UC4yxERkQgT0iA0s8vMbJuZ7TCzb3WxPtfM/mxmZWa2xcxuGegaNGFGRER6ErIgNLNE4FHgcmAi8Akzm9ip2ReArc65qcBC4MdmljLQtWh4VEREuhPKHuEsYIdzbpdzrgl4Fri6UxsHZJuZAVnAUaBloAvRhBkREelOKINwJLC3w3K5/7mOHgHOA/YDm4AvOefaOm/IzG43sxIzKzl06FCfC1GPUEREuhPKILQunnOdli8FSoERQCHwiJnlnPAi537hnCtyzhUNGzasz4WMSx9HblIu7ze9z/7G/X1+vYiIxK5QBmE5MLrD8ih8Pb+ObgF+73x2AO8CEwa6EDM7PmFGd6wXEZEOQhmEa4HxZjbWPwHmRuBPndrsAS4CMLNTgXOBXaEopijbPzxao+FRERE5LilUG3bOtZjZXcDfgUTgCefcFjO7w7/+MeC/gKVmtgnfUOo3nXOHQ1HPjBz/hBn1CEVEpIOQBSGAc+4l4KVOzz3W4ef9wCWhrKFdxx6hcw7fRFUREYl3MX9lmXZnpZ9FXlIeHzR9oAkzIiLiiZsgNDOdTygiIieImyCE48cJdT6hiIi0i68gVI9QREQ6iasg7HiFGec6n9svIiLxKK6CcGzaWPKT8jnYfJB9jfvCXY6IiESAuApCMzt+nFAn1ouICHEWhNDhOKFOrBcREeI5CDVhRkREiMMg1IQZERHpKO6CcEzaGPKT8jnUfIjyxvJwlyMiImEWd0FoZrpRr4iIeOIuCEHHCUVE5DgFoYiIxLW4DEJNmBERkXZxGYRnpp3JKUmncLj5MHsb94a7HBERCaO4DEJNmBERkXZxGYSg44QiIuITt0HY3iPUpdZEROJb3AZhe4+wpEYTZkRE4lncBuEZaWcwJHkIR5qPsKdhT7jLERGRMInbIDQzirL9E2Z0SyYRkbgVt0EIePcm1HFCEZH4FddBqB6hiIjEdRB27BFqwoyISHyK6yAcnTqaoclDOdpylN0Nu8NdjoiIhEFcB6GuMCMiInEdhKArzIiIxLu4D0KvR6gJMyIicSnug9DrEWrCjIhIXIr7IByVOorhycOpaKngvYb3wl2OiIgMsrgPQjPzTqPQhBkRkfgT90EImjAjIhLPFISgUyhEROKYgpDjPcL1Nes1YUZEJM4oCIGRqSM5NeVUKloqeLfh3XCXIyIig0hBiH/CTLYmzIiIxCMFoZ8mzIiIxCcFoZ8mzIiIxCcFoV/HHqEmzIiIxA8Fod+I1BGclnIaVS1V7KzfGe5yRERkkCgI/TpOmNFxQhGR+KEg7KD9OOG6agWhiEi8UBB24J1CoVsyiYjEDQVhB+0X315fs5421xbmakREZDAoCDsYkTqC01NO14QZEZE4oiDspL1XqOOEIiLxQUHYSVG2f8KMZo6KiMQFBWEnukmviEh8URB20vGWTJowIyIS+xSEnZyeejojUkdQ3VrNjvod4S5HRERCTEHYBe8KM5owIyIS8xSEXWifMKMT60VEYp+CsAs6hUJEJH4oCLugCTMiIvFDQdiF01JPY2TqSGpaa3in7p1wlyMiIiGkIOyGbskkIhIfFITdaL8lk06sFxGJbQrCbqhHKCISHxSE3dCEGRGR+KAg7MapqacyKnUUta21bK/bHu5yREQkRBSEPdDwqIhI7FMQ9kATZkREYl9Ig9DMLjOzbWa2w8y+1U2bhWZWamZbzGx5KOvpK/UIRURiX1KoNmxmicCjwIeBcmCtmf3JObe1Q5s84GfAZc65PWY2PFT19Ef7pdY21Gyg1bWSaIlhrkhERAZaKHuEs4Adzrldzrkm4Fng6k5tPgn83jm3B8A5dzCE9fTZ8JThjE4drQkzIiIxLJRBOBLY22G53P9cR+cA+Wa2zMzWmdlNXW3IzG43sxIzKzl06FCIyu2aLsAtIhLbQhmE1sVzrtNyEjAD+ChwKfBdMzvnhBc59wvnXJFzrmjYsGEDX2kPdEsmEZHYFrJjhPh6gKM7LI8C9nfR5rBz7hhwzMxWAFOBiBmHVI9QRCS2hbJHuBYYb2ZjzSwFuBH4U6c2fwTmmVmSmWUA5wNvhbCmPut4hZlW1xrmakREZKCFLAidcy3AXcDf8YXbb51zW8zsDjO7w9/mLeBvwEbgTeBx59zmUNXUH8NShnFG2hnUtdWx7di2cJcjIiIDLJRDozjnXgJe6vTcY52W7wPuC2UdJ6sou4g9DXtYV7OOiVkTw12OiIgMIF1ZJgjtxwl1hRkRkdijIAyCrjAjIhK7gh4aNbORwJkdX+OcWxGKoiJNexDqCjMiIrEnqCA0s/8BbgC2Au1TJx0QF0E4NGUoZ6adye6G3Q/5OOUAAB3ESURBVLx97G0mZU0Kd0kiIjJAgu0RXgOc65xrDGUxkawou4jdDbtZV7NOQSgiEkOCPUa4C0gOZSGRThNmRERiU7A9wjqg1Mz+CXi9QufcF0NSVQTShBkRkdgUbBD+iROvChNXOt6SqaWthaSEkJ6CKSIigySob3Pn3K/8l0lrvyD2Nudcc+jKijxDkocwJm0M7zW8x9t1bzM5a3K4SxIRkQEQ1DFCM1sIvIPvRrs/A7ab2fwQ1hWRinJ8d6LQ8KiISOwIdrLMj4FLnHMLnHPz8d0y6SehKysytR8n1IQZEZHYEWwQJjvnvCtOO+e2E4ezSNUjFBGJPcHO+Cgxs18CT/mXFwNxlwbTs6cDUFpTqgkzIiIxItge4eeBLcAXgS/hu8LMHaEqKlKdknwKY9PGUt9Wz1t1EXXbRBER6adgZ402Ag/4H3GtKKeIdxvepaS6hIKsgnCXIyIiJ6nHHqGZ/db/301mtrHzY3BKjCw6sV5EJLb01iP8kv+/V4S6kGjhTZipVhCKiMSCHnuEzrkD/h8PA3udc7uBVGAqsD/EtUUkb8JMrW/CjIiIRLdgJ8usANL89yT8J3ALsDRURUWy/OR8zko/i4a2BrYe2xruckRE5CQFG4TmnKsDPgb81Dl3LTAxdGVFtqJs3/BoSY1OrBcRiXZBB6GZzcF3/uBf/M/F7Ul07Rfg1nFCEZHoF2wQ/hvwbeBF59wWMzsLKA5dWZGtvUeomaMiItEv2PMIlwPLOyzvwndyfVzqOGGmua2Z5IS4u9qciEjM6DEIzexB59y/mdmfAdd5vXPuqpBVFsHykvMYlz6OnfU72XpsK1Ozp4a7JBER6afeeoTt1xa9P9SFRJui7CJ21u+kpKZEQSgiEsV6O4+w/SBYCbDSObfcP0y6Clgb6uIimSbMiIjEhmAny/wTyOiwnA68MvDlRA9NmBERiQ3BBmGac662fcH/c0YP7WPe9BzfhJmy2jKa25rDXI2IiPRXsEF4zMymty+Y2QygPjQlRYfcpFzGp4+nsa2RLce2hLscERHpp76cR/i8ma00s5XAc8BdoSsrOrQfJyyp1hVmRESiVVBB6JxbC0zAd4PeO4HzOkykiVu6JZOISPQLKgjNLAP4JvAl59wmYIyZxf2tmdpvyaQeoYhI9Ap2aPT/A5qAOf7lcuDekFQURaZlTwNgY+1GmtqawlyNiIj0R7BBOM459yOgGcA5Vw9YyKqKErlJuZyTcQ5NrkkTZkREolSwQdhkZun4L7NmZuOAxpBVFUXajxNqeFREJDoFG4T3AH8DRpvZ0/hOsP9GyKqKIpowIyIS3Xq9+4SZGfA2vpvyzsY3JPol59zhENcWFTRhRkQkuvUahM45Z2Z/cM7N4PhNecWvfcLMptpNNLU1kZKQEuaKRESkL4IdGl1jZjNDWkmUyknK4dyMc2lyTWyu3RzuckREpI+CDcJF+MJwp5ltNLNNZrYxlIVFE2/CTI2GR0VEok1Qd6gHLg9pFVGuKKeI33zwG98tmUaGuxoREemL3u5QnwbcAZwNbAJ+6ZxrGYzCool6hCIi0au3odFfAUX4QvBy4MchrygKTcuehmFsqt1EY5tOrxQRiSa9BeFE59ynnHP/C1wHzBuEmqJOdlI252acS7NrZlPtpnCXIyIifdBbEHp3nNWQaM/ab8m0rlon1ouIRJPegnCqmVX7HzXAlPafzax6MAqMFkXZvhPrdYUZEZHo0uNkGedc4mAVEu00YUZEJDoFex6h9KJ9wszm2s00tDaEuxwREQmSgnCAZCVlMSFjgm/CzDFNmBERiRYKwgGkCTMiItFHQTiANGFGRCT6KAgHUHuPULdkEhGJHgrCAVSYVUgCCWw+pgkzIiLRQkE4gLKSspiQOYEW18LGWt2cQ0QkGigIB1j7+YQ6TigiEh0UhAOsKEcTZkREoomCcIB5V5jRhBkRkaigIBxghdm+CTNbjm2hvrU+3OWIiEgvFIQDLDMxk/Myz9OEGRGRKKEgDAFNmBERiR4KwhBonzCj44QiIpFPQRgC6hGKiEQPBWEIaMKMiEj0UBCGQEZiBhMzJ9LqWimrLQt3OSIi0oOQBqGZXWZm28xsh5l9q4d2M82s1cyuC2U9g0m3ZBIRiQ4hC0IzSwQeBS4HJgKfMLOJ3bT7H+DvoaolHNpvyVRSowkzIiKRLJQ9wlnADufcLudcE/AscHUX7e4GfgccDGEtg87rEWrCjIhIRAtlEI4E9nZYLvc/5zGzkcC1wGM9bcjMbjezEjMrOXTo0IAXGgpTs6aSaIlsPbaVuta6cJcjIiLdCGUQWhfPuU7LDwLfdM619rQh59wvnHNFzrmiYcOGDViBoaQJMyIi0SGUQVgOjO6wPArY36lNEfCsmb0HXAf8zMyuCWFNg6r9OKEmzIiIRK5QBuFaYLyZjTWzFOBG4E8dGzjnxjrnxjjnxgAvAHc65/4QwpoGlXcnCk2YERGJWEmh2rBzrsXM7sI3GzQReMI5t8XM7vCv7/G4YCzQKRQiIpEvZEEI4Jx7CXip03NdBqBz7uZQ1hIOnSfMZCRmhLskERHpRFeWCaH0xHQmZU6ijTZKa0rDXY6IiHRBQRhi3oQZnU8oIhKRFIQh1n6cULdkEhGJTArCENMtmUREIpuCMMSmZE0hyZJ469hbHGs9Fu5yRESkEwVhiGnCjIhIZFMQDoKiHE2YERGJVArCQeBdYUYTZkREIo6CcBCoRygiErkUhIOgILPAmzBT21Ib7nJERKQDBeEgSEtMY3LmZByO0lpNmBERiSQKwkHiDY/qAtwiIhFFQThIdEsmEZHIpCAcJOoRiohEJgXhICnIKiDZknm77m1qWmrCXY6IiPgpCAdJakIqk7P8E2Z0hRkRkYihIBxEuiWTiEjkURAOIk2YERGJPArCQaQJMyIikUdBOIgmZ00m2ZLZVrdNE2ZERCKEgnAQpSakMiVrCg7HhpoN4S5HRERQEA46HScUEYksCsJBNiPHF4Q6TigiEhkUhINMp1CIiEQWBeEgm5w1mRRLYVvdNqpbqsNdjohI3FMQDrKUhBSmZE0B0IQZEZEIoCAMg/bjhCXVmjAjIhJuCsIwaJ85quOEIiLhpyAMA+8KMwpCEZGwUxCGwaTMSaRYCtvrtnPb1tv49YFfU95QHu6yRETiUlK4C4hHKQkpXDH0Cn5/6Pc8ceAJnjjwBADj0sexKH8RC/MXsjBvISPTRoa5UhGR2GfOuXDX0CdFRUWupCT6J5m0ulZKa0pZVrGM4opiVlSuoKY18Pqj49PH+0LR/xiROiJM1YqIRDYzW+ecK+rXaxWEkaGlrYXS2lKKK4pZVrGMlZUrTwjGczLOYWHeQhblL2JB/gJOTz09TNWKiEQWBWEMamlrYX3NepZV+nqMqypXUdtaG9Dm3IxzvaHUBXkLOC31tDBVKyISXgrCONDc1uwLRv9Q6qqqVRxrPRbQ5ryM8wKGUoenDA9TtSIig0tBGIea25opqSlhWcUyllUsY1XlKura6gLaTMycGDCUOixlWJiqFREJLQWh0NTWREl1iTeU+lrla9S31Qe0mZQ5KWAodWjK0DBVKyIysBSEcoKmtibWVq/1Jt+8VvUaDW0NAW0KsgpYmOcbRl2Qv4AhyUPCVK2IyMlREEqvGtsaA4JxddXqE4JxStYU31DqKYuYnzefU5JPCVO1IiJ9oyCUPmtsa+SNqje8yTevV79OY1ujt94wpmRN8YZS5+fNJz85P4wVi4h0T0EoJ62htYE3qt/weoyvV71Ok2vy1htGYXahN5Q6P28+ecl5YaxYROQ4BaEMuPrW+oBgXFO15oRgnJY9jYX5vlmp8/LmkZuUG8aKRSSeKQgl5Opb63m96nXf6RqVvmBsds3e+gQSmJY9zRtKnZs3V8EoIoNGQSiDrq61zgvG4opi3qx+84RgnJEzwxtKnZs3l5yknDBWLCKxTEEoYXes9RivV73uDaW+Wf0mLa7FW59oiczInuHdWWNu3lyyk7LDWLGIxBIFoUScY63HeK3yNW8odW312hOCsSi7yBtKvTD3QrKSssJYsYhEMwWhRLzallpeq3rNG0otqSmh1bV665MsiZk5M72h1AvzLiQzMTOMFYtINFEQStSpaanhtarXvKHUkuoS2mjz1idZErNyZnmzUi/IvYCMxIwwViwikUxBKFGvuqWaVZWrvKHUddXrAoIx2ZIDgnFO7hwFo4h4FIQSc6paqrxgLK4oZkPNhhOC8fyc871jjHNy55CemB7GikUknBSEEvMqmytZVbXKG0rdULMBx/Hf3RRLYXbubK/HODtnNmmJaWGsWEQGk4JQ4k5FcwUrK1d6Q6mlNaUBwZiakMrsnOPBeH7O+QpGkRimIJS4d7T5qBeMxRXFlNWWBaxPTUhlTs4cbyj1/NzzSU1IDVO1IjLQFIQinRxpPsLKypUUHy1mWeUyNtZuDFiflpDGBbkXeCf4z8qdpWAUiWIKQpFeHGk+wvKK5d5Q6qbaTQHr0xPSvWBclL+ImTkzSUlICVO1ItJXCkKRPjrUdIgVlSu8odQtx7YErE9PSOfCvAt9Q6l5CynKKVIwikQwBaHISTrYdJAVFSt8s1Irl7H12NaA9RkJGczNm+sbSs1fSFF2EckJyWGqVkQ6UxCKDLAPGj9geaV/KLViGW/VvRWwPjMxk7m5c73JNzOyZ5CUkBSmakVEQSgSYu83vu8FY3FFMdvqtgWsz0rMYm7eXG8odXr2dAWjyCBSEIoMsgONB1hesdwbSt1etz1gfXZiNvPy5nlDqdOypikYRUJIQSgSZvsb93vDqMsqlvFO/TsB63MSc7xgXJS/iMLsQhItMUzVisQeBaFIhClvKA84xrijfkfA+tykXOblzfOGUqdmT1UwipwEBaFIhNvbsDdgKHVX/a6A9XlJeczPm+8NpU7NmkqCJYSpWpHoE7FBaGaXAQ8BicDjzrkfdlq/GPimf7EW+LxzLvDaWJ0oCCUW7GnYEzCU+m7DuwHr85PyvWBclL+IgqwCBaNIDyIyCM0sEdgOfBgoB9YCn3DObe3Q5gLgLedchZldDixxzp3f03YVhBKLdtfvZlnlMm9W6u6G3QHr85PyWZC/wBtKnZw1WcEo0kGkBuEcfMF2qX/52wDOuR900z4f2OycG9nTdhWEEg/eq3/PC8XiimL2Nu4NWD8keQgL8hZ4Q6mTMicpGCWuRWoQXgdc5pz7V//yp4HznXN3ddP+a8CE9vad1t0O3A5wxhlnzNi9e3fnJiIxyznHew3vefdiLK4opryxPKDN0OShXjAuyl/ExMyJmFmYKhYZfJEahNcDl3YKwlnOubu7aLsI+Bkw1zl3pKftqkco8c45x676XQFDqfsa9wW0GZY8LGAo9bzM8xSMEtNOJghDeYZvOTC6w/IoYH/nRmY2BXgcuLy3EBQRMDPGZYxjXMY4bhtxG845dtbv9O6sUVxRzP7G/bxw8AVeOPgCAMOTh3vDqAvzFzIhY4KCUcQvlD3CJHyTZS4C9uGbLPNJ59yWDm3OAF4FbnLOrQ5mu+oRivTMOceO+h0BQ6nvN70f0ObUlFNZmLfQG0o9J+McBaNEtYgcGgUws48AD+I7feIJ59z3zewOAOfcY2b2OPBxoP2gX0tvH0RBKNI3zjm2120P6DF+0PRBQJvTUk7zQnFh/kLGp49XMEpUidggDAUFocjJcc6xrW6bdw5jcUUxB5sPBrQZkTrC6zEuzF/I2elnKxgloikIRaTfnHO8Xfe2F4rLKpZxqPlQQJuRqSN9oZjn6zWelX6WglEiioJQRAaMc463jr3lXQ5uWcUyDjcfDmgzKnVUwFDq2LSxCkYJKwWhiIRMm2tj67GtXo9xeeVyjjQHTvAenTraC8WF+QsZmz42TNVKvFIQisigaXNtbDm25XgwViznaMvRgDZnpp0ZMJR6ZvqZYapW4oWCUETCps21sbl2szeUurxiORUtFQFtxqSNCRhKPSPtjDBVK7FKQSgiEaPNtbGxdqM3K3V55XIqWyoD2oxNGxswlDo6bXQ3WxMJjoJQRCJWq2v1grG4opgVlSuoaqkKaDMufZw3lLowfyGj0kaFqVqJVgpCEYkara6Vspoybyh1RcUKqlurA9qcnX52wFDqiNQRYapWooWCUESiVqtrZUPNBm8odUXlCmpaawLajE8fHzCUenrq6WGqViKVglBEYkZLWwsbajd4Q6krK1dS21ob0ObcjHMDhlJPSz0tTNVKpFAQikjMamlrYX3Nem8odWXlSo61HgtoMyFjwvG7a+Qt5NTUU8NUrYSLglBE4kZzWzPratZ5Q6mrqladEIznZZznHWNckL+A4SnDw1StDBYFoYjErea2ZkpqSryh1NcqX6OurS6gzaTMSV5vcUH+AoalDAtTtRIqCkIREb+mtiZKqku8odTXKl+jvq0+oM3kzMneUOqCvAUMTRkapmploCgIRUS60dTWxJvVb3pDqa9VvUZDW0NAm4KsAu9ycPPz5zMkeUiYqpX+UhCKiASpsa2RN6ve9G5SvLpqNY1tjQFtpmRN8U7XmJ83n1OSTwlTtRIsBaGISD81tDbwZvWb3r0YX69+PSAYDWNq1lRvKHV+3nzyk/PDWLF0RUEoIjJAGlobWFO9xhtKfb3qdZpck7feMAqzC72h1Hl588hLzgtjxQIKQhGRkKlvrWdN1RpvKHVN1RqaXbO33jCmZU/zhlLn5c0jNyk3jBXHJwWhiMggqWutY03VGm8o9Y3qNwKCMYEEpmdP94ZS5+XNIycpJ4wVxwcFoYhImNS11rG6arU3lPpG9Ru0uBZvfQIJzMiZ4Q2lzs2bS3ZSdhgrjk0KQhGRCHGs9RirK1d7Q6lrq9cGBGOiJTIje4Y3lHph7oUKxgGgIBQRiVC1LbWsrlrtDaWurVlLq2v11idaIjOzZ3pDqRfmXkhWUlYYK45OCkIRkShR01LDa1WveUOpJTUlAcGYZEnMzJnpDaVekHcBmYmZYaw4OigIRUSiVE1LDasqV3lDqeuq19FGm7c+yZKYlTPLG0q9IPcCMhIzwlhxZFIQiojEiOqWalZVrvKGUtfXrA8IxmRL5vyc872h1AtyLyA9MT2MFUcGBaGISIyqaqliZeVKbyh1Q82GgGBMsRTOzz3fG0qdnTs7LoNRQSgiEicqmyt9wegfSi2tKcVx/Hs8xVKYnTvbG0qdnTObtMS0MFY8OBSEIiJxqqK5gpWVK72h1LLasoBgTE1IZU7OHG8odXbubFITUsNYcWgoCEVEBICjzUdZUbnCG0otqy0LWJ+WkMac3DneUOqs3FkxEYwKQhER6dKR5iOsqPAHY+UyNtZuDFiflpDGBbkXeEOps3JmkZKQEqZq+09BKCIiQTncdJgVlSu8odTNxzYHrE9PSOfCvAtZmOcbSp2ZMzMqglFBKCIi/XKo6RDLK5azrNI3lLrl2JaA9RkJGb5gzPcNpRZlF5GckBymarunIBQRkQFxsOmgLxj9Q6lbj20NWJ+ZmMmFuRd6Q6kzsmdERDAqCEVEJCQ+aPyA5ZW+YCyuKObturcD1mcmZjI3d25AMCYlJA16nQpCEREZFO83vs/yyuXeMcZtddsC1mclZjEvb543lDota9qgBKOCUEREwmJ/4/6AodTtddsD1mcnZjMvb57XYyzMKgxJMCoIRUQkIuxr2BcwlLqjfkfA+pzEHObnz/dmpRZmF5JoiSf9vgpCERGJSOUN5QFDqTvrdwasz03KpSi7iIKsAu8xKXNSn++woSAUEZGosLdhr3fVm2WVy9hVv+uENoYxLn0ck7MmU5B5PCDPTj+722FVBaGIiESlvQ17Ka0pZdOxTWyq9T221W2jxbWc0DY1IZWJmROZnDn5eA8ys4ARqSNISEhQEIqISGxoamtiW902Lxg31W5i07FN7GnY02X7/KR8KhZW9DsIB/9kDxERkR6kJKR4Pb6Oqlqq2FK7xQvG9pCsaKk4qfdTEIqISFTITcrlgrwLuCDvAu855xz7G/czilH93m7CQBQnIiISDmbGyLSRJ7UNBaGIiMQ1BaGIiMQ1BaGIiMQ1BaGIiMQ1BaGIiMQ1BaGIiMQ1BaGIiMQ1BaGIiMQ1BaGIiMQ1BaGIiMQ1BaGIiMQ1BaGIiMQ1BaGIiMQ1BaGIiMQ1BaGIiMQ1BaGIiMQ1BaGIiMQ1BaGIiMQ1BaGIiMS1kAahmV1mZtvMbIeZfauL9WZmD/vXbzSz6aGsR0REpLOQBaGZJQKPApcDE4FPmNnETs0uB8b7H7cDPw9VPSIiIl0JZY9wFrDDObfLOdcEPAtc3anN1cCTzmcNkGdmp4ewJhERkQBJIdz2SGBvh+Vy4Pwg2owEDnRsZGa34+sxAjSa2eaBLTUuDAUOh7uIKKT91n/ad/2j/dY/5/b3haEMQuviOdePNjjnfgH8AsDMSpxzRSdfXnzRfusf7bf+077rH+23/jGzkv6+NpRDo+XA6A7Lo4D9/WgjIiISMqEMwrXAeDMba2YpwI3Anzq1+RNwk3/26Gygyjl3oPOGREREQiVkQ6POuRYzuwv4O5AIPOGc22Jmd/jXPwa8BHwE2AHUAbcEselfhKjkWKf91j/ab/2nfdc/2m/90+/9Zs6dcEhOREQkbujKMiIiEtcUhCIiEtciNgh1ebb+CWK/Lfbvr41mttrMpoajzkjT237r0G6mmbWa2XWDWV+kCma/mdlCMys1sy1mtnywa4xEQfx/mmtmfzazMv9+C2b+RMwzsyfM7GB355L3OxeccxH3wDe5ZidwFpAClAETO7X5CPBXfOcizgbeCHfd4X4Eud8uAPL9P1+u/RbcfuvQ7lV8k7yuC3fd4X4E+fuWB2wFzvAvDw933eF+BLnfvgP8j//nYcBRICXctYf7AcwHpgObu1nfr1yI1B6hLs/WP73uN+fcaudchX9xDb5zN+NdML9vAHcDvwMODmZxESyY/fZJ4PfOuT0Azjntu+D2mwOyzcyALHxB2DK4ZUYe59wKfPuiO/3KhUgNwu4uvdbXNvGmr/vkNnx/PcW7XvebmY0ErgUeG8S6Il0wv2/nAPlmtszM1pnZTYNWXeQKZr89ApyH7wIjm4AvOefaBqe8qNavXAjlJdZOxoBdni3OBL1PzGwRviCcG9KKokMw++1B4JvOuVbfH+lCcPstCZgBXASkA6+b2Rrn3PZQFxfBgtlvlwKlwIeAccA/zGylc6461MVFuX7lQqQGoS7P1j9B7RMzmwI8DlzunDsySLVFsmD2WxHwrD8EhwIfMbMW59wfBqfEiBTs/6eHnXPHgGNmtgKYCsRzEAaz324Bfuh8B752mNm7wATgzcEpMWr1KxcidWhUl2frn173m5mdAfwe+HSc/1XeUa/7zTk31jk3xjk3BngBuDPOQxCC+//0j8A8M0syswx8d6B5a5DrjDTB7Lc9+HrRmNmp+O6ssGtQq4xO/cqFiOwRutBdni2mBbnfvgcMAX7m7920uDi/0n2Q+006CWa/OefeMrO/ARuBNuBx51xc30YtyN+3/wKWmtkmfMN933TOxf2tmczsGWAhMNTMyoF7gGQ4uVzQJdZERCSuRerQqIiIyKBQEIqISFxTEIqISFxTEIqISFxTEIqISFxTEIqEmf9uFqVmttl/x4G8Ad7+zWb2iP/nJWb2tYHcvki0UxCKhF+9c67QOTcZ3wWFvxDugkTiiYJQJLK8ToeLBJvZ181srf/eav/R4fmb/M+VmdlT/ueuNLM3zGyDmb3ivyKJiPQiIq8sIxKPzCwR32W1fulfvgQYj++2PQb8yczmA0eAfwcudM4dNrNT/JtYBcx2zjkz+1fgG8BXB/ljiEQdBaFI+KWbWSkwBlgH/MP//CX+xwb/cha+YJwKvNB+yS3nXPv92UYBz/nvv5YCvDso1YtEOQ2NioRfvXOuEDgTX4C1HyM04Af+44eFzrmznXO/9D/f1bURfwo84pwrAD4HpA1C7SJRT0EoEiGcc1XAF4GvmVkyvosy32pmWeC7ObCZDQf+CfyLmQ3xP98+NJoL7PP//JlBLV4kimloVCSCOOc2mFkZcKNz7ikzOw/fzWwBaoFP+e9U8H1guZm14hs6vRlYAjxvZvuANcDYcHwGkWiju0+IiEhc09CoiIjENQWhiIjENQWhiIjENQWhiIjENQWhiIjENQWhiIjENQWhiIjEtf8f61YOAUR+CLwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 504x504 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "skplt.metrics.plot_precision_recall(y, cervPredProb,\n",
    "        plot_micro=False,\n",
    "        figsize=(7,7),\n",
    "        classes_to_plot=1,\n",
    "        title='CART Classifier PR Curve');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question**:  Why do you think there's an \"elbow\" in the above Precision/Recall graph?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# UDU: Grid Search and CV for Better CARTing\n",
    "\n",
    "There are tools in other (non-Python) environments for \"pruning\" CART trees so as to reduce overfitting.  Another way of getting a good CART model is by doing a grid search over tree parameters.  Try that here.  The following code will give you a start.  Give at go at manipulating the parameter values that are searched.  Try adding additional parameters with values to search over.\n",
    "\n",
    "Did your results produce anything really useful?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=99)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the parameters that values can be searched for:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tree Parameters:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "dict_keys(['class_weight', 'criterion', 'max_depth', 'max_features', 'max_leaf_nodes', 'min_impurity_decrease', 'min_impurity_split', 'min_samples_leaf', 'min_samples_split', 'min_weight_fraction_leaf', 'presort', 'random_state', 'splitter'])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "treeclf=DecisionTreeClassifier(criterion='entropy',random_state=99)\n",
    "print(\"Tree Parameters:\")\n",
    "treeclf.get_params().keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/lynd/anaconda3/lib/python3.6/site-packages/sklearn/model_selection/_search.py:813: DeprecationWarning: The default of the `iid` parameter will change from True to False in version 0.22 and will be removed in 0.24. This will change numeric results when test-set sizes are unequal.\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=20, error_score='raise-deprecating',\n",
       "             estimator=DecisionTreeClassifier(class_weight=None,\n",
       "                                              criterion='entropy',\n",
       "                                              max_depth=None, max_features=None,\n",
       "                                              max_leaf_nodes=None,\n",
       "                                              min_impurity_decrease=0.0,\n",
       "                                              min_impurity_split=None,\n",
       "                                              min_samples_leaf=1,\n",
       "                                              min_samples_split=2,\n",
       "                                              min_weight_fraction_leaf=0.0,\n",
       "                                              presort=False, random_state=99,\n",
       "                                              splitter='best'),\n",
       "             iid='warn', n_jobs=None,\n",
       "             param_grid={'max_depth': [2, 3, 4], 'max_features': [3, 5, 7],\n",
       "                         'min_samples_leaf': [10, 20]},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring=None, verbose=0)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "param_grid={'max_depth':[2,3,4],'min_samples_leaf':[10,20],\n",
    "            'max_features':[3,5,7]}\n",
    "grid=GridSearchCV(treeclf,param_grid=param_grid,cv=20)\n",
    "grid.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best cross-validation accuracy: 0.9270\n",
      "Test set score: 0.9467\n",
      "Best parameters: {'max_depth': 2, 'max_features': 3, 'min_samples_leaf': 10}\n"
     ]
    }
   ],
   "source": [
    "print(\"Best cross-validation accuracy: {:.4f}\".format(grid.best_score_))\n",
    "print(\"Test set score: {:.4f}\".format(grid.score(X_test, y_test)))\n",
    "print(\"Best parameters: {}\".format(grid.best_params_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the following to examine cv_results. \n",
    "\n",
    "# pd.DataFrame(grid.cv_results_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## \"Best\" Tree Based On Grid Search  \n",
    "\n",
    "Assuming that you get a \"best\" tree that is really any better than any other tree, of course.\n",
    "\n",
    "\\*\\*grid_best_params_, below, is a dict of parameter key value pairs.  It should be like the dict that's output, above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the best tree based on the search\n",
    "\n",
    "bestTree=DecisionTreeClassifier(**grid.best_params_)  \n",
    "\n",
    "# Fit this tree using the training data\n",
    "\n",
    "bestTreeFit=bestTree.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Importances\n",
    "\n",
    "Importance here is a normalized measure of the extent to which splits based on a feature reduce the fitting criterion, either _gini impurity_ or _entropy_. \n",
    "\n",
    "Your results will vary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.64038214, 0.        , 0.        , 0.        , 0.        ,\n",
       "       0.        , 0.        , 0.        , 0.35961786])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bestTree.feature_importances_ "
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "313.4px"
   },
   "toc_section_display": false,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}