Adult Income Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adult Income Classification\n",
    "\n",
    "______\n",
    "\n",
    "In this example, we would try to classify if the income is high i.e. >50 k or low i.e. < 50 k.\n",
    "\n",
    "The dataset can be downloaded from : http://archive.ics.uci.edu/ml/datasets/Adult\n",
    "\n",
    "### Listing of attributes: \n",
    "\n",
    "Class: >50K, <=50K. \n",
    "\n",
    "age: continuous. \n",
    "\n",
    "workclass: Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, Local-gov, State-gov, Without-pay, Never-worked. \n",
    "\n",
    "fnlwgt: continuous. \n",
    "\n",
    "education: Bachelors, Some-college, 11th, HS-grad, Prof-school, Assoc-acdm, Assoc-voc, 9th, 7th-8th, 12th, Masters, 1st-4th, 10th, Doctorate, 5th-6th, Preschool. \n",
    "\n",
    "education-num: continuous. \n",
    "\n",
    "marital-status: Married-civ-spouse, Divorced, Never-married, Separated, Widowed, Married-spouse-absent, Married-AF-spouse. \n",
    "\n",
    "occupation: Tech-support, Craft-repair, Other-service, Sales, Exec-managerial, Prof-specialty, Handlers-cleaners, Machine-op-inspct, Adm-clerical, Farming-fishing, Transport-moving, Priv-house-serv, Protective-serv, Armed-Forces. \n",
    "\n",
    "relationship: Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried. \n",
    "\n",
    "race: White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other, Black. \n",
    "\n",
    "sex: Female, Male. \n",
    "\n",
    "capital-gain: continuous. \n",
    "\n",
    "capital-loss: continuous. \n",
    "\n",
    "hours-per-week: continuous. \n",
    "\n",
    "native-country: United-States, Cambodia, England, Puerto-Rico, Canada, Germany, Outlying-US(Guam-USVI-etc), India, Japan, \n",
    "Greece, South, China, Cuba, Iran, Honduras, Philippines, Italy, Poland, Jamaica, Vietnam, Mexico, Portugal, Ireland, France, Dominican-Republic, Laos, Ecuador, Taiwan, Haiti, Columbia, Hungary, Guatemala, Nicaragua, Scotland, Thailand, Yugoslavia, El-Salvador, Trinadad&Tobago, Peru, Hong, Holand-Netherlands.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from sklearn.tree import DecisionTreeClassifier, export_graphviz\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn import cross_validation, metrics\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.naive_bayes import BernoulliNB\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.svm import SVC\n",
    "from time import time\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.metrics import roc_auc_score , classification_report\n",
    "from sklearn.grid_search import GridSearchCV\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.metrics import precision_score, recall_score, accuracy_score, classification_report\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# read .csv from provided dataset\n",
    "csv_filename=\"datasets/adult.data\"\n",
    "\n",
    "df = pd.read_csv(csv_filename, header=None,\n",
    "                names=[\"Age\", \"Work-Class\", \"fnlwgt\",\n",
    "                \"Education\", \"Education-Num\",\n",
    "                \"Marital-Status\", \"Occupation\",\n",
    "                \"Relationship\", \"Race\", \"Sex\",\n",
    "                \"Capital-gain\", \"Capital-loss\",\n",
    "                \"Hours-per-week\", \"Native-Country\",\n",
    "                \"Earnings-Raw\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.replace([' <=50K',' >50K'],[0,1],inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Work-Class</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>Education</th>\n",
       "      <th>Education-Num</th>\n",
       "      <th>Marital-Status</th>\n",
       "      <th>Occupation</th>\n",
       "      <th>Relationship</th>\n",
       "      <th>Race</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Capital-gain</th>\n",
       "      <th>Capital-loss</th>\n",
       "      <th>Hours-per-week</th>\n",
       "      <th>Native-Country</th>\n",
       "      <th>Earnings-Raw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>32556</th>\n",
       "      <td>27</td>\n",
       "      <td>Private</td>\n",
       "      <td>257302</td>\n",
       "      <td>Assoc-acdm</td>\n",
       "      <td>12</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Tech-support</td>\n",
       "      <td>Wife</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>38</td>\n",
       "      <td>United-States</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32557</th>\n",
       "      <td>40</td>\n",
       "      <td>Private</td>\n",
       "      <td>154374</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32558</th>\n",
       "      <td>58</td>\n",
       "      <td>Private</td>\n",
       "      <td>151910</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Widowed</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Unmarried</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32559</th>\n",
       "      <td>22</td>\n",
       "      <td>Private</td>\n",
       "      <td>201490</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>United-States</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32560</th>\n",
       "      <td>52</td>\n",
       "      <td>Self-emp-inc</td>\n",
       "      <td>287927</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Wife</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>15024</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Age     Work-Class  fnlwgt    Education  Education-Num  \\\n",
       "32556   27        Private  257302   Assoc-acdm             12   \n",
       "32557   40        Private  154374      HS-grad              9   \n",
       "32558   58        Private  151910      HS-grad              9   \n",
       "32559   22        Private  201490      HS-grad              9   \n",
       "32560   52   Self-emp-inc  287927      HS-grad              9   \n",
       "\n",
       "            Marital-Status          Occupation Relationship    Race      Sex  \\\n",
       "32556   Married-civ-spouse        Tech-support         Wife   White   Female   \n",
       "32557   Married-civ-spouse   Machine-op-inspct      Husband   White     Male   \n",
       "32558              Widowed        Adm-clerical    Unmarried   White   Female   \n",
       "32559        Never-married        Adm-clerical    Own-child   White     Male   \n",
       "32560   Married-civ-spouse     Exec-managerial         Wife   White   Female   \n",
       "\n",
       "       Capital-gain  Capital-loss  Hours-per-week  Native-Country  \\\n",
       "32556             0             0              38   United-States   \n",
       "32557             0             0              40   United-States   \n",
       "32558             0             0              40   United-States   \n",
       "32559             0             0              20   United-States   \n",
       "32560         15024             0              40   United-States   \n",
       "\n",
       "       Earnings-Raw  \n",
       "32556             0  \n",
       "32557             1  \n",
       "32558             0  \n",
       "32559             0  \n",
       "32560             1  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The adult file itself contains two blank lines at the end of the file. By default, pandas\n",
    "will interpret the penultimate new line to be an empty (but valid) row. To remove\n",
    "this, we remove any line with invalid numbers (the use of inplace just makes sure\n",
    "the same Dataframe is affected, rather than creating a new one):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.dropna(how='all', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "feature_names = df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results show each of the feature names that are stored inside an Index object\n",
    "from pandas.\n",
    "\n",
    "The Adult dataset contains several categorical features, with Work-Class being one\n",
    "example. While we could argue that some values are of higher rank than others (for\n",
    "instance, a person with a job is likely to have a better income than a person without),\n",
    "it doesn't make sense for all values. For example, a person working for the state\n",
    "government is not more or less likely to have a higher income than someone working\n",
    "in the private sector.\n",
    "We can view the unique values for this feature in the dataset using the\n",
    "unique() function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([' State-gov', ' Self-emp-not-inc', ' Private', ' Federal-gov',\n",
       "       ' Local-gov', ' ?', ' Self-emp-inc', ' Without-pay', ' Never-worked'], dtype=object)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"Work-Class\"].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are some missing values in the preceding dataset, but they won't affect our\n",
    "computations in this example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Selecting the best individual features\n",
    "If we have a number of features, the problem of finding the best subset is a difficult\n",
    "task. \n",
    "\n",
    "The scikit-learn package has a number of transformers for performing univariate\n",
    "feature selection. They include SelectKBest, which returns the k best performing\n",
    "features, and SelectPercentile, which returns the top r% of features. In both cases,\n",
    "there are a number of methods of computing the quality of a feature.\n",
    "\n",
    "There are many different methods to compute how effectively a single feature\n",
    "correlates with a class value. A commonly used method is the chi-squared (χ2) test.\n",
    "Other methods include mutual information and entropy.\n",
    "\n",
    "We can observe single-feature tests in action using our Adult dataset. First, we\n",
    "extract a dataset and class values from our pandas DataFrame. We get a selection of\n",
    "the features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = df[[\"Age\", \"Education-Num\", \"Capital-gain\", \"Capital-loss\", \"Hours-per-week\"]].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also create a target class array by testing whether the Earnings-Raw value\n",
    "is above $50,000 or not. If it is, the class will be True. Otherwise, it will be False.\n",
    "Let's look at the code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "y = (df[\"Earnings-Raw\"]).values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, ..., 0, 0, 1], dtype=int64)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we create our transformer using the chi2 function and a SelectKBest\n",
    "transformer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import SelectKBest\n",
    "from sklearn.feature_selection import chi2\n",
    "transformer = SelectKBest(score_func=chi2, k=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running fit_transform will call fit and then transform with the same dataset.\n",
    "The result will create a new dataset, choosing only the best three features.\n",
    "Let's look at the code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Xt_chi2 = transformer.fit_transform(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The resulting matrix now only contains three features. We can also get the scores\n",
    "for each column, allowing us to find out which features were used. Let's look at\n",
    "the code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  8.60061182e+03   2.40142178e+03   8.21924671e+07   1.37214589e+06\n",
      "   6.47640900e+03]\n"
     ]
    }
   ],
   "source": [
    "print(transformer.scores_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The highest values are for the first, third, and fourth columns Correlates to the Age,\n",
    "Capital-Gain, and Capital-Loss features. Based on a univariate feature selection,\n",
    "these are the best features to choose.\n",
    "\n",
    "We could also implement other correlations, such as the Pearson's correlation\n",
    "coefficient. This is implemented in SciPy, a library used for scientific computing\n",
    "(scikit-learn uses it as a base)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from scipy.stats import pearsonr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The preceding function almost fits the interface needed to be used in scikit-learn's\n",
    "univariate transformers. The function needs to accept two arrays (x and y in our\n",
    "example) as parameters and returns two arrays, the scores for each feature and the\n",
    "corresponding p-values. The chi2 function we used earlier only uses the required\n",
    "interface, which allowed us to just pass it directly to SelectKBest.\n",
    "\n",
    "The pearsonr function in SciPy accepts two arrays; however, the X array it accepts is\n",
    "only one dimension. We will write a wrapper function that allows us to use this for\n",
    "multivariate arrays like the one we have. Let's look at the code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def multivariate_pearsonr(X, y):\n",
    "    scores, pvalues = [], []\n",
    "    for column in range(X.shape[1]):\n",
    "        cur_score, cur_p = pearsonr(X[:,column], y)\n",
    "        scores.append(abs(cur_score))\n",
    "        pvalues.append(cur_p)\n",
    "    return (np.array(scores), np.array(pvalues))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Pearson value could be between -1 and 1. A value of 1 implies a\n",
    "perfect correlation between two variables, while a value of -1 implies a\n",
    "perfect negative correlation, that is, high values in one variable give low\n",
    "values in the other and vice versa. Such features are really useful to have,\n",
    "but would be discarded. For this reason, we have stored the absolute\n",
    "value in the scores array, rather than the original signed value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.2340371   0.33515395  0.22332882  0.15052631  0.22968907]\n"
     ]
    }
   ],
   "source": [
    "transformer = SelectKBest(score_func=multivariate_pearsonr, k=3)\n",
    "Xt_pearson = transformer.fit_transform(X, y)\n",
    "print(transformer.scores_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This returns a different set of features! The features chosen this way are the first,\n",
    "second, and fifth columns: the Age, Education, and Hours-per-week worked. This\n",
    "shows that there is not a definitive answer to what the best features are— it depends\n",
    "on the metric.\n",
    "\n",
    "We can see which feature set is better by running them through a classifier. Keep in\n",
    "mind that the results only indicate which subset is better for a particular classifier\n",
    "and/or feature combination—there is rarely a case in data mining where one method\n",
    "is strictly better than another in all cases! Let's look at the code:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.cross_validation import cross_val_score\n",
    "clf = DecisionTreeClassifier(random_state=14)\n",
    "scores_chi2 = cross_val_score(clf, Xt_chi2, y, scoring='accuracy')\n",
    "scores_pearson = cross_val_score(clf, Xt_pearson, y, scoring='accuracy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Chi2 performance: 0.829\n",
      "Pearson performance: 0.771\n"
     ]
    }
   ],
   "source": [
    "print(\"Chi2 performance: {0:.3f}\".format(scores_chi2.mean()))\n",
    "print(\"Pearson performance: {0:.3f}\".format(scores_pearson.mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is worth remembering the goal of this data mining activity: predicting wealth.\n",
    "Using a combination of good features and feature selection, we can achieve 83\n",
    "percent accuracy using just three features of a person!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Principal Components Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "pca = PCA(n_components=5)\n",
    "Xd = pca.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.997,  0.003,  0.   ,  0.   ,  0.   ])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.set_printoptions(precision=3, suppress=True)\n",
    "pca.explained_variance_ratio_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result, array([ 0.997,  0.003,  0.   ,  0.   ,  0.   ]), shows\n",
    "us that the first feature accounts for 99.7 percent of the variance in the dataset,\n",
    "the second accounts for 0.3 percent, and so on. By the fourth feature, less than\n",
    "one-tenth of a percent of the variance is contained in the feature. The other\n",
    "features explain even less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clf = DecisionTreeClassifier(random_state=14)\n",
    "original_scores = cross_val_score(clf, X, y, scoring='accuracy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The average score from the original dataset is 0.8111\n"
     ]
    }
   ],
   "source": [
    "print(\"The average score from the original dataset is {:.4f}\".format(np.mean(original_scores)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clf = DecisionTreeClassifier(random_state=14)\n",
    "scores_reduced = cross_val_score(clf, Xd, y, scoring='accuracy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The average score from the reduced dataset is 0.8135\n"
     ]
    }
   ],
   "source": [
    "print(\"The average score from the reduced dataset is {:.4f}\".format(np.mean(scores_reduced)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.replace(' ?',np.nan,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Age                           19\n",
       "Work-Class               Private\n",
       "fnlwgt                    168294\n",
       "Education                HS-grad\n",
       "Education-Num                  9\n",
       "Marital-Status     Never-married\n",
       "Occupation          Craft-repair\n",
       "Relationship           Own-child\n",
       "Race                       White\n",
       "Sex                         Male\n",
       "Capital-gain                   0\n",
       "Capital-loss                   0\n",
       "Hours-per-week                40\n",
       "Native-Country     United-States\n",
       "Earnings-Raw                   0\n",
       "Name: 26, dtype: object"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[26,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(32561, 15)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "#We can also specify to only drop rows that are complete missing all data\n",
    "df.dropna(inplace=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(30162, 15)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Work-Class</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>Education</th>\n",
       "      <th>Education-Num</th>\n",
       "      <th>Marital-Status</th>\n",
       "      <th>Occupation</th>\n",
       "      <th>Relationship</th>\n",
       "      <th>Race</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Capital-gain</th>\n",
       "      <th>Capital-loss</th>\n",
       "      <th>Hours-per-week</th>\n",
       "      <th>Native-Country</th>\n",
       "      <th>Earnings-Raw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>77516</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>2174</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>Self-emp-not-inc</td>\n",
       "      <td>83311</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>United-States</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38</td>\n",
       "      <td>Private</td>\n",
       "      <td>215646</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53</td>\n",
       "      <td>Private</td>\n",
       "      <td>234721</td>\n",
       "      <td>11th</td>\n",
       "      <td>7</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Husband</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>Private</td>\n",
       "      <td>338409</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Wife</td>\n",
       "      <td>Black</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>Cuba</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Age         Work-Class  fnlwgt   Education  Education-Num  \\\n",
       "0   39          State-gov   77516   Bachelors             13   \n",
       "1   50   Self-emp-not-inc   83311   Bachelors             13   \n",
       "2   38            Private  215646     HS-grad              9   \n",
       "3   53            Private  234721        11th              7   \n",
       "4   28            Private  338409   Bachelors             13   \n",
       "\n",
       "        Marital-Status          Occupation    Relationship    Race      Sex  \\\n",
       "0        Never-married        Adm-clerical   Not-in-family   White     Male   \n",
       "1   Married-civ-spouse     Exec-managerial         Husband   White     Male   \n",
       "2             Divorced   Handlers-cleaners   Not-in-family   White     Male   \n",
       "3   Married-civ-spouse   Handlers-cleaners         Husband   Black     Male   \n",
       "4   Married-civ-spouse      Prof-specialty            Wife   Black   Female   \n",
       "\n",
       "   Capital-gain  Capital-loss  Hours-per-week  Native-Country  Earnings-Raw  \n",
       "0          2174             0              40   United-States             0  \n",
       "1             0             0              13   United-States             0  \n",
       "2             0             0              40   United-States             0  \n",
       "3             0             0              40   United-States             0  \n",
       "4             0             0              40            Cuba             0  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "For: Work-Class, Total unique values are 7 - \n",
      " [' State-gov' ' Self-emp-not-inc' ' Private' ' Federal-gov' ' Local-gov'\n",
      " ' Self-emp-inc' ' Without-pay']\n",
      "\n",
      "\n",
      "For: Education, Total unique values are 16 - \n",
      " [' Bachelors' ' HS-grad' ' 11th' ' Masters' ' 9th' ' Some-college'\n",
      " ' Assoc-acdm' ' 7th-8th' ' Doctorate' ' Assoc-voc' ' Prof-school'\n",
      " ' 5th-6th' ' 10th' ' Preschool' ' 12th' ' 1st-4th']\n",
      "\n",
      "\n",
      "For: Marital-Status, Total unique values are 7 - \n",
      " [' Never-married' ' Married-civ-spouse' ' Divorced'\n",
      " ' Married-spouse-absent' ' Separated' ' Married-AF-spouse' ' Widowed']\n",
      "\n",
      "\n",
      "For: Occupation, Total unique values are 14 - \n",
      " [' Adm-clerical' ' Exec-managerial' ' Handlers-cleaners' ' Prof-specialty'\n",
      " ' Other-service' ' Sales' ' Transport-moving' ' Farming-fishing'\n",
      " ' Machine-op-inspct' ' Tech-support' ' Craft-repair' ' Protective-serv'\n",
      " ' Armed-Forces' ' Priv-house-serv']\n",
      "\n",
      "\n",
      "For: Relationship, Total unique values are 6 - \n",
      " [' Not-in-family' ' Husband' ' Wife' ' Own-child' ' Unmarried'\n",
      " ' Other-relative']\n",
      "\n",
      "\n",
      "For: Race, Total unique values are 5 - \n",
      " [' White' ' Black' ' Asian-Pac-Islander' ' Amer-Indian-Eskimo' ' Other']\n",
      "\n",
      "\n",
      "For: Sex, Total unique values are 2 - \n",
      " [' Male' ' Female']\n",
      "\n",
      "\n",
      "For: Native-Country, Total unique values are 41 - \n",
      " [' United-States' ' Cuba' ' Jamaica' ' India' ' Mexico' ' Puerto-Rico'\n",
      " ' Honduras' ' England' ' Canada' ' Germany' ' Iran' ' Philippines'\n",
      " ' Poland' ' Columbia' ' Cambodia' ' Thailand' ' Ecuador' ' Laos' ' Taiwan'\n",
      " ' Haiti' ' Portugal' ' Dominican-Republic' ' El-Salvador' ' France'\n",
      " ' Guatemala' ' Italy' ' China' ' South' ' Japan' ' Yugoslavia' ' Peru'\n",
      " ' Outlying-US(Guam-USVI-etc)' ' Scotland' ' Trinadad&Tobago' ' Greece'\n",
      " ' Nicaragua' ' Vietnam' ' Hong' ' Ireland' ' Hungary'\n",
      " ' Holand-Netherlands']\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for col in df.columns:\n",
    "    if df[col].dtype != 'int64':\n",
    "        print (\"For: {}, Total unique values are {} - \\n {}\".format(col ,\n",
    "                                                        len(pd.Series(df[col].values.ravel()).unique()),\n",
    "                                                        pd.Series(df[col].values.ravel()).unique()))\n",
    "        print ('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Now all of these features are categorical classes, but most scikit-learn classifiers (in particular the Decision Trees we plan to use), expect real-valued attributes. We can easily convert sex  to a binary value (0=female,1=male). We will use the LabelEncoder class from scikit-learn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Categorical classes: [' Female' ' Male']\n",
      "Integer classes: [0 1]\n",
      "Feature names: Index(['Age', 'Work-Class', 'fnlwgt', 'Education', 'Education-Num',\n",
      "       'Marital-Status', 'Occupation', 'Relationship', 'Race', 'Sex',\n",
      "       'Capital-gain', 'Capital-loss', 'Hours-per-week', 'Native-Country',\n",
      "       'Earnings-Raw'],\n",
      "      dtype='object')\n",
      "Features: 10    1\n",
      "11    1\n",
      "12    0\n",
      "13    1\n",
      "15    1\n",
      "16    1\n",
      "Name: Sex, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "enc = LabelEncoder()\n",
    "label_encoder = enc.fit(df['Sex'])\n",
    "print (\"Categorical classes:\", label_encoder.classes_)\n",
    "integer_classes = label_encoder.transform(label_encoder.classes_)\n",
    "print (\"Integer classes:\", integer_classes)\n",
    "t = label_encoder.transform(df['Sex'])\n",
    "df['Sex'] = t\n",
    "print ('Feature names:',feature_names)\n",
    "print ('Features:', df['Sex'][10:16])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we have to convert other categorical features. Since we have three different classes, we cannot convert to binary values (and using 0/1/2 values would imply an order, something we do not want). We use pandas get_dummies method for it or we can also use the OneHotEncoder to get three different attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Work-Class',\n",
       " 'Education',\n",
       " 'Marital-Status',\n",
       " 'Occupation',\n",
       " 'Relationship',\n",
       " 'Race',\n",
       " 'Native-Country']"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categorical_features = []\n",
    "for col in df.columns:\n",
    "    if df[col].dtype != 'int64':\n",
    "        categorical_features.append(col)\n",
    "categorical_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30162"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "onehot_df = pd.get_dummies(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>Education-Num</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Capital-gain</th>\n",
       "      <th>Capital-loss</th>\n",
       "      <th>Hours-per-week</th>\n",
       "      <th>Earnings-Raw</th>\n",
       "      <th>Work-Class_ Federal-gov</th>\n",
       "      <th>Work-Class_ Local-gov</th>\n",
       "      <th>...</th>\n",
       "      <th>Native-Country_ Portugal</th>\n",
       "      <th>Native-Country_ Puerto-Rico</th>\n",
       "      <th>Native-Country_ Scotland</th>\n",
       "      <th>Native-Country_ South</th>\n",
       "      <th>Native-Country_ Taiwan</th>\n",
       "      <th>Native-Country_ Thailand</th>\n",
       "      <th>Native-Country_ Trinadad&amp;Tobago</th>\n",
       "      <th>Native-Country_ United-States</th>\n",
       "      <th>Native-Country_ Vietnam</th>\n",
       "      <th>Native-Country_ Yugoslavia</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39</td>\n",
       "      <td>77516</td>\n",
       "      <td>13</td>\n",
       "      <td>1</td>\n",
       "      <td>2174</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>83311</td>\n",
       "      <td>13</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38</td>\n",
       "      <td>215646</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53</td>\n",
       "      <td>234721</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>338409</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 104 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Age  fnlwgt  Education-Num  Sex  Capital-gain  Capital-loss  \\\n",
       "0   39   77516             13    1          2174             0   \n",
       "1   50   83311             13    1             0             0   \n",
       "2   38  215646              9    1             0             0   \n",
       "3   53  234721              7    1             0             0   \n",
       "4   28  338409             13    0             0             0   \n",
       "\n",
       "   Hours-per-week  Earnings-Raw  Work-Class_ Federal-gov  \\\n",
       "0              40             0                        0   \n",
       "1              13             0                        0   \n",
       "2              40             0                        0   \n",
       "3              40             0                        0   \n",
       "4              40             0                        0   \n",
       "\n",
       "   Work-Class_ Local-gov             ...              \\\n",
       "0                      0             ...               \n",
       "1                      0             ...               \n",
       "2                      0             ...               \n",
       "3                      0             ...               \n",
       "4                      0             ...               \n",
       "\n",
       "   Native-Country_ Portugal  Native-Country_ Puerto-Rico  \\\n",
       "0                         0                            0   \n",
       "1                         0                            0   \n",
       "2                         0                            0   \n",
       "3                         0                            0   \n",
       "4                         0                            0   \n",
       "\n",
       "   Native-Country_ Scotland  Native-Country_ South  Native-Country_ Taiwan  \\\n",
       "0                         0                      0                       0   \n",
       "1                         0                      0                       0   \n",
       "2                         0                      0                       0   \n",
       "3                         0                      0                       0   \n",
       "4                         0                      0                       0   \n",
       "\n",
       "   Native-Country_ Thailand  Native-Country_ Trinadad&Tobago  \\\n",
       "0                         0                                0   \n",
       "1                         0                                0   \n",
       "2                         0                                0   \n",
       "3                         0                                0   \n",
       "4                         0                                0   \n",
       "\n",
       "   Native-Country_ United-States  Native-Country_ Vietnam  \\\n",
       "0                              1                        0   \n",
       "1                              1                        0   \n",
       "2                              1                        0   \n",
       "3                              1                        0   \n",
       "4                              0                        0   \n",
       "\n",
       "   Native-Country_ Yugoslavia  \n",
       "0                           0  \n",
       "1                           0  \n",
       "2                           0  \n",
       "3                           0  \n",
       "4                           0  \n",
       "\n",
       "[5 rows x 104 columns]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "onehot_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Age', 'fnlwgt', 'Education-Num', 'Sex', 'Capital-gain', 'Capital-loss',\n",
       "       'Hours-per-week', 'Earnings-Raw', 'Work-Class_ Federal-gov',\n",
       "       'Work-Class_ Local-gov',\n",
       "       ...\n",
       "       'Native-Country_ Portugal', 'Native-Country_ Puerto-Rico',\n",
       "       'Native-Country_ Scotland', 'Native-Country_ South',\n",
       "       'Native-Country_ Taiwan', 'Native-Country_ Thailand',\n",
       "       'Native-Country_ Trinadad&Tobago', 'Native-Country_ United-States',\n",
       "       'Native-Country_ Vietnam', 'Native-Country_ Yugoslavia'],\n",
       "      dtype='object', length=104)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "onehot_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "enc = LabelEncoder()\n",
    "\n",
    "for f in categorical_features:\n",
    "\n",
    "    label_encoder = enc.fit(df[f])\n",
    "    print \"Categorical classes:\", label_encoder.classes_\n",
    "    \n",
    "    integer_classes = label_encoder.transform(label_encoder.classes_).reshape(\n",
    "        len(pd.Series(df[f].values.ravel()).unique()), 1)\n",
    "    print \"Integer classes:\", integer_classes\n",
    "    \n",
    "    enc = OneHotEncoder()\n",
    "    one_hot_encoder = enc.fit(integer_classes)\n",
    "    \n",
    "    # First, convert clases to 0-(N-1) integers using label_encoder\n",
    "    num_of_rows = df.shape[0]\n",
    "    t = label_encoder.transform(df[f]).reshape(num_of_rows, 1)\n",
    "    \n",
    "    # Second, create a sparse matrix with three columns, each one indicating if the instance belongs to the class\n",
    "    new_features = one_hot_encoder.transform(t)\n",
    "    \n",
    "    # Add the new features to df\n",
    "    df = np.concatenate([df, new_features.toarray()], axis = 1)\n",
    "    \n",
    "    #Eliminate converted columns\n",
    "    df.drop(f, axis=1, inplace=True)\n",
    "    \n",
    "    # Update feature names\n",
    "    feature_names = ['age', 'sex', 'first_class', 'second_class', 'third_class']\n",
    "    # Convert to numerical values\n",
    "    df = df.astype(float)\n",
    "    titanic_y = titanic_y.astype(float)\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "features_list = list(onehot_df.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "features_list.remove('Earnings-Raw')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = onehot_df[features_list]\n",
    "y= onehot_df['Earnings-Raw']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature importances with forests of trees\n",
    "\n",
    "This examples shows the use of forests of trees to evaluate the importance of features on an artificial classification task. The red bars are the feature importances of the forest, along with their inter-trees variability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# split dataset to 60% training and 40% testing\n",
    "X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, test_size=0.4, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(18097, 103) (18097,)\n"
     ]
    }
   ],
   "source": [
    "print (X_train.shape, y_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### feature\\_importances_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "\n",
    "# Build a classification task using 3 informative features\n",
    "\n",
    "# Build a forest and compute the feature importances\n",
    "forest = ExtraTreesClassifier(n_estimators=250,\n",
    "                              random_state=0)\n",
    "\n",
    "forest.fit(X, y)\n",
    "importances = forest.feature_importances_\n",
    "std = np.std([tree.feature_importances_ for tree in forest.estimators_],\n",
    "             axis=0)\n",
    "indices = np.argsort(importances)[::-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.17 ,  0.153,  0.093,  0.071,  0.066])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "importances[indices[:5]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. feature 1 - fnlwgt (0.170258)\n",
      "2. feature 0 - Age (0.152898)\n",
      "3. feature 6 - Hours-per-week (0.093478)\n",
      "4. feature 32 - Marital-Status_ Married-civ-spouse (0.070931)\n",
      "5. feature 4 - Capital-gain (0.065981)\n"
     ]
    }
   ],
   "source": [
    "features = features_list\n",
    "for f in range(5):\n",
    "    print(\"%d. feature %d - %s (%f)\" % (f + 1, indices[f], features[indices[f]] ,importances[indices[f]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "best_features = []\n",
    "for i in indices[:5]:\n",
    "    best_features.append(features[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqQAAAINCAYAAADoVW2PAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzs3X1c1fX9//HnOUByoZKKgRdJaKaoE+SYE5upwUKt1KXy\n9ZQh9Z3ZyDRx2YXpwsqLzXUaU6Nsrlp5cn6r2ZVjhnax5GsG6FoQWaaGxllnS/1tSBJ+fn/w9dQJ\nUA5ib7DH/XY7t+L9eb/fn9fnfBSft8+lzbIsSwAAAIAhdtMFAAAA4PuNQAoAAACjCKQAAAAwikAK\nAAAAowikAAAAMIpACgAAAKMIpAAAADCKQAoAAACjCKQAAAAwikAKAC1k//79stvteuqpp0yXAgBt\nCoEUQLM8+eSTstvtDX7uueees7bezZs3Kycn56zNf6ZsNpvpEs7II488oieffNJ0GQC+Z4JNFwCg\n7bLZbLr//vt10UUX+bUPGjTorK3z1Vdf1Zo1a/SLX/zirK2juWJjY3Xs2DGFhISYLqXZ1qxZo65d\nu2rGjBmmSwHwPUIgBXBGxo4dq6SkpO9sfZZlnZV5q6qqFB4efsbznHfeeS1QzXfv2LFjCgsLM10G\ngO8pTtkDOOuefvppDR06VOHh4erSpYucTqcqKir8+vz1r39Venq6YmNjFRoaql69eik7O1vV1dW+\nPjfeeKPWrFkjSb7LA4KCgiRJr7/+uux2u958802/eRu6rjMzM1MdOnTQ3r17NX78eHXs2FHTp0/3\nLd+xY4fGjh2r888/XxERERo9erS2b99+2u081bo+/fRTXX311erQoYN69uzp24733ntPKSkpat++\nvS666CK53W6/OU9eGvHWW29p1qxZioqKUmRkpGbMmKHDhw/Xq2HNmjUaNGiQQkND1aNHD82ePVtH\njhzx6zN69GgNHjxYxcXFuvzyyxUREaF77rlHcXFxev/9933fpd1u1xVXXCFJ+uKLL/Tzn/9cgwcP\nVocOHRQZGanx48frb3/7m9/cb7zxhux2uzZu3KgHH3xQF154ocLCwpSamqqPP/64Xr07duzQ+PHj\n1blzZ7Vv314JCQnKzc3161NeXq4pU6aoS5cuCgsL06WXXqqXXnrJr89XX32lnJwcXXLJJQoLC1NU\nVJRGjhypgoKC0+02AK0AR0gBnJEjR47on//8p19bly5dfP//4IMPavHixZo2bZpmzpypzz//XLm5\nuRo1apRKSkrUsWNHSdLGjRt17NgxZWVlqUuXLnrnnXf029/+VgcPHtSGDRskSbfccosOHTqk1157\nTc8884zf0VKbzdbk6zdtNpu++uorpaWlaeTIkfr1r3/tOzq6detWjR8/XkOHDtV9990nu92u3//+\n97riiiv017/+VUOHDg3o+7HZbDpx4oTGjRunUaNG6Ve/+pWeeeYZ3XbbbYqIiNDChQs1ffp0TZ48\nWXl5eZoxY4ZGjBih2NhYv3lmz56tTp06KScnR+Xl5VqzZo0OHDigbdu2+frcd999WrJkia688kpl\nZWX5+r377rt6++23feHdZrPJ6/Vq/PjxmjZtmjIyMhQdHa0xY8Zo9uzZ6tChg+69915ZlqXo6GhJ\n0t69e/Xiiy9q6tSpiouLk8fj0aOPPqrRo0ertLRUMTExfvUuX75cQUFBuuOOO3TkyBGtWLFC06dP\nV2Fhoa/Pli1bdM0116h79+66/fbbFRMTo7KyMr3yyiuaM2eOJOn999/Xj370I/Xs2VN33323IiIi\n9Mc//lGTJk3S888/r4kTJ0qSfvGLX2j58uW6+eabdemll+ro0aN69913VVxcrJSUlID2GQADLABo\nhieeeMKy2Wz1Pna73ddn//79VnBwsLV8+XK/se+//74VEhJiLVu2zNdWXV1dbx3Lly+3goKCrE8/\n/dTXNnv2bL91nPT6669bdrvdeuONN/za9+3bZ9lsNuvJJ5/0tWVmZlp2u91auHBhvXkuueQSa/z4\n8X5t1dXVVu/eva20tLTGvo7TrmvFihW+tsOHD1vh4eFWUFCQtXHjRl97eXm5ZbPZrJycHF/bye95\n2LBh1ldffeVr/9WvfmXZ7XbrpZdesizLsj7//HOrXbt21rhx4/xqWr16tWW3260nnnjC1zZ69GjL\nbrdba9eurbcNgwYNssaMGVOv/fjx4/Xa9u/fb4WGhloPPPCAr+3111+3bDabNXDgQL96c3NzLbvd\nbr3//vuWZVlWbW2tFRcXZ/Xu3ds6evRovblPSklJsRITE62amhq/9ssuu8zq16+f7+fExETrmmuu\naXQeAK0bp+wBNJvNZtMjjzyi1157zffZsmWLb/lzzz0ny7I0depU/fOf//R9LrjgAvXt29fv6F67\ndu18/19VVaV//vOfSk5O1okTJ1RSUnJW6r/lllv8ft61a5f27Nkjp9PpV+//+3//TykpKfUuBwjE\nf//3f/v+PzIyUv369VNERISmTJnia7/kkkt0/vnna+/evfXG33zzzb4jnJL0s5/9TEFBQXr11Vcl\n1R1trKmp0e233+43bubMmerQoYNeeeUVv/Z27dopMzOzyfV/80atEydO6F//+pfCw8PVr18/FRcX\n1+t/0003+dU7cuRIWZbl27aSkhLt27dPt99+uzp06NDgOr/44gtt27ZNU6dO9R2JP/m58sortWfP\nHn322WeSpPPPP1/vv/++PvrooyZvE4DWg1P2AM7IpZde2uhNTR999JFOnDihiy++uN4ym83mdwPQ\np59+qkWLFumll17SF1984dfv29dAtoTg4GD17NnTr23Pnj2SpIyMjAbH2O12HTlyRJGRkQGtKzQ0\n1O8yBqkulH57/Sfbv7n9Ut138O3vMCIiQt26ddO+ffskSQcOHJBUF2q/KSQkRL1799b+/fv92nv0\n6KHg4Kb/E2BZlh5++GE98sgj+uSTT1RbW+urLSoqql7/Cy+80O/nTp06SZJv2z7++GPZbDYNHDiw\n0XV+9NFHsixLixYt0r333ltvuc1m0z/+8Q9169ZNS5Ys0aRJk3TJJZdo0KBBGjt2rG644Qb94Ac/\naPI2AjCHQArgrDlx4oTsdrv+/Oc/y26vf0Kmffv2vn6pqak6fPiw7r77bt/Rw4MHD2rGjBk6ceLE\nadfV2PWjJ4PTt33ziOw365WkX//610pISGhw3MmaA/HNI4VNabfO0pMEvinQO+pPXgv805/+VA88\n8IA6d+4su92uuXPnNrh/WmLbTs7785//XGlpaQ32ORnUR44cqY8//libNm3SX/7yF/3ud7+Ty+XS\no48+qptuuqnJ6wRgBoEUwFnTp08fWZaliy66qMGjpCe999572rNnj/7whz/o+uuv97W/9tpr9fo2\nFjw7deoky7Lq3Xl+8ghiU+uVpA4dOvjuLm8NLMvSnj17NGrUKF/bf/7zH3322We66qqrJMl3E1R5\nebnfc2Framr0ySef6Mc//nGT1tXY9/vcc8/piiuu0GOPPebXfvjwYXXt2jWQzZH09Z+Nv//9741+\n171795ZUd5S3Kfvj/PPP14wZMzRjxgxVVVVp5MiRuu+++wikQBvANaQAzpprr71Wdru90Tcr/etf\n/5L09dG0bx9pe/jhh+sFpIiICEnS0aNH/dpjY2MVFBRU7zrPNWvWNPnue4fDoT59+mjlypX6z3/+\nU2+51+tt0jxnw2OPPaavvvrK9/OaNWtUW1ur8ePHS5JSU1MVEhJS75FJjz/+uI4ePaqrr766SeuJ\niIho8HFSQUFB9Y5ubty4UQcPHgx0UyRJSUlJiouL08MPP9zoJRldu3bV6NGj9eijj6qysrLe8m/u\nj5N/lk4KDw/XxRdfrC+//LJZ9QH4bnGEFECzne70a+/evfXAAw/onnvu0SeffKJJkyb5nv/5pz/9\nSbNmzVJ2drb69++vPn36aP78+aqoqFDHjh313HPPNRiMHA6HLMvSbbfdprS0NAUFBem//uu/1LFj\nR02dOtUXyPr06aOXX35Zn3/+eZO3x2az6fHHH9f48eM1cOBA3XjjjerRo4cOHjyobdu2KTIyUps2\nbQrsS2ohx48fV0pKitLT0/XBBx/okUce0ciRI31BMyoqSnfffbeWLFmisWPHasKECb5+w4YN8zvy\nfCoOh0N5eXl68MEHdfHFF+uCCy7QmDFjdPXVV+v+++/XTTfdpBEjRui9997TM8884zuqHKiTN8RN\nmDBBiYmJuvHGG9WtWzd98MEHKi0t1ebNmyVJq1ev1siRI/WDH/xAM2fOVO/eveXxeFRYWKiDBw/6\nbngbMGCARo8eLYfDoc6dO2vnzp36n//5H9/jowC0ckbu7QfQ5j3xxBOW3W63ioqKTtv3hRdesC6/\n/HKrQ4cOVocOHawBAwZYc+bMsfbs2ePr88EHH1hXXnml1bFjR+uCCy6wbrnlFuu9996z7Ha732OU\namtrrblz51rR0dFWUFCQ3yOgvF6vNXXqVKt9+/ZWly5drKysLKu0tLTeHJmZmVbHjh0brXf37t3W\nlClTrK5du1phYWFWXFycNW3aNGvbtm2n3M59+/Y1eV2jR4+2Bg8eXK89Li7OmjBhgu/nk9/zW2+9\nZd1yyy1Wly5drI4dO1oZGRnWF198UW/8mjVrrAEDBljt2rWzunXrZs2ePds6cuRIk9ZtWZbl8Xis\na665xoqMjLTsdrvvEVBffvmldccdd1g9evSwIiIirMsvv9zasWOHNWbMGOuKK67wjT/5+K3nnnvu\ntN+NZVnW9u3brbS0NCsyMtLq0KGDlZiYaK1Zs8avzyeffGJlZmZa3bt3t9q1a2ddeOGF1oQJE6zn\nn3/e12fp0qXW8OHDrc6dO1sRERHWgAEDrOXLl/s9egpA62WzrO/g6nkAQLM8+eSTuummm7Rz587v\n9BWtAPBdatY1pKtXr1ZcXJzCwsI0fPhw7dy5s9G+paWlmjJliuLi4mS32+td3yTVXTe2aNEi9e7d\n23fdzwMPPNCc0gAAANDGBBxIN2zYoPnz5ysnJ0clJSVKSEhQWlpaoxf7V1VVqU+fPlqxYoW6devW\nYJ/ly5fr0Ucf1Zo1a/TBBx/ol7/8pX75y19q1apVgZYHAOccTmQBONcFHEhdLpdmzZqljIwM9e/f\nX3l5eQoPD9e6desa7D906FCtWLFC6enpfg/B/qbCwkJNnDhRY8eOVa9evXTttdfqyiuv1DvvvBNo\neQBwzmnqUwIAoK0KKJDW1NSoqKhIKSkpvjabzabU1FQVFhY2u4gRI0aooKDA95aU3bt36+233/Y9\nzgQAvq9mzJih2tparh8FcE4L6LFPXq9XtbW1io6O9muPjo5WeXl5s4u46667dPToUfXv319BQUE6\nceKEHnzwQU2bNq3ROvLz83XRRRcF/LYRAAAAnH3Hjh3Tvn37lJaW1uArhr+pVTyHdMOGDVq/fr2e\nffZZDRgwQLt27dLcuXPVvXt33XDDDfX65+fna/r06QYqBQAAQCCefvrp0z4LOaBAGhUVpaCgIHk8\nHr92j8ejmJiYwCv8PwsWLNA999yjqVOnSpIGDhyoffv2admyZQ0G0pOvxXv66acVHx/f7PXi7Jk3\nb55cLpfpMtAA9k3rxb5p3dg/rRf7pnUqKyvT9OnT/V5n3JiAAmlISIgcDocKCgo0YcIESXV3fxYU\nFJzR2zCqqqrqXbRvt9vrvUbwpJOn6ePj47muqpWKjIxk37RS7JvWi33TurF/Wi/2TevWlMsrAz5l\nn52drczMTDkcDg0bNkwul0tVVVXKzMyUJGVkZKhnz55aunSppLoboUpLS2VZlo4fP66DBw9q9+7d\nat++ve+Vc9dcc40efPBBXXjhhRo4cKCKi4vlcrn005/+NNDyAAAA0MYEHEjT09Pl9Xq1ePFieTwe\nJSYmKj8/X127dpUkVVRUKDj462kPHTqkIUOG+I6Arly5UitXrtSoUaO0detWSdKqVau0aNEi3Xrr\nrfrHP/6h7t2762c/+5kWLVrUEtsIAACAVqxZNzVlZWUpKyurwWUnQ+ZJsbGxjZ56PykiIkIPPfSQ\nHnrooeaUAwAAgDasWa8OBU7H6XSaLgGNYN+0Xuyb1o3903qxb9o+m9UG30lXXFwsh8OhoqIiLmIG\nAABohQLJaxwhBQAAgFEEUgAAABhFIAUAAIBRBFIAAAAYRSAFAACAUQRSAAAAGEUgBQAAgFEEUgAA\nABjVrFeHAt/mdtd9JKm6Wtq/X4qNlUJD69qczroPAADAtxFI0SK+GTiLiyWHoy6g8iItAABwOpyy\nBwAAgFEEUgAAABhFIAUAAIBRBFIAAAAYRSAFAACAUQRSAAAAGMVjn9qYAwcOyOv1mi7jlMrKwiTF\nq6ysTNIx0+U0KioqSr169TJdBgAA33sE0jbkwIEDiu/XT1XV1aZLOY0hkoo1ffr1kkpMF9Oo8NBQ\nlZWXE0oBADCMQNqGeL1eVVVX62lJ8aaLOYUySdOlVl1nmaTp1dXyer0EUgAADCOQtkHxktrCC5Da\nSp0AAMAsbmoCAACAUQRSAAAAGEUgBQAAgFEEUgAAABhFIAUAAIBRBFIAAAAYRSAFAACAUTyHFC3C\nrWlyyylJqlY7XaJy3aVlCtWXkiSn3HLqWZMlAgCAVopAihbh1LMETgAA0CycsgcAAIBRBFIAAAAY\nRSAFAACAUQRSAAAAGEUgBQAAgFHcZQ+c49zuuo8kVVdL+/dLsbFSaGhdm9NZ9wEAwBQCKXCO+2bg\nLC6WHI66gJqUZLYuAABO4pQ9AAAAjCKQAgAAwCgCKQAAAIxqViBdvXq14uLiFBYWpuHDh2vnzp2N\n9i0tLdWUKVMUFxcnu92u3NzcBvsdOnRIN9xwg6KiohQeHq6EhAQVFxc3pzwAAAC0IQEH0g0bNmj+\n/PnKyclRSUmJEhISlJaWJq/X22D/qqoq9enTRytWrFC3bt0a7HP48GFddtllateunfLz81VWVqZf\n//rX6tSpU6DlAQAAoI0J+C57l8ulWbNmKSMjQ5KUl5enV155RevWrdOCBQvq9R86dKiGDh0qSbrz\nzjsbnHP58uXq1auXHn/8cV9bbGxsoKUBAACgDQroCGlNTY2KioqUkpLia7PZbEpNTVVhYWGzi3jp\npZc0dOhQpaenKzo6WklJSX7hFAAAAOeugAKp1+tVbW2toqOj/dqjo6NVWVnZ7CL27t2rRx55RP36\n9dNf/vIX/exnP9OcOXP0hz/8odlzAgAAoG1oFQ/GP3HihIYNG6b7779fkpSQkKC///3vysvL0w03\n3NDouHnz5ikyMtKvzel0yslrZwAAAL4zbrdb7pOvBfw/R44cafL4gAJpVFSUgoKC5PF4/No9Ho9i\nYmICmcpPt27dFB8f79cWHx+v559//pTjXC6XknjdDAAAgFENHRAsLi6Ww+Fo0viATtmHhITI4XCo\noKDA12ZZlgoKCjRixIhApvJz2WWXqby83K+tvLycG5sAAAC+BwI+ZZ+dna3MzEw5HA4NGzZMLpdL\nVVVVyszMlCRlZGSoZ8+eWrp0qaS6G6FKS0tlWZaOHz+ugwcPavfu3Wrfvr369Okjqe7U+2WXXaZl\ny5YpPT1dO3bs0OOPP661a9e23JYCAACgVQo4kKanp8vr9Wrx4sXyeDxKTExUfn6+unbtKkmqqKhQ\ncPDX0x46dEhDhgyRzWaTJK1cuVIrV67UqFGjtHXrVkl1j4Z64YUXdNddd+n+++9XXFycfvOb32ja\ntGktsY0AAABoxZp1U1NWVpaysrIaXHYyZJ4UGxurEydOnHbO8ePHa/z48c0pBwAAAG0Y77IHAACA\nUQRSAAAAGEUgBQAAgFEEUgAAABhFIAUAAIBRBFIAAAAYRSAFAACAUQRSAAAAGEUgBQAAgFEEUgAA\nABhFIAUAAIBRzXqXPYCGHThwQF6v13QZjSorC5MUr7KyMknHTJfTqKioKPXq1ct0GQCA7wiBFGgh\nBw4cUHy/fqqqrjZdyikMkVSs6dOvl1RiuphGhYeGqqy8nFAKAN8TBFKghXi9XlVVV+tpSfGmi2lE\nmaTpUuuvsbpaXq+XQAoA3xMEUqCFxUtKMl3EabSFGgEA3x/c1AQAAACjCKQAAAAwikAKAAAAowik\nAAAAMIpACgAAAKMIpAAAADCKQAoAAACjCKQAAAAwikAKAAAAowikAAAAMIpACgAAAKMIpAAAADCK\nQAoAAACjCKQAAAAwikAKAAAAowikAAAAMIpACgAAAKMIpAAAADCKQAoAAACjCKQAAAAwikAKAAAA\nowikAAAAMIpACgAAAKMIpAAAADCKQAoAAACjmhVIV69erbi4OIWFhWn48OHauXNno31LS0s1ZcoU\nxcXFyW63Kzc395RzL1++XHa7XdnZ2c0pDQAAAG1McKADNmzYoPnz5+uxxx7TsGHD5HK5lJaWpg8/\n/FBRUVH1+ldVValPnz5KT0/XvHnzTjn3zp079dhjjykhISHQsgA0wq1pcsspSapWO12ict2lZQrV\nl5Ikp9xy6lmTJQIAvucCDqQul0uzZs1SRkaGJCkvL0+vvPKK1q1bpwULFtTrP3ToUA0dOlSSdOed\ndzY677///W9Nnz5djz/+uO6///5AywLQCKeeJXACAFq1gE7Z19TUqKioSCkpKb42m82m1NRUFRYW\nnlEht956q6655hpdccUVZzQPAAAA2paAjpB6vV7V1tYqOjrarz06Olrl5eXNLuLZZ5/Vrl279O67\n7zZ7DgAAALRNAZ+yb2kVFRW6/fbb9dprrykkJMR0OQAAAPiOBRRIo6KiFBQUJI/H49fu8XgUExPT\nrAKKior0+eefKykpSZZlSZJqa2v15ptvatWqVfryyy9ls9kaHDtv3jxFRkb6tTmdTjmdzmbVAgAA\ngMC53W653W6/tiNHjjR5fECBNCQkRA6HQwUFBZowYYIkybIsFRQUaM6cOYFM5ZOamqr33nvPry0z\nM1Px8fG66667Gg2jUt0NVklJSc1aLwAAAFpGQwcEi4uL5XA4mjQ+4FP22dnZyszMlMPh8D32qaqq\nSpmZmZKkjIwM9ezZU0uXLpVUdyNUaWmpLMvS8ePHdfDgQe3evVvt27dXnz59FBERoQEDBvitIyIi\nQl26dFF8fHyg5QEAAKCNCTiQpqeny+v1avHixfJ4PEpMTFR+fr66du0qqe6a0ODgr6c9dOiQhgwZ\n4jvSuXLlSq1cuVKjRo3S1q1bG1zHqY6KAgAA4NzSrJuasrKylJWV1eCyb4fM2NhYnThxIqD5Gwuq\nAAAAOPcYv8seAIDWyO2u+0hSdbW0f78UGyuFhta1OZ11HwBnjkAKAEADvhk4i4slh6MuoHIvLdDy\nAnpTEwAAANDSCKQAAAAwikAKAAAAowikAAAAMIpACgAAAKMIpAAAADCKQAoAAACjCKQAAAAwikAK\nAAAAowikAAAAMIpACgAAAKMIpAAAADCKQAoAAACjCKQAAAAwikAKAAAAowikAAAAMIpACgAAAKMI\npAAAADCKQAoAAACjCKQAAAAwikAKAAAAowikAAAAMIpACgAAAKMIpAAAADCKQAoAAACjCKQAAAAw\nikAKAAAAowikAAAAMIpACgAAAKMIpAAAADCKQAoAAACjCKQAAAAwikAKAAAAowikAAAAMIpACgAA\nAKMIpAAAADCKQAoAAACjCKQAAAAwikAKAAAAo5oVSFevXq24uDiFhYVp+PDh2rlzZ6N9S0tLNWXK\nFMXFxclutys3N7den2XLlmnYsGHq2LGjoqOj9ZOf/EQffvhhc0oDAABAGxNwIN2wYYPmz5+vnJwc\nlZSUKCEhQWlpafJ6vQ32r6qqUp8+fbRixQp169atwT5vvfWWbrvtNu3YsUOvvfaaampqdOWVV+rY\nsWOBlgcAAIA2JjjQAS6XS7NmzVJGRoYkKS8vT6+88orWrVunBQsW1Os/dOhQDR06VJJ05513Njjn\nq6++6vfzE088oQsuuEBFRUX60Y9+FGiJAAAAaEMCOkJaU1OjoqIipaSk+NpsNptSU1NVWFjYYkUd\nPnxYNptNnTt3brE5AQAA0DoFFEi9Xq9qa2sVHR3t1x4dHa3KysoWKciyLN1+++360Y9+pAEDBrTI\nnAAAAGi9Aj5lf7ZlZWWptLRUb7/99mn7zps3T5GRkX5tTqdTTqfzbJUHAACAb3G73XK73X5tR44c\nafL4gAJpVFSUgoKC5PF4/No9Ho9iYmICmapBs2fP1quvvqq33nqr0RugvsnlcikpKemM1wsAAIDm\na+iAYHFxsRwOR5PGB3TKPiQkRA6HQwUFBb42y7JUUFCgESNGBDJVPbNnz9amTZu0bds29erV64zm\nAgAAQNsR8Cn77OxsZWZmyuFwaNiwYXK5XKqqqlJmZqYkKSMjQz179tTSpUsl1d0IVVpaKsuydPz4\ncR08eFC7d+9W+/bt1adPH0l1p+ndbrdefPFFRURE+I7ARkZGKjQ0tIU2FQAAAK1RwIE0PT1dXq9X\nixcvlsfjUWJiovLz89W1a1dJUkVFhYKDv5720KFDGjJkiGw2myRp5cqVWrlypUaNGqWtW7dKqnt0\nlM1m0+jRo/3W9fvf/973eCkAAACcm5p1U1NWVpaysrIaXHYyZJ4UGxurEydOnHK+0y0HAADAuYt3\n2QMAAMAoAikAAACMIpACAADAKAIpAAAAjCKQAgAAwCgCKQAAAIwikAIAAMAoAikAAACMIpACAADA\nKAIpAAAAjCKQAgAAwCgCKQAAAIwikAIAAMAoAikAAACMIpACAADAKAIpAAAAjCKQAgAAwCgCKQAA\nAIwikAIAAMAoAikAAACMIpACAADAKAIpAAAAjCKQAgAAwCgCKQAAAIwikAIAAMAoAikAAACMIpAC\nAADAKAIpAAAAjCKQAgAAwCgCKQAAAIwikAIAAMAoAikAAACMIpACAADAKAIpAAAAjCKQAgAAwCgC\nKQAAAIwikAIAAMAoAikAAACMIpACAADAKAIpAAAAjCKQAgAAwKhmBdLVq1crLi5OYWFhGj58uHbu\n3Nlo39LSUk2ZMkVxcXGy2+3Kzc094zkBAABw7gg4kG7YsEHz589XTk6OSkpKlJCQoLS0NHm93gb7\nV1VVqU8Dk5g9AAAgAElEQVSfPlqxYoW6devWInMCAADg3BFwIHW5XJo1a5YyMjLUv39/5eXlKTw8\nXOvWrWuw/9ChQ7VixQqlp6frvPPOa5E5AQAAcO4IKJDW1NSoqKhIKSkpvjabzabU1FQVFhY2q4Cz\nMScAAADajoACqdfrVW1traKjo/3ao6OjVVlZ2awCzsacAAAAaDuCTRdwJubNm6fIyEi/NqfTKafT\naagiAACA7x+32y232+3XduTIkSaPDyiQRkVFKSgoSB6Px6/d4/EoJiYmkKlaZE6Xy6WkpKRmrRcA\nAAAto6EDgsXFxXI4HE0aH9Ap+5CQEDkcDhUUFPjaLMtSQUGBRowYEchUZ3VOAAAAtB0Bn7LPzs5W\nZmamHA6Hhg0bJpfLpaqqKmVmZkqSMjIy1LNnTy1dulRS3U1LpaWlsixLx48f18GDB7V79261b99e\nffr0adKcAAAAOHcFHEjT09Pl9Xq1ePFieTweJSYmKj8/X127dpUkVVRUKDj462kPHTqkIUOGyGaz\nSZJWrlyplStXatSoUdq6dWuT5gSAc5HbXfeRpOpqaf9+KTZWCg2ta3M66z4AcK5r1k1NWVlZysrK\nanDZyZB5UmxsrE6cOHFGcwLAueibgbO4WHI46gIql8YD+L7hXfYAAAAwikAKAAAAowikAAAAMIpA\nCgAAAKMIpAAAADCKQAoAAACjCKQAAAAwikAKAAAAowikAAAAMIpACgAAAKOa9epQAGhrDhw4IK/X\na7qMRpWVhUmKV1lZmaRjpss5paioKPXq1ct0GQDOIQRSAOe8AwcOKL5fP1VVV5su5RSGSCrW9OnX\nSyoxXcwphYeGqqy8nFAKoMUQSAGc87xer6qqq/W0pHjTxTSiTNJ0qVXXKP1fndXV8nq9BFIALYZA\nCuB7I15SkukiTqMt1AgALY2bmgAAAGAUgRQAAABGEUgBAABgFIEUAAAARhFIAQAAYBSBFAAAAEYR\nSAEAAGAUgRQAAABGEUgBAABgFIEUAAAARvHqUAAwxK1pcsspSapWO12ict2lZQrVl5Ikp9xy6lmT\nJQLAd4JACgCGOPUsgRMAxCl7AAAAGEYgBQAAgFEEUgAAABhFIAUAAIBRBFIAAAAYRSAFAACAUQRS\nAAAAGEUgBQAAgFEEUgAAABhFIAUAAIBRBFIAAAAYRSAFAACAUQRSAAAAGEUgBQAAgFHNCqSrV69W\nXFycwsLCNHz4cO3cufOU/Tdu3Kj4+HiFhYUpISFBmzdv9lv+n//8R7Nnz9aFF16o8PBwDRw4UI8+\n+mhzSgMAAEAbE3Ag3bBhg+bPn6+cnByVlJQoISFBaWlp8nq9Dfbfvn27rrvuOs2cOVO7du3SxIkT\nNWnSJJWWlvr6zJs3T3/5y1+0fv16ffDBB7r99ts1e/Zsvfzyy83fMgAAALQJAQdSl8ulWbNmKSMj\nQ/3791deXp7Cw8O1bt26Bvvn5uZq3Lhxys7OVr9+/bRkyRIlJSVp1apVvj6FhYWaMWOGRo4cqV69\nemnmzJlKSEjQO++80/wtAwAAQJsQUCCtqalRUVGRUlJSfG02m02pqakqLCxscExhYaFSU1P92tLS\n0vz6jxgxQi+++KIOHTokSdq2bZv27NmjtLS0QMoDAABAGxQcSGev16va2lpFR0f7tUdHR6u8vLzB\nMZWVlQ32r6ys9P3829/+VjfffLN69uyp4OBgBQUFae3atbrssssCKQ8AAABtUECB9GzJzc3Vjh07\n9PLLL6tXr1568803lZWVpe7du+uKK64wXR4AAADOooACaVRUlIKCguTxePzaPR6PYmJiGhwTExNz\nyv7V1dVauHChNm3apLFjx0qSBg0apJKSEq1cufKUgXTevHmKjIz0a3M6nXI6nYFsFgAAAM6A2+2W\n2+32azty5EiTxwcUSENCQuRwOFRQUKAJEyZIkizLUkFBgebMmdPgmOTk5HrLt2zZouTkZEl116XW\n1NTIZrP5jQsKCtKJEydOWY/L5VJSUlIgmwAAAIAW1tABweLiYjkcjiaND/iUfXZ2tjIzM+VwODRs\n2DC5XC5VVVUpMzNTkpSRkaGePXtq6dKlkqS5c+dq9OjReuihh3TVVVfJ7XarqKhIa9eulSR16NBB\no0aN0h133KHQ0FDFxsbq9ddf11NPPaWHH3440PIAAADQxgQcSNPT0+X1erV48WJ5PB4lJiYqPz9f\nXbt2lSRVVFQoOPjraZOTk7V+/XotXLhQCxcuVN++fbVp0yYNGDDA12fDhg26++67NX36dP3rX/9S\nbGysli1bpptvvrkFNhEAAACtWbNuasrKylJWVlaDy7Zu3VqvbfLkyZo8eXKj811wwQX63e9+15xS\nAAAA0Ma1irvsAQAAmsrtrvtIUnW1tH+/FBsrhYbWtTmddR+0HQRSAADQpnwzcBYXSw5HXUDlPue2\nK+BXhwIAAAAtiUAKAAAAowikAAAAMIpACgAAAKMIpAAAADCKQAoAAACjCKQAAAAwikAKAAAAowik\nAAAAMIpACgAAAKMIpAAAADCKQAoAAACjCKQAAAAwikAKAAAAowikAAAAMCrYdAEAgO+3AwcOyOv1\nmi7jlMrKwiTFq6ysTNIx0+U0KioqSr169TJdBhAwAikAwJgDBw4ovl8/VVVXmy7lNIZIKtb06ddL\nKjFdTKPCQ0NVVl5OKEWbQyAFABjj9XpVVV2tpyXFmy7mFMokTZdadZ1lkqZXV8vr9RJI0eYQSAEA\nxsVLSjJdRBO0lTqBtoabmgAAAGAUgRQAAABGEUgBAABgFIEUAAAARhFIAQAAYBSBFAAAAEYRSAEA\nAGAUgRQAAABGEUgBAABgFG9qAgAAjTpw4IC8Xq/pMhpVVhYmKV5lZWWSjpkup1FRUVG80vUUCKQA\nAKBBBw4cUHy/fqqqrjZdyikMkVSs6dOvl1RiuphGhYeGqqy8nFDaCAIpAABokNfrVVV1tZ6WFG+6\nmEaUSZoutf4aq6vl9XoJpI0gkAIAgFOKl5RkuojTaAs1onHc1AQAAACjCKQAAAAwikAKAAAAowik\nAAAAMIpACgAAAKMIpAAAADCKQAoAAACjmhVIV69erbi4OIWFhWn48OHauXPnKftv3LhR8fHxCgsL\nU0JCgjZv3lyvT1lZmSZOnKjzzz9f7du31w9/+ENVVFQ0pzwAAAC0IQEH0g0bNmj+/PnKyclRSUmJ\nEhISlJaW1uh7brdv367rrrtOM2fO1K5duzRx4kRNmjRJpaWlvj4ff/yxRo4cqQEDBujNN9/Ue++9\np0WLFik0NLT5WwYAAIA2IeBA6nK5NGvWLGVkZKh///7Ky8tTeHi41q1b12D/3NxcjRs3TtnZ2erX\nr5+WLFmipKQkrVq1ytfn3nvv1VVXXaVly5Zp8ODBiouL09VXX62oqKjmbxkAAADahIACaU1NjYqK\nipSSkuJrs9lsSk1NVWFhYYNjCgsLlZqa6teWlpbm629Zll555RX17dtXY8eOVXR0tIYPH65NmzYF\nui0AAABogwIKpF6vV7W1tYqOjvZrj46OVmVlZYNjKisrT9n/H//4h/79739rxYoVGj9+vLZs2aKf\n/OQnuvbaa/XWW28FUh4AAADaoGDTBZw4cUKSNGnSJM2ZM0eSNHjwYG3fvl15eXkaOXJko2PnzZun\nyMhIvzan0ymn03n2CgYAAIAft9stt9vt13bkyJEmjw8okEZFRSkoKEgej8ev3ePxKCYmpsExMTEx\np+wfFRWl4OBgxcfH+/WJj4/X22+/fcp6XC6XkpKSAtkEAAAAtLCGDggWFxfL4XA0aXxAp+xDQkLk\ncDhUUFDga7MsSwUFBRoxYkSDY5KTk/36S9KWLVuUnJzsm/PSSy9VeXm5X58PP/xQsbGxgZQHAACA\nNijgU/bZ2dnKzMyUw+HQsGHD5HK5VFVVpczMTElSRkaGevbsqaVLl0qS5s6dq9GjR+uhhx7SVVdd\nJbfbraKiIq1du9Y35x133KFp06Zp5MiRGjNmjDZv3qyXX35Zb7zxRstsJQAAAFqtgANpenq6vF6v\nFi9eLI/Ho8TEROXn56tr166SpIqKCgUHfz1tcnKy1q9fr4ULF2rhwoXq27evNm3apAEDBvj6TJo0\nSXl5eVq6dKnmzp2rfv366fnnn/cdRQUAAMC5q1k3NWVlZSkrK6vBZVu3bq3XNnnyZE2ePPmUc2Zm\nZvqOsgIAAOD7g3fZAwAAwCgCKQAAAIwikAIAAMAoAikAAACMIpACAADAKAIpAAAAjCKQAgAAwCgC\nKQAAAIwikAIAAMAoAikAAACMatarQwEAAExxa5rcckqSqtVOl6hcd2mZQvWlJMkpt5x61mSJCBCB\nFAAAtClOPUvgPMdwyh4AAABGEUgBAABgFIEUAAAARhFIAQAAYBSBFAAAAEYRSAEAAGAUgRQAAABG\nEUgBAABgFIEUAAAARhFIAQAAYBSBFAAAAEYRSAEAAGAUgRQAAABGEUgBAABgFIEUAAAARhFIAQAA\nYBSBFAAAAEYRSAEAAGAUgRQAAABGEUgBAABgFIEUAAAARhFIAQAAYBSBFAAAAEYRSAEAAGAUgRQA\nAABGEUgBAABgFIEUAAAARhFIAQAAYBSBFAAAAEY1K5CuXr1acXFxCgsL0/Dhw7Vz585T9t+4caPi\n4+MVFhamhIQEbd68udG+t9xyi+x2u3Jzc5tTGgAAANqYgAPphg0bNH/+fOXk5KikpEQJCQlKS0uT\n1+ttsP/27dt13XXXaebMmdq1a5cmTpyoSZMmqbS0tF7fF154QTt27FCPHj0C3xIAAAC0SQEHUpfL\npVmzZikjI0P9+/dXXl6ewsPDtW7dugb75+bmaty4ccrOzla/fv20ZMkSJSUladWqVX79Dh48qLlz\n52r9+vUKDg5u3tYAAACgzQkokNbU1KioqEgpKSm+NpvNptTUVBUWFjY4prCwUKmpqX5taWlpfv0t\ny1JGRoYWLFig+Pj4QEoCAABAGxdQIPV6vaqtrVV0dLRfe3R0tCorKxscU1lZedr+y5cv13nnnafZ\ns2cHUg4AAADOAcbPjRcVFSk3N1clJSUBj503b54iIyP92pxOp5xOZ0uVBwAAgNNwu91yu91+bUeO\nHGny+IACaVRUlIKCguTxePzaPR6PYmJiGhwTExNzyv5//etf9fnnn+vCCy/0La+trVV2drYefvhh\n7d27t9F6XC6XkpKSAtkEAAAAtLCGDggWFxfL4XA0aXxAp+xDQkLkcDhUUFDga7MsSwUFBRoxYkSD\nY5KTk/36S9KWLVuUnJwsScrIyNDf/vY37d692/fp3r27FixYoPz8/EDKAwAAQBsU8Cn77OxsZWZm\nyuFwaNiwYXK5XKqqqlJmZqakuoDZs2dPLV26VJI0d+5cjR49Wg899JCuuuoqud1uFRUVae3atZKk\nTp06qVOnTn7rCAkJUUxMjPr27XuGmwcAAIDWLuBAmp6eLq/Xq8WLF8vj8SgxMVH5+fnq2rWrJKmi\nosLvsU3Jyclav369Fi5cqIULF6pv377atGmTBgwY0Og6bDZbMzYFAAAAbVGzbmrKyspSVlZWg8u2\nbt1ar23y5MmaPHlyk+c/1XWjAAAAOLfwLnsAAAAYRSAFAACAUQRSAAAAGEUgBQAAgFEEUgAAABhF\nIAUAAIBRBFIAAAAYRSAFAACAUQRSAAAAGEUgBQAAgFEEUgAAABhFIAUAAIBRBFIAAAAYRSAFAACA\nUQRSAAAAGEUgBQAAgFEEUgAAABhFIAUAAIBRBFIAAAAYRSAFAACAUQRSAAAAGEUgBQAAgFEEUgAA\nABhFIAUAAIBRBFIAAAAYRSAFAACAUQRSAAAAGEUgBQAAgFEEUgAAABhFIAUAAIBRBFIAAAAYRSAF\nAACAUQRSAAAAGEUgBQAAgFEEUgAAABhFIAUAAIBRBFIAAAAYRSAFAACAUQRSAAAAGEUgBQAAgFHN\nCqSrV69WXFycwsLCNHz4cO3cufOU/Tdu3Kj4+HiFhYUpISFBmzdv9i376quvdOedd2rw4MFq3769\nevTooRkzZuizzz5rTmkAAABoYwIOpBs2bND8+fOVk5OjkpISJSQkKC0tTV6vt8H+27dv13XXXaeZ\nM2dq165dmjhxoiZNmqTS0lJJUlVVlXbt2qVf/OIXKikp0QsvvKDy8nJNnDjxzLYMAAAAbULAgdTl\ncmnWrFnKyMhQ//79lZeXp/DwcK1bt67B/rm5uRo3bpyys7PVr18/LVmyRElJSVq1apUkqWPHjsrP\nz9fkyZPVt29fDRs2TKtWrVJRUZEqKirObOsAAADQ6gUUSGtqalRUVKSUlBRfm81mU2pqqgoLCxsc\nU1hYqNTUVL+2tLS0RvtL0uHDh2Wz2XT++ecHUh4AAADaoIACqdfrVW1traKjo/3ao6OjVVlZ2eCY\nysrKgPp/+eWXuuuuu3Tdddepffv2gZQHAACANqhV3WX/1VdfaerUqbLZbFqzZo3pcgAAAPAdCA6k\nc1RUlIKCguTxePzaPR6PYmJiGhwTExPTpP4nw+inn36qrVu3Nuno6Lx58xQZGenX5nQ65XQ6m7I5\nAAAAaAFut1tut9uv7ciRI00eH1AgDQkJkcPhUEFBgSZMmCBJsixLBQUFmjNnToNjkpOT6y3fsmWL\nkpOTfT+fDKN79+7Vtm3b1KlTpybV43K5lJSUFMgmAAAAoIU1dECwuLhYDoejSeMDCqSSlJ2drczM\nTDkcDg0bNkwul0tVVVXKzMyUJGVkZKhnz55aunSpJGnu3LkaPXq0HnroIV111VVyu90qKirS2rVr\nJdWF0cmTJ2vXrl16+eWXVVNT4zui2rlzZ4WEhARaIgAAANqQgANpenq6vF6vFi9eLI/Ho8TEROXn\n56tr166SpIqKCgUHfz1tcnKy1q9fr4ULF2rhwoXq27evNm3apAEDBkiSDh48qJdfflmSlJiYKKnu\nqKvNZtO2bdt0+eWXn/FGAgAAoPUKOJBKUlZWlrKyshpctnXr1nptkydP1uTJkxvsHxsbq9ra2uaU\nAQAAgHNAq7rLHgAAAN8/BFIAAAAYRSAFAACAUQRSAAAAGEUgBQAAgFEEUgAAABhFIAUAAIBRBFIA\nAAAYRSAFAACAUQRSAAAAGEUgBQAAgFEEUgAAABhFIAUAAIBRBFIAAAAYRSAFAACAUQRSAAAAGEUg\nBQAAgFEEUgAAABhFIAUAAIBRBFIAAAAYRSAFAACAUQRSAAAAGEUgBQAAgFEEUgAAABhFIAUAAIBR\nBFIAAAAYRSAFAACAUQRSAAAAGEUgBQAAgFEEUgAAABhFIAUAAIBRBFIAAAAYRSAFAACAUQRSAAAA\nGEUgBQAAgFEEUgAAABgVbLoAAABaI7emyS2nJKla7XSJynWXlilUX0qSnHLLqWdNlgicMwikAAA0\nwKlnCZzAd4RT9gAAADCKQAoAAACjCKQ4K9ymC0Cj2DetF/umdWP/tF7sm7avWYF09erViouLU1hY\nmIYPH66dO3eesv/GjRsVHx+vsLAwJSQkaPPmzfX6LF68WN27d1d4eLh+/OMf66OPPmpOaWgl+OXQ\nerFvWi/2TevG/mm92DdtX8CBdMOGDZo/f75ycnJUUlKihIQEpaWlyev1Nth/+/btuu666zRz5kzt\n2rVLEydO1KRJk1RaWurrs2LFCq1atUqPPfaY3nnnHUVERCgtLU3Hjx9v/pYBAACgTQg4kLpcLs2a\nNUsZGRnq37+/8vLyFB4ernXr1jXYPzc3V+PGjVN2drb69eunJUuWKCkpSatWrfL1+c1vfqNFixbp\n6quv1qBBg/TUU0/p0KFD+tOf/tT8LQMAAECbEFAgrampUVFRkVJSUnxtNptNqampKiwsbHBMYWGh\nUlNT/drS0tJ8/ffu3avKykq/OTt27Kgf/vCHjc4JAACAc0dAzyH1er2qra1VdHS0X3t0dLTKy8sb\nHFNZWdlg/8rKSkmSx+ORzWY7ZZ9vO3bsmCSprKwskPLbvJPb+6qk1r7lFZKeMV3EKXzyf/9tyT9D\nbWX/sG9ar9a+b6SW3z9tZd9IrX//8Hen9Tob+6YtOLm9J3PbqbTJB+Pv27dPkjR9+nSzhRiyyHQB\nTdQW9s7Z+DPUFvYP+6b1agv7Rmr5/dMW9o3UNvYPf3dar+9rbtm3b58uu+yyU/YJKJBGRUUpKChI\nHo/Hr93j8SgmJqbBMTExMafsHxMTI8uy5PF4/I6SejweDRkypME509LS9PTTT+uiiy5SWFhYIJsA\nAACA78CxY8e0b98+paWlnbZvQIE0JCREDodDBQUFmjBhgiTJsiwVFBRozpw5DY5JTk6ut3zLli1K\nTk6WJMXFxSkmJkYFBQUaPHiwJOno0aPasWOHbr311gbnjIqK0vXXXx9I6QAAAPiOne7I6ElB9913\n332BTNyxY0ctXrxYF154odq1a6d7771Xu3fv1uOPP66IiAhlZGRo586dvpuUevTooXvvvVcRERHq\n3LmzVq1apY0bN+p3v/udunbtKkmqra3VsmXLNGDAAB0/flxz5szRl19+qdzcXAUFBQW25QAAAGhT\nAr6GND09XV6vV4sXL5bH41FiYqLy8/N94bKiokLBwV9Pm5ycrPXr12vhwoVauHCh+vbtq02bNmnA\ngAG+PgsWLFBVVZVmzZqlw4cPa+TIkdq8ebPOO++8FthEAAAAtGY2y7Is00UAAADg+4t32QMAAMAo\nAikkSTfffLO6dOmioKAg/e1vfzttf7vdrhdffPE7qAxAQ5588kl17ty5Rebp1KlTC1TE74XmiouL\nU25u7hnNkZOT0+iTaZpi//79stvtTfr9fzbl5OQoKSmpReY5k++jqd544w0FBQXp6NGjZ31d5zoC\nKfTnP/9ZTz31lF599VV99tlnGjRokOmSWuQX9PfN//7v/yo4OFjXXHON6VK+EzfeeKOuvfbaeu1v\nvPGG7Ha7sX8gMjMzZbfblZWVVW/ZrbfeKrvdrptuuumM1zNt2jR9+OGHvp/P5B9gm812yuXHjh3T\n3XffrYsvvlhhYWG64IILNGbMGL300ku+PnFxcWrOFWBjxoxRdnZ2wONa0kUXXSS73a4//vGP9ZYN\nHDhQdrtdTz311Flb/7vvvqubb775jOc53X48lV69eqmysvKUv/89Ho9uu+029enTR6GhoYqNjdWE\nCRO0devWZq/32+644w4VFBT4fm7s73lT/P/2zjwqiiv749/qbqCbphFEZBEUCKLIIrIpYhARtwkq\nOhKDuLBNxGQA2w00ySiKivvCGMUddFBGJJJoIgmJBNBBFEKryOJERGNYouJCiDQN9/cHQ/0sQDSL\nYpL6nMM5VNVb73uv3u373n31a+TxvLi7u6Oqqgra2tovPK8/OrxCyoP//ve/MDIywtChQ9G7d28I\nBHy3+D2yb98+REREIDs7+6lfOfuz8KInIpVK1WXeffv2xdGjR9HY2Mjeb2xsxJEjR9CvX7/fJH8N\nDQ306tWLU5YXVe+5c+fixIkT2LFjB8rKypCRkYFp06bh7t27LyS/l01bmx04cIBz//z586ipqYGW\nltavSr+lpaVTZb2pqQkAoKenB7FY/Kvy+LUwDNPl+7+yshKOjo7IysrCpk2bcOXKFZw+fRqjRo3C\n3//+99+sHJqamr+Zxf5lIBKJ0Lt37+4uxh8D4vlTExgYSAzDkEAgIIZhyNzcnDw9PSkiIoKWLFlC\nPXv2JENDQ1qxYgUnHsMwlJ6eTkRE06ZNo/DwcPZZZGQkMQxDZWVlRESkVCpJKpXSl19+SUREjx49\nohkzZpBUKiVjY2PasmULeXp6klwuJyIiT09PTpkEAsHLEMXvmvr6epLJZFReXk7+/v60du1azvP0\n9HTq378/icVi8vLyosTERGIYhh48eMCGycnJoddff50kEgn17duXIiIi6Mcff3zZVXluAgMDacqU\nKR3uZ2VlkUAgYOuWmppKNjY2pKGhQWZmZrRp0yZO+Cf7chs6OjqUmJhIREQ3btwghmEoJSWFRo4c\nSRKJhBITE6myspJ8fHxIV1eXpFIp2dra0meffUaBgYHk6+tL9vb2lJyczKYZFhZGQqGQhg0bRtra\n2iQWi2ncuHF06NAhGjFiBOno6JCenh45OzuTjY0NicVieu2110gul3PyB0Bz5swhBwcHAkAxMTF0\n8ODBDmNGJpPR8ePHafPmzWRnZ0cCgYCEQiG98847VF9fTzk5OaShoUF79uwhXV1dun//PoWEhJC+\nvj5pa2vT6NGjSaFQsPJISkqiEydOkKOjI1u2mJgYUqlU7JgFwBmzixcvJolEQr179yZNTU2ys7Oj\nI0eOcNqwfbkrKyvpwIEDpKOjw2mTEydOEMMw7LVCoaBRo0aRTCYjbW1tcnZ2poKCgp/dj4iIzMzM\naNmyZSSRSOi7775j77/99tsUGRlJurq6bH8gIlamUqmUTE1NWZm2cfDgQdLR0aGPP/6YBg0aRGpq\nalRZWcn2jdWrV5OxsTFZWFiw+W/bto2N31VbtLF27VoyMDAgbW1tCgkJoejoaBoyZEiX9WxpaaF1\n69aRpaUlaWhoUL9+/WjNmjVE9P/9XKFQUEtLC5mYmNCuXbvYuBMmTCBDQ0NiGIZu3rzJSbdtrG3Y\nsIH09PTYviCTyWjlypVsOAAkkUjI0dGRGIYhhmHIwcGBbt26xYZ55513SEtLiyQSCUkkEgLA6R8O\nDg4kl8spKiqKrKysSFNTkzQ1NWnw4MGkUqmIiGjHjh3Us2dPYhiGDAwMyM/PjyODNWvWkLm5OYnF\nYtLV1SWxWNzpXEREdOjQIXJ2diaZTEaGhoY0Y8YMqq2tZZ9nZWVx3qVtbZ+RkUHW1takpaVF48eP\np+rq6i7bhoeIN4X9ydm+fTtWrlwJExMT1NTU4MKFCwCApKQkaGlpIT8/H+vXr8fKlSs5yyhPMnLk\nSGRlZbHX2dnZ0NfXZ+/l5+dDpVJh+PDhAAC5XI7//Oc/OHnyJL744gvk5OSgsLCQjZ+WlgYTExOs\nWok59FwAABTfSURBVLUK1dXVqKqqejGV/wORkpICa2tr9O/fHwEBAdi3bx/77MaNG/Dz88PUqVNx\n6dIlzJ07F++99x7Hmvbtt99iwoQJ8PPzw5UrV5CSkoKzZ88iPDy8O6rzq6AnLFEFBQWYPn06ZsyY\ngStXriAmJgYffPDBL1p+Xbp0KebPn4+SkhKMHTsW7777LpqampCbm4srV65g3bp1rCWNYRgEBwdj\n//79bPycnBwAQFlZGTw8PHDu3Dncv38fq1evxsKFC1FYWIiNGzeiqKgIDx8+RGlpKRISEpCamsrJ\nHwA+/fRTODs7o0ePHggODsb06dOxcOFC2NjYoKamBlVVVRg9ejSysrIgFAqxdu1aiEQiSCQSZGRk\nICoqCtnZ2XB1dYWamhoAsBbPjIwMFBYWwtHREd7e3rh//z4MDQ2RmJiI2bNnQy6Xs2VLTEzEmjVr\n2DELAAcPHkRVVRXCw8ORkpICuVyOzMxMFBcXY+7cuZg9ezYuXrwIANi2bRvc3Nzwt7/9jR3rpqam\nYBimU2vvk/cCAgJgamqKgoICFBYWIjo6mq3LL8HAwADjxo1DYmIigNZtCikpKQgODu5g3RQKhYiP\nj8fVq1eRlJSEM2fOICoqihOmoaEB69evx759+1BcXMwejfjll1+ivLwcmZmZOHnyZKdl6aotAODf\n//43YmJiEBcXh4sXL8LIyAgffvjhM+sYHR2N9evXY/ny5SgpKUFycjLnC4lt8mUYBv7+/khOTgYA\n1NXVISMjA1ZWVnj99ddhamrKSbdtuTo3NxcikQipqalISUmBnp4ecnNzOWF/+ukn1NXV4dixYwgJ\nCUFRURF8fX1ZmR06dAgikQgFBQU4fvw4pFIpjI2N2X7dlpe2tjaSkpJQUlICe3t7lJeXY8uWLbh4\n8SIiIyPh5eUFGxsbZGRkwMPDg81/zZo1OHz4MHbv3o3JkydDJBKhpaUFcXFxHeYioHU1IjY2Fpcu\nXUJ6ejoqKysRFBTECdO+rzY0NGDTpk3417/+hZycHNy8eROLFi16Zvv86elujZin+9m6dSuZm5uz\n156enuTh4cEJ4+rqSkuXLmWvn7QqXb58mYRCId25c4fq6upIQ0ODVq9eTf7+/kREtHr1ahoxYgQR\ntVpH1dXVKS0tjU3rwYMHJJVKOb9K21sMeLrG3d2d4uPjiYhIpVJR79696euvvyYioqioKLK3t+eE\nf//99zlWxNDQUAoLC+OEycnJIaFQSI2NjS+hBj+fwMBAEolEpKWlxfmTSCRs3QICAmjcuHGceEuW\nLCFbW1v2+nktpG3ybcPe3p5j/XmyXFOmTKEffviBxGIx3bx5k27cuEHq6urEMAx5enpSUFAQERGV\nlpYSwzB04cIFIiLy9vamDz74gBiGoeLiYiJqHZ8A2PwZhqGFCxfSwYMHSVdXl813xYoVHAtZfHw8\n2dnZEVGrhdzNzY2mTJlCc+fOJX19fRozZgy9//77dPDgQZLJZKSjo0NKpZJTF0tLS9qzZw9lZ2eT\nWCwmoVBILi4uJJfL6ezZs3T48GEyNjYmotYxC4BSU1NpxowZZGNjQ1VVVR3k4+PjQ4sXL2av21uk\niKhD3YhaLaRPrpZoa2tTUlJSh/R/CW3vm/T0dLK0tCQiosTERHJyciIibn/ojNTUVNLX1+eUXyAQ\n0OXLlznhAgMDycjIiJqamjrNn6h13HXVFkREw4cP56xKERENGzasSwvpo0ePSCwW0/79+zt9/qSF\nlIioqKiIhEIh3bp1i/Lz84lhGNLT06Pdu3c/NY+IiAjy9vZmr9vLBf+zoLf1dyKiwYMHEwC6cOEC\n7d69myQSCTk4OLDPvb29CQBrleysv/j6+pKLiwu5uLhQWloa6ejo0LJlyzrIo7GxkaRSKeXl5XHm\notDQUAoICOh0LmrPhQsXSCAQsKtH7Vdk2tq+oqKCjfPhhx+SkZHRU9PkaeVnH4zP8+eg7TOubRgZ\nGaG2trbTsLa2ttDV1cXXX38NNTU1ODo6wsfHBzt27ADQ6mTi6ekJALh+/TpUKhVcXFzY+Nra2hgw\nYMCLqcifgLKyMuTn5+PEiRMAWq03b775Jvbt2wcPDw+Ul5dz5A0Arq6unGuFQoHLly/j8OHD7D36\nn1WooqLilW0fLy8v7Nq1i2PBysvLw6xZswAAJSUlrPWlDXd3d2zbtg1E9LP2XDo5OXGuIyIiMG/e\nPGRkZMDb2xt//etfYWdnBwDIzMyEubk5mpqaMHr0aAQEBGDw4MFQKBTQ0dFh0xgwYABkMhnCw8NR\nW1uLiooKZGZmsvmJRCJ2n+GTziZOTk5QKpXsta2tLa5duwaVSoU33ngDp06dwsiRIzF//nykpaVB\nLpejrq4OhYWFUKlUICKcPXsW0dHRuHXrFpqbm9HQ0AAdHR00NDSw6QoEAly/fh2hoaHQ0tLCo0eP\noFAoUFhYiC1btkAkEoGI8PjxYzaOXC6HWCxGXl4edHR0sGrVKhw7dgy3b9+GUqmEUqmEVCp9brk/\njQULFiAkJARJSUnw9vaGn58fLCwsflWab7zxBsLCwpCdnY0DBw4gJCSk03CZmZmIi4tDaWkpHj58\nCJVKhcbGRjx+/JjdC6qurt6pg5CdnR3n4zHtuXTpEh49etThBIXHjx/j+vXrAFr79bx58zjP3dzc\n2FWp3NxcTJgwAUCr9S4hIQGWlpZQKpXw8vJ6LlkMHjwYAwcORHJyMvv+fvDgAaZNmwYAmDdvHvu+\nYBgGDx8+hLW1NXbu3Ak1NTUwDAOBQICmpiaOXIRCIZydndl8vLy8UFxcjJKSEpSWlsLQ0JAzLtss\nuGVlZayVGWhdFYqPj8e3336LO3fugIjQq1cvjBkzBv369cPWrVshFotZK7Cvry9aWlrQ0NAAT09P\nMAwDpVKJmTNnorm5GUOGDOl0LiooKEBMTAwUCgXq6urQ0tICALh58yYGDhzYqew0NTVhZmbGXnc1\nf/L8P/ySPU+ntF/6YhiGHYid4eHhgTNnzrDKp52dHRobG1FcXIxz585h5MiRL7rIf1r27duH5uZm\nGBkZQU1NDWpqati1axeOHz/+3J7m9fX1mDt3Li5dugSFQgGFQoFLly6hvLwcr7322guuwS9HKpXC\n3NwcFhYW7F+fPn1+VhoMw3RYkm1TAtvn9SQhISGoqKjA7NmzceXKFbi4uLA/wtzd3aFQKJCQkIDG\nxkYkJSVxlg2fpL6+Hg0NDdi7dy/U1dWxaNEiMAyDbdu2QaFQ4IsvvgDDMBwFpX1ZPvvsM8ybNw/W\n1tbYu3cvgFbFp0ePHvD390djYyNiY2Nx7NgxdluBSqWCm5sbgNYfH8bGxlAoFDhz5gz7V1BQwC41\n/vjjj4iNjUVJSQnKy8uxYMECCAQCXL16leOQM3bsWNy+fRunT5/G+vXrER8fj6VLlyIrKwsKhQJj\nx47lKNOdIRAIntkmy5cvx9WrV+Hj44OvvvoKNjY2SE9P7zLdZyEUCjFz5kwsX74c+fn5CAgI6BCm\nsrISEydOhIODA9LS0lBYWMi2+5P1kkgknebxLGW8vr4exsbGnLGoUChQVlb23Mu+zs7ObLyioiJM\nmjTpqeXpioCAACQnJ6N///4AACsrK9bhaNWqVZw8KisrIZfLERYWhpUrV2Lq1KkQCARoaWl5Znv/\nHAQCAb7//nvMnDkTPj4+OHXqFEaMGAF7e3solUpoaWmhsLAQfn5+UFNTw/LlyxEWFobc3FxWgf74\n44+RmpoKgUCAzz//HFevXmW3xjxJQ0MDxo8fDx0dHSQnJ+PixYv46KOPAKDLOnU2f7bvzzwd4RVS\nnt+Etn2kbQopwzB4/fXXsWHDBiiVSri7uwMALCwsIBKJ2L2qQOuv7iePrwFarQvNzc0vtQ6/R5qb\nm3Ho0CFs3ryZM3kpFAoYGxvjyJEjGDBgALtnr438/HzOtaOjI65evdpBuWtrr98r1tbWOHv2LOde\nbm4urKysWCuMvr4+Z5/ytWvXOFZC4One63369MHbb7+N1NRULFiwAHv27AHQqoxYWFggKCgIzc3N\nUKlUsLW1hUqlYvcBAq1e3C0tLYiIiMCoUaPg7OyM0tJSAIChoSEsLCzQt2/fZ1pyTU1NYWBgAKFQ\nCCMjI/a+paUlVCoVHj58iNDQUPj4+ECpVIKIMGTIEFZJEYlEqK6uhrq6Ojw9Pdk/BwcHVhF2dHRE\nWVkZ2y/c3d3R3NwMY2NjAGA/9Txp0iQkJycjNDQUqampmDx5Mvz9/WFnZwdzc/PnGuv6+vp49OgR\nfvrpJ/beN99806HelpaWiIyMREZGBqZMmdLBS/6XEBwcjOzsbPj6+nZ6lE9BQQGICBs3boSrqyss\nLS1x+/btX51vG46OjqiuroZQKOwwFtvawtraGufPn+fEy8vLY/8Xi8WceFKpFP3794dYLH6qLwDQ\nsZ+37b2uqKiAUChETU0N2ya9evVi09fT02Plsn37dixduhRHjx7Fm2++CQCcPt/c3Mx5H505cwYq\nlQqDBg2CtbU1qqurOcaPu3fvgmEY1nKpr6+PsrIymJmZITo6Gg4ODqioqEB9fT0bRyAQwNzcnP2R\nVVlZiYqKCowZMwZisRi1tbXw8PCASCRCbW0t+0O2/VxUWlqKe/fuYe3atXB3d4eVlRVqamqe0YI8\nv5Tf70zD80rh6ekJuVwODQ0NjBgxgr23aNEiuLq6shOflpYW5syZg0WLFkFXVxf6+vpYsWIFhEIh\n52VoZmaG7OxsTJ8+HRoaGtDT0+uWer3qfPLJJ7h//z6Cg4Mhk8k4z6ZOnYr9+/cjJSUFmzdvRnR0\nNEJCQvDNN9+wjhttMo+KioKbmxvCw8MRGhoKqVSK4uJiZGZmIj4+/qXX67di4cKFcHV1RWxsLKZP\nn45z585hx44d2LVrFxvGy8sL//znPzFs2DCoVCpER0ezylUbnVk35HI5JkyYACsrK9y7dw9nzpzB\noEGDOGEEAgGrYB4/fhwikQgKhQJaWlooKCjA/PnzIRKJkJ2dDU9PT0yaNAnR0dEAWpcES0tLkZmZ\n2eXqRBtmZmaoqKiAQqGAiYkJZDIZPD09kZ+fDwMDA9TU1LAOJkTEjlOgVSF1c3ODr68v1q1bBysr\nK9y+fRuffvoppk6dioULF8LZ2Rk7duyATCaDlZUVNmzYgEGDBmHdunVYtWoVzMzMUF5ejrt372LS\npEk4dOgQ3nrrLdy8eRPBwcHQ0dHBli1bUFNTAxsbG065z58/j8rKSmhpaUFPTw9Dhw6FpqYmli5d\nioiICOTl5bF9Fmhdvl68eDGmTZsGc3Nz3Lp1CxcuXICfn98z5fQsBg4ciDt37kBTU7PT55aWlmhq\nasL27dsxceJE5ObmIiEh4Vfn24a3t3eXbeHo6IjIyEgEBQXByckJ7u7uOHz4MIqLi7tczdDQ0EBU\nVBSWLFkCNTU1uLu744cffkBxcTF7Lm77ft6vXz+4ubkhJCQEEokEGhoacHV1RUxMDOzt7aFSqfD5\n558jISEBR48ehVKpRGBgIGbNmoWCggLW6vjkNhUiwltvvYW4uDhkZGSgqKgIjo6OcHJywqBBg7Bg\nwQJUVlaiuLgYtbW1yM/Ph1QqRV1dHQQCAUaOHIn09HSoVCps2bIF+fn5qKmpARFBU1MTp06dwvXr\n11FdXQ2lUonExEQQEQYMGAAtLS0sWrQIcrkczc3NmDJlCiIiIpCVlQVDQ0MUFhZy5qK+fftCXV0d\n27dvR1hYGC5fvozY2NgOsuWtn78RL3fLKs+rSHunplGjRnW6abzNEYOISCAQcBxBWlpaSE9Pj4YP\nH87eKyoqIoFAQO+99x4nrfr6epo5cyZpaWmRsbExbd26lYYOHUrLli1jw+Tl5ZGDgwOJxWL+2Kcu\nmDhxIk2cOLHTZ/n5+axjxSeffEJWVlYkkUjIy8uLEhISSCAQcByWLl68SOPGjSNtbW2SyWTk4ODQ\n4fioV4nnPfYpLS2NbG1t2WOfNm/ezAn//fff0/jx40kmk9GAAQPo9OnTnGN+bty4QQKBoMOxO+Hh\n4dS/f3+SSCRkYGBAgYGBdO/evaeWq81RZ+jQoSSTyUgikdC4cePo6NGjZGNjwzpzbNq0iQCQhoYG\n6ejo0JAhQzjOJm1jr73jT2NjI/n5+ZGuri4JBAJKTEykoqIiYhiGZDIZSaVSmjBhAs2cOZMA0Ecf\nfcQpV319PUVGRpKJiQl7JNCsWbPou+++o7i4OHJ3d6cePXqwR/BoaGiQs7Mz7d27l4haxywAUldX\nZ8fs/v37SSgUkqamJhkaGtI//vGPDvIpLy+n4cOHk6amJnvsE1GrI5aVlRVJpVKaNGkS7d27l01X\nqVSSv78/9evXj8RiMZmYmFBkZOQvdsAzNzfv0omy/bFPW7dupT59+rAyPXz4cAfHlvZOWURP77Pt\n8++qLdpYu3Yt9e7dm7S1tSkoKOi5jn0iIvbIo7bxEBcXR0RP7+c7d+4kgUBAQUFBVF1dTeHh4eyR\nSaampuTr60vZ2dlERDR9+nRSU1MjACQSicjW1pYjFwCkqalJDg4OXR77JJPJSFNTk3r16kVz5swh\nb29vkslkJBAI6KuvvqJ3332XdV7U0NAgExMTGjp0KOnq6lJubi55enqyzx0cHCg1NZVTp+3bt5O1\ntTWpq6uzznq9evXqdC46evQoWVhYkEQiIXd3dzp58iRHTp05NT3LIY+ncxgiXrXn6V4aGhrQp08f\nbN68ucNxGjwvhtWrV2P37t2orKzs7qL8aUhMTIRcLse9e/e6uyg8PN0CwzCsc9yrCD8XdS/8kj3P\nS6eoqAilpaVwdXXF/fv3sXLlSjAMg8mTJ3d30f6w7Ny5Ey4uLuy5gBs3bkRERER3F4uHh4en2+Dn\nolcLXiHl6RY2btyI8vJyqKurw8nJCbm5uR2OOeH57bh27RpiY2NRV1eHvn37YvHixexeRR6ePxJ/\n+ctf2I8QPAnDMFi2bBnf77uRl/Ft+Z8LPxe9OvBL9jw8PDw8fxiqqqo43vlP0rNnT46DDQ8Pz6sD\nr5Dy8PDw8PDw8PB0K/w5pDw8PDw8PDw8PN0Kr5Dy8PDw8PDw8PB0K7xCysPDw8PDw8PD063wCikP\nDw8PDw8PD0+3wiukPDw8PDw8PDw83QqvkPLw8PDw8PDw8HQrvELKw8PDw8PDw8PTrfwfCXb54eak\n9cMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xcc4f1d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the top 5 feature importances of the forest\n",
    "plt.figure(num=None, figsize=(8, 6), dpi=80, facecolor='w', edgecolor='k')\n",
    "plt.title(\"Feature importances\")\n",
    "plt.bar(range(5), importances[indices][:5], \n",
    "       color=\"r\",  yerr=std[indices][:5], align=\"center\")\n",
    "plt.xticks(range(5), best_features)\n",
    "plt.xlim([-1, 5])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Decision Tree accuracy and time elapsed caculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DecisionTree\n",
      "Acurracy:  0.827351844177\n",
      "time elapsed:  0.605034589767456\n"
     ]
    }
   ],
   "source": [
    "t0=time()\n",
    "print (\"DecisionTree\")\n",
    "\n",
    "dt = DecisionTreeClassifier(min_samples_split=20,random_state=99)\n",
    "# dt = DecisionTreeClassifier(min_samples_split=20,max_depth=5,random_state=99)\n",
    "\n",
    "clf_dt=dt.fit(X_train,y_train)\n",
    "\n",
    "print (\"Acurracy: \", clf_dt.score(X_test,y_test))\n",
    "t1=time()\n",
    "print (\"time elapsed: \", t1-t0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## cross validation for DT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cross result========\n",
      "[ 0.825  0.821  0.829  0.833  0.831]\n",
      "0.827962687655\n",
      "time elapsed:  2.6721527576446533\n"
     ]
    }
   ],
   "source": [
    "tt0=time()\n",
    "print (\"cross result========\")\n",
    "scores = cross_validation.cross_val_score(dt, X, y, cv=5)\n",
    "print (scores)\n",
    "print (scores.mean())\n",
    "tt1=time()\n",
    "print (\"time elapsed: \", tt1-tt0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Tuning our hyperparameters using GridSearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 27 candidates, totalling 81 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  34 tasks      | elapsed:    8.0s\n",
      "[Parallel(n_jobs=-1)]: Done  81 out of  81 | elapsed:   11.4s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best score: 0.638\n",
      "Best parameters set:\n",
      "\tclf__max_depth: 25\n",
      "\tclf__min_samples_leaf: 3\n",
      "\tclf__min_samples_split: 10\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.88      0.90      0.89      9043\n",
      "          1       0.66      0.62      0.64      3022\n",
      "\n",
      "avg / total       0.82      0.83      0.82     12065\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('clf', DecisionTreeClassifier(criterion='entropy'))\n",
    "])\n",
    "\n",
    "parameters = {\n",
    "    'clf__max_depth': (5, 25 , 50),\n",
    "    'clf__min_samples_split': (2, 5, 10),\n",
    "    'clf__min_samples_leaf': (1, 2, 3)\n",
    "}\n",
    "\n",
    "grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=1, scoring='f1')\n",
    "grid_search.fit(X_train, y_train)\n",
    "\n",
    "print ('Best score: %0.3f' % grid_search.best_score_)\n",
    "print ('Best parameters set:')\n",
    "\n",
    "best_parameters = grid_search.best_estimator_.get_params()\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print ('\\t%s: %r' % (param_name, best_parameters[param_name]))\n",
    "\n",
    "predictions = grid_search.predict(X_test)\n",
    "\n",
    "print (classification_report(y_test, predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Random Forest accuracy and time elapsed caculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RandomForest\n",
      "Acurracy:  0.846249481973\n",
      "time elapsed:  1.659095048904419\n"
     ]
    }
   ],
   "source": [
    "t2=time()\n",
    "print (\"RandomForest\")\n",
    "rf = RandomForestClassifier(n_estimators=100,n_jobs=-1)\n",
    "clf_rf = rf.fit(X_train,y_train)\n",
    "print (\"Acurracy: \", clf_rf.score(X_test,y_test))\n",
    "t3=time()\n",
    "print (\"time elapsed: \", t3-t2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## cross validation for RF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cross result========\n",
      "[ 0.851  0.843  0.849  0.858  0.855]\n",
      "0.850938729061\n",
      "time elapsed:  9.8535635471344\n"
     ]
    }
   ],
   "source": [
    "tt2=time()\n",
    "print (\"cross result========\")\n",
    "scores = cross_validation.cross_val_score(rf, X, y, cv=5)\n",
    "print (scores)\n",
    "print (scores.mean())\n",
    "tt3=time()\n",
    "print (\"time elapsed: \", tt3-tt2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Receiver Operating Characteristic (ROC) curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7706380677128094"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(y_test,rf.predict(X_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiAAAAGHCAYAAACJeOnXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzs3XecVNX5x/HPs5Slg7qAFLFGBTsbSFSioMTeogiiqNFY\nEGNB/anYSNBYYpBEBcHYAyxF1FiS2FCwIbgrigiKiiAgLIgiUhZhz++Pcwdmh+07M3fK9/163dfu\n3jn33mfuzs48e+45zzXnHCIiIiLJlBN2ACIiIpJ9lICIiIhI0ikBERERkaRTAiIiIiJJpwRERERE\nkk4JiIiIiCSdEhARERFJOiUgIiIiknRKQERERCTplIBIVjOzXc2s1MzOCzuWbBOc99vCjqMqZvam\nmX0cdhypxsyODH6HR8Rpf78P9tcpHvuT1KcERBLGzM4P3lAiy89mtsTMHjez9mHHFyUl70dgZruY\n2WgzW2hmG81shZk9a2aHhR1bdZnZ8WY2tIKHHSGeezNrbmZDzWy2ma01s/VmNsfM7jazdjFxpiUz\nG2JmpybwEDU+N5XEFOrrQZLPdC8YSRQzOx94DLgV+BpoBPwauABYCOzvnNsUWoABM2sI/OxS6I/B\nzA4H/gOUAv8E5gE7A78H9gKudM6NDC3AajKzB4BBzrl65TzWENjsnCsNIa49gNeAjsBk4G1gE3Ag\n0B9Y7ZzbN2j7BrCTc+7AZMdZV2a2FpjsnLswQftvWNO/4YpiMjMDGqTCe4IkR/2wA5Cs8D/nXFHw\n/WNm9h1wPXAK8HR4YXlhvOGZWRPn3PoKHmuFPy/rgMOcc19HPXYf8ArwdzMrdM7NSEa8UcevMO6K\nNqnogbA+aMysHvAM0Bo40jn3XszjNwM3hBBTjnPu52QetzbMLBfY5Ly4/Q6DfwCUfGQRXYKRMLyF\n/2DaM/aBoMt+upn9ZGY/mtmLZtalnHb7mNkkMysOus7nm9kdMW3am9ljZrY8uITxiZldENOmzBgQ\nM7s2+HmXco55l5mVmFnLqHW/MrP/mdkPZrYuGC9wWMx2fwr22dnMxpvZ6uAcVGQg0Aa4Ljr5AHDO\nlQDnBz9uHT8RdbnrN2Y2xsxWmdkaM3sySGhin0uV59nMngguTexhZv8xsx+BscFjPYLzvyg4t4vN\n7D4zaxS1/ePAoOD7yGW4LVGPlxkDEnWe9gyO/X1wXh+L3m/QtpGZ3W9mK4P4nwt+39UZV9IH39Nx\nR2zyEZzjn5xzt5Zzzjqb2RvB73mJmf1fzOMNzGyYmX0QxP1TcI57xrSLvOauMbOrzOwLYCPQubr7\nCPZjwfYfm9mG4G/hv2bWNXJ+gSZAZGxFqZk9FrV9df4+IuM8+pnZHWa2BJ8YN7dyxoCY2V5mNsXM\nvg1i+sbMCsyseVUxWQVjQILX6rTg97zGzGaaWf/tf62SbtQDImHYPfj6ffRKMzsXeAL4H76HpAlw\nGfCWmR3inFsctDsQ/wFeAowBFuGTmZOAW4I2bYD3gS3A/cAq4HjgUTNr7py7v4LYJgF/BfoCw2Me\nOxPfm7MmOMZR+MskHwB/wl8uuQCYamY9nHMfBNtFLu1MBj4HhlBJz0DwPDYG7bfjnPvazN4GjjKz\n3CApiXgQf16HAvvgE4BOQK9Ig+qe5yDu+sDL+PN9LRDp/TgTaAyMAr4DugNXAB2AfkGb0UB7oDdw\nThXPOXI88L+Dr4Abga7ARcAK/HmLeBKfSDyF/z0fCbxE9cYQnBK0G1uNthE7Av/F95xMCI59t5l9\n7Jx7OWjTArgQKAAeBpoDfwD+Z2bdnXOxA1kvBHLxr+ESYHUN9/EYPhl9CX+Zrj7wG/xlziJgAPAo\n/vw8HGzzJdTq7+PWIMZ7g5gjPRVbz7eZNcD3zjUI9rkc/3o4CWgFrK0sJsoZA2Jmvw/afwLcCfwA\nHAIcG5wjSWfOOS1aErLg3xy34D/8dsK/GZ2B/zBZB7SPatsU/wb8UMw+WuM/UEdHrZuGfyPqUMmx\nHwGWAK1i1o8PjpMb/LwrPnE4L6rNO8DMmO26Be3Ojlr3GfBSTLtc/Bvq/6LWDQ22/Vc1z9tqoKiK\nNn8Pzu1+Uee6FP/GXi+q3XVBu5NqcZ4fD7a9o5zj55az7gZgM9Axat0DwJYKnkMpcFs55+nhmHZT\ngOKonw8J2v0tpt1jQby3lXe8qHaF+DEe1X0dvxHsN/p33wBYBkyKWmdA/ZhtWwDfAv+MWhd5zX0P\n7BjTvrr76BXs474qYl8LPFaHv48jg+MsABrGtD0yOC9HBD8fFLT9XS1jirxfdIp63mvwf48NK9un\nlvRcdAlGEs2A14GVwDf4/+p/Ak5xzi2LavdboCUwwcx2iiz4/4jeJ/gP3szy8P/lPeqcW1rJcU8H\nXgDqxezvleA4XSvZdiKQb2a7R63rh++VeD6I42DgF0BBzP6bB883dmqiw/+nWx3N8W/SlYk83iJm\n/cPOuS1RPz+Ef1M/Ifj5GKpxnmOMjl3honpdzKxJsI/38Jd1D6ki9sqUd57eAnYys2bBz8cF7R6K\nafcAVfeygD9nVZ3fWD8558ZvDdKP1ZgJ7BG1zjnnNsPWyyM7AA3xPWTlvd6eds6tjl5Rg32cgf+w\nH1bD5xFR07+PJ1zV4z3WBF+PM7PGtYwr2m+BZsDd1Ti2pCFdgpFEc/jLAAvwb2wX4j+cY99QfoH/\n8Hijgn1E3twib/hzKzqgmbXGd/leAlxawf7aVBLzZOA+fNJxd7CuD/Bf59xPUfGCvwRQnlIza+mC\nyzWBhZUcM9pafBJSmcjj0R+kDvgiupFzbp2ZfQvsFqzai8rP848x6zY755bENjQ/RuZ24GRgh5h9\ntIxtX0OLY36OXKrbAZ+8RnoQYs/nF1TPj2y7DFhd252DIK4DoleYn/l1DbAvvpck4qtytv+6vANV\ncx97AMuccz9UFXg5+6/N30e5sZbZyF8aHI6PfYCZvYVP2Mc652JfV9URGSNW4d+6pDclIJIMs1ww\nC8bM/o2f8jjezPZx22ZU5ODf+AbgL9HE2lyD40V69sbixwqUp8LCUs65b4M3z7746/yH4sdRRA86\njBzjWuCjCnb1U8zPGyoLOso84GAza+AqnhVxEPAzPrGriZqe55LYBmaWg5/C2gq4C38pah3+EtuT\n1H1w+5YK1lend6M65uPPb4cqetFqFJOZDcBftnoGP46oONjuJqJ6SqJs93qoxT5qozZ/H9V67Trn\n/s/MngBOxfe23Q/caGa/junxFFECIsnlnCs1syH4/8D/iH+TBT9uwoCVzrmplewi8l/g/pW0WYnv\nGahXxb4qMxEYaWa/wPeErANejHo8MnBubR2OUZEX8QMJz8Rfky/DzHYDegCvuLIDUA3fMzMtqm1T\noB1+oGIk7uqc58ocEBznXOfcuKhj9S6nbSJqqyzCf4juzrbfA2zrlarKC/haHwOAe+IY1xnAl865\nPtErzawml0mqu48vgWPMrFUVvSDlnf94/H1UfEDn5uJ7Le40s18D7+JndkVmJ1X3NRF5re5P+T1I\nkuY0BkSSzjk3DX/9/GrzxajAz7T4EbjJzLZLjIOxHzjnVgHTgQutnKmyQZtS/MDFM8xsv4r2VYUp\nBINO8ZdfXnTORf8XWIh/g7wu+JCvzTEqMgb/IXFvzDiUSA2Gx4Mfy/tguyTm/A0C6uFn60A1z3MV\nIr0Bse8fV7P9h8u6YL+xY1Xq4mX8B9OgmPVXlHP88jwNzAFuDj4gyzBfIfWO7Ter0na9JGb2K+DQ\nBOxjCv78V1RlNmIdvqdqqzj9fWwnOG+xBefm4v+OciuLqQKv4BOlIcHrXjKMekAk0SrqNr8XP9bi\n9/iBk2vN7DL8mIoiM5uA/xDuBJyIv2xzZbDtlfiBiUVm9jB+LMDuwAnOucgAyBuBnsD7ZvZP4FP8\nVMp84Cig0jdZ59xK8xUwr8EPhJsY87gzs4vwH+xzzde8WIq/DNELP2alViWwnXOrzawPviekyMwe\nCeJvh58psCe+Eur75WzeEHjdzCbhxxBcBrzlnHsx2HdNznNF5uOTr+Fm1hGf0JxB+R8qhfjXwANm\n9jJ+RszEctpVm3OuyMym4BPYPGAGfkZGpAek0iTEObfZzE4HXgWmB+fqHfwlrf3wSedqgindNfAi\ncLqZPYfvcdoDP8ZiLv41FLd9OOfeNLN/AVea2d74KdU5+AHaU51zo4KmhUBvMxuMn7Wz0Dk3kzr+\nfUSJ/vs+CnjQzCLTzesD5+Ev602JaldRTGUEr9XB+CnGs8xsPH7czUFAY+fcBbHbSJoJexqOlsxd\n2Datrms5jxl+/MLnBLcECNYfgf9QX43/T+lzfB2AQ2K274z/T/a7oN2nwNCYNnn4a9Bf42ewLMX/\nV3VhVJtdgxjPKyfGPwSPfU8F0wDxBa0m46/Vr8d3FRcAPaPaDA32s2NF56qCfXfCz0BZGMS/Aj82\n4NBKznUP/OyQVfgk6EliplpW9zzje1rWVBDbPvieiDVBXA/hu8rLnEv8h+Lf8TUhNhM1JTdoe2tV\n54mY6ZnBukbB73ZlEMPT+AG2pcD/VfP8tgiOORv/n/Y6/Hie24E2Ue3eAD4qZ/vH8ZdLotfdELwG\n1uNnrhwf2y7qNTe4griq3EfU39A1+ORkQ3COXwQOjmqzdxD/T8ExH4t6rDp/H5GptqeXE2fsNNzd\n8MnC58G5XIkfK9QzZrtyYyrv9xysPxH/D8dP+L/F94C+8Xyv0hLOonvBiGQA23bfnW5uW9n7rBJM\njS4CznHOqUiVSIpLiTEg5stHP29mS82X4j2lGtv0NLNC8yWEPw/egEUkC1hMafbA1fj/oKcnORwR\nqYVUGQPSFN8N+ii+i7lSwSyAF/FloM/Gl3p+xMyWOedeTVyYIiktXtNU08H1ZpaP78rfjC+0diww\nxlV/aq2IhCglEhDn3P/wg6git2SuymXAV86564OfPzOzHsBg/MAykWyUTddT38X/43ELfnDmYvx4\njjvDDEpEqi/lxoCYv1viac655ytpMw0odM5dE7Xu98AI59wOFW0nIiIiqSElxoDUws5sX8VxBdBC\n88VFRERSX0pcgkmG4EZLx7JtypmIiIhUTyP8VOuXnXPfxWOH6ZqALAfaxqxrC/zoypamjnYsMK6C\nx0RERKRq51DOLSJqI10TkPfwxXmiHROsr8jXAGPHjqVz584JCktiDR48mBEjRoQdRlbROU++yDnf\nvBm+/94vq1f7Zc0a2LQJSkrg55+3fY1et2lT5Uv0dlsqui1eFerVgwYNIDfXf23YcNsSWVe/vm+X\nk+OXyPex68y2f6yi7aLblte+Ovsur/2oUYO58soR1Yq3urHk5Pg2WW/1arjrLpg6FY4+Gm68kXkr\nVjBgwACoxp2RqyslEpDgXhqR24QD7GFmBwGrnXPfmNldQHvnXKTWx2jgcjO7B1986Wj8/TpOqOQw\nGwE6d+5M165dE/E0pBwtW7bU+U4ynfP4cc4nEMXFlS9FRS357W+7snr19vvIzYXGjaFRI/99bu62\n7yNfmzev/PGq1lXn8Xqxd2lJcy++2JILL9TrPK6cg0mT4PLLfSY2aRKceaZ/rGhrfcO4DWFIiQQE\n+CV+Pr8LluHB+ieBC/GDTrfeeMw597WZnQiMwN+3YgnwB+fca8kMWkTSz6ZNsGJF1UlFZNm0qez2\n9epB69bQpo1fOnSAb76Biy7ati56abrdrQpFUtCaNfCHP8CUKdCnD4wc6V/ACZQSCYjzd0etcEaO\nK+emQ8656fgbJ4lIlov0VCxfDt9+65fI97Ffy+ulaNmybNLQrVv5yUSbNrDjjr6rPtopp8CQIcl5\nriIJ0aQJ/PRT2V6PBEuJBEREpDzO+YRh8WJYtqz8hCLy/caYjuGmTaFdO9h5Z/91v/22fd+2rV/a\ntPG9GbmavC/ZrkED+N//knpIJSCSUP379w87hKyTTud840ZYssQnGLHLN9/4r+vXl92mdetticXe\ne8ORR25LLCJf27WDZs3KP2YipNM5zxQ65+kv5SqhJoqZdQUKCwsLNUBPJEmc80nEnDnwxRfbJxkr\nYsoJtmkDnTptv+yyix9r0aaN/0dNRJKrqKiI/Px8gPx43XFbPSAiEhdr1sAnn8DHH/uEI/L1xx/9\n440bb0soDjwQTjqpbJLRsaOftSEiCeAcvPQSHHOMn3udApSAiEiN/PwzLFiwfaKxaJF/vH592Hdf\nOOAAn2QccIBPODp2VI0FkVAUF8OgQX6Gy4QJ0K9f2BEBSkBEpAIlJT7R+PTTbcvcufD557B5s2/T\noYNPMPr125Zo7LtvyvyDJZLdKqvrkQKUgIhkOefgs8/gww+3JRmffurHbESqbrZtC126QK9e8Mc/\n+u8POMBPSRWRFBTd63Hmmb6uR+vWYUdVhhIQkSyzbh3MnAnvvQfvvuu/RmpjtG/vk4vjjvNfu3SB\nzp1hp53CjVlEqinFez2iKQERyWCRWSjvvrtt+egj37PRogUceihcdZX/+stfwg47hB2xiNTJ5s1w\nxx2+uzIJ1UzrQgmISJorKfG1NBYt2ja9NfL9p5/6Al4Av/gFHHYYXHqp/9q5c+bdH0Qk6zVoAG+9\nBa1ahR1JlZSAiKSJyFiNadPg7bf9ANHFi30V0OhyPm3b+mmtu+4K553nezcOPTTlLv+KSKKkQfIB\nSkBEUpZzvgdj2rRty4oVvteia1dfWvzYY32iEUk4dtlFtTREJD0oARFJEWvXQmEhzJoFM2bA9Omw\napWvq9GtG1xwAfTs6S+fNG8edrQiEpqVK/2Arfrp/RGe3tGLpKmSEl/Aa9Ysv8ycCfPm+V6PJk38\ngNCBA/19Tg49VLd0FxHKznC58Ua47rqwI6oTJSAiSfLVV74S8n/+A2+84ZOQ+vXhoIPgiCPg2muh\ne3cNDhWRckTX9ejTxw/wSnNKQEQSZNMmPxg9knR89pkfoH7kkXDnnXD44T750JgNEalQGtX1qCkl\nICJxsGGDHzAafW+UGTPgp598ca8TToC774ajj9b4DRGppthejxSv61FTSkBEasg5P17j1Vd9Ya+P\nP/b3Rykt9Y/vuae/J8qQIT7xOOgg3YRNRGrh9NN912kG9XpEUwIiUg3ffguvvbZtWbbM33Cte3fo\n3RuuucYnHfvtB82ahR2tiGSE0aN9j0cG9XpEUwIiUolPP4Vhw/w/IM7BwQfD2Wf7pOM3v/EzVkRE\nEmL//cOOIKGUgIiUY/58n3hMmOCLe40cCWeckbH/iIiIJJ0SEJEo8+bBX/4CBQV+8OioUXDhhf5y\ni4hIXEXuoZClg8Rywg5AJBUUFvpB5vvtB2++CQ88AF984YuBKfkQkbgrLvYDS//+97AjCY0SEMlq\nb70Fxx3nK4/Ong0PPwxffulnvuXmhh2diGQc52DiROjSxd/gqVOnsCMKjRIQyTqbN8PTT/t7qhxx\nBCxdCuPH+3EfF12kxENEEiTS63HWWdCrF8yd6weXZSmNAZGssXYtPPaY7/H8+mtfkfTf/4aTToIc\npeIikigZXM20LpSASMYrLob77oOHHoL166FfP98Dkp8fdmQikhWuvRZGjMjIaqZ1oQREMtby5XDv\nvT7xqF8fLrsMrrgCOnYMOzIRySqnneZva61ejzKUgEjGWbYM/vpXGDPGj+f4v/+Dq66CHXcMOzIR\nyUpHHBF2BClJCYhkjEWL4J574NFHfYXSIUPgyiuhVauwIxMRkVhKQCTtffkl3HUXPPmkTzb+9Cc/\n1qtFi7AjExGRiigBkbS1ZQvcfjvccQfk5fnb3Q8cCE2bhh2ZiGQN52DyZNiwAc4/P+xo0oomH0pa\nWrUKTjzR36/l1lth4UI/0FzJh4gkTaSuR79+voSy1Ih6QCTtzJrlZ7OtXw8vvwy//W3YEYlIVlFd\nj7hQD4ikjdJSX0SsRw9o1w6KipR8iEiSlVfNVMlHragHRNLCl1/CBRf4e7dcdZWf7aKS6SKSVC+8\n4N+I1OsRF+oBkZTmnC8kduCBsGSJv8z6978r+RCREDRsqF6POFIPiKSsLVt85dKHHvKzW+69F5o1\nCzsqEclaxx7rF4kLJSCSkkpKYMAAeOYZeOQR+MMfwo5IRETiSQmIpJzFi/1l1nfe8QnIqaeGHZGI\niMSbxoBIyvjhB7jhBth7b/jkE3jlFSUfIpJExcUwdWrYUWQNJSASupISf6fqPfeEBx+EG2+EL77Q\n/ZtEJIkmTYL99oNLL4XNm8OOJisoAZFQffIJdO/u71h75pk+8fjTn6B587AjE5GsEF3NtGdPf+23\nvkYnJIPOsoSitBTuv9/3duy1F3zwARx8cNhRiUhWiVQzjXyvqbVJpR4QSbrFi/1MtsGD4bLLlHyI\nSJLF9nqorkco1AMiSVNaCmPGwPXXQ8uWfpCpSqmLSNK9/76vaqhej1ApAZGkWLoUzjkHpk2DSy6B\nv/7VJyEiIkl38sn+/g4tWoQdSVZTAiIJN3cuHHecL6v+2mtw9NFhRyQiWU/JR+g0BkQSato0OPxw\n2HFH3+up5ENEREAJiCSIczB6tB/j0a2bv4tthw5hRyUiWcE5f/dKSWlKQCTuNm6Eiy/2M1wuvRRe\nekm9nSKSJJEZLgcdBN9/H3Y0UomUSUDM7HIzW2hmG8xshpl1q6L91WY238zWm9liM7vPzHST9pDN\nnw89esDYsfDEE/DAA/4O1iIiCTdpEnTp4q/9jh4NO+wQdkRSiZRIQMysHzAcGAocAnwEvGxmeRW0\nPxu4K2i/L3Ah0A/4S1IClu2Ulvoy6occAj/95IsJnn9+2FGJSFaIruvRq5fqeqSJlEhAgMHAGOfc\nU865+cBAYD0+sSjPocDbzrmJzrnFzrnXgAKge3LClWjFxXD88XDFFf7SS1ER5OeHHZWIZIXIPVwi\ndT0mT4Y2bcKOSqoh9ATEzBoA+cDrkXXOOQe8hk80yvMukB+5TGNmewAnAC8lNlqJtXChn+Xy0Ufw\n8su+vHqTJmFHJSJZ4Z13VM00jaVCHZA8oB6wImb9CmCf8jZwzhUEl2feNjMLth/tnLsnoZFKGR9/\n7EuqN2sG770Hu+8edkQiklUOPxzeftt/lbQTeg9IbZhZT+Am/KWaQ4DTgZPM7JYw48om774LRxwB\n7dr5v38lHyISCiUfaSsVekBWAVuAtjHr2wLLK9hmGPCUc+7x4Oe5ZtYMGAPcUdnBBg8eTMuYGuD9\n+/enf//+NY07a02fDieeCF27wgsvaIqtiEgmKSgooKCgoMy6NWvWxP04oScgzrmfzawQOBp4HiC4\nrHI0cH8FmzUBSmPWlUa2DcaQlGvEiBF07dq1znFnq6lT/W0Ufv1reP55aNo07IhEJKOVlkJOWnbW\np63y/ikvKioiP86zC1Llt3ofcLGZnWdm+wKj8UnGEwBm9pSZ3RnV/gXgMjPrZ2a7mdlv8b0iz1eW\nfEjtOQePP+57Pnr0gBdfVPIhIgnkHEycCAceCKtWhR2NJEDoPSAAzrlJwaDSYfhLL7OBY51zK4Mm\nHYHNUZvcju/xuB3oAKzE955oDEgCrFkDAwfChAlw4YUwciQ0ahR2VCKSsYqLYdAgmDIF+vQJOxpJ\nkJRIQACcc6OAURU8dlTMz5Hk4/YkhJbVZs2Cs87y/4AUFPjvRUQSwjlfy+Pyy8HMf6+ptRkrVS7B\nSIpxDu67zw8w32kn+PBDJR8ikkArVvhk46yzVM00S6RMD4ikjpUr/aWWF1+Ea6+FO+/U/VxEJIEW\nLfLlk9XrkVWUgEgZr7zi7+GyebOfYnvSSWFHJCIZr1MnuP56+P3vVUY9i+gSjACwYQNcfbWvbHrg\ngb7KqZIPEUkKM5+AKPnIKuoBEWbPhnPOgS+/hL//3d9UTtPuRUQkkfQxk8Wcg3/+E371K2jQAAoL\n4aqrlHyIiEji6aMmS23cCBddBJdc4gecvv++v6O1iEjcFRf7YkI//BB2JJJClIBkIef8e8G4cfDE\nE/DQQ5CbG3ZUIpJxItVMu3TxRcXmzw87IkkhSkCy0MMPw5NP+ssv558fdjQikpGKi7fV9ejZ09f1\n+PWvw45KUogGoWaZGTPgyivhssvg3HPDjkZEMk5sNdOJE6Fv37CjkhSkHpAsMmsWHHccdOsGI0aE\nHY2IZJx167bv9VDyIRVQD0iWmDkTjjnGDzT9z3805kNEEqBJE182Wb0eUg1KQLLAnDnbko///hda\ntAg7IhHJSGYwfnzYUUia0CWYDFdcDCefDLvtpuRDRERShxKQDFZSAqef7mt+PP+8kg8REUkdSkAy\n1Ny5fgzYBx/Ac8/5ez2JiNSJc/DMM7B2bdiRSAZQApJhtmyBoUPhkEN80cGpUzX1XkTiYMUK6NMH\nzjgDJkwIOxrJAEpAMsy998Idd8CQIf4mc4cdFnZEIpLWItVM99sPpk/3NT4uvjjsqCQDKAHJIHPn\n+t6P666DP/9ZU21FpI6iq5n26uXfZM48M+yoJENoGm6G2LwZfv972HNPn3yIiNSaqplKEigByRD3\n3ANFRfDee9CoUdjRiEhacw5GjvQj2UeNgjZtwo5IMpASkAwwfbq/9DJkCHTvHnY0IpL2cnJ84aCm\nTcOORDKYxoCkuRUr/OXZHj3gT38KOxoRyRhKPiTBlICkMefgoougtBQKCqC++rNERCRN6CMrjRUU\nwIsv+kJj7dqFHY2IpJXiYmjWzN9ATiQE6gFJUytWwJVX+ssvp54adjQikjYidT26dPFFg0RCogQk\nDf30E5x2mh8ndv/9YUcjImkjtq7H1VeHHZFkMV2CSTORG8zNnQtvvAGtW4cdkYikvNi6HpMmqaCY\nhE49IGnm5pv9tNvnn4f8/LCjEZGUF93rcdRRqmYqKUM9IGnk00/hH/+AYcN8fSARkSpdcAHMnKle\nD0k5SkDShHNwxRWw++5wzTVhRyMiaePBB/1sF12vlRSjBCRNjBsHU6f64oS6yZyIVNvuu4cdgUi5\nNAYkDXz2GVx2GQwYAMcdF3Y0IiIidacEJMWtX+8v23boAA89FHY0IpJynIMtW8KOQqTGlICkuOuv\nhy++gMmT/WVcEZGtIjNcbr017EhEakwJSAp75x1/J+y77oIDDgg7GhFJGdHVTKdNg65dw45IpMaU\ngKSokhK4fFD4AAAgAElEQVS4+GLo3h3++MewoxGRlBFbzXTuXOjTJ+yoRGpMs2BSkHP+Pi8LFkBR\nEdSrF3ZEIhI6VTOVDKMekBQTST4eftgvuvQiIgDcckvZXg8lH5Lm1AOSYv7yF183aMwYX8BQRASA\nfv3g4IOVeEjGUAKSQqZOhdtug6FD4ZJLwo5GRFLKgQf6RSRD6BJMiiguhrPP9veK0ow6ERHJdEpA\nUoBzcNFFvpbQ2LEadCoiIplPCUgK+Oc/4YUX4LHHYOedw45GRJIuMsNlxIiwIxFJGiUgIVu+3N/d\n9pJL4OSTw45GRJIuUtejXz8oLPTJiEgW0CDUkA0bBg0bwj33hB2JiCSV6npIllMPSIgWLPCXX4YM\ngVatwo5GRJImuprpUUfBp58q+ZCsox6QEN1xB7Rtq1LrIlnlv/+Fc89Vr4dkPfWAhGTRIhg/Hq67\nDho3DjsaEUmavDzo3VvVTCXrqQckJPfdBy1a+Om3IpJFunWDCRPCjkIkdOoBCcHXX/uxH1dcAc2a\nhR2NiIhI8qVMAmJml5vZQjPbYGYzzKxbFe1bmtlIM1tmZhvNbL6ZHZeseGvLORg40PfCXntt2NGI\niIiEIyUSEDPrBwwHhgKHAB8BL5tZXgXtGwCvAZ2A04G9gYuBpUkJuA4mT4aXX4aHHoLmzcOORkTi\nrrgY/v3vsKMQSXkpkYAAg4ExzrmnnHPzgYHAeuDCCtr/AWgFnOacm+GcW+yce8s5NydJ8dZKaamv\n+3H88XDiiWFHIyJx5RxMnAhduvipbRs3hh2RSEoLPQEJejPygdcj65xzDt/DcWgFm50MvAeMMrPl\nZjbHzIaYWejPpzLPP+8Hvt9yS9iRiEhcRdf16NXLVzRt1CjsqERSWirMgskD6gErYtavAPapYJs9\ngKOAscDxwF7AQ/jnc3tiwqwb53zvx5FHwmGHhR2NiMSFqpmK1FoqJCC1kYNPUC4Jeks+NLOOwHVU\nkYAMHjyYli1bllnXv39/+vfvn6hYAd/78eGH8MYbCT2MiCRLcTEMGgRTpkCfPjByJLRpE3ZUInVW\nUFBAQUFBmXVr1qyJ+3HMhXzjo+ASzHrgDOfc81HrnwBaOud+V842bwKbnHPHRK07DngJyHXObS5n\nm65AYWFhIV27do3786iMc9C1qy+3rgREJEO88w6cfjo8+KB6PSTjFRUVkZ+fD5DvnCuKxz5D7wFx\nzv1sZoXA0cDzAGZmwc/3V7DZO0Bsl8U+wLflJR9he+opmD0b3nwz7EhEJG4OP9wX9VEpY5FaSZVB\nm/cBF5vZeWa2LzAaaAI8AWBmT5nZnVHtHwJ2NLP7zewXZnYiMAR4MMlxV2nePN9Le+65fvyHiGQQ\nJR8itRZ6DwiAc25SUPNjGNAWmA0c65xbGTTpCGyOar/EzI4FRuBrhiwNvv9rUgOvwvr10Lcv7Lqr\nr/shIiIiXkokIADOuVHAqAoeO6qcde8DKT2f5L774PPPoagImjYNOxoRqRHn4MsvYa+9wo5EJCOl\nyiWYjLNmjU9ALr0U9tsv7GhEpEYidT0OPBCWLQs7GpGMpAQkQR54wF+CueGGsCMRkWqLrmY6bRo8\n+SS0bx92VCIZSQlIAqxZA8OHwyWXQIcOYUcjItUSW8107lxNrxVJoJQZA5JJ7r8fNmyAG28MOxIR\nqZKqmYqEQj0gcbZqle/9uPRS9dyKpIVPPlGvh0gI1AMSZ7fd5r/qhnMiaeKAA/zN45JcIVkk2ykB\niaOPP4YxY3wPSOvWYUcjItWm5EMk6XQJJo4efBB22cVfShYREZGKKQGJk9JSeOEFf1PMBg3CjkZE\nyticcreIEsl61b4EY2YfAtW6da5zLuv6M2fOhOXL4dRTw45ERLaKzHAZMgSmToXddgs7IhEJ1GQM\nyHMJiyIDPPss5OXBYSldHF4kixQX+ztBTpniuyZ1PwSRlFLtBMQ59+dEBpLOfv4ZnnrKz+SrVy/s\naESynOp6iKQFjQGJg//8x19+ueiisCMRyXKqZiqSNmoyBuR7qj8GZMdaR5SGHngAfvlLOOigsCMR\nyWIrV26786N6PURSXk3GgFydsCjS2DvvwOuvw9NPhx2JSJZr3RruuAN+9zto0ybsaESkCjUZA/Jk\nIgNJVyNHwj77+Pc8EQnZpZeGHYGIVFOdx4CYWSMzaxG9xCOwdPDjj/Dcc/D730OORtOIiIhUW60+\nNs2sqZk9aGbFwDrg+5glKzzzDGzcCOecE3YkIiIi6aW2/7f/FTgKuAwoAS4ChgLLgPPiE1rq+/e/\n4dBDffl1EUmw4mLf3bh0adiRiEgc1DYBORkY5JybAmwG3nLO3QHcBGRFf8DGjfDqq3DSSWFHIpLh\nnIOJE6FLF3jpJfjyy7AjEpE4qG0CsiPwVfD9j8HPAG8DR9Q1qHTw9NOwbh2cfnrYkYhksPLqehyR\nFW8xIhmvtgnIV8Duwffzgb7B9ycDP9Q1qHQwciT07u1nwIhInEX3ekyb5ut6TJ6s6bUiGaS2Ccjj\nQKTs1t3A5Wa2ERgB3BuPwFLZRx/BjBm+0rOIxNmmTapmKpIFalKIbCvn3Iio718zs32BfOAL59zH\n8QouVY0b5288d+KJYUcikoEaNoT27X0PSN++VbcXkbRUqwQklnNuEbAoHvtKdaWlUFDg/yFr0CDs\naEQy1P33hx2BiCRYbeuA3G9mV5az/o9m9ve6h5W63nkHliyBs88OOxIREZH0VdsxIGcA75Sz/l2g\nT+3DSX0vv+wvvxx2WNiRiIiIpK/aJiA7AWvKWf8jkFf7cFLf9Ol+FqBKr4vUknN+RsuKFWFHIiIh\nqu3H6BfAceWsP55t9UEyzsaNMHOmyhCI1FqkrkffvjB+fNjRiEiIajsI9T7gQTNrDUwN1h0NXAtc\nHY/AUtErr0BJCRxzTNiRiKShSZNg0CAw899raq1IVqvtNNzHzCwXuBm4NVj9NXCZc+6pOMWWciZN\ngv33h86dw45EJI0UF/uiOU8/DX36+Cp+KigmkvVqPQ3XOfcQ8FDQC7LBOfdT/MJKPVu2wIsvwtUZ\n278jkgCTJm2r2KdeDxGJUuuhlGZW38x6A6cDFqxrb2bN4hVcKvnkE1izBo46KuxIRNJEpJx6z56q\nZioi26lVD4iZ7Qr8D+gE5AKvAmuBG4KfB8YrwFTx9tu+8Fi3bmFHIpImzPxA09zcsCMRkRRU2x6Q\nfwAfADsAG6LWP4sfjJpxXn0VfvlLaNw47EhE0oiSDxGpQG3HgPwGOMw5t8nMotd/DXSoa1CpprgY\nXnoJhg8POxIREZHMUNsekBygXjnrO+IvxWSUf/3LFx4755ywIxFJMcXFsHp12FGISBqqbQLyCmXr\nfbhg8Omfgf/UOaoUM24cnHoq7LRT2JGIpIjIANMuXeCmm8KORkTSUG0TkGuBw83sU6ARMB5/+aUj\nfiBqxvjxR5g9G449NuxIRFJEpJrpWWdBr14wbFjYEYlIGqptIbIlZnYQcBZwINAMeBQYF8fYUsKs\nWf6fvUMPDTsSkZA5t62uh6qZikgd1aUQ2WZgbOTnoDLq5cD1wM51Dy01vPcetGoF++4bdiQiISou\n9mXUp0xRNVMRiYsaXYIxs1wzu8vMPjCzd83stGD9BcBCYDAwIgFxhmbWLF/7Q3e/lax2zTUwbZrv\n9Zg8WcmHiNRZTT9WhwGX4cd77AZMNrOH8YnHNcBuzrl74hlg2D76CA4+OOwoREL2t7+pmqmIxFVN\nL8GcCZznnHvezPYHPg72cZBzzsU9upCtWQOLFsGBB4YdiUjIds6Yq6oikiJq2gPSESgEcM59ApQA\nIzIx+QD44AP/NT8/3DhEREQyTU0TkHrApqifNwMZexfcmTOhRQvYZ5+wIxFJgp9/DjsCEckiNb0E\nY8ATZlYS/NwIGG1m66IbOedOj0dwYZs5UwNQJQsUF/uptS1awKOPhh2NiGSJmn60PgkUA2uCZSyw\nLOrnyJL2Nm3yg/4PPzzsSEQSaNIk2G8/ePNNOO64sKMRkSxSox4Q59wFiQok1bz6Knz/vQb9S4aK\n9Ho8/bTqeohIKGpdiCzTTZjgb3Ox//5hRyISZ5FqppHvlWWLSAg0uqEcGzfCv/8N/fqFHYlInN11\nl39h9+ypuh4iEqqUSUDM7HIzW2hmG8xshpl1q+Z2Z5lZqZk9E69Y/vtfWLtWCYhkoLPP9nexVTVT\nEQlZSlyCMbN+wHDgEmAmvrLqy2a2t3NuVSXb7QbcC0yPZzzPPQcHHKDpt5KBdt3VLyIiIUuVHpDB\nwBjn3FPOufnAQGA9cGFFG5hZDn4Wzm34+9DEhXPw+utw7LHx2qOIiIjECj0BMbMGQD7wemRdUFn1\nNeDQSjYdCqxwzj0ez3gWLIClS+Goo+K5VxEREYkWegIC5OErrK6IWb8CKPcGFGbWA7gAuCjewbz1\nli881qNHvPcskmDO+fEdt9wSdiQiIlVKhQSkRsysGfAUcLFz7vt473/BAujUCZo3j/eeRRJoxQpf\nz+Oss+Dzz2HLlrAjEhGpVCoMQl0FbAHaxqxvCywvp/2ewK7AC2ZmwbocADPbBOzjnKtwTMjgwYNp\n2bJlmXX9+/enf//+ACxcCLvvXotnIRIG57bV9TBTXQ8RqbOCggIKCgrKrFuzJv5Fzi0VbmRrZjOA\n951zVwU/G7AYuN85d29M24bAXjG7+AvQDLgSWOCc21zOMboChYWFhXTt2rXCWLp1g4MOgkceqcsz\nEkmC4mIYNAimTFE1UxFJqKKiIvL9reHznXNF8dhnKvSAANyHv8ldIdum4TYBngAws6eAJc65m5xz\nm4BPozc2sx/wY1fn1SWILVt87/UZZ9RlLyJJ8Npr/nKLmR/30bdv2BGJiNRISiQgzrlJZpYHDMNf\nepkNHOucWxk06Qhs16sRb3PmwI8/wmGHJfpIInW0665+rviIEer1EJG0lBIJCIBzbhQwqoLHKp0U\nG6+b5E2fDrm50L17PPYmkkC/+AWMGxd2FCIitZZ2s2ASado0+NWvoFGjsCMRERHJbEpAosyYocsv\nIiIiyaAEJPDdd7BsGRx8cNiRiODrevzrX2FHISKSMEpAAnPm+K8HHBBuHJLlItVM99sPrrsOEjD3\nXkQkFSgBCcyZAw0bwt57hx2JZK3oaqa9evkXZUzRPBGRTJEys2DC9vHH0KUL1NcZkWRTNVMRyULq\nAQnMmaPLLxKC4uKyvR5z5yr5EJGsoAQEKC2FTz6BAw8MOxLJOqtWwcyZvtdj8mQVFRORrKELDsBX\nX8G6dUpAJARdusCXX/oBSCIiWUQ9IMCHH/qvmoIroVDyISJZSAkIMHs2tG+v3m8REZFkUQKCH4Cq\nyy+SEM75AUYiIlKGEhDg00/9pXiRuCou9jNaDj4YFiwIOxoRkZSS9QnIxo2wcKESEImjSDXTLl38\nHQ7Hj/d3rxURka2yPgGZP99Pw+3cOexIJCNEej3OOgt69vR1Pfr2DTsqEZGUk/XTcGfNgnr14KCD\nwo5E0lpsNdOJE5V4iIhUIut7QGbM8ANQmzYNOxJJa19/DQMGqNdDRKSasr4HZPZs6No17Cgk7e2+\nu7+hkK7liYhUS1b3gJSWwrx5/s7nInWm5ENEpNqyOgFZvBg2bNDnhoiISLJldQLy6af+q6bgSrWU\nlIQdgYhIxsjqBGTBAmjUCHbZJexIJKVF6npExnmIiEidZXUC8v33sOOOftakSLmi63r06AHt2oUd\nkYhIRsjqWTA//ggtW4YdhaSk2Loekyb5REREROIiq3tA1qyBFi3CjkJSTnSvx1FH+cFCSj5EROIq\nq3tA1qxRD4jEWLvWV6bbskW9HiIiCZTVCch338HOO4cdhaSU5s1hxAg4+mho0ybsaEREMlZWX4JZ\nvlwJiJSjf38lHyIiCZbVCciKFUpAREREwpC1CUhJiZ+G27Zt2JGIiIhkn6xNQL7+2n/ddddQw5Bk\nKy6Gs8/2NwESEZHQZG0CsmCB//qLX4QbhyRJpJpply7w6quwdGnYEYmIZLWsTkAaN4b27cOORBKu\nvLoevXuHHZWISFbL2mm4X30Fe+wBOVmbgmUBVTMVEUlZWfvxu2SJbkKX0UpLoV8/VTMVEUlRWdsD\nsmQJHHRQ2FFIwuTkwP77+6RDiYeISMrJ2gRk6VI44YSwo5CEuu22sCMQEZEKZOUlmE2bfBXUTp3C\njkRERCQ7ZWUC8s03fnzibruFHYmIiEh2ysoEREXIMkBkhssXX4QdiYiI1EJWJiCLFvlZmZoFk6Yi\ndT369YOCgrCjERGRWsjKQaiLFkG7dpCbG3YkUiOq6yEikjGysgdk6VLo2DHsKKRGoquZ9uoFc+cq\n+RARSWNZ2QOyfDnsvHPYUUi1TZoEgwap10NEJINkZQ+IEpA088Yb6vUQEckw6gGR1PfAA1A/K1+q\nIiIZK+t6QEpLYcUKaNs27Eik2pR8iIhknKxLQNasgc2b/SwYERERCUfWJSDffee/6hJMCiku9lOT\nREQkaygBkfA4BxMnQpcucO21YUcjIiJJlLUJiMaAhCy2rsf994cdkYiIJFHKJCBmdrmZLTSzDWY2\nw8y6VdL2IjObbmarg+XVytpH++EHaNwYmjSJX+xSA9G9HtOm+boekydDmzZhRyYiIkmUEgmImfUD\nhgNDgUOAj4CXzSyvgk2OBMYDPYFfA98Ar5hZlUNLS0qUfIRG1UxFRCSQEgkIMBgY45x7yjk3HxgI\nrAcuLK+xc+5c59xo59zHzrnPgYvwz+Xoqg5UUuJ7QCQEt9+uXg8REQFSIAExswZAPvB6ZJ1zzgGv\nAYdWczdNgQbA6qoabtyoBCQ0t9+uXg8REQFSoxJqHlAPWBGzfgWwTzX3cQ+wFJ+0VEo9ICFq1Srs\nCEREJEWkQgJSJ2Z2I9AXONI5t6mq9uoBERERCV8qJCCrgC1A7MTYtsDyyjY0s+uA64GjnXNzq3Ow\nd94ZTL16LTnllG3r+vfvT//+/WsSs5THOWV4IiJprqCggIKCgjLr1qxZE/fjmB9uES4zmwG875y7\nKvjZgMXA/c65eyvY5npgCHCMc25WNY7RFSjcf/9CunfvyqOPxi9+wc9wGTTIJyAvvABmYUckIiJx\nUlRURH5+PkC+c64oHvsMfRBq4D7gYjM7z8z2BUYDTYAnAMzsKTO7M9LYzG4AhuFnySw2s7bB0rSq\nA61apSqocRVb1+P885V8iIhIlVLhEgzOuUlBzY9h+Esvs4FjnXMrgyYdgc1RmwzEz3p5OmZXfw72\nUaHvvlMCEjeRXo8pU6BPHxg5UlNrRUSkWlIiAQFwzo0CRlXw2FExP+9e2+P8/LPKsNeZc76Wx+WX\n+96OSZM0tVZERGokVS7BJFX79mFHkOYeeEDVTEVEpE5SpgckmdpVWbBdKnX22T6L69Mn7EhERCRN\nZWUPiBKQOsrLU/IhIiJ1knUJSNOmuhmdiIhI2LIuAWnePOwIREREJOsSEBXprEKkrsfAgWFHIiIi\nGSzrEpBGjcKOIIUVF/sZLWed5QumlJSEHZGIiGSorJsFowSkHKrrISIiSaYekGwX3euhuh4iIpIk\n6gHJZm+9Bb/7nXo9REQk6bKuByQ3N+wIUsjee8Opp6rXQ0REki7rekAaNgw7ghTSti08+mjYUYiI\nSBbKuh4QJSAiIiLhUwIiIiIiSZd1CUhWjQEpLoaRI8OOQkREZDtZl4A0aBB2BEkQqWbapQv86U+w\nfHnYEYmIiJSRdYNQmzULO4IEKy6GQYNgyhR/x9qRI6FNm7CjEpEaWrx4MatWrQo7DMkSeXl5dOrU\nKanHzLoEpGXLsCNIEFUzFckYixcvpnPnzqxfvz7sUCRLNGnShHnz5iU1Ccm6BKRFi7AjSICVK+Gy\ny9TrIZIhVq1axfr16xk7diydO3cOOxzJcPPmzWPAgAGsWrVKCUgiZWQPSEkJfPSRej1EMkznzp3p\n2rVr2GGIJETWJSAZ2QPSsSPMnw/16oUdiYiISLVk3SyYVq3CjiBBlHyIiEgayboEpHnzsCMQERGR\nrEtA6qfrRadZs8KOQEREJG6yLgFJO8XFfmBp9+5QVBR2NCIiInGhBCSVTZoE++0Hb77pv9doeBHJ\nIKNGjSInJ4dDDz203McXLVpETk4O9913X7mP/+1vfyMnJ4fFixdv99izzz7LCSecQOvWrcnNzaVD\nhw7069ePN954I67PoTreffddevToQdOmTWnXrh1XXXUV69atq9a2xcXFXHDBBbRt25YmTZqQn5/P\n008/XW7bZcuW0bdvX3bYYQdatmzJaaedxsKFC+P5VOJKCUgqivR69OsHPXvC3LmaXisiGWf8+PHs\nvvvuzJw5k6+++qrG25sZZrbd+gsuuIAzzjiD4uJirr32WsaMGcMf//hHFi5cSO/evZkxY0Y8wq+W\n2bNn07t3bzZu3MiIESO4+OKLefjhh+nbt2+V265du5bDDz+cZ599lssuu4zhw4fTokUL+vbty4QJ\nE8q0XbduHT179uStt97illtuYdiwYXz44Yf07NmT77//PlFPr26cc1mxAF0BV1hY6FLaxInO5eX5\nZdKksKMRkRAUFha6tHi/qoOvvvrKmZl77rnnXJs2bdywYcO2a/P11187M3PDhw8vdx9/+9vfXE5O\njlu0aNHWdffee68zM3fttdeWu83YsWPdrFmz4vMkquH44493HTp0cD/99NPWdY888ojLyclxr776\naqXb/vWvf3U5OTnuzTff3LqutLTUde/e3bVv3979/PPPW9ffc889Licnp8xrZv78+a5+/fru5ptv\nrvQ41Xm9RdoAXV2cPpfVA5JKiovhwgvV6yEiGW/cuHHsuOOOnHjiifTp04dx48bVeZ8bN27k7rvv\npkuXLtx7773ltjnnnHP45S9/WedjVcfatWt57bXXOPfcc2natOnW9eeddx5NmzZl0qRJlW7/9ttv\n07p1a4488sit68yMvn37snz5cqZNm7Z1/ZQpU+jWrVuZwnX77LMPRx99dJXHCYsSkFTSpg3MmQOT\nJ6uUuohktPHjx3PGGWdQv359+vfvz4IFCygsLKzTPt9++21Wr17N2WefXe6lmer64Ycf+O6776pc\nNmzYUOl+5syZw+bNm8nPzy+zvkGDBhx88MF8+OGHlW5fUlJC48aNt1vfpEkTnHNbz5dzjo8//rjc\nxKp79+58+eWX1R5zkkxKQFLN7ruHHYGISEIVFhYyf/58zjrrLAB69OhBhw4d6twLMm/ePMyM/fff\nv077OeSQQ2jdunWlS5s2bSrsZYn49ttvMTPatWu33WPt2rVj2bJllW6/zz77sGTJEr755psy66dP\nn46ZsXTpUgBWr15NSUlJhccBqjxWGNK1KoaIiATWr/d3Y0ikffeFJk3is69x48ax884707Nnz63r\n+vXrx7hx4xg+fHitey9+/PFHAJrXseLk+PHjq+zdANhjjz0qfTyyj9zc3O0ea9SoUZXHuOiiixg9\nejRnnnkmI0aMoG3btkycOJHnnnuuzP6rOk50m1SiBCTZ1q2DqGuBIiJ1NX8+xPTyx11hYXwqAZSW\nljJx4kR69epVZuZL9+7dGT58OK+//jq9e/eu0T4jCUuL4GZfa9eurVOMFU0LrqnI5ZOSkpLtHtu4\ncWO5l1eiHXDAARQUFDBw4EB69OiBc4527drxj3/8g4EDB9KsWbNqHSe6TSpRApIszvlaHldc4b9G\nZf4iInWx774+QUj0MeJh6tSpfPvtt0yYMIGCgoIyj5kZ48aN25qAVPXf+/r168u023fffXHOMWfO\nHE455ZRax7hq1Sq2bNlSZbtmzZqVGVwaq127djjn+Pbbb7d77Ntvv6V9+/ZVHuP000/nlFNO4aOP\nPmLLli107dp1ay2TvffeG4Add9yR3NzcCo8DVOtYyaYEJBmKi2HQIJgyBfr0gS5dwo5IRDJIkybp\nU6dw7NixtG3bllGjRkVKJGw1ZcoUnn32WUaPHk1ubi6tW7emSZMmfPbZZ+Xua/78+TRp0oS8vDzA\njyXZYYcdKCgo4Kabbqr1pZxu3bqxaNGiStuYGUOHDuW2226rsM3+++9P/fr1+eCDD+jTp8/W9T//\n/DOzZ8+mX79+1Yqnfv36ZQayvvrqq5jZ1kTNzDjggAP44IMPttv2/fffZ4899qg0UQqLEpBEmzTJ\nJx9m/ntNrRWRLLVx40aeffZZ+vXrx+9+97vtHm/Xrh0FBQU8//zznHnmmeTk5HDMMcfwwgsv8M03\n37DLLrtsbbt48WJefPFFjj322K2JRuPGjbnhhhu48cYbuf7668sdJDpu3Dj22WefSqfixmsMSIsW\nLejduzdjx47l1ltv3ZoEPPXUU6xbt65MMbINGzawePFi8vLy2GmnnSrc54IFCxgzZgwnn3wye+21\n19b1ffr0YciQIRQVFW2divvZZ58xdepUrr/++iqfSyjiVVAk1ReSXYhsxQrn+vRxDvzXFSuSc1wR\nSXuZWohswoQJzszcCy+8UO7jpaWlrk2bNu7UU0/dum7evHmuVatWLi8vz910003u4YcfdkOGDHF5\neXluhx12cPPnz99uH+eff77Lyclx+fn57q677nKPP/64u+uuu9yvfvUrl5OT42bMmJHQ5xmtqKjI\nNW7c2HXt2tWNHj3a3Xzzza5x48bu+OOPL9PuzTffdGbm/vznP5dZ36VLFzd06FD36KOPuptvvtnt\ntNNObo899nDLli0r027t2rVur732cm3btnX33nuvGzFihOvUqZPbZZdd3KpVqyqNMaxCZKEnBsla\nkpqAbNzoXKdOqmYqIrWSqQnIKaec4po2beo2bNhQYZsLLrjA5ebmutWrV29d99lnn7n+/fu7nXfe\n2TVs2NDtvPPO7pxzznGfffZZhft55pln3HHHHefy8vJcw4YNXYcOHVz//v3dW2+9FdfnVB3vvPOO\n69Gjh2vSpIlr27atu/LKK8tURnXOJyA5OTnbVYQ9++yz3a677uoaNWrkOnbs6C6//HK3cuXKco+z\ndHecMbsAAAqvSURBVOlS17dvX9eqVSvXokULd+qpp7ovv/yyyvjCSkDMOVdex0jGMbOuQGFhYWGZ\nSnEJ89JL0K2bCoqJSI0VFRWRn59P0t6vJKtV5/UWaQPkO+ficmt2jQFJlBNPDDsCERGRlKVKqCIi\nIpJ0SkBEREQk6ZSA1EZxMfTtC+++G3YkIiIiaUkJSE04BxMn+kJib7wBP/wQdkQiIiJpSQlIdRUX\n+yJiZ50FvXrB3LlwwglhRyUiIpKWNAumKi64h8vll6uaqYiISJyoB6QyzsGAAWV7PZR8iIiI1Jl6\nQCpjBocdBqedpsRDRJJu3rx5YYcgWSCs15kSkKpcfnnYEYhIlsnLy6NJkyYMGDAg7FAkS0TfVThZ\nlICIiKSYTp06MW/ePFatWhV2KJIl8vLy6NSpU1KPqQRERCQFderUKekfCCLJlDKDUM3scjNbaGYb\nzGyGmXWrov2ZZjYvaP+RmR1f44NG6noUxeW+OlKOgoKCsEPIOjrnyadznnw65+kvJRIQM+sHDAeG\nAocAHwEvm1m5F6TM7DBgPPBP4GDg38BzZtal2geNrusxaVIdn4FURG8Syadznnw658mnc57+UiIB\nAQYDY5xzTznn5gMDgfXAhRW0vxL4r3PuPufcZ86524Ai4I9VHim6mum0aT75uPvuOD0NERERqY7Q\nExAzawDkA69H1jnnHPAacGgFmx0aPB7t5Urab3P99arrISIiErJUGISaB9QDVsSsXwHsU8E2O1fQ\nfucqj1ZUpGqmIiIiIUuFBCRZGgHMu/NO2HNPDTxNkjVr1lCkc51UOufJp3OefDrnyRVVrKxRvPaZ\nCgnIKmAL0DZmfVtgeQXbLK9he4DdAAYMGlTzCKVO8vPzww4h6+icJ5/OefLpnIdiN+DdeOwo9ATE\nOfezmRUCRwPPA5iZBT/fX8Fm75Xz+G+D9RV5GTgH+BrYWLeoRUREskojfPLxcrx2aH68Z7jMrC/w\nBH72y0z8rJg+wL7OuZVm9hSwxDl3U9D+UOBNYAjwEtAfuBHo6pz7NOlPQERERGok9B4QAOfcpKDm\nxzD8pZTZwLHOuZVBk47A5qj275nZ2cBfgmUBcKqSDxERkfSQEj0gIiIikl1CrwMiIiIi2UcJiIiI\niCRdxiQgodzMLsvV5Jyb2UVmNt3MVgfLq1X9jmR7NX2dR213lpmVmtkziY4x09TivaWlmY00s2Vm\nttHM5pvZccmKNxPU4pxfHZzn9Wa22MzuM7PcZMWb7szsN2b2vJktDd4nTqnGNj3NrDB4jX9uZufX\n9LgZkYCEcjO7LFfTcw4ciT/nPYFfA98Ar5hZu8RHmxlqcc4j2+0G3AtMT3CIGacW7y0N8LeJ6ASc\nDuwNXAwsTUrAGaAW5/xs4K6g/b74e4j1w09QkOppip/8MQiocmBo8J7yIv4WKgcB/wAeMbPf1uio\nzrm0X4AZwD+ifjZgCXB9Be0nAM/HrHsPGBX2c0mXpabnvJztc4A1wICwn0u6LLU558F5fhu4AHgc\neCbs55FOSy3eWwbiZ+XVCzv2dF1qcc4fAF6NWfc3YHrYzyUdF6AUOKWKNvcAH8esKwD+U5NjpX0P\nSNJvZie1PeexmgINgNVxDzAD1eGcDwVWOOceT2yEmaeW5/xkgn9mzGy5mc0xsyFmlvbvtclQy3P+\nLpAfuUxjZnsAJ+BrREli/Jo4fIamRB2QOkruzewEanfOY92D75aOfRFL+Wp8zs2sB77n46DEhpax\navM63wM4ChgLHA/sBTyEf6+9PTFhZpQan3PnXEFweebtoIp2PWC0c+6ehEaa3Sr6DG1hZrnOuZLq\n7CQTEhBJM2Z2I9AXONI5tynseDKRmTUDngIudu7/27u7kMuqOo7j359lkF1KMVJmNtqQVEMRhA1I\nMJWQGHhRQxC9Ul0IEjZoWVA6kZGC4J0lhRbhhd4YFXihXTR5I4iQDhqN+VITPQ1TM/k25fPvYq2D\nZ56e8TnnPHP2mfP0/cDinL332nuvszic899rr7VXHVl0ef6PnEH7If5yv3J/OMlbgL0YgMxFkg8B\n1/HKk7QvAG5NcqiqvrvIsunVbYUAZKjJ7PSKWeocgCR7gWuA3VX16HyKtyVNW+fbgfOAX/SrQuid\nzpMcB3ZU1ZNzKutWMcv3/BBwvAcfIweAbUleW1X/Ocl+amap8xuAO8duMz7aA/DbAAOQ+TjZf+jR\nSVs/YAuMgqmqfwOjyeyAEyazO9mMfQ+O5+82msxO3Yx1TpJrgG/SHrP/8LzLuZXMUOcHgHfTRnnt\n7Ole4P7+/pk5F3npzfg930+7Ah+3Azhk8LGxGev8LFrHyXGrY/vq1FvvP/SjTPsfuuget6eo1+4n\ngeeBz9CGYd0GHAbe2LffCXxvLP/FwEvA1bQfh+/QZsi9aNGfZVnSDHV+ba/jK2iR8ii9YdGfZVnS\ntHW+zv6OgplzndPmrfoHbabuC4HLaFeLX1/0Z1mWNEOdf7vX+R7abK0foY1E+vmiP8uyJNqggJ20\nC5ZV4Kt9+dy+/UbgjrH8bwOO0fry7aAN3z0OfHia826FWzCUk9kNbto6p92fPRO4e82hru/H0AZm\nqHNt0gy/Lc8muRS4hfb8ij/39z8YtOBLbIbv+T7an+Y+4M3ACq2171uDFXr5vR94gPYMkKI9hwXg\nDtpzVbYB544yV9WfklxG+25fRRsm/cWqmmpQgZPRSZKkwS19HxBJkrR8DEAkSdLgDEAkSdLgDEAk\nSdLgDEAkSdLgDEAkSdLgDEAkSdLgDEAkSdLgDEAkzVWS7UlWk1zUl3f35bMWXTZJi2MAImlDSX7S\ng4aX++vo/dsnPMTaRy7P/RHMBjrS6W1LzAUjaRC/Bj4HjM8wurJ+1v+xiFlJQwt0nBFVOg3ZAiJp\nUi9V1UpV/W0sVZKPJfltkiNJ/p7k3iTnb/ZkSa5M8sckLyZ5LMmnxradcFunrzu7r/tgku3AfX3T\nsd5a88PNlknSqWMAImmzXg/cBLwP2E1rcbhnMwdM8gnajJw3Au8Cfgz8NMmusWzr3cYZrTtIm9Yd\n4HzgHODqzZRJ0qnlLRhJk7o8ybGx5V9V1Z6qOiHYSPIl4C9J3lFVT8x4rq8BP6qq2/vyzUkuBvYC\n+0enWme/APSWmSN93UpVPT9jOSTNiS0gkiZ1P/AeYGdPVwEkuTDJXUkOJjkK/IHWEvHWjQ6Y5Iwk\nx3o6muTWvumdwO/WZN/f14/MvSOrpPmxBUTSpJ6rqifXWf9L4AngC8Ah4HXAI/31VVXVapKdY6v+\nOWFZVmmtHeOtIGdOuK+k04AtIJJmluRNwAXAvqr6TVU9DpzNFMNuq+rgWDrcVx8Adq3Jugt4rL8f\njb45Z2z7e9ec53h/fc1EH0bSoGwBkbQZh4EjwFeSrNA6fH5/nXzTDoW9CfhZkkeAB4ArgI8DlwBU\n1b+SPAR8I8mzwDbghjXHeKq/Xp7kPuCFqnpuynJImhNbQCTNrKpeBvYAHwB+Twsc9q6Xdcrj3kPr\niHptP+7ngU9X1YNj2T5LG4HzEHAzcN2aYzwNXE8bTfNX4JZpyiBpvlJlPy5JkjQsW0AkSdLgDEAk\nSdLgDEAkSdLgDEAkSdLgDEAkSdLgDEAkSdLgDEAkSdLgDEAkSdLgDEAkSdLgDEAkSdLgDEAkSdLg\nDEAkSdLg/gv+vlazMs4PvwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xcca00b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import roc_curve, auc\n",
    "\n",
    "predictions = rf.predict_proba(X_test)\n",
    "\n",
    "false_positive_rate, recall, thresholds = roc_curve(y_test, predictions[:, 1])\n",
    "roc_auc = auc(false_positive_rate, recall)\n",
    "plt.title('Receiver Operating Characteristic')\n",
    "plt.plot(false_positive_rate, recall, 'b', label='AUC = %0.2f' % roc_auc)\n",
    "plt.legend(loc='lower right')\n",
    "\n",
    "plt.plot([0, 1], [0, 1], 'r--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.0])\n",
    "plt.ylabel('Recall')\n",
    "plt.xlabel('Fall-out')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tuning Models using GridSearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 108 candidates, totalling 324 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  34 tasks      | elapsed:   12.6s\n",
      "[Parallel(n_jobs=-1)]: Done 184 tasks      | elapsed:  1.2min\n",
      "[Parallel(n_jobs=-1)]: Done 324 out of 324 | elapsed:  3.2min finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best score: 0.860\n",
      "Best parameters set:\n",
      "\tclf__max_depth: 50\n",
      "\tclf__min_samples_leaf: 2\n",
      "\tclf__min_samples_split: 5\n",
      "\tclf__n_estimators: 100\n",
      "Accuracy: 0.858682138417\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.88      0.94      0.91      9043\n",
      "          1       0.76      0.63      0.69      3022\n",
      "\n",
      "avg / total       0.85      0.86      0.85     12065\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "pipeline2 = Pipeline([\n",
    "('clf', RandomForestClassifier(criterion='entropy'))\n",
    "])\n",
    "\n",
    "parameters = {\n",
    "    'clf__n_estimators': (5, 25, 50, 100),\n",
    "    'clf__max_depth': (5, 25 , 50),\n",
    "    'clf__min_samples_split': (2, 5, 10),\n",
    "    'clf__min_samples_leaf': (1, 2, 3)\n",
    "}\n",
    "\n",
    "grid_search = GridSearchCV(pipeline2, parameters, n_jobs=-1, verbose=1, scoring='accuracy', cv=3)\n",
    "\n",
    "grid_search.fit(X_train, y_train)\n",
    "\n",
    "print ('Best score: %0.3f' % grid_search.best_score_)\n",
    "\n",
    "print ('Best parameters set:')\n",
    "best_parameters = grid_search.best_estimator_.get_params()\n",
    "\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print ('\\t%s: %r' % (param_name, best_parameters[param_name]))\n",
    "\n",
    "predictions = grid_search.predict(X_test)\n",
    "print ('Accuracy:', accuracy_score(y_test, predictions))\n",
    "print (classification_report(y_test, predictions))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Naive Bayes accuracy and time elapsed caculation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NaiveBayes\n",
      "Acurracy:  0.754910899295\n",
      "time elapsed:  0.43802499771118164\n"
     ]
    }
   ],
   "source": [
    "t4=time()\n",
    "print (\"NaiveBayes\")\n",
    "nb = BernoulliNB()\n",
    "clf_nb=nb.fit(X_train,y_train)\n",
    "print (\"Acurracy: \", clf_nb.score(X_test,y_test))\n",
    "t5=time()\n",
    "print (\"time elapsed: \", t5-t4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## cross-validation for NB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cross result========\n",
      "[ 0.765  0.763  0.762  0.766  0.76 ]\n",
      "0.763178730071\n",
      "time elapsed:  1.200068473815918\n"
     ]
    }
   ],
   "source": [
    "tt4=time()\n",
    "print (\"cross result========\")\n",
    "scores = cross_validation.cross_val_score(nb, X, y, cv=5)\n",
    "print (scores)\n",
    "print (scores.mean())\n",
    "tt5=time()\n",
    "print (\"time elapsed: \", tt5-tt4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# KNN accuracy and time elapsed caculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KNN\n",
      "Acurracy:  0.760464152507\n",
      "time elapsed:  2.1001200675964355\n"
     ]
    }
   ],
   "source": [
    "t6=time()\n",
    "print (\"KNN\")\n",
    "# knn = KNeighborsClassifier(n_neighbors=3)\n",
    "knn = KNeighborsClassifier()\n",
    "clf_knn=knn.fit(X_train, y_train)\n",
    "print (\"Acurracy: \", clf_knn.score(X_test,y_test) )\n",
    "t7=time()\n",
    "print (\"time elapsed: \", t7-t6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## cross validation for KNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cross result========\n",
      "[ 0.768  0.769  0.768  0.764  0.77 ]\n",
      "0.76788674415\n",
      "time elapsed:  10.268587589263916\n"
     ]
    }
   ],
   "source": [
    "tt6=time()\n",
    "print (\"cross result========\")\n",
    "scores = cross_validation.cross_val_score(knn, X, y, cv=5)\n",
    "print (scores)\n",
    "print (scores.mean())\n",
    "tt7=time()\n",
    "print (\"time elapsed: \", tt7-tt6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}