Tic-Tac-Toe Endgame Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tic-Tac-Toe Endgame Classification\n",
    "<hr>\n",
    "\n",
    "In this example, we would take up the binary classification task on possible configurations of tic-tac-toe game.\n",
    "\n",
    "Dataset : https://archive.ics.uci.edu/ml/datasets/Tic-Tac-Toe+Endgame\n",
    "\n",
    "#### Data Set Information:\n",
    "\n",
    "This database encodes the complete set of possible board configurations at the end of tic-tac-toe games, where \"x\" is assumed to have played first. The target concept is \"win for x\" (i.e., true when \"x\" has one of 8 possible ways to create a \"three-in-a-row\"). \n",
    "\n",
    "Interestingly, this raw database gives a stripped-down decision tree algorithm (e.g., ID3) fits. However, the rule-based CN2 algorithm, the simple IB1 instance-based learning algorithm, and the CITRE feature-constructing decision tree algorithm perform well on it.\n",
    "\n",
    "\n",
    "#### Attribute Information:\n",
    "\n",
    "1. top-left-square: {x,o,b} \n",
    "2. top-middle-square: {x,o,b} \n",
    "3. top-right-square: {x,o,b} \n",
    "4. middle-left-square: {x,o,b} \n",
    "5. middle-middle-square: {x,o,b} \n",
    "6. middle-right-square: {x,o,b} \n",
    "7. bottom-left-square: {x,o,b} \n",
    "8. bottom-middle-square: {x,o,b} \n",
    "9. bottom-right-square: {x,o,b} \n",
    "10. Class: {positive,negative}\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from sklearn.tree import DecisionTreeClassifier, export_graphviz\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn import cross_validation, metrics\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.naive_bayes import BernoulliNB\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.svm import SVC\n",
    "from time import time\n",
    "from sklearn import preprocessing\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.metrics import roc_auc_score , classification_report\n",
    "from sklearn.grid_search import GridSearchCV\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.metrics import precision_score, recall_score, accuracy_score, classification_report\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# read .csv from provided dataset\n",
    "csv_filename=\"tic-tac-toe.data\"\n",
    "\n",
    "# df=pd.read_csv(csv_filename,index_col=0)\n",
    "df=pd.read_csv(csv_filename, \n",
    "              names=[\"TopLeft\", \"TopMiddle\" , \"TopRight\" ,\n",
    "                     \"MiddleLeft\", \"MiddleMiddle\" , \"MiddleRight\" ,\n",
    "                     \"BottomLeft\", \"BottomMiddle\" , \"BottomRight\" , \"Class\" ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TopLeft</th>\n",
       "      <th>TopMiddle</th>\n",
       "      <th>TopRight</th>\n",
       "      <th>MiddleLeft</th>\n",
       "      <th>MiddleMiddle</th>\n",
       "      <th>MiddleRight</th>\n",
       "      <th>BottomLeft</th>\n",
       "      <th>BottomMiddle</th>\n",
       "      <th>BottomRight</th>\n",
       "      <th>Class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>x</td>\n",
       "      <td>x</td>\n",
       "      <td>x</td>\n",
       "      <td>x</td>\n",
       "      <td>o</td>\n",
       "      <td>o</td>\n",
       "      <td>x</td>\n",
       "      <td>o</td>\n",
       "      <td>o</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>x</td>\n",
       "      <td>x</td>\n",
       "      <td>x</td>\n",
       "      <td>x</td>\n",
       "      <td>o</td>\n",
       "      <td>o</td>\n",
       "      <td>o</td>\n",
       "      <td>x</td>\n",
       "      <td>o</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>x</td>\n",
       "      <td>x</td>\n",
       "      <td>x</td>\n",
       "      <td>x</td>\n",
       "      <td>o</td>\n",
       "      <td>o</td>\n",
       "      <td>o</td>\n",
       "      <td>o</td>\n",
       "      <td>x</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>x</td>\n",
       "      <td>x</td>\n",
       "      <td>x</td>\n",
       "      <td>x</td>\n",
       "      <td>o</td>\n",
       "      <td>o</td>\n",
       "      <td>o</td>\n",
       "      <td>b</td>\n",
       "      <td>b</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>x</td>\n",
       "      <td>x</td>\n",
       "      <td>x</td>\n",
       "      <td>x</td>\n",
       "      <td>o</td>\n",
       "      <td>o</td>\n",
       "      <td>b</td>\n",
       "      <td>o</td>\n",
       "      <td>b</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  TopLeft TopMiddle TopRight MiddleLeft MiddleMiddle MiddleRight BottomLeft  \\\n",
       "0       x         x        x          x            o           o          x   \n",
       "1       x         x        x          x            o           o          o   \n",
       "2       x         x        x          x            o           o          o   \n",
       "3       x         x        x          x            o           o          o   \n",
       "4       x         x        x          x            o           o          b   \n",
       "\n",
       "  BottomMiddle BottomRight     Class  \n",
       "0            o           o  positive  \n",
       "1            x           o  positive  \n",
       "2            o           x  positive  \n",
       "3            b           b  positive  \n",
       "4            o           b  positive  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['positive', 'negative'], dtype=object)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Class'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Convert animal labels to numbers\n",
    "le = preprocessing.LabelEncoder()\n",
    "for col in df.columns:\n",
    "    df[col] = le.fit_transform(df[col])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TopLeft</th>\n",
       "      <th>TopMiddle</th>\n",
       "      <th>TopRight</th>\n",
       "      <th>MiddleLeft</th>\n",
       "      <th>MiddleMiddle</th>\n",
       "      <th>MiddleRight</th>\n",
       "      <th>BottomLeft</th>\n",
       "      <th>BottomMiddle</th>\n",
       "      <th>BottomRight</th>\n",
       "      <th>Class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   TopLeft  TopMiddle  TopRight  MiddleLeft  MiddleMiddle  MiddleRight  \\\n",
       "0        2          2         2           2             1            1   \n",
       "1        2          2         2           2             1            1   \n",
       "2        2          2         2           2             1            1   \n",
       "3        2          2         2           2             1            1   \n",
       "4        2          2         2           2             1            1   \n",
       "\n",
       "   BottomLeft  BottomMiddle  BottomRight  Class  \n",
       "0           2             1            1      1  \n",
       "1           1             2            1      1  \n",
       "2           1             1            2      1  \n",
       "3           1             0            0      1  \n",
       "4           0             1            0      1  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for col in df.columns:\n",
    "    df[col] = pd.get_dummies(df[col])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TopLeft</th>\n",
       "      <th>TopMiddle</th>\n",
       "      <th>TopRight</th>\n",
       "      <th>MiddleLeft</th>\n",
       "      <th>MiddleMiddle</th>\n",
       "      <th>MiddleRight</th>\n",
       "      <th>BottomLeft</th>\n",
       "      <th>BottomMiddle</th>\n",
       "      <th>BottomRight</th>\n",
       "      <th>Class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   TopLeft  TopMiddle  TopRight  MiddleLeft  MiddleMiddle  MiddleRight  \\\n",
       "0      0.0        0.0       0.0         0.0           0.0          0.0   \n",
       "1      0.0        0.0       0.0         0.0           0.0          0.0   \n",
       "2      0.0        0.0       0.0         0.0           0.0          0.0   \n",
       "3      0.0        0.0       0.0         0.0           0.0          0.0   \n",
       "4      0.0        0.0       0.0         0.0           0.0          0.0   \n",
       "\n",
       "   BottomLeft  BottomMiddle  BottomRight  Class  \n",
       "0         0.0           0.0          0.0    0.0  \n",
       "1         0.0           0.0          0.0    0.0  \n",
       "2         0.0           0.0          0.0    0.0  \n",
       "3         0.0           1.0          1.0    0.0  \n",
       "4         1.0           0.0          1.0    0.0  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "features=(list(df.columns[:-1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['TopLeft',\n",
       " 'TopMiddle',\n",
       " 'TopRight',\n",
       " 'MiddleLeft',\n",
       " 'MiddleMiddle',\n",
       " 'MiddleRight',\n",
       " 'BottomLeft',\n",
       " 'BottomMiddle',\n",
       " 'BottomRight']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = df[features]\n",
    "y = df['Class']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TopLeft</th>\n",
       "      <th>TopMiddle</th>\n",
       "      <th>TopRight</th>\n",
       "      <th>MiddleLeft</th>\n",
       "      <th>MiddleMiddle</th>\n",
       "      <th>MiddleRight</th>\n",
       "      <th>BottomLeft</th>\n",
       "      <th>BottomMiddle</th>\n",
       "      <th>BottomRight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   TopLeft  TopMiddle  TopRight  MiddleLeft  MiddleMiddle  MiddleRight  \\\n",
       "0      0.0        0.0       0.0         0.0           0.0          0.0   \n",
       "1      0.0        0.0       0.0         0.0           0.0          0.0   \n",
       "2      0.0        0.0       0.0         0.0           0.0          0.0   \n",
       "3      0.0        0.0       0.0         0.0           0.0          0.0   \n",
       "4      0.0        0.0       0.0         0.0           0.0          0.0   \n",
       "\n",
       "   BottomLeft  BottomMiddle  BottomRight  \n",
       "0         0.0           0.0          0.0  \n",
       "1         0.0           0.0          0.0  \n",
       "2         0.0           0.0          0.0  \n",
       "3         0.0           1.0          1.0  \n",
       "4         1.0           0.0          1.0  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# split dataset to 60% training and 40% testing\n",
    "X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, test_size=0.4, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(574, 9) (574L,)\n"
     ]
    }
   ],
   "source": [
    "print X_train.shape, y_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature importances with forests of trees\n",
    "\n",
    "This examples shows the use of forests of trees to evaluate the importance of features on an artificial classification task. The red bars are the feature importances of the forest, along with their inter-trees variability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature ranking:\n",
      "1. feature 1 - TopMiddle (0.127575) \n",
      "2. feature 5 - MiddleRight (0.125550) \n",
      "3. feature 0 - TopLeft (0.117647) \n",
      "4. feature 6 - BottomLeft (0.115633) \n",
      "5. feature 3 - MiddleLeft (0.113785) \n",
      "6. feature 8 - BottomRight (0.112243) \n",
      "7. feature 2 - TopRight (0.107885) \n",
      "8. feature 7 - BottomMiddle (0.107579) \n",
      "9. feature 4 - MiddleMiddle (0.072104) \n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAz4AAAJZCAYAAABoTeCOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xu0lWWBP/Dv5kAXRQzUuBwQvHA55g0VdMrWnC6QWmmU\njhimYySV4yqz1WVNa1J+rcmwpkbD1pDNsnIM6o9GTAEtlsfSGWTGyzjZqF0mEBC8kShUBGf//mA8\nSVyEdxOb/ZzPZ6293Jv9vvt5/B72y/6e993vW6vX6/UAAAAUrE+zJwAAAPDnpvgAAADFU3wAAIDi\nKT4AAEDxFB8AAKB4ig8AAFA8xQeAfcaHP/zh/P3f/32zpwFAgWqu4wPQ+kaNGpUnn3wyffv2Tb1e\nT61Wy2OPPZYhQ4ZUfs277ror559/fh5//PE9ONPWcdFFF2XEiBH5f//v/zV7KgDsAfb4ABSgVqvl\ntttuy7p16/L8889n3bp1DZWeJD0FqqrNmzc3NH4zdXd3N3sKAOxhig9AIXa0A3/JkiV5wxvekIED\nB2b8+PG56667ep775je/maOOOioDBgzIkUcema9//etJkg0bNuSMM87IqlWrcsABB2TAgAFZvXp1\nLrroonz2s5/tWf+uu+7KiBEjeh4fdthhufrqq3Pcccelf//+6e7uzhNPPJGzzz47r33ta3PEEUfk\nq1/96g7/H176+i++9he/+MUMHjw47e3tmT9/fhYuXJixY8fm4IMPzlVXXdWz7syZM3POOedk6tSp\nGTBgQE466aQ89NBDPc8/8sgjedOb3pSBAwfmmGOOyQ9+8IOtxr3kkkvy9re/PQcccED++Z//OTfd\ndFOuvvrqDBgwIGeddVaSZNasWTnyyCMzYMCAHH300bn55pt7XuNb3/pW3vjGN+YTn/hEBg0alCOO\nOCKLFi3qeX7t2rV5//vfn/b29hx00EF597vf3fPcrbfemvHjx2fgwIE59dRT89///d89z82aNSvD\nhw/PgAED0tHRkTvvvHOH+QGwY4oPQMFWrVqVd7zjHfnsZz+btWvX5ktf+lLe85735JlnnkmSDB48\nOAsWLMi6detyww035GMf+1gefPDB7Lffflm4cGGGDRv2snuQ/nSv0Lx587Jw4cL85je/Sa1Wyzvf\n+c6MHz8+TzzxRBYvXpxrrrkmP/zhD3dp/qtXr87GjRuzatWqzJw5MxdffHFuuummPPDAA/nxj3+c\nz33uc1m2bFnP8rfcckvOPffcrF27Nuedd17e9a53ZfPmzdm0aVPe+c535rTTTstTTz2Va6+9NtOm\nTcvPf/7znnXnzp2bv/u7v8vzzz+fCy64INOmTcsnP/nJrFu3LvPnz0+SHHnkkbnnnnuybt26XHHF\nFTn//POzZs2antdYunRpOjo68swzz+QTn/hEpk+f3vPc+eefn9/+9rf5n//5nzz55JP52Mc+liR5\n4IEHMn369Fx//fV59tln88EPfjBnnnlm/vCHP+Sxxx7Lddddl/vuuy/r1q3L7bffnlGjRu1SdgBs\nTfEBKMS73vWuDBo0KIMGDerZm/Av//Ivefvb3563ve1tSZK3vOUtOemkk7JgwYIkyemnn97zQfqN\nb3xjJk+enJ/85CcNzeOjH/1ohg0blle+8pX5j//4jzz99NP5zGc+k7a2towaNSof+MAHMm/evF16\nrVe84hX527/927S1tWXq1Kl5+umnc9lll2W//fbLUUcdlaOOOir/9V//1bP8iSeemClTpqStrS2X\nX355fv/732fJkiVZsmRJ1q9fn0996lPp27dv3vSmN+Ud73hH5s6d27PuWWedlVNOOSVJ8spXvnK7\n83nPe96TwYMHJ0nOOeecjB49OkuXLu15fuTIkXn/+9+fWq2WCy+8ME888USefPLJrF69Orfffnvm\nzJmTAQMGpK2tLW984xuTJNdff30+9KEP5aSTTkqtVsv73ve+vPKVr8ySJUvS1taWjRs35qc//Wk2\nbdqUQw89NIcddtju/UAASKL4ABRj/vz5efbZZ/Pss8/m+9//fpJk2bJl+d73vtdTiAYOHJh77rkn\nTzzxRJJk4cKF+Yu/+IscdNBBGThwYBYuXJinn366oXkMHz685/6yZcuycuXKrca/6qqr8uSTT+7S\nax100EE9e5Re/epXJ0le+9rX9jz/6le/Oi+88ELP45cedler1dLe3p5Vq1Zl1apVWz2XbCkpK1eu\n3O66O/Ltb3+755C0gQMH5uGHH94qr5fuFXtxvi+88EIef/zxDBo0KAMGDNjmNZctW5Z/+Id/2Cqj\nFStWZNWqVTniiCPyj//4j7nyyiszePDgvPe97+352QGwexQfgEJs7zs+I0aMyAUXXNBTiNauXZvn\nn38+n/zkJ7Nx48acffbZ+eQnP5mnnnoqa9euzemnn97zOts7scH++++fDRs29Dze3ofwl643YsSI\nHH744VuN/9xzz231/Zo96aVnoKvX61mxYkWGDRuWYcOGZfny5Vstu3z58rS3t2933tt7vHz58syY\nMSNf+9rXsnbt2qxduzave93rdvjdqpcaMWJEnn322axbt267z33mM5/ZKqMXXngh5557bpJk6tSp\n+clPftJzSN+nP/3plx0PgG0pPgAFO//88/ODH/wgd9xxR7q7u/O73/0ud911V1atWpWNGzdm48aN\nOfjgg9OnT58sXLgwd9xxR8+6gwcPzjPPPLPVh/Xjjz8+CxYsyNq1a7N69epcc801Ox1/4sSJOeCA\nA3L11Vfnd7/7XTZv3pyHH344//mf//ln+f+97777cvPNN2fz5s35yle+kle96lU55ZRTcvLJJ2f/\n/ffP1VdfnU2bNqWrqyu33nprzjvvvB2+1uDBg/OrX/2q5/H69evTp0+fHHzwwenu7s4NN9yQn/70\np7s0ryFDhuT000/PJZdckt/85jfZtGlTzyGFF198cf7pn/6p55C59evXZ8GCBVm/fn0ee+yx3Hnn\nndm4cWNe8YpX5NWvfnX69PFPN0AVtp4ABdjRaaeHDx+e+fPn5/Of/3wOOeSQjBw5Ml/60pfS3d2d\n/v3759prr80555yTQYMGZd68eT1nL0uSsWPH5rzzzsvhhx+eQYMGZfXq1Xnf+96XY489NqNGjcpp\np52WqVOn7nQeffr0ya233poHH3wwhx12WF772tfm4osv3u6ejyr/n3/6+Kyzzsp3v/vdDBw4MDfd\ndFP+9V//NW1tbenXr19+8IMfZMGCBTn44INz6aWX5sYbb8zo0aN3mN/06dPz8MMP93xnqqOjI5df\nfnlOOeWUDBkyJA8//HBOPfXUXZ7vjTfemL59+2bcuHEZPHhwT2k88cQTc/311+fSSy/NoEGDMmbM\nmHzrW99Kkvz+97/Ppz/96RxyyCEZNmxYnnrqqa3OZAfArtsjFzBdtGhRLrvssnR3d2f69On51Kc+\ntdXz3/nOdzJr1qwkyQEHHJCvfe1rOfbYY5NsuejegQcemD59+qRfv35bfUkUAHbVzJkz88tf/jLf\n/va3mz0VAPZBfRt9ge7u7lx66aVZvHhxhg0blgkTJuSss87KuHHjepY5/PDD8+Mf/zgHHnhgFi1a\nlBkzZmTJkiVJtvw2sKurKwMHDmx0KgAAANvV8KFuS5cuzejRozNy5Mj069cvU6dO7bnewYtOOeWU\nHHjggT33X3oWnXq97grZAADAn1XDe3xWrly51SlAhw8fvtPD1b7xjW/k9NNP73lcq9UyadKktLW1\nZcaMGbn44osbnRIAvdAVV1zR7CkAsA9ruPjsjjvvvDM33HBD7r777p4/u+eeezJ06NA89dRTmTRp\nUjo6Ol72y6IAAAC7o+Hi097evtW1EVasWLHVdRFe9NBDD2XGjBlZtGjRVt/nGTp0aJLkkEMOyZQp\nU7J06dLtFp8dnbEIAADgRTs6d1vD3/GZMGFCfvGLX2TZsmXZuHFj5s2blzPPPHOrZZYvX573vOc9\nufHGG3PEEUf0/PmGDRt6rri9fv363HHHHTn66KN3+j/htnu3K664oulzaMWb3OQmt9a4yU5uctv3\nb3KT29687UzDe3za2toye/bsTJ48ued01h0dHZkzZ05qtVpmzJiRz33uc3n22WdzySWXpF6v95y2\nes2aNZkyZUpqtVo2bdqUadOmZfLkyY1OCQAAYCt75Ds+p512Wh599NGt/uyDH/xgz/3rr78+119/\n/TbrHXbYYXnwwQf3xBQAAAB2qOFD3di3dXZ2NnsKLUlu1citGrlVJ7tq5FaN3KqRWzVy2/Nq9Zc7\nGG4fUavVXva4PQAAoPfaWWewxwcAACie4gMAABRP8QEAAIqn+AAAAMVTfAAAgOIpPgAAQPEUHwAA\noHiKDwAAUDzFBwAAKJ7iAwAAFE/xAQAAiqf4AAAAxVN8AACA4ik+AABA8RQfAACgeIoPAABQPMUH\nAAAonuIDAAAUT/EBAACKp/gAAADFU3wAAIDiKT4AAEDxFB8AAKB4ig8AAFA8xQcAACie4gMAABRP\n8QEAAIqn+AAAAMVTfAAAgOIpPgAAQPEUHwAAoHiKDwAAUDzFBwAAKF7fZk+APa+ra8vtxfudnVvu\nd3b+8T4AAPQmtXq9Xm/2JHZFrVZLi0x1n1KrJWIDAKA32FlncKgbAABQPMUHAAAonuIDAAAUT/EB\nAACKp/gAAADFU3wAAIDiKT4AAEDxFB8AAKB4ig8AAFA8xQcAACie4gMAABRP8QEAAIqn+AAAAMXr\n2+wJwL6iq2vL7cX7nZ1b7nd2/vE+AACtqVav1+vNnsSuqNVqaZGp7lNqtURsu09uAACtZ2edwaFu\nAABA8RQfAACgeIoPAABQPMUHAAAonuIDAAAUT/EBAACKp/gAAADFU3wAAIDiKT4AAEDxFB8AAKB4\nig8AAFA8xQcAACie4gMAABRP8QEAAIqn+AAAAMVTfAAAgOIpPgAAQPEUHwAAoHiKDwAAUDzFBwAA\nKJ7iAwAAFE/xAQAAite32RMAgN3R1bXl9uL9zs4t9zs7/3gfAP5UrV6v15s9iV1Rq9XSIlPdp9Rq\nidh2n9ygNXivAvBSO+sMDnUDAACKp/gAAADFU3wAAIDiKT4AAEDxFB8AAKB4ig8AAFA8xQcAACie\n4gMAABRvjxSfRYsWZdy4cRkzZkxmzZq1zfPf+c53ctxxx+W4447LqaeemoceemiX1wUAoLV0dSVX\nXrnl1tn5x/tdXU2cFL1erb6jS5vuou7u7owZMyaLFy/OsGHDMmHChMybNy/jxo3rWWbJkiXp6OjI\ngQcemEWLFuXKK6/MkiVLdmndnonu5Cqs7JirmlcjN2gN3quw7/M+ZW/aWWfo2+iLL126NKNHj87I\nkSOTJFOnTs38+fO3Ki+nnHLKVvdXrly5y+sC+7aurj/+Bq+ra8tv9pIt/33xPtuSGwDsXQ0Xn5Ur\nV2bEiBE9j4cPH56lS5fucPlvfOMbOf300yutC+x7XvpBvVZzGMOukhsA7F0NF5/dceedd+aGG27I\n3XffvTeHBQAAermGi097e3uWL1/e83jFihVpb2/fZrmHHnooM2bMyKJFizJw4MDdWvdFV155Zc/9\nzs7OdDoeBAAAeq2urq507eJhEw2f3GDz5s0ZO3ZsFi9enKFDh2bixImZO3duOjo6epZZvnx53vKW\nt+TGG2/c6vs+u7Juz0Sd3KASXyisRm7VyK0auVUnO9j3eZ+yN/1ZT27Q1taW2bNnZ/Lkyenu7s70\n6dPT0dGROXPmpFarZcaMGfnc5z6XZ599Npdccknq9Xr69euXpUuX7nBdAACAPanhPT57iz0+1fgt\nSzVyq0Zu1citOtnBvs/7lL1pZ51hj1zAFAAAYF+2V8/qBgA0h2tHAb2dQ90KZ/dyNXKrRm7VyK06\n2VUjt12nMDbO3zf2pp11BsWncDY21citGrlVI7fqZFeN3KqRWzVyY2/yHR8AAKBXU3wAAIDiKT4A\nAEDxFB8AAKB4ig8AAFA8xQcAACie4gMAABRP8QEAAIqn+AAAAMVTfAAAgOIpPgAAQPEUHwAAoHiK\nDwAAUDzFBwAAKJ7iAwAAFE/xAQAAiqf4AAAAxVN8AACA4ik+AABA8RQfAACgeIoPAABQPMUHAAAo\nnuIDAAAUT/EBAACKp/gAAADFU3wAAIDiKT4AAEDxFB8AAKB4ig8AAFA8xQcAACie4gMAABRP8QEA\nAIqn+AAAAMVTfAAAgOIpPgAAQPEUHwAAoHiKDwAAUDzFBwAAKJ7iAwAAFE/xAQAAiqf4AAAAxVN8\nAACA4ik+AABA8RQfAACgeIoPAABQPMUHAAAonuIDAAAUT/EBAACKp/gAAADFU3wAAIDiKT4AAEDx\nFB8AAKB4ig8AAFA8xQcAACie4gMAABRP8QEAAIqn+AAAAMVTfAAAgOIpPgAAQPEUHwAAoHiKDwAA\nUDzFBwAAKJ7iAwAAFE/xAQAAiqf4AAAAxVN8AACA4ik+AABA8RQfAACgeIoPAABQPMUHAAAonuID\nAAAUT/EBAACKp/gAAADFU3wAAIDiKT4AAEDxFB8AAKB4ig8AAFA8xQcAACie4gMAABRP8QEAAIq3\nR4rPokWLMm7cuIwZMyazZs3a5vlHH300r3/96/OqV70qX/7yl7d6btSoUTnuuOMyfvz4TJw4cU9M\nBwAAYCt9G32B7u7uXHrppVm8eHGGDRuWCRMm5Kyzzsq4ceN6ljnooIPy1a9+NTfffPM26/fp0ydd\nXV0ZOHBgo1MBAADYrob3+CxdujSjR4/OyJEj069fv0ydOjXz58/fapmDDz44J554Yvr23bZn1ev1\ndHd3NzoNAACAHWp4j8/KlSszYsSInsfDhw/P0qVLd3n9Wq2WSZMmpa2tLTNmzMjFF1/c6JQAAKDl\ndHVtub14v7Nzy/3Ozj/ep7qGi0+j7rnnngwdOjRPPfVUJk2alI6Ojpx66qnNnhYAAOxVLy04tdof\nSxB7RsPFp729PcuXL+95vGLFirS3t+/y+kOHDk2SHHLIIZkyZUqWLl26w+Jz5ZVX9tzv7OxMp+oL\nAAC9VldXV7p2sSHW6vV6vZHBNm/enLFjx2bx4sUZOnRoJk6cmLlz56ajo2ObZWfOnJn+/fvn4x//\neJJkw4YN6e7uTv/+/bN+/fpMnjw5V1xxRSZPnrztRGu1NDjVXqlWS8S2++RWjdyqkVt1sqtGbtXI\nrRq5VSO3anbWGRre49PW1pbZs2dn8uTJ6e7uzvTp09PR0ZE5c+akVqtlxowZWbNmTU466aQ8//zz\n6dOnT6655pr87Gc/y1NPPZUpU6akVqtl06ZNmTZt2nZLDwAAQCMa3uOzt9jjU43fFlQjt2rkVo3c\nqpNdNXKrRm7VyK0auVWzs86wRy5gCgAAsC9TfAAAgOIpPgAAQPEUHwAAoHiKDwAAUDzFBwAAKJ7i\nAwAAFE/xAQAAiqf4AAAAxVN8AACA4ik+AABA8RQfAACgeIoPAABQPMUHAAAonuIDAAAUT/EBAACK\np/gAAADFU3wAAIDiKT4AAEDxFB8AAKB4ig8AAFA8xQcAACie4gMAABRP8QEAAIqn+AAAAMVTfAAA\ngOIpPgAAQPEUHwAAoHiKDwAAUDzFBwAAKJ7iAwAAFE/xAQAAiqf4AAAAxVN8AACA4ik+AABA8RQf\nAACgeIoPAABQPMUHAAAoXt9mT6A3GDVkSJatWdOk0eup1WpNGXnk4MH59erVTRkbAABeqlav1+vN\nnsSuqNVqaZGpbqNWq6VZM6+lnnqaU3xqSQv/zJIWnXpTya0auVUnu2rkVo3cqpFbNXKrZmedwaFu\nAABA8Rzqxj7LIYIAAOwpDnXbCxzqVnF9ubUcu+WrkVt1sqtGbtXIrRq5VSO3ahzqBgAA9GqKDwAA\nUDzFBwAAKJ7iAwAAFE/xAQAAiqf4AAAAxVN8AACA4ik+AABA8RQfAACgeIoPAABQPMUHAAAonuID\nAAAUT/EBAACK17fZEwD2rFFDhmTZmjVNGr2eWq3WpLGTkYMH59erVzdtfABg31Wr1+v1Zk9iV9Rq\ntbTIVLdRq9XSrJnXUk89zfkgWksa+pnJreL6vTS3LeM3ll2z1GpJC057nyC7auRWjdyqkVs1cqtm\nZ53BoW4AAEDxFB8AAKB4vuMDEN+N8t0oAErnOz57QW/9zoXvqlQdW27Vx6+endxadfvqGPgq5FaN\n3KqRWzVyq8Z3fAAAgF7NoW4AVNbcQwSTZh4m6BBBgNbiULe9oLceQuOQrapjy636+A51qzZ+a+aW\ntPZ7tZkcQlON3KqRWzVyq8ahbgAAQK/mUDcA2MscIugQQWDvc6jbXtBbD6FxyFbVseVWffzWPGRL\nbtW16ntVbq3677lDj6qQWzVyq8ahbgAAQK+m+AAAAMVTfAAAgOIpPgAAQPEUHwAAoHiKDwAAUDzX\n8QEAWkZzr4Hk+kfQylzHZy/ordcHcT2aqmPLrfr4rXldFblV16rvVbnZxu3+2I3l1uzCmCbl1sqF\n0XV8qtlZZ1B89gIb6Yrry63a+r00ty3jt+YHUblV16rvVbnZxu3+2HKrNrYL5vY2LmAKAAD0aooP\nAABQPMUHAAAonuIDAAAUT/EBAACKp/gAAADFU3wAAIDi7ZHis2jRoowbNy5jxozJrFmztnn+0Ucf\nzetf//q86lWvype//OXdWhcAAKBRDV/AtLu7O2PGjMnixYszbNiwTJgwIfPmzcu4ceN6lnn66aez\nbNmy3HzzzRk4cGAuv/zyXV63Z6IuYFpt7Ba+aJjcKq7fS3PbMn5rXlBSbtW16ntVbrZxuz+23KqN\n7QKmvc2f9QKmS5cuzejRozNy5Mj069cvU6dOzfz587da5uCDD86JJ56Yvn377va6AAAAjWq4+Kxc\nuTIjRozoeTx8+PCsXLnyz74uAADArnJyAwAAoHh9X36RnWtvb8/y5ct7Hq9YsSLt7e1/lnWvvPLK\nnvudnZ3p7Ozc7fkCAABl6OrqSldX1y4t2/DJDTZv3pyxY8dm8eLFGTp0aCZOnJi5c+emo6Njm2Vn\nzpyZ/v375+Mf//hur+vkBhXHbuEvFMqt4vq9NLct47fml83lVl2rvlflZhu3+2PLrdrYTm7Q2+ys\nMzS8x6etrS2zZ8/O5MmT093dnenTp6ejoyNz5sxJrVbLjBkzsmbNmpx00kl5/vnn06dPn1xzzTX5\n2c9+lv79+293XQAAgD2p4T0+e4s9PhXHbuHfssit4vq9NLct47fmb+DlVl2rvlflZhu3+2PLrdrY\n9vj0Nn/W01kDAADs6xQfAACgeIoPAABQPMUHAAAonuIDAAAUT/EBAACKp/gAAADFU3wAAIDiKT4A\nAEDxFB8AAKB4ig8AAFA8xQcAACie4gMAABRP8QEAAIqn+AAAAMVTfAAAgOIpPgAAQPEUHwAAoHiK\nDwAAUDzFBwAAKJ7iAwAAFE/xAQAAiqf4AAAAxVN8AACA4ik+AABA8RQfAACgeIoPAABQPMUHAAAo\nnuIDAAAUT/EBAACKp/gAAADFU3wAAIDiKT4AAEDxFB8AAKB4ig8AAFA8xQcAACie4gMAABRP8QEA\nAIqn+AAAAMVTfAAAgOIpPgAAQPEUHwAAoHiKDwAAUDzFBwAAKJ7iAwAAFE/xAQAAiqf4AAAAxVN8\nAACA4vVt9gQAAGBfNGrIkCxbs6ZJo9dTq9WaMvLIwYPz69WrmzL2n1OtXq/Xmz2JXVGr1dIiU91G\nrVZLs2ZeSz31NOdNU0sa+pnJreL6vTS3LeNXz05urZdb0rrvVbnZxu3+2HKrNrbcqo3dWG7NtLPO\n4FA3AACgeIoPAABQPMUHAAAonuIDAAAUT/EBAACKp/gAAADFU3wAAIDiKT4AAEDxFB8AAKB4ig8A\nAFA8xQcAACie4gMAABRP8QEAAIqn+AAAAMVTfAAAgOIpPgAAQPEUHwAAoHiKDwAAUDzFBwAAKJ7i\nAwAAFE/xAQAAiqf4AAAAxVN8AACA4ik+AABA8RQfAACgeIoPAABQPMUHAAAonuIDAAAUT/EBAACK\np/gAAADFU3wAAIDiKT4AAEDxFB8AAKB4ig8AAFC8PVJ8Fi1alHHjxmXMmDGZNWvWdpf5yEc+ktGj\nR+f444/PAw880PPno0aNynHHHZfx48dn4sSJe2I6AAAAW+nb6At0d3fn0ksvzeLFizNs2LBMmDAh\nZ511VsaNG9ezzMKFC/PLX/4yP//5z3Pvvffmwx/+cJYsWZIk6dOnT7q6ujJw4MBGpwIAALBdDe/x\nWbp0aUaPHp2RI0emX79+mTp1aubPn7/VMvPnz88FF1yQJDn55JPz3HPPZc2aNUmSer2e7u7uRqcB\nAACwQw0Xn5UrV2bEiBE9j4cPH56VK1fudJn29vaeZWq1WiZNmpQJEybk+uuvb3Q6AAAA22j4ULdG\n3XPPPRk6dGieeuqpTJo0KR0dHTn11FObPS0AAKAgDRef9vb2LF++vOfxihUr0t7evs0yjz/++HaX\nGTp0aJLkkEMOyZQpU7J06dIdFp8rr7yy535nZ2c6OzsbnT4AANCiurq60tXVtUvL1ur1er2RwTZv\n3pyxY8dm8eLFGTp0aCZOnJi5c+emo6OjZ5kFCxbkuuuuy2233ZYlS5bksssuy5IlS7Jhw4Z0d3en\nf//+Wb9+fSZPnpwrrrgikydP3naitVoanGrT1Gq1NGvmtdRTT61JY6ehn5ncKq7fS3PbMn717OTW\nerklrftelZtt3O6PLbdqY8ut2tiN5dZMO+sMDe/xaWtry+zZszN58uR0d3dn+vTp6ejoyJw5c1Kr\n1TJjxoycccYZWbBgQY488sjsv//+ueGGG5Ika9asyZQpU1Kr1bJp06ZMmzZtu6UHAACgEQ3v8dlb\n7PGpOHYL/7ZAbhXX76W5bRm/NX8DL7fqWvW9KjfbuN0fW27VxpZbtbHL3OOzRy5gCgAAsC9TfAAA\ngOIpPgAAQPEUHwAAoHiKDwAAUDzFBwAAKJ7iAwAAFE/xAQAAiqf4AAAAxVN8AACA4ik+AABA8RQf\nAACgeIoPAABQPMUHAAAonuIDAAAUT/EBAACKp/gAAADFU3wAAIDiKT4AAEDxFB8AAKB4ig8AAFA8\nxQcAACgvzQ+mAAAQG0lEQVSe4gMAABRP8QEAAIqn+AAAAMVTfAAAgOLV6vV6vdmT2BW1Wi0tMtVt\n1Gq1NGvmtdRTT61JY6ehn5ncKq6/l3Pryl+mK53/d78znelKknSmK525ay/OpLHseuvfty3jt2Zu\nSeu+V+XWOtu4rcaWW7Wx5VZt7BbOrZl21hkUn73Am6bi+nKrtn6TP1A1U6t+EFV8qmvV96rcWnMb\nJ7eKY8ut2tgtnFsz7awzONQNAAAonuIDAAAUT/EBAACKp/gAAADFU3wAAIDiOavbXuCMIBXX76Wn\nZW613PYlrXqWLWd1q65Vt3Fya81tnNwqji23amO3cG7N5HTWTeZNU3H9XvoBXm7VtdIH0X2laCet\nlds247foNk5urbmNk1vFseVWbewWzq2ZFJ8m86apuH4v/QAvt+pa+YNoM7Vybq26jZNba27j5FZx\nbLlVG7uFc2sm1/EBAAB6NcUHAAAonkPd9gK7SSuu30sPPZJbda186FEztXJurbqNa0Zu+8r3ylp5\nG9eqf98SuVVeX24tx3d8msybpuL6vfSDqNyqa7UPovuKVs6tVbdxzc6tmVp5G9eqf98SuVVeX24t\nR/Fpst56tqhW3tg0k9yq80G0mlbOrVU/GDQ7t2Zq5W1cq/59S+RWeX25tRzFp8l66z9wrbyxaSa5\nVeeDaDWtnFurfjBodm7N1MrbuFb9+5bIrfL6cms5zuoGAAD0aooPAABQPMUHAAAonuIDAAAUT/EB\nAACKp/gAAADFU3wAAIDiKT4AAEDxFB8AAKB4ig8AAFC8Wr1erzd7EruiVqulRaa6jVqtltaceWNq\nSUM/M7lVXL+X5pY0lp3cWie3rvxlutL5f/c705muJElnutKZu/baPFott31FK2/jaqmnnlqTxpZb\ntbHlVm3sxnJrpp11BsVnL+it/8C18sammeRWnQ+i1citGrlV02rbuBKKduIDfOX15dZyFJ8m663/\nwLXyxqaZ5FadD6LVyK0auVVjG1dNK+fWyh/g5dZ6dtYZfMcHAAAonuIDAAAUT/EBAACKp/gAAADF\nU3wAAIDiKT4AAEDxFB8AAKB4ig8AAFA8xQcAACie4gMAABSvVq/X682exK6o1Wppkaluo1arpTVn\n3pha0tDPTG4V1++luSWNZSc3ue0uuVVjG1dNq+XWlb9MVzr/735nOtOVJOlMVzpz116bR6vlttXY\nqaeeWpPGbiy3ZtpZZ1B89gIb6Yrry63a+r00t8QH0arkVo3cqrGNq0Zu1bRybopPNTvrDA51AwAA\niqf4AAAAxVN8AACA4ik+AABA8RQfAACgeIoPAABQPMUHAAAonuIDAAAUT/EBAACKp/gAAADFU3wA\nAIDiKT4AAEDxFB8AAKB4ig8AAFA8xQcAACie4gMAABRP8QEAAIq3R4rPokWLMm7cuIwZMyazZs3a\n7jIf+chHMnr06Bx//PF58MEHd2tdAACARjRcfLq7u3PppZfm9ttvz8MPP5y5c+fmkUce2WqZhQsX\n5pe//GV+/vOfZ86cOfnQhz60y+sCAAA0quHis3Tp0owePTojR45Mv379MnXq1MyfP3+rZebPn58L\nLrggSXLyySfnueeey5o1a3ZpXQAAgEY1XHxWrlyZESNG9DwePnx4Vq5cuUvL7Mq6AAAAjWrKyQ3q\n9XozhgUAAHqpvo2+QHt7e5YvX97zeMWKFWlvb99mmccff3ybZTZu3Piy677UlVde2XO/s7MznZ2d\njU5/rxg5eHBqa9Y0exp73cjBgxteX27V1u+NuSWNZSe36uvKrdq6cqu+fm/MTm7VtHputSaN22hu\ne1NXV1e6urp2adlavcHdL5s3b87YsWOzePHiDB06NBMnTszcuXPT0dHRs8yCBQty3XXX5bbbbsuS\nJUty2WWXZcmSJbu0bs9EazV7igAA6BVqtcRH3923s87Q8B6ftra2zJ49O5MnT053d3emT5+ejo6O\nzJkzJ7VaLTNmzMgZZ5yRBQsW5Mgjj8z++++fG264YafrAgAA7EkN7/HZW+zxAQCgt7DHp5qddYam\nnNwAAABgb1J8AACA4ik+AABA8RQfAACgeIoPAABQPMUHAAAonuIDAAAUT/EBAACKp/gAAADFq9V3\ndGnTfczOrsIKAACtrqtry+3F+52dW+53dv7xPju3s86g+AAAAEXYWWdwqBsAAFA8xQcAACie4gMA\nABRP8QEAAIqn+AAAAMVTfAAAgOIpPgAAQPEUHwAAoHiKDwAAUDzFBwAAKJ7iAwAAFE/xAQAAiqf4\nAAAAxVN8AACA4ik+AABA8RQfAACgeIoPAABQPMUHAAAonuIDAAAUT/EBAACKp/gAAADFU3wAAIDi\nKT4AAEDxFB8AAKB4ig8AAFA8xQcAACie4gMAABRP8QEAAIqn+AAAAMVTfAAAgOIpPgAAQPEUHwAA\noHiKDwAAUDzFBwAAKJ7iAwAAFE/xAQAAiqf4AAAAxVN8AACA4ik+AABA8RQfAACgeIoPAABQPMUH\nAAAonuIDAAAUT/EBAACKp/gAAADFU3wAAIDiKT4AAEDxFB8AAKB4ig8AAFA8xQcAACie4gMAABRP\n8QEAAIqn+AAAAMVTfAAAgOIpPgAAQPEUHwAAoHiKDwAAUDzFBwAAKJ7iAwAAFE/xAQAAiqf4AAAA\nxVN8AACA4ik+AABA8RQfAACgeIoPAABQPMUHAAAonuIDAAAUT/EBAACKp/gAAADFU3wAAIDiKT4A\nAEDxFB8AAKB4ig8AAFC8horP2rVrM3ny5IwdOzZve9vb8txzz213uUWLFmXcuHEZM2ZMZs2a1fPn\nM2fOzPDhw3PCCSfkhBNOyKJFixqZDgAAwHY1VHy+8IUv5K1vfWseffTRvPnNb85VV121zTLd3d25\n9NJLc/vtt+fhhx/O3Llz88gjj/Q8f/nll+f+++/P/fffn9NOO62R6bAdXV1dzZ5CS5JbNXKrRm7V\nya4auVUjt2rkVo3c9ryGis/8+fNz4YUXJkkuvPDC3Hzzzdsss3Tp0owePTojR45Mv379MnXq1Myf\nP7/n+Xq93sgUeBneNNXIrRq5VSO36mRXjdyqkVs1cqtGbnteQ8XnySefzODBg5MkQ4YMyZNPPrnN\nMitXrsyIESN6Hg8fPjwrV67seTx79uwcf/zx+cAHPrDDQ+UAAAAa8bLFZ9KkSTn22GN7bsccc0yO\nPfbY3HLLLdssW6vVdmvwSy65JL/61a/y4IMPZsiQIbn88st3a30AAIBdUm/AuHHj6qtXr67X6/X6\nE088UR83btw2y/z7v/97/W1ve1vP46uuuqr+hS98YZvlfv3rX9ePOeaYHY6VxM3Nzc3Nzc3Nzc3N\nbae3HembBpx55pn55je/mU996lP51re+lbPOOmubZSZMmJBf/OIXWbZsWYYOHZp58+Zl7ty5SZLV\nq1dnyJAhSZLvf//7Ofroo3c4Vt13gQAAgIpq9QYaxbPPPpu/+qu/yuOPP56RI0fme9/7Xl7zmtfk\niSeeyMUXX5xbb701yZbTWX/0ox9Nd3d3pk+fnk9/+tNJkgsuuCAPPvhg+vTpk1GjRmXOnDk93xkC\nAADYUxoqPgAAAK2gobO6se+aPn16Bg8enGOPPbbZU2k5o0aNynHHHZfx48dn4sSJzZ5Oy9jRhYrZ\nueeeey7nnHNOOjo68rrXvS733ntvs6e0z/v973+fk08+OePHj88xxxyTmTNnNntKLeMrX/lKjj76\n6Bx77LGZNm1aNm7c2Owp7fNWrFiRN7/5zXnd616XY445Jtdee22zp9QyHnvssYwfPz4nnHBCxo8f\nnwMPPFB+u6i7uzsnnHBCzjzzzGZPpSj2+BTq7rvvTv/+/XPBBRfkoYceavZ0Wsrhhx+e++67LwMH\nDmz2VFpGd3d3xowZk8WLF2fYsGGZMGFC5s2bl3HjxjV7avu8v/7rv85f/uVf5qKLLsqmTZuyYcOG\nDBgwoNnT2udt2LAh++23XzZv3pw3vOENufbaa/2i4mWsWrUqp556ah555JG84hWvyLnnnpu3v/3t\nueCCC5o9tX3a6tWrs3r16hx//PF54YUXcuKJJ2b+/Pm2b7upu7s7w4cPz7333rvVZU7Yvq985Su5\n7777sm7duu2eSZlq7PEp1KmnnuqDe0X1ej3d3d3NnkZLebkLFbN969aty09+8pNcdNFFSZK+ffsq\nPbtov/32S7Jl78+mTZt2+3IKvdXmzZuzfv36npI9bNiwZk9pnzdkyJAcf/zxSZL+/funo6Njq+sR\nsmt+9KMf5YgjjlB6dsGKFSuyYMGCfOADH2j2VIqj+MCfqNVqmTRpUiZMmJDrr7++2dNpCS93oWK2\n73//939z8MEH56KLLsoJJ5yQGTNm5Le//W2zp9USuru7M378+AwZMqTn/crODRs2LB//+Mdz6KGH\npr29Pa95zWvy1re+tdnTaim//vWv8+CDD+bkk09u9lRazne/+92cd955zZ5GS/jYxz6WL37xi36h\n82eg+MCfuOeee3L//fdnwYIFue6663L33Xc3e0oUatOmTbn//vvzN3/zN7n//vuz33775Qtf+EKz\np9US+vTpkwceeCArVqzIvffem5/97GfNntI+7ze/+U3mz5+fZcuWZdWqVXnhhRfyne98p9nTahkv\nvPBCzj777FxzzTXp379/s6fTUv7whz/klltuyTnnnNPsqezzbrvttgwePDjHH3986vW6y7nsYYoP\n/ImhQ4cmSQ455JBMmTIlS5cubfKM9n3t7e1Zvnx5z+MVK1akvb29iTNqDcOHD8+IESNy0kknJUnO\nPvvs3H///U2eVWsZMGBA3vSmN2XRokXNnso+70c/+lEOP/zwDBo0KG1tbXn3u9+df/u3f2v2tFrC\npk2bcvbZZ+d973vfdq9ZyM4tXLgwJ554Yg455JBmT2Wfd8899+SWW27J4YcfnvPOOy933nmn7+Ht\nQYpPwfymYPdt2LAhL7zwQpJk/fr1ueOOO3Z6YV22eOmFijdu3Jh58+Y5E80uGDx4cEaMGJHHHnss\nSbJ48eIcddRRTZ7Vvu/pp5/Oc889lyT57W9/mx/+8Ie+aL4LDj300CxZsiS/+93vUq/Xs3jx4nR0\ndDR7Wi3h/e9/f4466qh89KMfbfZUWtLcuXMd5raLPv/5z2f58uX51a9+lXnz5uXNb35zvv3tbzd7\nWsXo2+wJ8Ofx3ve+N11dXXnmmWdy6KGHZubMmT1foGbH1qxZkylTpqRWq2XTpk2ZNm1aJk+e3Oxp\n7fPa2toye/bsTJ48uedCxT5Q7Zprr70206ZNyx/+8IccfvjhueGGG5o9pX3eE088kQsvvDDd3d3p\n7u7OueeemzPOOKPZ09rnTZw4MWeffXbGjx+ffv36Zfz48ZkxY0azp7XPu+eee3LTTTflmGOOyfjx\n41Or1fL5z38+p512WrOn1hI2bNiQH/3oR/n617/e7KmA01kDAADlc6gbAABQPMUHAAAonuIDAAAU\nT/EBAACKp/gAAADFU3wAAIDiKT4AAEDxFB8AAKB4/x+oVutb62Y1KQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x9dfb588>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "\n",
    "# Build a classification task using 3 informative features\n",
    "\n",
    "# Build a forest and compute the feature importances\n",
    "forest = ExtraTreesClassifier(n_estimators=250,\n",
    "                              random_state=0)\n",
    "\n",
    "forest.fit(X, y)\n",
    "importances = forest.feature_importances_\n",
    "std = np.std([tree.feature_importances_ for tree in forest.estimators_],\n",
    "             axis=0)\n",
    "indices = np.argsort(importances)[::-1]\n",
    "\n",
    "# Print the feature ranking\n",
    "print(\"Feature ranking:\")\n",
    "\n",
    "for f in range(X.shape[1]):\n",
    "    print(\"%d. feature %d - %s (%f) \" % (f + 1, indices[f], features[indices[f]], importances[indices[f]]))\n",
    "\n",
    "# Plot the feature importances of the forest\n",
    "plt.figure(num=None, figsize=(14, 10), dpi=80, facecolor='w', edgecolor='k')\n",
    "plt.title(\"Feature importances\")\n",
    "plt.bar(range(X.shape[1]), importances[indices],\n",
    "       color=\"r\", yerr=std[indices], align=\"center\")\n",
    "plt.xticks(range(X.shape[1]), indices)\n",
    "plt.xlim([-1, X.shape[1]])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.12757459,  0.12554975,  0.11764683,  0.11563324,  0.11378505])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "importances[indices[:5]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. feature 1 - TopMiddle (0.127575)\n",
      "2. feature 5 - MiddleRight (0.125550)\n",
      "3. feature 0 - TopLeft (0.117647)\n",
      "4. feature 6 - BottomLeft (0.115633)\n",
      "5. feature 3 - MiddleLeft (0.113785)\n"
     ]
    }
   ],
   "source": [
    "for f in range(5):\n",
    "    print(\"%d. feature %d - %s (%f)\" % (f + 1, indices[f], features[indices[f]] ,importances[indices[f]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "best_features = []\n",
    "for i in indices[:5]:\n",
    "    best_features.append(features[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAecAAAF6CAYAAAAj9ZDJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt8VOWdx/HvSQioECBczc2AEEhAyIUEUgQaqkRBhKai\nhCIuypJaFt0uVLBaK2C1G2xdqciWRhfEC9T1QhaFSJcyCLwa4xUEVhCUAOEuKZcgCck8+wfllJiQ\nnHCbJ/B5v155vebMPM85z/zmTL5zLnPGMcYYAQAAawQFegAAAKAqwhkAAMsQzgAAWIZwBgDAMoQz\nAACWIZwBALAM4QxcZn7605/qySefDPQwAJwHh+85A6d06NBB+/fvV6NGjWSMkeM42rJli6699tpz\nnueqVat09913a+fOnRdwpA3Hvffeq+joaM2YMSPQQwEaFLacgb9zHEfvvvuujhw5oqNHj+rIkSPn\nFcyS3JA/V5WVlee1/EDy+/2BHgLQYBHOwBnOtiOpoKBAN954o8LCwpSUlKRVq1a5j82fP1/dunVT\n8+bN1blzZ/3xj3+UJB0/flxDhgzR7t27FRoaqubNm2vv3r2699579atf/crtv2rVKkVHR7vTHTt2\n1MyZM5WQkKBmzZrJ7/drz549GjFihNq1a6dOnTrpueeeO+tzOHP+p+f99NNPq3379oqMjFReXp6W\nLVumrl27qk2bNvrNb37j9p0+fbruvPNOZWVlqXnz5kpJSdH69evdx7/44gsNHDhQYWFh6tGjh5Ys\nWVJluRMmTNBtt92m0NBQvfjii3r11Vc1c+ZMNW/eXMOHD5ck5eTkqHPnzmrevLluuOEGLV682J3H\nSy+9pP79++uhhx5Sq1at1KlTJ+Xn57uPl5SU6L777lNkZKRat26tH/3oR+5j77zzjpKSkhQWFqZ+\n/frp888/dx/LyclRVFSUmjdvrvj4eK1cufKs9QOsYAAYY4zp0KGDWbFiRbX7i4uLTevWrU1+fr4x\nxpj//d//Na1btzYHDx40xhizdOlS8/XXXxtjjHn//ffNNddcYz799FNjjDE+n89ER0dXmd/YsWPN\nY4895k5/t02HDh1MUlKSKS4uNidOnDB+v9/06tXL/PrXvzYVFRXm66+/Np06dTLLly+v8XmcOX+f\nz2caNWrk9s3NzTVt27Y1o0ePNqWlpWbjxo3m6quvNtu3bzfGGDNt2jTTuHFj89Zbb5mKigrz29/+\n1nTs2NFUVFSYkydPms6dO5t///d/NydPnjR/+ctfTGhoqNmyZYu73JYtW5q//vWvxhhjTpw4Ue25\nGmPMG2+8Yfbu3WuMMeb11183TZs2dafnz59vGjdubF588UXj9/vNf/7nf5qIiAi375AhQ0xWVpY5\nfPiwqaioMO+//74xxphPPvnEtGvXznz44YfG7/ebBQsWmA4dOpjy8nKzefNmEx0d7S6jqKjIfPXV\nVzXWDrAFW87AGX74wx+qVatWatWqlbtV9sorr+i2227TLbfcIkm66aablJKSoqVLl0qSBg8erA4d\nOkiS+vfvr4yMDK1evfq8xvGv//qvioiIUJMmTfThhx/q4MGDevTRRxUcHKwOHTron//5n7Vo0SJP\n82rcuLEeeeQRBQcHKysrSwcPHtTPfvYzXXPNNerWrZu6deumdevWue179eqlzMxMBQcHa9KkSSor\nK1NBQYEKCgpUWlqqqVOnqlGjRho4cKCGDh2qhQsXun2HDx+utLQ0SVKTJk1qHM8dd9yh9u3bS5Lu\nvPNOxcbGqrCw0H08JiZG9913nxzH0T/90z9pz5492r9/v/bu3av33ntPc+fOVfPmzRUcHKz+/ftL\nknJzc3X//fcrJSVFjuNozJgxatKkiQoKChQcHKzy8nJt2LBBFRUVuu6669SxY8f6vSDAJUY4A2fI\ny8vToUOHdOjQIb311luSpKKiIr3++utuaIeFhWnt2rXas2ePJGnZsmX63ve+p9atWyssLEzLli3T\nwYMHz2scUVFR7u2ioiIVFxdXWf5vfvMb7d+/39O8Wrdu7R73vvrqqyVJ7dq1cx+/+uqrdezYMXf6\nzF3sjuMoMjJSu3fv1u7du6s8Jp0K0uLi4hr7ns2CBQvc3c9hYWHauHFjlXqdeZz/9HiPHTumnTt3\nqlWrVmrevHm1eRYVFel3v/tdlRrt2rVLu3fvVqdOnfTss89q2rRpat++vX784x+7rx1gK8IZOIOp\n4ZhzdHS07rnnHje0S0pKdPToUU2ZMkXl5eUaMWKEpkyZogMHDqikpESDBw9251PTyWBNmzbV8ePH\n3emaguLMftHR0br++uurLP/w4cNVjvdeSGeeWW6M0a5duxQREaGIiAjt2LGjStsdO3YoMjKyxnHX\nNL1jxw5lZ2drzpw5KikpUUlJibp3737WY/1nio6O1qFDh3TkyJEaH3v00Uer1OjYsWMaOXKkJCkr\nK0urV69WUVGRJOnhhx+uc3lAIBHOQB3uvvtuLVmyRMuXL5ff79eJEye0atUq7d69W+Xl5SovL1eb\nNm0UFBSkZcuWafny5W7f9u3b65tvvqkSKImJiVq6dKlKSkq0d+9ezZo1q9bl9+7dW6GhoZo5c6ZO\nnDihyspKbdy4UR999NFFeb4ff/yxFi9erMrKSv3Hf/yHrrrqKqWlpalPnz5q2rSpZs6cqYqKCvl8\nPr3zzjsaNWrUWefVvn17ffXVV+50aWmpgoKC1KZNG/n9fs2bN08bNmzwNK5rr71WgwcP1oQJE/S3\nv/1NFRUV7uGD8ePH6w9/+IO7e7y0tFRLly5VaWmptmzZopUrV6q8vFyNGzfW1VdfraAg/vXBbqyh\nwN+d7StPUVFRysvL01NPPaW2bdsqJiZGv/3tb+X3+9WsWTP9/ve/15133qlWrVpp0aJF7lnJktS1\na1eNGjVK119/vVq1aqW9e/dqzJgx6tmzpzp06KBbb71VWVlZtY4jKChI77zzjj777DN17NhR7dq1\n0/jx42vcgjyX5/nd6eHDh+tPf/qTwsLC9Oqrr+rtt99WcHCwQkJCtGTJEi1dulRt2rTRxIkT9fLL\nLys2Nvas9Rs3bpw2btzoHsOPj4/XpEmTlJaWpmuvvVYbN25Uv379PI/35ZdfVqNGjRQXF6f27du7\nH2x69eql3NxcTZw4Ua1atVKXLl300ksvSZLKysr08MMPq23btoqIiNCBAweqnKEO2MjTRUjy8/P1\ns5/9TH6/X+PGjdPUqVOrPP7aa68pJydHkhQaGqo5c+aoZ8+ekk5d2KFFixYKCgpSSEhIlRM/ANhl\n+vTp2rZtmxYsWBDooQBXtEZ1NfD7/Zo4caJWrFihiIgIpaamavjw4YqLi3PbXH/99Xr//ffVokUL\n5efnKzs7WwUFBZJOfer3+XwKCwu7eM8CAIDLSJ27tQsLCxUbG6uYmBiFhIQoKytLeXl5VdqkpaWp\nRYsW7u0zz940xnClIAAA6qHOLefi4uIqX4+Iioqqddf0Cy+8oMGDB7vTjuNo0KBBCg4OVnZ2tsaP\nH3+eQwZwsTz++OOBHgIAeQjn+li5cqXmzZunNWvWuPetXbtW4eHhOnDggAYNGqT4+Pg6TwABAOBK\nVmc4R0ZGVvlu465du6p8r/G09evXKzs7W/n5+VWOL4eHh0uS2rZtq8zMTBUWFtYYzufz4wAAADRE\nZz0nu67re1ZUVJhOnTqZ7du3m7KyMpOQkGA2bdpUpU1RUZHp3Lmze03d00pLS83Ro0eNMcYcO3bM\n9O3b17z33ns1LsfDUGCMefzxxwM9hAaBOnlHrbyhTt5RK29qy706t5yDg4M1e/ZsZWRkuF+lio+P\n19y5c+U4jrKzs/XEE0/o0KFDmjBhgowx7lem9u3bp8zMTDmOo4qKCo0ePVoZGRkX9mMHAACXGU/H\nnG+99VZt3ry5yn0/+clP3Nu5ubnKzc2t1q9jx4767LPPznOIAABcWbhCWAOTnp4e6CE0CNTJO2rl\nDXXyjlqdP09XCLsUHMfxdPF7AAAuB7XlHlvOAABYhnAGAMAyhDMAAJYhnAEAsAzhDACAZQhnAAAs\nQzgDAGAZwhkAAMsQzgAAWIZwBgDAMoQzAACWIZwBALAM4QwAgGUIZwAALEM4AwBgGcIZAADLEM4A\nAFiGcAYAwDKEMwAAliGcAQCwDOEMAIBlCGcAACxDOAMAYJlGgR4A6ubznfo7fTs9/dTt9PR/3AYA\nXD4cY4wJ9CAkyXEcWTIUqzmORJkAoOGrLffYrQ0AgGUIZwAALEM4AwBgGcIZAADLEM4AAFiGcAYA\nwDKEMwAAliGcAQCwDFcIw2WDK6kBuFxwhbAGhiuEeUOdANiOK4QBANCAEM4AAFiGcAYAwDKEMwAA\nliGcAQCwDOEMAIBlCGcAACxDOAMAYBnCGQAAyxDOAABYhnAGAMAyhDMAAJYhnAEAsAzhDACAZQhn\nAAAs0yjQAwAANHw+36m/07fT00/dTk//x21455iz/dLzJVbbj07jHxxHokx1o05A4PD+86a23GO3\nNgAAliGcAQCwDOEMAIBlCGcAACzjKZzz8/MVFxenLl26KCcnp9rjr732mhISEpSQkKB+/fpp/fr1\nnvsCAICq6jxb2+/3q0uXLlqxYoUiIiKUmpqqRYsWKS4uzm1TUFCg+Ph4tWjRQvn5+Zo2bZoKCgo8\n9XUHwtnannAWpDfUCQgc3n/enNfZ2oWFhYqNjVVMTIxCQkKUlZWlvLy8Km3S0tLUokUL93ZxcbHn\nvgAAoKo6L0JSXFys6OhodzoqKkqFhYVnbf/CCy9o8ODB59QXwKXBBSO8oU4IlAt6hbCVK1dq3rx5\nWrNmzYWcLYAL7MxwcZx/BBCqok4IlDrDOTIyUjt27HCnd+3apcjIyGrt1q9fr+zsbOXn5yssLKxe\nfU+bNm2aezs9PV3pfDQFAFwmfD6ffB4/4dV5QlhlZaW6du2qFStWKDw8XL1799bChQsVHx/vttmx\nY4duuukmvfzyy0pLS6tXX3cgnBDmCSdaeEOdvKNW3lAn76iVN7XlXp1bzsHBwZo9e7YyMjLk9/s1\nbtw4xcfHa+7cuXIcR9nZ2XriiSd06NAhTZgwQcYYhYSEqLCw8Kx9AQDA2fHDFw0Mn0i9oU7eUStv\nqJN31MobfvgCAIAGhHAGAMAyhDMAAJYhnAEAsAzhDACAZQhnAAAsQzgDAGAZwhkAAMsQzgAAWIZw\nBgDAMoQzAACWIZwBALAM4QwAgGUIZwAALEM4AwBgGcIZAADLEM4AAFiGcAYAwDKEMwAAliGcAQCw\nDOEMAIBlCGcAACxDOAMAYBnCGQAAyxDOAABYhnAGAMAyhDMAAJYhnAEAsAzhDACAZQhnAAAsQzgD\nAGAZwhkAAMsQzgAAWIZwBgDAMoQzAACWIZwBALAM4QwAgGUIZwAALEM4AwBgGcIZAADLEM4AAFiG\ncAYAwDKEMwAAliGcAQCwDOEMAIBlCGcAACxDOAMAYBnCGQAAyxDOAABYhnAGAMAyhDMAAJYhnAEA\nsAzhDACAZQhnAAAsQzgDAGAZwhkAAMsQzgAAWIZwBgDAMp7COT8/X3FxcerSpYtycnKqPb5582b1\n7dtXV111lZ555pkqj3Xo0EEJCQlKSkpS7969L8yoAQC4jDWqq4Hf79fEiRO1YsUKRUREKDU1VcOH\nD1dcXJzbpnXr1nruuee0ePHiav2DgoLk8/kUFhZ2YUcOAMBlqs4t58LCQsXGxiomJkYhISHKyspS\nXl5elTZt2rRRr1691KhR9aw3xsjv91+4EQMAcJmrM5yLi4sVHR3tTkdFRam4uNjzAhzH0aBBg5Sa\nmqrc3NxzGyUAAFeQOndrn6+1a9cqPDxcBw4c0KBBgxQfH69+/fpd7MUCANBg1RnOkZGR2rFjhzu9\na9cuRUZGel5AeHi4JKlt27bKzMxUYWHhWcN52rRp7u309HSlp6d7Xg4AADbz+Xzy+Xye2jrGGFNb\ng8rKSnXt2lUrVqxQeHi4evfurYULFyo+Pr5a2+nTp6tZs2aaPHmyJOn48ePy+/1q1qyZSktLlZGR\noccff1wZGRnVB+I4qmMokOQ4EmWqG3Xyjlp5Q528o1be1JZ7dW45BwcHa/bs2crIyJDf79e4ceMU\nHx+vuXPnynEcZWdna9++fUpJSdHRo0cVFBSkWbNmadOmTTpw4IAyMzPlOI4qKio0evToGoMZAAD8\nQ51bzpcKW87e8InUG+rkHbXyhjp5R628qS33uEIYAACWIZwBALAM4QwAgGUIZwAALEM4AwBgGcIZ\nAADLEM4AAFiGcAYAwDKEMwAAliGcAQCwDOEMAIBlCGcAACxDOAMAYBnCGQAAyxDOAABYhnAGAMAy\nhDMAAJYhnAEAsAzhDACAZQhnAAAsQzgDAGAZwhkAAMsQzgAAWIZwBgDAMoQzAACWIZwBALBMo0AP\noKHqcO21Ktq3LwBLNnIc55IvNaZ9e23fu/eSLxcArkSOMcYEehCS5DiOLBmKJ47jKBCjdWRkdOnD\n2ZEa2OsjNaDhBhS18oY6eUetvKkt99itDQCAZditjYsuMIcA2P0PoOFit/Y5Yrd2PfoGoFYNsU6B\nwi5Ib6iTd9TKG3ZrAwDQgBDOAABYhnAGAMAyhDMAAJYhnAEAsAzhDACAZQhnAAAsQzgDAGAZrhAG\nWIIfUwFwGlcIO0dcIawefblCmLe+rFNW46pX3lErb7hCGAAADQi7tQE0KOz+x5WA3drniF2Q9ejL\nbm1vfVmnvPWjTtZjt7Y3teUeW84AcJm6kvYyXG57GNhyPkd8eq9HX7acvfVlnfLWjzp573sF1aph\n7mHghDAAABoMwhkAAMsQzgAAWIZwBgDAMoQzAACWIZwBALAM4QwAgGUIZwAALEM4AwBgGcIZAADL\nEM4AAFiGcAYAwDKewjk/P19xcXHq0qWLcnJyqj2+efNm9e3bV1dddZWeeeaZevUFAABV1fmrVH6/\nX126dNGKFSsUERGh1NRULVq0SHFxcW6bgwcPqqioSIsXL1ZYWJgmTZrkua87EH6Vytty+WUcb8uk\nTt6X28BqRZ3q0fcKqtUV96tUhYWFio2NVUxMjEJCQpSVlaW8vLwqbdq0aaNevXqpUaNG9e4LAACq\nqjOci4uLFR0d7U5HRUWpuLjY08zPpy8AAFcqTggDAMAyjepqEBkZqR07drjTu3btUmRkpKeZ17fv\ntGnT3Nvp6elKT0/3tBwAAGzn8/nk8/k8ta3zhLDKykp17dpVK1asUHh4uHr37q2FCxcqPj6+Wtvp\n06erWbNmmjx5cr37ckKYx+VyUoq3ZVIn78ttYLWiTvXoewXV6nI7IazOLefg4GDNnj1bGRkZ8vv9\nGjdunOLj4zV37lw5jqPs7Gzt27dPKSkpOnr0qIKCgjRr1ixt2rRJzZo1q7EvAAA4uzq3nC8Vtpw9\nLpdP796WSZ28L7eB1Yo61aPvFVSry23LmRPCAACwDOEMAIBlCGcAACxDOAMAYBnCGQAAyxDOAABY\nhnAGAMAyhDMAAJYhnAEAsAzhDACAZQhnAAAsQzgDAGAZwhkAAMsQzgAAWIZwBgDAMoQzAACWIZwB\nALAM4QwAgGUIZwAALEM4AwBgGcIZAADLEM4AAFiGcAYAwDKEMwAAliGcAQCwDOEMAIBlCGcAACxD\nOAMAYBnCGQAAyxDOAABYhnAGAMAyhDMAAJYhnAEAsAzhDACAZQhnAAAsQzgDAGAZwhkAAMsQzgAA\nWIZwBgDAMoQzAACWIZwBALAM4QwAgGUIZwAALEM4AwBgGcIZAADLEM4AAFiGcAYAwDKEMwAAliGc\nAQCwDOEMAIBlCGcAACxDOAMAYBnCGQAAyxDOAABYhnAGAMAyhDMAAJYhnAEAsAzhDACAZTyFc35+\nvuLi4tSlSxfl5OTU2ObBBx9UbGysEhMT9emnn7r3d+jQQQkJCUpKSlLv3r0vzKgBALiMNaqrgd/v\n18SJE7VixQpFREQoNTVVw4cPV1xcnNtm2bJl2rZtm7788kt98MEH+ulPf6qCggJJUlBQkHw+n8LC\nwi7eswAA4DJS55ZzYWGhYmNjFRMTo5CQEGVlZSkvL69Km7y8PN1zzz2SpD59+ujw4cPat2+fJMkY\nI7/ffxGGDgDA5anOcC4uLlZ0dLQ7HRUVpeLi4lrbREZGum0cx9GgQYOUmpqq3NzcCzVuAAAuW3Xu\n1j5fa9euVXh4uA4cOKBBgwYpPj5e/fr1u9iLBQCgwaoznCMjI7Vjxw53eteuXYqMjKzWZufOnTW2\nCQ8PlyS1bdtWmZmZKiwsPGs4T5s2zb2dnp6u9PR0z08EAACb+Xw++Xw+b41NHSoqKkynTp3M9u3b\nTVlZmUlISDCbNm2q0ubdd981Q4YMMcYY89e//tX06dPHGGNMaWmpOXr0qDHGmGPHjpm+ffua9957\nr8bleBiKVSQZE4C/U0fxA7Hcc399AlEr6nT51oo6UasLXadAqW3MdW45BwcHa/bs2crIyJDf79e4\nceMUHx+vuXPnynEcZWdna8iQIVq6dKk6d+6spk2bat68eZKkffv2KTMzU47jqKKiQqNHj1ZGRkZ9\nP2wAAHBFcf6e3gHnOI4sGYonjuMoEKN1ZGTkBGC5OufXJxC1ok71WG4DqxV1qkffK6hW51OnQKkt\n97hCGAAAliGcAQCwDOEMAIBlCGcAACxDOAMAYBnCGQAAyxDOAABYhnAGAMAyhDMAAJYhnAEAsAzh\nDACAZQhnAAAsQzgDAGAZwhkAAMsQzgAAWIZwBgDAMoQzAACWIZwBALCMY4wxgR6EJDmOI0uG4onj\nOArEaB0ZGTkBWK7O+fUJRK2oU+18+r58Sv/77XSlyydJSpdP6Vp1ScZwrrXivVePvldQrc6nToFS\nW+4RzufoSlrpTy23Yf2DoE72I5y9LrfhrVOEsze15R67tQEAsAzhDACAZQhnAAAswzHnc3QlHcs5\ntVz7j3s15JOcJI45e+7HiXPe+15B/6cut2POhPM5upJW+lPLbXj/IAKBOnnXEMLZBg1xnSKcveGE\nMAAAGhDCGQAAy7Bb+xxx3Ksefa+g3ZDUyTt2a3vTENcpdmt7wzHni4B/EPXoewXVijp5Rzh70xDX\nKcLZG445AwDQgBDOAABYhnAGAMAyhDMAAJYhnAEAsAzhDACAZQhnAAAsw/eczxHftaxH3yuoVtTJ\nO77n7E1DWacCfbGky+17zoTzOeIfRD36XkG1ok7eEc7esE55c7mFM7u1AQCwDOEMAIBlCGcAACxD\nOAMAYBnCGQAAyxDOAABYhnAGAMAyhDMAAJYhnAEAsAzhDACAZQhnAAAsQzgDAGAZwhkAAMsQzgAA\nWIZwBgDAMoQzAACWIZwBALAM4QwAgGUIZwAALEM4AwBgGU/hnJ+fr7i4OHXp0kU5OTk1tnnwwQcV\nGxurxMREffbZZ/XqCwAAzmDqUFlZaTp16mS2b99uysvLTUJCgvm///u/Km2WLl1qhgwZYowxpqCg\nwPTp08dz39M8DMUqkowJwN/KAC33fF6fQNSKOl2+taJO1OpC1ylQahtznVvOhYWFio2NVUxMjEJC\nQpSVlaW8vLwqbfLy8nTPPfdIkvr06aPDhw9r3759nvqifnyBHkAD4Qv0ABoQX6AH0ED4Aj2ABsQX\n6AFcBuoM5+LiYkVHR7vTUVFRKi4u9tTGS18AAFDVRTkh7NTWOgAAOBeN6moQGRmpHTt2uNO7du1S\nZGRktTY7d+6s1qa8vLzOvmdyHKdegw+0QI12eoCWez6vTyBqRZ28a2i1ok716HsBx1EfgahVQ8uQ\n2tQZzqmpqdq6dauKiooUHh6uRYsWaeHChVXaDBs2TM8//7xGjhypgoICtWzZUu3bt1ebNm3q7Hsa\nW9sAAJxSZzgHBwdr9uzZysjIkN/v17hx4xQfH6+5c+fKcRxlZ2dryJAhWrp0qTp37qymTZtq3rx5\ntfYFAABn5xg2WQEAsApXCLtADh06pKSkJCUnJys8PFxRUVHudEVFRb3m1a9fP3Xq1KnKfUOHDlVY\nWJikU8fuR40aVWPf/v37a/369dXuf/HFFzVp0iRJ0mOPPabf//739RrT+QoKCnK/bidJlZWVatu2\nrYYNGyZJWrJkiWbOnFlj39DQ0Brvv/fee/XWW29JkgYOHKhPPvmk1jEMHDhQcXFxSkxMVJ8+farU\naejQoTpy5Eid/Wtaxrp167Rs2bJa+15oF3J9O9s6czaTJk1Sjx499Mgjj+jtt9/Wli1b6jv8CyY4\nOFjJyclKTExUSkqKCgoKam1fVFRU5dDahX7tvKyHZ3rooYfUo0cPTZ06VXl5efriiy8u2FhqYsv7\n0OYa2aLO3drwplWrVvr0008lSTNmzFCzZs3cMKwvx3EUGhqqwsJC9e7dWyUlJdq/f797skNUVNRZ\nj93bqmnTptqwYYPKysrUpEkT/fnPf67yNbvbb79dt99+e419L+RJHgsXLlRSUpLmz5+vn//851q+\nfLkk6Z133jnneX722Wf66KOPNHjw4As1zDpdyPWtPowxmjdvnkpKSiRJY8aMUVBQkLp06XLRl12T\npk2buv/oly9frocfflg+n++s7b/++mu99tpr7ofbQLx2Z8rNzVVJSYkcx9G9996roUOHKi4u7qIt\nz5b3YX1c6hrZgi3ni+C7RwpmzpypHj16qGfPnpo9e7Ykadu2bbrhhhs0atQodevWTVlZWSorK3P7\nZGVluQH8xhtvaMSIEe5j27ZtU1JSkiTp22+/1V133aXu3btrxIgRVebxwgsvqGvXrkpLSzvrFsXW\nrVt16623KjU1Venp6dq6deuFKUINhgwZonfffVfSqZA8c+v/pZde0gMPPCBJ2r59u/r27auEhAQ9\n9thjVeYxceJExcfHKyMjQ/v3769xOX/+85/Vt29fpaSkaOTIkTp+/Lj72OnX5nvf+552797t3t+x\nY0cdOnRIkvTEE08oLi5OAwYM0I9//GM988wzbrvXX39dffr0UVxcnNauXauTJ0/qV7/6lV5//XUl\nJyfrv//7v8+nROfkQqxv31VZWanJkycrLS1NiYmJ+q//+i9Jp/YwHDt2TMnJyfr1r3+tpUuXatKk\nSUpOTq5spJImAAAIhUlEQVTyzYxL5cznfvjwYbVq1cqdPr3FlZCQ4L4uv/jFL7RmzRolJydr5syZ\n1V67kpISZWZmKiEhQX379tWGDRskSdOnT9fYsWM1YMAAdezYUW+//bamTp2qnj17asiQIaqsrDzr\nGP1+v6ZMmaI+ffooMTFRubm5kqThw4fr2LFj6tWrl2bMmKH/+Z//0ZQpU5ScnKyvv/76YpRLkh3v\nw++yrUZWuPgXKLvyTJs2zfzud78zxhjzwQcfmMTERFNWVmaOHj1q4uPjzYYNG8zWrVuN4zimsLDQ\nGGPMPffcY2bNmmWMMaZfv37mo48+MgkJCcbv95ubbrrJFBUVmbCwMGOMMVu3bjVJSUnGGGNmzpxp\nfvKTnxhjjPn0009No0aNzLp168yuXbtMTEyMKSkpMSdPnjRpaWnm3/7t34wxxvzyl790lzVw4EDz\n1VdfGWOMWbt2rcnIyLgoNQkNDTWff/65GTFihDlx4oRJTEw0q1atMrfffrsxxpj58+ebBx54wBhj\nzLBhw8wrr7xijDHm+eefN6GhocYYY9588013fLt37zYtW7Y0b775pjHGmPT0dPPxxx+bgwcPmgED\nBpjjx48bY4zJyckxTzzxRJU2xhjz7LPPmkcffdQdX8eOHc0333xjPvzwQ5OUlGTKy8vN0aNHTWxs\nrPtapqenm5///OfGmFOXrL355purjT0QLsT6tm7duirznDNnjsnJyTHGGFNWVmaSkpLMzp07TUVF\nhbseGmPM3XffbfLy8i7F06xRcHCwSUpKMnFxcaZly5bmk08+McZUXVf27dtnrrvuOrN3717j8/nc\ndc6Y6q/dAw88YGbMmGGMMeYvf/mLSUxMNMacqnH//v1NZWWlWbdunbnmmmvMe++9Z4wxJjMz063B\nmevYaX/84x/Nk08+aYw5VcuUlBSzfft2Y4xx121jjBk7dqy7Pl8str0PT7OpRrZgt/ZFtmbNGt1x\nxx1q3LixGjdurB/+8IdavXq1Bg0apOuvv16pqamSpLvvvlu5ubl68MEHJUkhISFKS0vTokWL5Pf7\nFRkZWePXzd5//31NnTpVkpSYmKju3btLkgoKCnTzzTerZcuWkqS77rqrynfRpVNbGgUFBbrjjjvc\nefv9/otTCEk33HCDtm/froULF+q2224769fn1q5d6x7DGjNmjB5++GFJ0urVq91P+eHh4frBD35Q\nrW9BQYE2bdqkG2+8UcYYnTx5Un379nUfHz16tMrKylRaWlrlB1rOXPbw4cMVEhKikJCQarv4fvSj\nH0mSevXqpaKionOowsV1ruvbdy1fvlxffPGFu/fmyJEj+vLLLxUeHm7V1x6vueYad7d2QUGBxowZ\now0bNmjNmjXuutKuXTulp6frww8/POtx09PWrFlT5fjpoUOHdOzYMUnS4MGDFRQUpB49esjv9ysj\nI0OS1KNHD23fvv2s81y+fLk+//xzd+v9dC1jYmICUksb3offZVuNbEA4W+S7x3RGjhypO++8U089\n9VSNj9fkzBW5rpXaGKO2bdvW6+SM8zVs2DA99NBD8vl8OnjwYI1tHMdxn2t935jGGGVkZOjVV1+t\n8fHXXntNSUlJmjJliiZOnKg333yzXvNv0qSJpFMnItX3xCvb1LY+GWM0Z84cDRw4sMr9lZWV1l7o\nIS0tTQcPHqxxvfK6HtX23E6/9o7jKCQkxL0/KCio1nXBGKPnnntOgwYNqtfyLqZAvw9ram9bjQKN\nY84XWf/+/fX222+rrKxMx44dU15envr37y/p1MkpH3/8saRToXH6/tPS09P1i1/8QnfddZekmt8g\nAwYMcN8A69at08aNGyWd+ke1cuVK/e1vf1N5ebneeOONan1btmyp8PBwLV682J1/fc7arY/TY7/v\nvvv0+OOPu1v4NbnxxhvdLbYz39wDBgzQn/70J/n9fu3Zs0crV66s1jctLU1r167Vtm3bJEnHjx/X\nl19+WW0cM2bM0AcffOCeaXz6/htvvFFLlixxX6/aThQ73Sc0NLTOM70vlXNd3767bt1yyy16/vnn\n3WOpW7ZscY9Rn9k20M/9zLF88cUX8vv9at26tfr37++uKwcOHNDq1avVu3fvauP97nT//v31yiuv\nSJJ8Pp/atGmjZs2a1brcuh675ZZbNGfOHDfAv/zyS3377bfV2l6KWtr2PjzNphrZgnC+yFJTUzVq\n1CilpKSob9+++pd/+Rf3DREfH69nnnlG3bp107fffqvx48dL+scnRcdxNHnyZHfXdE2fICdOnKhv\nvvlG3bt315NPPqnk5GRJpy6p+stf/lJ9+vTR97///bO+CRcuXKg//OEPSkxM1A033OCeKHKhnR57\nZGSkJk6cWGvbZ599Vs8//7wSEhK0Z88e9/7MzEx17txZ3bt319ixY6vsJjs9/zZt2mj+/PkaNWqU\ne1LP5s2bq7SRpKuuukqTJ0/W008/XeWxlJQUDRs2TAkJCbrtttvUs2dPtWjRolr/M6cHDhyoTZs2\nBeyEsDOdy/omSbfeequuu+46XXfddRo9erTuv/9+9/fZe/bsqQkTJrj/OM+sw6hRo/TUU08F7ISw\nEydOKDk5WUlJSRo1apQWLFggx3GUmZmpnj17KiEhQTfffLOefvpptWvXTj179lRwcLCSkpI0a9as\naq/dtGnT9PHHHyshIUGPPPKIFixYUONya9uaGzp0qFvLkSNHavz48erWrZuSk5PVo0cP3X///TXW\nMisrS08//bR69ep10U52suF9KNldI1twEZIA2bZtm0aMGOF+HQb2KC0tVdOmTfXtt99qwIABys3N\nVWJiYqCHdV5Y34CGhWPOAXSlHkuxXXZ2tjZt2qSysjKNHTu2wQfzaaxvQMPBljMAAJbhmDMAAJYh\nnAEAsAzhDACAZQhnAAAsQzgDAGAZwhkAAMv8P3JUW438ZuTYAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x8e89ef0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the top 5 feature importances of the forest\n",
    "plt.figure(num=None, figsize=(8, 6), dpi=80, facecolor='w', edgecolor='k')\n",
    "plt.title(\"Feature importances\")\n",
    "plt.bar(range(5), importances[indices][:5], \n",
    "       color=\"r\",  yerr=std[indices][:5], align=\"center\")\n",
    "plt.xticks(range(5), best_features)\n",
    "plt.xlim([-1, 5])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Decision Tree accuracy and time elapsed caculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DecisionTree\n",
      "Acurracy:  0.690104166667\n",
      "time elapsed:  0.0\n"
     ]
    }
   ],
   "source": [
    "t0=time()\n",
    "print \"DecisionTree\"\n",
    "\n",
    "dt = DecisionTreeClassifier(min_samples_split=20,random_state=99)\n",
    "# dt = DecisionTreeClassifier(min_samples_split=20,max_depth=5,random_state=99)\n",
    "\n",
    "clf_dt=dt.fit(X_train,y_train)\n",
    "\n",
    "print \"Acurracy: \", clf_dt.score(X_test,y_test)\n",
    "t1=time()\n",
    "print \"time elapsed: \", t1-t0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## cross validation for DT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cross result========\n",
      "[ 0.603125    0.646875    0.36792453]\n",
      "0.539308176101\n",
      "time elapsed:  0.0350000858307\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tt0=time()\n",
    "print \"cross result========\"\n",
    "scores = cross_validation.cross_val_score(dt, X, y, cv=3)\n",
    "print scores\n",
    "print scores.mean()\n",
    "tt1=time()\n",
    "print \"time elapsed: \", tt1-tt0\n",
    "print \"\\n\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Tuning our hyperparameters using GridSearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  34 tasks      | elapsed:   17.3s\n",
      "[Parallel(n_jobs=-1)]: Done  81 out of  81 | elapsed:   18.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 27 candidates, totalling 81 fits\n",
      "Best score: 0.714\n",
      "Best parameters set:\n",
      "\tclf__max_depth: 25\n",
      "\tclf__min_samples_leaf: 1\n",
      "\tclf__min_samples_split: 1\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        0.0       0.87      0.87      0.87       253\n",
      "        1.0       0.76      0.76      0.76       131\n",
      "\n",
      "avg / total       0.83      0.83      0.83       384\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('clf', DecisionTreeClassifier(criterion='entropy'))\n",
    "])\n",
    "\n",
    "parameters = {\n",
    "    'clf__max_depth': (5, 25 , 50),\n",
    "    'clf__min_samples_split': (1, 5, 10),\n",
    "    'clf__min_samples_leaf': (1, 2, 3)\n",
    "}\n",
    "\n",
    "grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=1, scoring='f1')\n",
    "grid_search.fit(X_train, y_train)\n",
    "\n",
    "print 'Best score: %0.3f' % grid_search.best_score_\n",
    "print 'Best parameters set:'\n",
    "\n",
    "best_parameters = grid_search.best_estimator_.get_params()\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print '\\t%s: %r' % (param_name, best_parameters[param_name])\n",
    "\n",
    "predictions = grid_search.predict(X_test)\n",
    "\n",
    "\n",
    "print classification_report(y_test, predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DecisionTree\n",
      "Acurracy:  0.833333333333\n",
      "time elapsed:  0.00399994850159\n"
     ]
    }
   ],
   "source": [
    "t0=time()\n",
    "print \"DecisionTree\"\n",
    "\n",
    "dt1 = DecisionTreeClassifier(min_samples_split=1,max_depth=25,min_samples_leaf=1, random_state=99)\n",
    "\n",
    "clf_dt1=dt1.fit(X_train,y_train)\n",
    "\n",
    "print \"Acurracy: \", clf_dt1.score(X_test,y_test)\n",
    "t1=time()\n",
    "print \"time elapsed: \", t1-t0\n",
    "\n",
    "export_graphviz(clf_dt1, \n",
    "                out_file='tree.dot', \n",
    "                feature_names=features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we have installed GraphViz on our computer, we can convert the tree.dot file into a PNG file by executing the following command from the command line in the location where we saved the tree.dot file:\n",
    "\n",
    "> dot -Tpng tree.dot -o tree.png"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Random Forest accuracy and time elapsed caculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RandomForest\n",
      "Acurracy:  0.822916666667\n",
      "time elapsed:  1.14800000191\n"
     ]
    }
   ],
   "source": [
    "t2=time()\n",
    "print \"RandomForest\"\n",
    "rf = RandomForestClassifier(n_estimators=100,n_jobs=-1)\n",
    "clf_rf = rf.fit(X_train,y_train)\n",
    "print \"Acurracy: \", clf_rf.score(X_test,y_test)\n",
    "t3=time()\n",
    "print \"time elapsed: \", t3-t2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## cross validation for RF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cross result========\n",
      "[ 0.890625    0.890625    0.27987421]\n",
      "0.687041404612\n",
      "time elapsed:  3.69499993324\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tt2=time()\n",
    "print \"cross result========\"\n",
    "scores = cross_validation.cross_val_score(rf, X, y, cv=3)\n",
    "print scores\n",
    "print scores.mean()\n",
    "tt3=time()\n",
    "print \"time elapsed: \", tt3-tt2\n",
    "print \"\\n\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tuning Models using GridSearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  34 tasks      | elapsed:   19.8s\n",
      "[Parallel(n_jobs=-1)]: Done 184 tasks      | elapsed:   28.4s\n",
      "[Parallel(n_jobs=-1)]: Done 324 out of 324 | elapsed:   36.3s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 108 candidates, totalling 324 fits\n",
      "Best score: 0.810\n",
      "Best parameters set:\n",
      "\tclf__max_depth: 50\n",
      "\tclf__min_samples_leaf: 1\n",
      "\tclf__min_samples_split: 1\n",
      "\tclf__n_estimators: 50\n",
      "Accuracy: 0.825520833333\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        0.0       0.87      0.86      0.87       253\n",
      "        1.0       0.74      0.76      0.75       131\n",
      "\n",
      "avg / total       0.83      0.83      0.83       384\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "pipeline2 = Pipeline([\n",
    "('clf', RandomForestClassifier(criterion='entropy'))\n",
    "])\n",
    "\n",
    "parameters = {\n",
    "    'clf__n_estimators': (5, 25, 50, 100),\n",
    "    'clf__max_depth': (5, 25 , 50),\n",
    "    'clf__min_samples_split': (1, 5, 10),\n",
    "    'clf__min_samples_leaf': (1, 2, 3)\n",
    "}\n",
    "\n",
    "grid_search = GridSearchCV(pipeline2, parameters, n_jobs=-1, verbose=1, scoring='accuracy', cv=3)\n",
    "\n",
    "grid_search.fit(X_train, y_train)\n",
    "\n",
    "print 'Best score: %0.3f' % grid_search.best_score_\n",
    "\n",
    "print 'Best parameters set:'\n",
    "best_parameters = grid_search.best_estimator_.get_params()\n",
    "\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print '\\t%s: %r' % (param_name, best_parameters[param_name])\n",
    "\n",
    "predictions = grid_search.predict(X_test)\n",
    "print 'Accuracy:', accuracy_score(y_test, predictions)\n",
    "print classification_report(y_test, predictions)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Naive Bayes accuracy and time elapsed caculation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NaiveBayes\n",
      "Acurracy:  0.658854166667\n",
      "time elapsed:  0.048999786377\n"
     ]
    }
   ],
   "source": [
    "t4=time()\n",
    "print \"NaiveBayes\"\n",
    "nb = BernoulliNB()\n",
    "clf_nb=nb.fit(X_train,y_train)\n",
    "print \"Acurracy: \", clf_nb.score(X_test,y_test)\n",
    "t5=time()\n",
    "print \"time elapsed: \", t5-t4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## cross-validation for NB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cross result========\n",
      "[ 0.5125      0.6125      0.47169811]\n",
      "0.532232704403\n",
      "time elapsed:  0.0410001277924\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tt4=time()\n",
    "print \"cross result========\"\n",
    "scores = cross_validation.cross_val_score(nb, X,y, cv=3)\n",
    "print scores\n",
    "print scores.mean()\n",
    "tt5=time()\n",
    "print \"time elapsed: \", tt5-tt4\n",
    "print \"\\n\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# KNN accuracy and time elapsed caculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KNN\n",
      "Acurracy:  0.736979166667\n",
      "time elapsed:  0.0230000019073\n"
     ]
    }
   ],
   "source": [
    "t6=time()\n",
    "print \"KNN\"\n",
    "# knn = KNeighborsClassifier(n_neighbors=3)\n",
    "knn = KNeighborsClassifier(n_neighbors=3)\n",
    "clf_knn=knn.fit(X_train, y_train)\n",
    "print \"Acurracy: \", clf_knn.score(X_test,y_test) \n",
    "t7=time()\n",
    "print \"time elapsed: \", t7-t6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## cross validation for KNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cross result========\n",
      "[ 0.85492228  0.81770833  0.78534031  0.7382199   0.30890052]\n",
      "0.701018269222\n",
      "time elapsed:  0.0859999656677\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tt6=time()\n",
    "print \"cross result========\"\n",
    "scores = cross_validation.cross_val_score(knn, X,y, cv=5)\n",
    "print scores\n",
    "print scores.mean()\n",
    "tt7=time()\n",
    "print \"time elapsed: \", tt7-tt6\n",
    "print \"\\n\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fine tuning the model using GridSearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 2 candidates, totalling 6 fits\n",
      "Best score: 0.517\n",
      "Best parameters set:\n",
      "\tn_neighbors: 1\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          1       0.64      0.56      0.60        16\n",
      "          2       0.40      0.29      0.33         7\n",
      "          3       1.00      0.25      0.40         4\n",
      "          4       0.25      0.12      0.17         8\n",
      "          5       0.20      1.00      0.33         1\n",
      "          6       0.00      0.00      0.00         3\n",
      "          7       0.09      0.50      0.15         2\n",
      "\n",
      "avg / total       0.47      0.37      0.38        41\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done   6 out of   6 | elapsed:   12.9s finished\n"
     ]
    }
   ],
   "source": [
    "from sklearn.svm import SVC\n",
    "from sklearn.cross_validation import cross_val_score\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn import grid_search\n",
    "\n",
    "knn = KNeighborsClassifier()\n",
    "\n",
    "parameters = {'n_neighbors':[1,10]}\n",
    "\n",
    "grid = grid_search.GridSearchCV(knn, parameters, n_jobs=-1, verbose=1, scoring='accuracy')\n",
    "\n",
    "\n",
    "grid.fit(X_train, y_train)\n",
    "\n",
    "print 'Best score: %0.3f' % grid.best_score_\n",
    "\n",
    "print 'Best parameters set:'\n",
    "best_parameters = grid.best_estimator_.get_params()\n",
    "\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print '\\t%s: %r' % (param_name, best_parameters[param_name])\n",
    "    \n",
    "predictions = grid.predict(X_test)\n",
    "print classification_report(y_test, predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SVM accuracy and time elapsed caculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SVM\n",
      "Acurracy:  0.658854166667\n",
      "time elapsed:  0.0620000362396\n"
     ]
    }
   ],
   "source": [
    "t7=time()\n",
    "print \"SVM\"\n",
    "\n",
    "svc = SVC()\n",
    "clf_svc=svc.fit(X_train, y_train)\n",
    "print \"Acurracy: \", clf_svc.score(X_test,y_test) \n",
    "t8=time()\n",
    "print \"time elapsed: \", t8-t7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## cross validation for SVM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cross result========\n",
      "[ 0.65284974  0.65104167  0.65445026  0.65445026  0.65445026]\n",
      "0.653448438588\n",
      "time elapsed:  14.5099999905\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tt7=time()\n",
    "print \"cross result========\"\n",
    "scores = cross_validation.cross_val_score(svc,X,y, cv=5)\n",
    "print scores\n",
    "print scores.mean()\n",
    "tt8=time()\n",
    "print \"time elapsed: \", tt7-tt6\n",
    "print \"\\n\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 4 candidates, totalling 12 fits\n",
      "Best score: 0.650"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  12 out of  12 | elapsed:   17.6s finished\n",
      "C:\\Miniconda2\\lib\\site-packages\\sklearn\\metrics\\classification.py:1074: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Best parameters set:\n",
      "\tC: 1\n",
      "\tkernel: 'linear'\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        0.0       0.66      1.00      0.79       253\n",
      "        1.0       0.00      0.00      0.00       131\n",
      "\n",
      "avg / total       0.43      0.66      0.52       384\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.svm import SVC\n",
    "from sklearn.cross_validation import cross_val_score\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn import grid_search\n",
    "\n",
    "svc = SVC()\n",
    "\n",
    "parameters = {'kernel':('linear', 'rbf'), 'C':[1, 10]}\n",
    "\n",
    "grid = grid_search.GridSearchCV(svc, parameters, n_jobs=-1, verbose=1, scoring='accuracy')\n",
    "\n",
    "\n",
    "grid.fit(X_train, y_train)\n",
    "\n",
    "print 'Best score: %0.3f' % grid.best_score_\n",
    "\n",
    "print 'Best parameters set:'\n",
    "best_parameters = grid.best_estimator_.get_params()\n",
    "\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print '\\t%s: %r' % (param_name, best_parameters[param_name])\n",
    "    \n",
    "predictions = grid.predict(X_test)\n",
    "print classification_report(y_test, predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  34 tasks      | elapsed:   18.2s\n",
      "[Parallel(n_jobs=-1)]: Done  90 out of  90 | elapsed:   18.7s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 30 candidates, totalling 90 fits\n",
      "Best score: 0.650\n",
      "Best parameters set:\n",
      "\tclf__C: 0.1\n",
      "\tclf__gamma: 0.01\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        0.0       0.66      1.00      0.79       253\n",
      "        1.0       0.00      0.00      0.00       131\n",
      "\n",
      "avg / total       0.43      0.66      0.52       384\n",
      "\n"
     ]
    }
   ],
   "source": [
    "pipeline = Pipeline([\n",
    "    ('clf', SVC(kernel='linear', gamma=0.01, C=10))\n",
    "])\n",
    "\n",
    "parameters = {\n",
    "    'clf__gamma': (0.01, 0.03, 0.1, 0.3, 1),\n",
    "    'clf__C': (0.1, 0.3, 1, 3, 10, 30),\n",
    "}\n",
    "\n",
    "grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=1, scoring='accuracy')\n",
    "\n",
    "grid_search.fit(X_train, y_train)\n",
    "\n",
    "print 'Best score: %0.3f' % grid_search.best_score_\n",
    "\n",
    "print 'Best parameters set:'\n",
    "best_parameters = grid_search.best_estimator_.get_params()\n",
    "\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print '\\t%s: %r' % (param_name, best_parameters[param_name])\n",
    "    \n",
    "predictions = grid_search.predict(X_test)\n",
    "print classification_report(y_test, predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}