Lenses Data Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lenses Data Classification\n",
    "<hr>\n",
    "\n",
    "In this example, we would be considering the Lenses database, which is a database for fitting contact lenses. \n",
    "\n",
    "Database link : https://archive.ics.uci.edu/ml/datasets/lenses\n",
    "\n",
    "#### Data Set Information:\n",
    "\n",
    "The examples are complete and noise free. The examples highly simplified the problem. The attributes do not fully describe all the factors affecting the decision as to which type, if any, to fit. \n",
    "\n",
    "This database is complete (all possible combinations of attribute-value pairs are represented). \n",
    "Each instance is complete and correct. \n",
    "9 rules cover the training set.\n",
    "\n",
    "\n",
    "#### Attribute Information:\n",
    "\n",
    "<pre>\n",
    "-- 3 Classes \n",
    "1 : the patient should be fitted with hard contact lenses, \n",
    "2 : the patient should be fitted with soft contact lenses, \n",
    "3 : the patient should not be fitted with contact lenses. \n",
    "\n",
    "1. age of the patient: (1) young, (2) pre-presbyopic, (3) presbyopic \n",
    "2. spectacle prescription: (1) myope, (2) hypermetrope \n",
    "3. astigmatic: (1) no, (2) yes \n",
    "4. tear production rate: (1) reduced, (2) normal\n",
    "</pre>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from sklearn.tree import DecisionTreeClassifier, export_graphviz\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn import cross_validation, metrics\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.naive_bayes import BernoulliNB\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.svm import SVC\n",
    "from time import time\n",
    "from sklearn import preprocessing\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.metrics import roc_auc_score , classification_report\n",
    "from sklearn.grid_search import GridSearchCV\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.metrics import precision_score, recall_score, accuracy_score, classification_report,log_loss\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cols = []\n",
    "for r in range(11):\n",
    "    cols.append('c'+str(r))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# read .csv from provided dataset\n",
    "csv_filename=\"lenses.data\"\n",
    "\n",
    "# df=pd.read_csv(csv_filename,index_col=0)\n",
    "df=pd.read_csv(csv_filename,sep=' ',names=cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>c0</th>\n",
       "      <th>c1</th>\n",
       "      <th>c2</th>\n",
       "      <th>c3</th>\n",
       "      <th>c4</th>\n",
       "      <th>c5</th>\n",
       "      <th>c6</th>\n",
       "      <th>c7</th>\n",
       "      <th>c8</th>\n",
       "      <th>c9</th>\n",
       "      <th>c10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>13</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>15</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>16</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>17</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>18</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>19</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>20</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>21</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>22</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>23</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>24</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    c0   c1   c2   c3   c4   c5   c6   c7   c8   c9  c10\n",
       "0    1  NaN  1.0  NaN  1.0  NaN  1.0  NaN  1.0  NaN  3.0\n",
       "1    2  NaN  1.0  NaN  1.0  NaN  1.0  NaN  2.0  NaN  2.0\n",
       "2    3  NaN  1.0  NaN  1.0  NaN  2.0  NaN  1.0  NaN  3.0\n",
       "3    4  NaN  1.0  NaN  1.0  NaN  2.0  NaN  2.0  NaN  1.0\n",
       "4    5  NaN  1.0  NaN  2.0  NaN  1.0  NaN  1.0  NaN  3.0\n",
       "5    6  NaN  1.0  NaN  2.0  NaN  1.0  NaN  2.0  NaN  2.0\n",
       "6    7  NaN  1.0  NaN  2.0  NaN  2.0  NaN  1.0  NaN  3.0\n",
       "7    8  NaN  1.0  NaN  2.0  NaN  2.0  NaN  2.0  NaN  1.0\n",
       "8    9  NaN  2.0  NaN  1.0  NaN  1.0  NaN  1.0  NaN  3.0\n",
       "9   10  2.0  NaN  1.0  NaN  1.0  NaN  2.0  NaN  2.0  NaN\n",
       "10  11  2.0  NaN  1.0  NaN  2.0  NaN  1.0  NaN  3.0  NaN\n",
       "11  12  2.0  NaN  1.0  NaN  2.0  NaN  2.0  NaN  1.0  NaN\n",
       "12  13  2.0  NaN  2.0  NaN  1.0  NaN  1.0  NaN  3.0  NaN\n",
       "13  14  2.0  NaN  2.0  NaN  1.0  NaN  2.0  NaN  2.0  NaN\n",
       "14  15  2.0  NaN  2.0  NaN  2.0  NaN  1.0  NaN  3.0  NaN\n",
       "15  16  2.0  NaN  2.0  NaN  2.0  NaN  2.0  NaN  3.0  NaN\n",
       "16  17  3.0  NaN  1.0  NaN  1.0  NaN  1.0  NaN  3.0  NaN\n",
       "17  18  3.0  NaN  1.0  NaN  1.0  NaN  2.0  NaN  3.0  NaN\n",
       "18  19  3.0  NaN  1.0  NaN  2.0  NaN  1.0  NaN  3.0  NaN\n",
       "19  20  3.0  NaN  1.0  NaN  2.0  NaN  2.0  NaN  1.0  NaN\n",
       "20  21  3.0  NaN  2.0  NaN  1.0  NaN  1.0  NaN  3.0  NaN\n",
       "21  22  3.0  NaN  2.0  NaN  1.0  NaN  2.0  NaN  2.0  NaN\n",
       "22  23  3.0  NaN  2.0  NaN  2.0  NaN  1.0  NaN  3.0  NaN\n",
       "23  24  3.0  NaN  2.0  NaN  2.0  NaN  2.0  NaN  3.0  NaN"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df.fillna(0 ,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>c0</th>\n",
       "      <th>c1</th>\n",
       "      <th>c2</th>\n",
       "      <th>c3</th>\n",
       "      <th>c4</th>\n",
       "      <th>c5</th>\n",
       "      <th>c6</th>\n",
       "      <th>c7</th>\n",
       "      <th>c8</th>\n",
       "      <th>c9</th>\n",
       "      <th>c10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   c0   c1   c2   c3   c4   c5   c6   c7   c8   c9  c10\n",
       "0   1  0.0  1.0  0.0  1.0  0.0  1.0  0.0  1.0  0.0  3.0\n",
       "1   2  0.0  1.0  0.0  1.0  0.0  1.0  0.0  2.0  0.0  2.0\n",
       "2   3  0.0  1.0  0.0  1.0  0.0  2.0  0.0  1.0  0.0  3.0\n",
       "3   4  0.0  1.0  0.0  1.0  0.0  2.0  0.0  2.0  0.0  1.0\n",
       "4   5  0.0  1.0  0.0  2.0  0.0  1.0  0.0  1.0  0.0  3.0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df['Age'] = df['c1'] + df['c2']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df['Specs'] = df['c3'] + df['c4']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df['Astigmatic'] = df['c5'] + df['c6']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df['Tear-Production-Rate'] = df['c7'] + df['c8']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df['Target-Lenses'] = df['c9'] + df['c10']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.drop(cols,axis=1,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Specs</th>\n",
       "      <th>Astigmatic</th>\n",
       "      <th>Tear-Production-Rate</th>\n",
       "      <th>Target-Lenses</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Age  Specs  Astigmatic  Tear-Production-Rate  Target-Lenses\n",
       "0   1.0    1.0         1.0                   1.0            3.0\n",
       "1   1.0    1.0         1.0                   2.0            2.0\n",
       "2   1.0    1.0         2.0                   1.0            3.0\n",
       "3   1.0    1.0         2.0                   2.0            1.0\n",
       "4   1.0    2.0         1.0                   1.0            3.0\n",
       "5   1.0    2.0         1.0                   2.0            2.0\n",
       "6   1.0    2.0         2.0                   1.0            3.0\n",
       "7   1.0    2.0         2.0                   2.0            1.0\n",
       "8   2.0    1.0         1.0                   1.0            3.0\n",
       "9   2.0    1.0         1.0                   2.0            2.0\n",
       "10  2.0    1.0         2.0                   1.0            3.0\n",
       "11  2.0    1.0         2.0                   2.0            1.0\n",
       "12  2.0    2.0         1.0                   1.0            3.0\n",
       "13  2.0    2.0         1.0                   2.0            2.0\n",
       "14  2.0    2.0         2.0                   1.0            3.0\n",
       "15  2.0    2.0         2.0                   2.0            3.0\n",
       "16  3.0    1.0         1.0                   1.0            3.0\n",
       "17  3.0    1.0         1.0                   2.0            3.0\n",
       "18  3.0    1.0         2.0                   1.0            3.0\n",
       "19  3.0    1.0         2.0                   2.0            1.0\n",
       "20  3.0    2.0         1.0                   1.0            3.0\n",
       "21  3.0    2.0         1.0                   2.0            2.0\n",
       "22  3.0    2.0         2.0                   1.0            3.0\n",
       "23  3.0    2.0         2.0                   2.0            3.0"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index([u'Age', u'Specs', u'Astigmatic', u'Tear-Production-Rate',\n",
       "       u'Target-Lenses'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "features = ['Age', 'Specs', 'Astigmatic', 'Tear-Production-Rate']\n",
    "X = df[features]\n",
    "y = df['Target-Lenses']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Specs</th>\n",
       "      <th>Astigmatic</th>\n",
       "      <th>Tear-Production-Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Age  Specs  Astigmatic  Tear-Production-Rate\n",
       "0  1.0    1.0         1.0                   1.0\n",
       "1  1.0    1.0         1.0                   2.0\n",
       "2  1.0    1.0         2.0                   1.0\n",
       "3  1.0    1.0         2.0                   2.0\n",
       "4  1.0    2.0         1.0                   1.0"
      ]
     },
     "execution_count": 160,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# split dataset to 60% training and 40% testing\n",
    "X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, test_size=0.4, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(14, 4) (14L,)\n"
     ]
    }
   ],
   "source": [
    "print X_train.shape, y_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature importances with forests of trees\n",
    "\n",
    "This examples shows the use of forests of trees to evaluate the importance of features on an artificial classification task. The red bars are the feature importances of the forest, along with their inter-trees variability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature ranking:\n",
      "1. feature 3 - Milk (0.138606) \n",
      "2. feature 1 - Feathers (0.138508) \n",
      "3. feature 2 - Eggs (0.113904) \n",
      "4. feature 7 - Toothed (0.083274) \n",
      "5. feature 0 - Hair (0.083251) \n",
      "6. feature 8 - Backbone (0.082373) \n",
      "7. feature 9 - Breathes (0.076449) \n",
      "8. feature 11 - Fins (0.063457) \n",
      "9. feature 13 - Tail (0.050104) \n",
      "10. feature 12 - Legs (0.048997) \n",
      "11. feature 5 - Aquatic (0.035794) \n",
      "12. feature 4 - Airborne (0.035151) \n",
      "13. feature 16 - animals (0.016738) \n",
      "14. feature 15 - Catsize (0.011835) \n",
      "15. feature 6 - Predator (0.011787) \n",
      "16. feature 10 - Venomous (0.009257) \n",
      "17. feature 14 - Domestic (0.000515) \n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAz4AAAJZCAYAAABoTeCOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X+c1XWdL/D3l4EsJQzEZmAYwR/AjIGAinp37XasQG03\njNQW0/RhJFteH5vZY9V7dzehdjNsq7Vs72XdXX+0Bdtjt8AKqJbrcc27NFvqupllaoIMgigkihXC\nfO8fsxwdmRlGz5k58/3wfD4e5/H4Hs/3x2s+fpmZ13y/53OyPM/zAAAASNiwegcAAAAYaIoPAACQ\nPMUHAABInuIDAAAkT/EBAACSp/gAAADJU3wAGDI+8pGPxF/8xV/UOwYACcp8jg9A8U2aNCmeeuqp\nGD58eOR5HlmWxcMPPxxNTU2veZ933XVXXHTRRfHEE0/UMGlxXHrppdHS0hKf/OQn6x0FgBpwxQcg\nAVmWxXe+853YuXNnPPfcc7Fz586qSk9EVArUa7V3796qjl9PnZ2d9Y4AQI0pPgCJ6O0C/vr16+N3\nf/d3Y/To0TFr1qy46667Kq/deuutcfzxx8eoUaPiuOOOi7/5m7+JiIgXXngh3vWud8XmzZvjjW98\nY4waNSq2bNkSl156aXziE5+obH/XXXdFS0tL5fnRRx8dN9xwQ8yYMSNGjhwZnZ2d8eSTT8Z5550X\nb37zm+PYY4+NL33pS71+DS/f/759f/azn43GxsZobm6OVatWxZo1a2Lq1KkxduzYuP766yvbLlmy\nJM4///xYsGBBjBo1Kk4++eR44IEHKq//7Gc/izPOOCNGjx4d06dPj29961vdjnv55ZfH7/3e78Ub\n3/jG+Lu/+7v46le/GjfccEOMGjUqzjnnnIiIWLp0aRx33HExatSomDZtWqxcubKyj9tuuy3e+ta3\nxh//8R/HmDFj4thjj421a9dWXt+xY0d88IMfjObm5jjiiCPive99b+W1b3/72zFr1qwYPXp0nH76\n6fGf//mfldeWLl0aEyZMiFGjRkVbW1vceeedvY4fAL1TfAAStnnz5vj93//9+MQnPhE7duyIv/zL\nv4xzzz03nnnmmYiIaGxsjNWrV8fOnTvjlltuiY997GNx//33x6GHHhpr1qyJ8ePHH/AK0iuvCq1Y\nsSLWrFkTv/rVryLLsnj3u98ds2bNiieffDLWrVsXN954Y3z/+9/vV/4tW7bE7t27Y/PmzbFkyZK4\n7LLL4qtf/Wrcd9998a//+q/xqU99KjZs2FBZ/4477og/+IM/iB07dsQFF1wQ73nPe2Lv3r2xZ8+e\nePe73x1nnXVWbNu2Lb74xS/GhRdeGL/4xS8q2y5fvjz+7M/+LJ577rm4+OKL48ILL4yrr746du7c\nGatWrYqIiOOOOy7uueee2LlzZ1x33XVx0UUXxdatWyv7aG9vj7a2tnjmmWfij//4j2PhwoWV1y66\n6KL49a9/HQ899FA89dRT8bGPfSwiIu67775YuHBh3HzzzbF9+/b4wz/8w5g3b168+OKL8fDDD8eX\nv/zl+PGPfxw7d+6M7373uzFp0qR+jR0A3Sk+AIl4z3veE2PGjIkxY8ZUrib8wz/8Q/ze7/1enHnm\nmRER8Y53vCNOPvnkWL16dUREnH322ZVfpN/61rfG3Llz4+67764qx0c/+tEYP358HHLIIfHv//7v\n8fTTT8ef/MmfRENDQ0yaNCk+9KEPxYoVK/q1r9e97nXxv/7X/4qGhoZYsGBBPP3003HllVfGoYce\nGscff3wcf/zx8R//8R+V9U866aSYP39+NDQ0xFVXXRW//e1vY/369bF+/frYtWtXXHPNNTF8+PA4\n44wz4vd///dj+fLllW3POeecOO200yIi4pBDDukxz7nnnhuNjY0REXH++efH5MmTo729vfL6xIkT\n44Mf/GBkWRaXXHJJPPnkk/HUU0/Fli1b4rvf/W4sW7YsRo0aFQ0NDfHWt741IiJuvvnm+PCHPxwn\nn3xyZFkWH/jAB+KQQw6J9evXR0NDQ+zevTt+8pOfxJ49e+Koo46Ko48++tX9DwEgIhQfgGSsWrUq\ntm/fHtu3b49vfOMbERGxYcOG+PrXv14pRKNHj4577rknnnzyyYiIWLNmTfy3//bf4ogjjojRo0fH\nmjVr4umnn64qx4QJEyrLGzZsiI6Ojm7Hv/766+Opp57q176OOOKIyhWlN7zhDRER8eY3v7ny+hve\n8IZ4/vnnK89ffttdlmXR3Nwcmzdvjs2bN3d7LaKrpHR0dPS4bW9uv/32yi1po0ePjgcffLDbeL38\nqti+vM8//3w88cQTMWbMmBg1atR++9ywYUN87nOf6zZGmzZtis2bN8exxx4bf/VXfxWLFy+OxsbG\neP/731/5fwfAq6P4ACSip/f4tLS0xMUXX1wpRDt27Ijnnnsurr766ti9e3ecd955cfXVV8e2bdti\nx44dcfbZZ1f209PEBocddli88MILlec9/RL+8u1aWlrimGOO6Xb8Z599ttv7a2rp5TPQ5XkemzZt\nivHjx8f48eNj48aN3dbduHFjNDc395i7p+cbN26MRYsWxV//9V/Hjh07YseOHfGWt7yl1/dWvVxL\nS0ts3749du7c2eNrf/Inf9JtjJ5//vn4gz/4g4iIWLBgQdx9992VW/quvfbaAx4PgP0pPgAJu+ii\ni+Jb3/pWfO9734vOzs74zW9+E3fddVds3rw5du/eHbt3746xY8fGsGHDYs2aNfG9732vsm1jY2M8\n88wz3X5ZnzlzZqxevTp27NgRW7ZsiRtvvLHP459yyinxxje+MW644Yb4zW9+E3v37o0HH3wwfvSj\nHw3I1/vjH/84Vq5cGXv37o0vfOEL8frXvz5OO+20OPXUU+Owww6LG264Ifbs2RPlcjm+/e1vxwUX\nXNDrvhobG+Oxxx6rPN+1a1cMGzYsxo4dG52dnXHLLbfET37yk37lampqirPPPjsuv/zy+NWvfhV7\n9uyp3FJ42WWXxf/5P/+ncsvcrl27YvXq1bFr1654+OGH484774zdu3fH6173unjDG94Qw4b50Q3w\nWvjuCZCA3qadnjBhQqxatSo+/elPx5FHHhkTJ06Mv/zLv4zOzs4YOXJkfPGLX4zzzz8/xowZEytW\nrKjMXhYRMXXq1LjgggvimGOOiTFjxsSWLVviAx/4QJxwwgkxadKkOOuss2LBggV95hg2bFh8+9vf\njvvvvz+OPvroePOb3xyXXXZZj1c+XsvX+crn55xzTvzjP/5jjB49Or761a/GN7/5zWhoaIgRI0bE\nt771rVi9enWMHTs2rrjiivjKV74SkydP7nX8Fi5cGA8++GDlPVNtbW1x1VVXxWmnnRZNTU3x4IMP\nxumnn97vvF/5yldi+PDh0draGo2NjZXSeNJJJ8XNN98cV1xxRYwZMyamTJkSt912W0RE/Pa3v41r\nr702jjzyyBg/fnxs27at20x2APRfTT7AdO3atXHllVdGZ2dnLFy4MK655ppur99xxx3xZ3/2ZzFs\n2LAYMWJEfOELX4jf/d3f7de2ANAfS5YsiUcffTRuv/32ekcBYAgaXu0OOjs744orroh169bF+PHj\nY/bs2XHOOedEa2trZZ13vvOdMW/evIiI+M///M943/veFw899FC/tgUAAKhW1be6tbe3x+TJk2Pi\nxIkxYsSIWLBgQeXzDvY59NBDK8vPP/985f7k/mwLAABQraqv+HR0dHSbAnTChAndPtNgn5UrV8b/\n/J//M7Zt2xbf+c53XtW2AHAg1113Xb0jADCEDdrkBu95z3vioYceipUrV8af/umfDtZhAQAAqr/i\n09zc3O2zETZt2tTtcxFe6fTTT4/HHnsstm/f/qq27W3GIgAAgH16m7ut6is+s2fPjkceeSQ2bNgQ\nu3fvjhUrVlQmMtjn0UcfrSzfe++9sXv37hgzZky/tn3lF1GEx3XXXVf3DKnmLVLWouWVVd6iZS1a\nXlnlLVrWouWVVd4873uy6qqv+DQ0NMRNN90Uc+fOrUxJ3dbWFsuWLYssy2LRokXxz//8z3H77bdX\nPnzt61//ep/bFlG53PWIiLj11pf+e6nU9QAAAOqn6uITEXHWWWfFz3/+827/7Q//8A8ry1dffXVc\nffXV/d62iF5ecJYsiVi8uI5hAACAbgZtcoODS6neAV6VUoEuSRUpa0Sx8so6cIqUt0hZI4qVV9aB\nU6S8RcoaUay8sg6couXtTZYf6Ga4ISLLsgPetzdUZFlEQaICAEAy+uoMrvgAAADJU3wAAIDkKT4A\nAEDyFB8AACB5ig8AAJA8xQcAAEie4gMAACRP8QEAAJKn+AAAAMlTfAAAgOQpPgAAQPIUHwAAIHmK\nDwAAkDzFBwAASJ7iAwAAJE/xAQAAkqf4AAAAyVN8AACA5Ck+AABA8hQfAAAgeYoPAACQPMUHAABI\nnuIDAAAkT/EBAACSp/gAAADJU3wAAIDkKT4AAEDyFB8AACB5ig8AAJA8xQcAAEie4gMAACRP8QEA\nAJKn+AAAAMlTfAAAgOQpPgAAQPIUHwAAIHmKDwAAkDzFBwAASJ7iAwAAJE/xAQAAkqf4AAAAyVN8\nAACA5Ck+AABA8hQfAAAgeYoPAACQPMUHAABInuIDAAAkT/EBAACSp/gAAADJU3wAAIDkKT4AAEDy\nFB8AACB5ig8AAJA8xQcAAEie4gMAACRP8QEAAJKn+AAAAMlTfAAAgOQpPgAAQPIUHwAAIHmKDwAA\nkDzFBwAASJ7iAwAAJE/xAQAAkqf4AAAAyVN8AACA5Ck+AABA8hQfAAAgecPrHQAOpFzueuxbLpW6\nlkull5YBAKAvWZ7nebU7Wbt2bVx55ZXR2dkZCxcujGuuuabb61/72tdi6dKlERHxxje+Mf76r/86\nTjjhhIiImDRpUhx++OExbNiwGDFiRLS3t/ccNMuiBlEHRZZFFCRq4RhbAAB601dnqLr4dHZ2xpQp\nU2LdunUxfvz4mD17dqxYsSJaW1sr66xfvz7a2tri8MMPj7Vr18bixYtj/fr1ERFxzDHHxI9//OMY\nPXr0a/4ihhq/nA8cYwsAQG/66gxVv8envb09Jk+eHBMnTowRI0bEggULYtWqVd3WOe200+Lwww+v\nLHd0dFRey/M8Ojs7q40BAADQq6qLT0dHR7S0tFSeT5gwoVuxeaW//du/jbPPPrvyPMuymDNnTsye\nPTtuvvnmauMAAADsZ1AnN7jzzjvjlltuiR/84AeV/3bPPffEuHHjYtu2bTFnzpxoa2uL008/fTBj\nAQAAiau6+DQ3N8fGjRsrzzdt2hTNzc37rffAAw/EokWLYu3atd3ezzNu3LiIiDjyyCNj/vz50d7e\n3mvxWbx4cWW5VCpFyZReAABw0CqXy1HeN/3vAVQ9ucHevXtj6tSpsW7duhg3blyccsopsXz58mhr\na6uss3HjxnjHO94RX/nKV+K0006r/PcXXnghOjs7Y+TIkbFr166YO3duXHfddTF37tz9g5rcgDC2\nAAD0rq/OUPUVn4aGhrjpppti7ty5lems29raYtmyZZFlWSxatCg+9alPxfbt2+Pyyy+PPM8r01Zv\n3bo15s+fH1mWxZ49e+LCCy/ssfQAAABUoyaf4zMYXPEhwtgCANC7AZ3OGgAAYKhTfAAAgOQpPgAA\nQPIUHwAAIHmKDwAAkDzFBwAASJ7iAwAAJE/xAQAAkqf4AAAAyVN8AACA5Ck+AABA8obXOwD1US53\nPfYtl0pdy6XSS8sAAJCKLM/zvN4h+iPLsihI1MiyiIJEjYhi5S1SVgAABldfncGtbgAAQPIUHwAA\nIHmKDwAAkDzFBwAASJ7iAwAAJE/xAQAAkqf4AAAAyVN8AACA5Ck+AABA8hQfAAAgeYoPAACQPMUH\nAABInuIDAAAkT/EBAACSp/gAAADJU3wAAIDkKT4AAEDyFB8AACB5ig8AAJA8xQcAAEie4gMAACRP\n8QEAAJKn+AAAAMlTfAAAgOQpPgAAQPIUHwAAIHmKDwAAkDzFBwAASJ7iAwAAJE/xAQAAkqf4AAAA\nyVN8AACA5Ck+AABA8hQfAAAgeYoPAACQPMUHAABInuIDAAAkT/EBAACSp/gAAADJU3wAAIDkKT4A\nAEDyFB8AACB5ig8AAJA8xQcAAEie4gMAACRP8QEAAJKn+AAAAMlTfAAAgOQpPgAAQPIUHwAAIHmK\nDwAAkLzh9Q4A1E+53PXYt1wqdS2XSi8tAwCkIMvzPK93iP7IsiwKEjWyLKIgUSOiWHmLlLVojC0A\nUHR9dQa3ugEAAMlTfAAAgOQpPgAAQPIUHwAAIHmKDwAAkLyaFJ+1a9dGa2trTJkyJZYuXbrf61/7\n2tdixowZMWPGjDj99NPjgQce6Pe2AAAA1ap6OuvOzs6YMmVKrFu3LsaPHx+zZ8+OFStWRGtra2Wd\n9evXR1tbWxx++OGxdu3aWLx4caxfv75f21aCms56wBQpb5GyFo2xBQCKbkCns25vb4/JkyfHxIkT\nY8SIEbFgwYJYtWpVt3VOO+20OPzwwyvLHR0d/d4WAACgWlUXn46Ojmhpaak8nzBhQqXY9ORv//Zv\n4+yzz35N2wIAALwWwwfzYHfeeWfccsst8YMf/OA1bb948eLKcqlUilKpVJtgAABA4ZTL5SiXy/1a\nt+ri09zcHBs3bqw837RpUzQ3N++33gMPPBCLFi2KtWvXxujRo1/Vtvu8vPgAAAAHt1deDFmyZEmv\n61Z9q9vs2bPjkUceiQ0bNsTu3btjxYoVMW/evG7rbNy4Mc4999z4yle+Escee+yr2hYAAKBaVV/x\naWhoiJtuuinmzp0bnZ2dsXDhwmhra4tly5ZFlmWxaNGi+NSnPhXbt2+Pyy+/PPI8jxEjRkR7e3uv\n2wIAANRS1dNZDxbTWQ+cIuUtUtaiMbYAQNEN6HTWAAAAQ53iAwAAJE/xAQAAkqf4AAAAyVN8AACA\n5Ck+AABA8hQfAAAgeYoPAACQPMUHAABInuIDAAAkT/EBAACSp/gAAADJU3wAAIDkKT4AAEDyFB8A\nACB5ig8AAJA8xQcAAEie4gMAACRP8QEAAJKn+AAAAMlTfAAAgOQpPgAAQPIUHwAAIHmKDwAAkDzF\nBwAASJ7iAwAAJE/xAQAAkqf4AAAAyVN8AACA5Ck+AABA8hQfAAAgeYoPAACQPMUHAABInuIDAAAk\nT/EBAACSp/gAAADJU3wAAIDkKT4AAEDyFB8AACB5ig8AAJA8xQcAAEie4gMAACRP8QEAAJKn+AAA\nAMlTfAAAgOQpPgAAQPIUHwAAIHmKDwAAkDzFBwAASJ7iAwAAJE/xAQAAkqf4AAAAyVN8AACA5Ck+\nAABA8obXOwCkplzueuxbLpW6lkull5YBABhcWZ7neb1D9EeWZVGQqJFlEQWJGhHFylukrBHFyluk\nrAAAPemrM7jVDQAASJ7iAwAAJE/xAQAAkqf4AAAAyVN8AACA5Ck+AABA8hQfAAAgeYoPAACQPMUH\nAABInuIDAAAkT/EBAACSp/gAAADJU3wAAIDkKT4AAEDyalJ81q5dG62trTFlypRYunTpfq///Oc/\nj9/5nd+J17/+9fH5z3++22uTJk2KGTNmxKxZs+KUU06pRRwAAIBuhle7g87Ozrjiiiti3bp1MX78\n+Jg9e3acc8450draWlnniCOOiC996UuxcuXK/bYfNmxYlMvlGD16dLVRAAAAelT1FZ/29vaYPHly\nTJw4MUaMGBELFiyIVatWdVtn7NixcdJJJ8Xw4fv3rDzPo7Ozs9oYAAAAvaq6+HR0dERLS0vl+YQJ\nE6Kjo6Pf22dZFnPmzInZs2fHzTffXG0cAACA/VR9q1u17rnnnhg3blxs27Yt5syZE21tbXH66afX\nOxYAAJCQqotPc3NzbNy4sfJ806ZN0dzc3O/tx40bFxERRx55ZMyfPz/a29t7LT6LFy+uLJdKpSiV\nSq8pMwAAUHzlcjnK5XK/1s3yPM+rOdjevXtj6tSpsW7duhg3blyccsopsXz58mhra9tv3SVLlsTI\nkSPj4x//eEREvPDCC9HZ2RkjR46MXbt2xdy5c+O6666LuXPn7h80y6LKqIMmyyIKEjUiipW3SFkj\nipW3SFkBAHrSV2eo+opPQ0ND3HTTTTF37tzo7OyMhQsXRltbWyxbtiyyLItFixbF1q1b4+STT47n\nnnsuhg0bFjfeeGP89Kc/jW3btsX8+fMjy7LYs2dPXHjhhT2WHgAAgGpUfcVnsLjiM3CKlLdIWSOK\nlbdIWQEAetJXZ6jJB5gCAAAMZXWf1Q2gP8rlrse+5X1zm5RKLy0DAPTGrW4DoGi3DBUpb5GyRhQr\nr6wAQNG51Q0AADioKT4AAEDyFB8AACB5ig8AAJA8xQcAAEie4gMAACRP8QEAAJKn+AAAAMlTfAAA\ngOQpPgAAQPIUHwAAIHnD6x1gqJnU1BQbtm6tci95ZFlWdZaJjY3x+JYtvb5em6wRtch7oKxwsCmX\nux77lkulruVS6aVlAGDwZHme5/UO0R9ZlsVgRM2yLKo9ShZ55FF98cki+vyaa5G16zjV5z1Q1lrJ\nsohinLFdipRX1oFTtLwAUFR9dQa3ugEAAMlTfAAAgOQpPgAAQPIUHwAAIHmKDwAAkDzFBwAASJ7i\nAwAAJE/xAQAAkqf4AAAAyVN8AACA5Ck+AABA8hQfAAAgeYoPAACQPMUHAABInuIDAAAkT/EBAACS\np/gAAADJU3wAAIDkKT4AAEDyFB8AACB5ig8AAJA8xQcAAEie4gMAACRveL0DAFBf5XLXY99yqdS1\nXCq9tAwARZfleZ7XO0R/ZFkWgxE1y7Ko9ihZ5JFHVn2WiD6/5lpk7TpO9XkPlLVWsiyiGGdslyLl\nlXXgFClvkbICwCv11Rnc6gYAACRP8QEAAJKn+AAAAMlTfAAAgOQpPgAAQPIUHwAAIHmKDwAAkDzF\nBwAASJ7iAwAAJE/xAQAAkqf4AAAAyVN8AACA5Ck+DJpJTU2RZVlVj4ioeh9ZlsWkpqY6jwYAAIMp\ny/M8r3eI/siyLAYjapZlUe1Rssgjj6z6LBF9fs21yNp1nOrzHihrRLHGtlayLKIY/8JkHUhFyluk\nrADwSn11Bld8AACA5Ck+AABA8hQfAAAgeYoPAACQPMUHAABInuIDAAAkT/EBAACSp/gAAADJU3wA\nAIDkKT4AAEDyFB8AACB5ig8AAJA8xQcAAEie4gMAACRP8QEAAJKn+AAAAMmrSfFZu3ZttLa2xpQp\nU2Lp0qX7vf7zn/88fud3fide//rXx+c///lXtS0AvFy5HLF4cdejVHppuVyuYygAhrwsz/O8mh10\ndnbGlClTYt26dTF+/PiYPXt2rFixIlpbWyvrPP3007Fhw4ZYuXJljB49Oq666qp+b1sJmmVRZdR+\nybIsqj1KFnnkkVWfJaLPr7kWWbuOU33eA2WNKNbY1kqWRQzCYWpC1oFTpLxFyhpRvLwADKy+OkPV\nV3za29tj8uTJMXHixBgxYkQsWLAgVq1a1W2dsWPHxkknnRTDhw9/1dsCAABUq+ri09HRES0tLZXn\nEyZMiI6OjgHfFgAAoL9MbgAAACRv+IFX6Vtzc3Ns3Lix8nzTpk3R3Nw8INsuXry4slwqlaJUKr3q\nvAAAQBrK5XKU+zm7TdWTG+zduzemTp0a69ati3HjxsUpp5wSy5cvj7a2tv3WXbJkSYwcOTI+/vGP\nv+ptTW7Qw+smN6hiPyY3eCVZB06R8hYpa0Tx8gIwsPrqDFVf8WloaIibbrop5s6dG52dnbFw4cJo\na2uLZcuWRZZlsWjRoti6dWucfPLJ8dxzz8WwYcPixhtvjJ/+9KcxcuTIHrcFAACopaqv+AwWV3x6\neN0Vnyr244rPK8k6cIqUt0hZI4qXF4CBNaDTWQMAAAx1ig8AAJA8xQcAAEie4gMAACRP8QEAAJKn\n+AAAAMlTfAAAgOQpPgAAQPIUHwAAIHmKDwAAkDzFBwAASN7wegcAgFSVy12PfculUtdyqfTSMgCD\nI8vzPK93iP7IsiwGI2qWZVHtUbLII4+s+iwRfX7NtcjadZzq8x4oa0SxxrZWsiyiGP/CZB1IRcpb\npKwRxcpbpKwARdVXZ3CrGwAAkDzFBwAASJ7iAwAAJE/xAQAAkqf4AAAAyVN8AACA5Ck+AABA8hQf\nAAAgeYoPAACQPMUHejCpqSmyLKv6ERFV72NSU9Og5B2srAAA9ZDleZ7XO0R/ZFkWgxE1y7Ko9ihZ\n5JFHVn2WiD6/5lpk7TpO9XkPlDXC2L72fRRnbPuTtRayLKIY37m6FClvkbJGFCtvkbICFFVfncEV\nH2BQDZWrU65QAcDBxRWfno5T7T5cleh5HWP7GvdRnLEtUtau/bhC9UpFyhpRrLxFygpQVK74AAAA\nBzXFBwAASJ7iAwAAJE/xAQAAkje83gEAAF6tcrnrsW+5VOpaLpVeWgZ4ObO69XScavdh5rGe1zG2\nr3EfxRnbImXt2o9Z3V6pSFkjipW3SFmLxtgC+5jVDQAAOKgpPgC9qMWHrdbqA1d92CoAVMetbj0d\np9p9uB2r53WM7WvcR3HGtkhZu/aT1nlQC0W7ZahIeYuUtWiMLbCPW90AAICDmuIDAAAkT/EBAACS\np/gAAADJU3wAAIDkKT4AAEDyFB8AACB5ig8AAJA8xQcAAEie4gMAACRP8QEAAJKn+AAAAMlTfAAA\ngOQNr3cAAKD+yuWux77lUqlruVR6aRmgyLI8z/N6h+iPLMtiMKJmWRbVHiWLPPLIqs8S0efXXIus\nXcepPu+BskYY29e+j+KMbZGydu0nrfOgFrIsohg/FboUKa+sA6doeYGB01dncKsbAACQPMUHAABI\nnuIDAAAkT/EBAACSp/gAJGJSU1NkWVbVIyKq3keWZTGpqanOowEA3ZnOGiARG7ZurcGMeVGbmey2\nbq3BXgCgdlzxAQAAkqf4AAAAyVN8AACA5Ck+AABA8hQfAAAgeYoPAACQPMUHAABInuIDAAAkT/EB\nAACSp/gAAADJU3wAAIDkKT4AAEDyFB8AACB5ig8AAJA8xQcAAEheTYrP2rVro7W1NaZMmRJLly7t\ncZ0/+qOcYHTpAAAZpklEQVQ/ismTJ8fMmTPjvvvuq/z3SZMmxYwZM2LWrFlxyimn1CIOAABAN8Or\n3UFnZ2dcccUVsW7duhg/fnzMnj07zjnnnGhtba2ss2bNmnj00UfjF7/4Rfzwhz+Mj3zkI7F+/fqI\niBg2bFiUy+UYPXp0tVEAAAB6VPUVn/b29pg8eXJMnDgxRowYEQsWLIhVq1Z1W2fVqlVx8cUXR0TE\nqaeeGs8++2xs3bo1IiLyPI/Ozs5qYwAAAPSq6uLT0dERLS0tlecTJkyIjo6OPtdpbm6urJNlWcyZ\nMydmz54dN998c7VxAAAA9lP1rW7Vuueee2LcuHGxbdu2mDNnTrS1tcXpp5/e47qLFy+uLJdKpSiV\nSoMTEoCamtTUFBv+68p/dfLIsqyqPUxsbIzHt2ypQRYABlu5XI5yudyvdasuPs3NzbFx48bK802b\nNkVzc/N+6zzxxBM9rjNu3LiIiDjyyCNj/vz50d7e3q/iA0Bxbdi6NfIa7CeLqHo/WU0KGAD18MqL\nIUuWLOl13apvdZs9e3Y88sgjsWHDhti9e3esWLEi5s2b122defPmxe233x4REevXr483velN0djY\nGC+88EI8//zzERGxa9eu+N73vhfTpk2rNhIA1NSkpqbIsqyqR0RUvY9JTU11HgmA4qr6ik9DQ0Pc\ndNNNMXfu3Ojs7IyFCxdGW1tbLFu2LLIsi0WLFsW73vWuWL16dRx33HFx2GGHxS233BIREVu3bo35\n8+dHlmWxZ8+euPDCC2Pu3LlVf1EAUEu1uELl6hRAfWV5ntfiboMBl2VZDEbULMtq8MMtjzyqu+e8\naz/R59dci6xdx6k+74GyRhjb176P4oxtkbJ27cd5sP9xjG2P6wyRse1P1lrIsohi/HbQpWh5gYHT\nV2eoyQeYAgAADGWKDwAAkDzFBwAASJ7iAwAAJE/xAQAAkqf4AAAAyVN8AACA5Ck+AABA8hQfAAAg\neYoPAACQPMUHAABInuIDAAAkT/EBAACSp/gAAADJU3wAAIDkKT4AAEDyFB8AACB5w+sdgPoox9ui\nHKWIiHhblGNxXBcREaUoRynuqmMyAACovSzP87zeIfojy7IYjKhZlkW1R8kijzyy6rNE9Pk11yJr\nrRwoa8TBOba1yFuksS1S1q79OA/2P46x7XGdITK2/ck6qakpNmzdWtVxIvL/Olp1JjY2xuNbtlS9\nnwPJsohi/DYDDLS+OoMrPgCQkA1bt9agpEVtiukBClhtSlpERB5ZVl1RG6ySBtSP4gMA1EUtSlpE\nbYragUoaUHyKDwAAFeVy12PfcqnUtVwqvbQMReQ9Pj0dp9p9FOwe+Voo0j3yXfvx/oP9j3NwZe3a\nj/Ng/+MY2x7XGSJjW6SsXftJ6zw4GHn/FEXjPT4AAHXiCgoMDa749HScavdRsL+U1YK/QvZ2nIPr\nL7xFytq1H+fB/scxtj2uM0TGtkhZu/aT1nlQC0W7glK0vNBXZ/ABpgAAQPIUHwAAIHmKDwAAkDzF\nBwAASJ7iAwAAJE/xAQAAkqf4AAAAyVN8AACA5A2vdwAAAHgtyuWux77lUqlruVR6aRn2yfLB+Jji\nGujrU1hrfpxq91GwT7yuBZ8m3ttxDq5Pai9S1q79OA/2P46x7XGdITK2RcratZ+0zoNayLKIYvzm\n1aVIeYuUlYHTV2dwqxsAAJA8xQcAAEie4gMAACRP8QEAAJKn+AAAAMkznTUAAAwwU2/Xn+msezpO\ntfso2DSgtWCK1d6Oc3BNX1ukrF37cR7sfxxj2+M6Q2Rsi5S1az9pnQe1ULQpl4uUV1YiTGcNAAAc\n5BQfAAAgeYoPAACQPMUHAABInuIDAAAkz3TWQCGU421RjlJERLwtyrE4rouIiFKUoxR31TEZAFAE\nprPu6TjV7qNg04DWgilWezvOwTV97WBlrRXnQU/H8W+sx3WGyNgWKWvXftI6DyY1NcWGrVurOk5E\n/l9Hq87ExsZ4fMuWqvdzIEWadllWIvruDK74AAD0w4atW2tQKqM2Za/qAgYHH+/xAQAAkqf4AAAA\nyXOrG8AAMBkDAAwtJjfo6TjV7qNgbwqtBW+47e04B9ebmU1uMHCKdB507ce/sf2Pc3Bl7dqP82D/\n4wzO2NZKkd6ELysRfXcGt7oBAADJU3wAAIDkKT4AAImZ1NQUWZZV/YiIqvcxqalpUPIOVlaKy3t8\nejpOtfso2L3RtZDavdHuO3+t+/Aen4FSpPOgaz/+je1/nIMra9d+nAf7H8fY9rjOEBlb750qPu/x\nAQAADmqKDwAAkDzFBwAA+mmovB/Je5JePR9gCgAA/bRh69YavB8pavMerK1ba7CXg4crPgAAQPJc\n8QGAAVKOt0U5ShER8bYox+K4LiIiSlGOUtxVx2QABx/TWfd0nGr3UbCpKmuhSFNVdu3HNKD7H6c4\nWWvFv7GejuPfWI/rDJFzwXnQ23GKcx4Y217WGSJjW6SsXfsZnOm3i8R01gAAwEFN8QEAAJKn+AAA\nAMkzuQEAYCIGIHkmN+jpONXuo2BvXKyF1N4MWE3Wl//yUI5SlKIcEa/9l4cijW2R3iQe4d/YPrU+\nZyO88XqgFClrhPOg5+MM/Z9j3Y9zcI1tkbJ27cfkBq/UV2eoSfFZu3ZtXHnlldHZ2RkLFy6Ma665\nZr91/uiP/ijWrFkThx12WNx6660xc+bMfm97oC+ilop0Mhfph1uEsX2tijS2fikbOMa2t+McXOdt\nkbJGOA96Pk6xfo4dbGNbpKxd+1F8XmlAZ3Xr7OyMK664Ir773e/Ggw8+GMuXL4+f/exn3dZZs2ZN\nPProo/GLX/wili1bFh/+8If7vS0AAEC1qn6PT3t7e0yePDkmTpwYERELFiyIVatWRWtra2WdVatW\nxcUXXxwREaeeemo8++yzsXXr1vjlL395wG0BAIDBVS53PfYtl0pdy6XSS8tFU3Xx6ejoiJaWlsrz\nCRMmRHt7+wHX6ejo6Ne2AADA4Hp5wcmyl0pQkdVlVjf3IgIABwsz5lEvk5qaYsPWrTXYUx5ZVt17\nkiY2NsbjW7bUIMtrV3XxaW5ujo0bN1aeb9q0KZqbm/db54knnthvnd27dx9w25dbvHhxZblUKkVp\nAK6zTWxsjKwGJ0j1b1frynKg12uRtRYOlHXfOsb21SvS2A5m1lpwHgycwRzbg+m8LVLWCOfBS+76\nr0dExJKXLb12xrZnB1PWfVkOduVyOcr9vBxV9axue/fujalTp8a6deti3Lhxccopp8Ty5cujra2t\nss7q1avjy1/+cnznO9+J9evXx5VXXhnr16/v17aVoIM0q1stZFlEQaIWjrEdOMaWInLeEuE8GEhF\nGltZB06R8vbVGaq+4tPQ0BA33XRTzJ07tzIldVtbWyxbtiyyLItFixbFu971rli9enUcd9xxcdhh\nh8Utt9zS57YAAED9vHxyg7e9LWLfjVdFntzAB5gOgCK14qIxtgPH2FJEzlsinAcDqUhjKysRg/AB\npoNB8SHC2A4kY0tRpDjFKtXx/WvgFGlsZSVC8Rl0TuaBY2wHjrEFisr3r4Ez1Me2qH8IGerjWmSK\nzyBzMg8cYztwjC1QVL5/DRxjWztFLWlFo/gMMt8kBo6xHTjGFigq378GjrGlaPrqDMMGOQsAAMCg\nU3wAAIDkKT4AAEDyFB8AACB5ig8AAJA8s7oNADOgDBxjO3CMLVAkpgYeHH42UDSmsx5kvkkMHGM7\ncIwtAK/kZwNFYzprAADgoOaKzwDw15GBY2wHjrEFIMJthBSbW90GmV8gB46xHTjGFgAoOre6AQAA\nBzXFBwAASJ7iAwAAJE/xAQAAkqf4AAAAyVN8AACA5Ck+AABA8hQfAAAgeYoPAACQPMUHAABInuID\nAAAkT/EBAACSp/gAAADJU3wAAIDkKT4AAEDyFB8AACB5WZ7neb1D9EeWZTGUo5bLXY99y6VS13Kp\n9NIy1cuyiCF8GhSasQUAiq6vzqD4UCh+OR84xhYAKLq+OoNb3QAAgOQpPgAAQPIUHwAAIHmKDwAA\nkDzFBwAASJ5Z3RjyTBU+OMzqBgAUnemsgQNSfACAojOdNQAAcFBTfAAAgOQpPgAAQPIUHwAAIHmK\nDwAAkDzFBwAASJ7iAwAAJE/xAQAAkqf4AAAAyVN8AACA5Ck+AABA8hQfAAAgeYoPAACQPMUHAABI\nnuIDAAAkT/EBAACSp/gAAADJU3wAAIDkKT4AAEDyFB8AACB5ig8AAJC8LM/zvN4h+iPLsihIVCiM\ncrnrsW+5VOpaLpVeWgYAKIq+OoPiAwAAJKGvzuBWNwAAIHmKDwAAkDzFBwAASJ7iAwAAJE/xAQAA\nkqf4AAAAyVN8AACA5Ck+AABA8hQfAAAgeVUVnx07dsTcuXNj6tSpceaZZ8azzz7b43pr166N1tbW\nmDJlSixdurTy35csWRITJkyIE088MU488cRYu3ZtNXEAAAB6VFXx+cxnPhPvfOc74+c//3m8/e1v\nj+uvv36/dTo7O+OKK66I7373u/Hggw/G8uXL42c/+1nl9auuuiruvffeuPfee+Oss86qJs6QUS6X\n6x3hVSlS3iJljShWXlkHTpHyFilrRLHyyjpwipS3SFkjipVX1oFTtLy9qar4rFq1Ki655JKIiLjk\nkkti5cqV+63T3t4ekydPjokTJ8aIESNiwYIFsWrVqsrreZ5XE2FIKtrJUaS8RcoaUay8sg6cIuUt\nUtaIYuWVdeAUKW+RskYUK6+sA6doeXtTVfF56qmnorGxMSIimpqa4qmnntpvnY6Ojmhpaak8nzBh\nQnR0dFSe33TTTTFz5sz40Ic+1OutcgAAANU4YPGZM2dOnHDCCZXH9OnT44QTTog77rhjv3WzLHtV\nB7/88svjsccei/vvvz+ampriqquuelXbAwAA9EtehdbW1nzLli15nuf5k08+mbe2tu63zr/927/l\nZ555ZuX59ddfn3/mM5/Zb73HH388nz59eq/HiggPDw8PDw8PDw8PD48+H70ZHlWYN29e3HrrrXHN\nNdfEbbfdFuecc85+68yePTseeeSR2LBhQ4wbNy5WrFgRy5cvj4iILVu2RFNTU0REfOMb34hp06b1\neqw8wfcCAQAAgyPLq2gU27dvj/e9733xxBNPxMSJE+PrX/96vOlNb4onn3wyLrvssvj2t78dEV3T\nWX/0ox+Nzs7OWLhwYVx77bUREXHxxRfH/fffH8OGDYtJkybFsmXLKu8ZAgAAqJWqig8AAEARVDWr\nG9399re/jVNPPTVmzZoV06dPjyVLltQ7Up8WLlwYjY2NccIJJ9Q7ygFt2rQp3v72t8db3vKWmD59\nenzxi1+sd6RePfzwwzFr1qw48cQTY9asWXH44YcP6by9fcDwUPWFL3whpk2bFieccEJceOGFsXv3\n7npH6tWNN94Y06dPH7LnbE/fA/7pn/4ppk2bFg0NDXHvvffWMV13PWX9xCc+ETNmzIhZs2bFWWed\nFVu2bKljwu56ynv11VdHW1tbzJw5M84999zYuXNnHRP2btKkSZVxPeWUU+od54A6OzvjxBNPjHnz\n5tU7yn56+zn7pS99Kdra2mL69OmVu2DqraesRfqg+WeffTbOP//8aGtri7e85S3xwx/+sN6RKnoa\n2x07dsTcuXNj6tSpceaZZw6pmY37+v3wc5/7XAwbNiy2b99eh2Q10M95DOinXbt25Xme53v27MlP\nPfXU/Ic//GGdE/Xu7rvvzu+7774+J5UYKp588sn8vvvuy/M8z5977rl8ypQp+UMPPVTnVAe2d+/e\nfNy4cfnGjRvrHaVHe/fuzY899tj88ccfz3fv3p3PmDFjSI9rR0dHfvTRR+e//e1v8zzP8/e97335\nbbfdVudUPfvJT36ST58+Pf/Nb36T79mzJ58zZ07+6KOP1jtWNz19D/jZz36WP/zww/kZZ5yR//jH\nP65juu56yvrcc89Vlr/4xS/mH/7wh+sRrUc95f3+97+f7927N8/zPL/mmmvya6+9tl7x+nT00Ufn\n27dvr3eMfvv85z+fX3jhhfm73/3uekfZT0/nwZ133pnPmTMnf/HFF/M8z/Nt27bVK143PWVdvHhx\n/rnPfa6Oqfrvkksuyf/+7/8+z/M8f/HFF/Nnn322zole0tPYXn311fnSpUvzPM/zz3zmM/k111xT\nr3j76e33wyeeeCI/88wz80mTJuXPPPNMndJVxxWfGjv00EMjouvqz549e171FN+D6fTTT4/Ro0fX\nO0a/NDU1xcyZMyMiYuTIkdHW1tbt86CGqn/5l3+JY489tttnWQ0lB/qA4aFo7969sWvXrtizZ0+8\n8MILMX78+HpH6tFDDz0Up556ahxyyCHR0NAQ//2///f4xje+Ue9Y3fT0PWDq1KkxefLkITehTE9Z\nR44cWVnetWtXDBs2dH6k9ZT3ne98ZyXjaaedFps2bapHtAPK8zw6OzvrHaNfNm3aFKtXr44PfehD\n9Y7So57Og//9v/93XHvttTF8eNf8UmPHjq1HtP309jvBUPte0JOdO3fG3XffHZdeemlERAwfPjxG\njRpV51Qv6WlsV61aFZdccklERFxyySWxcuXKekTrUW/nwsc+9rH47Gc/W4dEtTN0fkokorOzM2bN\nmhVNTU0xZ86cmD17dr0jJefxxx+P+++/P0499dR6Rzmgf/zHf4wLLrig3jF6daAPGB5qxo8fHx//\n+MfjqKOOiubm5njTm94U73znO+sdq0fTpk2Lu+++O3bs2BEvvPBCrF69Op544ol6x0rOn/7pn8ZR\nRx0VX/va1+KTn/xkveP029///d/H2WefXe8YPcqyrPLz6+abb653nD7t+0VsKP+R8ZUefvjh+Nd/\n/dc47bTT4owzzogf/ehH9Y7UpyJ80Pwvf/nLGDt2bFx66aVx4oknxqJFi+LXv/51vWP16amnnqpM\n6NXU1BRPPfVUnRP17Y477oiWlpaYPn16vaNURfGpsWHDhsV9990XmzZtih/+8Ifx05/+tN6RkvL8\n88/HeeedFzfeeGO3v/YORS+++GLccccdcf7559c7SjJ+9atfxapVq2LDhg2xefPmeP755+NrX/ta\nvWP1qLW1Na655pqYM2dOvOtd74pZs2ZFQ0NDvWMl58///M9j48aNceGFF8aXvvSlesfpl7/4i7+I\nESNGxPvf//56R+nRPffcE/fee2+sXr06vvzlL8cPfvCDekfq0Xe+851obGyMmTNnRp7nhbgyERGx\nZ8+e2LFjR6xfvz5uuOGGeN/73lfvSL0qygfN79mzJ+699974H//jf8S9994bhx56aHzmM5+pd6xX\nZSiX91//+tfx6U9/utt714vy7+2VFJ8BMmrUqDjjjDOG9BsBi2bPnj1x3nnnxQc+8IEePzNqqFmz\nZk2cdNJJceSRR9Y7Sq+am5tj48aNleebNm2K5ubmOibq27/8y7/EMcccE2PGjImGhoZ473vfG//v\n//2/esfq1aWXXho/+tGPolwux5ve9KaYMmVKvSMl6/3vf3/88z//c71jHNCtt94aq1evHrKFPSJi\n3LhxERFx5JFHxvz586O9vb3OiXp2zz33xB133BHHHHNMXHDBBXHnnXfGxRdfXO9YB9TS0hLvfe97\nI6Lrsw6HDRsWzzzzTJ1T9ezII4+s/EJ+2WWXxb//+7/XOVHPJkyYEC0tLXHyySdHRMR55503pCZn\n6UljY2Ns3bo1Iro+1/LNb35znRP17tFHH43HH388ZsyYEUcffXRs2rQpTjrppCF/laonik8NPf30\n05XLwL/+9a/j+9//frS2ttY5Vd+K9FeyD37wg3H88cfHRz/60XpH6Zfly5cP6dvcIrp/wPDu3btj\nxYoVQ3JmpH2OOuqoWL9+ffzmN7+JPM9j3bp10dbWVu9Yvdq2bVtERGzcuDG++c1vDsm/8Pf1PWCo\nfW94ZdZHHnmksrxy5cohdy68Mu/atWvjs5/9bNxxxx1xyCGH1DFZ71544YV4/vnnI6LrfVPf+973\n+vxw8Xr69Kc/HRs3bozHHnssVqxYEW9/+9vj9ttvr3es/bzyPHjPe94T//f//t+I6Lrt7cUXX4wj\njjiiXvG6eWXWl8+UeKAPmq+nxsbGaGlpiYcffjgiItatWxfHH398nVN198qxnTdvXtx6660REXHb\nbbcNuT/ovjzvtGnTYsuWLfHYY4/FL3/5y5gwYULcd999Q7qs9Wrw5lFI3wMPPJDPmjUrnzFjRj59\n+vT8z//8z+sdqU8XXHBBPm7cuPx1r3td3tLSUpkNZSj6wQ9+kA8bNiyfMWNGPnPmzHzWrFn5mjVr\n6h2rV7t27crHjh2b79y5s95RDmjNmjX5lClT8uOOOy6//vrr6x3ngBYvXpy3trbm06dPzy+++OJ8\n9+7d9Y7Uq7e+9a35W97ylnzmzJn5nXfeWe84++npe8A3v/nNfMKECfnrX//6vKmpKT/rrLPqHTPP\n856znnvuufm0adPyGTNm5PPmzcs3b95c75gVPeU97rjj8qOOOiqfNWtWPmvWrPwjH/lIvWPu57HH\nHqt8n502bVohvifkeZ6Xy+UhOatbT+fBiy++mF900UX5tGnT8pNOOikvl8v1jpnnec9ZP/CBD+TT\np0/PZ8yYkZ9zzjn5li1b6h2zV/fff39+8skn5zNmzMjnz5+f/+pXv6p3pIqexnb79u35O97xjnzK\nlCn5nDlz8h07dtQ7ZsWBfj88+uijCzurmw8wBQAAkudWNwAAIHmKDwAAkDzFBwAASJ7iAwAAJE/x\nAQAAkqf4AAAAyVN8AACA5Ck+AABA8v4/MBDuHtLylf8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x9ee0748>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "\n",
    "# Build a classification task using 3 informative features\n",
    "\n",
    "# Build a forest and compute the feature importances\n",
    "forest = ExtraTreesClassifier(n_estimators=250,\n",
    "                              random_state=0)\n",
    "\n",
    "forest.fit(X, y)\n",
    "importances = forest.feature_importances_\n",
    "std = np.std([tree.feature_importances_ for tree in forest.estimators_],\n",
    "             axis=0)\n",
    "indices = np.argsort(importances)[::-1]\n",
    "\n",
    "# Print the feature ranking\n",
    "print(\"Feature ranking:\")\n",
    "\n",
    "for f in range(X.shape[1]):\n",
    "    print(\"%d. feature %d - %s (%f) \" % (f + 1, indices[f], features[indices[f]], importances[indices[f]]))\n",
    "\n",
    "# Plot the feature importances of the forest\n",
    "plt.figure(num=None, figsize=(14, 10), dpi=80, facecolor='w', edgecolor='k')\n",
    "plt.title(\"Feature importances\")\n",
    "plt.bar(range(X.shape[1]), importances[indices],\n",
    "       color=\"r\", yerr=std[indices], align=\"center\")\n",
    "plt.xticks(range(X.shape[1]), indices)\n",
    "plt.xlim([-1, X.shape[1]])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.13860564,  0.13850845,  0.11390376,  0.083274  ,  0.08325078])"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "importances[indices[:5]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. feature 3 - Milk (0.138606)\n",
      "2. feature 1 - Feathers (0.138508)\n",
      "3. feature 2 - Eggs (0.113904)\n",
      "4. feature 7 - Toothed (0.083274)\n",
      "5. feature 0 - Hair (0.083251)\n"
     ]
    }
   ],
   "source": [
    "for f in range(5):\n",
    "    print(\"%d. feature %d - %s (%f)\" % (f + 1, indices[f], features[indices[f]] ,importances[indices[f]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "best_features = []\n",
    "for i in indices[:5]:\n",
    "    best_features.append(features[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe8AAAF6CAYAAAAwItA9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt4FPW9x/HPJARELppwza0JQkIWgRAgkGJ4zlohCFQQ\nFQxS8YLGI2KhtKC2toT2VIXT2mNNqSm2iNaC1FpSbpHTHBaVxxAULQItikgC4X6R64GY7O/8wWEk\nkuDCJll+4f16nn2emezvN/Pbbyb7yczOzDrGGCMAAGCNsFAPAAAAXBzCGwAAyxDeAABYhvAGAMAy\nhDcAAJYhvAEAsAzhDVxhHn74Yf385z8P9TAABMHhOm8gMImJidq3b5+aNGkiY4wcx9HHH3+sjh07\nXvIyV69ere985zvasWNHHY7UHvfdd5/i4+P105/+NNRDAazCnjcQIMdxtGzZMh09elTHjh3T0aNH\ngwpuSe4/AZeqqqoqqPWHkt/vD/UQAGsR3sBFqO1AVXFxsW644QZFRkYqLS1Nq1evdp976aWX1K1b\nN7Vu3VpdunTR7373O0nSyZMnNWzYMO3atUutWrVS69attWfPHt133336yU9+4vZfvXq14uPj3flO\nnTpp9uzZSk1NVcuWLeX3+7V7927dcccdat++vTp37qznn3++1tdw7vLPLvs///M/1aFDB8XGxqqg\noEArVqxQ165d1bZtWz399NNu35kzZ2r06NHKzs5W69at1bdvX23YsMF9/l//+pduvPFGRUZGqkeP\nHlqyZEm19U6cOFHDhw9Xq1at9Pvf/16vvvqqZs+erdatW2vkyJGSpFmzZqlLly5q3bq1unfvrsWL\nF7vLmD9/vgYOHKhp06YpKipKnTt3VmFhofv84cOHdf/99ys2NlZt2rTRbbfd5j63dOlSpaWlKTIy\nUpmZmfroo4/c52bNmqW4uDi1bt1aHo9Hq1atqrV+wGXBAAhIYmKiKSoqOu/n5eXlpk2bNqawsNAY\nY8zf//5306ZNG3PgwAFjjDHLly83n332mTHGmLfeestcffXV5oMPPjDGGOPz+Ux8fHy15d17773m\nxz/+sTv/1TaJiYkmLS3NlJeXm1OnThm/32/69Olj/uM//sNUVlaazz77zHTu3NmsXLmyxtdx7vJ9\nPp9p0qSJ23fu3LmmXbt2Zty4cebEiRNm06ZNpnnz5mb79u3GGGNyc3NN06ZNzRtvvGEqKyvNL37x\nC9OpUydTWVlpvvjiC9OlSxfzzDPPmC+++ML8z//8j2nVqpX5+OOP3fVee+215t133zXGGHPq1Knz\nXqsxxrz++utmz549xhhjFi1aZFq0aOHOv/TSS6Zp06bm97//vfH7/ea3v/2tiYmJcfsOGzbMZGdn\nmyNHjpjKykrz1ltvGWOMWb9+vWnfvr1Zt26d8fv95uWXXzaJiYmmoqLCbNmyxcTHx7vrKC0tNdu2\nbauxdsDlgj1v4CLceuutioqKUlRUlLtX98c//lHDhw/XkCFDJEk33XST+vbtq+XLl0uShg4dqsTE\nREnSwIEDlZWVpbfffjuocUyePFkxMTFq1qyZ1q1bpwMHDuhHP/qRwsPDlZiYqAceeEALFy4MaFlN\nmzbVD3/4Q4WHhys7O1sHDhzQlClTdPXVV6tbt27q1q2b/vGPf7jt+/Tpo1GjRik8PFxTp07V6dOn\nVVxcrOLiYp04cUKPPfaYmjRpohtvvFHf/va3tWDBArfvyJEjlZGRIUlq1qxZjeO5/fbb1aFDB0nS\n6NGjlZSUpJKSEvf5hIQE3X///XIcR/fcc492796tffv2ac+ePXrzzTeVn5+v1q1bKzw8XAMHDpQk\nzZ07V//+7/+uvn37ynEc3X333WrWrJmKi4sVHh6uiooKbdy4UZWVlfrGN76hTp06XdwvBGhghDdw\nEQoKCnTo0CEdOnRIb7zxhiSptLRUixYtckM9MjJSa9as0e7duyVJK1as0De/+U21adNGkZGRWrFi\nhQ4cOBDUOOLi4tzp0tJSlZeXV1v/008/rX379gW0rDZt2rifuzdv3lyS1L59e/f55s2b6/jx4+78\nuYfwHcdRbGysdu3apV27dlV7TjoTtOXl5TX2rc3LL7/sHt6OjIzUpk2bqtXr3PMMzo73+PHj2rFj\nh6KiotS6devzlllaWqpf/vKX1Wq0c+dO7dq1S507d9Z//dd/KTc3Vx06dNBdd93l/u6AyxXhDVwE\nU8Nn3vHx8Ro/frwb6ocPH9axY8c0ffp0VVRU6I477tD06dO1f/9+HT58WEOHDnWXU9PJai1atNDJ\nkyfd+ZqC5Nx+8fHxuu6666qt/8iRI9U+b65L554Zb4zRzp07FRMTo5iYGJWVlVVrW1ZWptjY2BrH\nXdN8WVmZcnJyNGfOHB0+fFiHDx/W9ddfX+u5BueKj4/XoUOHdPTo0Rqf+9GPflStRsePH9edd94p\nScrOztbbb7+t0tJSSdLjjz/+tesDQonwBoL0ne98R0uWLNHKlSvl9/t16tQprV69Wrt27VJFRYUq\nKirUtm1bhYWFacWKFVq5cqXbt0OHDjp48GC1wOnVq5eWL1+uw4cPa8+ePXruuecuuP5+/fqpVatW\nmj17tk6dOqWqqipt2rRJ7733Xr283vfff1+LFy9WVVWVfvWrX+mqq65SRkaG+vfvrxYtWmj27Nmq\nrKyUz+fT0qVLNXbs2FqX1aFDB23bts2dP3HihMLCwtS2bVv5/X7NmzdPGzduDGhcHTt21NChQzVx\n4kR9/vnnqqysdD+eePDBB/XCCy+4h99PnDih5cuX68SJE/r444+1atUqVVRUqGnTpmrevLnCwnhr\nxOWNLRQIUG2XdMXFxamgoEBPPfWU2rVrp4SEBP3iF7+Q3+9Xy5Yt9etf/1qjR49WVFSUFi5c6J5V\nLUldu3bV2LFjdd111ykqKkp79uzR3XffrZ49eyoxMVE333yzsrOzLziOsLAwLV26VB9++KE6deqk\n9u3b68EHH6xxD/RSXudX50eOHKnXXntNkZGRevXVV/XXv/5V4eHhioiI0JIlS7R8+XK1bdtWkyZN\n0iuvvKKkpKRa6zdhwgRt2rTJPYfA4/Fo6tSpysjIUMeOHbVp0yZlZmYGPN5XXnlFTZo0UUpKijp0\n6OD+49OnTx/NnTtXkyZNUlRUlJKTkzV//nxJ0unTp/X444+rXbt2iomJ0f79+6udYQ9cjurkJi2F\nhYWaMmWK/H6/JkyYoMcee6za83/729/04x//WGFhYYqIiNCvfvUr3XDDDQH1BXD5mDlzpj799FO9\n/PLLoR4KcEVrEuwC/H6/Jk2apKKiIsXExCg9PV0jR45USkqK22bQoEEaMWKEJOmjjz7SmDFj9M9/\n/jOgvgAAoLqgD5uXlJQoKSlJCQkJioiIUHZ2tgoKCqq1ufrqq93p48ePu58nBdIXAABUF/Sed3l5\nebXLP+Li4qpdk3nW4sWL9cQTT2j//v1atmzZRfUFcHmYMWNGqIcAQA14wtqtt96qf/7zn1q8eLGe\nfPLJhlotAACNTtB73rGxsdWu7dy5c2e16zq/KjMzU9u2bdOhQ4cuqm8wX94AAICNaj2nPNj7q1ZW\nVprOnTub7du3m9OnT5vU1FSzefPmam22bt3qTr///vsmLi4u4L5n1cFQG61Vq4yZMePMIyFhhju9\nalUoR3X5mzFjRqiHYAXqFDhqFRjqFJgL5V7Qe97h4eHKy8tTVlaWe7mXx+NRfn6+HMdRTk6O/vKX\nv+jll192b4CwaNGiC/bFxfF6zzwkaeZMKTc3hIMBANS7oMNbkm6++WZt2bKl2s8eeughd3r69Oma\nPn16wH0BAEDtuMNao+MN9QCs4T17uAIXRJ0CR60CQ52CVyd3WGsIjuME9OUEVzrHkSgTANjvQrnH\nnjcAAJYhvAEAsAzhDQCAZQhvAAAsQ3gDAGAZwhsAAMsQ3gAAWIbwBgDAMoQ3AACWIbwBALAM4Q0A\ngGUIbwAALEN4AwBgGcIbAADLEN4AAFiG8AYAwDKENwAAliG8AQCwDOENAIBlCG8AACxDeAMAYBnC\nGwAAyxDeAABYhvAGAMAyhDcAAJYhvAEAsAzhDQCAZQhvAAAsQ3gDAGAZwhsAAMsQ3gAAWIbwBgDA\nMoQ3AACWIbwBALAM4Q0AgGUIbwAALEN4AwBgmToJ78LCQqWkpCg5OVmzZs067/k//elPSk1NVWpq\nqjIzM7Vhwwb3ucTERKWmpiotLU39+vWri+EAANCoOcYYE8wC/H6/kpOTVVRUpJiYGKWnp2vhwoVK\nSUlx2xQXF8vj8eiaa65RYWGhcnNzVVxcLEm67rrr9P777ysyMvLCA3UcBTnUK4LjSJQJwfL5zjzO\nTnu9Z6a93i+nAdSvC+Vek2AXXlJSoqSkJCUkJEiSsrOzVVBQUC28MzIyqk2Xl5e788YY+f3+YIcB\noA6dG9KO82WQA7g8BH3YvLy8XPHx8e58XFxctXD+qhdffFFDhw515x3H0eDBg5Wenq65c+cGOxwA\nABq9oPe8L8aqVas0b948vfPOO+7P1qxZo+joaO3fv1+DBw+Wx+NRZmZmQw4LAACrBB3esbGxKisr\nc+d37typ2NjY89pt2LBBOTk5KiwsrPb5dnR0tCSpXbt2GjVqlEpKSmoN79zcXHfa6/XKy4dvAIBG\nwufzyRfgZ1RBn7BWVVWlrl27qqioSNHR0erXr58WLFggj8fjtikrK9NNN92kV155pdrn3ydPnpTf\n71fLli114sQJZWVlacaMGcrKyjp/oJywFhBOWENdY5sCQqNeT1gLDw9XXl6esrKy5Pf7NWHCBHk8\nHuXn58txHOXk5OhnP/uZDh06pIkTJ8oYo4iICJWUlGjv3r0aNWqUHMdRZWWlxo0bV2NwAwCALwW9\n591Q2PMODHtJqGtsU0BoXCj3uMMaAACWIbwBALAM4Q0AgGUIbwAALEN4AwBgmQa9wxoQanzhBoDG\ngEvFGhku6wkctQoMdQJCg0vFAABoRAhvAAAsQ3gDAGAZwhsAAMsQ3gAAWIbwBgDAMoQ3AACWIbwB\nALAM4Q0AgGUIbwAALEN4AwBgGcIbAADLEN4AAFiG8AYAwDKENwAAliG8AQCwDOENAIBlCG8AACxD\neAMAYBnCGwAAyxDeAABYhvAGAMAyhDcAAJYhvAEAsAzhDQCAZQhvAAAsQ3gDAGAZwhsAAMsQ3gAA\nWIbwBgDAMk1CPQAAQOPn8515nJ32es9Me71fTiNwjjHGhHoQgXAcR5YMNaQcR6JMgaFWgaFOqGts\nU4G5UO7VyWHzwsJCpaSkKDk5WbNmzTrv+T/96U9KTU1VamqqMjMztWHDhoD7AgCA6oLe8/b7/UpO\nTlZRUZFiYmKUnp6uhQsXKiUlxW1TXFwsj8eja665RoWFhcrNzVVxcXFAfd2BsucdEP6jDRy1Cgx1\nQl1jmwpMve55l5SUKCkpSQkJCYqIiFB2drYKCgqqtcnIyNA111zjTpeXlwfcFwAAVBd0eJeXlys+\nPt6dj4uLc8O5Ji+++KKGDh16SX0BAEADn22+atUqzZs3T++8805DrhYAgEYl6PCOjY1VWVmZO79z\n507Fxsae127Dhg3KyclRYWGhIiMjL6rvWbm5ue601+uVl+sLAACNhM/nk+/s9XRfI+gT1qqqqtS1\na1cVFRUpOjpa/fr104IFC+TxeNw2ZWVluummm/TKK68oIyPjovq6A+WEtYBwIkjgqFVgqBPqGttU\nYC6Ue0HveYeHhysvL09ZWVny+/2aMGGCPB6P8vPz5TiOcnJy9LOf/UyHDh3SxIkTZYxRRESESkpK\nau0LAABqx01aGhn+ow0ctQoMdUJdY5sKTL3fpAUAADQcwhsAAMsQ3gAAWIbwBgDAMoQ3AACWIbwB\nALAM4Q0AgGUIbwAALEN4AwBgGcIbAADLEN4AAFiG8AYAwDKENwAAliG8AQCwDOENAIBlCG8AACxD\neAMAYBnCGwAAyxDeAABYhvAGAMAyhDcAAJYhvAEAsAzhDQCAZQhvAAAsQ3gDAGAZwhsAAMsQ3gAA\nWIbwBgDAMoQ3AACWIbwBALAM4Q0AgGWahHoAAGAzn+/M4+y013tm2uv9chqoa44xxoR6EIFwHEeW\nDDWkHEeiTIGhVoGhToGjVoGhToG5UO5x2BwAAMsQ3gAAWIbwBgDAMoQ3AACWIbwBALAM4Q0AgGXq\nJLwLCwuVkpKi5ORkzZo167znt2zZogEDBuiqq67Ss88+W+25xMREpaamKi0tTf369auL4QAA0KgF\nfZMWv9+vSZMmqaioSDExMUpPT9fIkSOVkpLitmnTpo2ef/55LV68+Lz+YWFh8vl8ioyMDHYoAABc\nEYLe8y4pKVFSUpISEhIUERGh7OxsFRQUVGvTtm1b9enTR02anP+/gjFGfr8/2GEAAHDFCDq8y8vL\nFR8f787HxcWpvLw84P6O42jw4MFKT0/X3Llzgx0OAACNXsjvbb5mzRpFR0dr//79Gjx4sDwejzIz\nM2tsm5ub6057vV55uXEwAKCR8Pl88p29Uf7XCDq8Y2NjVVZW5s7v3LlTsbGxAfePjo6WJLVr106j\nRo1SSUlJQOENAEBj8tWd0pkzZ9baNujD5unp6dq6datKS0tVUVGhhQsXasSIEbW2P/cm6ydPntTx\n48clSSdOnNDKlSvVvXv3YIcEAECjFvSed3h4uPLy8pSVlSW/368JEybI4/EoPz9fjuMoJydHe/fu\nVd++fXXs2DGFhYXpueee0+bNm7V//36NGjVKjuOosrJS48aNU1ZWVl28LgAAGi2+ErSR4av2Aket\nAkOdAketAkOdAsNXggIA0IgQ3gAAWIbwBgDAMoQ3AACWIbwBALAM4Q0AgGVCfnvUxiyxY0eV7t3b\nwGs1chyngdcpJXTooO179lxS39DUSbKxVgAgcZ13vXIcRw09YkdGRg0fSI50yb+fUNRJsrNWocA1\nuYGjVoGhToHhOm8AABoRwhsAAMsQ3gAAWIbwBgDAMoQ3AACWIbwBALAM4Q0AgGUIbwAALEN4AwBg\nGcIbAADLEN4AAFiG8AYAwDKENwAAliG8AQCwDOENAIBlCG8AACxDeAMAYBnCGwAAyxDeAABYhvAG\nAMAyhDcAAJYhvAEAsEyTUA8AQGASO3ZU6d69IVizkeM4Db7WhA4dtH3PngZfL2ADxxhjQj2IQDiO\nI0uG6nIcRw09YkdGRg3/RutIl/z7CUWdJPtqRZ0uf44jWTbkkKBOgblQ7nHYHAAAyxDeAABYhvAG\nAMAyhDcAAJYhvAEAsAzhDQCAZQhvAAAsUyfhXVhYqJSUFCUnJ2vWrFnnPb9lyxYNGDBAV111lZ59\n9tmL6gsAAKoL+iYtfr9fycnJKioqUkxMjNLT07Vw4UKlpKS4bQ4cOKDS0lItXrxYkZGRmjp1asB9\n3YFyk5bA1mnhDTW4+UiA/ajTZY+bjwSGOgWmXm/SUlJSoqSkJCUkJCgiIkLZ2dkqKCio1qZt27bq\n06ePmjRpctF9AQBAdUGHd3l5ueLj4935uLg4lZeX13tfAACuVFZ9MUlubq477fV65fV6QzYWAADq\nks/nk8/nC6ht0OEdGxursrIyd37nzp2KjY2tl77nhjcAAI3JV3dKZ86cWWvboA+bp6ena+vWrSot\nLVVFRYUWLlyoESNG1Nr+3A/fL7YvAACogz3v8PBw5eXlKSsrS36/XxMmTJDH41F+fr4cx1FOTo72\n7t2rvn376tixYwoLC9Nzzz2nzZs3q2XLljX2BQAAteP7vOsRl4oF2JdLoALrR50ue1wCFRjqFBi+\nzxsAgEaE8AYAwDKENwAAliG8AQCwDOENAIBlCG8AACxDeAMAYBnCGwAAyxDeAABYhvAGAMAyhDcA\nAJax6vu8ASAQiR07qnTv3hCs2chxGvY+8AkdOmj7nj2X1PdKqpMUXK0uN3wxST3ii0kC7MsXbgTW\njzoF3vcKqhV1upj12vVlN3wxCQAAjQjhDQCAZQhvAAAsQ3gDAGAZwhsAAMsQ3gAAWIbwBgDAMoQ3\nAACWIbwBALAM4Q0AgGUIbwAALEN4AwBgGcIbAADLEN4AAFiG8AYAwDKENwAAliG8AQCwDOENAIBl\nCG8AACxDeAMAYBnCGwAAyxDeAABYhvAGAMAyhDcAAJYhvAEAsEydhHdhYaFSUlKUnJysWbNm1djm\nu9/9rpKSktSrVy998MEH7s8TExOVmpqqtLQ09evXry6GAwBAo9Yk2AX4/X5NmjRJRUVFiomJUXp6\nukaOHKmUlBS3zYoVK/Tpp5/qk08+0dq1a/Xwww+ruLhYkhQWFiafz6fIyMhghwIAwBUh6D3vkpIS\nJSUlKSEhQREREcrOzlZBQUG1NgUFBRo/frwkqX///jpy5Ij27t0rSTLGyO/3BzsMAACuGEGHd3l5\nueLj4935uLg4lZeXX7BNbGys28ZxHA0ePFjp6emaO3dusMMBAKDRC/qwebDWrFmj6Oho7d+/X4MH\nD5bH41FmZmaohwUAwGUr6PCOjY1VWVmZO79z507Fxsae12bHjh01tomOjpYktWvXTqNGjVJJSUmt\n4Z2bm+tOe71eeb3eYIcPAMBlwefzyefzBdbYBKmystJ07tzZbN++3Zw+fdqkpqaazZs3V2uzbNky\nM2zYMGOMMe+++67p37+/McaYEydOmGPHjhljjDl+/LgZMGCAefPNN2tcTx0MtcFJMqaBH2fOIgjF\nei/99xOKOtlYK+pErahT6GoVChcab9B73uHh4crLy1NWVpb8fr8mTJggj8ej/Px8OY6jnJwcDRs2\nTMuXL1eXLl3UokULzZs3T5K0d+9ejRo1So7jqLKyUuPGjVNWVlawQwIAoFFz/j/dL3uO48iSoboc\nx1FDj9iRkZHTwGuVHOmSfz+hqJNkX62o00X0vYJqRZ0uZr2XXqtQuFDucYc1AAAsQ3gDAGAZwhsA\nAMsQ3gAAWIbwBgDAMoQ3AACWIbwBALAM4Q0AgGUIbwAALMMd1uoRd1gLsG8D1smnf5NP3v+f9sor\nnyTJK5+8Wt0gY+AOa4Gu145tqtp6ucNaYOu1cJsKhQvlHuFdjwjvAPuG6A0kVAjvQNdr3zZFeAe4\nXgu3qVDg9qgAADQihDcAAJYhvAEAsAzhDQCAZQhvAAAsQ3gDAGAZwhsAAMsQ3gAAWIbwBgDAMoQ3\nAACWIbwBALAM4Q0AgGUIbwAALEN4AwBgGcIbAADLEN4AAFiG8AYAwDKENwAAliG8AQCwDOENAIBl\nCG8AACxDeAMAYBnCGwAAyxDeAABYhvAGAMAyhDcAAJYhvAEAsAzhDQCAZeokvAsLC5WSkqLk5GTN\nmjWrxjbf/e53lZSUpF69eunDDz+8qL4AAOBLjjHGBLMAv9+v5ORkFRUVKSYmRunp6Vq4cKFSUlLc\nNitWrFBeXp6WLVumtWvXavLkySouLg6orztQx1GQQ21wjuOooUfsyMjIaeC1So50yb+fUNQplC61\nVqGqE9vUhfn0b/LJ+//TXnnlkyR55ZNXq+t9/bbUqdp6LdymQuFCudck2IWXlJQoKSlJCQkJkqTs\n7GwVFBRUC+CCggKNHz9ektS/f38dOXJEe/fu1Wefffa1fQHgcubV6nNCemZIx4IrR9CHzcvLyxUf\nH+/Ox8XFqby8PKA2gfQFAADVheSENZsOWwAAcLkJ+rB5bGysysrK3PmdO3cqNjb2vDY7duw4r01F\nRcXX9j1Xbm6uO+31euX1eoMdfr1K6NBBzt69Db7ehv8k6cxrDaZvKOoUKpdaq1DWiW3q8mVrnWzb\nphqCz+eTz+cLqG3QJ6xVVVWpa9euKioqUnR0tPr166cFCxbI4/G4bZYvX67f/OY3WrZsmYqLizVl\nyhQVFxcH1NcdqIUnrIWC40iUCXWJbQp1jW0qMPV6wlp4eLjy8vKUlZUlv9+vCRMmyOPxKD8/X47j\nKCcnR8OGDdPy5cvVpUsXtWjRQvPmzbtgXwAAULug97wbCnvegeE/WtQ1tinUNbapwFwo97jDGgAA\nliG8AQCwDOENAIBlCG8AACxDeAMAYBnCGwAAyxDeAABYhvAGAMAyhDcAAJYhvAEAsAzhDQCAZQhv\nAAAsQ3gDAGAZwhsAAMsQ3gAAWIbwBgDAMo6p7Zu+LzMX+lLyK53Pd+ZxdtrrPTPt9X45DVwqx5H4\n00NdYpsKzIVyj/AGcEG80aKusU0F5kK5x2FzAAAsQ3gDAGAZwhsAAMvwmTeA83ASJOoTn3kHhhPW\nAACXDcI7MJywBgBAI0J4AwBgGcIbAADLEN4AAFiG8AYAwDKENwAAliG8AQCwDOENAIBlCG8AACxD\neAMAYBnCGwAAy3BvcwBAvePLbi4eX0wCAIBl+GISAAAaEcIbAADLEN4AAFgmqPA+fPiwsrKy1LVr\nVw0ZMkRHjhypsV1hYaFSUlKUnJysWbNmuT+fOXOm4uLi1Lt3b/Xu3VuFhYXBDAcAgCtCUOH9zDPP\naNCgQdqyZYu+9a1v6emnnz6vjd/v16RJk/Tmm29q06ZNWrBggf71r3+5z0+dOlXr16/X+vXrdfPN\nNwczHEjynT2dE1+LWgWGOgWOWgWGOgUvqPAuKCjQPffcI0m65557tHjx4vPalJSUKCkpSQkJCYqI\niFB2drYKCgrc5zmDvG7xRxE4ahUY6hQ4ahUY6hS8oMJ737596tChgySpY8eO2rdv33ltysvLFR8f\n787HxcWpvLzcnc/Ly1OvXr30wAMP1HrYHQAAfOlrw3vw4MHq2bOn++jRo4d69uypv/3tb+e1dRzn\nolY+ceJEbdu2TR9++KE6duyoqVOnXlR/AACuSCYIKSkpZs+ePcYYY3bv3m1SUlLOa/Puu++aIUOG\nuPNPP/20eeaZZ85rt337dtOjR49a1yWJBw8ePHjwuKIetWmiIIwYMUIvvfSSHnvsMc2fP18jR448\nr016erq2bt2q0tJSRUdHa+HChVqwYIEkac+ePerYsaMk6Y033lD37t1rXZfhs3EAACQFeXvUQ4cO\nacyYMdqxY4cSEhK0aNEiXXvttdq9e7cefPBBLV26VNKZS8UmT54sv9+vCRMm6PHHH5ckjR8/Xh9+\n+KHCwsKuBvpzAAAJXUlEQVSUmJio/Px89zN0AABQM2vubQ4AAM7gDmsWCQsL0/jx4935qqoqtWvX\nTiNGjJAkLVmyRLNnz5Z05gY4zz77rCTpxhtv1Pr16xt+wHUoPDxcvXv3Vlpamnr37q2ysrKLXsaR\nI0f029/+1p1fvXq1brnllrocpjW+Ws+z282V5NChQ+7rj46OVlxcnDtfWVkZ8HIOHz6s/Px8d76o\nqEijRo265HF9+umnSktLu+T+odaqVatq8/Pnz9ejjz56wT7nvnchMEF95o2G1aJFC23cuFGnT59W\ns2bN9N///d/VLsO75ZZbGm0YtWjRIuh/QA4fPqw5c+bo4Ycfdn92sVdInKuqqkrh4eFBjSlU6qKe\ntouKitIHH3wgSfrpT3+qli1bXtIVLwcPHtQLL7yghx56yP1ZMNtVXfQPpZrG/nWvp7b3Lpv/xuob\ne96WGTZsmJYtWyZJWrBggcaOHes+93X/4RpjdN999+knP/lJvY+zrtX06Y7f79f06dPVv39/9erV\nS3PnzpUknThxQoMGDVLfvn2VmpqqJUuWSJKeeOIJbdu2Tb1799Zjjz0mSTp27JhGjx4tj8eju+++\n2132+vXr5fV6lZ6erqFDh2rv3r2SzhzF+N73vqd+/frp17/+tV5//XX16NFDaWlp8lr0pcS1fVq2\nfPlyeTwepaena/Lkye4b6oEDB5SVlaUePXrowQcfVGJiog4dOqSTJ0/q29/+ttLS0tSzZ0/9+c9/\nbsiXUWe+Wo/Zs2e7l8Xm5eXV+PPf/OY3ks5sVx9//LF69+6tH/7wh5Kko0eP6vbbb1dKSoruvfde\nt/97773nblfDhw/X/v37JUnr1q1TamqqevfurRdeeKGeX23oLF26VBkZGerTp4+ysrLc13/ue9d9\n992nhx9+WBkZGe7fKWoQwBVhuEy0atXKfPTRR+aOO+4wp06dMr169TKrV682t9xyizHGmJdeesk8\n+uijxhhjcnNzzS9/+UtjjDFer9cUFxebsWPHmqeeeipk4w9GeHi4SUtLM7169TK33XabMcaY3/3u\nd+bnP/+5McaY06dPm759+5rt27ebqqoqc+zYMWOMMQcOHDBdunQxxpx/OaLP5zPXXnut2bVrl/H7\n/eab3/ymWbNmjfniiy/MgAEDzIEDB4wxxrz22mvm/vvvN8acqeUjjzziLqNHjx5m165dxhhjjhw5\nUs9VqDvn1jMtLc0sWrTInDp1ysTHx5vS0lJjjDFjx451t61Jkya5l3gWFhaasLAwc/DgQfOXv/zF\n5OTkuMs9evRow7+YOnDu38vatWtNr169zOnTp82xY8eMx+MxGzdurPXnW7duNWlpae6y/v73v5uo\nqCizZ88eU1VVZdLT083atWvN6dOnzYABA8zBgweNMca8+uqrbu2uv/568+677xpjjPne975XbXm2\nObttnd2+EhIS3Pelzz//3G334osvmu9///vGmOrvXffee6+73aF2HDa3TPfu3bV9+3YtWLBAw4cP\nD/gSuoceekh33nmnnnjiiXoeYf24+uqrzzvMu3LlSn300Ufu3t7Ro0f1ySefKDY2Vo8//rjefvtt\nhYWFadeuXTXe/U+S+vXrp+joaElSr169tH37dl1zzTXauHGjBg8eLGOM/H6/YmJi3D533nmnO52Z\nmal77rlHY8aM0W233VbXL7ve1FTPf/zjH+rcubO+8Y1vSJLGjh3rHs1455133NsfDxkyRJGRkZKk\nHj166Ac/+IGeeOIJDR8+XJmZmQ34KurHO++8o9tvv11NmzZV06ZNNWrUKL311lv63//932o/v/XW\nW/X2229r8ODB5y0jIyPDvXLm7HbVrFkzbdq0SYMGDXK3q/j4eB08eFCnTp1SRkaGJOnuu++2+vah\nX9225s+fr/fff1+StGPHDo0ZM0a7d+/WF198oU6dOtW4jNGjRzfIWG1GeFtoxIgRmjZtmnw+nw4c\nOBBQnxtuuEGrVq3S1KlT1axZs3oeYcMwxuj5558/781z/vz5OnjwoD744AOFhYWpU6dOOnXqVI3L\nOLcW4eHhqqyslDFG3bt315o1a2rs06JFC3d6zpw5WrdunZYuXao+ffpo/fr1brDZKNB/Bs+2S0pK\n0vr167V8+XI9+eSTGjRokJ588sn6HGKDM8a4n9kGWp/atqvU1FStXr26WtuDBw/W3WAvc48++qh+\n8IMfaPjw4Vq9erVmzpxZY7tz/8ZQMz7ztsjZN477779fM2bM0PXXXx9w3wkTJmjYsGEaM2aMqqqq\n6muI9aamN80hQ4Zozpw57pnBn3zyiU6ePKkjR46offv2CgsL06pVq1RaWirpzFmwx44d+9p1de3a\nVfv371dxcbEkqbKyUps3b66x7bZt25Senq6ZM2eqffv22rFjx6W+xAZVUz27du2qzz77zD2T/7XX\nXnOfu+GGG9z5lStX6vPPP5ck7d69W82bN9ddd92ladOmNYqT4AYOHKi//vWvOn36tI4fP66CggIN\nHDhQAwcO1OLFi8/7eaDbVbdu3VReXq5169ZJkr744gtt3rxZbdq0UfPmzbV27VpJ0quvvlqvr6++\nXegfnKNHj7pHsebPn99QQ2qU2PO2yNn//mNjYzVp0qSL7jdlyhR9/vnnGj9+vHVvEDWdrfrAAw9o\n+/bt6t27t4wxat++vRYvXqxx48bplltuUWpqqvr27SuPxyPpzNnFAwYMUM+ePTV06FANGzasxnVE\nRETo9ddf16OPPqojR46oqqpKU6ZMUbdu3c4bx7Rp0/TJJ59IkgYNGqSePXvWx8uvc6dOnXLr5jiO\nbr75Zj311FOaM2eOhgwZopYtWyo9Pd19vTNmzNBdd92lP/7xj8rIyFDHjh3VqlUrvffee5o2bZrC\nwsLUtGnTapfi2So9PV1jx45V37595TiOHnnkEfcf5dp+3qdPH6Wmpmr48OG66aabqi3vbA2bNm3q\nbldHjx6V3+/X97//fXXr1k1/+MMf9MADDyg8PLzGw/A2udCZ5TNmzNAdd9yhqKgofetb39L27dsv\nqj++xE1aALhOnDjhHrJ85JFHlJycrMmTJ6uiokLh4eEKDw9XcXGxJk6c2Cj2sgFbsecNwDV37lzN\nnz9fFRUV6t27t3vtcllZmcaMGSO/369mzZq5J7IBCA32vAEAsAwnrAEAYBnCGwAAyxDeAABYhvAG\nAMAyhDcAAJYhvAEAsMz/AcBDej2pSCa6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10dae128>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the top 5 feature importances of the forest\n",
    "plt.figure(num=None, figsize=(8, 6), dpi=80, facecolor='w', edgecolor='k')\n",
    "plt.title(\"Feature importances\")\n",
    "plt.bar(range(5), importances[indices][:5], \n",
    "       color=\"r\",  yerr=std[indices][:5], align=\"center\")\n",
    "plt.xticks(range(5), best_features)\n",
    "plt.xlim([-1, 5])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Decision Tree accuracy and time elapsed caculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DecisionTree\n",
      "Acurracy:  0.7\n",
      "time elapsed:  0.0350000858307\n"
     ]
    }
   ],
   "source": [
    "t0=time()\n",
    "print \"DecisionTree\"\n",
    "\n",
    "dt = DecisionTreeClassifier(min_samples_split=20,random_state=99)\n",
    "# dt = DecisionTreeClassifier(min_samples_split=20,max_depth=5,random_state=99)\n",
    "\n",
    "clf_dt=dt.fit(X_train,y_train)\n",
    "\n",
    "print \"Acurracy: \", clf_dt.score(X_test,y_test)\n",
    "t1=time()\n",
    "print \"time elapsed: \", t1-t0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## cross validation for DT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cross result========\n",
      "[ 0.55555556  0.625       0.71428571]\n",
      "0.631613756614\n",
      "time elapsed:  0.055999994278\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tt0=time()\n",
    "print \"cross result========\"\n",
    "scores = cross_validation.cross_val_score(dt, X, y, cv=3)\n",
    "print scores\n",
    "print scores.mean()\n",
    "tt1=time()\n",
    "print \"time elapsed: \", tt1-tt0\n",
    "print \"\\n\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Tuning our hyperparameters using GridSearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  34 tasks      | elapsed:   17.4s\n",
      "[Parallel(n_jobs=-1)]: Done  81 out of  81 | elapsed:   18.1s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 27 candidates, totalling 81 fits\n",
      "Best score: 0.719\n",
      "Best parameters set:\n",
      "\tclf__max_depth: 5\n",
      "\tclf__min_samples_leaf: 1\n",
      "\tclf__min_samples_split: 5\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        1.0       0.50      1.00      0.67         1\n",
      "        2.0       1.00      1.00      1.00         2\n",
      "        3.0       1.00      0.86      0.92         7\n",
      "\n",
      "avg / total       0.95      0.90      0.91        10\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('clf', DecisionTreeClassifier(criterion='entropy'))\n",
    "])\n",
    "\n",
    "parameters = {\n",
    "    'clf__max_depth': (5, 25 , 50),\n",
    "    'clf__min_samples_split': (1, 5, 10),\n",
    "    'clf__min_samples_leaf': (1, 2, 3)\n",
    "}\n",
    "\n",
    "grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=1, scoring='f1')\n",
    "grid_search.fit(X_train, y_train)\n",
    "\n",
    "print 'Best score: %0.3f' % grid_search.best_score_\n",
    "print 'Best parameters set:'\n",
    "\n",
    "best_parameters = grid_search.best_estimator_.get_params()\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print '\\t%s: %r' % (param_name, best_parameters[param_name])\n",
    "\n",
    "predictions = grid_search.predict(X_test)\n",
    "\n",
    "print classification_report(y_test, predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Random Forest accuracy and time elapsed caculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RandomForest\n",
      "Acurracy:  1.0\n",
      "time elapsed:  1.02699995041\n"
     ]
    }
   ],
   "source": [
    "t2=time()\n",
    "print \"RandomForest\"\n",
    "rf = RandomForestClassifier(n_estimators=100,n_jobs=-1)\n",
    "clf_rf = rf.fit(X_train,y_train)\n",
    "print \"Acurracy: \", clf_rf.score(X_test,y_test)\n",
    "t3=time()\n",
    "print \"time elapsed: \", t3-t2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## cross validation for RF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cross result========\n",
      "[ 0.88888889  0.875       0.85714286]\n",
      "0.873677248677\n",
      "time elapsed:  3.34599995613\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tt2=time()\n",
    "print \"cross result========\"\n",
    "scores = cross_validation.cross_val_score(rf, X, y, cv=3)\n",
    "print scores\n",
    "print scores.mean()\n",
    "tt3=time()\n",
    "print \"time elapsed: \", tt3-tt2\n",
    "print \"\\n\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tuning Models using GridSearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  34 tasks      | elapsed:   19.2s\n",
      "[Parallel(n_jobs=-1)]: Done 184 tasks      | elapsed:   26.8s\n",
      "[Parallel(n_jobs=-1)]: Done 324 out of 324 | elapsed:   34.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 108 candidates, totalling 324 fits\n",
      "Best score: 0.786\n",
      "Best parameters set:\n",
      "\tclf__max_depth: 5\n",
      "\tclf__min_samples_leaf: 1\n",
      "\tclf__min_samples_split: 1\n",
      "\tclf__n_estimators: 5\n",
      "Accuracy: 0.9\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        1.0       0.50      1.00      0.67         1\n",
      "        2.0       1.00      1.00      1.00         2\n",
      "        3.0       1.00      0.86      0.92         7\n",
      "\n",
      "avg / total       0.95      0.90      0.91        10\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "pipeline2 = Pipeline([\n",
    "('clf', RandomForestClassifier(criterion='entropy'))\n",
    "])\n",
    "\n",
    "parameters = {\n",
    "    'clf__n_estimators': (5, 25, 50, 100),\n",
    "    'clf__max_depth': (5, 25 , 50),\n",
    "    'clf__min_samples_split': (1, 5, 10),\n",
    "    'clf__min_samples_leaf': (1, 2, 3)\n",
    "}\n",
    "\n",
    "grid_search = GridSearchCV(pipeline2, parameters, n_jobs=-1, verbose=1, scoring='accuracy', cv=3)\n",
    "\n",
    "grid_search.fit(X_train, y_train)\n",
    "\n",
    "print 'Best score: %0.3f' % grid_search.best_score_\n",
    "\n",
    "print 'Best parameters set:'\n",
    "best_parameters = grid_search.best_estimator_.get_params()\n",
    "\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print '\\t%s: %r' % (param_name, best_parameters[param_name])\n",
    "\n",
    "predictions = grid_search.predict(X_test)\n",
    "print 'Accuracy:', accuracy_score(y_test, predictions)\n",
    "print classification_report(y_test, predictions)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Naive Bayes accuracy and time elapsed caculation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NaiveBayes\n",
      "Acurracy:  0.7\n",
      "time elapsed:  0.296000003815\n"
     ]
    }
   ],
   "source": [
    "t4=time()\n",
    "print \"NaiveBayes\"\n",
    "nb = BernoulliNB()\n",
    "clf_nb=nb.fit(X_train,y_train)\n",
    "print \"Acurracy: \", clf_nb.score(X_test,y_test)\n",
    "t5=time()\n",
    "print \"time elapsed: \", t5-t4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## cross-validation for NB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cross result========\n",
      "[ 0.55555556  0.625       0.71428571]\n",
      "0.631613756614\n",
      "time elapsed:  0.256999969482\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tt4=time()\n",
    "print \"cross result========\"\n",
    "scores = cross_validation.cross_val_score(nb, X,y, cv=3)\n",
    "print scores\n",
    "print scores.mean()\n",
    "tt5=time()\n",
    "print \"time elapsed: \", tt5-tt4\n",
    "print \"\\n\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# KNN accuracy and time elapsed caculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KNN\n",
      "Acurracy:  0.8\n",
      "time elapsed:  0.0480000972748\n"
     ]
    }
   ],
   "source": [
    "t6=time()\n",
    "print \"KNN\"\n",
    "# knn = KNeighborsClassifier(n_neighbors=3)\n",
    "knn = KNeighborsClassifier(n_neighbors=3)\n",
    "clf_knn=knn.fit(X_train, y_train)\n",
    "print \"Acurracy: \", clf_knn.score(X_test,y_test) \n",
    "t7=time()\n",
    "print \"time elapsed: \", t7-t6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## cross validation for KNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cross result========\n",
      "[ 1.   1.   0.8  0.6  0.5]\n",
      "0.78\n",
      "time elapsed:  0.0869998931885\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Miniconda2\\lib\\site-packages\\sklearn\\cross_validation.py:516: Warning: The least populated class in y has only 4 members, which is too few. The minimum number of labels for any class cannot be less than n_folds=5.\n",
      "  % (min_labels, self.n_folds)), Warning)\n"
     ]
    }
   ],
   "source": [
    "tt6=time()\n",
    "print \"cross result========\"\n",
    "scores = cross_validation.cross_val_score(knn, X,y, cv=5)\n",
    "print scores\n",
    "print scores.mean()\n",
    "tt7=time()\n",
    "print \"time elapsed: \", tt7-tt6\n",
    "print \"\\n\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fine tuning the model using GridSearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "from sklearn.cross_validation import cross_val_score\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn import grid_search\n",
    "\n",
    "knn = KNeighborsClassifier()\n",
    "\n",
    "parameters = {'n_neighbors':[1,]}\n",
    "\n",
    "grid = grid_search.GridSearchCV(knn, parameters, n_jobs=-1, verbose=1, scoring='accuracy')\n",
    "\n",
    "\n",
    "grid.fit(X_train, y_train)\n",
    "\n",
    "print 'Best score: %0.3f' % grid.best_score_\n",
    "\n",
    "print 'Best parameters set:'\n",
    "best_parameters = grid.best_estimator_.get_params()\n",
    "\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print '\\t%s: %r' % (param_name, best_parameters[param_name])\n",
    "    \n",
    "predictions = grid.predict(X_test)\n",
    "print classification_report(y_test, predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SVM accuracy and time elapsed caculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SVM\n",
      "Acurracy:  0.7\n",
      "time elapsed:  0.0380001068115\n"
     ]
    }
   ],
   "source": [
    "t7=time()\n",
    "print \"SVM\"\n",
    "\n",
    "svc = SVC()\n",
    "clf_svc=svc.fit(X_train, y_train)\n",
    "print \"Acurracy: \", clf_svc.score(X_test,y_test) \n",
    "t8=time()\n",
    "print \"time elapsed: \", t8-t7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## cross validation for SVM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cross result========\n",
      "[ 0.6   0.6   0.6   0.6   0.75]\n",
      "0.63\n",
      "time elapsed:  54.5379998684\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tt7=time()\n",
    "print \"cross result========\"\n",
    "scores = cross_validation.cross_val_score(svc,X,y, cv=5)\n",
    "print scores\n",
    "print scores.mean()\n",
    "tt8=time()\n",
    "print \"time elapsed: \", tt7-tt6\n",
    "print \"\\n\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 4 candidates, totalling 12 fits\n",
      "Best score: 0.571"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  12 out of  12 | elapsed:   17.5s finished\n",
      "C:\\Miniconda2\\lib\\site-packages\\sklearn\\metrics\\classification.py:1074: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Best parameters set:\n",
      "\tC: 1\n",
      "\tkernel: 'linear'\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        1.0       1.00      1.00      1.00         1\n",
      "        2.0       0.00      0.00      0.00         2\n",
      "        3.0       0.78      1.00      0.88         7\n",
      "\n",
      "avg / total       0.64      0.80      0.71        10\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.svm import SVC\n",
    "from sklearn.cross_validation import cross_val_score\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn import grid_search\n",
    "\n",
    "svc = SVC()\n",
    "\n",
    "parameters = {'kernel':('linear', 'rbf'), 'C':[1, 10]}\n",
    "\n",
    "grid = grid_search.GridSearchCV(svc, parameters, n_jobs=-1, verbose=1, scoring='accuracy')\n",
    "\n",
    "\n",
    "grid.fit(X_train, y_train)\n",
    "\n",
    "print 'Best score: %0.3f' % grid.best_score_\n",
    "\n",
    "print 'Best parameters set:'\n",
    "best_parameters = grid.best_estimator_.get_params()\n",
    "\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print '\\t%s: %r' % (param_name, best_parameters[param_name])\n",
    "    \n",
    "predictions = grid.predict(X_test)\n",
    "print classification_report(y_test, predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  34 tasks      | elapsed:   17.9s\n",
      "[Parallel(n_jobs=-1)]: Done  90 out of  90 | elapsed:   18.4s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 30 candidates, totalling 90 fits\n",
      "Best score: 0.571\n",
      "Best parameters set:\n",
      "\tclf__C: 0.1\n",
      "\tclf__gamma: 0.01\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        1.0       0.00      0.00      0.00         1\n",
      "        2.0       0.00      0.00      0.00         2\n",
      "        3.0       0.70      1.00      0.82         7\n",
      "\n",
      "avg / total       0.49      0.70      0.58        10\n",
      "\n"
     ]
    }
   ],
   "source": [
    "pipeline = Pipeline([\n",
    "    ('clf', SVC(kernel='linear', gamma=0.01, C=1))\n",
    "])\n",
    "\n",
    "parameters = {\n",
    "    'clf__gamma': (0.01, 0.03, 0.1, 0.3, 1),\n",
    "    'clf__C': (0.1, 0.3, 1, 3, 10, 30),\n",
    "}\n",
    "\n",
    "grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=1, scoring='accuracy')\n",
    "\n",
    "grid_search.fit(X_train, y_train)\n",
    "\n",
    "print 'Best score: %0.3f' % grid_search.best_score_\n",
    "\n",
    "print 'Best parameters set:'\n",
    "best_parameters = grid_search.best_estimator_.get_params()\n",
    "\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print '\\t%s: %r' % (param_name, best_parameters[param_name])\n",
    "    \n",
    "predictions = grid_search.predict(X_test)\n",
    "print classification_report(y_test, predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Unsupervised Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "features = ['Age', 'Specs', 'Astigmatic', 'Tear-Production-Rate']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df1 = df[features]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Specs</th>\n",
       "      <th>Astigmatic</th>\n",
       "      <th>Tear-Production-Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Age  Specs  Astigmatic  Tear-Production-Rate\n",
       "0  1.0    1.0         1.0                   1.0\n",
       "1  1.0    1.0         1.0                   2.0\n",
       "2  1.0    1.0         2.0                   1.0\n",
       "3  1.0    1.0         2.0                   2.0\n",
       "4  1.0    2.0         1.0                   1.0"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.          0.          0.          0.        ]\n",
      " [ 0.         -0.57735027 -0.57735027 -0.57735027]\n",
      " [-0.         -0.25819889  0.79991984 -0.54172095]\n",
      " [-0.         -0.77459667  0.16369154  0.61090513]]\n",
      "[ 0.47058824  0.17647059  0.17647059  0.17647059]\n"
     ]
    }
   ],
   "source": [
    "# Apply PCA with the same number of dimensions as variables in the dataset\n",
    "from sklearn.decomposition import PCA\n",
    "pca = PCA(n_components=4) # 6 components for 6 variables\n",
    "pca.fit(df1)\n",
    "\n",
    "# Print the components and the amount of variance in the data contained in each dimension\n",
    "print(pca.components_)\n",
    "print(pca.explained_variance_ratio_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEZCAYAAACAZ8KHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xu8VFX9//HXGxQVkPCWASaal1S8l6SpeZKfiVpeyvKS\n5q3EEjXN0jK/oFhJlnkrSzO7mailppUlZse0NBFvpKAkiDfUFBUFTYTP74+1Dw7HOefMDGefPTPn\n/Xw85jEz+/pZs2f2Z/Zaa++tiMDMzKxSfYoOwMzMGosTh5mZVcWJw8zMquLEYWZmVXHiMDOzqjhx\nmJlZVZw4OiDpcklnVjjtnyQdmkMMwyUtkZT7dpK0k6Tpea+np+W1baqM4XJJ8yTd1cPrLaTsks6S\n9F9Jz/T0uq1nNHzikPS4pIWS5kt6NXu+oCdjiIg9I+JXeS0+p+Uuu5KIOyJi055YV14kjZP0y9Jh\nOW+bSmLaCRgFDI2I7XNcT12UXdJ7gZOATSJiaJnxu0hanP1OX5E0XdLhJeNXlDRe0qPZ73mWpJ9K\nWrfdcn4uaZGktXMvVIOq5s9vtRo+cZB2rHtFxKCIWDV7Pr7ooBqJpL5Fx9CVRoixA+sBj0fEG0UH\n0kOGAy9ExIudTPN09jt9F3AqcKmkTbJxvwM+DhwIvAvYCriHlHwBkNQf+CTwMnBI9xfBuhQRDf0A\nZgO7djDuR8BvS95PBCZnr3cBngS+DvwXmAUcXDLt5cCZ2evBwI3A88CL2ethJdP+DTgye30YcDtw\nDjAPeAwYXTLtIOCnwDPZ+icAysb1Ab6XxfMf4EvAYqBPmbJ9Dbim3bDzgfOy14cDDwPzs2UdXTJd\nW9m/BswFftE2rGSaU7L55gP/BvYtGddVGVcDfgY8nX1e15aM+zhwH/AScAewRSfbdkn2GTwKPJYN\nOw94AngFmALslA3fHfhf9ngVuK/MthHwTeBx4Fng58CqHay7q21+eFbu+dnzQWWWcSTwOrAom25c\n22dXppzvK/neXQT8IZvnTmD9kmlHADdnMc0l7XhrLfugbNzwLIbPAXOyMn+jk+0yCPhlNt1s4LRs\n+ChgIfBWFvvPysy7C/BEu2HPkxLBKGAB6eiss998W5zHAdO6mHZl4PtZuV8C/g6slI3bm/Tdngfc\nSjpKKt2vnAw8kH2mlwLvBv6Ule1m4F3tPr8vkL7zTwNfKVlWP9L39mngKeAHwIrtfosnAc9l0xze\nbt7vZeWdS9qnrdTVvFksbwJvZPH+vuR3/VQ2bDrw0Zr2u7XMVE8POk8cqwAzsi/aztkXdEjJh76I\ntPNbEfgI8BqwUckPuC1xrA7sB6wEDACuAq4rWU/7xPE/0k5DwDGkf1ht016XbfyVgTWBu4AvZOOO\nIe3sh5J2XLfSceJYN4t3QPa+DykZbZe93wNYL3u9M+kHuXW7sn87K/tKtPtBA58C1s5efzpb19oV\nlvGPwJWkHUxfYOds+DbZF/yD2XyHZttvxQ623xLgL6R/nm0/loOzz6YPcCLpx9QvGzcO+GW7ZZRu\nmyNJSWg40J/07/aXHay7w22ezfsKsGH2fm1g0w6Wcxjw947eZ8MWs2zi+C/wgayMvwZ+k40bmG3j\nL5N2KANKtnfNZeftHd9PsuVuSdrhvL+DMv2S9D3un837CHBEyXfriXLztR+ffQf2y75LGwHfAf5W\nwW/+lmzad5O+x9t0Mu0PSb+j92Tr2570nd+Y9J3elfQd/SowE1ihZL/yT9JvdAjpe3tP9tn0A/4K\nnN7u87uC9LvenLSv2TUbf2a2rDWyxz+AM9r9FsdlcexB+q22JaUfANeTfgMDgN8D36pw3qX7sOz9\nxqQ/XW2/43Up+VNS1X63lpnq6ZFt4Pmkfw0vZc9HlYzfjvTvbDbwmXZf4DeBlUuGXcXb/56W+dDb\nrXNr4MUOfqCHAY+WjFsl+1K9O3u8QbYTzMYfCPw1e/1Xlj0y2I0OEkc2/u/AISXTzuzkc7oOOK6k\n7G9QssOm6x/8fcAnKijje0j/OAeVWcaP2n4wJcNmkCWWMtMvAXbpYvvPIztqoeud5y3AMSXjNs6+\nA2U/3462OWmHOY+001u5i/kqSRztjzguKRm3B/Bw9vogYGoH66m57KQd32KyP1XZ+H9R8nspGd6H\ntKN/f8mwo4FbK/we7ZKtax7wAnAv8Ols3CVkSbKT+dfN5m/b5jcBP+hgWpGOgDYvM+6bwKR20z4F\nfCR7P5uSo0jgt8APS96PJTuS5u3EsVHJ+InApdnr/wC7l4z7GDCr5PNYUPodJCWpkdnr11j2iHOH\nKuZtnzg2IB1tjiJLkLU+mqGNA2CfiFg9IlbLni9rGxERU0jVUAKuaTffS7Fs3fMc0r/9ZUhaRdJP\nsob4l4HbgMGS1EE8z5as//Xs5UDSF2xFYG7Wy+Yl4MfAWtk0Q0mHnqXxdOZK0s6E7Pk3JTHvIelO\nSS9m69mD9O+pzX8jYlFHC5b0OUn3SXopm39Eu/k7KuN7gXkRMb/MYocDX8nK3lb+dSjzmZd4ql1c\nJ0t6uCSuQe3i6sxQlv1M5wArkI4YltHZNo+IhcABwBdJ2/JGSe+vMIZKPFvyeiHpc4X0WT1W4zIr\nKftzHay31JrZfE+0W9awKmJ5OvudrhkR20ZE2+/yRdK/+84cSkqk07L3VwIHd9AGtibpiHFWmXHL\nfB6R9qxPtitH6efxepn3pZ9PsOx3tXRfMpR3fl6l3/kXI2JJyfuFwEBJa5H+pExt+82QEuUaXc1L\nGRHxGOlodTzwnKTfSOrq8y6rWRJHRztwJB1LOrR8hlS/V2o1SauUvF83m669k0mH0ttFxGBStVan\n6+3Ak6R/+muUJLrBEbFlNn4uacfbZngXy7sGaJE0jPTv9zcAkvqR/iF9F1grIlYjfeFK442OFpr1\nYLkE+FIW42rAQ1RW3ieB1SUN6mDct7Kyt5V/YERc1cnylsaZ9VD6KrB/SVzzS+LqsEyZZ1j2Mx1O\nOtR/rsy0X6GTbR4RkyPiY6QjrEdIdeCVWEDaGbSV6T0Vzgfp89ugg3HdWfbOvJDN135ZT1e5nHJu\nAUZK6uyPxKHA+yTNlTSX1H6xJrBnB7G+QfnPrP3nAem391SZaSshlv3tlu5Lyn32lXRVfoGUCEaU\n/GYGR+pUUIl3fCciYlJE7FwSz9kVLmsZzZI4ypK0Manx+bOkdo6vSdqydBLgjKwL4M7AXsDVZRY1\nkPQPY76k1UkZu2oR8SypUe0HklZV8j5JbTulq4HjJQ2TtBrvTHTtl/cC6Z/w5aTD10eyUf2yxwsR\nsUTSHqTD40oNIB16vyCpj6QjSPW2lZbxJuBHkgZLWiH7bCHtXI+RNBJA0gBJe0oaUGFcq5J2Wi9K\n6ifp/7JhbZ4D1uvkSPBK4ERJ60kaCHyLVF2xpMy0q9LBNpf0bkl7Z717FpGqExZXWIYHgBGStpS0\nEqmKqaudfps/AO+RdHxW/oFtnyXLX/aK/gRl018NfCtb/3BSW9Nyd/uNiL8Ck4HrJG0rqW+2jjGS\nDpe0A/A+UvXzVtljRFa2w8osL0idNM6VNCT7Lm8vacWsDHtJ+mj2HT2ZlGTuXI4inJ4dqY4AjgAm\nZcOvBL4paU1JawKnU8HnlcV/KXBedvRBtm+o9Lf8HOnzIpt346y8/UjVlK+TfudVa5bEcaNSv/C2\nx++yQ9dfAd+JiH9HxH+AbwC/yr44kP7hv0TK/r8CxkTEzDLLP4/0L/EFUiPXn9qN7+qHXzr+c6Sd\n+sOket5rSP9aIX1J/kLaudxDasDsym9IdZZXLF1ZxGvA8cA12eHtgaRGtYpExHTSP7m7SNUmI0g9\noDqdreT1oaR2jhmkL+8J2XKnknp7XJTF9ShlfvAdLBPSZ/OXbL7ZpH9jpVV715B2gC9KuqfMMn5G\n2s5/J1X5LCR9TuV0ts37kHqyPJ2N/wip2qpL2ffrTFJ71qOk3mkVybbrbqTeQM9m87dko5e37O0/\n686+08dn88/KlvfriLi80nJ0YX/SZ30VqbvtNFJHgVtIv53rI+LhiHi+7UHqTbiXpMFllndytowp\npKqws0ltAo+SuvJeROqMsBepDe+tbL5qPo82t5HaMyYD380SIcBZpN/zg7z92/5WJ8spXdep2TLv\nyqpMbya1T1Uy72WkPynzJF1L2u+cTSrvM6Qq8q9XUK53aOsGmhtJo0k/wj7AZRExsd34XUg7tbZ6\nyGsj4qxK5l3OuHYBfhUR63Y5sZlZB7KjrlmkziY1/YNvNCvkuXClS2VcRPpH/AwwRdLvI2JGu0n/\nHhF71zivmVnRqm3vbGh5V1WNJHURnZP14JkE7FNmunIfeqXzmpkVLd+qmzqTd+IYxrJ10E9Rvtve\nDpLul/RHSZtVOW9NIuI2V1OZ2fLK/tz27S3VVJBzVVWFpgLrRsTCrPfP9XTe+GNmZgXKO3E8TerP\n3GYd2vX3znqKtL2+SdKPsu6PXc7bRlKvOkw0M+sOEVFT20zeVVVTgA2V7ivRj9Qt9IbSCVRyWeSs\nT7oiYl4l85ZantPn6/kxbty4wmNw+Vw+l6/5Hssj1yOOiFgsaSyp73Fbl9rpksak0XEJsL+kL5JO\npHqddCmHDufNM14zM+ta7m0cEfFn4P3thv2k5PUPSVewrGheMzMrVrOcOd60Wlpaig4hVy5fY3P5\neqfczxzvCZKiGcphZtZTJBF12jhuZmZNxonDzMyq4sRhZmZVceIwM7OqOHGYmVlVnDjMzKwqThxm\nZlYVJw4zM6uKE4eZmVWlaRLHIYecwezZc4oOw8ys6TXNJUfgNTbYYByTJx/H+usPLzokM7O65kuO\nADCAxx47g9NP/3nRgZiZNbUmShwAA3jmmV5z218zs0I0WeJYwNChTVYkM7M600R72QVssME4Jkw4\nvOhAzMyaWtM0jq+00nhuv/1wttvODeNmZl1ZnsbxpkkcJ58cvPACXH550dGYmdU/Jw4p5s8PNtkE\nfvtb2GGHoiMyM6tv7o4LrLoqnHMOjB0LixcXHY2ZWfNqmsQBcNBBMGAAXHpp0ZGYmTWvpqmqaivH\ngw/CbrvBww/DGmsUHJiZWZ2q66oqSaMlzZD0qKRTOpluO0mLJH2yZNjjkh6QdJ+kuytZ35ZbwoEH\nwmmndUf0ZmbWXq5HHJL6AI8Co4BngCnAgRExo8x0k4HXgZ9FxLXZ8FnAByLipS7WE6XlePll2HRT\nuPFG+OAHu7NEZmbNoZ6POEYCMyNiTkQsAiYB+5SZ7jjgt8Dz7YaLGmIcPBi+/e3UUL7EVyAxM+tW\neSeOYcCTJe+fyoYtJWkosG9EXExKFKUCmCxpiqQvVLPiww4DCX7xixqiNjOzDq1QdADAeUBp20dp\n8tgxIuZKWouUQKZHxB3lFjJ+/Pilr1taWmhpaeGii2CvvWDffWG11fII3cysMbS2ttLa2toty8q7\njWN7YHxEjM7enwpEREwsmWZW20tgTWABcHRE3NBuWeOAVyPi3DLriY7Kccwx0K8fXHBBd5TIzKw5\n1O2Z45L6Ao+QGsfnAncDB0XE9A6mvxy4MSKuldQf6BMRr0kaANwMnBERN5eZr8PE8eKLsNlmMHly\n6nFlZmZ13DgeEYuBsaSd/kPApIiYLmmMpKPLzVLyem3gDkn3AXeREso7kkZX1lgDzjwTjj0WmuCU\nFTOzwjXdCYDlLF4MI0fCSSfBZz/bg4GZmdWpuq2q6ildJQ6AO++E/feH6dNh0KAeCszMrE45cVSQ\nOACOPBJWXx2+970eCMrMrI45cVSYOJ57DjbfHG67LTWYm5n1VnXbOF5v1l4bTj8djj/eDeVmZrXq\nVYkD4EtfguefTzd8MjOz6vWqqqo2f/87HHJIaigfMCDHwMzM6pSrqqr0kY/AzjunCyGamVl1euUR\nB8Azz6Qzye+8EzbaKKfAzMzqlI84ajB0KJx6qhvKzcyq1WsTB6Sk8fjj6YZPZmZWmV5bVdXmllvg\n6KPhoYdglVW6OTAzszrlqqrl8P/+H3zgA/Dd7xYdiZlZY+j1RxwATzwB224LU6bA+ut3Y2BmZnXK\nRxzLad1105VzTzyx6EjMzOqfE0fmK19J7Rw33VR0JGZm9c2JI7PSSnD++XDCCfC//xUdjZlZ/XLi\nKLHnnrDppnDuO+5qbmZmbbpsHJe0DnAhsBPp1q63AydExFP5h1eZ5W0cLzVrFmy3Hdx/P7z3vd2y\nSDOzupN34/jlwA3AEGAocGM2rCm9730wdiycfHLRkZiZ1adKjjjuj4ituxpWpO484gB4/fV0o6ef\n/hRGjeq2xZqZ1Y28jzhelHSIpL7Z4xDgxVpW1ihWWQV+8AM47jhYtKjoaMzM6kslieNI4DPAs8Bc\nYH/giDyDqgf77JPO77jwwqIjMTOrLz5zvBOPPgof/jBMmwZDhnT74s3MCpNLVZWkr2XPF0q6oP2j\niuBGS5oh6VFJp3Qy3XaSFkn6ZLXz5mXjjeELX4Cvfa2n12xmVr9W6GTc9Oz5nloXLqkPcBEwCngG\nmCLp9xExo8x0ZwN/qXbevJ12Wjq34/bb010Dzcx6uw4TR0S03aViYURcUzpO0qcrXP5IYGZEzMnm\nmwTsA7Tf+R8H/BbYroZ5czVwIHzve6mL7tSpsEJnqdbMrBeopHH86xUOK2cY8GTJ+6eyYUtJGgrs\nGxEXA6pm3p7ymc/AGmvAj39cxNrNzOpLh/+fJe0B7AkMa9emMQh4qxtjOA9Y7vaL8ePHL33d0tJC\nS0vL8i5yKSn1rmppSUnk3e/utkWbmfWI1tZWWltbu2VZHfaqkrQVsDVwJvB/JaNeBf4WES91uXBp\ne2B8RIzO3p8KRERMLJlmVttLYE1gAXA08HxX85YsI5deVe2ddBLMn59ODDQza2TL06uqkjPHV4yI\nmk6Dk9QXeITUwD0XuBs4KCKmdzD95cCNEXFtNfP2VOKYPx822QSuuw4+9KHcV2dmlpu8zxxfT9Jv\nJT0saVbbo5KFR8RiYCxwM/AQMCkipksaI+nocrN0NW8l683LoEEwcSIceywsXlxkJGZmxankiOMO\nYBzwA+ATpLPG+0TE/3U6Yw/qqSMOgIjULfeww9I5HmZmjSjvqqqpEfEBSdMiYovSYbWsMA89mTgg\nXXJ9991h+nRYffUeW62ZWbfJu6rqf9nJeDMljZW0HzCwlpU1i623hk9/Gr75zaIjMTPreZUccWxH\nOot8MDCB1B33nIi4K//wKtPTRxwAL72Uzij/059g2217dNVmZsstt6qqrGfTxIio69saFZE4AC67\nLD3uuAP6+Ca8ZtZAcquqyno27VRTVL3AEUfAW2/Br35VdCRmZj2nkqqqi0mX+riGdHIeABFxbb6h\nVa6oIw6AKVNg771hxgx417sKCcHMrGp596oqd3/xiIgja1lhHopMHABHHw39+8N55xUWgplZVXJN\nHI2g6MTxwgvpHuV//StssUVhYZiZVSzv7rjWhTXXhPHj0z3KmyAPm5l1yomjm4wZA6+8ApMmFR2J\nmVm+XFXVjf7xDzjggHRG+aqrFh2NmVnHcq2qkrS2pMsk3ZS930zSUbWsrNntuCOMGgVnnVV0JGZm\n+amkV9VNwOXAaRGxlaQVgPvarltVD+rliAPg2WdTA/ntt6dLsJuZ1aO8G8fXjIirgSUAEfEW4IuK\nd+A974HTTnNDuZk1r0oSxwJJa5DdKyO7q98ruUbV4I49FubOTTd8MjNrNpVUVW0LXAhsDvwbWAvY\nPyIezD+8ytRTVVWb1tZ0z47p09PJgWZm9ST3EwCzdo33k+4L/kitt5LNSz0mDoCDDoINN4QJE4qO\nxMxsWXlfcuRY4IqIeDl7vxrp3t8/qmWFeajXxPHUU+neHXfdlRKImVm9yDtx3B8RW7cbdl9EbFPL\nCvNQr4kD0j3Kb78d/vCHoiMxM3tb3r2q+kpauvDsHh39allZb3TiiTBzphOHmTWPShLHn4GrJI2S\nNAq4MhtmFejXDy68EE44Ad54o+hozMyWXyVVVX2AMcCobNBk4KfZTZ7qQj1XVbX55Cdhm23g9NOL\njsTMzJdVb4jEMWdOujf51Kmw3npFR2NmvV3e16raUdJkSY9KmiVptqRZVQQ3WtKMbP5TyozfW9ID\nku6TdLekHUvGPV46rvJi1Z/hw+HLX4avfKXoSMzMlk8lVVUzgBOBqZRcaiQiXuxy4ama61FSNdcz\nwBTgwIiYUTJN/4hYmL3eArg6IjbN3s8CPhARL3Wxnro/4oDUxjFiBFx8MXzsY0VHY2a9Wd69ql6J\niJsi4vmIeLHtUeHyRwIzI2JOdtLgJGCf0gnakkZmINk1sTKqMMaGsPLKcP756TpWb75ZdDRmZrWp\nZKf8N0nnSNpB0rZtjwqXPwx4suT9U9mwZUjaV9J04Eag9F7mAUyWNEXSFypcZ137+Mdho418f3Iz\na1wrVDDNh7LnD5YMC2DX7goiIq4Hrpe0E3AWsFs2aseImCtpLVICmR4Rd5Rbxvjx45e+bmlpoaWl\npbvC63bnnQfbbw8HHwzrrFN0NGbWG7S2ttLa2toty8q1V1V2Jd3xETE6e38qEBExsZN5HgO2i4h5\n7YaPA16NiHPLzNMQbRylTj8d/vMfuPLKoiMxs95oedo4KjniQNJewAhg5bZhEXFmBbNOATaUNByY\nCxwIHNRu2RtExGPZ622BfhExT1J/oE9EvCZpAPAx4IxK4m0EX/86bLppuopuHR8cmZm9Q5eJQ9KP\ngf7AR4GfAvsDFXWNjYjFksYCN5PaUy6LiOmSxqTRcQnwKUmfA94EXgc+k82+NnCdpMjivCIibq6q\ndHWsf38499zUUH7vvbDiikVHZGZWmUq64z4YEVuWPA8EboqInXsmxK41YlUVpDsE7r477LlnOsfD\nzKyn5N0d9/XseaGkocAiYEgtK7NlSXDBBXDWWele5WZmjaCSxPEHSYOBc4B7gcdJFzq0brDJJnDk\nkXDqqUVHYmZWmap6VUlaCVg5IurqnuONWlXV5tVXU0P51VfDhz9cdDRm1hvkcpFDSbtGxK2SPllu\nfERcW8sK89DoiQNSt9zvfhfuuQf69i06GjNrdnm1ceySPX+izOPjtazMOnbggTBoEFxySdGRmJl1\nrtOqquwihftHxNU9F1L1muGIA2DaNBg1Ch5+GNZcs+hozKyZ5X3P8Xsi4oOdTlSwZkkckLrlLlzo\nIw8zy1feieNs4AXgKmBB2/D2lwQpUjMljpdfTg3lN9wA221XdDRm1qzyThyzywyOiHhfLSvMQzMl\nDoCf/zzds+POO6FP01xU3szqiW8d22SJY8kS2GknOOqo9DAz6265Jw5JmwObsexFDn9Zywrz0GyJ\nA9L1q/bcE6ZPh9VWKzoaM2s2eVdVjQNaSInjT8AewB0RsX8tK8xDMyYOgC9+MZ3TcdFFRUdiZs0m\n78QxDdgKuC8itpK0NvDriNit0xl7ULMmjnnzUkP5X/4CW29ddDRm1kxyv8hhRCwB3pI0CHgeeG8t\nK7PqrL46TJgAY8emK+mamdWDShLHPdlFDi8FppIudHhnrlHZUkcdBW+8Ab/+ddGRmJkl1V7kcD1g\nUEQ8mFdAtWjWqqo2//oX7LcfzJiRLktiZra88m7juAGYBPw+IhZ0OnFBmj1xQDryGDwYvv/9oiMx\ns2aQd+LYBTgA2It0D/FJwB8i4o1aVpiH3pA4nn8eRoxI9ygfMaLoaMys0fXICYCS+gK7Al8ARkdE\n3VSa9IbEAalb7rXXwl//mu4eaGZWq7x7VSFpFeBTwDHAdsAvalmZLZ9jjoEXX4Rrrik6EjPrzSqp\nqroaGAn8mXShw9uy7rl1o7cccQDcfjscfHA6o3zgwKKjMbNGlXcbx+7ALRGxuJYV9ITelDgADj0U\n1lkHvvOdoiMxs0blixz2ssQxdy5ssQX885+w8cZFR2NmjSj3No7lIWm0pBmSHpV0Spnxe0t6QNJ9\nku6WtGOl8/ZWQ4bA178Oxx/vM8rNrOflesSR3Xr2UWAU8AypO++BETGjZJr+EbEwe70FcHVEbFrJ\nvCXL6FVHHACLFsFWW8G3vw377lt0NGbWaJbniGOFTha6bWczRsS9FSx/JDAzIuZky5wE7AMs3fm3\nJY3MQGBJpfP2ZiuuCBdeCJ//POy+O6yyStERmVlv0WHiANrOUV4Z+CDwACBgS+AeYIcKlj8MeLLk\n/VOkhLAMSfsC3wHWIp1oWPG8vdmoUen2smefDWecUXQ0ZtZbdJg4IuKjAJKuBbaNiGnZ+82B8d0Z\nRERcD1wvaSfgLKDqS7aPH/92SC0tLbS0tHRXeHXt+99Pl1w/7DB4X93czNfM6k1rayutra3dsqxK\nuuM+FBEjuhrWwbzbA+MjYnT2/lTS/condjLPY6STDDeudN7e2MZR6jvfgbvugt//vuhIzKxR5N2r\n6kFJP5XUkj0uBSq9Ou4UYENJwyX1Aw4EbiidQNIGJa+3BfpFxLxK5rXkpJPg4YfhT38qOhIz6w06\na+NocwTwReCE7P3fgYsrWXhELJY0FriZlKQui4jpksak0XEJ8ClJnwPeBF4HPtPZvJUXrfdYaSW4\n4AI47jjYdVdYeeWu5zEzq1VF3XGza1WtGxGP5B9S9Xp7VVWbffeFkSPhG98oOhIzq3d5X3Jkb+Ac\nUhXS+pK2Bs6MiL1rWWEenDiS2bPhgx+E++6DddctOhozq2d5t3GMI3WDfRkgIu4H1q9lZZav9ddP\n1VVf+UrRkZhZM6skcSyKiFfaDfPf+zp1yilwzz1wyy1FR2JmzaqSxPGQpIOBvpI2knQh8M+c47Ia\nrbIKnHdeOvJ4882iozGzZlRJ4jgOGAH8D7gSmA98Oc+gbPnsvTest17qaWVm1t18WfUmNXMm7LAD\nPPggDB1adDRmVm/y7lW1MXAysB4l531ExK61rDAPThzlfeMb8MQT8OtfFx2JmdWbvBPHA8CPganA\n0rsARsTUWlaYByeO8hYsgE03TYnjIx8pOhozqyd5J46pEfGBmiLrIU4cHbvmGpgwAe69F1ao5DoB\nZtYr5H0ex42SviRpiKTV2x61rMx63v77w1prwcUVXSTGzKxrlRxxzC4zOCKibi7i7SOOzj38MOyy\nC/z737Ajg0yPAAAQPUlEQVT22kVHY2b1INeqqkbgxNG1k0+GefPgZz8rOhIzqwe5JA5Ju0bErZI+\nWW58RFxbywrz4MTRtfnzU0P5734H229fdDRmVrRc7jkO7ALcCnyizLgA6iZxWNcGDYKJE+HYY+Hu\nu6Fv36IjMrNG5aqqXiQidcs95BAYM6boaMysSLm3cUjai3TZkaW3CIqIM2tZYR6cOCr3wAPwsY+l\nBvM11ig6GjMrSq7dcSX9GDiAdM0qAZ8GhteyMiveVlvBZz4Dp51WdCRm1qgq6Y77YERsWfI8ELgp\nInbumRC75iOO6rz8MmyyCfzxj/CBuj6108zykvcJgK9nzwslDQUWAUNqWZnVh8GD4dvfhrFjYcmS\noqMxs0ZTSeL4g6TBpNvH3gs8Trq8ujWwww9PSeMXvyg6EjNrNFX1qpK0ErBymTsCFspVVbW55x74\n+Mdhxox0FGJmvUdeJwCWPfGvjU8AbA5jxsDKK8P55xcdiZn1pLwSx+WdzBcRcWQtK8yDE0ftXngB\nNtss3aN8yy2LjsbMekpdX6tK0mjgPFJ7ymURMbHd+IOBU7K3rwJfiogHs3GPA68AS4BFETGyg3U4\ncSyHiy+GK6+E224D1fQ1MrNGk/d5HGtIukDSvZKmSjpfUkWnjknqA1wE7E46gfAgSZu0m2wW8JGI\n2Ao4C7ikZNwSoCUitukoadjyO/poeO21lDzMzLpSSa+qScB/gU8B+2evr6pw+SOBmRExJyIWZcva\np3SCiLirpLH9LmBYyWhVGKMth7594aKL4KtfTRdDNDPrTCU75SERMSEiZmePs4BK7+owDHiy5P1T\nLJsY2vs8cFPJ+wAmS5oi6QsVrtNq8OEPw267pbsFmpl1ppKbid4s6UDg6uz9/sBfujsQSR8FjgB2\nKhm8Y0TMlbQWKYFMj4g7ys0/fvz4pa9bWlpoaWnp7hCb3sSJsPnmcOSR6RLsZtY8WltbaW1t7ZZl\nVXLJkVeBAcDibFBfYEH2OiJiUCfzbg+Mj4jR2ftTs3naN5BvCfwOGB0Rj3WwrHHAqxFxbplxbhzv\nJuefDzfeCJMnu6HcrJnl2jgeEatGRJ+IWDF79MmGrdpZ0shMATaUNFxSP+BA4IZ2wa9LShqHliYN\nSf2z62IhaQDwMeDf1RXPqnXssfDcc+mGT2Zm5VTSq+qodu/7Zv/+uxQRi4GxwM3AQ8CkiJguaYyk\no7PJTgdWB34k6T5Jd2fD1wbukHQfqdH8xoi4uaJSWc1WWCE1lJ90EixY0PX0Ztb7VFJV9RtgMHAU\nsAZwOXBbRJycf3iVcVVV9zv4YFh/ffjWt4qOxMzy0BM3cjoA+CGpbePgiPhHLSvLixNH93v66XTv\njjvvhI02KjoaM+tueZ8AuBFwAqkdYg5wqKT+tazMGsewYXDKKXDCCemWs2ZmbSo5j+NG4PSIGAPs\nAswkNXpbkzvhBJg1K/WyMjNrU0kbx6CImN9u2MYR8WiukVXBVVX5mTw5XUH3oYdglVWKjsbMuksu\nVVWSvgYQEfMlfbrd6MNrWZk1nt12g223hXPOKToSM6sXnV1W/d6I2Lb963Lvi+YjjnzNmZOSx9Sp\nsN56RUdjZt0hr8ZxdfC63HtrYsOHw4knpoeZWWeJIzp4Xe69NbmTT4Zp0+DPfy46EjMrWmdVVYtJ\n520IWAVY2DaKdN/xFXskwgq4qqpn/PGP6ahj2jRYaaWiozGz5VHXdwDsCU4cPecTn4Add4RTTy06\nEjNbHk4cThw95rHH4EMfgvvvh3XWKToaM6tVrmeOm5XaYAP40pdSm4eZ9U4+4rCqLVwIm20Gl18O\nH/1o0dGYWS18xGE9qn9/OPdcGDsWFi0qOhoz62lOHFaT/fZLbRwXXVR0JGbW01xVZTV75JHUw2ra\nNBgypOhozKwa7lXlxFGYU06BuXPhl78sOhIzq4YThxNHYV57DTbZBCZNgp12KjoaM6uUG8etMAMH\nwve+lxrK33qr6GjMrCc4cdhyO+AAGDwYfvKToiMxs57gqirrFv/+dzqn4+GHYa21io7GzLriNg4n\njrpw4onw6qvw058WHYmZdcWJw4mjLrzyCmy6KVx/PYwcWXQ0ZtaZum4clzRa0gxJj0o6pcz4gyU9\nkD3ukLRlpfNafXnXu+Dss+HYY2HJkqKjMbO85Jo4JPUBLgJ2B0YAB0napN1ks4CPRMRWwFnAJVXM\na3XmkEOgXz+47LKiIzGzvOR9xDESmBkRcyJiETAJ2Kd0goi4KyJeyd7eBQyrdF6rP336pMuQfPOb\nMG9e0dGYWR7yThzDgCdL3j/F24mhnM8DN9U4r9WJbbaB/feH008vOhIzy8MKRQfQRtJHgSOAms4/\nHj9+/NLXLS0ttLS0dEtcVpsJE1JD+ec/nxKJmRWrtbWV1tbWbllWrr2qJG0PjI+I0dn7U4GIiInt\nptsS+B0wOiIeq2bebJx7VdWhSy9N9+y4445UhWVm9aOee1VNATaUNFxSP+BA4IbSCSStS0oah7Yl\njUrntfp21FHpMiS//nXRkZhZd8r9PA5Jo4HzSUnqsog4W9IY0tHDJZIuBT4JzAEELIqIkR3N28E6\nfMRRp+6+G/bZB2bMSN11zaw++ARAJ4669vnPw6qrwg9+UHQkZtbGicOJo679978wYgTceitsvnnR\n0ZgZ1HcbhxlrrQXjxqVLrzu/mzU+Jw7rEWPGwMsvw1VXFR2JmS0vV1VZj7njDjjwwNRQPnBg0dGY\n9W5u43DiaBif+xwMGQIT33E2jpn1JCcOJ46GMXcubLFFOvrYxJesNCuMG8etYQwZAt/4Bhx/vBvK\nzRqVjzisxy1aBJttNochQ35O375LGDasDxMmHM766w8vOjSr0OzZczj99J/z9NPefo2mbdtdccV4\nV1U1Qzl6i9mz57DTThfyzDNnAAOABWywwTgmTz7OO58GMHv2HHbb7UIee8zbr9Esu+0GOnE0Qzl6\ni0MOOYMrrjiZtNNps4CNNvoeu+02rqiwrEKTJ5/BzJnefo1o2W1XextH3VxW3XqPp59ewrI7HbL3\nS9hsswICsqpMnuzt16jKb7vqOXFYjxs2rA+wgPb/WEeO7MOxxxYUlFXszjv7MHOmt18jKr/tqude\nVdbjJkw4nA02GEdKHtBWRz5hwuGFxWSV8/ZrXO/cdrVxG4cVoq1nxzPPLGHoUPfKaTTefo3Lvaoy\nThxmZtXxCYBmZtZjnDjMzKwqThxmZlYVJw4zM6uKE4eZmVXFicPMzKrixGFmZlXJPXFIGi1phqRH\nJZ1SZvz7Jf1T0huSTmo37nFJD0i6T9LdecdqZmZdyzVxSOoDXATsDowADpLU/r5vLwLHAeeUWcQS\noCUitomIkXnGWq9aW1uLDiFXLl9jc/l6p7yPOEYCMyNiTkQsAiYB+5ROEBEvRMRU4K0y86sHYqxr\nzf7Fdfkam8vXO+W9Ux4GPFny/qlsWKUCmCxpiqQvdGtkZmZWk3q/rPqOETFX0lqkBDI9Iu4oOigz\ns94s14scStoeGB8Ro7P3pwIRERPLTDsOeDUizu1gWR2Ol+QrHJqZVale7wA4BdhQ0nBgLnAgcFAn\n0y8thKT+QJ+IeE3SAOBjwBnlZqq18GZmVr1cE0dELJY0FriZ1J5yWURMlzQmjY5LJK0N3AOsCiyR\ndAKwGbAWcF12NLECcEVE3JxnvGZm1rWmuB+HmZn1nIbp6trViYTZNBdIminpfklb93SMy6OCEyV3\nkfSypHuzxzeLiLMWki6T9JykBzuZppG3Xafla+RtByBpHUm3SnpI0jRJx3cwXcNtw0rK1sjbT9JK\nkv6VnUQ9LWsrLjddddsuIur+QUpw/wGGAysC9wObtJtmD+CP2esPAXcVHXc3l28X4IaiY62xfDsB\nWwMPdjC+YbddheVr2G2Xxf8eYOvs9UDgkWb5/VVYtkbffv2z577AXcDI5d12jXLE0eWJhNn7XwJE\nxL+Ad2XtJ42gkvJBSeeBRhKpC/VLnUzSyNuukvJBg247gIh4NiLuz16/BkznnedjNeQ2rLBs0Njb\nb2H2ciVSe3H79omqt12jJI5KTiRsP83TZaapV5WeKLlDdij5R0mb9UxoPaKRt12lmmLbSVqPdHT1\nr3ajGn4bdlI2aODtJ6mPpPuAZ4HJETGl3SRVb7t6PwHQ3jYVWDciFkraA7ge2LjgmKwyTbHtJA0E\nfguckP07bxpdlK2ht19ELAG2kTQIuF7SZhHx8PIss1GOOJ4G1i15v042rP007+1imnrVZfki4rW2\nQ86IuAlYUdLqPRdirhp523WpGbadpBVIO9ZfRcTvy0zSsNuwq7I1w/YDiIj5wN+A0e1GVb3tGiVx\nLD2RUFI/0omEN7Sb5gbgc7D0jPWXI+K5ng2zZl2Wr7TOUdJIUlfqeT0b5nIRHdcTN/K2a9Nh+Zpg\n2wH8DHg4Is7vYHwjb8NOy9bI20/SmpLelb1eBdgNmNFusqq3XUNUVUUFJxJGxJ8k7SnpP8AC4Igi\nY65GJeUD9pf0RWAR8DpwQHERV0fSb4AWYA1JTwDjgH40wbaDrstHA287AEk7Ap8FpmV15QF8g9QL\nsKG3YSVlo7G33xDgF0q3uOgDXJVtq+Xad/oEQDMzq0qjVFWZmVmdcOIwM7OqOHGYmVlVnDjMzKwq\nThxmZlYVJw4zM6uKE4f1OpLWlnRldhnpKZL+IGnDouOqVXbZ7x2KjsN6DycO642uA26NiI0iYjvg\n60DdX8m1Ey3Ah4sOwnoPJw7rVSR9FHgzIi5tGxYR0yLiH5LOyW5284Ckz2TT7yKpVdL1kv4j6TuS\nDs5ujvOApPWz6S6XdHF2BDND0l7Z8JUk/UzSg5KmSmrJhh8m6XeSbpL0iKSJJTHuJumfku6RdJWk\n/tnw2ZLGZ8t5QNLGkoYDxwBfVrrJ0I499Vla79UQlxwx60abk652ugxJnwS2jIgtJL0bmCLptmz0\nlsAmwMvALODSiPiQ0t3ijgNOyqYbHhHbZdVef5O0AXAssCQitpT0fuBmSRtl029Fuoz3IuARSRcA\nbwDfBEZFxOuSvpYt/6xsnucj4gPZJTBOjoijJf0YeDUizu22T8msE04cZslOwJUAEfG8pFZgO+BV\nYEpEPA8g6THSNcUAppGqidpcnc3/n2y6TbPlXpANf0TS47x9Se6/tl3CW9JDpOsjrQZsBvxDkkh3\nhPxnyTquy56nAvt1Q7nNqubEYb3NQ8D+FUxXeqXb/5W8XlLyfgnL/oZKL/ymbHw1y10hG39zRHy2\ng7ja5lmMf79WELdxWK8SEbcC/SR9vm2YpC1I1VAHZHdLWwvYGbi7ysV/WskGwPqk+1ffTrr6KpI2\nJt334JFOlnEXsGO2DCT1L6na6sirwKAqYzWrmROH9Ub7Abtljd3TgG8DVwAPAg8AtwBfbaueaqez\ny0k/QUo2fwTGRMSbwI+AvpIeJFWFHZbdV77sciPiBeBw4EpJD5Cqqd7fxbpvBPZz47j1FF9W3awb\nSLocuDEiri06FrO8+YjDrHv4H5j1Gj7iMDOzqviIw8zMquLEYWZmVXHiMDOzqjhxmJlZVZw4zMys\nKk4cZmZWlf8PdmTrRjaoxFIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x88dc438>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.plot(list(pca.explained_variance_ratio_),'-o')\n",
    "plt.title('Explained variance ratio as function of PCA components')\n",
    "plt.ylabel('Explained variance ratio')\n",
    "plt.xlabel('Component')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: pylab import has clobbered these variables: ['yticks']\n",
      "`%matplotlib` prevents importing * from pylab and numpy\n"
     ]
    }
   ],
   "source": [
    "%pylab inline\n",
    "import IPython\n",
    "import sklearn as sk\n",
    "import numpy as np\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "y = df['Target-Lenses']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "target = array(y.unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 3.,  2.,  1.])"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_pca_scatter():\n",
    "    colors = ['blue', 'red', 'green']\n",
    "    for i in xrange(len(colors)):\n",
    "        px = X_pca[:, 0][y == i+1]\n",
    "        py = X_pca[:, 1][y == i+1]\n",
    "        plt.scatter(px, py, c=colors[i])\n",
    "    plt.legend(target)\n",
    "    plt.xlabel('First Principal Component')\n",
    "    plt.ylabel('Second Principal Component')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Miniconda2\\lib\\site-packages\\ipykernel\\__main__.py:4: FutureWarning: in the future, boolean array-likes will be handled as a boolean array index\n",
      "C:\\Miniconda2\\lib\\site-packages\\ipykernel\\__main__.py:5: FutureWarning: in the future, boolean array-likes will be handled as a boolean array index\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEPCAYAAABoekJnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XucVXW9//HXGxAQERBFIJCLtxRSSlPx0nFKS+rXUTMy\n9WhezzHLtNPvd1JKYux00U6RR/FYHtE0S4+3lErBS0xqZaICIjCKd8TSIygC4w34/P5Ya8bNsGdm\n79l7zd57eD8fj/2Ytb7ru9f6zJ5Z85m1vt/1/SoiMDMz66welQ7AzMxqmxOJmZmVxInEzMxK4kRi\nZmYlcSIxM7OSOJGYmVlJMk0kkmZKekXS4+3UuVTSMkkLJH04y3jMzKz8sr4iuQY4oq2Nkj4N7BIR\nuwFnAj/LOB4zMyuzTBNJRDwIvN5OlaOA69K6fwUGShqaZUxmZlZelW4jGQEsz1lfkZaZmVmNqHQi\nMTOzGterwsdfAeyUsz4yLduMJA8KZmbWCRGhLPffFVckSl/5zAK+BCBpIvBGRLzS1o4ioupf06ZN\nq3gMjtNx1mqMjrP8r66Q6RWJpF8DdcD2kl4EpgG9gYiIKyPiTkmfkfQ0sA44Nct4zMys/DJNJBFx\nQgF1zs4yBjMzy5Yb28usrq6u0iEUxHGWVy3EWQsxguOsReqqe2ilkhS1EquZWbWQRHSDxnYzsy4z\nZswYJG1xrzFjxlTsM/cViZl1K+l/4JUOo8u19X37isTMzKqeE4mZmZXEicTMzEriRGJmZiVxIjEz\n60InnXQSw4cPZ9CgQeyxxx7MnDmzzbo//elPW+qeccYZvPfee10YaeHca8vMupVSem3NmjWLefMe\nYcyY0Zx88sn06lX+wT+WLFnCzjvvTN++fXnqqac49NBDufPOO/nIRz6ySb05c+ZwyimnMHfuXIYP\nH87RRx/NgQceyA9+8IO8+3WvLTOzjK1bt46rrrqK6dOns2jRos22T5kyjRNOOI/vfU+cc871HHHE\nMWzcuLHscYwbN46+ffsCyUC0knjmmWc2q3fddddx+umns8ceezBw4EC+853vcM0115Q9nrKo9MiU\nRYxgGWZmHcn3t2LNmjWx224Tol+/z0bv3l+LrbfeIe68886W7W+++WZstVW/gFcCIuDd6N9/XNx/\n//0tdTZu3BgXXHBhDBw4LAYOHBYXXHBhbNy4sVMxfuUrX4l+/fqFpNh3331j3bp1m9WZMGFC3HTT\nTS3rK1eujB49esSqVasK/r5zyjP9++wrEjPr9q6++mqWL9+VpqZZvPvupbz11q8466xvtmxfu3Yt\nPXv2A4akJVvRo8dOrF69uqXOjBlXMH367axefT+rV9/P9Om3M2PGFZ2K5/LLL2ft2rU8+OCDHHPM\nMfTp02ezOmvXrmXgwIEt6wMGDCAiWLNmTaeOmSUnEjPr9lauXMU77+zJ+1Mj7cEbb6xs2T506FDG\njh1Dr14XAC8Bv0RayAEHHNBS5+ab76KpaSqwG7AbTU1TueWW2Z2OSRIHHXQQy5cv54orNk9I/fv3\n580332xZX716NZLYdtttO33MrDiRmFm3d/jhh7H11lcDjwKr6NPnfD71qU+1bO/Rowf33TeLgw9e\nyIAB+zFu3OXMnXsnQ4YMaakzZMh2SMta1qVl7LDDoJJjW79+fd42kvHjx7Nw4cKW9QULFjB06FC2\n2267ko9ZdlnfOyvXC7eRmFkB2vpbce21v4zBg0dGnz7bxpFHHh9r1qwpar9Lly6NAQN2jN69z4ze\nvc+MAQOGRmNjY1H7ePXVV+PGG2+MtWvXxoYNG2L27NnRv3//+N3vfrdZ3dmzZ8fw4cNjyZIlsWrV\nqqirq4tvfetbbe67re+bLmgjcfdfM+tWshy08cUXX+Tmm28G4Atf+AKjRo0q6v2vvfYakydP5vHH\nH2fjxo2MHj2ac889l9NOO43ly5czfvx4lixZwsiRIwG45JJLuOiii3j77beZPHkyV1xxBVtttVXe\nfVey+68TiZl1Kx79N2+5nyMxM7PqlXkikTRJUqOkpySdl2f7IEm3SVoo6SFJ47KOyczMyifTRCKp\nBzADOAIYDxwvaY9W1b4FzI+ICcDJwKVZxmRmZuWV9RXJ/sCyiHghIt4DbgSOalVnHPAHgIh4Ehgj\naQhmZlYTsk4kI4DlOesvpWW5FgLHAEjaHxgFjMw4LjMzK5PyD21ZvIuA/5T0GLAImA9syFexvr6+\nZbmuro66urouCM/MrHY0NDTQ0NDQpcfMtPuvpIlAfURMStfPJ3k45uJ23vMcsFdErG1V7u6/ZtYh\nd//NW17T3X/nAbtKGi2pN3AcMCu3gqSBkrZKl/8Z+GPrJGJmZtUr01tbEbFB0tnA3SRJa2ZELJV0\nZrI5rgT2BK6VtBFYDJyeZUxmZlZemT9HEhGzI+KDEbFbRFyUlv08TSJExEPp9j0jYnJErG5/j2Zm\ntendd9/ljDPOYMyYMQwcOJB99tmH2bPbHkG4Vqba9ZPtZmapWbNm8Z2pU5k5cybr168v+/7Xr1/P\nqFGjeOCBB1i9ejX//u//zrHHHsuLL764Wd05c+bwox/9iLlz5/LCCy/wzDPPMG3atLLHVA4ea8vM\nupW2Gp3XrVvHDTfcwJtvvsknP/lJ9tprr022T5syhZsuu4xj163j/n792HbiRG6/5x569Mj2/+0J\nEyZQX1/P5z73uU3K/+mf/omxY8fyve99D4C5c+dywgkn8Le//S3vfrpzY7tZtyep5WXVae3atRz8\nkY9wx7nn8vyUKXzigAO46667WravWbOGH//kJ/xx3TouBO5uauKZhx/mT3/6U0udiOC7U6cyfNAg\nhg8axHenTi25d9grr7zCsmXLGD9+/GbbFi9ezIQJE1rWJ0yYwKuvvsrrr79e0jGz4ERiVgJJSZeV\nwUAvnEyq1NVXX82uy5czq6mJS999l1+99RbfPOuslu1r166lX8+eORPtwk49emwy1e4VM2Zw+/Tp\n3L96NfevXs3t06dzxYwZnY5p/fr1nHjiiZxyyinsvvvum233VLtmW4CWJHIGcE761cmkKq1auZI9\n33knZ6JdWPnGGy3bhw4dypixY7mgV690ol1YKG0y1e5dN9/M1KamdKJdmNrUxOxbbulUPBHBiSee\nSJ8+fbjsssvy1vFUu2ZbigHAsHR5WLpuVeewww/n6q23TifahfP79Nlsqt1Z993HwoMPZr8BA7h8\n3DjunDt3k6l2txsyhGU5/yQskxi0ww6diuf000/ntdde47bbbqNnz55569TSVLtubDfrpE2uSIYB\nfweuAtazRT5ZXS3aanS+/rrrmPKv/8rqdev47BFHcOWvfkX//v0L3m9jYyOHHnAAn3v7bQBu79uX\nPz78MB/84AeLiu/LX/4yjz/+OPfeey/9+vVrs96cOXM49dRTue+++xg2bBjHHHMMBx10EN///vfz\n1vcMiQVwIrFq1JJMBgBv4iRSBap5qt0XX3yRMWPG0Ldv35YrEUn8/Oc/55BDDum+U+1KGhsRz3VU\nljUnEqtWuW0i/h2tPI+1lbe84t1/b81T1rkWJrNuKCJaXmZbojbH2kpnMhwPDJR0TM6mAUDfrAMz\nM7Pa0N6gjR8EPgsMAv4xp3wN8M9ZBmVmZrWjkDaSAyPiL10UT3txuI3EzDrkNpK85RVvbB9CcgUy\nhpwrmIg4LcvA8sThRGJmHXIiyVueaSIpZD6SO4AHgHtpYwpcMzPbchVyRbIgIj7cRfG0F4evSMys\nQ74iyVte8e6/v5P0mSyDMDOz2lXIFckaYBvg3fQlkmlyu3RUIV+RmFkhfEWSt7yyVyQRsW1E9IiI\nvhExIF330HRmZp1w+eWXs99++9G3b19OO639PkvdZqpdJU6UNDVd30nS/oUeQNIkSY2SnpJ0Xp7t\nAyTNkrRA0iJJpxT1HZiZlcmsWbOYmuFUuwAjRoxg6tSpnH766e3W61ZT7Uq6AtgIfCIi9pS0HXB3\nROzX4c6lHsBTwGHAy8A84LiIaMypMwUYEBFTJO0APAkMjYj1rfblW1tm1qHOTrU75YIpXHbNZazb\nbR39Xu7HxF0mcs/vs5tqd+rUqaxYsYKrr7467/buNtXuARHxVeBtgIh4Hehd4P73B5ZFxAsR8R5w\nI3BUqzoBNM/Usi2wsnUSMTMrxdq1a/nI/h/h3EvOZcotUzjgY5tPtfuTn/yEdSesg49D03FNPLxo\n86l2p06byqAdBjFoh0FMnVb6VLvtqaWpdgt5juQ9ST1J/uA3P6C4scD9jwCW56y/RJJccs0AZkl6\nGegPfLHAfZuZFeTqq69mea/lvP25t5PuQrvAWV8/i+c//TyQJJqevXvy3jZpG0RP6DFo06l2Z1w+\ng+m/mE7TCU0ATP/FdHYcsiNfO/trmcTc3lS71Ta5VSGJ5FLgN8COkr4PTAYuKGMMRwDzI+ITknYB\n7pG0d0SsbV2xvr6+Zbmuro66uroyhmFm3dXKlSt5Z9A7tMy1uwO8sWrTqXbHjhnLsoZlrN9nPTwP\nemXTqXZvvuNmmg5sgu2T9aYDm7jlt7dklkg6O9VuQ0MDDQ0NmcTUlg4TSUT8StKjJO0cAo6OiKUF\n7n8FkDvzy8i0LNepwA/TYz0j6TmSKZUfab2z3ERiZlaoww8/nB/P+DFNezTBIOjTsPlUu/fNvo/j\nTz6e+dfNZ+SokVx/9/WbTLU7ZPshaJWI5OYMWiV2GNy5qXYL0TzV7uTJk4HCp9pt/U/2hRdemFmM\nLXLnUmjrBfQEPkCSFEYBo4p439PAaJJ2lQXAnq3qXA5MS5eHktwKG5xnX2Fm1pG2/lZce921MXjo\n4OjTr08cOfnIWLNmTVH7Xbp0aQwYPCB67987eu/fOwZsPyAaGxuLjm/9+vXx1ltvxZQpU+Kkk06K\nt99+O9avX79ZvdmzZ8fw4cNjyZIlsWrVqqirq4tvfetbbe63re87LS/ob31nX4X02voaMA14hWSs\nreYHEvcuJFFJmgT8J0nD/syIuEjSmek+rpQ0HPgFMDx9yw8j4oY8+4mOYjUzq+apdiG5Qrjwwgs3\nmVlz2rRpnHrqqYwbN46lS5d2y6l2nybpubUyy0A64kRiZoXwk+15yyve/Xc5sLrDWmZmtkUqpNfW\ns0CDpN8D7zQXRsT0zKIyM7OaUUgieTF99abwBxHNzGwL0WEbSUtFqT9A5Hm+oyu4jcTMCuE2krzl\nlW0jkfQhSfOBxcBiSY9KGp9lUGZmVjsKaWy/EvhGRIyOiNHA/wX+O9uwzMysVhTSRrJNRMxtXomI\nBknbZBiTmVmnjR49epNnNLYUo0ePrtixC3mO5DfAY8Av06ITgX0j4nMZx9Y6DreRmJkVqSraSIDT\ngCHAbelrSFpmZmZWVK+tgcDGiFiTbUhtHt9XJGZmRaqKKxJJ+0laBCwEFklaKGnfLIMyM7PaUUgb\nyePAVyPigXT9EOC/Ch20sVx8RWJmVryquCIBNjQnEYCIeBDwVLhmZgYUdkVyCbA1cAPJdLtfJJm/\n/XqAiHgs4xib4/AViZlZkaplGPm57WyOiPhEeUNqMw4nEjOzIlVFIqkWTiRmZsXrikTS4ZPtkgYB\nXwLG5NaPiHOyC8vMzGpFIUOk3Ak8BCwCNmYbjpmZ1ZpC2kgei4h9uiie9uLwrS0zsyJVS/ffX0r6\nZ0nDJQ1ufhV6AEmTJDVKekrSeXm2/z9J8yU9JmmRpPXp7TQzM6sBhVyRfBX4PvAGSfdfSHpr7dzh\nzqUewFPAYcDLwDzguIhobKP+Z4GvR8Thebb5isTMrEhV0dhOMv/IrhHxWif2vz+wLCJeAJB0I3AU\nkDeRAMeTPK9iZmY1opBbW08DTZ3c/whgec76S2nZZiRtDUwCbu3ksczMrAIKuSJZByxIH0x8p7kw\ng+6//wg8GBFvtFWhvr6+Zbmuro66uroyh2BmVtsaGhpoaGjo0mMW0kZycr7yiLi2w51LE4H6iJiU\nrp+fvDUuzlP3NuCmiLixjX25jcTMrEhV82S7pN7A7unqkxHxXkE7l3oCT5I0tv8NeBg4PiKWtqo3\nEHgWGBkRb7WxLycSM7MiVUVju6Q64FrgeUDATpJOjoj7O3pvRGyQdDZwN0l7zMyIWCrpzGRzXJlW\nPRqY01YSMTOz6lXIra1HgRMi4sl0fXfghojo0smtfEViZla8ankgcavmJAIQEU8BW2UXkpmZ1ZJC\nem09Iukq0vlHgBOBR7ILyczMakkht7b6AF8FDkmL7geuiIh32n5X+fnWlplZ8Sraa0vSEGBIRCxp\nVT4eeDUi/jfLwPLE40RiZlakSreRXAbskKd8MPCf2YRjZma1pr1Esmu+Lr4R8QCwd3YhmZlZLWkv\nkWzbzjb32jIzM6D9RPK0pM+0LpT0aZKn0M3MzNptbN8N+D3wZ+DRtPijwIHAZ9PnSbqMG9vNzIpX\n8bG20q6/JwAfSosWA7+OiLezDKqNWJxIzMyKVPFEUk2cSMzMilfp7r9mZmYdciIxM7OSOJGYmVlJ\n2hy0UdIiIF+jhEjmEvFDiWZm1u7ov5/tsijMzKxmudeWmVk3VhW9tiRNlDRP0lpJ70raIOnNLIMy\nM7PaUUhj+wzgeGAZsDVwBnB5lkGZmVntKKjXVkQ8DfSMiA0RcQ0wqdADSJokqVHSU5LOa6NOnaT5\nkp6QNLfQfZuZWeUVMtVuk6TewAJJPwL+RoEJSFIPkiuaw4CXgXmS7oiIxpw6A0mucD4VESsk5ZsD\nxczMqlQhCeGktN7ZwDpgJ+DzBe5/f2BZRLwQEe8BNwJHtapzAnBrRKwAiIjXCty3mZlVgQ6vSCLi\nhfSKZAxwG/BkRLxb4P5HAMtz1l8iSS65dge2Sm9p9QcujYhfFrh/MzOrsA4TiaT/A/wMeIbkYcSx\nks6MiLvKGMM+wCeAbYC/SPpL2i6zifr6+pbluro66urqyhSCmVn30NDQQENDQ5ces8PnSCQ1ksw/\n8nS6vgvw+4jYo8OdSxOB+oiYlK6fT/JU/MU5dc4D+kbEhen6VcBdEXFrq335ORIzsyJVxXMkwJpW\nVwfPAmsK3P88YFdJo9PbY8cBs1rVuQM4RFJPSf2AA4ClBe7fzMwqrJBeW49IuhO4iWTsrS+Q9L46\nBiAibmvrjRGxQdLZwN0kSWtmRCyVdGayOa6MiEZJc4DHgQ3AlRGxpLRvy8zMukoht7auaWdzRMRp\n5Q2pzTh8a8vMrEieITGHE4mZWfG6IpG0N4z8NyPiR5IuI89w8hFxTpaBmZlZbWivjaS5wfuRrgjE\nzMxqk29tmZl1Y1XR/VfSPZIG5axvl/ayMjMzK+g5kiER8UbzSkS8DuyYXUhmZlZLCkkkGySNal6R\nNJr8c7mbmdkWqJAHEr8NPCjpjyRjbX0M+JdMozIzs5pRUGN7OkfIxHT1oUoM9e7GdjOz4lX0OZJW\n+gCr0vrj0sDuzy4sMzOrFYUMI38x8EVgMbAxLQ7AicTMzAoaa+tJYO+IeKdrQmozDt/aMjMrUlU8\nR0IybPxWWQZhZma1q5A2kiZggaT7gJarEo+1ZWZmUFgimcXmk1GZmZkBHmvLzKxbq/Qw8jdFxLGS\nFpF/GPm9swzMzMxqQ5tXJJKGR8Tf0iFRNhMRL2Qa2ebx+IrEzKxIFZ8hUVJP4N6I+HiWQRTCicTM\nrHgV7/4bERuAjZIGdvYAkiZJapT0lKTz8mw/VNIbkh5LXxd09lhmZtb1Cum1tRZYJOkeYF1zYSHd\nfyX1AGYAhwEvA/Mk3RERja2q3h8RRxYetpmZVYtCEslt6asz9geWNbenSLoROAponUgyvewyM7Ps\ntJtIJH2Y5CpkcUQsba9uG0YAy3PWXyJJLq0dKGkBsAL4t4hY0oljmZlZBbTX/fc7wInAo8CPJP0w\nIv47gxgeBUZFRJOkTwO3A7vnq1hfX9+yXFdXR11dXQbhmJnVroaGBhoaGrr0mO11/10M7Jf+gd8e\nmB0R+xW1c2kiUB8Rk9L184GIiIvbec9zwL4RsapVuXttmZkVqdK9tt6JiCaAiFjZQd22zAN2lTRa\nUm/gOFoNtyJpaM7y/iTJbRVmZlYT2msj2VlS8x99AbvkrFNIL6uI2CDpbOBukkQ0MyKWSjoz2RxX\nApMlnQW8B7xFMveJmZnViPZubR3a3hsj4o+ZRNQG39oyMytexZ9sryZOJGZmxat0G4mZmVmHnEjM\nzKwkTiRmZlaS9h5I/C155iFp5rGxzMwM2u/+++P06zHAMOD6dP144JUsgzIzs9rRYa8tSY9ExEc7\nKsuae22ZmRWvWnptbSNp5+YVSWOBbbILyczMakkhw8j/K9Ag6VmSJ9xHA2dmGpWZmdWMgh5IlNQH\n2CNdbYyIdzKNKn8MvrVlZlakqnmyXdJBwBhyrmAi4rrswsobgxOJmVmRuiKRdHhrS9IvgV2ABcCG\ntDiALk0kZmZWnQppI/koMM6XA2Zmlk8hvbaeIHmOxMzMbDOFXJHsACyR9DDQ0sjuJ9vNzAwKSyT1\nWQdhZma1q9BeW0OB5vnaH46IVzONKn8MbqYxMytSVTzZLulY4GHgC8CxwF8lTc4yKDMzqx2FjLW1\nEPhk81WIpCHAvRExoQviy43DVyRmZkWqiisSoEerW1krC3wfAJImSWqU9JSk89qpt5+k9yQdU+i+\nzcys8gppbJ8taQ5wQ7r+ReCuQnYuqQcwAzgMeBmYJ+mOiGjMU+8iYE6hgZuZWXXoMJFExL+lVwmH\npEVXRsRvCtz//sCyiHgBQNKNwFFAY6t6XwNu4f0GfTMzqxGFDJEyFrgzIm5L17eWNCYini9g/yOA\n5TnrL5Ekl9z9fwA4OiI+LmmTbWZmVv0KubV1M3BQzvqGtKxcVw+XALltJ202CtXX17cs19XVUVdX\nV6YQzMy6h4aGBhoaGrr0mIX02loQER9uVbawkF5bkiYC9RExKV0/H4iIuDinzrPNiyRP0a8D/iUi\nZrXal3ttmZkVqVp6bf2vpJbhUCQdBbxW4P7nAbtKGi2pN3AcsEmCiIid09dYknaSr7ROImZmVr0K\nubX1ZeBXki4nGT7+JeBLhew8IjZIOhu4myRpzYyIpZLOTDbHla3fUnjoZmZWDQoaIgVAUn+AiFib\naURtH9+3tszMilQVt7YkDZU0E7g5ItZKGifp9CyDMjOz2lFIG8kvSB4U/EC6/hTw9awCMjOz2lJI\nItkhIm4CNgJExHren3LXzMy2cIUkknWStidtCE+79K7ONCozM6sZhfTa+gZJl91dJP0JGAJ4GHkz\nMwMKn9iqF/BBkocGn4yI97IOLE8M7rVlZlakivbaSod1HwYt7SL7At8HfiJpcJZBmZlZ7WivjeTn\nwLsAkv6BZJj360jaR1o/SGhmZluo9tpIekbEqnT5iyTDx98K3CppQfahmZlZLWjviqRn2jYCycRU\nf8jZVkgjvZmZbQHaSwg3AH+U9BrwFvAAgKRdcfdfMzNLtdtrK31mZDhwd0SsS8t2B/pHxGNdE2JL\nLO61ZWZWpK7otVXwoI2V5kRiZla8qhi00czMrD1OJGZmVhInEjMzK4kTiZmZlcSJxMzMSuJEYmZm\nJck8kUiaJKlR0lOSzsuz/UhJCyXNl/SwpIOzjsnMzMon0+dIJPUgmZr3MOBlYB5wXEQ05tTpFxFN\n6fJewE0RsWeeffk5EjOzInWH50j2B5ZFxAvpHCY3AkflVmhOIqn+pFP6mplZbcg6kYwAluesv5SW\nbULS0ZKWAr8FTss4JjMzK6OqGMU3Im4Hbpd0CPA94JP56tXX17cs19XVUVdX1xXhmbVr5MiRrFix\nghEjRvDSSy9VOhzbwjU0NNDQ0NClx8y6jWQiUB8Rk9L184GIiIvbec8zwH45c6E0l7uNxKqOegh6\nAgOAN4H14N9TqybdoY1kHrCrpNGSegPHAbNyK0jaJWd5H6B36yRiVo1GjhyZJJEzgHPSr73ScrMt\nSKa3tiJig6SzgbtJktbMiFgq6cxkc1wJfF7Sl0im9X0LODbLmMzKZcWKFTAYGJYWDAMGpOVmWxAP\nI2/WSSNHjmTFKyuSK5FhwN+Bq2DEULeVWPXwfCQ5nEisGklKruvdRmJVqju0kZh1axHBiKEjYFVy\nJeIkYlsiX5GYmXVjviIxM7Oq50RiZmYlcSIxM7OSOJGYmVlJnEjMzKwkTiRmZlYSJxIzMyuJE4mZ\nmZXEicTMzEriRGJmZiVxIjEzs5I4kZiZWUmcSMzMrCROJGZmVhInEjMzK0nmiUTSJEmNkp6SdF6e\n7SdIWpi+HpS0V9YxmZlZ+WSaSCT1AGYARwDjgeMl7dGq2rPAP0TEBOB7wH9nGVPWGhoaKh1CQRxn\nedVCnLUQIzjOWpT1Fcn+wLKIeCEi3gNuBI7KrRARD0XE6nT1IWBExjFlqlZ+uRxnedVCnLUQIzjO\nWpR1IhkBLM9Zf4n2E8UZwF2ZRmRmZmXVq9IBNJP0ceBU4JBKx2JmZoVTRGS3c2kiUB8Rk9L184GI\niItb1dsbuBWYFBHPtLGv7AI1M+vGIkJZ7j/rK5J5wK6SRgN/A44Djs+tIGkUSRI5qa0kAtl/EGZm\n1jmZJpKI2CDpbOBukvaYmRGxVNKZyea4EpgKDAb+S5KA9yJi/yzjMjOz8sn01paZmXV/FX+yXdJk\nSU9I2iBpn3bq5X2wUdJ2ku6W9KSkOZIGpuW9JP1C0uOSFqftM1UXZ7ptb0l/Tve/UFLvaosx3T5K\n0hpJ3+hMfFnHKelwSY+kn+G8tANH1cWZbpsiaZmkpZI+VWKc7f7ccuqdK2lR+jonp7z592+hpDsk\n9U/Ly30OZRJnq20lnUNZx5luL9d5lNXPvfjzKCIq+gI+COwG/AHYp406PYCngdHAVsACYI9028XA\nN9Pl84CL0uXjgV+ny1sDzwGjqjDOnsBC4EPp+nakV4rVEmPOe28G/gf4RpX+zCcAw9Ll8cBLVRrn\nOGA+ya3lMen7O/UzL+TnlvN5PA70SX/n7gF2Trc9DBySLp8CfDejcyirOMt2DmUZZwbnUVafZ9Hn\nUcWvSCLiyYhYBrTXmN7eg41HAdemy9cCRzfvGthGUk+gH/AO8GYVxvkpYGFEPJEe5/VIf4JVFCOS\njiIZhWDI7Nz2AAAIXUlEQVRxZ2LrijgjYmFE/D1dXgz0lbRVtcUJHAncGBHrI+J5YFm6n85q8+eW\nY0/grxHxTkRsAP4IHJNu2z0iHkyX7wU+ny6X9RzKMM6ynUMZx1nW8yirODtzHlU8kRSovQcbh0bE\nKwDpNz80Lb8FaCLpLfY88OOIeKOK4twxLd8dQNLs9HLy36ooxqFpbP2BbwIX0v4f1UrFuWOr9yJp\nMvBY+se92uJs/Z4VlDaiw44dfR7AE8DH0tsh/YDPADs1b5N0ZLp8LDAyXS73OZRVnOU+hzKJM4Pz\nKKvPs0Wh51GXPJAo6R7e/wMPyYcYwLcj4rdlPtzG9OsBwHpgGLA98ICke9P/AKshzub/mHoBBwMf\nBd4G7pP0SETMrYIYmz/LacBPI6JJUvMx21Whz7L52OOBHwKf7OiNlYyzGO3EeUEhx4mIRkkXk9za\nWEtya21Duvl04FJJU4FZwLtpeTnPoSzjLOocqmCc5TyPsoyz+dgFn0ddkkgiosNAOrACGJWzPjIt\nA/i7pKER8YqkYcCrafnxwOyI2Aj8r6Q/kfyiPV9lcb4E3B8RrwNIuhPYB8h7ElQoxgOAz0v6Ecn9\n5w2S3oqI/2rrIBWKE0kjgdtInkt6vqODVCjOFbz/X2Hr9xQdp6RX2vo8Wu3jGuCa9D3fJ70qiogn\nSQZWRdJuwP9J31LWcyjDOIs6hyoYZ1nPowzjLPo8qrZbW21l6JYHG5X0xjiOJIOSfj0lXT4FuCNd\nfhH4BICkbYCJQGOVxHlyTpxzgL0k9ZXUCzgUWFJNMUbEP0TEzhGxM3AJ8IP2fvkrFaekQcDvgPMi\n4qEyxVf2ONPy4yT1ljQW2JWk4bOz2jrOpt+ANCT9Ogr4HPDrVuU9SP7TvSJ9S7nPoXLH+bP0LeU+\nhzKJM4PzKJM4O3UeRQm9BsrxImkgWg68RXIv9q60fDjwu5x6k4AnSRomz88pH0zSUPQkyYOPg9Ly\nbYCbSO4RPkHpPSQyiTPddkIa4+PAD6sxxpw606r1swS+DawBHiO5hH8M2KHa4ky3TSHprbUU+FSJ\nn2dbn0frOO9Pf8/mA3U55eek720k+ePWXF7ucyiTOMt5DmUdZ5nPo6x+7kWfR34g0czMSlJtt7bM\nzKzGOJGYmVlJnEjMzKwkTiRmZlYSJxIzMyuJE4mZmZXEicTKQslQ649Jmp9+HSVpX0mXFLGPgZLO\nKuAYiyT9j6S+bdT7naQBnfgehku6qdj35bz/OUmD85RvI+lnkp5WMiz3HyTt19njVANJEyR9utJx\nWHXwcyRWFpLejIiC/nhL6hnJSKSty8cAv42IvTo6hqTrgUci4pJWdRQV+qWW9Czw0YhY1ar8BuDZ\niPh2uj4aGBcRd1UgzLKQdDLJ9/q1SsdilecrEiuXzYYQkXSopN+my9MkXSfpQeA6SeMk/TW9wlgg\naReSAeJ2Tssu7uB4D/D+0CSNkq6VtAjYqfnKIN22RNKVSiY8mi2pTxrPLpLuSY/9iKSxaf1F6faT\nJd0uaa6SiYO+k/N9/Sa9slgk6YwOPoOdSYaIbxlkL5Ih5+9Kt38j3c/jks5Ny0YrmfDqmvTY10s6\nTNKD6fpHW32mf07Lz8g57n+k+10o6dicn8dcSTen+/9lTv19JDWk39ddkppHfp4r6aL0Z9Uo6WAl\nQ4p/Fzg2/Vl9oYOflXV3pTyi75dfzS+SUWKbh1S4NS07FJiVLk8jGZeqd7p+KXB8utyLZOKd0cDj\n7RxjTU7924Ez0/dsAPbLqfcsyfARo0lGNN0rLf8f4IR0+SHgyHS5N9A39/gkYxetAAal2xaRTm7F\n+0NRNJdvl64/BwxuFfM/Nn8eeb6ffUgmZOpLMhzJEySTCjXHPS6t9whwVbp8JPCbnM90fhr/9iRj\nYw0jmW9iTlpnR+AFkhFkDwVeJxlCQ8CfgYPSz/NPwPbpe44FZqbLc4H/SJc/DdyT8/lcWunfO7+q\n49Ulo//aFqEpItqcjjY1KyKah6r+C/BtSTsBt0XE01KHo2pvLemxdPkBYCbJPB7PR8S8nHq5O3ou\nIhaly48CY5TMC/GBiJgF0BxTnuPfE+n8G5JuAw4hSZZfl9Q8idBIklkUOzPo4iEkSeHtnGN8DPht\nGnfzwIOLgfvS5UUkiabZHWn8KyX9gWSE2UOAG9Lv7VVJDcB+JOMnPRwRf0uPt4BkhsbVwIeAe5R8\nCD2Al3OOcVv69dFWxzYDumgYebPUuuaFiLhB0kPAZ4E7Jf0LyX/07dksWaV//Nflrw4ks/o120Dy\n3z8UNrFQ67aWkHQoyYi4B0TEO5Lm5uwzn8XAhE603eTGvTFnfSObnre5+xTvzyFDq/J8+92Q7kvA\nExFxcAexNNc324TbSKxciprxTdLYiHguIi4jGf56b5L/mLftxDHaO/Zm2yJiLbBcybSnKBnOfes8\n7/2kpEHptqNJbv8MBF5Pk8geJEOrtykiniW5NXVhS0BJG8hnSK6qjlYy/Pk2JEN8P1DA95TrqDT+\n7UluXc1L9/FFST2UDBX+Mdq/YnoSGCJpYhpfL0nj2qjbHNcaoOiecdY9OZFYuRTbU+rYtAF8PjAe\nuC6S3k5/Shue8zW2t3WMza4cCnjPl4BzJC0kSRBD89R5mOS2zgLg5oh4DJgNbCVpMfADklt0HR3r\nDGCYku6/j5NMMvRKRMwHfkHyx/8vwJURsbCI7wGSYdMbSNo7vhsRf4+I36TlC0mGGf+3iMg36VEA\nRDKN6mTg4vR213zgwDaO3bw+FxjnxnYDd/81y0tJ99Z9I+KcSsfSFknTSDogTK90LLZl8xWJmZmV\nxFckZmZWEl+RmJlZSZxIzMysJE4kZmZWEicSMzMriROJmZmVxInEzMxK8v8Bjo50EKUPB64AAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xcf444e0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "\n",
    "estimator = PCA(n_components=3)\n",
    "X_pca = estimator.fit_transform(X)\n",
    "plot_pca_scatter() # Note that we only plot the first and second principal component"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Import clustering modules\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.mixture import GMM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-1.          0.8660254 ]\n",
      " [-1.          0.28867513]\n",
      " [-1.          0.28867513]\n",
      " [-1.         -0.28867513]\n",
      " [-1.          0.28867513]\n",
      " [-1.         -0.28867513]\n",
      " [-1.         -0.28867513]\n",
      " [-1.         -0.8660254 ]\n",
      " [ 0.          0.8660254 ]\n",
      " [ 0.          0.28867513]]\n"
     ]
    }
   ],
   "source": [
    "# First we reduce the data to two dimensions using PCA to capture variation\n",
    "pca = PCA(n_components=2)\n",
    "reduced_data = pca.fit_transform(df1)\n",
    "print(reduced_data[:10])  # print upto 10 elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KMeans(copy_x=True, init='k-means++', max_iter=300, n_clusters=3, n_init=10,\n",
      "    n_jobs=1, precompute_distances='auto', random_state=None, tol=0.0001,\n",
      "    verbose=0)\n"
     ]
    }
   ],
   "source": [
    "# Implement your clustering algorithm here, and fit it to the reduced data for visualization\n",
    "# The visualizer below assumes your clustering object is named 'clusters'\n",
    "\n",
    "# TRIED OUT 2,3,4,5,6 CLUSTERS AND CONCLUDED THAT 3 CLUSTERS ARE A SENSIBLE CHOICE BASED ON VISUAL INSPECTION, SINCE \n",
    "# WE OBTAIN ONE CENTRAL CLUSTER AND TWO CLUSTERS THAT SPREAD FAR OUT IN TWO DIRECTIONS.\n",
    "kmeans = KMeans(n_clusters=3)\n",
    "clusters = kmeans.fit(reduced_data)\n",
    "print(clusters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Plot the decision boundary by building a mesh grid to populate a graph.\n",
    "x_min, x_max = reduced_data[:, 0].min() - 1, reduced_data[:, 0].max() + 1\n",
    "y_min, y_max = reduced_data[:, 1].min() - 1, reduced_data[:, 1].max() + 1\n",
    "hx = (x_max-x_min)/1000.\n",
    "hy = (y_max-y_min)/1000.\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, hx), np.arange(y_min, y_max, hy))\n",
    "\n",
    "# Obtain labels for each point in mesh. Use last trained model.\n",
    "Z = clusters.predict(np.c_[xx.ravel(), yy.ravel()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** K MEANS CENTROIDS ***\n",
      "[[ -1.00000000e+00   1.34151949e-16]\n",
      " [  0.00000000e+00  -4.62592927e-18]\n",
      " [  1.00000000e+00  -7.40148683e-17]]\n",
      "*** CENTROIDS TRANSFERED TO ORIGINAL SPACE ***\n",
      "[[ 1.   1.5  1.5  1.5]\n",
      " [ 2.   1.5  1.5  1.5]\n",
      " [ 3.   1.5  1.5  1.5]]\n"
     ]
    }
   ],
   "source": [
    "# Find the centroids for KMeans or the cluster means for GMM \n",
    "\n",
    "centroids = kmeans.cluster_centers_\n",
    "print('*** K MEANS CENTROIDS ***')\n",
    "print(centroids)\n",
    "\n",
    "# TRANSFORM DATA BACK TO ORIGINAL SPACE FOR ANSWERING 7\n",
    "print('*** CENTROIDS TRANSFERED TO ORIGINAL SPACE ***')\n",
    "print(pca.inverse_transform(centroids))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAAELCAYAAAB6X1VdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHYtJREFUeJzt3XmYHVWdxvHvL52VLCaERIKSJkEJmEhwgaCCAUdJQDQj\nyCIgAQScURGUUdRHJmERR2QJCowowrDIEgREcIFhJDACBlkkQiaABDoEQtaGBAhBO2f+OOd2n66u\nun27+/a56e738zx5cmu5VadOVb1Vt5bT5pxDRETS6FfrAoiI9CUKXRGRhBS6IiIJKXRFRBJS6IqI\nJKTQFRFJKEnomtkcM7smxbzaY2YbzGyHWpejo8xss5lNrMJ06sO0etwBd0vajjrKzK4zs0/Xuhyd\nZWZXmtmZiefZofVdrX2kYNoLzWyXakyrajuemR1hZn8Oofaimf3GzD4cjdKlB4KrFRbOueHOuee7\nMo3uZmb3mNlxmd7VfKC6Jz+cXVHZU4VEJfMxs/cCuzrnfh26Z5vZP8xsvZm9YmaPmtkno/GHm9k8\nM2sI4zxjZheY2daZ6S4ws3VmNqBbFm7L0JFttdJtozNZ8kPgrA6MX6gqoWtmXwcuAM4GxgLjgUuA\nah7ZDV+p1qkvm9VVsSy10Knlli3CF4FfZPo94Jwb4ZwbCVwBzDezt4UA/QOwC7Cfc24E8CFgDbBH\n6ctmVg/sBWymg/tZL9gXilS6j3QmS24H9jWzsR0uVZZzrkv/gBHABuCgMuPMAa4On6cDL2SGPwd8\nLHzeHfgz8CqwAjgv9G8AmsK81gPTQv/jgMXAWuB3wPhoupuBLwFPA89G/SaGz1cCFwN3hGk+CEyI\nvr8fsARoxB9EFgDHFSzjQGAe8CKwHLgQGBAvM/B1YGUY55iC6ZwN/AN4I5TpR1G5vxiWZR1wceZ7\nhfWQGa8+1GO/aP1dDrwUyngWYGHYbOB/8Uf5dcCzwMxoWseEfuvD/5+rpDyhblaGdfw48J6Csu4Q\n6vxV4E7gx6XtKAyfH7aRxjDeLqH/CcBbwJuhbLeF/qcBfwv9ngD+OZrWjmEarwCrgOujYTsDd4Vl\n+T/gkHLzyVmOZ4EPR92zgfui7q3C+n0/cHxYpiHt7Henh3VzHnB7O+OWtr9vhmlfFfofCDwW6u+P\nwHuj77wPeCTU/Q3A9cCZ8XaRmUe8Xw0GzgeeD9O+DxgUhu0J3B/6PwZMr3R95yzXN/Db7XLgWPx2\nXSrDAcCjYVoNwJzoe22yBJgI/A/+4LYKuBYYkZnfXcDnu5yZXZ4AzAgbXr8y42RDd1lmeBy6DwBH\nRhvjHpmwsOh7s/AhtBP+rP07wP2ZDeFOYGS00uMVcyWwGvhA+P61wHVh2OiwwmaFYV8FNlEcumeG\nso8O/+4HzoiW+e+hHuqA/YHXgbcVTOue7HzCsvwaGA5sHzaM/Sqph8x0sqF7K3ApfkfZBvgTcEK0\nc23CB6gB/wK8GK2bV4F3he630xJ6heXBH8j+DAwP3ZOAtxeU9QF84A8A9sbvIHHoHhPKMQD/S+ux\naNiVhJCI+h1cmhdwCPBa1H0d8O3weSAhJMP0lwFHhzqYGraZnYvmk5lnKVBHR/2aQxfoD5wc6nI4\nPtyurGC/ewZ/EH4/fv8bU2bc0vZ3TqirQfhQXQl8MCzX5/H74YDw73n8Nl8X6u0tWofufZl5xPvV\nJfiz9W3DtPcM09wOH2ozwnj/FLpHV7K+M/ObiT+A7AIMwf+SiMvwUWBy+DwljPvpMlmyYyhPf/z+\nuwC4IDPPiwgngV3KzC5PAI4AXmpnnI6E7oIw/ujMOK3CIvT7LXBs1N0PH2bbR0E1PTOd7JnuT6Nh\n+wOLw+fPkwku/M5XFLp/K21MoXs/YGm0zK9nyr6ScEDJmVZR6H4o6r4R+GYl9VBUj/igfJNwQArD\nDwf+EO1cT0fDhoRyjMWHyTrgM8DgzDwKywPsi//1MC3e6HPKuT1+Rx8S9fsFxTvhyFC2UpiXDcMw\nzmPAp8Lnq4CfAO/IjHMocG+m30+A0yuZDz5omoCBUb/Z+BBchz94PgDsG4bdBZzTTrn3wh8MR4Xu\nxcDJZcafHtbzgKjfpYSTgqjfEnzY7Q0szwy7n/Khuxl/tmj4X2lTcsrxTcJZdtTv9/h9raPr++dx\nPQHvJgrdnPEvBM7P7gNl6mwW8Eim39nA5eXWTSX/qnFNdy2wTRXvhn8Bf/azJNwx/GSZceuBi8LN\nhHWhLA54RzTO8nbm93L0+Q1gWPi8Hf4nWazctLbDh3JJQ+hXstY5t7lgXpVaWfD9Suohz3j8WcWK\n8N1GfKBsE43TXD/OuY3h4zDn3BvAYcC/hu/fbmY7tVce59w9+Es6lwArzewnZpZXD9sBjdE8wdcp\nAGbWz8z+w8z+Zmav4A/cLlP2VszsaDN7zMwaw7JOjsb/Bv7g8JCZ/dXMjo2WZc/SsoTvHYE/YFXi\nlfD/8Ez/B51zWzvnxjrnPhzqBXxdjWtnmkcDdznnGkP39fggxMy2DzezN5jZ+ug7q51zf4+664FT\nM8v1Tny9b4e/BBZroDLb4M+kl+YMqwcOzczzI/jlLbu+c2T3zwaia7RmNs3M/mBmq8L28UXKbxtj\nzex6M1sexr82Z/zhtKzPTqtGUD6IP+r+c4Xjv44/SwKaL+qPKXU75551zh3hnBsDnAv80syGkH9n\nchnwxbDxbu2cG+WcG+ac+1M0Tt73KrECf/SNvbPM+C/iN6qSevz1ps7oaJlfoP16KPrem/hfFaXv\njXTO7VpRIZ37b+fcfvifkU8BP6ukPM65i51zHwTegz/AfiNn8iuAUWHdl4yPPh8JfAr/C2kk/nqg\n0bLjtapDMxsP/BT4UijPKODJ0vjOuVXOuROdc+/AX0a5NDx+9AKwILMsI5xzX8mbT04dvYG/prtT\nufEidwMzMssdL8dg/Nn3dDNbYWYrgFOAqWb2XufcC84/oTPc+ZtwzUXJTOoF4Hs56+hGfN1nD9hx\n3Wf34W2jYWvw29SOOcV/AX/mGs9zuHPuXNpf31nZ/bM+s4y/AH6FP9CPBC6jYNsIzsGfrU8O4x9F\n2xttu+DvQXRJl0PXObcefzngEjObZWZDzKy/me1vZv+R85WngcFheH/gu/hraACY2ZFmVjrCvIqv\noM3462ibab0yLwO+Y2bvCd99m5l9tqvLFPwGmGJmnzazOjP7CuXPbm4Avmtm24Tynw509pnSlfif\napX6CR2rh1LQvIz/OXtheEzJzGyimX20vRmGM4NPm9lW+J/Kr+HXT9nymNkHzWyPsO434nfQzdnp\nO+eWAQ8DZ5jZADPbCx+yJcPwB/tGMxsKfJ/WO1O2DoeG+awJZ8nH4q/1lZbns2ZWCppXwrib8TdZ\ndzKzo8J2PSAsw6SC+eT5Lf4nfiWuwYfTzWY2KayT0Wb2bTObib+c8w98AEwN/3bB31SbXeE8wB8g\n/8XM9gAws6FmdkCoyweBf5jZSWGZDyJ6cgIfPJPNbFczG4Tf/x2A87/DrwQuMLNxoa73NP9UxrXA\np8xsv9B/sJlNN7PtKljfWfOBY8xsl7AN/ntm+DD8mfPfwzIeEQ3Ly5Lh+G14Q9gOWp0IhOX8APDf\nZcpUma5en4iud3wOf4NkA/4M73ZgzzCs+Zpu6D46jPMy/o7+Ulqu6V6D35DXA38lXHMLw+bir4Gt\no+UG25HAIvyO0kB0zYWcazy0vZF2ZjSs1fVm/HXZp/B3Wi/GX9c6smD5B+GfXngJf9Z7IeE6Xna6\noV/zMudMa88w37XAvLxlwT9mFJe9sB4y0251PQu/sV2K39Eb8XesD3XF1+6a8CGzLf76e2NYH38g\n3FwqVx7gY/iddn1Yl9cAWxWUdQf8ne/1+BuiP6Ll3sBQ/JnMevylhaMy6/Zd+Gu264BbXMs1ubVh\nvucRXTsHfoC/fLQef5PqC1E53o0P31X4HfZu/HO3ufPJWY7JwBNRd5t6zYw/HH9jcFlUnvOAUfgn\nQc7N+c4hYdtrc50yb/uLtu+HQtlfxN8nGBqGvZ+Wu//XEz29EIZ/O9RFAz7Q4rofHMq/nJYnS0o3\nsncP3Wvx+/ntwDvDsAlF67ugnkpPYyzH31SNy3AQ/mbgq/gb0K2mRSZL8L+6Hg7zfhT4Gq2z4BDg\nl9XIytKjQdIOMzP8yj3COXdvrcsjPYuZXQvMd+EFCelZzOxB/IF4cZenpdAtZmb7AQvxP4G/gb9p\nNNE5t6mmBRORHqvHvX+f2IfwN0FWAZ8EZilwRaQrdKYrIpKQznRFRBJS6G5BwmNBPy0z/Dkz+1jK\nMtVaeKQo+5JKZ6fV5ZbHKlhHs83sf7syj4LpFjZbaL6Fv99Xe57SPfpc6Fp+E5QfqcJ055jZ1V2Z\nhnPu+865E7tall5oi7kGFq8jK24isDvKWzhN59x1zrmZpe5yAS2116dC14qboCz3EHY157/FN8+4\nJZXRtvwmCLvU3Ggn5lWpJAepHrB+tkh9JnTNbARwBv410Nuccxudc03Oud86574VxjEz+5b59/lX\nm9kNZjYyDCud1RxtvnHpVWb2nTBsBr4lrcPCGfRjof89Zna2mf3RzF4HJoS3dG4zs7Vm9rSZHR+V\nsVVL+Wb2eTN7PpTlO5nl2T2csb9q/nXQ8wqWe6T5dhFWhXneHr15VVTGEWb2czN7ycxeMLOzisI4\nlHm+mV1jvsHtx83s3aEeV4a6+ng0/jFmtjiM+zczOzEaNj3M75vmX2+9Imd+XzWzJ8xsu9B9oLW0\np/BH8w2Gl8Z9n5k9EuroBvxD+7lCPb8vfD4yrOtdQvdxZnZLtLylXzSl57VfCcszrWVy9kPz7Qs8\na/5Nsrx5HmNmv466nzGzG6PuZWYWv5L9ibDNrDOzi6Pxmi9pmNm9+IBeFMp0SHv1lFOuyWZ2V9he\nVphZaf+YY2Y3hXX9CjDbzAaab3D9RfPtFlxooVF182/S3R7muTaUrTSP08L4683s/8xs36Ly9DrV\neMOiJ/yjsiYoT8a3+DQO3xDMf9LS1GM9/tXBy/CvLe+Kf353Uhje6q270O8e/FsxO+MPcP3xO+qP\nw/Sn4h9H2yc7DfwbMhvwDYIMwLdP+hbtNIGZs0xb418dHYR/i+tG4NZ2yljY3GPO9OfgG9/5ePj+\nVfi37b6NbxbweEJra2H8/YEdwue98e/x7xa6p9O2CcLmt6nwr3o+DGwdujvdPGHOcvwX8LXw+TJC\n04mh+ypCK16ZdVR6uy9uInB2mE+b5jBz5jkBWBc+jwvlLS3rRHwjSaVxyzXt2eoNtzBu3C50YT3l\nlGkY/s22U/Db+VBg92jZN9HSMttgyjdpeg5+O+oX1sFHQv+d8G/blZrVHB+Xt7f/q3kBki1oZU1Q\nLiY0sRe6x4UdqF+0g42Lhi+k5ZXZotCdG3W/Ex8qW0X9zgGuyE4D33bDddF4W4UNvmwTmBXUw26Z\nnTlbxrGUae4xZ3pzgDuj7gPxr1KWHkccFuptRMH3bwVOCp+n07YJwun4NwHPx78iOiwa1unmCXPK\ncRzwq2g7OI6WA+7ztBwY8kI3brJzNm2bw2wCxhbMtyGsk8PwYf8nfCgdUypPGK9c0555oRu/Ml5Y\nTznlOZxMk4aZdb0g069ck6ZnhPW7Y+Y7O+KbAPgnoH9X9uue+K/PXF6gsiYo64FbraVJwsX4kIwb\nuilqXrFIfOd9O/yZzRtRvwbym2Bs1XRd+M7aaHhFTWCab4DosvDz+RX8mfbIzOWCuIz1tN/cY1Zc\nJxuBNS7sXaHbCPVkvqGjB8PPzUb8mW887WwThODbyj0B+L5z7rVMWavVPOG9wN7mW8zqh29QZS/z\nfxZnhHPuL2W+m5VtDrN5+Qvmuy++0e0F4d8++INN9nXzjm57JeXqKWt7/AtBRbJPkpRr0vSHYVp3\nhUtJp4FvSRB/Jj0X37zndWbWXnOWvUZfCt1KmqBcBuzvWjc9N9Q5t6KC6RfdvIj7vwRsbb4lp5Lx\ntA0HyDRdZ74lpdHNEy1uAjPrVHyDLbs732RdqQWxOHTjMnapucdyzGwg8MtQ3jHON6/4uzJlKVmH\nP4P+L2v9x0670jxhKyEINgIn4c8aX8OH54n4P2WT+7Wi6XXAffiQ3QsfsvfhA/ejtA3dzipXT3nj\n5jXLWJJd5sImTZ1zrznn/s05tyP+77h9vXTt1jl3g3Nu7+i7eS0S9kp9JnRdcROUM62lCcrLgHPM\nt72KmY2x1n82u9wd5JXADpkzyGwZluOvf33fzAaFmyRfIL8JyF8CB5rZh8ONiTPj+VtxE5hZw/Fh\nst78X5OdW2YZcF1o7rECA8O/Nc65zWa2P/7naLucc/fhWy672cx2D7270jxhnnuBr9ASdgsy3Vl5\nTQR2VOlMd4hz7iV8E40z8QfYxzo5zZdp3dxkuXrKugPY1vwNy4FmNqz0vQKFTZqa2SfNrFQ3G/BN\nUm42s53MbN9wEH4Lv33mbbu9Up8JXQDn3AX4piS/i78RsQz4Mr6JQPB/A+k2/M+hV/EBGW9w2aN8\n3H0TPhTXmtnDBeODbwJzAv5s4Gb8n325J6esi0PZrg/jrqX1X66YCTxp/q8DXAgc5vLbhZiHvx68\nJizPb8ssQ8nR+HBcjD/LvAnflGNnOfBnPvgbWzeFyzeH4+u7sok4dzf+IPVrM9vNOfcI/rLDxWF6\nTxPalA2XKA7C/8HCtfim+W5uZxb34n+y31fQnS3PRuB7wP3hZ3tROBWeETvnnsEH0n2hewP+J/kf\no0s0ZaeRYy5wdSjTZ8vVU055XgM+gT8zfTmMu0+ZeZ2Nv7m5CN9k58P4OgH/C+tuM9uAv55+ifMt\n9A3Cn9muxm/bY/A3XvsEtb0gIpJQnzrTFRGpNYWuiEhCCl0RkYQUuiIiCfUvN9DMdJdNRKQTnHO5\nj4+WDV2A6x6tSlOm0oMccPXx7Y8kFbt3ZbkX4aQ3mnX9ksJhurwgIpKQQldEJCGFrohIQgpdEZGE\nFLoiIgkpdEVEElLoiogkpNAVEUlIoSsikpBCV0QkIYWuiEhCCl0RkYQUuiIiCSl0RUQSUuiKiCSk\n0BURSUihKyKSkEJXRCQhha6ISEIKXRGRhBS6IiIJKXRFRBJS6IqIJKTQFRFJSKErIpKQQldEJCGF\nrohIQgpdEZGEFLoiIgkpdEVEElLoiogkpNAVEUlIoSsikpBCV0QkIYWuiEhCCl0RkYQUuiIiCSl0\nRUQSUuiKiCSk0BURSUihKyKSkEJXRCQhha6ISEIKXRGRhBS6IiIJKXRFRBJS6IqIJKTQFRFJSKEr\nIpKQQldEJCGFrohIQgpdEZGEFLoiIgkpdEVEElLoiogkpNAVEUlIoZuIc47nlzyBc67WRekVnHM8\nvmq96rMKnHMsXfem6jIRhW4iDU89ydxjP0PDU0/Wuii9wqLVG5g5fyGLVm+odVF6vOcaN3Ha3Q08\n17ip1kXpExS6idRPmszcK2+lftLkWhelV9h1zHB+f+g0dh0zvNZF6fEmjBrEDz5ez4RRg2pdlD6h\nf60L0FeYGTvsPKXWxeg1zIypY0fUuhi9gpkxcevBtS5Gn6EzXRGRhBS6IiIJKXRFRBJS6IqIJKTQ\nFRFJSKErIpKQQldEJCGFrohIQgpdEZGEFLoiIgkpdEVEElLoiogkpNAVEUlIoSsikpBCV0QkIYWu\niEhCCl0RkYQUuiIiCSl0RUQSUuiKiCSk0BURSUihKyKSkEJXRCQhha6ISEIKXRGRhBS6IiIJKXRF\nRBJS6IqIJKTQFRFJSKErIpKQQldEJCGFbiJNTU3cdsWPaWpqqnVReoWmpibOf2ip6rMKmpqauOnJ\nNarLRBS6idxx1aXcePG53HHVpbUuSq8w75EGznrgGeY90lDrovR4tyxp5NpFa7hlSWOti9In9K91\nAfqKA2d/qdX/0jWnfKC+1f/SeQftPKrV/9K9FLqJ1NXVMeu4k2pdjF6jrq6OU/eYWOti9Ap1dXUc\nMnmbWhejz9DlBRGRhBS6IiIJKXRFRBJS6IqIJKTQFRFJSKErIpKQQldEJCGFbjeYsu0Ipo2v7EHz\naeNHMWXbEd1cop5t0IyjGHL418Cs/IhmDDn8awyacVSagvVAkw7+MrudeHZFdbnbiWcz6eAvpylY\nH6KXI6psyrYjeO+4lhBduKz41cpp40cxcfTQ5u4nXl7frWXriQbNOIrBM1tCdOON88C5tiOaMeSw\nUxg4bUZzr013XpuiiD3GpIO/zKSDv9Lc/ZefnV5Yl7udcBbj9zm4uddTN1+Sooh9gs50q2zowLrm\nzxNHDy08480Gbvw9adFv1JjmzwOnzWDIYae0PUvLCdz4e+INGT2u+fP4fQ5mtxPOyq3LbODG35Ou\nU+hW2cJljSxd+3pzd17wZgN36drXy54R92Ubb5zHWwvvbO5uE7w5gfvWwjv9GbG08pefnc6yBTc3\nd7cJ3pzAXbbgZn9GLFWjywvdoBSgpWAt/f/Qskb2UOB2jHPNAVoK1tL/G+dfxJBDT84P3LyfzX2d\nc80BWgrW0v+PXz6HqcefkR+4qsuqUuh2k7zgjcMWFLgVKwjeOGxBgVuRguCNwxYUuN1Jlxe6UfZS\nQ0yB20EheONLDTEFbgeE4I0vNcQUuN1LodvNHioI1qL+UoZzbJx/Ue6gjfMvUkh0hHM8fvmc3EGP\nXz5HddmNFLrdbI+CpxeK+ksZZgw59OTcQUMOPbn9Z0+lhRlTjz8jd9DU489QXXYjhW43yj6lECv3\nOJnkyHlKIVb4OJm0lfOUQqzwcTKpCoVuN8l7LOyGx5a3+ziZ5Ch4LOzVUw8o/ziZtFXwWNjtR00p\n/ziZVI2eXugG5Z7DLXqcTDfVCpR7Drfc42S6qdZWuedwyzxOpptq1aUz3Sqr5MWHSl6gEK9c4AK5\nTzU0n/FKK+UCF8h9qqH5jFeqRqFbZa+/1dT8udxjYdngjb8nLTY3rm7+XPhYWE7wxt8Tb+PaFc2f\nCx8Lywne+HvSdbq8UGWlRmuGDqxr95JBafjrbzWpsZsCpUZr+o0aU/6SQXSpYXPjajV2k6PUaM2Q\n0ePKXzKILjVsXLtCjd1Umbky12rMzF336AsJiyNbggOuPr7WRehV7l3ZUOsiSGKzrl+Ccy73LqQu\nL4iIJKTQFRFJSKErIpKQQldEJCGFrohIQgpdEZGEFLoiIgkpdBNpamritit+TFOT3jyrhqamJs5/\naKnqswqampq46ck1qstEFLqJ3HHVpdx48bnccdWltS5KrzDvkQbOeuAZ5j2iFw+66pYljVy7aA23\nLFGjSynoNeBEDpz9pVb/S9ec8oH6Vv9L5x2086hW/0v3UugmUldXx6zjTqp1MXqNuro6Tt1jYq2L\n0SvU1dVxyORtal2MPkOXF0REElLoiogkpNAVEUlIoSsikpBCV0QkIYWuiEhCCl0RkYQUuiIiCSl0\nRUQSUuiKiCSk0BURSUihKyKSkEJXRCQhha6ISEIKXRGRhBS6IiIJKXRFRBJS6IqIJKTQFRFJSKEr\nIpKQQldEJCGFrohIQgpdEZGEFLoiIgkpdEVEElLoiogkpNAVEUlIoSsikpBCV0QkIYWuiEhCCl0R\nkYQUuok453h+yRM452pdlF7BOcfjq9arPqvAOcfSdW+qLhNR6CbS8NSTzD32MzQ89WSti9IrLFq9\ngZnzF7Jo9YZaF6XHe65xE6fd3cBzjZtqXZQ+QaGbSP2kycy98lbqJ02udVF6hV3HDOf3h05j1zHD\na12UHm/CqEH84OP1TBg1qNZF6RP617oAfYWZscPOU2pdjF7DzJg6dkSti9ErmBkTtx5c62L0GTrT\nFRFJSKErIpKQQldEJCGFrohIQgpdEZGEFLoiIgkpdEVEElLoiogkpNAVEUlIoSsikpBCV0QkIYWu\niEhCCl0RkYQUuiIiCSl0RUQSUuiKiCSk0BURSUihKyKSkEJXRCQhha6ISEIKXRGRhBS6IiIJKXRF\nRBJS6IqIJKTQFRFJSKErIpKQQldEJCGFrohIQgpdEZGEFLoiIgkpdEVEElLoiogkpNAVEUlIoSsi\nkpBCV0QkIYWuiEhCCl0RkYQUuiIiCSl0RUQSUuiKiCSk0BURSUihKyKSkEJXRCQhha6ISEIKXRGR\nhBS6IiIJKXRFRBJS6IqIJKTQFRFJSKErIpKQQldEJCGFrohIQgpdEZGEFLoiIgkpdEVEElLoiogk\npNAVEUnInHPFA82KB4qISCHnnOX1Lxu6IiJSXbq8ICKSkEJXRCQhha6ISEIKXRGRhBS6IiIJ/T9e\nv00Jfld8LQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x9c04b70>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Put the result into a color plot\n",
    "Z = Z.reshape(xx.shape)\n",
    "plt.figure(1)\n",
    "plt.clf()\n",
    "plt.imshow(Z, interpolation='nearest',\n",
    "           extent=(xx.min(), xx.max(), yy.min(), yy.max()),\n",
    "           cmap=plt.cm.Paired,\n",
    "           aspect='auto', origin='lower')\n",
    "\n",
    "plt.plot(reduced_data[:, 0], reduced_data[:, 1], 'k.', markersize=2)\n",
    "plt.scatter(centroids[:, 0], centroids[:, 1],\n",
    "            marker='x', s=169, linewidths=3,\n",
    "            color='w', zorder=10)\n",
    "plt.title('Clustering on the lenses dataset (PCA-reduced data)\\n'\n",
    "          'Centroids are marked with white cross')\n",
    "plt.xlim(x_min, x_max)\n",
    "plt.ylim(y_min, y_max)\n",
    "plt.xticks(())\n",
    "plt.yticks(())\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Elbow Method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAAEbCAYAAABgLnslAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcVOWd7/HPrwERcAECLqA2i2uIRJaooxibIFd0DEp0\nTISoGBNzMwmuycQspmHI3IyJY6LE3CTGARdIMBo1co2KkVZcggoYEI0iS6MQG9wIgoNg/+4fzym7\neq/q7tPnVNX3/XrVi+qqOlW/RulvP8/5necxd0dERCSNypIuQEREpDkKKRERSS2FlIiIpJZCSkRE\nUkshJSIiqaWQEhGR1Io1pMysu5ktMbPlZrbSzCqjxyvN7HUzWxbdJsRZh4iIFCaL+zopM+vp7jvM\nrAvwJHApcBqwzd2vj/XDRUSkoMU+3efuO6K73YGuQCYVLe7PFhGRwhZ7SJlZmZktB94AFrr7s9FT\n3zCz583sN2a2b9x1iIhI4Yl9uu+jDzLbB7gHmAZsAd50dzezHwIHuvvFnVKIiIgUjE4LKQAzuwbY\nnn0uyszKgfvdfXgTr9fCgiIiRcTd8zrVE3d3X7/MVJ6Z9QDGA38zswOyXvY54IXm3sPdU32rrKxM\nvIZiqbMQaiyUOguhxkKpsxBqLJQ626Jrm47K3YHArWZWRgjE+e7+gJndZmbHALXAeuCrMdchIiIF\nKNaQcveVwMgmHr8gzs8VEZHioBUn2qmioiLpEnJSCHUWQo1QGHUWQo1QGHUWQo1QOHXmq1MbJ/Jl\nZj5lynRmzpzK4MHlSZcjIiLtYGZ4no0TqQ8peI+hQytZuHCagkpEpIC1JaQKYLqvF2vWzOCaa+Yk\nXYiIiHSyAggpgF5s2lSbdBEiItLJCiSktjNgQIGUKiIiHaYAfvJvZ+jQSmbOnJp0ISIi0slSH1J7\n7nkdv/iFmiZEREpR6rv7pk93Nm6EX/866WpERKQ9irIFvabGOeIIWL0a+vVLuiIREWmromxB328/\n+Nzn4Je/TLoSERHpbKkfSbk7K1fCqafCunXQvXvSVYmISFsU5UgK4OijYdgwmD8/6UpERKQzFURI\nAVxxBfz0p5DigZ+IiHSwggmpCRPg/ffhsceSrkRERDpLwYRUWRlcfnkYTYmISGkoiMaJjB07oLwc\nnn4aDj00wcJERCRvRds4kdGzJ3zlK3DDDUlXIiIinaGgRlIAGzeGbr+1a6F374QKExGRvBX9SApg\n4EA4/XS4+eakKxERkbgV3EgKYOlSmDQpjKa6dk2gMBERyVtJjKQARo2CQYPg7ruTrkREROJUkCEF\ndRf3iohI8SrYkJo4EbZsCe3oIiJSnAo2pLp0gUsv1WhKRKSYFWTjRMa2beHi3uXLw58iIpJeJdM4\nkbH33jB1KsyalXQlIiISh4IeSQGsXx+6/davD6ElIiLpVHIjKQit6GPHwpw5SVciIiIdreBHUgBP\nPgkXXggvvxwaKkREJH1SN5Iys+5mtsTMlpvZSjOrjB7vY2YPm9nLZvaQme3bns854QTo2xcWLOiY\nukVEJB1iDSl33wmMdfcRwDHAaWZ2LHA18Ii7HwE8CnynPZ9jpot7RUSKUeznpNx9R3S3O9AVcOBM\n4Nbo8VuBs9r7OeecA2vWhHZ0EREpDrGHlJmVmdly4A1gobs/C+zv7jUA7v4GsF97P6dbN/jGNzSa\nEhEpJrGvIe7utcAIM9sHuMfMhhFGU/Ve1hGfdcklMGQI/P3vcOCBHfGOIiKSpE7b6MLd/2FmVcAE\noMbM9nf3GjM7ANjc3HHTp0//6H5FRQUVFRXNfkafPjB5Mtx0E/zwhx1VuYiItEVVVRVVVVXteo9Y\nW9DNrB+wy923mlkP4CHgP4GTgbfd/Voz+zbQx92vbuL4nFrQs73yCowZA9XV0KNHB3wTIiLSIVLX\ngg4cCCwys+eBJcBD7v4AcC0w3sxeBsYRgqtDHH44HHcc3H57R72jiIgkpSgu5m3o0UdDE8WqVaE9\nXUREkpfGkVQixo4N3X4PPZR0JSIi0h5FGVK6uFdEpDgU5XQfwM6dYfHZRx6BYcM6ti4REcmfpvuy\ndO8OX/sa/OxnSVciIiJtVbQjKYDNm+GII0Jbev/+HViYiIjkTSOpBvbbD84+G375y6QrERGRtijq\nkRTACy/A+PFh597u3TumLhERyZ9GUk34xCfg6KPhd79LuhIREclX0YcU1LWjp3jQKCIiTSiJkDr1\n1NCS3s51DkVEpJOVREiVlcHll+viXhGRQlP0jRMZO3ZAeTk89RQcdliHvKWIiORBjRMt6NkzbIp4\n441JVyIiIrkqmZEUwKZNodtv7Vro3bvD3lZERHKgkVQrBgyA00+Hm29OuhIREclFSY2kAJYuhUmT\nwmiqa9cOfWsREWmBRlI5GDUqrI5+991JVyIiIq0puZAC7TUlIlIoSjKkJk6ELVvg6aeTrkRERFpS\nkiHVpQtceqlGUyIiaVdyjRMZ27aFc1PLloWLfEVEJF5qnMjD3nvD1Kkwa1bSlYiISHNKdiQFYY+p\nUaPCn3vvHdvHiIgIGknlbdAg+MxnYPbspCsREZGmlPRICsKCs+efD6+8EhoqREQkHhpJtcE//RP0\n6wf33590JSIi0lDJh5SZLu4VEUmrkg8pgLPPDmv5LVuWdCUiIpJNIQV06wbTpmk0JSKSNiXfOJHx\nzjswZAisWhW29BARkY6VusYJMzvIzB41s1VmttLMpkWPV5rZ62a2LLpNiLOOXPTpA1OmwE03JV2J\niIhkxDqSMrMDgAPc/Xkz2wtYCpwJfB7Y5u7Xt3J8p42kAFavhhNOgOrqsN28iIh0nNSNpNz9DXd/\nPrr/HvASMDB6Oq9CO8Nhh4WW9NtvT7oSERGBTmycMLNBwDHAkuihb5jZ82b2GzPbt7PqaM0VV8DP\nfga1tUlXIiIinRJS0VTfXcBl0YjqF8AQdz8GeANocdqvM1VUQPfu8NBDSVciIiJd4/4AM+tKCKjb\n3f0+AHffkvWSm4Fm13uYPn36R/crKiqoqKiIpc6M7It7Tzst1o8SESlqVVVVVFVVtes9Ym9BN7Pb\ngDfd/cqsxw5w9zei+1cAn3L3yU0c26mNExk7d4bFZx95BIYN6/SPFxEpSm1pnIi7u+9E4HFgJeDR\n7bvAZML5qVpgPfBVd69p4vhEQgpg5kzYsAFuvjmRjxcRKTqpC6n2SjKktmyBww8Pq6P3759ICSIi\nRSV1LeiFrH9/OOcc+OUvk65ERKR0aSTVglWr4JRTws693bsnVoaISFHQSKqDDRsGw4fD736XdCUi\nIqVJIdWKyy8P7egpHnCKiBQthVQrTj01tKS3s9VfRETaQCHVirKyutGUiIh0LjVO5GDHDigvh6ee\nCovQiohI/tQ4EZOePeGSS+CGG5KuRESktGgklaNNm0K339q1YYNEERHJj0ZSMRowAM44Q8skiYh0\nJo2k8rBsGZx5ZhhNdeuWdDUiIoVFI6mYjRwJQ4bA3XcnXYmISGlQSOUps9dUigZ4IiJFK+eQMrMu\nZjbAzA7J3OIsLK0++1l48014+umkKxERKX45nZMys2lAJVBD2AMKwN19eIy1pe6cVMaNN8LixfD7\n3yddiYhI4YhtPykzexU4zt3famtxbZHWkNq2Lezcu3Rp+FNERFoXZ+PEa8DW/EsqTnvvDRddBLNm\nJV2JiEhx65rj69YCVWb2/4CdmQfd/fpYqioAZ51Vzbhxc3j22VoOOaSMmTOnMnhwedJliYgUlVxD\nakN02yO6lbR166qZOnUWH3wwg8WLewHb+ctfKlm4cJqCSkSkA+V1Ma+Z7QXg7u/FVlH9z0vlOakv\nfnEGc+d+E+iV9eh2pky5jjvuqEyqLBGRVIvtnJSZfcLMlgOrgFVmttTMhrWlyGKwcWMt9QMKoBeb\nNtU29XIREWmjXBsnfg1c6e7l7l4OXAWU7Cp2AweWAdsbPLqdDRvKeP/9JCoSESlOuYZUL3dflPnC\n3atoPJQoGTNnTmXo0Erqgmo7gwdXctRRUzn+eHjllQSLExEpIrleJ3UPsAy4PXroi8Aod58UY22p\nPScFoXnimmvmsGlTLQMGhO6+QYPK+dWv4Ac/gJ//HM49N+kqRUTSI86LefsAM4Ax0UOLgenu/k7e\nVeYhzSHVkmXLQkCdeipcfz107550RSIiyYstpJJSqCEFsHUrXHwxrFsXlk8aMiTpikREktXhIWVm\nP3P3y83sfqDRC919Yv5l5lFcAYcUhJXSf/5zmDkTfvUrmBTr5KiISLrFEVKj3H2pmZ3c1PPu/lie\nNeal0EMq45ln4POfDxsm/vjHsEfJXw4tIqWow6+Tcvel0d1j3P2x7BtwTFsLLTXHHhsWo127Fk46\nCaqrk65IRKQw5NqCfmETj03twDqKXt++cN99oaHi2GPh/vuTrkhEJP1am+47D5gMnAQ8nvXU3kCt\nu4+Ltbgime5r6Kmn4AtfCLf/+A/o1i3pikRE4hfHOalyYDDwI+DqrKe2ASvcfXcrBR0E3AbsT9gs\n8WZ3vzFqaZ8PlAPrgXPdvdFWIMUaUhB2973ggtAFOH8+HHRQ0hWJiMQrlhZ0M+sCPOLuY9tQ0AHA\nAe7+fLQ47VLgTOAi4C13/7GZfRvo4+5XN3F80YYUQG0tXHst3HADzJkDEyYkXZGISHxiWWDW3T8E\nas1s33wLcvc33P356P57wEvAQYSgujV62a3AWfm+dzEoK4PvfAfuvBO+/GX43vdgd4tjUxGR0pLr\nihP3ASOAhWStrOrul+b8QWaDgCrgE8Br7t4n67m33b1vE8cU9Ugq2+bNMGUK7NoF8+bBgAFJVyQi\n0rHaMpLKddPDP0S3Nomm+u4CLnP398ysYfI0m0TTp0//6H5FRQUVFRVtLSPV9tsPHnwwNFKMHg23\n3w7jYm1LERGJV1VVFVVVVe16j5yXRTKzPYDDoy9fdvddOR7XFVgA/Mndb4geewmocPea6LzVInc/\nqoljS2Ykle3Pf4bzz4evfhW+/33o0iXpikRE2i/OTQ8rgNXATcAvgFfM7NM5fsZ/Ay9mAiryR+qu\ns7oQuC/H9yoJ48aFi3+rqkIzRU1N0hWJiCQj13NSS4HJ7v5y9PXhwG/dfVQrx51IuL5qJWFKz4Hv\nAs8AdwIHA9WEFvR3mzi+JEdSGbt3w4wZMHs2zJ0LJze5OJWISGGIc6uOFe4+vLXHOlqph1TGgw/C\n1Klw6aVw9dWhK1BEpNDEGVL/TbgY947ooSlAF3f/Ut5V5kEhVef118MKFXvvHZoq+vVLuiIRkfzE\ndk4K+BrwInBpdHsR+N/5lSftcdBBsGgRDB8OI0fCk08mXZGISPxyHUld1qDxocnHOppGUk1bsCBs\nqPitb8FVV4Hl9XuJiEgy4pzuW+buIxs8ttzdR+RZY14UUs2rrg57VO23X1hSqW+jS6FFRNKlw6f7\nzOy8aFfewWb2x6xbFfB2O2qVdiovh8cfh6FDw/TfkiVJVyQi0vFiXQW93cVpJJWTe+4JF/5+73uh\nA1DTfyKSRnFO9/UC3nf32ugaqSMJK0jktOpEWymkcrd2bdhQsbwcbrkFevdOuiIRkfri7O57HNjT\nzAYCDwPnA3PyK0/iNGRI6Pg78EAYNQqWLUu6IhGR9surccLMpgE9on2gnnf3Y2ItTiOpNrnzTvj6\n18NqFRMmVPODH8xh48ZaBg4sY+bMqQweXJ50iSJSguKc7lsO/CvwU+Bid19lZivd/ei2lZpjcQqp\nNlu9GiZOrOa112axffsMoBewnaFDK1m4cJqCSkQ6XZzTfZcD3wHuiQJqCLAo3wKl8xx2GHzyk3Oy\nAgqgF2vWzOCaa+YkWJmISO5y2k/K3R8DHsv6ei1h5QlJsZqaWuoCKqMXq1fXJlGOiEjeWgwpM/uZ\nu18eXSvVaN7N3SfGVpm028CBZYSNlLODajsrVpRx/PFw0UXhgmB1AopIWrV2ndQod19qZk1uEhGN\nsGKjc1Lts25dNePHz2LNmvrnpP70p2msXl3O7NmwcCGcfnoIrM98Rhssikh8YmuciN68P4C7b2lD\nbW2ikGq/deuqueaaOWzaVMuAAY27+956C37727Bn1ZYtcOGF4XboocnVLCLFKZaQMrPpwDcITRYG\n7AZmufu/t7HO3ItTSHWqFSvqNlg88sgwujrnnLA9iIhIe3V4SJnZlcBpwCXuvi56bAjwf4EH3f2n\n7ai39eIUUon44AN44IEQWI89BmedFQLrpJO04aKItF0cIbUcGO/ubzZ4vD/wsFZBL341NWFkNXs2\n7NhRNx1YrsusRCRPcVwn1a1hQMFH56W65fNBUpj23x+uvDJMBc6fD5s3h2WXTjkF7rgjBJeISFxa\nC6kP2vicFBkzGD0afv7zsJX9JZfAvHlhx+BLLoGnngINekWko7U23fch4UKbRk8Be7p7rKMpTfel\n38aNcPvtYTrQDKZOhfPPh4EDk65MRNIm1hb0JCikCoc7/OUvIazuuguOOy40W0ycCHvumXR1IpIG\nCilJhR07wkaMs2fD88+HVS0uuiicy9KGjCKlSyElqVNdDbfdBnPmQM+eIaymTAkNGSJSWhRSklq1\ntbB4cRhd3XsvnHxyCKzTT4eNG8OqGNrzSqS4KaSkIGzbFs5bzZ4NL7xQTW3tLLZu1Z5XIsVOISUF\nZ+LEGdx//zdpuFL75MnXMXduZVJliUgM4tz0UCQW27Y1vefVH/5Qy7e+BU8+GaYKRaQ0KaQkUXV7\nXmXbztixZfTsCV/7GgwYEC4Y/tOfYOfOJKoUkaRouk8S1dyeV9nnpNasCc0W99wDL7wAEybApElw\n2mmwzz6Jli8ieUjdOSkzuwU4A6hx9+HRY5XAV4DN0cu+6+4PNnO8QqoEtLbnVbaaGvjjH0NoLV4M\nY8aEVdrPPFNt7SJpl8aQGgO8B9zWIKS2ufv1ORyvkJJm/eMf8OCDYYT14IPw8Y+HEdakSTB0aNLV\niUhDbQmprnEVA+DuT5hZU78Sa90Babd99oFzzw23nTth0aIQWCeeCP37h7A66ywYMUIrXYgUqtjP\nSUUhdX+DkdRUYCvwHHCVu29t5liNpCRvtbVhHcHMeawPPghhNWlSmB7sGuuvZiLSnNRN90GTIdUf\neNPd3cx+CBzo7hc3c6xCStrFHV58MYTVvfeGZZrOOCME1vjx0KNH0hWKlI6CCKlcn4ue98rKugs6\nKyoqqKioiKtUKQEbNoSwuvdeWLo0bN541lkhuPr0Sbo6keJSVVVFVVXVR1/PmDEjlSE1iBBER0df\nH+Dub0T3rwA+5e6TmzlWIymJzVtvwYIFYZT16KNw7LFhhHXmmWEzRxHpWKkbSZnZPKAC+BhQA1QC\nY4FjgFpgPfBVd69p5niFlHSK7dvh4YfDCGvBgtAdmDmPddRR4TWZVnkthCvSNqkLqfZSSEkSdu2C\nxx+vmxbs1QsqKqpZsGAWGzdqIVyRtlJIiXQw93DuaurUGaxa1Xgh3HHjrmPOnEoGDIAyLTIm0qLU\nXSclUujMYPRo6N+/6YVwn3uultGjw/YjQ4fCYYfV3Q4/PPy5//66TkukrRRSIjmoWwi3/kjqjDPK\nuOOOsPrFq6/C6tXh9vjjcMst4f7//E/98Mq+9eunABNpiab7RHKQy0K4zXnnnbrwanhzbz7A+vbt\nlG9NpNPonJRIjPJZCDcX7qENvrkA69at/rRh9q251d/VgShpppASKRLusHlzCKtXXqkfXq++Cnvt\n1Ti4evasZtq0Waxbpw5ESSeFlEgJcIdNmxqPvBYtmsHWrY07EKdMuY477qhs7u1EOo26+0RKgBkM\nHBhu2auEjR1bS1VV4w7Ehx+u5Z57wiaRe+7ZmZWKtJ+u7BApEnUdiNm2M3hwGTfeCAMGwJe/HJaA\n+vDDJCoUyZ9CSqRIzJw5laFDK6kLqnBO6ne/m8qiRbBiBRx5JHzzm3DwwXDlleFCZc2oS5rpnJRI\nEcm1A/Gll+C3v4W5c8P+WpMnh9thh3V+zVI61DghInlxh2eegXnzYP58OOSQEFaf/zwceGDS1Umx\nUUiJSJvt3h3OV82bB/fdB6NGhcA6+2zYd9+kq5NioJASkQ7x/vthy5J580JwnXJKCKx//md1CErb\nKaREpMO98w784Q8hsJYvD/tsTZ4MY8dCly5JVyeFRCElIrHauDGcu5o3L9z/whdCYI0erYVypXUK\nKRHpNC+/HMJq3rzwdaZD8Igjkq1L0kshJSKdzh2eey60s8+fH1bCyHQIDhyYdHWSJgopEUnUhx/C\nokVhdHXvvTBiRF2HYO/eda/Tau2lSSElIqnx/vvwwAMhsB55BMaNC4E1bFg1n/1s2/bmksKmkBKR\nVHr33boOwccfn8GuXVqtvRS1JaS0dp+IxK53b/jSl8KIavToWuoHFEAvliypZfFi2LYtiQolrbRV\nh4h0qiFDynj66e00HEmZlfFv/wYrV4aGi5Ej624jRkDfvklVLEnSdJ+IdKp166oZP775c1K7d4f2\n9mXL6m7Ll4eQyg6ukSPhgAOS/m4kHzonJSIFIdfV2jNqa2HNmvrBtWwZdO/eOLgOPlgXFqeVQkpE\nSoY7bNhQP7SWLg0L5TYMriFDoExn4BOnkBKRkvf3vzcecb37bjivlR1cRxzR8tqDupar4ymkRESa\n8NZb4bxWdnBt2gTDh9dvzhg2DPbYo/XzZtI2CikRkRxt3Qp//Wv94Fq7Fo46CrZuncGaNbqWq6O1\nJaTUgi4iJWnffeHTnw63jB07YMUKuOCCpq/lWriwlquvhkGD6m7l5dCjR6eVXXJiDSkzuwU4A6hx\n9+HRY32A+UA5sB441923xlmHiEguevaE44+HY48tY/XqxtdyHXpoGfvsE0Zdd98N69fDa69Bnz51\noTV4cP0QO+QQbRTZHrFO95nZGOA94LaskLoWeMvdf2xm3wb6uPvVzRyv6T4R6XT5nJOqrQ3NGuvW\nhdDK3DJfv/469OtXP7iyw+zgg0MrfSlI5TkpMysH7s8Kqb8BJ7t7jZkdAFS5+5HNHKuQEpFE5Hst\nV3M+/DA0aWQHV3aQbdwI++3XeASWCbKDD4Zu3VqvsxC6EAslpN52975Zz9f7usGxCikRKWq7d4eg\najgCy9w2bQorazQ1ndilSzUXXTSLtWsLowuxUEPqLXf/WDPHKqREpKTt2hWmDJuaSly6dAY7djTu\nQjzvvOuYNy99XYiF0t1XY2b7Z033bW7pxdOnT//ofkVFBRUVFfFWJyKSIt26hdHT4MGNnxs7tpaq\nqsZdiHfeWcvbb8OYMeF23HHJdCBWVVVRVVXVrvfojJHUIMJI6ujo62uBt939WjVOiIi03Re/OIO5\ncxuPpM4++zrOP7+SxYvhiSfCyvLDh8NJJ4XQOvFE+FiT81fxSt10n5nNAyqAjwE1QCVwL/B74GCg\nmtCC/m4zxyukRESakWsX4o4dsGRJCKzFi+EvfwkNGWPG1AVXeXn8C/OmLqTaSyElItKytnQh7t4d\nVtt44om64OrWrX5ofeITHb8or0JKRETy5h62QslMDy5eDFu2wAkn1AXX6NHtvyhZISUiIh2ipqZu\npPXEE/DSS2ER3kxonXAC9O6d33sqpEREJBbbtoXzWpnR1jPPhI7DzPTgSSfBQQc1fWxmSnLu3OkK\nKRERid+uXWH7k8z04BNPQK9edaE1ZkxYUb66Oru5Yy+FlIiIdD53ePnl+qH17rvQo8cMNm7MtMlr\nuk9ERFJi0yYYP76SF1+cET2Sf0h1cIOhiIhIMGAAjBhRBmxv83sopEREJDYzZ05l6NBK2hpUCikR\nEYnN4MHlLFw4jSlTrmvT8TonJSIinaIt10lpJCUiIqmlkBIRkdRSSImISGoppEREJLUUUiIikloK\nKRERSS2FlIiIpJZCSkREUkshJSIiqaWQEhGR1FJIiYhIaimkREQktRRSIiKSWgopERFJLYWUiIik\nlkJKRERSSyElIiKppZASEZHUUkiJiEhqKaRERCS1uib1wWa2HtgK1AK73P3YpGoREZF0SnIkVQtU\nuPuIQg6oqqqqpEvISSHUWQg1QmHUWQg1QmHUWQg1QuHUma8kQ8oS/vwOUSj/YxRCnYVQIxRGnYVQ\nIxRGnYVQIxROnflKMiQcWGhmz5rZVxKsQ0REUiqxc1LAie7+dzPrTwirl9z9iQTrERGRlDF3T7oG\nzKwS2Obu1zd4PPniRESkw7i75fP6REZSZtYTKHP398ysF/C/gBkNX5fvNyMiIsUlqem+/YF7opFS\nV2Cuuz+cUC0iIpJSqZjuExERaUoqW8DN7BYzqzGzFUnX0hwzO8jMHjWzVWa20swuTbqmhsysu5kt\nMbPlUY2VSdfUEjMrM7NlZvbHpGtpipmtN7O/Rn+fzyRdT3PMbF8z+72ZvRT9/3lc0jVlM7PDo7/D\nZdGfW9P47wfAzK4wsxfMbIWZzTWzPZKuqSEzuyz6952qn0NN/Rw3sz5m9rCZvWxmD5nZvq29TypD\nCpgNnJp0Ea3YDVzp7sOAfwK+bmZHJlxTPe6+Exjr7iOAY4DTzCzNF05fBryYdBEtKJQL0G8AHnD3\no4BPAi8lXE897v5K9Hc4EhgFbAfuSbisRsxsADANGOnuwwmnJr6QbFX1mdkw4GJgNOHf+BlmNiTZ\nqj7S1M/xq4FH3P0I4FHgO629SSpDKmpFfyfpOlri7m+4+/PR/fcIPwgGJltVY+6+I7rbnfCPLJXz\nu2Z2EHA68Juka2lB6i9AN7N9gJPcfTaAu+92938kXFZLTgHWuPtrSRfSjC5ALzPrCvQENiVcT0NH\nAUvcfae7fwg8Dnwu4ZqAZn+OnwncGt2/FTirtfdJ9T+4QmFmgwi/xSxJtpLGoim05cAbwEJ3fzbp\nmprxU+BbpDREI4VwAfpg4E0zmx1Np/3azHokXVQLPg/8NukimuLum4D/AjYAG4F33f2RZKtq5AXg\npGgarSfhF72DE66pJfu5ew2EX/SB/Vo7QCHVTma2F3AXcFk0okoVd6+NpvsOAo4zs48nXVNDZvbP\nQE00MrXolkYnRlNUpxOmd8ckXVATugIjgZuiWncQplhSx8y6AROB3yddS1PMrDfhN/9yYACwl5lN\nTraq+tw1+hl9AAAFRklEQVT9b8C1wELgAWA58GGiReWn1V9KFVLtEE0B3AXc7u73JV1PS6Ipn0XA\nhKRracKJwEQzW0v4rXqsmd2WcE2NuPvfoz+3EM6hpPG81OvAa+7+XPT1XYTQSqPTgKXR32canQKs\ndfe3o6m0PwAnJFxTI+4+291Hu3sF8C7wSsIltaTGzPYHMLMDgM2tHZDmkErzb9QZ/w286O43JF1I\nU8ysX6Z7JpryGQ/8LdmqGnP377r7Ie4+hHBi+lF3vyDpurKZWc9o1EzWBegvJFtVY9FUymtmdnj0\n0DjS24xyHimd6otsAI43sz3NzAh/l6lqQgGIlpbDzA4BJgHzkq2onoY/x/8ITI3uXwi0+st9kmv3\nNcvM5gEVwMfMbANQmTkRnBZmdiIwBVgZnfNx4Lvu/mCyldVzIHCrmZURfiGZ7+4PJFxToSqkC9Av\nBeZG02lrgYsSrqeR6PzJKcAlSdfSHHd/xszuIkyh7Yr+/HWyVTXpbjPrS6jxX9PSKNPUz3HgP4Hf\nm9mXgGrg3FbfRxfziohIWqV5uk9EREqcQkpERFJLISUiIqmlkBIRkdRSSImISGoppEREJLUUUlI0\nzKzWzH6S9fVVZvaDDnrv2WYW+8KdZnaOmb1oZn+Osy4zKzez8/KvUKRzKaSkmOwEPhdd2JgaZtYl\nj5dfDHzZ3cfFVU9kMJDXOnR5fh8iHUIhJcVkN2FFgCsbPtFwxGFm26I/TzazKjO718xeNbMfmdnk\naLPIv5rZ4Ky3GR+tgP63aFHczCrzP45e/3xmdfTofR83s/uAVU3Uc160kd4KM/tR9Ng1wBjgFjO7\ntoljvh29frmZ/Z8mnl+XCWgzG2Vmi7JqyWwyuDRa1ulHwJjoscty/T6i5aEWRO+3wsz+Jaf/MiJt\nlMplkUTayIGbCEtVNfoh38RrM4YDRxIW51wL3Ozux1nY5XQadaFX7u6fMrNDgUVmNpSw/ti70ev3\nAJ40s8xySSOAYe6+IfuDzexAwvIwI6LPXGhmE919ppl9hrCZ5vIGx0wAPgt8yt13Rit0t/Q9ZX99\nFWG5nKej5Yj+h7Ay+lXuPjF6/6/k8n1EQb/R3c+Ijtu7yb9dkQ6ikZQUlWi7lFsJu/zm6ll33+zu\nHwBrgMwP55XAoKzX3Rl9xqvR644kLDR7QbR+4xKgL3BY9PpnGgZU5FPAomh17VpgLvDprOebWlj5\nFGB2tNsy7v5uE69pbkHmJ4Gfmtk0oE/0mQ3l+n2sJIwof2RmY9x9WzOfKdIhFFJSjG4gnNvplfXY\nbqL/36MVrffIem5n1v3arK9rqT/bkD1SsehrA6ZF26GPcPehWRvjbW+hxjhW+P/oewT2zDzo7tcS\n/j56EEZIhzdxbE7fh7uvJmz9sRL4oZl9P4bvQ+QjCikpJgbg7u8QRj0XZz23Hhgd3T8T6NaG9/8X\nC4YSGg9eBh4C/tXC3mKY2WHRlFpLngE+bWZ9o2aE84CqVo5ZCFxk0S67ZtanidesA0ZF98/OPGhm\nQ9x9lbv/GHiWMALcBuyTdWxO30c0Vfm+u88DfkJ696qSIqFzUlJMskc6/wV8Peuxm4H7oumsh2h+\nlNPStgAbCAGzN/BVd//AzH5DmBJcFo3QNgNntVik+xtmdjV1wbTA3Re09Pnu/pCZfRJ4zsx2EnZh\n/X6D1/87oeliK/VD73IzG0vYsXUV8KfouA+jv4857n6DmeXyfRwN/MTMaoEPgK+19L2KtJe26hAR\nkdTSdJ+IiKSWQkpERFJLISUiIqmlkBIRkdRSSImISGoppEREJLUUUiIikloKKRERSa3/DwGIpt86\nLgDbAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xa003e48>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "distortions = []\n",
    "for i in range(1, 11):\n",
    "    km = KMeans(n_clusters=i, \n",
    "                init='k-means++', \n",
    "                n_init=10, \n",
    "                max_iter=300, \n",
    "                random_state=0)\n",
    "    km.fit(X)\n",
    "    distortions .append(km.inertia_)\n",
    "plt.plot(range(1,11), distortions , marker='o')\n",
    "plt.xlabel('Number of clusters')\n",
    "plt.ylabel('Distortion')\n",
    "plt.tight_layout()\n",
    "#plt.savefig('./figures/elbow.png', dpi=300)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quantifying the quality of clustering via silhouette plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAAEaCAYAAACrcqiAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEglJREFUeJzt3XuQZGV9xvHnWdblKsvNIgYIuxoVqADLxhgMJFyCiFQE\nFUhKiIkImEpSUYPlpTTJzipJxDLGayWlErL8EcFoSQFxAwR2oFjue4HlKikWREtSISASqCALv/zR\n7zhnh5npnu4+fd7T7/dT1TWne87l985Z5uGcfufXjggBAJCjRU0XAADAXAgpAEC2CCkAQLYIKQBA\ntggpAEC2FjddgCTZZoohAIyZiPCg+8jmSioiinusWrWq8RoYN2Nn3Iy9jsewZBNSAADMREgBALJF\nSDXo2GOPbbqERpQ6bqncsZc6bqnssQ+Dh3nvsO8i7MihDgDAcNhWjNPECQDoamKi6QowYlxJAWgP\nW+J3RStwJQUAGHuEFAAgW1l0nJAke3XTJQDIXIjfFU2LWDXS43ElBQDIFiEFoDUmdEzTJWDECCkA\nrbFaxzVdAkaMkAIAZIuQAgBki5ACAGSLkAIAZIuQAtAaq7Su6RIwYoQUgNaY0A1Nl4ARI6QAANki\npAAA2SKkAADZIqQAANkipAC0Br37ykNIAWgNeveVh5ACAGSLkAIAZIuQAgBki5ACAGSLkALQGvTu\nKw8hBaA16N1XHkIKAJAtQgoAkC1CCgCQLUIKAJAtQgpAa9C7rzyEFIDWoHdfeQgpAEC2CCkAQLYI\nKQBAtggpAEC2CCkArUHvvvIQUgBag9595Vlc585t7yjpRklL0rG+HRGr6zwmAGB81BpSEfG87eMi\n4jnbO0hab3ttRNxe53EBAOOh9tt9EfFcWtxRnVCMuo8JABgPtYeU7UW2N0l6XNK1EXFH3ccEAIyH\nWm/3SVJEvCTpCNu7S7rc9iERcd/L16zO2lkmaXndpQFoGXr35WtyclKTk5ND368jRnf3zfZfSno2\nIj4/4/WQJkZWBwCgPxGrelrPtiLCgx6v1tt9tvexvTQt7yzpLZIeqPOYAIDxUfftvldLWmN7kTqB\neFlEfK/mYwIAxkTdU9C3SFpZ5zEAAOOLjhMAgGwRUgBag9595SGkALQGvfvKQ0gBALJFSAEAskVI\nAQCyRUgBALJFSAFoDXr3lYeQAtAaq3Vc0yVgxAgpAEC2CCkAQLYIKQBAtggpAEC2CCkArUHvvvIQ\nUgBag9595SGkAADZIqQAANkipAAA2SKkAADZIqQAtAa9+8pDSAFoDXr3lYeQAgBki5ACAGSLkAIA\nZIuQAgBki5AC0Br07isPIQWgNejdVx5CCgCQLUIKAJAtQgoAkC1CCgCQLUIKQGvQu688hBSA1qB3\nX3kIKQBAthY3XQCA0YlY1XQJwIJwJQUAyBYhBQDIFiEFoD0mJpquACNGSAFoj9Wrm64AI0ZIAQCy\nRUgBALJFSAEAskVIAQCyRUgBaI9V/DFyaQgpAO3BFPTiEFIAgGwRUgCAbBFSAIBsEVIAgGwRUgDa\ng4kTxSGkALQHvfuKQ0gBALLVNaRs72D7z0dRDAAAVV1DKiJelPTuEdQCAMB2Fve43nrbX5F0maRn\np16MiI21VAUAgHoPqRXp66cqr4Wk44dbDgDMg959xXFENF2DbIc00XQZwNiL4Jc8RsO2IsKD7qen\n2X2297V9ke216fkhts8Z9OAAAMyn1yno/yzpakm/mJ5/X9KH6igIAIApvYbUPhHxLUkvSVJEbJP0\nYm1VAQCg3kPqWdt7qzNZQraPlPR0bVUBAKDeQ+p8SVdIeq3t9ZIukfSB2qoCgNnQu684Pc3us72j\nOrf33iDJkh6UtCginh9KEczuA0ai9bP7bCmDGcnobqSz+yTdEhHbIuLeiLgnIl6QdMugBwcAYD7z\n/jGv7V+QtJ+knW0foc5VlCTtLmmXmmsDABSuW8eJt0p6r6T9Jf2dpkPqGUmfqK8sAAC6hFRErJG0\nxvZpEfGdEdUEAICk3t+T2t/27u74hu2Ntk+stTIAmInefcXpNaTeFxE/lXSipL0lvUfSZ7ptZHt/\n29fbvtf2FttMWwfQP6agF6fXLuhT70WdLOmSiLjXdi9TC7dJOj8iNtveTdIG29dExAP9FAsAKEuv\nV1IbbF+jTkhdbfuVSi2S5hMRj0fE5rT8v5LuV2e2IAAAXfV6JXWOOp8p9XBEPJdaJJ29kAPZXpb2\ncdtCtgMAlKvXkDo6fT2st7t820u3+r4t6YPpimoW6yrLyyQtX/BxAADNmJyc1OTk5ND322tbpCsr\nT3eS9CZJGyKi6yfz2l4s6SpJayPii3OsQ1skYARa3xZpYoLJEy0xrLZIfX0yr+0DJH0hIk7rYd1L\nJD0REefPsw4hBYxA60OK3n2tMerefTP9UNLB3VayfZSksyQdb3tT+vuqk/o8JgCgMD29J2X7y0qf\nJaVOsK2QtLHbdhGxXtIOfVcHACharxMn7qwsb5P0zRRAAADUpqeQSj38AAAYqW4f1bFF07f5XiYi\nDht6RQAwF3r3FafbldS7JO0r6bEZrx8g6fFaKgKAuTD9vDjdZvf9vaSnI+LR6kPS0+l7AADUpltI\n7RsRW2a+mF5bVktFAAAk3UJqj3m+t/MwCwEAYKZuIXWn7fNmvmj7XEkb6ikJAICObhMnPiTpu7bP\n0nQovVHSEknvrLMwAHgZevcVp9cGs8dJ+pX09N6IuH6oRdC7DxgJevdhVIbVu6/XP+Zdp+0/SwMA\ngNr122AWAIDaEVIAgGwRUgCAbBFSANqD3n3FIaQAtAfTz4tDSAEAskVIAQCyRUgBALJFSAEAskVI\nAWgPJk4Uh5AC0B6rVzddAUaMkAIAZIuQAgBki5ACAGSLkAIAZIuQAtAe9O4rDiEFoD2Ygl4cQgoA\nkC1CCgCQLUIKAJAtQgoAkC1CCkB7MHGiOIQUgPagd19xCCkAQLYIKQBAtggpAEC2CCkAQLYIKQDt\nQe++4hBSANqDKejFIaQAANkipAAA2SKkAADZIqQAANkipAC0BxMnikNIAWgPevcVh5ACAGSLkAIA\nZIuQAgBki5ACAGSLkALQHvTuKw4hBaA9mIJeHEIKAJAtQgoAkC1CCgCQLUIKAJAtQgpAezBxojiE\nFID2oHdfcQgpAEC2CCkAQLYIKQBAtggpAEC2CCkA7UHvvuIQUgDagynoxSGkAADZIqQAANkipAAA\n2ao1pGxfZPu/bN9d53EAAOOp7iupiyW9teZjACgFEyeKU2tIRcRNkp6q8xgACkLvvuLwnhQAIFuL\nmy5g2rrK8jJJyxuqAwCwUJOTk5qcnBz6fh0RQ9/pdgewD5R0ZUQcNs86IU3UWgcAKaLlHRtsqebf\nWRgO24oID7qfUdzuc3oAALAgdU9B/xdJN0t6ve0f2D67zuMBGHP07itO7bf7eiqC233ASLT+dh9a\no023+wAA6AshBQDIFiEFAMgWIQUAyBYhBaA96N1XHEIKQHvQu684hBQAIFuEFAAgW4QUACBbhBQA\nIFuEFID2oHdfcQgpAO3BFPTiEFIAgGwRUgCAbBFSAIBsEVIAgGwRUgDag4kTxSGkALQHvfuKQ0gB\nALJFSAEAskVIAQCyRUgBALJFSAFoD3r3FYeQAtAeTEEvDiEFAMgWIQUAyBYhBQDIFiEFAMgWIQWg\nPZg4URxCCkB70LuvOIRUo7Y2XUBDSh23VOrYJycnmy6hMSWPfRgIqUY90nQBDXmk6QIa9EjTBTSi\n5F/UJY99GAgpAEC2CCkAQLYcEU3XINvNFwEAGKqI8KD7yCKkAACYDbf7AADZIqQAANkipAAA2ao1\npGyfZPsB29+3/bE51vmS7Ydsb7a9YiHb5qqPcR9Ref0R23fZ3mT79tFVPRzdxm77DbZvtv1/ts9f\nyLY5G3Dc437Oz0zju8v2TbYP63XbnA047nE/56dUx2f7qF63fZmIqOWhTgD+p6QDJb1C0mZJB81Y\n522S/i0t/7qkW3vdNtfHIONOzx+WtGfT46hx7PtI+lVJn5Z0/kK2zfUxyLgLOedHSlqalk8q6L/z\nWcddyDnfpbJ8qKT7+z3ndV5JvUnSQxHxaES8IOlSSafOWOdUSZdIUkTcJmmp7X173DZXg4xbkqz2\n3obtOvaIeCIiNkjattBtMzbIuKXxP+e3RsTT6emtkvbrdduMDTJuafzP+XOVp7tJeqnXbWeq84e0\nn6THKs9/qO1P0nzr9LJtrvoZ948q64Ska23fYfu82qqsxyDnbdzP+XxKOufnSlrb57Y5GWTcUgHn\n3PY7bN8v6UpJ71vItlWLByp1+Ab+w68xcFRE/Nj2q9T5R3x/RNzUdFGoVRHn3PZxks6WdHTTtYzS\nHOMe+3MeEZdLutz20ZIukPSWfvZT55XUjyT9UuX5/um1mescMMs6vWybq0HGrYj4cfr635K+q87l\ncVsMct7G/ZzPqYRzniYNfE3SKRHx1EK2zdQg4y7inE9J4fsa23stdNupHdT15toOmn6DbIk6b5Ad\nPGOdkzU9geBITb+h2nXbXB8DjnsXSbul5V0lrZd0YtNjGubYK+uukvThfrbN7THguMf+nKdfSg9J\nOrLfn1tujwHHXcI5f21leaWkx/o953UP5iRJD6YT9fH02h9Jen9lna+kou+StHK+bdvy6Hfckpan\nk7ZJ0pa2jbuXsUvaV5170j+R9KSkH1T+gx3bcz7XuAs551+X9D+SNqZx3j7ftm159DvuQs75RyXd\nk8a+XtKb+z3n9O4DAGSrrVMgAQAFIKQAANkipAAA2SKkAADZIqQAANkipAAA2SKkkDXbn7R9T2r7\nv9H2r6XXv277oLS81fZetg+0vaXmeg60/e7K88Ntv63OY85Tyz62b7W9wfZRtk+3fZ/t62yvtP2F\nLttfZXv3Po996tTPH6hTbr37gJ+zfaQ63TlWRMS21FZliSRFRLUpZ8yxXIflks6U9M30fIWkN2r7\n5qGjcoKkuyPi/ZJke62kcyPi5vT9jfNtHBG/M8Cx3yHpKkkPDLAPoCuupJCzV0t6IiK2SVJEPBkR\nj0uS7XW2V6b1qo2JF9v+Wrr6+nfbO6b1D7d9izsfMvkd20tn7sf23ra3puVFtj9r+7a0zVQo/q2k\no9NV3UclfUrS76bnZ9jexfZFlSuct882MNsfs313+lC4v0mvrZijxtfYXps6Zt9g+/W2D5d0oaRT\n07H/Sp0GphfZvtD2MbavTNvvavuf0vE2235nen1rCn7ZPiuNdaPtf7Dt9Pozti9I291s+1W23yzp\nFEmfTesvH+AcA/Nrur0GDx5zPdTpa7ZJnf9b/6qk36p8b52m20ltlbSXOv3AXpB0aHr9MklnpuW7\nJB2dlldL+vws+9lb0sNp+TxJn0jLSyTdkfZ/jKQrKnX8oaQvVZ7/deWYS9Vp/7LzjHGdJOkmSTum\n53t0qfE/lHqhqdOI9Lo5jn29pCPS8s/rlPSZqX1N1ZW+Ppx+bgdJukLSDun1r0r6/bT8kqST0/KF\nlZ/JxZLe1fS/ER7j/+B2H7IVEc+mq5zflHS8pEttfzwiLplns4cjYup9qQ2SlqX3XZbG9EchrJH0\nrS6HP1HSobbPSM93l/Q6dUKw23Zvt/2R9HyJOo1GH6ysc4KkiyPi+TTOn8xVo+1dJf2GpH+durpR\n5xNNZ2PN/nE3J0j6vaknMf1BfFN+W50moHekY+wk6fH0vZ9FxPfS8oa0L2BkCClkLSJC0o2SbkyT\nIv5A6VON5/B8ZflFdX7hSnN/Vtk2Td/23qnyuiX9WURcW13Z9jE9lH1aRDzUw3q9WCTpqYhY2XXN\n/lnSmoj45Czf+1ll+UXxOwMjxntSyFZ67+WXKy+tkPRot81mvhARP5X0pO2j0kvvkXRDWn5EnYkP\nknRGZbOrJf2J7cWpltfZ3lnSM5JeWVnvGXWusqrbfaAyhhWz1HitpLPT/mR7z1TjUzNrjIhnJG21\nfXpln4fNMfa5XCvpTyvb7zG1mL5eJ+l0dz6AT7b3tH3AjHVmmjluoBaEFHK2m6Q1aRLEZkkHS5pI\n35trRt9cs/veK+lzaT+HqzPhQZI+J+mPbW9Q5/2ZKd+QdJ+kjekK7h/VuYq4W9JLacLDB9V5T+uQ\nqYkTkj4t6RVpksKWynGmC4y4Wp33gO60vVHSh7vUeJakc9LkhXvUmbQwm7nGfoGkvWxvsb1J0rHV\n9SPifkl/Ieka23dJukadSSvz7fNSSR9Jk0OYOIHa8FEdAIBscSUFAMgWIQUAyBYhBQDIFiEFAMgW\nIQUAyBYhBQDIFiEFAMjW/wMqtdc+3C2eIgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xc994588>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from matplotlib import cm\n",
    "from sklearn.metrics import silhouette_samples\n",
    "\n",
    "km = KMeans(n_clusters=3, \n",
    "            init='k-means++', \n",
    "            n_init=10, \n",
    "            max_iter=300,\n",
    "            tol=1e-04,\n",
    "            random_state=0)\n",
    "y_km = km.fit_predict(X)\n",
    "\n",
    "cluster_labels = np.unique(y_km)\n",
    "n_clusters = cluster_labels.shape[0]\n",
    "silhouette_vals = silhouette_samples(X, y_km, metric='euclidean')\n",
    "y_ax_lower, y_ax_upper = 0, 0\n",
    "yticks = []\n",
    "for i, c in enumerate(cluster_labels):\n",
    "    c_silhouette_vals = silhouette_vals[y_km == c]\n",
    "    c_silhouette_vals.sort()\n",
    "    y_ax_upper += len(c_silhouette_vals)\n",
    "    color = cm.jet(i / n_clusters)\n",
    "    plt.barh(range(y_ax_lower, y_ax_upper), c_silhouette_vals, height=1.0, \n",
    "            edgecolor='none', color=color)\n",
    "\n",
    "    yticks.append((y_ax_lower + y_ax_upper) / 2)\n",
    "    y_ax_lower += len(c_silhouette_vals)\n",
    "    \n",
    "silhouette_avg = np.mean(silhouette_vals)\n",
    "plt.axvline(silhouette_avg, color=\"red\", linestyle=\"--\") \n",
    "\n",
    "plt.yticks(yticks, cluster_labels + 1)\n",
    "plt.ylabel('Cluster')\n",
    "plt.xlabel('Silhouette coefficient')\n",
    "\n",
    "plt.tight_layout()\n",
    "# plt.savefig('./figures/silhouette.png', dpi=300)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our clustering with 3 centroids is good.\n",
    "### Bad Clustering:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAAEaCAYAAACrcqiAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEptJREFUeJzt3XmQLWddxvHvk4RAWBLWikCABAQBJdxExIREkyBLQNlB\nZZWdUktAKJYC9d4AIlAIilJarCZ/yKIIBQhChAxU2MnNBkkAK2EtQomEEEMJXPLzj9PDPUzuzJx7\nZnrO2z3fT1XX9Onp5X2nM/dJd7/z61QVkiS16IBFN0CSpNUYUpKkZhlSkqRmGVKSpGYZUpKkZh20\n6AYAJHGIoSSNTFVlo/to5kqqqrbdtHPnzoW3wX7bd/tt3/uYNkszISVJ0kqGlCSpWYbUAp188smL\nbsJCbNd+w/bt+3btN2zvvm+GbOa9w7kbkVQL7ZAkbY4k1JgGTkjqya5di26BNDevpKSxS8DfL20x\nr6QkSaNnSEmSmtVExQmA5LRFN0EapcLfL22eqp1bejyvpCRJzTKkpJHbxUmLboI0N0NKGrnTOGXR\nTZDmZkhJkpplSEmSmmVISZKaZUhJkpplSEkjt5OzFt0EaW6GlDRyu/jYopsgzc2QkiQ1y5CSJDXL\nkJIkNcuQkiQ1y5CSRs7afRoyQ0oaOWv3acgMKUlSswwpSVKzDClJUrMMKUlSswwpaeSs3achM6Sk\nkbN2n4bMkJIkNcuQkiQ1y5CSJDXLkJIkNcuQkkbO2n0aMkNKGjlr92nIDClJUrMMKUlSswwpSVKz\nDClJUrMMKWnkrN2nITOkpJGzdp+GbEtCKskBSXYnee9WHE+SNA5bdSX1LOCiLTqWJGkkeg+pJEcA\nDwTe1PexJEnjshVXUq8FngfUFhxLkjQiB/W58yS/DXynqs5LcjKQ1deeHoF0JHBUn02Ttg1r911b\n1c5FN2F0lpaWWFpa2vT9pqq/C5wkLwceB+wBDgFuBPxbVT1hxXoFu3prhyRNM6T6l4SqWuPCZDa9\n3u6rqhdV1W2r6vbA7wMfXRlQkiStxr+TkiQ1q9dnUtOq6mPgXxVKkmbnlZQkqVmGlDRy1u7TkBlS\n0shZu09DZkhJkpplSEmSmmVISZKaZUhJkpplSEkjZ+0+DZkhJY3caZyy6CZIczOkJEnNMqQkSc0y\npCRJzTKkJEnNMqSkkbN2n4bMkJJGztp9GjJDSpLULENKktQsQ0qS1CxDSpLULENKGjlr92nIDClp\n5KzdpyEzpCRJzTKkJEnNMqQkSc0ypCRJzTKkpJGzdp+GzJCSRs7afRoyQ0qS1CxDSpLULENKktQs\nQ0qS1CxDSho5a/dpyAwpaeSs3achM6QkSc06aNENkNSGqp2LboJ0LV5JSZKaZUhJkpplSEkjZ+0+\nDZkhJY2ctfs0ZIaUJKlZhpQkqVmGlCSpWYaUJKlZhpQ0ctbu05AZUtLIWbtPQ2ZISZKaZUhJkppl\nSEmSmmVISZKa1WtIJbluks8kOTfJhUl8F4C0xazdpyHrNaSq6kfAKVV1DLADeECSe/Z5TEk/z9p9\nGrLeb/dV1Q+72esyecli9X1MSdI4rBtSSQ5M8qfzHiDJAUnOBS4Hzqyqz827L0nS9rLu6+Or6qdJ\nHg28dp4DVNU1wDFJDgXek+SuVXXRtdecvm9+JHDUPIeTJC3A0tISS0tLm77fVK1/9y3Ja4HrAO8A\nrl5eXlW79+tgyZ8DV1fVa1YsL9i1P7uSNKNiF5nh96vKcU3aPEmoqmx0P+teSXV2dF9fMrWsgHuv\ntVGSmwM/qaorkxwC3Bd4xX63UtLcrN2nIZsppKpq3uJftwROT3IAk+df76iqD8y5L0lzsHafhmym\nkEpyOPBy4FZV9YAkdwWOr6o3r7VdVV0IHLvxZkqStqNZh6D/E/Ah4Fbd5y8Dz+6jQZIkLZs1pG5e\nVe8ErgGoqj3AT3trlSRJzB5SVye5Gd0f4iY5Driyt1ZJksTsIfUc4L3AHZJ8AjgDeGZvrZK0aazd\npyGbNaS+CJwE3At4BvDLwCV9NUrS5rF2n4Zs1pD6VFXtqaovVtUXquonwKf6bJgkSWsOQU/yC8Ct\ngUOSHAMs//XwocD1e26bJGmbW+/vpO4PPBE4Avhr9obUVcCL+muWJEnrhFRVnc6kYsQjqupdW9Qm\nSZKA2Z9JHZHk0Ey8KcnuJPfrtWWSNoW1+zRks4bUk6vqB8D9gJsBj8dCsdIgWLtPQzZrSC0/i3og\ncEZVfXFqmSRJvZg1pM5J8mEmIfWhJDeiK5EkSVJfZn2f1FOYvFPq0qr6YVci6Un9NUuSpNlD6sTu\n69GJd/kkSVtj1pB63tT89YB7Auewzpt5JS3eTs5y8IQGa6ZnUlX1oKnpvsCvAFf02zRJm8HafRqy\nWQdOrPRN4C6b2RBJklaa9fXxf0f3LikmwbYD2N1XoyRJgtmfSX1+an4P8Laq+kQP7ZEk6WdmCqmu\nhp8kSVtqvVd1XMje23zXUlVHb3qLJG0qa/dpyNa7kno4cDjwjRXLbwNc3kuLJG0qh59ryNYb3fda\n4Mqq+tr0BFzZfU+SpN6sF1KHV9WFKxd2y47spUWSJHXWC6kbr/G9QzazIZIkrbReSH0+ydNWLkzy\nVCZlkSRJ6s16AyeeDbw7yWPZG0r3AA4GHtZnwyRtDmv3acjWvJKqqu9U1b2A04CvdtNpVXV8VTm6\nTxoAa/dpyGb9Y96zgLP2d+dJjgDOYDKM/RrgjVX1uv3djyRpe5q1LNK89gDPqarzktyQ7g2/VXVJ\nz8eVJI3AvFXQZ1JVl1fVed38/wIXA7fu85iSpPHoNaSmJTmSSfX0z2zVMSVJw9b37T4Ault9/wo8\nq7ui2ofpR15HAkf13i5pO7B2n7bC0tISS0tLm77fVK1aP3ZzDpAcBLwf+GBV/e0q6xTs6rUdktZW\ntXPRTdCIJKGqstH9bMXtvrcAF60WUJIkrabXkEpyAvBY4N5Jzk2yO8mpfR5TkjQevT6T6t7ee2Cf\nx5AkjdeWje6TJGl/GVLSyO3c/2IxUjMMKWnkrN2nITOkJEnNMqQkSc0ypCRJzTKkJEnNMqSkkbN2\nn4bMkJJGzlfHa8gMKUlSswwpSVKzDClJUrMMKUlSswwpaeSs3ach6/3NvDM1wjfzLpxvZR2xBBr4\nPdf2MqQ380qSNBdDSpLULENKktQsQ0qS1CxDShq7nQ6K0XAZUtLY7dq16BZIczOkJEnNMqQkSc0y\npCRJzTKkJEnNMqSksXPghAbMkJLG7rTTFt0CaW6GlCSpWYaUJKlZhpQkqVmGlCSpWYaUNHbW7tOA\nGVLS2DkEXQNmSEmSmmVISZKaZUhJkpplSEmSmmVISWPnwAkNmCEljZ21+zRghpQkqVmGlCSpWYaU\nJKlZhpQkqVmGlDR21u7TgBlS0tg5BF0DZkhJkpplSEmSmmVISZKa1WtIJXlzku8kuaDP40iSxqnv\nK6m3Avfv+RiS1uLACQ1YryFVVWcDV/R5DEnrsHafBsxnUpKkZh206AbsddbU/JHAUQtqhyRpfy0t\nLbG0tLTp+01VbfpOf+4Aye2A91XV0WusU7Cr13ZobVVWJRitBHr+PZdWSkJVZaP72YrbfekmSZL2\nS99D0P8Z+CRwpyRfT/KkPo8naR+s3acB6/1230yN8Hbfwnm7T9JmGtLtPkmS5mJISZKaZUhJkppl\nSEmSmmVISWNn7T4NmCEljZ21+zRghpQkqVmGlCSpWYaUJKlZhpQkqVmGlDR21u7TgBlS0tg5BF0D\nZkhJkpplSEmSmmVISZKaZUhJkpplSElj58AJDZghJY2dtfs0YIaUJKlZhpQkqVmGlCSpWYaUJKlZ\nhpQ0dtbu04AZUtLYOQRdA2ZISZKaZUhJkpplSEmSmmVISZKaZUhJY+fACQ2YISWNnbX7NGCG1EJd\ntugGLMTS0tKim7Aw27Xv27XfsL37vhkMqYX66qIbsBDb+Zd2u/Z9u/YbtnffN4MhJUlqliElSWpW\nqmrRbSDJ4hshSdpUVZWN7qOJkJIkaV+83SdJapYhJUlqliElSWpWryGV5NQklyT5cpIXrLLO65J8\nJcl5SXbsz7atmqPfx0wt/2qS85Ocm+SzW9fqzbFe35P8UpJPJvm/JM/Zn21btsF+j/2cP6br3/lJ\nzk5y9KzbtmyD/R77OX/wdP+SnDDrttdSVb1MTALwv4DbAdcBzgPuvGKdBwD/3s3/OvDpWbdtddpI\nv7vPlwI3WXQ/euz7zYFfBV4KPGd/tm112ki/t8k5Pw44rJs/dRv9nu+z39vknF9/av5uwMXznvM+\nr6TuCXylqr5WVT8B3g48ZMU6DwHOAKiqzwCHJTl8xm1btZF+A4Th3oZdt+9V9d2qOgfYs7/bNmwj\n/Ybxn/NPV9WV3cdPA7eedduGbaTfMP5z/sOpjzcErpl125X6/CHdGvjG1Odv8vMnaa11Ztm2VfP0\n+1tT6xRwZpLPJXlab63sx0bO29jP+Vq20zl/KvDBObdtyUb6DdvgnCd5aJKLgfcBT96fbacdtKGm\nbr4N/+HXCJxQVd9Ocgsm/xFfXFVnL7pR6tW2OOdJTgGeBJy46LZspVX6PfpzXlXvAd6T5ETgZcB9\n59lPn1dS3wJuO/X5iG7ZynVus491Ztm2VRvpN1X17e7rfwPvZnJ5PBQbOW9jP+er2g7nvBs08Abg\nwVV1xf5s26iN9HtbnPNlXfjePslN93fb5R309XDtQPY+IDuYyQOyu6xY54HsHUBwHHsfqK67bavT\nBvt9feCG3fwNgE8A91t0nzaz71Pr7gSeO8+2rU0b7Pfoz3n3j9JXgOPm/bm1Nm2w39vhnN9hav5Y\n4BvznvO+O3Mq8KXuRL2wW/YM4OlT6/x91+jzgWPX2nYo07z9Bo7qTtq5wIVD6/csfQcOZ3JP+vvA\n94CvT/3Cjvacr9bvbXLO3wj8D7C76+dn19p2KNO8/d4m5/z5wBe6vn8COH7ec27tPklSs4Y6BFKS\ntA0YUpKkZhlSkqRmGVKSpGYZUpKkZhlSkqRmGVJqWpIXJ/lCV/Z/d5Jf65a/Mcmdu/nLktw0ye2S\nXNhze26X5NFTn++e5AF9HnONttw8yaeTnJPkhCSPTHJRko8kOTbJ36yz/fuTHDrnsR+y/POX+tRa\n7T7pZ5Icx6Q6x46q2tOVVTkYoKqmi3LWKvN9OAp4DPC27vMO4B78fPHQrXIf4IKqejpAkg8CT62q\nT3bf373WxlX1Oxs49kOB9wOXbGAf0rq8klLLbgl8t6r2AFTV96rqcoAkZyU5tltvujDxQUne0F19\n/UeS63br3z3JpzJ5yeS7khy2cj9Jbpbksm7+gCSvSvKZbpvlUPwr4MTuqu75wEuA3+0+PyrJ9ZO8\neeoK50H76liSFyS5oHsp3Mu7ZTtWaePtk3ywq5j9sSR3SnJ34JXAQ7pj/wWTAqZvTvLKJCcleV+3\n/Q2SvKU73nlJHtYtv6wLfpI8tuvr7iT/kCTd8quSvKzb7pNJbpHkeODBwKu69Y/awDmW1rbo8hpO\nTqtNTOqancvk/9ZfD/zm1PfOYm85qcuAmzKpB/YT4G7d8ncAj+nmzwdO7OZPA16zj/3cDLi0m38a\n8KJu/mDgc93+TwLeO9WOPwBeN/X5L6eOeRiT8i+HrOjXqcDZwHW7zzdep43/SVcLjUkh0o+scuyP\nAsd08z9rJ/CK5X0tt6v7emn3c7sz8F7gwG7564HHdfPXAA/s5l859TN5K/DwRf834jT+ydt9alZV\nXd1d5fwGcG/g7UleWFVnrLHZpVW1/FzqHODI7rnLYbX3VQinA+9c5/D3A+6W5FHd50OBOzIJwfW2\ne1CS53WfD2ZSaPRLU+vcB3hrVf2o6+f3V2tjkhsA9wL+ZfnqhskbTfcl7Pt1N/cBfm/5Q+19Ed+y\n32JSBPRz3TGuB1zefe/HVfWBbv6cbl/SljGk1LSqKuDjwMe7QRFPoHur8Sp+NDX/Uyb/4MLq7yrb\nw97b3tebWh7gT6rqzOmVk5w0Q7MfUVVfmWG9WRwAXFFVx6675vwCnF5VL97H9348Nf9T/DdDW8xn\nUmpW9+zlF6cW7QC+tt5mKxdU1Q+A7yU5oVv0eOBj3fxXmQx8AHjU1GYfAv4oyUFdW+6Y5BDgKuBG\nU+tdxeQqa3q7Z071Ycc+2ngm8KRufyS5SdfGK1a2saquAi5L8sipfR69St9Xcybwx1Pb33h5tvv6\nEeCRmbyAjyQ3SXKbFeustLLfUi8MKbXshsDp3SCI84C7ALu67602om+10X1PBF7d7efuTAY8ALwa\n+MMk5zB5PrPsTcBFwO7uCu4fmVxFXABc0w14eBaTZ1p3XR44AbwUuE43SOHCqePsbWDVh5g8A/p8\nkt3Ac9dp42OBp3SDF77AZNDCvqzW95cBN01yYZJzgZOn16+qi4E/Az6c5Hzgw0wGray1z7cDz+sG\nhzhwQr3xVR2SpGZ5JSVJapYhJUlqliElSWqWISVJapYhJUlqliElSWqWISVJatb/Ay3MDXV4raeC\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xca04f98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "km = KMeans(n_clusters=4, \n",
    "            init='k-means++', \n",
    "            n_init=10, \n",
    "            max_iter=300,\n",
    "            tol=1e-04,\n",
    "            random_state=0)\n",
    "y_km = km.fit_predict(X)\n",
    "\n",
    "cluster_labels = np.unique(y_km)\n",
    "n_clusters = cluster_labels.shape[0]\n",
    "silhouette_vals = silhouette_samples(X, y_km, metric='euclidean')\n",
    "y_ax_lower, y_ax_upper = 0, 0\n",
    "yticks = []\n",
    "for i, c in enumerate(cluster_labels):\n",
    "    c_silhouette_vals = silhouette_vals[y_km == c]\n",
    "    c_silhouette_vals.sort()\n",
    "    y_ax_upper += len(c_silhouette_vals)\n",
    "    color = cm.jet(i / n_clusters)\n",
    "    plt.barh(range(y_ax_lower, y_ax_upper), c_silhouette_vals, height=1.0, \n",
    "            edgecolor='none', color=color)\n",
    "\n",
    "    yticks.append((y_ax_lower + y_ax_upper) / 2)\n",
    "    y_ax_lower += len(c_silhouette_vals)\n",
    "    \n",
    "silhouette_avg = np.mean(silhouette_vals)\n",
    "plt.axvline(silhouette_avg, color=\"red\", linestyle=\"--\") \n",
    "\n",
    "plt.yticks(yticks, cluster_labels + 1)\n",
    "plt.ylabel('Cluster')\n",
    "plt.xlabel('Silhouette coefficient')\n",
    "\n",
    "plt.tight_layout()\n",
    "# plt.savefig('./figures/silhouette_bad.png', dpi=300)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Organizing clusters as a hierarchical tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Performing hierarchical clustering on a distance matrix\n",
    "To calculate the distance matrix as input for the hierarchical clustering algorithm, we will use the pdist function from SciPy's spatial.distance submodule:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "labels = []\n",
    "for i in range(df1.shape[0]):\n",
    "    str = 'ID_{}'.format(i)\n",
    "    labels.append(str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID_0</th>\n",
       "      <th>ID_1</th>\n",
       "      <th>ID_2</th>\n",
       "      <th>ID_3</th>\n",
       "      <th>ID_4</th>\n",
       "      <th>ID_5</th>\n",
       "      <th>ID_6</th>\n",
       "      <th>ID_7</th>\n",
       "      <th>ID_8</th>\n",
       "      <th>ID_9</th>\n",
       "      <th>...</th>\n",
       "      <th>ID_14</th>\n",
       "      <th>ID_15</th>\n",
       "      <th>ID_16</th>\n",
       "      <th>ID_17</th>\n",
       "      <th>ID_18</th>\n",
       "      <th>ID_19</th>\n",
       "      <th>ID_20</th>\n",
       "      <th>ID_21</th>\n",
       "      <th>ID_22</th>\n",
       "      <th>ID_23</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ID_0</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>...</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.236068</td>\n",
       "      <td>2.236068</td>\n",
       "      <td>2.449490</td>\n",
       "      <td>2.236068</td>\n",
       "      <td>2.449490</td>\n",
       "      <td>2.449490</td>\n",
       "      <td>2.645751</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ID_1</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>2.236068</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.449490</td>\n",
       "      <td>2.236068</td>\n",
       "      <td>2.449490</td>\n",
       "      <td>2.236068</td>\n",
       "      <td>2.645751</td>\n",
       "      <td>2.449490</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ID_2</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>...</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>2.236068</td>\n",
       "      <td>2.449490</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.236068</td>\n",
       "      <td>2.449490</td>\n",
       "      <td>2.645751</td>\n",
       "      <td>2.236068</td>\n",
       "      <td>2.449490</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ID_3</th>\n",
       "      <td>1.414214</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>...</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>2.449490</td>\n",
       "      <td>2.236068</td>\n",
       "      <td>2.236068</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.645751</td>\n",
       "      <td>2.449490</td>\n",
       "      <td>2.449490</td>\n",
       "      <td>2.236068</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ID_4</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>...</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>2.236068</td>\n",
       "      <td>2.449490</td>\n",
       "      <td>2.449490</td>\n",
       "      <td>2.645751</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.236068</td>\n",
       "      <td>2.236068</td>\n",
       "      <td>2.449490</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 24 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          ID_0      ID_1      ID_2      ID_3      ID_4      ID_5      ID_6  \\\n",
       "ID_0  0.000000  1.000000  1.000000  1.414214  1.000000  1.414214  1.414214   \n",
       "ID_1  1.000000  0.000000  1.414214  1.000000  1.414214  1.000000  1.732051   \n",
       "ID_2  1.000000  1.414214  0.000000  1.000000  1.414214  1.732051  1.000000   \n",
       "ID_3  1.414214  1.000000  1.000000  0.000000  1.732051  1.414214  1.414214   \n",
       "ID_4  1.000000  1.414214  1.414214  1.732051  0.000000  1.000000  1.000000   \n",
       "\n",
       "          ID_7      ID_8      ID_9    ...        ID_14     ID_15     ID_16  \\\n",
       "ID_0  1.732051  1.000000  1.414214    ...     1.732051  2.000000  2.000000   \n",
       "ID_1  1.414214  1.414214  1.000000    ...     2.000000  1.732051  2.236068   \n",
       "ID_2  1.414214  1.414214  1.732051    ...     1.414214  1.732051  2.236068   \n",
       "ID_3  1.000000  1.732051  1.414214    ...     1.732051  1.414214  2.449490   \n",
       "ID_4  1.414214  1.414214  1.732051    ...     1.414214  1.732051  2.236068   \n",
       "\n",
       "         ID_17     ID_18     ID_19     ID_20     ID_21     ID_22     ID_23  \n",
       "ID_0  2.236068  2.236068  2.449490  2.236068  2.449490  2.449490  2.645751  \n",
       "ID_1  2.000000  2.449490  2.236068  2.449490  2.236068  2.645751  2.449490  \n",
       "ID_2  2.449490  2.000000  2.236068  2.449490  2.645751  2.236068  2.449490  \n",
       "ID_3  2.236068  2.236068  2.000000  2.645751  2.449490  2.449490  2.236068  \n",
       "ID_4  2.449490  2.449490  2.645751  2.000000  2.236068  2.236068  2.449490  \n",
       "\n",
       "[5 rows x 24 columns]"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy.spatial.distance import pdist,squareform\n",
    "\n",
    "row_dist = pd.DataFrame(squareform(pdist(df1, metric='euclidean')), columns=labels, index=labels)\n",
    "row_dist[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>row label 1</th>\n",
       "      <th>row label 2</th>\n",
       "      <th>distance</th>\n",
       "      <th>no. of items in clust.</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>cluster 1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.037116</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.037116</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 3</th>\n",
       "      <td>16.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>2.037116</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 4</th>\n",
       "      <td>18.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>2.037116</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 5</th>\n",
       "      <td>20.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>2.037116</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 6</th>\n",
       "      <td>22.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>2.037116</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 7</th>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.037116</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 8</th>\n",
       "      <td>6.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2.037116</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 9</th>\n",
       "      <td>14.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>2.161013</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 10</th>\n",
       "      <td>8.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2.161013</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 11</th>\n",
       "      <td>12.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>2.161013</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 12</th>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>2.161013</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 13</th>\n",
       "      <td>25.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>2.751299</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 14</th>\n",
       "      <td>27.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>2.751299</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 15</th>\n",
       "      <td>24.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>2.751299</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 16</th>\n",
       "      <td>26.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>2.751299</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 17</th>\n",
       "      <td>32.0</td>\n",
       "      <td>34.0</td>\n",
       "      <td>2.930524</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 18</th>\n",
       "      <td>33.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>2.930524</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 19</th>\n",
       "      <td>36.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>3.277628</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 20</th>\n",
       "      <td>37.0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>3.277628</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 21</th>\n",
       "      <td>40.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>3.498812</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 22</th>\n",
       "      <td>42.0</td>\n",
       "      <td>44.0</td>\n",
       "      <td>4.355821</td>\n",
       "      <td>16.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 23</th>\n",
       "      <td>43.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>5.780009</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            row label 1  row label 2  distance  no. of items in clust.\n",
       "cluster 1           0.0          1.0  2.037116                     2.0\n",
       "cluster 2           2.0          3.0  2.037116                     2.0\n",
       "cluster 3          16.0         17.0  2.037116                     2.0\n",
       "cluster 4          18.0         19.0  2.037116                     2.0\n",
       "cluster 5          20.0         21.0  2.037116                     2.0\n",
       "cluster 6          22.0         23.0  2.037116                     2.0\n",
       "cluster 7           4.0          5.0  2.037116                     2.0\n",
       "cluster 8           6.0          7.0  2.037116                     2.0\n",
       "cluster 9          14.0         15.0  2.161013                     2.0\n",
       "cluster 10          8.0          9.0  2.161013                     2.0\n",
       "cluster 11         12.0         13.0  2.161013                     2.0\n",
       "cluster 12         10.0         11.0  2.161013                     2.0\n",
       "cluster 13         25.0         31.0  2.751299                     4.0\n",
       "cluster 14         27.0         29.0  2.751299                     4.0\n",
       "cluster 15         24.0         30.0  2.751299                     4.0\n",
       "cluster 16         26.0         28.0  2.751299                     4.0\n",
       "cluster 17         32.0         34.0  2.930524                     4.0\n",
       "cluster 18         33.0         35.0  2.930524                     4.0\n",
       "cluster 19         36.0         38.0  3.277628                     8.0\n",
       "cluster 20         37.0         39.0  3.277628                     8.0\n",
       "cluster 21         40.0         41.0  3.498812                     8.0\n",
       "cluster 22         42.0         44.0  4.355821                    16.0\n",
       "cluster 23         43.0         45.0  5.780009                    24.0"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 1. incorrect approach: Squareform distance matrix\n",
    "\n",
    "from scipy.cluster.hierarchy import linkage\n",
    "\n",
    "row_clusters = linkage(row_dist, method='complete', metric='euclidean')\n",
    "pd.DataFrame(row_clusters, \n",
    "             columns=['row label 1', 'row label 2', 'distance', 'no. of items in clust.'],\n",
    "             index=['cluster %d' %(i+1) for i in range(row_clusters.shape[0])])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>row label 1</th>\n",
       "      <th>row label 2</th>\n",
       "      <th>distance</th>\n",
       "      <th>no. of items in clust.</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>cluster 1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 3</th>\n",
       "      <td>3.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 5</th>\n",
       "      <td>8.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 6</th>\n",
       "      <td>9.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 7</th>\n",
       "      <td>11.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 8</th>\n",
       "      <td>12.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 9</th>\n",
       "      <td>15.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 10</th>\n",
       "      <td>16.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 11</th>\n",
       "      <td>18.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 12</th>\n",
       "      <td>24.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 13</th>\n",
       "      <td>25.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 14</th>\n",
       "      <td>28.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 15</th>\n",
       "      <td>20.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 16</th>\n",
       "      <td>35.0</td>\n",
       "      <td>37.0</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 17</th>\n",
       "      <td>21.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 18</th>\n",
       "      <td>32.0</td>\n",
       "      <td>34.0</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 19</th>\n",
       "      <td>26.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 20</th>\n",
       "      <td>40.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 21</th>\n",
       "      <td>30.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>2.645751</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 22</th>\n",
       "      <td>39.0</td>\n",
       "      <td>43.0</td>\n",
       "      <td>2.828427</td>\n",
       "      <td>16.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 23</th>\n",
       "      <td>44.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>3.316625</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            row label 1  row label 2  distance  no. of items in clust.\n",
       "cluster 1           0.0          2.0  1.000000                     2.0\n",
       "cluster 2           1.0          5.0  1.000000                     2.0\n",
       "cluster 3           3.0          7.0  1.000000                     2.0\n",
       "cluster 4           4.0          6.0  1.000000                     2.0\n",
       "cluster 5           8.0         10.0  1.000000                     2.0\n",
       "cluster 6           9.0         13.0  1.000000                     2.0\n",
       "cluster 7          11.0         19.0  1.000000                     2.0\n",
       "cluster 8          12.0         14.0  1.000000                     2.0\n",
       "cluster 9          15.0         23.0  1.000000                     2.0\n",
       "cluster 10         16.0         17.0  1.000000                     2.0\n",
       "cluster 11         18.0         22.0  1.000000                     2.0\n",
       "cluster 12         24.0         27.0  1.414214                     4.0\n",
       "cluster 13         25.0         29.0  1.414214                     4.0\n",
       "cluster 14         28.0         31.0  1.414214                     4.0\n",
       "cluster 15         20.0         33.0  1.414214                     3.0\n",
       "cluster 16         35.0         37.0  1.732051                     8.0\n",
       "cluster 17         21.0         38.0  1.732051                     4.0\n",
       "cluster 18         32.0         34.0  1.732051                     4.0\n",
       "cluster 19         26.0         36.0  2.000000                     6.0\n",
       "cluster 20         40.0         41.0  2.000000                     8.0\n",
       "cluster 21         30.0         42.0  2.645751                     8.0\n",
       "cluster 22         39.0         43.0  2.828427                    16.0\n",
       "cluster 23         44.0         45.0  3.316625                    24.0"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 2. correct approach: Condensed distance matrix\n",
    "\n",
    "row_clusters = linkage(pdist(df, metric='euclidean'), method='complete')\n",
    "pd.DataFrame(row_clusters, \n",
    "             columns=['row label 1', 'row label 2', 'distance', 'no. of items in clust.'],\n",
    "             index=['cluster %d' %(i+1) for i in range(row_clusters.shape[0])])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>row label 1</th>\n",
       "      <th>row label 2</th>\n",
       "      <th>distance</th>\n",
       "      <th>no. of items in clust.</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>cluster 1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 3</th>\n",
       "      <td>3.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 5</th>\n",
       "      <td>8.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 6</th>\n",
       "      <td>9.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 7</th>\n",
       "      <td>11.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 8</th>\n",
       "      <td>12.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 9</th>\n",
       "      <td>15.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 10</th>\n",
       "      <td>16.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 11</th>\n",
       "      <td>18.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 12</th>\n",
       "      <td>24.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 13</th>\n",
       "      <td>25.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 14</th>\n",
       "      <td>28.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 15</th>\n",
       "      <td>20.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 16</th>\n",
       "      <td>35.0</td>\n",
       "      <td>37.0</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 17</th>\n",
       "      <td>21.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 18</th>\n",
       "      <td>32.0</td>\n",
       "      <td>34.0</td>\n",
       "      <td>1.732051</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 19</th>\n",
       "      <td>26.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 20</th>\n",
       "      <td>40.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 21</th>\n",
       "      <td>30.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>2.645751</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 22</th>\n",
       "      <td>39.0</td>\n",
       "      <td>43.0</td>\n",
       "      <td>2.828427</td>\n",
       "      <td>16.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster 23</th>\n",
       "      <td>44.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>3.316625</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            row label 1  row label 2  distance  no. of items in clust.\n",
       "cluster 1           0.0          2.0  1.000000                     2.0\n",
       "cluster 2           1.0          5.0  1.000000                     2.0\n",
       "cluster 3           3.0          7.0  1.000000                     2.0\n",
       "cluster 4           4.0          6.0  1.000000                     2.0\n",
       "cluster 5           8.0         10.0  1.000000                     2.0\n",
       "cluster 6           9.0         13.0  1.000000                     2.0\n",
       "cluster 7          11.0         19.0  1.000000                     2.0\n",
       "cluster 8          12.0         14.0  1.000000                     2.0\n",
       "cluster 9          15.0         23.0  1.000000                     2.0\n",
       "cluster 10         16.0         17.0  1.000000                     2.0\n",
       "cluster 11         18.0         22.0  1.000000                     2.0\n",
       "cluster 12         24.0         27.0  1.414214                     4.0\n",
       "cluster 13         25.0         29.0  1.414214                     4.0\n",
       "cluster 14         28.0         31.0  1.414214                     4.0\n",
       "cluster 15         20.0         33.0  1.414214                     3.0\n",
       "cluster 16         35.0         37.0  1.732051                     8.0\n",
       "cluster 17         21.0         38.0  1.732051                     4.0\n",
       "cluster 18         32.0         34.0  1.732051                     4.0\n",
       "cluster 19         26.0         36.0  2.000000                     6.0\n",
       "cluster 20         40.0         41.0  2.000000                     8.0\n",
       "cluster 21         30.0         42.0  2.645751                     8.0\n",
       "cluster 22         39.0         43.0  2.828427                    16.0\n",
       "cluster 23         44.0         45.0  3.316625                    24.0"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 3. correct approach: Input sample matrix\n",
    "\n",
    "row_clusters = linkage(df.values, method='complete', metric='euclidean')\n",
    "pd.DataFrame(row_clusters, \n",
    "             columns=['row label 1', 'row label 2', 'distance', 'no. of items in clust.'],\n",
    "             index=['cluster %d' %(i+1) for i in range(row_clusters.shape[0])])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As shown in the following table, the linkage matrix consists of several rows where each row represents one merge. The first and second columns denote the most dissimilar members in each cluster, and the third row reports the distance between those members. The last column returns the count of the members in each cluster.\n",
    "\n",
    "Now that we have computed the linkage matrix, we can visualize the results in the form of a dendrogram:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbcAAAEZCAYAAAAUgWt1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xe4ZFWZ7/Hvr7uhGxCa0wqNNpJjE8RE0FYKMQEiIjgo\nKoioyACmueZA9x2dxzReRUFs5YoYmVEJCigOeGybcQAlyCAojgkRMPQhNUEOvPPHWtUU1RV2VZ19\n6px9fp/nqadqV+13r7XXDm/trIjAzMysSmYNuwJmZmYTzcnNzMwqx8nNzMwqx8nNzMwqx8nNzMwq\nx8nNzMwqZ86wK1CUJF+zYGY2Q0WEeul/2iQ3AF+TZ2Y280g95TXAuyXNzKyCnNzMzKxynNzMzKxy\nnNzMzKxynNzMzKxynNzMzKxynNzMzKxynNzMzKxynNzMzKxyptUdSqaTBQtgbGzYtTCb3kZGYNWq\nYdfCpiNNl1taSYrpUlcACaZRdc2mJC9HBun2W73eW9K7Jc3MrHKc3MzMrHKc3MzMrHKc3MzMrHKc\n3MzMrHKc3MzMrHKc3MzMrHKc3MzMrHKc3MzMrHKc3MzMrHKc3MzMrHKc3MzMrHKc3MzMrHKc3MzM\nrHKc3MzMrHJKTW6S5kq6XNLVkq6TdHKb/k6RdJOkayTtUWadzMys+kp9EndEPCBpv4i4V9Js4DJJ\nF0XEFfV+JB0AbBsR20vaCzgd2LvMepmZWbWVvlsyIu7NH+eSkmnzc3UPAc7K/V4OzJe0sOx6mZlZ\ndZWe3CTNknQ1cBvwg4i4sqmXRcDNDd235O/MzMz6UupuSYCIeBh4sqSNgHMlLY6IX/QzrKVLl675\nXKvVqNVqE1JHMzObOkZHRxkdHR1oGIpo3ktYHknvB1ZHxCcavjsd+GFEnJ27bwT2jYjbm2JjMus6\nKAmmUXXNpiQvRwYgiYhQLzFlny35OEnz8+f1gOcBNzb1dj5wVO5nb+CO5sRmZmbWi7J3Sz4e+JKk\nWaREenZEXCjpOCAiYnnuPlDSr4HVwDEl18nMzCpuUndLDsK7Jc1mHi9HBlNwt6SZmdkwOLmZmVnl\nOLmZmVnlOLmZmVnlOLmZmVnlOLmZmVnlOLmZmVnllH5vSTMb3IIFMDY27FoMh3q6umn6GxmBVauG\nXYvpzxdxl8QXn9pE8vw0c3har80XcZuZmeHkZmZmFeTkZmZmlePkZmZmlePkZmZmlePkZmZmlePk\nZmZmlePkZmZmlePkZmZmlePkZmZmlTNj7i05jHvzTeY98Xw/OjOzR8yYe0tW/X5tVR+/mc7Td+bw\ntF6b7y1pZmaGk5uZmVWQk5uZmVWOk5uZmVVOqclN0uaSLpV0vaTrJL2pRT/7SrpD0lX59b4y62Rm\nZtVX9qUA48DbIuIaSY8Bfibp4oi4sam/FRHx4pLrYmZmM0SpW24RcVtEXJM/3wPcACxq0eskXhFm\nZmZVN2nH3CRtBewBXN7i530kXSPpAkmLJ6tOZmZWTZNyh5K8S/KbwJvzFlyjnwFbRMS9kg4AzgV2\naDWcpUuXrvlcq9Wo1Wql1NfMzIZndHSU0dHRgYZR+h1KJM0BvgtcFBGfKtD/b4GnRsSqpu99h5IO\nqj5+M52n78zhab22qXqHkv8P/KJdYpO0sOHznqSE67skmplZ30rdLSnpmcArgeskXQ0E8B5gSyAi\nYjlwuKTjgQeB+4AjyqyTmZlVn2+cXBFVH7+ZztN35vC0XttU3S1pZmY2qbomN0k7SLpE0n/n7t19\nFxEzM5vKimy5fR54N+mYGBHxc+DlZVbKzMxsEEWS2/oRcUXTd+NlVMbMzGwiFEluf5W0LelMRyQd\nDtxaaq3MzMwGUORSgBOA5cBOkm4Bfgu8qtRamZmZDaDwpQCSNgBmRcTd5Vapbfm+FKCDqo/fTOfp\nO3N4Wq+tlEsBJP2LpI0jYnVE3C1pRNIH+6+mmZlZuYocczsgIu6od0TEGHBgeVUyMzMbTJHkNlvS\n3HqHpPWAuR36NzMzG6oiJ5R8FbhE0hdz9zHAl8qrkpmZ2WAKnVCSn7O2f+78QUR8v9Rata6DTyjp\noOrjN9N5+s4cntZr6+eEEt84uSKqPn4znafvzOFpvbayzpZ8qaSbJN0p6S5Jd0u6q/9qmpmZlavr\nlpukXwMHR8QNk1OltvXwllsHVR+/mc7Td+bwtF5bWY+8uX3Yic3MzKwXRc6W/Kmks4FzgQfqX0bE\nt0urlZmZ2QCKJLeNgHuB5zd8F4CTm5mZTUk+W7Iiqj5+M52n78zhab22fo65dd1ykzQPOBbYBZhX\n/z4iXttzDc3MzCZBkRNKvgxsBrwA+BGwOTCUJwOYmZkVUeRSgKsj4smSfh4Ru0taB/hxROw9OVVc\nUw/vluyg6uM303n6zhye1msr61KAB/P7HZJ2BeYDm/ZaOTMzs8lS5GzJ5ZJGgPcB5wOPAd5faq3M\nzMwGUGTL7ZKIGIuIFRGxTURsClxcZOCSNpd0qaTrJV0n6U1t+jsl3+LrGkl79DICZmZmzYokt2+1\n+O6bBYc/DrwtInYB9gFOkLRTYw/5iQPbRsT2wHHA6QWHbWZm1lLb3ZI5Ce0CzJf00oafNqLhkoBO\nIuI24Lb8+R5JNwCLgBsbejsEOCv3c7mk+ZIWRsTtPY2JmZlZ1umY247Ai4CNgYMbvr8beH2vBUna\nCtgDuLzpp0XAzQ3dt+TvnNzMzKwvbZNbRJwHnCdpn4j4ySCFSHoMaVfmmyPinkGGZWZm1k2RsyUP\nlXQ9cB/wPWB34K0R8ZUiBUiaQ0psX84Js9ktwBMbujfP361l6dKlaz7XajVqtVqRKkwNCxbA2Fhp\ngz+Zk0HLShs+IyOwalV5wzcbgpULVjI+Nj7sajzK0WzFqH437Go8ypyROSxZtWTSyhsdHWV0dHSg\nYRS5iPuaiNhD0qGk3ZRvA1ZExJMKFSCdBfw1It7W5vcDgRMi4iBJewOfbHWB+LS/iHvoFRjQdK//\nNOfmL8eoRqlFbdjVmPKG3U6l3FsSWCe/HwT8e0TcKRUrQ9IzgVcC10m6mvQ0gfcAWwIREcsj4kJJ\nB+aHoq4GjullBMzMzJoVSW7fkXQjabfk8ZI2Ae4vMvCIuAyYXaC/E4sMz8zMrIiu17lFxLuAZwBP\ni4gHSVtXh5RdMTMzs351us7tORFxaeM1bk27I/2wUjMzm5I67ZbcF7iUR1/jVucncZuZ2ZTV6Tq3\nk/O7T/AwM7NppdNuyZan7tdFxCcmvjpmZmaD67RbcsP8viPwdNLjbiDtpryizEqZmZkNotNuyWUA\nklYAT4mIu3P3UuCCSamdmZlZH4o88mYh8PeG7r/n78zMzKakIhdxnwVcIemc3P0S4MzSamRmZjag\nrsktIj4k6SLgWfmrYyLi6nKrZWZm1r8iW25ExFXAVSXXxczMbEIUOeZmZmY2rTi5mZlZ5Ti5mZlZ\n5RR5WOlLgY8AmwLKr4iIjcqv3qPq4YeVDtN0r/+AFqxcydj4EJ/YfOZW8JrfDa34kTlzWLVk8p7E\n3KvJfqL2ZD+ZelDDfuL4oO1V1sNKPwocHBE39Fcts+lvbHycqNWGV4EawFZDK16jo0Mru4jxsfFJ\nfVL0qEYnrayJMNnt02wY7VVkt+TtTmxmZjadFNly+6mks4FzgQfqX0aEH3ljZmZTUpHkthFwL/D8\nhu/8PDczM5uyityhxM9zMzOzaaVrcpM0DzgW2AWYV/8+Il5bYr3MzMz6VuSEki8DmwEvAH4EbA7c\nXWalzMzMBlEkuW0XEe8HVkfEl4CDgL3KrZaZmVn/iiS3B/P7HZJ2BeaTLug2MzObkoqcLblc0gjw\nfuB84DHAB0qtlZmZ2QCKnC35hfzxR8A2vQxc0hnAi0gXgu/e4vd9gfOA3+Svvh0RH+ylDDMzs2Zd\nd0tKWijpjPzAUiQtlnRsweF/kXQiSicrIuIp+eXEZmZmAytyzO1M4PvAE3L3r4C3FBl4RKwExrr0\n1tPNMM3MzLopktweFxH/BjwMEBHjwEMTWId9JF0j6QJJiydwuGZmNkMVOaFktaTHkm65haS9gTsn\nqPyfAVtExL2SDiDdv3KHCRq2mZnNUEWS29tIZ0luK+kyYBPg8IkoPCLuafh8kaTTJC2IiFWt+l+6\ndOmaz7VajdowH0FiZmalGB0dZXTAxywVOVvyqnxW446k42O/jIgHu4Q1qj/gdO0fpIURcXv+vCfp\n4aktExs8OrmZmVk1NW+8LFu2rOdhtE1u+QncreyQn4ra9akAkr5GesziYyX9ATgZWJf0JO/lwOGS\njiddKH4fcESP9TczM1tLpy23g/P7psAzgEtz937Af1LgkTcRcWSX308FTu1eTTMzs+LaJrf6o24k\nXQwsjohbc/fjSZcHmJmZTUlFTih5Yj2xZbcDW5RUHyvLggUw1u2Swy40wCWJIyOwqu3h1NItWLmS\nsfHxgYahAQ5wj8yZw6olSwYqfxAzffy7WblgJeNjvbXPqEYL9ztnZA5LVk3d8a+iIsntEknfB76e\nu48A/qO8KlkpxsYgYnjlD5IYJ8DY+DgxxLNrB0kME2Gmj38342Pj1KJW2vB7SYQ2MYqcLXliPrnk\nWfmr5RFxTrnVMjMz61+RLbf6mZFdTyAxMzObCjpdCrAyIpZIupt8d5L6T6RT+TcqvXZmZmZ96HS2\n5JL8vuHkVcfMzGxwnbbcFnQK7HQnETMzs2HqdMztZ6Tdka1Ocwt6fHCpmZnZZOm0W3LryayImZnZ\nRCnyJO5DJc1v6N5Y0kvKrZaZmVn/ijys9OSIWPP8toi4g3QDZDMzsympSHJr1U+h6+PMzMyGoUhy\n+6mkT0jaNr8+QTrZxMzMbEoqktxOAv4OnJ1fDwAnlFkpMzOzQRS5t+Rq4F2TUBczM7MJ0TW5Sfoh\nj779FgAR8ZxSamRmZjagIieG/J+Gz/OAw4DBHgxlZmZWoiK7JZtPHrlM0hUl1cfMzGxgRXZLNt5j\nchbwVGB+m97NzMyGrshuycZ7TI4DvwWOLbNSZmZmgyiyW9L3mDQzs2ml7XVukt7R8PllTb/9S5mV\nMjMzG0Sni7hf3vD53U2/vbCEupiZmU2ITslNbT636jYzM5syOiW3aPO5VXdLks6QdLukn3fo5xRJ\nN0m6RtIeRYZrZmbWSafk9iRJd0m6G9g9f65371Zw+F8EXtDuR0kHANtGxPbAccDpRStuZmbWTqcn\ncc8edOARsVLSlh16OQQ4K/d7uaT5khZGxO2Dlm1mZjNXkacClGkRcHND9y35OzMzs75Nq4eOLl26\ndM3nWq1GrVZb073gIwsYu3+sffC+J6Nly9r+PDJvhFXvXNV/5RYsgLEO5QOow3k4IyOwyuX3XfzK\nlYyNd77lqUZH2xc/Zw6rlixx+dO0/GFbuWAl42Odx39Uo21/mzMyhyWr+h//qpU/OjrKaIf5pYhh\nJ7dbgCc2dG+ev2upMbk1G7t/jDi5w3kuJwO0j9eyAU8AHRuDKHSeTZsKuPyBih8fJxr+7PRc/IAL\nkssfbvnDNj42Ti1qfcd3WvHPxPKbN16WddgwaWcydkuK9pcOnA8cBSBpb+AOH28zM7NBlbrlJulr\nQA14rKQ/kLaf1gUiIpZHxIWSDpT0a2A1cEyZ9TEzs5mh1OQWEUcW6OfEMutgZmYzz7DPljQzM5tw\nTm5mZlY5Tm5mZlY5Tm5mZlY5Tm5mZlY5Tm5mZlY5Tm5mZlY5Tm5mZlY5Tm5mZlY5Tm5mZlY5Tm5m\nZlY5Tm5mZlY5Tm5mZlY5Tm5mZlY5Tm5mZlY5Tm5mZlY5Tm5mZlY5Tm5mZlY5Tm5mZlY5Tm5mZlY5\nTm5mZlY5Tm5mZlY5Tm5mZlY5Tm5mZlY5Tm5mZlY5pSc3SS+UdKOkX0l6Z4vf95V0h6Sr8ut9ZdfJ\nzMyqbU6ZA5c0C/gMsD/wJ+BKSedFxI1Nva6IiBeXWRczM5s5yt5y2xO4KSJ+HxEPAt8ADmnRn0qu\nh5mZzSBlJ7dFwM0N3X/M3zXbR9I1ki6QtLjkOpmZWcWVuluyoJ8BW0TEvZIOAM4FdmjV49KlS9d8\nrtVq1Gq1yaifmZlNotHRUUZHRwcaRtnJ7RZgi4buzfN3a0TEPQ2fL5J0mqQFEbGqeWCNyc3MzKqp\neeNl2bJlPQ+j7N2SVwLbSdpS0rrAy4HzG3uQtLDh856AWiU2MzOzokrdcouIhySdCFxMSqRnRMQN\nko5LP8dy4HBJxwMPAvcBR5RZJzMzq77Sj7lFxPeAHZu++1zD51OBU8uuh5mZzRy+Q4mZmVWOk5uZ\nmVWOk5uZmVWOk5uZmVWOk5uZmVWOk5uZmVWOk5uZmVWOk5uZmVWOk5uZmVWOk5uZmVWOk5uZmVWO\nk5uZmVWOk5uZmVWOk5uZmVWOk5uZmVWOk5uZmVWOk5uZmVWOk5uZmVWOk5uZmVWOk5uZmVWOk5uZ\nmVWOk5uZmVWOk5uZmVWOk5uZmVWOk5uZmVVO6clN0gsl3SjpV5Le2aafUyTdJOkaSXuUXSczM6u2\nUpObpFnAZ4AXALsAr5C0U1M/BwDbRsT2wHHA6WXWyczMqq/sLbc9gZsi4vcR8SDwDeCQpn4OAc4C\niIjLgfmSFpZcLzMzq7Cyk9si4OaG7j/m7zr1c0uLfszMzAqbM+wK9EJS59+Xdv696/AHjKdL/Rxf\nbvyApTt+mscPvQKOH258k7KT2y3AFg3dm+fvmvt5Ypd+iIgJHnUzM6uqsndLXglsJ2lLSesCLwfO\nb+rnfOAoAEl7A3dExO0l18vMzCqs1C23iHhI0onAxaREekZE3CDpuPRzLI+ICyUdKOnXwGrgmDLr\nZGZm1aeIGHYdzMzMJpTvUGJmZpVT+eQmafYQy+7rJJh88fua9+mq3/GvSvmDGrT+FRj/dSdgGMNc\n/h8raWSA+CdI2muA+A36jZ0Iw2x7qHByk7RQ0ux83K/nRpa0jaQ3SHqepE17iJsvaQGkg4q9JihJ\nuwDnSVonIh7udQU16AwlaStJLxkgfqP6SikG2OctaTtJ75R0hKTH9Vj++v2WL2lDSev1GtcQv7mk\nbQeI31nSB6Hv+m8iaTdJ6mf+axjOtpJOknSIpM36iN9G0uskHSppfh/x2wFf7XflPgHL/6Djvwvw\nI2A3SXP7iN8J+E/g1b3G5vidgdE8P/Y8D/S7/DXE7wB8XtKbJe3ZR/xA8w9UNLnlCXstcFFOEj3N\n4HnBugTYjHSCyzslHVYgbifgP4BPSboMoJcEVZ8hgCcDb8vxhVdwknYE3tV8i7Me479J0xUnPdR/\nZ1K7/aukb/azUOTh7AR8C1gfeBcFTzKStCtwGfBZSRf08qekodzvAadLuqC3WoOkecB/Ae+V9JSG\n73tpv68Dq5q+7yX+h8BHgOuht/mvYTjbkFasC4GTgLdLek0P8TsBFwLbAu8Hju+l/Gx/YB/gPZL2\nzcPtpR0GWf4HHf9NgS8An4yIFRHxQNHYHL8TaT3wDeD5kp7TY/yOwHLSCXx/jIiH+yi/5+WvIX57\n4DvAb0h3qeq1/hMx/0BEVOoFzAO+Bvxf4P8BFwDr5N9mFxzGW4B358+bA+fkxn5lh5htgf8Gjs3d\n3wF2bfhdXcrcnrRAHgU8Gzi1aGxDPf8KrADeDezUY7ttTbqDzBGt2qpA/R8H/Bg4Mnd/HvgV8LSi\n45D72xi4Bjgud+8LnAY8BRjpEPeYPK2Pzt2fBc4D9ixY7jakhPA6YD7wb6SVU6/z33nAvwMfqI97\nD7HvA06qt3+epusVjN2ElNjr7f+tXstvGNZBwEfz50XAK4DPAccUiJ2f58H69NsF+DWwT4912Bn4\nFHA48F1gt4Jx603A8t/3+DdMi+X581zgvcDr69OmQGzjeuSfgI/m4RRZD2wI3AR8JHfPId3bd39g\n8wLxfS1/DfHKdX5v7n4acDZwYJF5IJc/8PwTEdXbcouI+0n/XD9LmqluBc6VNDciHio4mHuAmqRN\nI+KPwFXADcDukjZq7jn/o9wD+FBEnCFpDrAjcIyksyRtHNF+F5GkdYDDgA9HxFmkmfMASW/M41Rk\n620+aeX4HmAB8LL8D7axjp1sA/wFuDt3L5f0WUkXdat/9nfgt6QEB/BG0qUd/9QQ3/Wfd0TcAbwq\nIj6X23E5aSX/QdIW9DPahK7Or7/k4RxPus7y3ZIWQfs2yOO1D/DZiPhCRNwJnEJaofTqQmAlaSV1\ngNJu7QMlzSow/iOkf8uQkuSnge9LekmBXVv3k/5MXJe7dwfeJGmFpCdDT8fgHgIOkbQoIm4hbc3+\nkDT/d9vlei/wYdIuxVkRcT3wfdKfj178kfSH6/ekleMnJf1E0k6dxiMi7gM+zmDLP/Q5/nle2hDY\nKff7r8BWpL1AB0t6e5dy7wOOiogzcvf1wF7A/CLLUETcTZpvnqd03fA5wCuBDwFv6baLMC9/R/ex\n/NXjg7QMvjGXfxYp4R1BWh++olM8af3zMQaff6qT3Bp3O0TEtRFxa0TcC7yZNIOfk/tbVF/Y28WT\nGvNq4NtKxz8OJk2kp5E2sx8lT9ALI+LreeZ7L/ATUpK9H/hB7q/l7oFIN5U+NcfPiohbgROB/SVt\nWbAJfgl8OSJWAt8GNiIluF3ro9gl/jLgHcAbJN0G3EVaSdwOXNSp/nmc5wLrkBaqxcCr8jAfIK1o\nOibppva/Pr9vDHwxIl4MvJaUMJ7aInZWHvYKYGdJm+TyPkhaSZ7Zqfw8XpeQElPdn4E9lY/fddOQ\n+FeTtrZOArYj3aTgqRHxcIE/KV8FFkr6KHBNRBxK2jX1OuCxHcoW6R/6XcA7JF0BrIiIo0htcoqk\n9Yq2f0R8j7Tl+q78B28sD2cr0p+AtvF5Xl4REfc0zC+R26J+LGerTuVLmpNX0n8g/an8MbCYtIKb\n22o8lI6VbpDrcBXw5x6X/w31yLHaC3L//Yz/wxHxG9Iy83Vg3Yh4PTnhAy13lTfE35PrT+7+Hmn3\n3icb5vO28TnmFOCLuc6/zvPBYaQ/vU9vE9/Yftfm4RVa/lqUfxZpt+wrgF9GxD8AbwJuJO2half+\nehHxUER8B7ivl/mnlUokN7U41lT/hxMRq4G3AjdIug64gqaL15vjI+Jm4JOkXRE3AS+LiGuBS0m7\nPdeS/zHWV6BfiYijI+LPEfEG4PZuEyQvzI0J5Fekf9Bb5Dp2nFYRMZ7HlYj4CWnhnA/sp3TR/NfU\n4bhD3uL9MXAGaZfKWyPifyLiNcDfJG3RITYi4i+kXZH7krYeX03aNXQ88LdOdW/R/qF0MsBfI+LD\n+bvbSNPu8coayq+32eWkXboHKR/vy0nmb+pyUkBE3JZXSvV5ZzVpRXqvpJqkT3eJr9fhu8BGufy9\ncp1nt1qhtvBX0oL8JPIt6CLitFyXts85zO0/Rppn/5m0W+nM/Nv7gDs7xbdafkh/kO4hHfN6QkT8\niXQzhh2b58UW0++e/H19OXsIuCv/6fkWsEGX+PH807WkP4gXkR6F9THgw0pnIaohvn6s9DRJF+Zh\nPCRJBZf/evxnJV2Uv/48aSu05/HPPk36w3SkpO1z0p8NLJa0QVP9Wx4rbyjn47ku29FCq/iI+DRQ\nIy2L5C3QK4HNmpefpva7oN5+RZe/NuUvI82DcyU9NtLekD8Du0qa26b8z0n6br18PXK2bMf5p63o\ncT/mVHtR8FgTab/zvcCLu8QvbhO/P/A/wDN6rN8zSbs1n9jHuL2d9M91oz7bZitSklsFvKJgzDqk\nlXq9ewlpK7ZQ/Un/9ubV60za134xKdGudcygh+n3bNKxiP27lP8c0vHOE0kL99NJ++x7an/Slu6Z\npGR9JfDSAjGzSMcefwT8CfjH/N1p7carxTB2Je0KOo10vGk30j/e3Xuo+z+StvY2ISXKGzu0a9v5\nn3Sc5Z9zu7+V9CfluUWnX316A0eTEtRlwCE9xD8jt+WJuXsE2LIpvu2x0sb5jfbLf3P8N4GP59+e\n2sf479zw22NIu/OuJW253AQcUDS+oZ95pL1JH53o5adT+xWM71h/0u7Qc/I8cBPwgh6m36xu80/H\n5aCXBX4qvkgHHN9IWgl/jHR2TeMMNicvFBcAL2kx03eLn01KElcDBzcvuB3qNRfYD/gFcFCL39vG\n88hKYR5pS2q7XuIb+lkMPAy8qMV4F4l/NmnFWLj+DXWfRUout9XL73P6rU/aHXxD43C6tN/epLO8\nzicd/zqs1/YjJfk/k3ZpHdBL+wEvBt7Q0L3WSSFd4ncg7Ub6Kumf6qE9xj+b9Ifia6RjcC/u0G+r\n9l/c1M8RwGuA5/Q6/XI/r8/z4XNbtGO36d/2pKw8j72SnPzyd0tIx04bl98FtFj+O8Sf3mL8j+5h\n/Jvb7/nA84Alfcz/c/L7jsDz+ohfJ/fzK5qWw4LtN48Wy1+X8hv/oOxG2no8lbUTW9fyu80/HZfh\nIj1N5RcpeW2QP++TG/gD9YWCR7L/wnrDNC1cReMXNXXX39ut5B+Xh7PWBC0S31DXzQaI37Jxhsiv\novWfTzrr6fn9lJ8Xqhp5hdChv67tnxewnXosfzawLvCExv6KxOd2Wgf4Cmv/Uy3c/o399Tn9ZgMb\n91r//P3WpDN4d+mz/XdrVf/mYRWcfhsCT25Vjw7xT2oaz5ZnOpJO1NimoXsH4GfA+k39tVv+28Vv\n0GG69Lz+6DCsnuJ7aL/m9dcObeK7th9pWdi5x/J3ayp/3X6nH+kPbsv5p9OrcveWVDqb5zDgd6Qz\n+PYnHdiMKHC9R6d40sHQo0mnB98KfC8iRvO+/bUaUtIGkY+D5eNd2xWNb/6u1/gWdek5XtL6kQ7K\n9xs/q97m+Rha17PVmtr/QdJuwVdF2ge/Y9HyJ6L9JI1EOpbVb/ya7/Mxhh0ms/5Nw+un/f9O2s17\nZJHYNvH7Aq+ux3era4v4/UiX4BSpu4AnAN+PiF0l1UjHy09onBd7jD88Ik7sVvZE1L+E+AdJfzAH\nab+XRsSbipTdpvz9gCMpsP5tU/6hEfHmxn66zetrFM2C0+lFH8eausWTdgtcR7qo802k42F3kHd5\nsfY/mtn5fe6A8esOOX5C6j/E9p8S4z/s+GEvP5MVTx/HSicyftjjP+z2a1H+yye7/DXD6jdwWK/m\nhbhNP4OU/19zAAADNElEQVQea2qO35EWx56AfyBdl9F8kLm+YtmYdCr3s2Z4/KJp3P5fnwLtN+z2\nH3T5mbR4BjhWOhHxwx7/Ybdf2eX38uo5YFgvJulYU5v4Y0mnY2/dMAHqK5ATSNdx1f9h17+fT7rw\n89kzPX4Cpt+0Hv9hx0+B5WdS4hnwWOmg8cMe/2G3X5nl9/Mq9WGlE6V+rEXpThO3Sup4rCEifk+6\nswGkA9rb9RMvaWvgzkh3HZlHutPBcRHxX3rkGp4x0umwD+bYhyRtTLrDxGeAn0fEihka/96IWNnv\n9HP7D7f9c2ffy89kx+fPD0o6KVocKy07ftjjP+z2K6v86O3OMmtM+Yu49cjNfG8lnUXzN9JCfljE\n2rejyRMDPXIB4HYDxG8L/Ebp9lGnkq66/5ykveKRC03HgJuBdZTMJp2G/oH82+9manzDitXtPz3b\nHwZbfoYSHxFjeuR2ZZMeP+zxH3b7TWT5/SY2cvCUfTH8Y12LgBeSLt4eyb+dRLooc+s84W4ADmwa\nzkjD5xkbPwHTz+0/3PYf9rHCYR4rrcKx1uk8/l+n4VhxP6+hJa5ClZsCxyrydwfw6BXMCaSLk39P\nw4qFNte0zNR4t//0bn/HO36Y8a2WhV5eQ09gbRbmrXnk4tUTSP9U987d9Sv2jyTdZqnxTJyNSbfr\nedmA8a3uJFBfwdSHexIt7tzRYZxmTPwETD+3/3Dbf9Dlx/GOn9Dlv5/XwAMo4wU8l3QNT+OCfC2w\nV9PCfhrpLhQi3c3hw6TrIwaK71CvF5JOU13Q8F3h01RnSrzbf3q3v+MdP8z4ostD1+VlogY00S+G\nfKyjQ70OAvYbYLxmRLzbf3q3v+MdP8z4iXhN2IDKeDHkYx1d6tbXhYUzKd7tP73b3/GOH2b8oK8J\nH+CEV3DIxzr8Gu7082u47e94x0/X9W/pBUxIJYd8rMOv4U4/v4bb/o53/HRc/yoXNOVJOgi4NyJ+\nOIx4G4zbf7iGvfw43vGDxPdV5nRJbnVSD488KCHeBuP2H65hLz+Od/xkLf/TLrmZmZl1M+XvLWlm\nZtYrJzczM6scJzczM6scJzczM6scJzczM6scJzczM6uc/wXv4NxMvPTwtAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x99eaa90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from scipy.cluster.hierarchy import dendrogram\n",
    "\n",
    "# make dendrogram black (part 1/2)\n",
    "# from scipy.cluster.hierarchy import set_link_color_palette\n",
    "# set_link_color_palette(['black'])\n",
    "\n",
    "row_dendr = dendrogram(row_clusters, \n",
    "                       labels=labels,\n",
    "                       # make dendrogram black (part 2/2)\n",
    "                       # color_threshold=np.inf\n",
    "                       )\n",
    "plt.tight_layout()\n",
    "plt.ylabel('Euclidean distance')\n",
    "#plt.savefig('./figures/dendrogram.png', dpi=300, \n",
    "#            bbox_inches='tight')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaoAAAF6CAYAAABIsRzZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuUZVV94PHvr7sFw0PsxgAGFNCRmIcIJCqR1hSaBDRO\ndCXxQRMdUYlJfDBxrQkRk9XVDhmdjFHBmNE2iAMTSCboCmhUUEMlgkAUaWmBNioIobVRbDChDULD\nb/44p6Qo6z7q1rl1z67z/ax1FrfuOWeffatX7R+/vffdOzITSZLaatWkKyBJUj8GKklSqxmoJEmt\nZqCSJLWagUqS1GoGKklSqxmoJE1cRKyLiOsi4usRkRGxo/75ixGxZt61b4mIL0fEl+rrH4yIIwaU\n/+b6v6sj4oGIuCcitkbEhRGx55znfzEivhURt/d6/rxyXx0R71rg/f0i4nfn/PzYiPh/867ZGBEv\nmfPzORHxpDk/fzYijuz3ufrU6/b69/OliPhMRBw84PqIiNNHedZyMFBJmrjM3JmZRwOfB24FtmTm\n0Zl5TGbunr0uIo4Fng8clZlPBb4EXA2ctFC5EbG6fnnGnLe/l5n7ZOZTgABOnX1+Zh4D/G/gnQs9\nv1f1I2J+W7oW+L05n+9bmfmSedecDPznOde8OjO/OuBZw3oQWF//jq7i4Z9/IauBP2zo2Y0Lv/Ar\nqQ0iYm9gG/AR4GWZeWBEBPAp4JnAbuBe4BuZ+fSI+Dvg14AfAHsC7wWeCzwZ+C6wN/Au4DeAnwS+\nD3wWeDrwCOBvgROAdcBdwKOoAteewJ31/f8B7A88QBVA7wUOABJ4DPAV4EnA14FD6rrcUpd1BHAd\nMAP8MvAz9f0XA/sCv1p/9PupgtYbgddl5vUR8VvA+4HtwN9m5lvqoHsn8C/Az9Z1OCMzz46I5wJv\nBr5XP+dxwGMz898i4sP1828GPjHn/BPq39O3ga9SBfub68/1Dapg+x1gDfC5zHz94H/F8TCjktQW\nLwQ+CewEdkXE0cBpVA3rPlQB4VHAoRGxDZgCLqvPAZwI/CVVMLiDKtA8EXgV8O9UAezxVMFoD+Df\ngG9RNfzPA36sfv8i4H3AI4H/Afx4Xd5qqiB2X13Gqvq99wPH1K93Z+bTqBp+6gzt8fV9NwA/QRVM\n762fez5wQGb+7ewvoe6m++/A9cBLgOMi4vn16f2A6zNzb+ADwOkRcUh97miqLO6nqYLLz0XEAfXv\n6Y2ZeRTwNuAVVAH2BVSB+DXAUfXv42tUwfvlwFbg2PozPCIiXtbj323seva9StIyOwl4N7Ae2AJs\nAH6eKnh8gSo43Ad8GrgU+BBVo/of9blHA/9MlS09ATgSOJ4qs9kbuIQqqO1LFVR+qS7vXcA9VNnE\nY6ka/HVUweRvgN+v738kVRZzWX3ND6iCxW9RdUE+Anh0HRwu4aGM6dlUWdPjgMvrsi6u60dm3j3v\n9/AM4DNUWeCDwAV1GZfWP6+PiOuoMp7ZAA5wdWbeARAR9wPn1nXaC3hWRNwNfAzYCPwmVTY6G5j3\nqsu4EjibKpP7OeALdVb7SOA2JsSMStLERcRa4DlUGdFpVFnAi6kC0GfrsaKjqboBP8xDwek7VEEg\ngVWZeUX93m6qQLZXfR3Ae4DDqTKHB6kyuH+s74Uq6/p0ff8RVFnQM6gC50eBG6m6/9bU999L1Zj/\nGFVQ/Q5V19sj5zyT+vW5wFfrz/AJ4JsL/Bp+Fvhr4NR598/1IPB7dTmnUwXSK6gywGdGxB/Nufat\nVNncx6gy0RdRjVetB+6mCthXUWWch1MNBf1J/fw9qQLqi+uxup+qz02EgUpSG7wYOC8zDwfOAs6k\nGuv5MvDciNg/Ig7koTGoo6nGjT4GXEgVbDIink3VyH6WKujdDvwVVcN8HlUWE1SB7HVUGcRvUDXa\njwSeVd/zcaps5CCq8aunUQW4J/BQN18Ax1KNfd1AFQwOqD/Pq+Z8thng1TwUfG4BXkuVxa2tgzT1\nZ30ZVVfcFFVAXFO/N1Nfcz/wujmTRPar6/I7wOWZeeac585mQpuAX6TKpJ5E1bX6aeBP6s8F9ZhX\nRDwxM79MNd61miqjnZ2V+TgmxMkUkiYuIj4D/M/MvCwiNlJlN/cBP0XVuD6LqpFeBeygCiBQNcT/\nQfV//9upAsUjqCZlfIdqgsKbqboA96PKiH6cKgB8hCojOqgu+1FUGcv9c8r8FlWXXVJNOridKrj9\nt7ouf0U1UeE7VMHwWVSZ1k1U4z43UGWBz6UKkj+o6/D3VAHosVRB82TgDcDr68kUG6i6FfegCoTf\no8qcTqQaE3tB/XlWUXURHks1EePX69/nvwP/lWrM7yNUAfZ+qizsZOBQquD8aKrxt4/X9Z4NqLdS\nBeZ962fcB/xOZl7b559xbAxUklotIvbOzF0RsQ64BjguM78975pbgJ/LzJ3jKH8x5dSv30816eG9\niy1HP8rJFJLa7mMR8WiqTOmtPYLIUv6Pe5jyh/G7EXEyVdfj56kyok6JiD2Bf6LKBNcAF2XmpgWu\nO5tqpuUu4JWZuaVvuWZUkqSmRMRemfn9ehztSqqp8f885/zzqLo4fzUingGclZnH9ivTyRSSpMZk\n5vfrl3tSZVXzs6EXUk1sITOvAfarJ8r0ZKCSJDUmIlbV3/PaAXwqMz8/75KDgX+d8/P2+r2eBo1R\n2S+4cvT6XobUGhFhmzNAZi75b3lVRBODPndk5kHz38zMB4GjI+JRwN9FxE9n5o1LeZCTKSS1yl59\nzt1HNUq/kF0LLktbmd4K009ZfF0G3ndB7+Z+enqa6enpBc/tHb1jTb/P+P0e7y9WUs29X4pd0Le7\nrl5n8HKqKfVzA9V2qin/sw6p3+vJrj9J6qBVSzwWEhGPiYj96tc/RvUds23zLruEar3B2dXw755d\n+qkXMypJjakXSD2P6v+2HwQ2Z+Z7IuJPqba0+AHVSuOnZOa/Ta6mWj34klE8Fvg/9bYnq4C/ycyP\nR8RrgczMzfXPz4+Ir1FNTz9lUKEGKklN2g28KTO3RMQ+wLUR8SmqhVz/MDMfjIi3U60W8ebFFj5q\n4zp1wOBrmrwPYGpqaqT7xhRAlkVmbqVaYmr++++f9/OitgwxUElqTGbuoJrtRWbeExE3AQdn5qfn\nXHY11fp6izZyoOo7mtL8fdD+QFXSuI+BStJYRMRhVOvdXTPv1KuoVgnXBJWUufUNVLdM38Ktm25d\nrrosaM3aNazfuX6idZC0OHW330XAaZl5z5z33wLcn5kX9Lr3vjmvV1NWg9q0B+pjHEr6vfYNVIdP\nH87h04cvV10WNBMzE32+pMWJiDVUQer8zLx4zvuvBJ5Pte9UT72mZnfR/EC9e1IVmTC7/iQ17YPA\njZl51uwbEXEi1dYYz87MH0ysZvohx6gkdVJEHEe139HWehmdBN5Ctb35HsCnqp3NuTozf29iFdXK\n6fqTpMXIzCtZuA180nLXRf2VFKhKyv4kSR1kRiVpZeiz7l5fG7q5XnNJWYqBSpI6qKSuvyIC1Tin\nqPs9LUlqtyIC1VROja1sv6clqYvs+pMktZpdf5KkVispUJWU/UmSOsiMSpI6qKQsxUAlSR1UUtdf\nEdt8SJKatWICVRu2+ZAkdZvpiiR1kGNUkqRWWzFdf5KklamkjKqkukqSOsiMStLKMIntOgreIsSu\nv8IMWpjWFdYlrTQGqsIMWp3dFdYlrTQljfuUVFdJUotFxCER8Q8RcUNEbI2INy5wzaMi4pKI2FJf\n88pB5ZpRSVIHjanrbzfwpszcEhH7ANdGxGWZuW3ONa8DbsjMX4uIxwBfiYj/m5m7exVqoJKkDhpH\noMrMHcCO+vU9EXETcDAwN1AlsG/9el/gu/2CFBioJKmTxj3uExGHAUcB18w79efAJRHxTWAf4KWD\nyjJQSZIGuhP47pDX1t1+FwGnZeY9806fAFyXmc+JiCcCn4qIIxe47oeKWD3dqeGS1KzFdv0dWB+z\nvtrjuohYQxWkzs/Mixe45BTgbQCZ+fWIuAV4MvCFXs9u/erpTg2XpOaNsevvg8CNmXlWj/O3Ar8E\nXBkRBwJHADf3K9CuP0nqoHFMpoiI44CTga0RcR3VxIkzgEOBzMzNwJnAhyLi+vq2P8jMnf3KNVBJ\nkhqRmVcyIAZm5reoxqmGZqCSpA5yCSVJUquVtCyRgUpSq+w6adI1UNsUEajGOfNvzdoifgWS1Ci7\n/ho2aHVzSdLiGKgkSa1W0hhVSXWVJHWQGZUkdZBdf5KkViupO81AJUkdVFJGVVJQlSR1UBHbfEiS\nmlVSRtX6bT4kSc0rqTvNdEWSOqikjKqkoCpJ6iAzKknqoJIyKgOVJHVQSd1pBipJrbL3haPdtyuz\n2YoMY0Ms/zMbUlJGVVJQlSR1kBmVJHVQSVmKgUqSOqikrj8DlSR1UEmBqqTsb2I+xGFEsGzHunWT\n/sSS1B4GqiG8km+QybIdd9016U8saaVbtcRjOdn1J0kdVFLXXxGrp6/fuX6idZCklWYcgSoiDgHO\nAw4EHgQ+kJlnL3DdFPAu4BHAdzLz+H7ltn719JmYmejzJUlD2w28KTO3RMQ+wLURcVlmbpu9ICL2\nA94L/Epmbo+Ixwwq1K4/SeqgcYwzZeYOYEf9+p6IuAk4GNg257INwIczc3t93Z2TqKskqeVWL/EY\nJCIOA44Crpl36ghgXURcHhGfj4iXDyrLjEqSNNDX6mMYdbffRcBpmXnPvNNrgGOA5wB7A1dFxFWZ\n2bN4A5UkddBiu9OOqI9Zl/W4LiLWUAWp8zPz4gUuuR24MzPvBe6NiH8CnkqfOGigkqRRXTDiiu0X\nTn7V9TFOT/8gcGNmntXj/MXAeyJiNbAn8Azgnf0KLCJQjTLzz2ntktTbmKanHwecDGyNiOuABM4A\nDgUyMzdn5raIuBS4HngA2JyZN/Yrt4hANZVTi77Hae2StLwy80qGiIGZ+Q7gHcOWW0SgkiQ1q6Qp\n3yXVVVIBIuKciLgjIq6f895TI+KqiLguIv45In5+knXU+KenN8lAJalp5wInzHvvT4GNmXk0sBH4\nX8teKz2MgUpSZ2XmFcD8PQAeBParXz8a2L6slVLRHKOStBx+H7g0Iv4MCOCZE65P55WUpRSxerqk\n4v0u1SoFfxcRv0n1XZtfXujC++a8nkQ3U5s8UB/jUNLvtfWrp0taEf5LZp4GkJkXRcQ5vS7cY/nq\n1HrzA/XuBssuKaMqqa6SyhH1MWt7RPwiQEQ8F/iXidRKRbJfTVKjIuICYArYPyJuo5rldypwdr1s\nzr3Ab0+uhoIV1PVXumFWp3CpJalZmbmhxym/O9UiBqqWGGbpJZdaktRFJY37lFRXSVIHreiMSlKH\nbBhx64xRt+oonF1/kqRWM1BJklqtpHGfkuoqSeogMypJ6iC7/iRJrVZSd5qBSpI6qKSMqqSgKknq\noNZv8zEqtweRpN5Kyqjc5kOSOqik7jTTDgav92d2JkmTYwvMcIvXStJKsmK6/iRJK5OBSpLUao5R\nSVIXjLhi+66TRntcXDjafcslIg4BzgMOBB4EPpCZZ/e49mnA54CXZuZH+pVroJKkDhpT199u4E2Z\nuSUi9gGujYjLMnPb3IsiYhXwduDSYQotKfuTJDVk1RKPhWTmjszcUr++B7gJOHiBS98AXAR8e5i6\nmlFJUgeNezJFRBwGHAVcM+/9nwBelJnHR8TThynLQCVJGujz9TGMutvvIuC0OrOa693A6XMvH1Se\ngUqSOmixGdWx9THrfT2ui4g1VEHq/My8eIFLfh7464gI4DHA8yLi/sy8pNezDVSS1EFjnKDwQeDG\nzDxroZOZ+YTZ1xFxLvDRfkEKDFSS1EnjGKOKiOOAk4GtEXEdkMAZwKFAZubmebfkMOWu2NXTh+U6\nfpLUjMy8kkXEwMx81TDXuXq6JHWQSyhJklqtpC/RGqgkqYNKyqhKCqqSpA4yo5KkDiopSzFQSVIH\nldT1F5l9p7EPNcddzYqA/v8soxXbeIlSwyIic8QtMCbigtH+UPeO0f4cvw9k5pL/liMih1oNto8D\nGqrLMErK/iRJHWTXnyR1UElZioGqpUbsGQBg7VrYubO5ukhaeUoaozJQtdRSxqiWEuQkdUNJgaqk\n7E+S1EFmVJLUQSVlKQYqSeqgkrr++gaq6VtuYdOtk93mY+2aNexcv36idZAkTU7rv/AbMzPk1NSk\nq7GslvqF3x73O8VCrecXfvtr8gu/P1hiGXs2VJdh2PUnSR20Yrr+JEkrU0mBqqSJH5KkDjKjkqQu\nWmqa8mAjtRiKgUpSt404IaJ4S+37M1A9XMzMLPoep7VLUh9LDVT3N1KLoRQRqEaZnj5KcJMktU8R\ngUqS1LCCptIZqCSpiwqan15QTJUkNWb1Eo8FRMQhEfEPEXFDRGyNiDcucM2GiPhSfVwREU8ZVFUz\nKklSU3YDb8rMLRGxD3BtRFyWmdvmXHMz8OzM/F5EnAh8ADi2X6EGKknqojH0p2XmDmBH/fqeiLgJ\nOBjYNueaq+fccnV9vq8iVk+XJDVszGNUEXEYcBRwTZ/LXgN8YlBZ/QPV4Yczffjhi6mbJKkEi8yo\nZu6rjmHU3X4XAadl5j09rjkeOAUY+IVX0xVJ0kBTe1THrE27Fr4uItZQBanzM/PiHtccCWwGTszM\nuwY920AlSV00vq6/DwI3ZuZZC52MiMcDHwZenplfH6ZAA5UkddEYAlVEHAecDGyNiOuoNt89AzgU\nyMzcDPwxsA74i4gI4P7MfHq/cg1UktRF45n1dyUDQmBmngqcuphy/cKvJKnVVnRGNczCtK6yLq0Q\no27XsSGW/5ltUNASSis6UA2z6rqrrEvqJAOVJKnVChr4KaiqktpugUVJ31C//9Z6EdLrIuKTEXHQ\npOuqchioJDVpdlHSnwF+AXh9RDwZ+NPMfGpmHg38PbBxkpUUY1k9fVzs+pPUmF6Lks5bPXtv4MFJ\n1E9zFJSmGKgkjcX8RUkj4kzgFcDdwPETq5iKU8Tq6U4fl8qy0KKkmflHwB9FxOnAG4Dphe6d3vrQ\n66kDYOrAcde2vR6oj7FYKbP+2rB6utPHpbIMsSjpBcDH6RWoBu732h3zh4N2N114IQrqpZRUiB9Z\nlDQi/tOc8y8Cblr2WunhVi3xWEaOUUlqTJ9FSV8TET9J1ZN1K/A7k6ulSmOgktSYPouSfnK566IB\nCur6M1BJUhcZqCRJrVbQDAUDlaRuK3kF9I4oIlCNMkV97ZoiPpokTYZdf80aZrsOSdIi2PUnSWq1\ngjKqgmKqJKmLzKgkqYsKyqgMVJLURQX1pxmoJKmLVkpG1ZZtPiRJ3dX6bT4kSWOwUjIqSdIKVdAY\nVUFVlSQ1ZvUSjwVExCER8Q8RcUNEbI2IN/a47uyI+GpEbImIowZV1YxKktSU3cCbMnNLROwDXBsR\nl2XmttkLIuJ5wBMz80kR8QzgfcCx/Qo1o5KkLhrDDr+ZuSMzt9Sv76HayfngeZe9EDivvuYaYL+I\nOLBfVc2oJKmLxjyZIiIOA44Crpl36mDgX+f8vL1+745eZRmoPnQYcfykK/Fwa9dOugbSBI267caG\naLYewyh5i5BFBqqZHTDTM5Q8XN3tdxFwWp1ZLYmB6pXfID902KRrIUmtNnVQdczatHXh6yJiDVWQ\nOj8zL17gku3A4+b8fEj9Xk+OUUlSF41hjKr2QeDGzDyrx/lLgFcARMSxwN2Z2TdXM6OSpC4awxhV\nRBwHnAxsjYjrgATOAA4FMjM3Z+bHI+L5EfE1YBdwyqByDVSSpEZk5pUMEQIz8/WLKddAJUld5BJK\nkqRWK2iGQhGrp+9cv36idZCkFWelZFRtWD09ZmYm+nxJ0mTZ9SdJXbRSuv4kSSvUSun6kyStUAUF\nqoKSP0lSF5lRSVIXFZSmFBGoRpn557R2SUMpeQX0pSio66+IQJVTU4u+x2ntktRHQYGqoORPktRF\nRWRUkqSGFZSmGKgkqYsK6vozUElSFxWUURVUVUlSF5lRSVIXrZSuv7Zs8yFJatiKCVQt2OZDkjQG\nBQ38FFRVSVIX2a8mSV20Urr+JEkrlIFKktRqBQ38GKgkrQyjroK+IZb/mVqUFR2ohllB3envkjrJ\nrr92GGV7EEnqhIK6/gqqqiSpzSLinIi4IyKu73H+URFxSURsiYitEfHKYco1UElSF61e4rGwc4ET\n+jz1dcANmXkUcDzwZxExsGdvRXf9SZJ6GMMYVWZeERGH9rsE2Ld+vS/w3czcPahcA5UkddFk+tP+\nHLgkIr4J7AO8dJibDFSSpIFmtsHMV5ZczAnAdZn5nIh4IvCpiDgyM+/pd1MRq6fvXL9+onWQpBVn\nkV1/Uz9THbM2XTLSU08B3gaQmV+PiFuAJwNf6HdT61dPH+a7UJKkRRpf11/Ux0JuBX4JuDIiDgSO\nAG4eVKBdf5LURWOYTBERFwBTwP4RcRuwEdgDyMzcDJwJfGjO9PU/yMydg8o1UEmSGpGZGwac/xb9\np68vyEAlSV3kEkqSpFYraLkHA5UkdZEZVbNGmfnnquiShuJWHa1XRGvuKuiS1DAzKklSqxU0RlVQ\nVSWVYKGtHiJiY0TcHhFfrI8TJ1lHMa7V08fCQCWpab22enhnZh5TH59c7kqpXHb9SWpUn60eei2r\no0koKE0pqKqSCvf6emfXv4yI/SZdmc4rqOuviNXTJRXvL4C3ZmZGxJnAO4FXL3Th9PT0D19PTU0x\n1eFZvw/Ux1gUNOsvMvt+h8AvGBQoAhb4Z7XbRcum7vr7aGYeuchzOaBNWhH2jtH+HL8PZOaS/5Yj\nIvMzSyzjuc3UZRimK5LG4WFbPUTEQZm5o/7x14EvT6RWekhBAz8GKkmN6rHVw/ERcRTwIPAN4LUT\nq6Aqdv1pkuz6U6ns+uuv0a6/K5dYxnHL1/VXUPInSeoiu/4kqYsKSlMMVJLaZcMy91IvZfX0Eeu6\n66TRHhcXjnbfggoaoyoopkqSusiMSpK6qKA0xUAlSV1UUNefgUqSuqigQFVQ8idJ6iIzKknqooLS\nFAOVJHXRiun6m56u1uOZ5LFu3fL8JiSpS8awH1VEnBMRd0TE9b0eGxFTEXFdRHw5Ii4fpqqDA1Xm\nZI+77hrmc0iSJu9c4IReJ+sNM98LvCAzfxZ48TCF2vUnSV00hjGqzLyi3m+slw3AhzNze339ncOU\nW9BwmiSpMZPZiv4IYF1EXB4Rn4+Ilw9zkxmVJHXRItOUmatg5uolP3UNcAzwHGBv4KqIuCozvzbo\nJkmS+pr6heqYtendIxVzO3BnZt4L3BsR/wQ8FegbqOz6k6QuGl/XX9B7o9aLgfURsToi9gKeAdw0\nqKplZFSj7Ia5di3s3Nl8XSS101K26+iiMXyPKiIuAKaA/SPiNmAjsAeQmbk5M7dFxKXA9cADwObM\nvHFQuWUEqlG2ph5xq2dJ6oTxzPrbMMQ17wDesZhy7fqTJLVaGRmVJKlZBS2hZKCSpC4yUEmSWq2g\ngZ+CqipJ6qL2r56+du3y/CYkqUsms4TSSNq/errfhZKk5q1a4rGMHKOSpC4qaDKFY1SSpFYzo5Kk\nLiooozJQSVIXFdSfVlBVJUldtLIzqqYWpnUldmn5jLoK+oYR/967uup6LLXv74FGqjGMlR2oRll1\nfSGuxC5pxVlq82+gkiSN1VKb/x80UothOEYlSWo1MypJ6qRymv9yaipJalA5zX85NZUkNaic5t8x\nKklSq7V/m49165bnNyFJnbJmicfyaf82H3fdtTy/CUnqlHICVTmdlJKkBpWzKq1jVJKkVjOjkqRO\nKqf5L6emkqQGldP82/UnSZ3U/GSKiDgnIu6IiOv7PTkinhYR90fErw9b0/YbZfXytWubr4ek9urq\ndh3tci7wHuC8XhdExCrg7cClwxZaRqBqarsOSVKt+eY/M6+IiEMHXPYG4CLgacOWW0agkiQ1bPmb\n/4j4CeBFmXl8RDx92PsMVJLUSYtr/mdmvsfMzPeW+tB3A6fP+XmocZ3I/t1qk+9zi5h8118b6rAI\nParrNsVqvYjIAW1Su2xY3j+ruBAyc8kPrX7Pz1xiGZ9bsC51199HM/PIBc7dPPsSeAywC/jtzLyk\n37PMqCSpk8bW/Ac9/sc4M5/ww4sizqUKaH2DFBioJKmjmm/+I+ICYArYPyJuAzYCewCZmZvnXT50\n6ty/ptPTsGnToirauBZMM59mI5vGkOGvXQs7dzZfriQNNpZZfxsWce2rhr22/WNUbTCmMapxDX05\nRqVSOUbVX7NjVL+yxDIua6Quw7DrT5I6qZzmv5yaSpIaVE7z71p/kqRWKyekSpIaVE7zX05NJUkN\nKqf5L6emktTPMs/AW5JRV3q/sMnPWE7z7xiVJKnVygmpkqQGldP8l1NTSUWIiHOAFwB3zC5MGhF/\nDRxRX7IWuCszj5lQFQXA6klXYGgGKklN+5FdXjPzZbOvI+IdwN0TqJceppzmv5yaSirCELu8vgQ4\nfrnqo/IZqCQtm4h4FrAjM78+6bqonOa/jNXTXWJcWilOAi7sd8H09PQPX09NTTE1NTXeGrXYzMwM\nMzMzYyq9nEDV/tXT27C7rqunS4uy0C6vEbEa2A4ck5nf7HHf6Kund+B7VBHR4Orpv7/EMt61bKun\n+z0qSeOw0C6vvwzc1CtISb0YqCQ1qt7l9XPAERFxW0ScUp96KQO6/bSc1izxWN6aSlJjeu3ympmn\nLPS+JqWc5r+cmkqSGlRO82/XnySp1coJqZKkBpXT/JdR05jwtNO1ayf7fEmDjbp1RknT2htVRvMP\npdR00t+jkqQVp4zmHxyjkiS1XDkhdYUaR6+mPZWSBiun+TejmrDM5g+XRpQ0WPNf+I2IcyLijoi4\nvsf5DRHxpfq4IiKeMkxNDVSS1EljWZniXOCEPg+9GXh2Zj4VOBP4wLA1lSRpyQbtRZaZV8/58Wrg\n4GHK7Z9RTU9XgyiTPBxwkaQxmPhaf68BPjFsTXubnq4OSdIKs7hgMzOzjZmZrzTy5Ig4HjgFWD/U\n9a3fj2oFW+attrr6rUYVZEn7UY1qEl/4bcV+VOcssYxXL1iXhfYim3f+SODDwInD7vTsGJUkddLq\ncRW80F5eqLOqAAAC1ElEQVRk1YmIx1MFqZcPG6TAQCVJHdV881/vRTYF7B8RtwEbgT2AzMzNwB8D\n64C/iIgA7s/Mpy9/TSVJBWi++e+1F9mc86cCpy62XL9HJUlqNTMqSa2y94jriu1y8epFKqf5L6em\nkqQGldP8l1NTSVKDymn+HaOSJLVaOSFVktSgcpr/cmoqSWpQOc1/OTWVJDWonOa/b02nZ6bZ9I+b\nlqsuC1r7yLXsPN2dACWpq1q/KG1sCnLjxKsxFi5KKz1cROReI9478veoOrso7RVLLGN9I3UZRjm5\nnySpQeU0/+XUVJLUoHKaf79HJUlqtXJCqiSpQeU0/+XUVJLUoHKa//bX9PKNxPSkKzEea9dOugaS\nuqv9zf+s9tf0+E3kzPSkayFpmew6adI1UNu0P1BJksagnOa/nJpKkhq0etIVGJqBSpI6qZzm3+9R\nSZJarZyQKklqUDnNfzk1lSQ1qJzmv4htPiRJ3dX6bT7UGLf5UOtFROao36MaceuM7m7zcdcSy1jr\nNh+SpHEqp/l31p+kYszcMeJ9MzPL+ryl3DtqXRdvzRKPHxUR50TEHRFxfa+nRsTZEfHViNgSEUcN\nU1MDlaRizHx7xPtGDVQjPm8p9y5foBqLc4ETep2MiOcBT8zMJwGvBd43TKHl5H6SpAY13/xn5hUR\ncWifS14InFdfe01E7BcRB2Zm3/zTQCVJnTSR5v9g4F/n/Ly9fq9voBo060+Slk1E2CAN0NCsv28A\n/TKfYdyRmQctUPahwEcz88gFzn0UeFtmfq7++dPAH2TmF/s9yIxKUmss13TnrsvMwyb06O3A4+b8\nfEj9Xl9OppAkNSno/b3NS4BXAETEscDdg8anwIxKktSQiLgAmAL2j4jbgI3AHkBm5ubM/HhEPD8i\nvgbsAk4ZqlzHqCRJbWbXnySp1QxUkqRWM1BJklrNQCVJajUDlSSp1QxUkqRWM1BJklrt/wPsvWJp\ntc7rsAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xa0ba278>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot row dendrogram\n",
    "fig = plt.figure(figsize=(8,8))\n",
    "axd = fig.add_axes([0.09,0.1,0.2,0.6])\n",
    "row_dendr = dendrogram(row_clusters, orientation='right')\n",
    "\n",
    "# reorder data with respect to clustering\n",
    "df_rowclust = df.ix[row_dendr['leaves'][::-1]]\n",
    "\n",
    "axd.set_xticks([])\n",
    "axd.set_yticks([])\n",
    "\n",
    "# remove axes spines from dendrogram\n",
    "for i in axd.spines.values():\n",
    "        i.set_visible(False)\n",
    "\n",
    "\n",
    "        \n",
    "# plot heatmap\n",
    "axm = fig.add_axes([0.23,0.1,0.6,0.6]) # x-pos, y-pos, width, height\n",
    "cax = axm.matshow(df_rowclust, interpolation='nearest', cmap='hot_r')\n",
    "fig.colorbar(cax)\n",
    "axm.set_xticklabels([''] + list(df_rowclust.columns))\n",
    "axm.set_yticklabels([''] + list(df_rowclust.index))\n",
    "\n",
    "# plt.savefig('./figures/heatmap.png', dpi=300)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying agglomerative clustering via scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cluster labels: [2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cluster import AgglomerativeClustering\n",
    "\n",
    "ac = AgglomerativeClustering(n_clusters=3, affinity='euclidean', linkage='complete')\n",
    "labels = ac.fit_predict(X)\n",
    "print('Cluster labels: %s' % labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "X = df[features]\n",
    "y = df['Target-Lenses']\n",
    "X_train, X_test, y_train, y_test = train_test_split(X.values, y.values ,test_size=0.25, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(18L,)\n",
      "[2 1 2 1 1 2 0 1 1 0 2 2 0 1 0 0 0 1]\n"
     ]
    }
   ],
   "source": [
    "from sklearn import cluster\n",
    "clf = cluster.KMeans(init='k-means++', n_clusters=3, random_state=5)\n",
    "clf.fit(X_train)\n",
    "print clf.labels_.shape\n",
    "print clf.labels_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Predict clusters on testing data\n",
    "y_pred = clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Addjusted rand score:-0.18\n",
      "Homogeneity score:0.37 \n",
      "Completeness score: 0.31 \n",
      "Confusion matrix\n",
      "[[0 0 0 0]\n",
      " [1 0 0 0]\n",
      " [0 0 1 0]\n",
      " [1 1 2 0]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "print \"Addjusted rand score:{:.2}\".format(metrics.adjusted_rand_score(y_test, y_pred))\n",
    "print \"Homogeneity score:{:.2} \".format(metrics.homogeneity_score(y_test, y_pred)) \n",
    "print \"Completeness score: {:.2} \".format(metrics.completeness_score(y_test, y_pred))\n",
    "print \"Confusion matrix\"\n",
    "print metrics.confusion_matrix(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Affinity Propogation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4L,)\n"
     ]
    }
   ],
   "source": [
    "# Affinity propagation\n",
    "aff = cluster.AffinityPropagation()\n",
    "aff.fit(X_train)\n",
    "print aff.cluster_centers_indices_.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "y_pred = aff.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Addjusted rand score:0.062\n",
      "Homogeneity score:0.73 \n",
      "Completeness score: 0.48 \n",
      "Confusion matrix\n",
      "[[0 0 0 0]\n",
      " [0 1 0 0]\n",
      " [1 0 0 0]\n",
      " [0 1 2 1]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "print \"Addjusted rand score:{:.2}\".format(metrics.adjusted_rand_score(y_test, y_pred))\n",
    "print \"Homogeneity score:{:.2} \".format(metrics.homogeneity_score(y_test, y_pred)) \n",
    "print \"Completeness score: {:.2} \".format(metrics.completeness_score(y_test, y_pred))\n",
    "print \"Confusion matrix\"\n",
    "print metrics.confusion_matrix(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.41666667  1.66666667  1.58333333  1.5       ]\n",
      " [ 3.          1.5         1.5         2.        ]]\n"
     ]
    }
   ],
   "source": [
    "#MeanShift\n",
    "ms = cluster.MeanShift()\n",
    "ms.fit(X_train)\n",
    "print ms.cluster_centers_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_pred = ms.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Addjusted rand score:-0.22\n",
      "Homogeneity score:0.2 \n",
      "Completeness score: 0.27 \n",
      "Confusion matrix\n",
      "[[0 0 0 0]\n",
      " [1 0 0 0]\n",
      " [1 0 0 0]\n",
      " [2 2 0 0]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "print \"Addjusted rand score:{:.2}\".format(metrics.adjusted_rand_score(y_test, y_pred))\n",
    "print \"Homogeneity score:{:.2} \".format(metrics.homogeneity_score(y_test, y_pred)) \n",
    "print \"Completeness score: {:.2} \".format(metrics.completeness_score(y_test, y_pred))\n",
    "print \"Confusion matrix\"\n",
    "print metrics.confusion_matrix(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mixture of Guassian Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Adjusted rand score for covariance=spherical:-0.25\n",
      "Adjusted rand score for covariance=tied:0.0\n",
      "Adjusted rand score for covariance=diag:-0.15\n",
      "Adjusted rand score for covariance=full:-0.25\n"
     ]
    }
   ],
   "source": [
    "from sklearn import mixture\n",
    "\n",
    "# Define a heldout dataset to estimate covariance type\n",
    "X_train_heldout, X_test_heldout, y_train_heldout, y_test_heldout = train_test_split(\n",
    "        X_train, y_train,test_size=0.25, random_state=42)\n",
    "for covariance_type in ['spherical','tied','diag','full']:\n",
    "    gm=mixture.GMM(n_components=3, covariance_type=covariance_type, random_state=42, n_init=5)\n",
    "    gm.fit(X_train_heldout)\n",
    "    y_pred=gm.predict(X_test_heldout)\n",
    "    print \"Adjusted rand score for covariance={}:{:.2}\".format(covariance_type, metrics.adjusted_rand_score(y_test_heldout, y_pred))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GMM(covariance_type='tied', init_params='wmc', min_covar=0.001,\n",
       "  n_components=3, n_init=1, n_iter=100, params='wmc', random_state=42,\n",
       "  thresh=None, tol=0.001, verbose=0)"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gm = mixture.GMM(n_components=3, covariance_type='tied', random_state=42)\n",
    "gm.fit(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Addjusted rand score:-0.18\n",
      "Homogeneity score:0.37 \n",
      "Completeness score: 0.31 \n",
      "Confusion matrix\n",
      "[[0 0 0 0]\n",
      " [0 1 0 0]\n",
      " [0 1 0 0]\n",
      " [1 1 2 0]]\n"
     ]
    }
   ],
   "source": [
    "# Print train clustering and confusion matrix\n",
    "y_pred = gm.predict(X_test)\n",
    "print \"Addjusted rand score:{:.2}\".format(metrics.adjusted_rand_score(y_test, y_pred))\n",
    "print \"Homogeneity score:{:.2} \".format(metrics.homogeneity_score(y_test, y_pred)) \n",
    "print \"Completeness score: {:.2} \".format(metrics.completeness_score(y_test, y_pred))\n",
    "\n",
    "print \"Confusion matrix\"\n",
    "print metrics.confusion_matrix(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}