Ensemble Learning to Classify Patients with Heart Disease.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ensemble Learning to Classify Patients with Heart Disease\n",
    "<hr>\n",
    "## Learning with ensembles\n",
    "\n",
    "The goal behind ensemble methods is to combine different classifiers into a\n",
    "meta-classifier that has a better generalization performance than each individual classifier alone. For example, assuming that we collected predictions from 10 experts, ensemble methods would allow us to strategically combine these predictions by the 10 experts to come up with a prediction that is more accurate and robust than the predictions by each individual expert. As we will see later in this chapter, there are several different approaches for creating an ensemble of classifiers. In this section, we will introduce a basic perception about how ensembles work and why they are typically recognized for yielding a good generalization performance.\n",
    "\n",
    "In this chapter, we will focus on the most popular ensemble methods that use the majority voting principle. Majority voting simply means that we select the class label that has been predicted by the majority of classifiers, that is, received more than 50 percent of the votes. Strictly speaking, the term majority vote refers to binary class settings only. However, it is easy to generalize the majority voting principle to multi-class settings, which is called plurality voting. Here, we select the class label that received the most votes (mode). The following diagram illustrates the concept of majority and plurality voting for an ensemble of 10 classifiers where each unique symbol (triangle, square, and circle) represents a unique class label:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<img src = \"images/ensemble.png\" > "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the training set, we start by training m different classifiers (1,,mCC…). Depending on the technique, the ensemble can be built from different classification algorithms, for example, decision trees, support vector machines, logistic regression classifiers, and so on. Alternatively, we can also use the same base classification algorithm fitting different subsets of the training set. One prominent example of this approach would be the random forest algorithm, which combines different decision tree classifiers. The following diagram illustrates the concept of a general ensemble approach using majority voting:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<img src =\"images/majorityvoting.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[[Images source]](https://github.com/rasbt/python-machine-learning-book/tree/master/images/image_gallery)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementing a simple majority vote classifier \n",
    "\n",
    "After the short introduction to ensemble learning in the previous section, let's start with a warm-up exercise and implement a simple ensemble classifier for majority voting in Python.\n",
    "\n",
    "The algorithm that we are going to implement will allow us to combine different classification algorithms associated with individual weights for confidence. Our goal is to build a stronger meta-classifier that balances out the individual classifiers' weaknesses on a particular dataset.\n",
    "\n",
    "Let's assume that we have an ensemble of three base classifiers c1,c2,c3\u001c",
    " and want to predict the class label of a given sample instance x. Two out of three base classifiers predict the class label 0, and one c3 predicts that the sample belongs to class 1.\n",
    "If we weight the predictions of each base classifier equally, the majority vote will predict that the sample belongs to class 0.\n",
    "\n",
    "Now let's assign a weight of 0.6 to c3 and weight c1 and c2 by a coefficient of 0.2, respectively.\n",
    "\n",
    "More intuitively, since 3 x 0.2 = 0.6, we can say that the prediction made by c3 has three times more weight than the predictions by c1 or c2 (), respectively.To translate the concept of the weighted majority vote into Python code, we can use NumPy's convenient argmax and bincount functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "np.argmax(np.bincount([0, 0, 1], \n",
    "                      weights=[0.2, 0.2, 0.6]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Certain classifiers in scikit-learn can also return the probability of a predicted class label via the predict_proba method. Using the predicted class probabilities instead of the class labels for majority voting can be useful if the classifiers in our ensemble are well calibrated.\n",
    "\n",
    "Let's assume that our classifiers c1,c2,c3 returns the following class membership probabilities for a particular sample x:\n",
    "c1(x) -> [0.9,0.1] \n",
    "c2(x) -> [0.8,0.2] \n",
    "c3(x) -> [0.4,0.6] \n",
    "\n",
    "To implement the weighted majority vote based on class probabilities, we can again make use of NumPy using numpy.average and np.argmax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.58,  0.42])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex = np.array([[0.9, 0.1],\n",
    "               [0.8, 0.2],\n",
    "               [0.4, 0.6]])\n",
    "\n",
    "p = np.average(ex, \n",
    "               axis=0, \n",
    "               weights=[0.2, 0.2, 0.6])\n",
    "p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.argmax(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Putting everything together, let's now implement a MajorityVoteClassifier\n",
    "in Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.base import BaseEstimator\n",
    "from sklearn.base import ClassifierMixin\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.externals import six\n",
    "from sklearn.base import clone\n",
    "from sklearn.pipeline import _name_estimators\n",
    "import numpy as np\n",
    "import operator\n",
    "\n",
    "\n",
    "class MajorityVoteClassifier(BaseEstimator, \n",
    "                             ClassifierMixin):\n",
    "    \"\"\" A majority vote ensemble classifier\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    classifiers : array-like, shape = [n_classifiers]\n",
    "      Different classifiers for the ensemble\n",
    "\n",
    "    vote : str, {'classlabel', 'probability'} (default='label')\n",
    "      If 'classlabel' the prediction is based on the argmax of\n",
    "        class labels. Else if 'probability', the argmax of\n",
    "        the sum of probabilities is used to predict the class label\n",
    "        (recommended for calibrated classifiers).\n",
    "\n",
    "    weights : array-like, shape = [n_classifiers], optional (default=None)\n",
    "      If a list of `int` or `float` values are provided, the classifiers\n",
    "      are weighted by importance; Uses uniform weights if `weights=None`.\n",
    "\n",
    "    \"\"\"\n",
    "    def __init__(self, classifiers, vote='classlabel', weights=None):\n",
    "\n",
    "        self.classifiers = classifiers\n",
    "        self.named_classifiers = {key: value for key, value\n",
    "                                  in _name_estimators(classifiers)}\n",
    "        self.vote = vote\n",
    "        self.weights = weights\n",
    "\n",
    "    def fit(self, X, y):\n",
    "        \"\"\" Fit classifiers.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        X : {array-like, sparse matrix}, shape = [n_samples, n_features]\n",
    "            Matrix of training samples.\n",
    "\n",
    "        y : array-like, shape = [n_samples]\n",
    "            Vector of target class labels.\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        self : object\n",
    "\n",
    "        \"\"\"\n",
    "        if self.vote not in ('probability', 'classlabel'):\n",
    "            raise ValueError(\"vote must be 'probability' or 'classlabel'\"\n",
    "                             \"; got (vote=%r)\"\n",
    "                             % self.vote)\n",
    "\n",
    "        if self.weights and len(self.weights) != len(self.classifiers):\n",
    "            raise ValueError('Number of classifiers and weights must be equal'\n",
    "                             '; got %d weights, %d classifiers'\n",
    "                             % (len(self.weights), len(self.classifiers)))\n",
    "\n",
    "        # Use LabelEncoder to ensure class labels start with 0, which\n",
    "        # is important for np.argmax call in self.predict\n",
    "        self.lablenc_ = LabelEncoder()\n",
    "        self.lablenc_.fit(y)\n",
    "        self.classes_ = self.lablenc_.classes_\n",
    "        self.classifiers_ = []\n",
    "        for clf in self.classifiers:\n",
    "            fitted_clf = clone(clf).fit(X, self.lablenc_.transform(y))\n",
    "            self.classifiers_.append(fitted_clf)\n",
    "        return self\n",
    "\n",
    "    def predict(self, X):\n",
    "        \"\"\" Predict class labels for X.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        X : {array-like, sparse matrix}, shape = [n_samples, n_features]\n",
    "            Matrix of training samples.\n",
    "\n",
    "        Returns\n",
    "        ----------\n",
    "        maj_vote : array-like, shape = [n_samples]\n",
    "            Predicted class labels.\n",
    "            \n",
    "        \"\"\"\n",
    "        if self.vote == 'probability':\n",
    "            maj_vote = np.argmax(self.predict_proba(X), axis=1)\n",
    "        else:  # 'classlabel' vote\n",
    "\n",
    "            #  Collect results from clf.predict calls\n",
    "            predictions = np.asarray([clf.predict(X)\n",
    "                                      for clf in self.classifiers_]).T\n",
    "\n",
    "            maj_vote = np.apply_along_axis(\n",
    "                                      lambda x:\n",
    "                                      np.argmax(np.bincount(x,\n",
    "                                                weights=self.weights)),\n",
    "                                      axis=1,\n",
    "                                      arr=predictions)\n",
    "        maj_vote = self.lablenc_.inverse_transform(maj_vote)\n",
    "        return maj_vote\n",
    "\n",
    "    def predict_proba(self, X):\n",
    "        \"\"\" Predict class probabilities for X.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        X : {array-like, sparse matrix}, shape = [n_samples, n_features]\n",
    "            Training vectors, where n_samples is the number of samples and\n",
    "            n_features is the number of features.\n",
    "\n",
    "        Returns\n",
    "        ----------\n",
    "        avg_proba : array-like, shape = [n_samples, n_classes]\n",
    "            Weighted average probability for each class per sample.\n",
    "\n",
    "        \"\"\"\n",
    "        probas = np.asarray([clf.predict_proba(X)\n",
    "                             for clf in self.classifiers_])\n",
    "        avg_proba = np.average(probas, axis=0, weights=self.weights)\n",
    "        return avg_proba\n",
    "\n",
    "    def get_params(self, deep=True):\n",
    "        \"\"\" Get classifier parameter names for GridSearch\"\"\"\n",
    "        if not deep:\n",
    "            return super(MajorityVoteClassifier, self).get_params(deep=False)\n",
    "        else:\n",
    "            out = self.named_classifiers.copy()\n",
    "            for name, step in six.iteritems(self.named_classifiers):\n",
    "                for key, value in six.iteritems(step.get_params(deep=True)):\n",
    "                    out['%s__%s' % (name, key)] = value\n",
    "            return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A lot of comments are added to the code to better understand the individual parts. However, before we implement the remaining methods, let's take a quick break and discuss some of the code that may look confusing at first. We used the parent classes BaseEstimator and ClassifierMixin to get some base functionality for free, including the methods get_params and set_params to set and return the classifier's parameters as well as the score method to calculate the prediction accuracy, respectively. Also note that we imported six to make the MajorityVoteClassifier compatible with Python 2.7.\n",
    "\n",
    "Next we will add the predict method to predict the class label via majority vote based on the class labels if we initialize a new MajorityVoteClassifier object with vote='classlabel'. Alternatively, we will be able to initialize the ensemble classifier with vote='probability' to predict the class label based on the class membership probabilities. Furthermore, we will also add a predict_proba method to return the average probabilities, which is useful to compute the Receiver Operator Characteristic area under the curve (ROC AUC).\n",
    "\n",
    "Also, note that we defined our own modified version of the 'get_params' methods to use the '_name_estimators' function in order to access the parameters of individual classifiers in the ensemble. This may look a little bit complicated at first, but it will make perfect sense when we use grid search for hyperparameter-tuning in\n",
    "later sections."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Combining different algorithms for classification with majority vote\n",
    "\n",
    "Now it is about time to put the MajorityVoteClassifier that we implemented in the previous section into action. But first, let's prepare a dataset that we can test it on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Heart Disease Data Set \n",
    "\n",
    "We would using the Heart Disease Data Set. The dataset actually contains 76 attributes but for simplicity we would be dealing with only the most important 14 attributes. Using this 14 attributes, our job would be to predict if the patient has a heart disease or not.\n",
    "\n",
    "#### Attribute Information:\n",
    "\n",
    "The 14 attributes used: \n",
    "[age sex cp trestbps chol fbs restecg thalach exang oldpeak slope ca thal num]\n",
    "\n",
    "Complete attribute documentation: \n",
    "1. age: age in years \n",
    "2. sex: sex (1 = male; 0 = female) \n",
    "3. cp: chest pain type \n",
    "    - Value 1: typical angina \n",
    "    - Value 2: atypical angina \n",
    "    - Value 3: non-anginal pain \n",
    "    - Value 4: asymptomatic \n",
    "4. trestbps: resting blood pressure (in mm Hg on admission to the hospital) \n",
    "5. chol: serum cholestoral in mg/dl \n",
    "6. fbs: (fasting blood sugar > 120 mg/dl) (1 = true; 0 = false) \n",
    "7. restecg: resting electrocardiographic results \n",
    "    - Value 0: normal \n",
    "    - Value 1: having ST-T wave abnormality (T wave inversions and/or ST elevation or depression of > 0.05 mV) \n",
    "    - Value 2: showing probable or definite left ventricular hypertrophy by Estes' criteria \n",
    "8. thalach: maximum heart rate achieved \n",
    "9. exang: exercise induced angina (1 = yes; 0 = no) \n",
    "10. oldpeak = ST depression induced by exercise relative to rest \n",
    "11. slope: the slope of the peak exercise ST segment \n",
    "    - Value 1: upsloping \n",
    "    - Value 2: flat \n",
    "    - Value 3: downsloping \n",
    "12. ca: number of major vessels (0-3) colored by flourosopy \n",
    "13. thal: 3 = normal; 6 = fixed defect; 7 = reversable defect \n",
    "14. num: diagnosis of heart disease (angiographic disease status) \n",
    "\n",
    "We would be using the preprocessed dataset. It can be downloaded from here : http://archive.ics.uci.edu/ml/machine-learning-databases/heart-disease/processed.cleveland.data\n",
    "\n",
    "Once downloaded, we can move on to code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\hp\\appdata\\local\\programs\\python\\python35\\lib\\site-packages\\sklearn\\cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n",
      "c:\\users\\hp\\appdata\\local\\programs\\python\\python35\\lib\\site-packages\\sklearn\\grid_search.py:43: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. This module will be removed in 0.20.\n",
      "  DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from sklearn.tree import DecisionTreeClassifier, export_graphviz\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn import cross_validation, metrics\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.naive_bayes import BernoulliNB\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.svm import SVC\n",
    "from time import time\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.metrics import roc_auc_score , classification_report\n",
    "from sklearn.grid_search import GridSearchCV\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.metrics import precision_score, recall_score, accuracy_score, classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cols = ['age',\n",
    " 'sex',\n",
    " 'cp',\n",
    " 'trestbps',\n",
    " 'chol',\n",
    " 'fbs',\n",
    " 'restecg',\n",
    " 'thalach',\n",
    " 'exang',\n",
    " 'oldpeak',\n",
    " 'slope',\n",
    " 'ca',\n",
    " 'thal',\n",
    " 'num']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "# read .csv from provided dataset\n",
    "csv_filename=\"processed.cleveland.data\"\n",
    "\n",
    "# Seperator = ' ' i.e a single space\n",
    "df=pd.read_csv(csv_filename,sep=',',names=cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thalach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slope</th>\n",
       "      <th>ca</th>\n",
       "      <th>thal</th>\n",
       "      <th>num</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>63.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>145.0</td>\n",
       "      <td>233.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>67.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>160.0</td>\n",
       "      <td>286.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>108.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>67.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>229.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>129.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.6</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>37.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>250.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>187.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>41.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>204.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>172.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    age  sex   cp  trestbps   chol  fbs  restecg  thalach  exang  oldpeak  \\\n",
       "0  63.0  1.0  1.0     145.0  233.0  1.0      2.0    150.0    0.0      2.3   \n",
       "1  67.0  1.0  4.0     160.0  286.0  0.0      2.0    108.0    1.0      1.5   \n",
       "2  67.0  1.0  4.0     120.0  229.0  0.0      2.0    129.0    1.0      2.6   \n",
       "3  37.0  1.0  3.0     130.0  250.0  0.0      0.0    187.0    0.0      3.5   \n",
       "4  41.0  0.0  2.0     130.0  204.0  0.0      2.0    172.0    0.0      1.4   \n",
       "\n",
       "   slope   ca thal  num  \n",
       "0    3.0  0.0  6.0    0  \n",
       "1    2.0  3.0  3.0    2  \n",
       "2    2.0  2.0  7.0    1  \n",
       "3    3.0  0.0  3.0    0  \n",
       "4    1.0  0.0  3.0    0  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus we have our dataset. But we want our task to be a binary classification task i.e we would like to classify whether the patient has a heart disease or not. However our target variable 'num' contains 5 values 0,1,2,3,4. We would simply attempt to distinguish presence (values 1,2,3,4) from absence (value 0). We can make clean our target variable values accordingly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "164\n"
     ]
    }
   ],
   "source": [
    "count0 = 0\n",
    "for z in df['num']:\n",
    "    if z == 0:\n",
    "        count0 = count0 + 1\n",
    "print (count0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for v in df['num']:\n",
    "    if v != 0 :\n",
    "        df['num'].replace(v,1,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "164\n"
     ]
    }
   ],
   "source": [
    "count0 = 0\n",
    "for z in df['num']:\n",
    "    if z == 0:\n",
    "        count0 = count0 + 1\n",
    "print (count0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "139\n"
     ]
    }
   ],
   "source": [
    "count0 = 0\n",
    "for z in df['num']:\n",
    "    if z != 0:\n",
    "        count0 = count0 + 1\n",
    "print (count0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thalach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slope</th>\n",
       "      <th>ca</th>\n",
       "      <th>thal</th>\n",
       "      <th>num</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>63.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>145.0</td>\n",
       "      <td>233.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>67.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>160.0</td>\n",
       "      <td>286.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>108.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>67.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>229.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>129.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.6</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>37.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>250.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>187.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>41.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>204.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>172.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    age  sex   cp  trestbps   chol  fbs  restecg  thalach  exang  oldpeak  \\\n",
       "0  63.0  1.0  1.0     145.0  233.0  1.0      2.0    150.0    0.0      2.3   \n",
       "1  67.0  1.0  4.0     160.0  286.0  0.0      2.0    108.0    1.0      1.5   \n",
       "2  67.0  1.0  4.0     120.0  229.0  0.0      2.0    129.0    1.0      2.6   \n",
       "3  37.0  1.0  3.0     130.0  250.0  0.0      0.0    187.0    0.0      3.5   \n",
       "4  41.0  0.0  2.0     130.0  204.0  0.0      2.0    172.0    0.0      1.4   \n",
       "\n",
       "   slope   ca thal  num  \n",
       "0    3.0  0.0  6.0    0  \n",
       "1    2.0  3.0  3.0    1  \n",
       "2    2.0  2.0  7.0    1  \n",
       "3    3.0  0.0  3.0    0  \n",
       "4    1.0  0.0  3.0    0  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our data contains 6 row with missing values. These values are represented by \"?\". So first we replace these \"?\" with NaN and then drop all rows which contain NaNs. We can simply achive this by doing the following: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.replace(\"?\",np.NaN,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.dropna(axis=0, inplace=True, how='any')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df = df.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thalach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slope</th>\n",
       "      <th>ca</th>\n",
       "      <th>thal</th>\n",
       "      <th>num</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>63.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>145.0</td>\n",
       "      <td>233.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>67.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>160.0</td>\n",
       "      <td>286.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>108.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>67.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>229.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>129.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.6</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>37.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>250.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>187.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>41.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>204.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>172.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    age  sex   cp  trestbps   chol  fbs  restecg  thalach  exang  oldpeak  \\\n",
       "0  63.0  1.0  1.0     145.0  233.0  1.0      2.0    150.0    0.0      2.3   \n",
       "1  67.0  1.0  4.0     160.0  286.0  0.0      2.0    108.0    1.0      1.5   \n",
       "2  67.0  1.0  4.0     120.0  229.0  0.0      2.0    129.0    1.0      2.6   \n",
       "3  37.0  1.0  3.0     130.0  250.0  0.0      0.0    187.0    0.0      3.5   \n",
       "4  41.0  0.0  2.0     130.0  204.0  0.0      2.0    172.0    0.0      1.4   \n",
       "\n",
       "   slope   ca thal  num  \n",
       "0    3.0  0.0  6.0    0  \n",
       "1    2.0  3.0  3.0    1  \n",
       "2    2.0  2.0  7.0    1  \n",
       "3    3.0  0.0  3.0    0  \n",
       "4    1.0  0.0  3.0    0  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can move on to classification of our data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "features = df.columns[:-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['age', 'sex', 'cp', 'trestbps', 'chol', 'fbs', 'restecg', 'thalach',\n",
       "       'exang', 'oldpeak', 'slope', 'ca', 'thal'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = df[features]\n",
    "y = df['num']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1], dtype=int64)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our target variable y has 2 unique values 0  and 1. 0 means the patient doesn't have a heart disease; 1 means unfortunately he/she does. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We split our dataset into 70% training set and 30% testing set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "X_train, X_test, y_train, y_test =\\\n",
    "       train_test_split(X, y, \n",
    "                        test_size=0.3, \n",
    "                        random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((207, 13), (207,))"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " X_train.shape, y_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature importances with forests of trees\n",
    "This examples shows the use of forests of trees to evaluate the importance of features on an artificial classification task. The red bars are the feature importances of the forest, along with their inter-trees variability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "\n",
    "# Build a classification task using 3 informative features\n",
    "\n",
    "# Build a forest and compute the feature importances\n",
    "forest = ExtraTreesClassifier(n_estimators=250,\n",
    "                              random_state=0)\n",
    "\n",
    "forest.fit(X, y)\n",
    "importances = forest.feature_importances_\n",
    "std = np.std([tree.feature_importances_ for tree in forest.estimators_],\n",
    "             axis=0)\n",
    "indices = np.argsort(importances)[::-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. feature 12 - thal (0.167814)\n",
      "2. feature 11 - ca (0.124068)\n",
      "3. feature 2 - cp (0.110151)\n",
      "4. feature 7 - thalach (0.083754)\n",
      "5. feature 8 - exang (0.082387)\n"
     ]
    }
   ],
   "source": [
    "for f in range(5):\n",
    "    print(\"%d. feature %d - %s (%f)\" % (f + 1, indices[f], features[indices[f]] ,importances[indices[f]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqQAAAINCAYAAADoVW2PAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3X1c1eXh//H3OYDcKokgeC+aN6B5R5k30zRJ0qW5aSyM\nL2Kb05HdqK2yGwwrU3NRTs2ac97U0Fybbd9VVqS25U1O6OabyDTv0uKsY6lLRBGu3x/+OOsMMA6h\nF+jr+Xicx+A61/mc68PH2IvPuXMYY4wAAAAAS5y2FwAAAIDLG0EKAAAAqwhSAAAAWEWQAgAAwCqC\nFAAAAFYRpAAAALCKIAUAAIBVBCkAAACsIkgBAABgFUEKAHXk4MGDcjqdWrVqle2lAECDQpACqJWV\nK1fK6XRWeXnwwQcv2P2+/vrrysrKumDb/74cDoftJXwvzz33nFauXGl7GQAuM/62FwCg4XI4HHrs\nscfUvn17r/Hu3btfsPt87bXXtGTJEs2aNeuC3UdttWvXTqdOnVJAQIDtpdTakiVLFBUVpQkTJthe\nCoDLCEEK4Hu58cYb1adPn4t2f8aYC7Ld4uJihYSEfO/tNGrUqA5Wc/GdOnVKwcHBtpcB4DLFQ/YA\nLrgXX3xRV199tUJCQtSsWTOlpKTo8OHDXnP+/ve/Kzk5We3atVNQUJDatm2r6dOnq6SkxDNn4sSJ\nWrJkiSR5nh7g5+cnSdq0aZOcTqfeffddr+1W9bzO9PR0NW7cWPv27dPIkSPVpEkTpaameq7fvn27\nbrzxRl1xxRUKDQ3VkCFDtGXLlu/cz/Pd12effaabbrpJjRs3VuvWrT378fHHH2vYsGEKCwtT+/bt\nlZOT47XNiqdG/O1vf9PkyZMVGRmp8PBwTZgwQceOHau0hiVLlqh79+4KCgpSq1atNHXqVB0/ftxr\nzpAhQ9SjRw/l5eVp8ODBCg0N1YMPPqjY2Fh98sknnp+l0+nU9ddfL0n6+uuvde+996pHjx5q3Lix\nwsPDNXLkSH300Ude2968ebOcTqfWrVunJ554Qm3atFFwcLASExP16aefVlrv9u3bNXLkSEVERCgs\nLEw9e/bUwoULveYUFhZq3LhxatasmYKDg3XNNdfoL3/5i9ecs2fPKisrS507d1ZwcLAiIyM1aNAg\n5ebmftdhA1APcIYUwPdy/PhxHT161GusWbNmnq+feOIJZWZm6tZbb9WkSZP05ZdfauHChbruuuuU\nn5+vJk2aSJLWrVunU6dOKSMjQ82aNdP777+vX//61zpy5IjWrl0rSZoyZYo+//xzvf3223rppZe8\nzpY6HI4aP3/T4XDo7NmzSkpK0qBBg/SrX/3Kc3b0nXfe0ciRI3X11Vfr0UcfldPp1O9+9ztdf/31\n+vvf/66rr77ap5+Pw+FQeXm5RowYoeuuu05PPfWUXnrpJd15550KDQ3VQw89pNTUVI0dO1ZLly7V\nhAkTNGDAALVr185rO1OnTlXTpk2VlZWlwsJCLVmyRIcOHdLGjRs9cx599FHNnj1bw4cPV0ZGhmfe\nP/7xD7333nueeHc4HHK73Ro5cqRuvfVWpaWlKTo6WkOHDtXUqVPVuHFjPfzwwzLGKDo6WpK0b98+\n/fnPf9Ytt9yi2NhYuVwuPf/88xoyZIh27dqlmJgYr/XOnTtXfn5++uUvf6njx49r3rx5Sk1N1dat\nWz1z3nrrLY0aNUotW7bUPffco5iYGBUUFOivf/2r7rrrLknSJ598oh/84Adq3bq1Zs6cqdDQUL38\n8ssaM2aM/vjHP+rmm2+WJM2aNUtz587Vz3/+c11zzTU6ceKE/vGPfygvL0/Dhg3z6ZgBsMAAQC2s\nWLHCOByOShen0+mZc/DgQePv72/mzp3rddtPPvnEBAQEmCeffNIzVlJSUuk+5s6da/z8/Mxnn33m\nGZs6darXfVTYtGmTcTqdZvPmzV7jBw4cMA6Hw6xcudIzlp6ebpxOp3nooYcqbadz585m5MiRXmMl\nJSWmQ4cOJikpqbofx3fe17x58zxjx44dMyEhIcbPz8+sW7fOM15YWGgcDofJysryjFX8nPv27WvO\nnj3rGX/qqaeM0+k0f/nLX4wxxnz55ZcmMDDQjBgxwmtNixcvNk6n06xYscIzNmTIEON0Os1vfvOb\nSvvQvXt3M3To0ErjZ86cqTR28OBBExQUZB5//HHP2KZNm4zD4TDdunXzWu/ChQuN0+k0n3zyiTHG\nmLKyMhMbG2s6dOhgTpw4UWnbFYYNG2Z69eplSktLvcYHDhxounTp4vm+V69eZtSoUdVuB0D9xkP2\nAGrN4XDoueee09tvv+25vPXWW57rX3nlFRljdMstt+jo0aOeS/PmzdWpUyevs3uBgYGer4uLi3X0\n6FH1799f5eXlys/PvyDrnzJlitf3H3zwgfbs2aOUlBSv9f773//WsGHDKj0dwBc//elPPV+Hh4er\nS5cuCg0N1bhx4zzjnTt31hVXXKF9+/ZVuv3Pf/5zzxlOSfrFL34hPz8/vfbaa5LOnW0sLS3VPffc\n43W7SZMmqXHjxvrrX//qNR4YGKj09PQar//bL9QqLy/XV199pZCQEHXp0kV5eXmV5t9+++1e6x00\naJCMMZ59y8/P14EDB3TPPfeocePGVd7n119/rY0bN+qWW27xnImvuAwfPlx79uzRF198IUm64oor\n9Mknn2jv3r013icA9QcP2QP4Xq655ppqX9S0d+9elZeX68orr6x0ncPh8HoB0GeffaZHHnlEf/nL\nX/T11197zfvv50DWBX9/f7Vu3dprbM+ePZKktLS0Km/jdDp1/PhxhYeH+3RfQUFBXk9jkM5F6X/f\nf8X4t/dfOvcz+O+fYWhoqFq0aKEDBw5Ikg4dOiTpXNR+W0BAgDp06KCDBw96jbdq1Ur+/jX/vwBj\njJ555hk999xz2r9/v8rKyjxri4yMrDS/TZs2Xt83bdpUkjz79umnn8rhcKhbt27V3ufevXtljNEj\njzyihx9+uNL1DodD//rXv9SiRQvNnj1bY8aMUefOndW9e3fdeOON+p//+R9dddVVNd5HAPYQpAAu\nmPLycjmdTr3xxhtyOis/IBMWFuaZl5iYqGPHjmnmzJmes4dHjhzRhAkTVF5e/p33Vd3zRyvC6b99\n+4zst9crSb/61a/Us2fPKm9XsWZffPtMYU3GzQV6J4Fv8/UV9RXPBf7Zz36mxx9/XBEREXI6nbr7\n7rurPD51sW8V27333nuVlJRU5ZyKUB80aJA+/fRTvfrqq3rzzTf129/+VtnZ2Xr++ed1++231/g+\nAdhBkAK4YDp27ChjjNq3b1/lWdIKH3/8sfbs2aPVq1frtttu84y//fbbleZWF55NmzaVMabSK88r\nziDWdL2S1LhxY8+ry+sDY4z27Nmj6667zjN28uRJffHFF/rhD38oSZ4XQRUWFnq9L2xpaan279+v\nG264oUb3Vd3P95VXXtH111+vF154wWv82LFjioqK8mV3JP3n38b//d//Vfuz7tChg6RzZ3lrcjyu\nuOIKTZgwQRMmTFBxcbEGDRqkRx99lCAFGgCeQwrggvnxj38sp9NZ7ScrffXVV5L+czbtv8+0PfPM\nM5UCKTQ0VJJ04sQJr/F27drJz8+v0vM8lyxZUuNX3yckJKhjx45asGCBTp48Wel6t9tdo+1cCC+8\n8ILOnj3r+X7JkiUqKyvTyJEjJUmJiYkKCAio9JZJy5Yt04kTJ3TTTTfV6H5CQ0OrfDspPz+/Smc3\n161bpyNHjvi6K5KkPn36KDY2Vs8880y1T8mIiorSkCFD9Pzzz6uoqKjS9d8+HhX/liqEhIToyiuv\n1OnTp2u1PgAXF2dIAdTadz382qFDBz3++ON68MEHtX//fo0ZM8bz/p/r16/X5MmTNX36dHXt2lUd\nO3bUjBkzdPjwYTVp0kSvvPJKlWGUkJAgY4zuvPNOJSUlyc/PTz/5yU/UpEkT3XLLLZ4g69ixo/73\nf/9XX375ZY33x+FwaNmyZRo5cqS6deumiRMnqlWrVjpy5Ig2btyo8PBwvfrqq779kOrImTNnNGzY\nMCUnJ2v37t167rnnNGjQIE9oRkZGaubMmZo9e7ZuvPFGjR492jOvb9++XmeezychIUFLly7VE088\noSuvvFLNmzfX0KFDddNNN+mxxx7T7bffrgEDBujjjz/WSy+95Dmr7KuKF8SNHj1avXr10sSJE9Wi\nRQvt3r1bu3bt0uuvvy5JWrx4sQYNGqSrrrpKkyZNUocOHeRyubR161YdOXLE84K3+Ph4DRkyRAkJ\nCYqIiNCOHTv0hz/8wfP2UQDqOSuv7QfQ4K1YscI4nU6zc+fO75z7pz/9yQwePNg0btzYNG7c2MTH\nx5u77rrL7NmzxzNn9+7dZvjw4aZJkyamefPmZsqUKebjjz82TqfT622UysrKzN13322io6ONn5+f\n11tAud1uc8stt5iwsDDTrFkzk5GRYXbt2lVpG+np6aZJkybVrvfDDz8048aNM1FRUSY4ONjExsaa\nW2+91WzcuPG8+3ngwIEa39eQIUNMjx49Ko3Hxsaa0aNHe76v+Dn/7W9/M1OmTDHNmjUzTZo0MWlp\naebrr7+udPslS5aY+Ph4ExgYaFq0aGGmTp1qjh8/XqP7NsYYl8tlRo0aZcLDw43T6fS8BdTp06fN\nL3/5S9OqVSsTGhpqBg8ebLZv326GDh1qrr/+es/tK95+65VXXvnOn40xxmzZssUkJSWZ8PBw07hx\nY9OrVy+zZMkSrzn79+836enppmXLliYwMNC0adPGjB492vzxj3/0zJkzZ47p16+fiYiIMKGhoSY+\nPt7MnTvX662nANRfDmMuwrPnAQC1snLlSt1+++3asWPHRf2IVgC4mGr1HNLFixcrNjZWwcHB6tev\nn3bs2FHt3GXLlmnw4MGKiIhQRESEbrjhhkrzJ06c6PmYuopLxfOiAAAAcGnzOUjXrl2rGTNmKCsr\nS/n5+erZs6eSkpKqfbL/5s2bNX78eG3atEnbtm1TmzZtNHz4cM+bGVcYMWKEXC6XioqKVFRUVOnz\nnAHgcsUDWQAudT4HaXZ2tiZPnqy0tDR17dpVS5cuVUhIiJYvX17l/NWrV2vKlCnq0aOHOnfurGXL\nlqm8vFy5uble8wIDAxUVFaXmzZurefPmPr/xNABcqmr6LgEA0FD5FKSlpaXauXOnhg0b5hlzOBxK\nTEzU1q1ba7SNkydPqrS0VBEREV7jmzZtUnR0tLp27aqMjIxKb+EBAJejCRMmqKysjOePArik+fS2\nT263W2VlZYqOjvYaj46OVmFhYY22cf/996tVq1ZKTEz0jI0YMUJjx45VbGysPv30U82cOVMjR47U\n1q1bqzwz4Ha7tWHDBrVv397nTxsBAADAhXfq1CkdOHBASUlJVX7E8Ldd1PchnTt3rl5++WVt3rzZ\n6zOsk5OTPV9369ZNV111lTp27KhNmzZp6NChlbazYcMGpaamXpQ1AwAAoPZefPHF73wvZJ+CNDIy\nUn5+fnK5XF7jLpdLMTEx573tggULNH/+fOXm5qpbt27nnRsbG6vIyEjt3bu3yiCt+Fi8F198UXFx\ncb7sAi6SadOmKTs72/YyUAWOTf3FsanfOD71F8emfiooKFBqaqrXxxlXx6cgDQgIUEJCgnJzczV6\n9GhJ5179mZube95Pw5g/f76efPJJvfnmm+rdu/d33s/hw4d19OhRtWjRosrrKx6mj4uL43lV9VR4\neDjHpp7i2NRfHJv6jeNTf3Fs6reaPL3S54fsp0+frvT0dCUkJKhv377Kzs5WcXGx0tPTJUlpaWlq\n3bq15syZI0maN2+eZs2apZycHLVt29ZzdjUsLEyhoaE6efKksrKyNHbsWMXExGjv3r26//771blz\nZyUlJfm6PFiSk3PuIkklJdKWLdLw4VJQ0LmxlJRzFwAAgP/mc5AmJyfL7XYrMzNTLpdLvXr10oYN\nGxQVFSXp3NlNf///bHbp0qUqLS3VuHHjvLYza9YsZWZmys/PTx999JFWrVqlY8eOqWXLlkpKStLs\n2bMVEBDwPXcPF8u3gzMvT0pIkObOlfiDFQAAfJdavagpIyNDGRkZVV73zjvveH2/f//+824rKChI\nb7zxRm2WAQAAgEtArT46FPhuPD5fX6Xw3Il6i2NTv3F86i+OTcNHkOIC4ZdDfcUv7vqLY1O/cXzq\nL45Nw0eQAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBVBCkAAACsIkgBAABgFUEKAAAAqwhSAAAAWEWQ\nAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBVBCkAAACsIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqC\nFAAAAFYRpAAAALCKIAUAAIBVBCkAAACsIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYR\npAAAALCKIAUAAIBVBCkAAACsIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALCK\nIAUAAIBVBCkAAACsIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBV\nBCkAAACsIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBVBCkAAACs\nIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBVBCkAAACsIkgBAABg\nFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBVBCkAAACsIkgBAABgFUEKAAAA\nqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBVBCkAAACsIkgBAABgFUEKAAAAqwhSAAAA\nWFWrIF28eLFiY2MVHBysfv36aceOHdXOXbZsmQYPHqyIiAhFRETohhtuqHJ+ZmamWrZsqZCQEN1w\nww3au3dvbZYGAACABsbnIF27dq1mzJihrKws5efnq2fPnkpKSpLb7a5y/ubNmzV+/Hht2rRJ27Zt\nU5s2bTR8+HB98cUXnjnz5s3TokWL9MILL+j9999XaGiokpKSdObMmdrvGQAAABoEn4M0OztbkydP\nVlpamrp27aqlS5cqJCREy5cvr3L+6tWrNWXKFPXo0UOdO3fWsmXLVF5ertzcXM+cZ599Vo888ohu\nuukmde/eXatWrdLnn3+u9evX137PAAAA0CD4FKSlpaXauXOnhg0b5hlzOBxKTEzU1q1ba7SNkydP\nqrS0VBEREZKk/fv3q6ioyGubTZo00bXXXlvjbQIAAKDh8ilI3W63ysrKFB0d7TUeHR2toqKiGm3j\n/vvvV6tWrZSYmChJKioqksPh+F7bBAAAQMPlfzHvbO7cuXr55Ze1efNmNWrU6Htvb9q0aQoPD/ca\nS0lJUUpKyvfeNgAAAGomJydHOTk5XmPHjx+v8e19CtLIyEj5+fnJ5XJ5jbtcLsXExJz3tgsWLND8\n+fOVm5urbt26ecZjYmJkjJHL5fI6S+pyudS7d+/zbjM7O1t9+vTxZRcAAABQx6o6IZiXl6eEhIQa\n3d6nh+wDAgKUkJDg9YIkY4xyc3M1YMCAam83f/58PfHEE9qwYUOlyIyNjVVMTIzXNk+cOKHt27ef\nd5sAAAC4NPj8kP306dOVnp6uhIQE9e3bV9nZ2SouLlZ6erokKS0tTa1bt9acOXMknXtLp1mzZikn\nJ0dt27b1nF0NCwtTaGioJOmee+7R448/riuvvFLt27fXI488otatW+vmm2+uo90EAABAfeVzkCYn\nJ8vtdiszM1Mul0u9evXShg0bFBUVJUk6fPiw/P3/s9mlS5eqtLRU48aN89rOrFmzlJmZKUm67777\nVFxcrMmTJ+vYsWMaNGiQXn/99Tp5nikAAADqN4cxxthehK8qnpOwc+dOnkNaD+XlSQkJ0s6dEocH\nAIDLky+9xmfZAwAAwCqCFAAAAFYRpAAAALCKIAUAAIBVBCkAAACsIkgBAABgFUEKAAAAqwhSAAAA\nWEWQAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBVBCkAAACsIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAA\nwCqCFAAAAFYRpAAAALCKIAUAAIBVBCkAAACsIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAA\nAFYRpAAAALCKIAUAAIBVBCkAAACsIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAA\nALCKIAUAAIBVBCkAAACsIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALCKIAUA\nAIBVBCkAAACsIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALDK3/YCAFxYOTnn\nLpJUUiIdPCi1aycFBZ0bS0k5d8HFx7EBgHMIUuAS9+2oycuTEhLORVCfPnbXBY4NAFQgSBuYQ4cO\nye12217GeRUUBEuKU0FBgaRTtpdTrcjISLVt29b2MgAAuOwRpA3IoUOHFNeli4pLSmwv5Tv0lpSn\n1NTbJOXbXky1QoKCVFBYSJQCAGAZQdqAuN1uFZeU6EVJcbYXcx4FklKler3OAkmpJSVyu90EKQAA\nlhGkDVCcpIbwFLOGsk4AAGAXb/sEAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBV\nBCkAAACsIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBVBCkAAACs\nIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBVBCkAAACsIkgBAABg\nFUEKAAAAq2oVpIsXL1ZsbKyCg4PVr18/7dixo9q5u3bt0rhx4xQbGyun06mFCxdWmpOVlSWn0+l1\niY+Pr83SAAAA0MD4HKRr167VjBkzlJWVpfz8fPXs2VNJSUlyu91Vzi8uLlbHjh01b948tWjRotrt\ndu/eXS6XS0VFRSoqKtLf//53X5cGAACABsjnIM3OztbkyZOVlpamrl27aunSpQoJCdHy5curnH/1\n1Vdr3rx5Sk5OVqNGjardrr+/v6KiotS8eXM1b95cERERvi4NAAAADZBPQVpaWqqdO3dq2LBhnjGH\nw6HExERt3br1ey1kz549atWqlTp27KjU1FR99tln32t7AAAAaBh8ClK3262ysjJFR0d7jUdHR6uo\nqKjWi+jXr59WrFihDRs2aOnSpdq/f78GDx6skydP1nqbAAAAaBj8bS9AkpKSkjxfd+/eXX379lW7\ndu308ssva+LEidXebtq0aQoPD/caS0lJUUpKygVbKwAAALzl5OQoJyfHa+z48eM1vr1PQRoZGSk/\nPz+5XC6vcZfLpZiYGF82dV7h4eHq3Lmz9u7de9552dnZ6tOnT53dLwAAAHxX1QnBvLw8JSQk1Oj2\nPj1kHxAQoISEBOXm5nrGjDHKzc3VgAEDfNnUeX3zzTf69NNPz/uqfAAAAFwafH7Ifvr06UpPT1dC\nQoL69u2r7OxsFRcXKz09XZKUlpam1q1ba86cOZLOvRBq165dMsbozJkzOnLkiD788EOFhYWpY8eO\nkqRf/vKXGjVqlNq1a6cjR45o1qxZ8vf356F3AACAy4DPQZqcnCy3263MzEy5XC716tVLGzZsUFRU\nlCTp8OHD8vf/z2Y///xz9e7dWw6HQ5K0YMECLViwQNddd53eeecdz23Gjx+vo0ePKioqSj/4wQ+0\nbds2NWvWrC72EQAAAPVYrV7UlJGRoYyMjCqvq4jMCu3atVN5efl5t/ffT4IFAADA5YPPsgcAAIBV\nBCkAAACsIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBVBCkAAACs\nIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBVBCkAAACsIkgBAABg\nFUEKAAAAq/xtLwAAAMAXOTnnLpJUUiIdPCi1aycFBZ0bS0k5d0HDQZACAIAG5dvBmZcnJSScC9Q+\nfeyuC7XHQ/YAAACwiiAFAACAVQQpAAAArCJIAQAAYBVBCgAAAKsIUgAAAFhFkAIAAMAqghQAAABW\nEaQAAACwiiAFAACAVQQpAAAArCJIAQAAYBVBCgAAAKsIUgAAAFhFkAIAAMAqghQAAABWEaQAAACw\niiAFAACAVQQpAAAArCJIAQAAYJW/7QUAl5JDhw7J7XbbXka1CgqCJcWpoKBA0inby6lWZGSk2rZt\na3sZAICLhCAF6sihQ4cU16WLiktKbC/lPHpLylNq6m2S8m0vplohQUEqKCwkSgHgMkGQAnXE7Xar\nuKREL0qKs72YahRISpXq/xpLSuR2uwlSALhMEKRAHYuT1Mf2Ir5DQ1gjAODywYuaAAAAYBVBCgAA\nAKsIUgAAAFhFkAIAAMAqXtQEAEAVcnLOXSSppEQ6eFBq104KCjo3lpJy7gLg+yNIAQCowreDMy9P\nSkg4F6h9eIsKoM7xkD0AAACsIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALCK\nIAUAAIBVBCkAAACsIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBV\n/rYXAAAXw6FDh+R2u20vo1oFBcGS4lRQUCDplO3lnFdkZKTatm1rexkALiEEKYBL3qFDhxTXpYuK\nS0psL+U8ekvKU2rqbZLybS/mvEKCglRQWEiUAqgzBCmAS57b7VZxSYlelBRnezHVKJCUKtXrNUr/\nf50lJXK73QQpgDpDkAK4bMRJ6mN7Ed+hIawRAOoaL2oCAACAVQQpAAAArCJIAQAAYBVBCgAAAKsI\nUgAAAFhFkAIAAMAqghQAAABWEaQAAACwiiAFAACAVQQpAAAArKpVkC5evFixsbEKDg5Wv379tGPH\njmrn7tq1S+PGjVNsbKycTqcWLlz4vbcJAACAS4fPQbp27VrNmDFDWVlZys/PV8+ePZWUlCS3213l\n/OLiYnXqASjyAAAX+0lEQVTs2FHz5s1TixYt6mSbAAAAuHT4HKTZ2dmaPHmy0tLS1LVrVy1dulQh\nISFavnx5lfOvvvpqzZs3T8nJyWrUqFGdbBMAAACXDp+CtLS0VDt37tSwYcM8Yw6HQ4mJidq6dWut\nFnAhtgkAAICGw6cgdbvdKisrU3R0tNd4dHS0ioqKarWAC7FNAAAANBz+thfwfUybNk3h4eFeYykp\nKUpJSbG0IgAAgMtPTk6OcnJyvMaOHz9e49v7FKSRkZHy8/OTy+XyGne5XIqJifFlU3WyzezsbPXp\n06dW9wsAAIC6UdUJwby8PCUkJNTo9j49ZB8QEKCEhATl5uZ6xowxys3N1YABA3zZ1AXdJgAAABoO\nnx+ynz59utLT05WQkKC+ffsqOztbxcXFSk9PlySlpaWpdevWmjNnjqRzL1ratWuXjDE6c+aMjhw5\nog8//FBhYWHq2LFjjbYJAACAS5fPQZqcnCy3263MzEy5XC716tVLGzZsUFRUlCTp8OHD8vf/z2Y/\n//xz9e7dWw6HQ5K0YMECLViwQNddd53eeeedGm0TAAAAl65avagpIyNDGRkZVV5XEZkV2rVrp/Ly\n8u+1TQAAAFy6+Cx7AAAAWEWQAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBVBCkAAACsIkgBAABgFUEK\nAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBVBCkAAACsIkgBAABgFUEKAAAAqwhS\nAAAAWEWQAgAAwCp/2wvApSFHtypHKZKkEgWqswr1gJ5UkE5LklKUoxStsblEAABQTxGkqBMpWkNw\nAgCAWuEhewAAAFhFkAIAAMAqghQAAABWEaQAAACwihc1AQCsOnTokNxut+1lnFdBQbCkOBUUFEg6\nZXs51YqMjFTbtm1tLwPwGUEKALDm0KFDiuvSRcUlJbaX8h16S8pTauptkvJtL6ZaIUFBKigsJErR\n4BCkAABr3G63iktK9KKkONuLOY8CSalSvV5ngaTUkhK53e46DdL6fgabs9eXBoIUAGBdnKQ+thdR\nAw1lnXWlYZzB5uz1pYAgBQAAVWoIZ7Av57PXlxKCFAAAnFdDODPcENaI6vG2TwAAALCKIAUAAIBV\nBCkAAACsIkgBAABgFS9qAi5xObpVOUqRJJUoUJ1VqAf0pIJ0WpKUohylaI3NJQIALnMEKXCJS9Ea\nghMAUK/xkD0AAACsIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBV\nBCkAAACsIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBVBCkAAACs\nIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBVBCkAAACsIkgBAABg\nFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBVBCkAAACsIkgBAABgFUEKAAAA\nqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBVBCkAAACsIkgBAABgFUEKAAAAqwhSAAAA\nWEWQAgAAwCqCFAAAAFbVKkgXL16s2NhYBQcHq1+/ftqxY8d5569bt05xcXEKDg5Wz5499frrr3td\nP3HiRDmdTq/LyJEja7M0AAAANDA+B+natWs1Y8YMZWVlKT8/Xz179lRSUpLcbneV87ds2aLx48dr\n0qRJ+uCDD3TzzTdrzJgx2rVrl9e8ESNGyOVyqaioSEVFRcrJyandHgEAAKBB8TlIs7OzNXnyZKWl\npalr165aunSpQkJCtHz58irnL1y4UCNGjND06dPVpUsXzZ49W3369NGiRYu85gUGBioqKkrNmzdX\n8+bNFR4eXrs9AgAAQIPiU5CWlpZq586dGjZsmGfM4XAoMTFRW7durfI2W7duVWJiotdYUlJSpfmb\nNm1SdHS0unbtqoyMDH311Ve+LA0AAAANlL8vk91ut8rKyhQdHe01Hh0drcLCwipvU1RUVOX8oqIi\nz/cjRozQ2LFjFRsbq08//VQzZ87UyJEjtXXrVjkcDl+WCAANRo5uVY5SJEklClRnFeoBPakgnZYk\npShHKVpjc4kAcFH4FKQXSnJysufrbt266aqrrlLHjh21adMmDR06tNrbTZs2rdJD+ykpKUpJSblg\nawWAupKiNQQngEtCTk5Opdf/HD9+vMa39ylIIyMj5efnJ5fL5TXucrkUExNT5W1iYmJ8mi9JsbGx\nioyM1N69e88bpNnZ2erTp48PewAAAIC6VtUJwby8PCUkJNTo9j49hzQgIEAJCQnKzc31jBljlJub\nqwEDBlR5m/79+3vNl6S33npL/fv3r/Z+Dh8+rKNHj6pFixa+LA8AAAANkM+vsp8+fbp+85vfaNWq\nVdq9e7emTJmi4uJipaenS5LS0tL04IMPeubffffdeuONN/T000+rsLBQjz76qHbu3KmpU6dKkk6e\nPKn77rtP27dv18GDB5Wbm6sxY8aoc+fOSkpKqpu9BAAAQL3l83NIk5OT5Xa7lZmZKZfLpV69emnD\nhg2KioqSdO7spr//fzbbv39//f73v9dDDz2khx56SJ06ddKrr76q+Ph4SZKfn58++ugjrVq1SseO\nHVPLli2VlJSk2bNnKyAgoI52EwAAAPVVrV7UlJGRoYyMjCqve+eddyqNjR07VmPHjq1yflBQkN54\n443aLAMAgAuGd0EALp568Sp7AADqG94FAbh4avVZ9gAAAEBdIUgBAABgFUEKAAAAqwhSAAAAWEWQ\nAgAAwCqCFAAAAFYRpAAAALCKIAUAAIBVBCkAAACsIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqC\nFAAAAFYRpAAAALCKIAUAAIBVBCkAAACsIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYR\npAAAALCKIAUAAIBVBCkAAACsIkgBAABgFUEKAAAAqwhSAAAAWEWQAgAAwCqCFAAAAFYRpAAAALCK\nIAUAAIBVBCkAAACs8re9AAAAAF/k6FblKEWSVKJAdVahHtCTCtJpSVKKcpSiNTaXCB8RpAAAoEFJ\n0RqC8xLDQ/YAAACwiiAFAACAVQQpAAAArCJIAQAAYBVBCgAAAKsIUgAAAFhFkAIAAMAqghQAAABW\nEaQAAACwiiAFAACAVQQpAAAArCJIAQAAYBVBCgAAAKsIUgAAAFhFkAIAAMAqghQAAABWEaQAAACw\niiAFAACAVQQpAAAArCJIAQAAYBVBCgAAAKsIUgAAAFhFkAIAAMAqghQAAABWEaQAAACwiiAFAACA\nVQQpAAAArCJIAQAAYBVBCgAAAKsIUgAAAFhFkAIAAMAqghQAAABWEaQAAACwiiAFAACAVQQpAAAA\nrCJIAQAAYBVBCgAAAKsIUgAAAFhFkAIAAMAqghQAAABWEaQAAACwiiAFAACAVQQpAAAArCJIAQAA\nYBVBCgAAAKsIUlwQObYXgGpxbOovjk39xvGpvzg2DV+tgnTx4sWKjY1VcHCw+vXrpx07dpx3/rp1\n6xQXF6fg4GD17NlTr7/+eqU5mZmZatmypUJCQnTDDTdo7969tVka6gl+OdRfHJv6i2NTv3F86i+O\nTcPnc5CuXbtWM2bMUFZWlvLz89WzZ08lJSXJ7XZXOX/Lli0aP368Jk2apA8++EA333yzxowZo127\ndnnmzJs3T4sWLdILL7yg999/X6GhoUpKStKZM2dqv2cAAABoEHwO0uzsbE2ePFlpaWnq2rWrli5d\nqpCQEC1fvrzK+QsXLtSIESM0ffp0denSRbNnz1afPn20aNEiz5xnn31WjzzyiG666SZ1795dq1at\n0ueff67169fXfs8AAADQIPgUpKWlpdq5c6eGDRvmGXM4HEpMTNTWrVurvM3WrVuVmJjoNZaUlOSZ\nv2/fPhUVFXlts0mTJrr22mur3SYAAAAuHf6+THa73SorK1N0dLTXeHR0tAoLC6u8TVFRUZXzi4qK\nJEkul0sOh+O8c/7bqVOnJEkFBQW+LL/Bq9jf1yTV9z0/LOkl24s4j/3//3/r8t9QQzk+HJv6q74f\nG6nuj09DOTZS/T8+/LdTf12IY9MQVOxvRbedj09BWl8cOHBAkpSammp3IZY8YnsBNdQQjs6F+DfU\nEI4Px6b+agjHRqr749MQjo3UMI4P/+3UX5drtxw4cEADBw487xyfgjQyMlJ+fn5yuVxe4y6XSzEx\nMVXeJiYm5rzzY2JiZIyRy+XyOkvqcrnUu3fvKreZlJSkF198Ue3bt1dwcLAvuwAAAICL4NSpUzpw\n4ICSkpK+c65PQRoQEKCEhATl5uZq9OjRkiRjjHJzc3XXXXdVeZv+/ftXuv6tt95S//79JUmxsbGK\niYlRbm6uevToIUk6ceKEtm/frjvuuKPKbUZGRuq2227zZekAAAC4yL7rzGgFv0cfffRRXzbcpEkT\nZWZmqk2bNgoMDNTDDz+sDz/8UMuWLVNoaKjS0tK0Y8cOz4uUWrVqpYcfflihoaGKiIjQokWLtG7d\nOv32t79VVFSUJKmsrExPPvmk4uPjdebMGd111106ffq0Fi5cKD8/P9/2HAAAAA2Kz88hTU5Oltvt\nVmZmplwul3r16qUNGzZ44vLw4cPy9//PZvv376/f//73euihh/TQQw+pU6dOevXVVxUfH++Zc999\n96m4uFiTJ0/WsWPHNGjQIL3++utq1KhRHewiAAAA6jOHMcbYXgQAAAAuX3yWPWpl8+bNcjqdOnHi\nxPfaTmxsrBYuXFhHqwKAc+rr76ihQ4dq+vTpdbY94FJBkKJGqvol6nA4LK0GALzxOwpo2AhSAAAA\nWEWQ4jtNnDhRmzdv1rPPPiun0yk/Pz/PhxP84x//0DXXXKPQ0FANHDhQ//znPz2327dvn8aMGaOY\nmBg1btxYffv2VW5urqW9uPwYYzR//nx16tRJQUFBat++vZ588klJ0gMPPKAuXbooNDRUHTt2VGZm\npsrKyiyvGKidi/k7Kjs7Wz169FBYWJjatm2rO+64Q8XFxV5z3nvvPQ0dOtTz7jIjRozQ8ePHPdeX\nl5fr/vvvV7NmzdSiRQtlZWXV3Q8DaKAIUnynZ599Vv3799ekSZPkcrn0xRdfqE2bNjLG6OGHH1Z2\ndrZ27twpf39//fSnP/Xc7ptvvtEPf/hDbdy4UR988IFGjBih0aNH6/Dhwxb35vLxwAMPaP78+Zo1\na5YKCgr0+9//3vPhE02aNNGqVatUUFCghQsXatmyZcrOzra84stHdX8sHDx4UE6nU2vXrtXAgQMV\nHBysq666Su+++67tJddrF/N3lJ+fn379619r165dWrVqlTZu3Kj77rvPc/0HH3ygxMREde/eXdu2\nbdN7772nUaNGef3Bt3LlSoWFhen999/X/PnzNXv27Mvuj3VjjJ588kl16NBBISEh6t27t1555RVJ\n0mOPPaZWrVrp66+/9sz/4Q9/6Hk7SanqPwxOnjzpuX7lypVq2rSp3nzzTcXHx6tx48YaMWKE1wf1\nlJWV6a677lLTpk0VFRWlBx54QOnp6frRj350EX4CqMQANTBkyBAzbdo0z/ebNm0yTqfTbNy40TP2\n2muvGafTaU6fPl3tdrp3724WL17s+b59+/bm2WefvSBrvpz9+9//NkFBQWb58uU1mr9gwQJzzTXX\nXOBVocJ9991nmjVrZlavXm327dtn3nvvPfPb3/7WHDhwwDgcDtO2bVvzpz/9yezevdtMmjTJhIeH\nm6+++sr2sus1W7+j/vCHP5ioqCjP9+PHjzeDBg067zoHDx7sNda3b18zc+bMam9zKXr88cdNfHy8\neeutt8z+/fvNypUrTXBwsHn33XdNWVmZGThwoPnxj39sjDFm0aJFJiIiwhw+fNhz+2effdZs2rTJ\nHDx40GzcuNHExcWZO+64w3P9ihUrTKNGjczw4cNNXl6eyc/PN/Hx8SY1NdVrDZGRkebVV181hYWF\n5he/+IUJDw83P/rRjy7eDwIeBClqpLpf9m632zOWn59vnE6n+eyzz4wxxnzzzTdmxowZJi4uzlxx\nxRUmLCzM+Pv7m/vvv99zG4L0wnj//feN0+k0Bw4cqPL6NWvWmIEDB5qYmBgTFhZmgoKCTHR09EVe\n5eXpfH8sVATpU0895Rk7e/asadOmjdcYKrtYv6PeeustM2zYMNOqVSvTuHFjExwcbJxOpzl16pQx\nxpj4+Hjz6KOPnnedU6dO9Rq7+eabzU9/+tPv9wNoQE6fPm1CQ0PNtm3bvMZ/9rOfmdtuu80YY8y+\nfftMeHi4eeCBB0xISIhZs2bNebf5338YrFixwjidTrN//37P2JIlS0yLFi0838fExJinn37a831Z\nWZlp164dQWqJz2+MD3xbQECA5+uKV7SWl5dLkmbMmKHc3Fz96le/UseOHRUcHKyxY8fqzJkzVtZ6\nOQkODq72um3btik1NVWPPfaYhg8frvDwcOXk5Ojpp5++iCu8fBUUFOjMmTO6/vrrq53Tr18/z9d+\nfn66+uqrVVBQcDGWd8mpy99RBw8e1KhRo3THHXdozpw5ioiI0N/+9jf97Gc/05kzZxQUFHTe//aq\nWlPFuirWdDnYu3eviouLdcMNN8h8663QS0tL1bt3b0nn3m7rqaee0uTJk3XrrbfqJz/5idc23n77\nbc2dO1e7d+/WiRMndPbsWZ0+fVolJSUKCgqSJIWEhKh9+/ae27Ro0UL/+te/JJ37iHKXy6VrrrnG\nc73T6VRCQoLXmnDxEKSokUaNGvn8opctW7YoPT1do0ePlnTu+VoVLzTAhVXx3MTc3FzdfvvtXtdt\n2bJF7du31wMPPOAZ47hcPDUJFvjuYvyO2rlzp4wxWrBggWdszZo1XnN69Oih3NxczZo1y6e1XE6+\n+eYbSdJrr72mli1bel0XGBjo+Xrz5s3y9/fXgQMHVF5eLqfz3MteavKHgVR1+BOb9RcvakKNtG/f\nXtu3b9fBgwd19OhRlZeXV/kf9rfHOnXqpD/+8Y/68MMP9eGHH+q2227jl8FFEhgYqPvvv1/33Xef\nVq9erX379mn79u1avny5OnXqpEOHDmnt2rXat2+fFi5cqPXr19te8mXj238sVGfbtm2er8vKyrRz\n507FxcVdjOU1WBfjd9SVV16p0tJSLVy4UPv379fq1av1/PPPe82ZOXOmduzYoTvuuEMff/yxdu/e\nraVLl+qrr76qu51t4OLj4xUYGKiDBw+qQ4cOXpdWrVpJktauXav169dr06ZNOnjwoGbPnu25/bf/\nMOjbt6+uvPJKHTlyxKc1NGnSRNHR0dqxY4dnrLy8XHl5eXWzk/AZQYoauffee+Xn56f4+Hg1b95c\nhw4dqvJNp7899vTTT6tp06YaOHCgbr75Zt14443q06dPtfNRtzIzMzVjxgzNmjVL8fHxuvXWW/Xl\nl19q1KhRmjZtmu6880717t1b27ZtU2Zmpu3lXjbO98dChcWLF2v9+vUqLCxURkaGjh07VulMN7xd\njN9RPXr00NNPP6358+frqquuUk5OjubOnes1v1OnTnrzzTf10Ucf6dprr9XAgQP15z//Wf7+/pW2\nd7kKCwvTvffeq2nTpmnVqlXat2+f8vPztWjRIq1evVqHDx9WRkaG5s+frwEDBuh3v/ud5syZo/ff\nf19Szf4wqIk777xTc+bM0Z///Gf985//1N13361jx45xjGyx9NxVALiszZkzx8TGxprAwEDTvn17\nM3fuXM+LmtasWWOuvfZaExQUZLp37242b95se7lAnVu4cKGJi4szgYGBJjo62owYMcK8++67JjEx\n0YwcOdJr7t133206depkTp48aYwx5plnnjGtWrUyoaGhZsSIEebFF180TqfTHD9+3Bhz7kVNTZs2\n9drG+vXrjdPp9Hx/9uxZc9ddd5krrrjCNGvWzMycOdMkJyeb8ePHX+A9R1UcxvAYKgDUBxUPYebn\n56tHjx62lwNcVowxiouL009+8hM+rMACXtQEAPUI5wiAi+PQoUN68803dd1116mkpESLFi3SgQMH\nNH78eNtLuyzxHFIAqEd4/hpwcTidTq1YsUJ9+/bVoEGD9Mknnyg3N1ddunSxvbTLEg/ZAwAAwCrO\nkAIAAMAqghQAAABWEaQAAACwiiAFAACAVQQpAAAArCJIAQAAYBVBCgAAAKsIUgAAAFj1/wA19N//\nb268TQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x9f4b6a0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "best_features = []\n",
    "for i in indices[:5]:\n",
    "    best_features.append(features[i])\n",
    "\n",
    "# Plot the top 5 feature importances of the forest\n",
    "plt.figure(num=None, figsize=(8, 6), dpi=80, facecolor='w', edgecolor='k')\n",
    "plt.title(\"Feature importances\")\n",
    "plt.bar(range(5), importances[indices][:5], \n",
    "       color=\"r\",  yerr=std[indices][:5], align=\"center\")\n",
    "plt.xticks(range(5), best_features)\n",
    "plt.xlim([-1, 5])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decision Tree accuracy and time elapsed caculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DecisionTree\n",
      "Acurracy:  0.722222222222\n",
      "time elapsed:  0.009000539779663086\n"
     ]
    }
   ],
   "source": [
    "from time import time\n",
    "t0=time()\n",
    "print (\"DecisionTree\")\n",
    "\n",
    "dt = DecisionTreeClassifier(min_samples_split=20,random_state=99)\n",
    "# dt = DecisionTreeClassifier(min_samples_split=20,max_depth=5,random_state=99)\n",
    "\n",
    "clf_dt=dt.fit(X_train,y_train)\n",
    "\n",
    "print (\"Acurracy: \", clf_dt.score(X_test,y_test))\n",
    "t1=time()\n",
    "print (\"time elapsed: \", t1-t0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cross result========\n",
      "[ 0.71666667  0.86666667  0.79661017  0.76271186  0.74576271]\n",
      "0.777683615819\n",
      "time elapsed:  0.04300236701965332\n"
     ]
    }
   ],
   "source": [
    "tt0=time()\n",
    "print (\"cross result========\")\n",
    "scores = cross_validation.cross_val_score(dt, X, y, cv=5)\n",
    "print (scores)\n",
    "print (scores.mean())\n",
    "tt1=time()\n",
    "print (\"time elapsed: \", tt1-tt0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tuning our hyperparameters using GridSearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 9 candidates, totalling 27 fits\n",
      "Best score: 0.743\n",
      "Best parameters set:\n",
      "\tclf__max_depth: 25\n",
      "\tclf__min_samples_leaf: 3\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.80      0.69      0.74        51\n",
      "          1       0.65      0.77      0.71        39\n",
      "\n",
      "avg / total       0.73      0.72      0.72        90\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  27 out of  27 | elapsed:    7.3s finished\n"
     ]
    }
   ],
   "source": [
    "from sklearn.grid_search import GridSearchCV\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('clf', DecisionTreeClassifier(criterion='entropy'))\n",
    "])\n",
    "\n",
    "parameters = {\n",
    "    'clf__max_depth': (15, 20 , 25),\n",
    "    'clf__min_samples_leaf': (3, 5, 10)\n",
    "}\n",
    "\n",
    "grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=1, scoring='f1')\n",
    "grid_search.fit(X_train, y_train)\n",
    "\n",
    "print ('Best score: %0.3f' % grid_search.best_score_)\n",
    "print ('Best parameters set:')\n",
    "\n",
    "best_parameters = grid_search.best_estimator_.get_params()\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print ('\\t%s: %r' % (param_name, best_parameters[param_name]))\n",
    "\n",
    "predictions = grid_search.predict(X_test)\n",
    "\n",
    "print (classification_report(y_test, predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Naive Bayes accuracy and time elapsed caculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NaiveBayes\n",
      "Acurracy:  0.788888888889\n",
      "time elapsed:  0.024001598358154297\n"
     ]
    }
   ],
   "source": [
    "t4=time()\n",
    "print (\"NaiveBayes\")\n",
    "nb = BernoulliNB()\n",
    "clf_nb=nb.fit(X_train,y_train)\n",
    "print (\"Acurracy: \", clf_nb.score(X_test,y_test))\n",
    "t5=time()\n",
    "print (\"time elapsed: \", t5-t4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cross result========\n",
      "[ 0.78333333  0.8         0.77966102  0.77966102  0.81355932]\n",
      "0.791242937853\n",
      "time elapsed:  0.0500028133392334\n"
     ]
    }
   ],
   "source": [
    "tt4=time()\n",
    "print (\"cross result========\")\n",
    "scores = cross_validation.cross_val_score(nb, X, y, cv=5)\n",
    "print (scores)\n",
    "print (scores.mean())\n",
    "tt5=time()\n",
    "print (\"time elapsed: \", tt5-tt4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### KNN accuracy and time elapsed caculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KNN\n",
      "Acurracy:  0.677777777778\n",
      "time elapsed:  0.0060002803802490234\n"
     ]
    }
   ],
   "source": [
    "t6=time()\n",
    "print (\"KNN\")\n",
    "knn = KNeighborsClassifier()\n",
    "clf_knn=knn.fit(X_train, y_train)\n",
    "print (\"Acurracy: \", clf_knn.score(X_test,y_test) )\n",
    "t7=time()\n",
    "print (\"time elapsed: \", t7-t6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cross result========\n",
      "[ 0.61666667  0.7         0.61016949  0.74576271  0.69491525]\n",
      "0.673502824859\n",
      "time elapsed:  0.06500363349914551\n"
     ]
    }
   ],
   "source": [
    "tt6=time()\n",
    "print (\"cross result========\")\n",
    "scores = cross_validation.cross_val_score(knn, X, y, cv=5)\n",
    "print (scores)\n",
    "print (scores.mean())\n",
    "tt7=time()\n",
    "print (\"time elapsed: \", tt7-tt6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SVM accuracy and time elapsed caculation¶"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SVM\n",
      "Acurracy:  0.566666666667\n",
      "time elapsed:  0.017000913619995117\n"
     ]
    }
   ],
   "source": [
    "t7=time()\n",
    "print (\"SVM\")\n",
    "\n",
    "svc = SVC()\n",
    "clf_svc=svc.fit(X_train, y_train)\n",
    "print (\"Acurracy: \", clf_svc.score(X_test,y_test) )\n",
    "t8=time()\n",
    "print (\"time elapsed: \", t8-t7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cross result========\n",
      "[ 0.53333333  0.53333333  0.54237288  0.54237288  0.54237288]\n",
      "0.538757062147\n",
      "time elapsed:  0.27701568603515625\n"
     ]
    }
   ],
   "source": [
    "tt7=time()\n",
    "print (\"cross result========\")\n",
    "scores = cross_validation.cross_val_score(svc, X,y, cv=5)\n",
    "print (scores)\n",
    "print (scores.mean())\n",
    "tt8=time()\n",
    "print (\"time elapsed: \", tt7-tt6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the training dataset, we now will train two different classifiers— a decision tree classifier, and a k-nearest neighbors classifier—and look at their individual performances via a 10-fold cross-validation\n",
    "on the training dataset before we combine them into an ensemble classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10-fold cross validation:\n",
      "\n",
      "ROC AUC: 0.69 (+/- 0.08) [Decision Tree]\n",
      "ROC AUC: 0.75 (+/- 0.09) [KNN]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cross_validation import cross_val_score\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier \n",
    "from sklearn.pipeline import Pipeline\n",
    "import numpy as np\n",
    "\n",
    "clf2 = DecisionTreeClassifier(max_depth=1, \n",
    "                              criterion='entropy', \n",
    "                              random_state=0)\n",
    "\n",
    "clf3 = KNeighborsClassifier(n_neighbors=1, \n",
    "                            p=2, \n",
    "                            metric='minkowski')\n",
    "\n",
    "pipe3 = Pipeline([['sc', StandardScaler()],\n",
    "                  ['clf', clf3]])\n",
    "\n",
    "clf_labels = ['Decision Tree', 'KNN']\n",
    "\n",
    "print('10-fold cross validation:\\n')\n",
    "for clf, label in zip([clf2, pipe3], clf_labels):\n",
    "    scores = cross_val_score(estimator=clf, \n",
    "                             X=X_train, \n",
    "                             y=y_train, \n",
    "                             cv=10, \n",
    "                             scoring='roc_auc')\n",
    "    print(\"ROC AUC: %0.2f (+/- %0.2f) [%s]\" \n",
    "               % (scores.mean(), scores.std(), label))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see the accuracies o our individual classifiers are almost same and are on the high side. Now let's move on to the more exciting part and combine the individual classifiers for majority rule voting in our MajorityVoteClassifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROC AUC: 0.69 (+/- 0.08) [Decision Tree]\n",
      "ROC AUC: 0.75 (+/- 0.09) [KNN]\n",
      "ROC AUC: 0.80 (+/- 0.09) [Majority Voting]\n"
     ]
    }
   ],
   "source": [
    "# Majority Rule (hard) Voting\n",
    "\n",
    "mv_clf = MajorityVoteClassifier(\n",
    "                classifiers=[clf2, pipe3])\n",
    "\n",
    "clf_labels += ['Majority Voting']\n",
    "all_clf = [clf2, pipe3, mv_clf]\n",
    "\n",
    "for clf, label in zip(all_clf, clf_labels):\n",
    "    scores = cross_val_score(estimator=clf, \n",
    "                             X=X_train, \n",
    "                             y=y_train, \n",
    "                             cv=10, \n",
    "                             scoring='roc_auc')\n",
    "    print(\"ROC AUC: %0.2f (+/- %0.2f) [%s]\" \n",
    "               % (scores.mean(), scores.std(), label))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the performance of the MajorityVotingClassifier has substantially improved over the individual classifiers in the 10-fold cross-validation evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating and tuning the ensemble classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we are going to compute the ROC curves from the test set to check if the MajorityVoteClassifier generalizes well to unseen data. We should remember that the test set is not to be used for model selection; its only purpose is to report an unbiased estimate of the generalization performance of a classifier system. The code is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk4AAAGGCAYAAACNCg6xAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzs3Xd4VFX6wPHvSQKkUaVLkSJFpQmKgKAgRggwlkURdRUU\nUAQLrrrr2gs/hbWtii4iqIjGAogCghRFQASlSAcFgUCAECBAIKTO+f1xkyGTZFImc+fO3Hk/zzPP\nZs7cufPO693w5txTlNYaIYQQQghRujCrAxBCCCGECBZSOAkhhBBClJEUTkIIIYQQZSSFkxBCCCFE\nGUnhJIQQQghRRlI4CSGEEEKUkRROQgghhBBlJIWTEEIIIUQZRVgdgD8opaKBjsAFwF4gw8p4hBBC\nCBFQIjFqhO+11sdKOjAkCiegDfCz1UEIIYQQIqDdDnxW0gGhUjjtwEjGpzNmzKBt27ZWx1OqwYMH\nM3PmTKvDsCXJrXkkt+aR3JpHcmueYMnt9u3bueOOO8C4K1WikCictNbpSqkdAG3btuXSSy+1OqRS\nVapUKSjiDEaSW/NIbs0juTWP5NY8QZjbUofyyODwANW6dWurQ7Atya15JLfmkdyaR3JrHjvmVgon\nIYQQQogyksJJCCGEEKKMpHAKUAMHDrQ6BNuS3JpHcmseya15JLfmsWNupXAKUPPmzbM6BNuS3JpH\ncmseya15JLfmsWNupXAKUM8995zVIdiW5NY8klvzSG7NI7k1jx1zq7TWVsfgF0qpS4F169atC7ap\nkUIIIYQw0fr16+ncuTNAZ631+pKOlR4nIYQQQogyksJJCCGEEKKMpHAKUFOnTrU6BNuS3JpHcmse\nya15JLfmsWNupXAKUOvXl3iLVVSA5NY8klvzSG7NI7k1jx1zK4PDhRBCCBHSZHC4EEIIIYQJpHAS\nQgghhCgjKZyEEEIIIcpICqcA5XA4rA7BtiS35pHcmkdyax7JrXnsmFspnALU2LFjrQ7BtiS35pHc\nmkdyax7JrXnsmFuZVSeEEEKIkBZ0s+qUUj2VUt8qpZKUUk6lVKl9e0qpq5VS65RSGUqpP5RSd/kj\nViGEEEKEroAonIAY4HfgfqDULjCl1AXAPGAp0AH4L/CBUupa80IUQgghRKgLiMJJa71Qa/2M1vob\nQJXhLaOBv7TWj2utd2qtJwEzgXGmBupHc+bMsToE25Lcmkdyax7JrXkkt+axY24DonDywhXAkkJt\n3wPdLIjFFAkJCVaHYFuSW/NIbs0juTWP5NY8dsxtwA0OV0o5gRu01t+WcMxOYJrWekKBtv4Yt++i\ntdaZxbxHBocLIYQQoojyDA6P8E9IQgghhLAzrWHLFjhyBJKToXbtdOLioq0Oy+eC9VbdYaBeobZ6\nwKniepsKio+Px+FwuD26detW5D7sokWLil24a8yYMUydOtWtbf369TgcDo4ePerW/uyzzzJhwgS3\ntsTERBwOBzt27HBrf/vtt3nsscfc2tLT03E4HKxcudKtPSEhgeHDhxeJbciQIfI95HvI95DvId9D\nvocp32PPHhg/HkbdM4Kpbz0Nh5cCkJOTw0svvUSfPr257LKj9O0Lt98OH32kA/J7JCQk4HA4aNas\nGR07dsThcDBuXNmHSAfrrbpXgP5a6w4F2j4Damit4z28R27VCSGEEAVs2QKffWb0Eh05ArVrw7Rp\neS+e3gtHfoLTuyFtFytWV+VvL7zEmucvp1ndvRBZD246jNaaOnXq8NRTT9Gz58PUrAl160JsrIVf\nrJyCcR2nGKVUB6VUx7ym5nnPG+e9/rJS6uMCb/lf3jETlFKtlVL3A4OB1/0cummKq6SFb0huzSO5\nNY/k1jzBnNszZ4yHJ88/D/fdV8wLzhzIzSIx0SicNm82brXVrXvukNzDP8DqYbD7A0jfT8/umcx/\ncwRPfbWPs72WwYCtZGdn88MPP7Bp0yYefvhhOneG5s3PFU3BnFtPAmWMUxfgR4w1nDTwWl77x8Dd\nQH2gcf7BWuu9SqkBwBvAg8AB4B6tdeGZdkErLi7O6hBsS3JrHsmteSS35gmk3GoNKSkQFQVVqxZ/\nzNixMH++0UOUng6vvQaPPFLMgTlnaVQrlejsE7Dje0jbbfQend4Np/fAFR8SH38He/cW/zlX3Pwa\nffqMYcJr77jamrc6xrM9jhF5/oXs/usv5s2bwYkTJ0hOTmbo0KEo5b6iUCDl1lcC7ladWeRWnRBC\nCCvk5kJi4rlB00eOwF13QaVKRY/NyoIqVYzbZZ46ayZPhr17jd6hevXgssvgwguLOXBWXchMMX4O\nj4LY5lC1JcS2MB4N4qBqS1atWsUtt9zC2rVrqV+/vuvt8+fPp1GjRnTo0MHttGfOnOH7779n8+bN\nrrbw8HDuvfde6tSpU87sBAaZVSeEEEL4yY4dsHQpjBlT/OvHjxu3rwoaMAAaNCh6bOXKMHcudMwf\nuKI1ZByGtF15Y412c+8lu6FdOHT/pOTAuk6FyjWMIimqASjFxIkTycjI4Jln7ncddsEFF3DHHXdQ\nuCNlwIABbs+11mzcuJFFixZx9uxZV3vTpk0ZOHAgtWvXLjkem5DCSQghhChk3Tr4/vtzg6bbtIFn\nnin+2N9+g0cfhXvugcjIoq/XqgWLFxs9RHXrGgOwI0r413fgQCBxFsx/Dk7/Bbnp516MOh+qtoAa\nHTy9HYBjx47x3/d/4+677+aCug1d7bm5ueTk5Lgd27BhQ1555ZUSz5eTk8Nnn33Gnj17XG2RkZHE\nxcXRsWPHIrfo7CwgBoeLogpPvRS+I7k1j+TWPJLb8nE6oVB94ObRRyF/lnxxuf3lF3j9dVi0CA4d\nMs7nye23G2ONiiuaAMLDoe/VZ2nfZCv1c74h4s/X4eSO4g/OF1kX6vaCDi9Br28gfgvckg43HoC+\nP0GXt1yHbt++nXnz5rm9PSIigvfff59du3a5tT/xxBO88MILJX92MSIiIqhWrZrr+SWXXMKYMWPo\n1KlTiUWTHa9b6XEKUBMnTuTKK6+0OgxbktyaR3JrHsmtUZxUqlT82CCAYcNg/XqjhyglBb78Ev72\nt+KPjY2F6Ly1GYvL7dixxqMsworrgtj6CqT94ZrKz9mD514Lj4LoRlC9jeeT1u1pPArQWrN3zx5i\nY2PdxhLNmDGDGTNmMHDgQFdb9erVOXTokE97guLi4khJSaF37960bNmyTO+x43Urg8MDVHp6OtHR\n9ltxNRBIbs0juTWPHXPrdBrjf/IHTJ88CTfcUPyxu3dDy5bGWKI+fYo/5umn4cSJc4Om+/YtOrao\nOGXOrdZw9hBkpUKNi0s+dkEnUJXODcaumjcgu2pLiKwPZShoMjIyiCzQjZWdnU21atV4+eWXefjh\nh13tJ0+eJDIykipVqpT+HSpIa12uYixYrlsZHG4DwXChBSvJrXkkt+YJxtxu2QI7d3ru9Vm/3pgR\nli8iwphVVty/y+efDzNmQNu2nj/vxRe9i7NIbjOOwPENBabu5/Uanf4Lcs9C1VYwaGfJJ+2/wbtg\n8kydOpVx48Zx/PhxIvIGRFWqVInFixfTtlASqlevXqHPKo/y9mAF43VbGimchBBClNnq1cZg6Pyp\n9d26eZ42/8038N//ei6cWrWC2bON3qH8gdOeREYaY4n8Yt/nsO4hUBEQ28zoKap7NbS4B2JbQtXi\n5v577+abb+ayyy7j8ccfd7X17NmT//73v+Tk5LgKJ8C0216nTp1iyZIl9O3b120skyhKCichhAgx\nWnu+UzR2LHTvDrfdVvzrM2fCO++cK3ZatfL8OY88AgVqgSKqVYMbbyx73GWmNWQdd1/wMX86/+nd\ncPn7cP5Az+9vehucPwiiG0OY7/6Z3LJlCy+++CJTp04ltsB+JJ06daJ5oXuKrVq1olVJyfURp9PJ\n2rVrWbp0KVlZWeTk5HDLLbeY/rnBTGbVBajCGx0K35Hcmkdya56Scut0GrWCJ0OGGMVQy5bGatRr\n13o+9swZyMjw/PrLL8PZs7Bv37lp+J5ERXkeyG0areHrhjCrNizqCqtug51vwqkdEN0EWow0FoIs\noEhuI2sbPU0VKJrmzp3LzJkz3drCwsI4ePAgR44ccWv/97//zeDBg73+LG8lJyczbdo0FixYQFZW\nFmBswJuWluazz7Dj7wTpcQpQTZo0sToE25Lcmkdy6ztnz55bQygry3NuV66Eq682FmH0NNGpcmVo\n3Rp69jR6iho2LP44gA8/LDkuvxZCzmw4sy+v52iX8b8RUdBhvOf3KAXtnoXKtc4NzK5c8higily3\nZ8+e5aeffqJr167UrFnT1f7ll1+SlZXlVhBddNFFrFixwuvP8pXs7GyWL1/OqlWrcBZYZ+HSSy+l\nb9++REVF+eyz7Pg7QWbVCSGEn23ZYswm69Wr+NfnzHG/hdW8uTGrrDjJyfD113DLLcZCi0Fv/xz4\n813j1lp6Iuhcoz2sEsRcALW7Q7ePLAktOzubgwcP0rRpU1fbgQMHaNy4MXPmzOH66693tefm5hIe\nHm5FmCU6e/YsU6ZMITU11dV23nnnMWjQILfvFWpkVp0QQvjZ6tWwffu5qfXXXWc8ivP228bK1J5u\nmXXpAp98cm7AdL16nj+3Xj24776Kx28a13ijvDFGda+C6PNLfk+lqtBk8Lnp+7Et8sYb+a8Q0Vrj\ndDrdip9HH32UhQsXsnPnuRl1jRo1YteuXUXGKAVi0QQQFRVFw4YNSU1NJSwsjJ49e3LllVe6DUAX\nJZNMCSFEKZxOGD0ahg41bosV5/XX4auvjF6funWNLTo8efnlkrfcaNQI7rijQiFbQzvhrw/db62d\n3g3ZJ88dc+VMaOJhmh1A4xuMh4WysrJo3rw5L774IsMLTBm87777uPXWW4sc36JFC3+GV2H9+vUj\nOzubvn37Bu2mvFaSweEBaseOUpbjF16T3JonmHJ7+rRRnMTFGRuqNmxo9BYVJyzM6E06dszz+T74\nADIzjWO2b4dRozwfW6uWMaOsPCzPbW4WnPWQIBcFGx6HvZ9C5lGodSlc/ARc+RX0Ww83nyy5aLLA\nN998w+WXX+62wW3lypV55JFH6NSpk9uxbdu2pVu3bv4O0ediY2MZOnSoX4omy69bE0iPU4B6/PHH\n+fbbb60Ow5Ykt+bxR26dTkhNNW6HaQ0XXVT8cbNmGesLpaRAcQsqR0bC/v1G79CFFxq3vEoa+Lx8\neclxmb30jV+u25wz526puf43r9coPRFqXgr9fvP8fqXgpsPGeKQANH78eNq2bctNN93kaqtVqxbJ\nyclkZGS4DYp+5JFHrAjRduz4+1YKpwD1zjvvWB2CbUluzVPR3G7bZhRGl1xS/OuTJxvrDOVv3nr1\n1fDjj8Ufe/HFxm72njZnjYiAn36qULh+5ZfrdscbsOlp4+eI2HNjjJrcYmwZUq2EZbvzBUDRlJSU\nxMyZM7n//vupVKAaXr9+PZUrV3Y7tmfPnqxYscKnM8mslJSUxJIlSxg8eDAxMTFWh2PL37cyq04I\nYapffzXW/MkfNH3TTcatseL062dsvDp7dvGvb95sTL/PHzB9/vnQrJl5sQcl7YT0JPcxRvk9SF3e\ngTrdPb/39F5jL7aqLaBKnTLtp2a1PXv2kJaWRvv27V1tP//8M9dccw0bN26kdevWFkbnP5mZmfzw\nww/8+uuvALRr186tZ02UTGbVCSH84sQJY3+wESM87yH2wANG8VSpklHstG/vuXCaPNlYNNGTdu2M\nhyiGMxu+62Dsp+bMzGtUENPE6Dmq1cWYrVaS2AuMR4A6ceIE0dHRbr1GDz74IFlZWXz//feutiuu\nuIJTp04V6V2yq507d/Ldd99x6tQpV1tKSgpZWVkhkwN/ksJJCOEmKckohvL3Ijtxwlh3qLjOB6Vg\n/nwYMMBz4fT110YvUvXqpXdghPAyMkVlp7mPMYqIgVZjPB8fVgka3wSR9fJusbUw1j0KL2aAVxDQ\nWrttKPvnn3/SqlUrli5dSp8+fVztb775ZpFNbsPDwwN2OQBfSktLY8GCBWzfvt3VFhERQe/evbni\niisIC5P5X2aQrAaoCRMmWB2Cbdk5txkZxoDnpCTPx0yeDIUmC7lxOo2tNM6eNVaijo8/N6aosOrV\njRWr8/8dKy63DRtCjRpBcdfHWknfwaq/w6LuMLsefFUNFnSClYNh63gmvP6/0s/R4SVo/QCcHw/V\nWgdt0TR+/Hi34giMKf8ff/wxlxQaANeiRQtq165doc8Lxt8JJ0+eZNKkSW5FU4sWLbj//vvp3r17\nwBRNwZjb0kiPU4BKT0+3OgTbCsbc/vmnMQuscePiXx8/HiZOhPye+rvugo8+Kv7Yiy7yvFs9GJ+x\nbp13cQZjbk3lzIWzB4xeo1qdoHJNz8eePWjcZqt6ITTod67XKLYlVDmP9C3P+S1sf8nIyOCOO+5g\n5MiRXFdgtdAuXbpQtar7bcWwsDDuvPNOU+IIxuu2evXqNGvWjB07dhAdHU2/fv245JJL3HrpAkEw\n5rY0MjhcCIts2ACHD58bND10qOfC6KKLjFWo33ij+Nd/+MEodvIHTbdoYUyxF37izIZDiwrcWsub\nyn96DziNzVO5+jto2N/aOC20Zs0a5s6dy0svveTWPnjwYO6++27i4+Mtiix4nTp1iuXLl3PNNdfY\nZlagVWRwuBAWO3AApkwxps57WmMuPt4onMC45dW1q+fC6fPP4bzzPH9enz7nbpcJEzizS5lmr2D5\nDaDCIba50Vvk6jXK2zIkNnSm/y1cuJA6derk/0MEGLPfvv76a5566ikiIyNd7TNnzrQiRFuoVq0a\nAwcOtDqMkCOFkxAmOHECpk6FwYM9F05LlxqLJtapU/wCjQUVmGktzKA1ZBwpMHW/0JYhNdrDNUs8\nvz8sAm5INAZmq8AYW+IPWVlZbNq0ic6dO7vdInr88cfp3bu3W+F06623FrtdifCs8AB5ERikcApQ\nR48erfCAR1E8f+T2kkuMXqeSeFrxOpgF7XW78UnY9vK555H18xZ8bAUN46Gmh/UTCopqYF58WJ9b\nrTXp6eluiyouWbKEAQMGsHv3brdNbpctW0bNmiWM5wowVue2MK01mzdvZuXKlQwbNozo6GirQ/Ja\noOXWF0LnT6Mgc/fdd1sdgm35Krc7d8Ivv/jkVLZh+XWbmwEnt0PSPNjxJqx9AH6MN9pK0nQI9JwN\n/TfCzWlw0yG4diV0+xjaPQ2NBvkn/hJYndsbb7yRYcOGubX17NmTX375hUaNGrm116pVK6h6SqzO\nbUGpqal8+umnfP3116SkpLB48WKrQ6qQQMqtr0iPU4B67rnnrA7Btiqa27Nnjd3tJ0yAq66CRYt8\nE5cdWHLdZp+GnwYZt9bSk4C8CS/hkXnjjVoaY5RKUrOD8Qhg/srtnj17GDRoENOnT3ebSDN69Gi3\n7UsAqlatyhVXXOGXuMwUCL9vc3NzWb16NcuWLSOnwPofmZmZOJ3OgFleoLwCIbe+JoVTgJKZf+ap\nSG6TkoxiKTER/vUveOIJHwZmAz65bl3jjfLGGEXEQJMS1k+IiIHoRlC727mB2FVbGrfObDTeyIzf\nCTNmzGDTpk1MnDjR1dagQQN69OhRZJZWweUC7Mbq37dJSUnMnTuX5ORkV1vVqlWJj4+nTZs2FkZW\ncVbn1gxSOAlRDg0awPXXw8iREOS/zwJH8o9wcIGxl1r+4OycM+debzig5MJJKej+iflxBrHMzEwm\nTZpE37593fZ0S0tL48iRI27HRkZGMnnyZH+HGLIOHTrE1KlTKbg00OWXX06fPn2oUtqsEWEJKZyE\nKIewMHjtNaujCBK5GcaCjjHNIKKENWZSfob9s4yeotrdodmdBRZ/bG70KIkyS0pKYsOGDW7T1CtV\nqsSrr75KrVq13Aqn0aNHM3r0aCvCFHnq169PixYt2LVrF/Xq1WPgwIFFxoyJwGKffmybmTp1qtUh\n2Jbk1oecOXB8Hez7Erb+H1P/3ROWXA1zGsMXUTD/YkjdUPI5LnkKHLuhzyK4/D1o+w9ofAPUaCdF\nUwHFXbepqalut3cAvv32W2666SYyMzNdbWFhYRw4cKDI4G5hsPJ3glKKAQMG0LdvX0aOHGm7osmO\nv2+lcApQ69eXuHCpqIDScnv6NDz/vPG/ohS5Z2FhF/h5CGx/lfWbdxpji5rdBV2nQd+fjDWQRIWt\nXbu2SFunTp149dVX3dpuu+02kpOTi9zmCdbBxf5g9e/bGjVq0KNHD1tuTGx1bs0gW64IUciePXDZ\nZfDllyGyGrczB9ITz40xyl/0MW2XsX5RaeOHjq83VsUuaR82USGLFi3ihhtuYN++fdQpsKLq8uXL\nadq0KU2bNrUwOlGanJwcIiJkZEwgky1XhKiAZs1g3z6ICYW7RBufhG0TQedNf1bhEHOBMcaobk9j\nzFFpaskfIr706KOPEhERwSuvvOJqu+SSSxg/fnyRHolevXr5OzxRDjk5OaxYsYKNGzdy3333uW01\nI4KXFE5CFCMoi6as1HO9Rfk9Ru1fhOjzPb+nfl9jKn9sC+MR06SUPdmEryQlJfHCCy/wxBNPcMEF\nF7jaGzVqVKR3omHDhowbN87PEYqK2Lt3L/PmzePYsWOAscq67CtnD1I4iZC1dy8U+Pcq+GSlwm/3\nnyuWso6fe61yLaMQyjpecuFUr7fxEKb65Zdf2LNnD7fddpurLSoqilWrVnH48GG3wunhhx+2IELh\nK2fPnmXx4sVs2HBuUkRYWBjR0dGy95xNyGjBAOVwOKwOwbb693cwejS0aAHFjLe1ljPbKIQOLYIj\ny0s+NjwGzh4yZp+1/Qf0+Byu+w0GH4fBx6Dfr8ZrfhTq121ubi5r1qzh0KFDbu3ffPNNkUHctWrV\nYvPmzWVeeTvUc2smX+RWa82WLVuYNGmSW9HUqFEjRo0aRZ8+fUKyaLLjdSs9TgFq7NixVodgSzNn\nwurVY/n5Z3jjDejUycJgjq6BlJUFBmTvgjP7QOcarzfoB3VLGMMSXhn6LvNLqGUVStet1prk5GTq\n16/vasvJyaFXr1689tprbrl4/vnnefnll4s7TZmFUm79zRe53b17N7NmzXI9r1y5Mn379qVz584h\nPaPRjtetzKoTIeWNN2DVKnjzTTi/hDtYFZZ5HCpVg7AS/jZZNw52vV9gm5AW57YLiW0B0Y1Lfr/w\nq8K3WV5//XWee+45UlNT3QZtb9u2jZYtW1K5cmUrwhQW0VrzySefsGfPHtq0aUP//v2pVq2a1WGJ\nMirPrDopnERI0drYocMnJzp70L23qOA0/uwTEL8Zalzi+Ry5GRBWxUcBCTNdeeWV9O/fnyeffNLV\n9tdff7Fjxw7i4uJkqrkA4Pjx4yQnJ9O2bVurQxHlJMsRCOGBz2qUswdhToEVfvNnptVoD41vyusx\nKqVLK1ymJgea1atX88QTTzB//nyio6Nd7UOGDOGiiy5yO7Z58+Y0b97c3yGKAFarVi1q1apldRjC\nZKF74zXAzZkzx+oQglZKCmze7Pl1t9zmnIETm2H/HNj+Kvw6Gn6Ig9//VfKHRDWAXt/CgG0w5Czc\nsN8Yb3TFVLj4CWh6S0guCBlM1+20adOYPn26W1vVqlU577zzOHHihFv7Aw88wDXXXOPP8IoIptwG\nm7LmNj093eRI7MeO160UTgEqISHB6hCC1j33wL33en49ISEBNj0HsxvAl7HwXXtYcSNsfg6OroJK\nVY1FIEuiwqDRIKjeVnqOCgjE6/bEiRN8/PHHHD9+3K191apV/Prrr25tF198MTNnzqRhw4b+DLFM\nAjG3dlFabk+fPs2sWbOYPHmy2x6AonR2vG5ljJOwh8xjRs/R6d3s2XGc2I6jqHN+dc/HJ840ji84\nGDuyrow3CnLHjh3j4MGDtGt3bhmGffv2ccEFF7BkyRLLe41EcNFas2HDBhYvXkxGRgYAXbt2pV+/\nfhZHJnwtKMc4KaXGAI8C9YGNwANa699KOP5h4D6gCXAUmAk8obWWPwdCTeZxmNsqbwFIRbPoxhA7\nCCihcGoy2HiIoJWVlYXT6XTbxuKpp55i+fLlbN261dXWpEkTUlNTqVGjhhVhiiB19OhR5s2bx759\n+1xtUVFRNGjQwMKoRCAIiMJJKTUEeA0YBfwKjAO+V0q10lofLeb424CXgWHAL0Ar4GPAiVF8iVBy\n4GtjFtt1eQs+yq0z2ztz5gx16tRh8uTJ/P3vf3e1P/bYYzz6qPuvAKWUFE2izHJycvj5559ZsWIF\nubm5rvb27dsTFxdHTFDuxyR8KVDGOI0DJmutp2utd2D0JKUDd3s4vhuwUmv9hdY6UWu9BEgALvdP\nuCKg7Psc6l4F510mRZMNTZ8+nUsvvZSCwwpiYmKYNGkS3bu7b0LcvHlzWrRo4e8QhY1s27aNZcuW\nuYqmmjVrcscdd3DjjTdK0SSAACiclFKVgM7A0vw2bfyGXIJRIBVnFdBZKXVZ3jmaA/HAfHOj9Z/h\nw4dbHULA++UXeOXF0+QeWgZNhpT5fZJb81Q0t2PHjuXzzz93a2vZsiXx8fHk5OQU+axQKpLkujVP\nwdy2a9eOJk2aoJSiR48ejB49OqSuM1+z43UbCLfqagPhQHKh9mSgdXFv0FonKKVqAyuVsZRvOPA/\nrfUEUyP1o7i4OKtDCGhnzsCdd0Lt2DM8+o8waPy3Mr9Xcmuesub2jz/+YPr06Tz//PNuq26npaVx\n5swZt2O7d+9epGcpFMl1a56CuVVK4XA4yM7OdttOR3jHjtet5bPqlFINgCSgm9Z6TYH2CUAvrXWR\nXiel1NUYt+b+jTEmqiXwFjBFa/2Sh8+RWXU2Mno0TJ8Ov6/P5cK6m6FmR6tDEh6sXbuWnJwct81s\nly9fzq233srq1atp0qSJhdEJIUT5ZtVZfqsOY0ZcLlCvUHs94LCH97wATNdaf6i13qq1/gajiCpl\n1UKIj4/H4XC4Pbp161Zkka5FixYVu6vzmDFjmDp1qlvb+vXrcTgcHD3qPo792WefZcIE906wxMRE\nHA4HO3bscGt/++23eeyxx9za0tPTcTgcrFy50q09ISGh2O7PIUOGhMT3+PNPeP99ePVVePOtB5k6\ne11Qfo8wAmXBAAAgAElEQVSCgvm/Rz6tNbfffjv/+9//3NofeughbrjhBrfv0bNnT0aMGFFkjZdA\n+B5gj/8e8j2K/x5Op9O1wGkwf4+C5HuU73skJCTgcDho1qwZHTt2xOFwMG7cuCLn8sTyHicApdRq\nYI3W+qG85wpIBN7SWv+nmOPXAou11k8UaBsKTAGq6mK+lPQ42cumTdCunSy7ZKXs7GwqVarker5p\n0yY6dOjATz/9RK9evVztycnJ1KpVy+1YIaxw+PBh5s6dS3p6Ovfff79ck8Il2HqcAF4HRiql7lRK\ntQH+B0QDHwEopaYrpf6vwPFzgdFKqSFKqQuUUtdi9EJ9W1zRFIwKV9fCXfv23hdNktuKGzduHFdf\nfbVb20UXXcR//vMfOnXq5NZer149+QfKB+S69V52djaLFy/m/fff5+DBg5w4cYJly5a5XpfcmseO\nuQ2Iwklr/SXG+ksvABuA9sB1WuuUvEMaYSyMme9FjHWfXgS2YvQ0LcBYxsAWJk6caHUItiW5Lbu0\ntDR69uzJggUL3NodDgcPPvigW1tERATLly+natWq/gwxZMh1651du3bx7rvvsmrVKteSFnXq1KF1\n63NzjyS35rFjbgPiVp0/BNutuvT0dLfd2YXvSG6Lt3jxYr799lvefvttV5vWmpEjRzJs2DCuvPLK\nUs8huTWP5LZ8zpw5w/fff8/mAjt+h4eH06tXL3r06OE2m1Nya55gyW1Qbrki3AXDhRasJLfw0Ucf\n0apVK7dp/qmpqWzbts1t7JJSig8++KDM55XcmkdyWz5bt251K5qaNm3KwIEDqV27dpFjJbfmsWNu\nA+JWnRAl+eor+OOPAg3ZaZArWxKWxYkTJ1iwYAGFe5ZfffVVFi9e7NZ2yy23sHTpUhmPJGyhS5cu\nNGzYkMjISBwOB3fddVexRZMQ5SU9TiKg5ebCSy9Bnz7wxht5jX+8Yzyu3wdhcgnny8zM5OTJk9St\nW9fVtnz5cq6//nr27t1L06ZNXe0bNmyQAknYWlhYGH/729+oUqWKbJUifEp6nAJU4bUsQlV4OKxa\nBf9XcE7lvi+gzpVeF012yK3Wukgv0rXXXstDDz3k1nbNNdewZ8+eIotMmlU02SG3gUpyW361atUq\nU9EkuTWPHXMrhVOAktWUz4mJgaiovCendsKJjdC07HvTFRbsud26dSv16tVj06ZNbu3jx4/nX/9y\nXwM2JiaGCy64AOWnBa+CPbeBTHLrLiMjg5SUlNIPLAPJrXnsmFuZVSeCy+YXYPt/4KYjEBFV+vFB\n7q233mLHjh28++67rrYzZ87wyiuvMGLECLfbb0KEiu3bt7NgwQIqV67MfffdR0SE3LIXFROMC2AK\nUTaJX0Cj621XNJ0+fZp//vOfbNiwwa09JiaGatWqFWl78cUXpWgSIefUqVN8/vnnfPnll6SlpXHs\n2DFbLrAoApuU6SJ4nNgCJ7dBh1esjqRCduzYwdq1a7njjjtcbVFRUXzzzTdcccUVbitv33PPPVaE\nKERAcTqdrF27lqVLl5KVleVqv/DCC+nYUTb4Fv4lPU4BqvDmh6Fi8WL473/B6SzmxX1fQKUa0CCu\nQp/hr9xqrdm5cyeJiYlu7YsXL+bBBx8kJyfH1RYeHs6OHTu48cYb/RKbWUL1uvWHUM1tcnIy06ZN\nY8GCBa6iKSYmhsGDBzN06FBq1KhR4c8I1dz6gx1zK4VTgHr88cetDsHvUlNh+HCYN8/DASe3QuMb\nIbxKhT7HrNzm77ieTylFnz593MYnAYwYMYIjR47YclxGKF63/hKqud21axdJSUmu55deeiljxozh\n4osv9tmkh1DNrT/YMbcyODxAJSYm2nI2Qkluuw0WLIDNm6FRIw8H5WZWuHAyI7ezZ89m8ODBHDt2\njJo1a7raN27cSNOmTX3yV3EwCMXr1l9CNbdOp5MpU6aQk5PDwIEDTRnbF6q59Ydgya1suWIDwXCh\n+dLvv0NCAnz6aQlFE1S4aIKK5/auu+6iZs2avPnmm662bt26MWPGDCpXrux2bIcOHSr0WcEm1K5b\nfwrV3IaFhXHrrbcSExNjWi9tqObWH+yYW7lVJwJCx46wcSMMHWp1JOfs3r2bG264gX379rm19+jR\ng65du7q1NWjQgNtuu01WKBbCBNWrV7flrW0RnORKFAGjfXvrPnv+/PkkJSUxatQoV1u1atU4ffo0\np06dcju24DFCiIo5fvw4WVlZ1K9f3+pQhCgT6XEKUBMmTLA6BFvKzMzkrrvuYv/+/W7tP/30E998\n841bW506dViyZAnt2rXzZ4hBTa5b89gtt7m5uaxcuZL33nuP2bNnk5uba1ksdsttILFjbqVwClDp\n6elWhxD0MjIy2LZtm1tbbm4un3zyCYsWLXJrnzBhAvPnz/dneLYk16157JTbAwcOMGXKFJYuXUpO\nTg4pKSn8+uuvlsVjp9wGGjvm1qtZdUqpy4FRQAvgdq31QaXUrcBerfVqH8foE8E2q06Uj9aarKws\nqlQ5N3j8pZde4tVXX+X48eOEhZ37G+Hw4cPUq1fPb/u3CSEMmZmZLF26lN9++83VppSia9eu9O7d\nu8jkCiH8xdQtV5RSDuAnoArQDYjMe6ku8FR5zydC04wZsGdPGQ48shIySt7I88yZM1x88cW88cYb\nbu133nknS5cuLXJ8/fr1pWgSws927tzJpEmT3Iqm+vXrM2LECK677jopmkTQ8OZW3bPAWK3134Hs\nAu0rgc4+iUrYWmYmPP88fPxxKQc6c2HlzbD1/0o8LCoqitdff53LLrvMrb1JkyZ07tzZrbdJCGGN\n5ORk0tLSAKhUqRLXXnstI0eOpGHDhhZHJkT5eDOrrg1Q9M94OAHULKZdeOHo0aPUrl3b6jBMUaUK\nrFsHkZGlHJiyHDIOQ9NbSzwsLCyMfv36lfnz7Zxbq0luzRPsue3RowdbtmyhevXqxMfHuy0Ua7Vg\nz20gs2NuvflT/AjQrJj2bkBZbr6IMrj77rutDsFU1apBqT3z+76AmAvgvMt9+tl2z62VJLfmCfbc\nhoeHM2zYMG677baAKpog+HMbyOyYW28Kpw+BN5VSHQANnKeU+hvwKvC+L4MLZc8995zVIVjLmQ37\nZ0GTW6CE8UjZ2dkeX/Mk5HNrIsmteeyQ2+jo6IAcX2iH3AYqO+bWm8LpJeBb4BcgFlgNfAbMAN4s\n4X2iHEJ+5t/hHyDzKDQd4vGQrKwsWrZsyfTp08t16pDPrYkkt+YJ9Nzu2bOHw4cPWx2GVwI9t8HM\njrkt9xgnrbUTeFop9QrQGqN42qy1TvV1cCKEJX4BsS2hZiePh+Tk5DB27Ngig8KFEP6Tnp7O4sWL\n+f3336lfvz4jR46UCRnC1spdOCml3gUe11qfBtYXaI8GXtVa3+/D+IQNfP01HDkCo0aVeNftnNws\n2P81tBpT4huio6N57LHHfBeoEKLMtNZs2bKFhQsXuhY5PHz4MJs3bw65za1FaPHmz4J7gehi2qMx\nFsUUPjB16lSrQ/CJQ4dg5EgotFB3yTJTjAHhpcym85ZdchuIJLfmCaTcpqam8umnnzJ79mxX0VSl\nShUGDBhAeys3nfRSIOXWbuyY2zIXTkqpykqpKoACKuc9z39EAX2Ao2YFGmrWry9x4dKgoDWMGAER\nETB5chl7mwCiz4c+30ONS0yJyw65DVSSW/MESm5Xr17Nu+++y+7du11tbdu2ZcyYMXTp0iUgB3+X\nJlBya0d2zG2Zt1xRSjkxZtGVZLzW+pkKR2UC2XLF/1asgF69YN48GDDAd+d1Op3079+fhx9+mP79\n+/vuxEKIUv3www+sWLECgGrVqhEfH0/r1q0tjkqIiinPlivlGePUH6O36TvgNqDgYPAsjH3qZB0n\n4dKzJ2zaBO3a+fa8aWlp1KlTJ+DWghEiFPTq1Yvt27fTvHlz+vTp47Y/pBChoMyFk9b6ewClVFvg\nz7zZdUKUyNdFE0D16tWZMWOG708shChVREQEo0aNolKlSlaHIoQlvFmOYCeAUioCaARULvT6H74J\nTQghRCCSokmEsnLPqlNKnaeUmgmcBXYD2ws9hA84HA6rQ7Atya15JLfm8UdutdasW7eO5ORk0z8r\nkMh1ax475tab5QheBxoDvTGKp+sxlij4C7jRd6GFtrFjx1odQsDRWvOPf/yDdevWVeg8klvzSG7N\nY3ZuU1JS+Oijj5g3bx5z587F6Qyd0Rhy3ZrHjrn1pnC6FnhIa70ScAI7tdYfAP8EHvFlcKEsLi7O\n6hDKbdo0OHDAyzdnp8Hm5+Gs5790jx8/zqJFizhy5IiXH2IIxtwGC8mteczKbU5ODsuWLeN///sf\niYmJACQlJbFnT+jM9ZHr1jx2zG25xzgBVYFDeT+nAnWAPzFWEfftNvYiaKSlwbPPwsmTMG6cFyc4\n8C1sfg6ae95J+7zzzmPTpk1exyiEcLd3717mzZvHsWPHXG21atViwIABNG/e3MLIhAhc3hROfwAX\nAvuAzcDdSqmdwN1AaN0YFy5VqxpLD1Sr5uUJEr+A2t0hpnGJhwXj4npCBBqtNfPnz3e77R0WFkb3\n7t3p1auXDP4WogTe3Kp7B7gg7+cXgZuAIxi36gJy8ctgNGfOHKtDKLeaNSE83Is3ZqXCoYXQdIjP\nYypOMOY2WEhuzePL3CqlqFz53IToRo0aMWrUKK655pqQLJrkujWPHXNb7sJJa/1h3pgmtNZrgGZA\nT6CZ1voTH8cXshISEqwOwX/2zwFnDjQe7PGQadOmcfDgQZ98XEjl1s8kt+bxdW6vvvpq6tWrR3x8\nPMOHD6devXo+PX8wkevWPHbMbZm3XCnTyZRqp7Xe7LMT+pBsuRLAfuwHuRnQd1mxLx85coTmzZvz\nwQcfcOut5mz8K0Qo0lrL7W8hMG/LFcDY7Bdwaq1zCrRdBDyPsRyBN+OmRKjKOAqHl0CXtz0eUrdu\nXQ4ePEhkZKQfAxPC/qRoEqL8ynyrTinVUCn1I3AGOK2U+j+lVBWl1PvA70Al4BqT4hQBaPduOHOm\ngic5MBvQ0PhvJR5WrVo1tzEZQgjPsrKyWLRoESkpKVaHIoTtlKd3aCLG0gNPYCx6+U+MRTC3Am20\n1n/5PjwRyP7+d2jSBD7/vAInqdEBOrwMkXV9FpcQoezPP/9k/vz5nDx5kqSkJIYNGyY9S0L4UHkG\nh/cG7tVavwrcAihgttZ6hBRNvjd8+HCrQyhRejr89hv06lXBE9XuChc97vHl1atXk5OT4/F1bwR6\nboOZ5NY8peX29OnTzJo1i88++4yTJ08CxkKWobZ9ijfkujWPHXNbnh6n+hh706G1PqSUSgfmmhKV\nCPjVVlevhpwcHxROJTh8+DA9evRgypQp3H2354UxyyvQcxvMJLfm8ZRbrTUbNmxg8eLFZGRkuNqb\nNWvGgAEDOO+88/wVYtCS69Y8dsxtmWfVKaVygfpa65S856eADlprn6zLr5QaAzyKUaBtBB7QWv9W\nwvHVgf/DGJBeC9gLPKy1XujheJlV50Pvvw/PPAMHD0KYN6uBldGGDRto2bIlVatWNe9DhAhSTqeT\nTz75hL1797raoqKiiIuLo0OHDnKLTogyMmtWnQI2K6Xyd36MAVbnFVQuWuuG5QkWQCk1BHgNGAX8\nCowDvldKtdJaHy3m+ErAEuAwxgKcB4GmwInyfrbwzqhRMHy4uUUTQKdOncz9ACGCWFhYGHXr1nUV\nTu3btycuLo6YmBhrAxPCxspTOI02LQqjUJqstZ4OoJS6DxiAsY3LxGKOvweoAVyhtc4v3BJNjE8U\nIwQXGBYi4PTp04fk5GR69uxJixYtrA5HCNsrc3+B1npyWR7lDSCv96gzsLTAZ2mMHqVuHt42CPgF\neFcpdVgptVkp9YRSyuT+D/9ZuXKl1SFYquCmo74W6rk1k+TWPJ5yW6VKFYYNGyZFUwXIdWseO+Y2\nEAqN2kA4RTcITsYY71Sc5sDNGPH3B14A/gE8aVKMfjdxYnEdbaHh2LFjNGrUiM8rtM6BZ6GcW7NJ\nbs0juTWP5NY8dsxtIBRO3gjDKKxGaa03aK2/AsYD91kblu+YVTQEhD8nw7pxHl+Ojo7mvffeo0+f\nPqZ8vK1zazHJrW+dPHmSr776iuPHj0tuTSS5NY8dcxsIhdNRIBcovMNkPYzB38U5BPyh3acEbgfq\nK6VKHLcVHx+Pw+Fwe3Tr1q3IDs6LFi3C4XAUef+YMWOYOnWqW9v69etxOBwcPeo+jv3ZZ59lwoQJ\nbm2JiYk4HA527Njh1v7222/z2GOPuZ5HR0eTnp6Ow+Eo0tWZkJBQ7NoYQ4YMCbjvART9Hrveh/QD\nHr/HsGHDqFGjBnXrnlsU05ffIzo62jffI0/Q//fw4fd47LHHbPE9rP7v4XQ6eeGFF+jTpw/btm1j\n3rx5REVFBd33gOD471Hwd0Iwf4+CAuV7TJgwIeC+R0JCAg6Hg2bNmtGxY0ccDgfjxnn+Y74wn27y\n6y2l1Gpgjdb6obznCmOw91ta6/8Uc/x4YKjWunmBtoeAx7TWjTx8hixH4AP79sH69RAfD1WqeHGC\ntF0w90K48itoMtjn8QkR7A4fPszcuXM5ePCgqy02NpYRI0ZQvXp1CyMTwr5M3eQ3X95A7MbAgQIz\n27z1OvCRUmod55YjiAY+yvus6Xmf8++8498Dxiil3gLeBlphbAXzZgXjEKVYtAgeeABOeLvww74v\nICIGGsYX+7Ls1i5CVXZ2NsuWLeOXX36h4B+0nTt3pm/fvrLJtRABoty36pRSkUqpScBZjJXEm+a1\nv6GUesSbILTWX2IsfvkCsAFoD1yXv9gm0IgCA8W11geA64AuGItlvgm8Abj3Bwaxwt2VgWLkSNi/\nH7z+HZ74BZzvgIjoIi+dOXOGVq1asXBhsWuY+kyg5tYOJLfeycrK4r333mPVqlWuoql27doMHz6c\ngQMHEhkZKbk1keTWPHbMrTc9Ti8BPYB44JsC7cuBpzB6j8pNa/0u8K6H14qMEtZarwG6e/NZwaBJ\nkyZWh+BRnTpevvHkNjixGdq/WOzLGRkZDBo0iNatW3sfXBkEcm6DneTWO5UrV6ZZs2akpqYSHh5O\nz5496dGjBxER535FS27NI7k1jx1zW+4xTkqpvcDtWuuflVJpGNuu/KWUuhBYq7UOyJvwMsYpAGx6\nFna+CTcdgXBvBkgJYV8ZGRnMnTuX3r17U7t2bavDESKkmD3GqS7GFieFRWFsyyJEUVobt+ka3SBF\nkxDFiIyM5Oabb7Y6DCFEKbwpnDYA/TAGaBc0DFhT0YCEjfX4HMKkaBKhSSY+CGEP3qzj9BQwUSn1\nBsaK3/cqpeZi7GX3lC+DC2WF17cIekpBzY5QvW2Rl7Kzs7nxxhv59ddf/RKK7XIbQCS3xdu/fz9T\np07lhNfTUSW3ZpLcmseOuS134aS1/hG4HGOrlF0YW59kAj3yBmwLH3j88cetDsHNsWPw88+Qne37\ncx85coRTp075bbp1oOXWTiS37jIyMpg/fz7Tpk0jKSmJ7777Dm/XzpPcmkdyax475jYgFsD0h2Ab\nHJ6YmBhQsxGmT4dhw4wCqmZNq6OpmEDLrZ1Ibs/Zvn07CxYsIC0tzdXWoEED7rzzTq/+SJDcmkdy\na55gya2pg8OVUvOAGcA3Wuuz3oUoShNoF9ry5dCuXfAXTRB4ubUTyS2cOnWK7777jp07d7raKlWq\nRO/evenatSthYd7tdCW5NY/k1jx2zK03g8OTgHeA95VS32AUUYu11k6fRiYCyvLlcN11VkchRGA7\nffo0kyZNIisry9V24YUXEh8fT40aNSyMTAjhK96McboXYxXvO4BKwGzgoFLqLaVUVx/HJwJAejpE\nRcFVV/n2vE6nk6effpq//vrLtycWwiKxsbG0bWtMgIiJiWHw4MEMHTpUiiYhbMSrPmOtdY7W+lut\n9a1APeAxoBfwsy+DC2WFd5O2UnQ0bNwIg73Zk9fpeTT53r17mTJlittmpv4QSLm1G8ktxMXF0bVr\nV8aMGcPFF1/ssyUIJLfmkdyax4659XqTXwClVC3gFozep3bAZl8EJSA9Pd3qECrOmQtzW8PFT0DL\nkUVebt68Ofv373fbVsIfbJHbACW5hejoaPr16+fz80puzSO5NY8dc+vNlitRwA3AbUAccAj4DPhU\na73V5xH6SLDNqrOF5J9g6dVw7c9Qx7bbCooQ43Q6vR7gLYQITGZvuZICnAVmAtdorVd6cQ4RChK/\ngOjGUPsKqyMRosK01mzatImffvqJYcOGUa1aNatDEkJYwJs/m4YCDbTWo6VoEh45cyBxJjS5BZT7\nZaa15quvvnJb30aIQHb8+HFmzJjBnDlzSE1NZeHChVaHJISwiDez6uZqrXPMCEacc/ToUatDqJgj\nyyAzBZoOKfLSn3/+yZAhQ1ixYoX/48IGuQ1gdsttbm4uK1eu5L333nOb/RkWFkZOjn9/Ddott4FE\ncmseO+a2TIWTUmqVUqpG3s+/5D0v9mFuuKHj7rvvtjoEAN55B37/3Ys37vsCYptDrS5FXmrVqhV7\n9+7lOosWhgqU3NqRnXJ74MABpkyZwtKlS11FUrVq1Rg6dCiDBw/2+6QGO+U20EhuzWPH3Jb1//k/\nAVkFfg6NfVos9Nxzz1kdAtnZ8Mor8Mwz0LFjOd7ozIb9s6HlvcbmvsWwcjXZQMitXdkltykpKUyd\nOtX1XClF165d6d27N5UrV7YkJrvkNhBJbs1jx9zKXnWiRFpDTg5UqlSON6X8Aot7QP8NULODabEJ\nYaZZs2axZcsW6tevz6BBg2jYsKHVIQkhTGL2XnXbgCu11scLtVcHftFaX1Tec4rApVQ5iyaAOt3g\nxoMQWa/IS9u3b6dNmzY+WxRQCLNcd911NGzYsEL7ywkh7Meb3wZtKL7gigRaVCwcYRtR9Yvcpvvj\njz+46KKLmDdvnkVBCVF2sbGxdOvWTYomIYSbMv9GUErFKaXi8p5enf8879EfeBxINCXKEFRwfIVd\ntGjRgvnz53PttddaGocdcxsogim3BTfiDQbBlNtgI7k1jx1zW54/pRbmPTTweYHnC4F5wGCM4kn4\nwPr1Jd5iDUrh4eHEx8cTGRlpaRx2zG2gCIbc5uTk8MMPP/DWW29x5swZq8Mps2DIbbCS3JrHjrkt\n8+BwpVQVQAF7gMswVhDPl6O1zvV9eL4jg8OFEHv27GHevHkcP24M0WzXrh033XSTxVEJIaxmyuBw\nrXVm3o8NKhCbCBKzZ0O7dnDhhb4539mzZ4mKivLNyYQop/T0dBYvXszvBRYlCwsLo0aNGmitZbKC\nEKLMylQ4KaVGAR9rrTPzfvZIa/2+TyITlsnJgTvvhKefhn/+s+LnS0pKok2bNnz77bf07t274icU\nooy01mzZsoWFCxe67dLeuHFjBg4cSN26dS2MTggRjMra4/Q8MAvIzPvZEw1I4RTkNmyAM2egV69y\nvCknHTKSIbZZkZeioqJ48skn87tBhfCbxMREZs+e7XpepUoV+vbtS+fOnaWXSQjhlTINDtdaN9Ba\nHyvws6eHrBDnIw6Hw7LPXr4coqKgXHXOgW/g2+aQnlTkpVq1avGvf/0rYHaTtzK3dhdouW3atClt\n2rQBoG3btowZM4YuXboEZdEUaLm1E8mteeyY2wpvtqSM30Ctgf1a6+CZohLgxo4da9lnV6kCQ4ZA\nuXaWSPwCzrscos83LS5fsTK3dheIue3fvz8dO3akdevWVodSIYGYW7uQ3JrHjrkt95YrSqmJwDat\n9UdKqTBgKXAVkAbEa61/9n2YFSez6kyUdRJm14UOL0PbR6yORgghhCiX8syq82ZJ3FuBrXk/DwDa\nAh2B/wGveHE+EewOfAPOLGh6i1tzamoql156Kb/99ptFgYlQcPr0aatDEEKEEG8Kp7rAobyfBwBf\naq03AZOB9r4KTASRxC+gzpUQ3cit+dSpU7Ru3ZomTZpYFJiws7S0NL766ivee+89txlzQghhJm8K\npyNA67zbdP2AJXntkRiz6oQPzJkzx+oQyibzGBxaBE2GFHmpadOmJCQkUK9e0c1+rRQ0uQ1C/sit\n1pp169YxadIktm3b5lqjye7kujWP5NY8dsytN4XTJ8AXwAaMweWL8tovA3b6KK6Ql5CQYHUIZbP/\na8AJTQZbHUmZBU1ug5DZuU1JSeHDDz9k3rx5ZGYaa/JGR0fTrFnRZTDsRq5b80huzWPH3JZ7cDiA\nUuoOoDHwudZ6T17bPcBJrfVM34boGzI43CRbX4GUn+HquVZHImwsJyeHFStWsHLlSpxOp6u9Q4cO\nxMXFER0dbWF0QohgZ8qWKwVprWcU02a/LZBDzPffw/r18K9/QZmXubn4X1Co+D5z5gz3338/Tz75\nJK1atfJ9oCLk7Nq1i+XLl7ue16xZk4EDB9K8eXMLoxJChCJvbtWhlOqqlPpKKbUl7/GlUupyXwcn\n/GvrVli8uBxFU75Cb9i3bx9r164lIqLCy4QJAUDr1q1p2bIlYWFhXHnllYwePVqKJiGEJbxZx+kW\n4DNgPpC/ZlMPIB64TWv9lU8j9BG5VedfsnGq8LUTJ06QmZkZcJMNhBDBz+x1nJ4FntRaX6+1npj3\nuB54CnjOi/OJYgwfPtzqECokkIumYM9tIDMztzVq1AjpokmuW/NIbs1jx9x6Uzi1xNjwt7BZQIuK\nhSPyxcXFWR2CbUluzeNtbp1OJ0ePHvVxNPYi1615JLfmsWNuvblV9xfwktZ6WqH2e4B/a60DsniS\nW3Xmy87O5q233uLOO++kTp06VocjgsTBgweZN28eaWlpjBkzhsjISKtDEkKEGLNn1b0JTFJKtQNW\n5bX1AEYB//TifMImNm3axNNPP821114rhZMoVVZWFj/++CNr1qwh/w+4JUuWMHDgQIsjE0IIz8pd\nOGmt31JKpQD/AEbmNe8Ahmutv/BlcCJAJf8I4dFQu6tbc+fOnUlOTqZq1aoWBSaCxZ9//sn8+fM5\neX9UbIoAACAASURBVPKkq61u3bp07NjRwqiEEKJ0Xi1HoLVO0Fp30VrH5j26SNHkWytXrvTbZ/32\nG0yZAgXWFSzZhn/CtgnFvhQMRZM/cxtqSsvt6dOnmTVrFp999pmraAoPD6dPnz6MGjWKRo0alfj+\nUCbXrXkkt+axY27LVTgppa5XSk1VSn2ilBpmUkwCmDhxot8+69NP4eWXIawsV8Ppv+D4b9C06N50\nwcKfuQ01peV29+7dbNmyxfW8WbNmjB49mp49exIeHm52eEFNrlvzSG7NY8fclrlwUkqNAL4GrsHY\nl26qUmq8rwJRSo1RSu1RSp1VSq1WSl1WxvfdqpRyKqVm+yqWQPD555/77bOWL4errirjwfu+MG7T\nnX9uHIrT6WTx4sXk5uaaE6CP+TO3oaa03LZv355mzZoRFRXFDTfcwN///nfOO+88P0UX3OS6NY/k\n1jx2zG15epweAl7WWl+gtW6DMRj8QV8EoZQaAryGsUZUJ2Aj8L1SqnYp77sA+A+wvKTjgpG/9t46\neRJ+/x169SrjG/Z9YRRNETGuptWrVxMXF8fq1avNCdLHZF8z85SWW6UU119/PWPGjKFDhw4Bvd5X\noJHr1jySW/PYMbflKZxaAB8UeP4hUEUp1cAHcYwDJmutp2utdwD3AenA3Z7eoJQKA2YAzwB7fBBD\nSDp1CoYMgauvLsvBO+HERmh6q1tzt27dWL9+Pd27dzclRmEv1atXJyYmpvQDhRAiAJWncIoETuc/\n0Vo7gUwgqiIBKKUqAZ2BpQXOrYElQLcS3voskKy1/rAinx/qGjeGhARo1qwMB+/7AiKqQsP+bs1K\nKTp16iS9B4KMjAwOHTpkdRhCCGGa8s6qe0op9X/5D6Ay8GihtvKqDYQDyYXak4H6xb1BKXUlMBwY\n4cXnBYXHHnvM6hCKSvwCGl0P4cG9QGFA5jbIaa3ZunUr/fr14/PPPyczM9PqkGxHrlvzSG7NY8fc\nlmcdp1+Bywu1rccYk5SvfMuQe0EpFQtMB0ZqrVPN/jyrNGnSxOoQ3OVmQf1r4XyHq0lrzaFDh2jY\nsKGFgZVfwOU2yJ08eZLvvvuOP/74g+joaE6dOsWyZcu47rrrrA7NVuS6NY/k1jx2zG2Ze5y01ldo\nrbuV8vBmkMtRIBcovHtnPeBwMce3AJoCc5VS2UqpbOBO4HqlVJZSqsSbTvHx8TgcDrdHt27dmDNn\njttxixYtwuFwFHn/mDFjmDp1qlvb+vXrcTgcRfbaevbZZ5kwwX29o8TERBwOBzt27HBrf/vtt90q\n8wceeID09HQcDkeRdTASEhKK3ThxyJAh5n2PV9+Azm9C/T6u73HVVVdx/vnns2bNGo/fAwis7zFh\nAg888ICrraz/PQLxexRkxfcYNGgQq1evZtKkSfzxxx8AHD16lIMHD3LFFVcEzfcIlv8e+ddtsH+P\nfIH0PQr+Tgjm71FQoHyPo0ePBtz3SEhIwOFw0KxZMzp27IjD4WDcuHFFzuVJufeqM4NSajWwRmv9\nUN5zBSQCb2mt/1Po2MoYGw0XNB6IxZjl96fWOqeYz5C96nwsfzHD22+/nYgIb3bvEcHq8OHDzJ07\nl4MHD7raYmNj6d+/P23btpXxbkKIoGL2XnVmeB34SCm1DuOW4DggGvgIQCk1HTigtf631joL2Fbw\nzUqpExhjyrf7NeoQFxsby1133WV1GMICSUlJbkVT586d6du3r2zQK4SwPa+2XPE1rfWXwKPAC8AG\noD1wndY6Je+QRngYKG5XhbswfS0rCwYPhnXrTP2YgGR2bkPBpZdeSpMmTahTpw7Dhw9n4MCBREZG\nSm5NJLk1j+TWPHbMbUAUTgBa63fzFteMyhsvtbbAa3201h7XdNJaD9da3+SfSP3j8ccfN/X8KSmQ\nmgre3lEJllXCi2N2bkOBUoqbb76Ze++9123wp+TWPJJb80huzWPH3AbEGCd/CLYxTomJiQE7G+GP\nP/7gqquuYuHChXTo0MHqcMotkHMb7CS35pHcmkdya55gyW15xjh51eOklLpcKfWBUupHpVTDvLZb\nlVJXlPZeUTaBfKFFRUVx++2307p1a6tD8Uog5zZQHDt2jAMHDpT7fZJb80huzSO5NY8dc1vuwkkp\n5QB+AqpgrOydPxq0LvCU70ITltNO+PVeOL7Brblx48a8+uqrMhDYhnJzc1m+fDnvvfces2bNIisr\ny+qQhBAioHjT4/QsMFZr/Xcgu0D7SoytU4RdpKyCXe9DzhmrIxF+sH//fiZPnsyPP/5Ibm4uJ06c\nCJqNm4UQwl+8KZzaUGBfuQJOADUrFo7IV3jBMEskfgFR50Mde23eGxC5DSAZGRnMnz+fadOmkZJi\nTGRVStGtWze3hSzLQnJrHsmteSS35rFjbr1Zx+kI0AzYW6i9G7CnogEJQ3p6urUBOHMhcSY0HQrK\nqK+TkpK46667eO+997jwwgutja8CLM9tANm5cyfz5s3j9GnX/t00aNCAQYMG0aBBg3KfT3JrHsmt\neSS35rFjbss9q04p9QzwN4xtTlYA1wBNgHeAiVrrN3wdpC8E26w6s2gN48bBsGHQsWMJByb/CEv7\nQNxqqN0VgE2bNvHoo4/y1VdfUb16db/EK8y1Zs0aFi5cCEClSpXo06cPl19+OWFhAbNSiRBCmM7s\nlcNf+n/27jyupvSPA/jnnPZSUVkitIxIgwhTaEWFylqGyZJl7AbzYxaGMGQnxpJh7NkVyhJDZR0q\n6yBLCSkpiVbden5/pDud7tIt93Zvt+f9et3XS895zjnf85R7v/c5z3keAGoArqF0YPh1ADyULo+i\nkEkT9Z9Hj4CgIKBfv0oqJh8EdEwBw//WdW7fvj0iIyNlGh9Vs7p06YK7d+9CR0cHffv2Rf369eUd\nEkVRlEKrcuJECCkB8BvDMMsAtEbpGnH3CCFZ0g6Okr6YGEBFBbC3F1OphAe8PAqYj6n+DJlUrcCy\nLEaMGAENDQ26vhxFUZQEqt0fTwjJJYTEE0JiaNIkfRVXmJaWmBjA1haoV09MpTcXgMIMoOVQmcQg\nb7Jq29pKU1NTakkTbVvZoW0rO7RtZUcZ27Y68zidEveSRZB10ZgxIleY+SI9ewKTJlVSSf9roPMf\nQIOOAICsrCzMmjULaWlpMomppsmqbRVRYmIiXr58WWPnq0ttW9No28oObVvZUca2rU6PU3KF12uU\nTn7Z7fPPlBQEBATI5LhjxpQODBdLuylgOYV/m+7Bgwc4ePCg0tzKkVXbKpK8vDyEhYVhz549OH78\nOHg8Xo2cty60rbzQtpUd2rayo4xtK7W16hiGWfr5eL9I5YBSRp+q+zIlJSX0SatagBCCu3fv4uzZ\ns8jPz+eXe3p6lj0xQlEURVUg66fqRNmB0iftFDJxor4MTZoU37t37xAeHo6kpP+mU9PU1ETv3r3R\nsWNHOUZGURSlPKSZOHUCdwkWiqJqyPXr1/H3339zbslZW1vDw8MD9cQ+CUBRFEVVRXUGh4dUeO1n\nGCYKwF4Af0k9wjpq+/bt8g4BeXl52L17t9LN/KoIbSttJSUl/KRJX18fw4YNw5AhQ2o8aVLGtlUU\ntG1lh7at7Chj21bn/gtT4VUC4DaAwYSQuVKMrU6Ljxd7i7VGXLp0Cf7+/krzNF0ZRWhbabOzs4Ox\nsTHs7OwwefJkWFpayiUOZWxbRUHbVnZo28qOMrZtlQaHMwyjAsAWQAIhJFtmUclAXR8cHhAA5OUB\nK1ZUbb+0tDQ0adJEJjFR0lVcXAwVFRV5h0FRFFXrVGVweJV6nAghxShdn86w+uFR8mBgUPoS6cVh\n4LwTwMvnFNOkqfagSRNFUZTsVWdw+AMAzQEkSjkWSoamT6+kwvP9AC8PUNWqkXgoyRFCEBsbiyZN\nmqB58+byDoeiKKpOq84YpzkAVjEM04thmAYMw6iXf0k7QKoGFH0AXp/iL7FSVFSE27dvyzkoCgDS\n09OxY8cOnDp1CidPnkRxcbG8Q6IoiqrTqpM4nUXpOKezADIA5Fd4UVLg7e1dcyd7dQIoKQRa+AIA\nzpw5g44dO+Lhw4c1F0MNqtG2rSYej4cLFy4gODiYv2TK27dv8fTpUzlHJl5taNvairat7NC2lR1l\nbNvq3KrrI/UoKAFTp06tuZMlHwSM7AGdFgCAPn364MKFC7Cysqq5GGpQjbZtNSQlJSE8PBzv3r3j\nlxkaGsLT0xOmpqbyC0wCit62tRltW9mhbSs7yti2Ej9VxzDMfACrCCG1clKfuv5UnUifsoBjjQGb\nlUCbH+QdTZ0XERGB2NhY/s8sy6JHjx5wcHCAqqo056ulKIqiyshqyZUFALYAqJWJEyXCy1CghAe0\n8JF3JBQAXV1d/r+bN28OT09PNGrUSI4RURRFUeVVJXFiZBYFJTNbtwJpacD8+SIqvDwKNHIEtJui\npKQEubm5nA9vqmZ1794dT58+Rfv27WFrawuGof/tKIqiFElVB4dLPlsm9UXCwsKkcpx9+4C7d8VU\n6BYCdN0KAIiOjkaTJk3w5MkTqZxbUUmrbWVBRUUF/v7+6Ny5c61MmhS5bWs72rayQ9tWdpSxbaua\nOD1mGOaduJdMoqyD9u/f/8XHKCgA/vkHcHQUU0ldH9ArXZ7DysoKS5cuxVdfffXF51Zk0mhbWaqN\nCVMZRW/b2oy2rezQtpUdZWzbqgwOLwEwA4DYpVYIIbukEJfU1cXB4ZculSZNt24BNjbyjob69OkT\nLly4gPbt26Np06byDoeiKIr6TFaDwwHgACEkvdqRUTWqeXNgyRKgXTt5R0I9fvwYERER+PDhA5KT\nkzF+/HiwbHWmUaMoiqLkqSqJEx3fVMuYmgK//lp5vbJex9p8i0hRffz4EWfOnMGDBw/4ZRkZGUhN\nTUWzZs3kGBlFURRVHVX5yks/VZVUfHw82rdvj+TkZHmHojQIIYiLi8PGjRs5SZO5uTkmTZpEkyaK\noqhaSuLEiRDC0tt0Ncff37/GzqWuro5vvvmmznyYy7ptCSHYs2cPwsPDUVhYCADQ1tbGwIED4efn\nBwMDA5meX55q8u+2rqFtKzu0bWVHGduWTkWsoNzc3GR38MJ3gHoD4POtuXbt2mHbtm2yO5+CkWnb\novSWp4mJCZKSkgAAHTp0gJubG7S1tWV6XkUg67aty2jbyg5tW9lRxraV+Km62q4uPlUn0lk7oIEN\n0HWLvCNRWjweD4cOHYKdnR3Mzc3lHQ5FURQlhiyfqqNqu5znQOY/QGu6Lp0sqaqqYvjw4fIOg6Io\nipIy+jy0Erp3D2jVCkhIELLxxSFARQto5oUnT57Az88Pb968qfEYKYqiKKo2oomTgrp8+XK199XU\nBDw8SudxEpB8AGjaD1Crh9evX+Pp06fQ19evfqC10Je0LQC8f/8eBw4coAmnEF/atpRotG1lh7at\n7Chj29LESUGtWLGi2vu2agVs2AAIjEX+8ATIugW0HAoAcHJywvXr16GpqfkFkdY+1W3bkpISXLt2\nDZs2bUJCQgJOnjyJkpISKUdXu33J3y0lHm1b2aFtKzvK2LZ0cLiCysvLk/5TWPd/Bx4sAwalA6rK\n/4SXKNVp29TUVJw8eRKpqan8Ml1dXYwePVqppxeoqry8PGRkZCAjI0PeoSid/Px8aGlpyTsMpUTb\nVnYUqW2NjIzQokULodvo4HAlIJNH15MPAs2863TSBFStbT99+oSoqChcv34d5b9kdOnSBa6urnWu\nt64yGRkZsLKyQl5enrxDoSiK4tDW1sbDhw9FJk+SoolTXVGQDhSkAS1/x+vXr7F3715MnDgRenp6\n8o5MYfF4PAQHB+Pdu3f8skaNGsHT0xPNhQ4gozIyMpCXl4e9e/fCyspK3uFQFEUBAB4+fAg/Pz9k\nZGTQxImSkGYjYGDpbaYbJ8KxbNkyTJgwQc5BKTZVVVW0adMGV69ehYqKCpycnNCtWzeoqKjIOzSF\nZ2VlVStuiVMURVUVHRyuoGbPni39g7KqAKuKAQMG4PXr13XuaboyVWlbZ2dndOjQAZMmTYKDgwNN\nmiqxbt06eYdAURQlUwqTODEMM4VhmCSGYfIZhrnOMEwXMXXHMQwTwzDMu8+vc+Lq10bV6Up89Qro\n0QP499/K69blsTlVaVs1NTUMGDAAhoaGMoxIeTRp0kTeIVAURcmUQiRODMMMBbAawAIAHQHcAXCW\nYRgjEbs4AQgB4AzADsBLAJEMwxjLPtqaMW3atCrvExMDXLkCNGokg4CUSPm2rStPldaUb7/9Vt4h\nUBRFyZRCJE4AZgIIJoTsJoQ8AjARQB6AMcIqE0JGEEK2EELuEkIeAxiH0mvpWWMRK6CYGMDKCmjY\nUPj29+/fIzIyks499NmLFy+wdetWvH37Vt6hUBRFUbWE3BMnhmHUANgC+LusjJR2A5wHYC/hYXQA\nqAF4V1lFZRYTAzg6it5+4sQJ9O3bF+np6TUXlAIqKChAeHg4duzYgbS0NISHh9OeJ0rhJScng2VZ\n7N69u0r7OTs7w8XFRUZRKZdFixbBxsZG3mHUSbdv34a6ujqePXsm71AqJffECYARABUAFdeveANA\n0gETywGkoDTZUgqPHj2q8j5r1gATJ4rePmLECDx48KDOjkMhhODff//FggULEBcXxy/n8Xh03iEp\nSUpKkncIMrNr1y6wLMt/aWlpoVmzZvDw8MCGDRuQk5Mj8xgYhqnWPixbs2/1ZmZmnLYS9lJRUaly\nEihLWVlZWLt2LX799Vd5h1JjoqOj0a1bN+jo6KBp06b48ccfUVBQUOl+wcHBYn+3oaGh/LrGxsYi\n63Xo0IFfz8bGBq6urliwYIFMrlWaav10BAzD/AzAF4ATIeSTvOORljlz5uDEiRNV2sfDo0IBKQFS\nzwGNXQAVdTAMA0tLS+kFWYtkZ2fj1KlTePz4McLCwjB8+HCoq6vD1dUVXbp0qfEPFmUVFBQk7xBk\nimEYLF68GKampigqKkJaWhqioqIwY8YMrFmzBidOnEC7du1kcu6WLVsiPz8fampqVdrv3LlzMolH\nnKCgIE4iGRERgQMHDmDdunWcBy26detW47GJEhwcDDU1NQwZMkTeodSIGzduwN3dHR07dsS6devw\n/PlzrF69Gs+fP8fRo0fF7turVy/s3btXoHz58uVISEiAs7Mzv2zTpk3Iz8/n1Hv69CkWLlwId3d3\nTvnEiRPh4+ODFStWoGnTptW/OFkjhMj1hdJbbEUAvCuU7wQQWsm+/0Pp7bmOEpynEwDSuHFj4uXl\nxXnZ2dmR0NBQUt7Zs2eJl5cXqWjy5Mlk27ZtnLK4uDji5eVF3r59yymfP38+WbZsGacsOTmZeHl5\nkYcPH3LK169fT/73v/9x6uXm5hIvLy9y6dIlTt2QkBAyevRogdh8fX2515F+hZz9CcTLrZvcroMQ\n8uXXQb7s95Gbm0sCAwOJk5MT6dWrF5kxYwYJCQkh79+/r1XXUUaRfx99+/YlAEhcXJzAttpu586d\nhGVZodd28eJFoq2tTczMzEhBQYEcolNsq1atIizLkuTkZInqFxUVkaKiIhlHxdWmTRvy/fff1+g5\n5cnFxYWYmpqS/Px8ftkff/xBWJYVeG+QxMePH4m2tjYZMGBApXXnzp1LWJYlt27d4pQXFBQQPT09\nEhgYWOXzVyYuLo7/3hQSEkK8vLyIqakp6dChA/Hy8iKOjo4EAAHQiVSWT1RWoSZeAK4DCCr3M4PS\nJ+Vmi9lnDoAsAF0kPEcnZX1DF+nmdEKOGZO83I/k1atX8o5Grk6dOkUCAgLIqlWryL///ktKSkrk\nHZJSKv/mpGzEJU6EEBIYGEhYlhVIgB89ekQGDx5MDAwMiKamJuncuTM5ceKEwP7v378nM2bMIKam\npkRDQ4OYmJiQkSNHkszMTEIIIc+fPycMw5Bdu3bx90lLSyOjR48mJiYmRENDgxgbG5P+/ftzEhQn\nJyfi4uLCOVd6ejoZM2YMady4MdHU1CQdOnTgHLf8+VavXk22bt1KLCwsiIaGBunSpQu5efNmldpO\nXOL06NEjwjAM+eOPP8jKlSuJmZkZUVVVJQkJCYQQQvLz88mvv/5KzM3NiYaGBmnZsiWZO3cu+fTp\nk8Cxtm/fTjp27Ei0tLSIoaEh8fPzI6mpqZXG9/DhQ8IwDDl06JDAtqVLlxJ7e3tiYGBAtLS0SNeu\nXcnx48eFXsPBgwc55QUFBYRhGLJ8+XJO+YsXL8ioUaNIkyZNiKamJrGwsCBTp06tsfeljIwMoqKi\nQgICAjjl+fn5RFNTk0ybNq3Kx9y1axdhWZYcPny40roWFhakbdu2Qrf17duX2NnZVfn8lansvals\nuySJk6LcqlsDYCfDMHEAbqD0KTttlPY6gWGY3QBeEUJ+/fzzTwAWAhgG4AXDMI0/HyeHEJJbw7Er\nppJi4OVhoLkPjhwNhb+/P1JSUtC4cePK91VCrq6uUFVVhYODQ52ew4qSnREjRuDXX39FZGQkxo4d\nCwD4999/0aNHD5iYmOCXX36Bjo4ODh06hAEDBuDYsWPo378/ACA3Nxc9evRAQkICxo4di44dOyIj\nIwMnTpzAq1evRC4kPWjQIDx8+BDTp09Hy5YtkZ6ejnPnzuHFixf8+coqjosqKCiAk5MTEhMTMW3a\nNJiamuLw4cMYPXo0srOzBaZC2bdvH3JycjBx4kQwDIPly5dj8ODBSExMlOqEsJs3b0ZxcTEmT54M\nVVVV6Ovro6SkBH369EF8fDwmTpyIVq1a4datW1i+fDkSExMREhLC3/+3335DYGAgvvvuO0ycOBFp\naWkICgrCjRs3cOvWLbFrVF69ehUMwwid7T4oKAhDhw7FyJEjUVhYiL1792LQoEGIjIyEq6trla/z\n5cuX6NKlC/Lz8zFhwgRYWlrixYsXOHToEIqKiqCuri5y36ysLImeitbR0RH7Pnfnzh0QQsoWteXT\n1NREu3btcOvWLckv6LN9+/ZBV1cXXl5eYutdu3YNiYmJWLJkidDttra2WLFiBQoLC6GhoVHlOGpE\nZZlVTb0ATAbwHEA+gGsAOpfbdgHAX+V+TgJQLOQ1X8zx61aPU1oUIftASPoVkp2dTcLCwuQdEVUH\n1OUeJ0IIqV+/PrG1teX/3LNnT2JjYyNw26l79+6kdevW/J/nz59PWJYV6Mkor2KP0/v37/k9QuI4\nOztzepzWrVtHWJYl+/fv55fxeDzSrVs3oqenR3Jycjjna9iwIcnOzubXPXHiBGFZlkRERIg9b3mS\n9DgZGRlxzkMIIX/++SdRU1MjsbGxnPKgoCDOrZ6EhASioqJC1q1bx6l369YtoqKiQtauXSs2vtmz\nZxOWZUlxcbHAtoq3Xj99+kRat25NPD09Ba5Bkh4nX19foq6uTu7fvy82JmGaNGlCGIYR+2JZVqCH\nq6K9e/cSlmUF2pUQQry9vYm5uXmV4nrz5g1RVVUl/v7+ldadMmUKYVmWJCUlCd2+Y8cOwrIsuXfv\nXpViqIw0e5wUZkQsIWQTIcSUEKJFCLEnhMSW2+ZKCBlT7mczQoiKkNci+UQvfcuXL5e4blGRkMIX\nBwHt5oCRHfT09PjfbJVVcXGxxHWr0rZU1ezcubNK9VNTU3Hv3j2B8tu3b+PNG+6DthkZGYiPjxeo\n++DBA7x69YpT9uHDB8THx+PTp5p9XqRevXr4+PEjgNLegYsXL8LHxwfZ2dnIzMzkv9zc3PDkyROk\nppauH3ns2DF06NAB3t7eEp9LS0sL6urqiIqKwvv37yXe7/Tp02jSpAlnslIVFRVMnz4dOTk5iI6O\n5tT/9ttvOYuBOzg4gBCCxMREic8piYrnAYAjR46gQ4cOMDU15bSfq6srCCG4ePEiAODo0aNgWRaD\nBg3i1DMxMYGpqSm/niiZmZmoV6+e0IdEyvd6vH//HtnZ2ejevbvQv8XK8Hg8hIeHY8iQIbC2tq7y\n/keOHMH58+fFvs6dO4dhw4aJPU7ZYG1hPTqampoCg7krc+DAAZSUlOC7774TW6+4uBiHDx9Gt27d\nYGpqKrROgwYNAJT+f1dUinKrjqqgKo/Hu7gA9vbAypWfC0p4wIsjgNlIgFGY3FgmCCG4c+cOLly4\ngNGjR4u8pVEenXpAdiR5lLm84OBgbNu2TSDxcXR0REBAAGbNmsUvCwsLw/jx48t6kPl8fHzg7u6O\nNWvW8MuuXbsGDw8PvHz5EiYmJtW4kurJycnh3w5/+vQpCCH47bffMG/ePIG6DMMgPT0dxsbGePbs\nWZWf5lJXV8fy5cvxv//9D40bN4adnR08PT0xcuRIsbfkk5OT0apVK4FyKysrEEKQnJzMKW/evDnn\n5/r16wMoTQylSdgH6ZMnT/D8+XM0FDKrb1n7AaVtzePx0LJlS6H1hO1fUcW/qzKhoaEIDAzEvXv3\nUFhYyC8Xd+tPlNevXyM/P79aSRMAdO/evVr7VaSlpQUAnOspU1BQwN8uqX379qFx48aV3rqMjIzE\n27dvsXDhQpF1yn4P1Zl6o6bQxElBifvDqmjyZIAzNVN6FFD4FsUmQ0B4PKiqKuevOTMzE+Hh4Xj+\n/DkAIDw8HCNGjKj0P1xV2paqmokTJ+LPP/+UuP6ECRMwePBggfKYmBgYG3NXUBowYIDQMSiHDx8W\n6Kmwt7dHXFwcGtXg+kMpKSnIzs7GV199BQD8sSj/+9//BB67LlNWt7p++OEHeHt7IywsDGfPnsX8\n+fMRGBiIixcvcubI+RKixjGJSjSqS9iHdUlJCWxtbbF8+XKh5ytLlEpKSqCuro7Tp08LrVfx76Mi\nQ0ND5Obmori4mHO9586dw+DBg9G7d28EBwejSZMmUFVVxZYtWxAeHs6vJ+o9pyo94ZLIyMiQ6Ji6\nurpiEztjY2MQQvg9nuWlpqZWaSqAZ8+e4ebNm5g5c2al77379u2DmpoafHx8RNYpS8iNjEStuCZ/\nyvmJWscMH16hoLgAMO6D8KtpmDylJW7fvi3RN67aori4GFeuXEFMTAznTURHRwc8Hq/K89xQKChP\n/wAAIABJREFU8mNsbCyQIAEQOnuzkZGR0DfTtm3bCpTp6ekJTbJkaffu3WAYBh6fJ1QzNzcHULpQ\ndGXfxC0sLHD//v1qndfMzAwzZ87EzJkz8ezZM3To0AGrV68WOblky5Ythd4effjwIX+7orCwsEBy\ncnKlM59bWFigqKgIrVq1qlYPY5s2bQCUTuBaPpk9duwY9PX1cfr0ac5tvI0bN3L2L7u9VPGWacXe\nu6ZNm0JLS6vav+t27doJ3MKuiGEYBAYGYs6cOSLrtG/fHgzDIDY2Fp6envzygoIC3Lt3D+PGjZM4\npr1794JhGAwX+CDiys/Px/Hjx+Hu7i520fSkpCSoqal98ZcKWVLu+zh1VTNPwOUUWrdpgylTpihV\n0vTy5UsEBwfj4sWL/KSpfv36+O677zB48GCaNFFyceHCBfz+++8wNzfnf4A0bNgQzs7OCA4ORlpa\nmsA+5cdwDB48GHfu3MHx48clPmd+fr7ArRYzMzPo6uoKvQVTpm/fvkhLS8PBgwf5ZcXFxdiwYQN0\ndXXh5OQkcQyy5uvri8TEROzZs0dgW15eHn8sTtltTmG9yYSQSm8r2tvbgxCC2NhYTrmKigpYluV8\nQXvy5AlOnTrFqdewYUPo6uoiJiaGU75x40ZOL4yqqiq8vLxw9OjRaiVP0hrjZGRkBCcnJ+zatYtz\ne3379u349OkTfH19+WV5eXlISEgQ2Yb79+9Hq1atBJ7QqygsLAx5eXmVjoOKi4uDjY2N4j5RB9rj\npLAyMjK+uKuyTZs2SrV8QFZWFnbs2MG5B25nZwdnZ2exj/BWJI22pYST9rgXRUMIwalTp/Dw4UPw\neDy8efMGFy5cwLlz52BmZoYTJ05w/hY3btwIBwcHtGvXDuPHj4e5uTnevHmDa9euISUlhf/Y9+zZ\ns3HkyBH4+PjA398ftra2yMzMxMmTJxEcHCx0NvLHjx+jZ8+e8PX1Rdu2baGqqopjx44hPT1d7Afn\n999/j+DgYIwePRqxsbH86QiuXbuGoKAg6OjoSL/hqmns2LE4fPgw/P39ERkZCXt7exQVFeHBgwc4\nfPgwLl++jLZt26JNmzaYP38+Fi1ahCdPnsDLyws6Ojp49uwZQkNDMWvWLEyePFnkeaysrNCqVSuc\nP3+eM2je09MTmzZtgoeHB4YOHYrXr19j06ZNaNOmDRISEgRiDQoKgq6uLmxsbHDhwgUkJSUJ3Dpc\nvnw5oqKi0K1bN0yYMAGtW7fGq1evcOjQIf56baJIa4wTAAQGBsLJyQlOTk4YO3YskpKSsHbtWnh7\ne8PBwYFf79KlS+jTpw+WLVsm0IsVGxuLx48fY9Giyp/L2rdvH3R0dMQ+AFFYWIjLly/jl19+qf6F\n1YTKHrtTlhdq2XQEwmZlpkofhQ4ICCDBwcHk9evX1ToGbVvZcXBwUPrpCMpempqapGnTpsTd3Z38\n8ccf/Mf4K0pKSiKjR48mTZs2JRoaGqR58+bE29tbYDb2rKwsMn36dNK8eXOiqalJWrRoQcaMGUPe\nvXtHCCmdHoBlWf50BJmZmWTatGmkbdu2RFdXlzRo0IDY29uTo0ePco7r7OxMXF1dOWVv374lY8eO\nJY0aNeJPgLl7925OnbLzrVmzRuCaWJYlixYtkrjtKpuOgGVZsnHjRqH7FhUVkcDAQGJtbU00NTWJ\nkZER+eabb0hgYCDJzc3l1D106BDp0aMH0dXVJXp6esTa2prMnDmTJCYmVhpjYGAgMTQ0JDwej1O+\ndetW0qpVK6KlpUW+/vprEhISQn7++WeipaXFqZebm0v8/f2Jvr4+qV+/PhkxYgRJTU0lLMuSFStW\ncOo+f/6cjBgxgjRq1IhoaWmRVq1akZkzZ9b4xLxRUVHE3t6eaGtrE2NjYzJr1izOTOKEEHLmzBmh\n10AIITNnziQsy5KnT5+KPU9mZibR0NAgI0aMEFsvNDSUqKqqymTCZmlOR8AQKQ/wU1QMw3QCEBcX\nF1fjYx+qIz4+vlbEWdMKCgpw9+5ddO7cudrry9G2lZ19+/bBz88PteX/GUWVeffuHSwsLLBp06ZK\nb3VRsuHh4QEjIyOh6+B9qfj4eNja2op8byrbDsCWECJ2rgk6xklBfcmHTnR0NNzd3fHu3TspRqQY\nNDU10bVr1y9alJd+oMuOlZWVvEOgqGoxMDDAzJkz6TxvcnLnzh1cvHixVjz1TBOnWmzoUEDUECZD\nQ0P+kx61SVXnAaIoipKW+fPn4/bt2/IOo07q0KEDCgsLYWFhIe9QKkUTp1qquBg4cwYQNo7TyckJ\nISEhCj2BWEVFRUX4+++/ERQUVKVZkCmKoiiqJtHESUFt375d7Pa7d4EPHwBHx88F6ZeAZ9uBWjhm\nLTExEZs3b8bly5dRUFCAiIgIqU+uV15lbUtVX1hYmLxDoCiKkimaOCmoytZBiokBNDSALl0+FySs\nBx5vAmpRL1NeXh7CwsKwZ88e/mPsKioqaNasmUwTp+qsMUVJ5tGjR/IOgaIoSqZo4qSgKs5MW5Gf\nH3D2LKCpCaAoB3gdgdfqDpg9ezZ/kVFFRT6vL/fHH3/gzp07/PIWLVpgwoQJcHZ2/qLB35WprG2p\n6vv555/lHQJFUZRM0cSpljI0BPgT/KacBIrzcfudJSIjI6u8QGNNS01NRVhYGGeFbk9PT4wePVqp\nZjmnKIqilA9NnJTBi4OA4Tfo6zMZt2/fVvhFfZs2bcpfgNTa2hpTp06Fra1trRrMTlEURdVNiv0J\nS1XuUzbw+jRgswyA6FW6FY2bmxvatm0LS0tLeYdCURRFURKjPU4KStx6PhyvjgMln4AWPrINSMq0\ntbXlljRJ3LZUlc2YMUPeIVAURckUTZwU1NSpUyWr+OIgCnQ7Y++xKHz69Em2QVVBdna2vEMQSeK2\npaps6NCh8g6BoihKpmjipKDc3NyEluflAbm55QqaD0HUmy6YPn06iouLayY4MT5+/IhDhw5h8+bN\n+PDhg7zDEUpU21Jfzt7eXt4hUAAOHToEQ0ND5OXlyTuUOofH46FFixbYsmWLvEOhZIQmTrXM4cOA\ngQHAn3HAwh8e4zbh2bNncn2ajhCCmzdvYuPGjXj48CEKCwtx5swZucVDUdK2a9cusCwrMA/Yhw8f\n0LVrV2hrayMyMhIAsHDhQrAsC2NjY6HLCJmamgrcMmZZFizLYu3atRKfW5iSkhIEBATghx9+gLa2\ndlUusdZ69OgRPDw8oKurC0NDQ4wcORIZGRkS7VtYWIjAwEBYW1tDR0cHJiYm8PX1xYMHDzj1Lly4\ngLFjx6J169bQ0dGBhYUFxo8fj7S0NE49VVVVzJo1C7///rtC3QWgpIcmTrWMiwuwcyegq8stl+e6\ndOnp6dixYwdOnTqFwsJCAKVjmKysrGQ6kSVF1bSKD198/PgRvXv3xv379xEWFibQm5meno7NmzdX\nepzy5StXrhSabEn64MeJEyfw+PFjjB8/XqL6tV1KSgocHByQmJiIZcuWYfbs2YiIiICbmxt4PF6l\n+w8fPhwBAQFwdXXFhg0bMHHiRMTExKBbt254+fIlv95PP/2E6OhoDBo0CBs2bMCwYcNw6NAhdOrU\nCenp6Zxj+vv7IyMjAyEhIVK/Xkr+6FN1CiosLAwDBgwQKG/RovSlCHg8HmJiYnDlyhWUlJTwy21s\nbODm5qaw80mJalvqy128eFHeIdSYnJwcuLm54e7duwgNDRV6C9jGxgYrV67E5MmToaGhUekxbWxs\ncPv2bWzZsqXaA+137tyJ7t27w9jYuFr71zZLlixBfn4+bt++jWbNmgEAunTpgt69e2Pnzp0YN26c\nyH1fv36N0NBQzJkzB8uWLeOX9+jRA66urjh27Bh++OEHAMDatWvRo0cPzv7u7u5wcnLCH3/8gUWL\nFvHL9fX14ebmhp07d2L06NFSvFpKEdAeJwW1f//+Suu8ffsW9+7dq4FohHvx4gUuXbrET5oMDAww\ncuRI9O/fX2GTJkCytqWq5+zZs/IOoUbk5ubC3d0dt2/fxrFjx+Dh4SFQh2EYzJ8/H2lpaUJ7nYTp\n3r07XF1dsWLFCn7vbVWU3SLv1auXwLYdO3agZ8+eaNy4MTQ1NWFtbS10HA7LspwkoIypqSnGjBnD\nKcvOzsbMmTNhZmYGTU1NNG/eHKNGjcK7d++qHHt1HTt2DJ6envykCQB69uwJS0tLHDp0SOy+Zass\nNGrUiFPepEkTAOC8j1VMmgDAwcEBBgYGePjwocC23r174/Lly3TRciVEEycFdfDgwUrr7Nq1C998\n843cllgxNzfH119/DZZl4ejoiEmTJsHMzEwusVSFJG1LVU/5b+3KKicnBx4eHoiLi8ORI0fQp08f\nkXUdHByqnAgFBARUKdkqLy4uDp8+fUKnTp0Etm3ZsgWmpqaYO3cu1qxZgxYtWmDy5MkSn6fircLc\n3Fz06NEDGzduhIeHB9avX49JkyYhISEBr169EnusDx8+IDMzs9JXLudJGEGvX79Geno6OnfuLLCt\na9euuHXrltj9LSwsYGJigtWrVyM8PBwpKSm4ceMGJk2aBAsLC3z77bdi98/NzUVOTg6MjIwEttna\n2qKkpARXr14Vewyq9qG36mqxH374Ac7OztCtOOCpBrm7u8PBwUHgGxtFKSNCCEaNGoXU1FQcOXIE\n/fr1q3SfBQsWwMnJCVu2bOHf9hGne/fucHFxwcqVKzFp0iSJbvGVefToERiGEfoFJiYmhnOsyZMn\no0+fPlizZg0mTZok8TnKrFixAg8ePEBoaChnoPuvv/5a6b79+/dHdHS02DoMw2DUqFH466+/RNZJ\nTU0FAKG3JY2NjfHu3TsUFRVBTU1N6P6qqqo4duwYhg0bxrmGzp0748qVK9DT0xMb49q1a1FUVCQ0\nwTI3NwcAPHjwAH379hV7HKp2oYlTLaampib0m1ZNqlevHurVqyfXGKhaLD+19CWKiiag31b8MbIf\nAMWCg6kBAFrGpS8pSk9Ph6amJkxMTCSq7+DgABcXF6xYsQITJ06UKBEKCAioUrJVJjMzE4Dwh0XK\nn/fDhw8oKiqCo6MjIiMj8fHjxyp/ATt27Bg6dOhQrQll16xZg6ysrErrNW3aVOz28utdVqSpqcmv\nIypxAoD69evDxsYGQ4cOxTfffIOnT58iMDAQQ4YMwfnz56Guri50v5iYGCxatAhDhw6FE3/h0P+U\n/Q4kfbqPqj1o4lRL8HgAy5a+cH0M0LSPzGcLLykpwdu3b9G4cWOZnoeqw54EA/cXit6u3xbo96/4\nY1z2KU2ehPl6AdA+oNrhVcQwDIKDgzFz5ky4u7vj8uXLaNWqVaX7VTURqphsVZWwp1mvXLmCBQsW\n4Pr165z5nRiGQXZ2dpUTp2fPnmHIkCFVjg0AOnbsWK39KiobgyTsNmjZk4nixlt++PABDg4OmDNn\nDmbOnMkvt7W1hbOzM3bs2IEJEyYI7Pfo0SMMGjQI7du3x59//in02GW/g9qyDBYlOZo4KSh/f3/s\n2LGD//OpU4C/P/AoLgUNE3egQL8bNGV4/tevX+PkyZN4//49pkyZolS9ShXblpKegICAqu3QagJg\nIqbHQkWCv/Ieh8X3OElZ27Ztcfr0abi6uqJ37964cuUKZ2CyMA4ODnB2dsaKFSuEfhALs2DBAjg7\nOyM4OBj6+voS7WNoaAgAyMrK4vTWJCYmolevXrCyssLatWvRvHlzqKurIyIiAuvWreM8FSuKNCfY\nzcrKkmiOIy0tLbG3y8pu0ZXdsisvNTUVBgYGYnubjhw5gvT0dIFeM0dHR+jp6eHKlSsCv6+XL1/C\nzc0NDRo0QEREBHR0dIQeu6xHTdj4J6p2o4mTgqr4aHNMDKCjAxjlHQCPqKJ1z9l48GSYyP+01fXp\n0ydcuHABN27c4H9jOnv2LAYPHizV88gTnTlcduzs7HDy5EnJd5DGrbTKbuXJQOfOnREWFoZ+/fqh\nd+/euHTpEj9pESUgIAAuLi4IDg6W6ByOjo5wdnbG8uXL8dtvv0m0T5s2bUAIQVJSEqytrfnlJ0+e\nxKdPn3Dy5ElOkvf3338LHKNBgwYCT4IVFRUJJCcWFha4f/++RHFVNGjQIKmMcWratCkaNmyI2NhY\ngW03btyAjY2N2HOUzb8kLCksLi4WmAfq3bt3/PmhoqKixPbGJyUlAQCsrKzExkDVPjRxUlDDhg3j\n/BwTAzg6AsyLg/hk1AuBq0ZIPWl6/PgxTp06xVlnrnHjxvjmm2+keh55q9i2lPR4eHhg7ty58g6j\nRri6umL//v3w8fGBh4cHLl68KLZn1tHREU5OTli+fLnEE8MGBATA2dkZW7dulai+ra0t1NXVERsb\nC09PT365iooKAHB6lrKzs7Fz506BY1hYWCAmJoZTFhwcLJBcDB48GIsXL8bx48fRv39/ieIrI60x\nTmVx7N69GykpKfyk8O+//8bjx4/x448/8uvxeDw8e/YM+vr6/OkGLC0tQQjBgQMHMH/+fH7d48eP\nIzc3l/N0Yl5eHvr06YPU1FRERUXxB3+LEhsbC5Zl6TJESogmTrVATg4QHw+M83sLvLsJ7e4HMLyl\n9BZTzcnJwZkzZ/Dvv/+NJVFVVYWTkxPs7e35b7oUVddVTHgGDBiAP//8E2PGjIGXlxfOnDkjdvD3\nggUL4OLiIvH5ypKt6OhoicbKaGhowM3NDefPn+fcNnVzc4Oamho8PT0xYcIEfPz4Edu2bUPjxo0F\nlgwZN24cJk6ciCFDhqB37964c+cOIiMj0bBhQ0692bNn48iRI/Dx8YG/vz9sbW2RmZmJkydPIjg4\nGO3atRMZp7TGOAGlT/EdOXIEzs7O+OGHH/Dx40esWrUKHTp04Ew+mZKSAisrK4wePZrfi+Xl5QVr\na2ssWrQIz58/h52dHZ48eYKNGzeiWbNmnHmrhg8fjps3b2Ls2LH4999/Oe+X9erVE0gez58/j+7d\nu8t1VQdKNmjiVAvUqwckJgK6r/YCL7WBZp6V71QFr1694rwJmJubo1+/fjAwMJDqeSiqthOWvIwe\nPRrv3r3D7Nmz4evri9DQUJH7Ozk5wcnJCTExMQLHYhhG6PHLlgORdJDxmDFjMGTIEE4PjKWlJY4e\nPYp58+Zh9uzZaNKkCSZPngxDQ0OMHTuWs//48ePx/PlzbN++HWfPnoWjoyPOnTuHnj17cmLQ0dHB\n5cuXsWDBAoSGhmL37t1o1KgRevXqJfETh9JgYmKC6OhozJo1C7/88gvU1dXh6emJVatWCYxvqtjG\nampquHz5MhYvXoyIiAgcOHAAurq6GDRoEJYsWcJ5D7xz5w4YhsFff/0lcPuwZcuWnMTpw4cPiIyM\npAv9KimmrqwlxjBMJwBxcXFxQieHUzSXL18WmKmWnLIB9NqA6XFA6uc7ePAgXrx4ATc3N7Rv316p\nnwQR1raUdGzfvh3jxo1Dbfl/poxKSkpgbW0NHx8foTOAU7K3bt06rFq1Cs+ePavSPFyU7MTHx8PW\n1lbke1PZdgC2hBCxq2nTmcMV1IoVK7gFHxLAvL+DH9f/w5+7RJr69euHKVOmoEOHDkqdNAFC2paS\nml27dsk7hDqPZVksXLgQmzZt4kw7QNUMHo+HdevW4bfffqNJk5Kit+oU1IEDFXqVdEzxsNFy1LfK\nksk6cMo03UBlBNqWkprAwEDam6cAfH194evrK+8w6iRVVVU8f/5c3mFQMkQTJwWlra3NLVDRgFWv\nOZgvuHZnpfLz85GRkYHmzZtLJ7haTqBtKalR5MWdKYqipIEmTkqMEIJ///0XZ86cASEEU6ZMoUkD\nRVEURX0BOsZJwT14AEgwwa6A9+/fIyQkBEePHkVubi7y8vJw4cIF6QdIURRFUXUITZwU1OzZs8Hj\nAV27AhMnJmDUqFH8tZfEKSkpwbVr17Bp0yY8ffqUX96mTRs4OjrKMuRaY/bs2fIOQWmtW7dO3iFQ\nFEXJFL1Vp6BatGgBlgWiooD4+GeIjMzlr/YtSmpqKk6ePMlZGkFXVxd9+vSh0/6X06JFC3mHoLTK\nZmSmKIpSVrTHSUFNmzYNLAt07gx8/31fHDlypNJ9MjMzOUlTly5dMHnyZJo0VTBt2jR5h6C0vv32\nW3mHQFEUJVO0x0mREQKQEoCVbMkTa2tr3LlzBx8+fICnpyd9io6iKIqipIwmToos8wYQ0x/odQnQ\na1VpdYZhMHDgQGhoaND15SiKoihKBuitOgX16NEjpP0ThNzcPHzSkHzdJ21tbZo0VeLRo0fyDkFp\nJSUlyTsEiqIomVKYxIlhmCkMwyQxDJPPMMx1hmG6VFLfh2GYh5/r32EYpk9NxVoT5syZjXpZpxF2\nSw1q6qWDwjMyMugHkxTMmTNH3iEoraCgIHmHQFEUJVMKkTgxDDMUwGoACwB0BHAHwFmGYYxE1O8G\nIATAnwBsABwHEMYwTNuaiVj2Fk4fCx3mPYbOPori4mJER0djy5YtOHbsmETTElCi/fHHH/IOQWn9\n9NNP8g6hVjM1NcWYMWOkeszo6GiwLIuYmBipHlfeiouLwbIsli5dKrcYrl69Ck1NTaSkpMgthrqq\nqKgIJiYm2LZtW42fWyESJwAzAQQTQnYTQh4BmAggD4Cod5DpAE4TQtYQQhIIIfMBxAOYWjPhyhYh\nQL9hLvg9fAVefzJFcHAwoqKiUFxcjJycHFy5ckXeIdZqdDoC2TE2NpZ3CDKza9cusCwLlmVx9epV\noXWaN28OlmXh7e1drXOwLCuTRbYrHnP//v1S7R0MDQ0Fy7L466+/RNY5d+4cWJat8heXiIgILF68\nWOg2hmHkuij5vHnzMHLkSDRr1kxuMdSkV69ewcfHBw0aNIC+vj4GDhwo8bp8JSUl2LRpE2xsbKCr\nqwtjY2N4enrin3/+EVo/NjYWnp6eMDQ0RL169dC+fXts3ryZv11NTQ0zZszA4sWLUVRUJI3Lk5jc\nEyeGYdQA2AL4u6yMEEIAnAdgL2I3+8/byzsrpn6t8uxJMbJyNGDYHtixcxcyMjIAlL5JdOvWjU5k\nSVFypKWlhZCQEIHy6OhopKSkVDrfmjgJCQnYunXrl4QnwMnJCfn5+Zz3jZCQEKkmTv369YO+vr7Q\ndil/TlVV1SpPWREeHi40cVJRUUF+fj5+/vnnKscrDbGxsYiKisLEiRPlcv6a9vHjRzg5OeHq1av4\n7bffsHDhQsTGxsLFxQXZ2dmV7j9z5kxMnToVnTp1wtq1azFr1iw8ePAATk5OuHXrFqfu6dOn0b17\nd7x//x4LFizAunXr0K9fP7x8+ZJTb+zYsUhNTcXBgweleq2VUYSn6owAqAB4U6H8DYDWIvZpIqK+\nUsy+d2TvSUyZ8gxvP+Xxy5o2bQovLy86wSBFyVnfvn1x+PBhrF+/Hiz733fPkJAQdO7cmf9FpzrU\n1NSkESIAoLCwEOrq6mAYBurq6lI7rjDq6uoYMmQIdu7cibS0NIH3qcLCQoSFhcHNzQ1GRkJHYIhU\n+j1a9HnlZceOHTA3N0enTp3kFkNN2rBhA5KTkxEfH4/27dsDANzc3NC+fXusXbsWAQEBIvctKirC\n1q1bMXz4cE6v5MCBA2FpaYmQkBB07NgRAJCdnY3Ro0dj4MCBOHDggNiYGjRogF69emHnzp3w8/P7\n8ouUkNx7nChBjx9l4M6dMwBK3xg8PDwwduxYmjRJyfLly+UdgtLauXOnvEOQKYZhMGzYMGRmZuLc\nuXP88qKiIhw5cgTDhw8X+kG/atUqdO/eHUZGRtDW1kbnzp1x9OhRgXrCxjglJSXBx8cHhoaG0NHR\ngb29PU6dOsWpUzaO6eDBg5g3bx5MTEygo6ODjx8/CoxxcnFxQUREBJKTk/m3Hs3NzZGbm4t69eph\n5syZAnGlpKRAVVVV7P8dPz8/FBcXC/2wCw8PR3Z2Nr777jtO+YEDB9CpUydoaWmhUaNGGDVqFNLS\n0vjbR4wYga1bt/LHM7Esy0+WhI1xmjdvHliWxfPnzzFy5EjUr18fDRo0wPjx41FYWMg5d35+PqZO\nnQojIyPo6elh0KBBePnypcTjpo4fP46ePXsKlIeFhaFfv35o1qwZNDU10apVKyxdulTg78LExATf\nf/+9wP49evSAm5sbp6ygoADz58+HpaUlNDU10bRpU/j4+CA5ObnSOKXl6NGjsLOz4ydNANC2bVs4\nOzvj0KFDYvctLCxEYWEhGjVqxClv3LgxGIbhLD6/Z88eZGRk8H8Hubm5YpPn3r17Izo6Gh8/fqzO\nZVWLIiROGQCKATSuUN4YQJpgdeBzeVXq8/Xt2xfe3t6cl729PcLCwjj1IiMjhY5TmDJlCrZv384p\ni4+Ph7e3t8A3zQULFgi80bx48QLe3t4Cj8Rv2LCBv4banyH+UFNVQ8uWLRETE4OioiLON9v9+/fD\n399fILahQ4cq1HWUycvLg7e3Ny5fvswpl9d15OX915NXm6+jPEW5jvPnK95BVz6mpqaws7PD/v37\n+WWnTp3Chw8fRN6GWr9+PTp16oTFixcjMDAQampq8PX1xenTpzn1Ko7XSU9Ph729Pc6dO4epU6di\n6dKlKCwshLe3N44fPy5wnsWLF+P06dOYPXs2li5dyk8yyh933rx5sLGxgZGREfbt24e9e/di3bp1\n0NHRwcCBA3Hw4EGBD6qyW3DivtU7OjrCxMRE6O26kJAQ6OjooH///vyybdu2Yfjw4dDU1MSKFSsw\nbtw4HD58GA4ODsjJyQFQ+v/C1dUVLMvyY929e7fIGMrGPA0ePBiFhYVYvnw5hgwZgr/++gu///47\np66fnx82b96M/v37Y8WKFVBVVYW3t7dEY6ZevHiB169fC+1t2rFjB/T19fHjjz8iKCgIHTt2xLx5\n8zBv3jyBWEVdQ3nFxcXo06cPlixZAjs7O6xbtw4zZsxAVlYWHjx4IDbO3NxcZGZmVvqqLOkoLi7G\n/fv30blzZ4FtXbt2xePHj8U+tFSvXj3Y2tpi+/btOHDgAF69eoU7d+5g1KhRaNiwIca8BwASAAAb\nVUlEQVSOHcuv+/fff8PAwADPnj2DpaUldHV1oa+vj2nTpuGTkBXvbW1t+Wu0Smr//v3w9vaGmZkZ\nbGxs4O3tLfQLg0iEELm/AFwHEFTuZwbASwCzRdQ/AOB4hbIrADaJOUcnACQuLo7UBoWFhaSkpETe\nYVBUlcTFxZHa9P+sKnbu3ElYliVxcXFk48aNRF9fnxQUFBBCCPH19SU9e/YkhBBiampKvLy8OPuW\n1SvD4/FIu3btSK9evTjlpqamxN/fn//zjBkzCMuy5OrVq/yynJwcYm5uTszNzfllUVFRhGEY8tVX\nX5HCwkLOMaOiogjLsiQ6Oppf5unpSczMzASuMTIykrAsS86ePcsp79ChA3FxcRHdOJ/NmTOHsCxL\nnjx5wi/78OED0dLSIn5+fvyywsJCYmRkRDp16kQ+ffrELz9+/DhhGIb8/vvv/LKJEycSNTU1gXPx\neDzCMAxZsmQJv2zevHmEYRgyadIkTl1vb29ibGzM//nGjRuEYRjy008/ceqNGDGCsCzLOaYwZ8+e\nFdpOhAj+rgkhZNy4cURPT4/weDx+mYmJCRk/frxA3R49epDevXvzf966dSthGIZs3LhRbEzC+Pn5\nEYZhKn2VP58waWlphGEYsmzZMoFt69evJyzLksTERLHHePLkCenYsSPnvJaWluTp06ecetbW1qRe\nvXpER0eHzJo1i4SGhpJp06YRhmHIyJEjBY778uVLwjAMWbt2rdjzV/beVLYdQCdSSc6iCD1OALAG\nwHiGYUYyDNMGwBYA2gB2AgDDMLsZhinfdxoEwINhmFkMw7RmGCYApQPMleY587KxCRSlzFJTgfj4\n/16JiaLrZmSU1hGluJh7rPh4QMgXVKnw9fVFXl4ewsPDkZOTg/DwcIHbUOVpaGjw//3+/XtkZWXB\nwcEB8eIuCKWDZLt27Qp7+/+ee9HR0cH333+P58+fC/Q4jB49+ovG/fTq1QvGxsbYt28fv+z+/fu4\ne/cuRowYUen+fn5+IIRwep2OHDmCwsJCTvvcuHEDmZmZmDJlCmdcl7e3N7766itERERU+xoYhsGE\nCRM4ZQ4ODnjz5g2/V+TMmTNgGAaTJk3i1Js2bZrY20JlMjMzAZSOsamo/O86JycHmZmZ6NGjB3Jy\ncvD48eMqX8+xY8fQpEkTgVgl8euvv+L8+fOVvlauXCn2OPn5+QC411am7GGIsjqi6OrqwtraGtOn\nT0dYWBg2bdqEwsJC9O/fH+/fv+fXy8nJQV5eHsaNG4fVq1djwIABWL9+PcaOHYt9+/YJ3J4s+x18\nydjCqlKEweEghBz6PGfTIpTecrsNwJ0Q8vZzFRMAvHL1rzEMMxzAks+vJwD6E0LE91tSFKVQgoOB\nhQv/+3nIEODwYeF1w8KA8eNLp+sQJjcXsLXllr18CZhIPvG+xIyMjNCrVy+EhIQgNzcXJSUlGDJk\niMj64eHhWLJkCW7fvs0Za1P+FrwwycnJsLOzEygvW7g7OTkZbdv+N32dqalpFa+Ei2EYfPfdd9iy\nZQsKCgqgqamJffv2QUtLS+z1lWnXrh2+/vpr7N+/H/PnzwdQepvOyMiIM24nOTkZDMPA0tJS4Bht\n2rRBXFzcF11HxSlHyj5cs7KyYGxsjOTkZKiqqqJly5acel999VWVziMsybp//z7mzp2LqKgozi0w\nhmEkevqsomfPnqFNmzbV+iJtZWUllUXetbS0AEBgnBgAfjJaVkcYHo8HV1dXeHh4YPXq1fxyFxcX\ntGvXDqtXr+Y/OVl2nIq3vYcPH47t27fj2rVrnN9b2e+gJjsaFCJxAgBCyCYAm0RscxVSdhSA4OhK\nJZGRkVHlp08oydC2lZ2srKwq1Z8wASg/VKp+fdF1BwwAxD3ApKMDVPy8rTAWVaqGDx+O8ePHIzU1\nFX369IGurq7QepcuXUL//v3h7OyMzZs3w9jYGGpqavjrr78446SkQdyHl6RGjhyJlStXIiwsDN9+\n+y32798PLy8vkddXkZ+fH3755RfEx8ejWbNmiIqKwqRJkypNEqVJ1LJTkvQmScLQ0BCEEIG/96ys\nLDg6OsLQ0BCBgYEwNTWFpqYmbty4gblz56KkpIRfV9QHfXFxsVRiBIAPHz5U2hMElPYk1Rfzn8/I\nyAhqampITU0V2JaamgqGYcTO4Xbx4kU8evQImzZxP+Jbt24NS0tLztyETZs2xePHj9G4MXcYc9nA\ncmFtXhZjTVGYxIniGjNmDE6cOCHvMJQSbVvZWVi++0gCxsalL0kYGZW+RFFREZ9YSdvAgQMxYcIE\n/PPPP2LnkTl27Bi0tLRw9uxZqKr+95Zb8WEAYVq2bImEhASB8ocPH/K3V4e4b+fW1tbo2LEj9u3b\nh2bNmuHFixfYuHGjxMceNmwYfvnlF4SEhKBFixYoKSkRuI3ZsmVLEEKQkJCAHj16cLYlJCRwrksW\nPQktW7YEj8dDcnIy51xPnjyRaP82bdoAEFyb8cKFC8jOzsbp06fxzTff8MuF/Q4bNGjAuUVVJjk5\nGdbW1vyfLSwscOfOHZSUlFQ5+ZwyZQrntqsovXr1QmRkpMjtKioqsLa2RmxsrMC2f/75B61atRI7\nf9mbN6WzBwlLCouKisDj8W8owdbWFlFRUUhJSYGZmRm//PXr1wCAhg0bcvYv+x1Io2dNUooyxomq\nQNycGNSXoW0rOxXHligzHR0dbNmyBQEBAfDy8hJZT0VFBQzDcD4cnj9/LvSpuIr69u2LGzducGZX\nzs3NxdatW2FmZsa5TVfV2MXdNhoxYgTOnj2LdevWwcjICB4eHhIfu3nz5nBwcMCBAwewd+9emJmZ\ncZIIoPRJLENDQ2zevJnTLidPnsSTJ0/g6enJibW4uJjzNOyXcnd3ByFEoAdkw4YNEiVqLVq0gLGx\nsUAiUdbTVb5nqbCwkDPjdRkLCwtcu3aNk0yEhYUJ9OoMHjwYaWlpQo9RGWmNcQKAIUOG4Pr167hz\n5w6/7MGDB4iOjoavry+nbkJCAl69esX/2dLSEoQQgakqbt68iadPn3KeTvT19QUhROCLxbZt26Ch\noQEnJydOeWxsLFiWFXpLW1Zoj5OCqiuTqskDbVvZqclvffJQ8VaPJAOm+/XrhzVr1sDd3R3Dhw/H\nmzdvsGnTJrRq1Qp3794Vu+/PP/+M/fv3w8PDA9OnT4eBgQF27tyJ5ORkHDt2rNpx29ra4tChQ/jx\nxx/RpUsX1KtXj5OsDB8+HHPmzEFYWBgmT54s8taXKH5+fvj++++Rmpoq8Bg+UPrwy7Jly/D999/D\n0dERw4YNw+vXr7F+/Xp89dVXmD59OidWAJg6dSp69eoFNTU1+Pj4VCmeirp27Yr+/ftj1apVePv2\nLbp06YKLFy/i2bNnACTr5erfv7/AdBI9evSAnp4e/Pz8MG3aNJSUlGDPnj2cnsYy48aNQ1hYGDw8\nPDB48GA8ffoUISEhMDc359Tz9/fHnj17MH36dFy7dg3du3dHTk4Ozp07h5kzZ6JPH9Hr20trjBNQ\n2v7btm1Dnz598OOPP4JlWaxZswYmJiaYMWMGv15xcTGsrKw4vVhdu3aFi4sLtm/fjqysLPTq1Quv\nXr3Cxo0boaenx/l9d+7cGSNHjsTu3btRWFgIR0dHnD9/HqGhoZg/f75Aj9P58+fh5OQEPT09qVyn\nRCp77E5ZXqhl0xFQVG1UV6YjEMfMzIx4e3tzynbs2EFat25NtLS0SNu2bcmuXbtIQEAAYVmWU6/i\ndASEEJKUlER8fX2JgYEB0dbWJnZ2duT06dOcOmVTDhw9elQgHmHTEeTm5hI/Pz9iYGBAWJYVOjVB\nv379CMuy5Pr162KvV5isrCyiqalJVFRUyKNHj0TWO3DgAOnUqRPR0tIiDRs2JKNGjSKpqamcOsXF\nxWTq1KmkUaNGREVFhT81AY/HIyzLkqVLl/Lrzps3j7AsS7KzsznH2LZtG2FZlqSkpPDL8vLyyJQp\nU4ihoSHR09MjPj4+JCEhgTAMQ9asWVPpNd68eZOwLEv++ecfTvmVK1eInZ0d0dHRISYmJmTevHnk\nzJkzhGVZcuXKFU7dVatWERMTE6KtrU2cnJzI7du3iYODA3Fzc+PUy8/PJ3PnziXm5uZEQ0ODNGvW\njHz77bckOTm50jil6eXLl2TIkCGkfv36RF9fnwwcOJAkJSVx6pT9XoRdw6JFi8jXX39NdHR0iIGB\nARkwYAC5d++ewHmKiopIQEAAMTU1JRoaGqR169ZCp2N49+4dUVdXJ3v27Kk0dmlORyD3hKamXjRx\noijZU+bEqSY0b95c6Nw+8jBw4EDSqlUreYdRo27evEkYhiGHDh2SqL6Tk5NAokvVnJUrV5IWLVpw\n5gITRRnncaIqkGTgKFU9tG1lp+IM45TkeDweMjMzFeKJz9TUVERERGDkyJHyDkVmhM10HRQUBBUV\nFTg4OEh0jKVLlyIkJAQpKSnSDo+qRFFREYKCgjB//nyprvEoCTrGSUHFx8dzpqGnpIe2rexUXPKF\nkkxkZCT279+PgoICoeuf1ZTnz5/j8uXL2LZtG9TV1YWupaYsAgMDcffuXTg7O4NlWURERODcuXOY\nMmWKxOuCduvWTexSI5TsqKmp4eXLl3I5N+1xUlBVefyXqhratrLz888/yzuEWmnZsmW4cOECli5d\nKtfEKTo6GiNHjsSLFy+we/dugUVZlUm3bt2QkZGBRYsWYfbs2UhMTMTixYsRFBQk79AoBUd7nCiK\nouTswoUL8g4BADBq1CiMGjVK3mHUCHd3d7i7u8s7DKoWoj1OFEVRFEVREqKJE0VRFEVRlIRo4qSg\nvMsv4EVJFW1b2Sk/ER5FUZQyoomTgpo6daq8Q1BatG1lZ+jQofIOgaIoSqbo4HAF5ebmJu8QlBZt\nW9mxt7cH8N8itBRFUYpAmu9JNHGiKEpqjIyMoK2tDT8/P3mHQlEUxaGtrS2VCWZp4kRRlNS0aNEC\nDx8+REZGhrxDoSiK4jAyMkKLFi2++Dg0cVJQYWFhGDBggLzDUEq0bWWnrG2l8eZEcdG/W9mhbSs7\nyti2dHC4glq+fLm8Q1BatG1lh7at7NC2lR3atrKjjG1LEycF1bBhQ3mHoLRo28oObVvZoW0rO7Rt\nZUcZ25YmThRFURRFURKiiRNFURRFUZSEaOJEURRFURQloTrxVB3DMNoA2gC1Z2K+GzduID4+Xt5h\nKCXatrJD21Z2aNvKDm1b2aktbVsuN9CsrC5DCJFtNAqAYZhOAP7f3r1HS1WWcRz//lKy0HK1rNSV\nkHnXSjQzwyuFRGpQ2koszVZeumilq8ySWmmSectKXJlUZhqSWpmRaJZloYW6RLKLuqQV5iUUOGiQ\nyEV4+uN9RzfDzDkz+5yZOeP5fdbaizN7v/vdz35nmHnmfd+9Z26n4zAzM7NB7eiImNFbgaGSOA0H\n9gC2BR4GVnYyHjMzMxtUXkbKEW6JiJ7eCg6JxMnMzMxsIHhyuJmZmVmDnDiZmZmZNciJk5mZmVmD\nnDh1iKSTJS2Q9KykOyXt3Uf5D0h6IJe/T9Ih7Yq12zTTtpJOkDRb0tK8/Lav52Ioa/Z1W9jvKEnr\nJF3f6hi7VYn3hM0lfUfSfyStlPSgpHe3K95uUqJtT83tuULSI5K+KWmTdsXbDSQdIGmmpMfz/+2J\nDewzRtLc/Hp9SNJH2hHrQHPi1AGSJgEXAWcCewL3AbdIenWd8vsCM4Dvk64O/CVwg6Td2hNx92i2\nbYGDSG07Bng78CjwG0lbtz7a7lKibSv7bQtcCMxucYhdq8R7wjDgVmAkcASwE3Ai8HhbAu4iJdr2\nQ8C5ufwuwHHAJOCctgTcPTYF/gKcBPR5lVl+H7gR+B0wCrgY+IGkca0LsTV8VV0HSLoTuCsiTsmP\nRfrAnhoRF9Qofw0wPCImFtbNAeZFxEltCrsrNNu2NfZ/CfAUcHJETG9psF2mTNvm9pwNXA4cCGwe\nEUe0KeSuUeI94RPA54BdImJtW4PtMiXa9hJSu44rrPsG8LaIOLBNYXcVSeuA90XEzF7KnA8cEhG7\nF9b9hPSecGgbwhww7nFqs/xNcS9S1g1ApOz1VmB0nd1G5+1Ft/RSfkgq2bbVNgWGAUsHPMAu1o+2\nPRN4MiKuaG2E3atk204A5gCXSnpC0t8knZETVctKtu2fgb0qw3mStgMOBWa1NtoXvbfzIvkcGxI/\nuTLIvBrYCHiyav2TwM519tmqTvmtBja0rlembaudTxruqP4PPtQ13baS9gc+SuqWt/rKvG63A94J\nTAcOAXYAvkt6T5/SmjC7UtNtGxE/ycN4d+TeqY2AyyLi/JZG+uJX73PslZI2iYhVHYipFCdOZpmk\nLwJHAgdFxOpOx9PNJG0GXAWcGBFPdTqeF6GXkD50PpZ7UOZJ2gY4DSdO/SJpDDAZ+ARwNykpnSpp\nYUR8rZOx2eDgxKn9lgBrgS2r1m8JPFFnnyeaLD9UlWlbACSdBpwOjI2If7QmvK7WbNtuD7we+FX+\n1g55aoCk1cDOEbGgRbF2mzKv24XA6lh/kuoDwFaSNo6I5wY+zK5Upm3PBq4qDC//I38RmAY4cSqv\n3ufYsm7qbQLPcWq7iFhD+sHhsZV1+YNlLGlsvZY5xfLZuLzespJti6TTgS8B4yNiXqvj7EYl2vYB\n4M2kq0BH5WUm8Pv896MtDrlrlHzd/onUE1K0M7DQSdMLSrbtcGBd1bp1hX2tnFqfY++iGz/HIsJL\nmxfScNAK4FjS5a7TgB7gNXn7VcDXC+VHA6uAz5LeHM8i/VDxbp0+l8G2lGjbL+S2PJz07aeybNrp\ncxlsS7NtW2P/K4DrO30eg3Ep8brdBngamArsCBxG+kb/xU6fy2BbSrTtmbltJ5F+9HUcMB+Y0elz\nGUwL6UKaUaQvR+uAU/PjEXn7ucCVhfLbAstJ80h3Jt3GYDVwcKfPpdnFQ3UdEBHX5cmHZ5M+pP9C\n6u1YnItsAzxXKD8n31vknLzMB94bEfe3N/LBr9m2Jc1jGAb8rKqqr+Y6LCvRttagEu8Jj0kaD3yL\ndF+ix/Pffd5yY6gp8bqdQkoEpgCvAxaTeku/3Lagu8NbgdtI93AK0r2yAK4k3ftqK2BEpXBEPCzp\nMNLr9DPAY8DxEdF1F+L4Pk5mZmZmDfIcJzMzM7MGOXEyMzMza5ATJzMzM7MGOXEyMzMza5ATJzMz\nM7MGOXEyMzMza5ATJzMzM7MGOXEyMzMza5ATJzMzM7MGOXEys1IkbS9pnaTdOh1LGZLGSloraXgf\n5R6VdFK74jKzwc2Jk9kQJemKnPiszf9W/t6uiWpa9ptNhcSssiyW9GtJuw/QIf4IbB0RK/Lxjpe0\nuEa5PYAfDtAxa5J0R+E8n5X0oKTPl6jnx5Kua0WMZpY4cTIb2m4m/RhnZdkaWNDE/mpFUAUBHEiK\n7d3A5sBNkjbrd8URz0XEosIqUSMRjIieiFjZ3+P1FQ5wKek8dyL9WO85ko5v8XHNrElOnMyGtlUR\nsTgiFhWWAJB0aO4JeUrSEkkzJb2hXkWSXiVphqRFklbkXpNjCttHSvppob5fSBpRr77KbsDSHNdc\n4HRScrd34ZjTc53/k3RjscdM0raSfiVpad7+V0nj8raxuYdnuKSxwPeALQo9b5NzueeH6iRdK2l6\n1XkPk9Qj6aj8WJK+JOlfuR3ulXR4A8/Finyej0bED4H7gXGF42ws6XJJCwrt+6nC9inA0cD7C+ew\nbz/a3sxqcOJkZvW8HLgQeAswlpTE/LyX8ucCOwDjgV2Ak4AeSMkF8BtgCbAfsD/wLHCzpGbeh1bm\nOF6aH08HdgcOAfYFhgGzCnVeRnqf2x94E3AGsKJQX6WHaTbwOWApsCUpOftWjeNfDUyU9LLCusPy\ncX+ZH38FOAo4AdgVmArMkDS60ZOUNIbU87S6sHoj4N/AEbneKcB5kt6Xt59Hen5uLJzDXQPY9mYG\nbNzpAMysoyZIWl54fFNETAKIiPWSJEknAv+RtFNEPFSjrhHAvIiYlx8/Utj2IWB1RHyyUN9HgadJ\nQ3F/6CtQSa8CvgwsA+6RtCspYdo790aRe7geASaQEpkRwPSIuD9X83CtuiNijaRl6c+oNc+p4mZg\nDfBe4Nq87oPADRHxbE6oTgcOrMQE/EjSQcDHgTm91H2KpE+SksJhpARvaiHGVcDZhfL/lrQ/cGQ+\n/jOSVlafQ26TfrW9mb3A3zbMhrbfk3psRuXlM5UNknaUdE0ecloGzCf10IysU9elwIclzZV0nqR9\nCttGAbtKWl5ZSD0gw4Dt+4jx7ly+h9TT8oGI6CH1aq0qJCjkhGF+LgdwMfBVSbdLOlPSG/tukvoi\nYg3wU9KQGHmu1QRSzxekXqKXA7dVnesHGzjPK0nPxX7ALcDZEXFPsYCkT0u6R2mi/HLgOOo/HxX9\naXszq+IeJ7Oh7ZmIqDcZfBbwEOnDeSGpJ+Q+XhgmW09EzJI0kjR0dTApefh2REwGNgPuBI5lwwnl\nvfXwQBqamg/0RMSyvk9pvZi+J+mmHNN4YLKkUyLismbqqXI18NvcAzaR1AN2a95WmbQ+Hniyar++\nJpg/nZ+LBZKOBP4p6c6ImA3P9xydB5wK3A0sJw09juqj3v60vZlVceJkZhuQ9FrSfKUPR8Rded0Y\nNrzqbL3HEbGE1HNypaQ5pKGlycC9pOGtRRHxTBOhBPBYneTuAeClkt5a6ZnJce9ImlhdiekxYBow\nTdIFpLlHtRKn1aR5RL0HFHG7pIXAJOBw4NqIWJc3/z3XMzIiehuW6+sYyyVdAlxEnghPmsM1OyK+\nXyknaYca51B9X6qybW9mNXiozsxq6QGeAj4uabt81dmFNco934MhaYqkCUr3X3oTcCgvJDA/Bv4L\n3CBpv3y12zskXSJpy17iqHu7g4h4ELgJuFzSaEmjSENm/yJNkEbSxZLG5ePtBYwpxFTtYWBzSQdJ\n2qJqAni1a4CTgXeQeqAqMS0jTSq/WNIxue32zENsR/dSXy2XAW+UNDE/ng/sI+ngPIx6DrBnjXMY\nlbdvIWkjyre9mdXgxMnMNhARa0k9KvuQelEuBE6rVbTw9xrSUNJ9wG2koaljcn3PAAcAjwPXk5KX\naaQenv/1FkofoR6bjzcLuANYBbyn0AO0MWnu1f2kZOrvFOZxrXegiNuBHwA/AxYBn+0lhquB3YAF\nEXF3VT1nkK4wnJyPezPpHlS93R+r1v2jluTjnJVXXQrMBK4jTTJ/BRv2nE0jJY5z8zns04+2N7Ma\nlG/ZYmZmZmZ9cI+TmZmZWYOcOJmZmZk1yImTmZmZWYOcOJmZmZk1yImTmZmZWYOcOJmZmZk1yImT\nmZmZWYOcOJmZmZk1yImTmZmZWYOcOJmZmZk1yImTmZmZWYOcOJmZmZk16P/C0OuUtsD2HwAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xa1c5630>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import roc_curve\n",
    "from sklearn.metrics import auc\n",
    "\n",
    "colors = ['black', 'orange', 'blue', 'green']\n",
    "linestyles = [':', '--', '-.', '-']\n",
    "for clf, label, clr, ls \\\n",
    "        in zip(all_clf, \n",
    "               clf_labels, colors, linestyles):\n",
    "\n",
    "    # assuming the label of the positive class is 1\n",
    "    y_pred = clf.fit(X_train, \n",
    "                     y_train).predict_proba(X_test)[:, 1]\n",
    "    fpr, tpr, thresholds = roc_curve(y_true=y_test, \n",
    "                                     y_score=y_pred)\n",
    "    roc_auc = auc(x=fpr, y=tpr)\n",
    "    plt.plot(fpr, tpr, \n",
    "             color=clr, \n",
    "             linestyle=ls, \n",
    "             label='%s (auc = %0.2f)' % (label, roc_auc))\n",
    "\n",
    "plt.legend(loc='lower right')\n",
    "plt.plot([0, 1], [0, 1], \n",
    "         linestyle='--', \n",
    "         color='gray', \n",
    "         linewidth=2)\n",
    "\n",
    "plt.xlim([-0.1, 1.1])\n",
    "plt.ylim([-0.1, 1.1])\n",
    "plt.grid()\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "\n",
    "plt.tight_layout()\n",
    "# plt.savefig('./figures/roc.png', dpi=300)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see in the resulting ROC, the ensemble classifier also performs well on\n",
    "the test set (ROC AUC = 0.86)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we tune the individual classifier parameters for ensemble classification, let's call the get_params method to get a basic idea of how we can access the individual parameters inside a GridSearch object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'decisiontreeclassifier': DecisionTreeClassifier(class_weight=None, criterion='entropy', max_depth=1,\n",
       "             max_features=None, max_leaf_nodes=None,\n",
       "             min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "             min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "             presort=False, random_state=0, splitter='best'),\n",
       " 'decisiontreeclassifier__class_weight': None,\n",
       " 'decisiontreeclassifier__criterion': 'entropy',\n",
       " 'decisiontreeclassifier__max_depth': 1,\n",
       " 'decisiontreeclassifier__max_features': None,\n",
       " 'decisiontreeclassifier__max_leaf_nodes': None,\n",
       " 'decisiontreeclassifier__min_impurity_split': 1e-07,\n",
       " 'decisiontreeclassifier__min_samples_leaf': 1,\n",
       " 'decisiontreeclassifier__min_samples_split': 2,\n",
       " 'decisiontreeclassifier__min_weight_fraction_leaf': 0.0,\n",
       " 'decisiontreeclassifier__presort': False,\n",
       " 'decisiontreeclassifier__random_state': 0,\n",
       " 'decisiontreeclassifier__splitter': 'best',\n",
       " 'pipeline': Pipeline(steps=[['sc', StandardScaler(copy=True, with_mean=True, with_std=True)], ['clf', KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "            metric_params=None, n_jobs=1, n_neighbors=1, p=2,\n",
       "            weights='uniform')]]),\n",
       " 'pipeline__clf': KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "            metric_params=None, n_jobs=1, n_neighbors=1, p=2,\n",
       "            weights='uniform'),\n",
       " 'pipeline__clf__algorithm': 'auto',\n",
       " 'pipeline__clf__leaf_size': 30,\n",
       " 'pipeline__clf__metric': 'minkowski',\n",
       " 'pipeline__clf__metric_params': None,\n",
       " 'pipeline__clf__n_jobs': 1,\n",
       " 'pipeline__clf__n_neighbors': 1,\n",
       " 'pipeline__clf__p': 2,\n",
       " 'pipeline__clf__weights': 'uniform',\n",
       " 'pipeline__sc': StandardScaler(copy=True, with_mean=True, with_std=True),\n",
       " 'pipeline__sc__copy': True,\n",
       " 'pipeline__sc__with_mean': True,\n",
       " 'pipeline__sc__with_std': True,\n",
       " 'pipeline__steps': [['sc',\n",
       "   StandardScaler(copy=True, with_mean=True, with_std=True)],\n",
       "  ['clf',\n",
       "   KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "              metric_params=None, n_jobs=1, n_neighbors=1, p=2,\n",
       "              weights='uniform')]]}"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mv_clf.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the values returned by the get_params method, we now know how to access the individual classifier's attributes. Let's now tune the decision tree depth via a grid search for demonstration purposes. The code is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.850+/-0.04 {'pipeline__clf__n_neighbors': 5, 'decisiontreeclassifier__max_depth': 1}\n",
      "0.861+/-0.03 {'pipeline__clf__n_neighbors': 15, 'decisiontreeclassifier__max_depth': 1}\n",
      "0.870+/-0.03 {'pipeline__clf__n_neighbors': 20, 'decisiontreeclassifier__max_depth': 1}\n",
      "0.856+/-0.04 {'pipeline__clf__n_neighbors': 5, 'decisiontreeclassifier__max_depth': 2}\n",
      "0.868+/-0.04 {'pipeline__clf__n_neighbors': 15, 'decisiontreeclassifier__max_depth': 2}\n",
      "0.875+/-0.04 {'pipeline__clf__n_neighbors': 20, 'decisiontreeclassifier__max_depth': 2}\n"
     ]
    }
   ],
   "source": [
    "from sklearn.grid_search import GridSearchCV\n",
    "\n",
    "params = {'decisiontreeclassifier__max_depth': [1,2],\n",
    "          'pipeline__clf__n_neighbors': [5,15,20]}\n",
    "\n",
    "grid = GridSearchCV(estimator=mv_clf, \n",
    "                    param_grid=params, \n",
    "                    cv=10, \n",
    "                    scoring='roc_auc')\n",
    "grid.fit(X_train, y_train)\n",
    "\n",
    "for params, mean_score, scores in grid.grid_scores_:\n",
    "    print(\"%0.3f+/-%0.2f %r\"\n",
    "            % (mean_score, scores.std() / 2, params))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the grid search has completed, we can print the different hyperparameter\n",
    "value combinations and the average ROC AUC scores computed via 10-fold\n",
    "cross-validation. The code is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameters: {'pipeline__clf__n_neighbors': 20, 'decisiontreeclassifier__max_depth': 2}\n",
      "Accuracy: 0.87\n"
     ]
    }
   ],
   "source": [
    "print('Best parameters: %s' % grid.best_params_)\n",
    "print('Accuracy: %.2f' % grid.best_score_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, we get the best cross-validation results when we choose a higher n_neighbors (n = 20) whereas the tree depth does not seem to affect the performance at all, suggesting that a decision stump is sufficient to separate the data. To remind ourselves that it is a bad practice to use the test dataset more than once for model evaluation, we are not going to estimate the generalization performance of the tuned hyperparameters in this section. We will move on swiftly to an alternative approach for ensemble learning: bagging."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bagging -- Building an ensemble of classifiers from bootstrap samples\n",
    "\n",
    "Bagging is an ensemble learning technique that is closely related to the MajorityVoteClassifier that we implemented in the previous section,\n",
    "as illustrated in the following diagram:\n",
    "\n",
    "<img src = \"images/bagging.png\">\n",
    "\n",
    "However, instead of using the same training set to fit the individual classifiers in the ensemble, we draw bootstrap samples (random samples with replacement) from the initial training set, which is why bagging is also known as bootstrap aggregating."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = df[features]\n",
    "y = df['num']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((297, 13), (297,))"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape , y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we encode the class labels into binary format and split the dataset into\n",
    "60 percent training and 40 percent test set, respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.cross_validation import train_test_split\n",
    "\n",
    "le = LabelEncoder()\n",
    "y = le.fit_transform(y)\n",
    "\n",
    "X_train, X_test, y_train, y_test =\\\n",
    "            train_test_split(X, y, \n",
    "                             test_size=0.40, \n",
    "                             random_state=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A BaggingClassifier algorithm is already implemented in scikit-learn, which we can import from the ensemble submodule. Here, we will use an unpruned decision tree as the base classifier and create an ensemble of 500 decision trees fitted on different bootstrap samples of the training dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import BaggingClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "tree = DecisionTreeClassifier(criterion='entropy', \n",
    "                              max_depth=None)\n",
    "\n",
    "bag = BaggingClassifier(base_estimator=tree,\n",
    "                        n_estimators=500, \n",
    "                        max_samples=1.0, \n",
    "                        max_features=1.0, \n",
    "                        bootstrap=True, \n",
    "                        bootstrap_features=False, \n",
    "                        n_jobs=1, \n",
    "                        random_state=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we will calculate the accuracy score of the prediction on the training and test dataset to compare the performance of the bagging classifier to the performance of a single unpruned decision tree. Based on the accuracy values, the unpruned decision tree predicts all class labels of the training samples correctly; however, the substantially lower test accuracy indicates high variance (overfitting) of the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision tree train/test accuracies 1.000/0.714\n",
      "Bagging train/test accuracies 1.000/0.756\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "tree = tree.fit(X_train, y_train)\n",
    "y_train_pred = tree.predict(X_train)\n",
    "y_test_pred = tree.predict(X_test)\n",
    "\n",
    "tree_train = accuracy_score(y_train, y_train_pred)\n",
    "tree_test = accuracy_score(y_test, y_test_pred)\n",
    "print('Decision tree train/test accuracies %.3f/%.3f'\n",
    "      % (tree_train, tree_test))\n",
    "\n",
    "bag = bag.fit(X_train, y_train)\n",
    "y_train_pred = bag.predict(X_train)\n",
    "y_test_pred = bag.predict(X_test)\n",
    "\n",
    "bag_train = accuracy_score(y_train, y_train_pred) \n",
    "bag_test = accuracy_score(y_test, y_test_pred) \n",
    "print('Bagging train/test accuracies %.3f/%.3f'\n",
    "      % (bag_train, bag_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although the training accuracies of the decision tree and bagging classifier are similar on the training set (both 1.0), we can see that the bagging classifier has a slightly better generalization performance as estimated on the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In practice, more complex classification tasks and datasets' high dimensionality can easily lead to overfitting in single decision trees and this is where the bagging algorithm can really play out its strengths. Finally, we shall note that the bagging algorithm can be an effective approach to reduce the variance of a model. However, bagging is ineffective in reducing model bias, which is why we want to choose an ensemble of classifiers with low bias, for example, unpruned decision trees."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Leveraging weak learners via adaptive boosting \n",
    "\n",
    "In this section about ensemble methods, we will discuss boosting with a special focus on its most common implementation, AdaBoost (short for Adaptive Boosting).\n",
    "\n",
    "In boosting, the ensemble consists of very simple base classifiers, also often referred to as weak learners, that have only a slight performance advantage over random guessing. A typical example of a weak learner would be a decision tree stump. The key concept behind boosting is to focus on training samples that are hard to classify, that is, to let the weak learners subsequently learn from misclassified training samples to improve the performance of the ensemble. In contrast to bagging, the initial formulation of boosting, the algorithm uses random subsets of training samples drawn from the training dataset without replacement. The original boosting procedure is summarized in four key steps as follows:\n",
    "\n",
    "1. Draw a random subset of training samples d1 without replacement from the training set D to train a weak learner C1.\n",
    "2. Draw second random training subset d2 without replacement from the training set and add 50 percent of the samples that were previously misclassified to train a weak learner C2.\n",
    "3. Find the training samples d3 in the training set D on which C1 and C2 disagree to train a third weak learner C3.\n",
    "4. Combine the weak learners C1, C2, and C3 via majority voting.\n",
    "\n",
    "Boosting can lead to a decrease in bias as well as variance compared to bagging models. In practice, however, boosting algorithms such as AdaBoost are also known for their high variance, that is, the tendency to overfit the training data. In contrast to the original boosting procedure as described here, AdaBoost uses the complete training set to train the weak learners where the training samples are reweighted in each iteration to build a strong classifier that learns from the mistakes of the previous weak learners in the ensemble. Take a look at the following figure to get a better grasp of the basic concept behind AdaBoost:\n",
    "\n",
    "<img src = \"images/adaboost.png\">\n",
    "\n",
    "To walk through the AdaBoost illustration step by step, we start with subfigure 1, which represents a training set for binary classification where all training samples are assigned equal weights. Based on this training set, we train a decision stump (shown as a dashed line) that tries to classify the samples of the two classes (triangles and circles) as well as possible by minimizing the cost function (or the impurity score in the special case of decision tree ensembles). For the next round (subfigure 2),\n",
    "we assign a larger weight to the two previously misclassified samples (circles). Furthermore, we lower the weight of the correctly classified samples. The next decision stump will now be more focused on the training samples that have the largest weights, that is, the training samples that are supposedly hard to classify. The weak learner shown in subfigure 2 misclassifies three different samples from the circle-class, which are then assigned a larger weight as shown in subfigure 3. Assuming that our AdaBoost ensemble only consists of three rounds of boosting, we would then combine the three weak learners trained on different reweighted training subsets by a weighted majority vote, as shown in subfigure 4."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Skipping to the more practical part, let's now train an AdaBoost ensemble classifier via scikit-learn. We will use the same Wine subset that we used in the previous section to train the bagging meta-classifier. Via the base_estimator attribute, we will train the AdaBoostClassifier on 500 decision tree stumps:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "\n",
    "tree = DecisionTreeClassifier(criterion='entropy', \n",
    "                              max_depth=1)\n",
    "\n",
    "ada = AdaBoostClassifier(base_estimator=tree,\n",
    "                         n_estimators=500, \n",
    "                         learning_rate=0.1,\n",
    "                         random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision tree train/test accuracies 0.770/0.756\n"
     ]
    }
   ],
   "source": [
    "tree = tree.fit(X_train, y_train)\n",
    "y_train_pred = tree.predict(X_train)\n",
    "y_test_pred = tree.predict(X_test)\n",
    "\n",
    "tree_train = accuracy_score(y_train, y_train_pred)\n",
    "tree_test = accuracy_score(y_test, y_test_pred)\n",
    "print('Decision tree train/test accuracies %.3f/%.3f'\n",
    "      % (tree_train, tree_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the decision tree stump seems to overfit the training data in contrast with the unpruned decision tree that we saw in the previous section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AdaBoost train/test accuracies 0.961/0.790\n"
     ]
    }
   ],
   "source": [
    "ada = ada.fit(X_train, y_train)\n",
    "y_train_pred = ada.predict(X_train)\n",
    "y_test_pred = ada.predict(X_test)\n",
    "\n",
    "ada_train = accuracy_score(y_train, y_train_pred) \n",
    "ada_test = accuracy_score(y_test, y_test_pred) \n",
    "print('AdaBoost train/test accuracies %.3f/%.3f'\n",
    "      % (ada_train, ada_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the AdaBoost model predicts all class labels of the training set correctly and also shows a slightly improved test set performance compared to the decision tree stump. However, we also see that we introduced additional variance by our attempt to reduce the model bias.\n",
    "\n",
    "Although we used another simple example for demonstration purposes, we can see that the performance of the AdaBoost classifier is slightly improved compared to the decision stump and achieved very similar accuracy scores to the bagging classifier that we trained in the previous section. However, we should note that it is considered as bad practice to select a model based on the repeated usage of the test set. The estimate of the generalization performance may be too optimistic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As concluding remarks about ensemble techniques, it is worth noting that\n",
    "ensemble learning increases the computational complexity compared to individual classifiers. In practice, we need to think carefully whether we want to pay the price of increased computational costs for an often relatively modest improvement of predictive performance.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_____\n",
    "## Summary\n",
    "\n",
    "In this chapter, we looked at some of the most popular and widely used techniques for ensemble learning. Ensemble methods combine different classification models\n",
    "to cancel out their individual weakness, which often results in stable and\n",
    "well-performing models that are very attractive for industrial applications\n",
    "as well as machine learning competitions.\n",
    "\n",
    "In the beginning of this chapter, we implemented a MajorityVoteClassifier in Python that allows us to combine different algorithm for classification. We then looked at bagging, a useful technique to reduce the variance of a model by drawing random bootstrap samples from the training set and combining the individually trained classifiers via majority vote. Then we discussed AdaBoost, which is an algorithm that is based on weak learners that subsequently learn from mistakes.\n",
    "___"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}