07-08-Ensemble of Ensembles 1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ensemble of ensembles - model stacking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Ensemble with different types of classifiers**: \n",
    "  * Different types of classifiers (E.g., logistic regression, decision trees, random forest, etc.) are fitted on the same training data\n",
    "  * Results are combined based on either \n",
    "    * majority voting (classification) or \n",
    "    * average (regression)\n",
    "  \n",
    "\n",
    "* **Ensemble with a single type of classifier**: \n",
    "  * Bootstrap samples are drawn from training data \n",
    "  * With each bootstrap sample, model (E.g., Individual model may be decision trees, random forest, etc.) will be fitted \n",
    "  * All the results are combined to create an ensemble. \n",
    "  * Suitabe for highly flexible models that is prone to overfitting / high variance. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "## Combining Method\n",
    "\n",
    "* **Majority voting or average**: \n",
    "  * Classification: Largest number of votes (mode) \n",
    "  * Regression problems: Average (mean).\n",
    "  \n",
    "  \n",
    "* **Method of application of meta-classifiers on outcomes**: \n",
    "  * Binary outcomes: 0 / 1 from individual classifiers\n",
    "  * Meta-classifier is applied on top of the individual classifiers. \n",
    "  \n",
    "  \n",
    "* **Method of application of meta-classifiers on probabilities**: \n",
    "  * Probabilities are obtained from individual classifiers. \n",
    "  * Applying meta-classifier\n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/anthonyn/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ipykernel_launcher.py:20: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version\n",
      "of pandas will change to not sort by default.\n",
      "\n",
      "To accept the future behavior, pass 'sort=True'.\n",
      "\n",
      "To retain the current behavior and silence the warning, pass sort=False\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(1470, 51)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"data//WA_Fn-UseC_-HR-Employee-Attrition.csv\")\n",
    "df.pop('EmployeeNumber')\n",
    "df.pop('Over18')\n",
    "df.pop('StandardHours')\n",
    "df.pop('EmployeeCount')\n",
    "y = df['Attrition']\n",
    "X = df\n",
    "X.pop('Attrition')\n",
    "from sklearn import preprocessing\n",
    "le = preprocessing.LabelBinarizer()\n",
    "y = le.fit_transform(y)\n",
    "ind_BusinessTravel = pd.get_dummies(df['BusinessTravel'], prefix='BusinessTravel')\n",
    "ind_Department = pd.get_dummies(df['Department'], prefix='Department')\n",
    "ind_EducationField = pd.get_dummies(df['EducationField'], prefix='EducationField')\n",
    "ind_Gender = pd.get_dummies(df['Gender'], prefix='Gender')\n",
    "ind_JobRole = pd.get_dummies(df['JobRole'], prefix='JobRole')\n",
    "ind_MaritalStatus = pd.get_dummies(df['MaritalStatus'], prefix='MaritalStatus')\n",
    "ind_OverTime = pd.get_dummies(df['OverTime'], prefix='OverTime')\n",
    "df1 = pd.concat([ind_BusinessTravel, ind_Department, ind_EducationField, ind_Gender, \n",
    "                 ind_JobRole, ind_MaritalStatus, ind_OverTime])\n",
    "df1 = pd.concat([ind_BusinessTravel, ind_Department, ind_EducationField, ind_Gender, \n",
    "                 ind_JobRole, ind_MaritalStatus, ind_OverTime, df.select_dtypes(['int64'])], axis=1)\n",
    "df1.dropna(inplace=True)\n",
    "df1.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(df1, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn import preprocessing\n",
    "from sklearn.model_selection import cross_val_score, cross_val_predict\n",
    "from sklearn.metrics import accuracy_score, classification_report\n",
    "from sklearn.metrics import confusion_matrix, roc_auc_score\n",
    "def print_score(clf, X_train, X_test, y_train, y_test, train=True):\n",
    "    '''\n",
    "    v0.1 Follow the scikit learn library format in terms of input\n",
    "    print the accuracy score, classification report and confusion matrix of classifier\n",
    "    '''\n",
    "    lb = preprocessing.LabelBinarizer()\n",
    "    lb.fit(y_train)\n",
    "    if train:\n",
    "        '''\n",
    "        training performance\n",
    "        '''\n",
    "        print(\"Train Result:\\n\")\n",
    "        print(\"accuracy score: {0:.4f}\\n\".format(accuracy_score(y_train, clf.predict(X_train))))\n",
    "        print(\"Classification Report: \\n {}\\n\".format(classification_report(y_train, clf.predict(X_train))))\n",
    "        print(\"Confusion Matrix: \\n {}\\n\".format(confusion_matrix(y_train, clf.predict(X_train))))\n",
    "        print(\"ROC AUC: {0:.4f}\\n\".format(roc_auc_score(lb.transform(y_train), \n",
    "                                                        lb.transform(clf.predict(X_train)))))\n",
    "\n",
    "        #cv_res = cross_val_score(clf, X_train, y_train, cv=10, scoring='accuracy')\n",
    "        #print(\"Average Accuracy: \\t {0:.4f}\".format(np.mean(cv_res)))\n",
    "        #print(\"Accuracy SD: \\t\\t {0:.4f}\".format(np.std(cv_res)))\n",
    "        \n",
    "    elif train==False:\n",
    "        '''\n",
    "        test performance\n",
    "        '''\n",
    "        res_test = clf.predict(X_test)\n",
    "        print(\"Test Result:\\n\")        \n",
    "        print(\"accuracy score: {0:.4f}\\n\".format(accuracy_score(y_test, clf.predict(X_test))))\n",
    "        print(\"Classification Report: \\n {}\\n\".format(classification_report(y_test, clf.predict(X_test))))\n",
    "        print(\"Confusion Matrix: \\n {}\\n\".format(confusion_matrix(y_test, clf.predict(X_test))))      \n",
    "        print(\"ROC AUC: {0:.4f}\\n\".format(roc_auc_score(lb.transform(y_test), lb.transform(res_test))))\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model 1: Decision Tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
       "            splitter='best')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_clf = DecisionTreeClassifier()\n",
    "tree_clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Result:\n",
      "\n",
      "accuracy score: 1.0000\n",
      "\n",
      "Classification Report: \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "          0       1.00      1.00      1.00       922\n",
      "          1       1.00      1.00      1.00       180\n",
      "\n",
      "avg / total       1.00      1.00      1.00      1102\n",
      "\n",
      "\n",
      "Confusion Matrix: \n",
      " [[922   0]\n",
      " [  0 180]]\n",
      "\n",
      "ROC AUC: 1.0000\n",
      "\n",
      "Test Result:\n",
      "\n",
      "accuracy score: 0.7935\n",
      "\n",
      "Classification Report: \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "          0       0.88      0.87      0.88       311\n",
      "          1       0.34      0.37      0.36        57\n",
      "\n",
      "avg / total       0.80      0.79      0.80       368\n",
      "\n",
      "\n",
      "Confusion Matrix: \n",
      " [[271  40]\n",
      " [ 36  21]]\n",
      "\n",
      "ROC AUC: 0.6199\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print_score(tree_clf, X_train, X_test, y_train, y_test, train=True)\n",
    "print_score(tree_clf, X_train, X_test, y_train, y_test, train=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model 2: Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "            oob_score=False, random_state=None, verbose=0,\n",
       "            warm_start=False)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_clf = RandomForestClassifier()\n",
    "rf_clf.fit(X_train, y_train.ravel())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Result:\n",
      "\n",
      "accuracy score: 0.9864\n",
      "\n",
      "Classification Report: \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "          0       0.98      1.00      0.99       922\n",
      "          1       1.00      0.92      0.96       180\n",
      "\n",
      "avg / total       0.99      0.99      0.99      1102\n",
      "\n",
      "\n",
      "Confusion Matrix: \n",
      " [[922   0]\n",
      " [ 15 165]]\n",
      "\n",
      "ROC AUC: 0.9583\n",
      "\n",
      "Test Result:\n",
      "\n",
      "accuracy score: 0.8696\n",
      "\n",
      "Classification Report: \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "          0       0.87      0.99      0.93       311\n",
      "          1       0.80      0.21      0.33        57\n",
      "\n",
      "avg / total       0.86      0.87      0.84       368\n",
      "\n",
      "\n",
      "Confusion Matrix: \n",
      " [[308   3]\n",
      " [ 45  12]]\n",
      "\n",
      "ROC AUC: 0.6004\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print_score(rf_clf, X_train, X_test, y_train, y_test, train=True)\n",
    "print_score(rf_clf, X_train, X_test, y_train, y_test, train=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "en_en = pd.DataFrame()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 0.],\n",
       "       [1., 0.],\n",
       "       [1., 0.],\n",
       "       ...,\n",
       "       [1., 0.],\n",
       "       [1., 0.],\n",
       "       [1., 0.]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_clf.predict_proba(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mSignature:\u001b[0m \u001b[0mtree_clf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict_proba\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcheck_input\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "Predict class probabilities of the input samples X.\n",
       "\n",
       "The predicted class probability is the fraction of samples of the same\n",
       "class in a leaf.\n",
       "\n",
       "check_input : boolean, (default=True)\n",
       "    Allow to bypass several input checking.\n",
       "    Don't use this parameter unless you know what you do.\n",
       "\n",
       "Parameters\n",
       "----------\n",
       "X : array-like or sparse matrix of shape = [n_samples, n_features]\n",
       "    The input samples. Internally, it will be converted to\n",
       "    ``dtype=np.float32`` and if a sparse matrix is provided\n",
       "    to a sparse ``csr_matrix``.\n",
       "\n",
       "check_input : bool\n",
       "    Run check_array on X.\n",
       "\n",
       "Returns\n",
       "-------\n",
       "p : array of shape = [n_samples, n_classes], or a list of n_outputs\n",
       "    such arrays if n_outputs > 1.\n",
       "    The class probabilities of the input samples. The order of the\n",
       "    classes corresponds to that in the attribute `classes_`.\n",
       "\u001b[0;31mFile:\u001b[0m      ~/anaconda3/envs/tensorflow/lib/python3.6/site-packages/sklearn/tree/tree.py\n",
       "\u001b[0;31mType:\u001b[0m      method\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tree_clf.predict_proba?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "en_en['tree_clf'] = pd.DataFrame(tree_clf.predict_proba(X_train))[1]\n",
    "en_en['rf_clf'] =  pd.DataFrame(rf_clf.predict_proba(X_train))[1]\n",
    "col_name = en_en.columns\n",
    "en_en = pd.concat([en_en, pd.DataFrame(y_train).reset_index(drop=True)], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tree_clf</th>\n",
       "      <th>rf_clf</th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   tree_clf  rf_clf  0\n",
       "0       0.0     0.0  0\n",
       "1       0.0     0.2  0\n",
       "2       0.0     0.0  0\n",
       "3       1.0     0.8  1\n",
       "4       0.0     0.1  0"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en_en.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tmp = list(col_name)\n",
    "tmp.append('ind')\n",
    "en_en.columns = tmp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tree_clf</th>\n",
       "      <th>rf_clf</th>\n",
       "      <th>ind</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   tree_clf  rf_clf  ind\n",
       "0       0.0     0.0    0\n",
       "1       0.0     0.2    0\n",
       "2       0.0     0.0    0\n",
       "3       1.0     0.8    1\n",
       "4       0.0     0.1    0"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en_en.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Meta Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "m_clf = LogisticRegression(fit_intercept=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=False,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m_clf.fit(en_en[['tree_clf', 'rf_clf']], en_en['ind'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "en_test = pd.DataFrame()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "en_test['tree_clf'] = pd.DataFrame(tree_clf.predict_proba(X_test))[1]\n",
    "en_test['rf_clf'] =  pd.DataFrame(rf_clf.predict_proba(X_test))[1]\n",
    "col_name = en_en.columns\n",
    "en_test['combined'] = m_clf.predict(en_test[['tree_clf', 'rf_clf']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "col_name = en_test.columns\n",
    "tmp = list(col_name)\n",
    "tmp.append('ind')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['tree_clf', 'rf_clf', 'combined', 'ind']"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "en_test = pd.concat([en_test, pd.DataFrame(y_test).reset_index(drop=True)], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "en_test.columns = tmp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "combined    0   1\n",
      "ind              \n",
      "0         271  40\n",
      "1          36  21\n"
     ]
    }
   ],
   "source": [
    "print(pd.crosstab(en_test['ind'], en_test['combined']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7935\n"
     ]
    }
   ],
   "source": [
    "print(round(accuracy_score(en_test['ind'], en_test['combined']), 4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.88      0.87      0.88       311\n",
      "          1       0.34      0.37      0.36        57\n",
      "\n",
      "avg / total       0.80      0.79      0.80       368\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(en_test['ind'], en_test['combined']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}