Phase_01.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Set Up \n",
    "\n",
    "#### What programs you will need installed: \n",
    "- Anaconda 5.0.1 with Python 3.6   \n",
    "https://www.anaconda.com/download/   \n",
    "-- Python 3.6 for its better handling of text data  \n",
    "-- Anacond for its collection of popular data science libraries and packages.\n",
    "\n",
    " \n",
    "- XGboost   \n",
    "-- Arguably the most popular classifer package\n",
    "\n",
    "You can install XGBoost at the comand line via:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "conda install -c conda-forge xgboost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Embeddings Data  \n",
    "\n",
    "You will also need the Data for today, which is available at:\n",
    "\n",
    "https://github.com/GeorgeMcIntire/fake_real_news_dataset"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "git pull ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What We Will Cover\n",
    "\n",
    "1. Logistic classifier \n",
    "2. Metrics: precision, recal, F1\n",
    "3. XGBoost classifer \n",
    "4. Embedded text\n",
    "\n",
    "\n",
    "## 1) Logistic Regression\n",
    "\n",
    "#### Binary Classification\n",
    "\n",
    "In a binary classification problem, we try to predict a binary outcome and we assign labels of 0 and 1 to our data.\n",
    "\n",
    "Here are some example questions:\n",
    "\n",
    "* Does a patient have cancer?\n",
    "* Will a team win the next game?\n",
    "* Will the customer buy my product?\n",
    "* Will I get the loan?\n",
    "* Is this article fake?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by looking at an example. We're going to be using some NFL data. The x axis is the number of touchdowns scored by team over a season and the y axis is whether they lost or won the game indicated by a value of 0 or 1 respectively.\n",
    "\n",
    "![NFL data](images/nfl.png)\n",
    "\n",
    "So, how do we predict whether we have a win or a loss if we are given a score? Note that we are going to be predicting values between 0 and 1. Close to 0 means we're sure it's in class 0, close to 1 means we're sure it's in class 1, and closer to 0.5 means we don't know.\n",
    "\n",
    "If we use linear regression for the NFL example above, we will certainly do better than randomly guessing, but it doesn't accurately represent the data:\n",
    "\n",
    "![NFL linear regression](images/linefit.png)\n",
    "\n",
    "So clearly a line is not the best way to model this data. So we need to find a better curve."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Logistic Function\n",
    "\n",
    "First, we will just pull a function out of the data science bag of tricks and show that it works reasonably well.\n",
    "\n",
    "And, second, we are going to understand how we came up with that function and how it is related to binary outcomes and odds. But before that let's understand this a bit better.\n",
    "\n",
    "This function will need to have a value of 0 for the loss scores and 1 for the win scores. To make sense it will need to be 0 for some score and all scores below it and be 1 for some other score and all scores above it. And it will need to smoothly increase from 0 to 1 in the intermediate range.\n",
    "\n",
    "It will need to look something like this:\n",
    "\n",
    "![logistic](images/standardLogisticFunction.png)\n",
    "\n",
    "\n",
    "A function that has the above shape is:\n",
    "\n",
    "$$ f(t) = \\frac{e^t}{1 + e^t} = \\frac{1}{1 + e^{-t}} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the logistic function, also known as the sigmoid function. Note that as t approaches infinity, the value of the logistic function approaches 1 and as t approaches negative infinity, the value of the logistic function approaches 0.\n",
    "\n",
    "We will use $ t = \\beta_0 + \\beta_1x $, which means we'll be dealing with a familiar looking linear function.  \n",
    "\n",
    "This gives us:\n",
    "\n",
    "$$ p(x) = \\frac{e^{\\beta_0 + \\beta_1 x}}{1 + e^{\\beta_0 + \\beta_1 x}} = \\frac{1}{1 + e^{-(\\beta_0 + \\beta_1 x)}} $$\n",
    "\n",
    "* p(x) is our hypothesis and it represents the probability of a score of x leading to a win. \n",
    "\n",
    "\\begin{align*}\n",
    "& P(Y = 1 | X) = p(X) \\\\\n",
    "& P(Y = 0 | X) = 1 - p(X)\\\\\n",
    "\\end{align*}\n",
    "\n",
    "* $\\beta_0$ and $\\beta_1$ are parameters that we will optimize to best fit our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# inital imports \n",
    "\n",
    "# pandats is like a more powerful version of excel, we're using it to read a .csv file and manipulate tables\n",
    "import pandas as pd\n",
    "# a very popular graph plotting library \n",
    "import matplotlib.pyplot as plt\n",
    "# makes matplotlib look a bit better \n",
    "import seaborn\n",
    "# this makes the plots display in the notebook (here) rather than open as a file \n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>admit</th>\n",
       "      <th>gre</th>\n",
       "      <th>gpa</th>\n",
       "      <th>rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>380</td>\n",
       "      <td>3.61</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>660</td>\n",
       "      <td>3.67</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>800</td>\n",
       "      <td>4.00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>640</td>\n",
       "      <td>3.19</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>520</td>\n",
       "      <td>2.93</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   admit  gre   gpa  rank\n",
       "0      0  380  3.61     3\n",
       "1      1  660  3.67     3\n",
       "2      1  800  4.00     1\n",
       "3      1  640  3.19     4\n",
       "4      0  520  2.93     4"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load the dataset into pandas: data/grad.csv\n",
    "df = pd.read_csv('data/grad.csv')\n",
    "\n",
    "# view the first 5 rows \n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>admit</th>\n",
       "      <th>gre</th>\n",
       "      <th>gpa</th>\n",
       "      <th>rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>400.000000</td>\n",
       "      <td>400.000000</td>\n",
       "      <td>400.000000</td>\n",
       "      <td>400.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.317500</td>\n",
       "      <td>587.700000</td>\n",
       "      <td>3.389900</td>\n",
       "      <td>2.48500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.466087</td>\n",
       "      <td>115.516536</td>\n",
       "      <td>0.380567</td>\n",
       "      <td>0.94446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>220.000000</td>\n",
       "      <td>2.260000</td>\n",
       "      <td>1.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>520.000000</td>\n",
       "      <td>3.130000</td>\n",
       "      <td>2.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>580.000000</td>\n",
       "      <td>3.395000</td>\n",
       "      <td>2.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>660.000000</td>\n",
       "      <td>3.670000</td>\n",
       "      <td>3.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>800.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.00000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            admit         gre         gpa       rank\n",
       "count  400.000000  400.000000  400.000000  400.00000\n",
       "mean     0.317500  587.700000    3.389900    2.48500\n",
       "std      0.466087  115.516536    0.380567    0.94446\n",
       "min      0.000000  220.000000    2.260000    1.00000\n",
       "25%      0.000000  520.000000    3.130000    2.00000\n",
       "50%      0.000000  580.000000    3.395000    2.00000\n",
       "75%      1.000000  660.000000    3.670000    3.00000\n",
       "max      1.000000  800.000000    4.000000    4.00000"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use the pandas describe method to get some preliminary summary statistics on the data. \n",
    "# In particular look at the mean values of the features.\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAHk5JREFUeJzt3X+UXGWd5/H3ZwggBoYAcdqYRBrXHOcwRvmRE+JhXHtAd2Jgic4wHNwICRsn6whHHLNHg7srMo5ncXbwB8ysGgETJfJTNBlE2Yjp8TirEQJIQiJLwDBJDAkqBIIO2vDdP+7TsahUdVdXV9176+bzOqdO319V9a2nn/utp5773HsVEZiZWXX9XtEBmJlZdznRm5lVnBO9mVnFOdGbmVWcE72ZWcU50ZuZVZwTfclI6pcUkia0+fx9kl7T6bjMrHe1lUysvCLiyOFpSSuAHRHx34uLyMyK5ha9mVnFOdHnRNIySY9KelbSZknvTMsPkfT3kn4u6THgrLrnDUr6W0n/N3XL/JOk4yStkvSMpHsk9ddsH5JeK2kJsAD40PDzcvy4Zk1JOkXS/WlfuFXSzamOD0jaIekjaX/YJmlBzfPOSs97RtJ2SR8r8GP0FCf6/DwKvBk4GrgCuEHSFOAvgbOBk4FZwLkNnns+cAEwFfh3wA+ALwHHAluAy+ufEBHLgVXA30XEkRHxHzv9gczGStJhwNeBFWT190bgnTWbvBKYTFbXFwLLJb0urXsOuBCYRNYg+itJ78gn8t7mRJ+TiLg1In4WES9GxM3AI8Bs4DzgMxGxPSJ+CfzPBk//UkQ8GhF7gW8Bj0bEdyJiCLiV7EvCrBfMITs2eHVE/DYibgd+VLfN/4iI5yPin4Fvku0jRMRgRGxM+9CDZF8Sb8kz+F7lRJ8TSRdKekDS05KeBl5P1nJ5FbC9ZtPHGzx9d830rxvMH4lZb3gVsDNeejXF2vr/VEQ8VzP/eHoOkk6TtE7Sk5L2Au8l24dsFE70OZB0PPBF4BLguIiYBGwCBOwCptds/uoOvrUvTWplswuYKkk1y2rr/zGSJtbMvxr4WZr+KrAGmB4RRwOfJ9uHbBRO9PmYSJZ0nwSQdBFZix7gFuD9kqZJOgZY1sH33Q14TL2VyQ+AF4BLJE2QNJ+sC7PWFZIOk/RmsuNXt6blRwG/jIh/kzQb+E+5Rd3jnOhzEBGbgavIKvluYCbwL2n1F4G7gB8D9wG3d/CtrwNOTN1F3+jg65q1JSJ+A/wZsBh4Gng3cAfwfNrkCeApslb8KuC9EfGTtO59wN9Iehb4KFkjyVog33jEzIokaT1ZN8xPgRsiYlrBIVWOW/RmlitJb5H0ytR1sxB4A/DtouOqMl8Cwczy9jqybpeJwGPAuRGxq2a8vHWYu27MzCrOXTdmZhVXiq6byZMnR39/f8N1zz33HBMnTmy47mDicsiMVA4bNmz4eUS8IueQ2jJSne+UstYZxzU2HanzEVH449RTT41m1q1b13TdwcTlkBmpHIB7owT1uZXHSHW+U8paZxzX2HSizrvrxsys4pzozcwqzonezKziSnEwdiQbd+5l0bJvjrjNtivPGnG9mVkZ9Y+S2wBWzB3/AWK36M3MKq70LXozs1a10kI+GHsA3KI3M6s4J3ozs4pzojczqzgnejOzinOiNzOrOCd6M7OKc6I3M6s4j6M3s4NKs7H2S2cO7T8Lv2pj7d2iNzOrOCd6M7OKc6I3M6s4J3ozs4pzojczqzgnejOzivPwSjOzOlW73LFb9GZmFedEbzZGkg6RdL+kO9L8CZLWS9oq6WZJhxUdo1ktJ3qzsbsU2FIz/0ng0xHxWuApYHEhUZk1MWqilzRd0jpJmyU9JOnStPxYSWslPZL+HpOWS9LVqXXzoKRTuv0hzPIiaRpwFnBtmhdwBnBb2mQl8I5iojNrrJWDsUPA0oi4T9JRwAZJa4FFwN0RcaWkZcAy4MPA24EZ6XEa8Ln016wKPgN8CDgqzR8HPB0RQ2l+BzC10RMlLQGWAPT19TE4ONjVQPft29f192hHN+NaOnNo9I2a6DtibM/vxGdo5f06UV6jJvqI2AXsStPPStpCVpHnAwNps5XAIFminw98OSIC+KGkSZKmpNcx61mSzgb2RMQGSQNjfX5ELAeWA8yaNSsGBsb8EmMyODhIt9+jHd2Ma1ELo2WaWTpziKs2tj4QcduCgbbfa1gr8a6YO3Hc5TWm4ZWS+oGTgfVAX03yfgLoS9NTge01Txtu4bwk0bfaumnlW7aMrZZOK2vrLG8Fl8PpwDmS5gEvA34f+CwwSdKE1KqfBuwsKkCzRlpO9JKOBL4GfCAinsm6JjMREZJiLG/cauvmmlWrR/2W7cQ3a9mVtXWWtyLLISIuAy4DSC36/xoRCyTdCpwL3AQsBFYXEqBZEy2NupF0KFmSXxURt6fFuyVNSeunAHvS8p3A9Jqnu4VjVfdh4IOStpL12V9XcDxmL9HKqBuRVdwtEfGpmlVryFov8NJWzBrgwjT6Zg6w1/3zVjURMRgRZ6fpxyJidkS8NiL+IiKeLzo+s1qtdN2cDlwAbJT0QFr2EeBK4BZJi4HHgfPSujuBecBW4FfARR2N2MzMxqSVUTffB9Rk9ZkNtg/g4nHGZWZmHeKLmplZKVTtQmJl4ksgmJlVnBO9mVnFOdGbmVWc++jNrGe00o9vB3KL3sys4pzozcwqzl03ZtZ1tV0uS2cOjesqkzZ2btGbmVWcE72ZWcU50ZuZVZwTvZlZxTnRm5lVnBO9mVnFOdGbmVWcE72ZWcX5hCkzGxdff6b8nOjNzLqkLF+CTvRWGq3sFCvmTswhErNqcaI3M2tDWVrrrfDBWDOzinOiNzOrOCd6M7OKc6I3a5Gk6ZLWSdos6SFJl6blx0paK+mR9PeYomM1qzVqopd0vaQ9kjbVLGtYsZW5WtJWSQ9KOqWbwZvlbAhYGhEnAnOAiyWdCCwD7o6IGcDdad6sNFpp0a8A5tYta1ax3w7MSI8lwOc6E6ZZ8SJiV0Tcl6afBbYAU4H5wMq02UrgHcVEaNbYqMMrI+J7kvrrFs8HBtL0SmAQ+HBa/uWICOCHkiZJmhIRuzoVsFkZpH3iZGA90FdTx58A+po8ZwlZA4i+vj4GBwe7GuO+ffu6/h6Q3RpwLPqOGPtz8lDWuDrxf2x3HH2zij0V2F6z3Y607IBE32qlb6Xw86jMRctrpy1SKztZGcpB0pHA14APRMQzkvavi4iQFI2eFxHLgeUAs2bNioGBga7GOTg4SLffAxjz/V+Xzhziqo3lO4WnrHGtmDtx3P/HcX+qkSr2KM9rqdJfs2r1qIW/bUHj51ZJXjttkVpJGJ2o9OMh6VCyJL8qIm5Pi3cP/3KVNAXYU1iAZg20O+pmd6rQ1FXsncD0mu2mpWVmPU9Z0/06YEtEfKpm1RpgYZpeCKzOOzazkbSb6JtV7DXAhWn0zRxgr/vnrUJOBy4AzpD0QHrMA64E3ibpEeCtad6sNEbtupF0I9mB18mSdgCXk1XkWyQtBh4Hzkub3wnMA7YCvwIu6kLMZoWIiO8DarL6zDxjMRuLVkbdvKvJqgMqdhptc/F4gzIzs87xmbFmZhXnRG9mVnFO9GZmFedEb2ZWcU70ZmYV50RvZlZxTvRmZhVXviv4mFkueunm1jY+btGbmVWcE72ZWcU50ZuZVZwTvZlZxTnRm5lVnBO9mVnFOdGbmVWcE72ZWcU50ZuZVZwTvZlZxTnRm5lVnBO9mVnFOdGbmVWcE72ZWcX5MsVmPaiVSwwvnTnEIl+K2HCL3sys8rqS6CXNlfSwpK2SlnXjPczKxvXeyqrjXTeSDgH+EXgbsAO4R9KaiNjc6fcyK4tO1nvf+ck6rRst+tnA1oh4LCJ+A9wEzO/C+5iVieu9lVY3DsZOBbbXzO8ATqvfSNISYEma3Sfp4SavNxn4+UhvqE+2EWXvGbUcDgZ/8skRy+H4PGOpM2q9H0Od74j3l7TOOK6x6USdL2zUTUQsB5aPtp2keyNiVg4hlZrLIdPL5dBqne+UspaV4xqbTsTVja6bncD0mvlpaZlZlbneW2l1I9HfA8yQdIKkw4DzgTVdeB+zMnG9t9LqeNdNRAxJugS4CzgEuD4iHhrHS+b2U7fkXA6ZUpZDF+p9J5SyrHBcYzXuuBQRnQjEzMxKymfGmplVnBO9mVnFlSLRj3bquKTDJd2c1q+X1J9/lPlooSwWSXpS0gPp8Z4i4uwmSddL2iNpU5P1knR1KqMHJZ2Sd4xlIWm6pHWSNkt6SNKlDbYZkLS3ps58NIe4XibpR5J+nOK6osE2ue/XLcZV2D4m6RBJ90u6o8G69ssrIgp9kB24ehR4DXAY8GPgxLpt3gd8Pk2fD9xcdNwFlsUi4B+KjrXL5fDvgVOATU3WzwO+BQiYA6wvOuYCy2oKcEqaPgr4fw3qzABwR85xCTgyTR8KrAfm1G2T+37dYlyF7WPAB4GvNvp/jae8ytCib+XU8fnAyjR9G3CmJOUYY158Gj0QEd8DfjnCJvOBL0fmh8AkSVPyia5cImJXRNyXpp8FtpCdpVuo9L/Zl2YPTY/6kR+579ctxlUISdOAs4Brm2zSdnmVIdE3OnW8vqLu3yYihoC9wHG5RJevVsoC4M9Tl8VtkqY3WF91rZbTQSX9lD+ZrJVa702pu+Jbkv4op3gOkfQAsAdYGxH1cRWyX7cQFxSzj30G+BDwYpP1bZdXGRK9jc0/Af0R8QZgLb/7hreDmKQjga8BH4iIZ+pW3wccHxFvBK4BvpFHTBHxQkScRHaW8GxJr8/jfUfTQly572OSzgb2RMSGbrx+GRJ9K6eO799G0gTgaOAXuUSXr1HLIiJ+ERHPp9lrgVNziq1MfLmBGpIOJUvyqyLi9vr1EfHMcHdFRNwJHCppcl7xRcTTwDpgbt2qtvdrSYPjPUjaLK6C9rHTgXMkbSPrsj1D0g1127RdXmVI9K2cOr4GWJimzwW+G+mIRMWMWhZ1fdHnkPXJHmzWABem0TdzgL0RsavooIqQ+mivA7ZExKeabPPK4b5cSbPJ9vuuNpQkvULSpDR9BNl1+n9St1nu+3UrcRWxj0XEZRExLSL6yfb770bEu+s2a7u8Cr9nbDQ5dVzS3wD3RsQasor8FUlbyQ7SnV9cxN0zWlkAdwLvl3QOMERWFouKirdbJN1INlJksqQdwOVkB82IiM+TlcM8YCvwK+CiYiIthdOBC4CNqd8Z4CPAq2F/eZ0L/JWkIeDXwPk5NJSmACuV3ZDl94BbIuKOurpcxH49Ylwp35RmH+tUHvQlEEogjQO/Dngt8G2ygzGPAN8BbiDrV/1rsgNHF6T+vL8F+oHNwHsj4sECQjdrWeqW+BywAHgd8HGyL+k/IDvI+N8i4utp20XAe4AfAouBp4H3RcS30vpB4IaIuDa1wO8CvhIR/yvHj9QzytB1c1BLXTRfB1YAxwI3Au+s2eSVafnxwBJJJwPXA/+F7Ij7F4A1kg7PMWyzdr2LbAjhJOBh4M1kfc1XADfUdZuclraZDPwdcF39cEJJJwD/TDbu3Um+CSf64s0h60K7OiJ+mw6m/ahm/YvA5RHxfET8muwORV+IiPVp9MBK4Pn0OmZld3VEbI+IX0fErRHxs4h4MSJuJvsVO7tm28cj4osR8QLZyJcpQF/N+hPJDqZeHtlNXawJJ/rivQrYWddnWjtG/MmI+Lea+eOBpZKeHn6QHYl/VQ6xmo3X/rot6cJ0iYHhevx6stb7sCeGJyLiV2nyyJr1C8hGotzWxXgrwYm+eLuAqXU/SWuHDtYfRNkOfCIiJtU8Xh4RN3Y9UrPxy65DIB0PfBG4BDguIiYBm8guUdCqj5HdS/Wr6eCqNeFEX7wfAC8Al0iaIGk+L/35Wu+LwHslnZaGF06UdJako3KJ1qwzJpIl/ScBJF1E1qIfi98Cf5Fe68uSnM+acMEULF3T5s/43ciCdwN3kPW7N9r+XuAvgX8AniIbYrgoj1jNOiUiNgNXkTV0dgMzgX9p43WG958+4Hon+8Y8vLKEJK0nu0rdl4qOxcx6n7/9SkDSW9LZixMkLQTeQDae3sxs3Ao/M9aA7OSRW8j6Gh8Dzj1YT+k3s85z142ZWcW568bMrOJK0XUzefLk6O/v57nnnmPixIlFhzNmvRh3L8YMI8e9YcOGn0fEK3IOqS3Ddb6RXvvfON7u6kidb+V+g91+nHrqqRERsW7duuhFvRh3L8YcMXLcZFf5K7w+t/IYrvNj/Yxl5Hi7qxN13l03ZnUkXS9pj6RNNcs+JmlnOmX/AUnzatZdJmmrpIcl/WkxUZs150RvdqAVHHg3JIBPR8RJ6XEngKQTya4L/kfpOf/bp+Nb2TjRm9WJiO+R3dihFfOBmyK7uuhPyc5UHukSFma5K8XBWOuO/mXfbLpu6cwhFi37JtuuPCvHiHreJZIuJLtD0tKIeAqYSnZzjGE70rIDSFpCdplp+vr6GBwcbPgm+/bta7qujPb8ci/XrFo94jYzpx6dUzSj67Xy7US8TvQ9aqQk3unX8ZcBkN0Z6eNkF+L6ONl1Wv7zWF4gsmumLweYNWtWDAwMNNxucHCQZuvK6JpVq7lq48ipZNuCgXyCaUGvlW8n4nXXjVkLImJ3ZDd6eZHsCqLD3TM7eellpaelZWal4URv1oK6W9y9k+za6QBrgPMlHZ5uazeDl94hzKxw7roxqyPpRmAAmCxpB3A5MCDpJLKum21k9+wlIh6SdAvZTdqHgIsju/WdWWk40ZvViYh3NVh83QjbfwL4RPciMhsfd92YmVWcE72ZWcU50ZuZVZwTvZlZxTnRm5lVnBO9mVnFeXhlCXXq8gZmZuAWvZlZ5blFb6Pyhc/Meptb9GZmFedEb2ZWceNO9JIOkXS/pDvS/AmS1qd7aN4s6bDxh2lmZu3qRB/9pcAW4PfT/CfJ7q15k6TPA4vJbtpgZla4jTv3smiU405VO+Y0rha9pGnAWcC1aV7AGcBtaZOVwDvG8x5mZjY+4+26+QzwIeDFNH8c8HREDKX5pvfPNDOzfLTddSPpbGBPRGyQNNDG8w+4UXKv3bR3WKfjXjpzaPSNxqnviM6+T17/t16tI2ZFGk8f/enAOZLmAS8j66P/LDBJ0oTUqm96/8xGN0rutZv2Dut03KP1H3bC0plDo97QeSzyuvlzr9YRsyK13XUTEZdFxLSI6AfOB74bEQuAdcC5abOFwOpxR2lmZm3rxjj6DwMflLSVrM++6S3YzMys+zry2z0iBoHBNP0YMLsTr2tmZuPnM2PNzCrOid6sAUnXS9ojaVPNsmMlrZX0SPp7TFouSVens8EflHRKcZGbHciJ3qyxFcDcumXLgLsjYgZwd5oHeDswIz2W4DPBrWSc6M0aiIjvAb+sWzyf7GxveOlZ3/OBL0fmh2RDjKfkE6nZ6Hw9erPW9UXErjT9BNCXpqcC22u2Gz4jfFfNsoYnCTbSayeFtXLyXZk+T6/F24n64ERv1oaICEkxxucccJJgI712Utg1q1aPevJdXifUtaLX4u1EfXDXjVnrdg93yaS/e9LyncD0mu2anhFuVgS36K0jRrvdYEUu+7qG7GzvK3npWd9rgEsk3QScBuyt6eIxK5wTvVkDkm4EBoDJknYAl5Ml+FskLQYeB85Lm98JzAO2Ar8CLso9YLMRONGbNRAR72qy6swG2wZwcXcjMmuf++jNzCrOid7MrOLcdVOA0Q5cHqxaKZcVcyfmEIlZtbhFb2ZWcU70ZmYV50RvZlZxTvRmZhXnRG9mVnFO9GZmFedEb2ZWcU70ZmYV50RvZlZxTvRmZhXnRG9mVnG+1o2ZWUHyur6TW/RmZhXXdqKXNF3SOkmbJT0k6dK0/FhJayU9kv4e07lwzcxsrMbTdTMELI2I+yQdBWyQtBZYBNwdEVdKWgYsAz48/lDNDg4bd+5l0cFxD17LSdst+ojYFRH3pelngS3AVGA+sDJtthJ4x3iDNDOz9nXkYKykfuBkYD3QFxG70qongL4mz1kCLAHo6+tjcHCQffv2MTg42ImQcjXWuJfOHOpeMC3qOyLfOFopn1bi6dU6YlakcSd6SUcCXwM+EBHPSNq/LiJCUjR6XkQsB5YDzJo1KwYGBhgcHGRgYGC8IeVurHGP9rM8D0tnDnHVxvwGXW1bMDDqNq2Uy4q5E3uyjpgVaVx7uqRDyZL8qoi4PS3eLWlKROySNAXYM94gy6LZUKilM4f2Jyn3nVabpG3As8ALwFBEzJJ0LHAz0A9sA86LiKeKitGs3nhG3Qi4DtgSEZ+qWbUGWJimFwKr2w/PrJT+JCJOiohZaX4Z2QCEGcDdad6sNMYzjv504ALgDEkPpMc84ErgbZIeAd6a5s2qzAMQrNTa7rqJiO8DarL6zHZft9e1cqab9bQA/k869vSFdKyp7QEIjbRyoLxMB6Qdb/vyGoDgSyCYjc0fR8ROSX8ArJX0k9qVYx2A0Mg1q1aPeqC8lYPbeXG87ctrAIITveWiKr90ImJn+rtH0teB2VR4AIJVg691Y9YiSRPTWeBImgj8B2ATHoBgJecWvVnr+oCvp3NFJgBfjYhvS7oHuEXSYuBx4LwCYzQ7gBO9WYsi4jHgjQ2W/4KDeACClZ+7bszMKs6J3sys4pzozcwqzonezKzinOjNzCrOid7MrOKc6M3MKs6J3sys4pzozcwqzonezKzifAmEpCpXVzQzq+cWvZlZxTnRm5lVnBO9mVnFOdGbmVWcE72ZWcU50ZuZVZwTvZlZxTnRm5lV3EFxwpRPhjKzg5lb9GZmFdeVRC9prqSHJW2VtKwb72FWNq73VlYd77qRdAjwj8DbgB3APZLWRMTmdl6vlW6XbVee1c5Lm3VMp+u9WSd1o0U/G9gaEY9FxG+Am4D5XXgfszJxvbfSUkR09gWlc4G5EfGeNH8BcFpEXFK33RJgSZp9HfAwMBn4eUcDykcvxt2LMcPIcR8fEa/IM5hhrdT7JnW+kV773zje7hp3nS9s1E1ELAeW1y6TdG9EzCoopLb1Yty9GDP0btzQuM430muf0fF2Vyfi7UbXzU5ges38tLTMrMpc7620upHo7wFmSDpB0mHA+cCaLryPWZm43ltpdbzrJiKGJF0C3AUcAlwfEQ+1+PRRf9aWVC/G3YsxQ0njHme9r1fKzzgCx9td44634wdjzcysXHxmrJlZxTnRm5lVXG6JXtJ0SeskbZb0kKRL0/JjJa2V9Ej6e0xaLklXp9PJH5R0Sl6xNon/EEn3S7ojzZ8gaX2K7+Z0AA5Jh6f5rWl9f4ExT5J0m6SfSNoi6U1lL29Jf53qxyZJN0p6WS+UdSdIul7SHkmbio6lFc326bJKdelHkn6c4r2i6JhaUZ972pFni34IWBoRJwJzgIslnQgsA+6OiBnA3Wke4O3AjPRYAnwux1gbuRTYUjP/SeDTEfFa4ClgcVq+GHgqLf902q4onwW+HRF/CLyRLP7SlrekqcD7gVkR8Xqyg5rn0xtl3QkrgLlFBzEGzfbpsnoeOCMi3gicBMyVNKfgmFpRn3vGLiIKeQCrya4L8jAwJS2bAjycpr8AvKtm+/3bFRDrNLKkeAZwByCyM9UmpPVvAu5K03cBb0rTE9J2KiDmo4Gf1r93mcsbmApsB45NZXcH8KdlL+sOl0E/sKnoONqMfTXwtqLjaDHWlwP3kZ29XHg8I8T5ktzT7usU0keffmKfDKwH+iJiV1r1BNCXpod3+mE70rIifAb4EPBimj8OeDoihtJ8bWz7407r96bt83YC8CTwpfSz71pJEylxeUfETuDvgX8FdpGV3QbKX9YHvbp9urRSN8gDwB5gbUSUOl4OzD1tyT3RSzoS+BrwgYh4pnZdZF9hpRrvKelsYE9EbCg6ljGaAJwCfC4iTgae43fdNED5yjsdL5hP9iX1KmAivdWVcVAaaZ8um4h4ISJOImspz5b0+qJjaqaTuSfXRC/pULIKsSoibk+Ld0uaktZPIfumhfKcUn46cI6kbWRXJDyDrO97kqThE85qY9sfd1p/NPCLPANOdgA7alost5El/jKX91uBn0bEkxHxW+B2svIve1kftJrs06UXEU8D6yh3Q+KA3CPphnZeKM9RNwKuA7ZExKdqVq0BFqbphWT9fMPLL0yjQeYAe2u6HHITEZdFxLSI6Cc7MPjdiFhAVknObRL38Oc5N22fe6s5Ip4Atkt6XVp0JrCZcpf3vwJzJL081ZfhmEtd1gerEfbpUpL0CkmT0vQRZMcIf1JsVM01yT3vbvfF8jqo8Mdk3QQPAg+kxzyyPtW7gUeA7wDHpu1FdiOHR4GNZCMxij4wMkA6IAK8BvgRsBW4FTg8LX9Zmt+a1r+mwHhPAu5NZf4N4JiylzdwBdnOtwn4CnB4L5R1hz77jWTHJn5L9otscdExjRJvw3266LhGiPcNwP0p3k3AR4uOaQyx78897Tx8CQQzs4rzmbFmZhXnRG9mVnFO9GZmFedEb2ZWcU70ZmYV50RvZlZxTvRmZhX3/wHseCoPi3XOfAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x114cf8208>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# look at the distribution of values \n",
    "pd.DataFrame.hist(df, bins=15);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.                 nan -0.35810345  0.12649425  0.17166667]\n",
      " [ 1.                 nan  0.12465517  0.16097701  0.17166667]\n",
      " [ 1.                 nan  0.36603448  0.35063218 -0.495     ]\n",
      " ...\n",
      " [ 1.                 nan -0.22017241 -0.43672414 -0.16166667]\n",
      " [ 1.                 nan  0.19362069  0.14948276 -0.16166667]\n",
      " [ 1.                 nan  0.0212069   0.28741379  0.17166667]]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/lifesaver/miniconda3/lib/python3.6/site-packages/ipykernel_launcher.py:11: RuntimeWarning: invalid value encountered in true_divide\n",
      "  # This is added back by InteractiveShellApp.init_path()\n"
     ]
    }
   ],
   "source": [
    "# data transformation  \n",
    "\n",
    "import numpy as np\n",
    "\n",
    "X = np.asarray(df[['gre','gpa','rank']])\n",
    "X = np.insert(X, 0, 1, axis=1)             #  11111111111111!\n",
    "y = np.asarray(df['admit'])\n",
    "\n",
    "# data preprocessing - normalization \n",
    "\n",
    "X_norm = (X - X.mean(axis=0)) / (X.max(axis=0) - X.min(axis=0)) # normed by col\n",
    "X_norm = np.insert(X_norm, 0, 1, axis=1)                        # col of 1s                         \n",
    "y = np.asarray(df['admit'])                                     # true values\n",
    "print(X_norm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(268, 4) (132, 4) (268,) (132,)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# split data into train and test sets\n",
    "seed = 42\n",
    "test_size = 0.33\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size, random_state=seed)\n",
    "print (X_train.shape, X_test.shape, y_train.shape, y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.75"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# instantiate a logistic regression model, and fit with X and y\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "model = LogisticRegression()\n",
    "model = model.fit(X_train, y_train)\n",
    "\n",
    "y_pred = model.predict(X_test)\n",
    "\n",
    "# check the accuracy on the training set\n",
    "model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### But wait, what exactly does that number mean? \n",
    "\n",
    "\n",
    "### Accuracy\n",
    "The simplest measure is **accuracy**. This is the number of correct predictions over the total number of predictions. It's the percent you predicted correctly. In `sklearn`, this is what the `score` method calculates.\n",
    "\n",
    "### Shortcomings of Accuracy\n",
    "Accuracy is often a good first glance measure, but it has many shortcomings. If the classes are unbalanced, accuracy will not measure how well you did at predicting. Say you are trying to predict whether or not an email is spam. Only 2% of emails are in fact spam emails. You could get 98% accuracy by always predicting not spam. This is a great accuracy but a horrible model!\n",
    "\n",
    "### Confusion Matrix\n",
    "We can get a better picture our model but looking at the confusion matrix. We get the following four metrics:\n",
    "\n",
    "* **True Positives (TP)**: Correct positive predictions\n",
    "* **False Positives (FP)**: Incorrect positive predictions (false alarm)\n",
    "* **True Negatives (TN)**: Correct negative predictions\n",
    "* **False Negatives (FN)**: Incorrect negative predictions (a miss)\n",
    "\n",
    "|            | Predicted Yes  | Predicted No   |\n",
    "| ---------- | -------------- | -------------- |\n",
    "| Actual Yes | True positive  | False negative |\n",
    "| Actual No  | False positive | True negative  |\n",
    "\n",
    "With logistic regression, we can visualize it as follows:\n",
    "\n",
    "![logistic confusion matrix](images/logistic.png)\n",
    "\n",
    "\n",
    "## 2) Metrics "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "Not Admitted       0.73      1.00      0.85        90\n",
      "    Admitted       1.00      0.21      0.35        42\n",
      "\n",
      " avg / total       0.82      0.75      0.69       132\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "target_names = ['Not Admitted', 'Admitted']\n",
    "\n",
    "print(classification_report(y_test, y_pred, target_names=target_names))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "71.5% accuracy only tells part of the story.  \n",
    "\n",
    "### Precision, Recall and F1\n",
    "\n",
    "![](http://i.stack.imgur.com/ysM0Z.png)\n",
    "Instead of accuracy, there are some other scores we can calculate:\n",
    "\n",
    "* **Precision**: A measure of how good your positive predictions are\n",
    "    ```\n",
    "    Precison = TP / (TP + FP)\n",
    "             = TP / (predicted yes)\n",
    "    ```\n",
    "* **Recall**: A measure of how well you predict positive cases. Aka *sensitivity*.\n",
    "    ```\n",
    "    Recall = TP / (TP + FN) \n",
    "           = TP / (actual yes)\n",
    "    ```\n",
    "* **F1 Score**: The harmonic mean of Precision and Recall\n",
    "    ```\n",
    "    F1 = 2 / (1/Precision + 1/Recall)\n",
    "       = 2 * Precision * Recall / (Precision + Recall)\n",
    "       = 2TP / (2TP + FN + FP)\n",
    "    ```\n",
    "\n",
    "Accuracy can also be written in this notation:\n",
    "```\n",
    "Accuracy = (TP + TN) / (TP + FP + TN + FN)\n",
    "```\n",
    "\n",
    "![](http://upload.wikimedia.org/wikipedia/commons/2/26/Precisionrecall.svg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3) A More Advanced Classifier \n",
    "\n",
    "## Random Forests\n",
    "\n",
    "Probably the most common ensemble method is a *Random Forest*, which consists of a collection of Decision Trees.\n",
    "\n",
    "They were developed by Leo Breimen, who has the most extensive notes about them on his [webpage](http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm).\n",
    "\n",
    "The idea is to repeatedly randomly select data from the dataset (*with replacement*) and build a Decision Tree with each new sample. The default is to have the randomly selected data be the same size as the initial dataset. Note that since we are sampling with replacement, many data points will be repeated in the sample and many won't be included.\n",
    "\n",
    "Random Forests also limit each node of the Decision Tree to only consider splitting on a random subset of the features.\n",
    "\n",
    "Here is the pseudocode for creating a Random Forest:\n",
    "\n",
    "    CreateRandomForest(data, num_trees, num_features):\n",
    "        Repeat num_trees times:\n",
    "            Create a random sample of the test data with replacement\n",
    "            Build a decision tree with that sample (only consider num_features features at each node)\n",
    "        Return the list of the decision trees created\n",
    "\n",
    "To classify a new document, use each tree to get a prediction. Choose the label that gets the most votes.\n",
    "\n",
    "The default parameters that sklearn uses, which are also standard defaults, are 10 trees and only considering sqrt(m) features (where m is the total number of features).\n",
    "\n",
    "\n",
    "### Out of Bag Error\n",
    "\n",
    "We can analyze a Random Forest using the standard cross validation method of splitting the dataset into a training set and a testing set. However, if we're clever, we notice that each tree doesn't see all of the training data, so we can use the skipped data to cross validate each tree individually.\n",
    "\n",
    "We'll skip the mathematical proof, but when selecting from the dataset, about one third of the data is left out (discussed [here](http://math.stackexchange.com/questions/203491/expected-coverage-after-sampling-with-replacement-k-times) if you want to think about the math). So every data point can be tested with about 1/3 of the trees. We calculate the percent of these that we get correct, and this is the *out-of-bag error*.\n",
    "\n",
    "It has been proven that this is sufficient and that cross validation is not strictly necessary for a random forest, but we often still use it as that makes it easier to compare with other models.\n",
    "\n",
    "\n",
    "### Feature Importance\n",
    "\n",
    "We can use the random forest to determine which features are the most importance in predicting the class.\n",
    "\n",
    "Breiman, the originator of random forests, uses out-of-bag error to determine feature importance, discussed [here](http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#varimp). The idea is to compare the out-of-bag error of the trees with the out-of-bag error of the trees if you change the feature's value (basically, if we screw with the value of the feature, how much does that impact the total error?). Here is the pseudocode for calculating the feature importance for a single feature:\n",
    "\n",
    "        For every tree:\n",
    "            Take the data that is not covered by the tree.\n",
    "            Randomly permute the values of the feature (i.e. keep the same values,\n",
    "                but shuffle them around the data points).\n",
    "            Calculate the OOB error on the data with the feature values permuted.\n",
    "            Subtract the permutated OOB from the OOB of the original data to get the\n",
    "                feature importance on this tree.\n",
    "        Average all the individual feature importances to get the feature importance.\n",
    "\n",
    "sklearn uses a different method, described [here](http://scikit-learn.org/stable/modules/ensemble.html#feature-importance-evaluation). Their method doesn't involve using the out-of-bag score. Basically, the higher in the tree the feature is, the more important it is in determining the result of a data point. The expected fraction of data points that reach a node is used as an estimate of that feature's importance for that tree. Then average those values across all trees to get the feature's importance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Boosting in General\n",
    "\n",
    "You will often hear the word \"Bagging\" dropped around the same time as \"Boosting.\" The two are very different techniques and not really used in the same context. You already know what bagging is. \n",
    "\n",
    "Boosting, properly stated, the use of the output of one model as the input of another. Thus we can use Pipelines to construct whole groups of models that are technically \"boosted\" by each other. Any and all models can be boosted, according to the theory. \n",
    "\n",
    "The word \"boosting\" is most commonly used to describe boosting used with trees, and most particularly Gradient Boosted Regression Trees (GBRTs). You can think of it as \"combining weak learners\" to make a \"strong learner\". This is mumbo-jumbo. The \"weak\" learners are just normal decision trees (as opposed to an estimator which is especially \"weak\" in some way). \n",
    "\n",
    "## Boosting in Trees\n",
    "\n",
    "There are two types of boosting used in trees in common practice today. This is the most common jargon usage of this word in data science. If you hear fellow data scientists say \"boosting\", they are often referring to boosted trees.\n",
    "\n",
    "### Adding bias into the training process\n",
    "\n",
    " In the boosted tree algorithms, a \"family\" of trees is \"grown\" over successive generations of trees, by making each successor tree an expert on attacking the weaknesses of the other. What results is a single \"master\" tree designed to defeat all weaknesses of the training set selection. Members of the family are weighted by the ratio of errors not covered by the previous members. \n",
    "\n",
    "The process of boosting (in any case) is based on measuring the degree to which a given tree fails in making a good prediction. This, of course, is the cost or more commonly in the case of trees \"loss\" function.\n",
    "\n",
    "### Some notes about the boosting community\n",
    "1. They try to make algorithms sound like products (\"LogitBoost\",\"AnyBoost\", etc.)\n",
    "2. \"loss function\" = cost function\n",
    "3. \"weak learners\" = normal estimators\n",
    "4. \"model is complex/review of the theory\" = we don't understand it very well/it's all Black Magic to us"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One method of boosting is called AdaBoost it looks like:\n",
    "\n",
    "![adaboost_part](./images/AdaBoost_part.png)\n",
    "\n",
    "Another is Gradient booting:\n",
    "![gradient_boosting_scheme](./images/GradientBoosting_scheme.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why does any of this work?\n",
    "\n",
    "Most of the machine learning community seems baffled by the success of this approach, and in general, GB tends to outperform even very sophisticated methods, including neural networks. \n",
    "\n",
    "Why? The intuition is that the algorithm ends up partitioning on the noise rather than the signal. What I think happens is that signal gets partitioned into a few relatively powerful trees (perhaps 1/5th to 1/3 of them), and most of the trees are set up to shunt out the noise. I suspect that the weights (lagrange multipliers) provide a degree of freedom so that probably reflects the dominance of that particular noise, not unlike an eigenvector/eigenvalue relationship.\n",
    "\n",
    "Trying to get a clear statement like that out of famous people like Hastie or Tibishirani seems impossible. Maybe because they're afraid they might be wrong."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
       "       colsample_bytree=1, gamma=0, learning_rate=0.2, max_delta_step=0,\n",
       "       max_depth=7, min_child_weight=1, missing=None, n_estimators=1000,\n",
       "       n_jobs=1, nthread=-1, objective='binary:logistic', random_state=0,\n",
       "       reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=0, silent=True,\n",
       "       subsample=1)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from xgboost import XGBClassifier\n",
    "\n",
    "XGmodel = XGBClassifier(max_depth=7, learning_rate=0.2, \n",
    "                        n_estimators=1000, silent=True, \n",
    "                        objective='binary:logistic', nthread=-1, \n",
    "                        gamma=0, min_child_weight=1, max_delta_step=0, \n",
    "                        subsample=1, colsample_bytree=1, \n",
    "                        colsample_bylevel=1, reg_alpha=0, \n",
    "                        reg_lambda=1, scale_pos_weight=1, \n",
    "                        base_score=0.5, seed=0, missing=None)\n",
    "\n",
    "XGmodel.fit(X_train, y_train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "Not Admitted       0.72      0.81      0.76        90\n",
      "    Admitted       0.45      0.33      0.38        42\n",
      "\n",
      " avg / total       0.64      0.66      0.64       132\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/lifesaver/miniconda3/lib/python3.6/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n"
     ]
    }
   ],
   "source": [
    "y_pred = XGmodel.predict(X_test)\n",
    "\n",
    "target_names = ['Not Admitted', 'Admitted']\n",
    "\n",
    "print(classification_report(y_test, y_pred, target_names=target_names))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note, boosting performed worse than Logistic Regression here. Why do you think that is?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4) Embedded Text \n",
    "\n",
    "Remember those familiar column names?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['admit', 'gre', 'gpa', 'rank']\n"
     ]
    }
   ],
   "source": [
    "print(list(df.columns))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if we didn't know the test scores or the rank of the student in the class?  \n",
    "\n",
    "What if all we had were the student's essays?  \n",
    "\n",
    "In that case, our problem is a little harder. First, we need to decide what our colums are going to be. What is a feature of an essay? What aspects or parts of the essays do we care about? Once we know that we have to read each one and enter values for each student for each column, according to what they wrote.\n",
    "\n",
    "We could do that ourselves, but it would take a very long time.  \n",
    "\n",
    "In machine learning there is a concept called an *embedding*. Embeddings are where we choose how many columns and the computer/algorithm/AI chooses what aspect goes in each one, then scores the essays for us in each column. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>41</th>\n",
       "      <th>42</th>\n",
       "      <th>43</th>\n",
       "      <th>44</th>\n",
       "      <th>45</th>\n",
       "      <th>46</th>\n",
       "      <th>47</th>\n",
       "      <th>48</th>\n",
       "      <th>49</th>\n",
       "      <th>50</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>19995</th>\n",
       "      <td>-5.230067</td>\n",
       "      <td>-3.597191</td>\n",
       "      <td>-3.962436</td>\n",
       "      <td>4.357621</td>\n",
       "      <td>-8.503902</td>\n",
       "      <td>2.594657</td>\n",
       "      <td>1.948463</td>\n",
       "      <td>-4.259420</td>\n",
       "      <td>-7.848333</td>\n",
       "      <td>-0.198398</td>\n",
       "      <td>...</td>\n",
       "      <td>2.696597</td>\n",
       "      <td>2.526998</td>\n",
       "      <td>2.212848</td>\n",
       "      <td>6.105873</td>\n",
       "      <td>5.207789</td>\n",
       "      <td>-1.095498</td>\n",
       "      <td>-8.406791</td>\n",
       "      <td>1.995144</td>\n",
       "      <td>2.922570</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19996</th>\n",
       "      <td>8.655161</td>\n",
       "      <td>3.216463</td>\n",
       "      <td>3.180928</td>\n",
       "      <td>3.735920</td>\n",
       "      <td>1.518733</td>\n",
       "      <td>6.659177</td>\n",
       "      <td>3.347452</td>\n",
       "      <td>-0.577658</td>\n",
       "      <td>-7.307533</td>\n",
       "      <td>-8.166327</td>\n",
       "      <td>...</td>\n",
       "      <td>1.056458</td>\n",
       "      <td>4.289464</td>\n",
       "      <td>11.064774</td>\n",
       "      <td>3.595927</td>\n",
       "      <td>0.350768</td>\n",
       "      <td>5.447138</td>\n",
       "      <td>-1.876241</td>\n",
       "      <td>8.191900</td>\n",
       "      <td>-4.402915</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19997</th>\n",
       "      <td>5.787154</td>\n",
       "      <td>3.571592</td>\n",
       "      <td>1.132058</td>\n",
       "      <td>-0.635032</td>\n",
       "      <td>2.682277</td>\n",
       "      <td>8.982737</td>\n",
       "      <td>-0.363522</td>\n",
       "      <td>-0.991837</td>\n",
       "      <td>-1.214867</td>\n",
       "      <td>-0.102986</td>\n",
       "      <td>...</td>\n",
       "      <td>4.381866</td>\n",
       "      <td>0.025675</td>\n",
       "      <td>1.275467</td>\n",
       "      <td>0.232545</td>\n",
       "      <td>4.025445</td>\n",
       "      <td>5.082364</td>\n",
       "      <td>1.341273</td>\n",
       "      <td>2.689050</td>\n",
       "      <td>-0.143825</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19998</th>\n",
       "      <td>-2.560885</td>\n",
       "      <td>-7.999623</td>\n",
       "      <td>-8.849792</td>\n",
       "      <td>3.775523</td>\n",
       "      <td>-6.862742</td>\n",
       "      <td>-2.963762</td>\n",
       "      <td>7.276275</td>\n",
       "      <td>2.637631</td>\n",
       "      <td>-0.669727</td>\n",
       "      <td>3.637837</td>\n",
       "      <td>...</td>\n",
       "      <td>11.104369</td>\n",
       "      <td>9.130438</td>\n",
       "      <td>2.408080</td>\n",
       "      <td>-0.311689</td>\n",
       "      <td>0.189652</td>\n",
       "      <td>-1.000840</td>\n",
       "      <td>-4.385654</td>\n",
       "      <td>-1.742498</td>\n",
       "      <td>-1.524839</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19999</th>\n",
       "      <td>-0.025104</td>\n",
       "      <td>-0.688953</td>\n",
       "      <td>-3.751254</td>\n",
       "      <td>1.800386</td>\n",
       "      <td>0.176604</td>\n",
       "      <td>2.892242</td>\n",
       "      <td>-2.041802</td>\n",
       "      <td>-3.385113</td>\n",
       "      <td>0.674931</td>\n",
       "      <td>-3.920923</td>\n",
       "      <td>...</td>\n",
       "      <td>6.433015</td>\n",
       "      <td>0.781282</td>\n",
       "      <td>-2.134592</td>\n",
       "      <td>0.244282</td>\n",
       "      <td>5.747131</td>\n",
       "      <td>1.039919</td>\n",
       "      <td>-7.127241</td>\n",
       "      <td>-3.419089</td>\n",
       "      <td>2.414535</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 51 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             0         1         2         3         4         5         6   \\\n",
       "19995 -5.230067 -3.597191 -3.962436  4.357621 -8.503902  2.594657  1.948463   \n",
       "19996  8.655161  3.216463  3.180928  3.735920  1.518733  6.659177  3.347452   \n",
       "19997  5.787154  3.571592  1.132058 -0.635032  2.682277  8.982737 -0.363522   \n",
       "19998 -2.560885 -7.999623 -8.849792  3.775523 -6.862742 -2.963762  7.276275   \n",
       "19999 -0.025104 -0.688953 -3.751254  1.800386  0.176604  2.892242 -2.041802   \n",
       "\n",
       "             7         8         9  ...          41        42         43  \\\n",
       "19995 -4.259420 -7.848333 -0.198398 ...    2.696597  2.526998   2.212848   \n",
       "19996 -0.577658 -7.307533 -8.166327 ...    1.056458  4.289464  11.064774   \n",
       "19997 -0.991837 -1.214867 -0.102986 ...    4.381866  0.025675   1.275467   \n",
       "19998  2.637631 -0.669727  3.637837 ...   11.104369  9.130438   2.408080   \n",
       "19999 -3.385113  0.674931 -3.920923 ...    6.433015  0.781282  -2.134592   \n",
       "\n",
       "             44        45        46        47        48        49   50  \n",
       "19995  6.105873  5.207789 -1.095498 -8.406791  1.995144  2.922570  1.0  \n",
       "19996  3.595927  0.350768  5.447138 -1.876241  8.191900 -4.402915  1.0  \n",
       "19997  0.232545  4.025445  5.082364  1.341273  2.689050 -0.143825  1.0  \n",
       "19998 -0.311689  0.189652 -1.000840 -4.385654 -1.742498 -1.524839  1.0  \n",
       "19999  0.244282  5.747131  1.039919 -7.127241 -3.419089  2.414535  1.0  \n",
       "\n",
       "[5 rows x 51 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('data/onion_reuters_embeddings.csv', sep='\\s+', header=None)\n",
    "\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "51"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "50"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>title</th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "      <th>title_vectors</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6330</th>\n",
       "      <td>4490</td>\n",
       "      <td>State Department says it can't find emails fro...</td>\n",
       "      <td>The State Department told the Republican Natio...</td>\n",
       "      <td>REAL</td>\n",
       "      <td>[ 0.04833125  0.01000964  0.09758892  0.089780...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6331</th>\n",
       "      <td>8062</td>\n",
       "      <td>The ‘P’ in PBS Should Stand for ‘Plutocratic’ ...</td>\n",
       "      <td>The ‘P’ in PBS Should Stand for ‘Plutocratic’ ...</td>\n",
       "      <td>FAKE</td>\n",
       "      <td>[ 0.01861144  0.00860776  0.01915705  0.074776...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6332</th>\n",
       "      <td>8622</td>\n",
       "      <td>Anti-Trump Protesters Are Tools of the Oligarc...</td>\n",
       "      <td>Anti-Trump Protesters Are Tools of the Oligar...</td>\n",
       "      <td>FAKE</td>\n",
       "      <td>[-2.33481918e-02  2.35741232e-02  3.15252207e-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6333</th>\n",
       "      <td>4021</td>\n",
       "      <td>In Ethiopia, Obama seeks progress on peace, se...</td>\n",
       "      <td>ADDIS ABABA, Ethiopia —President Obama convene...</td>\n",
       "      <td>REAL</td>\n",
       "      <td>[ 5.14873676e-02  7.35515803e-02  3.77446413e-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6334</th>\n",
       "      <td>4330</td>\n",
       "      <td>Jeb Bush Is Suddenly Attacking Trump. Here's W...</td>\n",
       "      <td>Jeb Bush Is Suddenly Attacking Trump. Here's W...</td>\n",
       "      <td>REAL</td>\n",
       "      <td>[ 0.0595002   0.06272358 -0.04209072  0.104249...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Unnamed: 0                                              title  \\\n",
       "6330        4490  State Department says it can't find emails fro...   \n",
       "6331        8062  The ‘P’ in PBS Should Stand for ‘Plutocratic’ ...   \n",
       "6332        8622  Anti-Trump Protesters Are Tools of the Oligarc...   \n",
       "6333        4021  In Ethiopia, Obama seeks progress on peace, se...   \n",
       "6334        4330  Jeb Bush Is Suddenly Attacking Trump. Here's W...   \n",
       "\n",
       "                                                   text label  \\\n",
       "6330  The State Department told the Republican Natio...  REAL   \n",
       "6331  The ‘P’ in PBS Should Stand for ‘Plutocratic’ ...  FAKE   \n",
       "6332   Anti-Trump Protesters Are Tools of the Oligar...  FAKE   \n",
       "6333  ADDIS ABABA, Ethiopia —President Obama convene...  REAL   \n",
       "6334  Jeb Bush Is Suddenly Attacking Trump. Here's W...  REAL   \n",
       "\n",
       "                                          title_vectors  \n",
       "6330  [ 0.04833125  0.01000964  0.09758892  0.089780...  \n",
       "6331  [ 0.01861144  0.00860776  0.01915705  0.074776...  \n",
       "6332  [-2.33481918e-02  2.35741232e-02  3.15252207e-...  \n",
       "6333  [ 5.14873676e-02  7.35515803e-02  3.77446413e-...  \n",
       "6334  [ 0.0595002   0.06272358 -0.04209072  0.104249...  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('data/fake_or_real_news.csv', )\n",
    "\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}