TitanicPythonIntro.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:955cdfd414bd9788fb3483f3316f33e5590b0e3a242c7804a1e1a680b6f2c21a"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Pandas and Scikit-Learn for the Titanic Challenge\n",
      "---\n",
      "There 5 broad steps that are covered in this Notebook:\n",
      "1. Loading the Titanic CSV data into a Pandas Dataframe\n",
      "2. Use Pandas for 'Data Munging' and 'Feature Engineering'\n",
      "    i.e. Format + clean the training and test data to feed into Scikit-learn for machine-learning.\n",
      "3. Scikit-learn for machine learning is incompatible with Pandas,so convert the data from a Pandas DataFrame into a Numpy array.\n",
      "4. Load the resulting training data and supervision labels to train or fit a model using Scikit-learn.\n",
      "5. See how well your model did by predicting outcomes on the test data."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Import the needed modules into Python\n",
      "----"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import pandas as pd\n",
      "import numpy as np\n",
      "import pylab as P\n",
      "from sklearn.linear_model import LogisticRegression"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 94
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Load the training and test data using the pandas CSV reader\n",
      "----"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_train = pd.read_csv(\"data/train.csv\") \n",
      "df_train.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>PassengerId</th>\n",
        "      <th>Survived</th>\n",
        "      <th>Pclass</th>\n",
        "      <th>Name</th>\n",
        "      <th>Sex</th>\n",
        "      <th>Age</th>\n",
        "      <th>SibSp</th>\n",
        "      <th>Parch</th>\n",
        "      <th>Ticket</th>\n",
        "      <th>Fare</th>\n",
        "      <th>Cabin</th>\n",
        "      <th>Embarked</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td> 3</td>\n",
        "      <td>                           Braund, Mr. Owen Harris</td>\n",
        "      <td>   male</td>\n",
        "      <td> 22</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>        A/5 21171</td>\n",
        "      <td>  7.2500</td>\n",
        "      <td>  NaN</td>\n",
        "      <td> S</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 2</td>\n",
        "      <td> 1</td>\n",
        "      <td> 1</td>\n",
        "      <td> Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
        "      <td> female</td>\n",
        "      <td> 38</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>         PC 17599</td>\n",
        "      <td> 71.2833</td>\n",
        "      <td>  C85</td>\n",
        "      <td> C</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 3</td>\n",
        "      <td> 1</td>\n",
        "      <td> 3</td>\n",
        "      <td>                            Heikkinen, Miss. Laina</td>\n",
        "      <td> female</td>\n",
        "      <td> 26</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> STON/O2. 3101282</td>\n",
        "      <td>  7.9250</td>\n",
        "      <td>  NaN</td>\n",
        "      <td> S</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 4</td>\n",
        "      <td> 1</td>\n",
        "      <td> 1</td>\n",
        "      <td>      Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
        "      <td> female</td>\n",
        "      <td> 35</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>           113803</td>\n",
        "      <td> 53.1000</td>\n",
        "      <td> C123</td>\n",
        "      <td> S</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 5</td>\n",
        "      <td> 0</td>\n",
        "      <td> 3</td>\n",
        "      <td>                          Allen, Mr. William Henry</td>\n",
        "      <td>   male</td>\n",
        "      <td> 35</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>           373450</td>\n",
        "      <td>  8.0500</td>\n",
        "      <td>  NaN</td>\n",
        "      <td> S</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 95,
       "text": [
        "   PassengerId  Survived  Pclass  \\\n",
        "0            1         0       3   \n",
        "1            2         1       1   \n",
        "2            3         1       3   \n",
        "3            4         1       1   \n",
        "4            5         0       3   \n",
        "\n",
        "                                                Name     Sex  Age  SibSp  \\\n",
        "0                            Braund, Mr. Owen Harris    male   22      1   \n",
        "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female   38      1   \n",
        "2                             Heikkinen, Miss. Laina  female   26      0   \n",
        "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female   35      1   \n",
        "4                           Allen, Mr. William Henry    male   35      0   \n",
        "\n",
        "   Parch            Ticket     Fare Cabin Embarked  \n",
        "0      0         A/5 21171   7.2500   NaN        S  \n",
        "1      0          PC 17599  71.2833   C85        C  \n",
        "2      0  STON/O2. 3101282   7.9250   NaN        S  \n",
        "3      0            113803  53.1000  C123        S  \n",
        "4      0            373450   8.0500   NaN        S  "
       ]
      }
     ],
     "prompt_number": 95
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_test = pd.read_csv(\"data/test.csv\") \n",
      "df_test.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>PassengerId</th>\n",
        "      <th>Pclass</th>\n",
        "      <th>Name</th>\n",
        "      <th>Sex</th>\n",
        "      <th>Age</th>\n",
        "      <th>SibSp</th>\n",
        "      <th>Parch</th>\n",
        "      <th>Ticket</th>\n",
        "      <th>Fare</th>\n",
        "      <th>Cabin</th>\n",
        "      <th>Embarked</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> 892</td>\n",
        "      <td> 3</td>\n",
        "      <td>                             Kelly, Mr. James</td>\n",
        "      <td>   male</td>\n",
        "      <td> 34.5</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>  330911</td>\n",
        "      <td>  7.8292</td>\n",
        "      <td> NaN</td>\n",
        "      <td> Q</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 893</td>\n",
        "      <td> 3</td>\n",
        "      <td>             Wilkes, Mrs. James (Ellen Needs)</td>\n",
        "      <td> female</td>\n",
        "      <td> 47.0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>  363272</td>\n",
        "      <td>  7.0000</td>\n",
        "      <td> NaN</td>\n",
        "      <td> S</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 894</td>\n",
        "      <td> 2</td>\n",
        "      <td>                    Myles, Mr. Thomas Francis</td>\n",
        "      <td>   male</td>\n",
        "      <td> 62.0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>  240276</td>\n",
        "      <td>  9.6875</td>\n",
        "      <td> NaN</td>\n",
        "      <td> Q</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 895</td>\n",
        "      <td> 3</td>\n",
        "      <td>                             Wirz, Mr. Albert</td>\n",
        "      <td>   male</td>\n",
        "      <td> 27.0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>  315154</td>\n",
        "      <td>  8.6625</td>\n",
        "      <td> NaN</td>\n",
        "      <td> S</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 896</td>\n",
        "      <td> 3</td>\n",
        "      <td> Hirvonen, Mrs. Alexander (Helga E Lindqvist)</td>\n",
        "      <td> female</td>\n",
        "      <td> 22.0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 1</td>\n",
        "      <td> 3101298</td>\n",
        "      <td> 12.2875</td>\n",
        "      <td> NaN</td>\n",
        "      <td> S</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 96,
       "text": [
        "   PassengerId  Pclass                                          Name     Sex  \\\n",
        "0          892       3                              Kelly, Mr. James    male   \n",
        "1          893       3              Wilkes, Mrs. James (Ellen Needs)  female   \n",
        "2          894       2                     Myles, Mr. Thomas Francis    male   \n",
        "3          895       3                              Wirz, Mr. Albert    male   \n",
        "4          896       3  Hirvonen, Mrs. Alexander (Helga E Lindqvist)  female   \n",
        "\n",
        "    Age  SibSp  Parch   Ticket     Fare Cabin Embarked  \n",
        "0  34.5      0      0   330911   7.8292   NaN        Q  \n",
        "1  47.0      1      0   363272   7.0000   NaN        S  \n",
        "2  62.0      0      0   240276   9.6875   NaN        Q  \n",
        "3  27.0      0      0   315154   8.6625   NaN        S  \n",
        "4  22.0      1      1  3101298  12.2875   NaN        S  "
       ]
      }
     ],
     "prompt_number": 96
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Use Pandas DataFrame info() and describe() routines to get more information about the data\n",
      "---"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Info about the training data\n",
      "df_train.info()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "<class 'pandas.core.frame.DataFrame'>\n",
        "Int64Index: 891 entries, 0 to 890\n",
        "Data columns (total 12 columns):\n",
        "PassengerId    891 non-null int64\n",
        "Survived       891 non-null int64\n",
        "Pclass         891 non-null int64\n",
        "Name           891 non-null object\n",
        "Sex            891 non-null object\n",
        "Age            714 non-null float64\n",
        "SibSp          891 non-null int64\n",
        "Parch          891 non-null int64\n",
        "Ticket         891 non-null object\n",
        "Fare           891 non-null float64\n",
        "Cabin          204 non-null object\n",
        "Embarked       889 non-null object\n",
        "dtypes: float64(2), int64(5), object(5)"
       ]
      }
     ],
     "prompt_number": 97
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Description of the training data\n",
      "df_train.describe()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>PassengerId</th>\n",
        "      <th>Survived</th>\n",
        "      <th>Pclass</th>\n",
        "      <th>Age</th>\n",
        "      <th>SibSp</th>\n",
        "      <th>Parch</th>\n",
        "      <th>Fare</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>count</th>\n",
        "      <td> 891.000000</td>\n",
        "      <td> 891.000000</td>\n",
        "      <td> 891.000000</td>\n",
        "      <td> 714.000000</td>\n",
        "      <td> 891.000000</td>\n",
        "      <td> 891.000000</td>\n",
        "      <td> 891.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>mean</th>\n",
        "      <td> 446.000000</td>\n",
        "      <td>   0.383838</td>\n",
        "      <td>   2.308642</td>\n",
        "      <td>  29.699118</td>\n",
        "      <td>   0.523008</td>\n",
        "      <td>   0.381594</td>\n",
        "      <td>  32.204208</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>std</th>\n",
        "      <td> 257.353842</td>\n",
        "      <td>   0.486592</td>\n",
        "      <td>   0.836071</td>\n",
        "      <td>  14.526497</td>\n",
        "      <td>   1.102743</td>\n",
        "      <td>   0.806057</td>\n",
        "      <td>  49.693429</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>min</th>\n",
        "      <td>   1.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   1.000000</td>\n",
        "      <td>   0.420000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>25%</th>\n",
        "      <td> 223.500000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   2.000000</td>\n",
        "      <td>  20.125000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   7.910400</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>50%</th>\n",
        "      <td> 446.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   3.000000</td>\n",
        "      <td>  28.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>  14.454200</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>75%</th>\n",
        "      <td> 668.500000</td>\n",
        "      <td>   1.000000</td>\n",
        "      <td>   3.000000</td>\n",
        "      <td>  38.000000</td>\n",
        "      <td>   1.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>  31.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>max</th>\n",
        "      <td> 891.000000</td>\n",
        "      <td>   1.000000</td>\n",
        "      <td>   3.000000</td>\n",
        "      <td>  80.000000</td>\n",
        "      <td>   8.000000</td>\n",
        "      <td>   6.000000</td>\n",
        "      <td> 512.329200</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 98,
       "text": [
        "       PassengerId    Survived      Pclass         Age       SibSp  \\\n",
        "count   891.000000  891.000000  891.000000  714.000000  891.000000   \n",
        "mean    446.000000    0.383838    2.308642   29.699118    0.523008   \n",
        "std     257.353842    0.486592    0.836071   14.526497    1.102743   \n",
        "min       1.000000    0.000000    1.000000    0.420000    0.000000   \n",
        "25%     223.500000    0.000000    2.000000   20.125000    0.000000   \n",
        "50%     446.000000    0.000000    3.000000   28.000000    0.000000   \n",
        "75%     668.500000    1.000000    3.000000   38.000000    1.000000   \n",
        "max     891.000000    1.000000    3.000000   80.000000    8.000000   \n",
        "\n",
        "            Parch        Fare  \n",
        "count  891.000000  891.000000  \n",
        "mean     0.381594   32.204208  \n",
        "std      0.806057   49.693429  \n",
        "min      0.000000    0.000000  \n",
        "25%      0.000000    7.910400  \n",
        "50%      0.000000   14.454200  \n",
        "75%      0.000000   31.000000  \n",
        "max      6.000000  512.329200  "
       ]
      }
     ],
     "prompt_number": 98
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Info about the test data\n",
      "df_test.info()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "<class 'pandas.core.frame.DataFrame'>\n",
        "Int64Index: 418 entries, 0 to 417\n",
        "Data columns (total 11 columns):\n",
        "PassengerId    418 non-null int64\n",
        "Pclass         418 non-null int64\n",
        "Name           418 non-null object\n",
        "Sex            418 non-null object\n",
        "Age            332 non-null float64\n",
        "SibSp          418 non-null int64\n",
        "Parch          418 non-null int64\n",
        "Ticket         418 non-null object\n",
        "Fare           417 non-null float64\n",
        "Cabin          91 non-null object\n",
        "Embarked       418 non-null object\n",
        "dtypes: float64(2), int64(4), object(5)"
       ]
      }
     ],
     "prompt_number": 99
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Description of the training data\n",
      "df_test.describe()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>PassengerId</th>\n",
        "      <th>Pclass</th>\n",
        "      <th>Age</th>\n",
        "      <th>SibSp</th>\n",
        "      <th>Parch</th>\n",
        "      <th>Fare</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>count</th>\n",
        "      <td>  418.000000</td>\n",
        "      <td> 418.000000</td>\n",
        "      <td> 332.000000</td>\n",
        "      <td> 418.000000</td>\n",
        "      <td> 418.000000</td>\n",
        "      <td> 417.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>mean</th>\n",
        "      <td> 1100.500000</td>\n",
        "      <td>   2.265550</td>\n",
        "      <td>  30.272590</td>\n",
        "      <td>   0.447368</td>\n",
        "      <td>   0.392344</td>\n",
        "      <td>  35.627188</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>std</th>\n",
        "      <td>  120.810458</td>\n",
        "      <td>   0.841838</td>\n",
        "      <td>  14.181209</td>\n",
        "      <td>   0.896760</td>\n",
        "      <td>   0.981429</td>\n",
        "      <td>  55.907576</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>min</th>\n",
        "      <td>  892.000000</td>\n",
        "      <td>   1.000000</td>\n",
        "      <td>   0.170000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>25%</th>\n",
        "      <td>  996.250000</td>\n",
        "      <td>   1.000000</td>\n",
        "      <td>  21.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   7.895800</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>50%</th>\n",
        "      <td> 1100.500000</td>\n",
        "      <td>   3.000000</td>\n",
        "      <td>  27.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>  14.454200</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>75%</th>\n",
        "      <td> 1204.750000</td>\n",
        "      <td>   3.000000</td>\n",
        "      <td>  39.000000</td>\n",
        "      <td>   1.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>  31.500000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>max</th>\n",
        "      <td> 1309.000000</td>\n",
        "      <td>   3.000000</td>\n",
        "      <td>  76.000000</td>\n",
        "      <td>   8.000000</td>\n",
        "      <td>   9.000000</td>\n",
        "      <td> 512.329200</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 100,
       "text": [
        "       PassengerId      Pclass         Age       SibSp       Parch        Fare\n",
        "count   418.000000  418.000000  332.000000  418.000000  418.000000  417.000000\n",
        "mean   1100.500000    2.265550   30.272590    0.447368    0.392344   35.627188\n",
        "std     120.810458    0.841838   14.181209    0.896760    0.981429   55.907576\n",
        "min     892.000000    1.000000    0.170000    0.000000    0.000000    0.000000\n",
        "25%     996.250000    1.000000   21.000000    0.000000    0.000000    7.895800\n",
        "50%    1100.500000    3.000000   27.000000    0.000000    0.000000   14.454200\n",
        "75%    1204.750000    3.000000   39.000000    1.000000    0.000000   31.500000\n",
        "max    1309.000000    3.000000   76.000000    8.000000    9.000000  512.329200"
       ]
      }
     ],
     "prompt_number": 100
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "To discard or to not discard, that is the question\n",
      "----\n",
      "There are many places where information about a passenger is unavailable, e.g. 'Age' has many NaN values.\n",
      "How do you deal with this incomplete information?"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Use the Pandas DataFrame isnull() function, DataFrame logical indexing and DataFrame Field referencing \n",
      "to see some of the entries which have NaN/null age values"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Field Referencing ---> df['Age']\n",
      "#isNull Function ----> df['Age'].isnull()\n",
      "#Logical Indexing ---> df[df['Age'].isnull()]\n",
      "nullAgeDataFrame=df_train[df_train['Age'].isnull()]\n",
      "#To view sub-fields of a frame, provide the subfields as a list ---> ['Sex', 'Pclass', 'Age']\n",
      "nullAgeDataFrame[['Sex', 'Pclass', 'Age']]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>Sex</th>\n",
        "      <th>Pclass</th>\n",
        "      <th>Age</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>5  </th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>17 </th>\n",
        "      <td>   male</td>\n",
        "      <td> 2</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>19 </th>\n",
        "      <td> female</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>26 </th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>28 </th>\n",
        "      <td> female</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>29 </th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>31 </th>\n",
        "      <td> female</td>\n",
        "      <td> 1</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>32 </th>\n",
        "      <td> female</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>36 </th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>42 </th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>45 </th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>46 </th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>47 </th>\n",
        "      <td> female</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>48 </th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>55 </th>\n",
        "      <td>   male</td>\n",
        "      <td> 1</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>64 </th>\n",
        "      <td>   male</td>\n",
        "      <td> 1</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>65 </th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>76 </th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>77 </th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>82 </th>\n",
        "      <td> female</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>87 </th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>95 </th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>101</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>107</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>109</th>\n",
        "      <td> female</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>121</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>126</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>128</th>\n",
        "      <td> female</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>140</th>\n",
        "      <td> female</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>154</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>...</th>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>718</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>727</th>\n",
        "      <td> female</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>732</th>\n",
        "      <td>   male</td>\n",
        "      <td> 2</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>738</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>739</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>740</th>\n",
        "      <td>   male</td>\n",
        "      <td> 1</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>760</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>766</th>\n",
        "      <td>   male</td>\n",
        "      <td> 1</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>768</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>773</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>776</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>778</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>783</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>790</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>792</th>\n",
        "      <td> female</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>793</th>\n",
        "      <td>   male</td>\n",
        "      <td> 1</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>815</th>\n",
        "      <td>   male</td>\n",
        "      <td> 1</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>825</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>826</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>828</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>832</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>837</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>839</th>\n",
        "      <td>   male</td>\n",
        "      <td> 1</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>846</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>849</th>\n",
        "      <td> female</td>\n",
        "      <td> 1</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>859</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>863</th>\n",
        "      <td> female</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>868</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>878</th>\n",
        "      <td>   male</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>888</th>\n",
        "      <td> female</td>\n",
        "      <td> 3</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "<p>177 rows \u00d7 3 columns</p>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 101,
       "text": [
        "        Sex  Pclass  Age\n",
        "5      male       3  NaN\n",
        "17     male       2  NaN\n",
        "19   female       3  NaN\n",
        "26     male       3  NaN\n",
        "28   female       3  NaN\n",
        "29     male       3  NaN\n",
        "31   female       1  NaN\n",
        "32   female       3  NaN\n",
        "36     male       3  NaN\n",
        "42     male       3  NaN\n",
        "45     male       3  NaN\n",
        "46     male       3  NaN\n",
        "47   female       3  NaN\n",
        "48     male       3  NaN\n",
        "55     male       1  NaN\n",
        "64     male       1  NaN\n",
        "65     male       3  NaN\n",
        "76     male       3  NaN\n",
        "77     male       3  NaN\n",
        "82   female       3  NaN\n",
        "87     male       3  NaN\n",
        "95     male       3  NaN\n",
        "101    male       3  NaN\n",
        "107    male       3  NaN\n",
        "109  female       3  NaN\n",
        "121    male       3  NaN\n",
        "126    male       3  NaN\n",
        "128  female       3  NaN\n",
        "140  female       3  NaN\n",
        "154    male       3  NaN\n",
        "..      ...     ...  ...\n",
        "718    male       3  NaN\n",
        "727  female       3  NaN\n",
        "732    male       2  NaN\n",
        "738    male       3  NaN\n",
        "739    male       3  NaN\n",
        "740    male       1  NaN\n",
        "760    male       3  NaN\n",
        "766    male       1  NaN\n",
        "768    male       3  NaN\n",
        "773    male       3  NaN\n",
        "776    male       3  NaN\n",
        "778    male       3  NaN\n",
        "783    male       3  NaN\n",
        "790    male       3  NaN\n",
        "792  female       3  NaN\n",
        "793    male       1  NaN\n",
        "815    male       1  NaN\n",
        "825    male       3  NaN\n",
        "826    male       3  NaN\n",
        "828    male       3  NaN\n",
        "832    male       3  NaN\n",
        "837    male       3  NaN\n",
        "839    male       1  NaN\n",
        "846    male       3  NaN\n",
        "849  female       1  NaN\n",
        "859    male       3  NaN\n",
        "863  female       3  NaN\n",
        "868    male       3  NaN\n",
        "878    male       3  NaN\n",
        "888  female       3  NaN\n",
        "\n",
        "[177 rows x 3 columns]"
       ]
      }
     ],
     "prompt_number": 101
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "A Solution : Replace null/Nan age entries with median age values. \n",
      "---\n",
      "Median age values are to be calculated over Sex and PClass.\n",
      "    * Sex : M,F\n",
      "    * PClass : 1,2,3\n",
      "    * Therefore, 6 median age values are to be calculated"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "An aside, replace Sex - F/M with numeric values 0/1\n",
      "---\n",
      "This would make data processing easier"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#create a new column called Gender and populate it with 1 : Male, and 0:Female \n",
      "#Training data \n",
      "df_train['Gender'] = df_train['Sex'].map( {'female': 0, 'male': 1} ).astype(int)\n",
      "df_train.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>PassengerId</th>\n",
        "      <th>Survived</th>\n",
        "      <th>Pclass</th>\n",
        "      <th>Name</th>\n",
        "      <th>Sex</th>\n",
        "      <th>Age</th>\n",
        "      <th>SibSp</th>\n",
        "      <th>Parch</th>\n",
        "      <th>Ticket</th>\n",
        "      <th>Fare</th>\n",
        "      <th>Cabin</th>\n",
        "      <th>Embarked</th>\n",
        "      <th>Gender</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td> 3</td>\n",
        "      <td>                           Braund, Mr. Owen Harris</td>\n",
        "      <td>   male</td>\n",
        "      <td> 22</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>        A/5 21171</td>\n",
        "      <td>  7.2500</td>\n",
        "      <td>  NaN</td>\n",
        "      <td> S</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 2</td>\n",
        "      <td> 1</td>\n",
        "      <td> 1</td>\n",
        "      <td> Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
        "      <td> female</td>\n",
        "      <td> 38</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>         PC 17599</td>\n",
        "      <td> 71.2833</td>\n",
        "      <td>  C85</td>\n",
        "      <td> C</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 3</td>\n",
        "      <td> 1</td>\n",
        "      <td> 3</td>\n",
        "      <td>                            Heikkinen, Miss. Laina</td>\n",
        "      <td> female</td>\n",
        "      <td> 26</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> STON/O2. 3101282</td>\n",
        "      <td>  7.9250</td>\n",
        "      <td>  NaN</td>\n",
        "      <td> S</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 4</td>\n",
        "      <td> 1</td>\n",
        "      <td> 1</td>\n",
        "      <td>      Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
        "      <td> female</td>\n",
        "      <td> 35</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>           113803</td>\n",
        "      <td> 53.1000</td>\n",
        "      <td> C123</td>\n",
        "      <td> S</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 5</td>\n",
        "      <td> 0</td>\n",
        "      <td> 3</td>\n",
        "      <td>                          Allen, Mr. William Henry</td>\n",
        "      <td>   male</td>\n",
        "      <td> 35</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>           373450</td>\n",
        "      <td>  8.0500</td>\n",
        "      <td>  NaN</td>\n",
        "      <td> S</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 102,
       "text": [
        "   PassengerId  Survived  Pclass  \\\n",
        "0            1         0       3   \n",
        "1            2         1       1   \n",
        "2            3         1       3   \n",
        "3            4         1       1   \n",
        "4            5         0       3   \n",
        "\n",
        "                                                Name     Sex  Age  SibSp  \\\n",
        "0                            Braund, Mr. Owen Harris    male   22      1   \n",
        "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female   38      1   \n",
        "2                             Heikkinen, Miss. Laina  female   26      0   \n",
        "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female   35      1   \n",
        "4                           Allen, Mr. William Henry    male   35      0   \n",
        "\n",
        "   Parch            Ticket     Fare Cabin Embarked  Gender  \n",
        "0      0         A/5 21171   7.2500   NaN        S       1  \n",
        "1      0          PC 17599  71.2833   C85        C       0  \n",
        "2      0  STON/O2. 3101282   7.9250   NaN        S       0  \n",
        "3      0            113803  53.1000  C123        S       0  \n",
        "4      0            373450   8.0500   NaN        S       1  "
       ]
      }
     ],
     "prompt_number": 102
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#create a new column called Gender and populate it with 1 : Male, and 0:Female \n",
      "#Test data\n",
      "df_test['Gender'] = df_test['Sex'].map( {'female': 0, 'male': 1} ).astype(int)\n",
      "df_test.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>PassengerId</th>\n",
        "      <th>Pclass</th>\n",
        "      <th>Name</th>\n",
        "      <th>Sex</th>\n",
        "      <th>Age</th>\n",
        "      <th>SibSp</th>\n",
        "      <th>Parch</th>\n",
        "      <th>Ticket</th>\n",
        "      <th>Fare</th>\n",
        "      <th>Cabin</th>\n",
        "      <th>Embarked</th>\n",
        "      <th>Gender</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> 892</td>\n",
        "      <td> 3</td>\n",
        "      <td>                             Kelly, Mr. James</td>\n",
        "      <td>   male</td>\n",
        "      <td> 34.5</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>  330911</td>\n",
        "      <td>  7.8292</td>\n",
        "      <td> NaN</td>\n",
        "      <td> Q</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 893</td>\n",
        "      <td> 3</td>\n",
        "      <td>             Wilkes, Mrs. James (Ellen Needs)</td>\n",
        "      <td> female</td>\n",
        "      <td> 47.0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>  363272</td>\n",
        "      <td>  7.0000</td>\n",
        "      <td> NaN</td>\n",
        "      <td> S</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 894</td>\n",
        "      <td> 2</td>\n",
        "      <td>                    Myles, Mr. Thomas Francis</td>\n",
        "      <td>   male</td>\n",
        "      <td> 62.0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>  240276</td>\n",
        "      <td>  9.6875</td>\n",
        "      <td> NaN</td>\n",
        "      <td> Q</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 895</td>\n",
        "      <td> 3</td>\n",
        "      <td>                             Wirz, Mr. Albert</td>\n",
        "      <td>   male</td>\n",
        "      <td> 27.0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>  315154</td>\n",
        "      <td>  8.6625</td>\n",
        "      <td> NaN</td>\n",
        "      <td> S</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 896</td>\n",
        "      <td> 3</td>\n",
        "      <td> Hirvonen, Mrs. Alexander (Helga E Lindqvist)</td>\n",
        "      <td> female</td>\n",
        "      <td> 22.0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 1</td>\n",
        "      <td> 3101298</td>\n",
        "      <td> 12.2875</td>\n",
        "      <td> NaN</td>\n",
        "      <td> S</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 103,
       "text": [
        "   PassengerId  Pclass                                          Name     Sex  \\\n",
        "0          892       3                              Kelly, Mr. James    male   \n",
        "1          893       3              Wilkes, Mrs. James (Ellen Needs)  female   \n",
        "2          894       2                     Myles, Mr. Thomas Francis    male   \n",
        "3          895       3                              Wirz, Mr. Albert    male   \n",
        "4          896       3  Hirvonen, Mrs. Alexander (Helga E Lindqvist)  female   \n",
        "\n",
        "    Age  SibSp  Parch   Ticket     Fare Cabin Embarked  Gender  \n",
        "0  34.5      0      0   330911   7.8292   NaN        Q       1  \n",
        "1  47.0      1      0   363272   7.0000   NaN        S       0  \n",
        "2  62.0      0      0   240276   9.6875   NaN        Q       1  \n",
        "3  27.0      0      0   315154   8.6625   NaN        S       1  \n",
        "4  22.0      1      1  3101298  12.2875   NaN        S       0  "
       ]
      }
     ],
     "prompt_number": 103
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now back to finding out the median age values over Sex, and PClass\n",
      "----"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "median_ages = np.zeros((2,3))#Rows correspond to females, males (2 Rows). Columns correspond to the class (3 Columns)\n",
      "median_ages"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 104,
       "text": [
        "array([[ 0.,  0.,  0.],\n",
        "       [ 0.,  0.,  0.]])"
       ]
      }
     ],
     "prompt_number": 104
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Use a nested loop to calculate the medians\n",
      "for i in range(0, 2):#loop over the Sex (Rows)\n",
      "    for j in range(0, 3):#loop over the classes (Columns)\n",
      "        #Here first short list all entries that correspond to the required Sex\n",
      "        # ---> (df_train['Gender'] == i) \n",
      "        # And ---> &\n",
      "        #In that short list, pick out the ones that correspond to a particular class\n",
      "        #---->df_train['Pclass'] == j+1\n",
      "        # Drop null age values using dropna() on a dataFrame and calulate the median\n",
      "        #Store median for Sex i, and Class j in median(i,j)\n",
      "        median_ages[i,j] = df_train[(df_train['Gender'] == i) & \\\n",
      "                              (df_train['Pclass'] == j+1)]['Age'].dropna().median()\n",
      "        \n",
      "median_ages"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 105,
       "text": [
        "array([[ 35. ,  28. ,  21.5],\n",
        "       [ 40. ,  30. ,  25. ]])"
       ]
      }
     ],
     "prompt_number": 105
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Create a new column called 'AgeFill' for train and test data frames\n",
      "----"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Training AgeFill\n",
      "df_train['AgeFill'] = df_train['Age']\n",
      "df_test['AgeFill'] = df_test['Age']"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 106
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for i in range(0, 2):#loop over the Sex (Rows)\n",
      "    for j in range(0, 3):#loop over the classes (Columns)\n",
      "        #Get the locations corresponding to Class and and Gender where the age is null and replace it\n",
      "        #with the median age in our new field AgeFill\n",
      "        df_train.loc[ (df_train.Age.isnull()) & (df_train.Gender == i) & (df_train.Pclass == j+1),'AgeFill'] = median_ages[i,j]\n",
      "        \n",
      "#Test AgeFill\n",
      "for i in range(0, 2):#loop over the Sex (Rows)\n",
      "    for j in range(0, 3):#loop over the classes (Columns)\n",
      "        #Get the locations corresponding to Class and and Gender where the age is null and replace it\n",
      "        #with the median age in our new field AgeFill\n",
      "        df_test.loc[ (df_test.Age.isnull()) & (df_test.Gender == i) & (df_test.Pclass == j+1),'AgeFill'] = median_ages[i,j]\n",
      "        \n",
      "df_train.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>PassengerId</th>\n",
        "      <th>Survived</th>\n",
        "      <th>Pclass</th>\n",
        "      <th>Name</th>\n",
        "      <th>Sex</th>\n",
        "      <th>Age</th>\n",
        "      <th>SibSp</th>\n",
        "      <th>Parch</th>\n",
        "      <th>Ticket</th>\n",
        "      <th>Fare</th>\n",
        "      <th>Cabin</th>\n",
        "      <th>Embarked</th>\n",
        "      <th>Gender</th>\n",
        "      <th>AgeFill</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td> 3</td>\n",
        "      <td>                           Braund, Mr. Owen Harris</td>\n",
        "      <td>   male</td>\n",
        "      <td> 22</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>        A/5 21171</td>\n",
        "      <td>  7.2500</td>\n",
        "      <td>  NaN</td>\n",
        "      <td> S</td>\n",
        "      <td> 1</td>\n",
        "      <td> 22</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 2</td>\n",
        "      <td> 1</td>\n",
        "      <td> 1</td>\n",
        "      <td> Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
        "      <td> female</td>\n",
        "      <td> 38</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>         PC 17599</td>\n",
        "      <td> 71.2833</td>\n",
        "      <td>  C85</td>\n",
        "      <td> C</td>\n",
        "      <td> 0</td>\n",
        "      <td> 38</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 3</td>\n",
        "      <td> 1</td>\n",
        "      <td> 3</td>\n",
        "      <td>                            Heikkinen, Miss. Laina</td>\n",
        "      <td> female</td>\n",
        "      <td> 26</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> STON/O2. 3101282</td>\n",
        "      <td>  7.9250</td>\n",
        "      <td>  NaN</td>\n",
        "      <td> S</td>\n",
        "      <td> 0</td>\n",
        "      <td> 26</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 4</td>\n",
        "      <td> 1</td>\n",
        "      <td> 1</td>\n",
        "      <td>      Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
        "      <td> female</td>\n",
        "      <td> 35</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>           113803</td>\n",
        "      <td> 53.1000</td>\n",
        "      <td> C123</td>\n",
        "      <td> S</td>\n",
        "      <td> 0</td>\n",
        "      <td> 35</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 5</td>\n",
        "      <td> 0</td>\n",
        "      <td> 3</td>\n",
        "      <td>                          Allen, Mr. William Henry</td>\n",
        "      <td>   male</td>\n",
        "      <td> 35</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>           373450</td>\n",
        "      <td>  8.0500</td>\n",
        "      <td>  NaN</td>\n",
        "      <td> S</td>\n",
        "      <td> 1</td>\n",
        "      <td> 35</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 107,
       "text": [
        "   PassengerId  Survived  Pclass  \\\n",
        "0            1         0       3   \n",
        "1            2         1       1   \n",
        "2            3         1       3   \n",
        "3            4         1       1   \n",
        "4            5         0       3   \n",
        "\n",
        "                                                Name     Sex  Age  SibSp  \\\n",
        "0                            Braund, Mr. Owen Harris    male   22      1   \n",
        "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female   38      1   \n",
        "2                             Heikkinen, Miss. Laina  female   26      0   \n",
        "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female   35      1   \n",
        "4                           Allen, Mr. William Henry    male   35      0   \n",
        "\n",
        "   Parch            Ticket     Fare Cabin Embarked  Gender  AgeFill  \n",
        "0      0         A/5 21171   7.2500   NaN        S       1       22  \n",
        "1      0          PC 17599  71.2833   C85        C       0       38  \n",
        "2      0  STON/O2. 3101282   7.9250   NaN        S       0       26  \n",
        "3      0            113803  53.1000  C123        S       0       35  \n",
        "4      0            373450   8.0500   NaN        S       1       35  "
       ]
      }
     ],
     "prompt_number": 107
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_test.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>PassengerId</th>\n",
        "      <th>Pclass</th>\n",
        "      <th>Name</th>\n",
        "      <th>Sex</th>\n",
        "      <th>Age</th>\n",
        "      <th>SibSp</th>\n",
        "      <th>Parch</th>\n",
        "      <th>Ticket</th>\n",
        "      <th>Fare</th>\n",
        "      <th>Cabin</th>\n",
        "      <th>Embarked</th>\n",
        "      <th>Gender</th>\n",
        "      <th>AgeFill</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> 892</td>\n",
        "      <td> 3</td>\n",
        "      <td>                             Kelly, Mr. James</td>\n",
        "      <td>   male</td>\n",
        "      <td> 34.5</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>  330911</td>\n",
        "      <td>  7.8292</td>\n",
        "      <td> NaN</td>\n",
        "      <td> Q</td>\n",
        "      <td> 1</td>\n",
        "      <td> 34.5</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 893</td>\n",
        "      <td> 3</td>\n",
        "      <td>             Wilkes, Mrs. James (Ellen Needs)</td>\n",
        "      <td> female</td>\n",
        "      <td> 47.0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>  363272</td>\n",
        "      <td>  7.0000</td>\n",
        "      <td> NaN</td>\n",
        "      <td> S</td>\n",
        "      <td> 0</td>\n",
        "      <td> 47.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 894</td>\n",
        "      <td> 2</td>\n",
        "      <td>                    Myles, Mr. Thomas Francis</td>\n",
        "      <td>   male</td>\n",
        "      <td> 62.0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>  240276</td>\n",
        "      <td>  9.6875</td>\n",
        "      <td> NaN</td>\n",
        "      <td> Q</td>\n",
        "      <td> 1</td>\n",
        "      <td> 62.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 895</td>\n",
        "      <td> 3</td>\n",
        "      <td>                             Wirz, Mr. Albert</td>\n",
        "      <td>   male</td>\n",
        "      <td> 27.0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>  315154</td>\n",
        "      <td>  8.6625</td>\n",
        "      <td> NaN</td>\n",
        "      <td> S</td>\n",
        "      <td> 1</td>\n",
        "      <td> 27.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 896</td>\n",
        "      <td> 3</td>\n",
        "      <td> Hirvonen, Mrs. Alexander (Helga E Lindqvist)</td>\n",
        "      <td> female</td>\n",
        "      <td> 22.0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 1</td>\n",
        "      <td> 3101298</td>\n",
        "      <td> 12.2875</td>\n",
        "      <td> NaN</td>\n",
        "      <td> S</td>\n",
        "      <td> 0</td>\n",
        "      <td> 22.0</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 108,
       "text": [
        "   PassengerId  Pclass                                          Name     Sex  \\\n",
        "0          892       3                              Kelly, Mr. James    male   \n",
        "1          893       3              Wilkes, Mrs. James (Ellen Needs)  female   \n",
        "2          894       2                     Myles, Mr. Thomas Francis    male   \n",
        "3          895       3                              Wirz, Mr. Albert    male   \n",
        "4          896       3  Hirvonen, Mrs. Alexander (Helga E Lindqvist)  female   \n",
        "\n",
        "    Age  SibSp  Parch   Ticket     Fare Cabin Embarked  Gender  AgeFill  \n",
        "0  34.5      0      0   330911   7.8292   NaN        Q       1     34.5  \n",
        "1  47.0      1      0   363272   7.0000   NaN        S       0     47.0  \n",
        "2  62.0      0      0   240276   9.6875   NaN        Q       1     62.0  \n",
        "3  27.0      0      0   315154   8.6625   NaN        S       1     27.0  \n",
        "4  22.0      1      1  3101298  12.2875   NaN        S       0     22.0  "
       ]
      }
     ],
     "prompt_number": 108
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now for some feature engineering\n",
      "---\n",
      "    a. Create a new feature called Age*Class\n",
      "    b. Create a new feature called FamilySize = Number of Siblings/Spouses + Number of Parents/Children on -board"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_train['FamilySize'] = df_train['SibSp'] + df_train['Parch']\n",
      "df_test['FamilySize'] = df_test['SibSp'] + df_test['Parch']\n",
      "\n",
      "df_train['Age*Class'] = df_train.AgeFill * df_train.Pclass\n",
      "df_test['Age*Class'] = df_test.AgeFill * df_test.Pclass\n",
      "\n",
      "df_train.loc[888,:]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 109,
       "text": [
        "PassengerId                                         889\n",
        "Survived                                              0\n",
        "Pclass                                                3\n",
        "Name           Johnston, Miss. Catherine Helen \"Carrie\"\n",
        "Sex                                              female\n",
        "Age                                                 NaN\n",
        "SibSp                                                 1\n",
        "Parch                                                 2\n",
        "Ticket                                       W./C. 6607\n",
        "Fare                                              23.45\n",
        "Cabin                                               NaN\n",
        "Embarked                                              S\n",
        "Gender                                                0\n",
        "AgeFill                                            21.5\n",
        "FamilySize                                            3\n",
        "Age*Class                                          64.5\n",
        "Name: 888, dtype: object"
       ]
      }
     ],
     "prompt_number": 109
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_test.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>PassengerId</th>\n",
        "      <th>Pclass</th>\n",
        "      <th>Name</th>\n",
        "      <th>Sex</th>\n",
        "      <th>Age</th>\n",
        "      <th>SibSp</th>\n",
        "      <th>Parch</th>\n",
        "      <th>Ticket</th>\n",
        "      <th>Fare</th>\n",
        "      <th>Cabin</th>\n",
        "      <th>Embarked</th>\n",
        "      <th>Gender</th>\n",
        "      <th>AgeFill</th>\n",
        "      <th>FamilySize</th>\n",
        "      <th>Age*Class</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> 892</td>\n",
        "      <td> 3</td>\n",
        "      <td>                             Kelly, Mr. James</td>\n",
        "      <td>   male</td>\n",
        "      <td> 34.5</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>  330911</td>\n",
        "      <td>  7.8292</td>\n",
        "      <td> NaN</td>\n",
        "      <td> Q</td>\n",
        "      <td> 1</td>\n",
        "      <td> 34.5</td>\n",
        "      <td> 0</td>\n",
        "      <td> 103.5</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 893</td>\n",
        "      <td> 3</td>\n",
        "      <td>             Wilkes, Mrs. James (Ellen Needs)</td>\n",
        "      <td> female</td>\n",
        "      <td> 47.0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>  363272</td>\n",
        "      <td>  7.0000</td>\n",
        "      <td> NaN</td>\n",
        "      <td> S</td>\n",
        "      <td> 0</td>\n",
        "      <td> 47.0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 141.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 894</td>\n",
        "      <td> 2</td>\n",
        "      <td>                    Myles, Mr. Thomas Francis</td>\n",
        "      <td>   male</td>\n",
        "      <td> 62.0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>  240276</td>\n",
        "      <td>  9.6875</td>\n",
        "      <td> NaN</td>\n",
        "      <td> Q</td>\n",
        "      <td> 1</td>\n",
        "      <td> 62.0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 124.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 895</td>\n",
        "      <td> 3</td>\n",
        "      <td>                             Wirz, Mr. Albert</td>\n",
        "      <td>   male</td>\n",
        "      <td> 27.0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>  315154</td>\n",
        "      <td>  8.6625</td>\n",
        "      <td> NaN</td>\n",
        "      <td> S</td>\n",
        "      <td> 1</td>\n",
        "      <td> 27.0</td>\n",
        "      <td> 0</td>\n",
        "      <td>  81.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 896</td>\n",
        "      <td> 3</td>\n",
        "      <td> Hirvonen, Mrs. Alexander (Helga E Lindqvist)</td>\n",
        "      <td> female</td>\n",
        "      <td> 22.0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 1</td>\n",
        "      <td> 3101298</td>\n",
        "      <td> 12.2875</td>\n",
        "      <td> NaN</td>\n",
        "      <td> S</td>\n",
        "      <td> 0</td>\n",
        "      <td> 22.0</td>\n",
        "      <td> 2</td>\n",
        "      <td>  66.0</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 110,
       "text": [
        "   PassengerId  Pclass                                          Name     Sex  \\\n",
        "0          892       3                              Kelly, Mr. James    male   \n",
        "1          893       3              Wilkes, Mrs. James (Ellen Needs)  female   \n",
        "2          894       2                     Myles, Mr. Thomas Francis    male   \n",
        "3          895       3                              Wirz, Mr. Albert    male   \n",
        "4          896       3  Hirvonen, Mrs. Alexander (Helga E Lindqvist)  female   \n",
        "\n",
        "    Age  SibSp  Parch   Ticket     Fare Cabin Embarked  Gender  AgeFill  \\\n",
        "0  34.5      0      0   330911   7.8292   NaN        Q       1     34.5   \n",
        "1  47.0      1      0   363272   7.0000   NaN        S       0     47.0   \n",
        "2  62.0      0      0   240276   9.6875   NaN        Q       1     62.0   \n",
        "3  27.0      0      0   315154   8.6625   NaN        S       1     27.0   \n",
        "4  22.0      1      1  3101298  12.2875   NaN        S       0     22.0   \n",
        "\n",
        "   FamilySize  Age*Class  \n",
        "0           0      103.5  \n",
        "1           1      141.0  \n",
        "2           0      124.0  \n",
        "3           0       81.0  \n",
        "4           2       66.0  "
       ]
      }
     ],
     "prompt_number": 110
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Drop values that are strings, keep only numeric values. Drop any rows that might have null values\n",
      "--"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "numpy_trainSupervisionLabels = df_train['Survived'].values\n",
      "#numpy_trainSupervisionLabels\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 111
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_train = df_train.drop(['Name', 'Sex', 'Ticket', 'Cabin', 'Embarked','Age','Survived','Fare'], axis=1)\n",
      "df_train.dropna()\n",
      "df_train.tail()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>PassengerId</th>\n",
        "      <th>Pclass</th>\n",
        "      <th>SibSp</th>\n",
        "      <th>Parch</th>\n",
        "      <th>Gender</th>\n",
        "      <th>AgeFill</th>\n",
        "      <th>FamilySize</th>\n",
        "      <th>Age*Class</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>886</th>\n",
        "      <td> 887</td>\n",
        "      <td> 2</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 27.0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 54.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>887</th>\n",
        "      <td> 888</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 19.0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 19.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>888</th>\n",
        "      <td> 889</td>\n",
        "      <td> 3</td>\n",
        "      <td> 1</td>\n",
        "      <td> 2</td>\n",
        "      <td> 0</td>\n",
        "      <td> 21.5</td>\n",
        "      <td> 3</td>\n",
        "      <td> 64.5</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>889</th>\n",
        "      <td> 890</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 26.0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 26.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>890</th>\n",
        "      <td> 891</td>\n",
        "      <td> 3</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 32.0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 96.0</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 112,
       "text": [
        "     PassengerId  Pclass  SibSp  Parch  Gender  AgeFill  FamilySize  Age*Class\n",
        "886          887       2      0      0       1     27.0           0       54.0\n",
        "887          888       1      0      0       0     19.0           0       19.0\n",
        "888          889       3      1      2       0     21.5           3       64.5\n",
        "889          890       1      0      0       1     26.0           0       26.0\n",
        "890          891       3      0      0       1     32.0           0       96.0"
       ]
      }
     ],
     "prompt_number": 112
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "numpy_train=df_train.values\n",
      "numpy_train\n",
      "clf = LogisticRegression()\n",
      "logReg = clf.fit(numpy_train,numpy_trainSupervisionLabels)\n",
      "#output = logReg.predict(numpy_test)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 113
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_test = df_test.drop(['Name', 'Sex', 'Ticket', 'Cabin', 'Embarked','Age','Fare'], axis=1)\n",
      "df_test.dropna()\n",
      "df_test.tail()\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>PassengerId</th>\n",
        "      <th>Pclass</th>\n",
        "      <th>SibSp</th>\n",
        "      <th>Parch</th>\n",
        "      <th>Gender</th>\n",
        "      <th>AgeFill</th>\n",
        "      <th>FamilySize</th>\n",
        "      <th>Age*Class</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>413</th>\n",
        "      <td> 1305</td>\n",
        "      <td> 3</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 25.0</td>\n",
        "      <td> 0</td>\n",
        "      <td>  75.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>414</th>\n",
        "      <td> 1306</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 39.0</td>\n",
        "      <td> 0</td>\n",
        "      <td>  39.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>415</th>\n",
        "      <td> 1307</td>\n",
        "      <td> 3</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 38.5</td>\n",
        "      <td> 0</td>\n",
        "      <td> 115.5</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>416</th>\n",
        "      <td> 1308</td>\n",
        "      <td> 3</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 25.0</td>\n",
        "      <td> 0</td>\n",
        "      <td>  75.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>417</th>\n",
        "      <td> 1309</td>\n",
        "      <td> 3</td>\n",
        "      <td> 1</td>\n",
        "      <td> 1</td>\n",
        "      <td> 1</td>\n",
        "      <td> 25.0</td>\n",
        "      <td> 2</td>\n",
        "      <td>  75.0</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 114,
       "text": [
        "     PassengerId  Pclass  SibSp  Parch  Gender  AgeFill  FamilySize  Age*Class\n",
        "413         1305       3      0      0       1     25.0           0       75.0\n",
        "414         1306       1      0      0       0     39.0           0       39.0\n",
        "415         1307       3      0      0       1     38.5           0      115.5\n",
        "416         1308       3      0      0       1     25.0           0       75.0\n",
        "417         1309       3      1      1       1     25.0           2       75.0"
       ]
      }
     ],
     "prompt_number": 114
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Do Logistic Regression with SKLearn\n",
      "--"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_test.head()\n",
      "#numpy_test=df_test.values"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>PassengerId</th>\n",
        "      <th>Pclass</th>\n",
        "      <th>SibSp</th>\n",
        "      <th>Parch</th>\n",
        "      <th>Gender</th>\n",
        "      <th>AgeFill</th>\n",
        "      <th>FamilySize</th>\n",
        "      <th>Age*Class</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> 892</td>\n",
        "      <td> 3</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 34.5</td>\n",
        "      <td> 0</td>\n",
        "      <td> 103.5</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 893</td>\n",
        "      <td> 3</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 47.0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 141.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 894</td>\n",
        "      <td> 2</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 62.0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 124.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 895</td>\n",
        "      <td> 3</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 27.0</td>\n",
        "      <td> 0</td>\n",
        "      <td>  81.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 896</td>\n",
        "      <td> 3</td>\n",
        "      <td> 1</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td> 22.0</td>\n",
        "      <td> 2</td>\n",
        "      <td>  66.0</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 115,
       "text": [
        "   PassengerId  Pclass  SibSp  Parch  Gender  AgeFill  FamilySize  Age*Class\n",
        "0          892       3      0      0       1     34.5           0      103.5\n",
        "1          893       3      1      0       0     47.0           1      141.0\n",
        "2          894       2      0      0       1     62.0           0      124.0\n",
        "3          895       3      0      0       1     27.0           0       81.0\n",
        "4          896       3      1      1       0     22.0           2       66.0"
       ]
      }
     ],
     "prompt_number": 115
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_test.tail()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>PassengerId</th>\n",
        "      <th>Pclass</th>\n",
        "      <th>SibSp</th>\n",
        "      <th>Parch</th>\n",
        "      <th>Gender</th>\n",
        "      <th>AgeFill</th>\n",
        "      <th>FamilySize</th>\n",
        "      <th>Age*Class</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>413</th>\n",
        "      <td> 1305</td>\n",
        "      <td> 3</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 25.0</td>\n",
        "      <td> 0</td>\n",
        "      <td>  75.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>414</th>\n",
        "      <td> 1306</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 39.0</td>\n",
        "      <td> 0</td>\n",
        "      <td>  39.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>415</th>\n",
        "      <td> 1307</td>\n",
        "      <td> 3</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 38.5</td>\n",
        "      <td> 0</td>\n",
        "      <td> 115.5</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>416</th>\n",
        "      <td> 1308</td>\n",
        "      <td> 3</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 25.0</td>\n",
        "      <td> 0</td>\n",
        "      <td>  75.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>417</th>\n",
        "      <td> 1309</td>\n",
        "      <td> 3</td>\n",
        "      <td> 1</td>\n",
        "      <td> 1</td>\n",
        "      <td> 1</td>\n",
        "      <td> 25.0</td>\n",
        "      <td> 2</td>\n",
        "      <td>  75.0</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 116,
       "text": [
        "     PassengerId  Pclass  SibSp  Parch  Gender  AgeFill  FamilySize  Age*Class\n",
        "413         1305       3      0      0       1     25.0           0       75.0\n",
        "414         1306       1      0      0       0     39.0           0       39.0\n",
        "415         1307       3      0      0       1     38.5           0      115.5\n",
        "416         1308       3      0      0       1     25.0           0       75.0\n",
        "417         1309       3      1      1       1     25.0           2       75.0"
       ]
      }
     ],
     "prompt_number": 116
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_test.describe()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>PassengerId</th>\n",
        "      <th>Pclass</th>\n",
        "      <th>SibSp</th>\n",
        "      <th>Parch</th>\n",
        "      <th>Gender</th>\n",
        "      <th>AgeFill</th>\n",
        "      <th>FamilySize</th>\n",
        "      <th>Age*Class</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>count</th>\n",
        "      <td>  418.000000</td>\n",
        "      <td> 418.000000</td>\n",
        "      <td> 418.000000</td>\n",
        "      <td> 418.000000</td>\n",
        "      <td> 418.000000</td>\n",
        "      <td> 418.000000</td>\n",
        "      <td> 418.000000</td>\n",
        "      <td> 418.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>mean</th>\n",
        "      <td> 1100.500000</td>\n",
        "      <td>   2.265550</td>\n",
        "      <td>   0.447368</td>\n",
        "      <td>   0.392344</td>\n",
        "      <td>   0.636364</td>\n",
        "      <td>  29.357656</td>\n",
        "      <td>   0.839713</td>\n",
        "      <td>  60.799234</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>std</th>\n",
        "      <td>  120.810458</td>\n",
        "      <td>   0.841838</td>\n",
        "      <td>   0.896760</td>\n",
        "      <td>   0.981429</td>\n",
        "      <td>   0.481622</td>\n",
        "      <td>  12.957412</td>\n",
        "      <td>   1.519072</td>\n",
        "      <td>  26.602739</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>min</th>\n",
        "      <td>  892.000000</td>\n",
        "      <td>   1.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   0.170000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   0.510000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>25%</th>\n",
        "      <td>  996.250000</td>\n",
        "      <td>   1.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>  21.500000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>  43.125000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>50%</th>\n",
        "      <td> 1100.500000</td>\n",
        "      <td>   3.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   1.000000</td>\n",
        "      <td>  25.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>  60.500000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>75%</th>\n",
        "      <td> 1204.750000</td>\n",
        "      <td>   3.000000</td>\n",
        "      <td>   1.000000</td>\n",
        "      <td>   0.000000</td>\n",
        "      <td>   1.000000</td>\n",
        "      <td>  36.000000</td>\n",
        "      <td>   1.000000</td>\n",
        "      <td>  75.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>max</th>\n",
        "      <td> 1309.000000</td>\n",
        "      <td>   3.000000</td>\n",
        "      <td>   8.000000</td>\n",
        "      <td>   9.000000</td>\n",
        "      <td>   1.000000</td>\n",
        "      <td>  76.000000</td>\n",
        "      <td>  10.000000</td>\n",
        "      <td> 181.500000</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 117,
       "text": [
        "       PassengerId      Pclass       SibSp       Parch      Gender  \\\n",
        "count   418.000000  418.000000  418.000000  418.000000  418.000000   \n",
        "mean   1100.500000    2.265550    0.447368    0.392344    0.636364   \n",
        "std     120.810458    0.841838    0.896760    0.981429    0.481622   \n",
        "min     892.000000    1.000000    0.000000    0.000000    0.000000   \n",
        "25%     996.250000    1.000000    0.000000    0.000000    0.000000   \n",
        "50%    1100.500000    3.000000    0.000000    0.000000    1.000000   \n",
        "75%    1204.750000    3.000000    1.000000    0.000000    1.000000   \n",
        "max    1309.000000    3.000000    8.000000    9.000000    1.000000   \n",
        "\n",
        "          AgeFill  FamilySize   Age*Class  \n",
        "count  418.000000  418.000000  418.000000  \n",
        "mean    29.357656    0.839713   60.799234  \n",
        "std     12.957412    1.519072   26.602739  \n",
        "min      0.170000    0.000000    0.510000  \n",
        "25%     21.500000    0.000000   43.125000  \n",
        "50%     25.000000    0.000000   60.500000  \n",
        "75%     36.000000    1.000000   75.000000  \n",
        "max     76.000000   10.000000  181.500000  "
       ]
      }
     ],
     "prompt_number": 117
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_test.info()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "<class 'pandas.core.frame.DataFrame'>\n",
        "Int64Index: 418 entries, 0 to 417\n",
        "Data columns (total 8 columns):\n",
        "PassengerId    418 non-null int64\n",
        "Pclass         418 non-null int64\n",
        "SibSp          418 non-null int64\n",
        "Parch          418 non-null int64\n",
        "Gender         418 non-null int64\n",
        "AgeFill        418 non-null float64\n",
        "FamilySize     418 non-null int64\n",
        "Age*Class      418 non-null float64\n",
        "dtypes: float64(2), int64(6)"
       ]
      }
     ],
     "prompt_number": 118
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "numpy_test=df_test.values"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 119
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "testPassengerStartIndex=892\n",
      "outputs = [[x[0]+testPassengerStartIndex,x[1]] for x in enumerate(logReg.predict(numpy_test))]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 133
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "np.savetxt('data/submission.csv', outputs, delimiter=',', fmt='%d,%d', \n",
      "            header='PassengerId,Survived', comments = '')\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 136
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Score on Kaggle:0.74163\n",
      "---"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}