regression_airline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Airline flight delay dataset\n",
    "\n",
    "This notebook looks to assess the uncertainty estimates of Mondrian Forests on a (albeit simple) regression problem.  The data consists of flight delays, with 8 features as predictors.  This is briefly described in [the paper](http://www.gatsby.ucl.ac.uk/~balaji/mfregression_aistats16.pdf).\n",
    "\n",
    "I find that the MSE that models report below agree qutie well with the paper.  The main goal was to assess uncertainty estimates in a more interpretible way than presented in the paper.  I find that, indeed, the uncertainty estimates for this particular problem do align well with expection from Gaussian statistics.\n",
    "\n",
    "## Load the flight data\n",
    "\n",
    "This data was originally cleaned by by James Hensman, and lives in a cryptic site mirror thanks to Neil Lawrence (yay semi-open science! (not actually sarcastic)).  The data is a pickled pandas DF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# this pickle file can be found as 'filtered_data.pickle' at\n",
    "# this url: http://staffwww.dcs.shef.ac.uk/people/N.Lawrence/dataset_mirror/airline_delay/\n",
    "data = pd.read_pickle('./process_data/airline-delays/airline-delays.p')\n",
    "\n",
    "num_train = 7000\n",
    "num_test = 1000\n",
    "\n",
    "# WARNING: removing year\n",
    "data.pop('Year')\n",
    "\n",
    "# Get data matrices\n",
    "Yall = data.pop('ArrDelay').values[:,None]\n",
    "Xall = data.values\n",
    "\n",
    "# Subset the data (memory!!)\n",
    "all_data = num_train+num_test\n",
    "Xall = Xall[:all_data]\n",
    "Yall = Yall[:all_data]\n",
    "\n",
    "np.random.seed(seed=1234)\n",
    "N_shuffled = np.random.permutation(Yall.shape[0])\n",
    "train, test = N_shuffled[num_test:], N_shuffled[:num_test]\n",
    "\n",
    "X_train, X_test, y_train, y_test = Xall[train], Xall[test], Yall[train], Yall[test]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This next cell creates a dict that the MF code plays nice with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(7000, 8)\n"
     ]
    }
   ],
   "source": [
    "def make_data(xtrain, xtest, ytrain, ytest):\n",
    "    data = {}\n",
    "    data['n_dim'] = xtrain.shape[1]\n",
    "    data['x_train'] = xtrain\n",
    "    data['y_train'] = ytrain.ravel()\n",
    "    data['n_train'] = xtrain.shape[0]\n",
    "    data['x_test'] = xtest\n",
    "    data['y_test'] = ytest.ravel()\n",
    "    data['n_test'] = xtest.shape[0]\n",
    "    data['n_class'] = 1\n",
    "    data['is_sparse'] = 0\n",
    "    data['train_ids_partition'] = {}\n",
    "    \n",
    "    data['train_ids_partition']['current'] = {0:np.arange(xtrain.shape[0])}\n",
    "    data['train_ids_partition']['cumulative'] = {0:np.arange(xtrain.shape[0])}\n",
    "    return data\n",
    "data = make_data(X_train, X_test, y_train, y_test)\n",
    "print X_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Models for comparison\n",
    "\n",
    "For benchmarking purposes, run RF and ERT models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from time import time\n",
    "from sklearn.ensemble import RandomForestRegressor, ExtraTreesRegressor\n",
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "np.random.seed(12345)\n",
    "def run_regressor(rgr, data):\n",
    "    \"\"\"\n",
    "    Run a sklearn regressor, return time and score\n",
    "    \"\"\"\n",
    "    t0 = time()\n",
    "    rgr.fit(data['x_train'].copy(), data['y_train'].copy())\n",
    "    score = mean_squared_error(rgr.predict(data['x_test'].copy()), data['y_test'].copy())\n",
    "    run_time = time()-t0\n",
    "    return run_time, score\n",
    "\n",
    "rf_time, rf_score = run_regressor(RandomForestRegressor(10, min_samples_leaf=5), data)\n",
    "et_time, et_score = run_regressor(ExtraTreesRegressor(10, min_samples_leaf=5), data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Punking the command line\n",
    "\n",
    "I do not personally enjoy the command line interface of the MF code.  Below is a class containing the settings necessary to run the code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class ParamSettings(object):\n",
    "    def __init__(self):\n",
    "        self.dataset = 'toy'\n",
    "        self.normalize_features = 1\n",
    "        self.select_features = 0\n",
    "        self.optype = 'real'\n",
    "        self.data_path = './process_data/'\n",
    "        self.debug = 0\n",
    "        self.op_dir = 'results'\n",
    "        self.tag = ''\n",
    "        self.save = 0\n",
    "        self.verbose = 1\n",
    "        self.init_id = 1\n",
    "        self.n_mondrians = 10\n",
    "        self.budget = -1 # -1 sets lifetime to inf\n",
    "        self.discount_factor = 10\n",
    "        self.n_minibatches = 1\n",
    "        self.draw_mondrian = 0\n",
    "        self.smooth_hierarchically = 0 # the code doesnt seem to play nice for regression with this on\n",
    "        self.store_every = 0\n",
    "        self.bagging = 0\n",
    "        self.min_samples_split = 10 # this is the rec in the paper\n",
    "        self.name_metric = 'mse'\n",
    "        \n",
    "        if self.optype == 'class':\n",
    "            self.alpha = 0    # normalized stable prior\n",
    "            assert self.smooth_hierarchically\n",
    "            \n",
    "        if self.budget < 0:\n",
    "            self.budget_to_use = np.inf\n",
    "        else:\n",
    "            self.budget_to_use = settings.budget\n",
    "        \n",
    "settings = ParamSettings()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "minibatch\tmetric_train\tmetric_test\tnum_leaves\n",
      "log_prob (using Gaussian approximation) = -4.118255\n",
      "log_prob (using mixture of Gaussians) = -3.558153\n",
      "log_prob (using Gaussian approximation) = -4.730813\n",
      "log_prob (using mixture of Gaussians) = -4.311993\n",
      "        0\t505.538\t\t633.025\t\t1479.700\n"
     ]
    }
   ],
   "source": [
    "from src.mondrianforest_utils import load_data, reset_random_seed, precompute_minimal\n",
    "from src.mondrianforest import process_command_line, MondrianForest\n",
    "np.random.seed(12345)\n",
    "\n",
    "# data loading and cache\n",
    "param, cache = precompute_minimal(data, settings)\n",
    "\n",
    "mf = MondrianForest(settings, data)\n",
    "print '\\nminibatch\\tmetric_train\\tmetric_test\\tnum_leaves'\n",
    "t0=time()\n",
    "\n",
    "# loop over minibatches\n",
    "for idx_minibatch in range(settings.n_minibatches):\n",
    "    train_ids_current_minibatch = data['train_ids_partition']['current'][idx_minibatch]\n",
    "    if idx_minibatch == 0:\n",
    "        # Batch training for first minibatch\n",
    "        mf.fit(data, train_ids_current_minibatch, settings, param, cache)\n",
    "    else:\n",
    "        # Online update\n",
    "        mf.partial_fit(data, train_ids_current_minibatch, settings, param, cache)\n",
    "\n",
    "    # produce predictions\n",
    "    weights_prediction = np.ones(settings.n_mondrians) * 1.0 / settings.n_mondrians\n",
    "    train_ids_cumulative = data['train_ids_partition']['cumulative'][idx_minibatch]\n",
    "    pred_forest_train, metrics_train = \\\n",
    "        mf.evaluate_predictions(data, data['x_train'][train_ids_cumulative, :], \\\n",
    "        data['y_train'][train_ids_cumulative], \\\n",
    "        settings, param, weights_prediction, False)\n",
    "    pred_forest_test, metrics_test = \\\n",
    "        mf.evaluate_predictions(data, data['x_test'], data['y_test'], \\\n",
    "        settings, param, weights_prediction, False)\n",
    "    name_metric = settings.name_metric     # acc or mse\n",
    "    metric_train = metrics_train[name_metric]\n",
    "    metric_test = metrics_test[name_metric]\n",
    "    tree_numleaves = np.zeros(settings.n_mondrians)\n",
    "    for i_t, tree in enumerate(mf.forest):\n",
    "        tree_numleaves[i_t] = len(tree.leaf_nodes)\n",
    "    forest_numleaves = np.mean(tree_numleaves)\n",
    "    print '%9d\\t%.3f\\t\\t%.3f\\t\\t%.3f' % (idx_minibatch, metric_train, metric_test, forest_numleaves)\n",
    "mf_time = time() - t0\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prediction summary\n",
    "\n",
    "Below here we compare the point estimate predictions, and the time taken to produce them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RF time, rmse = 0.30, 23.97\n",
      "ET time, rmse = 0.11, 24.02\n",
      "MF time, rmse = 7.87, 25.16\n"
     ]
    }
   ],
   "source": [
    "print 'RF time, rmse = %0.2f, %0.2f' % (rf_time, np.sqrt(rf_score))\n",
    "print 'ET time, rmse = %0.2f, %0.2f' % (et_time, np.sqrt(et_score))\n",
    "print 'MF time, rmse = %0.2f, %0.2f' % (mf_time, np.sqrt(metric_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Uncertainty estimates\n",
    "\n",
    "Below here we test if the uncertainty estimates are reasonable. Indeed they are, falling close to the expected ideal for a Gaussian distribution (2sigma is right on the money). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percentage within 1sigma = 0.80\n",
      "Percentage within 2sigma = 0.95\n"
     ]
    }
   ],
   "source": [
    "chi = np.abs(data['y_test'].ravel() - pred_forest_test['pred_mean']) / np.sqrt(pred_forest_test['pred_var']) \n",
    "\n",
    "print 'Percentage within 1sigma = %0.2f' % (1. * np.where(chi < 1)[0].size / data['y_test'].size)\n",
    "print 'Percentage within 2sigma = %0.2f' % (1. * np.where(chi < 2)[0].size / data['y_test'].size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}