notebooks/19.Feature_Selection.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true,
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Automatic Feature Selection\n",
    "Often we collected many features that might be related to a supervised prediction task, but we don't know which of them are actually predictive. To improve interpretability, and sometimes also generalization performance, we can use automatic feature selection to select a subset of the original features. There are several types of feature selection methods available, which we'll explain in order of increasing complexity.\n",
    "\n",
    "For a given supervised model, the best feature selection strategy would be to try out each possible subset of the features, and evaluate generalization performance using this subset. However, there are exponentially many subsets of features, so this exhaustive search is generally infeasible. The strategies discussed below can be thought of as proxies for this infeasible computation.\n",
    "\n",
    "### Univariate statistics\n",
    "The simplest method to select features is using univariate statistics, that is by looking at each feature individually and running a statistical test to see whether it is related to the target. This kind of test is also known as analysis of variance (ANOVA).\n",
    "\n",
    "We create a synthetic dataset that consists of the breast cancer data with an additional 50 completely random features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_breast_cancer, load_digits\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "cancer = load_breast_cancer()\n",
    "\n",
    "# get deterministic random numbers\n",
    "rng = np.random.RandomState(42)\n",
    "noise = rng.normal(size=(len(cancer.data), 50))\n",
    "# add noise features to the data\n",
    "# the first 30 features are from the dataset, the next 50 are noise\n",
    "X_w_noise = np.hstack([cancer.data, noise])\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X_w_noise, cancer.target,\n",
    "                                                    random_state=0, test_size=.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We have to define a threshold on the p-value of the statistical test to decide how many features to keep. There are several strategies implemented in scikit-learn, a straight-forward one being ``SelectPercentile``, which selects a percentile of the original features (we select 50% below):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import SelectPercentile\n",
    "\n",
    "# use f_classif (the default) and SelectPercentile to select 50% of features:\n",
    "select = SelectPercentile(percentile=50)\n",
    "select.fit(X_train, y_train)\n",
    "# transform training set:\n",
    "X_train_selected = select.transform(X_train)\n",
    "\n",
    "print(X_train.shape)\n",
    "print(X_train_selected.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We can also use the test statistic directly to see how relevant each feature is. As the breast cancer dataset is a classification task, we use f_classif, the F-test for classification. Below we plot the p-values associated with each of the 80 features (30 original features + 50 noise features). Low p-values indicate informative features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import f_classif, f_regression, chi2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "F, p = f_classif(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "plt.plot(p, 'o')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Clearly most of the first 30 features have very small p-values.\n",
    "\n",
    "Going back to the SelectPercentile transformer, we can obtain the features that are selected using the ``get_support`` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "mask = select.get_support()\n",
    "print(mask)\n",
    "# visualize the mask. black is True, white is False\n",
    "plt.matshow(mask.reshape(1, -1), cmap='gray_r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Nearly all of the original 30 features were recovered.\n",
    "We can also analize the utility of the feature selection by training a supervised model on the data.\n",
    "It's important to learn the feature selection only on the training set!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "# transform test data:\n",
    "X_test_selected = select.transform(X_test)\n",
    "\n",
    "lr = LogisticRegression()\n",
    "lr.fit(X_train, y_train)\n",
    "print(\"Score with all features: %f\" % lr.score(X_test, y_test))\n",
    "lr.fit(X_train_selected, y_train)\n",
    "print(\"Score with only selected features: %f\" % lr.score(X_test_selected, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Model-based Feature Selection\n",
    "A somewhat more sophisticated method for feature selection is using a supervised machine learning model and selecting features based on how important they were deemed by the model. This requires the model to provide some way to rank the features by importance. This can be done for all tree-based models (which implement ``get_feature_importances``) and all linear models, for which the coefficients can be used to determine how much influence a feature has on the outcome.\n",
    "\n",
    "Any of these models can be made into a transformer that does feature selection by wrapping it with the ``SelectFromModel`` class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import SelectFromModel\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "select = SelectFromModel(RandomForestClassifier(n_estimators=100, random_state=42), threshold=\"median\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "select.fit(X_train, y_train)\n",
    "X_train_rf = select.transform(X_train)\n",
    "print(X_train.shape)\n",
    "print(X_train_rf.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "mask = select.get_support()\n",
    "# visualize the mask. black is True, white is False\n",
    "plt.matshow(mask.reshape(1, -1), cmap='gray_r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "X_test_rf = select.transform(X_test)\n",
    "LogisticRegression().fit(X_train_rf, y_train).score(X_test_rf, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This method builds a single model (in this case a random forest) and uses the feature importances from this model.\n",
    "We can do a somewhat more elaborate search by training multiple models on subsets of the data. One particular strategy is recursive feature elimination:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Recursive Feature Elimination\n",
    "Recursive feature elimination builds a model on the full set of features, and similar to the method above selects a subset of features that are deemed most important by the model. However, usually only a single feature is dropped from the dataset, and a new model is built with the remaining features. The process of dropping features and model building is repeated until there are only a pre-specified number of features left:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import RFE\n",
    "select = RFE(RandomForestClassifier(n_estimators=100, random_state=42), n_features_to_select=40)\n",
    "\n",
    "select.fit(X_train, y_train)\n",
    "# visualize the selected features:\n",
    "mask = select.get_support()\n",
    "plt.matshow(mask.reshape(1, -1), cmap='gray_r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "X_train_rfe = select.transform(X_train)\n",
    "X_test_rfe = select.transform(X_test)\n",
    "\n",
    "LogisticRegression().fit(X_train_rfe, y_train).score(X_test_rfe, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "select.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    <b>EXERCISE</b>:\n",
    "     <ul>\n",
    "      <li>\n",
    "      Create the \"XOR\" dataset as in the first cell below:\n",
    "      </li>\n",
    "      <li>\n",
    "      Add random features to it and compare how univariate selection compares to model based selection using a Random Forest in recovering the original features.\n",
    "      </li>\n",
    "    </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "rng = np.random.RandomState(1)\n",
    "\n",
    "# Generate 400 random integers in the range [0, 1]\n",
    "X = rng.randint(0, 2, (200, 2))\n",
    "y = np.logical_xor(X[:, 0] > 0, X[:, 1] > 0)  # XOR creation\n",
    "\n",
    "plt.scatter(X[:, 0], X[:, 1], c=plt.cm.spectral(y.astype(float)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# %load solutions/19_univariate_vs_mb_selection.py"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}