nested_cv.ipynb

{
  "metadata": {
    "name": "Nested Cross-Validation"
  }, 
  "nbformat": 3, 
  "nbformat_minor": 0, 
  "worksheets": [
    {
      "cells": [
        {
          "cell_type": "heading", 
          "level": 1, 
          "metadata": {}, 
          "source": [
            "Nested Cross-Validation"
          ]
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "Often it is desired to explore multiple models (classifiers,\nparameterizations) but it becomes an easy trap for introducing an\noptimistic bias into generalization estimate.  The easiest but\ncomputationally intensive solution to overcome such a bias is to carry\nmodel selection by estimating the same (or different) performance\ncharacteristic while operating only on training data.  If such\nperformance is a cross-validation, then it leads to the so called\n\"nested cross-validation\" procedure.\n\n", 
            "This example will demonstrate on how to implement such nested\ncross-validation while selecting the best performing classifier from\nthe warehouse of available within PyMVPA."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "from mvpa2.suite import *\n# increase verbosity a bit for now\nverbose.level = 3\n# pre-seed RNG if you want to investigate the effects, thus\n# needing reproducible results\n#mvpa2.seed(3)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "For this simple example lets generate some fresh random data with 2\nrelevant features and low SNR."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "dataset = normal_feature_dataset(perlabel=24, nlabels=2, nchunks=3,\n                                 nonbogus_features=[0, 1],\n                                 nfeatures=100, snr=3.0)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "For the demonstration of model selection benefit, lets first compute\ncross-validated error using simple and popular kNN."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "clf_sample = kNN()\ncv_sample = CrossValidation(clf_sample, NFoldPartitioner())\n\nverbose(1, \"Estimating error using a sample classifier\")\nerror_sample = np.mean(cv_sample(dataset))"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "For the convenience lets define a helpful function which we will use\ntwice -- once within cross-validation, and once on the whole dataset"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "def select_best_clf(dataset_, clfs):\n    \"\"\"Select best model according to CVTE\n\n    Helper function which we will use twice -- once for proper nested\n    cross-validation, and once to see how big an optimistic bias due\n    to model selection could be if we simply provide an entire dataset.\n\n    Parameters\n    ----------\n    dataset_ : Dataset\n    clfs : list of Classifiers\n      Which classifiers to explore\n\n    Returns\n    -------\n    best_clf, best_error\n    \"\"\"\n    best_error = None\n    for clf in clfs:\n        cv = CrossValidation(clf, NFoldPartitioner())\n        # unfortunately we don't have ability to reassign clf atm\n        # cv.transerror.clf = clf\n        try:\n            error = np.mean(cv(dataset_))\n        except LearnerError, e:\n            # skip the classifier if data was not appropriate and it\n            # failed to learn/predict at all\n            continue\n        if best_error is None or error < best_error:\n            best_clf = clf\n            best_error = error\n        verbose(4, \"Classifier %s cv error=%.2f\" % (clf.descr, error))\n    verbose(3, \"Selected the best out of %i classifiers %s with error %.2f\"\n            % (len(clfs), best_clf.descr, best_error))\n    return best_clf, best_error"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "First lets select a classifier within cross-validation, thus\neliminating model-selection bias"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "best_clfs = {}\nconfusion = ConfusionMatrix()\nverbose(1, \"Estimating error using nested CV for model selection\")\npartitioner = NFoldPartitioner()\nsplitter = Splitter('partitions')\nfor isplit, partitions in enumerate(partitioner.generate(dataset)):\n    verbose(2, \"Processing split #%i\" % isplit)\n    dstrain, dstest = list(splitter.generate(partitions))\n    best_clf, best_error = select_best_clf(dstrain, clfswh['!gnpp'])\n    best_clfs[best_clf.descr] = best_clfs.get(best_clf.descr, 0) + 1\n    # now that we have the best classifier, lets assess its transfer\n    # to the testing dataset while training on entire training\n    tm = TransferMeasure(best_clf, splitter,\n                         postproc=BinaryFxNode(mean_mismatch_error,\n                                               space='targets'),\n                         enable_ca=['stats'])\n    tm(partitions)\n    confusion += tm.ca.stats"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "And for comparison, lets assess what would be the best performance if\nwe simply explore all available classifiers, providing all the data at\nonce"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "verbose(1, \"Estimating error via fishing expedition (best clf on entire dataset)\")\ncheating_clf, cheating_error = select_best_clf(dataset, clfswh['!gnpp'])\n\nprint \"\"\"Errors:\n sample classifier (kNN): %.2f\n model selection within cross-validation: %.2f\n model selection via fishing expedition: %.2f with %s\n \"\"\" % (error_sample, 1 - confusion.stats['ACC'],\n        cheating_error, cheating_clf.descr)\n\nprint \"# of times following classifiers were selected within \" \\\n      \"nested cross-validation:\"\nfor c, count in sorted(best_clfs.items(), key=lambda x:x[1], reverse=True):\n    print \" %i times %s\" % (count, c)\n\nprint \"\\nConfusion table for the nested cross-validation results:\"\nprint confusion"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }
      ], 
      "metadata": {}
    }
  ]
}