3-Cross-Validation-Intro-v1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Creative Commons CC BY 4.0 Lynd Bacon & Associates, Ltd. Not warranted to be suitable for any particular purpose. (You're on your own!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><h1>Cross-Validation</h1></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Cross-Validation: What is It?\n",
    "\n",
    "A procedure consisting of randomly splitting data into parts, using some parts to _train_ an ML algorithm, and the other parts to _test_ how well the algorithm performs on new data, data it \"hasn't seen\" while learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Cross-Validation:  Why Do it?\n",
    "\n",
    "An important ML goal is to train models that _generalize_ well, that perform well on _future data_.   Most ML methods are prone to _over-fitting_ the data used to train them.   \n",
    "\n",
    "This over-fitting has been referred to by some as \"learning the data,\" rather than learning _patterns_ in the data. Models that are over-fit will perform poorly on new data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## ML Validation Flavors\n",
    "\n",
    "(See [James et al. 2017, Chapter 5](http://www-bcf.usc.edu/~gareth/ISL/))\n",
    "\n",
    "* __Validation Set Method__: randomly divide available data in to a _training set_ and a _test set_. But,\n",
    "    * test data performance measures may be highly variable due to small N\n",
    "    * may result in overestimation of test error rate, again, due to small N\n",
    "    * Not really a _cross_ validation technique\n",
    "* __\"Leave One Out\"__ cross-validation: Each observation is used as validation data with training on all the other observations.\n",
    "    * test performance measures tend to be highly variable\n",
    "* __K-fold__ cross-validation:  data is randomly split into \"folds\" of similar size. Each fold is used as test data after training on the observations in in the fold. \n",
    "    * Perhaps more biased test performance measures than Leave One Out, but possibly less variable ones.\n",
    "    * A very frequently used \"compromise.\"\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Bias-Variance Trade-off</h2>\n",
    "\n",
    "(Based on [James et al. 2017, Chapter 2](http://www-bcf.usc.edu/~gareth/ISL/))\n",
    "\n",
    "For a regression-type ML \"learner,\" the expected MSE for some new data point x<sub>0</sub> is  \n",
    "\n",
    "\\begin{align*}\n",
    "\\large\n",
    "E{(y_0 - \\hat{g}(x_0))}^2 = Var(\\hat{g}(x_0))+[Bias(\\hat{g}(x_0))]^2+Var(\\epsilon)\n",
    "\\end{align*}\n",
    "\n",
    "Where:\n",
    " \n",
    "Var($\\hat{g}(x_0)$) refers to the amount by which $\\hat{g}(x_0)$ would change when training is done using different data sets.  Different training datasets will result in different $\\hat{g}(x_0)$.  That is, the model will vary with different datasets.\n",
    "\n",
    "_Bias_ refers to error in approximating the \"true\" g(x<sub>i</sub>), e.g. the error in approximating a nonlinear relationship with a linear one.\n",
    "\n",
    "Var($\\epsilon$) is \"noise\" that's independent of the model and the data. (From the Big Bang?) It's the lower limit on the MSE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Best Possible Fit in Test Data\n",
    "\n",
    "To minimize _expected_ error when predicting new (or test) data, a ML algorithm needs to minimize _both_ bias and variance.   \n",
    "\n",
    "Heuristics:\n",
    "\n",
    "* Using more \"flexible\" methods (i.e.., more complicated models) will tend to result in _increased_ variance, and _decreased_ bias.\n",
    "* At some point, increases in flexibility will have little additional effect on bias, but variance will _continue_ to _increase_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Regression Models for Three Different Datasets\n",
    "\n",
    "Examples of MSE, bias, and variance for regression models trained on three data sets from [James et at. (2017, Chapter 5)](http://www-bcf.usc.edu/~gareth/ISL/):\n",
    "\n",
    "<img src=\"./images/bias-variance-james-2017.png\" alt=\"bias-variance trade-off examples\" width=\"1500\"/>\n",
    "\n",
    "\n",
    "<h1>There's No Free Lunch!</h1> \n",
    "\n",
    "* David Wolpert(1996): \"...averaged over all possible data-generating distributions, every classification algorithm same error rate when classifying previously unobserved points.\"  (Goodfellow et al., 2016, p. 113.)\n",
    "    * The bottom line: No _machine learning algorithm is universally better than any other one._\n",
    "    * Wolpert's [NASA presentation](http://no-free-lunch.org/coev.pdf)\n",
    "* _Two_ [Free Lunch Theorems](http://no-free-lunch.org/)\n",
    "    * w.r.t. Wolpert (1996) \"noise-free scenario where the loss function is the misclassification rate, if one is interested in off-training-set error, then there are no a priori distinctions between learning algorithms.\"\n",
    "    \n",
    "* Bottom Line:  There is no _best_ algorithm.  (Also, no _best regularization_.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Bias,Variance, and Model Fit Bullseyes!</h2>\n",
    "\n",
    "[Bias, variance, algorithm overfitting, and algorithm underfitting](https://medium.com/@martinezbielosdaniel/bias-variance-tradeoff-overfitting-and-underfitting-c63799cb4851) are in general related.  Here's a rather famous \"Bullseye\" respresentation:\n",
    "\n",
    "\n",
    "<img src=\"./images/bias-variance-bullseyes.jpeg\" alt=\"bias-variance bullseyes\" width=\"400\"/>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "384px"
   },
   "toc_section_display": false,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}