2016-06-27-data-science-intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "jupyter nbconvert --to slides --reveal-prefix \"http://cdn.jsdelivr.net/reveal.js/2.6.2\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import pods\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Introduction to Machine Learning and Data Science\n",
    "\n",
    "### Neil D. Lawrence\n",
    "\n",
    "### 27th June 2016"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Text\n",
    "\n",
    "<img src=\"diagrams/a-first-course-in-machine-learning.jpg\" height=100% align=center>\n",
    "\n",
    "@Rogers:book11\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Another Text\n",
    "\n",
    "\n",
    "![](diagrams/978-0-387-31073-2.png)\n",
    "\n",
    "@Bishop:book06\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What is Machine Learning?\n",
    "\n",
    "\n",
    "**<center>data + model = prediction</center>**\n",
    "\n",
    " -   **data** : observations, could be actively or passively\n",
    "    acquired (meta-data).\n",
    "\n",
    " -   **model** : assumptions, based on previous experience (other data!\n",
    "    transfer learning etc), or beliefs about the regularities of\n",
    "    the universe. Inductive bias.\n",
    "\n",
    " -   **prediction** : an action to be taken or a categorization or a\n",
    "    quality score.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Fitting Data\n",
    "\n",
    "- **data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Create some data\n",
    "x = np.array([1, 3])\n",
    "y = np.array([3, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **model**\n",
    "$$y=mx + c$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Model Fitting\n",
    "\n",
    "$$m = \\frac{y_2- y_1}{x_2-x_1}$$\n",
    "$$ c = y_1 - m x_1 $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "xvals = np.linspace(0, 5, 2);\n",
    "\n",
    "m = (y[1]-y[0])/(x[1]-x[0]);\n",
    "c = y[0]-m*x[0];\n",
    "\n",
    "yvals = m*xvals+c;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVwAAAFaCAYAAABST+UxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcVOWV//HP6UUWkRA0w+KIYKImGlpAjWwCGnWigpEB\nlIlhcSE6yi/GhKCgMZjEfRk1GhONIKiDgIhRo44RaIFINGy2ijFkUEBUMgiGYANC9/n98XTTW3V3\ndXdV3Vq+79erXl1V9/at57Zy+vRzzz2PuTsiIpJ8eVEPQEQkVyjgioikiAKuiEiKKOCKiKSIAq6I\nSIoo4IqIpEhB1ANoCjN7H9gBlAF73f0b0Y5IRCR+GRVwAQeGuPu2qAciItJUmTilYFEPQESkOTIt\n4DrwspmtMLMJUQ9GRKQpMm1KYYC7f2RmXwL+YGZ/cfelUQ9KRCQeGRVw3f2jiq//Z2YLgG8ASwHM\nTE0hRCQp3D0hU5kZM6VgZm3N7KCK5wcCZwBvVt/H3bP28dOf/jTyMej8dH65eH6JlEkZbidggZlB\nGPfj7v5StEMSEYlfxgRcd38P6BX1OEREmitjphTisW5d1CNIniFDhkQ9hKTS+WW2bD+/RLFEz1FE\nxcy8qMhZvhzato16NCKSLcwMz7WLZvEoKYErroAs+R0iIlkmqwJumzbwyCPw8MNRj0REpK6sCrgP\nPBC+TpwIq1ZFOxYRkdqyKuCOGwcTJsCePTByJGzfHvWIRESqZNVFM3dn924YMCBkuMOGwdNPQ15W\n/VoRkVTSRbMGtG4NTz4JHTrAs8/C7bdHPSIRkSDrMtxKzz0XMty8PFi4EFQmKCLNoQw3DkOHwpQp\nUF4Oo0fDRx9FPSIRyXVZm+EC7NsHZ5wBixfDySeHTLewMKIBikhGUoYbp4ICmD0bunSBpUth6tSo\nRyQiuSyrM9xKy5aFOdyyMnjqKRg+PLVjE5HMpQy3iQYOhNtuC8/Hj8/uJjcikr5yIsOF0F9h1CiY\nPx+KilCTGxGJizLcZjCD6dPhyCPV5EZEopEzARegffuQ4arJjYhEIacCLkDPnmpyIyLRyLmAC2py\nIyLRyJmLZrWpyY2IxEMXzRJATW5EJNVyNsOtpCY3ItIQZbgJpCY3IpIqOZ/hgprciEj9lOEmmJrc\niEgqKMOtRk1uRKQ2ZbhJoiY3Itln0KBB/OEPf4h6GIACbh1XXQUjRsCOHeGmiNLSqEckIi2xadMm\nPkqTq+EKuLWoyY2IJIsCbgxqciMiyaCAWw81uRGRRFPAbYCa3Ei8PvzwQ0aOHEn//v056qijePzx\nx2Put3z5cv7lX/6FG2+8McUjrOvBBx9kwIABHHvssVx77bWUl5cDsG7dOi677DIGDx5M//79KSoq\n4q677tq/fefOndx44430799///d///vfZ8eOHQ1+3syZMxkyZAgDBw6kqKiIX/7ylwDs2rWLiRMn\n0q9fPwYPHsyYMWPYunVrck++mcys0MwuMbOFZvZnMys2s6fN7Mdm1nj1vrtnxSOcSuLt2uXep487\nuA8b5l5WlpSPkQy2Z88eHzhwoK9Zs8bd3c8991wvKCjwd955p86+48aNczPzKVOmpHqYNSxdutTP\nPfdcd3efO3eum5nffPPN/t///d9+5plneklJyf5977nnHjcz//GPf+x//etffeDAgT5r1qz92994\n4w03M//3f//3ej/v4osv9kmTJnlpaam7uy9btszNzG+66SY/++yzffbs2e7u/pvf/MYPOuggHz9+\nfMLOtXv37j5z5sxmf39FbAH4KrAWeBs4watiz0XAB8Dt3licamyHTHkkK+C6u69f796hQ/hp3XJL\n0j5GMtRNN93kv/zlL/e/7tu3r5uZz5s3r86+hx9+uJuZv/jiizGP9e677/pRRx3l3bt3T8jj17/+\ndczPGT58uC9btszd3efNm+dm5l27dvWRI0f6vn37auz7zjvvuJn5oYce6n379o35i6RTp05eWFjo\ne/bsqbPtvvvu8x/+8Id13j/44IM9Ly/PL730Und33759u+fl5bmZ+dixY2OOuzkSEXCBwcCnwCbg\nYK+KO+2BfUA58LQ3Fqca2yFTHskMuO7uzz4bflp5ee6LFyf1oySD7Nu3z4855hj//PPP3d39gw8+\n8Ly8PC8oKPDNmzfX2Pe9995zM/MDDjjAd+7cGcVw3d19165d/uUvf3n/6+uvv97NzLt37x5zXK+9\n9pqbmefl5fnzzz8f85gHHnig5+Xl+SeffFLj/dLSUj/mmGN8165ddd7Pz8/3vLw8f/PNN93dvays\nzC+44AI/99xzfdOmTS09zf0SFHD/DpQBg71m3CkE1lQE4oHeSJwqaOmcRq6obHJz882hyc3q1eFW\nYMltGzZsYOzYsRRWNN+YMWMG7s7QoUPp2rVrjX0XL14MQJ8+fTjwwANTPtZKH330ERMmTNj/unJc\nU6dOjTmulStXAnDKKadw5pln1tm+YcMGSktLad++PR07dqyxbd26dUycOJHWrVvXeH/VqlWUl5fT\ntWtXvv71rwOQl5fHY4891rKTq0dFcGyJQ4AX3f2VWsfdC/SK9yAKuE3ws5/Bn/4Umtycf76a3Agc\nccQRXH311ftfT58+HYDLL7+8zr7FxcUADIm4B2iPHj32j/mzzz7jT3/6E2bGGWecEXP/yoB82mmn\nxdy+aNEiINzRVVtRURFFRUV13n/55ZcB+OY3v9n0E6iwadMm/u3f/o1du3Y1uN8HH3zApEmTmDZt\nWoP7XXjhhVx//fU13luxYkXlUwfuafZg9x8lDaYDEvEgyVMKlT7+2L1LlzC9MGlSSj5SMsTy5cvd\nzPzwww+Pub1bt24Nzt9G4YUXXnAzqzHFUF15ebl/6Utf8ry8PH/99ddj7jN06FA3M3/wwQfj/twB\nAwa4mdW4+JYsLZlSuOaaayqnFHYChd7COKWysCbq1AnmzoX8fLjjDliwIOoRSbqYN28eAMNjdD1a\nv349mzZtorCwkIEDB6Z6aPVauHAhUH+mWVJSwtatW/nCF77ACSecUGf7p59+yksvvURBQUHM845l\nx44dvPbaa5hZizLcVHjrrbcqn77uYfqgRTSl0AyVTW5+9KPQ5ObrXw+3AktuW758ORDmOmuLd/72\nr3/9K8OGDePzzz9PyJiuueYaLr300nq3VwbcU089tcHtgwcPxqxuw6w5c+awd+9ezj77bA455JD9\n73Xr1o1+/frFPObixYspKyvj6KOPrjPPvXfvXqZOncrtabLmVWXtMbAxEcdTwG2mq66CV18NtwCP\nHAnLl0PbtlGPSqK0detWzIwvf/nLdbbFO3971FFH8e677yZhdHV98sknrFmzBjNrNODWl4k+8cQT\nAIwdO3b/e/feey/PPPMMAL/73e+4//77ufTSSxkxYgQAL774IgB9+/atc7zf/e53lJWVNfOMEu9r\nX/sazz//PMDuhvYzs+OAye5+QYMHbOmcRLo8SNEcbnX/+If7kUeG+dzx493Ly1M+BEkjp556qpvZ\n/hsgKr3xxhvepk2btJu/ray/LSoqirn9888/93bt2nleXp6//fbbMfc56KCDvFWrVvtvaFi8eLGP\nGTPG3d0/++yz/ec9YsQIdw+1tgcffLCbmU+qdRHkk08+8d69e/sHH3yQqFN095bN4ZaUlFTO4a7w\n+mNPf6AE+Fp9++zft7EdMuURRcB1dy8pcW/TJvwkH3ookiFImpg/f76bmX//+9/f/97ChQv9sMMO\nS4v629ouu+wyNzP/wQ9+EHN75d1gXbp0qfcYAwYM8IMPPtjd3T/++GPv37+/f/jhh+7uvnPnTm/b\ntq0XFRX5O++846Wlpf7tb3/bZ86c6ccee6z36tXLd+/e7e7u69ev98GDB/vChQsTfJYJq8MtByZ6\nzZhzPPAb4AXgCI8nTsWzUyY8ogq47u6PPBJ+kq1aua9cGdkwJA3MmTPHTzzxRO/Vq5efcMIJfsEF\nF/gtt9ziZuZ9+/aNeng1XHXVVX7ooYf6W2+9FXP70qVLvWPHjn7nnXfWe4y//e1vPnjwYO/du7ef\neuqpvmrVqhrbX3zxRR88eLAPGjTI+/Xr508++aS7u2/ZssXHjBnjxx57rA8cONDPOeecOn8ZJEqC\nAu5ZwMvAG8ASYCnwILVuhGjsEXmgTNQjyoDr7j5hQvhp9ujhvm1bpEORNDN69Gg3M79F94VHIlG9\nFBLxUFlYgtx7L/TpA++9F7qMVV3clGy3Y8cORo8ezRlnnME///nPGts++eQTFixYQNu2bRk3blxE\nI5R0oYCbIK1bw5NPQocO8OyzkCZVLZIC999/P3PnzmXhwoW89957Nbb94he/4PPPP+f666+nc+fO\nEY0wt3Xr1q1O+VlUMirgmlm+ma02s2ejHkssPXrAo4+G51OnQkUlkGS5zZs3U1BQwLhx42rcxlpc\nXMyvfvUrzj//fCZPnhzhCHPbK6+8Uu9tyamWUQEXuJLQjzJtVxmrbHJTXh6a3KTJ2nWSROeeey69\nevXijjvu2P/e888/z8iRI/nP//xPHq38LSw5L2NufDCzfyVcKbwR+GGsfdzDIpBRU5Ob3HLaaaex\nfft2hg0bRn5+PmbGYYcdxu9//3tOOumkqIcnacTCRbj0Z2bzgJsIDX8nufuwWtv92mudn/88PYLu\nli3Qu3fIcCdN0pyuSKYyM9w9IVElI6YUzGwo8Hd3Xw3Ue+I33gg/+Ul6LGuuJjciUltGZLhmdhMw\nhrCURWtCljvf3cdW28fhpwCcfDLccMMQTjllSASjremuu0KTm/btYcUKNbkRSXfFxcX7e18A3HDD\nDQnLcDMi4FZnZoOpZ0qh+rW0a68lLaYX3GHUqNDkpqhITW5EMk3OTSnEEPO3xNChVc/TZXrBDKZP\nD5ltSQlccUX0YxKRaGRchlsfM/Pdu52RI+G556reT5dM98034aSTYNcueOghuOSSaMcjIvFRhluP\nVq3C3V7pmOn27AkPPBCeT5wIq1ZFOx4RSb2sCriQ3kF33DiYMAH27AlNy7dvj3Y8IpJaWTWlUP1c\nKoNauk0v7N4NAwaEDHfYMHj6acjLul97ItlDUwpxSNdMV01uRHJX1ma4ldI1033uuZDh5uWFW38b\nWepKRCKiDLcJ0jXTVZMbkdyT9RlupXTMdPftgzPOCE1uTj5ZTW5E0pEy3GZIx0y3oABmz4YuXWDp\n0tBDV0SyV85kuJXSMdNdtizM4ZaVwVNPwfDh0YxDROpShtsC6ZjpDhwIt90Wno8fD+vWRTMOEUmu\nnMtwK6VbpqsmNyLpSRluPVaujH/fdMt01eRGJPtlVcDt2zcEzbKy+PZPt6Dbvn3IcNu0gUcegYcf\nTv0YRCR5smpKobJr44ABYfXcHj3i+950m16YOTPM5bZqBa++Cn36pH4MIhJoSqEeL70EXbvCH/8Y\n5kFnzIgvU023TFdNbkSyU1ZluO7Otm1w2WUwb154f/hwePBBOOSQxo+RTpmumtyIpAdluA3o2BHm\nzIFZs8Kc6IIFoRftiy82/r3plOmqyY1I9sm6DLe6DRtg7FhYsiS8vuKKUO/aWLlVOmW6anIjEi1l\nuHE6/HBYtAhuvTX0KLj/fjj++MbLx9Ip01WTG5HskdUZbnVr1sAFF8DataGHwbRpcM01kJ9f/zHT\nJdNVkxuR6CjDbYZevWDFCrjyyhDArrsOBg+G996r/3vSJdNVkxuR7JAzGW51f/hDqHP98ENo1w7u\nvTe8ri9rTZdMV01uRFJPGW4LnX56WLZ81CjYuRMuughGjICtW2Pvny6ZrprciGS2nMxwK7nDY4+F\nZct37IDOncPNEt/6Vuz90yHTVZMbkdRShpsgZjBmTGgWM2gQfPwxnHlmCMClpXX3T4dMV01uRDJX\nTgfcSk0pH0uHoKsmNyKZSQG3Qn4+TJ4Mr78OxxwDf/lL/d3H0iHo9uwJDzwQnk+cGG4BFpH0ltNz\nuPXZtSvcbHDPPeF1fd3H0mFO93vfg4ceCmNbuRK++MXUfK5IrkjkHK4CbgPiKR+LOuiqyY1Icumi\nWYrEUz4W9fSCmtyIZA5luHGIVT42fXqoaKgUdaarJjciyaEMN8VilY+ddVbN8rGoM101uRFJf8pw\nm6isDO68M/Ri2LsXvvrVkP0ef3zYHmWmqyY3IomnDDdCjZWPRZnpqsmNSHpThtsCDZWPRZnpqsmN\nSOIow00TbdrA3XfHXrzygAOiy3TV5EYkPSnDTZD6Fq886KBoMl01uRFJDGW4aai+xSsXLYom01WT\nG5H0o4CbQPWVj/3oRyEQpzroqsmNSHpRwE2CWN3H+vcPVQOpDrpqciOSPjSHm2S1F6+87jr485/h\n97+v2icVc7pqciPSPGpeE0O6BlyoWz7Wr1/4M3/Roqp9kh10qze5OeecMMesJjcijVPAjSGdA26l\n2t3HvvxleOONqu3JDrrvvQd9+sCnn8Itt8DVVyfnc0SyiQJuDJkQcKFu+VjnzuHiWqVkB101uRFp\nGpWFZbDa5WMffxxuB66U7AtpanIjEh1luBHasAHGjoUlS+puS2amqyY3IvHTlEIMmRhwoW73seqm\nTS7l+iFLsPlPwsaN0K1buG1t0KAW3za2ZQv07h0y3EmT1LhcpD45GXDNrDXwCtAKOAD4nbtPqbY9\nIwNupTVr4DvfgXfeCa/PZzY/5zp65G2ioLxaJC4shEMPhRNPhLlzW/SZanIj0ricDLgAZtbW3UvN\nrABYBkxy92UV2zI64EIoH5s8Ge67L7wewDIeZQw9eL/uzt27w9tvtzjTveuucCdc+/awYkW4FVhE\nquTsRTN3r1hfgQOAfGBbhMNJuDZt4Je/hN/f8Ge6spk/MpAiSpjBeOr8Ktm8OfbkbxNddVVYp23H\njjBbUbmChYgkXkYFXDPLM7M1wBZgsbuvjXpMyXDWxt/wJj0ZxVx2chAXMYMRzGcrB1fttHdv6IrT\nQmpyI5I6BVEPoCncvRzoZWZfAP7HzIa4e3Hl9mnTpu3fd8iQIQzJ1CLTjRvpyHbmcD7DeJaJ3McC\n/p3l9GM6F3EmL+7fLxEqm9ycdFJocjNgAFxySUIOLZJxiouLKS4uTsqxM2oOtzoz+wmwy93vqHid\n8XO4+11ySY3WXhvoxlhmsYTBAFzBfdzGZNpe/B347W8T9rEzZ4Y74Vq1gldfDXelieS6nJzDNbND\nzKxDxfM2wOnA6mhHlSQjR9YojD2cjSziVG5lMoV8zv1M5HhWsaLoooR+7LhxMGFC1fJA27cn9PAi\nOS9jMlwz6wnMJPySyAMedffbq23Pngy3tBSOPRbef7/OpjUcxwU8zlqOJS/PueEGY8qUsLhlIqjJ\njUhNOVsW1pCsCrgA550X+jhu3lzjjoi9Vsjf/Ah+xJ28wNlA6LX76KNwxBGJ+Wg1uRGpooAbQ9YF\nXAiZ7pIloRqh4k6zz88ZyXd+PYj5L9Ssv23XDu69N8zBJuJ2YDW5EQkUcGPIyoBbj1hLsFeqXLzy\nkENa/jlTp8LNN0OnTrB6NXTp0vJjimQaBdwYcingQuyg26pVeL9z51Bbe+aZLfsMNbkRydEqBamp\nVau6qwHv2RP621QuXjlxYsvuHCsogNmzQ2a7dGnIeEWk+ZThZrhYme6pp4YAuXcvfPWr8NhjcPzx\nzf8MNbmRXKYMV/aLlekuWhT67B5zDPzlL9C3b2hsXlbWvM8YOBBuuy08Hz8e1q1r8bBFcpIy3CwR\nK9OdPDnU1d57b3jdkvIxdxg1KtwCXFQEy5e3uFGZSEZQhit1xMp0b7sNDjoI/ud/oGvXcLvuccfB\njBlNb1CjJjciLaeAm0ViBd0bbwylvCUlIUPduRMuuii0ZNy6tWnHr2xy06ZNaHJTrd2DiMRBATfL\n1Bd0/+u/4IknqhavXLAAevaEF15o2vF79oQHHgjPJ04MtwCLSHwUcLNQfUH3+uvhu98N2e6gQc0v\nH1OTG5Hm0UWzLBbrQlrlasDl5TUXr2xq+Zia3Eiu0EUziUt9me5PfhKC4+TJ8PrrzSsfa906HLtD\nB3jmGa36KxIPZbg5oKFM1ywsXjllCtxzT9jWlPIxNbmRbKcMV5qkoUzXPVQd3H03vPRS08vHhg4N\nwbq8HEaPho8+Su65iGQyZbg5pLFMF2DbNrjsMpg3L7yOp/uYmtxINlOGK83SWKYL0LEjzJnTtPIx\nNbkRiU+zM1wzuxY4EHgFWObunyVyYM0YjzLcOMWT6QJs2BB6MixZEl5ffnm4OFbfLb1qciPZKC36\n4ZrZ34HKPzT3AquA4opHygOwAm7TxBt0y8pqlo8dfTQ8/nj95WN33QU/+lHIjlesCLcCi2SydAm4\n/YBLgSFAt1qbUx6AFXCbLt6gC7BmDVxwAaxdG6YQpk2Da66pu3ilmtxItkmLgFvjIGbdgVMIwXcI\ncFitXSoD8NPATHf/uMUfWncMCrjN0JSgG2/52I4dcMIJoY3j+PGh6U0i1lkTiUIiAy7unvAHcARw\nMfAoUAqUV3vsBCYn4TNdmmf3bvehQ91Dfhoe117rXl4ee/+XXnLv2jXs166d+/TpdfctKXFv0ybs\n89BDyT8HkWSpiC2JiVOJOlC9HwBFwEbgamA2sLsi8M4GChL4OQn54eaqpgbdTz5xHzWqat/hw93/\n7/9q7vPII2Fbq1buK1cm/xxEkiGjAm4YL6cCj1U8Pxj4L2APcHcCPyMBP9rc1tSgW17uPmuWe/v2\nYd/Ond2ff77mPhMmhG09erhv25b8cxBJtLQJuMBXgZFA1zj2/XOt10OArcBJCTkRBdyEaGrQdXd/\n/333QYOq9r/8cvfPPgvbdu1y79MnvH/OOe5lZak5D5FESYuAC1xAuBhWXvF1AXB6A/uvjvHeWZWZ\nb4tPRAE3YZoTdPftc7/1VvfCwrD/0Ue7r1gRtq1f796hQ3j/lltScw4iiZLIgNuSsrC/Am8DnYFv\nAJVX8TZWBN/FwP8CBcBFQD93/0aM47zi7oObNYiax/HmnovU1ZTqherqKx974QU1uZHMlBZVCsCr\n1Z4fClwHrKdmRUJZtefnVex7EFBU7XuXJOI3B8pwE645ma67e2mp+5VXVn1P//7u//u/7lOmhNed\nOrl/+GFqzkGkpUhghtuSXgofmtlxFZFus7v/wt2PAPoTLor9iZDhLgZGuvvciu+bD6w2s+lm1hHI\nj3FsSQPx9F6Ipb7uY0ccETLbLVvg/PPDnWsiuaQlUwpdgTkVj9+4e1z/fMzsCeA8YB+wBrjd3ec1\naxA1j+vNPRdpWHOnF6Bu97GzzoKVK0PQnXTlXm7/1sIQ1TduhG7dwgcNGqTb0yRtpM2dZmZ2APAj\n4LvALHe/NY7vaQ9cQZiGeNrdX272AGoeVwE3iVoSdN3D8j0TJ4a70Dp2hE+3lVFOPk/lj2J42ZNV\nOxcWwqGHwoknwty59R9UJEXSJuDuP4hZPnC4u69v+ZCaPQYF3CRrSdCFut3HAA7iH6zkBI7kbzV3\n7t4d3n5bma5ELu364bp7WZTBVlKjuXO6lQ4/HBYtglsvepdCPgfgn3yBs/g9pbSpufPmzTUjs0gW\nUANyaZKWBt38fJhst/M63+CrvAPA3ziKgSxlX/X/HffuDR8kkkUUcKXJWhp02biRXrzBKvpwAY8B\nsJrj+Srvsp4eNfYTySYKuNIsLQq63UL75Dbs5jHGMInbAPhfvkJP3mQG4/Fq+4lkCwVcabZmB92R\nI2usMnk7VzOGmQCUciAXMYMRtoCtp/9HkkYuEg2t2ist1uTqhdJSOPZYeP/9/W/tphX9+SOrOZ4C\n9rKPQjp3cqbPMM48M+mnIFKvtKtSkNzW5Ey3bdtQZ9u9+/5MtzV7mM9IOrCdfRTS48AtfLzFOOss\nuOKKEKNFMp0yXEmYZmW6S5bUuNPsucP+k2HTjscMJkyAGTPiW7xSJFnS7saHdKCAmx5aenMEwNSp\ncPPN0KlTWDPtBz9ofPFKkWRRwI1BATd9tDTo7tsHZ5wBixfDySeH41x/feOLV4okg+ZwJa21tE63\noABmz4YuXWDp0hCoY3UfmzEjzrpfkTShDFeSpqWZ7rJloZ1jWRk89RQMH163+9jw4fDgg3DIIUk5\nBRFluJIZWprpDhwIt4V7Ihg/HtatC53G5syBWbOgfXtYsAB69gwrSoikO2W4knQtbe04ahTMnw9F\nRbB8eVUDsdrdxy6/HG6/XQ3GJLGU4UpGaUmmawbTp8ORR0JJSajJrfye/d3Hbg3lvL/6FfTpExqc\ni6SjjAm4ZnaYmS02s7fN7C0z+37UY5L4tSTotm8fvrdNG3jkEXj44apt+fkweTK8/joccwy8+y70\n7RuOXVaWlFMRab5ELY6W7AdhdeBeFc/bAe8CX6u2ve7qb5J2mrswpbv7I4+E/Vu1cl+1qu72+hav\nFGkJ0mQRyZRy94/dfU3F853AO0DXaEclTdWSTHfcuHD3WeWc8Kef1txe3+KVKh+TdJExAbc6M+sO\n9AZei3Yk0hwtCbr33hvmadevDwG4vLzuPqefDm++GS627dwJF10EI0bA1q2JPQ+Rpsq4KgUzawcU\nA79w96erve8//elP9+83ZMgQhgwZkvLxSfyaW73w3nsh6H76KdxyC1x9dez9ai9e2blzuACn7mPS\nkOLiYoqLi/e/vuGGG3Lz1l4zKwSeA15w97trbfNMOhcJmht0n3sOhg2DvDxYuDDcIFEflY9JS+Rk\nWZiZGfAwsLZ2sJXM1dzphaFDYcqUMKUwejR89FH9+6p8TNJFxmS4ZjYQWAKUAJWDnuLuL1ZsV4ab\nwZqT6dZucrNwYY2FJGJaswYuuEDdxyR+6hYWgwJu5mtO0N2yBXr3DhnupElhqqAxu3aF7FjdxyQe\nCrgxKOBmh+YE3VhNbuLxhz+EHg0ffgjt2oUKiPHj4+/bK7khJ+dwJTc0Z043VpObeKh8TFJNGa6k\npaZmug01uWmMysekIcpwJes1NdNtqMlNY8xgzJjwfYMGwccfo8UrJSkUcCVtNTXoNtTkJh4qH5Nk\nU8CVtNbUoFtUBA88EJ5PnAirVzft8yq7j732mrqPSeJpDlcyQlPndL/3PXjooVDqtXIldOjQ9M9U\n+ZiAysLe6thqAAAP/klEQVRiUsDNfk0Jurt3w4ABsGoVnHNOWIonr5l/z6l8LLfpopnkpKZML7Ru\nHfbt0AGeeSa+GyLqo/IxSRRluJJxmpLpNqXJTWNUPpablOFKTmtKptuUJjeNUfmYtJQyXMlY8Wa6\nzWly05iyMrjzTrjuOti7F44+Gh5/HI4/vmXHlfSjDFeE+DPdggKYPRu6dIGlS2Hq1JZ/tsrHpDmU\n4UrGizfTbW6Tm8aofCy7KcMVqSbeTLe5TW4ao8UrJV7KcCVrxJPptqTJTTy2bYPLLoN588Lr4cPh\nwQfhkEMS9xmSWspwRWKIJ9NtSZObeHTsCHPmwKxZobfDggXQsye88ELiPkMylwKuZJV4gm7tJjfT\npyd2DCofk/oo4ErWiSfoVm9yc8UVTW9yEw91H5PaFHAlK8UTdMeNgwkTquZ+P/008eNQ+ZhUp4tm\nktUau5CWyCY3jVH5WGbSRTORODWW6SayyU1jVD4mynAlJzSW6SayyU08VD6WOZThijRRY5luIpvc\nxEPlY7lJGa7klIYy3bKyxDe5iceGDTB2LCxZEl5ffnmY2kjkDRnSfMpwRZqpoUw3Pz/xTW7iofKx\n3KEMV3JSQ5nuH/+YnCY38Vi9Gr77XVi7NnQ5mzYNrrkm/DKQaCjDFWmhhjLdAQOS0+QmHr17w4oV\ncOWVoY/vddeFu9XWr0/dGCR5lOFKTqsv0/3Zz+C885LX5CYeWrwyPWjV3hgUcKW56gu6P/4xnHhi\nyHDHjw89F1Id7FQ+Fj0F3BgUcKUl6gu6o0ZBv37hLrHf/hYuvjj1Y9PildFSwI1BAVdaqr6g+5Wv\nwIUXhnnf5cvDPGsUVD4WDQXcGBRwJRHqC7pbtoQM94gjQrlWhw7RjE+LV6aeqhREkqS+6oWDDw61\nsevXhy5j5eXRjE/dxzKbAq5ILbGC7q23hrncVDS5iYfKxzKTphRE6hFreuG882Du3NQ1uYmHyseS\nS3O4MSjgSjLECrr9+4fWip06hTvDunSJbnyVVD6WPAq4MSjgSrLECrqHHx6qBlLZ5KYxKh9LDl00\nE0mhWHO6GzaEP99T2eSmMVq8Mv0pwxWJU6xM1yxklqluctMYlY8ljqYUYlDAlVSIFXQhNBFfsQKO\nPDKacdUnVvexq68OzyU+mlIQiUis6QUIc6YjR6bfn+6xyscGD1b5WFQUcEWaqL6gW1IS5kvT7Q8t\nLV6ZPhRwRZqhvqD7yCPw8MORDKlRp58Ob74ZGvLs3AkXXQQjRsDWrVGPLHdoDlekBWLN6ebnw+uv\nh1uB05HKx5pGF81iUMCVqMQKuh06hHnSL34xunE1Rt3H4pOTAdfMpgNnA393954xtivgSmRiBd2j\nv7KPt+9+mfwFT8LGjdCtW9hp0KC0iWoqH2tcrgbck4GdwCwFXElH1YPugezkM9pxk01lit9ctVNh\nIRx6aFhKYu7c6AZbi8rH6peTZWHuvhTYHvU4ROpTeSFtxJml3M2VAFznP6eYwVU77d0L778Pf/5z\nWtWQqXwsNTIm4Ipkglat4L8vW8I4e5Qp3EQ5+YzmCT6ic80dN2+umjxNEyofSz4FXJEEO+CZJyn0\nvfyM6zmFRWyhM+czh71U+/t8796QDqchlY8lT1bN0EybNm3/8yFDhjAkHZqVSu7ZuBGAAsqYzX/Q\nm9UsZRBTuYnbmVxnv3TUsSPMmQPDhoXysQULwnpuuVA+VlxcTHFxcVKOnTEXzQDMrDvwrC6aSVq7\n5JIadz8sYwBDKKaMAp5iOMN5Omy4+OKwUFqay/XysZy8aGZms4FXgaPMbJOZXRj1mERiGjmyRoPc\ngfyR2yoy2/E8wjq+wr68QnzEyKhG2CSHHw6LFoVlhgoL4Ve/Cjd1rFwZ9cgyT0ZluA1Rhitpo7QU\njj02VCNUcGAU85jPSIp4g9mMZt7klVx/S9uMWgonF8vHcjLDFckYbduGOtvu3fdnugZM5yK+wjpK\nOI7L+RXTbmvLT36SWRUAKh9rGWW4IslSWhomPp+sutNsZdF4TrpqAGXlVQnTtdfCz3+eeYs+5sri\nlTl5p1ljFHAlU/z2tzBhQs33MjXo5sLilZpSEMlgl1wSaluru/FGMm56AarKx2bNCqteLFgAPXvC\nCy9EPbL0pAxXJAK7d4fl1levrvl+pma6kL3lY8pwRTJc69Ywf35o41hdpma6oPKxeCjgikSkRw94\n9NG672dy0M3Ph8mT4bXX4Jhj4N13oW/fcE779kU9uugp4IpEaOhQmDIlPG/Vqur9TA66oPKx+mgO\nVyRi+/bBGWfA4sXhItS2bVXbMnlOt1Kml49pDlckixQUwOzZ0KVLCLZHHFG1LdMzXVD3seoUcEXS\nQKdOYQGI/PzwZ3f1JW6yIeiqfCxQwBVJEwMHwm23hefr1kH17qLZEHTNYMwYKCkJy7p9/DGcdRZc\ncUVaLX6RVAq4ImnkqqvCn9s7dsAnn9TsPZsNQRdyu3xMAVckjZiFJt9HHhnmPb/0pVDJUClbgm6u\nlo8p4IqkmfbtQ7+bNm3CnOfZZ2dn0IXcKx9TWZhImpo5M5RPtWoFr7wCv/hFWIK9UjaUjFWXruVj\nKgsTyQHjxoWuYnv2wHe+E7qMZWumC7lRPqYMVySN7d4NAwbAqlVwzjnwxBNw3nnZnem6w2OPhcUr\nd+yAzp2jXbxSGa5IjmjdOszndugAzzwT/sx+8snsznSzuXxMGa5IBnjuubBkeV4eLFwI/fqFtSqz\nOdMFKCuDO+8MF9P27oWjj4bHH695Y0iyKcMVyTGVTW7Ky2H06HALcLZnupB95WPKcEUyRPUmNyef\nHDLd8vLcyHQBdu0Kv3TuuSe87t8/tLes3nsiGZThiuSg6k1uli6FqVNDyVguZLoQ6pLvvhteegm6\ndoVXX4XjjgsX1DLlXJXhimSYZctCn4WyMnjqqbBw4549uZPpQmoXr1SGK5LDqje5GT8+NLrJpUwX\nMrf7mDJckQzkHm4QmD8fiopg+fKwWGOuZbqQ/MUrleGK5LjqTW5KSsJNApB7mS5kVvcxBVyRDFW9\nyc2MGfDww+H9XAy6mVI+pikFkQxXvcnN8uWhAxfk5vQCJL58TFMKIrJf9SY3I0fCp5+G93Mx04X0\nLh9ThiuSBWo3uVmwINwGDLmb6UJiyseU4YpIDbWb3Nx+e9W2XM10If3Kx5ThimSR2k1uqi9EmcuZ\nLjS/fEwZrojEVLvJzUcfVW3L5UwX0qN8TBmuSJaJ1eSmsLBqe65nugCrV8N3vwtr14YeFdOmwdVX\nh+e1KcMVkXrFanJTXa5nuhDd4pXKcEWyVKwmN9Up0w1qL155zz1w4YVVPwNluCLSqFhNbqpTphvU\nXrzy4ouTt3ilMlyRLFZfk5vqlOkG9S1eedZZictwFXBFstyOHXDCCSHDvfDCEERqU9CtUrt8DDSl\nICJxqq/JTXWaXqhSu3wskZThiuSI+prcVKdMt6bVq6FPH00p1KGAK9K4730PHnoodM5auTLcClyb\ngm5NiaxSUMAVySENNbmpTkG3isrCRKRZGmpyU53mdJNDGa5IDmqoyU11ynSV4YpICzXU5KY6ZbqJ\nlTEB18y+ZWZ/MbN1ZnZ11OMRyXQ/+xmccgps2QLnnw9798beT0E3cTIi4JpZPnAf8C3gGOA/zOxr\n0Y4qtYqLi6MeQlLp/FKvsSY31TUWdNPx/NJRRgRc4BvA39z9fXffCzwBfDviMaVUtv8PrfOLRqdO\nMHduWPX2jjtC1UJ9Ggq6ixcXJ32s2SBTAu6hwKZqrz+oeE9EWqixJjfV1Rd0Fy/W9EI8MiXg6j+l\nSBJddVXokLVjR6hKKC2tf99YQXfp0tBTVkG3YRlRFmZmfYFp7v6titdTgHJ3v7XaPul/IiKSkXLq\nTjMzKwDeBb4JfAi8DvyHu78T6cBERJogxgo+6cfd95nZROB/gHzgYQVbEck0GZHhiohkg0y5aNag\nbL4pwsymm9kWM3sz6rEkg5kdZmaLzextM3vLzL4f9ZgSycxam9lrZrbGzNaa2c1RjynRzCzfzFab\n2bNRjyXRzOx9MyupOL/XW3y8TM9wK26KeBc4DdgM/Jksmt81s5OBncAsd+8Z9XgSzcw6A53dfY2Z\ntQNWAudmy38/ADNr6+6lFdcilgGT3H1Z1ONKFDP7IXA8cJC7nxP1eBLJzN4Djnf3bYk4XjZkuFl9\nU4S7LwW2Rz2OZHH3j919TcXzncA7QNdoR5VY7l5ZZHUA4RpEQv7xpgMz+1fgLOC3QLa2s0nYeWVD\nwNVNEVnCzLoDvYHXoh1JYplZnpmtAbYAi919bdRjSqD/An4MlEc9kCRx4GUzW2FmE1p6sGwIuJk9\nJyIAVEwnPAlcWZHpZg13L3f3XsC/AoPMbEjEQ0oIMxsK/N3dV5O92e0Ad+8NnAlcUTHF12zZEHA3\nA4dVe30YIcuVDGFmhcB84DF3fzrq8SSLu/8D+D1wQtRjSZD+wDkV85yzgVPNbFbEY0ood/+o4uv/\nAQsIU5jNlg0BdwVwpJl1N7MDgPOBZyIek8TJzAx4GFjr7ndHPZ5EM7NDzKxDxfM2wOnA6mhHlRju\nPtXdD3P3HsBoYJG7j416XIliZm3N7KCK5wcCZwAtqhbK+IDr7vuAypsi1gJzsuwK92zgVeAoM9tk\nZhdGPaYEGwB8FzilovRmtZl9K+pBJVAXYFHFHO5rwLPuvjDiMSVLtk3vdQKWVvtv95y7v9SSA2Z8\nWZiISKbI+AxXRCRTKOCKiKSIAq6ISIoo4IqIpIgCrohIiijgioikiAKuiEiKKOCKiKSIAq6ISIoo\n4IqIpIgCrohIiijgioikiAKuiEiKKOCKiKSIAq6ISIoo4IqIpIgCruQEMxtnZsVmtszMSszs/1W8\n38bM7jOz5Wb2ipk9amaHRD1eyU4FUQ9AJNnM7LfAduBMd99lZgMIS6e0Iyzx85i7TzSz7wF3APuA\nbFvKSNKAMlzJamZ2BfAPd/+xu+8CcPc/AtuAXwAfuPsTFQs9PgC0Q/8uJEm0pplkrYpVclcAx7v7\n7lrv/xMw4Dh3f8vM8oBZwIHA/3P3D6IYs2Q3TSlINjsSuK96sK3Qh5DFfujubwG4ezlh9WCRpFHA\nlazl7iVASYxNp1V8zdblyiVNaa5KctHpFV9fjnQUknM0hys5xczaA58Qko3D3P3DiIckOUQZruSa\nU4B84K+1g62ZFZrZ7dEMS3KBAq5kLTP7tpm9ZGYjqr39rYqvf4rxLd8mBGORpNBFM8lKZtYWmA20\nBnYA8ytqbUdV7LK11v4dganAsFSOU3KLMlzJVgY48CZwXUXt7SPAD4G1wGlm1grAzHoATwGT3H1z\nNMOVXKCLZpK1zOzfgCmE4FsI3Onu883sXwi38PYh3PK7Dbje3d+IbLCSExRwRURSRFMKIiIpooAr\nIpIiCrgiIimigCsikiIKuCIiKaKAKyKSIgq4IiIpooArIpIiCrgiIimigCsikiIKuCIiKaKAKyKS\nIv8fTArE8xhOKFcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10923cb70>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "xvals = np.linspace(0, 5, 2);\n",
    "\n",
    "m = (y[1]-y[0])/(x[1]-x[0]);\n",
    "c = y[0]-m*x[0];\n",
    "\n",
    "yvals = m*xvals+c;\n",
    "\n",
    "ylim = np.array([0, 5])\n",
    "xlim = np.array([0, 5])\n",
    "\n",
    "f, ax = plt.subplots(1,1,figsize=(5,5))\n",
    "a = ax.plot(xvals, yvals, '-', linewidth=3);\n",
    "\n",
    "ax.set_xlim(xlim)\n",
    "ax.set_ylim(ylim)\n",
    "\n",
    "plt.xlabel('$x$', fontsize=30)\n",
    "plt.ylabel('$y$',fontsize=30)\n",
    "plt.text(4, 4, '$y=mx+c$',  horizontalalignment='center', verticalalignment='bottom', fontsize=30)\n",
    "plt.savefig('diagrams/straight_line1.svg')\n",
    "ctext = ax.text(0.15, c+0.15, '$c$',  horizontalalignment='center', verticalalignment='bottom', fontsize=20)\n",
    "xl = np.array([1.5, 2.5])\n",
    "yl = xl*m + c;\n",
    "mhand = ax.plot([xl[0], xl[1]], [yl.min(), yl.min()], color=[0, 0, 0])\n",
    "mhand2 = ax.plot([xl.min(), xl.min()], [yl[0], yl[1]], color=[0, 0, 0])\n",
    "mtext = ax.text(xl.mean(), yl.min()-0.2, '$m$',  horizontalalignment='center', verticalalignment='bottom',fontsize=20);\n",
    "plt.savefig('diagrams/straight_line2.svg')\n",
    "\n",
    "a2 = ax.plot(x, y, '.', markersize=20, linewidth=3, color=[1, 0, 0])\n",
    "plt.savefig('diagrams/straight_line3.svg')\n",
    "\n",
    "xs = 2\n",
    "ys = m*xs + c + 0.3\n",
    "\n",
    "ast = ax.plot(xs, ys, '.', markersize=20, linewidth=3, color=[0, 1, 0])\n",
    "plt.savefig('diagrams/straight_line4.svg')\n",
    "\n",
    "\n",
    "m = (y[1]-ys)/(x[1]-xs);\n",
    "c = ys-m*xs;\n",
    "yvals = m*xvals+c;\n",
    "\n",
    "for i in a:\n",
    "    i.set_visible(False)\n",
    "for i in mhand:\n",
    "    i.set_visible(False)\n",
    "for i in mhand2:\n",
    "    i.set_visible(False)\n",
    "mtext.set_visible(False)\n",
    "ctext.set_visible(False)\n",
    "a3 = ax.plot(xvals, yvals, '-', linewidth=2, color=[0, 0, 1])\n",
    "for i in ast:\n",
    "    i.set_color([1, 0, 0])\n",
    "plt.savefig('diagrams/straight_line5.svg')\n",
    "\n",
    "m = (ys-y[0])/(xs-x[0])\n",
    "c = y[0]-m*x[0]\n",
    "yvals = m*xvals+c\n",
    "\n",
    "for i in a3:\n",
    "    i.set_visible(False)\n",
    "a4 = ax.plot(xvals, yvals, '-', linewidth=2, color=[0, 0, 1]);\n",
    "for i in ast:\n",
    "    i.set_color([1, 0, 0])\n",
    "plt.savefig('diagrams/straight_line6.svg')\n",
    "for i in a:\n",
    "    i.set_visible(True)\n",
    "for i in a3:\n",
    "    i.set_visible(True)\n",
    "plt.savefig('diagrams/straight_line7.svg')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src='./diagrams/straight_line4.svg'>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pods\n",
    "pods.notebook.display_plots('straight_line{plot}.svg', \n",
    "                            directory='./diagrams', plot=(1, 7))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# $y = mx + c$\n",
    "\n",
    "point 1: $x = 1$, $y=3$ $$3 = m + c$$ \n",
    "point 2: $x = 3$, $y=1$ $$1 = 3m + c$$ \n",
    "point 3: $x = 2$, $y=2.5$ $$2.5 = 2m + c$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"diagrams/Pierre-Simon_Laplace.png\" align=center width=50%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"diagrams/laplacesDeterminismFrench.png\" align=center width=50%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"diagrams/laplacesDeterminismEnglish.png\" align=center width=50%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"diagrams/philosophicaless00lapliala.png\" height=800>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# $y = mx + c + \\epsilon$\n",
    "\n",
    "point 1: $x = 1$, $y=3$ \n",
    "$$3 = m + c + \\epsilon_1$$ \n",
    "\n",
    "point 2: $x = 3$, $y=1$ \n",
    "$$1 = 3m + c + \\epsilon_2$$ \n",
    "\n",
    "point 3: $x = 2$, $y=2.5$ \n",
    "$$2.5 = 2m + c + \\epsilon_3$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Regression Examples\n",
    "\n",
    "-   Predict a real value, $y_i$ given some inputs\n",
    "    $x_i$.\n",
    "\n",
    "-   Predict quality of meat given spectral measurements (Tecator data).\n",
    "\n",
    "-   Radiocarbon dating, the C14 calibration curve: predict age given\n",
    "    quantity of C14 isotope.\n",
    "\n",
    "-   Predict quality of different Go or Backgammon moves given expert\n",
    "    rated training data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Olympic 100m Data\n",
    "\n",
    "-  Gold medal times for Olympic 100 m runners since 1896.\n",
    "\n",
    "![image](./diagrams/100m_final_start.jpg)\n",
    "Image from Wikimedia Commons <http://bit.ly/191adDC>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Olympic 100m Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x109316208>]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbkAAAGnCAYAAAAqiCnDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHm5JREFUeJzt3X+QHOV95/HP12jQabArDnc+MAObdV1hJ9ZxNnFW2UnO\nk80WIXK4Anwe5HC5FDGQSp0rZRc4ie3ggETWKf8ol5wflR9VGBe6q2BUcwmlC5YMyN6M7jwLS2Js\nWSATKt5aMYDI1eHj0CjS2P7eH9MrRquZ2e3ZnpnuZ96vqin19Dzd+zys6I+65/lh7i4AAEL0ulFX\nAACAQSHkAADBIuQAAMEi5AAAwSLkAADBIuQAAMHqGXJmdq+ZHTezw237PmtmT5vZN83sr8zsR7oc\nu93MjprZP5jZR5OuOAAAa1nrTu6Lkrav2vewpK3u/g5Jz0j6+OqDzOw8SX8SHft2STea2U9svLoA\nAKxfz5Bz90OSXl617xF3/2H09jFJl3Y4dJukZ919yd2bkr4k6boE6gsAwLpt9Du5myV9ucP+gqRj\nbe+fi/YBADA0m/o90MzukHTa3f+yw8frnivMzJhXDABwFne3JM7T152cmf2apF+S9CtditQlXdb2\n/jK17uY6cvdUvuZuuUUn1Ersbq8TkuZuuaXj8XfdddfI2zDIF+3L9ov2ZfcVctvck73viR1yZrZd\n0m9Lus7d/7lLsSckXW5mk2Z2vqT3S9rXfzVHo1guayGX61mmlsupWC4PqUYAgDjWGkJwv6SvS3qb\nmR0zs5sl/bGk10t6xMy+YWZ/GpW9xMwekiR3/76k35T0FUlPSXrA3Z8eYDsGYrpUUq3Q+6vEWqGg\n6VJpSDUCAMTR8zs5d7+xw+57u5R9XtI1be/3S9q/odqNWD6f16mpKc1JKtbrKjabyktqqHUHVysU\ndHpqSvl8vuPxMzMzQ6zt8NG+bKN92RVy25JmST//jF0BMx91HdbSaDS0UK2qVqno5PKytkxMqFgu\na7pU6hpwAID+mJk8oY4nhBwAIFWSDDnmrgQABIuQAwAEi5ADAASLkAMABIuQAwAEi5ADAASLkAMA\nBIuQAwAEi5ADAASLkAMABIuQAwAEi5ADAASLkAMABIuQAwAEi5ADAASLkAMABIuQAwAEi5ADAASL\nkAMABIuQAwAEi5ADAASLkAMABIuQAwAEi5ADAASLkAMABIuQAwAEi5ADAASLkAMABIuQAwAEi5AD\nAASLkAMABIuQAwAEi5ADAASLkAMABIuQAwAEi5ADAASLkAMABIuQAwAEi5ADAASLkAMABIuQAwAE\ni5ADAASLkAMABIuQAwAEi5ADAASLkAMABIuQAwAEi5ADAASLkAMABIuQAwAEi5ADAASLkAMABIuQ\nAwAEi5ADAASLkAMABIuQAwAEi5ADAASLkAMABIuQAwAEi5ADAASLkAMABIuQAwAEi5ADAASLkAMA\nBIuQAwAEi5ADAASLkAMABKtnyJnZvWZ23MwOt+27wcyOmNkPzOwnexy7ZGbfMrNvmNnjSVYaAID1\nWOtO7ouStq/ad1jSeyVV1zjWJc24+5Xuvq3P+gEA0LdNvT5090NmNrlq31FJMrP1nH9dhQAAGIRB\nfifnkh41syfM7NcH+HMAAOio553cBv2su79gZm+S9IiZHXX3Q50K7ty588z2zMyMZmZmBlgtAECa\nzM/Pa35+fiDnNnfvXaD1uPJ/uPsVq/Z/TdJH3P3v1/whZndJetXdP9fhM1+rDgCA8WFmcvdEvu7a\n6OPKjpUws7yZvSHavkDS1Wp1WAEAYGjWGkJwv6SvS3qbmR0zs5vN7HozOyZpWtJDZrY/KnuJmT0U\nHXqxpENm9qSkxyT9jbs/PLhmAABwrjUfVw68AjyuBAC0SdPjSgAAUouQAwAEi5ADAASLkAMABIuQ\nAwAEi5ADAASLkAMABIuQAwAEi5ADAASLkAMABIuQAwAEi5ADAASLkAMABIuQAwAEi5ADAASLkAMA\nBIuQAwAEi5ADAASLkAMABIuQAwAEi5ADAASLkAMABIuQAwAEi5ADAASLkAMABIuQAwAEi5ADAASL\nkAMABIuQAwAEi5ADAASLkAMABIuQAwAEi5ADAASLkAMABIuQAwAEi5ADAASLkAMABGvTqCuAczUa\nDS1Uq6pVKjq5vKwtExMqlsuaLpWUz+dHXT0AyAxz99FWwMxHXYc0uXPHDm1eXFSxXtd0s6m8pIak\nhVxOtUJBp6amdPfevaOuJgAMjJnJ3S2Jc3EnlyKNRkObFxd1x9LSWfvzkmabTc0uLWkuKscdHQCs\nje/kUmShWlWxXu9Zpliva6FaHVKNACDbCLkUqVUqmm42e5YpNpuqVSpDqhEAZBshlyInl5e11kPI\nfFQOALA2Qi5FtkxMqLFGmUZUDgCwNkIuRYrlshZyuZ5larmciuXykGoEANlGyKXIdKmkWqHQs0yt\nUNB0qTSkGgFAtjGEIEXy+bxOTU1pTq1elMW2cXK1aJzc6akphg8AwDoxGDyFmPEEwDhLcjA4IQcA\nSJUkQ47v5AAAwSLkAADBIuQAAMEi5AAAwSLkAADBIuQAAMEi5AAAwSLkAADBIuQAAMEi5AAAwSLk\nAADBIuQAAMEi5AAAwSLkAADBIuQAAMEi5AAAwSLkAADBIuQAAMEi5AAAwdo06gqEptFoaKFaVa1S\n0cnlZW2ZmFCxXNZ0qaR8Pr/h8gCA9TN37/6h2b2SrpH0krtfEe27QdJOST8uacrd/77LsdslfV7S\neZLucfdPdynnveqQJXfu2KHNi4sq1uuabjaVl9SQtJDLqVYo6NTUlO7eu7fv8gAwDsxM7m5JnGut\nO7kvSvpjSXva9h2W9F5Jf9HtIDM7T9KfSLpKUl3Sopntc/enN1bd9Go0Gtq8uKg7lpbO2p+XNNts\nanZpSXNRuXw+H7s8ACC+nt/JufshSS+v2nfU3Z9Z47zbJD3r7kvu3pT0JUnXbaimKbdQrapYr/cs\nU6zXtVCt9lUeABDfoDqeFCQda3v/XLQvWLVKRdPNZs8yxWZTtUqlr/IAgPgG1fEk1pdsO3fuPLM9\nMzOjmZmZhKszeCeXl7XWQ8V8VK6f8gAQqvn5ec3Pzw/k3IMKubqky9reX6bW3VxH7SGXVVsmJtSQ\negZXIyrXT3kACNXqm5tdu3Yldu6NPq7s1vvlCUmXm9mkmZ0v6f2S9m3wZ6VasVzWQi7Xs0wtl1Ox\nXO6rPAAgvp4hZ2b3S/q6pLeZ2TEzu9nMrjezY5KmJT1kZvujspeY2UOS5O7fl/Sbkr4i6SlJD4Tc\ns1KSpksl1Qq9v3asFQqaLpX6Kg8AiK/n40p3v7HLRw92KPu8WmPqVt7vl7R/Q7XLkHw+r1NTU5pT\nq1dksW3cWy0a93Z6aurMcIC45QEA8fUcDD6UCgQ0GFxixhMA2KgkB4MTcgCAVEky5JigGQAQLEIO\nABAsQg4AECxCDgAQLEIOABAsQg4AECxCDgAQLEIOABAsQg4AECxCDgAQLEIOABAsQg4AECxCDgAQ\nLEIOABAsQg4AECxCDgAQLEIOABAsQg4AECxCDgAQLEIOABCsTaOuANav0WhooVpVrVLRyeVlbZmY\nULFc1nSppHw+33fZQdUhTecGMJ7M3UdbATMfdR2y4M4dO7R5cVHFel3TzabykhqSFnI51QoFnZqa\n0t1798YuO6g6pOncALLFzOTulsS5uJPLgEajoc2Li7pjaems/XlJs82mZpeWNBeVk7TusnHujuLU\nIe5d1yDPDWC88Z1cBixUqyrW6z3LFOt1LVSrscoOqg5xDfLcAMYbIZcBtUpF081mzzLFZlO1SiVW\n2UHVIa5BnhvAeONxZQacXF7WWg/p8lG5le31lh1UHdJybgDjjZDLgC0TE2qod3g1onIr2+stO6g6\npOXcAMYbjyszoFguayGX61mmlsupWC7HKjuoOsQ1yHMDGG+EXAZMl0qqFQo9y9QKBU2XSrHKDqoO\ncQ3y3ADGG48rMyCfz+vU1JTm1OplWGwbR1aLxpGdnpo6070+TtlB1SEt5wYw3hgMniHMeAJgHCQ5\nGJyQAwCkSpIhx3dyAIBgEXIAgGARcgCAYBFyAIBgEXIAgGARcgCAYBFyAIBgEXIAgGARcgCAYBFy\nAIBgEXIAgGARcgCAYBFyAIBgEXIAgGARcgCAYBFyAIBgEXIAgGARcgCAYBFyAIBgbRp1BRCuRqOh\nhWpVtUpFJ5eXtWViQsVyWdOlkvL5/KirB2AMmLuPtgJmPuo6IHl37tihzYuLKtbrmm42lZfUkLSQ\ny6lWKOjU1JTu3rt31NUEkEJmJne3JM7FnRwS12g0tHlxUXcsLZ21Py9pttnU7NKS5qJy3NEBGCS+\nk0PiFqpVFev1nmWK9boWqtUh1QjAuCLkkLhapaLpZrNnmWKzqVqlMqQaARhXhBwSd3J5WWs9hMxH\n5QBgkAg5JG7LxIQaa5RpROUAYJAIOSSuWC5rIZfrWaaWy6lYLg+pRgDGFSGHxE2XSqoVCj3L1AoF\nTZdKQ6oRgHHFEAIkLp/P69TUlObU6kVZbBsnV4vGyZ2emmL4AICBYzA4BoYZTwD0I8nB4IQcACBV\nkgw5vpMDAASLkAMABIuQAwAEi5ADAASLIQRAxtGLFeiuZ+9KM7tX0jWSXnL3K6J9F0p6QNKPSVqS\ntMPdv9fh2CVJr0j6gaSmu2/r8jPoXQn0iXX7EKJh9q78oqTtq/Z9TNIj7v5WSQej9524pBl3v7Jb\nwAHoX/u6fbNRwEmvrdt3x9KSzl9cVKOx1kyiQLh6hpy7H5L08qrd10q6L9q+T9L1PU6RSBIDOBfr\n9gFr66fjyUXufjzaPi7poi7lXNKjZvaEmf16X7UD0BXr9gFr21DHE3d3M+v2hdrPuvsLZvYmSY+Y\n2dHozvAcO3fuPLM9MzOjmZmZjVQLGAus24dQzM/Pa35+fiDn7ifkjpvZxe7+opm9WdJLnQq5+wvR\nn/9kZn8taZukNUMOwPqsrNvXK+hYtw9ZsPrmZteuXYmdu5/Hlfsk3RRt3yTpwdUFzCxvZm+Iti+Q\ndLWkw/1WEsC5WLcPWFvPkDOz+yV9XdLbzOyYmX1A0qck/YKZPSNpNnovM7vEzB6KDr1Y0iEze1LS\nY5L+xt0fHlQjgHHEun3A2no+rnT3G7t8dFWHss+rNaZO7v6Pkt654doB6Ip1+4C1sdQOkHHMeILQ\nsJ4cACBYrCcHAMA6EHIAgGARcgCAYLHUDpBCdCYBkkHHEyBlWD4H4y7JjifcyQEp0r58TruV5XNm\nl5Y0F5Xjjg5YG9/JASnC8jlAsgg5IEVYPgdIFiEHpAjL5wDJIuSAFFlZPqcXls8B1o+QA1KE5XOA\nZBFyQIqwfA6QLIYQACnC8jlAshgMDqQQM55gnLHUDgAgWCy1AwDAOhByAIBgEXIAgGARcgCAYDGE\nAJkUp/dhGnoqpqEOwDiidyUyJ856a2lYmy0NdQCyhPXkMLbirLcmaeRrs7E+HDBafCeHTImz3loa\n1mZLQx2AcUbIIVPirLeWhrXZ0lAHYJzxuBKZEne9tVGvzcb6cMBoEXLIlJX11noFR/t6a3HKDkLc\n+gJIFo8rkSlx1ltLw9psaagDMM4IOWRKnPXW0rA2WxrqME4ajYa+euCAPnnrrfrE1Vfrk7feqq8e\nOHCmty3GD48rkSlx11sb9dpsrA83PO3jEW9rH4+4Z492Mx5xbDEYHJnEjCdo12g0tHvr1nPGI7ab\nm5zU7UeO8N87A1hPDgDafPXAAenaazXbY7jGwVxOtm+fZrdvH2LN0A/WkwOANoxHRDeEHIDMYzwi\nuiHkAGTeynjEXhiPOJ7oXQmMkVA7wBTLZS3s2dPzOznGI44nOp4AYyLkJX/oXRkWltoBEEvoS/4w\nHhHdcCcHjIFx6WIf6uPYccM4OQCxfPLWW3XbF76w5kTRu2+5RXfcc8+wqgV0xDg5ALHQxR7jipAD\nxgBd7DGuCDlgDLDkD8YVIQeMAZb8wbhiCAEwBuhij3FF70pgjKShi30a6oB0YwgBgEwKedYVJIcZ\nTwBkTuizriCd6HgCYCgWqlUV6/WeZYr1uhaq1SHVCOOAkAMwFCxsilEg5AAMBbOuYBQIOQBDwawr\nGAVCDsBQMOsKRoGQAzAUzLqCUWAIAYChYNYVjAKDwQEMFTOeYC3MeAIACBYzngAYG9z5YSO4kwOQ\nWsx1OZ64kwMQPOa6RBIYQgAglZjrEkkg5ACkEnNdIgmEHIBUYq5LJIGQA5BKzHWJJBByAFKJuS6R\nBEIOQCox1yWSwBACAKnEXJdIAoPBAXSUlplG0lIPDM/Q5q40s3slXSPpJXe/Itp3oaQHJP2YpCVJ\nO9z9ex2O3S7p85LOk3SPu3+6y88g5ICUGYeZRgjP9BpmyL1b0quS9rSF3Gck/W93/4yZfVTSj7r7\nx1Ydd56k70i6SlJd0qKkG9396Q4/g5ADUqTRaGj31q3nzDTSbm5yUrcfOZLZMBiHEM+yJEOuZ8cT\ndz8k6eVVu6+VdF+0fZ+k6zscuk3Ss+6+5O5NSV+SdN0G6wpgCEKfaaR9urDZKOCk16YLu2NpSecv\nLqrRWGsAA7Kgn96VF7n78Wj7uKSLOpQpSDrW9v65aB+AlAt9ppHQQxxn21DvSnd3M+v0rDHW88ed\nO3ee2Z6ZmdHMzMxGqgVgA0KfaaRWqei2dYT47kpFs9u3D6lW421+fl7z8/MDOXc/IXfczC529xfN\n7M2SXupQpi7psrb3l6l1N9dRe8gBGK2VmUZ6BV2WZxoZdIjToSW+1Tc3u3btSuzc/YTcPkk3Sfp0\n9OeDHco8IelyM5uU9Lyk90u6sb8qAhimYrmshT17NNvjbmf1TCNZurAPMsTbO7Tc1t6hZc8e7aZD\ny2i4e9eXpPvVCqnTan3H9gFJF0p6VNIzkh6W9Mao7CWSHmo79j1q9bB8VtLHe/wMB5AeJ06c8LnJ\nSXep6+v3Jyf9xIkT7u7+ezfc4HOTk34wl/MT0ecnJD+Yy/nc5KT/3g03jLhFZzu4f78fzOV6tu/R\nXM4P7t8f67xx/7uhuygXeubTel897+Tcvdvd11Udyj6v1pi6lff7Je1fZ9YCSIk4M41kcWHT6VJJ\nuwsFzfYYIlErFHR7zOnC4nRo4bu+4WFaLwDnuHvv3jOPIHevegR5e9sjyCxe2Ac1XRgdWtKJkAPQ\nUT6f1+z27T0vyFm9sK83xOMIvVdqVhFyAPqW5Qv7ekJ8xXo61oTeKzWrCDkAfRuHC/t6e0z20ysV\ng8d6cgD6FvrCpnGmAGP9u3Qi5AD0LfQLe5yONWc6tExO6mAup5WZLxuSDuZympucZP27EeBxJYC+\nhb6wadyONYPo0IKNIeQAbEjIF/Z+OtbE6dCCwSPkAGxYqBf2cehYEzq+kwOALkLvWDMOuJMDgC4G\nNQVYP7I0CXaaEHIA0EVaOtawukH/rDXh8wgrYOajrgMA9DLKu6hGo6HdW7eeMwl2u7nJSd1+5Egw\nd3RmJne3RM416oAh5ACgu68eOCBde23PmVQO5nKyffvO6viT5cebSYYcjysBIMX6mQSbx5uvIeQA\nYETWc7cVd6xeFtf4GyRCDgBGYL13W3HH6mVxjb9BYpwcAAxZnImf447Vq1Uqml7H481apZJAS9KP\nkAOAIYtztxV3Euwsr/E3CIQcAAxZnLutuKsbrDze7GWcpiLjOzkAGLK4d1txJsFm8dazEXIAMGT9\nTPy83kmw0zQVWRrwuBIAhmyQEz+zeOvZmPEEAIZsGFN1MeNJdK5RBwwhB2Ac3bljh86Pxsl1m/h5\nXGYlWY2QA4AAZPlua5AIOQBAsJIMOTqeAACCRcgBAIJFyAEAgkXIAQCCRcgBAIJFyAEAgkXIAQCC\nRcgBAIJFyAEAgkXIAQCCRcgBAILFoqkAMOZCniiaCZoBYIzduWOHNkdL/ky3LfmzEC35c2oES/4k\nOUEzd3IAMKYajYY2Ly6es3hrXtJss6nZpSXNReWyekfHd3IAMKYWqlUV6/WeZYr1uhaq1SHVKHmE\nHACMqVqloulms2eZYrOpWqUypBolj5ADgDF1cnlZaz2EzEflsoqQA4AxtWViQo01yjSicllFyAHA\nmCqWy1rI5XqWqeVyKpbLQ6pR8gg5ABhT06WSaoVCzzK1QkHTpdKQapQ8hhAAwJjK5/M6NTWlObV6\nURbbxsnVonFyp6emMjt8QGIwOACMvbTNeJLkYHBCDgCQKkmGHN/JAQCCRcgBAIJFyAEAgkXIAQCC\nRcgBAIJFyAEAgkXIAQCCRcgBAIJFyAEAgkXIAQCCRcgBAIJFyAEAgkXIAQCCRcgBAIJFyAEAgkXI\nAQCCRcgBAIJFyAEAgkXIAQCC1XfImdmHzeywmX3bzD7c4fMZM/u/ZvaN6PWJjVU1e+bn50ddhYGi\nfdlG+7Ir5LYlra+QM7N/K+lWSVOS3iHpP5jZv+lQ9G/d/croNbeBemZS6H8RaV+20b7sCrltSev3\nTu7HJT3m7v/s7j+Q9LeS/mOHctZ3zQAA2KB+Q+7bkt5tZheaWV7SNZIuXVXGJf2MmX3TzL5sZm/f\nSEUBAIjL3L2/A81ulvRBSSckHZF0yt1va/v8DZJ+4O4NM3uPpD9097d2OE9/FQAABMvdE3kS2HfI\nnXUSsz+QtOzuf96jzHclvcvd/8+GfyAAAOuwkd6V/zr6c0LSeyX95arPLzIzi7a3qRWoBBwAYGg2\nbeDYipn9S0lNSR9091fM7Dckyd3/QlJZ0n8xs+9Lakj65Q3XFgCAGBJ5XAkAQBolPuOJmd1rZsfN\n7HDbvm1m9ng0KHzRzKai/f/CzO43s2+Z2VNm9rG2Y94VDTb/BzP7w6Tr2a8u7XuHmdWiduyLOt2s\nfPbxqA1Hzezqtv2pa1+ctpnZL5jZE9H+J8zs59uOSV3bpPi/u+jzCTN71cw+0rYviPaZ2b+LPvt2\n9Pn50f7Mty9r1xYzu8zMvmZmR6Lfx4ei/Rea2SNm9oyZPWxmb2w7JkvXlljtS/T64u6JviS9W9KV\nkg637ZuX9IvR9nskfS3a/jVJ90fbWyR9V9JE9P5xSdui7S9L2p50XRNs36Kkd0fbH5B0d7T9dklP\nSspJmpT0rF67e05d+2K27Z2SLo62t0p6ru2Y1LUtbvvaPq9IekDSR0Jqn1pfVXxT0hXR+x+V9LqA\n2pepa4ukiyW9M9p+vaTvSPoJSZ+R9DvR/o9K+lS0nbVrS9z2JXZ9SfxOzt0PSXp51e4XJP1ItP1G\nSfW2/ReY2XmSLpB0WtIrZvZmSW9w98ejcnskXZ90XfvRpX2XR/sl6VFJ74u2r1Prf7Smuy+p9Rfx\np9Pavjhtc/cn3f3FaP9TkraYWS6tbZNi/+5kZtdL+ke12reyL5T2XS3pW+5+ODr2ZXf/YUDty9S1\nxd1fdPcno+1XJT0tqSDpWkn3RcXu02t1zdq1JVb7kry+DGuC5o9J+pyZLUv6rKTflSR3/4qkV9T6\nC7kk6bPu/j21Gv9c2/H1aF9aHTGz66LtGyRdFm1forPb8Zxa7Vi9P83t69a2du+T9Hfu3lQgvzsz\ne72k35G0c1X5INon6a2S3MwOmNnfmdlvR/uDaF+Wry1mNqnWHetjki5y9+PRR8clXRRtZ/bass72\ntdvQ9WVYIfcFSR9y9wlJt0XvZWb/Wa1HCW+W9BZJv2VmbxlSnZJ0s6QPmtkTat2Knx5xfZLUs21m\ntlXSpyT9xgjqloRu7dspabe7N5Tt6em6tW+TpH8v6T9Ff77XzGbVmqkoSzq2L6vXlugfV/9d0ofd\n/f+1f+at53NZ+/2cJW77kri+bGQIQRzb3P2qaLsi6Z5o+2ck/bW35r/8JzP7X5LeJel/6uxpwi7V\na484U8fdvyPpFyXJzN6q1jRnUqvO7Xc+l6r1r5C6MtK+Hm2TmV0q6a8k/aq7fzfanZm2SR3b90vR\nR9skvc/MPqPWI/YfmtlJtdqb5fat/P6OSap6NHbVzL4s6Scl/Tdlu30rv7/MXVvMLKdWAPxXd38w\n2n3czC529xejR3UvRfszd22J2b7Eri/DupN71sx+LtqelfRMtH00ei8zu0DStKSj0bPYV8zsp83M\nJP2qpAeVUmb2pujP10n6hKQ/iz7aJ+mXzez86F+Rl0t6PEvt69a2qBfUQ5I+6u61lfLu/oIy0jap\nY/v+XJLcveTub3H3t0j6vKRPuvufZul3J/X8u/kVSVeY2RYz2yTp5yQdCaB9K7MuZeraEtXlC5Ke\ncvfPt320T9JN0fZNeq2umbq2xG1foteXAfSiuV/S82o9NjimVo+nn1Lr+euTkmqSrozKblbrX46H\n1Zr/sr0H27ui/c9K+qOk65lg+26W9CG1egt9R9IfrCr/u1EbjirqYZrW9sVpm1oXlFclfaPt9a/S\n2rZ+fndtx90l6fY0/+76/Lv5K2pNtn5YUa+2UNqXtWuLWo+MfxhdI1f+f9ou6UK1OtQ8I+lhSW9s\nOyZL15ZY7Uvy+sJgcABAsIb1uBIAgKEj5AAAwSLkAADBIuQAAMEi5AAAwSLkAADBIuQAAMH6/4RY\nftDYwo4uAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x109316160>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = pods.datasets.olympic_100m_men()\n",
    "f, ax = plt.subplots(figsize=(7,7))\n",
    "ax.plot(data['X'], data['Y'], 'ro', markersize=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Olympic Marathon Data\n",
    "\n",
    "-   Gold medal times for Olympic Marathon since 1896.\n",
    "\n",
    "-   Marathons before 1924 didn’t have a standardised distance.\n",
    "\n",
    "-   Present results using pace per km.\n",
    "\n",
    "-   In 1904 Marathon was badly organised leading to very slow times.\n",
    "\n",
    "![image](./diagrams/Stephen_Kiprotich.jpg)\n",
    "Image from Wikimedia Commons <http://bit.ly/16kMKHQ>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Olympic Marathon Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1094398d0>]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbMAAAGnCAYAAAA9qrkKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAH1BJREFUeJzt3X+M3PV95/HXGzwYD6XhokROOma1SA1S4+sdlFtnh7TD\n3Opwja/n9pTBQndN2tTuofSiRHZJm8QEHLJUUX9ok0hpYglOwr0qPWu4EHJHnYDJsOE0C0trCDFH\nEhRWC3OEVKJpGn8tMwnv+2O+Xo9nd2dnZr8z8/3MPh/Siu98v5+Z/Xx2l+/L38/38/l8zd0FAEDI\nLhp2BQAAWC/CDAAQPMIMABA8wgwAEDzCDAAQPMIMABC8TZ0UMrMFST+W9DNJdXff0XK8KOkrkr4f\n77rf3aeTqyYAAKvrKMwkuaSiu7/Wpsxj7r4ngToBANCVbroZbZ3HAQDoi07DzCU9YmZPmdnvr3L8\nejN7xsweMrN3JldFAADa67Sb8d3u/oqZvVXSw2b2vLt/s+n430u60t0jM7tJ0gOSrm7+ADNj3SwA\nwDLuvu6evY6uzNz9lfi//yDpy5J2tBz/Z3eP4u2/lZQxszev8Dkj+3XnnXcOvQ60j7bRvtH7GvX2\nJWXNMDOzrJldHm9fJmmnpGdbymw1M4u3d0gybz9YBACAxHTSzbhV0pfjrNok6a/d/etmdqskufsR\nSSVJHzCzn0qKJN3Sp/oCALDMmmHm7i9KumaF/Ueatj8v6fPJVi0sxWJx2FXoq1Fu3yi3TaJ9oRv1\n9iXFkuyzbPuNzHxQ3wsAEAYzkw9qAAgAAGlGmAEAgkeYAQCCR5gBAIJHmAEAgkeYAQCCR5gBAIJH\nmAEAgkeYAQCCR5gBAIJHmAEAgkeYAQCCR5gBAIJHmAEAgkeYAQCCR5gBAIJHmAEAgkeYAQCCR5gB\nAIJHmAEAgkeYAQCCR5gBAIJHmAEAgkeYAQCCR5gBAIK3adgV2CiiKNLc7Kyq5bLOLC5qy9iY8qWS\nJgsFZbPZYVcPAIJm7j6Yb2Tmg/peaXPH3r3aPD+vfK2myXpdWUmRpLlMRtVcTmcnJnTXsWPDriYA\nDJyZyd1tvZ/DlVmfRVGkzfPzOrSwcMH+rKSpel1TCwuajstxhQYAveGeWZ/Nzc4qX6u1LZOv1TQ3\nOzugGgHA6CHM+qxaLmuyXm9bJl+vq1ouD6hGADB6CLM+O7O4qLU6D7NxOQBAbwizPtsyNqZojTJR\nXA4A0BvCrM/ypZLmMpm2ZaqZjPKl0oBqBACjhzDrs8lCQdVcrm2Zai6nyUJhQDUCgNHD0Pw+y2az\nOjsxoWk1Ri3mm+aZVeN5Zq9PTDAsHwDWgUnTA8IKIACwXFKTpgkzAMDQJBVm3DMDAASPMAMABI8w\nAwAEjzADAASPMAMABI8wAwAEjzADAASPMAMABI8wAwAEjzADAASPMAMABI8wAwAEjzADAASPMAMA\nBI8wAwAEjzADAASPMAMABI8wAwAEr6MwM7MFM/uWmZ00sydXKfM5M/uemT1jZtcmW00AAFa3qcNy\nLqno7q+tdNDMdkv6RXd/h5m9S9IXJE0mVEcAANrqppvR2hzbI+k+SXL3JyRdYWZb11MxAAA61WmY\nuaRHzOwpM/v9FY7nJL3U9PplSdvWWzkAADrRaTfju939FTN7q6SHzex5d/9mS5nWKzdv/ZDDhw8v\nbReLRRWLxS6qCgAIXaVSUaVSSfxzzX1Z5rR/g9mdkn7i7n/RtO+Lkiru/jfx6+cl3eDurzaV8W6/\nFwBgtJmZ3L3dbayOrNnNaGZZM7s83r5M0k5Jz7YUe1DS++Iyk5J+1BxkAAD0UyfdjFslfdnMzpX/\na3f/upndKknufsTdHzKz3Wb2gqTTkt7ftxoDANCi627Gnr8R3YwAgBYD62YEACDtCDMAQPAIMwBA\n8AgzAEDwCDMAQPAIMwBA8AgzAEDwCDMAQPAIMwBA8AgzAEDwCDMAQPAIMwBA8AgzAEDwCDMAQPAI\nMwBA8AgzAEDwCDMAQPAIMwBA8AgzAEDwCDMAQPAIMwBA8AgzAEDwCDMAQPAIMwBA8AgzAEDwCDMA\nQPAIMwBA8AgzAEDwCDMAQPAIMwBA8AgzAEDwCDMAQPAIMwBA8AgzAEDwCDMAQPAIMwBA8AgzAEDw\nCDMAQPAIMwBA8AgzAEDwCDMAQPAIMwBA8AgzAEDwCDMAQPAIMwBA8AgzAEDwCDMAQPAIMwBA8Agz\nAEDwCDMAQPAIMwBA8AgzAEDwCDMAQPAIMwBA8AgzAEDwCDMAQPA6CjMzu9jMTprZV1c4VjSzf4qP\nnzSz25OvJgAAq9vUYbkPS3pO0uWrHH/M3fckUyUAALqz5pWZmW2TtFvSPZJstWJJVgoAgG50cmU2\nI+kjkn5+leMu6Xoze0ZSTdJt7v5cQvUbuCiKNDc7q2q5rDOLi9oyNqZ8qaTJQkHZbHbY1QMArKBt\nmJnZb0j6obufNLPiKsX+XtKV7h6Z2U2SHpB09UoFDx8+vLRdLBZVLK72kcNxx9692jw/r3ytpgP1\nurKSIklzR49qJpfT2YkJ3XXs2FJ5gg8AulOpVFSpVBL/XHP31Q+a/Ymk90r6qaRL1bg6u9/d39fm\nPS9Kus7dX2vZ7+2+17BFUaSZ7dt1aGFh1TLT4+M6eOqUstnsBcE32Rx8mYyqKwQfAGA5M5O7r/tW\nVdt7Zu7+cXe/0t2vknSLpEdbg8zMtpqZxds71AjI11b4uFSbm51VvlZrWyZfq2ludlZRFGnz/LwO\nLSxoKg4yScpKmqrXdWhhQZfMzyuKor7XGwDQ/TwzlyQzu9XMbo33lSQ9a2ZPS/qMGqEXnGq5rMl6\nvW2ZfL2uarncVfABAPqv4zBz96Xh9+5+xN2PxNufd/d/6e7XuPv17j7Xr8r205nFRa11lysbl+sm\n+AAA/ccKILEtY2Naq1Mwist1E3wAgP4jzGL5UklzmUzbMtVMRvlSqavgAwD0H2EWmywUVM3l2pap\n5nKaLBS6Cj4AQP8RZrFsNquzExOaHh/XiUxm6corknQik9H0+Lhen5hQNpvtKvgAAP3X6dqMG8Jd\nx44tTYSeaZkIfbBpIvRS8KkxajHfNM+sGs8zOxd8AID+aztpOtFvlPJJ071gBRAAWJ+kJk0TZgCA\noRnICiAAAISAMAMABI8wAwAEjzADAASPMAMABI8wAwAEjzADAASPMAMABI8wAwAEjzADAASPMAMA\nBI8wAwAEjzADAASPMAMABI8wAwAEjzADAASPMAMABI8wAwAEjzADAASPMAMABI8wAwAEjzADAASP\nMAMABI8wAwAEjzADAASPMAMABI8wAwAEjzADAARv07ArgOWiKNLc7Kyq5bLOLC5qy9iY8qWSJgsF\nZbPZYVcPAFLH3H0w38jMB/W9QnbH3r3aPD+vfK2myXpdWUmRpLlMRtVcTmcnJnTXsWPDriYAJMLM\n5O623s/hyixFoijS5vl5HVpYuGB/VtJUva6phQVNx+W4QgOA87hnliJzs7PK12pty+RrNc3Nzg6o\nRgAQBsIsRarlsibr9bZl8vW6quXygGoEAGEgzFLkzOKi1uo8zMblAADnEWYpsmVsTNEaZaK4HADg\nPMIsRfKlkuYymbZlqpmM8qXSgGoEAGEgzFJkslBQNZdrW6aay2myUBhQjQAgDAzNT5FsNquzExOa\nVmPUYr5pnlk1nmf2+sQEw/IBoAWTplOIFUAAbBRJTZomzAAAQ5NUmHHPDAAQPMIMABA8wgwAEDzC\nDAAQPMIMABA8wgwAEDzCDAAQPMIMABA8wgwAELyOwszMLjazk2b21VWOf87Mvmdmz5jZtclWEQCA\n9jq9MvuwpOckLVuPysx2S/pFd3+HpP8i6QvJVQ8AgLWtGWZmtk3Sbkn3SFpp/aw9ku6TJHd/QtIV\nZrY1yUoCANBOJ1dmM5I+IumNVY7nJL3U9PplSdvWWS8AADrWNszM7Dck/dDdT2rlq7Kloi2vWR4f\nADAwaz2c83pJe+L7YpdK+nkzO+ru72sqU5N0ZdPrbfG+ZQ4fPry0XSwWVSwWe6gyACBUlUpFlUol\n8c/t+HlmZnaDpNvc/T+07N8t6YPuvtvMJiV9xt0nV3g/zzMDAFwgqeeZrXVl1srjb36rJLn7EXd/\nyMx2m9kLkk5Lev96KwUAQDd40jQAYGh40jQAADHCDAAQPMIMABC8bgeAIGWiKNLc7Kyq5bLOLC5q\ny9iY8qWSJgsFZbPZYVcPAAaCASABu2PvXm2en1e+VtNkva6spEjSXCajai6nsxMTuuvYsWFXEwBW\nNayh+UiJKIq0eX5ehxYWLtiflTRVr2tqYUHTcTmu0ACMOu6ZBWpudlb52ooLrSzJ12qam50dUI0A\nYHgIs0BVy2VN1utty+TrdVXL5QHVCACGhzAL1JnFRa3VeZiNywHAqCPMArVlbEzRGmWiuBwAjDrC\nLFD5UklzmUzbMtVMRvlSaUA1AoDhIcwCNVkoqJrLtS1TzeU0WSgMqEYAMDwMzQ9UNpvV2YkJTasx\najHfNM+sGs8ze31igmH5ADYEJk0HLi0rgKSlHgDCktSkacIM68ZKJAB6xQogSAVWIgGQBgwAwbqw\nEgmANCDMsC6sRAIgDQgzrAsrkQBIA8IM68JKJADSgDDDurASCYA0IMywLqxEAiANGJqPdWElEgBp\nwKRpJIIVQAD0ghVAAADBSyrMuGcGAAgeYQYACB5hBgAIHmEGAAgeYQYACB5hBgAIHmEGAAgeYQYA\nCB7LWQEJYzUUYPBYAQRI0B1792rz/LzytZomm9apnIvXqTw7MaG7jh0bdjWB1EhqBRCuzICERFGk\nzfPzOrSwcMH+rKSpel1TCwuajstxhQYki3tmQELmZmeVr9XalsnXapqbnR1QjYCNgzADElItlzVZ\nr7ctk6/XVS2XB1QjYOMgzICEnFlc1Fqdh9m4HIBkcc8MAzeqo/22jI0pktoGWhSXA5AsRjNioEZ5\ntN+jx49Le/Zoqk1X44lMRvbgg5ratWuANQPSi+eZITjNo/2m4iCTzo/2O7SwoEvm5xVF0TCr2bPJ\nQkHVXK5tmWoup8lCYUA1AjYOwgwDM+qj/bLZrM5OTGh6fFwnMhmdi+RIjSuy6fFxvT4xEXRXKpBW\ndDNiYO7ev18H7r13zXtKM/v26dA99wyqWokb1XuCQD8waRrB6WW0X4jBkM1mNbVrF/fFgAEizDAw\n3Y72ax4scqB5sMjRo5oJfLAIgGRxzwwDky+VNJfJtC1TzWSUL5VGfrAIgGQRZlhVFEV69Phx3b1/\nv27fuVN379+vR48f7zlAuhntN+qDRQAki25GrKgfXXxLo/3UCKJ80+dW43lm50b7VctlHehgaaiZ\ncpl7UwAIMyzXz9Xf7zp2bGlQx0zLoI6DTYM6WBoKQDcIMyzTTRdfL1dFnYz2Y2koAN3gnhmWScPq\n790MFjkn6Xt8AMLBlRmWSUMX32ShoJlcTlMtXZ3NqrmcDsZLQzGMH9jYCDMsk4Yuvm4Gi/CEZwAs\nZ4Vl0rT6eycrgKSpvgC6k9RyVoQZlomiSDPbty+70mk2PT6ug6dOpeJKZ6Os+QiMItZmRN9008WX\nBmm4xwdguNYMMzO7VNJjkjZLukTSV9z9Yy1lipK+Iun78a773X062apikDqdD5YGabjHB2C4Oupm\nNLOsu0dmtknS45Juc/fHm44XJR109z1tPoNuRvQF98yAcA30SdPufm6iziWSLpb02kp1Wm9lgF7w\nhGcAHYWZmV1kZk9LelXSN9z9uZYiLul6M3vGzB4ys3cmXVFgNTzhGUBXoxnN7E2Svibpo+5eadp/\nuaSfxV2RN0n6rLtf3fJev/POO5deF4tFFYvF9dUeaBLigzyBjaZSqahSqSy9/uQnPzmcoflm9glJ\nZ9z9z9uUeVHSde7+WtM+7pkBAC4wsHtmZvYWM7si3t4i6UZJJ1vKbDUzi7d3qBGSK91XAwAgcZ3M\nM3u7pPvM7CI1wu+v3P2Emd0qSe5+RFJJ0gfM7Kdq3Kq4pV8VBjYqulGB1bECCBCA5oWUJ5sXUo4n\nsbOQMkLFCiDABsFCysDaeJ4ZkHLdPCwV2KgIMyDl0vCwVCDtCDMg5VhIGVgb98yAIepkhCILKQNr\nYzQjMCSdjlBkIWWMsoEuNAwgWc0jFKfiIJPOj1A8tLCgS+bnFUURCykDHSDMgCHoZoQiCykDa6Ob\nERiCu/fv14F7713zPtjMvn06dM89jdesAIIRxKRpIGC9jFDMZrOa2rWL+2LACuhmBIbg3AjFdhih\nCHSOMAOGIF8qaS6TaVummskoXyoNqEZA2OhmBIZgslDQTC6nqZb1FptVczkdTOEIRe7dIY0IM2AI\nlkYoqjFqMd80z6wazzNL4wjF5rlxB5rnxh09qhlW78cQMZoRGKKQrnKiKNLM9u3LVu9vNj0+roOn\nTqWu7kivpEYzEmYAOsJKJOgHVgABMFCs3o80I8wAdITV+5FmhBmAjjA3DmlGmAHoCHPjkGaEGYCO\nsHo/0ox5ZgA6EurcOGwMDM0H0JWQ5sYh/ZhnBgAIHvPMAACIEWYAgOARZgCA4BFmAIDgMTQfGEGM\nOMRGw2hGYMQ0P3NssvmZY/FcMJ45hjRJajQjV2bACImiSJvn55c9cywraape19TCgqbjclyhYZRw\nzwwYIXOzs8rXam3L5Gs1zc3ODqhGwGAQZsAI4Zlj2KgIM2CE8MwxbFTcMwNGyLlnjrULNJ45lgxG\njKYLoxmBEfLo8ePSnj2aatPVeCKTkT34oKZ27RpgzUYLI0aTw2hGAMtMFgqayeU01TKasVk1l9NB\nnjnWs15HjHIl11+EGTBCennmWD9PsqN4Au9mxOi5q9/mK7kDzVdyR49qhiu5RNDNCIygTkOkn91l\no9oVd/f+/Tpw771r3pec2bdPh+65R1EUaWb79mVXcs2mx8d18NSpYAN+PehmBLCqbDarqV272t4X\n6+cE61GevN3tiNFeruTQPYbmAxtUPydYj/Lk7XMjRttpHjHK3L/BIMyADaqfJ9l+n8CjKNKjx4/r\n7v37dfvOnbp7/349evy4omitmFm/fKmkuUymbZlqJqN8qSSp/3P/hvmzSBO6GYENqp8n2X5+9rAH\nU3Q7YrSfc/+G/bNIE8IM2KD6eZLt12f3+15cJwNnuh0xmi+VNHf0aNu5f81Xct3UdVTvS/aC0YzA\nBtXPCda9fHYnQdLPOnc7+rLTEaO9jGYc9s9ikJIazSh3H8hX41sBSIvTp0/79Pi4u7Tq16fGx/30\n6dN9/+xP3HyzT4+P+4lMxk/Hx09LfiKT8enxcf/EzTe7u/v0vn1Lx1f7Oi359L59qflZnGvfp8bH\n/ZGW9j2SyfinmtqXhp/FoMXZsO6MoZsR2KB6mWDdj8/uprusl3txnVzl9Hv4/F3Hji3VY6alHgeb\n6tHvn8UoI8yADazTk2w/P7ubIOn2XlynAySq5bIOdDD6cqZc7rnLrpO5f/38WYw6wgzY4Do5yfbz\ns7sJkm4GU4R4ldOvn8VGwDwzAEPVTZBMFgqq5nJty1ZzOU0WCj1d5bQziKucfv0sNgKuzAAMVTfd\nZd3ciwvxKqdfP4tehLZINEPzAQxVv4bx375zp6YffnjN73/7jTfq4w88kIrFgPv1s+jWIBeJZqFh\nACOhl2ewdXIvLk1XOZ3q18+iG6FOxibMAAxVv4Kk267Dfo7s7FQaQjXUVf7pZgSQCkl3l4X8HLFh\n3q/q9nlt60U3I4CRknR3WRqucnrVr+kSnYRkWqYpdKttmJnZpZIek7RZ0iWSvuLuH1uh3Ock3aTG\n38nvuvvJPtQVALqShq7DtOh0Anmok7HX7GY0s6y7R2a2SdLjkm5z98ebju+W9EF3321m75L0WXef\nXOFz6GYEgCHopst1bnZ2oAsYJ9XNuOakaXc/N5fwEkkXS3qtpcgeSffFZZ+QdIWZbV1vxQAAyehm\nUEeok7HXDDMzu8jMnpb0qqRvuPtzLUVykl5qev2ypG3JVREAsB7dPPl76V7j+LhOZDJLK6NEalyR\nTY+Pp/Je45oDQNz9DUnXmNmbJH3NzIruXmkp1nqJuGJ/4uHDh5e2i8WiisViN3UFAPSg20Ed/bzX\nWKlUVKlUen7/aroamm9mn5B0xt3/vGnfFyVV3P1v4tfPS7rB3V9teS/3zABgCAY93L4bA7lnZmZv\nMbMr4u0tkm6U1DpS8UFJ74vLTEr6UWuQAQCGJ18qaS6TaVsm9BX217pn9nZJj8b3zJ6Q9FV3P2Fm\nt5rZrZLk7g9J+r6ZvSDpiKQ/6GuNAQBdCXVQRzfa3jNz92cl/coK+4+0vP5gwvUCACQk5AnknWI5\nKwDYINL4WJek7pkRZgCAoRnYpGkAANKOMAMABI8wAwAEjzADAASPMAMABI8wAwAEjzADAASPMAMA\nBI8wAwAEjzADAASPMAMABI8wAwAEjzADAASPMAMABI8wAwAEjzADAASPMAMABI8wAwAEjzADAASP\nMAMABI8wAwAEjzADAASPMAMABI8wAwAEjzADAASPMAMABI8wAwAEjzADAASPMAMABI8wAwAEjzAD\nAASPMAMABI8wAwAEjzADAASPMAMABI8wAwAEjzADAASPMAMABI8wAwAEjzADAASPMAMABI8wAwAE\njzADAASPMAMABI8wAwAEjzADAASPMAMABI8wAwAEjzADAASPMAMABI8wAwAEjzADAASPMAMABI8w\nAwAEb80wM7MrzewbZnbKzL5tZh9aoUzRzP7JzE7GX7f3p7rpValUhl2Fvhrl9o1y2yTaF7pRb19S\nOrkyq0s64O7bJU1K+q9m9ksrlHvM3a+Nv6YTrWUARv0PbpTbN8ptk2hf6Ea9fUlZM8zc/Qfu/nS8\n/RNJ/1fSL6xQ1BKuGwAAHenqnpmZjUu6VtITLYdc0vVm9oyZPWRm70ymegAArM3cvbOCZj8nqSJp\n2t0faDl2uaSfuXtkZjdJ+qy7X91SprNvBADYUNx93T17HYWZmWUk/S9Jf+vun+mg/IuSrnP319Zb\nQQAA1tLJaEaTdK+k51YLMjPbGpeTme1QIyQJMgDAQGzqoMy7Jf22pG+Z2cl438cljUmSux+RVJL0\nATP7qaRI0i19qCsAACvq+J4ZAABp1fMKIGb238zsVTN7tmnfDjN7Mp44PW9mE/H+S83sS2b2LTN7\nzsw+2vSe68zsWTP7npl9dn3NSc4q7fvXZlaN2/FgPPDl3LGPxW143sx2Nu0Pvn1mdqOZPRXvf8rM\n/m3Te4JvX9PxMTP7iZn9YdO+1LWvh7/NfxUf+3Z8/JJ4f+raJnX9txniuWXFhSjM7M1m9rCZfdfM\nvm5mVzS9J5jzS7ftS+z84u49fUn6NTWG6T/btK8i6dfj7ZskfSPe/l1JX4q3t0h6UdJY/PpJSTvi\n7Yck7eq1Tkl+rdK+eUm/Fm+/X9Jd8fY7JT0tKSNpXNILOn/VOwrtu0bS2+Lt7ZJebnpP8O1rOl6W\n9D8k/WGa29fl726TpGck/XL8+l9IuiitbeuhfSGeW94m6Zp4++ckfUfSL0n6U0l/FO//Y0mfjreD\nOr/00L5Ezi89X5m5+zcl/WPL7lckvSnevkJSrWn/ZWZ2saTLJL0u6cdm9nZJl7v7k3G5o5J+q9c6\nJWmV9r0j3i9Jj0h6T7z9m2r8D1V39wU1/tjeNSrtc/en3f0H8f7nJG0xs8yotE+SzOy3JH1fjfad\n25fK9nXZtp2SvuXuz8bv/Ud3fyOtbZO6bl+I55aVFqLISdoj6b642H06X9+gzi/dti+p80vSCw1/\nVNJfmNmipD9TY6CI3P1rkn6sxh/egqQ/c/cfqdHAl5veX4v3pdUpM/vNePtmSVfG27+gC9vxshrt\naN0favuavUfS37l7XSPy+7PGHMo/knS4pXxI7Vvtd3e1JDez42b2d2b2kXh/SG2TVmlf6OcWu3Ah\niq3u/mp86FVJW+PtYM8vHbavWc/nl6TD7F5JH3L3MUkH4tcys99Wowvg7ZKuknSbmV2V8PcehN+T\n9Adm9pQal8+vD7k+SWvbPjPbLunTkm4dQt2SsFr7DkuacfdI4S7LtlrbNkn6VUn/Kf7vfzSzKTVW\n7QnJiu0L+dwS/yPqfkkfdvd/bj7mjX610H5HF+i2fes9v3QyNL8bO9z938XbZUn3xNvXS/qyu/9M\n0j+Y2f+RdJ2kxyVta3r/Np3vmkwdd/+OpF+XJDO7WtK/jw/VdOFVzDY1/kVR02i0T2a2TdL/lPRe\nd38x3h16+3bHh3ZIeo+Z/aka3eNvmNkZNdobRPva/O5ekjTr8bxPM3tI0q9I+u8KpG1S299dkOcW\nayxEcb+kv/LzKyq9amZvc/cfxF1sP4z3B3d+6bJ9iZxfkr4ye8HMboi3pyR9N95+Pn4tM7tMjdX3\nn4/7SX9sZu8yM5P0XkkPKKXM7K3xfy+SdLukL8SHHpR0i5ldEv+r8B2SnhyV9sWjjv63pD929+q5\n8u7+isJu3xclyd0L7n6Vu18l6TOS7nb3vwzp99fmb/Nrkn7ZzLaY2SZJN0g6FVLbpNV/dwrw3BLX\nZ6WFKB6U9Dvx9u/ofH2DOr90277Ezi/rGLHyJUn/T43L/ZfUGGH0b9ToG31aUlXStXHZzWr8S/BZ\nSad04Wix6+L9L0j63HpG0ST5tUL7fk/Sh9QYmfMdSX/SUv7jcRueVzyic1Tap8bJ4yeSTjZ9vWVU\n2tfyvjslHUzz76+Hv83/LOnbcTs+nea29fC3GeK55VclvRGfJ8/9/7RL0pvVGNzyXUlfl3RF03uC\nOb90276kzi9MmgYABC/pbkYAAAaOMAMABI8wAwAEjzADAASPMAMABI8wAwAEjzADAATv/wNxtuBj\npScp3gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1093250f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = pods.datasets.olympic_marathon_men()\n",
    "f, ax = plt.subplots(figsize=(7,7))\n",
    "ax.plot(data['X'], data['Y'], 'ro',markersize=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What is Machine Learning?\n",
    "\n",
    "$$ \\text{data} + \\text{model} = \\text{prediction}$$\n",
    "\n",
    "-   $\\text{data}$ : observations, could be actively or passively\n",
    "    acquired (meta-data).\n",
    "\n",
    "-   $\\text{model}$ : assumptions, based on previous experience (other data!\n",
    "    transfer learning etc), or beliefs about the regularities of\n",
    "    the universe. Inductive bias.\n",
    "\n",
    "-   $\\text{prediction}$ : an action to be taken or a categorization or a\n",
    "    quality score.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Regression: Linear Releationship\n",
    "\n",
    "$$y_i = m x_i + c$$\n",
    "\n",
    "-   $y_i$ : winning time/pace.\n",
    "\n",
    "-   $x_i$ : year of Olympics.\n",
    "\n",
    "-   $m$ : rate of improvement over time.\n",
    "\n",
    "-   $c$ : winning time at year 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Overdetermined System\n",
    "\n",
    "![](diagrams/straight_line7.svg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# $y = mx + c$\n",
    "\n",
    "point 1: $x = 1$, $y=3$ $$3 = m + c$$ \n",
    "point 2: $x = 3$, $y=1$ $$1 = 3m + c$$ \n",
    "point 3: $x = 2$, $y=2.5$ $$2.5 = 2m + c$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"diagrams/Pierre-Simon_Laplace.png\" align=center width=50%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# $y = mx + c + \\epsilon$\n",
    "\n",
    "point 1: $x = 1$, $y=3$ \n",
    "$$3 = m + c + \\epsilon_1$$ \n",
    "\n",
    "point 2: $x = 3$, $y=1$ \n",
    "$$1 = 3m + c + \\epsilon_2$$ \n",
    "\n",
    "point 3: $x = 2$, $y=2.5$ \n",
    "$$2.5 = 2m + c + \\epsilon_3$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The Gaussian Density\n",
    "- Perhaps the most common probability density.\n",
    "\\begin{align*}\n",
    "p(y| \\mu, \\sigma^2) & = \\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp\\left(-\\frac{(y - \\mu)^2}{2\\sigma^2}\\right)\\\\\n",
    "& \\buildrel\\triangle\\over = \\mathcal{N}(y|\\mu, \\sigma^2)\n",
    "\\end{align*}\n",
    "- The Gaussian density.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAc8AAAECCAYAAACYDlxBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYVNWd//H3l6VRZBcFWRQUMOCKyhI32kQdJIlJJvpo\nHDPRZCZOJsYlM78xMYtkkkk0sxk1MSaPOiZmjIkmjImKOhMaESMuICA7gg+IAoZVaGWR7++PU2VV\nF910Vdete6tufV7PUw/n3nvq3m8X3f3tc+4555q7IyIiIsXrlHQAIiIitUbJU0REpERKniIiIiVS\n8hQRESmRkqeIiEiJlDxFRERKVPXJ08wOMrM5ZvaymS02s++3Ue82M1thZvPNbGzccYqISP3oknQA\n7XH3d83sHHdvNrMuwDNmdqa7P5OtY2ZTgBHuPtLMJgB3AhOTillERNKt6lueAO7enCk2AJ2BzQVV\nLgTuy9SdA/QxswHxRSgiIvWkJpKnmXUys5eBDcAMd19cUGUwsDZv+3VgSFzxiYhIfamJ5Onu+9z9\nZEJCPNvMGlupZoVvq3hgIiJSl6r+nmc+d99mZo8CpwFNeYfWAUPztodk9r3PzJRMRURkP+5e2Phq\nV9W3PM2sv5n1yZQPBs4D5hVUewT460ydicBWd99QeC53r+vXTTfdlHgMSb/q/jMAbiJ0yyQei74X\nEn/pM+h4m6oWWp5HAPeZWSdCsv+Fu/+fmV0F4O53uftjZjbFzFYCO4ErE4xXRERSruqTp7svBE5p\nZf9dBdtXxxaUiIjUtarvtpXoNDY2Jh1C4vQZQGPSAVQJfS/oMyiHldPnW0vMzOvlaxVpk+WNi9DP\ngwhmhqdxwJCIiEi1UfIUEREpkZKniIhIiZQ8RURESqTkKSIiUiIlTxERkRIpeYqIiJRIyVNERKRE\nSp4iIiIlUvIUEREpkZKniIhIiZQ8RURESqTkKSIiUiIlTxERkRJV/cOwRSQa69fDN/kpe+jKGczm\n8nfg4IOTjkqkNul5niJ1YNMmmDQJFi3K7TvhBHjuOejePbm4RJKm53mKSKv27IGPfrRl4gRYuBC+\n+c1kYhKpdUqeIin3m9+EFmbWENa+X771VpgzJ4GgRGqckqdIyt1+e658I//CGo7kPJ4EYN8++M53\nEgpMpIbpnqdIir3wAowfH8oNDbBm9wAGsJHljORYlgPQuTO8+SYcdliCgYokRPc8RWQ/d9yRK19y\nCQxgIwCjWMHpp4f9770HDz6YQHAiNUzJUySl9uyBadNy21df3fL4Zz6TK99/fzwxiaRF1SdPMxtq\nZjPMbJGZvWJm17RSp9HMtpnZvMzrG0nEKlJNnn0Wtm8P5aOOgnHjWh6/+GLo2jWU58yBlSvjjU+k\nllV98gT2ANe7+3HAROBLZja6lXoz3X1s5vXdeEMUqT6PP54rX3ABWMFdnUMPhcmTc9vTp8cTl0ga\nVH3ydPf17v5yprwDWAIMaqVqyTd8RdIsP3nmJ8l855+fK8+YUdl4RNKk6pNnPjMbBowFCmemOXC6\nmc03s8fMbEzcsYlUkzfegAULQrlrV/jQh1qvl7+/qSlMXRGR9tXM2rZm1gN4CLg20wLNNxcY6u7N\nZnYBMA0YVXiOqVOnvl9ubGyksbGxYvGKJOmJJ3LlM8+Enj1brzd6NAwYABs2wObNIeGefHI8MYok\noampiaamprLPUxPzPM2sK/AH4HF3v7WI+quBU919c94+zfOUuvH5z8M994Ty978PX/1q5kD+jc/M\nz8Oll+amqvzHf8D118cXp0jSUjvP08wMuBtY3FbiNLMBmXqY2XjCHwWbW6srUg/yl+PLzudsyznn\n5Mp//GNl4hFJm6pveZrZmcDTwALCvU2AG4EjAdz9LjP7EvBFYC/QDHzF3Z8rOI9anlIXtm6Fvn1D\nuXNn2LYNDjkkc7CVlueKFTAqc5Ojd2/YsmX/kbkiadXRlmfVJ8+oKHlKvXjqqdwo2rFjYe7cvIOt\nJE/3sDTfpk1h94oVMGJEPLGKJC213bYiUpr8LtuJE9uvbwannZbbfvHF6GMSSRslT5GUyX/EWDHJ\nE5Q8RUql5CmSIu6ltzxByVOkVLrnKZIiq1bBMceEct++4T5mi8E/rdzzBHj9dRg6NJR79AiDjDrp\nT2upA7rnKSLMn58rn3JK8aNmBw8OiyUA7NgBy5dHH5tImih5iqRIdkk+gBNPLP59GjQkUholT5EU\nyW95nnRSae/NT54vvRRNPCJppeQpkiLlJM/8+q+8Ek08ImmlAUMiKbF9e1ghCKBLl3Dvslu3gkpt\nDBiC8DDskSNDecAAWL++crGKVAsNGBKpc/mtxQ98oJXE2Y7hw+Hgg0N5wwZ4663oYhNJGyVPkZQo\np8sWwjq4xx2X2160qPyYRNJKyVMkJfKTZykjbfMdf3yuvHBhefGIpJmSp0hK5Ce7jibPE07IlTVo\nSKRtSp4iKeAOS5bktvO7X0uR3/JU8hRpm0bbiqTAhg0wcGAo9+gRRt62urrQAUbbArz5JgwaFMq9\neoVng+rZnpJmGm0rUscWL86VR4/ueMIbOBD69Qvl7dthzZryYxNJIyVPkRTI77IdPbrj5zFr2eW7\ndGnHzyWSZkqeIimQnzzHjCnvXB/4QK6s5CnSOiVPkRSIquUJSp4ixVDyFEmBwnue5VDyFGlfl3JP\nYGYnA38BnAQMB/oABmwFVgEvAU+5+4I2TyIiHbZtWxglC2FJvuHDyzufkqdI+zqUPM2sM/BZ4Abg\nMOAZYDmwCNhEaNH2y7zOA24yszXAvwP/pTkjItHJ77IdNSosCl+Oo44KSXjXrrA4/Nat0KdPeecU\nSZuSf8zM7Fjg58Bi4NPAy+6+r533dAHGA9cDXzKzy9xdz6oXiUCU9zshrHE7alRuxaJly2DChPLP\nK5ImJd3zNLOJhNbjxe5+pbvPbS9xArj7Xnd/1t0vBv4K+JGZjSvymkPNbIaZLTKzV8zsmjbq3WZm\nK8xsvpmNLeXrEqllK1fmyqNGRXNOdd2KHFjRyTPTVXse8Al37/DUaXdfBnws8yrGHuB6dz8OmEho\nubb4+9rMpgAj3H0k8AXgzo7GJ1JrVqzIlbPP4yyXkqfIgRWdPN39PXf/jrvvLfei7v6uu3+ryLrr\n3f3lTHkHsAQYVFDtQuC+TJ05QB8zG1BunCK1IL/lOWJENOdU8hQ5sJqaqmJmw4CxwJyCQ4OBtXnb\nrwND4olKJDnulW95LlsWzTlF0qRmkqeZ9QAeAq7NtED3q1KwrRG9knobN8KOzE9Dr17Qv380581v\nwb76Krz3XjTnFUmLsud5xsHMugIPA/e7+7RWqqwDhuZtD8nsa2Hq1KnvlxsbG2lsbIw0TpG4FbY6\no3oCSq9ecPjhITnv3g2vvx6msIjUuqamJpqamso+T9U/kszMjHA/c5O7X99GnSnA1e4+JTMi+FZ3\nn1hQR9NLJXXuvRc+97lQvvRSeOCBdt7QziPJ8p15JsyeHcpPPQXnntvxOEWqVZofSXYGcDlwjpnN\ny7wuMLOrzOwqAHd/DFhlZiuBu4C/TzBekdhUYrBQVv790/wWrojUQLetuz9DEUne3a+OIRyRqlKJ\nwUJZ+clYyVOkpVpoeYpIG+JqeeZfR0SKTJ5mNsjMLjKzk/L2HWVmEzOjYEUkZpWaptLa+dTyFGmp\n3eRpZmcDK4BfA/PM7F8zh9YDRwDbKheeiLSlUtNUsvJbsqtWabqKSL5iWp7fIDxBpQ9wPDDQzG52\n913An9h/fqWIxKBS01SyevaEAZl1unbvhrVrD1xfpJ4UkzyfdfeH3H27uy92988AK8zsSrQQgUhi\nKnm/M0tdtyKtKyZ5bgcws6OzO9z9bmAj8JEKxSUi7ajk/c4sjbgVaV0xyXO2mX0fWJlZgAAAd38U\neBVobak8EakwtTxFklPM/Mk5wHeAk939uYJjM4GTzezCzAOvRSQmcbQ8NV1FpHVFTVVx92Z3X9DG\nsVXANOCDUQYmIm1zj6flqW5bkdZFuUhC1wjPJSIHsHEjvP12KPfqBYcdVpnrFE5X2Vv203xF0iHK\n5HmRmc0wsyVmNj2z9mxDhOcXkYz8VuCIEdFPU8nq2RMGDgzlPXs0XUUkK8rk+TngNeD3wLvAD4D5\nZjY6wmuICC27bCt1vzNLXbci+4syeX7L3a90939y908Ag4C7gcfMrEKdSiL1KY7BQq2dX8lTJIgq\neb4NLM/f4e473f3fgCuAmyK6jogQz2ChLCVPkf1FlTybgPNbO5CZzqJpLCIRirPlmZ+cNV1FJIgq\neX4D+KyZ/W0bxzVGTyQicU1TyVLLU2R/5h7N8rRmNoXw5JWFwH3A84TVh84Fznb3SyO5UAeZmUf1\ntYokacOG3AjYXr1g69YSRtvmVyzy52HHjjDqFqBLF3jnnfCvSBqYGe5e8nj1yAYMuftjwFhgHXAb\n8CKwFLgMuCaq64jUu8JWZ6WmqWT16JFL1nv3wpo1lb2eSC2IcrQt7r7C3S8CDiesOPQBdz/T3TdG\neR2Rehbn/c4s3fcUaSnS5Jnl7lvdfY67L2+/toiUIs77nVla41akpYokTxGpHLU8RZJX0dv+ZtYN\n+AVwKvBzoAGY7+6/ruR1RdKscGm+OCh5irRU6ZbnMOAioKe7f9vdvw6sNbN/r/B1RVKpcJqKWp4i\nyYhsqkqbFzA7B1jv7kvy9nVy930VvfD+cWiqitS8/GkqPXvCtm0ljrbtwFQVCNfp0yeUGxqguRk6\ndy7huiJVKvGpKm1x9xn5iTOzr+jEaWb3mNkGM1vYxvFGM9tmZvMyr2+UG7NItSpsdVZ6mkpW7965\nx57t3g3r1sVzXZFqVbHkaWajzeyvzKx3mae6F5jcTp2Z7j428/pumdcTqVpJDBbKUtetSE4kyTOT\nJM8xs4Pydi8HVgP/z8z+vqPndvdZwJb2Qujo+UVqSRLTVFq7npKn1LuoRtteDUwA9pjZ88DTwExg\ntrs/a2Z3RnSd1jhwupnNJ6xu9I/uvriC1xNJjFqeItUhkuTp7h/MPPT6LOBs4HLga4Cb2Z8J69xW\nylxgqLs3m9kFwDRgVGsVp06d+n65sbGRxsbGCoYlEj21PEXK09TURFNTU9nnqdhoWzMbRnhM2RXA\nue7eXOa5fu/uJxRRdzVwqrtvLtiv0bZS09zDwJ233w7bGzbA4YeXeJIOjrYFeP55mDAhlE84ARYs\nKPHaIlWo6kbbuvtr7v5T4PPAlyp1HTMbYBZ+I5jZeMIfBJvbeZtIzXnrrVzi7NkzN/o1LoUtT/0t\nKvWs4g8WcvclZvY3HX2/mT0ATAL6m9la4Caga+bcdxEWYfiime0FmoFEH30mUimF9zvjmqaS1a8f\n9O0LW7aEx5K9+SYMGhRvDCLVIpLkaWargE3AjMxrlrvvyKtyaEfP7e6fbuf4j4AfdfT8IrUiyfud\nWSNHhu7bbDxKnlKvouq2/TKwjPDszkeBLWY218weNLNZEV1DpK4lOdI2Kz9p58cjUm+iGm37KCFp\nYmYjgUbgnMy/A4EJmf1/JLRMZ7v7riiuLVIvklgQvpBG3IoEkd/zdPcVwArgZwBmdiy5ZPo3wNcz\nx4+N+toiaZbEgvCFlDxFgjgGDC0jdOneBWHZPuDISl9XJE3c1fIUqSYVf6pKtdA8T6ll69fDEUeE\ncu/eYcRrh0bbljHPE8J0mezc0h49YPv2+Ef9ikSpo/M8S2p5mtkA4KB2KxbnHXffGNG5RFJt+fJc\nedSo5BJW//7Qq1dImjt2wMaNMGBAMrGIJKnUbtsbgV4RXXs7cG1E5xJJtWXLcuVRrS4+GQ+z0HU7\nd27YXrlSyVPqU0nJ092V7EQSUNjyTFJ+8ly+HM44I9l4RJJQ8Ydhi0j5qil5Hps3Tj4/LpF6ouQp\nUgOqNXnmdyeL1JOKJU8zO9rMfmxmWmtWpAx798Krr+a2k06e+ddX8pR6FclUFTPrRliarx8wH5jp\n7nsyxyYTHhH2L2VfqAyaqiK16tVXc/MrBw2CdevKOFmZU1UgjLTt3TuUGxqguRk6dy4jJpEExTJV\n5QD+C/gIsI8wGnebmT0M3Ovu083soxFdR6TuVFOXLYSpKgMHhrmnu3fDa6/BMcckHZVIvKLqtt0L\n9HX3PsBo4GbgNGCWmW0CEloPRaT2Vcs0lXy67yn1Lqrk+aa7vwdhOT53v8XdTwZOBP4W+MuIriNS\nd6qt5QkacStScretmf0P8DQwE5jr7vuAHWY2yN3fyK/r7q8Ar0QSqUidqvbkqZan1KOOtDw/BnwX\neJ7w3M7Hga7Ag2Y2JsrgRETJU6QalTza1sxmA5OA8YRHjTUCHwQOAfYQntc5A2gCXsi0TBOn0bZS\ni5qb4ZBDQrlz57Dd0FDGCSMYbQvhCS/ZRF72CGCRBHV0tG1HkudZ7j6rYF9XYBz7J9MdwDPA/e7+\n36UGFyUlT6lFCxbASSeF8siREdxfjCh57t0L3bvDnj1he/t26NmzzNhEEtDR5Flyt21h4szs2+Pu\nz7r799z9fMJ8zzMJo267ANeUeh0Rqc4uW4AuXVpOT9GgIak3FVlhqDCZuvvESlxHJO2qNXmC7ntK\nfdPatiJVrBrneGZpuorUs6KTp5l1NrMrorioBerKFWmHWp4i1ano5JlZBGG7md1qZgd19IJm1hf4\nDbCkyPr3mNkGM1t4gDq3mdkKM5tvZmM7GptItVHyFKlOJXXbuvtvgd8BM83smkwiLIqZDTKzWwiL\nK9zi7k8V+dZ7gckHOO8UYIS7jwS+ANxZbEwi1WzTJti8OZS7dw9TQqpJYbetBrNLPSl5hSF3n2lm\n5wE3AivNbDXwLLAQ2Jp5dSKMuD0UGAOcDQwE7gAmuntzCdebZWbDDlDlQuC+TN05ZtbHzAa4+4ZS\nvzaRarIkr29m1CjoVGUjFA49FPr2hS1bYOfOMNdzyJCkoxKJR4eequLu24Gvmtl3CE9TOY/Q6hsG\n9AackERXE+Z5XgfMcvddEcRcaDCwNm/7dWAIoOQpNW3x4lx5TBWu3WUWWp/PPRe2ly1T8pT6UdYj\nydx9J/DrzCtJhRNc1YEkNa/akye0TJ7Ll8OHP5xsPCJxiep5nklaBwzN2x6S2befqVOnvl9ubGyk\nsbGxknGJlKVWkmfW0qXJxSFSrKamJpqamso+T8nL87V6krA839eAS4CDgKXAA8B/R7G2beae5+/d\n/YRWjk0Brnb3KWY2Ebi1tUUZtDyf1JohQ3Jrxi5d2jJRdVhEy/NlTZsGn/xkKJ97LjxV7DBAkSoR\n29q2bVz8TuAU4FXgaGAs4Ukr84CL3X1VGed+gLAQfX/CfcybMufG3e/K1LmDMCJ3J3Clu89t5TxK\nnlIztm2DPn1CuWvXsCB8lyj6iSJOnitXhjV3AQYOhDffLPuUIrHqaPKMqtu2s7tPyAumO/Ah4EtA\nk5mN6+joV3f/dBF1ru7IuUWqVf5I22OPjShxVsDw4XDQQfDuu7B+fZha069f0lGJVF5Ug9+35G+4\ne7O7/8HdLwC+T3j+p4gUqRbud0J4TNro0bntRYuSi0UkTlElz5fMrNVxdu5+J1CJKSoiqVUryRPg\nuONy5VdeSS4OkThF1Rn0DPC4mf0eeBJ4zt13V+A6InWhlpLn8cfnymp5Sr2IquX5S2A38HdAE7DN\nzP5oZv9sZr8FHsqvbGY3RHRdkVTKb8FVe/LMb3kqeUq9iGq07e3u/mUzM8JyfJMyr7OBAYRu2xeA\nWYSl/P7V3WP9laDRtlIrtmzJDbppaAhL30U2YCji0bYAq1fD0UeHcv/+8NZbkZxWJBZJT1X5ONBI\nSI6P5i/DZ2ajyCXTSYTl9NzdO5d94dJiVPKUmvD00zBpUiiPHQtz95t4VYYKJM99+6BnzzCdBmDD\nBjj88EhOLVJxHU2ekXTbuvv/ADcA24G+BceWu/vP3P1ydx8KHAusieK6Imm0YEGufOKJycVRrE6d\nWnbdLmzz4YEi6RHZcxrcfbe7/6+7r2+n3grg1qiuK5I2tZY8AU46KVeePz+5OETikshDjtz9h0lc\nV6QW1HryfPnl5OIQiUuVPSFQpL7t29ey2/OE/VZzrk4nn5wrK3lKPYhkwFAt0IAhqQX5a8UefngY\nfBOpCgwYAti+HXr3DuUuXWDHDujWLbLTi1RMogOGRCQatdhlC9CrV266yt69LRd5EEkjJU+RKpI/\nLaWWkieo61bqi5KnSBV56aVc+bTTkoujIzTiVuqJkqdIlXBvmTxPPTW5WDoiv+U5b15ycYjEQQOG\nRKrE2rVw5JGh3LMnbN0aFiCIVIUGDAGsWQNHHRXKFYtfJGIaMCRS4/JbnaecUnuJZ+jQ3LJ8b78N\ny5YlG49IJdXYj6dIetVyly2ERu24cbntF15ILhaRSlPyFKkStZ48AcaPz5WVPCXNlDxFqkDhYKFa\nG2mbld/yfP755OIQqTQNGBKpAq+9BsOHh3KvXuGZnhW551nBAUMAf/4zHHZYKDc0hHufDQ2RX0Yk\nMhowJFLDnn02V544sfYGC2X175/7I2D3bj2eTNKrRn9ERdIlP3l+8IPJxRGF/K7bOXOSi0OkkpQ8\nRapAfvI8/fTk4ojCxIm58uzZycUhUkk1kTzNbLKZLTWzFWZ2QyvHG81sm5nNy7y+kUScIh2xY0du\nOTszmDAh2XjKdeaZufIzzyQXh0gldUk6gPaYWWfgDuBcYB3wgpk94u5LCqrOdPcLYw9QpEzPPx+e\n4wlw/PG5R3vVqrFj4ZBDYOfOsOrQmjW5lZNE0qIWWp7jgZXu/pq77wF+BXy8lXolj5YSqQZ/+lOu\nXOtdthCe55l/33bWrORiEamUWkieg4G1eduvZ/blc+B0M5tvZo+Z2ZjYohMp09NP58ppSJ6grltJ\nv6rvtiUkxvbMBYa6e7OZXQBMA0YVVpo6der75cbGRhobGyMKUaRjdu9u2TKbNCm5WKJ01lm5slqe\nUk2amppoamoq+zxVv0iCmU0Eprr75Mz214B97n7LAd6zGjjV3Tfn7dMiCVJ1nn46lzCPOQZWrqzw\nBSu8SEJWc3O4d7t3b9h+660wB1Sk2qR5kYQXgZFmNszMGoBLgEfyK5jZALPwW8HMxhP+KNi8/6lE\nqssf/5grf/jDycURte7dW65z+7//m1wsIpVQ9cnT3fcCVwNPAIuBB919iZldZWZXZapdBCw0s5eB\nW4FLk4lWpDT/93+58oc+lFwclXD++bnyk08mF4dIJVR9t21U1G0r1WbnTujTJ9e1uWFD7nmYFRNT\nty2EUcTZAVCDB4eHfZvGxEuVSXO3rUgqNTXlEucJJ8SQOGM2blz44wBg3TpYUjgzW6SGKXmKJOT3\nv8+V87s406JLl5b3cZ94IrlYRKKm5CmSgH37WibPj7e27EcK5P9R8PjjycUhEjXd8xRJwIsv5p4+\ncuihsH59aKlVXIz3PCHc58wuzdelC2zcCH37VvyyIkXTPU+RGpLf6pwyJabEmYChQ3NTVvbuhUce\nOXB9kVqh5CmSgGnTcuULU/44g099Klf+7W+Ti0MkSuq2FYnZokXh6SkA3bqFrsxevWK6eMzdthBW\nTRo5MpS7dQurDfXsGculRdqlbluRGnH//bnyhRfGmDgTMmIEnHhiKO/apa5bSQclT5EY7dsHv/xl\nbvvyy5OLJU6XXJIr33NPcnGIREXdtiIxamqCc84J5UMPhTfegIaGGANIoNsWwiIJRx6Ze+j3q6/C\n0UfHdnmRNqnbVqQG/PSnufKll8acOBM0eDBMnpzbvvfe5GIRiYJaniIxWbcOhg3LLcn30ktwyikx\nB5FQyxPg4YfhootCefBgWL0aunaNNQSR/ajlKVLl7rwzlzjPOiuBxJmwj30st37vunXwwAPJxiNS\nDiVPkRg0N8NPfpLbvvba5GJJSkNDy6/7llty90BFao2Sp0gMbr8dNm0K5SOPTO9atu354hehR49Q\nXrwY/vCHZOMR6SglT5EK27IFbr45t33jjeldjq89ffvC3/1dbvvGG3Nd2SK1RMlTpMK+9z3YujWU\nR46Ez30u2XiS9g//kGt9LloEd92VbDwiHaHkKVJBL70E//mfue3vflcjTAcOhK9/Pbf9zW+GJQpF\naommqohUyO7dcNppsHBh2D77bJgxAzol+SdrglNV8r37LowZE6arQHiyzB/+0DI8kThoqopIFXGH\nL385lzgPPhjuvjvhxFlFDjqo5ejjxx6D225LLh6RUulHWaQCbrut5WpCN98cFkiXnPPPh698Jbf9\nla/A736XXDwipVC3rUjEfvhDuO663PZll4UnqVRFl2SVdNtm7doVFox44YWw3a0b/PrX6X/GqVQP\ndduKJOzdd+Gaa1omzgkTQndtVSTOKtStW7jXmW2V79oFn/gE/OAHWkBBqltNJE8zm2xmS81shZnd\n0Ead2zLH55vZ2LhjlPr2xBNhcNDtt+f2nXEGTJ8e7u9J2w4/HJ58EoYPD9vucMMNYYDV3LnJxibS\nlqpPnmbWGbgDmAyMAT5tZqML6kwBRrj7SOALwJ2xB1oDmpqakg4hcVF+Blu2wH33wcSJ4Ykhixbl\njn3ykyGh9ukT2eUi05R0AK0YPhzmzIHTT8/tmz0bTj0VLrgAHnooLHEYJf086DMoR9UnT2A8sNLd\nX3P3PcCvgMLFzS4E7gNw9zlAHzMbEG+Y1U8/KKV/Bu7w9tuwfDnMnBlGiH7hCzBuHBx2GFxxRfil\nn9W9O/z4x+EJIoccEmnokWlKOoA2HHZYmMrzrW+1XIFp+nS4+GLo1y88C/W66+BnPwvJdfVq2Lmz\nY9fTz4M+g3LUwiJhg4G1eduvAxOKqDME2JBf6aMf3f/kBxozkbZjq1bBrFnJx5HksTVrQouwrffs\n3h1aOM3N4Zfyzp3hPlx7Ghrgs5+Fb38bjjii/frSuoaG8Bleckn49ze/yf1f7toVHibe2u/7gw+G\nnj3DvwcdFP49+OAwNahTp3DPufDfVavgueda7qs3y5fDiy8mHUVtqoXkWeyQwMJv/f3e9+ij5QdT\n6157LekIkrduXXTnGj8ePvWp0ALNPm5LyjdmDDz4YFja8Oc/D0l0yZK267/zTniVatWqjseYFitW\nJB1BbaopLYfaAAAHkElEQVT6qSpmNhGY6u6TM9tfA/a5+y15dX4CNLn7rzLbS4FJ7r4hr051f6Ei\nIpKIjkxVqYWW54vASDMbBrwBXAJ8uqDOI8DVwK8yyXZrfuKEjn04IiIiran65Onue83sauAJoDNw\nt7svMbOrMsfvcvfHzGyKma0EdgJXJhiyiIikXNV324qIiFSbWpiqIiIiUlVSlTzN7B4z22BmC9up\nN87M9prZX8YVW1za+wzMrNHMtpnZvMzrG3HHWGnFfB9kPod5ZvaKmTXFGF4sivg++Me874GFmZ+H\nKlzSoeOK+Az6m9l0M3s5831wRcwhxqKIz6Gvmf0uszrbHDM7Lu4YK8nMhprZDDNblPl/vqaNeiWt\nUpeq5AncS1iJqE2ZFYtuAaaz//SWNGj3MwBmuvvYzOu7cQQVswN+Bpkk8SPgY+5+PHBRXIHF6ICf\ngbv/W/Z7APgaYbT61tiii0d7PwtXA/Pc/WSgEfh3M6v6cSAd0N7ncCMw191PAv4a+GEsUcVnD3C9\nux8HTAS+FMUqdalKnu4+C9jSTrUvAw8Bb1U+ovgV+Rmk8Y+G9xXxGVwGPOzur2fq/zmWwGJU5PdB\n1mXAAxUMJxFFfAZvAr0y5V7AJnffW/HAYlbE5zAamJGpuwwYZmaHxRFbHNx9vbu/nCnvAJYAgwqq\nlbxKXaqSZ3vMbDBhab/sXxX1OFrKgdMzXROPmdmYpANKwEigX6Yr50Uz+0zSASXFzLoDfwE8nHQs\nCfgZcJyZvQHMB65NOJ6kzAf+EsDMxgNHEVZoS53MlMexwJyCQ22tUtemNHZRHMitwFfd3c3MSHkL\nrA1zgaHu3mxmFwDTgFEJxxS3rsApwIeB7sCfzOw5d6/HtVY+BjyTwi7bYtwIvOzujWZ2DPCUmZ3k\n7m8nHVjMbgZ+aGbzgIXAPOC9ZEOKnpn1IPQ6Xptpge5XpWD7gI2rekuepxIWUgDoD1xgZnvc/ZFk\nw4pP/i8Gd3/czH5sZv3cfXOSccVsLfBnd38HeMfMngZOAuoxeV5KCrtsi3Q68C8A7v6qma0GjiUs\nzFI3Mr8TPpfdznwOqVq40My6EnpX7nf3aa1UWQcMzdsektnXprrqtnX3o919uLsPJ/wF8sV6SpwA\nZjYg0+rOdtFYnSVOgP8BzjSzzpluywnA4oRjip2Z9QbOJnwe9WgpcC6EnwtC4kxV0iiGmfU2s4ZM\n+W8JAwpba5nVpMzvu7uBxe5+axvVHiEMlsouCbvfKnWFUtXyNLMHgElAfzNbC9xE6KLD3e9KMra4\nFPEZXAR80cz2As2ElkeqtPcZuPtSM5sOLAD2AT9z91QlzyJ/Fj4BPJFpgadOEZ/B94B7zWw+oSHx\nT2n8Q7KIz2EM8F+Z9b9fAT6fVKwVcgZwObAg0zUNocv+SOj4KnVaYUhERKREddVtKyIiEgUlTxER\nkRIpeYqIiJRIyVNERKRESp4iIiIlUvIUEREpkZKniIhIiZQ8RURESqTkKVJnzOyTSccgUuuUPEXq\niJl9gMwaniLScUqeIjXEzM4ys9lmts/MOrKg+2XAf0cdl0i9UfIUqSHuPouwyPcu4OkOnOIjhCdI\niEgZlDxFas94oBsws5Q3ZR5Bt8jdd1UkKpE6ouQpUnsagbeBuSW+76+AX0YejUgd0iPJRGqMmT0J\nvOfuF5Twns6EZDvW3fdVLDiROqHkKVJDzKwrsBm4FdgD9ACOIDzY/oq2umTN7DzgQnf/csH+jwMf\nBk4CPgscSnhgugOnA/8GTAeuyxw7HGgArnT3vVF/fSK1okvSAYhIScYBhxC6bi9x9zcyrcothJG0\n97bxvsuAn+bvMLMGoNHdrzGzF4BfAL91969ljv8TcHdm/23uvsbMOgFbM+f7edRfnEit0D1PkdrS\nSGhxXuXubwC4+3vAPqBfa28ws4OAk939TwWHzgZmmZkBRwNvuvt/5h3fmznnL919TeZa+4D3CC1Q\nkbqllqdIbWkEZrj74uwOMzsW6AXMb+M9HwEea2X/K4RW5AlAX+CHBcfHAXPcfV7etY4GegOLOhi/\nSCqo5SlSIzL3Oz8IzCg49AlgG21PXbmUVkbZuvt6d38X+BDwDjCnoEoj0FSwbzLw7gGuJVIXlDxF\nakf2fmdTwf5PAw+7+55My/B9ZtYbODK/pdqKc4Bn8wcAmdloYEAr1/ok8Li7N5vZsA58DSKpoOQp\nUjsagR3AC9kdZnY8cCK5gUL/WPCeTwEPt3XCzACgs9k/SZ5DuLc6O69uv0wM2VbsV0oLXyQ9lDxF\nakcj8EzBPM1RwBZ3n21mk9l/yb5LgAcOcM6xhHuYTQX7zwGed/d38vYNAzoDT5nZJPKSuEi9UfIU\nqR29gPsL9k0HXjCzO4BT3f1X2QNmNhDo5u5rD3DOwYSBQ4X3O/uz/1SUl4GHgB8AZ7j7L0r/EkTS\nQYskiKSUmV0HvOvuP0k6FpG00VQVkfS6CLgw6SBE0kjdtiIpZGYjgK3uvjnpWETSSMlTJJ300GuR\nClLyFEmn04BpSQchklYaMCQiIlIitTxFRERKpOQpIiJSIiVPERGREil5ioiIlEjJU0REpERKniIi\nIiVS8hQRESnR/wdQjfEN0tRcSQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x109480ac8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "h = np.linspace(0, 2.5, 1000)\n",
    "sigma2 = 0.0225\n",
    "mu = 1.7\n",
    "p = 1./np.sqrt(2*np.pi*sigma2)*np.exp(-(h-mu)**2/(2*sigma2**2))\n",
    "f2, ax2 = plt.subplots(figsize=(7, 3.5))\n",
    "ax2.plot(h, p, 'b-', linewidth=3)\n",
    "ylim = (0, 3)\n",
    "ax2.vlines(mu, ylim[0], ylim[1], colors='r', linewidth=3)\n",
    "ax2.set_ylim(ylim)\n",
    "ax2.set_xlim(1.4, 2.0)\n",
    "ax2.set_xlabel('$h/m$', fontsize=20)\n",
    "ax2.set_ylabel('$p(h|\\mu, \\sigma^2)$', fontsize = 20)\n",
    "f2.savefig('./diagrams/gaussian_of_height.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Gaussian Density\n",
    "![](./diagrams/gaussian_of_height.svg)\n",
    "The Gaussian PDF with $\\mu=1.7$ and variance $\\sigma^2=\n",
    "  0.0225$. Mean shown as red line. It could represent the heights of a population of\n",
    "  students."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Gaussian Density\n",
    "$$\n",
    "\\mathcal{N}(y|\\mu, \\sigma^2) =  \\frac{1}{\\sqrt{2\\pi\\sigma^2}} \\exp\\left(-\\frac{(y-\\mu)^2}{2\\sigma^2}\\right)\n",
    "$$\n",
    "$\\sigma^2$ is the variance of the density and $\\mu$ is the mean.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "### Two Important Gaussian Properties\n",
    "\n",
    "**Sum of Gaussian**\n",
    "\n",
    "-   Sum of Gaussian variables is also Gaussian.\n",
    "    $$y_i \\sim \\mathcal{N}(\\mu, \\sigma^2)$$ \n",
    "    And the sum is distributed as\n",
    "    $$\\sum_{i=1}^{n} y_i \\sim \\mathcal{N}\\left(\\sum_{i=1}^n \\mu_i,\\sum_{i=1}^n \\sigma_i^2\\right)$$\n",
    "    (*Aside*: As sum increases, sum of non-Gaussian, finite variance variables is\n",
    "    also Gaussian [central limit theorem](https://en.wikipedia.org/wiki/Central_limit_theorem).)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Two Important Gaussian Properties\n",
    "\n",
    "**Scaling a Gaussian**\n",
    "\n",
    "-   Scaling a Gaussian leads to a Gaussian.\n",
    "    $$y \\sim \\mathcal{N}(\\mu, \\sigma^2)$$\n",
    "    And the scaled density is distributed as\n",
    "    $$w y \\sim \\mathcal{N}(w\\mu,w^2 \\sigma^2)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Laplace's Idea\n",
    "\n",
    "### A Probabilistic Process\n",
    "\n",
    "-   Set the mean of Gaussian to be a function.\n",
    "    $$p\\left(y_i|x_i\\right)=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp \\left(-\\frac{\\left(y_i-f\\left(x_i\\right)\\right)^{2}}{2\\sigma^2}\\right).$$\n",
    "\n",
    "-   This gives us a ‘noisy function’.\n",
    "\n",
    "-   This is known as a stochastic process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Height as a Function of Weight\n",
    "\n",
    "-   In the standard Gaussian, parametized by mean and variance.\n",
    "\n",
    "-   Make the mean a linear function of an *input*.\n",
    "\n",
    "-   This leads to a regression model. \n",
    "    \\begin{align*}\n",
    "       y_i=&f\\left(x_i\\right)+\\epsilon_i,\\\\\n",
    "         \\epsilon_i \\sim &\\mathcal{N}(0, \\sigma^2).\n",
    "     \\end{align*}\n",
    "        \n",
    "-   Assume $y_i$ is height and $x_i$ is weight."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Data Point Likelihood\n",
    "\n",
    "-   Likelihood of an individual data point\n",
    "    $$p\\left(y_i|x_i,m,c\\right)=\\frac{1}{\\sqrt{2\\pi \\sigma^2}}\\exp \\left(-\\frac{\\left(y_i-mx_i-c\\right)^{2}}{2\\sigma^2}\\right).$$\n",
    "\n",
    "-   Parameters are gradient, $m$, offset, $c$ of the function and noise\n",
    "    variance $\\sigma^2$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Data Set Likelihood\n",
    "\n",
    "-   If the noise, $\\epsilon_i$ is sampled independently for each\n",
    "    data point.\n",
    "\n",
    "-   Each data point is independent (given $m$ and $c$).\n",
    "\n",
    "-   For independent variables:\n",
    "    $$p(\\mathbf{y}) = \\prod_{i=1}^n p(y_i)$$\n",
    "    $$p(\\mathbf{y}|\\mathbf{x}, m, c) = \\prod_{i=1}^n p(y_i|x_i, m, c)$$\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### For Gaussian \n",
    "\n",
    "- i.i.d. assumption\n",
    "    \n",
    "    $$p(\\mathbf{y}|\\mathbf{x}, m, c) = \\prod_{i=1}^n \\frac{1}{\\sqrt{2\\pi \\sigma^2}}\\exp \\left(-\\frac{\\left(y_i-mx_i-c\\right)^{2}}{2\\sigma^2}\\right).$$\n",
    "    $$p(\\mathbf{y}|\\mathbf{x}, m, c) = \\frac{1}{\\left(2\\pi \\sigma^2\\right)^{\\frac{n}{2}}}\\exp \\left(-\\frac{\\sum_{i=1}^n\\left(y_i-mx_i-c\\right)^{2}}{2\\sigma^2}\\right).$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Log Likelihood Function\n",
    "\n",
    "-   Normally work with the log likelihood:\n",
    "    $$L(m,c,\\sigma^{2})=-\\frac{n}{2}\\log 2\\pi -\\frac{n}{2}\\log \\sigma^2 -\\sum _{i=1}^{n}\\frac{\\left(y_i-mx_i-c\\right)^{2}}{2\\sigma^2}.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Consistency of Maximum Likelihood\n",
    "\n",
    "\n",
    "-   If data was really generated according to probability we specified.\n",
    "\n",
    "-   Correct parameters will be recovered in limit as\n",
    "    $n \\rightarrow \\infty$.\n",
    "\n",
    "-   This can be proven through sample based approximations (law of\n",
    "    large numbers) of “KL divergences”.\n",
    "\n",
    "-   Mainstay of classical statistics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Probabilistic Interpretation of the Error Function\n",
    "\n",
    "-   Probabilistic Interpretation for Error Function is Negative\n",
    "    Log Likelihood.\n",
    "\n",
    "-   *Minimizing* error function is equivalent to *maximizing*\n",
    "    log likelihood.\n",
    "\n",
    "-   Maximizing *log likelihood* is equivalent to maximizing the\n",
    "    *likelihood* because $\\log$ is monotonic.\n",
    "\n",
    "-   Probabilistic interpretation: Minimizing error function is\n",
    "    equivalent to maximum likelihood with respect to parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Error Function\n",
    "\n",
    "-   Negative log likelihood is the error function leading to an error\n",
    "    function\n",
    "    $$E(m,c,\\sigma^{2})=\\frac{n}{2}\\log \\sigma^2 +\\frac{1}{2\\sigma^2}\\sum _{i=1}^{n}\\left(y_i-mx_i-c\\right)^{2}.$$\n",
    "\n",
    "-   Learning proceeds by minimizing this error function for the data\n",
    "    set provided."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Connection: Sum of Squares Error\n",
    "\n",
    "-   Ignoring terms which don’t depend on $m$ and $c$ gives\n",
    "    $$E(m, c) \\propto \\sum_{i=1}^n (y_i - f(x_i))^2$$\n",
    "    where $f(x_i) = mx_i + c$.\n",
    "\n",
    "-   This is known as the *sum of squares* error function.\n",
    "\n",
    "-   Commonly used and is closely associated with the\n",
    "    Gaussian likelihood."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Reminder\n",
    "\n",
    "- Two functions involved:\n",
    "  - Prediction function: $f(x_i)$\n",
    "  - Error, or Objective function: $E(m, c)$\n",
    "- Error function depends on parameters through prediction function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Mathematical Interpretation\n",
    "\n",
    "-   What is the mathematical interpretation?\n",
    "\n",
    "    -   There is a cost function.\n",
    "\n",
    "    -   It expresses mismatch between your prediction and reality.\n",
    "        $$E(m, c)=\\sum_{i=1}^n \\left(y_i - mx_i -c\\right)^2$$\n",
    "\n",
    "    -   This is known as the sum of squares error."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Learning is Optimization\n",
    "\n",
    "-   Learning is minimization of the cost function.\n",
    "\n",
    "-   At the minima the gradient is zero.\n",
    "\n",
    "-   Coordinate ascent, find gradient in each coordinate and set to zero.\n",
    "    $$\\frac{\\text{d}E(m)}{\\text{d}m} =\n",
    "          -2\\sum_{i=1}^n x_i\n",
    "          \\left(y_i- m x_i - c \\right)$$\n",
    "    $$0 =\n",
    "          -2\\sum_{i=1}^n x_i\n",
    "          \\left(y_i-\n",
    "            m x_i - c \\right)$$            \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Learning is Optimization\n",
    "\n",
    "- Fixed point equations\n",
    "    $$0 =\n",
    "          -2\\sum_{i=1}^n x_iy_i\n",
    "          +2\\sum_{i=1}^n\n",
    "            m x_i^2 +2\\sum_{i=1}^n cx_i$$\n",
    "    $$m  =    \\frac{\\sum_{i=1}^n \\left(y_i\n",
    "          -c\\right)x_i}{\\sum_{i=1}^nx_i^2}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Learning is Optimization\n",
    "\n",
    "-   Learning is minimization of the cost function.\n",
    "\n",
    "-   At the minima the gradient is zero.\n",
    "\n",
    "-   Coordinate ascent, find gradient in each coordinate and set to zero.\n",
    "     $$\\frac{\\text{d}E(c)}{\\text{d}c} =\n",
    "          -2\\sum_{i=1}^n \n",
    "          \\left(y_i- m x_i - c \\right)$$\n",
    "    $$0 =\n",
    "          -2\\sum_{i=1}^n\\left(y_i-\n",
    "            m x_i - c \\right)$$\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Learning is Optimization\n",
    "\n",
    "- Fixed point equations\n",
    "    $$0 = -2\\sum_{i=1}^n y_i +2\\sum_{i=1}^n m x_i +2n c$$\n",
    "    $$c  =    \\frac{\\sum_{i=1}^n \\left(y_i\n",
    "          -mx_i\\right)}{n}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Fixed Point Updates\n",
    "\n",
    "Worked example. $$\\begin{aligned}\n",
    "    c^{*}=&\\frac{\\sum _{i=1}^{n}\\left(y_i-m^{*}x_i\\right)}{n},\\\\\n",
    "    m^{*}=&\\frac{\\sum _{i=1}^{n}x_i\\left(y_i-c^{*}\\right)}{\\sum _{i=1}^{n}x_i^{2}},\\\\\n",
    "    \\left.\\sigma^2\\right.^{*}=&\\frac{\\sum _{i=1}^{n}\\left(y_i-m^{*}x_i-c^{*}\\right)^{2}}{n}\n",
    "  \\end{aligned}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Important Concepts Not Covered\n",
    "\n",
    "-   Other optimization methods:\n",
    "\n",
    "    -   Second order methods, conjugate gradient, quasi-Newton\n",
    "        and Newton.\n",
    "\n",
    "    -   Effective heuristics such as momentum.\n",
    "\n",
    "-   Local vs global solutions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Reading\n",
    "\n",
    "- Section 1.1-1.2 of @Rogers:book11 for fitting linear models. \n",
    "- Section 1.2.5 of @Bishop:book06 up to equation 1.65.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Multi-dimensional Inputs\n",
    "\n",
    "-   Multivariate functions involve more than one input.\n",
    "\n",
    "-   Height might be a function of weight and gender.\n",
    "\n",
    "-   There could be other contributory factors.\n",
    "\n",
    "-   Place these factors in a feature vector $\\mathbf{x}_i$.\n",
    "\n",
    "-   Linear function is now defined as\n",
    "    $$f(\\mathbf{x}_i) = \\sum_{j=1}^p w_j x_{i, j} + c$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Vector Notation\n",
    "\n",
    "-   Write in vector notation,\n",
    "    $$f(\\mathbf{x}_i) = \\mathbf{w}^\\top \\mathbf{x}_i + c$$\n",
    "\n",
    "-   Can absorb $c$ into $\\mathbf{w}$ by assuming extra input $x_0$\n",
    "    which is always 1.\n",
    "    $$f(\\mathbf{x}_i) = \\mathbf{w}^\\top \\mathbf{x}_i$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Log Likelihood for Multivariate Regression\n",
    "\n",
    "-   The likelihood of a single data point is\n",
    "    $$p\\left(y_i|x_i\\right)=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp\n",
    "        \\left(-\\frac{\\left(y_i-\\mathbf{w}^{\\top}\\mathbf{x}_i\\right)^{2}}{2\\sigma^2}\\right).$$\n",
    "\n",
    "-   Leading to a log likelihood for the data set of\n",
    "    $$L(\\mathbf{w},\\sigma^2)= -\\frac{n}{2}\\log \\sigma^2\n",
    "          -\\frac{n}{2}\\log 2\\pi -\\frac{\\sum\n",
    "            _{i=1}^{n}\\left(y_i-\\mathbf{w}^{\\top}\\mathbf{x}_i\\right)^{2}}{2\\sigma^2}.$$\n",
    "\n",
    "-   And a corresponding error function of\n",
    "    $$E(\\mathbf{w},\\sigma^2)= \\frac{n}{2}\\log\n",
    "          \\sigma^2 + \\frac{\\sum\n",
    "            _{i=1}^{n}\\left(y_i-\\mathbf{w}^{\\top}\\mathbf{x}_i\\right)^{2}}{2\\sigma^2}.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Expand the Brackets\n",
    "\n",
    "\\begin{align*}\n",
    "  E(\\mathbf{w},\\sigma^2)  = & \\frac{n}{2}\\log \\sigma^2 + \\frac{1}{2\\sigma^2}\\sum _{i=1}^{n}y_i^{2}-\\frac{1}{\\sigma^2}\\sum _{i=1}^{n}y_i\\mathbf{w}^{\\top}\\mathbf{x}_i\\\\&+\\frac{1}{2\\sigma^2}\\sum _{i=1}^{n}\\mathbf{w}^{\\top}\\mathbf{x}_i\\mathbf{x}_i^{\\top}\\mathbf{w} +\\text{const}.\\\\\n",
    "    = & \\frac{n}{2}\\log \\sigma^2 + \\frac{1}{2\\sigma^2}\\sum _{i=1}^{n}y_i^{2}-\\frac{1}{\\sigma^2}\n",
    "  \\mathbf{w}^\\top\\sum_{i=1}^{n}\\mathbf{x}_iy_i\\\\&+\\frac{1}{2\\sigma^2} \\mathbf{w}^{\\top}\\left[\\sum\n",
    "    _{i=1}^{n}\\mathbf{x}_i\\mathbf{x}_i^{\\top}\\right]\\mathbf{w} +\\text{const}.\n",
    "    \\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Multivariate Derivatives\n",
    "\n",
    "-   We will need some multivariate calculus.\n",
    "\n",
    "-   For now some simple multivariate differentiation:\n",
    "    $$\\frac{\\text{d}{\\mathbf{a}^{\\top}}{\\mathbf{w}}}{\\text{d}\\mathbf{w}}=\\mathbf{a}$$\n",
    "    and\n",
    "    $$\\frac{\\mathbf{w}^{\\top}\\mathbf{A}\\mathbf{w}}{\\text{d}\\mathbf{w}}=\\left(\\mathbf{A}+\\mathbf{A}^{\\top}\\right)\\mathbf{w}$$\n",
    "    or if $\\mathbf{A}$ is symmetric (*i.e.*\n",
    "    $\\mathbf{A}=\\mathbf{A}^{\\top}$)\n",
    "    $$\\frac{\\text{d}\\mathbf{w}^{\\top}\\mathbf{A}\\mathbf{w}}{\\text{d}\\mathbf{w}}=2\\mathbf{A}\\mathbf{w}.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Differentiate\n",
    "\n",
    "Differentiating with respect to the vector $\\mathbf{w}$ we obtain\n",
    "$$\\frac{\\partial L\\left(\\mathbf{w},\\sigma^2 \\right)}{\\partial \\mathbf{w}}=\\frac{1}{\\sigma^2} \\sum _{i=1}^{n}\\mathbf{x}_iy_i-\\frac{1}{\\sigma^2} \\left[\\sum _{i=1}^{n}\\mathbf{x}_i\\mathbf{x}_i^{\\top}\\right]\\mathbf{w}$$\n",
    "Leading to\n",
    "$$\\mathbf{w}^{*}=\\left[\\sum _{i=1}^{n}\\mathbf{x}_i\\mathbf{x}_i^{\\top}\\right]^{-1}\\sum _{i=1}^{n}\\mathbf{x}_iy_i,$$\n",
    "Rewrite in matrix notation:\n",
    "$$\\sum _{i=1}^{n}\\mathbf{x}_i\\mathbf{x}_i^\\top = \\mathbf{X}^\\top \\mathbf{X}$$\n",
    "$$\\sum _{i=1}^{n}\\mathbf{x}_iy_i = \\mathbf{X}^\\top \\mathbf{y}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Update Equations\n",
    "\n",
    "-   Update for $\\mathbf{w}^{*}$.\n",
    "    $$\\mathbf{w}^{*} = \\left(\\mathbf{X}^\\top \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top \\mathbf{y}$$\n",
    "\n",
    "-   The equation for $\\left.\\sigma^2\\right.^{*}$ may also be found\n",
    "    $$\\left.\\sigma^2\\right.^{{*}}=\\frac{\\sum _{i=1}^{n}\\left(y_i-\\left.\\mathbf{w}^{*}\\right.^{\\top}\\mathbf{x}_i\\right)^{2}}{n}.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Reading\n",
    "\n",
    "- Section 1.3 of @Rogers:book11 for Matrix & Vector Review.\n",
    " "
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}