04 - Ensemble Methods.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ensemble methods\n",
    "### Combiding models together to improve performance\n",
    "### Can act as regularization\n",
    "### With decision trees, provides the closest thing to \"It just works\" algorithm for a diversity of cases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are three main ways of aggregating models:\n",
    "- **bagging**: train multiple copies of the same model on randomly subsamples datasets, average their output\n",
    "- **boosting**: train successive versions of a model on samples the aggregate of the previous fails on\n",
    "- **stacking**: train different models, and stack a meta-predictor using their outputs as inputs on top"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The best known is probably random forests.\n",
    "# Technically, this is not a boosting algorithm, but is conceptually very similar\n",
    "# The idea: train a large bunch of very shallow, simple trees (that do not overfit), with random parameters.\n",
    "# i.e. looking at a different subset of features, taking different splits, etc.\n",
    "\n",
    "# let's see it in action"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6oAAAGTCAYAAAAhlu9SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3X+o5el9H/bPoxlXCrGruNmZVbqz1hqrRVElk3VHwtJG6nolGysWMhUUr1KHkASWQAIyxKixBf2n9C9D7EJCw+IWWmysGrIhJbFI5OwMsnatsKNfkR0p6dZY9lXs0WxinDhtZa/u0z/unN07d8659/z4fr/Pr9cLBml2Zu59zjnfc8/zfj6f5/mmnHMAAABALV5TegAAAABwmqAKAABAVQRVAAAAqiKoAgAAUBVBFQAAgKoIqgAAAFRFUAUAAKAqgioAAABVEVQBAACoyuUS3/SBBx7IjzzySIlvDUCnPvvZz76Uc75Sehy98FkNwBy2/bwuElQfeeSRuHXrVolvDUCnUkpfLT2Gnlz5gz+I//n3/6D0MBjMW5/8rtJDICJe9+RjpYfAKd948/tLD2ESr7v0aERs/3mt9RcAAIgIIbU2vYTUfQiqAAAUp5oKnCaoAgAAqqmVGbmaGiGoAgBQmGpqeUJqXUYPqRGCKgAABQmp5QmpdRFSTwiqAAAAVEVQBQCgCNXU8lRT66Ka+ipBFQAABiSk1kVIvZegCgDA4lRTyxJS6yKk3u9y6QEAAPX5Y//Ja+8JEr/28f+74GjojZBalpBaFyF1PUEVALjQ2WAhuLIvIRXYhqAKAOxMtZV9CKnlqabWRTV1M0EVADiIaiu0QUiti5B6PkEVAJiUaivrqKaWJaTWRUi9mKAKAMxGaCVCSC1NSK2LkLodQRUYxvFxxJ07EVevRqRUejQwHi3CYxJSyxJS6yKkbs99VKEWx8cRt29H5Fx6JLtrYOzHxxHf930R165FPP74ye+Bst765He98os+eW3LElLrIqTuRlCFGrScohoZ+507Ec8/H/Hyyyf/e+dO6RHNqPTCQenvT5OE1r54LcsTUusipO5OUIUatJyiGhn71asR73pXxOXLJ/979WrpEc2k9MJB6e9PF06HVmGnPV4zuJeQuh9BFWowd4qas8LVSAJMKeLGjYijo4ibNzveo1p64aD096dLQms7vEZ1UE2th5C6P0EVajBnipq7wtVQAnzNayIefLDqIR6u9MJB6e9P94TWenlN6iCk1kNIPYxTf6EWqxQ1tXUVrqm/z1xjZ3erhYNSxxuX/v4Mxa1v6iCg1kNIrYOAOg1BdQrueUHNVhWu559X4RpF6YWD0t+fIbn1zfIE1LoIqXUQUqcjqB5q1Va5CgE3bpxM0qAWKlzAgFRb5yWk1kVIrYOQOq1+gmqpquYSbZVwKBUuYGBC63QE1PoIqXUQUqfXR1Cdoqq5b9DVVgnT0koPzEiL8H4E1DoJqeUJqPPpI6geWtU8JOhqq2yXQFSfGlrpXRcwFMH1fAJqvYTU8oTUefURVA+tah4adGtoqzS53k0NgYj7lW6ld13A8ATXEwJq3YTUsgTUZfQRVA+tarbevjvi5PrQYL5tILIAsKwF3ovnvqSlgzJQnZGCq3DaBiG1LCF1OX0E1YjDqpqtt++ONrmeIphvE4hGXAAobeb34qsvaY53vf2P4savfEu85tKp79H6ohUwu96Cq3DaFiG1HAF1eSnnvPg3vX79er5169bi37dbOUc8/virk+ubN08m+L1WA2/fjrh27SSYX74ccXS0XzC/6PmZ6vtQjZOXNMfLL6e4HH8UR9/738SDzz1z7wJEr++bAaSUPptzvl56HL34nv/09fm5v/JnSw+jWbUGWMG0XUJqGQLqdF536dGI2P7zup+K6sjWVaF6rgZOVfW6qAqvutadq1cj3vX2P4rnfzXFu+K5uPrCP7q/A6GGPedA8zYFwiUDrFDaDyG1DCF1Op86emP8wBt3+zeC6r5qq7qcnVzX1g485fO1VKt26y3h3CeliBu/8i1x58/+13H1hX8U6TELEIwnpXQpIm5FxNdyzh8oPZ7RCI/sSkhdnoA6rU8d7ZhQ7+qkxLawVbXy2rWTltvj49Ijut+qGnj5cvlq4BzP1yqYzx0el/o+tTs+PumbLbBVYGqvuZTiweeeifS1o1fb5GEsH4mIL5ceBHAxIXVZ33jz+4XUigiq+1hXrazNqhp4VMFkvIXni81aWJjZlQUIBpVSuhYRPxQRP1t6LMD5hNTlCKjz2beaGiGo7qemauV5apmMt/J8sZ6FBujJz0TERyNi7YpTSumplNKtlNKtl/7DHy47MuAVQuoyBNS62aO6D3sXdzPa81Xb/uVDOVQKupBS+kBEfD3n/NmU0uPr/k7O+emIeDri5NTfBYcH3CWkzk84bYOK6r5qqVa2ovTztdQeyx7bZGtqIwcO8VhEfDCl9JsR8fGIeCKl9HNlhwScJqTOSwV1WYe0/UYIqkR0dVDOWqvw+NBDJxXBb35zvu/Va5ts6YUG4GA555/IOV/LOT8SEU9GxLM55x8tPCzgLiF1PgJqm8YKqr0Hsn30WAE8686diOeeOwmon/lMxLvfPd/jtB8XANiRkDoPAbWcQ6upESMF1REC2T56rQCedvVqxNvf/urvX3hhvsepTRZoQM75pnuoQh2E1OkJqH0YJ6iOEMj2MUIFMKWIT3864p3vXOZxapMFALYgpE5LQK3DFNXUiJFO/XVy6XqjnMh76dJJWO39cQIATRBSpyGY9mucoDpKINvHqgLYu1EeJwBQNSH1cAJqnaaqpkaMFFQjBBUAAIoSUg8joI5jrKDKGI6PVc4BgOoIqfsTUOs3ZTU1YsLDlFJKl1JKn08p/cOpvibszOnONOieO2e5jRZAl4TU/TggqQ1Th9SIaU/9/UhEfHnCrwe7c7ozjbl3bSXH8eNPWGgB6MjrnnxMSN2DgMokQTWldC0ifigifnaKrzcblYr+jXC7Hbpy39rK8/+XhRaATgiou1mFUwG1LXNUUyOmq6j+TER8NCI2Lv+nlJ5KKd1KKd26U2Ly1VNLqMC92ep056OjiJs37VGlevetrbzrTRZaADogpG5POG3XXCE1YoKgmlL6QER8Pef82fP+Xs756Zzz9Zzz9StXrhz6bXfXS0toT4F7LqvTnQ8JqRYDWMi9aysp0s3DFlpcugDlCanbEVA5zxQV1cci4oMppd+MiI9HxBMppZ+b4OtOq5eW0F4Cd80sBrCwe9ZWDlhocekClCekXkxA7cOc1dSICYJqzvkncs7Xcs6PRMSTEfFszvlHDx7Z1KZqCS1druglcNfMYgCNcukClCWknk9A7cfcITVi2lN/63doS2gN5Qp7MOc3ymJA6UUXJjfKpQtQIyF1MwG1L0uE1IiIy1N+sZzzzYi4OeXXrMq6csWDDy4/jlXgZh6rxYA7d05m+j0uBqwWXZ5//iTR3Lhxcl217Pi479dsCyNcugA1ElLXE077s1RIjRitonoo5YpxTHEg01k1VTB76xGtoduhEnNcugBsJqTeTwWVKQiqp10UJLTdsq/aglRviy69BW8AmiCk3k9A7deS1dQIQfVV2wYJ5Qr2UVuQ6m3RpbfgDUD1hNR7qaL2bemQGjHxHtWm1bL/lD6tgtRqT2gNQaqnvc42ZwKwEAH1XsJp30oE1BVBdaXGIEE/BKn5zRS8ndEEwIqQ+ioBtX8lQ2qEoPoqQYK59VTBHESPhyMDsB8h9YSAOobSITXCHtV7jbr/tKbTaKEitW0tBqAMIfWEkDqGGkJqhKBKbafRMi+LEjtxRhMAQqqDkkbxqaM3VhNSI7T+zqOlTW0OkRqHPtad2REwj5Z+RAJjGz2kCqdjqCmcnmaWOrXWKpQ9loxUDdfTx7qXUXcEzKW1H5HAuIRUIXUEtYbUCEF1eq2Fgd7up2kWvFmPixI0p7UfkcCYRg6p2nzHUFub7zpaf6fW4m1uejqNVivzZvpYqUCLPyKBcYwcUCNUUUdQezg9TVCdmjBQllnw+XpalKBJfkQCtRo5pAqofWspnJ4mqM5BGCjHLBiq50ckUBshlR61GlBXBFXmU+poT7NgAGBLo4ZUAbVPrYfT0wRV5uFWKGNxvxEAGiSk0rqegulZkgPzcLTnIqq4E4+TlgFokJBKq1Yn9vYcUiNUVJmLQ41mV03R2knLADRmxJAqoLar90C6iaDKPGo71KjD1tRq8qFFCQAaMWJAjRBSWzJqKF1HUGU+hx5qtGu43PT3qyk9TquafFjbogQArCGkUhuh9HyCagkdVvcmt2u4PO/vV1N6nFZV+dBJy2zBjz6glBFDqoBaF6F0d+2XlVrj4Jnt7HoY03l/f1V6vHz55H8feKCCE4imscqHJv3Uzo8+oBQhlaWdPuxohEOP5qKiurROq3uT27Wv9by/f7r0+MADEU880V0b8IWUsijMjz6gBCGVuQmh8xFUl1bNxsLK7drXetHfX5Ueb98eb7bc6R5d2uJHH7AkAZU5CKXLElR3dWhl6mygyjni619X6Vpn132P2/z9EWfLSllUoKo91UDXhFSmIJSWp6yyi6k2Wa0CVc42bS1tNVs+Ooq4eXOM2fLZPbojhPN1jo+72ZvcKnuqgbkJqezLntL6qKjuYurKlEpXGTWcULvknlGlLO3PAAMQUtmFMFo/M7VdTF2ZUukaU4njT0cvZZ1ZFDq+fUdxFaAjQioXUTFtj4rqLqauTKl0jUklfXmn9iYfv/Ox+L4nryquAnRgxIAaIaRuQxhtn6C6q6nbRmtoQ2VZIx7oVNqpRaE7+Wo8/3CyTgDQuBFDqoB6PuG0L4IqLE0lvYy7i0JXs3UCgNYJqawIp/0SVKEElfRirBMAtGvEgBohpJ4lnI5BUAWGY50AoD1C6tiE0/EIqoxpydvDtDAOmuKyAUYjpI5LQB2Xsy6ndnwc7ntRuRK3h6l5HDTFZQOM5HVPPiakDshtZIgQVHd3XhA1g2zDutvDjDwOmuKyAUYxakCNGDekCqecNm5Q3afyeVEQNYNsw+r2MJcvlz32tZZxdKjnxgaXDTACIXUsAirrjBlU9618XhRES8wge56Rz2V17OvRUcTNm8ts8lv3OpUYxwB6b2xw2QA9G7nVN2K8kCqgcp4xg+q+lc+LgujSM8jeZ+RzWh37ulRI3fQ6LTmOQYzQ2OCyAXo0ckCNGCukCqhsY8xTf1eB8/nnd6t8bnMDxiXve7FuRu6eG/XxOi1q37c3AGWMHlAjxgmpwim7GDOobhM4N6npBoxm5G3wOi3qkLd3q9yuhjm85tu/dbYA8f99/LlZvi7tEVLHCKkCKvsYM6hG1BU49zXijLxFrb9ODaagHt7e21p1lq/WQW7cOHn8ULPzwokQOwYB9YSQCpuNG1R7MdKMvGWtvk4HpKAG822TdJbTm3UBRnjti5A6BgGVQwmqwGZ7piBVvuXU2FlukYKpnQ02gmubBNR79VpNFVCZiqAKbLZnClLlW05tneUWKViC4NoeIfVeQipcTFAFNtszBdVY5etZTZ3lFikoQXCtl4B6vx5DqoDKHATVi+hhY3R7pKDaqnwsxyIFNRBcyxNQ1xNSYXuC6nn0sMHeaqrytWjfNbLSa2sWKajR6dAktM5LQN1MSIXdCKrn0cMGFPDyyxHveU/ECy/stkZWy9qaRQpqpto6DwF1LAIqSxBUz6OHDVjY8XHEu98d8ZnPnPx+lzUya2uwO8H1MALqdnqqpgqpLEVQPY8eNuCMuVtr79w5qaSuvP3t26+RWVvjPCmlhyPif4+IN0TEcUQ8nXP+n8qOqj7ahLcjoG5PSIX9CKoX0cMG3DVVa+15Yffq1YjHHot47rmTkPrpT28fiK2tcYGXI+Jv5Jw/l1L6toj4bErpkznnf1F6YLVSbb2fgLobIRX2J6huUvpEEqbl9WQCU7TWXhR2Dw2b1tbYJOf8OxHxO3f//79PKX05Ih6KCEF1S6MGV+F0P0IqHEZQXWeuE0mEpTJqOWGG5k3RWrtN2BU2mVtK6ZGIeDQi/lnZkbRtXYDrJbwKp0QIqJQlqK4zx4kkwlI5TphhIlO01tpHSmkppW+NiL8XET+Wc/53Z/7sqYh4KiLiO/7U6wuMrn0tV12F0+n0UE0VUilNUF1njpmksFTONq+najdbWlft3OXy6W0fqbdOW1JK3xInIfXnc87PnP3znPPTEfF0RMT1/+KhvPDwurQp/JUOsELpfIRUmIagus4cM8mpw+8+s8NRZ5QXvZ6q3U2o9fLd5/LppbXXW6ctKaUUEf9LRHw55/y3So9ndOcFxSlDrEDKroRUaiGobjL1THLK8LvP7HD0GeV5r6dqd/V2uXyXDrQjXz4jP/ZGPRYRfyEivpRS+sLd//aTOedfKjgm1hAu29V6NVVIpSYHJ5WU0sMppRsppS+nlH49pfSRKQbWpVVYOnT2vG52OMe/GcWq2n35sk2Dldr28l0F2mvXIh5//OT3cxv58hn5sbco5/zpnHPKOX93zvnP3P0lpMJEWg+pUJspSmqr+7L96Yj43oj4aymlt0zwddlkn9mhGeVmq2r30VHEzZt19ZUSEdtfviXWY0a+fEZ+7AC9UU2lNge3/rovWwH7tBHXdIJLjZsNe9k02KltL99SJ+qOfPmM/NgBVlqvpgqp1GjSTYrn3ZctpfRUSulWSunWHW2nh9unjXiq1uNDlOjNpAvbXL4qfAAsTUiFeUwWVM+7L1vEyZH3OefrOefrV65cmerbzuf4OOL27YjsdP5J2SvLzGpYjwGAFgip1GySoHrRfdmao+o3H3tlAYBOtFxNFVKp3RSn/pa7L9tcVU9Vv/nozVxPBR8AAF4xRUV1dV+2J1JKX7j7689N8HXPN2fVc+Sq3xKBSW/mvVTwtyLLA1AT1VSY1xSn/n46IpZPHHPeab6mE3KXtApMqyNTb9w4CZXMa85ruRMuTQBqIqSyrWeefXHyr/mhJ940+des0cFBtZi570Mx4j0XaghMNd66Zm6l7qnSkBouTQBonZA6rTlC6NTft+VQ225QHbXqOafSgWnUsplr+UKlL00AWGm5msp+SgXSKVw09pqDbLtBNWLMquecSgemkctmruVzlb40AaB1qqnbazmY7mrTY60hwLYdVJleycCkbLafQdqlZXkASlNN7ddI4XQbNQRYQZV6KJvtbtR2aQBga6qp6wmnu1sywAqq1EXZbDcjt0sDwIJaraYKqfcSTuex7nk9NLwKqixrkDbVxWiXBgC4kIC6vLPP+Q/8pXfs9O8FVU4sESC1qU5PuzQAsIFqqoDaMimBVwPktWsRjz9+8vs5rGtT5XCrdmkhFQBm0Wrb78ieefZFIbVxKqost89RmyoAwCJGraYKp/0QVFkuQGpThW7Ybg6MQjW1DQJqf7T+8mqAPDqKuHlz3lmnNlVo3lK7BQBgG0JqnwRVTgiQFzs+jrh9OyLn0iMZ2kUvg5dpfrabA9RtpLZfIbVf7QfV1az0m980O2U+SkhVuOhl8DItY7Vb4PJl282Bvmn7rZfDkvrXdlA9PSv9k38y4qGHzE6ZhxJSFS56GbxMy1hytwAAuxmhmiqgjqHtoHp6Vvr7v39SVd11dqpPkG0oIVXhopfBy7QcuwUAKEFIHUfbQfX0rPT1r4+4dGm32ak+wWW1vCighFSFi14GLxMAU9H2Wx8hdSxtB9XTs9J/828ivva13Wan+gSX08OigBJSFS56GbxM02t5jQlgJD23/Qqp42k7qEa8Oiu9dGn32ak+weVYFIAm9bDGBEDbhNQxtR9UD6FPcDkWBaBJ1piAEWn7rYeQOq6xg2qEPsGlWBSAJlljAqAUIXVsl0sPgIGsFgWAZqzWmO7cOQmp1pgA6tTb/lQhFRVVgAEcciCSxhMAYGmCKkDnHIgEsD37U8tTTSVCUAXongORAGiFkMqKoAotcmNLduBAJACgNYLqROQGFqOPkx05dBugb70cpKSaymmC6gTkBhalj5M9OBAJgJoJqZwlqE5AbmBR+jgBAOicoDoBuYFF6eMEgFk48bcM1VTWuVx6AD1Y5YY7d05CqtzA7FZ9nAAA0CFBdSJyAwAAwDS0/gLMqOcTwXt+bAAsQ9svmwiqADPp+UTwnh8bAFCeoApUo7cKXc8ngvf82ABYhmoq5xFUgSr0WKHr+UTwnh8bAFCew5SAKqyr0LV+QFnPJ4L3/NgAgPJUVIEq9FqhW50IXjrIzdFWXctjAwD6I6jWorfNea3yOhSzqtAdHUXcvCn8TKnHtmoA2mZ/KhcRVGtgFlkHr0NxKnTzcPARANAaQbUGZpF18DrQqQceiLh+PeLSpb7aqgGAfgmqNeh1c15rvA506Pg44oknIm7dinjHOyKefVbFGgCo3xin/h4f1300peMz6+B1oEOnGwVeeCHipZfaP00ZAOhf/xXVVvYdHro5zyFA07BJks5oFAAAWtR/UB1h32ErYRwq1fM6j9OUAaiNE3/ZRv9BdYRywghhHGYywjqPRgEAoDX9B9URygkjhHEiou/KXynWeQAA6tN/UI3ov5wwQhhniMpfCaXXeSw+AHCo91z7aukhwOTGCKqHamEm2XsYR+XvrqnfjiXXeSw+ANTltV/5ROkhAHcJquc4Po64/TvHkR83k6S80pW/GswV7Eqt81h8AABYT1Dd4JUJ8XekePxX/oc4fvmbZpKtaaESvgMd3v0FO4sPAADrCaobvDohTvF8elfcufSnzCRb0mlPZasd3lOtGfQW7Cw+AACsJ6hucM+E+N2X4urR58wkW9Jb6a1hU64Z9BjsWl18AKAuDlSiN4LqBvdOiFOkN5hJNqW30tsWau10nnrNQLCrQ63XGwDQB0H1HM1NiM0cX9Vj6e0cNXc6n7dm4JJtU83XG8ChnPwLdeg+qA4zETZzvF9zKw37q7nTedOagUu2XTVfbwBAH7oOqkNNhM0ch1Z7p/O6NYMeL9lRFsZqv94ARtXKPtUPPfGm0kOgAV0H1R4nwhuZOQ6txU7n3i7ZkRbGWrzeOJFS+sGU0r9MKb2YUvqbpccDAJtMElRr/eDrbSJ8LjPH4bXW6dzbJTvUwli0d70RkVK6FBF/JyLeHxFviYgPp5TeUnZUUCf7VKG8g4NqzR98vU2EL2TmSGN6umRbWRgbpT2Ztd4RES/mnH8j5/yHEfHxiPjhwmMCJtZK+y9cZIqKatUffD1NhDmQGTozamFhbKT2ZNZ6KCJ++9Tvj+7+t1eklJ5KKd1KKd2683v/YdHBAcBpUwTVCz/4Is58+PXeE0d9zNBZQOmFsYvWYkZrT+Y+667Me66WnPPTOefrOefrV779jy80LKiT9l8oa4qgeuEHX8SZD78rVyb4trADM3Q6t81aTCvtyczmKCIePvX7axHxrwuNBZiR9l96MEVQ9cFH/czQ96NduhnbrMW00J7MrF6IiP8spfSdKaX/KCKejIj/s/CYoGqqqvNxixouMkVQ9cFH/czQd6dduinbrsWUbk+mnJzzyxHx1yPiH0fElyPiF3POv152VACw3uVDv0DO+eWU0uqD71JE/K8++KjSaobOdtaV6Dx/1Vqtxdy5cxJSBVHWyTn/UkT8UulxAPN7z7WvxqeO3lh6GLC3Se6jmnP+pZzzf55z/q6c8/84xdfsmW5KmqBdujmqpQDT0/4LZRxcUWU3q27K558/mfvfuHEyuYTqdFKiOz5u/iFAEfmPvT6+8eb3r/0zE3dgCh964k3xzLMvlh4GlRJUF6abkqY03i5tYQjmsSnAriPU0oPXfuUTO133tdD+S8sE1YWtuilXE2fdlDAfC0NQ3raTe4EWgNME1YV10k3JHPSoTs7CELTjokAryFKaqiosS1AtoPFuSuagR3UWFoagH/bLQp/sU2UTQRVqoEd1NhaGlqMpgBIEWJakqgrLUbKBGrgVzFB6vEXVqing2rWIxx8/+T2U9I03v/+eXwC0RVCFgl4JLHG3R/XoKOLmTeWojvUa6NY1BUBNBFem0mq1/j3Xvlp6CBt96Ik3lR4CFdL6C4Xcvy31NfGalnpU9XnupdcubwdX0ZqzYbXV8AFwqM//wpdm/fqPfvhte/07QRUKaTqwOPxpb70GOgdX0brTwVVo5SL2qtKyuYPpxu/3l96x078TVKGQpgNL0ym7rF0DXUuFawdX0QuhlW20GlZr5fTfeS0dTqegBAKFpJa3pc5w+FOPBwxtsgp024TUHvezQkvsa6U3Ne9VZXqf/4UvNRlSIwTV6Y002+ZgawNLC9fQxClbIFvPAUVQF6GVs1TcqVXLAXVFUJ2S2fZ0Wghrc2jpGtq2LLiF0QPZpsvdXYugXkIrLau1qur038P1EFBXBNUpjT7bnsq+Ya2WcHvIOAa9hkYOZOdd7k23h8NABNaxqapSi14C6oqgOqWRZ9tT2ies1VKJPHQcg15DIweyiy73CQvXi6hlvQhKEFjH1WJYVVXtS28hNUJQndbIs+0p7RPWpq5E7jvbXjeOXb7WwNdQa4FsKj2tTdSyXgSlCazAknoMqRGC6vRmmG0PV6HYJ6xNOds/ZLZ9dhwPPLD71xo1sV2g1/dBT2sTg3auw0YC61hUVSmh15AaIahWb9gKxa5hbcrZ/iGz7bPjeOklM/cJ9P4+6GVtoqfqMExJYB2HsDoN7b/b6TmkRgiq1VOh2MFUs/1DZ9unx2HmPgnvgzb0VB2GOQisY2gxrEKNBNXKyTkFTDnbNnOfhPdBO3qpDsOcBFZqo6rant6rqRGCavXOyzm97tmrwpSzbTP3g8n7QI+E1X6pqsLhBNUGrMs5ve/Z65bVhb3J+0CPVFf71VpYVVVtxwjV1AhBtVn37dm7LQBVz+oCABsIrH1qLaxCTQTVRt27Zy/H1R8RgKrnRCAALiCsUlKNVVXuNUo1NUJQbdY9e/b+j69H+lUBqHpOBAJgC6qrfWmtqlpbWNX+Oy5BtWGv7Nl7UABqghOBANiBsNqP1sIq1EBQ7cE2AWiXQ3wc+DMfJwIBsANhtR8thVVV1TqN1PYbIaj247wAtMshPg78AYCqaAXuh7AK2xNUR7DLIT4O/AGAKgmrfWgprNZEVXU8guoMquuc3eUQHwf+AEC1hFVHwT12AAAVKElEQVSWpKpKSYLqxIp2zm5KyLsc4uPAHwComrDavpaqqjWFVVXVsQiqO7qoWlqsc/aihLzLIT4O/AGAqgmr7WsprFLeaAcpRQiqO9mmWlqsc9beUgAYirDavlbCqqoqJQiqO9gmCxbrnLW3FACG40Tg9gmrsJ6guoNts2CRzll7S+tS3YlaAPRMWG1bK2G1FqqqYxBUd1B9Fqx9b2mL4W2fMbsXLQAFCKttayGsqqqyJEF1R7VnwWq1GN72HbP9wuypxbUcoC7CatuE1e2pqvZPUGUZjYS3e4LCvmO2X5g9tLiWA8D0WgirsARBdQrKIBerKLxternuCwoP7Dnm6nvEqVEjazlAA1RVmZuqKksQVA+lDLKdSsLbeS/XfUHhpQPGrEecLZxeNKloLQfogLDathaqqrWEVfo1bFCdrAhaSRmkiaJuBeHtvJdrbVCoYMz06eyiSc5VrOUAHRFW29ZCWK2Bqmq/hgyqkxZBKyiDKOpu77yXq5KiL4NYt2hiXQSYmrDattrDqqoqcxoyqE5aBK0g3VRS1G3CRS+XoMBSKljjAgYhrLZNWL2Yqmqfhgyqk08QC6cbE97dCKPUoII1LgAaUXtYrYGw2p8hg2pvE8TeHg+MwqIJsBRV1fbVHFZrqKrSnyGDakR/E8TeHg8AMC1htX3C6vlUVfsybFAFYFpNnD4OgxNW21dzWGU+j374baWHsDhBFWBAU4fKdaePC64A86g1rKqqMiVBlUmYkEI75ril1dnTx2/fdtssqJWqah+EVXonqHIw93GFtsxxS6uzp4+n5LZZUDNhtQ+1htXSVFX7IKhyMPdxhfnM0a0wxy2tzp4+/uCDbpsFsIQaw6qqKlMQVDmY+7jCPObqVpjrllanTx932yyon6oqcyodVlVV2yeocjATUpjHnN0KS9zSym2zAJZRY1UVDiWoMgkTUpiebgVgbqqq/agxrKqqTmu0W9QIqsB8HAd9EN0KAOxCWKUnguqIhAeWcMEGS5fhdnQrAHNTVe2LsHqv3qqqIxFUR+NeMizlnA2WLkMAmE+NYbUkYbVNBwXVlNJPpZS+klL65ymlv59S+hNTDYyZuJcMSzlng6XLEKAuqqr9qS2sagFmV4dWVD8ZEW/NOX93RPyriPiJw4fErJzOwlLO2WBZ02WoBRmAXgmrr1JVbc9BQTXn/E9yzi/f/e1nIuLa4UNiVk5nYUkbNljWchlqQQZ4lapqn2oLqxxmpJN/p9yj+pcjYuM7IaX0VErpVkrp1h19fmU5nYUK1HAZakEGYAQ1hVVVVbZ1YVBNKf1ySunX1vz64VN/52MR8XJE/Pymr5NzfjrnfD3nfP3KlSvTjB7gADW1IAPUQFW1X8Iqrbl80V/IOb/vvD9PKf3FiPhARLw350Z3eR0fn5RSrl5VZYQObXqLr1qQa3r7+3EEAPP50BNvimeefbH0MNjCoaf+/mBE/HcR8cGc8/8zzZAWZpMarXH6z04ueovX0IK8ssuPI5cBu3BKP7tQVe2XqmofRtmneuge1b8dEd8WEZ9MKX0hpfR3JxjTsvbcpNblJLHLB9WZ1hZWKrimWtqHuu1YW7sMqIJT+oGIEFYj7FVtxaGn/r4p5/xwzvnP3P31V6ca2GL22KTW5SSxywe1pwrC1UYtpa5KrqmW9qFuO9aWLgPq4JR+4LSawipsMuWpv23a4z4ZXU4Su3xQe6gkXG3UUuqq5Jqq5VY429h2rC1dBlRp4yn9957Q/3sLD4uaaP/tXy1hVVWVTQTViJ03qXU5SezyQe2hknC1Uc4RH/94xG//dv2pq6JrqqZ9qBfZZqwthW+WM8Up/fee0P/tSw0dKGT0sNqyEfapCqp76HKS2OWD2kNF4eo+q2rvd3xHxI/8SJ2tyae5pmbVUvhmGTnn9+Wc37rm1z+IuOeU/v+22VP6WZSq6hhqCaslqKrWTVDd02qSmHO92xl3ZuZbd7iqvdq7jmsKqtDFKf1A11RVOUtQPUDt2xnZU63hquZqL1C79k/pB2ZTS1W1RFhVVa2XoHqAFgtc3an5hN6p1VztrcFI1wLsqItT+ilC++84agmrJbQaVnvfpyqoHkCBq7ARS9q1VntLG/FaAICJ1RBWtQCzIqie54IKjQJXYUrarLgWAGASNYTVElqtqvZMUN1kywqNAldBStqsdHAttNK53Mo4Adhf6bCqqkqEoLqZCk39eippm/0fpvFrYdvO5dKXiQ5rGJN9qpTgYKXt9LxPVVDd5LwKTenZIq/qoaRt9j+Nhq+FbdbFarhMrN8BjKN0VRUE1U02VWhqmC1W4LysLsfvyOx/eNt0LtdwmXTQYQ3ADkqHVVXVsQmq51lXoalhtljYeVldjt+D2f/wtulcruEyabzDGoA9jBhWqYOguqsaZouFnZfVu8vxS5SHzf6JizuXa7lMGu6wBg5gn+rYSofVpbVWVe11n6qguqtaZosz2SaXnZfVu8rxS5aHzf7ZgssEgBGpqo5JUN1Hp7PFbXPZeVm9qxxfU3nYxl8AoCBVVZYmqPKKXXLZeVm9mxxfS3nYxl8AoAIlw6qq6vl6bP8VVHlFLbmsGrWUh2uq7MIZiv0AYxkprKqqliWo8opacllVaigPW0GgUor9MBYHKgFLElS5Rw25jDOsIFApxX6AMamqsgRBFVpgBYEKKfYDjGu0w5Va0Ns+VUGVadmwBsNQ7AegBFXVMQiqTMeGNdio1zUcxX6AcY3UAszyBFWmM/GGtV4n9ozHGg4AvRqlBbiVqmpP7b+CKtOZcMPaXhN7yZZKOXQIgJ6VCquqqn0TVJnOhBvWdp7YK1lRsd4PHbJGBONwixpgKYIq05pow9rOE3slK9aoJUD1fOiQNSIAIsaoqmr/XZaguqBaJs0t2Hli33vJip3VFqB6PXTIGhEAK6PsV2UZgupCzp00S7Br7TSx77lkxV4EqGVYIwKgNFXVPgmqC9k4aa6t7NOyXktW7EWAWoY1IgBOU1VlKoLqQjZOmpV9aElD1X8BajnWiAAoTVX1Xj3sUxVUF7Jx0qzsQysarP4LUACwPFVVpiCoLmjtpFnZh1ao/gMAWyoRVlVV+yKo1kDZhxao/gMANKP19l9BFdiO6j8AsIPeq6rMS1BdSENn0MBmqv8AABGh/XduguoCDjmDRsDlPi4KznBJAFArVdWyWm7/FVQXsO8ZNA0essrcXBSc4ZIAoHY9nwKsqjofQfWMOSoT+55B45BV7nP7dsRzz7kolBBf4ecEANxPVbV9guopc1Um9j2DxiGrDZozQB0fRzz55KsX5jvfOcxFcc/TqoR4Dz8nAGiBqiq7ElRPmbMysc8ZNNUdsqqKdb65A9TqAs35JJX84i/udlE0+vrd97TeVkI8rbqfE0DXeg4b9EdV9USr+1QF1VNqrExUc8iqKtbF5u7BPHuBPvjg9v+24dfvvqc1VfhGLezQnxONrmEA0BgLHexCUD1FZeIcNsJdbO6VjpQi/uk/jfj8508u1F0u0IZfv/ue1ge9UafU8BoGAFxoqaqq9t/pCapnVFPBrE2N5ebazL3ScXwc8d73Rjz66Emy2CVRNPz6rX1avVEn0/AaBgANUlUto8X2X0GV7Sg3b2fOAHVIomj89ZNL59PwGgYAbEVVtU2CKtuTFso6NFF4/fbW8x7OxtcwAGiQqirbEFShFRLFbiZKlyPs4VytYeTcbyAHDidcAEsSVKElqqLbmTBdjrKHc4RADkA9ll740P7b3j5VQRVoz0XV0gnT5Sh7OEcJ5ABAGwRVoC3blP4mTJejdFyPEsiB/Wj7pQdLVVWZhqAKtGWb0t/E6XKEjutRAjkA9eh1AUT77zQEVaAt25b+RkiXE/OUAQC1EFSBtij9ASyq16oXdXCoEptcLj0AgJ2tSn8AAHRJRRUAgLVUU6E/rexTFVQBFnTRnXV6MtJjBWB/2n9ZR1AFWMg2d9bpxUiPFXqlmgqUNElQTSn9eEopp5QemOLrAfRomzvr9GKkxwpAe9xTtX4HB9WU0sMR8f0R8VuHD4fF6MmDxW17Z50ejPRYoUeqqSyt12uu1vbfFvapTlFR/emI+GhESDyt0JMHRYx0Z52RHisAML2DgmpK6YMR8bWc8xcnGg9L0JMHxazurDNCcBvpsUJPeq1sAW25MKimlH45pfRra379cER8LCL++22+UUrpqZTSrZTSrTuCUVl68gCANYRUSur19N9a1d7+e2FQzTm/L+f81rO/IuI3IuI7I+KLKaXfjIhrEfG5lNIbNnydp3PO13PO169cuTLlY2BXevKgWraPA6UIqTCPWvep1m7v1t+c85dyzldzzo/knB+JiKOI+J6c8+9ONjrmoyfvMNIEM7B9HChFSAVq4z6qsKua0oTA3BXbx4EShFRqov2XlcmC6t3K6ktTfb3hCBztqCVN1BSYmYTt48DShFRYRq3tvzXvU1VRrYHA0ZYt0sQi6w61BGYmc9H2cetZwJSEVKBmgmoNBI62XJAmFlt3UH7r0qbt4y2tZwnUULfXfuUTQipQPUG1BgJHe845jGqxdQenNw+llfWslgI1jEhAhfvZp1onQbUGAkdXFl13cHrzMFpZz2olUMNoVFFpSa/Xqn2qu7lcegDctQocNG+17nDnzkmYkCGZQivX1SpQP/983YEaRtLrpB/om6AKM7DuwBxauK5aCdSjSin9eET8VERccVJ//wRUoGWCKgCTaiFQjyil9HBEfH9E/FbpsTAvARV2955rX41PHb2x9DA4xR7VQzjaEoB2/HREfDQifGh1aLUHVUiFutmnuj1BdV+OtmRkFmmgKSmlD0bE13LOX7zg7z2VUrqVUrp1587vLTQ6DiGcAr3S+ruvdUdb6nVjBKtFmtVpOTdunPR6AkWllH45It6w5o8+FhE/GRE/cNHXyDk/HRFPR0T8l9ffYiWqQkIpMApBdV+OtmRUFmmgSjnn96377ymlt0XEd0bEF9PJ6VbXIuJzKaV35Jx/d8EhsifhlBG99iufiG+8+f2lh0FByiD7cu9TRtXKDT2BiIjIOX8p53w15/xIzvmRiDiKiO8RUut1er+pkArLec+1ry7yfexT3Y6K6iEcbcmI3H8EYFLCKMD9BFXqd3wsFNXGIs0wTr/9cvZW7MHdqioFCKQA2xNUqZuDe6CYs2+/nCN+9Ve9FWEbQinAYQRV6ubgnmopdPfv7Nsv54hvftNbEVaEUYD5WA+nbg7uqZLbCI/h7NvPW5FRnT3cyCFHwKEcqHQxFVXq5uCeKil0j+Hs288eVXomeALURVClfg7uqY7bCI/j9NsvJW9F2ieQArRBUIU5dbqRU6EbqJ1ACuzjPde+Gp86emPpYRCCKsyn8xOLFbqBWgilAP0RVEvptNLGKTZyAkxOKIUxfOPN7y89hGE9+uG3xed/4UulhyGoFtF5pY27bOQEOJhgCjAmQbUElbYx2MhZjIaF6XguKUE4BUbwoSfeFM88+2LpYVRLGa8E9wYdx2ojpxn+YtzjdTqeS5bi/qQAnKWiWoJKG8xGw8J0PJfMSSAF4DwqqqWotPXl+Dji9u2InEuPZHgaFqbjuWRqqqYAbXj0w28rPQQVVTiYw7GqomFhOp5LpiKYArtw4i8RgiocTn9kddzjdTqeS/YlnAJwCEEVDuU2NACvEFABmIKgCofSHwkgoAIwKUEVpqA/EhiUgApMyf5UVpz4AgDszOm9AH0rffKviioAsDXhFOjZp47eWHoI3CWoAgAXElCBuWn75TStvwDAuYRUAJamogoArCWgAlCKiioAcJ/0//5+6SEAA9H2y1mCKgAAAFURVAEAgGJUU+tV8hY1gioAADA8t6api6AKAAAUoZrKJoIqAAAAVRFUAQCAxammch5BFQAAgKqknPPy3zSlOxHx1cW/8TIeiIiXSg+iAZ6ni3mOtuN52s4Iz9Mbc85XSg+iF418Vo9wXS/B83g4z+HhPIfTaOF53OrzukhQ7VlK6VbO+XrpcdTO83Qxz9F2PE/b8TzRI9f1NDyPh/McHs5zOI2enketvwAAAFRFUAUAAKAqgur0ni49gEZ4ni7mOdqO52k7nid65LqehufxcJ7Dw3kOp9HN82iPKgAAAFVRUQUAAKAqgioAAABVEVRnlFL68ZRSTik9UHostUkp/VRK6SsppX+eUvr7KaU/UXpMNUkp/WBK6V+mlF5MKf3N0uOpUUrp4ZTSjZTSl1NKv55S+kjpMdUqpXQppfT5lNI/LD0WmIvP3P35TN6fz+vD+TyfTm+f94LqTFJKD0fE90fEb5UeS6U+GRFvzTl/d0T8q4j4icLjqUZK6VJE/J2IeH9EvCUiPpxSekvZUVXp5Yj4GznnPx0R3xsRf83ztNFHIuLLpQcBc/GZezCfyXvweT0Zn+fT6erzXlCdz09HxEcjwmlVa+Sc/0nO+eW7v/1MRFwrOZ7KvCMiXsw5/0bO+Q8j4uMR8cOFx1SdnPPv5Jw/d/f///s4+cH8UNlR1SeldC0ifigifrb0WGBGPnMP4DN5bz6vJ+DzfBo9ft4LqjNIKX0wIr6Wc/5i6bE04i9HxCdKD6IiD0XEb5/6/VH4gX2ulNIjEfFoRPyzsiOp0s/EyQT+uPRAYA4+cyfnM3l7Pq8n5vP8IN193l8uPYBWpZR+OSLesOaPPhYRPxkRP7DsiOpz3nOUc/4Hd//Ox+Kk5ePnlxxb5dKa/6ZKsEFK6Vsj4u9FxI/lnP9d6fHUJKX0gYj4es75symlx0uPB/blM/dwPpNn4fN6Qj7P99fr572guqec8/vW/feU0tsi4jsj4osppYiT9pnPpZTekXP+3QWHWNym52glpfQXI+IDEfHe7Ia+px1FxMOnfn8tIv51obFULaX0LXHyofbzOednSo+nQo9FxAdTSn8uIl4XEf9xSunncs4/WnhcsBOfuYfzmTwLn9cT8Xl+sC4/75OfRfNKKf1mRFzPOb9Ueiw1SSn9YET8rYj4r3LOd0qPpyYppctxcpjFeyPiaxHxQkT8+ZzzrxcdWGXSyaz0f4uIf5tz/rHS46nd3RXWH885f6D0WGAuPnP34zN5Pz6vp+HzfFo9fd7bo0opfzsivi0iPplS+kJK6e+WHlAt7h5o8dcj4h/HyYECv+hDb63HIuIvRMQTd6+hL9xdSQRgNz6T9+DzejI+z1lLRRUAAICqqKgCAABQFUEVAACAqgiqAAAAVEVQBQAAoCqCKgAAAFURVAEAAKiKoAoAAEBV/n8oZbnW1y1xUwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1008x432 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from ml_helper import helper\n",
    "import matplotlib.pyplot as plt\n",
    "from ipywidgets import interact\n",
    "import ipywidgets as widgets\n",
    "\n",
    "X, Y, TRUE_SURFACE = helper.gen_planar_samples(seed=1346)\n",
    "plt.figure()\n",
    "ax_l = plt.subplot(1, 2, 1)\n",
    "helper.plot_red_blue(X, Y, ax=ax_l)\n",
    "ax_r = plt.subplot(1, 2, 2)\n",
    "helper.plot_decision_surface(TRUE_SURFACE, ax=ax_r)\n",
    "plt.gcf().set_size_inches(14, 6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a6dd2d31f40a4eeb9b0abf57232d4ef0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=2, description='max_depth', max=5, min=1), IntSlider(value=10, descripti…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "def train_random_forest(max_depth=1, num_trees=10):\n",
    "    model = RandomForestClassifier(max_depth=max_depth, n_estimators=num_trees).fit(X, Y)\n",
    "    helper.plot_decision_surface(model.predict_proba)\n",
    "    helper.plot_red_blue(X, Y, ax=plt.gca())\n",
    "    plt.gcf().set_size_inches(7, 6)\n",
    "    \n",
    "interact(\n",
    "    train_random_forest,\n",
    "    max_depth=widgets.IntSlider(min=1, max=5, value=2),\n",
    "    num_trees=widgets.IntSlider(min=1, max=50, value=10),\n",
    ");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "509394fc457a4034903bc69d196953a6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=10, description='num_classifiers', min=1), Output()), _dom_classes=('wid…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# My backup career goal is to try and sell a print of the above to the NAC to replace the \"Voice of Fire\"\n",
    "# Now let's look at boosted trees:\n",
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "def train_adaboost(num_classifiers=10):\n",
    "    \n",
    "    model = AdaBoostClassifier(\n",
    "        base_estimator=DecisionTreeClassifier(max_depth=2), n_estimators=num_classifiers).fit(X, Y)\n",
    "    \n",
    "    helper.plot_decision_surface(model.predict_proba)\n",
    "    helper.plot_red_blue(X, Y, ax=plt.gca())\n",
    "    plt.gcf().set_size_inches(7, 6)\n",
    "    \n",
    "interact(\n",
    "    train_adaboost,\n",
    "    num_classifiers=widgets.IntSlider(min=1, max=100, value=10),\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notes on AdaBoost\n",
    "\n",
    "Notice how the AdaBoostClassifier actually takes another predictor as the first argument.\n",
    "\n",
    "AdaBoosting is a general process that can be applied to arbitrary base classifiers. It's most often used with trees, however, since it needs to train very many version of a model, and only shallow trees are fast enough to train. \n",
    "\n",
    "You don't really want to do it with a neural net!\n",
    "\n",
    "### Weak Learners\n",
    "An important thing to note is that AdaBoost works best with \"weak learners\", defined as a simple model which is capable of achieving just above 50% accuracy (or just above 1/#Classes more generally). Without going into the details: if we can't do better than 50/50 out of any single learner, then aggregating them won't help!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5858601724364ac4a58a324fea64e2b7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(RadioButtons(description='kernel', options=('linear', 'rbf'), value='linear'), IntSlider…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# For kicks, let's see what adaboosting a linear SVM looks like...\n",
    "from sklearn.svm import SVC\n",
    "from warnings import filterwarnings\n",
    "filterwarnings('ignore', '.*')\n",
    "def train_adasvm(kernel='linear', num_classifiers=10):\n",
    "    \n",
    "    # adaboost needs probabilities to work\n",
    "    model = AdaBoostClassifier(\n",
    "        base_estimator=SVC(kernel=kernel, probability=True, gamma=2), n_estimators=num_classifiers).fit(X, Y)\n",
    "    \n",
    "    helper.plot_decision_surface(model.predict_proba)\n",
    "    helper.plot_red_blue(X, Y, ax=plt.gca())\n",
    "    plt.gcf().set_size_inches(7, 6)\n",
    "    \n",
    "interact(\n",
    "    train_adasvm,\n",
    "    num_classifiers=widgets.IntSlider(min=1, max=20, value=3),\n",
    "    kernel=widgets.RadioButtons(options=['linear', 'rbf'])\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For the linear SVM\n",
    "This washed out mess is an illustration of why domain knowledge is important: the random samples are not linearly separable, and red and blue are generated in equal amounts. The end result is that trying to layer linear models results in every point being labelled 50/50. The linear classifier is simply not up to the task!\n",
    "### For the RBF SVM\n",
    "Think a little bit about how it works using the \"shining light\" analogy from the Decision Surfaces notebook. This is a potentially-useful classifier! (In any case, someone wrote a thesis about it: https://arxiv.org/pdf/0812.2575.pdf) though it does appear to suffer from \"washout\" as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}