07-01-Ensemble Methods 1 - Ensemble Learning Methods Introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ensemble Learning Methods Introduction\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Supervised Learning\n",
    "\n",
    "Group of predictors (classifiers / regressors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ensemble Methods:\n",
    "\n",
    "### **Bootstrap Aggregating** or [Bagging](https://en.wikipedia.org/wiki/Bootstrap_aggregating)\n",
    "* [Scikit- Learn Reference](http://scikit-learn.org/stable/modules/ensemble.html#bagging)\n",
    "* Bootstrap sampling: Sampling with replacement\n",
    "* Combine by averaging the output (regression)\n",
    "* Combine by voting (classification)\n",
    "* Can be applied to many classifiers which includes ANN, CART, etc.\n",
    "\n",
    "### [Pasting](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingClassifier.html)\n",
    "* Sampling without replacement\n",
    "\n",
    "### [Boosting](https://en.wikipedia.org/wiki/Boosting_(machine_learning)\n",
    "* Train weak classifiers \n",
    "* Add them to a final strong classifier by weighting. Weighting by accuracy (typically)\n",
    "* Once added, the data are reweighted\n",
    "  * Misclassified samples gain weight \n",
    "  * Correctly classified samples lose weight (Exception: Boost by majority and BrownBoost - decrease the weight of repeatedly misclassified examples). \n",
    "  * Algo are forced to learn more from misclassified samples\n",
    "  \n",
    "    \n",
    "### [Stacking](http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/)\n",
    "* Also known as Stacked generalization\n",
    "* [From Kaggle:](http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/) Combine information from multiple predictive models to generate a new model. Often times the stacked model (also called 2nd-level model) will outperform each of the individual models due its smoothing nature and ability to highlight each base model where it performs best and discredit each base model where it performs poorly. For this reason, stacking is most effective when the base models are significantly different. \n",
    "* Training a learning algorithm to combine the predictions of several other learning algorithms. \n",
    "  * Step 1: Train learning algo\n",
    "  * Step 2: Combiner algo is trained using algo predictions from step 1.  \n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Ensemble Methods:\n",
    "\n",
    "[Wikipedia](https://en.wikipedia.org/wiki/Ensemble_learning)\n",
    "* Bayes optimal classifier\n",
    "  * An ensemble of all the hypotheses in the hypothesis space. \n",
    "  * Each hypothesis is given a vote proportional to the likelihood that the training dataset would be sampled from a system if that hypothesis were true. \n",
    "  * To facilitate training data of finite size, the vote of each hypothesis is also multiplied by the prior probability of that hypothesis. \n",
    "* Bayesian parameter averaging\n",
    "  * an ensemble technique that seeks to approximate the Bayes Optimal Classifier by sampling hypotheses from the hypothesis space, and combining them using Bayes' law.\n",
    "  * Unlike the Bayes optimal classifier, Bayesian model averaging (BMA) can be practically implemented. \n",
    "  * Hypotheses are typically sampled using a Monte Carlo sampling technique such as MCMC. \n",
    "* Bayesian model combination\n",
    "  * Instead of sampling each model in the ensemble individually, it samples from the space of possible ensembles (with model weightings drawn randomly from a Dirichlet distribution having uniform parameters). \n",
    "  * This modification overcomes the tendency of BMA to converge toward giving all of the weight to a single model. \n",
    "  * Although BMC is somewhat more computationally expensive than BMA, it tends to yield dramatically better results. The results from BMC have been shown to be better on average (with statistical significance) than BMA, and bagging.\n",
    "* Bucket of models\n",
    "  * An ensemble technique in which a model selection algorithm is used to choose the best model for each problem. \n",
    "  * When tested with only one problem, a bucket of models can produce no better results than the best model in the set, but when evaluated across many problems, it will typically produce much better results, on average, than any model in the set.\n",
    "\n",
    "\n",
    "R released\n",
    "* BMS (an acronym for Bayesian Model Selection) package\n",
    "* BAS (an acronym for Bayesian Adaptive Sampling) package\n",
    "* BMA package"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note: Ensemble methods**\n",
    "\n",
    "* Work best with indepedent predictors\n",
    "\n",
    "* Best to utilise different algorithms\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}