mlnotebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Machine Learning with Python \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Module 2\n",
    "\n",
    "### Learning Activity 1: Load the required libraries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import scipy\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import plotly.plotly as py\n",
    "\n",
    "import visplots\n",
    "\n",
    "from plotly.graph_objs import *\n",
    "from plotly.offline import download_plotlyjs, init_notebook_mode, iplot\n",
    "from sklearn import preprocessing, metrics\n",
    "from sklearn.cross_validation import train_test_split, cross_val_score\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.grid_search import GridSearchCV, RandomizedSearchCV\n",
    "from scipy.stats.distributions import randint\n",
    "\n",
    "init_notebook_mode()\n",
    "\n",
    "print(\"libraries all imported, ready to go\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning Activity 2: Importing the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset we will be using throughout this workshop is an adapted version of the Wine Quality case study, available from the UCI Machine Learning repository (https://archive.ics.uci.edu/ml/datasets/Wine+Quality). The goal of this case study is to model the wine quality (into \"low\" or \"high\" quality) based on physicochemical tests (such as fixed and volatile acidity, citric acid, etc.).\n",
    "\n",
    "\n",
    "The first thing you will need to do in order to work with the wine quality dataset is to read the contents from the provided `wine_quality.csv` data file using the `read_csv` command. You should also try to explore the first few rows of the imported wine DataFrame using the `head` function from the `pandas` package (http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.head.html):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Import the data and explore the first few rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to feed the data into our classification models and sklearn, the imported wine quality DataFrame needs to be converted into a `numpy` array. For more information on numpy arrays, see http://scipy-lectures.github.io/intro/numpy/array_object.html. \n",
    "\n",
    "In addition, it is **always** a good practice to check the dimensionality of the imported data using the `shape` command prior to constructing any classification model to make sure you have really imported all the data, and imported it in the correct way (e.g. one common mistake is to get the separator wrong and end up with only one column). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Convert to numpy array and check the dimensionality"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test Activity 3: Inspect your data by indexing and index slicing\n",
    "\n",
    "To select elements in an array, you specify their indices with square bracket notation. For a two-dimensional array, the first index indicates the row number and the second index indicates the column number. Try selecting the values of the first and second columns of the first sample in the npArray:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Print the 1st row and 1st column of npArray"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Print the 1st row and 2nd column of npArray"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To select ranges of elements, we use \"index slicing\". Index slicing is the technical name for the syntax A[lower:upper], where lower refers to the lower bound index that is included, and upper refers to the upper bound index that is not included. Try selecting the first three samples (rows):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Print the first 3 rows of npArray"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and also the first three samples (rows) of the last column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Print the first 3 rows from the last column of npArray"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning Activity 4: Split the data into input features, X, and outputs, y\n",
    "\n",
    "Subsequently, we need to split our initial dataset into the data matrix X (independent variable) and the associated class vector y (dependent or target variable). The input features, _X_,  are the variables that you use to predict the outcome. In this data set, there are ten input features stored in columns 1-10 (index 0-9, although the upper bound is not included so the range for indexing is 0:10), all of which have continuous values. The output label, _y_, holds the information of whether the wine has been rated as high or low quality, and is stored in the final (eleventh) column (index 10). To split the data, we need to assign the columns of the input features and the columns of the output labels to different arrays:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Split to input matrix X and class vector y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try printing the size of the input matrix _X_ and class vector _y_ using the \"`shape`\" command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Print the dimensions of X and y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploratory Data Analysis\n",
    "\n",
    "Visualisation is an integral part of Data Science. Exploratory data analysis (EDA) is the field dealing with the analysis of data sets as a means of summarising their main characteristics, most often using visual methods.\n",
    "\n",
    "Plotly is an online collaborative data analysis and graphing tool that we will use in order to construct fully interactive graphs. The Plotly API allows you to access all of the library's interactive functionality directly from Python (or other programming languages such as R, JavaScript and MATLAB, among others). Crucially, Plotly has recently been made **open-source**, which now enables plotting **offline** without requiring access to their API. _Plotly Offline_ brings interactive Plotly graphs to the _offline_ Jupyter (IPython) Notebook environment.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning Activity 5:  Investigate the y frequencies\n",
    "\n",
    "An important aspect to understand before applying any classification algorithms is how the output labels are distributed. Are they evenly distributed? Imbalances in distribution of labels can often lead to poor classification results for the minority class even if the classification results for the majority class are very good. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Print the y frequencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our current dataset, you can see that the _y_ values are categorical (i.e. they can only take one of a discrete set of values) and have a non-numeric representation, \"high\" vs. \"low\". This can be problematic for scikit-learn and plotting functions in Python, since they assume numerical values, so we need to map the text categories to numerical representations using `LabelEncoder`  and the `fit_transform` function from the `preprocessing` module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Convert the categorical to numeric values, and print the y frequencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualising the data in some way is a good way to get a feel for how the data is distributed. As a simple example, try plotting the frequencies of the class labels (held in _yFreq_), 1 and 0, and see how they are distributed using a barplot from Plotly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Display the y frequencies in a barplot with Plotly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "More examples on Plotly barplots can be found at https://plot.ly/python/bar-charts/. In addition, a full list of arguments on barplots can be found at https://plot.ly/python/reference/#bar/.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning Activity 6: Data scaling\n",
    "\n",
    "It is usually advisable to scale your data prior to fitting a classification model to avoid attributes with\n",
    "greater numeric ranges dominating those with smaller numeric ranges. In order to investigate the range and descriptive statistics of our features, we can apply the `describe()` function from `pandas` to the original `wineQ` DataFrame (**_not_** the numpy array!). For instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Print the descriptive statistics of the wineQ DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Boxplots are a powerful visual aid, commonly used in order to investigate simultaneously the range differences of the input features. Boxplots are a standardised way of displaying the distribution of the data based on the \"five number summary\" (minimum, first quartile, median, third quartile, and maximum). For example, try and plot the features of the _raw_ matrix _X_ using the script for the boxplots:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Create a boxplot of the raw data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many ways of scaling but one common scaling mechanism is auto-scaling, where for each\n",
    "column, the values are centred around the mean and divided by their standard deviation. This scaling\n",
    "mechanism can be applied by calling the `scale()` function in scikit-learn’s `preprocessing` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Auto-scale the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try to re-run the previous plotting script and have a look at the outcome of the boxplot after scaling. Alternatively, \n",
    "if you feel more adventurous, you create a more enhanced version of the boxplot. You can find more online examples at https://plot.ly/python/box-plots/, and also a full list of boxplot arguments at https://plot.ly/python/reference/#box.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create a boxplot of the scaled data (simple or enhanced)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning Activity 7: Investigate the relationship between input features\n",
    "\n",
    "You can visualise the relationship between two variables (features) using a simple scatter plot. This step can give you a good first indication of the ML model model to apply and its complexity (linear vs. non-linear). At this stage, let’s plot the first two variables against each other:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create a scatter plot of the first two features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also relate associations between features to their y classifications by making the colour of\n",
    "the points dependent on the corresponding _y_ value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create an enhanced scatter plot of the first two features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Examples of Plotly scatterplots can be found at https://plot.ly/python/line-and-scatter/ (or for a list of arguments refer to https://plot.ly/python/reference/#scatter/).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bonus Activity 8:  Try plotting different combinations of three features (f1, f2, f3) in the same plot.\n",
    "\n",
    "\n",
    "The scatterplots we have seen so far investigated the relationship between two variables (features). A three-dimensional graph lets you introduce a third axis, typically called the _z_ axis, and can help you understand the relationship between three variables. Plotly's fully interactive functionality allows you to plot, hover, zoom and rotate 3-dimensional scatterplots. For a full list of arguments on 3d plots in Plotly visit https://plot.ly/python/reference/#scatter3d. Other examples on 3D scatterplots using Plotly can be found at https://plot.ly/python/3d-scatter-plots/.\n",
    "\n",
    "_Hint: Investigate the Scatter3d object from Plotly_\n",
    "\n",
    "_Axes in 3D Plotly plots work a bit differently than in 2D (axes are bound to a Scene object -- use help(Scene))._\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create a 3D scatterplot using the first three features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bonus Activity 9: Try different combinations of f1 and f2 (in a grid/scatterplot matrix if you can).\n",
    "\n",
    "\n",
    "A scatterplot matrix shows a grid of scatterplots where each attribute is plotted against all other attributes. For example, try to create a scatterplot matrix of the first four features.\n",
    "You can find further information on how to create and customise subplots with Plotly at https://plot.ly/python/subplots/.\n",
    "\n",
    "_Hints: You may want to use nested loops that iterate through the rows and columns of the grid, and also import and make use of the_ `make_subplots()` _function from Plotly_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create a grid plot of scatterplots using a combination of features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bonus Activity 10: Create a correlation matrix and plot a heatmap of correlations between the input features in X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often, the different features (variables) in X are not completely independent from each other. For example,\n",
    "fixed acidity is related to volatile acidity. To quickly identify which features are related and to\n",
    "what extent, it is useful to see how they are correlated. You can do this by creating a correlation matrix\n",
    "from X using `corrcoef()` in the `numpy` module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Calculate the correlation coefficient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To search for linear relationships between features across all pairs of features, you can use a heatmap\n",
    "of correlations (directly from X), which is simply a matrix of subplots whose colours represent the\n",
    "sizes of the correlations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create a heatmap of the correlation coefficients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Module 3\n",
    "\n",
    "### Learning Activity 1: Split the data into training and test sets\n",
    "\n",
    "Training and testing a classification model on the same dataset is a methodological mistake: a model that would just repeat the labels of the samples that it has just seen would have a perfect score but would fail to predict anything useful on yet-unseen data (poor generalisation). To use different datasets for training and testing, we need to split the wine dataset into two disjoint sets: train and test (**Holdout method**) using the `train_test_split` function. <br/> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Split into training and test sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "XTrain and yTrain are the two arrays you use to train your model. XTest and yTest are the two arrays that you use to evaluate your model. By default, scikit-learn splits the data so that 25% of it is used for testing, but you can also specify the proportion of data you want to use for training and testing.\n",
    "\n",
    "<br/>You can check the sizes of the different training and test sets by using the `shape` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Print the dimensionality of the individual splits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also investigate how the class labels are distributed within the *yTest* vector by using the `itemfreq` function as previously"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Calculate the frequency of classes in yTest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that 129 random samples of class 0 (high quality) and 243 random samples of class 1 (low quality) are included in the yTest set.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning Activity 2: Apply KNN classification algorithm with scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To build KNN models using scikit-learn, you will be using the `KNeighborsClassifier` object, which allows you to set the value of K using the `n_neighbors` parameter (http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html). The optimal choice for the value K is highly data-dependent: in general a larger K suppresses the effects of noise, but makes the classification boundaries less distinct. <br/>\n",
    "\n",
    "\n",
    "For every classification model built with scikit-learn, we will follow four main steps: 1) **Building** the classification model (using either default, pre-defined or optimised parameters), 2) **Training** the model with data, 3) **Testing** the model, and 4) **Performance evaluation** using various metrics. <br/> <br/>\n",
    "\n",
    "We are going to start by trying two pre-defined random values of K and compare their performance. Let us start with a small number of K such as K=3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Build a KNN classifier with 3 nearest neighbors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us try a larger value of K, for instance K = 99 or another number of your own choice; remember, it is good practice to select an **odd** number for K in a binary classification problem to avoid ties. Can you generate the KNN model and print the overall performance for a larger K (such as K=99) using as guidance the previous example? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Build a KNN classifier with 99 nearest neighbors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning Activity 3: Calculate validation metrics for your classifier\n",
    "\n",
    "In a classification task, once you have created your predictive model, you will need to evaluate it. Evaluation functions help you to do this by reporting the performance of the model through four main performance metrics: precision, recall and specificity for the different classes, and overall accuracy. To understand these metrics, it is useful to create a _confusion matrix_, which records all the true positive, true negative, false positive and false negative values.\n",
    "\n",
    "We can compute the confusion matrix for our classifier using the `confusion_matrix` function in the `metrics` module.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Get the confusion matrix for your classifier using metrics.confusion_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Because performance metrics are such an important step of model evaluation, scikit-learn offers a wrapper around these functions, `metrics.classification_report`, to facilitate their computation. It also offers the function `metrics.accuracy_score` that we tried before to compute the overall accuracy.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Report the metrics using metrics.classification_report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Learning Activity 4: Plot the decision boundaries for different models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can visualise the classification boundary created by the KNN classifier using the built-in function `visplots.knnDecisionPlot`. For easier visualisation, you can (interactively) select to view only the test samples from the plot. Remember though that the decision boundary has been built using the _training_ data! <br/> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Check the arguments of the function\n",
    "\n",
    "# Visualise the boundaries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** For smaller values of K the decision boundaries present many \"creases\". In this case the models may suffer from instances of overfitting. For larger values of K, we can see that the decision boundaries are less distinct and tend towards linearity. In these cases the boundaries may be too simple and unable to learn thus leading to cases of underfitting. **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test Activity 5: Try different weight configurations\n",
    "\n",
    "Under some circumstances, it is better to give more importance (\"weight\" in computing terms) to nearer neighbors. This can be accomplished through the `weights` parameter.  When `weights = 'distance'`, weights are assigned to the training data points in a way that is proportional to the inverse of the distance from the query point. In other words, nearer neighbors contribute more to the fit. <br/>\n",
    "\n",
    "What if we use weights based on distance? Does it improve the overall performance?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Build the classifier with two pre-defined parameters (n_neighbors and weights)\n",
    "\n",
    "# Visualise the boundaries of a KNN model with weights equal to \"distance\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Module 4\n",
    "\n",
    "### Learning Activity 1: Implement k-fold cross-validation\n",
    "\n",
    "Let us estimate the accuracy of the classifier on the wine quality dataset by splitting the data 5 consecutive times (the parameter cv gives the number of samples the data is split into) using the cross_val_score function. For example, try to implement cross-validation for knn3, your KNN model with k=3:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Implement cross-validation for knn3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What happens if we change the number of _K_?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parameter Tuning\n",
    "\n",
    "### Learning Activity 2: Grid search on hyperparameters\n",
    "\n",
    "Rather than trying one-by-one predefined values of K, we can automate this process. The scikit-learn library provides the grid search function `GridSearchCV` (http://scikit-learn.org/stable/modules/generated/sklearn.grid_search.GridSearchCV.html), which allows us to exhaustively search for the optimum combination of parameters by evaluating models trained with a particular algorithm with all provided parameter combinations. Further details and examples on grid search with scikit-learn can be found at http://scikit-learn.org/stable/modules/grid_search.html <br/>\n",
    "\n",
    "You can use the `GridSearchCV` function with the validation technique of your choice (in this example, 10-fold cross-validation has been applied) to search for a parametisation of the KNN algorithm that gives a more optimal model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Grid search with 10-fold cross-validation using a dictionary of parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also graphically represent the results of the grid search using a heatmap:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Visualise the grid search results using a heatmap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When evaluating the resulting model it is important to do it on held-out samples that were not seen during the grid search process (XTest). <Br/>\n",
    "So, we are testing our independent XTest dataset using the optimised model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Build the classifier using the optimal parameters detected by grid search "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning Activity 3: Systematic variation of the K neighbors, and the bias-variance trade-off\n",
    "\n",
    "We can graphically represent and investigate how the systematic increase of the number of _K_ neighbors influences the validation, train and test accuracy (and attempt to detect cases of over- or under-fitting). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Explore the benefit of cross-validated results vs. simple training and test data separation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Plot the train, test and validation accuracies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test Activity 4: Randomized search on hyperparameters\n",
    "\n",
    "Unlike `GridSearchCV`, `RandomizedSearchCV` does not exhaustively try all the parameter settings. Instead, it samples a fixed number of parameter settings based on the distributions you specify (e.g. you might specify that one parameter should be sampled uniformly while another is sampled following a Gaussian distribution). The number of parameter settings that are tried is given by `n_iter`. If all parameters are presented as a list, sampling without replacement is performed. If at least one parameter is given as a distribution, sampling with replacement is used. You should use continuous distributions for continuous parameters. Further details can be found at http://scikit-learn.org/stable/modules/grid_search.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Conduct a randomised search on hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Print the optimal n_neighbors detected by randomised search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also graphically represent the results of the randomised search using a scatterplot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Visualise the randomised search results using a scatterplot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Build the classifier using the optimal parameters detected by randomised search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Module 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning Activity 1:  Decision Tree\n",
    "\n",
    "Decision Tree classifiers construct classification models in the form of a tree structure. A decision tree progressively splits the training set into smaller subsets. Each node of the tree represents a subset of the data. Once a new sample is presented to the data, it is classified according to the test condition generated for each node of the tree.\n",
    "\n",
    "Let us build a simple decision tree with 3 layers. We will first evaluate the accuracy, then plot the decision boundaries just as we did for knn. (See http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html for the documentation of the classifier.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Build a Decision Tree classifier with 3 layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning Activity 2:  Random Forests\n",
    "\n",
    "The random forests model is an `ensemble method` since it aggregates a group of decision trees into an ensemble (http://scikit-learn.org/stable/modules/ensemble.html). Ensemble learning involves the combination of several models to solve a single prediction problem. It works by generating multiple classifiers/models which learn and make predictions independently. Those predictions are then combined into a single (mega) prediction that should be as good or better than the prediction made by any one classifer. Unlike single decision trees which are likely to suffer from high Variance or high Bias (depending on how they are tuned) Random Forests use averaging to find a natural balance between the two extremes. <br/> \n",
    "\n",
    "Let us start by building a simple Random Forest model which consists of 100 independently trained decision trees. For further details and examples on how to construct a Random Forest, see http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Build a Random Forest classifier with 100 decision trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning Activity 3: Visualising the RF accuracy\n",
    "\n",
    "We can also investigate how the overall test accuracy gets influenced with the increase of `n_estimators` (decision trees) in our model. In order to do so, we can use the provided `rfAvgAcc` function from `visplots`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Visualise the average accuracy "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning Activity 4: Feature Importance \n",
    "\n",
    "Random forests allow you to compute a heuristic for determining how “important” a feature is in predicting a target. This heuristic measures the change in prediction accuracy if you take a given feature and permute (scramble) it across the datapoints in the training set. The more the accuracy drops when the feature is permuted, the more “important” we can conclude the feature is.\n",
    "\n",
    "We can use the `feature_importances_` attribute of the RF classifier to obtain the relative importance of each feature, which we can then visualise using a simple bar plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Display the importance of the features in a barplot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Learning activity 5: Boundary visualisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can visualise the classification boundary created by the Random Forest using the `visplots.rfDecisionPlot` function. You can check the arguments passed in this function by using the `help` command. For easier visualisation, only the test samples have been included in the plot. And remember that the decision boundary has been built using the _training_ data!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Check the arguments of the function\n",
    "\n",
    "# Visualise the boundaries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning Activity 6: Tuning Random Forests with grid search\n",
    "\n",
    "Random forests offer several parameters that can be tuned. In this case, parameters such as `n_estimators`, `max_features`, `max_depth` and `min_samples_leaf` can be some of the parameters to be optimised. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# View the list of arguments to be optimised\n",
    "help(RandomForestClassifier())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a dictionary of allowed parameter ranges for `n_estimators` and `max_depth` (or include more of the parameters you would like to tune) and conduct a grid search with cross validation using the `GridSearchCV` function as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Conduct a grid search with 10-fold cross-validation using the dictionary of parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, testing our independent XTest dataset using the optimised model: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Build the classifier using the optimal parameters detected by grid search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also graphically represent the results of the grid search using a heatmap:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Create a heatmap like the one you made when you applied GridSearchCV to KNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Bonus Activity 7: Parallelisation\n",
    "\n",
    "\n",
    "The scikit-learn implementation of Random Forests also features the parallel construction of the trees and the parallel computation of the predictions through the n_jobs parameter.\n",
    "If `n_jobs=k` then computations are partitioned into k jobs, and run on k cores of the machine.\n",
    "If `n_jobs=-1` then all cores available on the machine are used.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Change the value of n_jobs and estimate the excution time of fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Plot a graph"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}