03_model_performance.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/ascarrambad/ml-21-22/blob/main/03_model_performance.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zwPfDFnz1ECg"
   },
   "source": [
    "# Machine Learning SP 2021/2022\n",
    "\n",
    "Prof. Cesare Alippi<br>\n",
    "Giorgia Adorni ([`giorgia.adorni@usi.ch`](mailto:giorgia.adorni@usi.ch))<br>\n",
    "Luca Butera ([`luca.butera@usi.ch`](mailto:luca.butera@usi.ch))<br>\n",
    "Matteo Riva ([`matteo.riva@usi.ch`](mailto:matteo.riva@usi.ch))\n",
    "\n",
    "---\n",
    "# Lab 03: Model Performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "YMfbdgMUQvJX"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# --- Auxiliary code -------------------- #\n",
    "\n",
    "# function to plot decision boundaries\n",
    "def plot_decision_surface(model, x, y, transform=lambda x:x):\n",
    "    \n",
    "  from matplotlib.colors import ListedColormap\n",
    "  # color_maps\n",
    "  cm = plt.cm.RdBu\n",
    "  cols = ['#FF0000', '#0000FF']\n",
    "  cm_bright = ListedColormap(cols)  \n",
    "\n",
    "  #init figure\n",
    "  fig = plt.figure()\n",
    "\n",
    "  # Create mesh\n",
    "  h = .1  # step size in the mesh\n",
    "  x_min, x_max = x[:, 0].min() - .5, x[:, 0].max() + .5\n",
    "  y_min, y_max = x[:, 1].min() - .5, x[:, 1].max() + .5\n",
    "  xx, yy = np.meshgrid(np.arange(x_min, x_max, h), \n",
    "                       np.arange(y_min, y_max, h))\n",
    "\n",
    "  # plot train data\n",
    "  cy = [cols[int(yi)] for yi in y]\n",
    "  plt.scatter(x[:, 0], x[:, 1], c=cy, cmap=cm_bright,\n",
    "              edgecolors='k')\n",
    "  plt.xlim(xx.min(), xx.max())\n",
    "  plt.ylim(yy.min(), yy.max())\n",
    "\n",
    "  plt.xlabel(r'$x_1$')\n",
    "  plt.ylabel(r'$x_2$');\n",
    "\n",
    "  y_pred = model.predict(transform(np.c_[xx.ravel(), yy.ravel()]))\n",
    "\n",
    "  y_pred = y_pred.reshape(xx.shape)\n",
    "  plt.contourf(xx, yy, y_pred > 0.5, cmap=cm, alpha=.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "klMKmWT8KpcD"
   },
   "source": [
    "## Use-case scenario\n",
    "Let's pretend a company asked us to develop a machine learning model for one of their machinery. \n",
    "\n",
    "We are given some labelled data $(x_i, y_i)$ for $i=1, ..., N$, and asked to provide\n",
    "- the best model $f(x; \\hat \\theta)$ we can find;\n",
    "- an estimate of its performance $V(\\hat \\theta)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "j1fYESi8MXe4"
   },
   "outputs": [],
   "source": [
    "# Prepare some data\n",
    "N = 400\n",
    "\n",
    "np.random.seed(200402)\n",
    "Xa = np.random.randn(N//4, 2)\n",
    "Xb = np.random.randn(N//4, 2) + np.array([ 8.,  1.])\n",
    "Xc = np.random.randn(N//4, 2) + np.array([-4., -1.])\n",
    "Xd = np.random.randn(N//4, 2) + np.array([ 4., -1.])\n",
    "\n",
    "X = np.vstack([Xa, Xb, Xc, Xd])\n",
    "\n",
    "plt.scatter(X[:, 0], X[:, 1]);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vtEtGAdPMxRE"
   },
   "outputs": [],
   "source": [
    "# Add labels\n",
    "y = np.zeros((N,))\n",
    "y[N//2:] = 1\n",
    "\n",
    "plt.scatter(X[:, 0], X[:, 1], c=y);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ud6eXh23MVyF"
   },
   "source": [
    "## Train some models\n",
    "\n",
    "Let's start from a logistic regression ([sklearn doc](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LKgpcppqNpoH"
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "logreg = LogisticRegression()\n",
    "logreg.fit(X, y)\n",
    "\n",
    "plot_decision_surface(model=logreg, x=X, y=y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "weLS_eJHKpQ4"
   },
   "source": [
    "Let's try a feed-forward neural net ([keras doc](https://keras.io/models/sequential/)). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PI31Ut3nQb5I"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras.losses import BinaryCrossentropy\n",
    "\n",
    "def create_ffnn(neurons=3, activation=\"tanh\"):\n",
    "    model = Sequential()\n",
    "    model.add(Dense(neurons, input_shape=(2,), activation=activation))\n",
    "    model.add(Dense(1, activation=\"sigmoid\"))    \n",
    "    model.compile(loss=BinaryCrossentropy(), metrics=[\"accuracy\"])\n",
    "    return model\n",
    "epochs = 200\n",
    "\n",
    "ffnn = create_ffnn() \n",
    "ffnn.fit(X, y, epochs=epochs, verbose=0)\n",
    "\n",
    "plot_decision_surface(model=ffnn, x=X, y=y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "myKkNJY6AMrM"
   },
   "source": [
    "## Performance assessment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HVhXxlw9KpHn"
   },
   "source": [
    "### Split the data: cross-validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "iAUs0grQS-cx"
   },
   "outputs": [],
   "source": [
    "# Set sizes\n",
    "n = int(N * .8)  # Training points\n",
    "l = N - n        # Test points\n",
    "print(\"num training points: n=\",  n)\n",
    "print(\"num test points:     l= \", l)\n",
    "\n",
    "# Data split\n",
    "X_train, y_train = X[:n], y[:n]\n",
    "X_test, y_test = X[n:], y[n:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6eprfe6CUAJR"
   },
   "outputs": [],
   "source": [
    "# Train the two models\n",
    "#logistic\n",
    "logreg = LogisticRegression()\n",
    "logreg.fit(X_train, y_train)\n",
    "#nn\n",
    "ffnn = create_ffnn() \n",
    "ffnn.fit(X_train, y_train, epochs=epochs, verbose=0)\n",
    "\n",
    "# Accuracy: rate of correct classifications:\n",
    "#logistic\n",
    "correct_classif = (logreg.predict(X_test) == y_test).astype(int)\n",
    "print(\"LR acc   :\", np.mean(correct_classif))\n",
    "#nn\n",
    "y_pred = np.array(ffnn.predict(X_test) > .5, dtype=int)[:, 0]\n",
    "correct_classif = (y_pred == y_test).astype(int)\n",
    "print(\"NN acc   :\", np.mean(correct_classif))\n",
    "\n",
    "# Plot boundaries\n",
    "plot_decision_surface(model=logreg, x=X_test, y=y_test)\n",
    "plot_decision_surface(model=ffnn,   x=X_test, y=y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Sw7Qh0rRRPgA"
   },
   "source": [
    "#### What's wrong?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fY0HGwt_RMA6"
   },
   "outputs": [],
   "source": [
    "# Plot split data\n",
    "plt.subplot(121)\n",
    "plt.scatter(X_train[:, 0], X_train[:, 1], c=y_train);\n",
    "plt.subplot(122)\n",
    "plt.scatter(X_test[:, 0], X_test[:, 1], c=y_test);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MIYgf6cZeWEo"
   },
   "source": [
    "We did not shuffled the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Nbgcu11LehCi"
   },
   "outputs": [],
   "source": [
    "# Shuffle the data!\n",
    "p = np.random.permutation(N)\n",
    "idx_train = p[:n]\n",
    "idx_test = p[n:]\n",
    "\n",
    "# Data split\n",
    "X_train, y_train = X[idx_train], y[idx_train]\n",
    "X_test, y_test = X[idx_test], y[idx_test]\n",
    "\n",
    "# Plot split data\n",
    "plt.subplot(121)\n",
    "plt.scatter(X_train[:, 0], X_train[:, 1], c=y_train);\n",
    "plt.subplot(122)\n",
    "plt.scatter(X_test[:, 0], X_test[:, 1], c=y_test);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0zuleJgtVYrk"
   },
   "source": [
    "\n",
    "SkLearn provides many [utilities](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection). For example, `train_test_split`, `ShuffleSplit` and `StratifiedShuffleSplit`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "m3ljaUwnOQ1O"
   },
   "outputs": [],
   "source": [
    "# or with SkLearn\n",
    "from sklearn.model_selection import train_test_split\n",
    "idx_train, idx_test = train_test_split(np.arange(N), test_size=0.2, shuffle=True)\n",
    "#X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle=True)\n",
    "\n",
    "# Data split\n",
    "X_train, y_train = X[idx_train], y[idx_train]\n",
    "X_test, y_test = X[idx_test], y[idx_test]\n",
    "\n",
    "# Plot split data\n",
    "plt.subplot(121)\n",
    "plt.scatter(X_train[:, 0], X_train[:, 1], c=y_train);\n",
    "plt.subplot(122)\n",
    "plt.scatter(X_test[:, 0], X_test[:, 1], c=y_test);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OtF6T3xGgukx"
   },
   "outputs": [],
   "source": [
    "# Train the two models\n",
    "#logistic\n",
    "logreg = LogisticRegression()\n",
    "logreg.fit(X_train, y_train)\n",
    "#nn\n",
    "ffnn = create_ffnn(8) \n",
    "ffnn.fit(X, y, epochs=epochs, verbose=0)\n",
    "\n",
    "# Evaluate accuracy\n",
    "acc_lr = logreg.score(X_test, y_test)\n",
    "[loss_nn, acc_nn] = ffnn.evaluate(X_test, y_test)\n",
    "print(\"acc_lr\", acc_lr)\n",
    "print(\"acc_nn\", acc_nn)\n",
    "\n",
    "# Plot boundaries\n",
    "plot_decision_surface(model=logreg, x=X_test, y=y_test)\n",
    "plot_decision_surface(model=ffnn, x=X_test, y=y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vr3mqNKX2y8B"
   },
   "source": [
    "## Can we say which model is the best?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WMK-joaW8CEU"
   },
   "source": [
    "\n",
    "$$\n",
    "e_i = \n",
    "\\begin{cases}\n",
    "1, & \\text{if } y_i =   f(x_i;\\hat \\theta)\\\\\n",
    "0, & \\text{if } y_i \\ne f(x_i;\\hat \\theta)\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "$$\\overline e = \\frac{1}{l}\\sum_{i=1}^l e_i ;\\qquad s^2 = \\overline e (1 - \\overline e)$$\n",
    "\n",
    "$$T = \\frac{\\overline e_{nn} - \\overline e_{lr}}\n",
    "           {\\sqrt{ \\frac{s^2_{nn}}{l} + \\frac{s^2_{lr}}{l}}}$$\n",
    "           "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wtY5rl712zZR"
   },
   "outputs": [],
   "source": [
    "# Logistic Regression part\n",
    "e_lr = (y_test == logreg.predict(X_test)).astype(int)\n",
    "mean_e_lr = e_lr.mean()\n",
    "s2_lr = mean_e_lr * (1 - mean_e_lr)\n",
    "print(\"mean: {} -- s2: {}\".format(mean_e_lr, s2_lr))\n",
    "\n",
    "# Neural Net part\n",
    "y_pred = (ffnn.predict(X_test) > .5)[:, 0].astype(int)\n",
    "e_nn = (y_test == y_pred).astype(int)\n",
    "mean_e_nn = e_nn.mean()\n",
    "s2_nn = mean_e_nn * (1 - mean_e_nn)\n",
    "print(\"mean: {} -- s2: {}\".format(mean_e_nn, s2_nn))\n",
    "\n",
    "# Test statistics\n",
    "T = (mean_e_nn - mean_e_lr) \n",
    "T /= np.sqrt( s2_nn / l + s2_lr / l )\n",
    "print(\"is T={} in 95\\% confidence interval (-1.96, 1.96) ?\".format(T))\n",
    "\n",
    "# t-test\n",
    "from scipy.stats import ttest_ind\n",
    "tt, p_val = ttest_ind(e_lr, e_nn, equal_var=False)\n",
    "print('t-test: T={:.2f}, p-value={:.4f}'.format(tt, p_val))\n",
    "\n",
    "# paired t-test\n",
    "from scipy.stats import ttest_rel\n",
    "tt, p_val = ttest_rel(e_lr, e_nn)\n",
    "print('t-test: T={:.2f}, p-value={:.4f}'.format(tt, p_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aYsXf0KBI5O5"
   },
   "source": [
    "## Did we finish?\n",
    "\n",
    "- The best model was the neural net,\n",
    "- We estimated its performance,\n",
    "- ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b40Y9ecmYmhS"
   },
   "source": [
    "We retrain the best model on the entire dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "aL4xRLUDYsSo"
   },
   "outputs": [],
   "source": [
    "ffnn_final = create_ffnn()  # Remember: train the model from scratch.\n",
    "ffnn_final.fit(X, y, epochs=epochs, verbose=0)\n",
    "\n",
    "from tensorflow.keras.models import save_model \n",
    "save_model(ffnn_final, \"my_final_model.tf\")\n",
    "\n",
    "from tensorflow.keras.models import load_model \n",
    "loaded_model = load_model(\"my_final_model.tf\")\n",
    "\n",
    "# Check they are actually the same\n",
    "print(ffnn_final.evaluate(X, y))\n",
    "print(loaded_model.evaluate(X, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zPTt0iNdS9xZ"
   },
   "source": [
    "---\n",
    "\n",
    "\n",
    "## K-fold cross-validation\n",
    "\n",
    "Say we have a single model and we want to identify a confidence interval for its accuracy.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "P01zmRGnlUA1"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "# Utility to split the data\n",
    "kfcv = StratifiedKFold(n_splits=10, shuffle=True)\n",
    "fold_iterator = kfcv.split(X, y)\n",
    "\n",
    "# Utility to split the data\n",
    "acc_nn = []\n",
    "\n",
    "for idx_train, idx_val in fold_iterator:\n",
    "\n",
    "    # split data\n",
    "    X_train, y_train = X[idx_train], y[idx_train]\n",
    "    X_val, y_val = X[idx_val], y[idx_val]\n",
    "\n",
    "    # train model\n",
    "    ffnn = create_ffnn(8)  # Remember: train the model from scratch.\n",
    "    ffnn.fit(X_train, y_train, epochs=epochs, verbose=0)\n",
    "\n",
    "    # evaluate model\n",
    "    _, current_acc = ffnn.evaluate(X_val, y_val)\n",
    "    acc_nn.append(current_acc)\n",
    "\n",
    "print(\"Acc list:\", acc_nn)\n",
    "print(\"This is our estimated accuracy:  {:.3f} +- {:.3f}\".format(np.mean(acc_nn), np.std(acc_nn)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iz4k0X72uVg1"
   },
   "source": [
    "Even from here we could have compared the two models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0f4-k1e8xG3O"
   },
   "outputs": [],
   "source": [
    "# Utility to split the data\n",
    "kfcv = StratifiedKFold(n_splits=10, shuffle=True)\n",
    "fold_iterator = kfcv.split(X, y)\n",
    "\n",
    "# Utility to split the data\n",
    "acc_nn = []\n",
    "acc_lr = []\n",
    "\n",
    "for idx_train, idx_val in fold_iterator:\n",
    "\n",
    "    X_train, y_train = X[idx_train], y[idx_train]\n",
    "    X_val, y_val = X[idx_val], y[idx_val]\n",
    "\n",
    "    ffnn = create_ffnn(8)  # Remember: train the model from scratch.\n",
    "    ffnn.fit(X_train, y_train, epochs=epochs, verbose=0)\n",
    "\n",
    "    logreg = LogisticRegression() \n",
    "    logreg.fit(X_train, y_train)\n",
    "\n",
    "    _, current_acc = ffnn.evaluate(X_val, y_val)\n",
    "    acc_nn.append(current_acc)\n",
    "    current_acc = logreg.score(X_val, y_val)\n",
    "    acc_lr.append(current_acc)\n",
    "\n",
    "print(\"LogReg list:   \", acc_lr)\n",
    "print(\"NeuralNet list:\", acc_nn)\n",
    "\n",
    "print(\"LogReg:     {:.3f} +- {:.3f}\".format(np.mean(acc_lr), np.std(acc_lr)))\n",
    "print(\"NeuralNet:  {:.3f} +- {:.3f}\".format(np.mean(acc_nn), np.std(acc_nn)))\n",
    "\n",
    "# Paired two sample test\n",
    "T, p_val = ttest_rel(acc_lr, acc_nn)\n",
    "print('t-test: T={:.2f}, p-value={:.4f}'.format(T, p_val))\n",
    "print(\"is T={:.2f} in 95\\% confidence interval (-1.96, 1.96) ?\".format(T))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XSeLHmXkM6Ih"
   },
   "source": [
    "\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "## More than two models and hyper-parameter tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zdr8cuYxNbTz"
   },
   "outputs": [],
   "source": [
    "# neurons - activation\n",
    "model_parameters = [(3, \"tanh\"), \n",
    "                    (3, \"relu\"), \n",
    "                    (6, \"tanh\"), \n",
    "                    (6, \"relu\")]  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KFr9fuUZZiJL"
   },
   "source": [
    "### Data split\n",
    "\n",
    "```\n",
    "|               Entire data (N)               |\n",
    "|---------------------------------------------|\n",
    "\n",
    "|          Training data          | Test data |\n",
    "|---------------------------------|-----------|\n",
    "\n",
    "|          Training data          | Test data |\n",
    "| Actual fitting data  | Val.data |           |\n",
    "|----------------------|----------|-----------|\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "exBdNL6WcmDX"
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, shuffle=True)\n",
    "\n",
    "X_atr, X_val, y_atr, y_val = train_test_split(X_train, y_train, test_size=0.3, shuffle=True)\n",
    "\n",
    "epochs = 200\n",
    "\n",
    "# Model selection\n",
    "acc_list = []\n",
    "for (neurons, activation) in model_parameters:\n",
    "    print(\"Training NN with {} neurons and {} activation\".format(neurons, activation))\n",
    "\n",
    "    model = create_ffnn(neurons=neurons, activation=activation)\n",
    "    model.fit(X_atr, y_atr, epochs=epochs, verbose=0)\n",
    "\n",
    "    _, acc = model.evaluate(X_val, y_val)\n",
    "    acc_list.append(acc)\n",
    "\n",
    "imax = np.argmax(acc_list)\n",
    "print(\"Best model parameters:\", model_parameters[imax])\n",
    "\n",
    "# Performance of best model\n",
    "(neurons, activation) = model_parameters[imax]\n",
    "best_model = create_ffnn(neurons=neurons, activation=activation)\n",
    "best_model.fit(X_train, y_train, epochs=epochs, verbose=0)\n",
    "_, final_acc = best_model.evaluate(X_test, y_test, verbose=0)\n",
    "print(\"Best model accuracy:\", final_acc)\n",
    "\n",
    "# Final trained model\n",
    "final_model = create_ffnn(neurons=neurons, activation=activation)\n",
    "final_model.fit(X, y, epochs=epochs, verbose=0);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gGHvrBL4Dv08"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "04_model_performance.py",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}