Extra-01.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MNIST handwritten digits classification with naive Bayes \n",
    "\n",
    "In this notebook, we'll use [naive Bayes classifiers](https://scikit-learn.org/stable/modules/naive_bayes.html) to classify MNIST digits using scikit-learn (version 0.20 or later required).\n",
    "\n",
    "First, the needed imports. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "from pml_utils import get_mnist, show_failures\n",
    "\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import datasets, __version__\n",
    "from sklearn.naive_bayes import GaussianNB, BernoulliNB\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set()\n",
    "\n",
    "from packaging.version import Version\n",
    "assert(Version(__version__) >= Version(\"0.20\")), \"Version >= 0.20 of sklearn is required.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we load the MNIST data. First time we need to download the data, which can take a while."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, y_train, X_test, y_test = get_mnist('MNIST')\n",
    "\n",
    "print('MNIST data loaded: train:',len(X_train),'test:',len(X_test))\n",
    "print('X_train:', X_train.shape)\n",
    "print('y_train:', y_train.shape)\n",
    "print('X_test', X_test.shape)\n",
    "print('y_test', y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training data (`X_train`) is a matrix of size (60000, 784), i.e. it consists of 60000 digits expressed as 784 sized vectors (28x28 images flattened to 1D). `y_train` is a 60000-dimensional vector containing the correct classes (\"0\", \"1\", ..., \"9\") for each training digit.\n",
    "\n",
    "Let's take a closer look. Here are the first 10 training digits:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "pltsize=1\n",
    "plt.figure(figsize=(10*pltsize, pltsize))\n",
    "\n",
    "for i in range(10):\n",
    "    plt.subplot(1,10,i+1)\n",
    "    plt.axis('off')\n",
    "    plt.imshow(X_train[i,:].reshape(28, 28), cmap=\"gray\")\n",
    "    plt.title('Class: '+y_train[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive Bayes classifiers\n",
    "\n",
    "Naive Bayes classifiers are a family of simple classifiers based on applying Bayes' theorem. The classifiers are called \"naive\" as we make a strong assumption that the features are conditionally independent given the value of the class variable. While this assumption is not usually true, a naive Bayes classifier may in practice work reasonably well. Naive Bayes classifiers are also simple and fast compared to many more sophisticated methods.\n",
    "\n",
    "The classification rule for naive Bayes is\n",
    "\\begin{equation}\n",
    "\\hat{y} = \\arg\\max_yP(y)\\prod_{i=1}^nP(x_i|y)\n",
    "\\end{equation}\n",
    "where $P(y)$ is the prior probability of class $y$ and $P(x_i|y)$ is the class-conditional likelihood of feature $i$.\n",
    "\n",
    "## Gaussian naive Bayes\n",
    "\n",
    "In Gaussian naive Bayes, the likelihood of the features is assumed to be Gaussian\n",
    "\\begin{equation}\n",
    "P(x_i|y) = \\mathcal{N}(x_i\\,|\\,\\mu_{iy},\\sigma_{iy}^2) \n",
    "\\end{equation}\n",
    "where $\\mu_{iy}$ and $\\sigma_{iy}^2$ are the mean and variance, respectively, of feature $i$ in objects of class $y$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mu = 192.\n",
    "sigma = 32.\n",
    "x = np.arange(255.)\n",
    "plt.plot(x, 1/(sigma * np.sqrt(2 * np.pi)) * np.exp( - (x - mu)**2 / (2 * sigma**2)),\n",
    "         lw=3)\n",
    "plt.xticks([0,127,255])\n",
    "plt.title('Gaussian distribution with $\\mu={}$ and $\\sigma={}$'.format(mu, sigma));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The prior probabilities $P(y)$ are learned from training data by default. \n",
    "\n",
    "### Learning\n",
    "\n",
    "Training a naive Bayes classifier is fast:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "clf_gnb = GaussianNB()\n",
    "clf_gnb.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can take a look at the mean and variance of features for each class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pltsize=1\n",
    "plt.figure(figsize=(10*pltsize, pltsize))\n",
    "plt.suptitle('Mean of each feature', y=1.3)\n",
    "\n",
    "for i in range(10):\n",
    "    plt.subplot(1,10,i+1)\n",
    "    plt.axis('off')\n",
    "    plt.imshow(clf_gnb.theta_[i,:].reshape(28, 28), cmap=\"gray\")\n",
    "    plt.title('Class: '+str(i))\n",
    "\n",
    "plt.figure(figsize=(10*pltsize, pltsize))\n",
    "plt.suptitle('Variance of each feature', y=1.1)\n",
    "for i in range(10):\n",
    "    plt.subplot(1,10,i+1)\n",
    "    plt.axis('off')\n",
    "    plt.imshow(clf_gnb.var_[i,:].reshape(28, 28), cmap=\"gray\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference\n",
    "\n",
    "Evaluating a naive Bayes classifier is also fast:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "predictions = clf_gnb.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy of the classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "print('Predicted', len(predictions), 'digits with accuracy:', accuracy_score(y_test, predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also inspect the results in more detail. Let's use the `show_failures()` helper function to show the wrongly classified test digits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_failures(predictions, y_test, X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_failures(predictions, y_test, X_test, trueclass='5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can observe that the classifier makes rather easy mistakes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bernoulli naive Bayes\n",
    "\n",
    "Gaussian naive Bayes assumes that the features are normally distributed, which is not a good assumption for the MNIST digits. Let's therefore use a second approach and model each feature as a binary variable (\"black\" or \"white\"). A suitable distribution in this case is the Bernoulli\n",
    "\\begin{equation}\n",
    "P(x_i|y) = \\mathrm{Ber}(x_i\\,|\\,\\theta_{iy}) \n",
    "\\end{equation}\n",
    "where $\\theta_{iy}$ is the probability that feature $i$ is \"white\" in objects of class $y$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "theta = 0.4\n",
    "plt.bar([0, 255], [1-theta, theta], width=16.)\n",
    "plt.xticks([0,127,255])\n",
    "plt.title('Bernoulli distribution with $\\\\theta={}$'.format(theta));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning\n",
    "\n",
    "Bernoulli naive Bayes assumes binary data, so we'll binarize the digits with a threshold in the middle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "clf_bnb = BernoulliNB(binarize=128.)\n",
    "clf_bnb.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can take a look at the probabilities of features for each class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pltsize=1\n",
    "plt.figure(figsize=(10*pltsize, pltsize))\n",
    "plt.suptitle('Probability of each feature', y=1.3)\n",
    "\n",
    "for i in range(10):\n",
    "    plt.subplot(1,10,i+1)\n",
    "    plt.axis('off')\n",
    "    plt.imshow(np.exp(clf_bnb.feature_log_prob_[i,:]).reshape(28, 28), cmap=\"gray\")\n",
    "    plt.title('Class: '+str(i))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "predictions = clf_bnb.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "print('Predicted', len(predictions), 'digits with accuracy:', accuracy_score(y_test, predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "show_failures(predictions, y_test, X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_failures(predictions, y_test, X_test, trueclass='5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}