02 Perceptron.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Clasificación con el Algoritmo del Perceptrón\n",
    "\n",
    "Implementaremos una versión básica del algoritmo del perceptrón para clasificación binaria.\n",
    "\n",
    "En este caso los datos de entrada serán de **dos dimensiones**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "np.set_printoptions(suppress=True)  # no usar notacion \"e\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Datos\n",
    "\n",
    "Usaremos scikit-learn ([make_classification](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_classification.html)) para generar datos aleatorios de dos dimensiones, divididos en dos clases etiquetadas con los valores 1 y -1.\n",
    "\n",
    "Los datos serán linealmente separables. Fijamos la semilla aleatoria para obtener siempre los mismos resultados."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_classification\n",
    "\n",
    "size = 100\n",
    "X, y_true = make_classification(n_samples=size, n_features=2, n_classes=2,\n",
    "                                   n_redundant=0, n_informative=2,\n",
    "                                   n_clusters_per_class=1, class_sep=0.5,\n",
    "                                   random_state=1)\n",
    "y_true[y_true == 0] = -1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_true.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Graficamos para ver el resultado. Acá, los azules son 1, los rojos -1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(X[y_true==1, 0], X[y_true==1, 1], color=\"royalblue\", label=\"1\")\n",
    "plt.scatter(X[y_true==-1, 0], X[y_true==-1, 1], color=\"red\", label=\"-1\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### División en Entrenamiento y Evaluación"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_size = 60\n",
    "val_size = size - train_size\n",
    "X_train, X_val = X[:train_size], X[train_size:]\n",
    "y_train, y_val = y_true[:train_size], y_true[train_size:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train.shape, X_val.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Algoritmo del Perceptrón"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inicialización\n",
    "\n",
    "Comenzamos a partir de un vector cualquiera:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w =np.ones(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predicción\n",
    "\n",
    "Definimos la función de predicción:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(X, w):\n",
    "    return np.sign(np.dot(X, w))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Probamos predecir con el primer modelo:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = predict(X_val, w)\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_val"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluación\n",
    "\n",
    "Calcularemos la \"accuracy\" sobre los datos de evaluación."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def accuracy(y_true, y_pred):\n",
    "    return (y_true == y_pred).sum() / y_true.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy(y_val, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "En este caso el resultado es muy bajo.\n",
    "\n",
    "Graficamos la frontera de decisión:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import plot_decision_boundary\n",
    "\n",
    "def predfun(x):\n",
    "    return predict(x, w)\n",
    "    \n",
    "plot_decision_boundary(predfun, \n",
    "                       X_val, \n",
    "                       y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vemos que la región rosa debería contener a los puntos rojos y la celeste a los puntos azules."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Un Paso del Algoritmo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Primero debemos buscar un elemento de entrenamiento mal clasificado:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = predict(X_train, w)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.where(y_train != y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train[0], y_pred[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tomamos el primero:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i = 0\n",
    "xi, yi = X_train[i], y_train[i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xi.shape  # qué da?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verificamos que la clasificación es incorrecta:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "yi_pred = predict(xi, w)\n",
    "yi, yi_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Probemos graficar el punto:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y2 = np.ones(size)\n",
    "y2[i] = -1\n",
    "plot_decision_boundary(lambda x: predict(x, w), X, y2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ahora actualicemos el modelo y veamos si se corrige la clasificación:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = 1.0  # learning rate\n",
    "new_w = w + r * xi * yi\n",
    "predict(xi, new_w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Probemos otras tasas de aprendizaje, a ver qué pasa:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for r in [0.001, 0.01, 0.1, 0.5, 1.0]:\n",
    "    new_w = w + r * xi * yi\n",
    "    print(r, predict(xi, new_w))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lo dejamos en $r=1$. Evaluemos el nuevo modelo, para ver si mejora el resultado:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = predict(X_val, new_w)\n",
    "accuracy(y_val, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_decision_boundary(lambda x: predict(x, new_w), X, y_true)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_w"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perceptrón: Algoritmo completo\n",
    "\n",
    "Implementaremos el algoritmo completo, iterando hasta que todos los datos de entrenamiento estén clasificados correctamente:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w = np.ones(2)\n",
    "\n",
    "r = 0.5  # qué pasaría\n",
    "step = 0\n",
    "finished = False\n",
    "while not finished:\n",
    "    # 1. predecir usando el modelo actual\n",
    "    y_pred = predict(X_train, w)\n",
    "    y_val_pred = predict(X_val, w)\n",
    "    \n",
    "    # 2. evaluar\n",
    "    train_acc = accuracy(y_train, y_pred)\n",
    "    val_acc = accuracy(y_val, y_val_pred)\n",
    "    print(f'step={step}\\ttrain_acc={train_acc:0.2}\\tval_acc={val_acc:0.2}')\n",
    "\n",
    "    # 3. si no clasifica todo bien, actualizar modelo\n",
    "    if (y_train == y_pred).all():\n",
    "        finished = True\n",
    "    else:\n",
    "        indices = np.where(y_train != y_pred)\n",
    "        i = indices[0][0]\n",
    "        xi, yi = X_train[i], y_train[i]\n",
    "        w = w + r * xi * yi\n",
    "        \n",
    "        step = step + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Grafiquemos el resultado final:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "    plot_decision_boundary(lambda x: predict(x, w), X, y_true)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ejercicios\n",
    "\n",
    "1. Experimentar con otras tasas de aprendizaje.\n",
    "2. Generar datos no linealmente separables (probar achicando el valor de 'class_sep' en 'make_classification').\n",
    "3. Adaptar el algoritmo del perceptrón para que funcione con estos datos (ver variante \"pocket\" en [Wikipedia](https://en.wikipedia.org/wiki/Perceptron)).\n",
    "4. Hacer todo con el perceptrón de Scikit-learn ([Perceptron](https://scikit-learn.org/stable/modules/linear_model.html#perceptron))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Referencias\n",
    "\n",
    "- [Perceptron (Wikipedia)](https://en.wikipedia.org/wiki/Perceptron)\n",
    "- [Perceptron Learning Algorithm: A Graphical Explanation Of Why It Works](https://towardsdatascience.com/perceptron-learning-algorithm-d5db0deab975)\n",
    "- Gallant, S. I. (1990). [Perceptron-based learning algorithms.](https://ftp.cs.nyu.edu/~roweis/csc2515-2006/readings/gallant.pdf) IEEE Transactions on Neural Networks, vol. 1, no. 2, pp. 179–191. \n",
    "\n",
    "Scikit-learn:\n",
    "\n",
    "- [make_classification](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_classification.html)\n",
    "- [Perceptron](https://scikit-learn.org/stable/modules/linear_model.html#perceptron)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}