TP1_MAPI3_sujet.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    },
    "colab": {
      "name": "TP1_MAPI3_sujet.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/DavidBert/TP-MAPI3/blob/master/TP1_MAPI3_sujet.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i6ZxpPrwYu9x"
      },
      "source": [
        "# Travaux pratiques: Reseaux de neurones"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RzGYKyI2Yu95"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import numpy as np"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xj_cLNzEYu96"
      },
      "source": [
        "Dans ce TP nous allons implémenter les differents éléments qui composent un réseau de neurones "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4fjKfsltYu98"
      },
      "source": [
        "# Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h7FJpptAYu99"
      },
      "source": [
        "Dans la première partie, nous allons essayer de prédire le prix de vente de biens immobiliers de la ville Boston.  \n",
        "Nous allons pour cela utiliser le classique jeu de données boston house-prices, disponible directement dans la librairie de machine learning [scikit-learn](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html)  \n",
        "Commençons donc par importer le dataset:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jH-L-U41Yu99"
      },
      "source": [
        "from sklearn.datasets import load_boston\n",
        "dataset = load_boston()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZWBv3HBQYu9-"
      },
      "source": [
        "Nous allons donc essayer de prédire le prix des maisons à partir de 13 features présentées [ici](https://scikit-learn.org/stable/datasets/toy_dataset.html#boston-dataset).   \n",
        "\n",
        "Commençons par séparer le dataset en 2:\n",
        "* Un ensemble d'apprentissage pour entrainer le modèle\n",
        "* Un ensemble de test pour tester le modèle appris\n",
        "\n",
        "Utilisez la méthode [train_test_split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) de scikit-learn pour séparer le dataset de la façon suivante:\n",
        "* ```X_train``` -> les features pour entrainer le modèle\n",
        "* ```y_train``` -> les prix à prédire durant l'apprentissage\n",
        "* ```X_test``` -> les features pour tester le modèle\n",
        "* ```y_test``` -> les prix à prédire pour tester le modèle  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n8rwQT2BYu-H"
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "import numpy as np\n",
        "\n",
        "data = np.asarray(dataset.data, dtype='float32')\n",
        "target = np.asarray(dataset.target.reshape(-1, 1), dtype='float32')\n",
        "\n",
        "X_train, X_test, y_train, y_test = ..."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VLr2_t4nYu-I"
      },
      "source": [
        "Parmi les bonnes pratiques permettant d'améliorer l'apprentissage des réseaux de neurones, il est important de normaliser les données en entrée pour obtenir un moyenne proche de 0 et une variance à 1.  \n",
        "Utilisez la méthode [StandardScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html) de scikit-learn pour normaliser ```X_train``` et ```X_test``` ainsi que ```y_train``` et ```y_test```:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XnGt7KT_Yu-I"
      },
      "source": [
        "from sklearn import preprocessing\n",
        "..."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WnpEWfHXYu-K"
      },
      "source": [
        "# Réseaux de neurones"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "41rRO3dDYu-L"
      },
      "source": [
        "Nous allons construire nos réseaux de neurones comme des successions de couches.  \n",
        "Nous distinguerons, similairement aux frameworks les plus courants (Tensorflow, Pytorch, Keras ...), deux types de couches:\n",
        "* des couches de poids et gradients\n",
        "* des couches d'activations \n",
        "\n",
        "Chaque couche possèdera:\n",
        "* une méthode ```forward``` qui recevra en entrée un vecteur $x$ auquel sera appliqué une transformation qui servira d'inputs pour les couches suivantes\n",
        "* une méthode ```backward``` qui recevra en entrée des gradients et les retropopages vers les couches précédentes. \n",
        "* un argument ```layer_type``` permettant de définir le type de couche\n",
        "\n",
        "La classe suivante définit la structure d'une couche:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pi1Eef3uYu-L"
      },
      "source": [
        "class Layer:\n",
        "    def __init__(self):\n",
        "        self.layer_type = 'abstract'\n",
        "\n",
        "    def forward(self, inputs):\n",
        "        pass\n",
        "    \n",
        "    def backward(self, grad):\n",
        "        pass"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PV7ZaM-MYu-M"
      },
      "source": [
        "Un réseau de neurones sera donc une classe:\n",
        "* contenant une liste de couches\n",
        "* possédant une méthode ```forward``` (qui fera appel aux méthodes ```forward``` de ses couches) qui pour un vecteur $x$ fournira une prédiction $\\hat{y}$\n",
        "* et une méthode ```backward``` (composition des méthodes ```backward``` de ses couches) qui rétropopagera les gradients depuis sa sortie jusqu'a ses premières couches."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ChW_stmIYu-M"
      },
      "source": [
        "![NeuralNetwork](https://github.com/DavidBert/TP-MAPI3/blob/master/layers.png?raw=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PxfXkUbOYu-N"
      },
      "source": [
        "# Layers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s4-onA9JYu-N"
      },
      "source": [
        "Nous allons maintenant implémenter les couches de notre réseau.  \n",
        "Commençons par les couches \"linéaires\" composées de poids $W$ et de biais $b$.  \n",
        "Ces couches reçoivent en entrée un vecteur $x$ et retournent en sortie le résultat de $xW + b$.\n",
        "\n",
        "Complétez la definition de la classe LinearLayer:\n",
        "* les poids et les biais serons stockés dans un dictionnaire \"params\". Les poids seront une matrice $input \\times output$ initialisée uniformémént entre $[-0.1, 0.1]$ (pensez à la fonction random.uniform de numpy)\n",
        "* implémentez la méthode forward (vous pouvez utiliser la fonction np.matmul pour calculer $xW$), il faudra garder en mémoire le vecteur d'entrées qui sera utilisé lors de l'appel à la méthode backward:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uPfpVTeGYu-N"
      },
      "source": [
        "class Linear(Layer):\n",
        "    def __init__(self, input_size, output_size):\n",
        "        super().__init__()\n",
        "        \n",
        "        self.layer_type = 'linear'\n",
        "        self.params ={}\n",
        "        self.grads = {}\n",
        "        #weights est la matrice [input x output] contenant les poids de la couche \n",
        "        self.params[\"weights\"] = ...\n",
        "        #biais est un vecteur de dimension: (output_size)\n",
        "        self.params[\"bias\"] = ...\n",
        "\n",
        "    def forward(self, inputs):\n",
        "        # outputs = inputs @ weights + biais\n",
        "        #Nous devons garder en mémoire les inputs car ils seront utilisés dans la méthode backward\n",
        "        self.inputs = inputs\n",
        "        return ...\n",
        "    \n",
        "    def backward(self, grad):\n",
        "        # we need to sum gradients over the batch axis\n",
        "        self.grads[\"weights\"] = np.matmul(self.inputs.T, grad)\n",
        "        self.grads[\"bias\"] = np.sum(grad, axis=0)\n",
        "        return np.matmul(grad, self.params[\"weights\"].T)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PPfYWrfdYu-N"
      },
      "source": [
        "Testez votre layer, la cellule suivante ne doit pas renvoyer d'erreur:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B3QVA1fVYu-N"
      },
      "source": [
        "input_size, output_size = 10, 5\n",
        "X = np.random.rand(input_size)\n",
        "grads = np.random.rand(10,5)\n",
        "test_linear_layer = Linear(input_size, output_size)\n",
        "assert test_linear_layer.forward(X).shape == (5,)\n",
        "assert test_linear_layer.backward(grads).shape == (10, 10)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ALle91hrYu-O"
      },
      "source": [
        "## Activation Layers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H-WJvNFCYu-O"
      },
      "source": [
        "Nous allons maintenant implémenter les fonctions d'activation utilisées par les neurones.  \n",
        "Implémentez une fonction d'activation ainsi que sa dérivée (par exemple la fonction ReLU: $f(x) = max(0,x)$:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "snXgjlxvYu-P"
      },
      "source": [
        "def relu(x):\n",
        "    return np.maximum(0,x)\n",
        "\n",
        "def relu_prime(x):\n",
        "    return np.where(x > 0, 1.0, 0.0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s_bVNafDYu-P"
      },
      "source": [
        "Implémentez maintenant les couches d'activations:  \n",
        "Elles seront initialisées à partir d'une fonction $f$ et de sa dérivée"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f7MYfSnQYu-P"
      },
      "source": [
        "class Activation(Layer):\n",
        "    def __init__(self, f, f_prime):\n",
        "        super().__init__()\n",
        "        self.layer_type = 'activation'\n",
        "        self.f = f\n",
        "        self.f_prime = f_prime\n",
        "\n",
        "    def forward(self, inputs):\n",
        "        #Nous devons garder en mémoire les inputs car ils seront utilisés dans la méthode backward\n",
        "        self.intputs = inputs\n",
        "        return ...\n",
        "    \n",
        "    def backward(self, grad):\n",
        "        \"\"\" si y = f(x) et x = g(z)\n",
        "        alors dy/dz = f'(x) * g'(z)\n",
        "        Dans notre cas g'(z) correspond aux gradients en entrée\n",
        "        \"\"\"\n",
        "        return self.f_prime(self.intputs) * grad"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nDUd114uYu-Q"
      },
      "source": [
        "Testez votre couche d'activation.  \n",
        "La cellule suivante ne dois pas renvoyer d'erreur:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6u-pRcbsYu-Q"
      },
      "source": [
        "relu_layer = Activation(f=relu, f_prime=relu_prime)\n",
        "\n",
        "#forward\n",
        "x = np.array([-1, 5, -2, 6])\n",
        "y = relu_layer.forward(x)\n",
        "assert np.array_equal(y, np.array([0, 5, 0, 6]))\n",
        "\n",
        "#backward\n",
        "incomming_grads = np.array([-0.1, 0.6, -0.4, -0.1])\n",
        "grads = relu_layer.backward(incomming_grads)\n",
        "assert np.array_equal(grads, np.array([-0. ,  0.6, -0. , -0.1]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vQcb-PKlYu-Q"
      },
      "source": [
        "## Neural network"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y4B9umGnYu-Q"
      },
      "source": [
        "Nous allons maintenant définir la classe qui sera utilisée pour représenter nos réseaux de neurones.  \n",
        "Un réseau de neurones possédera une liste de couches et deux méthodes: ```forward``` et ```backward```.  \n",
        "\n",
        "Complétez la méthode ```forward``` et la méthode ```backward``` de la classe NeuralNetwork en utilisant les methodes ```forward``` et ```backward``` des couches du réseau:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-7wzQJvCYu-Q"
      },
      "source": [
        "class NeuralNetwork:\n",
        "    def __init__(self, layers):\n",
        "        self.layers = layers\n",
        "\n",
        "    def forward(self, inputs):\n",
        "        ...\n",
        "\n",
        "    def backward(self, grad):\n",
        "        ..."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f4g7NHOwYu-R"
      },
      "source": [
        "Nous avons maintenant tous les éléments constituant un réseau de neurones.  \n",
        "Instanciez un reseau de neurones constitué de 3 couches:\n",
        "* une couche d'entrée recevant un vecteur de dimension 13 et constituée de 40 neurones\n",
        "* une couche cachée de 40 neurones\n",
        "* une couche de sortie constituée d'un seul neurone"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WVW1z3KuYu-R"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rf8NeoNqYu-S"
      },
      "source": [
        "Testez votre réseau sur le jeu de test:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F7cozFj_Yu-S"
      },
      "source": [
        "from sklearn.metrics import mean_absolute_error\n",
        "\n",
        "y_pred = net.forward(X_test)\n",
        "print(mean_absolute_error(y_pred, y_test))\n",
        "    \n",
        "plt.figure(figsize=(20,5))\n",
        "plt.scatter(range(len(y_test)), scaler.inverse_transform(y_test), label='target')\n",
        "plt.scatter(range(len(y_test)), scaler.inverse_transform(y_pred), label='prediction')\n",
        "plt.legend()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IU6qtdYrYu-T"
      },
      "source": [
        "# Loss function"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aNsic9eeYu-T"
      },
      "source": [
        "Les prédictions du réseau sont mauvaises.  \n",
        "C'est normal on ne l'a pas encore entrainé.  \n",
        "Commençons par implémenter la fonction de perte qu'utilisera le réseau de neurones lors de son apprentissage.  \n",
        "Nous sommes ici dans un problème de régression, nous utiliserons l'erreur quadratique moyenne (Mean Squared Error) pour apprendre notre modèle:\n",
        "$$\\sum_{i}^{n}\\frac{(f(x_i) - y_i)^2}{n}$$  \n",
        "Implémentez la fonction de perte ainsi que sa dérivée qui sera utilisée lors de la retro-propagation:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P0K2tjhuYu-T"
      },
      "source": [
        "class MSE():\n",
        "\n",
        "    def loss(self, y_pred, y_true):\n",
        "        #returns a scalar\n",
        "        return ...\n",
        "    \n",
        "    def grad(self, y_pred, y_true):\n",
        "        #returns a tensor of gradients\n",
        "        return ..."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s6olrHzDYu-T"
      },
      "source": [
        "# Optimizer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y0-IGRAFYu-T"
      },
      "source": [
        "Nous allons maintenant implémenter la méthode d'optimisation utilisée durant l'apprentissage.  \n",
        "Nous utiliserons ici la descente de gradient stochastique.  \n",
        "La méthode est simple: à chaque pas nous allons mettre à jour les paramètres du réseau grace à la formule suivante: $w_{t+1} = w_t - \\eta \\nabla$ où $\\eta$ est le learning rate."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SKO67-CTYu-U"
      },
      "source": [
        "class SGD():\n",
        "    def __init__(self, lr=0.001):\n",
        "        self.lr = lr\n",
        "    \n",
        "    def step(self, net):\n",
        "        for layer in net.layers:\n",
        "            if layer.layer_type == 'linear':\n",
        "                for param, grad in zip(layer.params.values(), layer.grads.values()):\n",
        "                    ..."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Cd9qvGwYu-U"
      },
      "source": [
        "# Fit function"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yVkuTtS7Yu-U"
      },
      "source": [
        "Il ne reste plus qu'à implémenter la fonction fit qui réalise une époque d'apprentissage.  \n",
        "La fonction doit:\n",
        "* calculer les predictions du réseau sur le batch d'apprentissage\n",
        "* calculer la loss\n",
        "* calculer le gradient de la loss\n",
        "* retropropager les gradients\n",
        "* realiser un pas d'optimisation\n",
        "* retourner la perte "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0wpRP4NNYu-V"
      },
      "source": [
        "def fit(net, loss, optimizer, X, y):\n",
        "    ...\n",
        "    return prediction_loss"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XYabi61yYu-V"
      },
      "source": [
        "On peut maintenant entrainer notre réseau:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PFFzqwBQYu-V"
      },
      "source": [
        "%%capture\n",
        "import matplotlib \n",
        "matplotlib.use('Agg')\n",
        "\n",
        "optimizer = SGD(lr=0.01)\n",
        "loss = MSE()\n",
        "k=0\n",
        "for i, (x, y) in enumerate(zip(X_train, y_train)):\n",
        "    training_loss = fit(net, loss, optimizer, np.array([x]), np.array([y]))\n",
        "    y_pred = net.forward(X_test)\n",
        "    fig = plt.figure(figsize=(20,5));\n",
        "    # cette boucle sert à visualiser la precision au cours des batchs\n",
        "    if i%5 == 0:\n",
        "      plt.scatter(range(len(y_test)), scaler.inverse_transform(y_test), label='target');\n",
        "      plt.scatter(range(len(y_test)), scaler.inverse_transform(y_pred), label='prediction');\n",
        "      plt.legend();\n",
        "      plt.savefig(f'test{100 + k}.png');\n",
        "      plt.close(fig);\n",
        "      k+=1\n",
        "print(mean_absolute_error(y_pred, y_test))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oQZmOqWjuAlS"
      },
      "source": [
        "print('mean absolute error: ', mean_absolute_error(y_pred, y_test))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4p0aEDiZuJo9"
      },
      "source": [
        "%%capture\r\n",
        "!apt-get update\r\n",
        "!apt-get install imagemagick\r\n",
        "!convert -delay 40 *.png animated_chart.gif"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MZTePzDnuM9y"
      },
      "source": [
        "from IPython.display import Image\r\n",
        "Image(open('animated_chart.gif','rb').read())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-xN-7Tf4uG0y"
      },
      "source": [
        "Les cellules suivantes montrent la progression de l' entrainement durant le batch:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aV7SytaiYu-V"
      },
      "source": [
        "Nous venons de réaliser un entrainement sur une époque.  \r\n",
        "Entrainez votre réseau sur 10 époques et comparez les resultats:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kDwCPp-YYu-V"
      },
      "source": [
        "for epoch in range(10):\n",
        "  ...\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "print(mean_absolute_error(y_pred, y_test))\n",
        "    \n",
        "plt.figure(figsize=(20,5))\n",
        "plt.scatter(range(len(y_test)), scaler.inverse_transform(y_test), label='target')\n",
        "plt.scatter(range(len(y_test)), scaler.inverse_transform(y_pred), label='prediction')\n",
        "plt.legend()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TuVz4NXAYu-V"
      },
      "source": [
        "# Digits"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ECKsiD9AYu-W"
      },
      "source": [
        "Essayons maintenant notre classe ```NeuralNetwork``` sur un problème de classification.  \n",
        "Nous allons chercher à classifier des chiffres manuscrits à partir des pixels.  \n",
        "Nous utiliserons pour cela le dataset [digits de scikit-learn](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x6duZymXYu-W"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "from sklearn.datasets import load_digits\n",
        "\n",
        "dataset = load_digits()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eO6FlSYiYu-X"
      },
      "source": [
        "Affichez les 10 premiers éléments du dataset:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IqghpjbfYu-X"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qhs6rfb9Yu-Y"
      },
      "source": [
        "Similairement au dataset précédent, séparez votre jeu de données en deux et normalisez les vecteurs d'entrées. (Pas la peine cette fois çi de normaliser les $y$)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HYx9BLLCYu-Y"
      },
      "source": [
        "from sklearn import preprocessing\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "data = np.asarray(dataset.data, dtype='float32')\n",
        "target = np.asarray(dataset.target, dtype='int32')\n",
        "\n",
        "X_train, X_test, y_train, y_test = ...\n",
        "\n",
        "...\n",
        "X_train = ...\n",
        "X_test = ..."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wI_h6KnSYu-Y"
      },
      "source": [
        "# Cross entropy"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A_5bOoQGj7pj"
      },
      "source": [
        "Nous sommes cette fois-ci en présence d'un problème de classification.  \r\n",
        "La fonction de perte MSE n'est plus appropriée pour notre problème.  \r\n",
        "Nous allons utiliser ici la [cross-entropy.](https://en.wikipedia.org/wiki/Cross_entropy#Cross-entropy_loss_function_and_logistic_regression) comme fonction de perte:  \r\n",
        "En considerant que notre réseau produit en sortie un vecteur $\\hat{y}$ de dimension $C$.  \r\n",
        "Où $C$ correspond au nombre de neurones de la couche de sortie (chaque neurone correspond à une classe).  \r\n",
        "Alors la cross-entropy entre le vecteur prédit par le réseau et la véritable classe notée $c$ peut s'écrire:\r\n",
        "\r\n",
        "\r\n",
        "$$\r\n",
        "\\begin{align}\r\n",
        " L(\\hat{y}, c) &= -log(\\frac{e^{\\hat{y_c}}}{\\sum_{i}^{C}e^\\hat{y_c}_k}) \\\\ \r\n",
        " &= - \\hat{y_c} + log(\\sum_{i}^{C}e^\\hat{y_c}_k)\r\n",
        " \\end{align}\r\n",
        " $$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7YtvoQXJYu-Z"
      },
      "source": [
        "def softmax(x):\n",
        "    return np.exp(x) / np.exp(x).sum(axis=-1,keepdims=True)\n",
        "\n",
        "class CrossEntropy():\n",
        "\n",
        "    def loss(self, logits, y_true):\n",
        "        logits_for_answers = logits[np.arange(len(logits)), y_true]\n",
        "        xentropy = - logits_for_answers + np.log(np.sum(np.exp(logits),axis=-1))\n",
        "        return xentropy\n",
        "\n",
        "\n",
        "\n",
        "    def grad(self, logits, y_true):\n",
        "        ones_for_answers = np.zeros_like(logits)\n",
        "        ones_for_answers[np.arange(len(logits)), y_true] = 1\n",
        "        y_softmax = softmax(logits)\n",
        "        return (- ones_for_answers + y_softmax) / logits.shape[0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U8xZnIx-Yu-Z"
      },
      "source": [
        "Instanciez un reseau de neurones capable de prédire la classe d'une image:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RFI-CK_nYu-Z"
      },
      "source": [
        "...\n",
        "net = ...\n",
        "optimizer = SGD(lr=0.05)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5fW0mhhEYu-Z"
      },
      "source": [
        "Calculez la précision de votre réseau sur le jeu de test. (Vous pouvez pour cela utiliser la fonction [acuracy_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html) de scikit-learn)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PrdGFeELYu-Z"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pssRXNVFYu-Z"
      },
      "source": [
        "La fonction suivante permet de visualiser nos prédictions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TBl1SMEwYu-a"
      },
      "source": [
        "def plot_prediction(net, sample_idx=range(3), classes=range(10)):\n",
        "    \n",
        "    for idx in sample_idx:\n",
        "        plt.figure()\n",
        "        logits = net.forward(X_test[idx])\n",
        "        probas = softmax(logits)\n",
        "        prediction = np.argmax(probas)\n",
        "\n",
        "        fig, (ax0, ax1) = plt.subplots(nrows=1, ncols=2, figsize=(10, 4))\n",
        "\n",
        "        ax0.imshow(scaler.inverse_transform(X_test[idx]).reshape(8, 8), cmap=plt.cm.gray_r,\n",
        "                   interpolation='nearest')\n",
        "        ax0.set_title(\"True image label: %d\" % y_test[idx]);\n",
        "        ax1.bar(classes, np.eye(len(classes))[y_test[idx]], label='true')\n",
        "        ax1.bar(classes, probas, label='prediction', color=\"red\")\n",
        "        ax1.set_xticks(classes)\n",
        "\n",
        "        ax1.set_title('Output probabilities (prediction: %d)'\n",
        "                      % prediction)\n",
        "        ax1.set_xlabel('Digit class')\n",
        "        ax1.legend()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M8bYhAUZYu-b"
      },
      "source": [
        "plot_prediction(net, sample_idx=range(3))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-BJzay6XYu-b"
      },
      "source": [
        "Nous allons cette fois-çi, utiliser des mini-batchs durant notre apprentissage.  \n",
        "La fonction suivante permet d' itérer sur des mini-batchs:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eb6IPkWhYu-b"
      },
      "source": [
        "def iterate_minibatches(inputs, targets, batchsize, shuffle=True):\n",
        "    assert inputs.shape[0] == targets.shape[0]\n",
        "    if shuffle:\n",
        "        indices = np.arange(inputs.shape[0])\n",
        "        np.random.shuffle(indices)\n",
        "    for start_idx in range(0, inputs.shape[0] - batchsize + 1, batchsize):\n",
        "        if shuffle:\n",
        "            excerpt = indices[start_idx:start_idx + batchsize]\n",
        "        else:\n",
        "            excerpt = slice(start_idx, start_idx + batchsize)\n",
        "        yield inputs[excerpt], targets[excerpt]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sUI7wDcFYu-b"
      },
      "source": [
        "Et la fonction suivant de réaliser une époque d'apprentissage à l'aide de mini-batchs:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zcHEdyz6Yu-b"
      },
      "source": [
        "def fit_one_epoch(X, y_true):\n",
        "    for x, y in iterate_minibatches(X, y_true, 32):\n",
        "        training_loss = fit(net, loss, optimizer, x, y)\n",
        "        \n",
        "    return training_loss.mean()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UXVuh04wYu-b"
      },
      "source": [
        "Entrainez votre reseau sur 15 époques.  \n",
        "A chaque époque, calculez votre précision sur le jeu de d'apprentissage et sur le jeu de test et affichez les sur un même graphique une fois l'entrainement terminé."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2RNxTrCGYu-c"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4l_XpelIYu-c"
      },
      "source": [
        "Affichez votre précision sur le jeu de test et visualisez vos prédiction à l'aide de la fonction plot_prediction)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kfFX30iMYu-c"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4AYe1bNcYu-d"
      },
      "source": [
        "Essayez d'afficher votre matrice de confusion en vous inspirant de [ce code:](https://scikit-learn.org/dev/auto_examples/classification/plot_digits_classification.html#sphx-glr-auto-examples-classification-plot-digits-classification-py) "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nmTEOPRYYu-d"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "up4JIwLAYu-d"
      },
      "source": [
        "Affichez les exemples sur lesquels le réseau se trompe.  \n",
        "Auriez-vous donné une meilleure prédiction que le réseau sur ces exemples?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7zlS8-saYu-d"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}