07-Avanzado.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "07-Avanzado.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZeJXqef3dOmc",
        "colab_type": "text"
      },
      "source": [
        "# Autoencoders\n",
        "\n",
        "Un autoencoder son dos redes neuronales, una cuyo objetivo es reducir la representación de las instancias a un espacio dimensional más bajo. Por ejemplo, en el ejemplo de la MNIST existen 784 caractéristicas y se podría desear reducirlas a un número menor. Esto se llama reducción de dimensionalidad y tiene diversas aplicaciones, como entrenar modelos que no se comportan bien cuando hay muchas caractéristicas, eliminar caractéristicas redundantes, o reducir el nivel de ruido. Un ejemplo de utilización de autoencoders puede ser para comprimir imagenes, de hecho se ha probado que son competitivos cuando se comparan con estandares de la industria como JPEG2000 [1]. La arquitectura de un autoencoder es:\n",
        "\n",
        "<img src=\"https://upload.wikimedia.org/wikipedia/commons/2/28/Autoencoder_structure.png\"/>\n",
        "\n",
        "> [Autoencoder](https://en.wikipedia.org/wiki/Autoencoder)\n",
        "\n",
        "\n",
        "\n",
        "[1] Theis, L., Shi, W., Cunningham, A., & Huszár, F. (2017). Lossy image compression with compressive autoencoders. arXiv preprint arXiv:1703.00395.\n",
        "\n",
        "\n",
        "## Ejemplo de autoencoder\n",
        "\n",
        "En este ejemplo, se proyectan las imágeenes  eel conjunto de datos NMIST a un espacio 2D, que permite graficar las intancias en un plano, para posteriormente reconstruir las imágenes.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ELA_u72zblhT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%matplotlib inline\n",
        "import keras\n",
        "from keras.layers import Activation, Dense, Input\n",
        "from keras.layers import Conv2D, Flatten\n",
        "from keras.layers import Reshape, Conv2DTranspose\n",
        "from keras.models import Model\n",
        "from keras import backend as K\n",
        "from keras.datasets import mnist\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from PIL import Image\n",
        "\n",
        "# MNIST dataset\n",
        "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
        "\n",
        "image_size = x_train.shape[1]\n",
        "x_train = np.reshape(x_train, [-1, image_size * image_size])\n",
        "x_test = np.reshape(x_test, [-1, image_size * image_size])\n",
        "x_train = x_train.astype('float32') / 255\n",
        "x_test = x_test.astype('float32') / 255"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GXVfbvcHeEdn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "i = Input((image_size * image_size,))\n",
        "d = Dense(100, activation='relu')(i)\n",
        "d = Dense(2, activation='linear')(d)\n",
        "encoder = Model(i, d, name='Encoder')\n",
        "encoder.summary()\n",
        "\n",
        "d_i = Input((2,))\n",
        "d_d = Dense(100, activation='relu')(d_i)\n",
        "d_d = Dense(image_size * image_size, activation='sigmoid')(d_d)\n",
        "decoder = Model(d_i, d_d, name='Decoders')\n",
        "decoder.summary()\n",
        "\n",
        "autoencoder = Model(i, decoder(encoder(i)), name='Autoencoder')\n",
        "autoencoder.compile(loss='mse', optimizer='nadam')\n",
        "\n",
        "autoencoder.fit(x_train,\n",
        "                x_train,\n",
        "                validation_data=(x_test, x_test),\n",
        "                epochs=30,\n",
        "                batch_size=128, verbose=2)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rORQsurzgs1l",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "img = np.empty((image_size*10, image_size*10))\n",
        "img_pred = np.empty((image_size*10, image_size*10))\n",
        "x_test_pred = autoencoder.predict(x_test)\n",
        "for i in range(10):\n",
        "    for j in range(10):\n",
        "        img[i*image_size:(i+1)*image_size, j*image_size:(j+1)*image_size] = np.reshape(x_test[i*10+j, :], (image_size, image_size))\n",
        "        img_pred[i*image_size:(i+1)*image_size, j*image_size:(j+1)*image_size] = np.reshape(x_test_pred[i*10+j, :], (image_size, image_size))\n",
        "plt.rcParams['figure.figsize'] = [10, 10]\n",
        "plt.imshow(img, cmap='gray')\n",
        "plt.show()\n",
        "plt.imshow(img_pred, cmap='gray')\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_Ps6rSunGk0B",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "emb = encoder.predict(x_test)\n",
        "plt.rcParams['figure.figsize'] = [10, 10]\n",
        "plt.scatter(emb[:, 0], emb[:, 1], c=y_test)\n",
        "plt.colorbar()\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GU-0zuc7JEFU",
        "colab_type": "text"
      },
      "source": [
        "## Denoiser autoencoder\n",
        "\n",
        "El siguiente ejemplo, basado en los [ejemplos de Keras](https://github.com/keras-team/keras/blob/master/examples/mnist_denoising_autoencoder.py), utilizaremos un autoencoder para sacar ruido del MNIST. En el caso del ejemplo, se agregará ruido artificialmente. En particular a cada pixel se le agregará un ruido de media 0.5 y desviación estandard de 0.5. Notese que los pixeles están normalizados a valores entre 0 y 1, por lo que el ruido es significativo.\n",
        "\n",
        "El encoder tiene las siguiente arquitectura:\n",
        "\n",
        "1. Entrada de 28 x 28 x 1\n",
        "1.  Convolucional de 32 filtros y kernel de 3x3\n",
        "1.  Convolucional de 64 filtros y kernel de 3x3\n",
        "1. Capa de aplanado. Cada imagen resulta en vectores de 3136 elementos\n",
        "1. Densa con 16 neuronas\n",
        "\n",
        "\n",
        "Es decir, al final del encoder cada imagen queda representada por un vector de 16 caractéristicas en lugar de 784 pixeles.\n",
        "\n",
        "El decoder, quien es el encargado de regenerar la imagen tiene la siguiente arquitectura:\n",
        "\n",
        "1. Entrada de 16\n",
        "1. Una capa densa con 3136 salidas\n",
        "1. Deconvolución de 64 filtros\n",
        "1. Deconvolución de 32 filtros\n",
        "1. Deconvolución de 1 filtro. Reconstruendo la imagen original.\n",
        "\n",
        "\n",
        "Las deconvoluciones son operaciones que permiten reconstruir imagenes a las que se le aplicaron filtros convolucionales. Ver: [Deconvolutional Networks](https://www.matthewzeiler.com/mattzeiler/deconvolutionalnetworks.pdf).\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3S0A5NuNJOLz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "(x_train, _), (x_test, _) = mnist.load_data()\n",
        "\n",
        "image_size = x_train.shape[1]\n",
        "x_train = np.reshape(x_train, [-1, image_size, image_size, 1])\n",
        "x_test = np.reshape(x_test, [-1, image_size, image_size, 1])\n",
        "x_train = x_train.astype('float32') / 255\n",
        "x_test = x_test.astype('float32') / 255\n",
        "\n",
        "# Generate corrupted MNIST images by adding noise with normal dist\n",
        "# centered at 0.5 and std=0.5\n",
        "noise = np.random.normal(loc=0.5, scale=0.5, size=x_train.shape)\n",
        "x_train_noisy = x_train + noise\n",
        "noise = np.random.normal(loc=0.5, scale=0.5, size=x_test.shape)\n",
        "x_test_noisy = x_test + noise\n",
        "\n",
        "x_train_noisy = np.clip(x_train_noisy, 0., 1.)\n",
        "x_test_noisy = np.clip(x_test_noisy, 0., 1.)\n",
        "\n",
        "# Network parameters\n",
        "input_shape = (image_size, image_size, 1)\n",
        "batch_size = 128\n",
        "kernel_size = 3\n",
        "latent_dim = 16\n",
        "# Encoder/Decoder number of CNN layers and filters per layer\n",
        "layer_filters = [32, 64]\n",
        "\n",
        "# Build the Autoencoder Model\n",
        "# First build the Encoder Model\n",
        "inputs = Input(shape=input_shape, name='encoder_input')\n",
        "x = inputs\n",
        "# Stack of Conv2D blocks\n",
        "# Notes:\n",
        "# 1) Use Batch Normalization before ReLU on deep networks\n",
        "# 2) Use MaxPooling2D as alternative to strides>1\n",
        "# - faster but not as good as strides>1\n",
        "for filters in layer_filters:\n",
        "    x = Conv2D(filters=filters,\n",
        "               kernel_size=kernel_size,\n",
        "               strides=2,\n",
        "               activation='relu',\n",
        "               padding='same')(x)\n",
        "\n",
        "# Shape info needed to build Decoder Model\n",
        "shape = K.int_shape(x)\n",
        "\n",
        "# Generate the latent vector\n",
        "x = Flatten()(x)\n",
        "latent = Dense(latent_dim, name='latent_vector')(x)\n",
        "\n",
        "# Instantiate Encoder Model\n",
        "encoder = Model(inputs, latent, name='encoder')\n",
        "print('Encoder')\n",
        "encoder.summary()\n",
        "\n",
        "# Build the Decoder Model\n",
        "latent_inputs = Input(shape=(latent_dim,), name='decoder_input')\n",
        "x = Dense(shape[1] * shape[2] * shape[3])(latent_inputs)\n",
        "x = Reshape((shape[1], shape[2], shape[3]))(x)\n",
        "\n",
        "# Stack of Transposed Conv2D blocks\n",
        "# Notes:\n",
        "# 1) Use Batch Normalization before ReLU on deep networks\n",
        "# 2) Use UpSampling2D as alternative to strides>1\n",
        "# - faster but not as good as strides>1\n",
        "for filters in layer_filters[::-1]:\n",
        "    x = Conv2DTranspose(filters=filters,\n",
        "                        kernel_size=kernel_size,\n",
        "                        strides=2,\n",
        "                        activation='relu',\n",
        "                        padding='same')(x)\n",
        "\n",
        "x = Conv2DTranspose(filters=1,\n",
        "                    kernel_size=kernel_size,\n",
        "                    padding='same')(x)\n",
        "\n",
        "outputs = Activation('sigmoid', name='decoder_output')(x)\n",
        "\n",
        "# Instantiate Decoder Model\n",
        "decoder = Model(latent_inputs, outputs, name='decoder')\n",
        "print('Decoder')\n",
        "decoder.summary()\n",
        "\n",
        "# Autoencoder = Encoder + Decoder\n",
        "# Instantiate Autoencoder Model\n",
        "print('Encoder-decoder apliado para entrenamiento')\n",
        "autoencoder = Model(inputs, decoder(encoder(inputs)), name='autoencoder')\n",
        "autoencoder.summary()\n",
        "\n",
        "autoencoder.compile(loss='mse', optimizer='adam')\n",
        "\n",
        "# Train the autoencoder\n",
        "autoencoder.fit(x_train_noisy,\n",
        "                x_train,\n",
        "                validation_data=(x_test_noisy, x_test),\n",
        "                epochs=10,\n",
        "                batch_size=batch_size)\n",
        "\n",
        "# Predict the Autoencoder output from corrupted test images\n",
        "x_decoded = autoencoder.predict(x_test_noisy)\n",
        "\n",
        "# Display the 1st 8 corrupted and denoised images\n",
        "rows, cols = 10, 30\n",
        "num = rows * cols\n",
        "imgs = np.concatenate([x_test[:num], x_test_noisy[:num], x_decoded[:num]])\n",
        "imgs = imgs.reshape((rows * 3, cols, image_size, image_size))\n",
        "imgs = np.vstack(np.split(imgs, rows, axis=1))\n",
        "imgs = imgs.reshape((rows * 3, -1, image_size, image_size))\n",
        "imgs = np.vstack([np.hstack(i) for i in imgs])\n",
        "imgs = (imgs * 255).astype(np.uint8)\n",
        "plt.rcParams['figure.figsize'] = [25, 25]\n",
        "plt.figure()\n",
        "plt.axis('off')\n",
        "plt.title('Original images: top rows, '\n",
        "          'Corrupted Input: middle rows, '\n",
        "          'Denoised Input:  third rows')\n",
        "plt.imshow(imgs, interpolation='none', cmap='gray')\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tJt8py4LOuuD",
        "colab_type": "text"
      },
      "source": [
        "## Variational Autoencoder\n",
        "Los Variational Autoencoders intentan aprender una representación estádistica de las dimensiones latentes. El encoder retorna la distribución de las dimensiones latentes, retornando su media y desviación estándar. Por su parte, el decoder utiliza un muestreo sobre esta distribución para generar las imágenes.\n",
        "\n",
        "$autoencoder(x)=P(z|x)$\n",
        "\n",
        "$decoder(z)=p(x|z)$\n",
        "\n",
        "La función de perdida utilizada para este tipo de autoencoders es la **Divergencia de Kullback-Leibler**, también conocida como divergencia de información. Es una función de perdida no simetrica que evalúa cuan similares son dos distribuciones de probabilidad.\n",
        "\n",
        "$KL(p||q)=\\sum{p_i \\ln{\\frac{p_i}{q_i}}}$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oTFogKngR2px",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%matplotlib inline\n",
        "from keras.layers import Lambda, Input, Dense\n",
        "from keras.models import Model\n",
        "from keras.datasets import mnist\n",
        "from keras.losses import mse, binary_crossentropy\n",
        "from keras.utils import plot_model\n",
        "from keras import backend as K\n",
        "\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import argparse\n",
        "import os\n",
        "\n",
        "\n",
        "# reparameterization trick\n",
        "# instead of sampling from Q(z|X), sample epsilon = N(0,I)\n",
        "# z = z_mean + sqrt(var) * epsilon\n",
        "def sampling(args):\n",
        "    \"\"\"Reparameterization trick by sampling from an isotropic unit Gaussian.\n",
        "    # Arguments\n",
        "        args (tensor): mean and log of variance of Q(z|X)\n",
        "    # Returns\n",
        "        z (tensor): sampled latent vector\n",
        "    \"\"\"\n",
        "\n",
        "    z_mean, z_log_var = args\n",
        "    batch = K.shape(z_mean)[0]\n",
        "    dim = K.int_shape(z_mean)[1]\n",
        "    # by default, random_normal has mean = 0 and std = 1.0\n",
        "    epsilon = K.random_normal(shape=(batch, dim))\n",
        "    return z_mean + K.exp(0.5 * z_log_var) * epsilon\n",
        "\n",
        "\n",
        "def plot_results(models,\n",
        "                 data,\n",
        "                 batch_size=128):\n",
        "    \"\"\"Plots labels and MNIST digits as a function of the 2D latent vector\n",
        "    # Arguments\n",
        "        models (tuple): encoder and decoder models\n",
        "        data (tuple): test data and label\n",
        "        batch_size (int): prediction batch size\n",
        "        model_name (string): which model is using this function\n",
        "    \"\"\"\n",
        "\n",
        "    encoder, decoder = models\n",
        "    x_test, y_test = data\n",
        "    # display a 2D plot of the digit classes in the latent space\n",
        "    z_mean, z_log_var, _ = encoder.predict(x_test,\n",
        "                                   batch_size=batch_size)\n",
        "    plt.figure(figsize=(12, 10))\n",
        "    plt.scatter(z_mean[:, 0], z_mean[:, 1], c=y_test)\n",
        "    plt.colorbar()\n",
        "    plt.xlabel(\"z[0]\")\n",
        "    plt.ylabel(\"z[1]\")\n",
        "    plt.show()\n",
        "\n",
        "    plt.figure(figsize=(12, 10))\n",
        "    plt.scatter(z_log_var[:, 0], z_log_var[:, 1], c=y_test)\n",
        "    plt.colorbar()\n",
        "    plt.xlabel(\"z[0]\")\n",
        "    plt.ylabel(\"z[1]\")\n",
        "    plt.show()\n",
        "\n",
        "    # display a 30x30 2D manifold of digits\n",
        "    n = 30\n",
        "    digit_size = 28\n",
        "    figure = np.zeros((digit_size * n, digit_size * n))\n",
        "    # linearly spaced coordinates corresponding to the 2D plot\n",
        "    # of digit classes in the latent space\n",
        "    grid_x = np.linspace(-4, 4, n)\n",
        "    grid_y = np.linspace(-4, 4, n)[::-1]\n",
        "\n",
        "    for i, yi in enumerate(grid_y):\n",
        "        for j, xi in enumerate(grid_x):\n",
        "            z_sample = np.array([[xi, yi]])\n",
        "            x_decoded = decoder.predict(z_sample)\n",
        "            digit = x_decoded[0].reshape(digit_size, digit_size)\n",
        "            figure[i * digit_size: (i + 1) * digit_size,\n",
        "                   j * digit_size: (j + 1) * digit_size] = digit\n",
        "\n",
        "    plt.figure(figsize=(10, 10))\n",
        "    start_range = digit_size // 2\n",
        "    end_range = (n - 1) * digit_size + start_range + 1\n",
        "    pixel_range = np.arange(start_range, end_range, digit_size)\n",
        "    sample_range_x = np.round(grid_x, 1)\n",
        "    sample_range_y = np.round(grid_y, 1)\n",
        "    plt.xticks(pixel_range, sample_range_x)\n",
        "    plt.yticks(pixel_range, sample_range_y)\n",
        "    plt.xlabel(\"z[0]\")\n",
        "    plt.ylabel(\"z[1]\")\n",
        "    plt.imshow(figure, cmap='Greys_r')\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "# MNIST dataset\n",
        "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
        "\n",
        "image_size = x_train.shape[1]\n",
        "original_dim = image_size * image_size\n",
        "x_train = np.reshape(x_train, [-1, original_dim])\n",
        "x_test = np.reshape(x_test, [-1, original_dim])\n",
        "x_train = x_train.astype('float32') / 255\n",
        "x_test = x_test.astype('float32') / 255\n",
        "\n",
        "# network parameters\n",
        "input_shape = (original_dim, )\n",
        "intermediate_dim = 512\n",
        "batch_size = 128\n",
        "latent_dim = 2\n",
        "epochs = 50\n",
        "\n",
        "# VAE model = encoder + decoder\n",
        "# build encoder model\n",
        "inputs = Input(shape=input_shape, name='encoder_input')\n",
        "x = Dense(intermediate_dim, activation='relu')(inputs)\n",
        "z_mean = Dense(latent_dim, name='z_mean')(x)\n",
        "z_log_var = Dense(latent_dim, name='z_log_var')(x)\n",
        "\n",
        "# use reparameterization trick to push the sampling out as input\n",
        "# note that \"output_shape\" isn't necessary with the TensorFlow backend\n",
        "z = Lambda(sampling, output_shape=(latent_dim,), name='z')([z_mean, z_log_var])\n",
        "\n",
        "# instantiate encoder model\n",
        "encoder = Model(inputs, [z_mean, z_log_var, z], name='encoder')\n",
        "encoder.summary()\n",
        "plot_model(encoder, to_file='vae_mlp_encoder.png', show_shapes=True)\n",
        "\n",
        "# build decoder model\n",
        "latent_inputs = Input(shape=(latent_dim,), name='z_sampling')\n",
        "x = Dense(intermediate_dim, activation='relu')(latent_inputs)\n",
        "outputs = Dense(original_dim, activation='sigmoid')(x)\n",
        "\n",
        "# instantiate decoder model\n",
        "decoder = Model(latent_inputs, outputs, name='decoder')\n",
        "decoder.summary()\n",
        "plot_model(decoder, to_file='vae_mlp_decoder.png', show_shapes=True)\n",
        "\n",
        "# instantiate VAE model\n",
        "outputs = decoder(encoder(inputs)[2])\n",
        "vae = Model(inputs, outputs, name='vae_mlp')\n",
        "\n",
        "def run():\n",
        "    models = (encoder, decoder)\n",
        "    data = (x_test, y_test)\n",
        "\n",
        "    # VAE loss = mse_loss or xent_loss + kl_loss\n",
        "    #reconstruction_loss = mse(inputs, outputs)\n",
        "    reconstruction_loss = binary_crossentropy(inputs, outputs)\n",
        "\n",
        "    reconstruction_loss *= original_dim\n",
        "    kl_loss = 1 + z_log_var - K.square(z_mean) - K.exp(z_log_var)\n",
        "    kl_loss = K.sum(kl_loss, axis=-1)\n",
        "    kl_loss *= -0.5\n",
        "    vae_loss = K.mean(reconstruction_loss + kl_loss)\n",
        "    vae.add_loss(vae_loss)\n",
        "    vae.compile(optimizer='adam')\n",
        "    vae.summary()\n",
        "    \n",
        "    # train the autoencoder\n",
        "    vae.fit(x_train,\n",
        "            epochs=epochs,\n",
        "            batch_size=batch_size,\n",
        "            validation_data=(x_test, None))\n",
        "    #vae.save_weights('vae_mlp_mnist.h5')\n",
        "\n",
        "    plot_results(models,\n",
        "                 data,\n",
        "                 batch_size=batch_size)\n",
        "    \n",
        "run()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wG6fF0zNrgh7",
        "colab_type": "text"
      },
      "source": [
        "# Transfer learning\n",
        "\n",
        "\n",
        "Trasnfer learning es otra manera de utilizar las técnicas de Deep Learning. Se utiliza en casos donde los datos de entrenamiento son escasos, pero se tiene modelos entrenados para tareas similares. Para ejemplificar, utilizaremos el dataset conocido como [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html).\n",
        "\n",
        "\n",
        "| Propiedad | Valor |\n",
        "| --- | --- |\n",
        "| Clases | 10 |\n",
        "| Tamaño de las imágenes | 32 X 32  |\n",
        "| Canales de las imágenes | 3 (RGB)  |\n",
        "| Instancias de entrenamiento | 50.000 |\n",
        "| Instancias de testeo | 10.000 |\n",
        "| Valor mínimo de cada pixel | 0 |\n",
        "| Valor máximo de cada pixel | 255 |\n",
        "\n",
        "El dataset contiene imágenes en color de 32 X 32 pixeles divididas en 10 clases:\n",
        "1. Avión\n",
        "1. Auto\t\t\t\t\t\t\t\t\t\t\n",
        "1. Pájaro\t\t\t\t\t\t\t\t\t\n",
        "1. Gato\t\t\t\t\t\t\t\n",
        "1. Venado\t\t\t\t\t\t\t\t\t\t\n",
        "1. Perro\t\t\t\t\t\t\n",
        "1. Rana\t\t\t\t\t\t\t\t\t\n",
        "1. Caballo\t\t\t\t\t\t\t\t\t\t\n",
        "1. Barco\t\t\t\t\t\t\t\n",
        "1. Camión\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lWNzf9HxskQs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.datasets import cifar10\n",
        "\n",
        "(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
        "\n",
        "print('100 primeros elementos del conjunto de entrenamiento')\n",
        "f = plt.figure(111)\n",
        "for i in range(10):\n",
        "    for j in range(10):\n",
        "        ax = f.add_subplot(10, 10, i + j*10 + 1)\n",
        "        ax.set_xticklabels('')\n",
        "        ax.set_yticklabels('')\n",
        "        ax.imshow(x_train[i + j*10, :, :], cmap='gray')\n",
        "plt.show()\n",
        "\n",
        "x_train = x_train / 255\n",
        "x_test = x_test / 255"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e6Gg3Xgdsw0T",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.layers import Conv2D, Flatten\n",
        "from keras.utils import to_categorical\n",
        "from sklearn.metrics import accuracy_score as acc, confusion_matrix\n",
        "\n",
        "def show_confusion_matrix_nl(cm):\n",
        "    fig = plt.figure()\n",
        "    ax = fig.add_subplot(111)\n",
        "    cax = ax.matshow(cm)\n",
        "    plt.title('Matriz de confusión')\n",
        "    fig.colorbar(cax)\n",
        "    plt.xlabel('Verdadero')\n",
        "    plt.ylabel('Predicho')\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "i = Input(shape=(32, 32, 3))\n",
        "d = Conv2D(5, (5,5), activation='relu')(i)\n",
        "d = Conv2D(5, (5,5), activation='relu')(d)\n",
        "d = Conv2D(5, (5,5), activation='relu')(d)\n",
        "d = Conv2D(10, (5,5), activation='relu')(d)\n",
        "d = Flatten()(d)\n",
        "d = Dense(10, activation='softmax')(d)\n",
        "model = Model(inputs=i, outputs=d)\n",
        "model.summary()\n",
        "model.compile(loss='categorical_crossentropy', optimizer='nadam', metrics=['categorical_accuracy'])\n",
        "\n",
        "predict = lambda x: np.argmax(model.predict(x), axis=-1)\n",
        "show_confusion_matrix_nl(confusion_matrix(y_test, predict(x_test)))\n",
        "print('La accuracy antes de entrenar es {}'.format(acc(y_test, predict(x_test))))\n",
        "\n",
        "h = model.fit(x_train, to_categorical(y_train), epochs=10, batch_size=100, \n",
        "              validation_data=(x_test, to_categorical(y_test)), verbose=0)\n",
        "\n",
        "\n",
        "print('Función de pérdidad:')\n",
        "plt.plot(h.history['loss'], 'b-', h.history['val_loss'], 'r-')\n",
        "plt.show()\n",
        "print('Accuracy:')\n",
        "plt.plot(h.history['categorical_accuracy'], 'b-', h.history['val_categorical_accuracy'], 'r-')\n",
        "plt.show()\n",
        "\n",
        "show_confusion_matrix_nl(confusion_matrix(y_test, predict(x_test)))\n",
        "print('La accuracy después de entrenar es {}'.format(acc(y_test, predict(x_test))))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TzDdFEgJs-CQ",
        "colab_type": "text"
      },
      "source": [
        "Supongamos que tenemos solo una porción de datos para entrenar, por ejemplo 2000 imágenes (200 por cada clases). ¿Sería posible entrenar la red neuronal?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9x52WfBwtfdW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sample_per_class = 200\n",
        "\n",
        "x_small = np.empty((sample_per_class * 10, 32, 32, 3))\n",
        "y_small = np.empty((sample_per_class * 10,))\n",
        "\n",
        "\n",
        "counter = [0] * 10\n",
        "\n",
        "i = 0\n",
        "for x, y in zip(x_train, y_train):\n",
        "    if counter[y[0]] == sample_per_class:\n",
        "      continue\n",
        "    counter[y[0]] += 1\n",
        "    x_small[i, :, :, :] = x\n",
        "    y_small[i] = y\n",
        "    i += 1\n",
        "    if i == sample_per_class * 10: \n",
        "        break"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GzJObRxatyrx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "i = Input(shape=(32, 32, 3))\n",
        "d = Conv2D(5, (5,5), activation='relu')(i)\n",
        "d = Conv2D(5, (5,5), activation='relu')(d)\n",
        "d = Conv2D(5, (5,5), activation='relu')(d)\n",
        "d = Conv2D(10, (5,5), activation='relu')(d)\n",
        "d = Flatten()(d)\n",
        "d = Dense(10, activation='softmax')(d)\n",
        "model = Model(inputs=i, outputs=d)\n",
        "model.summary()\n",
        "model.compile(loss='categorical_crossentropy', optimizer='nadam', metrics=['categorical_accuracy'])\n",
        "\n",
        "show_confusion_matrix_nl(confusion_matrix(y_test, predict(x_test)))\n",
        "print('La accuracy antes de entrenar es {}'.format(acc(y_test, predict(x_test))))\n",
        "\n",
        "h = model.fit(x_small, to_categorical(y_small), epochs=10, batch_size=100, \n",
        "              validation_data=(x_test, to_categorical(y_test)), verbose=0)\n",
        "\n",
        "\n",
        "print('Función de pérdidad:')\n",
        "plt.plot(h.history['loss'], 'b-', h.history['val_loss'], 'r-')\n",
        "plt.show()\n",
        "print('Accuracy:')\n",
        "plt.plot(h.history['categorical_accuracy'], 'b-', h.history['val_categorical_accuracy'], 'r-')\n",
        "plt.show()\n",
        "\n",
        "show_confusion_matrix_nl(confusion_matrix(y_test, predict(x_test)))\n",
        "print('La accuracy después de entrenar es {}'.format(acc(y_test, predict(x_test))))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xJPqG4V8t5QQ",
        "colab_type": "text"
      },
      "source": [
        "En el gráfico de accuracy podemos ver que la red aprende muy bien a identificar los ejemplos de entrenamiento. Llega a un accuracy del $40\\%$, pero cuando hacemos la evaluación con el conjunto de test, el valor es del $30\\%$. Este fenómeno se conoce como *overfitting* y es un problema importante cuando se usa este tipo de técnicas con pocos datos.\n",
        "\n",
        "Para este tipo de problemas se utiliza el *transfer learning*. Para esto, se debe considerad alguna red neuronal arbitraría entrenada para clasificar imágenes con un dataset grande. Hay muchas disponibles públicamente. Keras provee varias [redes preentrenadas](https://keras.io/applications/) con el dataset de [ImageNet](http://www.image-net.org/), más de 14 millones de imágenes con 1000 clases. Por ser una arquictura simple, podemos tomar VGG16 que tiene más de **138 millones de parámetros**. A continuación, se puede observar la arquitectura de la red."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c_zT9jXmt9dt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.applications.vgg16 import VGG16\n",
        "model = VGG16(include_top=True)\n",
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ALUUtgx7uDA-",
        "colab_type": "text"
      },
      "source": [
        "Si consideramos que las capas ocultas aprenden las características de las imágenes, podemos separar la red en dos partes:\n",
        "\n",
        "1. Desde la capa de `Input` hasta la capa `block5_pool` como un extractor de características.\n",
        "2. Las capas `fc1` y `fc2` como un clasificador. \n",
        "\n",
        "Si nos quedamos con la primera parte podemos tener un extractor de características para imágenes genéricas:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wQSfVfvGuDv5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#El modelo es pesado y no queremos que se rompa por falta de memoria en la GPU\n",
        "del model \n",
        "#Ahora si, sin el tope!!\n",
        "model = VGG16(include_top=False)\n",
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xSCutVEKuHM5",
        "colab_type": "text"
      },
      "source": [
        "Por comparación vamos a crear 2 dataset nuevos:\n",
        "\n",
        "\n",
        "1. **x_small_t** y **x_test_t**: dataset small transformado con el modelo VGG16.\n",
        "2. **x_small_f** y **x_test_f**: dataset small con forma cambiada para que cada pixel de la imagen sea un valor en un vector.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q-5DLIwJuKn_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Dataset de transfer learning\n",
        "x_small_t = model.predict(x_small)\n",
        "#Esto hace las veces de flatten\n",
        "x_small_t = np.reshape(x_small_t, (x_small.shape[0], 512))\n",
        "print('Forma del dataset transformado con VGG16 {}'.format(x_small_t.shape))\n",
        "#Test set\n",
        "x_test_t = model.predict(x_test)\n",
        "x_test_t = np.reshape(x_test_t, (x_test.shape[0], 512))\n",
        "\n",
        "\n",
        "#Dataset de imagenes\n",
        "x_small_f = np.reshape(x_small, (x_small.shape[0], 32 * 32 * 3))\n",
        "print('Forma del dataset original {}'.format(x_small_f.shape))\n",
        "x_test_f = np.reshape(x_test, (x_test.shape[0], 32 * 32 * 3))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DuflBbvruNAz",
        "colab_type": "text"
      },
      "source": [
        "Podemos probar los dos tipos de características con una regresión logística:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uJyfH0TruQf1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.svm import LinearSVC, SVC\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "#Los parámetros son para evitar warnings, estandades hasta la versión 0.22\n",
        "mt = LogisticRegression(solver='liblinear', multi_class='ovr')\n",
        "mf = LogisticRegression(solver='liblinear', multi_class='ovr')\n",
        "\n",
        "print('Entrenando Transfer')\n",
        "mt.fit(x_small_t, y_small)\n",
        "print('Entrenando Full')\n",
        "mf.fit(x_small_f, y_small)\n",
        "\n",
        "print('Accuracy: {}'.format(acc(y_test, mt.predict(x_test_t))))\n",
        "print('Accuracy: {}'.format(acc(y_test, mf.predict(x_test_f))))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VNpZrCokuT6E",
        "colab_type": "text"
      },
      "source": [
        "Podemos observar que las características transferidas tienen una mejor performance que usar los pixeles de forma cruda.\n",
        "\n",
        "## Fine Tuning\n",
        "\n",
        "Otro uso de las redes preentrenadas para extraer características es incorporarlas en otras redes neuronales para acelerar su entrenamiento. Por ejemplo, en el siguiente caso se utiliza la VGG16 como una capa inicial en una red neuronal. Para que esto funcione, es necesario que las modificaciones en los pesos más sutil que cuando se entrena una red de cero, ya que se supone que la mayoría de los pesos ya están cerca de un valor óptimo. En consecuencia, podemos cambiar el **learning rate** del optimizador, en este caso **Stocastic Gradiant Descent**, de $0.01$ a $0.001$, es decir un orden de magnitud menor."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hq0dO5HCuXrA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.optimizers import SGD\n",
        "\n",
        "\n",
        "i = Input((32, 32, 3))\n",
        "model = VGG16(include_top=False)(i)\n",
        "\n",
        "d = Flatten()(model)\n",
        "d = Dense(512, activation='relu')(d)\n",
        "d = Dense(10, activation='softmax')(d)\n",
        "model = Model(inputs=i, outputs=d)\n",
        "model.summary()\n",
        "model.compile(loss='categorical_crossentropy', optimizer= \\\n",
        "              SGD(lr=1e-3, momentum=0.0, decay=0.0, nesterov=False), \\\n",
        "              metrics=['categorical_accuracy'])#1e-4:ok y 30 epocs\n",
        "\n",
        "show_confusion_matrix_nl(confusion_matrix(y_test, predict(x_test)))\n",
        "print('La accuracy antes de entrenar es {}'.format(acc(y_test, predict(x_test))))\n",
        "\n",
        "h = model.fit(x_train, to_categorical(y_train), epochs=4, batch_size=100, \n",
        "              validation_data=(x_test, to_categorical(y_test)), verbose=1)\n",
        "\n",
        "\n",
        "print('Función de pérdidad:')\n",
        "plt.plot(h.history['loss'], 'b-', h.history['val_loss'], 'r-')\n",
        "plt.show()\n",
        "print('Accuracy:')\n",
        "plt.plot(h.history['categorical_accuracy'], 'b-', h.history['val_categorical_accuracy'], 'r-')\n",
        "plt.show()\n",
        "\n",
        "show_confusion_matrix_nl(confusion_matrix(y_test, predict(x_test)))\n",
        "print('La accuracy después de entrenar es {}'.format(acc(y_test, predict(x_test))))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tADHmxUcwfMG",
        "colab_type": "text"
      },
      "source": [
        "# GAN\n",
        "\n",
        "Generative Adeversarial Networks es una técnica para generar nuevas instancias a partir de dos redes neuronales que compiten entre ellas:\n",
        "\n",
        "* El generador: es la red neuronal encargadas de generar instancias falsas.\n",
        "* El discriminador: es la red neuronal encargada de decidir si una instancia es falsa o verdadera.\n",
        "\n",
        "Para el entrenamiento, se realizan pasadas en batch. Por un lado, al discriminador se lo alimenta con mitad de datos reales y mitad de datos falso, y como objetivo se espera que clasifique los reales como reales y los falsos como falso. En una segunda pasada, se fijan los pesos del discriminador, se conecta el generador con el discriminador, y como objetivo se fija que determine que todos los datos salidos del discriminador son verdaderos.\n",
        "\n",
        "A continuación, se presenta un ejemplo basado en ek [AC-GAN](https://github.com/keras-team/keras/blob/master/examples/mnist_acgan.py) implementado en los ejemplos de Keras."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YR3mLgTqwfdF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from collections import defaultdict\n",
        "try:\n",
        "    import cPickle as pickle\n",
        "except ImportError:\n",
        "    import pickle\n",
        "from PIL import Image\n",
        "\n",
        "from six.moves import range\n",
        "\n",
        "from keras.datasets import mnist\n",
        "from keras import layers\n",
        "from keras.layers import Input, Dense, Reshape, Flatten, Embedding, Dropout\n",
        "from keras.layers import BatchNormalization\n",
        "from keras.layers.advanced_activations import LeakyReLU\n",
        "from keras.layers.convolutional import Conv2DTranspose, Conv2D\n",
        "from keras.models import Sequential, Model\n",
        "from keras.optimizers import Adam\n",
        "from keras.utils.generic_utils import Progbar\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "\n",
        "np.random.seed(1337)\n",
        "num_classes = 10\n",
        "\n",
        "\n",
        "def build_generator(latent_size):\n",
        "    # we will map a pair of (z, L), where z is a latent vector and L is a\n",
        "    # label drawn from P_c, to image space (..., 28, 28, 1)\n",
        "    cnn = Sequential()\n",
        "\n",
        "    cnn.add(Dense(3 * 3 * 384, input_dim=latent_size, activation='relu'))\n",
        "    cnn.add(Reshape((3, 3, 384)))\n",
        "\n",
        "    # upsample to (7, 7, ...)\n",
        "    cnn.add(Conv2DTranspose(192, 5, strides=1, padding='valid',\n",
        "                            activation='relu',\n",
        "                            kernel_initializer='glorot_normal'))\n",
        "    cnn.add(BatchNormalization())\n",
        "\n",
        "    # upsample to (14, 14, ...)\n",
        "    cnn.add(Conv2DTranspose(96, 5, strides=2, padding='same',\n",
        "                            activation='relu',\n",
        "                            kernel_initializer='glorot_normal'))\n",
        "    cnn.add(BatchNormalization())\n",
        "\n",
        "    # upsample to (28, 28, ...)\n",
        "    cnn.add(Conv2DTranspose(1, 5, strides=2, padding='same',\n",
        "                            activation='tanh',\n",
        "                            kernel_initializer='glorot_normal'))\n",
        "\n",
        "    # this is the z space commonly referred to in GAN papers\n",
        "    latent = Input(shape=(latent_size, ))\n",
        "\n",
        "    # this will be our label\n",
        "    image_class = Input(shape=(1,), dtype='int32')\n",
        "\n",
        "    cls = Embedding(num_classes, latent_size,\n",
        "                    embeddings_initializer='glorot_normal')(image_class)\n",
        "\n",
        "    # hadamard product between z-space and a class conditional embedding\n",
        "    h = layers.multiply([latent, cls])\n",
        "\n",
        "    fake_image = cnn(h)\n",
        "\n",
        "    return Model([latent, image_class], fake_image)\n",
        "\n",
        "\n",
        "def build_discriminator():\n",
        "    # build a relatively standard conv net, with LeakyReLUs as suggested in\n",
        "    # the reference paper\n",
        "    cnn = Sequential()\n",
        "\n",
        "    cnn.add(Conv2D(32, 3, padding='same', strides=2,\n",
        "                   input_shape=(28, 28, 1)))\n",
        "    cnn.add(LeakyReLU(0.2))\n",
        "    cnn.add(Dropout(0.3))\n",
        "\n",
        "    cnn.add(Conv2D(64, 3, padding='same', strides=1))\n",
        "    cnn.add(LeakyReLU(0.2))\n",
        "    cnn.add(Dropout(0.3))\n",
        "\n",
        "    cnn.add(Conv2D(128, 3, padding='same', strides=2))\n",
        "    cnn.add(LeakyReLU(0.2))\n",
        "    cnn.add(Dropout(0.3))\n",
        "\n",
        "    cnn.add(Conv2D(256, 3, padding='same', strides=1))\n",
        "    cnn.add(LeakyReLU(0.2))\n",
        "    cnn.add(Dropout(0.3))\n",
        "\n",
        "    cnn.add(Flatten())\n",
        "\n",
        "    image = Input(shape=(28, 28, 1))\n",
        "\n",
        "    features = cnn(image)\n",
        "\n",
        "    # first output (name=generation) is whether or not the discriminator\n",
        "    # thinks the image that is being shown is fake, and the second output\n",
        "    # (name=auxiliary) is the class that the discriminator thinks the image\n",
        "    # belongs to.\n",
        "    fake = Dense(1, activation='sigmoid', name='generation')(features)\n",
        "    aux = Dense(num_classes, activation='softmax', name='auxiliary')(features)\n",
        "\n",
        "    return Model(image, [fake, aux])\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EelLcPI9wkly",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# batch and latent size taken from the paper\n",
        "epochs = 1\n",
        "batch_size = 100\n",
        "latent_size = 100\n",
        "# Adam parameters suggested in https://arxiv.org/abs/1511.06434\n",
        "adam_lr = 0.0002\n",
        "adam_beta_1 = 0.5\n",
        "\n",
        "# build the discriminator\n",
        "print('Discriminator model:')\n",
        "discriminator = build_discriminator()\n",
        "discriminator.compile(\n",
        "    optimizer=Adam(lr=adam_lr, beta_1=adam_beta_1),\n",
        "    loss=['binary_crossentropy', 'sparse_categorical_crossentropy']\n",
        ")\n",
        "discriminator.summary()\n",
        "\n",
        "# build the generator\n",
        "generator = build_generator(latent_size)\n",
        "\n",
        "latent = Input(shape=(latent_size, ))\n",
        "image_class = Input(shape=(1,), dtype='int32')\n",
        "\n",
        "# get a fake image\n",
        "fake = generator([latent, image_class])\n",
        "\n",
        "# we only want to be able to train generation for the combined model\n",
        "discriminator.trainable = False\n",
        "fake, aux = discriminator(fake)\n",
        "combined = Model([latent, image_class], [fake, aux])\n",
        "\n",
        "print('Combined model:')\n",
        "combined.compile(\n",
        "    optimizer=Adam(lr=adam_lr, beta_1=adam_beta_1),\n",
        "    loss=['binary_crossentropy', 'sparse_categorical_crossentropy']\n",
        ")\n",
        "combined.summary()\n",
        "\n",
        "# get our mnist data, and force it to be of shape (..., 28, 28, 1) with\n",
        "# range [-1, 1]\n",
        "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
        "x_train = (x_train.astype(np.float32) - 127.5) / 127.5\n",
        "x_train = np.expand_dims(x_train, axis=-1)\n",
        "\n",
        "x_test = (x_test.astype(np.float32) - 127.5) / 127.5\n",
        "x_test = np.expand_dims(x_test, axis=-1)\n",
        "\n",
        "num_train, num_test = x_train.shape[0], x_test.shape[0]\n",
        "\n",
        "train_history = defaultdict(list)\n",
        "test_history = defaultdict(list)\n",
        "\n",
        "for epoch in range(1, epochs + 1):\n",
        "    print('Epoch {}/{}'.format(epoch, epochs))\n",
        "\n",
        "    num_batches = int(np.ceil(x_train.shape[0] / float(batch_size)))\n",
        "    progress_bar = Progbar(target=num_batches)\n",
        "\n",
        "    epoch_gen_loss = []\n",
        "    epoch_disc_loss = []\n",
        "\n",
        "    for index in range(num_batches):\n",
        "        # get a batch of real images\n",
        "        image_batch = x_train[index * batch_size:(index + 1) * batch_size]\n",
        "        label_batch = y_train[index * batch_size:(index + 1) * batch_size]\n",
        "\n",
        "        # generate a new batch of noise\n",
        "        noise = np.random.uniform(-1, 1, (len(image_batch), latent_size))\n",
        "\n",
        "        # sample some labels from p_c\n",
        "        sampled_labels = np.random.randint(0, num_classes, len(image_batch))\n",
        "\n",
        "        # generate a batch of fake images, using the generated labels as a\n",
        "        # conditioner. We reshape the sampled labels to be\n",
        "        # (len(image_batch), 1) so that we can feed them into the embedding\n",
        "        # layer as a length one sequence\n",
        "        generated_images = generator.predict(\n",
        "            [noise, sampled_labels.reshape((-1, 1))], verbose=0)\n",
        "\n",
        "        x = np.concatenate((image_batch, generated_images))\n",
        "\n",
        "        # use one-sided soft real/fake labels\n",
        "        # Salimans et al., 2016\n",
        "        # https://arxiv.org/pdf/1606.03498.pdf (Section 3.4)\n",
        "        soft_zero, soft_one = 0, 0.95\n",
        "        y = np.array(\n",
        "            [soft_one] * len(image_batch) + [soft_zero] * len(image_batch))\n",
        "        aux_y = np.concatenate((label_batch, sampled_labels), axis=0)\n",
        "\n",
        "        # we don't want the discriminator to also maximize the classification\n",
        "        # accuracy of the auxiliary classifier on generated images, so we\n",
        "        # don't train discriminator to produce class labels for generated\n",
        "        # images (see https://openreview.net/forum?id=rJXTf9Bxg).\n",
        "        # To preserve sum of sample weights for the auxiliary classifier,\n",
        "        # we assign sample weight of 2 to the real images.\n",
        "        disc_sample_weight = [np.ones(2 * len(image_batch)),\n",
        "                              np.concatenate((np.ones(len(image_batch)) * 2,\n",
        "                                              np.zeros(len(image_batch))))]\n",
        "\n",
        "        # see if the discriminator can figure itself out...\n",
        "        epoch_disc_loss.append(discriminator.train_on_batch(\n",
        "            x, [y, aux_y], sample_weight=disc_sample_weight))\n",
        "\n",
        "        # make new noise. we generate 2 * batch size here such that we have\n",
        "        # the generator optimize over an identical number of images as the\n",
        "        # discriminator\n",
        "        noise = np.random.uniform(-1, 1, (2 * len(image_batch), latent_size))\n",
        "        sampled_labels = np.random.randint(0, num_classes, 2 * len(image_batch))\n",
        "\n",
        "        # we want to train the generator to trick the discriminator\n",
        "        # For the generator, we want all the {fake, not-fake} labels to say\n",
        "        # not-fake\n",
        "        trick = np.ones(2 * len(image_batch)) * soft_one\n",
        "\n",
        "        epoch_gen_loss.append(combined.train_on_batch(\n",
        "           [noise, sampled_labels.reshape((-1, 1))],\n",
        "           [trick, sampled_labels]))\n",
        "\n",
        "        progress_bar.update(index + 1)\n",
        "\n",
        "    print('Testing for epoch {}:'.format(epoch))\n",
        "\n",
        "    # evaluate the testing loss here\n",
        "\n",
        "    # generate a new batch of noise\n",
        "    noise = np.random.uniform(-1, 1, (num_test, latent_size))\n",
        "\n",
        "    # sample some labels from p_c and generate images from them\n",
        "    sampled_labels = np.random.randint(0, num_classes, num_test)\n",
        "    generated_images = generator.predict(\n",
        "        [noise, sampled_labels.reshape((-1, 1))], verbose=False)\n",
        "\n",
        "    x = np.concatenate((x_test, generated_images))\n",
        "    y = np.array([1] * num_test + [0] * num_test)\n",
        "    aux_y = np.concatenate((y_test, sampled_labels), axis=0)\n",
        "\n",
        "    # see if the discriminator can figure itself out...\n",
        "    discriminator_test_loss = discriminator.evaluate(\n",
        "        x, [y, aux_y], verbose=False)\n",
        "\n",
        "    discriminator_train_loss = np.mean(np.array(epoch_disc_loss), axis=0)\n",
        "\n",
        "    # make new noise\n",
        "    noise = np.random.uniform(-1, 1, (2 * num_test, latent_size))\n",
        "    sampled_labels = np.random.randint(0, num_classes, 2 * num_test)\n",
        "\n",
        "    trick = np.ones(2 * num_test)\n",
        "\n",
        "    generator_test_loss = combined.evaluate(\n",
        "        [noise, sampled_labels.reshape((-1, 1))],\n",
        "        [trick, sampled_labels], verbose=False)\n",
        "\n",
        "    generator_train_loss = np.mean(np.array(epoch_gen_loss), axis=0)\n",
        "\n",
        "    # generate an epoch report on performance\n",
        "    train_history['generator'].append(generator_train_loss)\n",
        "    train_history['discriminator'].append(discriminator_train_loss)\n",
        "\n",
        "    test_history['generator'].append(generator_test_loss)\n",
        "    test_history['discriminator'].append(discriminator_test_loss)\n",
        "\n",
        "    print('{0:<22s} | {1:4s} | {2:15s} | {3:5s}'.format(\n",
        "        'component', *discriminator.metrics_names))\n",
        "    print('-' * 65)\n",
        "\n",
        "    ROW_FMT = '{0:<22s} | {1:<4.2f} | {2:<15.4f} | {3:<5.4f}'\n",
        "    print(ROW_FMT.format('generator (train)',\n",
        "                         *train_history['generator'][-1]))\n",
        "    print(ROW_FMT.format('generator (test)',\n",
        "                         *test_history['generator'][-1]))\n",
        "    print(ROW_FMT.format('discriminator (train)',\n",
        "                         *train_history['discriminator'][-1]))\n",
        "    print(ROW_FMT.format('discriminator (test)',\n",
        "                         *test_history['discriminator'][-1]))\n",
        "\n",
        "    # generate some digits to display\n",
        "    num_rows = 40\n",
        "    noise = np.tile(np.random.uniform(-1, 1, (num_rows, latent_size)),\n",
        "                     (num_classes, 1))\n",
        "\n",
        "    sampled_labels = np.array([\n",
        "        [i] * num_rows for i in range(num_classes)\n",
        "    ]).reshape(-1, 1)\n",
        "\n",
        "    # get a batch to display\n",
        "    generated_images = generator.predict(\n",
        "        [noise, sampled_labels], verbose=0)\n",
        "\n",
        "    # prepare real images sorted by class label\n",
        "    real_labels = y_train[(epoch - 1) * num_rows * num_classes:\n",
        "                          epoch * num_rows * num_classes]\n",
        "    indices = np.argsort(real_labels, axis=0)\n",
        "    real_images = x_train[(epoch - 1) * num_rows * num_classes:\n",
        "                         epoch * num_rows * num_classes][indices]\n",
        "\n",
        "    # display generated images, white separator, real images\n",
        "    img = np.concatenate(\n",
        "        (generated_images,\n",
        "         np.repeat(np.ones_like(x_train[:1]), num_rows, axis=0),\n",
        "         real_images))\n",
        "\n",
        "    # arrange them into a grid\n",
        "    img = (np.concatenate([r.reshape(-1, 28)\n",
        "                           for r in np.split(img, 2 * num_classes + 1)\n",
        "                           ], axis=-1) * 127.5 + 127.5).astype(np.uint8)\n",
        "    plt.figure(figsize=(30, 30))\n",
        "    plt.imshow(img)\n",
        "    plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}