Copy_of_IIC_TP_LSTMs.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of IIC - TP LSTMs.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "[View in Colaboratory](https://colab.research.google.com/github/calicinio/Testando/blob/master/Copy_of_IIC_TP_LSTMs.ipynb)"
      ]
    },
    {
      "metadata": {
        "id": "ksSAxy0Ck5qU",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Introdução à Inteligência Computacional\n",
        "## Exercício sobre LSTM\n",
        "## Autores:\n",
        "* Álvaro Lemos\n",
        "* Danilo Caldeira\n",
        "* Felipe Carvalho\n",
        "* Rogério Lima"
      ]
    },
    {
      "metadata": {
        "id": "giyrozIUU-Kk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Algumas observações\n",
        "\n",
        "### LSTM é complicado!\n",
        "Pessoal, LSTM é uma técnica bastante complexa e de difícil entendimento, principalmente quando é a primeira vez que se estuda esse tema. Por conta disso, o texto introdutório está bem grande, pois ele tenta passar várias intuições relacionadas às LSTMs. É super tranquilo não entender todas as fórmulas que serão expostas aqui.\n",
        "\n",
        "\n",
        "### Esse notebook pode parecer complicado... Mas não é :D\n",
        "Tentamos mostrar através desse notebook a superioridade das LSTMs em relação às redes neurais tradicionais. Por conta disso, pode parecer que tem muito código e tudo mais, mas tentamos deixar tudo bem claro e, principalmente, a parte que vocês terão que responder é bastante tranquila. De todo modo, qualquer problema que vocês tiverem, não exitem em nos procurar. Nossos contatos são:\n",
        "\n",
        "* alvarolemos@gmail.com\n",
        "* felipefrecar93@gmail.com\n",
        "* rlima.rogerio@gmail.com\n",
        "\n",
        "Para quem tiver interesse em se aprofundar um pouco mais nesse tema, seguem algumas referências que achamos interessantes:\n",
        "\n",
        "* [Blog post](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) *Understanding LSTMs*, do Christopher Olah. Trata-se de um texto bem didático e cheio de imagens para facilitar o entendimento\n",
        "* [Curso online](https://www.coursera.org/learn/nlp-sequence-models) *Sequence Models*, ministrado pelo prof. Andrew Ng. Esse é o último curso da especialização de Deep Learning do Coursera. Trata-se de uma série de cursos extremamente didáticos e de mão na massa, recomendamos fortemente\n",
        "* [Livro](http://www.deeplearningbook.org/) *Deep Learning*, um livro relativamente novo, cujos autores são personalidades muito conhecidas na comunidade de *Machine Learning*. O conteúdo aqui é muito mais aprofundado e não recomendamos que seja o primeiro contato\n",
        "\n",
        "\n",
        "## O problema das sequências\n",
        "As Redes Neurais *feed forward* (FFNN - *Feed Forward Neural Networks*) - as tradicionais apresentadas em sala - são boas para várias aplicações, mas pecam em muitos aspectos para problemas de sequência. Problemas de sequência são àqueles onde a ordem das características de uma amostra importa. Alguns exemplos são:\n",
        "\n",
        "* Análise de sentimento\n",
        "* Tradução de texto\n",
        "* Previsão de séries temporais\n",
        "* Speech-to-text\n",
        "* etc.\n",
        "\n",
        "São várias os problemas que uma FFNN encontra em um problema de sequência:\n",
        "\n",
        "### Problema #1 - Sequências podem ter tamanho variável\n",
        "FFNNs são treinadas para amostras com uma dimensão específica de entrada, mas sequências podem ter tamanhos variados. Por exemplo, um sistema de Análise de Sentimento para classificar críticas de filmes entre *positiva* e *negativa*, poderia ter que classificar amostras como:\n",
        "\n",
        "* *\"Filme excelente\"* (tamanho = 2)\n",
        "* *\"Filme com atuação muito boa e excelente roteiro\"* (tamanho = 8)\n",
        "* *\"Melhor filme desse ano\"* (tamanho = 4)\n",
        "\n",
        "Todas as críticas são claramente positivas, mas uma FFNN teria dificuldades em lidar elas, pois ela esperaria um número fixo de características no seu vetor de entradas\n",
        "\n",
        "### Problema #2: Os padrões aprendidos pelos parâmetros não são compartilhados entre as posições das sequências\n",
        "Isso fica evidente no problema anterior. Nas duas primeiras críticas, *filme* é a primeira palavra, enquanto que na terceira, ele é a segunda. Assim, os pesos relacionados direta (primeira camada escondida) ou indiretamente (nas mais profundas) a cada entrada precisariam aprendar a representação de todas as palavras, o que é bem ruim, pois nem todas as palavras aparecem em todas as posições. Seria necessário um conjunto de amostras muito maior para resolver esse problema. E mesmo que resolvesse, seriam necessários muitos neurônios e camadas para aprender o mesmo padrão individualmente, o que aumentaria o custo computacional consideravalmente.\n",
        "\n",
        "Uma abordagem melhor para resolver esse tipo de problema é através de uma outra arquitetura de Redes Neurais, as Redes Recorrentes, que serão apresentadas a seguir.\n",
        "\n",
        "## Redes Neurais Recorrentes\n",
        "Diferetente das FFNN, as Redes Neurais Recorrentes (RNNs - *Recurrent Neural Networks*) lidam com sequências muito bem, pois foram feitas justamente para isso. O poder reside no fato de que cada amostra da sequência é computada individualmente por uma mesma rede, juntamente com o resultado das amostras anteriores, o que possibilita a representação não só da palavra, mas do contexto em que ela está.\n",
        "\n",
        "Essa explicação ficará mais clara através da figura abaixo\n",
        "\n",
        "![Diagrama de uma RNN](https://bitbucket.org/alvarolemos/lstm-notebook/raw/f1efe2926175e25f27f01454435214ec1d3e4a66/rnn_diagram.svg =300x)\n",
        "\n",
        "Na figura acima, cada quadrado está realizando as seguintes operações\n",
        "\n",
        "$$\n",
        "a^{<t>} = g_1(W_a[a^{<t-1>},x^{<t>}] + b_a)\n",
        "$$\n",
        "$$\n",
        "\\hat{y}^{<t>} = g_2(W_{y}[a^{<t-1>},x^{<t>}] + b_y)\n",
        "$$\n",
        "\n",
        "onde $g_1$ é uma função de ativação $tanh$ ou $relu$ e $g_2$, uma $sigmoid$. Já $[a^{<t-1>},x^{<t>}]$ é uma notação para representar a concatenação entre $a^{<t-1>}$ e $x^{<t>}$. Ou seja, se $a^{<t-1>}$ tem dimensão $(3, 1)$ e $x^{<t>}$ $(4, 1)$, então $[a^{<t-1>},x^{<t>}]$ terá $(7, 1)$.\n",
        "\n",
        "Na figura, o contexto das palavras anteriores é representada por $a^{<t>}$, que nada mais é que a ativação de uma camada de uma rede neural. Fica assim, claro que o contexto da frase vai sendo mantido ao longo da sequência. Outro ponto que é muito importante, é que pelas equações acima, vemos que para todas as palavras, são utilizados os mesmo pesos $W_{a}$, $W_{y}$, $b_a$ e $b_{y}$, o que resolve o **Problema #2** das FFNN. O **Problema #1** também é resolvido, uma vez que cada unidade recorrente (os quadradinhos!) são iguais, o que possibilita que frases de qualquer tamanho sejam aplicadas a um modelo RNN como o da figura.\n",
        "\n",
        "## RNNs são bem legais mesmo... Mas então por que LSTMs?\n",
        "Então, se RNNs fossem tão boas assim, esse notebook seria sobre elas e não sobre LSTM, certo? O ponto fraco das RNNs são sequências **muito longas**. Considere as seguintes frase:\n",
        "\n",
        "*\"**O gato comeu** maçã, laranja, banana e mais um tanto de coisa, então **ele está cheio** agora\"*\n",
        "\n",
        "*\"**Os gatos comeram** maçã, laranja, banana e mais um tanto de coisa, então **eles estão cheios** agora\"*\n",
        "\n",
        "Talvez essa frase não seja tão longa assim, mas passa a ideia. Nessas ocasiões, as RNNs têm dificuldade de manter uma informação (plural, nesse caso) ativada ao longo de várias passos de uma sequência. Esse problema é conhecido como *vanishing gradients*, e de forma bem simplificada, o que ocorre é que, a medida que os a informação vai sendo passada a frente entre as células de $a^{<1>}$ para $a^{<2>}$, depois para $a^{<3>}$, e assim por diante, as ativações vão sendo multiplicadas sucessivamente pelos pesos $W_a$. Como os pesos em $W_a$ são normalmente valores dentro do intervalo $[0, 1]$, quão mais longa for uma sequência, menor será a ativação a cada passo. Assim, uma ativação do início da série (como *\"O gato comeu\"*) dificilmente alcança as células mais ao final da sequência. E mesmo que chegue, da mesma forma o backpropagation é afetado, ou seja, as atualizações dos pesos no início da sequência tendem a ser ínfimos.\n",
        "\n",
        "Para lidar com esse problema, foi proposta a arquitetura de Redes Recorrentes LSTM (*\"Long Short-Term Memory\"*), que trouxe o conceito de memória de célula. A LSTM continua sendo recorrente, mas as operações que sua célula (quadrado da figura anterior) computa são diferentes. A figura abaixo mostra a nova estrutura. Ela é relativamente complexa, então vamos tentar entender a intuição por trás dela.\n",
        "\n",
        "![Diagrama de uma RNN](https://github.com/Kulbear/deep-learning-coursera/raw/1beef8d17e804448c302b21a0b19833a7572f379/Sequence%20Models/images/LSTM.png =900x)\n",
        "\n",
        "Vemos que, assim como numa RNN simples, a célula que processa os dados no instante $t$ recebe as ativações da célula anterior ($a^{<t-1>}$) e a entrada do instante atual ($x^{<t>}$), tendo como saída as ativações do instante atual. O que muda é que temos uma nova entrada, $c^{<t-1>}$, a memória da célula anterior, e uma nova saída, $c^{<t>}$, a memória da célula atual. Internamente, o que rege o funcionamento dessa *memória* são os portões de atualização e de esquecimento: $\\Gamma_u$ (*update*) e $\\Gamma_f$ (*forget*), respectivamente.\n",
        "\n",
        "O que é importante nas suas fórmulas: ambos são resultados da aplicação de uma função sigmoidal ($\\sigma(\\cdot)$), ou seja, seus valores então entre $0$ e $1$. Por conta disso, o que é multiplicado por eles será reforçado ou atenuado, a depender se seus valores estiverem próximos de $1$ ou $0$, respectivamente.\n",
        "\n",
        "Com isso em mente, repare que o portão de esquecimento ($\\Gamma_f$) está sendo multiplicado por pelo estado da célula no instante anterior, $c^{<t-1>}$. Isso quer dizer que, se $\\Gamma_f \\approx 1$, o estado anterior será **reforçado**, ao passo que, se $\\Gamma_f \\approx 0$, ele será esquecido. Da mesma forma, o portão de atualização está sendo multiplicado pelo estado atual, ou seja, se  $\\Gamma_u \\approx 1$, a memória da célula será atualizada. E assim, o problema dos *vanishing gradients* está resolvido.\n",
        "\n",
        "Vamos agora ao nosso exercício!\n",
        "\n",
        "## Emojify\n",
        "A aplicação que você irá completar aqui foi inspirada em um exercício do curso Sequence Models da especialização de Deep Learning do Coursera, em parceria com a deeplearning.ai. Essa aplicação funciona da seguinte forma: dada uma frase como entrada, ela retorna um emoji que transmita a mensagem que a frase de entrada está passando. Ela será implementada antes em uma FFNN, para vermos alguns problemas inerentes da incapacidade de lidar com sequência. Depois, em uma LSTM, onde poderemos ver sua superioridade para lidar com esse tipo de problema. \n",
        "\n",
        "Antes de começarmos, vale ressaltar uma nota sobre a representação das palavras de cada frase, que utiliza de **word embeddings**.\n",
        "\n",
        "### Word Embeddings\n",
        "A ideia por trás dos word embeddings é que eles possibilitam que palavras semelhantes tenham representações paracidas. Por exemplo, se durante o treinamento do Emojify, a frase *\"hoje o almoço foi bom\"* tinha como saída esperada o emoji 🍴 (talheres! :P), se o modelo aprendeu corretamente, muito provavelmente o mesmo emoji seria retornado para a frase *\"hoje o jantar foi bom\"*, pois ambos são substantivos que descrevem uma refeição.\n",
        "\n",
        "Mas como ele sabe a representação das palavras *almoço* e *jantar*? Através de uma matriz de word embeddings, que já está pronta para ser usada. Existem algoritmos que treinam essas matrizes, mas esse problema não é o foco deste notebook. Existem vários word embeddings disponíveis online justamente para poupar o tempo de pessoas que querem implementar alguma aplicação de NLP (*Natural Language Processing* - Processamento de Linguagem Natural), sem precisar ter que treinar uma matriz de embeddings, que é uma tarefa muito cara computacionalmente.\n",
        "\n",
        "Abaixo temos um exemplo de uma matriz de embeddings. As linhas são palavras do vocabulário do embedding, enquanto que as colunas são as dimensões de representação do mesmo. Nessa representação, os valores das colunas variam entre $[-1, 1]$ e quanto mais próximo desses limites estiver uma coluna de uma dada palavra, mais ela é representada por aquela característica. Vemos, por exemplo, que Homem e Mulher são muito bem representados pela característica gênero. Como são gêneros distintos, o sinal dessas palavras são opostos.\n",
        "\n",
        "|            |   Gênero   |   Realeza  |    Idade   |  Alimento  |\n",
        "| ---------- | ---------- | ---------- | ---------- | ---------- |\n",
        "| Homem      |   -0.96    |   -0.02    |   -0.02    |    0.09    |\n",
        "| Mulher     |   0.98     |   -0.03    |   -0.03    |    0.07    |\n",
        "| Rei        |  -0.94     |  -0.93     |   0.09     |   0.00     |\n",
        "| Rainha     |   0.92     |   0.98     |   0.12     |   0.01     |\n",
        "| Maçã       |   0.00     |   0.01     |   0.01     |   0.98     |\n",
        "| Laranja    |   0.01     |   0.03     |   0.04     |   0.97     |\n",
        "\n",
        "É isso pessoal... Mãos a obra! ;)"
      ]
    },
    {
      "metadata": {
        "id": "EcaD7a3Y8vnf",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Exercício\n",
        "\n",
        "Neste notebook resolveremos um problema de classificação de texto utilizando MLPs, que são modelos de redes neurais tradicionais e LSTMs que são modelos de redes neurais recorrentes (RNN).\n",
        "\n",
        "Atualmente, a LSTM representa o estado da arte no que tange problemas de sequencias. LSTMs, por serem capazes de aprender hieraquias temporais tem comportamentos melhores para problemas com essas caracteristicas. Exemplos de problemas assim são problemas de classificação de texto por exemplo, onde a ordem das palavras pode mudar completamente o sentido da frase, de forma que palavras que aparecem antes em uma frase inflenciam quais aparecem depois além de influenciar outros fatores como conjulgação verbal, etc. \n",
        "\n",
        "Neste trabalho um dataset de frases onde cada label é um emoji correspondente à frase será utilizado e a performance de MLPs e LSTMs será comparada na tarefa de classificação. Ao final, deverá ficar evidente a capacidade superior da arquitetura LSTM de lidar melhor com modelos de sequencia."
      ]
    },
    {
      "metadata": {
        "id": "bhmYo1G-HO7v",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 591
        },
        "outputId": "535830b1-2ff0-4bea-d160-bacedb1e4685"
      },
      "cell_type": "code",
      "source": [
        "#Baixando os dados necessários para o TP\n",
        "\n",
        "!wget \"https://docs.google.com/uc?export=download&id=1ECmCxGL2Sa36WtPVUc-qFWrCP14I4CmR\" -O data.tar.gz\n",
        "!tar -zxvf data.tar.gz\n",
        "!pip install emoji"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2018-06-09 01:09:02--  https://docs.google.com/uc?export=download&id=1ECmCxGL2Sa36WtPVUc-qFWrCP14I4CmR\r\n",
            "Resolving docs.google.com (docs.google.com)... 74.125.141.138, 74.125.141.113, 74.125.141.139, ...\r\n",
            "Connecting to docs.google.com (docs.google.com)|74.125.141.138|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Moved Temporarily\n",
            "Location: https://doc-14-74-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/qf5t1nb7ca4ulkprf3m4f4g8hj98kg6o/1528502400000/02256098521353866195/*/1ECmCxGL2Sa36WtPVUc-qFWrCP14I4CmR?e=download [following]\n",
            "Warning: wildcards not supported in HTTP.\n",
            "--2018-06-09 01:09:07--  https://doc-14-74-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/qf5t1nb7ca4ulkprf3m4f4g8hj98kg6o/1528502400000/02256098521353866195/*/1ECmCxGL2Sa36WtPVUc-qFWrCP14I4CmR?e=download\n",
            "Resolving doc-14-74-docs.googleusercontent.com (doc-14-74-docs.googleusercontent.com)... 74.125.141.132, 2607:f8b0:400c:c06::84\n",
            "Connecting to doc-14-74-docs.googleusercontent.com (doc-14-74-docs.googleusercontent.com)|74.125.141.132|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: unspecified [application/gzip]\n",
            "Saving to: ‘data.tar.gz’\n",
            "\n",
            "data.tar.gz             [  <=>               ]  65.98M  55.4MB/s    in 1.2s    \n",
            "\n",
            "2018-06-09 01:09:08 (55.4 MB/s) - ‘data.tar.gz’ saved [69186137]\n",
            "\n",
            "data/\n",
            "data/test_set.csv\n",
            "data/utils.py\n",
            "data/glove.6B.50d.txt\n",
            "data/train_set.csv\n",
            "Collecting emoji\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/47/e7/cf76e0774b13459e61ffe16a042a35f42e292aeebba060b919c05b0854ab/emoji-0.5.0.tar.gz (41kB)\n",
            "\u001b[K    100% |████████████████████████████████| 51kB 2.5MB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: emoji\n",
            "  Running setup.py bdist_wheel for emoji ... \u001b[?25l-\b \bdone\n",
            "\u001b[?25h  Stored in directory: /content/.cache/pip/wheels/5d/60/eb/49bb6f75caa1bd0b277d43bd14ac3ea1db3e5ed030d2013852\n",
            "Successfully built emoji\n",
            "Installing collected packages: emoji\n",
            "Successfully installed emoji-0.5.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "ys7wpZW5IjOP",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "7f5bf19d-ea17-48da-8085-e73a5a60dc6b"
      },
      "cell_type": "code",
      "source": [
        "import sys\n",
        "sys.path.insert(0, './data')\n",
        "\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import emoji\n",
        "from utils import *\n",
        "\n",
        "from keras.models import Model\n",
        "from keras.layers import Dense, Input, Dropout, LSTM, Activation\n",
        "from keras.layers.embeddings import Embedding\n",
        "from keras.preprocessing import sequence\n",
        "from keras.initializers import glorot_uniform\n",
        "np.random.seed(1)\n",
        "\n",
        "%matplotlib inline"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "0N1BB9tqDVYi",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 106
        },
        "outputId": "93a57094-295b-4ccf-b034-347916354e9a"
      },
      "cell_type": "code",
      "source": [
        "# Exemplo dos labels de treinamento e teste utilizados : Emojis\n",
        "print_emoji_labels()"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "label: 0, emoji: ❤\n",
            "label: 1, emoji: ⚾\n",
            "label: 2, emoji: 😄\n",
            "label: 3, emoji: 😞\n",
            "label: 4, emoji: 🍴\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "AWtymkhBDMe7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Data preprocessing\n",
        "\n",
        "Os exemplos de treinamento e teste do dataset sao feitos de frases, e essas frases por consequencia constituidas por uma serie de palavras.\n",
        "\n",
        "Uma representacao numerica a nivel de palavra (cada palavra sera posteriormente utilizada como entrada de um timestep da LSTM) será feita utilizado-se um embedding conhecido como glove vectors, de dimensão tamanho 50. Cade dimensao deste embedding pode ser intepretada como representando uma caracteristica da palavra, como sexo ou negacao/afirmacao, etc.\n",
        "\n",
        "Embeddings, como ja discutido anteriormente sao uma forma muito poderosa de representar sequencias e texto, e foram um grande avanco na area de Deep Learning para esse tipo de modelo. Embeddings trazem uma capacidade de generalizacao muito boa para o problema, de forma que dados que nao necessariamente estao contidos no conjunto de treinamento, mas que estao no dicionario de embeddings do problema podem ser inferidos atraves do modelo por terem representacoes na forma de vetor parecidas.\n",
        "\n",
        "Abaixo serão carregados embeddings pre treinados e criados dicionários que mapeam palavras para indices (sera usado posteriormente na LSTM), indices para palavras e palavras para sua representacao na forma de vetor glove 50 dimensional (os vetores serao utilizados em ambos modelos).\n"
      ]
    },
    {
      "metadata": {
        "id": "GwRvZ2aOGlfc",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8hJxqJxArqQr",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "word_to_index, index_to_word, word_to_vec_map = read_glove_vecs('data/glove.6B.50d.txt')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ix1IC8pH8KIS",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "0c32c3ba-f8a4-4a15-c72d-36d798547c13"
      },
      "cell_type": "code",
      "source": [
        "# visualização dos conjuntos de treinamento\n",
        "train_set = pd.read_csv('data/train_set.csv')\n",
        "train_set.head()"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>never talk to me again</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>I am proud of your achievements</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>It is the worst day in my life</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Miss you so much</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>food is life</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                          sentence  label\n",
              "0           never talk to me again      3\n",
              "1  I am proud of your achievements      2\n",
              "2   It is the worst day in my life      3\n",
              "3                 Miss you so much      0\n",
              "4                     food is life      4"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "metadata": {
        "id": "iqfHT8VcNvFs",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "bdecc1c7-0ada-401b-bf8d-abe511387cf8"
      },
      "cell_type": "code",
      "source": [
        "test_set = pd.read_csv('data/test_set.csv')\n",
        "test_set.head()"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>I want to eat</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>he did not answer</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>he got a very nice raise</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>she got me a nice present</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ha ha ha it was so funny</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                    sentence  label\n",
              "0              I want to eat      4\n",
              "1          he did not answer      3\n",
              "2   he got a very nice raise      2\n",
              "3  she got me a nice present      2\n",
              "4   ha ha ha it was so funny      2"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "metadata": {
        "id": "Wmq9S2AXN000",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "d418003b-188e-43f9-e082-2bf2b0aed985"
      },
      "cell_type": "code",
      "source": [
        "print('Numero de amostras de treino: {}'.format(len(train_set)))\n",
        "print('Numero de amostras de teste: {}'.format(len(test_set)))"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Numero de amostras de treino: 132\n",
            "Numero de amostras de teste: 56\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "mazqywJfNxQ4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Como pode ser visto o tamanho dos conjuntos de treinamento não são grandes, porem a quantidade de embeddings de palavras disponívels fazem com que mesmo com pouco dado o modelo ainda tenha uma capacidade de generalizacao para dados nao vistos relativamente alta, para os labels selecionados."
      ]
    },
    {
      "metadata": {
        "id": "StaqS1P3OVsR",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "1b80a876-3e9f-4b56-ef4d-a693561bd713"
      },
      "cell_type": "code",
      "source": [
        "print('Numero de palavras no dicionario de embeddings: {}'.format(len(word_to_index)))"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Numero de palavras no dicionario de embeddings: 400000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "C6jMimdfOfY3",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#turn into arrays\n",
        "#train set\n",
        "X_train = train_set['sentence'].values\n",
        "y_train = train_set['label'].values\n",
        "#test set\n",
        "X_test = test_set['sentence'].values\n",
        "y_test = test_set['label'].values"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "hBooI0Z6OsAf",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Como pode ser visto na visualizaçao dos dados acima, existe rotulos (labels) de 0 a 4 para os conjuntos de treinamento e teste.\n",
        "\n",
        "Isso faz com que o nosso problema de classificao seja multi-classe.\n",
        "\n",
        "Uma maneira de representar isto em redes neurais é fazer com que existam um numero de neuronios igual ao numero de classes que se deseja classificar. Dado uma ativação do tipo Softmax, o neuronio com maior valor de probabilidade atrelado recebe 1 e os demais 0.\n",
        "\n",
        "Logo, y deve ser transformado em um vetor de 5 posicoes, onde deve ser colocado 1 naquela cujo indice corresponde ao label do y original. A esse tipo de representacao se da um nome de Onehot encoding.\n",
        "\n",
        "Crie uma função que transforme um vetor y em um vetor one hot encoded (http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html)\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "MPr2n0yRIfzL",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "e0999066-b785-4162-bcaa-b4a5f351110a"
      },
      "cell_type": "code",
      "source": [
        "from sklearn.preprocessing import OneHotEncoder\n",
        "\n",
        "one = OneHotEncoder()\n",
        "new_y = np.reshape(y_test, (-1, 1))\n",
        "\n",
        "type(y_test)\n",
        "new_y.shape\n",
        "\n",
        "#new_y = np.ndarray([len(y_test), 1])\n",
        "#new_y = y_test\n",
        "\n",
        "#one.fit(new_y)\n",
        "\n",
        "#saida = one.transform(new_y)\n",
        "\n",
        "#one.n_values_\n",
        "\n",
        "#saida.toarray()\n"
      ],
      "execution_count": 95,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(56, 1)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 95
        }
      ]
    },
    {
      "metadata": {
        "id": "cXqCKNjSRDAz",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from sklearn.preprocessing import OneHotEncoder\n",
        "\n",
        "# Cria uma funcao que transform os Y em vetores one-hot. Basicamente, iremos de\n",
        "# uma representacao do tipo:\n",
        "#\n",
        "#   3\n",
        "#   2\n",
        "#   1\n",
        "#   4\n",
        "# \n",
        "# Para algo do tipo:\n",
        "#\n",
        "#   0 0 1 0\n",
        "#   0 1 0 0\n",
        "#   1 0 0 0\n",
        "#   0 0 0 1\n",
        "#\n",
        "# Para tal, utilize a classe OneHotEncoder do sklearn. O array de entrada deve ter\n",
        "# uma coluna so, entao, para garantir isso, y.reshape(-1, 1). O resultado do one\n",
        "# hot encoding deve ser transformado em em array, atravez da funcao .toarray()\n",
        "def one_hot_encode(y):\n",
        "    \n",
        "    one = OneHotEncoder()\n",
        "    y = np.reshape(y, (-1, 1))\n",
        "    \n",
        "    one.fit(y)\n",
        "    \n",
        "    y_1hot = one.transform(y).toarray()\n",
        "    \n",
        "    return y_1hot\n",
        "\n",
        "y_train_1hot = one_hot_encode(y_train)\n",
        "y_test_1hot = one_hot_encode(y_test)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "heMDRnsnRUSb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Expected output:**  \n",
        "y_train_1hot.shape = (132, 5)  \n",
        "y_test_1hot.shape = (56, 5)"
      ]
    },
    {
      "metadata": {
        "id": "iccM1GceRVZ0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "9134fef7-48a7-4e1f-b936-ea4dc16987b2"
      },
      "cell_type": "code",
      "source": [
        "print('y_train_1hot.shape = {}'.format(y_train_1hot.shape))\n",
        "print('y_test_1hot.shape = {}'.format(y_test_1hot.shape))"
      ],
      "execution_count": 98,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "y_train_1hot.shape = (132, 5)\n",
            "y_test_1hot.shape = (56, 5)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "Q_aMFwUoSsz7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Uma vez que o dataset foi preparado partiremos para o modelo de MLP seguido da LSTM. O input de cada um destes modelos e ligeiramente diferente.\n",
        "\n",
        "No entanto vale ressaltar que, como ambos representam palavras na forma de vetores, e embora exista um dicionario de embeddings contendo 400k palavras, todas estas sao case sensitive e além disso cada exemplo de treinamento é constituido de uma frase. Logo, um pre processamento em comum para tanto a MLP quanto para a LSTM é garantir que todas as palavras tenham todas suas letras minusculas, assim como no dicionario, e alem disso que as frases sejam divididas em palavras individuais para cada observação."
      ]
    },
    {
      "metadata": {
        "id": "-pwjfN4QUHi3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Modelo 1 - MLP\n",
        "\n",
        "No caso da MLP, uma vez separadas cada palavra de uma frase, será tirado a media destas para servir como um exemplo de treinamento. Ou seja, supondo que a frase tenha 5 palavras por exemplo, a media destas 5 na sua forma de vetor 50 dimensional será um exemplo de treinamento para a rede MLP.\n",
        "\n",
        "Isso se faz necessário pois as frases tem tamanho variável e o numero de caracteristicas na MLP nao pode ser variável, logo como solução tirou-se a media dos vetores de cada palavra. \n",
        "\n",
        "Vamos implementar a função que faz a média de uma frase utilizando os dicionarios de embeddings disponíveis:"
      ]
    },
    {
      "metadata": {
        "id": "LCconJVqTPuj",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "e823f3fb-bfbc-4547-8b1c-75ffeed7bbe8"
      },
      "cell_type": "code",
      "source": [
        "#Exemplo\n",
        "sentence = 'i LiKe poTaTos'\n",
        "#use lower e split em uma frase para transformar tudo em minusculo e separar as palavras\n",
        "words = sentence.lower().split()\n",
        "print(words)"
      ],
      "execution_count": 99,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['i', 'like', 'potatos']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "8KNKStcxRXh7",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 178
        },
        "outputId": "9f6dac1f-1eff-41c8-d1dc-88d35512c621"
      },
      "cell_type": "code",
      "source": [
        "print(word_to_vec_map['potatos'])\n",
        "print(word_to_vec_map['potatos'].shape)"
      ],
      "execution_count": 100,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.77033  -0.57765  -1.4086    0.28488   1.3224    0.025708 -0.39524\n",
            "  0.1565    0.73926   0.07319   0.39573   0.81557   1.7829   -1.1323\n",
            " -1.2555   -0.31784  -1.0604   -0.19973   2.2133   -0.13883   0.50197\n",
            "  1.5084    0.58036   0.3317    0.56073   2.767     0.22387  -1.1203\n",
            "  1.3909    0.51904   0.91258   1.2222    1.3356   -0.01816   1.855\n",
            "  0.87767  -0.51949  -0.60569  -0.33336  -0.68257   0.18867  -1.2253\n",
            " -0.3839   -0.61717   1.311     1.0898    0.23086   1.3371   -0.060975\n",
            "  1.4675  ]\n",
            "(50,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "62weqx0NV30f",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 249
        },
        "outputId": "b35c8fac-9152-4424-be48-4c00dfba945a"
      },
      "cell_type": "code",
      "source": [
        "#média dessas 3 palavras em um unico vetor de 50 dimensoes, para ser um exemplo de treinamento para a MLP\n",
        "avg = (word_to_vec_map[words[0]] + word_to_vec_map[words[1]] + word_to_vec_map[words[2]])/3\n",
        "\n",
        "avg"
      ],
      "execution_count": 104,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([-9.44466667e-02, -7.22533333e-02, -5.71287667e-01, -1.36759000e-01,\n",
              "        7.60850000e-01, -6.08073333e-02, -4.92200000e-01, -3.22600000e-02,\n",
              "       -1.98810000e-01,  1.93335857e-01,  4.30000000e-04,  7.90340000e-01,\n",
              "        1.23240000e-01, -4.23786667e-01,  1.70286667e-01,  2.23070000e-01,\n",
              "       -1.62426667e-01,  2.49016667e-01,  6.94771000e-01, -6.00296667e-01,\n",
              "       -1.71700000e-01,  8.59043333e-01,  5.61106667e-01,  3.30116667e-01,\n",
              "        6.74366667e-01, -5.22300000e-01, -7.96943333e-01, -1.19553333e-01,\n",
              "        9.86853333e-01, -5.62653333e-01,  2.54052667e+00,  8.28670000e-01,\n",
              "        2.11421000e-01,  8.96473333e-02,  5.04962000e-01,  2.39480000e-01,\n",
              "       -1.62270000e-01,  6.08300000e-02,  1.46910000e-01, -3.42720000e-01,\n",
              "       -3.12081000e-02, -1.48692667e-01, -2.21085733e-01,  1.30266667e-01,\n",
              "        7.44175333e-01,  5.71426667e-01, -3.73400000e-02,  2.98733333e-02,\n",
              "       -2.41591667e-01,  8.80180000e-01])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 104
        }
      ]
    },
    {
      "metadata": {
        "id": "TG61J56lWDNR",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# implemente a funcao que dada uma frase, calcula a media da sua representacao na forma de vetor\n",
        "\n",
        "def average_sentence(sentence):\n",
        "    # a frase ja esta sendo lower e split pra voce :)\n",
        "    sentence = sentence.lower().split()\n",
        "    avg = np.zeros((50,))    \n",
        "    n = len(sentence)\n",
        "    for word in sentence:\n",
        "        avg = word_to_vec_map[word] + avg\n",
        "    avg = avg/n\n",
        "    # FIM DO CODIGO\n",
        "    return avg"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "si-Xm0k_WuCq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Expected output:**  \n",
        "\n",
        "average_sentence('hello bob')\n",
        "\n",
        "array([-0.729985  ,  0.49204   ,  0.196528  ,  0.17776   , -0.2934595 ,\n",
        "        0.112665  , -1.169915  ,  0.05839   , -0.496285  , -0.1215    ,\n",
        "       -0.4811865 ,  0.69023   , -0.03501   ,  0.2782145 ,  0.51474   ,\n",
        "       -0.187681  , -0.124935  ,  0.4923305 , -0.114805  , -0.3723525 ,\n",
        "       -0.29856   ,  0.616055  ,  0.331685  ,  0.00125   ,  0.82339   ,\n",
        "       -0.874855  , -0.83978   ,  0.18487   ,  0.19692085, -0.681915  ,\n",
        "        1.06786   ,  0.09266   , -0.401135  ,  0.4549565 , -0.65187   ,\n",
        "       -0.22757   ,  0.39745   , -0.5354514 ,  0.2234785 , -0.585075  ,\n",
        "        0.342345  ,  0.23009   , -0.89711   , -0.391322  ,  0.316475  ,\n",
        "        0.071785  , -0.4854155 , -0.54965   , -0.3647985 ,  1.23727   ])"
      ]
    },
    {
      "metadata": {
        "id": "JPojoN5kWaKK",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 196
        },
        "outputId": "bf5f734d-5a36-4da5-e195-5edf131a9dd6"
      },
      "cell_type": "code",
      "source": [
        "average_sentence('hello bob')"
      ],
      "execution_count": 106,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([-0.729985  ,  0.49204   ,  0.196528  ,  0.17776   , -0.2934595 ,\n",
              "        0.112665  , -1.169915  ,  0.05839   , -0.496285  , -0.1215    ,\n",
              "       -0.4811865 ,  0.69023   , -0.03501   ,  0.2782145 ,  0.51474   ,\n",
              "       -0.187681  , -0.124935  ,  0.4923305 , -0.114805  , -0.3723525 ,\n",
              "       -0.29856   ,  0.616055  ,  0.331685  ,  0.00125   ,  0.82339   ,\n",
              "       -0.874855  , -0.83978   ,  0.18487   ,  0.19692085, -0.681915  ,\n",
              "        1.06786   ,  0.09266   , -0.401135  ,  0.4549565 , -0.65187   ,\n",
              "       -0.22757   ,  0.39745   , -0.5354514 ,  0.2234785 , -0.585075  ,\n",
              "        0.342345  ,  0.23009   , -0.89711   , -0.391322  ,  0.316475  ,\n",
              "        0.071785  , -0.4854155 , -0.54965   , -0.3647985 ,  1.23727   ])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 106
        }
      ]
    },
    {
      "metadata": {
        "id": "IpCvIj3dXDuA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Utilizando a funcao anterior temos uma funcao que faz a mesma coisa para todos as observacoes de um dataset.\n",
        "\n",
        "E aproveitando isso vamos calcular como ficaria o conjunto de teste e treino, uma vez tirado as medias das palavras de suas frases."
      ]
    },
    {
      "metadata": {
        "id": "-FrZhqrIW7zS",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def average_dataset(X):\n",
        "    m = X.shape[0]\n",
        "    avg_X = np.zeros((m, 50))\n",
        "    for i, sentence in enumerate(X):\n",
        "        avg = average_sentence(sentence)\n",
        "        avg_X[i,:] = avg\n",
        "    return avg_X\n",
        "\n",
        "avg_X_train = average_dataset(X_train)\n",
        "avg_X_test = average_dataset(X_test)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PeMfPf2uX0jt",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "b96b864d-6c05-422d-aa88-81f97951030c"
      },
      "cell_type": "code",
      "source": [
        "#conferindo a dimensao para ver se temos um vetor com as 132 observacoes originais e 50 dimensoes\n",
        "avg_X_train.shape"
      ],
      "execution_count": 108,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(132, 50)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 108
        }
      ]
    },
    {
      "metadata": {
        "id": "EkUSXw7jbeCf",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Agora vamos finalmente construir um modelo MLP para classificar cada frase (que foi transformada numa media das palavras com 50 dimensoes) e ver seu emoji correspondente.\n",
        "\n",
        "Vamos utilizar o framework Keras (https://keras.io/)\n",
        "\n",
        "Primeiramente, para ficar familiarizado com o Keras, vamos construir um modelo de uma rede neural teste, uma MLP simples de 1 camada escondida com 5 neuronios e 1 camada de saida.\n",
        "\n",
        "\n",
        "        "
      ]
    },
    {
      "metadata": {
        "id": "foPgRwL4cvyB",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def modelo_teste(input_shape):\n",
        "    #Vamos criar um placeholder para a entrada da rede\n",
        "    #Input é uma funcao do Keras que faz isso\n",
        "    #Este shape seria o numero de dimensoes do tensor\n",
        "    #Por exemplo no nosso problema nosso vetor tem 50 dimensoes\n",
        "    X_input = Input(input_shape)\n",
        "    #Adicionando uma camada escondida com 5 neuronios\n",
        "    #Dense significa fully connected: todos os pesos da camada anterior estao\n",
        "    #conectados a todos os pesos da seguinte (padrao das MLPs)\n",
        "    X = Dense(5)(X_input)\n",
        "    #Repare como essa camada recebe a entrada X_input e retorna X\n",
        "    #Esta camada tem uma ativacao, vamos escolher a sigmoid\n",
        "    X = Activation('sigmoid')(X)\n",
        "    #Agora uma camada de saida\n",
        "    #Vamos supor um problema de classificacao com 3 classes para este teste (precisamos de 3 neuronios entao, seguindo nossa logica)\n",
        "    X = Dense(3)(X)\n",
        "    #uma ativacao sofmax é adequada\n",
        "    X = Activation('softmax')(X)\n",
        "\n",
        "    model = Model(inputs=X_input, outputs=X, name='MLP_teste')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "v41FvrWZexWx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Agora construa a MLP para nosso modelo de sequencia dos emojis.\n",
        "\n",
        "A MLP tem uma camada escondida com 128 neuronios, ativacaco sigmoid, uma camada de saida com 5 neuronios, ativacao softmax."
      ]
    },
    {
      "metadata": {
        "id": "WShakQ18YaVi",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def MLP_avg(input_shape):\n",
        "    # SEU CODIGO AQUI\n",
        "    X_input = Input(input_shape)\n",
        "    X = Dense(128)(X_input)\n",
        "    X = Activation('sigmoid')(X)\n",
        "    X = Dense(5)(X)\n",
        "    X = Activation('softmax')(X)\n",
        "    # FIM DO CODIGO\n",
        "    model = Model(inputs=X_input, outputs=X, name='MLP')\n",
        "    \n",
        "    return model   "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "-cjaSleofKOL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "  Vamos compilar e rodar pra ver os resultados dessa MLP das medias das palavras!\n",
        "  \n",
        "**Expected output:**  \n",
        "\n",
        "  \n",
        "Layer (type)                 Output Shape              Param #   \n",
        "\n",
        "input_1 (InputLayer)         (None, 50)                0         \n",
        "_________________________________________________________________\n",
        "dense_1 (Dense)              (None, 128)               6528      \n",
        "_________________________________________________________________\n",
        "activation_1 (Activation)    (None, 128)               0         \n",
        "_________________________________________________________________\n",
        "dense_2 (Dense)              (None, 5)                 645       \n",
        "_________________________________________________________________\n",
        "activation_2 (Activation)    (None, 5)                 0         \n",
        "\n",
        "Total params: 7,173  \n",
        "Trainable params: 7,173  \n",
        "Non-trainable params: 0"
      ]
    },
    {
      "metadata": {
        "id": "r7aXZev7fJDC",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 321
        },
        "outputId": "e74514ab-f346-480f-d320-8f9d7d862863"
      },
      "cell_type": "code",
      "source": [
        "MLP = MLP_avg((avg_X_train.shape[1],))\n",
        "MLP.compile('adam', 'categorical_crossentropy', metrics=['accuracy'])\n",
        "MLP.summary()"
      ],
      "execution_count": 113,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_2 (InputLayer)         (None, 50)                0         \n",
            "_________________________________________________________________\n",
            "dense_4 (Dense)              (None, 128)               6528      \n",
            "_________________________________________________________________\n",
            "activation_3 (Activation)    (None, 128)               0         \n",
            "_________________________________________________________________\n",
            "dense_5 (Dense)              (None, 5)                 645       \n",
            "_________________________________________________________________\n",
            "activation_4 (Activation)    (None, 5)                 0         \n",
            "=================================================================\n",
            "Total params: 7,173\n",
            "Trainable params: 7,173\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "5y3joixHfU7G",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 14328
        },
        "outputId": "df22b7a2-347c-4920-a424-5bcf13805e92"
      },
      "cell_type": "code",
      "source": [
        "MLP.fit(avg_X_train, y_train_1hot, epochs=400, batch_size=avg_X_train.shape[0], shuffle=True)"
      ],
      "execution_count": 115,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/400\n",
            "132/132 [==============================] - 0s 38us/step - loss: 0.3298 - acc: 0.9318\n",
            "Epoch 2/400\n",
            "132/132 [==============================] - 0s 41us/step - loss: 0.3289 - acc: 0.9318\n",
            "Epoch 3/400\n",
            "132/132 [==============================] - 0s 30us/step - loss: 0.3281 - acc: 0.9318\n",
            "Epoch 4/400\n",
            "132/132 [==============================] - 0s 29us/step - loss: 0.3272 - acc: 0.9318\n",
            "Epoch 5/400\n",
            "132/132 [==============================] - 0s 18us/step - loss: 0.3263 - acc: 0.9318\n",
            "Epoch 6/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.3255 - acc: 0.9318\n",
            "Epoch 7/400\n",
            "132/132 [==============================] - 0s 26us/step - loss: 0.3246 - acc: 0.9318\n",
            "Epoch 8/400\n",
            "132/132 [==============================] - 0s 19us/step - loss: 0.3237 - acc: 0.9318\n",
            "Epoch 9/400\n",
            "132/132 [==============================] - 0s 22us/step - loss: 0.3229 - acc: 0.9318\n",
            "Epoch 10/400\n",
            "132/132 [==============================] - 0s 17us/step - loss: 0.3220 - acc: 0.9318\n",
            "Epoch 11/400\n",
            "132/132 [==============================] - 0s 19us/step - loss: 0.3212 - acc: 0.9318\n",
            "Epoch 12/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.3203 - acc: 0.9318\n",
            "Epoch 13/400\n",
            "132/132 [==============================] - 0s 24us/step - loss: 0.3195 - acc: 0.9318\n",
            "Epoch 14/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.3187 - acc: 0.9318\n",
            "Epoch 15/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.3178 - acc: 0.9318\n",
            "Epoch 16/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.3170 - acc: 0.9318\n",
            "Epoch 17/400\n",
            "132/132 [==============================] - 0s 26us/step - loss: 0.3162 - acc: 0.9318\n",
            "Epoch 18/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.3154 - acc: 0.9318\n",
            "Epoch 19/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.3145 - acc: 0.9318\n",
            "Epoch 20/400\n",
            "132/132 [==============================] - 0s 22us/step - loss: 0.3137 - acc: 0.9318\n",
            "Epoch 21/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.3129 - acc: 0.9318\n",
            "Epoch 22/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.3121 - acc: 0.9318\n",
            "Epoch 23/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.3113 - acc: 0.9318\n",
            "Epoch 24/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.3105 - acc: 0.9318\n",
            "Epoch 25/400\n",
            "132/132 [==============================] - 0s 19us/step - loss: 0.3097 - acc: 0.9318\n",
            "Epoch 26/400\n",
            "132/132 [==============================] - 0s 19us/step - loss: 0.3089 - acc: 0.9318\n",
            "Epoch 27/400\n",
            "132/132 [==============================] - 0s 23us/step - loss: 0.3081 - acc: 0.9318\n",
            "Epoch 28/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.3074 - acc: 0.9318\n",
            "Epoch 29/400\n",
            "132/132 [==============================] - 0s 22us/step - loss: 0.3066 - acc: 0.9318\n",
            "Epoch 30/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.3058 - acc: 0.9318\n",
            "Epoch 31/400\n",
            "132/132 [==============================] - 0s 24us/step - loss: 0.3050 - acc: 0.9318\n",
            "Epoch 32/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.3043 - acc: 0.9318\n",
            "Epoch 33/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.3035 - acc: 0.9318\n",
            "Epoch 34/400\n",
            "132/132 [==============================] - 0s 19us/step - loss: 0.3027 - acc: 0.9394\n",
            "Epoch 35/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.3020 - acc: 0.9470\n",
            "Epoch 36/400\n",
            "132/132 [==============================] - 0s 23us/step - loss: 0.3012 - acc: 0.9470\n",
            "Epoch 37/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.3005 - acc: 0.9470\n",
            "Epoch 38/400\n",
            "132/132 [==============================] - 0s 18us/step - loss: 0.2997 - acc: 0.9470\n",
            "Epoch 39/400\n",
            "132/132 [==============================] - 0s 19us/step - loss: 0.2990 - acc: 0.9470\n",
            "Epoch 40/400\n",
            "132/132 [==============================] - 0s 26us/step - loss: 0.2982 - acc: 0.9470\n",
            "Epoch 41/400\n",
            "132/132 [==============================] - 0s 56us/step - loss: 0.2975 - acc: 0.9470\n",
            "Epoch 42/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.2967 - acc: 0.9470\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 43/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.2960 - acc: 0.9470\n",
            "Epoch 44/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.2953 - acc: 0.9470\n",
            "Epoch 45/400\n",
            "132/132 [==============================] - 0s 22us/step - loss: 0.2946 - acc: 0.9470\n",
            "Epoch 46/400\n",
            "132/132 [==============================] - 0s 23us/step - loss: 0.2938 - acc: 0.9470\n",
            "Epoch 47/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.2931 - acc: 0.9470\n",
            "Epoch 48/400\n",
            "132/132 [==============================] - 0s 26us/step - loss: 0.2924 - acc: 0.9470\n",
            "Epoch 49/400\n",
            "132/132 [==============================] - 0s 40us/step - loss: 0.2917 - acc: 0.9470\n",
            "Epoch 50/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.2910 - acc: 0.9470\n",
            "Epoch 51/400\n",
            "132/132 [==============================] - 0s 46us/step - loss: 0.2903 - acc: 0.9470\n",
            "Epoch 52/400\n",
            "132/132 [==============================] - 0s 28us/step - loss: 0.2895 - acc: 0.9545\n",
            "Epoch 53/400\n",
            "132/132 [==============================] - 0s 30us/step - loss: 0.2888 - acc: 0.9545\n",
            "Epoch 54/400\n",
            "132/132 [==============================] - 0s 18us/step - loss: 0.2881 - acc: 0.9545\n",
            "Epoch 55/400\n",
            "132/132 [==============================] - 0s 24us/step - loss: 0.2874 - acc: 0.9545\n",
            "Epoch 56/400\n",
            "132/132 [==============================] - 0s 18us/step - loss: 0.2868 - acc: 0.9545\n",
            "Epoch 57/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.2861 - acc: 0.9545\n",
            "Epoch 58/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.2854 - acc: 0.9545\n",
            "Epoch 59/400\n",
            "132/132 [==============================] - 0s 19us/step - loss: 0.2847 - acc: 0.9545\n",
            "Epoch 60/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.2840 - acc: 0.9545\n",
            "Epoch 61/400\n",
            "132/132 [==============================] - 0s 17us/step - loss: 0.2833 - acc: 0.9545\n",
            "Epoch 62/400\n",
            "132/132 [==============================] - 0s 18us/step - loss: 0.2826 - acc: 0.9545\n",
            "Epoch 63/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.2820 - acc: 0.9545\n",
            "Epoch 64/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.2813 - acc: 0.9545\n",
            "Epoch 65/400\n",
            "132/132 [==============================] - 0s 46us/step - loss: 0.2806 - acc: 0.9545\n",
            "Epoch 66/400\n",
            "132/132 [==============================] - 0s 22us/step - loss: 0.2800 - acc: 0.9545\n",
            "Epoch 67/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.2793 - acc: 0.9545\n",
            "Epoch 68/400\n",
            "132/132 [==============================] - 0s 38us/step - loss: 0.2786 - acc: 0.9545\n",
            "Epoch 69/400\n",
            "132/132 [==============================] - 0s 33us/step - loss: 0.2780 - acc: 0.9545\n",
            "Epoch 70/400\n",
            "132/132 [==============================] - 0s 30us/step - loss: 0.2773 - acc: 0.9545\n",
            "Epoch 71/400\n",
            "132/132 [==============================] - 0s 26us/step - loss: 0.2767 - acc: 0.9545\n",
            "Epoch 72/400\n",
            "132/132 [==============================] - 0s 34us/step - loss: 0.2760 - acc: 0.9545\n",
            "Epoch 73/400\n",
            "132/132 [==============================] - 0s 29us/step - loss: 0.2754 - acc: 0.9545\n",
            "Epoch 74/400\n",
            "132/132 [==============================] - 0s 38us/step - loss: 0.2747 - acc: 0.9545\n",
            "Epoch 75/400\n",
            "132/132 [==============================] - 0s 47us/step - loss: 0.2741 - acc: 0.9545\n",
            "Epoch 76/400\n",
            "132/132 [==============================] - 0s 26us/step - loss: 0.2734 - acc: 0.9545\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 77/400\n",
            "132/132 [==============================] - 0s 30us/step - loss: 0.2728 - acc: 0.9545\n",
            "Epoch 78/400\n",
            "132/132 [==============================] - 0s 28us/step - loss: 0.2722 - acc: 0.9545\n",
            "Epoch 79/400\n",
            "132/132 [==============================] - 0s 32us/step - loss: 0.2715 - acc: 0.9545\n",
            "Epoch 80/400\n",
            "132/132 [==============================] - 0s 28us/step - loss: 0.2709 - acc: 0.9545\n",
            "Epoch 81/400\n",
            "132/132 [==============================] - 0s 41us/step - loss: 0.2703 - acc: 0.9545\n",
            "Epoch 82/400\n",
            "132/132 [==============================] - 0s 48us/step - loss: 0.2697 - acc: 0.9545\n",
            "Epoch 83/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.2690 - acc: 0.9545\n",
            "Epoch 84/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.2684 - acc: 0.9545\n",
            "Epoch 85/400\n",
            "132/132 [==============================] - 0s 34us/step - loss: 0.2678 - acc: 0.9545\n",
            "Epoch 86/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.2672 - acc: 0.9545\n",
            "Epoch 87/400\n",
            "132/132 [==============================] - 0s 16us/step - loss: 0.2666 - acc: 0.9545\n",
            "Epoch 88/400\n",
            "132/132 [==============================] - 0s 18us/step - loss: 0.2660 - acc: 0.9545\n",
            "Epoch 89/400\n",
            "132/132 [==============================] - 0s 19us/step - loss: 0.2654 - acc: 0.9545\n",
            "Epoch 90/400\n",
            "132/132 [==============================] - 0s 53us/step - loss: 0.2647 - acc: 0.9545\n",
            "Epoch 91/400\n",
            "132/132 [==============================] - 0s 38us/step - loss: 0.2641 - acc: 0.9545\n",
            "Epoch 92/400\n",
            "132/132 [==============================] - 0s 41us/step - loss: 0.2635 - acc: 0.9545\n",
            "Epoch 93/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.2629 - acc: 0.9545\n",
            "Epoch 94/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.2623 - acc: 0.9545\n",
            "Epoch 95/400\n",
            "132/132 [==============================] - 0s 28us/step - loss: 0.2618 - acc: 0.9545\n",
            "Epoch 96/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.2612 - acc: 0.9545\n",
            "Epoch 97/400\n",
            "132/132 [==============================] - 0s 26us/step - loss: 0.2606 - acc: 0.9545\n",
            "Epoch 98/400\n",
            "132/132 [==============================] - 0s 29us/step - loss: 0.2600 - acc: 0.9545\n",
            "Epoch 99/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.2594 - acc: 0.9545\n",
            "Epoch 100/400\n",
            "132/132 [==============================] - 0s 31us/step - loss: 0.2588 - acc: 0.9545\n",
            "Epoch 101/400\n",
            "132/132 [==============================] - 0s 22us/step - loss: 0.2582 - acc: 0.9545\n",
            "Epoch 102/400\n",
            "132/132 [==============================] - 0s 31us/step - loss: 0.2577 - acc: 0.9545\n",
            "Epoch 103/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.2571 - acc: 0.9545\n",
            "Epoch 104/400\n",
            "132/132 [==============================] - 0s 28us/step - loss: 0.2565 - acc: 0.9545\n",
            "Epoch 105/400\n",
            "132/132 [==============================] - 0s 24us/step - loss: 0.2559 - acc: 0.9545\n",
            "Epoch 106/400\n",
            "132/132 [==============================] - 0s 24us/step - loss: 0.2554 - acc: 0.9545\n",
            "Epoch 107/400\n",
            "132/132 [==============================] - 0s 33us/step - loss: 0.2548 - acc: 0.9545\n",
            "Epoch 108/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.2542 - acc: 0.9545\n",
            "Epoch 109/400\n",
            "132/132 [==============================] - 0s 23us/step - loss: 0.2537 - acc: 0.9545\n",
            "Epoch 110/400\n",
            "132/132 [==============================] - 0s 26us/step - loss: 0.2531 - acc: 0.9545\n",
            "Epoch 111/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.2526 - acc: 0.9545\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 112/400\n",
            "132/132 [==============================] - 0s 36us/step - loss: 0.2520 - acc: 0.9545\n",
            "Epoch 113/400\n",
            "132/132 [==============================] - 0s 22us/step - loss: 0.2514 - acc: 0.9545\n",
            "Epoch 114/400\n",
            "132/132 [==============================] - 0s 35us/step - loss: 0.2509 - acc: 0.9545\n",
            "Epoch 115/400\n",
            "132/132 [==============================] - 0s 28us/step - loss: 0.2503 - acc: 0.9545\n",
            "Epoch 116/400\n",
            "132/132 [==============================] - 0s 19us/step - loss: 0.2498 - acc: 0.9545\n",
            "Epoch 117/400\n",
            "132/132 [==============================] - 0s 19us/step - loss: 0.2492 - acc: 0.9545\n",
            "Epoch 118/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.2487 - acc: 0.9545\n",
            "Epoch 119/400\n",
            "132/132 [==============================] - 0s 29us/step - loss: 0.2482 - acc: 0.9545\n",
            "Epoch 120/400\n",
            "132/132 [==============================] - 0s 18us/step - loss: 0.2476 - acc: 0.9545\n",
            "Epoch 121/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.2471 - acc: 0.9545\n",
            "Epoch 122/400\n",
            "132/132 [==============================] - 0s 98us/step - loss: 0.2465 - acc: 0.9545\n",
            "Epoch 123/400\n",
            "132/132 [==============================] - 0s 42us/step - loss: 0.2460 - acc: 0.9545\n",
            "Epoch 124/400\n",
            "132/132 [==============================] - 0s 35us/step - loss: 0.2455 - acc: 0.9545\n",
            "Epoch 125/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.2449 - acc: 0.9545\n",
            "Epoch 126/400\n",
            "132/132 [==============================] - 0s 44us/step - loss: 0.2444 - acc: 0.9621\n",
            "Epoch 127/400\n",
            "132/132 [==============================] - 0s 22us/step - loss: 0.2439 - acc: 0.9621\n",
            "Epoch 128/400\n",
            "132/132 [==============================] - 0s 34us/step - loss: 0.2433 - acc: 0.9621\n",
            "Epoch 129/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.2428 - acc: 0.9621\n",
            "Epoch 130/400\n",
            "132/132 [==============================] - 0s 22us/step - loss: 0.2423 - acc: 0.9621\n",
            "Epoch 131/400\n",
            "132/132 [==============================] - 0s 44us/step - loss: 0.2418 - acc: 0.9621\n",
            "Epoch 132/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.2413 - acc: 0.9621\n",
            "Epoch 133/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.2407 - acc: 0.9621\n",
            "Epoch 134/400\n",
            "132/132 [==============================] - 0s 29us/step - loss: 0.2402 - acc: 0.9621\n",
            "Epoch 135/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.2397 - acc: 0.9621\n",
            "Epoch 136/400\n",
            "132/132 [==============================] - 0s 32us/step - loss: 0.2392 - acc: 0.9621\n",
            "Epoch 137/400\n",
            "132/132 [==============================] - 0s 34us/step - loss: 0.2387 - acc: 0.9621\n",
            "Epoch 138/400\n",
            "132/132 [==============================] - 0s 28us/step - loss: 0.2382 - acc: 0.9621\n",
            "Epoch 139/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.2377 - acc: 0.9621\n",
            "Epoch 140/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.2372 - acc: 0.9621\n",
            "Epoch 141/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.2367 - acc: 0.9621\n",
            "Epoch 142/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.2362 - acc: 0.9621\n",
            "Epoch 143/400\n",
            "132/132 [==============================] - 0s 30us/step - loss: 0.2357 - acc: 0.9621\n",
            "Epoch 144/400\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "132/132 [==============================] - 0s 36us/step - loss: 0.2352 - acc: 0.9621\n",
            "Epoch 145/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.2347 - acc: 0.9621\n",
            "Epoch 146/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.2342 - acc: 0.9621\n",
            "Epoch 147/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.2337 - acc: 0.9621\n",
            "Epoch 148/400\n",
            "132/132 [==============================] - 0s 29us/step - loss: 0.2332 - acc: 0.9621\n",
            "Epoch 149/400\n",
            "132/132 [==============================] - 0s 26us/step - loss: 0.2327 - acc: 0.9621\n",
            "Epoch 150/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.2322 - acc: 0.9621\n",
            "Epoch 151/400\n",
            "132/132 [==============================] - 0s 33us/step - loss: 0.2317 - acc: 0.9621\n",
            "Epoch 152/400\n",
            "132/132 [==============================] - 0s 46us/step - loss: 0.2313 - acc: 0.9621\n",
            "Epoch 153/400\n",
            "132/132 [==============================] - 0s 31us/step - loss: 0.2308 - acc: 0.9621\n",
            "Epoch 154/400\n",
            "132/132 [==============================] - 0s 33us/step - loss: 0.2303 - acc: 0.9621\n",
            "Epoch 155/400\n",
            "132/132 [==============================] - 0s 43us/step - loss: 0.2298 - acc: 0.9621\n",
            "Epoch 156/400\n",
            "132/132 [==============================] - 0s 34us/step - loss: 0.2293 - acc: 0.9621\n",
            "Epoch 157/400\n",
            "132/132 [==============================] - 0s 36us/step - loss: 0.2289 - acc: 0.9621\n",
            "Epoch 158/400\n",
            "132/132 [==============================] - 0s 38us/step - loss: 0.2284 - acc: 0.9621\n",
            "Epoch 159/400\n",
            "132/132 [==============================] - 0s 43us/step - loss: 0.2279 - acc: 0.9621\n",
            "Epoch 160/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.2274 - acc: 0.9621\n",
            "Epoch 161/400\n",
            "132/132 [==============================] - 0s 43us/step - loss: 0.2270 - acc: 0.9621\n",
            "Epoch 162/400\n",
            "132/132 [==============================] - 0s 48us/step - loss: 0.2265 - acc: 0.9621\n",
            "Epoch 163/400\n",
            "132/132 [==============================] - 0s 48us/step - loss: 0.2260 - acc: 0.9621\n",
            "Epoch 164/400\n",
            "132/132 [==============================] - 0s 31us/step - loss: 0.2256 - acc: 0.9621\n",
            "Epoch 165/400\n",
            "132/132 [==============================] - 0s 31us/step - loss: 0.2251 - acc: 0.9621\n",
            "Epoch 166/400\n",
            "132/132 [==============================] - 0s 44us/step - loss: 0.2246 - acc: 0.9621\n",
            "Epoch 167/400\n",
            "132/132 [==============================] - 0s 41us/step - loss: 0.2242 - acc: 0.9621\n",
            "Epoch 168/400\n",
            "132/132 [==============================] - 0s 40us/step - loss: 0.2237 - acc: 0.9621\n",
            "Epoch 169/400\n",
            "132/132 [==============================] - 0s 49us/step - loss: 0.2233 - acc: 0.9621\n",
            "Epoch 170/400\n",
            "132/132 [==============================] - 0s 33us/step - loss: 0.2228 - acc: 0.9621\n",
            "Epoch 171/400\n",
            "132/132 [==============================] - 0s 46us/step - loss: 0.2224 - acc: 0.9621\n",
            "Epoch 172/400\n",
            "132/132 [==============================] - 0s 26us/step - loss: 0.2219 - acc: 0.9621\n",
            "Epoch 173/400\n",
            "132/132 [==============================] - 0s 42us/step - loss: 0.2214 - acc: 0.9621\n",
            "Epoch 174/400\n",
            "132/132 [==============================] - 0s 39us/step - loss: 0.2210 - acc: 0.9621\n",
            "Epoch 175/400\n",
            "132/132 [==============================] - 0s 37us/step - loss: 0.2205 - acc: 0.9621\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 176/400\n",
            "132/132 [==============================] - 0s 48us/step - loss: 0.2201 - acc: 0.9621\n",
            "Epoch 177/400\n",
            "132/132 [==============================] - 0s 37us/step - loss: 0.2197 - acc: 0.9621\n",
            "Epoch 178/400\n",
            "132/132 [==============================] - 0s 32us/step - loss: 0.2192 - acc: 0.9621\n",
            "Epoch 179/400\n",
            "132/132 [==============================] - 0s 34us/step - loss: 0.2188 - acc: 0.9621\n",
            "Epoch 180/400\n",
            "132/132 [==============================] - 0s 40us/step - loss: 0.2183 - acc: 0.9621\n",
            "Epoch 181/400\n",
            "132/132 [==============================] - 0s 29us/step - loss: 0.2179 - acc: 0.9621\n",
            "Epoch 182/400\n",
            "132/132 [==============================] - 0s 38us/step - loss: 0.2174 - acc: 0.9621\n",
            "Epoch 183/400\n",
            "132/132 [==============================] - 0s 42us/step - loss: 0.2170 - acc: 0.9621\n",
            "Epoch 184/400\n",
            "132/132 [==============================] - 0s 30us/step - loss: 0.2166 - acc: 0.9621\n",
            "Epoch 185/400\n",
            "132/132 [==============================] - 0s 23us/step - loss: 0.2161 - acc: 0.9621\n",
            "Epoch 186/400\n",
            "132/132 [==============================] - 0s 36us/step - loss: 0.2157 - acc: 0.9621\n",
            "Epoch 187/400\n",
            "132/132 [==============================] - 0s 34us/step - loss: 0.2153 - acc: 0.9621\n",
            "Epoch 188/400\n",
            "132/132 [==============================] - 0s 34us/step - loss: 0.2148 - acc: 0.9621\n",
            "Epoch 189/400\n",
            "132/132 [==============================] - 0s 31us/step - loss: 0.2144 - acc: 0.9621\n",
            "Epoch 190/400\n",
            "132/132 [==============================] - 0s 26us/step - loss: 0.2140 - acc: 0.9621\n",
            "Epoch 191/400\n",
            "132/132 [==============================] - 0s 34us/step - loss: 0.2135 - acc: 0.9621\n",
            "Epoch 192/400\n",
            "132/132 [==============================] - 0s 32us/step - loss: 0.2131 - acc: 0.9621\n",
            "Epoch 193/400\n",
            "132/132 [==============================] - 0s 39us/step - loss: 0.2127 - acc: 0.9621\n",
            "Epoch 194/400\n",
            "132/132 [==============================] - 0s 37us/step - loss: 0.2123 - acc: 0.9621\n",
            "Epoch 195/400\n",
            "132/132 [==============================] - 0s 36us/step - loss: 0.2118 - acc: 0.9621\n",
            "Epoch 196/400\n",
            "132/132 [==============================] - 0s 53us/step - loss: 0.2114 - acc: 0.9621\n",
            "Epoch 197/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.2110 - acc: 0.9621\n",
            "Epoch 198/400\n",
            "132/132 [==============================] - 0s 22us/step - loss: 0.2106 - acc: 0.9621\n",
            "Epoch 199/400\n",
            "132/132 [==============================] - 0s 30us/step - loss: 0.2102 - acc: 0.9697\n",
            "Epoch 200/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.2098 - acc: 0.9697\n",
            "Epoch 201/400\n",
            "132/132 [==============================] - 0s 23us/step - loss: 0.2093 - acc: 0.9697\n",
            "Epoch 202/400\n",
            "132/132 [==============================] - 0s 19us/step - loss: 0.2089 - acc: 0.9697\n",
            "Epoch 203/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.2085 - acc: 0.9697\n",
            "Epoch 204/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.2081 - acc: 0.9697\n",
            "Epoch 205/400\n",
            "132/132 [==============================] - 0s 19us/step - loss: 0.2077 - acc: 0.9697\n",
            "Epoch 206/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.2073 - acc: 0.9697\n",
            "Epoch 207/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.2069 - acc: 0.9697\n",
            "Epoch 208/400\n",
            "132/132 [==============================] - 0s 29us/step - loss: 0.2065 - acc: 0.9697\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 209/400\n",
            "132/132 [==============================] - 0s 37us/step - loss: 0.2061 - acc: 0.9697\n",
            "Epoch 210/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.2057 - acc: 0.9697\n",
            "Epoch 211/400\n",
            "132/132 [==============================] - 0s 29us/step - loss: 0.2053 - acc: 0.9697\n",
            "Epoch 212/400\n",
            "132/132 [==============================] - 0s 34us/step - loss: 0.2049 - acc: 0.9697\n",
            "Epoch 213/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.2045 - acc: 0.9697\n",
            "Epoch 214/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.2041 - acc: 0.9697\n",
            "Epoch 215/400\n",
            "132/132 [==============================] - 0s 34us/step - loss: 0.2037 - acc: 0.9697\n",
            "Epoch 216/400\n",
            "132/132 [==============================] - 0s 23us/step - loss: 0.2033 - acc: 0.9697\n",
            "Epoch 217/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.2029 - acc: 0.9697\n",
            "Epoch 218/400\n",
            "132/132 [==============================] - 0s 23us/step - loss: 0.2025 - acc: 0.9697\n",
            "Epoch 219/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.2021 - acc: 0.9697\n",
            "Epoch 220/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.2017 - acc: 0.9697\n",
            "Epoch 221/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.2013 - acc: 0.9697\n",
            "Epoch 222/400\n",
            "132/132 [==============================] - 0s 22us/step - loss: 0.2009 - acc: 0.9697\n",
            "Epoch 223/400\n",
            "132/132 [==============================] - 0s 30us/step - loss: 0.2005 - acc: 0.9697\n",
            "Epoch 224/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.2001 - acc: 0.9697\n",
            "Epoch 225/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.1997 - acc: 0.9697\n",
            "Epoch 226/400\n",
            "132/132 [==============================] - 0s 23us/step - loss: 0.1994 - acc: 0.9697\n",
            "Epoch 227/400\n",
            "132/132 [==============================] - 0s 24us/step - loss: 0.1990 - acc: 0.9697\n",
            "Epoch 228/400\n",
            "132/132 [==============================] - 0s 31us/step - loss: 0.1986 - acc: 0.9697\n",
            "Epoch 229/400\n",
            "132/132 [==============================] - 0s 28us/step - loss: 0.1982 - acc: 0.9697\n",
            "Epoch 230/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.1978 - acc: 0.9697\n",
            "Epoch 231/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.1974 - acc: 0.9697\n",
            "Epoch 232/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.1971 - acc: 0.9697\n",
            "Epoch 233/400\n",
            "132/132 [==============================] - 0s 38us/step - loss: 0.1967 - acc: 0.9697\n",
            "Epoch 234/400\n",
            "132/132 [==============================] - 0s 26us/step - loss: 0.1963 - acc: 0.9697\n",
            "Epoch 235/400\n",
            "132/132 [==============================] - 0s 24us/step - loss: 0.1959 - acc: 0.9697\n",
            "Epoch 236/400\n",
            "132/132 [==============================] - 0s 24us/step - loss: 0.1956 - acc: 0.9697\n",
            "Epoch 237/400\n",
            "132/132 [==============================] - 0s 34us/step - loss: 0.1952 - acc: 0.9697\n",
            "Epoch 238/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.1948 - acc: 0.9697\n",
            "Epoch 239/400\n",
            "132/132 [==============================] - 0s 28us/step - loss: 0.1944 - acc: 0.9697\n",
            "Epoch 240/400\n",
            "132/132 [==============================] - 0s 31us/step - loss: 0.1941 - acc: 0.9697\n",
            "Epoch 241/400\n",
            "132/132 [==============================] - 0s 40us/step - loss: 0.1937 - acc: 0.9697\n",
            "Epoch 242/400"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "\r132/132 [==============================] - 0s 27us/step - loss: 0.1933 - acc: 0.9697\n",
            "Epoch 243/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.1930 - acc: 0.9697\n",
            "Epoch 244/400\n",
            "132/132 [==============================] - 0s 32us/step - loss: 0.1926 - acc: 0.9697\n",
            "Epoch 245/400\n",
            "132/132 [==============================] - 0s 26us/step - loss: 0.1922 - acc: 0.9697\n",
            "Epoch 246/400\n",
            "132/132 [==============================] - 0s 30us/step - loss: 0.1919 - acc: 0.9697\n",
            "Epoch 247/400\n",
            "132/132 [==============================] - 0s 29us/step - loss: 0.1915 - acc: 0.9697\n",
            "Epoch 248/400\n",
            "132/132 [==============================] - 0s 35us/step - loss: 0.1911 - acc: 0.9697\n",
            "Epoch 249/400\n",
            "132/132 [==============================] - 0s 28us/step - loss: 0.1908 - acc: 0.9697\n",
            "Epoch 250/400\n",
            "132/132 [==============================] - 0s 35us/step - loss: 0.1904 - acc: 0.9697\n",
            "Epoch 251/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.1901 - acc: 0.9697\n",
            "Epoch 252/400\n",
            "132/132 [==============================] - 0s 28us/step - loss: 0.1897 - acc: 0.9697\n",
            "Epoch 253/400\n",
            "132/132 [==============================] - 0s 33us/step - loss: 0.1893 - acc: 0.9697\n",
            "Epoch 254/400\n",
            "132/132 [==============================] - 0s 28us/step - loss: 0.1890 - acc: 0.9697\n",
            "Epoch 255/400\n",
            "132/132 [==============================] - 0s 32us/step - loss: 0.1886 - acc: 0.9697\n",
            "Epoch 256/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.1883 - acc: 0.9697\n",
            "Epoch 257/400\n",
            "132/132 [==============================] - 0s 19us/step - loss: 0.1879 - acc: 0.9697\n",
            "Epoch 258/400\n",
            "132/132 [==============================] - 0s 23us/step - loss: 0.1876 - acc: 0.9697\n",
            "Epoch 259/400\n",
            "132/132 [==============================] - 0s 37us/step - loss: 0.1872 - acc: 0.9697\n",
            "Epoch 260/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.1869 - acc: 0.9697\n",
            "Epoch 261/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.1865 - acc: 0.9697\n",
            "Epoch 262/400\n",
            "132/132 [==============================] - 0s 22us/step - loss: 0.1862 - acc: 0.9697\n",
            "Epoch 263/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.1858 - acc: 0.9697\n",
            "Epoch 264/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.1855 - acc: 0.9697\n",
            "Epoch 265/400\n",
            "132/132 [==============================] - 0s 26us/step - loss: 0.1851 - acc: 0.9697\n",
            "Epoch 266/400\n",
            "132/132 [==============================] - 0s 33us/step - loss: 0.1848 - acc: 0.9697\n",
            "Epoch 267/400\n",
            "132/132 [==============================] - 0s 23us/step - loss: 0.1844 - acc: 0.9773\n",
            "Epoch 268/400\n",
            "132/132 [==============================] - 0s 31us/step - loss: 0.1841 - acc: 0.9773\n",
            "Epoch 269/400\n",
            "132/132 [==============================] - 0s 30us/step - loss: 0.1837 - acc: 0.9773\n",
            "Epoch 270/400\n",
            "132/132 [==============================] - 0s 31us/step - loss: 0.1834 - acc: 0.9773\n",
            "Epoch 271/400\n",
            "132/132 [==============================] - 0s 39us/step - loss: 0.1830 - acc: 0.9773\n",
            "Epoch 272/400\n",
            "132/132 [==============================] - 0s 31us/step - loss: 0.1827 - acc: 0.9773\n",
            "Epoch 273/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.1824 - acc: 0.9773\n",
            "Epoch 274/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.1820 - acc: 0.9773\n",
            "Epoch 275/400\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "132/132 [==============================] - 0s 31us/step - loss: 0.1817 - acc: 0.9773\n",
            "Epoch 276/400\n",
            "132/132 [==============================] - 0s 19us/step - loss: 0.1814 - acc: 0.9773\n",
            "Epoch 277/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.1810 - acc: 0.9773\n",
            "Epoch 278/400\n",
            "132/132 [==============================] - 0s 30us/step - loss: 0.1807 - acc: 0.9773\n",
            "Epoch 279/400\n",
            "132/132 [==============================] - 0s 17us/step - loss: 0.1803 - acc: 0.9773\n",
            "Epoch 280/400\n",
            "132/132 [==============================] - 0s 43us/step - loss: 0.1800 - acc: 0.9773\n",
            "Epoch 281/400\n",
            "132/132 [==============================] - 0s 51us/step - loss: 0.1797 - acc: 0.9773\n",
            "Epoch 282/400\n",
            "132/132 [==============================] - 0s 43us/step - loss: 0.1793 - acc: 0.9773\n",
            "Epoch 283/400\n",
            "132/132 [==============================] - 0s 53us/step - loss: 0.1790 - acc: 0.9773\n",
            "Epoch 284/400\n",
            "132/132 [==============================] - 0s 48us/step - loss: 0.1787 - acc: 0.9773\n",
            "Epoch 285/400\n",
            "132/132 [==============================] - 0s 32us/step - loss: 0.1784 - acc: 0.9773\n",
            "Epoch 286/400\n",
            "132/132 [==============================] - 0s 23us/step - loss: 0.1780 - acc: 0.9773\n",
            "Epoch 287/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.1777 - acc: 0.9773\n",
            "Epoch 288/400\n",
            "132/132 [==============================] - 0s 31us/step - loss: 0.1774 - acc: 0.9773\n",
            "Epoch 289/400\n",
            "132/132 [==============================] - 0s 33us/step - loss: 0.1770 - acc: 0.9773\n",
            "Epoch 290/400\n",
            "132/132 [==============================] - 0s 43us/step - loss: 0.1767 - acc: 0.9773\n",
            "Epoch 291/400\n",
            "132/132 [==============================] - 0s 33us/step - loss: 0.1764 - acc: 0.9773\n",
            "Epoch 292/400\n",
            "132/132 [==============================] - 0s 29us/step - loss: 0.1761 - acc: 0.9773\n",
            "Epoch 293/400\n",
            "132/132 [==============================] - 0s 41us/step - loss: 0.1758 - acc: 0.9773\n",
            "Epoch 294/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.1754 - acc: 0.9773\n",
            "Epoch 295/400\n",
            "132/132 [==============================] - 0s 44us/step - loss: 0.1751 - acc: 0.9773\n",
            "Epoch 296/400\n",
            "132/132 [==============================] - 0s 35us/step - loss: 0.1748 - acc: 0.9773\n",
            "Epoch 297/400\n",
            "132/132 [==============================] - 0s 32us/step - loss: 0.1745 - acc: 0.9773\n",
            "Epoch 298/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.1742 - acc: 0.9773\n",
            "Epoch 299/400\n",
            "132/132 [==============================] - 0s 32us/step - loss: 0.1738 - acc: 0.9773\n",
            "Epoch 300/400\n",
            "132/132 [==============================] - 0s 33us/step - loss: 0.1735 - acc: 0.9773\n",
            "Epoch 301/400\n",
            "132/132 [==============================] - 0s 24us/step - loss: 0.1732 - acc: 0.9773\n",
            "Epoch 302/400\n",
            "132/132 [==============================] - 0s 31us/step - loss: 0.1729 - acc: 0.9773\n",
            "Epoch 303/400\n",
            "132/132 [==============================] - 0s 36us/step - loss: 0.1726 - acc: 0.9773\n",
            "Epoch 304/400\n",
            "132/132 [==============================] - 0s 32us/step - loss: 0.1723 - acc: 0.9773\n",
            "Epoch 305/400\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\r132/132 [==============================] - 0s 45us/step - loss: 0.1719 - acc: 0.9773\n",
            "Epoch 306/400\n",
            "132/132 [==============================] - 0s 34us/step - loss: 0.1716 - acc: 0.9773\n",
            "Epoch 307/400\n",
            "132/132 [==============================] - 0s 32us/step - loss: 0.1713 - acc: 0.9773\n",
            "Epoch 308/400\n",
            "132/132 [==============================] - 0s 22us/step - loss: 0.1710 - acc: 0.9773\n",
            "Epoch 309/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.1707 - acc: 0.9773\n",
            "Epoch 310/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.1704 - acc: 0.9773\n",
            "Epoch 311/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.1701 - acc: 0.9773\n",
            "Epoch 312/400\n",
            "132/132 [==============================] - 0s 31us/step - loss: 0.1698 - acc: 0.9773\n",
            "Epoch 313/400\n",
            "132/132 [==============================] - 0s 38us/step - loss: 0.1695 - acc: 0.9773\n",
            "Epoch 314/400\n",
            "132/132 [==============================] - 0s 36us/step - loss: 0.1692 - acc: 0.9773\n",
            "Epoch 315/400\n",
            "132/132 [==============================] - 0s 29us/step - loss: 0.1689 - acc: 0.9773\n",
            "Epoch 316/400\n",
            "132/132 [==============================] - 0s 35us/step - loss: 0.1685 - acc: 0.9773\n",
            "Epoch 317/400\n",
            "132/132 [==============================] - 0s 26us/step - loss: 0.1682 - acc: 0.9773\n",
            "Epoch 318/400\n",
            "132/132 [==============================] - 0s 35us/step - loss: 0.1679 - acc: 0.9773\n",
            "Epoch 319/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.1676 - acc: 0.9773\n",
            "Epoch 320/400\n",
            "132/132 [==============================] - 0s 30us/step - loss: 0.1673 - acc: 0.9773\n",
            "Epoch 321/400\n",
            "132/132 [==============================] - 0s 23us/step - loss: 0.1670 - acc: 0.9773\n",
            "Epoch 322/400\n",
            "132/132 [==============================] - 0s 33us/step - loss: 0.1667 - acc: 0.9773\n",
            "Epoch 323/400\n",
            "132/132 [==============================] - 0s 23us/step - loss: 0.1664 - acc: 0.9773\n",
            "Epoch 324/400\n",
            "132/132 [==============================] - 0s 34us/step - loss: 0.1661 - acc: 0.9773\n",
            "Epoch 325/400\n",
            "132/132 [==============================] - 0s 22us/step - loss: 0.1658 - acc: 0.9773\n",
            "Epoch 326/400\n",
            "132/132 [==============================] - 0s 22us/step - loss: 0.1655 - acc: 0.9773\n",
            "Epoch 327/400\n",
            "132/132 [==============================] - 0s 22us/step - loss: 0.1652 - acc: 0.9773\n",
            "Epoch 328/400\n",
            "132/132 [==============================] - 0s 30us/step - loss: 0.1649 - acc: 0.9773\n",
            "Epoch 329/400\n",
            "132/132 [==============================] - 0s 38us/step - loss: 0.1646 - acc: 0.9773\n",
            "Epoch 330/400\n",
            "132/132 [==============================] - 0s 37us/step - loss: 0.1644 - acc: 0.9773\n",
            "Epoch 331/400\n",
            "132/132 [==============================] - 0s 28us/step - loss: 0.1641 - acc: 0.9773\n",
            "Epoch 332/400\n",
            "132/132 [==============================] - 0s 39us/step - loss: 0.1638 - acc: 0.9773\n",
            "Epoch 333/400\n",
            "132/132 [==============================] - 0s 30us/step - loss: 0.1635 - acc: 0.9773\n",
            "Epoch 334/400\n",
            "132/132 [==============================] - 0s 26us/step - loss: 0.1632 - acc: 0.9773\n",
            "Epoch 335/400\n",
            "132/132 [==============================] - 0s 24us/step - loss: 0.1629 - acc: 0.9773\n",
            "Epoch 336/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.1626 - acc: 0.9773\n",
            "Epoch 337/400\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "132/132 [==============================] - 0s 27us/step - loss: 0.1623 - acc: 0.9773\n",
            "Epoch 338/400\n",
            "132/132 [==============================] - 0s 23us/step - loss: 0.1620 - acc: 0.9773\n",
            "Epoch 339/400\n",
            "132/132 [==============================] - 0s 30us/step - loss: 0.1617 - acc: 0.9773\n",
            "Epoch 340/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.1614 - acc: 0.9773\n",
            "Epoch 341/400\n",
            "132/132 [==============================] - 0s 34us/step - loss: 0.1612 - acc: 0.9773\n",
            "Epoch 342/400\n",
            "132/132 [==============================] - 0s 36us/step - loss: 0.1609 - acc: 0.9773\n",
            "Epoch 343/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.1606 - acc: 0.9773\n",
            "Epoch 344/400\n",
            "132/132 [==============================] - 0s 36us/step - loss: 0.1603 - acc: 0.9848\n",
            "Epoch 345/400\n",
            "132/132 [==============================] - 0s 36us/step - loss: 0.1600 - acc: 0.9848\n",
            "Epoch 346/400\n",
            "132/132 [==============================] - 0s 26us/step - loss: 0.1597 - acc: 0.9848\n",
            "Epoch 347/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.1594 - acc: 0.9848\n",
            "Epoch 348/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.1592 - acc: 0.9848\n",
            "Epoch 349/400\n",
            "132/132 [==============================] - 0s 30us/step - loss: 0.1589 - acc: 0.9848\n",
            "Epoch 350/400\n",
            "132/132 [==============================] - 0s 28us/step - loss: 0.1586 - acc: 0.9848\n",
            "Epoch 351/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.1583 - acc: 0.9848\n",
            "Epoch 352/400\n",
            "132/132 [==============================] - 0s 32us/step - loss: 0.1580 - acc: 0.9848\n",
            "Epoch 353/400\n",
            "132/132 [==============================] - 0s 23us/step - loss: 0.1578 - acc: 0.9848\n",
            "Epoch 354/400\n",
            "132/132 [==============================] - 0s 35us/step - loss: 0.1575 - acc: 0.9848\n",
            "Epoch 355/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.1572 - acc: 0.9848\n",
            "Epoch 356/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.1569 - acc: 0.9848\n",
            "Epoch 357/400\n",
            "132/132 [==============================] - 0s 23us/step - loss: 0.1566 - acc: 0.9848\n",
            "Epoch 358/400\n",
            "132/132 [==============================] - 0s 29us/step - loss: 0.1564 - acc: 0.9848\n",
            "Epoch 359/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.1561 - acc: 0.9848\n",
            "Epoch 360/400\n",
            "132/132 [==============================] - 0s 18us/step - loss: 0.1558 - acc: 0.9848\n",
            "Epoch 361/400\n",
            "132/132 [==============================] - 0s 23us/step - loss: 0.1555 - acc: 0.9848\n",
            "Epoch 362/400\n",
            "132/132 [==============================] - 0s 17us/step - loss: 0.1553 - acc: 0.9848\n",
            "Epoch 363/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.1550 - acc: 0.9848\n",
            "Epoch 364/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.1547 - acc: 0.9848\n",
            "Epoch 365/400\n",
            "132/132 [==============================] - 0s 26us/step - loss: 0.1545 - acc: 0.9848\n",
            "Epoch 366/400\n",
            "132/132 [==============================] - 0s 19us/step - loss: 0.1542 - acc: 0.9848\n",
            "Epoch 367/400\n",
            "132/132 [==============================] - 0s 21us/step - loss: 0.1539 - acc: 0.9848\n",
            "Epoch 368/400\n",
            "132/132 [==============================] - 0s 22us/step - loss: 0.1536 - acc: 0.9848\n",
            "Epoch 369/400\n",
            "132/132 [==============================] - 0s 41us/step - loss: 0.1534 - acc: 0.9848\n",
            "Epoch 370/400\n",
            "132/132 [==============================] - 0s 29us/step - loss: 0.1531 - acc: 0.9848\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 371/400\n",
            "\r132/132 [==============================] - 0s 29us/step - loss: 0.1528 - acc: 0.9848\n",
            "Epoch 372/400\n",
            "132/132 [==============================] - 0s 43us/step - loss: 0.1526 - acc: 0.9848\n",
            "Epoch 373/400\n",
            "132/132 [==============================] - 0s 28us/step - loss: 0.1523 - acc: 0.9848\n",
            "Epoch 374/400\n",
            "132/132 [==============================] - 0s 33us/step - loss: 0.1520 - acc: 0.9848\n",
            "Epoch 375/400\n",
            "132/132 [==============================] - 0s 29us/step - loss: 0.1518 - acc: 0.9848\n",
            "Epoch 376/400\n",
            "132/132 [==============================] - 0s 31us/step - loss: 0.1515 - acc: 0.9848\n",
            "Epoch 377/400\n",
            "132/132 [==============================] - 0s 23us/step - loss: 0.1512 - acc: 0.9848\n",
            "Epoch 378/400\n",
            "132/132 [==============================] - 0s 32us/step - loss: 0.1510 - acc: 0.9848\n",
            "Epoch 379/400\n",
            "132/132 [==============================] - 0s 38us/step - loss: 0.1507 - acc: 0.9848\n",
            "Epoch 380/400\n",
            "132/132 [==============================] - 0s 24us/step - loss: 0.1504 - acc: 0.9848\n",
            "Epoch 381/400\n",
            "132/132 [==============================] - 0s 32us/step - loss: 0.1502 - acc: 0.9848\n",
            "Epoch 382/400\n",
            "132/132 [==============================] - 0s 30us/step - loss: 0.1499 - acc: 0.9848\n",
            "Epoch 383/400\n",
            "132/132 [==============================] - 0s 31us/step - loss: 0.1497 - acc: 0.9848\n",
            "Epoch 384/400\n",
            "132/132 [==============================] - 0s 24us/step - loss: 0.1494 - acc: 0.9848\n",
            "Epoch 385/400\n",
            "132/132 [==============================] - 0s 33us/step - loss: 0.1491 - acc: 0.9848\n",
            "Epoch 386/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.1489 - acc: 0.9848\n",
            "Epoch 387/400\n",
            "132/132 [==============================] - 0s 33us/step - loss: 0.1486 - acc: 0.9848\n",
            "Epoch 388/400\n",
            "132/132 [==============================] - 0s 30us/step - loss: 0.1484 - acc: 0.9848\n",
            "Epoch 389/400\n",
            "132/132 [==============================] - 0s 20us/step - loss: 0.1481 - acc: 0.9848\n",
            "Epoch 390/400\n",
            "132/132 [==============================] - 0s 33us/step - loss: 0.1478 - acc: 0.9848\n",
            "Epoch 391/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.1476 - acc: 0.9848\n",
            "Epoch 392/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.1473 - acc: 0.9848\n",
            "Epoch 393/400\n",
            "132/132 [==============================] - 0s 31us/step - loss: 0.1471 - acc: 0.9848\n",
            "Epoch 394/400\n",
            "132/132 [==============================] - 0s 19us/step - loss: 0.1468 - acc: 0.9848\n",
            "Epoch 395/400\n",
            "132/132 [==============================] - 0s 23us/step - loss: 0.1466 - acc: 0.9848\n",
            "Epoch 396/400\n",
            "132/132 [==============================] - 0s 30us/step - loss: 0.1463 - acc: 0.9848\n",
            "Epoch 397/400\n",
            "132/132 [==============================] - 0s 25us/step - loss: 0.1461 - acc: 0.9848\n",
            "Epoch 398/400\n",
            "132/132 [==============================] - 0s 33us/step - loss: 0.1458 - acc: 0.9848\n",
            "Epoch 399/400\n",
            "132/132 [==============================] - 0s 27us/step - loss: 0.1456 - acc: 0.9848\n",
            "Epoch 400/400\n",
            "132/132 [==============================] - 0s 24us/step - loss: 0.1453 - acc: 0.9848\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f1ccc3ccc50>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 115
        }
      ]
    },
    {
      "metadata": {
        "id": "QYhXOajDgQ8W",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Testando a acuracia no conjunto de teste:"
      ]
    },
    {
      "metadata": {
        "id": "EK_ZekcWgD54",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "664db705-c532-4ff3-c708-469e84eb173d"
      },
      "cell_type": "code",
      "source": [
        "loss, accuracy = MLP.evaluate(avg_X_test, y_test_1hot, batch_size=avg_X_test.shape[0])\n",
        "print('Loss: {}, Acuracia: {}'.format(loss, accuracy))"
      ],
      "execution_count": 116,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\r56/56 [==============================] - 0s 593us/step\n",
            "Loss: 0.285441726446, Acuracia: 0.875\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "jWTB-kA1gnB4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Muito bom, a rede aprendeu bem nossos dados, vamos ver o que ela esta errando."
      ]
    },
    {
      "metadata": {
        "id": "QkDggzQ5gsnz",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 142
        },
        "outputId": "5a6aa08f-d043-41bb-9132-8fe25de0ce4d"
      },
      "cell_type": "code",
      "source": [
        "preds = MLP.predict(avg_X_test)\n",
        "for i, phrase in enumerate(X_test):\n",
        "    if np.argmax(preds[i]) != y_test[i]:\n",
        "        x = X_test[i]\n",
        "        y = y_test[i]\n",
        "        y_emoji = label_to_emoji(y)\n",
        "        y_hat = np.argmax(preds[i])\n",
        "        y_hat_emoji = label_to_emoji(y_hat)\n",
        "        print('Sentence: {}, Expected Label: {}, Predicted Label: {}'.format(x, y_emoji, y_hat_emoji))"
      ],
      "execution_count": 117,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sentence: work is hard, Expected Label: 😞, Predicted Label: 😄\n",
            "Sentence: This girl is messing with me, Expected Label: 😞, Predicted Label: ❤\n",
            "Sentence: work is horrible, Expected Label: 😞, Predicted Label: 😄\n",
            "Sentence: you brighten my day, Expected Label: 😄, Predicted Label: ❤\n",
            "Sentence: she is a bully, Expected Label: 😞, Predicted Label: ❤\n",
            "Sentence: My life is so boring, Expected Label: 😞, Predicted Label: ❤\n",
            "Sentence: will you be my valentine, Expected Label: 😄, Predicted Label: ❤\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "QHixjZPXhPiw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Agora repare nesta frase e no output da rede:"
      ]
    },
    {
      "metadata": {
        "id": "6ewXOjnuguG_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "ca0cdf26-f6d4-4570-897e-043dda578278"
      },
      "cell_type": "code",
      "source": [
        "def predict_emoji_MLP(model, sentence):\n",
        "    pred = model.predict(average_sentence(sentence).reshape(1,-1))\n",
        "    return label_to_emoji(np.argmax(pred))\n",
        "  \n",
        "print(predict_emoji_MLP(MLP, 'not feeling very happy'))"
      ],
      "execution_count": 118,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "😄\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "jh_OSWcWhZ1p",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "a926f436-d3d9-41d8-b05a-60b324c0dfbb"
      },
      "cell_type": "code",
      "source": [
        "print(predict_emoji_MLP(MLP, 'very happy not feeling'))"
      ],
      "execution_count": 119,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "😄\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "DJLDMlSkh-F1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Podemos ver que o 'not' nao esta sendo levado em conta, a MLP ve o happy e classifica emoji feliz.\n",
        "\n",
        "A ordem da frase tambem pouco importa, nosso modelo ve afinal a media das palavras.\n",
        "\n",
        "Um modelo de sequencia, que leve em conta a ordem temporal das palavras deve ser capaz de resolver estes problemas."
      ]
    },
    {
      "metadata": {
        "id": "U3YuDHvaiYd9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Modelo 2 - LSTM"
      ]
    },
    {
      "metadata": {
        "id": "tIb_pWGsmz-F",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Para o modelo LSTM cada palavra das frases sera utilizada individualmente em vez de tirado a média das palavras como na MLP. Isso se deve ao fato de que cada palavra pode ocupar um timestep da LSTM e ser usada para tentar descobrir uma hierarquia temporal entre as palavras treinando o modelo de sequencia.\n",
        "\n",
        "Por esse mesmo motivo, a LSTM nao tem problemas com frases de tamanhos variados, dado que cada observacao de treinamento pode ser treinada por um numero de timesteps equiavalente ao numero de palavras.\n",
        "\n",
        "A saida da LSTM tambem sera uma camada Dense seguida de uma ativacao softmax, assim como na MLP.\n",
        "\n",
        "Para fins de vetorizacao no Keras, embora em teoria a LSTM seja capaz de lidar com frases de tamanhos variados, sera feito um padding nas frases para que todas tenham um mesmo numero de palavras igual à maior frase. (por exemplo, se no conjunto de treino a maior frase tem 10 palavras, uma frase com 3 palavras tera 7 posicoes seguintes preenchidas com vetores de 0 uma vez que esta for transformada para o vetor de embeddings de 50 dimensoes). Vale ressaltar que isso so sera feito devido para ganhar desempenho na vetorizacao do treinamento com o keras."
      ]
    },
    {
      "metadata": {
        "id": "HETRa_nVmnaQ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#funcoes cujo objetivo e criar uma layer que mapea as palavras para seus embeddings no keras\n",
        "#alem disso as frases sao 0 padded ate max_len (que no caso do nosso dataset e 10)\n",
        "\n",
        "def sentence_to_index(X, word_to_index, max_len):\n",
        "    m = X.shape[0]\n",
        "    X_indices = np.zeros((m,max_len))\n",
        "    for i in range(m):\n",
        "        words = X[i].lower().split()[:max_len]\n",
        "        for j, word in enumerate(words):\n",
        "            X_indices[i, j] = word_to_index[word]\n",
        "    return X_indices\n",
        "\n",
        "def create_emb_matrix(word_to_vec_map, word_to_index):\n",
        "    vocab_len = len(word_to_index) + 1\n",
        "    emb_dim = 50 #glove vectors are 50D in our dataset\n",
        "    \n",
        "    emb_matrix = np.zeros((vocab_len, emb_dim))\n",
        "    for word, index in word_to_index.items():\n",
        "        emb_matrix[index,:] = word_to_vec_map[word]\n",
        "        \n",
        "    return emb_matrix\n",
        "\n",
        "def create_emb_layer(emb_matrix):\n",
        "    vocab_len, emb_dim = emb_matrix.shape\n",
        "    embedding_layer = Embedding(vocab_len, emb_dim, trainable=False)\n",
        "    embedding_layer.build((None,))\n",
        "    embedding_layer.set_weights([emb_matrix])\n",
        "    return embedding_layer\n",
        "\n",
        "emb_matrix = create_emb_matrix(word_to_vec_map, word_to_index)\n",
        "embedding_layer = create_emb_layer(emb_matrix)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "NZZzSkefqkjn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Com a camada de embeddings pronta para traduzir as frases para seus vetores e coloca-los corretamente na LSTM, implemente o modelo da LSTM.\n",
        "\n",
        "Este recebe 'embeddings' para uma camada LSTM de 128 neuronios (return_sequence=True, todos os outros parametros default), passa por uma tecnica de regularizacao chamada dropout com 0.5 de chance (nao se preocupe com isso, mas se quiser aprender sobre dropout saiba que e uma tecnica muito utilizada em deep learning, principalmente em visao computacional), depois uma outra camada LSTM de 128 neuronios (return_sequence=False, todos os outros parametros default), outro droput 0.5, uma camada densa de saida de 5 neuronios e uma ativacao softmax.\n",
        "\n",
        "documentacao sobre LSTM : https://keras.io/layers/recurrent/\n",
        "\n",
        "*oBS: entre a conecao das duas celulas de LSTM, ou seja, na primeira celula LSTM declarada, use return_sequences=True, que conecta cada timestep da primeira celula LSTM com a proxima.*\n",
        "\n",
        "*ja na segunda celula lstm return_sequences=False, que significa que so no ultimo timestep da LSTM existe uma saida, que seria pra previsao de y*\n",
        "\n",
        "![](https://github.com/Kulbear/deep-learning-coursera/raw/1beef8d17e804448c302b21a0b19833a7572f379/Sequence%20Models/images/emojifier-v2.png)"
      ]
    },
    {
      "metadata": {
        "id": "vGM9GFBBo7E-",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def LSTM_emoji(input_shape, embedding_layer):\n",
        "    indices = Input(input_shape, dtype='int32')\n",
        "    embeddings = embedding_layer(indices)\n",
        "    # SEU CODIGO AQUI\n",
        "    # Adiciona a primeira camada de LSTM aqui. Devera ter 128 neuronios,\n",
        "    # recebe embeddings e retorna X. Utilize return_sequences=True\n",
        "    X = LSTM(128, return_sequences=True)(embeddings)\n",
        "    # FIM DO CODIGO \n",
        "    X = Dropout(0.5)(X)\n",
        "    # SEU CODIGO AQUI\n",
        "    # Adiciona a segunda camada de LSTM aqui. Devera ter 128 neuronios,\n",
        "    # recebe e retorna X. Utilize return_sequences=False\n",
        "    X = LSTM(128, return_sequences=False)(X)\n",
        "    # FIM DO CODIGO \n",
        "    X = Dropout(0.5)(X)\n",
        "    X = Dense(5)(X)\n",
        "    X = Activation('softmax')(X)\n",
        "    model = Model(inputs=indices, outputs=X)\n",
        "    return model\n",
        "\n",
        "maxLen = len(max(X_train, key=lambda string: len(string.split())).split())\n",
        "LSTM = LSTM_emoji((maxLen,), embedding_layer)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "QEp9fl97t7z9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Expected output:**  \n",
        "\n",
        "Layer (type)                 Output Shape              Param #   \n",
        "\n",
        "input_2 (InputLayer)         (None, 10)                0         \n",
        "_________________________________________________________________\n",
        "embedding_1 (Embedding)      (None, 10, 50)            20000050  \n",
        "_________________________________________________________________\n",
        "lstm_1 (LSTM)                (None, 10, 128)           91648     \n",
        "_________________________________________________________________\n",
        "dropout_1 (Dropout)          (None, 10, 128)           0         \n",
        "_________________________________________________________________\n",
        "lstm_2 (LSTM)                (None, 128)               131584    \n",
        "_________________________________________________________________\n",
        "dropout_2 (Dropout)          (None, 128)               0         \n",
        "_________________________________________________________________\n",
        "dense_3 (Dense)              (None, 5)                 645       \n",
        "_________________________________________________________________\n",
        "activation_3 (Activation)    (None, 5)                 0         \n",
        "\n",
        "Total params: 20,223,927\n",
        "Trainable params: 223,877\n",
        "Non-trainable params: 20,000,050"
      ]
    },
    {
      "metadata": {
        "id": "dvvSlK2DtrFt",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 428
        },
        "outputId": "89078f43-d247-412b-82e7-fa91fec64ea3"
      },
      "cell_type": "code",
      "source": [
        "LSTM.summary()"
      ],
      "execution_count": 124,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_5 (InputLayer)         (None, 10)                0         \n",
            "_________________________________________________________________\n",
            "embedding_1 (Embedding)      (None, 10, 50)            20000050  \n",
            "_________________________________________________________________\n",
            "lstm_3 (LSTM)                (None, 10, 128)           91648     \n",
            "_________________________________________________________________\n",
            "dropout_3 (Dropout)          (None, 10, 128)           0         \n",
            "_________________________________________________________________\n",
            "lstm_4 (LSTM)                (None, 128)               131584    \n",
            "_________________________________________________________________\n",
            "dropout_4 (Dropout)          (None, 128)               0         \n",
            "_________________________________________________________________\n",
            "dense_6 (Dense)              (None, 5)                 645       \n",
            "_________________________________________________________________\n",
            "activation_5 (Activation)    (None, 5)                 0         \n",
            "=================================================================\n",
            "Total params: 20,223,927\n",
            "Trainable params: 223,877\n",
            "Non-trainable params: 20,000,050\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "_9CxfYPZuqGv",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Nao se assuste com o numero de parametros, destes muitos sao os embeddings que agora sao um layer do keras para este modelo de LSTM. Mas eles sao nao treinaveis e ja estao pre treinados, (embora possam ser treinaveis) entao nao vai demorar muito pra treinar o modelo."
      ]
    },
    {
      "metadata": {
        "id": "Ed5KOWzmuawb",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 3608
        },
        "outputId": "ca6b8725-b708-4de3-d775-90a94f48c365"
      },
      "cell_type": "code",
      "source": [
        "LSTM.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "X_train_indices = sentence_to_index(X_train, word_to_index, maxLen)\n",
        "X_test_indices = sentence_to_index(X_test, word_to_index, maxLen)\n",
        "LSTM.fit(X_train_indices, y_train_1hot, epochs=100, batch_size=X_train_indices.shape[0], shuffle=True)"
      ],
      "execution_count": 125,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/100\n",
            "132/132 [==============================] - 1s 10ms/step - loss: 1.6237 - acc: 0.1212\n",
            "Epoch 2/100\n",
            "132/132 [==============================] - 0s 668us/step - loss: 1.5803 - acc: 0.3030\n",
            "Epoch 3/100\n",
            "132/132 [==============================] - 0s 653us/step - loss: 1.5551 - acc: 0.3106\n",
            "Epoch 4/100\n",
            "132/132 [==============================] - 0s 695us/step - loss: 1.5310 - acc: 0.3333\n",
            "Epoch 5/100\n",
            "132/132 [==============================] - 0s 681us/step - loss: 1.5055 - acc: 0.3258\n",
            "Epoch 6/100\n",
            "132/132 [==============================] - 0s 679us/step - loss: 1.5098 - acc: 0.3106\n",
            "Epoch 7/100\n",
            "132/132 [==============================] - 0s 680us/step - loss: 1.4881 - acc: 0.3333\n",
            "Epoch 8/100\n",
            "132/132 [==============================] - 0s 694us/step - loss: 1.4503 - acc: 0.4015\n",
            "Epoch 9/100\n",
            "132/132 [==============================] - 0s 680us/step - loss: 1.4348 - acc: 0.4015\n",
            "Epoch 10/100\n",
            "132/132 [==============================] - 0s 699us/step - loss: 1.3988 - acc: 0.4015\n",
            "Epoch 11/100\n",
            "132/132 [==============================] - 0s 712us/step - loss: 1.3732 - acc: 0.4545\n",
            "Epoch 12/100\n",
            "132/132 [==============================] - 0s 748us/step - loss: 1.3426 - acc: 0.5076\n",
            "Epoch 13/100\n",
            "132/132 [==============================] - 0s 679us/step - loss: 1.3003 - acc: 0.4621\n",
            "Epoch 14/100\n",
            "132/132 [==============================] - 0s 678us/step - loss: 1.2598 - acc: 0.4773\n",
            "Epoch 15/100\n",
            "132/132 [==============================] - 0s 689us/step - loss: 1.2324 - acc: 0.5379\n",
            "Epoch 16/100\n",
            "132/132 [==============================] - 0s 675us/step - loss: 1.1762 - acc: 0.5530\n",
            "Epoch 17/100\n",
            "132/132 [==============================] - 0s 691us/step - loss: 1.1234 - acc: 0.5530\n",
            "Epoch 18/100\n",
            "132/132 [==============================] - 0s 714us/step - loss: 1.0998 - acc: 0.5758\n",
            "Epoch 19/100\n",
            "132/132 [==============================] - 0s 673us/step - loss: 1.0640 - acc: 0.5682\n",
            "Epoch 20/100\n",
            "132/132 [==============================] - 0s 701us/step - loss: 1.0032 - acc: 0.6439\n",
            "Epoch 21/100\n",
            "132/132 [==============================] - 0s 696us/step - loss: 0.9176 - acc: 0.6742\n",
            "Epoch 22/100\n",
            "132/132 [==============================] - 0s 699us/step - loss: 0.8985 - acc: 0.6818\n",
            "Epoch 23/100\n",
            "132/132 [==============================] - 0s 697us/step - loss: 0.8741 - acc: 0.6515\n",
            "Epoch 24/100\n",
            "132/132 [==============================] - 0s 724us/step - loss: 0.8321 - acc: 0.7121\n",
            "Epoch 25/100\n",
            "132/132 [==============================] - 0s 711us/step - loss: 0.7593 - acc: 0.7424\n",
            "Epoch 26/100\n",
            "132/132 [==============================] - 0s 687us/step - loss: 0.7007 - acc: 0.7424\n",
            "Epoch 27/100\n",
            "132/132 [==============================] - 0s 686us/step - loss: 0.6791 - acc: 0.7576\n",
            "Epoch 28/100\n",
            "132/132 [==============================] - 0s 729us/step - loss: 0.6558 - acc: 0.8030\n",
            "Epoch 29/100\n",
            "132/132 [==============================] - 0s 714us/step - loss: 0.5934 - acc: 0.7803\n",
            "Epoch 30/100\n",
            "132/132 [==============================] - 0s 685us/step - loss: 0.6008 - acc: 0.7803\n",
            "Epoch 31/100\n",
            "132/132 [==============================] - 0s 769us/step - loss: 0.5201 - acc: 0.7955\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 32/100\n",
            "132/132 [==============================] - 0s 703us/step - loss: 0.4878 - acc: 0.8409\n",
            "Epoch 33/100\n",
            "132/132 [==============================] - 0s 728us/step - loss: 0.4655 - acc: 0.8409\n",
            "Epoch 34/100\n",
            "132/132 [==============================] - 0s 737us/step - loss: 0.3812 - acc: 0.8561\n",
            "Epoch 35/100\n",
            "132/132 [==============================] - 0s 711us/step - loss: 0.3725 - acc: 0.8636\n",
            "Epoch 36/100\n",
            "132/132 [==============================] - 0s 681us/step - loss: 0.3073 - acc: 0.8788\n",
            "Epoch 37/100\n",
            "132/132 [==============================] - 0s 693us/step - loss: 0.3544 - acc: 0.8939\n",
            "Epoch 38/100\n",
            "132/132 [==============================] - 0s 703us/step - loss: 0.3243 - acc: 0.8864\n",
            "Epoch 39/100\n",
            "132/132 [==============================] - 0s 689us/step - loss: 0.2936 - acc: 0.8939\n",
            "Epoch 40/100\n",
            "132/132 [==============================] - 0s 685us/step - loss: 0.2727 - acc: 0.8939\n",
            "Epoch 41/100\n",
            "132/132 [==============================] - 0s 779us/step - loss: 0.2927 - acc: 0.9167\n",
            "Epoch 42/100\n",
            "132/132 [==============================] - 0s 681us/step - loss: 0.2148 - acc: 0.9318\n",
            "Epoch 43/100\n",
            "132/132 [==============================] - 0s 690us/step - loss: 0.2129 - acc: 0.9091\n",
            "Epoch 44/100\n",
            "132/132 [==============================] - 0s 672us/step - loss: 0.1716 - acc: 0.9318\n",
            "Epoch 45/100\n",
            "132/132 [==============================] - 0s 675us/step - loss: 0.2199 - acc: 0.9167\n",
            "Epoch 46/100\n",
            "132/132 [==============================] - 0s 674us/step - loss: 0.1604 - acc: 0.9470\n",
            "Epoch 47/100\n",
            "132/132 [==============================] - 0s 666us/step - loss: 0.1326 - acc: 0.9545\n",
            "Epoch 48/100\n",
            "132/132 [==============================] - 0s 678us/step - loss: 0.1258 - acc: 0.9697\n",
            "Epoch 49/100\n",
            "132/132 [==============================] - 0s 673us/step - loss: 0.1599 - acc: 0.9242\n",
            "Epoch 50/100\n",
            "132/132 [==============================] - 0s 707us/step - loss: 0.1477 - acc: 0.9318\n",
            "Epoch 51/100\n",
            "132/132 [==============================] - 0s 682us/step - loss: 0.1012 - acc: 0.9697\n",
            "Epoch 52/100\n",
            "132/132 [==============================] - 0s 689us/step - loss: 0.1627 - acc: 0.9394\n",
            "Epoch 53/100\n",
            "132/132 [==============================] - 0s 691us/step - loss: 0.0890 - acc: 0.9773\n",
            "Epoch 54/100\n",
            "132/132 [==============================] - 0s 699us/step - loss: 0.0997 - acc: 0.9621\n",
            "Epoch 55/100\n",
            "132/132 [==============================] - 0s 684us/step - loss: 0.0907 - acc: 0.9697\n",
            "Epoch 56/100\n",
            "132/132 [==============================] - 0s 682us/step - loss: 0.0628 - acc: 0.9848\n",
            "Epoch 57/100\n",
            "132/132 [==============================] - 0s 666us/step - loss: 0.1194 - acc: 0.9545\n",
            "Epoch 58/100\n",
            "132/132 [==============================] - 0s 672us/step - loss: 0.0601 - acc: 0.9697\n",
            "Epoch 59/100\n",
            "132/132 [==============================] - 0s 675us/step - loss: 0.1119 - acc: 0.9697\n",
            "Epoch 60/100\n",
            "132/132 [==============================] - 0s 695us/step - loss: 0.1120 - acc: 0.9697\n",
            "Epoch 61/100\n",
            "132/132 [==============================] - 0s 703us/step - loss: 0.1187 - acc: 0.9848\n",
            "Epoch 62/100\n",
            "132/132 [==============================] - 0s 676us/step - loss: 0.1519 - acc: 0.9470\n",
            "Epoch 63/100\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\r132/132 [==============================] - 0s 673us/step - loss: 0.0452 - acc: 0.9848\n",
            "Epoch 64/100\n",
            "132/132 [==============================] - 0s 795us/step - loss: 0.1503 - acc: 0.9318\n",
            "Epoch 65/100\n",
            "132/132 [==============================] - 0s 703us/step - loss: 0.0825 - acc: 0.9697\n",
            "Epoch 66/100\n",
            "132/132 [==============================] - 0s 648us/step - loss: 0.0242 - acc: 1.0000\n",
            "Epoch 67/100\n",
            "132/132 [==============================] - 0s 761us/step - loss: 0.0661 - acc: 0.9848\n",
            "Epoch 68/100\n",
            "132/132 [==============================] - 0s 676us/step - loss: 0.0786 - acc: 0.9773\n",
            "Epoch 69/100\n",
            "132/132 [==============================] - 0s 699us/step - loss: 0.0544 - acc: 0.9848\n",
            "Epoch 70/100\n",
            "132/132 [==============================] - 0s 695us/step - loss: 0.0410 - acc: 0.9924\n",
            "Epoch 71/100\n",
            "132/132 [==============================] - 0s 737us/step - loss: 0.0520 - acc: 0.9848\n",
            "Epoch 72/100\n",
            "132/132 [==============================] - 0s 710us/step - loss: 0.0696 - acc: 0.9773\n",
            "Epoch 73/100\n",
            "132/132 [==============================] - 0s 699us/step - loss: 0.0315 - acc: 1.0000\n",
            "Epoch 74/100\n",
            "132/132 [==============================] - 0s 763us/step - loss: 0.1309 - acc: 0.9545\n",
            "Epoch 75/100\n",
            "132/132 [==============================] - 0s 697us/step - loss: 0.0565 - acc: 0.9848\n",
            "Epoch 76/100\n",
            "132/132 [==============================] - 0s 710us/step - loss: 0.0278 - acc: 0.9924\n",
            "Epoch 77/100\n",
            "132/132 [==============================] - 0s 683us/step - loss: 0.0303 - acc: 0.9924\n",
            "Epoch 78/100\n",
            "132/132 [==============================] - 0s 669us/step - loss: 0.0357 - acc: 0.9924\n",
            "Epoch 79/100\n",
            "132/132 [==============================] - 0s 691us/step - loss: 0.0321 - acc: 0.9924\n",
            "Epoch 80/100\n",
            "132/132 [==============================] - 0s 699us/step - loss: 0.0256 - acc: 1.0000\n",
            "Epoch 81/100\n",
            "132/132 [==============================] - 0s 682us/step - loss: 0.0323 - acc: 0.9924\n",
            "Epoch 82/100\n",
            "132/132 [==============================] - 0s 681us/step - loss: 0.0129 - acc: 1.0000\n",
            "Epoch 83/100\n",
            "132/132 [==============================] - 0s 689us/step - loss: 0.0201 - acc: 0.9924\n",
            "Epoch 84/100\n",
            "132/132 [==============================] - 0s 674us/step - loss: 0.0230 - acc: 0.9924\n",
            "Epoch 85/100\n",
            "132/132 [==============================] - 0s 694us/step - loss: 0.0267 - acc: 0.9924\n",
            "Epoch 86/100\n",
            "132/132 [==============================] - 0s 671us/step - loss: 0.0187 - acc: 0.9924\n",
            "Epoch 87/100\n",
            "132/132 [==============================] - 0s 688us/step - loss: 0.0104 - acc: 1.0000\n",
            "Epoch 88/100\n",
            "132/132 [==============================] - 0s 695us/step - loss: 0.0183 - acc: 0.9924\n",
            "Epoch 89/100\n",
            "132/132 [==============================] - 0s 671us/step - loss: 0.0159 - acc: 1.0000\n",
            "Epoch 90/100\n",
            "132/132 [==============================] - 0s 702us/step - loss: 0.0213 - acc: 1.0000\n",
            "Epoch 91/100\n",
            "132/132 [==============================] - 0s 677us/step - loss: 0.0218 - acc: 0.9924\n",
            "Epoch 92/100\n",
            "132/132 [==============================] - 0s 697us/step - loss: 0.0163 - acc: 0.9924\n",
            "Epoch 93/100\n",
            "132/132 [==============================] - 0s 684us/step - loss: 0.0112 - acc: 1.0000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 94/100\n",
            "132/132 [==============================] - 0s 709us/step - loss: 0.0068 - acc: 1.0000\n",
            "Epoch 95/100\n",
            "132/132 [==============================] - 0s 790us/step - loss: 0.0082 - acc: 1.0000\n",
            "Epoch 96/100\n",
            "132/132 [==============================] - 0s 766us/step - loss: 0.0088 - acc: 1.0000\n",
            "Epoch 97/100\n",
            "132/132 [==============================] - 0s 646us/step - loss: 0.0130 - acc: 0.9924\n",
            "Epoch 98/100\n",
            "132/132 [==============================] - 0s 681us/step - loss: 0.0064 - acc: 1.0000\n",
            "Epoch 99/100\n",
            "132/132 [==============================] - 0s 707us/step - loss: 0.0054 - acc: 1.0000\n",
            "Epoch 100/100\n",
            "132/132 [==============================] - 0s 735us/step - loss: 0.0059 - acc: 1.0000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f1ccbe98410>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 125
        }
      ]
    },
    {
      "metadata": {
        "id": "HzwAa_EYw5Tp",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "9923dcbf-2a34-4658-a014-35f1b48c2f7c"
      },
      "cell_type": "code",
      "source": [
        "loss, accuracy = LSTM.evaluate(X_test_indices, y_test_1hot)\n",
        "print('Loss: {}, Acuracia: {}'.format(loss, accuracy))"
      ],
      "execution_count": 126,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "56/56 [==============================] - 0s 4ms/step\n",
            "Loss: 0.68515251364, Acuracia: 0.857142865658\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "wc52_948xZkY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "A LSTM tem uma acuracia ligeiramente melhor que a MLP no treino e teste, mas os resultados sao similares. Vamos ver o que ela errou no conjunto de teste:"
      ]
    },
    {
      "metadata": {
        "id": "NpigLHIIxDr9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 160
        },
        "outputId": "0fe16d27-e29a-4041-e74a-baaa3797a67e"
      },
      "cell_type": "code",
      "source": [
        "preds = LSTM.predict(X_test_indices)\n",
        "for i, phrase in enumerate(X_test):\n",
        "    if np.argmax(preds[i]) != y_test[i]:\n",
        "        x = X_test[i]\n",
        "        y = y_test[i]\n",
        "        y_emoji = label_to_emoji(y)\n",
        "        y_hat = np.argmax(preds[i])\n",
        "        y_hat_emoji = label_to_emoji(y_hat)\n",
        "        print('Sentence: {}, Expected Label: {}, Predicted Label: {}'.format(x, y_emoji, y_hat_emoji))"
      ],
      "execution_count": 127,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sentence: she got me a nice present, Expected Label: 😄, Predicted Label: ❤\n",
            "Sentence: work is hard, Expected Label: 😞, Predicted Label: 😄\n",
            "Sentence: This girl is messing with me, Expected Label: 😞, Predicted Label: ❤\n",
            "Sentence: any suggestions for dinner, Expected Label: 🍴, Predicted Label: 😄\n",
            "Sentence: you brighten my day, Expected Label: 😄, Predicted Label: ❤\n",
            "Sentence: she is a bully, Expected Label: 😞, Predicted Label: 😄\n",
            "Sentence: will you be my valentine, Expected Label: 😄, Predicted Label: ❤\n",
            "Sentence: go away, Expected Label: 😞, Predicted Label: ⚾\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "ST_4PFCkywQ5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Qual a diferenca principal entao entre os modelos?"
      ]
    },
    {
      "metadata": {
        "id": "7lucCEEQy1bt",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "efa50960-11dd-45a1-cf26-417dbed3b2a6"
      },
      "cell_type": "code",
      "source": [
        "def predict_emoji_LSTM(model, sentence):\n",
        "    pred = model.predict(sentence_to_index(np.array([sentence]), word_to_index, maxLen).reshape(1,-1))\n",
        "    return label_to_emoji(np.argmax(pred))\n",
        "  \n",
        "print(predict_emoji_LSTM(LSTM, 'not feeling very happy'))"
      ],
      "execution_count": 128,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "😄\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "KJJ6aTG9zJ-v",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "c3553476-324e-4eab-e4ce-c5d4f5a03f4f"
      },
      "cell_type": "code",
      "source": [
        "print(predict_emoji_LSTM(LSTM, 'it is not good'))"
      ],
      "execution_count": 129,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "😄\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "JOVGNERjzP4l",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "a3206ca5-8159-4036-f885-c891ff330402"
      },
      "cell_type": "code",
      "source": [
        "print(predict_emoji_LSTM(LSTM, 'it is good not'))"
      ],
      "execution_count": 130,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "😄\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "R32ozlWGzOPL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Conclusão\n",
        "\n",
        "Como pode ser visto, a LSTM leva em consideracao a ordem da sequencia, como esperado, a tornando mais apropriada para este tipo de problema, mostrando uma capacidade maior de entender o contexto da frase e portanto se provando um modelo mais robusto.\n",
        "\n",
        "Enquanto que a MLP aprendeu bem o dataset, ela 'overfitta' mais pois mudando um pouco o contexto ou a ordem ela nao consegue lidar. Ja a LSTM aprende as dependencias temporais e portanto faz previsoes fora do conjunto de treino bem melhores, aprendendo de verdade. "
      ]
    },
    {
      "metadata": {
        "id": "mZcMDCRc0Lwn",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "cd60cae1-5026-4a11-ca1f-4df944372c27"
      },
      "cell_type": "code",
      "source": [
        "#tente voce com uma frase qualquer\n",
        "print(predict_emoji_LSTM(LSTM, 'i need a coffee it is 3 am help'))"
      ],
      "execution_count": 131,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "🍴\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}