lectures/cap05_1_extracao_caracteristicas.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "cap05_1_extracao_caracteristicas.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyOytjoRjD9gKr7VNgulx0Ni",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/viniciusrpb/cic0269_natural_language_processing/blob/main/cap05_1_extracao_caracteristicas.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Capítulo 5 - Extração de Características de Textos\n",
        "\n",
        "Nesta seção vamos estudar como obter representações estruturadas de textos que são conhecidas como *embeddings*.\n",
        "\n"
      ],
      "metadata": {
        "id": "3NR4MddSh66H"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd"
      ],
      "metadata": {
        "id": "cssPeBEhk0gS"
      },
      "execution_count": 43,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 5.1. O Modelo Bag-of-Words (BoW)\n",
        "\n",
        "Suponha o seguinte corpus contendo \"apenas\" cinco sentenças (considerados aqui como documentos):"
      ],
      "metadata": {
        "id": "eVjloctN0Tp6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "corpus = []\n",
        "corpus.append('Batatinha quando nasce esparrama pelo chão igual batatinha')\n",
        "corpus.append('A pior experiência da minha vida')\n",
        "corpus.append('Quero meu dinheiro de volta pois é meu e é meu')\n",
        "corpus.append('A experiência do dinheiro esparrama minha vida')\n",
        "corpus.append('Acai é a melhor coisa da vida')"
      ],
      "metadata": {
        "id": "Js0FTZ7Jz0Bn"
      },
      "execution_count": 44,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "frase = \" \".join(corpus)"
      ],
      "metadata": {
        "id": "mrdfI6FL0Rao"
      },
      "execution_count": 45,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "frase = frase.lower()\n",
        "corpus = [x.lower() for x in corpus]\n",
        "frase"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "id": "FjiazLl50mo_",
        "outputId": "650485bf-dbef-491b-e8e2-af29f3f0db5a"
      },
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'batatinha quando nasce esparrama pelo chão igual batatinha a pior experiência da minha vida quero meu dinheiro de volta pois é meu e é meu a experiência do dinheiro esparrama minha vida acai é a melhor coisa da vida'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "corpus"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NGg9qkYHmTVe",
        "outputId": "7cf554da-e3e6-464a-961c-f939ee132a6b"
      },
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['batatinha quando nasce esparrama pelo chão igual batatinha',\n",
              " 'a pior experiência da minha vida',\n",
              " 'quero meu dinheiro de volta pois é meu e é meu',\n",
              " 'a experiência do dinheiro esparrama minha vida',\n",
              " 'acai é a melhor coisa da vida']"
            ]
          },
          "metadata": {},
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Vamos construir o vocabulário, em que cada índice:"
      ],
      "metadata": {
        "id": "-QXzlgFAlt5R"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tokens = frase.split()\n",
        "vocab = sorted(set(tokens))"
      ],
      "metadata": {
        "id": "xeS2FLBv0puZ"
      },
      "execution_count": 48,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "vocab"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "x0lSBJeJ00ki",
        "outputId": "12786e18-c62f-46f2-9e39-366bd8730ba4"
      },
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['a',\n",
              " 'acai',\n",
              " 'batatinha',\n",
              " 'chão',\n",
              " 'coisa',\n",
              " 'da',\n",
              " 'de',\n",
              " 'dinheiro',\n",
              " 'do',\n",
              " 'e',\n",
              " 'esparrama',\n",
              " 'experiência',\n",
              " 'igual',\n",
              " 'melhor',\n",
              " 'meu',\n",
              " 'minha',\n",
              " 'nasce',\n",
              " 'pelo',\n",
              " 'pior',\n",
              " 'pois',\n",
              " 'quando',\n",
              " 'quero',\n",
              " 'vida',\n",
              " 'volta',\n",
              " 'é']"
            ]
          },
          "metadata": {},
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Bag-of-Words Binário (*One-hot encoding*)\n",
        "\n",
        "O primeiro passo é construirmos o \"saco de palavras\" do corpus como um dicionário, em que cada documento é um índice, que armazena a presença de cada palavra do documento no vocabulário: "
      ],
      "metadata": {
        "id": "A2T6YVX0jkZQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "bow_binario = {}\n",
        "\n",
        "for i,doc in enumerate(corpus):\n",
        "    bow_binario['sentenca {}'.format(i+1)] = dict()\n",
        "    for word in doc.split():\n",
        "        bow_binario['sentenca {}'.format(i+1)][word] = 1"
      ],
      "metadata": {
        "id": "onhhoKIR1wCo"
      },
      "execution_count": 50,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "bow_binario"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YoThFhsI6Pmi",
        "outputId": "393145f1-908f-4036-8d12-ad48129c95a9"
      },
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'sentenca 1': {'batatinha': 1,\n",
              "  'chão': 1,\n",
              "  'esparrama': 1,\n",
              "  'igual': 1,\n",
              "  'nasce': 1,\n",
              "  'pelo': 1,\n",
              "  'quando': 1},\n",
              " 'sentenca 2': {'a': 1,\n",
              "  'da': 1,\n",
              "  'experiência': 1,\n",
              "  'minha': 1,\n",
              "  'pior': 1,\n",
              "  'vida': 1},\n",
              " 'sentenca 3': {'de': 1,\n",
              "  'dinheiro': 1,\n",
              "  'e': 1,\n",
              "  'meu': 1,\n",
              "  'pois': 1,\n",
              "  'quero': 1,\n",
              "  'volta': 1,\n",
              "  'é': 1},\n",
              " 'sentenca 4': {'a': 1,\n",
              "  'dinheiro': 1,\n",
              "  'do': 1,\n",
              "  'esparrama': 1,\n",
              "  'experiência': 1,\n",
              "  'minha': 1,\n",
              "  'vida': 1},\n",
              " 'sentenca 5': {'a': 1,\n",
              "  'acai': 1,\n",
              "  'coisa': 1,\n",
              "  'da': 1,\n",
              "  'melhor': 1,\n",
              "  'vida': 1,\n",
              "  'é': 1}}"
            ]
          },
          "metadata": {},
          "execution_count": 51
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "A partir do \"saco de palavras\" obtido, vamos construir a representação estruturada como uma matriz de incidência binária:"
      ],
      "metadata": {
        "id": "IPT8GGhUj3sw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df = pd.DataFrame().from_records(bow_binario).fillna(0).T.astype(int)\n",
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 386
        },
        "id": "nhTGUEmi7V-1",
        "outputId": "1b972456-b804-4b22-d98a-0bc772e33588"
      },
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "            batatinha  quando  nasce  esparrama  pelo  chão  igual  a  pior  \\\n",
              "sentenca 1          1       1      1          1     1     1      1  0     0   \n",
              "sentenca 2          0       0      0          0     0     0      0  1     1   \n",
              "sentenca 3          0       0      0          0     0     0      0  0     0   \n",
              "sentenca 4          0       0      0          1     0     0      0  1     0   \n",
              "sentenca 5          0       0      0          0     0     0      0  1     0   \n",
              "\n",
              "            experiência  ...  dinheiro  de  volta  pois  é  e  do  acai  \\\n",
              "sentenca 1            0  ...         0   0      0     0  0  0   0     0   \n",
              "sentenca 2            1  ...         0   0      0     0  0  0   0     0   \n",
              "sentenca 3            0  ...         1   1      1     1  1  1   0     0   \n",
              "sentenca 4            1  ...         1   0      0     0  0  0   1     0   \n",
              "sentenca 5            0  ...         0   0      0     0  1  0   0     1   \n",
              "\n",
              "            melhor  coisa  \n",
              "sentenca 1       0      0  \n",
              "sentenca 2       0      0  \n",
              "sentenca 3       0      0  \n",
              "sentenca 4       0      0  \n",
              "sentenca 5       1      1  \n",
              "\n",
              "[5 rows x 25 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-012e3a42-62f5-4de0-bbe6-84c23a616a79\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>batatinha</th>\n",
              "      <th>quando</th>\n",
              "      <th>nasce</th>\n",
              "      <th>esparrama</th>\n",
              "      <th>pelo</th>\n",
              "      <th>chão</th>\n",
              "      <th>igual</th>\n",
              "      <th>a</th>\n",
              "      <th>pior</th>\n",
              "      <th>experiência</th>\n",
              "      <th>...</th>\n",
              "      <th>dinheiro</th>\n",
              "      <th>de</th>\n",
              "      <th>volta</th>\n",
              "      <th>pois</th>\n",
              "      <th>é</th>\n",
              "      <th>e</th>\n",
              "      <th>do</th>\n",
              "      <th>acai</th>\n",
              "      <th>melhor</th>\n",
              "      <th>coisa</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>sentenca 1</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>sentenca 2</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>sentenca 3</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>sentenca 4</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>sentenca 5</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 25 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-012e3a42-62f5-4de0-bbe6-84c23a616a79')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-012e3a42-62f5-4de0-bbe6-84c23a616a79 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-012e3a42-62f5-4de0-bbe6-84c23a616a79');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Bag-of-Words de Contagem\n",
        "\n",
        "O \"saco de palavras\" do corpus também é construído como se fosse um dicionário, em que cada índice armazena a frequência de cada palavra do associado documento em relação ao vocabulário: "
      ],
      "metadata": {
        "id": "fsM36x22kVig"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "bow_contagem = {}\n",
        "\n",
        "for i,doc in enumerate(corpus):\n",
        "    bow_contagem['sentenca {}'.format(i+1)] = dict()\n",
        "    for word in doc.split():\n",
        "        if word in bow_contagem['sentenca {}'.format(i+1)]:\n",
        "            bow_contagem['sentenca {}'.format(i+1)][word] += 1\n",
        "        else:\n",
        "            bow_contagem['sentenca {}'.format(i+1)][word] = 1"
      ],
      "metadata": {
        "id": "Eo3TySc_kluF"
      },
      "execution_count": 53,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "bow_contagem"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-cdkqnRKm8gS",
        "outputId": "5751469c-cd0d-48d7-f4c1-015de6fec492"
      },
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'sentenca 1': {'batatinha': 2,\n",
              "  'chão': 1,\n",
              "  'esparrama': 1,\n",
              "  'igual': 1,\n",
              "  'nasce': 1,\n",
              "  'pelo': 1,\n",
              "  'quando': 1},\n",
              " 'sentenca 2': {'a': 1,\n",
              "  'da': 1,\n",
              "  'experiência': 1,\n",
              "  'minha': 1,\n",
              "  'pior': 1,\n",
              "  'vida': 1},\n",
              " 'sentenca 3': {'de': 1,\n",
              "  'dinheiro': 1,\n",
              "  'e': 1,\n",
              "  'meu': 3,\n",
              "  'pois': 1,\n",
              "  'quero': 1,\n",
              "  'volta': 1,\n",
              "  'é': 2},\n",
              " 'sentenca 4': {'a': 1,\n",
              "  'dinheiro': 1,\n",
              "  'do': 1,\n",
              "  'esparrama': 1,\n",
              "  'experiência': 1,\n",
              "  'minha': 1,\n",
              "  'vida': 1},\n",
              " 'sentenca 5': {'a': 1,\n",
              "  'acai': 1,\n",
              "  'coisa': 1,\n",
              "  'da': 1,\n",
              "  'melhor': 1,\n",
              "  'vida': 1,\n",
              "  'é': 1}}"
            ]
          },
          "metadata": {},
          "execution_count": 54
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Bag-of-Words Ponderado"
      ],
      "metadata": {
        "id": "n30f9HpgB7Al"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "bow_ponderado = {}\n",
        "N = {} # quantidade de termos em cada documento\n",
        "\n",
        "for i,doc in enumerate(corpus):\n",
        "    bow_ponderado['sentenca {}'.format(i+1)] = dict()\n",
        "    N['sentenca {}'.format(i+1)] = len(doc.split())\n",
        "    for word in doc.split():\n",
        "        if word in bow_ponderado['sentenca {}'.format(i+1)]:\n",
        "            bow_ponderado['sentenca {}'.format(i+1)][word] += 1\n",
        "        else:\n",
        "            bow_ponderado['sentenca {}'.format(i+1)][word] = 1\n",
        "\n",
        "for doc in bow_ponderado:\n",
        "    for word in bow_ponderado[doc]:\n",
        "        bow_ponderado[doc][word] /= N[doc]"
      ],
      "metadata": {
        "id": "0uihbVedB7HG"
      },
      "execution_count": 55,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "bow_ponderado"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QXE_T9YeD7Yk",
        "outputId": "6b080bff-f175-49c7-cb7c-8b9a0ea99397"
      },
      "execution_count": 56,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'sentenca 1': {'batatinha': 0.25,\n",
              "  'chão': 0.125,\n",
              "  'esparrama': 0.125,\n",
              "  'igual': 0.125,\n",
              "  'nasce': 0.125,\n",
              "  'pelo': 0.125,\n",
              "  'quando': 0.125},\n",
              " 'sentenca 2': {'a': 0.16666666666666666,\n",
              "  'da': 0.16666666666666666,\n",
              "  'experiência': 0.16666666666666666,\n",
              "  'minha': 0.16666666666666666,\n",
              "  'pior': 0.16666666666666666,\n",
              "  'vida': 0.16666666666666666},\n",
              " 'sentenca 3': {'de': 0.09090909090909091,\n",
              "  'dinheiro': 0.09090909090909091,\n",
              "  'e': 0.09090909090909091,\n",
              "  'meu': 0.2727272727272727,\n",
              "  'pois': 0.09090909090909091,\n",
              "  'quero': 0.09090909090909091,\n",
              "  'volta': 0.09090909090909091,\n",
              "  'é': 0.18181818181818182},\n",
              " 'sentenca 4': {'a': 0.14285714285714285,\n",
              "  'dinheiro': 0.14285714285714285,\n",
              "  'do': 0.14285714285714285,\n",
              "  'esparrama': 0.14285714285714285,\n",
              "  'experiência': 0.14285714285714285,\n",
              "  'minha': 0.14285714285714285,\n",
              "  'vida': 0.14285714285714285},\n",
              " 'sentenca 5': {'a': 0.14285714285714285,\n",
              "  'acai': 0.14285714285714285,\n",
              "  'coisa': 0.14285714285714285,\n",
              "  'da': 0.14285714285714285,\n",
              "  'melhor': 0.14285714285714285,\n",
              "  'vida': 0.14285714285714285,\n",
              "  'é': 0.14285714285714285}}"
            ]
          },
          "metadata": {},
          "execution_count": 56
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Term Frequency"
      ],
      "metadata": {
        "id": "-KStCO632ueB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import math\n",
        "idf = {}\n",
        "for word in vocab:\n",
        "    idf[word] = 0\n",
        "    #contabilizar em quantos documentos a palavra word aparece\n",
        "    for doc in bow_contagem:\n",
        "        if word in bow_contagem[doc]:\n",
        "            idf[word] += 1\n",
        "\n",
        "for word in idf:\n",
        "    idf[word] = math.log(len(corpus)/idf[word])   "
      ],
      "metadata": {
        "id": "BVPtmXasGkBD"
      },
      "execution_count": 57,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "idf"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "R2lO9No-HUGR",
        "outputId": "b94bb6bb-3860-4a97-e0fc-83c213387144"
      },
      "execution_count": 58,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'a': 3,\n",
              " 'acai': 1,\n",
              " 'batatinha': 1,\n",
              " 'chão': 1,\n",
              " 'coisa': 1,\n",
              " 'da': 2,\n",
              " 'de': 1,\n",
              " 'dinheiro': 2,\n",
              " 'do': 1,\n",
              " 'e': 1,\n",
              " 'esparrama': 2,\n",
              " 'experiência': 2,\n",
              " 'igual': 1,\n",
              " 'melhor': 1,\n",
              " 'meu': 1,\n",
              " 'minha': 2,\n",
              " 'nasce': 1,\n",
              " 'pelo': 1,\n",
              " 'pior': 1,\n",
              " 'pois': 1,\n",
              " 'quando': 1,\n",
              " 'quero': 1,\n",
              " 'vida': 3,\n",
              " 'volta': 1,\n",
              " 'é': 2}"
            ]
          },
          "metadata": {},
          "execution_count": 58
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Term Frequency-Inverse Document Frequency"
      ],
      "metadata": {
        "id": "HlnswJ5eRv4f"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "bow_tfidf = {}\n",
        "\n",
        "for doc in bow_ponderado:\n",
        "    bow_tfidf[doc] = dict()\n",
        "    for word in bow_ponderado[doc]:\n",
        "        bow_tfidf[doc][word] = bow_ponderado[doc][word]*idf[word]"
      ],
      "metadata": {
        "id": "qMNHYA1pSjRc"
      },
      "execution_count": 67,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "bow_tfidf"
      ],
      "metadata": {
        "id": "Hd3qyMS9TRXg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df = pd.DataFrame().from_records(bow_tfidf).fillna(0).T\n",
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 386
        },
        "id": "XHZnuiUPTJFo",
        "outputId": "efe8497b-0d00-4069-8cab-ff026e30bd43"
      },
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "            batatinha  quando  nasce  esparrama   pelo   chão  igual  \\\n",
              "sentenca 1       0.25   0.125  0.125   0.250000  0.125  0.125  0.125   \n",
              "sentenca 2       0.00   0.000  0.000   0.000000  0.000  0.000  0.000   \n",
              "sentenca 3       0.00   0.000  0.000   0.000000  0.000  0.000  0.000   \n",
              "sentenca 4       0.00   0.000  0.000   0.285714  0.000  0.000  0.000   \n",
              "sentenca 5       0.00   0.000  0.000   0.000000  0.000  0.000  0.000   \n",
              "\n",
              "                   a      pior  experiência  ...  dinheiro        de  \\\n",
              "sentenca 1  0.000000  0.000000     0.000000  ...  0.000000  0.000000   \n",
              "sentenca 2  0.500000  0.166667     0.333333  ...  0.000000  0.000000   \n",
              "sentenca 3  0.000000  0.000000     0.000000  ...  0.181818  0.090909   \n",
              "sentenca 4  0.428571  0.000000     0.285714  ...  0.285714  0.000000   \n",
              "sentenca 5  0.428571  0.000000     0.000000  ...  0.000000  0.000000   \n",
              "\n",
              "               volta      pois         é         e        do      acai  \\\n",
              "sentenca 1  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000   \n",
              "sentenca 2  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000   \n",
              "sentenca 3  0.090909  0.090909  0.363636  0.090909  0.000000  0.000000   \n",
              "sentenca 4  0.000000  0.000000  0.000000  0.000000  0.142857  0.000000   \n",
              "sentenca 5  0.000000  0.000000  0.285714  0.000000  0.000000  0.142857   \n",
              "\n",
              "              melhor     coisa  \n",
              "sentenca 1  0.000000  0.000000  \n",
              "sentenca 2  0.000000  0.000000  \n",
              "sentenca 3  0.000000  0.000000  \n",
              "sentenca 4  0.000000  0.000000  \n",
              "sentenca 5  0.142857  0.142857  \n",
              "\n",
              "[5 rows x 25 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-965ee3e0-6e5c-4857-a179-bc2737f01af4\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>batatinha</th>\n",
              "      <th>quando</th>\n",
              "      <th>nasce</th>\n",
              "      <th>esparrama</th>\n",
              "      <th>pelo</th>\n",
              "      <th>chão</th>\n",
              "      <th>igual</th>\n",
              "      <th>a</th>\n",
              "      <th>pior</th>\n",
              "      <th>experiência</th>\n",
              "      <th>...</th>\n",
              "      <th>dinheiro</th>\n",
              "      <th>de</th>\n",
              "      <th>volta</th>\n",
              "      <th>pois</th>\n",
              "      <th>é</th>\n",
              "      <th>e</th>\n",
              "      <th>do</th>\n",
              "      <th>acai</th>\n",
              "      <th>melhor</th>\n",
              "      <th>coisa</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>sentenca 1</th>\n",
              "      <td>0.25</td>\n",
              "      <td>0.125</td>\n",
              "      <td>0.125</td>\n",
              "      <td>0.250000</td>\n",
              "      <td>0.125</td>\n",
              "      <td>0.125</td>\n",
              "      <td>0.125</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>...</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>sentenca 2</th>\n",
              "      <td>0.00</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.500000</td>\n",
              "      <td>0.166667</td>\n",
              "      <td>0.333333</td>\n",
              "      <td>...</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>sentenca 3</th>\n",
              "      <td>0.00</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>...</td>\n",
              "      <td>0.181818</td>\n",
              "      <td>0.090909</td>\n",
              "      <td>0.090909</td>\n",
              "      <td>0.090909</td>\n",
              "      <td>0.363636</td>\n",
              "      <td>0.090909</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>sentenca 4</th>\n",
              "      <td>0.00</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.285714</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.428571</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.285714</td>\n",
              "      <td>...</td>\n",
              "      <td>0.285714</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.142857</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>sentenca 5</th>\n",
              "      <td>0.00</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.000</td>\n",
              "      <td>0.428571</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>...</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.285714</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.142857</td>\n",
              "      <td>0.142857</td>\n",
              "      <td>0.142857</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 25 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-965ee3e0-6e5c-4857-a179-bc2737f01af4')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-965ee3e0-6e5c-4857-a179-bc2737f01af4 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-965ee3e0-6e5c-4857-a179-bc2737f01af4');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 74
        }
      ]
    }
  ]
}