Tarea_1_NLP.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/bacchus00/tareas-ds/blob/main/Tarea_1_NLP.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zbrMqMPyncTC"
      },
      "source": [
        "# **Tarea 1 - CC6205 Natural Language Processing 📚**\n",
        "\n",
        "**Integrantes:** Melanie Marquez, Franco Zautzik\n",
        "\n",
        "**Fecha límite de entrega 📆:** Martes 29 de Marzo.\n",
        "\n",
        "**Tiempo estimado de dedicación:** 5 hrs\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D1HFX-9PpxF9"
      },
      "source": [
        "` ` \n",
        "\n",
        "\n",
        "Bienvenid@s a la primera tarea en el curso de Natural Language Processing (NLP). Esta tarea tiene como objetivo evaluar los contenidos teóricos de las primeras semanas de clases, enfocado principalmente en ***Information Retrieval (IR)*** y ***Vector Space Models***. Si aún no han visto las clases, se recomienda visitar los links de las referencias.\n",
        "\n",
        "La tarea consta de una parte teórica que busca evaluar conceptos vistos en clases. Seguido por una parte práctica con el fín de introducirlos a la programación en Python enfocada en NLP. \n",
        "\n",
        "` ` \n",
        "\n",
        "\n",
        "\n",
        "\n",
        "**Instrucciones:**\n",
        "- La tarea se realiza en grupos de **máximo** 2 personas. Puede ser invidivual pero no es recomendable.\n",
        "- La entrega es a través de u-cursos a más tardar el día estipulado arriba. No se aceptan atrasos.\n",
        "- El formato de entrega es este mismo Jupyter Notebook.\n",
        "- Al momento de la revisión tu código será ejecutado. Por favor verifica que tu entrega no tenga errores de compilación. \n",
        "- Está **PROHIBIDO** usar cualquier librería que implemente los algoritmos pedidos (Spacy, scikit, etc). Sólo se podrán utilizar las librerías importadas al inicio de la sección de práctica.\n",
        "- En el horario de auxiliar pueden realizar consultas acerca de la tarea a través del canal de Discord del curso. \n",
        "\n",
        "\n",
        "\n",
        "Ahora sí, empecemos! 😄🚀\n",
        "\n",
        "` ` \n",
        "\n",
        "**Referencias:**\n",
        "\n",
        "\n",
        "Slides:\n",
        "    \n",
        "- [Introducción al curso](https://github.com/dccuchile/CC6205/blob/master/slides/NLP-introduction.pdf)\n",
        "- [Vector Space Model / Information Retrieval](https://github.com/dccuchile/CC6205/blob/master/slides/NLP-IR.pdf)    \n",
        "\n",
        "Videos: \n",
        "\n",
        "- [CC6205 - Procesamiento de Lenguaje Natural: CC6205 - Procesamiento de Lenguaje Natural: Introducción parte I](https://www.youtube.com/watch?v=HEKTNOttGvU)\n",
        "- [CC6205 - Procesamiento de Lenguaje Natural: CC6205 - Procesamiento de Lenguaje Natural: Introducción parte II](https://www.youtube.com/watch?v=P8cwnI-f-Kg)\n",
        "- [CC6205 - Procesamiento de Lenguaje Natural: Vector Space Model and Information Retrieval parte 1](https://youtu.be/FXIVClF370w)\n",
        "- [CC6205 - Procesamiento de Lenguaje Natural: Vector Space Model and Information Retrieval parte 2](https://youtu.be/f8nG1EMmPZk)\n",
        "\n",
        "` ` \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hJshpe1yrKJr"
      },
      "source": [
        "## **1 - Preguntas teóricas 📕 (2 puntos).** ##\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MBEDKXBPoA7w"
      },
      "source": [
        "Las siguientes celdas contienen preguntas acerca del contenido visto en clases y en el material del curso.  Contestar cada pregunta en su celda correspondiente y **no extenderse más de 5 lineas** . 🙏\n",
        "` ` "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xNJPR1kMrw9R"
      },
      "source": [
        "**Pregunta 1 (0.2 puntos): ¿Por qué el análisis del lenguaje humano es una tarea compleja? Mencione dos razones.**\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mlTBYHEptdde"
      },
      "source": [
        "` ` \n",
        "\n",
        "Es compleja por que el lenguaje humano es ambiguo en cuanto a sus reglas y adicionalmente esta siempre cambiando. Tambien tiene la cualidad de ser Sparse, puesto que se pueden combinar de infinitas formas las palabras para generar una oracion que tenga sentido, y es composicional (Compositionality) que significa que el significado de una oracion es mas que solo el significado individual de cada palabra.\n",
        "` ` "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zv8yjG94r6FC"
      },
      "source": [
        "**Pregunta 2 (0.3 puntos): ¿Cuál es la diferencia entre Natural Language Processing (NLP) y Computational Linguistics?**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0vmC3uYHtctH"
      },
      "source": [
        "` ` \n",
        "\n",
        "NLP es una diciplina aplicada que se enfoca en resolver tasks bien de definidas que involucran el lenguaje humano mientras que computational linguistics busca, con ayuda de la computacion, entender de que manera funciona el lenguaje humano.\n",
        "\n",
        "` ` "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U-0imOaFsRK5"
      },
      "source": [
        "**Pregunta 3 (0.3 puntos): ¿Qué es el Machine Learning supersivado? Donde podría aplicar este tipo de Aprendizaje**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kY6mfCE7tb1x"
      },
      "source": [
        "` ` \n",
        "\n",
        "El machine learning supervisado es un tipo de ML que en base a un data set previamente anotado de pares input, output aprende a mapear nuevos inputs a su ouput correspondiente. Un ejemplo de aplicacion en hacer un clasificador de imagenes que distinga entres si son perros o gatos.\n",
        "\n",
        "` ` "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tVMXilrYsiSZ"
      },
      "source": [
        "**Pregunta 4 (0.3 puntos): ¿Cuáles son las diferencias entre usar Deep learning y Machine Learning clásico (empirismo) para un problema de NLP? Ejemplifique con alguna task.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_QXWMzpTtBZL"
      },
      "source": [
        "` ` \n",
        "\n",
        "El Machine Learning tradicional utiliza una representacion inicial del input y en base a la data de entrenamiento previamente anotada aprende la relacion entre input y output,\n",
        "al Deep Learning, encambio, se le entega el input en un formato \"raw\" y este por si solo aprende formas de representar la data y patrones en los datos.\n",
        "\n",
        "` ` "
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        " **Pregunta 5 (0.3 puntos) Según las primeras clases, ¿Qué método clásico nos permite rankear las similitudes existentes entre documentos?, ¿Cómo son las representaciones que genera y problemas que podrían experimentar estas soluciones simples?** "
      ],
      "metadata": {
        "id": "XXc4XuVG7Loa"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "` ` \n",
        "\n",
        "Puede ocupar la representacion bag of words de los documentos llevando estos a si a un espacio vectorial. Luego se puede utilizar la similitud coseno para medir la similitud entre documentos en base a su representacion vectorial y rankear. El problema con esta forma es que las bag of words obvian completamente el orden que tienen las palabras dentro de un documento.\n",
        "\n",
        "` ` "
      ],
      "metadata": {
        "id": "rtaQ_aVE8LhH"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Pregunta 6 (0.3 puntos) Usted se encuentra realizando un modelo de clasificación de sentimientos con texto, su jefe le señala que debe eliminar las palabras mas comunes para obtener una mejor clasificación. ¿Qué palabras le señala que elimine su jefe?, ¿es acaso esto una buena idea?.**"
      ],
      "metadata": {
        "id": "EKNDNy047tcY"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "` ` \n",
        "\n",
        "El jefe se refiere a las Stop Words, palabras que son muy frecuentes en un vocabulario, cosa que comprende los articulos, nombres, pronombres, preposiciones, conjunciones, etc... En analisis de sentimiento eliminar todas las stop words puede ser problematico por que podria eliminar la palabra \"not\" cambiando completamente el sentimiento de alguna oracion. Por eljemplo \"I do not like bananas\" -> \"like bananas\"\n",
        "\n",
        "` ` "
      ],
      "metadata": {
        "id": "cRe4QaGS8MqY"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Pregunta 7 (0.3 puntos) Nombre las ventajas y desventajas de utilizar un modelo Bag of Words y TFIDF**"
      ],
      "metadata": {
        "id": "uO-79Xth8Ajd"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "` ` \n",
        "\n",
        "Desventajas:\n",
        "\n",
        "*   El modelo bag of words pierde completamente el orden en el que estan puestas las palabra.\n",
        "*   No puede capturar la semantica de un documento solo su lexico\n",
        "\n",
        "Ventajas:\n",
        "\n",
        "\n",
        "*   Es facil de calcular\n",
        "*   Elemento de la listaPemite tener la nocion de cuan frecuentes son las palabras en un documento (TF)\n",
        "*   Toma en consideracion cuan relevante es una palabra en un documento considerando su aparicion en los demas documentos (IDF) \n",
        "\n",
        "` ` "
      ],
      "metadata": {
        "id": "PROa1Csq8OAR"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6Fpsz2pQt8x5"
      },
      "source": [
        "## **2 - Preguntas prácticas 💻 (4 puntos).** ##"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YB92kQXspvbR"
      },
      "source": [
        "Esta segunda sección incluye ejercicios de programación 🤙. Leer atentamente las instrucciones entregadas a continuación para facilitar el proceso de revisión de sus trabajos.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PosWgWgRxHKp"
      },
      "source": [
        "` ` \n",
        "\n",
        "**Instrucciones:**\n",
        "\n",
        "\n",
        "\n",
        "- Escribe tu código entre las lineas de comentarios **### Aquí inicia tu código ###** y **### Aquí termina tu código ###**.\n",
        "- Cuando el ejercicio incluya un bloque llamado ***Test***, comprueba que el resultado de la ejecución coincida con el resultado esperado.\n",
        "- Recuerde siempre mantener buenas prácticas de código.\n",
        "- Está permitido sólo utilizar las librerías importadas antes del Ejercicio 1.\n",
        "- **Recordar** que: *Documento = Oración. Dataset = Corpus. Vocabulario = Typos*.\n",
        "- El **orden de los resultados** pueden variar dependiendo de su máquina, pero los valores de los resultados son los mismos.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jBrmFHXhqUww"
      },
      "source": [
        "**Ejemplo:** Implemente una función **`hello_world()`** que imprima en pantalla `\"Hello World\"`. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tu7cIsawyJHx"
      },
      "source": [
        "def hello_world():\n",
        "  ### Aquí inicia tu código ###\n",
        "  print(\"Hello World\")\n",
        "  ### Aquí termina tu código ###"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fz6klw12lwbW"
      },
      "source": [
        "***Test:***"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ac-WMk2dyQbp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d98277e5-1757-492f-8410-b23fa956cd8d"
      },
      "source": [
        "hello_world()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Hello World\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fIoiAMxtyUjQ"
      },
      "source": [
        "***Resultado esperado***: \n",
        "<table>\n",
        "    <tr> \n",
        "        <td> Hello World </td> \n",
        "    </tr>\n",
        "</table> \n",
        "\n",
        "``\n",
        "``\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dlPyrPXiH0l4"
      },
      "source": [
        "Estas son las librerías permitidas. Si quieren utilizar alguna librería adicional, pueden realizar la consulta a través de Discord. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CtP6Emjo1kF0"
      },
      "source": [
        "import codecs\n",
        "import re\n",
        "import numpy as np\n",
        "import pandas as pd"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "En caso de desarrollar la tarea desde colab, utilizar el siguiente código para cargar los archivos desde el drive:"
      ],
      "metadata": {
        "id": "Bj9Do0LdC9w2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "try:\n",
        "    from google.colab import drive\n",
        "\n",
        "    drive.mount(\"/content/drive\")\n",
        "    path = \"/content/drive/MyDrive/marcianeke.txt\"\n",
        "except: \n",
        "    print('Ignorando conexión drive-colab')"
      ],
      "metadata": {
        "id": "9EXq1VDeC-ml",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "57f9c47c-6d79-435e-9790-3402bb62bf6a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tSN4bBoY2Td4"
      },
      "source": [
        "` ` \n",
        "\n",
        "**Ejercicio 1 - *Tokenización* (0.5 puntos).** \n",
        "` `  \n",
        "` ` \n",
        "\n",
        "En el primer ejercicio veremos la dificultad 😨 de tokenizar textos no estructurados, destacando la importancia de tener librerías que realicen este trabajo. \n",
        "\n",
        "El archivo adjunto al enunciado de la tarea contiene la letra de una canción del marcianeke 👽. Utilice este texto para realizar su primera tokenización y ver que tan bien funciona su función. \n",
        "\n",
        "Ejecute el código a continuación para cargar el ejemplo. Recuerde realizar la modificación al directorio en caso que el archivo no se encuentre en el mismo directorio del Jupyter Notebook."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FnNUYlWo21g4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c409578e-c681-4f48-c7de-4ff79813a377"
      },
      "source": [
        "text = codecs.open(path, 'r', 'UTF-8').read()\n",
        "print(text)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Brr\r\n",
            "Marcianeke\r\n",
            "Vamo' a estar con Pailita\r\n",
            "Dimelo má\r\n",
            "Ando en busca de una criminal (ah, ah)\r\n",
            "Esa que el gatillo le gusta jalar (rata-ta)\r\n",
            "Que le guste flotar y fumar (brr)\r\n",
            "Tussi, keta quiere' mezclar\r\n",
            "Dimelo má\r\n",
            "\r\n",
            "Ando en busca de una criminal (ah, ah)\r\n",
            "Esa que el gatillo le gusta jalar (rata-ta)\r\n",
            "Que le guste flotar y fumar\r\n",
            "Tussi, keta pura quiere' mezclar\r\n",
            "Di-dimelo má\r\n",
            "Di-dimelo má\r\n",
            "Di-dimelo má\r\n",
            "Di-dimelo má\r\n",
            "Di-dimelo má\r\n",
            "Di-dimelo má\r\n",
            "Di-dimelo má\r\n",
            "\r\n",
            "Esperame que ahora entro yo\r\n",
            "Y lo que pide yo lo traje\r\n",
            "No visto de traje\r\n",
            "Puro corte calle, no de maquillaje\r\n",
            "Pronto coronamos y nos vamo' de viaje\r\n",
            "Tanto hit que hago que lo' culo bajen\r\n",
            "Ella se va de shopping\r\n",
            "Sale positivo si se hace el doping\r\n",
            "Baila twerk con un poco de popping\r\n",
            "Los fardos en el botín\r\n",
            "\r\n",
            "Si quieren letra llamen pa' mi booking\r\n",
            "Generando, sigo en la mía lowkey\r\n",
            "Cooking en el estudio con tu woman\r\n",
            "Tanto whisky, pisco que hasta lo' vecinos toman\r\n",
            "Si se tiran pa' aca puede que la arena coman\r\n",
            "Ja, en el chanteo titulado sin diploma\r\n",
            "Di-dimelo má\r\n",
            "Di-dimelo má\r\n",
            "Di-dimelo má\r\n",
            "Di-dimelo má\r\n",
            "Di-dimelo má\r\n",
            "Di-dimelo má\r\n",
            "Di-dimelo má\r\n",
            "\r\n",
            "Ah, pe-peligrosa\r\n",
            "Quiero ver como perreando me acosa\r\n",
            "Eso de atra' con el Gucci me lo roza\r\n",
            "Tengo tussi del naranjo me aburrio el rosa\r\n",
            "Capaz que tosa con el blunt\r\n",
            "Sprite con Flemibron\r\n",
            "Louis Vuitton, le quito la polera Champion\r\n",
            "A tu pretendiente con la fory lo espanto\r\n",
            "Puro perro, le doy de comer Champion Dog\r\n",
            "Ese toto lo corono yo\r\n",
            "La movie en play no hay stop\r\n",
            "Flow de sobra no hay stock\r\n",
            "Me la topo en la disco queda en shock\r\n",
            "My love, rica en las redes y en persona\r\n",
            "No usa Photoshop\r\n",
            "La llevo a comprar blone' al growshop\r\n",
            "En ropa interior los do'\r\n",
            "Me roza su vicky con mi boxer Top\r\n",
            "Dimelo má\r\n",
            "Ando en busca de una cri\r\n",
            "minal (ah, ah)\r\n",
            "Esa que el gatillo le gusta jalar (rata-ta)\r\n",
            "Que le guste flotar y fumar\r\n",
            "Tussi, keta quiere' mezclar\r\n",
            "Dimelo má\r\n",
            "\r\n",
            "Ando en busca de una criminal (ah, ah)\r\n",
            "Esa que el gatillo le gusta jalar (rata-ta)\r\n",
            "Que le guste flotar y fumar\r\n",
            "Tussi, keta pura quiere' mezclar\r\n",
            "Marcianeke, Pailita\r\n",
            "Young Varas\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UIaOYzMk3v1X"
      },
      "source": [
        "Implementen una función **`get_tokens()`** que reciba un texto y entregue una lista con sus tokens. Son libres de elegir la forma de tokenizar mientras no utilicen librerías con tokenizadores ya implementados. Pueden utilizar la librería **re** importada para trabajar símbolos.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5dl42-hgIhqB"
      },
      "source": [
        "Ejemplo de uso:\n",
        "\n",
        "`get_tokens('Este es un ejemplo de prueba.')` \n",
        "\n",
        "Nos entregaría:\n",
        "\n",
        "`['Este', 'es', 'un', 'ejemplo', 'de', 'prueba', '.']`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IF1RcIwq4G2x"
      },
      "source": [
        "def get_tokens(text):\n",
        "  ### Inicio del código ###\n",
        "  return re.findall(r\"[A-Za-záéíóú\\-']+|[.,()]\",text,re.UNICODE)\n",
        "  ### Fín del código ###"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KqlSpefv4_EH",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f3f1230a-251f-4a24-d1b6-d86fa7b3219f"
      },
      "source": [
        "tokens = get_tokens(text)\n",
        "tokens"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Brr',\n",
              " 'Marcianeke',\n",
              " \"Vamo'\",\n",
              " 'a',\n",
              " 'estar',\n",
              " 'con',\n",
              " 'Pailita',\n",
              " 'Dimelo',\n",
              " 'má',\n",
              " 'Ando',\n",
              " 'en',\n",
              " 'busca',\n",
              " 'de',\n",
              " 'una',\n",
              " 'criminal',\n",
              " '(',\n",
              " 'ah',\n",
              " ',',\n",
              " 'ah',\n",
              " ')',\n",
              " 'Esa',\n",
              " 'que',\n",
              " 'el',\n",
              " 'gatillo',\n",
              " 'le',\n",
              " 'gusta',\n",
              " 'jalar',\n",
              " '(',\n",
              " 'rata-ta',\n",
              " ')',\n",
              " 'Que',\n",
              " 'le',\n",
              " 'guste',\n",
              " 'flotar',\n",
              " 'y',\n",
              " 'fumar',\n",
              " '(',\n",
              " 'brr',\n",
              " ')',\n",
              " 'Tussi',\n",
              " ',',\n",
              " 'keta',\n",
              " \"quiere'\",\n",
              " 'mezclar',\n",
              " 'Dimelo',\n",
              " 'má',\n",
              " 'Ando',\n",
              " 'en',\n",
              " 'busca',\n",
              " 'de',\n",
              " 'una',\n",
              " 'criminal',\n",
              " '(',\n",
              " 'ah',\n",
              " ',',\n",
              " 'ah',\n",
              " ')',\n",
              " 'Esa',\n",
              " 'que',\n",
              " 'el',\n",
              " 'gatillo',\n",
              " 'le',\n",
              " 'gusta',\n",
              " 'jalar',\n",
              " '(',\n",
              " 'rata-ta',\n",
              " ')',\n",
              " 'Que',\n",
              " 'le',\n",
              " 'guste',\n",
              " 'flotar',\n",
              " 'y',\n",
              " 'fumar',\n",
              " 'Tussi',\n",
              " ',',\n",
              " 'keta',\n",
              " 'pura',\n",
              " \"quiere'\",\n",
              " 'mezclar',\n",
              " 'Di-dimelo',\n",
              " 'má',\n",
              " 'Di-dimelo',\n",
              " 'má',\n",
              " 'Di-dimelo',\n",
              " 'má',\n",
              " 'Di-dimelo',\n",
              " 'má',\n",
              " 'Di-dimelo',\n",
              " 'má',\n",
              " 'Di-dimelo',\n",
              " 'má',\n",
              " 'Di-dimelo',\n",
              " 'má',\n",
              " 'Esperame',\n",
              " 'que',\n",
              " 'ahora',\n",
              " 'entro',\n",
              " 'yo',\n",
              " 'Y',\n",
              " 'lo',\n",
              " 'que',\n",
              " 'pide',\n",
              " 'yo',\n",
              " 'lo',\n",
              " 'traje',\n",
              " 'No',\n",
              " 'visto',\n",
              " 'de',\n",
              " 'traje',\n",
              " 'Puro',\n",
              " 'corte',\n",
              " 'calle',\n",
              " ',',\n",
              " 'no',\n",
              " 'de',\n",
              " 'maquillaje',\n",
              " 'Pronto',\n",
              " 'coronamos',\n",
              " 'y',\n",
              " 'nos',\n",
              " \"vamo'\",\n",
              " 'de',\n",
              " 'viaje',\n",
              " 'Tanto',\n",
              " 'hit',\n",
              " 'que',\n",
              " 'hago',\n",
              " 'que',\n",
              " \"lo'\",\n",
              " 'culo',\n",
              " 'bajen',\n",
              " 'Ella',\n",
              " 'se',\n",
              " 'va',\n",
              " 'de',\n",
              " 'shopping',\n",
              " 'Sale',\n",
              " 'positivo',\n",
              " 'si',\n",
              " 'se',\n",
              " 'hace',\n",
              " 'el',\n",
              " 'doping',\n",
              " 'Baila',\n",
              " 'twerk',\n",
              " 'con',\n",
              " 'un',\n",
              " 'poco',\n",
              " 'de',\n",
              " 'popping',\n",
              " 'Los',\n",
              " 'fardos',\n",
              " 'en',\n",
              " 'el',\n",
              " 'botín',\n",
              " 'Si',\n",
              " 'quieren',\n",
              " 'letra',\n",
              " 'llamen',\n",
              " \"pa'\",\n",
              " 'mi',\n",
              " 'booking',\n",
              " 'Generando',\n",
              " ',',\n",
              " 'sigo',\n",
              " 'en',\n",
              " 'la',\n",
              " 'mía',\n",
              " 'lowkey',\n",
              " 'Cooking',\n",
              " 'en',\n",
              " 'el',\n",
              " 'estudio',\n",
              " 'con',\n",
              " 'tu',\n",
              " 'woman',\n",
              " 'Tanto',\n",
              " 'whisky',\n",
              " ',',\n",
              " 'pisco',\n",
              " 'que',\n",
              " 'hasta',\n",
              " \"lo'\",\n",
              " 'vecinos',\n",
              " 'toman',\n",
              " 'Si',\n",
              " 'se',\n",
              " 'tiran',\n",
              " \"pa'\",\n",
              " 'aca',\n",
              " 'puede',\n",
              " 'que',\n",
              " 'la',\n",
              " 'arena',\n",
              " 'coman',\n",
              " 'Ja',\n",
              " ',',\n",
              " 'en',\n",
              " 'el',\n",
              " 'chanteo',\n",
              " 'titulado',\n",
              " 'sin',\n",
              " 'diploma',\n",
              " 'Di-dimelo',\n",
              " 'má',\n",
              " 'Di-dimelo',\n",
              " 'má',\n",
              " 'Di-dimelo',\n",
              " 'má',\n",
              " 'Di-dimelo',\n",
              " 'má',\n",
              " 'Di-dimelo',\n",
              " 'má',\n",
              " 'Di-dimelo',\n",
              " 'má',\n",
              " 'Di-dimelo',\n",
              " 'má',\n",
              " 'Ah',\n",
              " ',',\n",
              " 'pe-peligrosa',\n",
              " 'Quiero',\n",
              " 'ver',\n",
              " 'como',\n",
              " 'perreando',\n",
              " 'me',\n",
              " 'acosa',\n",
              " 'Eso',\n",
              " 'de',\n",
              " \"atra'\",\n",
              " 'con',\n",
              " 'el',\n",
              " 'Gucci',\n",
              " 'me',\n",
              " 'lo',\n",
              " 'roza',\n",
              " 'Tengo',\n",
              " 'tussi',\n",
              " 'del',\n",
              " 'naranjo',\n",
              " 'me',\n",
              " 'aburrio',\n",
              " 'el',\n",
              " 'rosa',\n",
              " 'Capaz',\n",
              " 'que',\n",
              " 'tosa',\n",
              " 'con',\n",
              " 'el',\n",
              " 'blunt',\n",
              " 'Sprite',\n",
              " 'con',\n",
              " 'Flemibron',\n",
              " 'Louis',\n",
              " 'Vuitton',\n",
              " ',',\n",
              " 'le',\n",
              " 'quito',\n",
              " 'la',\n",
              " 'polera',\n",
              " 'Champion',\n",
              " 'A',\n",
              " 'tu',\n",
              " 'pretendiente',\n",
              " 'con',\n",
              " 'la',\n",
              " 'fory',\n",
              " 'lo',\n",
              " 'espanto',\n",
              " 'Puro',\n",
              " 'perro',\n",
              " ',',\n",
              " 'le',\n",
              " 'doy',\n",
              " 'de',\n",
              " 'comer',\n",
              " 'Champion',\n",
              " 'Dog',\n",
              " 'Ese',\n",
              " 'toto',\n",
              " 'lo',\n",
              " 'corono',\n",
              " 'yo',\n",
              " 'La',\n",
              " 'movie',\n",
              " 'en',\n",
              " 'play',\n",
              " 'no',\n",
              " 'hay',\n",
              " 'stop',\n",
              " 'Flow',\n",
              " 'de',\n",
              " 'sobra',\n",
              " 'no',\n",
              " 'hay',\n",
              " 'stock',\n",
              " 'Me',\n",
              " 'la',\n",
              " 'topo',\n",
              " 'en',\n",
              " 'la',\n",
              " 'disco',\n",
              " 'queda',\n",
              " 'en',\n",
              " 'shock',\n",
              " 'My',\n",
              " 'love',\n",
              " ',',\n",
              " 'rica',\n",
              " 'en',\n",
              " 'las',\n",
              " 'redes',\n",
              " 'y',\n",
              " 'en',\n",
              " 'persona',\n",
              " 'No',\n",
              " 'usa',\n",
              " 'Photoshop',\n",
              " 'La',\n",
              " 'llevo',\n",
              " 'a',\n",
              " 'comprar',\n",
              " \"blone'\",\n",
              " 'al',\n",
              " 'growshop',\n",
              " 'En',\n",
              " 'ropa',\n",
              " 'interior',\n",
              " 'los',\n",
              " \"do'\",\n",
              " 'Me',\n",
              " 'roza',\n",
              " 'su',\n",
              " 'vicky',\n",
              " 'con',\n",
              " 'mi',\n",
              " 'boxer',\n",
              " 'Top',\n",
              " 'Dimelo',\n",
              " 'má',\n",
              " 'Ando',\n",
              " 'en',\n",
              " 'busca',\n",
              " 'de',\n",
              " 'una',\n",
              " 'cri',\n",
              " 'minal',\n",
              " '(',\n",
              " 'ah',\n",
              " ',',\n",
              " 'ah',\n",
              " ')',\n",
              " 'Esa',\n",
              " 'que',\n",
              " 'el',\n",
              " 'gatillo',\n",
              " 'le',\n",
              " 'gusta',\n",
              " 'jalar',\n",
              " '(',\n",
              " 'rata-ta',\n",
              " ')',\n",
              " 'Que',\n",
              " 'le',\n",
              " 'guste',\n",
              " 'flotar',\n",
              " 'y',\n",
              " 'fumar',\n",
              " 'Tussi',\n",
              " ',',\n",
              " 'keta',\n",
              " \"quiere'\",\n",
              " 'mezclar',\n",
              " 'Dimelo',\n",
              " 'má',\n",
              " 'Ando',\n",
              " 'en',\n",
              " 'busca',\n",
              " 'de',\n",
              " 'una',\n",
              " 'criminal',\n",
              " '(',\n",
              " 'ah',\n",
              " ',',\n",
              " 'ah',\n",
              " ')',\n",
              " 'Esa',\n",
              " 'que',\n",
              " 'el',\n",
              " 'gatillo',\n",
              " 'le',\n",
              " 'gusta',\n",
              " 'jalar',\n",
              " '(',\n",
              " 'rata-ta',\n",
              " ')',\n",
              " 'Que',\n",
              " 'le',\n",
              " 'guste',\n",
              " 'flotar',\n",
              " 'y',\n",
              " 'fumar',\n",
              " 'Tussi',\n",
              " ',',\n",
              " 'keta',\n",
              " 'pura',\n",
              " \"quiere'\",\n",
              " 'mezclar',\n",
              " 'Marcianeke',\n",
              " ',',\n",
              " 'Pailita',\n",
              " 'Young',\n",
              " 'Varas']"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TPbgTvAW-stF"
      },
      "source": [
        "**Describa cuáles fueron sus supuestos para realizar la tokenización y compare sus tokens con los entregados por la librería nltk en el bloque de código de más abajo.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jGQ7CJy3-3aH"
      },
      "source": [
        "El apostrofe \" ' \" y el guion \"-\" son considerados parte de las palabras en el tokenizador creado, en el de nltk en cambio son separados de las palabras de las cuales forma parte. Por otro lado el punto \".\", coma \",\" y los parentesis \"(\" \")\" son considerados como tokens por si solos en el tokenizador creado y el nltk "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YYtmAXTr9KXK",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "beaeb71d-055b-40f1-f886-c56737f53fb4"
      },
      "source": [
        "from nltk.tokenize import wordpunct_tokenize \n",
        "nltk_tokens = wordpunct_tokenize(text)\n",
        "nltk_tokens"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Brr',\n",
              " 'Marcianeke',\n",
              " 'Vamo',\n",
              " \"'\",\n",
              " 'a',\n",
              " 'estar',\n",
              " 'con',\n",
              " 'Pailita',\n",
              " 'Dimelo',\n",
              " 'má',\n",
              " 'Ando',\n",
              " 'en',\n",
              " 'busca',\n",
              " 'de',\n",
              " 'una',\n",
              " 'criminal',\n",
              " '(',\n",
              " 'ah',\n",
              " ',',\n",
              " 'ah',\n",
              " ')',\n",
              " 'Esa',\n",
              " 'que',\n",
              " 'el',\n",
              " 'gatillo',\n",
              " 'le',\n",
              " 'gusta',\n",
              " 'jalar',\n",
              " '(',\n",
              " 'rata',\n",
              " '-',\n",
              " 'ta',\n",
              " ')',\n",
              " 'Que',\n",
              " 'le',\n",
              " 'guste',\n",
              " 'flotar',\n",
              " 'y',\n",
              " 'fumar',\n",
              " '(',\n",
              " 'brr',\n",
              " ')',\n",
              " 'Tussi',\n",
              " ',',\n",
              " 'keta',\n",
              " 'quiere',\n",
              " \"'\",\n",
              " 'mezclar',\n",
              " 'Dimelo',\n",
              " 'má',\n",
              " 'Ando',\n",
              " 'en',\n",
              " 'busca',\n",
              " 'de',\n",
              " 'una',\n",
              " 'criminal',\n",
              " '(',\n",
              " 'ah',\n",
              " ',',\n",
              " 'ah',\n",
              " ')',\n",
              " 'Esa',\n",
              " 'que',\n",
              " 'el',\n",
              " 'gatillo',\n",
              " 'le',\n",
              " 'gusta',\n",
              " 'jalar',\n",
              " '(',\n",
              " 'rata',\n",
              " '-',\n",
              " 'ta',\n",
              " ')',\n",
              " 'Que',\n",
              " 'le',\n",
              " 'guste',\n",
              " 'flotar',\n",
              " 'y',\n",
              " 'fumar',\n",
              " 'Tussi',\n",
              " ',',\n",
              " 'keta',\n",
              " 'pura',\n",
              " 'quiere',\n",
              " \"'\",\n",
              " 'mezclar',\n",
              " 'Di',\n",
              " '-',\n",
              " 'dimelo',\n",
              " 'má',\n",
              " 'Di',\n",
              " '-',\n",
              " 'dimelo',\n",
              " 'má',\n",
              " 'Di',\n",
              " '-',\n",
              " 'dimelo',\n",
              " 'má',\n",
              " 'Di',\n",
              " '-',\n",
              " 'dimelo',\n",
              " 'má',\n",
              " 'Di',\n",
              " '-',\n",
              " 'dimelo',\n",
              " 'má',\n",
              " 'Di',\n",
              " '-',\n",
              " 'dimelo',\n",
              " 'má',\n",
              " 'Di',\n",
              " '-',\n",
              " 'dimelo',\n",
              " 'má',\n",
              " 'Esperame',\n",
              " 'que',\n",
              " 'ahora',\n",
              " 'entro',\n",
              " 'yo',\n",
              " 'Y',\n",
              " 'lo',\n",
              " 'que',\n",
              " 'pide',\n",
              " 'yo',\n",
              " 'lo',\n",
              " 'traje',\n",
              " 'No',\n",
              " 'visto',\n",
              " 'de',\n",
              " 'traje',\n",
              " 'Puro',\n",
              " 'corte',\n",
              " 'calle',\n",
              " ',',\n",
              " 'no',\n",
              " 'de',\n",
              " 'maquillaje',\n",
              " 'Pronto',\n",
              " 'coronamos',\n",
              " 'y',\n",
              " 'nos',\n",
              " 'vamo',\n",
              " \"'\",\n",
              " 'de',\n",
              " 'viaje',\n",
              " 'Tanto',\n",
              " 'hit',\n",
              " 'que',\n",
              " 'hago',\n",
              " 'que',\n",
              " 'lo',\n",
              " \"'\",\n",
              " 'culo',\n",
              " 'bajen',\n",
              " 'Ella',\n",
              " 'se',\n",
              " 'va',\n",
              " 'de',\n",
              " 'shopping',\n",
              " 'Sale',\n",
              " 'positivo',\n",
              " 'si',\n",
              " 'se',\n",
              " 'hace',\n",
              " 'el',\n",
              " 'doping',\n",
              " 'Baila',\n",
              " 'twerk',\n",
              " 'con',\n",
              " 'un',\n",
              " 'poco',\n",
              " 'de',\n",
              " 'popping',\n",
              " 'Los',\n",
              " 'fardos',\n",
              " 'en',\n",
              " 'el',\n",
              " 'botín',\n",
              " 'Si',\n",
              " 'quieren',\n",
              " 'letra',\n",
              " 'llamen',\n",
              " 'pa',\n",
              " \"'\",\n",
              " 'mi',\n",
              " 'booking',\n",
              " 'Generando',\n",
              " ',',\n",
              " 'sigo',\n",
              " 'en',\n",
              " 'la',\n",
              " 'mía',\n",
              " 'lowkey',\n",
              " 'Cooking',\n",
              " 'en',\n",
              " 'el',\n",
              " 'estudio',\n",
              " 'con',\n",
              " 'tu',\n",
              " 'woman',\n",
              " 'Tanto',\n",
              " 'whisky',\n",
              " ',',\n",
              " 'pisco',\n",
              " 'que',\n",
              " 'hasta',\n",
              " 'lo',\n",
              " \"'\",\n",
              " 'vecinos',\n",
              " 'toman',\n",
              " 'Si',\n",
              " 'se',\n",
              " 'tiran',\n",
              " 'pa',\n",
              " \"'\",\n",
              " 'aca',\n",
              " 'puede',\n",
              " 'que',\n",
              " 'la',\n",
              " 'arena',\n",
              " 'coman',\n",
              " 'Ja',\n",
              " ',',\n",
              " 'en',\n",
              " 'el',\n",
              " 'chanteo',\n",
              " 'titulado',\n",
              " 'sin',\n",
              " 'diploma',\n",
              " 'Di',\n",
              " '-',\n",
              " 'dimelo',\n",
              " 'má',\n",
              " 'Di',\n",
              " '-',\n",
              " 'dimelo',\n",
              " 'má',\n",
              " 'Di',\n",
              " '-',\n",
              " 'dimelo',\n",
              " 'má',\n",
              " 'Di',\n",
              " '-',\n",
              " 'dimelo',\n",
              " 'má',\n",
              " 'Di',\n",
              " '-',\n",
              " 'dimelo',\n",
              " 'má',\n",
              " 'Di',\n",
              " '-',\n",
              " 'dimelo',\n",
              " 'má',\n",
              " 'Di',\n",
              " '-',\n",
              " 'dimelo',\n",
              " 'má',\n",
              " 'Ah',\n",
              " ',',\n",
              " 'pe',\n",
              " '-',\n",
              " 'peligrosa',\n",
              " 'Quiero',\n",
              " 'ver',\n",
              " 'como',\n",
              " 'perreando',\n",
              " 'me',\n",
              " 'acosa',\n",
              " 'Eso',\n",
              " 'de',\n",
              " 'atra',\n",
              " \"'\",\n",
              " 'con',\n",
              " 'el',\n",
              " 'Gucci',\n",
              " 'me',\n",
              " 'lo',\n",
              " 'roza',\n",
              " 'Tengo',\n",
              " 'tussi',\n",
              " 'del',\n",
              " 'naranjo',\n",
              " 'me',\n",
              " 'aburrio',\n",
              " 'el',\n",
              " 'rosa',\n",
              " 'Capaz',\n",
              " 'que',\n",
              " 'tosa',\n",
              " 'con',\n",
              " 'el',\n",
              " 'blunt',\n",
              " 'Sprite',\n",
              " 'con',\n",
              " 'Flemibron',\n",
              " 'Louis',\n",
              " 'Vuitton',\n",
              " ',',\n",
              " 'le',\n",
              " 'quito',\n",
              " 'la',\n",
              " 'polera',\n",
              " 'Champion',\n",
              " 'A',\n",
              " 'tu',\n",
              " 'pretendiente',\n",
              " 'con',\n",
              " 'la',\n",
              " 'fory',\n",
              " 'lo',\n",
              " 'espanto',\n",
              " 'Puro',\n",
              " 'perro',\n",
              " ',',\n",
              " 'le',\n",
              " 'doy',\n",
              " 'de',\n",
              " 'comer',\n",
              " 'Champion',\n",
              " 'Dog',\n",
              " 'Ese',\n",
              " 'toto',\n",
              " 'lo',\n",
              " 'corono',\n",
              " 'yo',\n",
              " 'La',\n",
              " 'movie',\n",
              " 'en',\n",
              " 'play',\n",
              " 'no',\n",
              " 'hay',\n",
              " 'stop',\n",
              " 'Flow',\n",
              " 'de',\n",
              " 'sobra',\n",
              " 'no',\n",
              " 'hay',\n",
              " 'stock',\n",
              " 'Me',\n",
              " 'la',\n",
              " 'topo',\n",
              " 'en',\n",
              " 'la',\n",
              " 'disco',\n",
              " 'queda',\n",
              " 'en',\n",
              " 'shock',\n",
              " 'My',\n",
              " 'love',\n",
              " ',',\n",
              " 'rica',\n",
              " 'en',\n",
              " 'las',\n",
              " 'redes',\n",
              " 'y',\n",
              " 'en',\n",
              " 'persona',\n",
              " 'No',\n",
              " 'usa',\n",
              " 'Photoshop',\n",
              " 'La',\n",
              " 'llevo',\n",
              " 'a',\n",
              " 'comprar',\n",
              " 'blone',\n",
              " \"'\",\n",
              " 'al',\n",
              " 'growshop',\n",
              " 'En',\n",
              " 'ropa',\n",
              " 'interior',\n",
              " 'los',\n",
              " 'do',\n",
              " \"'\",\n",
              " 'Me',\n",
              " 'roza',\n",
              " 'su',\n",
              " 'vicky',\n",
              " 'con',\n",
              " 'mi',\n",
              " 'boxer',\n",
              " 'Top',\n",
              " 'Dimelo',\n",
              " 'má',\n",
              " 'Ando',\n",
              " 'en',\n",
              " 'busca',\n",
              " 'de',\n",
              " 'una',\n",
              " 'cri',\n",
              " 'minal',\n",
              " '(',\n",
              " 'ah',\n",
              " ',',\n",
              " 'ah',\n",
              " ')',\n",
              " 'Esa',\n",
              " 'que',\n",
              " 'el',\n",
              " 'gatillo',\n",
              " 'le',\n",
              " 'gusta',\n",
              " 'jalar',\n",
              " '(',\n",
              " 'rata',\n",
              " '-',\n",
              " 'ta',\n",
              " ')',\n",
              " 'Que',\n",
              " 'le',\n",
              " 'guste',\n",
              " 'flotar',\n",
              " 'y',\n",
              " 'fumar',\n",
              " 'Tussi',\n",
              " ',',\n",
              " 'keta',\n",
              " 'quiere',\n",
              " \"'\",\n",
              " 'mezclar',\n",
              " 'Dimelo',\n",
              " 'má',\n",
              " 'Ando',\n",
              " 'en',\n",
              " 'busca',\n",
              " 'de',\n",
              " 'una',\n",
              " 'criminal',\n",
              " '(',\n",
              " 'ah',\n",
              " ',',\n",
              " 'ah',\n",
              " ')',\n",
              " 'Esa',\n",
              " 'que',\n",
              " 'el',\n",
              " 'gatillo',\n",
              " 'le',\n",
              " 'gusta',\n",
              " 'jalar',\n",
              " '(',\n",
              " 'rata',\n",
              " '-',\n",
              " 'ta',\n",
              " ')',\n",
              " 'Que',\n",
              " 'le',\n",
              " 'guste',\n",
              " 'flotar',\n",
              " 'y',\n",
              " 'fumar',\n",
              " 'Tussi',\n",
              " ',',\n",
              " 'keta',\n",
              " 'pura',\n",
              " 'quiere',\n",
              " \"'\",\n",
              " 'mezclar',\n",
              " 'Marcianeke',\n",
              " ',',\n",
              " 'Pailita',\n",
              " 'Young',\n",
              " 'Varas']"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s5QKlXAZwN1L"
      },
      "source": [
        "` `  \n",
        "` ` \n",
        "\n",
        "**Ejercicio 2 - *Stopwords y Stemming* (1 punto).** \n",
        "` `  \n",
        "` ` \n",
        "\n",
        "Considere el siguiente corpus:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sEp83zESwb2j"
      },
      "source": [
        "dataset = [\"I like human languages\", \"I like programming languages\", \"Spanish is my favorite language\"]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gmjdlJWuyS2E"
      },
      "source": [
        "Diseñe una función **`get_vocab()`** que extraiga los typos de este corpus solamente tokenizando. Puede utilizar la función del Ejercicio 1."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UudC-b6TzZgw"
      },
      "source": [
        "def get_vocab(dataset):\n",
        "  ### Aquí inicia tu código ###\n",
        "  tokens = []\n",
        "  for doc in dataset:\n",
        "    tokens.extend(get_tokens(doc))\n",
        "  return list(set(tokens))\n",
        "  ### Aquí termina tu código ###"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5-m32IoNmSwM"
      },
      "source": [
        "***Test:***"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BNzPKiAx0Aa1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7ec5095f-7d23-46f9-deba-51faeb402a84"
      },
      "source": [
        "vocab = get_vocab(dataset)\n",
        "vocab"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['my',\n",
              " 'like',\n",
              " 'programming',\n",
              " 'languages',\n",
              " 'Spanish',\n",
              " 'language',\n",
              " 'is',\n",
              " 'favorite',\n",
              " 'human',\n",
              " 'I']"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qZLV2hWf9FN7"
      },
      "source": [
        "``\n",
        "``\n",
        "\n",
        "***Resultado esperado***: \n",
        "<table>\n",
        "    <tr> \n",
        "        <td>['favorite',\n",
        " 'Spanish',\n",
        " 'language',\n",
        " 'I',\n",
        " 'like',\n",
        " 'programming',\n",
        " 'languages',\n",
        " 'my',\n",
        " 'human',\n",
        " 'is'] </td> \n",
        "    </tr>\n",
        "</table> \n",
        "\n",
        "``\n",
        "``"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L3KB0fL2zk2v"
      },
      "source": [
        "Ahora diseñe reglas que usted estime convenientes tanto de **Stemming** como de **Stopwords**. Implemente una función que reciba una lista con los elementos del vocabulario, le aplique sus reglas y devuelva el vocabulario actualizado. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iY7g67Ml0aby"
      },
      "source": [
        "**Reglas:** Estan hechas ad hoc al corpus\n",
        "\n",
        "*   En cuanto a las stopwords en la funcion remove_stopwords hay una lista harcodeada de stopwords, esta no es exauhstiva (como lo descrito en la **Pregunta 6**) y esta hecha especificamente para el corpus dado. \n",
        "*   En cuanto a al stemming se ocupo la reglA 's' -> '' del algoritmo de porter puesto que el la unica que aplica para el corpus."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def remove_stopwords(vocab):\n",
        "  stopwords = [\"is\", \"my\", \"I\"]\n",
        "  for stopword in stopwords:\n",
        "    if stopword in vocab:\n",
        "      vocab.remove(stopword)\n",
        "  return vocab\n",
        "\n",
        "def stemming(token):\n",
        "  if token[-1] == \"s\":\n",
        "    return token[:-1]\n",
        "  else:\n",
        "    return token"
      ],
      "metadata": {
        "id": "Zg2j3abctXjM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lrKDtM_H0XlE"
      },
      "source": [
        "def pre_processing(vocab): \n",
        "  ### Aquí inicia tu código ###\n",
        "  vocab = remove_stopwords(vocab)\n",
        "  vocab = list(set(map(stemming, vocab)))\n",
        "  return vocab\n",
        "  ### Aquí termina tu código ###"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "onOSuS-_mL2F",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2333e0a6-8960-4174-cf53-2a53952378d0"
      },
      "source": [
        "vocab = pre_processing(vocab)\n",
        "vocab"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['like', 'programming', 'Spanish', 'language', 'favorite', 'human']"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "65IwHx11uA75"
      },
      "source": [
        "\n",
        "**Ejercicio 3 - *Bag of Words* 🐶🐈(0.5 puntos).** \n",
        " \n",
        "\n",
        "\n",
        "Considere el siguiente corpus, donde cada elemento del arreglo representa un documento:\n",
        "\n",
        "**disclaimer: El orden de los resultados pueden variar**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Rh-utMuozhsK"
      },
      "source": [
        "d0 = 'El perro se come la comida y después se duerme'\n",
        "d1 = 'El perro se despierta y después empieza a ladrar'\n",
        "d2 = 'El perro ladra y después se come la comida'\n",
        "d3 = 'El gato se come la comida y después se duerme'\n",
        "d4 = 'El gato se despierta y después empieza a maullar'\n",
        "d5 = 'El gato maulla y después se come la comida'\n",
        "dataset = [d0, d1, d2, d3, d4, d5]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WH7ne8C6ltvE"
      },
      "source": [
        "El objetivo de este ejercicio es determinar cuáles de  los documentos entregados son los más similares entre sí. Para ello utilizaremos la técnica TF-IDF. \n",
        "\n",
        "Como los algoritmos de Machine Learning no comprenden el texto en lenguaje natural, estos documentos deben ser convertidos a vectores numéricos. La representación más simple vista en clases es el **Bag of Words**, método mediante el cuál se cuentan las apariciones de cada palabra en cada uno de los documentos entregados.\n",
        "\n",
        "Implemente la función **`bag_of_words()`**, que reciba como input un arreglo de documentos y devuelva un pandas dataframe con la representación Bag of Words de los documentos entregados. En esta representación las columnas son el vocabulario y las filas representa las apariciones de cada una de las palabras en los documentos. En otras palabras, cada fila representa el bow de un determinado documento.\n",
        "\n",
        "\n",
        "Por ejemplo para el siguiente dataset: \n",
        "\n",
        "```\n",
        "dataset = ['El perro ladra', 'El perro come']\n",
        "```\n",
        "\n",
        "Debiese entregarnos lo siguiente:\n",
        "\n",
        "\n",
        "|   | el | perro | ladra | come |\n",
        "|---|----|-------|------|-------|\n",
        "| 0 | 1  | 1     | 1    | 0     |\n",
        "| 1 | 1  | 1     | 0    | 1     |\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kXDMAyFmnq5j"
      },
      "source": [
        "def bag_of_words(dataset):\n",
        "  ### Aquí inicia tu código ###\n",
        "  doc_names = [\"d0\",\"d1\",\"d2\",\"d3\",\"d4\",\"d5\"]\n",
        "  vocab = get_vocab(dataset)\n",
        "  df = pd.DataFrame(index=doc_names, columns=vocab)\n",
        "  print(df[\"perro\"][\"d0\"])\n",
        "  for i in range(len(dataset)):\n",
        "    for j in range(len(vocab)):\n",
        "      df[vocab[j]][doc_names[i]] = get_tokens(dataset[i]).count(vocab[j])\n",
        "  return df\n",
        "  ### Aquí termina tu código ###"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "okfo-nEQmW1R"
      },
      "source": [
        "***Test:***"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T_Kk9GwCoDW8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 256
        },
        "outputId": "ff35db93-8da2-40c7-f298-35965d88bd51"
      },
      "source": [
        "dataset_bow = bag_of_words(dataset)\n",
        "dataset_bow"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "nan\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   después despierta empieza maullar  y comida come gato El duerme  a ladra  \\\n",
              "d0       1         0       0       0  1      1    1    0  1      1  0     0   \n",
              "d1       1         1       1       0  1      0    0    0  1      0  1     0   \n",
              "d2       1         0       0       0  1      1    1    0  1      0  0     1   \n",
              "d3       1         0       0       0  1      1    1    1  1      1  0     0   \n",
              "d4       1         1       1       1  1      0    0    1  1      0  1     0   \n",
              "d5       1         0       0       0  1      1    1    1  1      0  0     0   \n",
              "\n",
              "   maulla la se ladrar perro  \n",
              "d0      0  1  2      0     1  \n",
              "d1      0  0  1      1     1  \n",
              "d2      0  1  1      0     1  \n",
              "d3      0  1  2      0     0  \n",
              "d4      0  0  1      0     0  \n",
              "d5      1  1  1      0     0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-5655f06c-9634-4d3c-8230-d9803c188a06\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>después</th>\n",
              "      <th>despierta</th>\n",
              "      <th>empieza</th>\n",
              "      <th>maullar</th>\n",
              "      <th>y</th>\n",
              "      <th>comida</th>\n",
              "      <th>come</th>\n",
              "      <th>gato</th>\n",
              "      <th>El</th>\n",
              "      <th>duerme</th>\n",
              "      <th>a</th>\n",
              "      <th>ladra</th>\n",
              "      <th>maulla</th>\n",
              "      <th>la</th>\n",
              "      <th>se</th>\n",
              "      <th>ladrar</th>\n",
              "      <th>perro</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>d0</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>d1</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>d2</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>d3</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>d4</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>d5</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5655f06c-9634-4d3c-8230-d9803c188a06')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-5655f06c-9634-4d3c-8230-d9803c188a06 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-5655f06c-9634-4d3c-8230-d9803c188a06');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oeR5ADGz-MPa"
      },
      "source": [
        "``\n",
        "``\n",
        "\n",
        "***Resultado esperado***: \n",
        "\n",
        "|    | El | perro | se | come | la | comida | y | después | duerme | despierta | empieza | a | ladrar | ladra | gato | maullar | maulla |\n",
        "|----|---:|------:|---:|-----:|---:|-------:|--:|--------:|-------:|----------:|--------:|--:|-------:|------:|-----:|--------:|-------:|\n",
        "| d0 |  1 |     1 |  2 |    1 |  1 |      1 | 1 |       1 |      1 |         0 |       0 | 0 |      0 |     0 |    0 |       0 |      0 |\n",
        "| d1 |  1 |     1 |  1 |    0 |  0 |      0 | 1 |       1 |      0 |         1 |       1 | 1 |      1 |     0 |    0 |       0 |      0 |\n",
        "| d2 |  1 |     1 |  1 |    1 |  1 |      1 | 1 |       1 |      0 |         0 |       0 | 0 |      0 |     1 |    0 |       0 |      0 |\n",
        "| d3 |  1 |     0 |  2 |    1 |  1 |      1 | 1 |       1 |      1 |         0 |       0 | 0 |      0 |     0 |    1 |       0 |      0 |\n",
        "| d4 |  1 |     0 |  1 |    0 |  0 |      0 | 1 |       1 |      0 |         1 |       1 | 1 |      0 |     0 |    1 |       1 |      0 |\n",
        "| d5 |  1 |     0 |  1 |    1 |  1 |      1 | 1 |       1 |      0 |         0 |       0 | 0 |      0 |     0 |    1 |       0 |      1 |\n",
        "\n",
        "``\n",
        "``"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K4OXMz7opWcd"
      },
      "source": [
        "` `  \n",
        "` ` \n",
        "\n",
        "**Ejercicio 4 - *Calcular TF* (0.5 puntos):** Ahora debemos usar el dataframe del ejercicio anterior para calcular la matriz de TF normalizada por la máxima frecuencia ${max_i({\\text{tf}_{i,j})}}$, donde\n",
        "i corresponde al índice de las filas (bow) y j al de las columnas (palabras). Es decir, dividir cada bow en la cantidad de veces de la palabra que aparezca más veces en ese vector. \n",
        "\n",
        "\n",
        "$$\\text{nft}_{i,j} = \\frac{\\text{tf}_{i,j}}{max_i({\\text{tf}_{i,j})}}$$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xWE16xhBpswc"
      },
      "source": [
        "def calc_tf(dataset_bow):\n",
        "    ### Aquí inicia tu código ###\n",
        "    doc_names = [\"d0\",\"d1\",\"d2\",\"d3\",\"d4\",\"d5\"]\n",
        "    vocab = dataset_bow.columns\n",
        "    max_tf = dataset_bow.max(axis=1)\n",
        "    for i in range(len(dataset_bow.index)):\n",
        "      for j in range(len(dataset_bow.columns)):\n",
        "        dataset_bow[vocab[j]][doc_names[i]] = dataset_bow[vocab[j]][doc_names[i]]/max_tf[i]\n",
        "        \n",
        "    return dataset_bow\n",
        "    ### Aquí termina tu código ###"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AZQPZe3JmYqy"
      },
      "source": [
        "***Test:***"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YQ2h8jEYp4nZ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 237
        },
        "outputId": "2b6f0dd8-f3df-4ac0-e52a-4eb80f4127e9"
      },
      "source": [
        "tf = calc_tf(dataset_bow)\n",
        "tf"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   después despierta empieza maullar    y comida come gato   El duerme    a  \\\n",
              "d0     0.5       0.0     0.0     0.0  0.5    0.5  0.5  0.0  0.5    0.5  0.0   \n",
              "d1     1.0       1.0     1.0     0.0  1.0    0.0  0.0  0.0  1.0    0.0  1.0   \n",
              "d2     1.0       0.0     0.0     0.0  1.0    1.0  1.0  0.0  1.0    0.0  0.0   \n",
              "d3     0.5       0.0     0.0     0.0  0.5    0.5  0.5  0.5  0.5    0.5  0.0   \n",
              "d4     1.0       1.0     1.0     1.0  1.0    0.0  0.0  1.0  1.0    0.0  1.0   \n",
              "d5     1.0       0.0     0.0     0.0  1.0    1.0  1.0  1.0  1.0    0.0  0.0   \n",
              "\n",
              "   ladra maulla   la   se ladrar perro  \n",
              "d0   0.0    0.0  0.5  1.0    0.0   0.5  \n",
              "d1   0.0    0.0  0.0  1.0    1.0   1.0  \n",
              "d2   1.0    0.0  1.0  1.0    0.0   1.0  \n",
              "d3   0.0    0.0  0.5  1.0    0.0   0.0  \n",
              "d4   0.0    0.0  0.0  1.0    0.0   0.0  \n",
              "d5   0.0    1.0  1.0  1.0    0.0   0.0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-86db59e9-3c75-49f9-ac23-489f7b6a6509\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>después</th>\n",
              "      <th>despierta</th>\n",
              "      <th>empieza</th>\n",
              "      <th>maullar</th>\n",
              "      <th>y</th>\n",
              "      <th>comida</th>\n",
              "      <th>come</th>\n",
              "      <th>gato</th>\n",
              "      <th>El</th>\n",
              "      <th>duerme</th>\n",
              "      <th>a</th>\n",
              "      <th>ladra</th>\n",
              "      <th>maulla</th>\n",
              "      <th>la</th>\n",
              "      <th>se</th>\n",
              "      <th>ladrar</th>\n",
              "      <th>perro</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>d0</th>\n",
              "      <td>0.5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.5</td>\n",
              "      <td>0.5</td>\n",
              "      <td>0.5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.5</td>\n",
              "      <td>0.5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.5</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>d1</th>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>d2</th>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>d3</th>\n",
              "      <td>0.5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.5</td>\n",
              "      <td>0.5</td>\n",
              "      <td>0.5</td>\n",
              "      <td>0.5</td>\n",
              "      <td>0.5</td>\n",
              "      <td>0.5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.5</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>d4</th>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>d5</th>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-86db59e9-3c75-49f9-ac23-489f7b6a6509')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-86db59e9-3c75-49f9-ac23-489f7b6a6509 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-86db59e9-3c75-49f9-ac23-489f7b6a6509');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TOzdRwx9_UMM"
      },
      "source": [
        "``\n",
        "``\n",
        "\n",
        "***Resultado esperado***: \n",
        "\n",
        "|    |  El | perro |  se | come |  la | comida |   y | después | duerme | despierta | empieza |   a | ladrar | ladra | gato | maullar | maulla |\n",
        "|----|----:|------:|----:|-----:|----:|-------:|----:|--------:|-------:|----------:|--------:|----:|-------:|------:|-----:|--------:|-------:|\n",
        "| d0 | 0.5 |   0.5 | 1.0 |  0.5 | 0.5 |    0.5 | 0.5 |     0.5 |    0.5 |       0.0 |     0.0 | 0.0 |    0.0 |   0.0 |  0.0 |     0.0 |    0.0 |\n",
        "| d1 | 1.0 |   1.0 | 1.0 |  0.0 | 0.0 |    0.0 | 1.0 |     1.0 |    0.0 |       1.0 |     1.0 | 1.0 |    1.0 |   0.0 |  0.0 |     0.0 |    0.0 |\n",
        "| d2 | 1.0 |   1.0 | 1.0 |  1.0 | 1.0 |    1.0 | 1.0 |     1.0 |    0.0 |       0.0 |     0.0 | 0.0 |    0.0 |   1.0 |  0.0 |     0.0 |    0.0 |\n",
        "| d3 | 0.5 |   0.0 | 1.0 |  0.5 | 0.5 |    0.5 | 0.5 |     0.5 |    0.5 |       0.0 |     0.0 | 0.0 |    0.0 |   0.0 |  0.5 |     0.0 |    0.0 |\n",
        "| d4 | 1.0 |   0.0 | 1.0 |  0.0 | 0.0 |    0.0 | 1.0 |     1.0 |    0.0 |       1.0 |     1.0 | 1.0 |    0.0 |   0.0 |  1.0 |     1.0 |    0.0 |\n",
        "| d5 | 1.0 |   0.0 | 1.0 |  1.0 | 1.0 |    1.0 | 1.0 |     1.0 |    0.0 |       0.0 |     0.0 | 0.0 |    0.0 |   0.0 |  1.0 |     0.0 |    1.0 |\n",
        "\n",
        "``\n",
        "``"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZqgW4Ni4t0xC"
      },
      "source": [
        "` `  \n",
        "` ` \n",
        "\n",
        "**Ejercicio 5 - *Calcular IDF* (0.5 punto):**\n",
        "\n",
        "\n",
        "Implementar `calc_idf(dataset_bow)`. Este debe retornar un diccionario en donde las llaves sean las palabras y los valores sean el calculo de cada idf por palabra.\n",
        "\n",
        "Recordar que $idf_{t_i} = log_{10}\\frac{N}{n_i}$ con $N = $ número de documentos y $n_i = $ Número de documentos que contienen la palabra $t_i$ "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "thhDY1-Ht6T5"
      },
      "source": [
        "def calc_idf(dataset_bow):\n",
        "    ### Aquí inicia tu código ###\n",
        "    N = len(dataset_bow.index)\n",
        "    vocab = dataset_bow.columns\n",
        "    doc_names = dataset_bow.index\n",
        "    dic = {}\n",
        "    for j in range(len(dataset_bow.columns)):\n",
        "      suma = 0\n",
        "      for i in range(len(dataset_bow.index)):\n",
        "        if dataset_bow[vocab[j]][doc_names[i]] > 0:\n",
        "          suma+=1\n",
        "      dic[vocab[j]] = np.log10(N/suma)\n",
        "    return dic\n",
        "    pass\n",
        "    ### Aquí termina tu código ###"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yR_j3pYemcAc"
      },
      "source": [
        "***Test:***"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ro-OMGpduC0R",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b193a143-ee56-4696-f6d7-eb0f114c6b40"
      },
      "source": [
        "idf = calc_idf(dataset_bow)\n",
        "idf"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'El': 0.0,\n",
              " 'a': 0.47712125471966244,\n",
              " 'come': 0.17609125905568124,\n",
              " 'comida': 0.17609125905568124,\n",
              " 'despierta': 0.47712125471966244,\n",
              " 'después': 0.0,\n",
              " 'duerme': 0.47712125471966244,\n",
              " 'empieza': 0.47712125471966244,\n",
              " 'gato': 0.3010299956639812,\n",
              " 'la': 0.17609125905568124,\n",
              " 'ladra': 0.7781512503836436,\n",
              " 'ladrar': 0.7781512503836436,\n",
              " 'maulla': 0.7781512503836436,\n",
              " 'maullar': 0.7781512503836436,\n",
              " 'perro': 0.3010299956639812,\n",
              " 'se': 0.0,\n",
              " 'y': 0.0}"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Resultado esperado***: \n",
        "\n",
        "```python\n",
        "{'El': 0.0, \n",
        " 'a': 0.47712125471966244,\n",
        " 'come': 0.17609125905568124,\n",
        " 'comida': 0.17609125905568124,\n",
        " 'despierta': 0.47712125471966244,\n",
        " 'después': 0.0,\n",
        " 'duerme': 0.47712125471966244,\n",
        " 'empieza': 0.47712125471966244,\n",
        " 'gato': 0.3010299956639812,\n",
        " 'la': 0.17609125905568124,\n",
        " 'ladra': 0.7781512503836436,\n",
        " 'ladrar': 0.7781512503836436,\n",
        " 'maulla': 0.7781512503836436,\n",
        " 'maullar': 0.7781512503836436,\n",
        " 'perro': 0.3010299956639812,\n",
        " 'se': 0.0,\n",
        " 'y': 0.0}\n",
        "```"
      ],
      "metadata": {
        "id": "Ioy_HicQDr-a"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IzKAzJtSJ7gx"
      },
      "source": [
        "Puede notar el bajo puntaje otorgado a las palabras que más se repiten! 😮"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D17lm6l9uJPo"
      },
      "source": [
        "**Ejercicio 6 - *Calcular TF-IDF & concluir similitud de documentos.* (1 punto)**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bc7FTQ19Kcwo"
      },
      "source": [
        "Programe la función `calc_tf_idf(tf, idf)` que entrega el dataframe TF-IDF asociado al dataset que estamos analizando."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9knMl0KguMwo"
      },
      "source": [
        "def calc_tf_idf(tf, idf):\n",
        "    ### Aquí inicia tu código ###\n",
        "    doc_names = tf.index\n",
        "    vocab = tf.columns\n",
        "    for i in range(len(tf.index)):\n",
        "      for j in range(len(tf.columns)):\n",
        "        tf[vocab[j]][doc_names[i]] = idf[vocab[j]] * tf[vocab[j]][doc_names[i]]\n",
        "    return tf\n",
        "    pass\n",
        "    ### Aquí termina tu código ### "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tRzIr1nQmepp"
      },
      "source": [
        "***Test.***"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H8z6jaq2uPEo",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 237
        },
        "outputId": "98fdd184-f8e3-4f94-fdd6-cb06795d2c02"
      },
      "source": [
        "tf_idf = calc_tf_idf(tf, idf)\n",
        "tf_idf"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   después despierta   empieza   maullar    y    comida      come      gato  \\\n",
              "d0     0.0       0.0       0.0       0.0  0.0  0.088046  0.088046       0.0   \n",
              "d1     0.0  0.477121  0.477121       0.0  0.0       0.0       0.0       0.0   \n",
              "d2     0.0       0.0       0.0       0.0  0.0  0.176091  0.176091       0.0   \n",
              "d3     0.0       0.0       0.0       0.0  0.0  0.088046  0.088046  0.150515   \n",
              "d4     0.0  0.477121  0.477121  0.778151  0.0       0.0       0.0   0.30103   \n",
              "d5     0.0       0.0       0.0       0.0  0.0  0.176091  0.176091   0.30103   \n",
              "\n",
              "     El    duerme         a     ladra    maulla        la   se    ladrar  \\\n",
              "d0  0.0  0.238561       0.0       0.0       0.0  0.088046  0.0       0.0   \n",
              "d1  0.0       0.0  0.477121       0.0       0.0       0.0  0.0  0.778151   \n",
              "d2  0.0       0.0       0.0  0.778151       0.0  0.176091  0.0       0.0   \n",
              "d3  0.0  0.238561       0.0       0.0       0.0  0.088046  0.0       0.0   \n",
              "d4  0.0       0.0  0.477121       0.0       0.0       0.0  0.0       0.0   \n",
              "d5  0.0       0.0       0.0       0.0  0.778151  0.176091  0.0       0.0   \n",
              "\n",
              "       perro  \n",
              "d0  0.150515  \n",
              "d1   0.30103  \n",
              "d2   0.30103  \n",
              "d3       0.0  \n",
              "d4       0.0  \n",
              "d5       0.0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-0cf7cded-cc3f-4762-a351-a98d8d75335e\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>después</th>\n",
              "      <th>despierta</th>\n",
              "      <th>empieza</th>\n",
              "      <th>maullar</th>\n",
              "      <th>y</th>\n",
              "      <th>comida</th>\n",
              "      <th>come</th>\n",
              "      <th>gato</th>\n",
              "      <th>El</th>\n",
              "      <th>duerme</th>\n",
              "      <th>a</th>\n",
              "      <th>ladra</th>\n",
              "      <th>maulla</th>\n",
              "      <th>la</th>\n",
              "      <th>se</th>\n",
              "      <th>ladrar</th>\n",
              "      <th>perro</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>d0</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.088046</td>\n",
              "      <td>0.088046</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.238561</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.088046</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.150515</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>d1</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.477121</td>\n",
              "      <td>0.477121</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.477121</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.778151</td>\n",
              "      <td>0.30103</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>d2</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.176091</td>\n",
              "      <td>0.176091</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.778151</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.176091</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.30103</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>d3</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.088046</td>\n",
              "      <td>0.088046</td>\n",
              "      <td>0.150515</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.238561</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.088046</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>d4</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.477121</td>\n",
              "      <td>0.477121</td>\n",
              "      <td>0.778151</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.30103</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.477121</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>d5</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.176091</td>\n",
              "      <td>0.176091</td>\n",
              "      <td>0.30103</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.778151</td>\n",
              "      <td>0.176091</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-0cf7cded-cc3f-4762-a351-a98d8d75335e')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-0cf7cded-cc3f-4762-a351-a98d8d75335e button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-0cf7cded-cc3f-4762-a351-a98d8d75335e');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZBG2qfwv_6HK"
      },
      "source": [
        "``\n",
        "``\n",
        "\n",
        "***Resultado esperado***: \n",
        "\n",
        "|    |  El |    perro |  se |     come |       la |   comida |   y | después |   duerme | despierta |  empieza |        a |   ladrar |    ladra |     gato |  maullar |   maulla |\n",
        "|----|----:|---------:|----:|---------:|---------:|---------:|----:|--------:|---------:|----------:|---------:|---------:|---------:|---------:|---------:|---------:|---------:|\n",
        "| d0 | 0.0 | 0.150515 | 0.0 | 0.088046 | 0.088046 | 0.088046 | 0.0 |     0.0 | 0.238561 |  0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 |\n",
        "| d1 | 0.0 | 0.301030 | 0.0 | 0.000000 | 0.000000 | 0.000000 | 0.0 |     0.0 | 0.000000 |  0.477121 | 0.477121 | 0.477121 | 0.778151 | 0.000000 | 0.000000 | 0.000000 | 0.000000 |\n",
        "| d2 | 0.0 | 0.301030 | 0.0 | 0.176091 | 0.176091 | 0.176091 | 0.0 |     0.0 | 0.000000 |  0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.778151 | 0.000000 | 0.000000 | 0.000000 |\n",
        "| d3 | 0.0 | 0.000000 | 0.0 | 0.088046 | 0.088046 | 0.088046 | 0.0 |     0.0 | 0.238561 |  0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.150515 | 0.000000 | 0.000000 |\n",
        "| d4 | 0.0 | 0.000000 | 0.0 | 0.000000 | 0.000000 | 0.000000 | 0.0 |     0.0 | 0.000000 |  0.477121 | 0.477121 | 0.477121 | 0.000000 | 0.000000 | 0.301030 | 0.778151 | 0.000000 |\n",
        "| d5 | 0.0 | 0.000000 | 0.0 | 0.176091 | 0.176091 | 0.176091 | 0.0 |     0.0 | 0.000000 |  0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.301030 | 0.000000 | 0.778151 |\n",
        "\n",
        "\n",
        "``\n",
        "``"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WmVlbpzMp5NU"
      },
      "source": [
        "Ahora que tenemos el dataframe de TF-IDF, nos queda calcular la similitud coseno entre todos los vectores. Notar que la matriz resultante será una matriz simétrica. Implemente la función *cosine_similarity(v1, v2)* que recibe dos vectores (v1 y v2) y calcula la similitud coseno entre ambos vectores. Concluya cuáles son los dos documentos más similares."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HEgUtgBkQAae",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8eb19435-06a2-41b7-ce1e-ea1e5ea07758"
      },
      "source": [
        "def cosine_similarity(v1, v2):\n",
        "  ### Aquí inicia tu código ###\n",
        "  norma = 0\n",
        "  for i in range(len(v1)):\n",
        "    norma += (v1[i]*v1[i])\n",
        "  norma = np.sqrt(norma)\n",
        "  norma2 = 0\n",
        "  for i in range(len(v2)):\n",
        "    norma2 += (v2[i]*v2[i])\n",
        "  norma2 = np.sqrt(norma2)\n",
        "  cos = np.dot(v1, v2)/(norma*norma2)\n",
        "  return cos\n",
        "  pass\n",
        "  ### Aquí termina tu código ### \n",
        "\n",
        "similarity_matrix = np.zeros((6,6))\n",
        "for i, v1 in enumerate(tf_idf.index.values):\n",
        "  for j, v2 in enumerate(tf_idf.index.values):\n",
        "      similarity = cosine_similarity(tf_idf.loc[v1].values, tf_idf.loc[v2].values)\n",
        "      similarity_matrix[i][j] = similarity\n",
        "print(similarity_matrix)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[1.         0.12032418 0.3223436  0.77967014 0.         0.1632828 ]\n",
            " [0.12032418 1.         0.08686457 0.         0.4952126  0.        ]\n",
            " [0.3223436  0.08686457 1.         0.1632828  0.         0.11787732]\n",
            " [0.77967014 0.         0.1632828  1.         0.12032418 0.3223436 ]\n",
            " [0.         0.4952126  0.         0.12032418 1.         0.08686457]\n",
            " [0.1632828  0.         0.11787732 0.3223436  0.08686457 1.        ]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u9FlltqIqRxf"
      },
      "source": [
        "   Los documentos mas similares son aquellos con angulo mas cercano a 0, por lo tanto, con coseno mas cercano a 1. Notese que en la diagonal todos los valores son 1 porque se compara cada documento consigo mismo. Finalmente, los documentos mas similares son d0 con d3, pues su similitud coseno es de 0.8"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FUAc1zX0Lg16"
      },
      "source": [
        "![gato](https://www.elagoradiario.com/wp-content/uploads/2020/05/Gato-mascarilla.jpg)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s1A95IaXLHaB"
      },
      "source": [
        "**Cualquier recomendación que nos quisieran dar para una futura tarea es bienvenid@!**"
      ]
    }
  ]
}