Competencia_2_NLP.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/bacchus00/tareas-ds/blob/main/Competencia_2_NLP.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z0tyIsliieNr"
      },
      "source": [
        "# **Competencia 2 - CC6205 Natural Language Processing 📚**\n",
        "\n",
        "Integrantes: Ricardo García, Nicolás Canales, Melanie Márquez, Franc Zautzik\n",
        "\n",
        "Usuario del equipo en CodaLab (Obligatorio): FraNiMeRi\n",
        "\n",
        "Fecha límite de entrega 📆: 24 de Junio.\n",
        "\n",
        "Tiempo estimado de dedicación:\n",
        "\n",
        "Link competencia: Poner el link aquí"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MocJN22HSJ1x"
      },
      "source": [
        "### **Objetivo**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OwdgXS8FSLvc"
      },
      "source": [
        "El objetivo de esta competencia es resolver una de las tareas más importantes en el área del procesamiento de lenguage natural, relacionada con la extracción de información: [Named Entity Recognition (NER)](http://www.cs.columbia.edu/~mcollins/cs4705-spring2019/slides/tagging.pdf). \n",
        "\n",
        "En particular, y al igual que en la competencia anterior, deberán crear distintos modelos que apunten a resolver la tarea de NER en Español. Para esto, les entregaremos un dataset real perteneciente a la lista de espera NO GES en Chile. Es importante destacar que existe una falta de trabajos realizados en el área de NER en Español y aún más en el contexto clínico, por ende puede ser considerado como una tarea bien desafiante y quizás les interesa trabajar en el área más adelante en sus carreras.\n",
        "\n",
        "En este notebook les entregaremos un baseline como referencia de los resultados que esperamos puedan obtener. **Recuerden que el no superar a los baselines en alguna de las tres métricas conlleva un descuento de 0.5 puntos hasta 1.5 puntos**.\n",
        "\n",
        "Como hemos estado viendo redes neuronales tanto en catedras, tareas y auxiliares (o próximamente lo harán), esperamos que (por lo menos) utilicen Redes Neuronales Recurrentes (RNN) para resolverla. \n",
        "\n",
        "Nuevamente, hay total libertad para utilizar el software y los modelos que deseen, siempre y cuando estos no traigan los modelos ya implementados. (De todas maneras como es un corpus nuevo, es difícil que haya algún modelo ya implementado con estas entidades)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wnjgmvjBSReb"
      },
      "source": [
        "### **Explicación de la competencia**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NH4HqnCjSWs-"
      },
      "source": [
        "La tarea **NER** que van a resolver en esta competencia es comúnmente abordada como un problema de Sequence Labeling.\n",
        "\n",
        "**¿Qué es Sequence Labeling?** \n",
        "\n",
        "En breves palabras, dada una secuencia de tokens (oración) sequence labeling tiene por objetivo asignar una etiqueta a cada token de dicha secuencia. En pocas palabras, dada una lista de tokens esperamos encontrar la mejor secuencia de etiquetas asociadas a esa lista. Ahora veamos de qué se trata este problema.\n",
        "\n",
        "**Named Entity Recognition (NER)**\n",
        "\n",
        "NER es un ejemplo de un problema de Sequence Labeling. Pero antes de definir formalmente esta tarea, es necesario definir algunos conceptos claves para poder entenderla de la mejor manera:\n",
        "\n",
        "- *Token*: Un token es una secuencia de caracteres, puede ser una palabra, un número o un símbolo.\n",
        "\n",
        "- *Entidad*: No es más que un trozo de texto (uno o más tokens) asociado a una categoría predefinida. Originalmente se solían utilizar categorías como nombres de personas, organizaciones, ubicaciones, pero actualmente se ha extendido a diferentes dominios.\n",
        "\n",
        "- *Límites de una entidad*: Son los índices de los tokens de inicio y fín dentro de una entidad.\n",
        "\n",
        "- *Tipo de entidad*: Es la categoría predefinida asociada a la entidad.\n",
        "\n",
        "Dicho esto, definimos formalmente una entidad como una tupla: $(s, e, t)$, donde $s, e$ son los límites de la entidad (índices de los tokens de inicio y fin, respectivamente) y t corresponde al tipo de entidad o categoría. Ya veremos más ejemplos luego de describir el Dataset.\n",
        "\n",
        "**Corpus de la Lista de espera**\n",
        "\n",
        "Trabajaran con un conjunto de datos reales correspondiente a interconsultas de la lista de espera NO GES en Chile. Si quieren saber más sobre cómo fueron generados los datos pueden revisar el paper publicado hace unos meses atrás en el workshop de EMNLP, una de las conferencias más importantes de NLP: [https://www.aclweb.org/anthology/2020.clinicalnlp-1.32/](https://www.aclweb.org/anthology/2020.clinicalnlp-1.32/).\n",
        "\n",
        "Este corpus Chileno está constituido originalmente por 7 tipos de entidades pero por simplicidad en esta competencia trabajarán con las siguientes:\n",
        "\n",
        "- **Disease**\n",
        "- **Body_Part**\n",
        "- **Medication** \n",
        "- **Procedures** \n",
        "- **Family_Member**\n",
        "\n",
        "Si quieren obtener más información sobre estas entidades pueden consultar la [guía de anotación](https://plncmm.github.io/annodoc/). Además, mencionar que este corpus está restringido bajo una licencia que permite solamente su uso académico, así que no puede ser compartido más allá de este curso o sin permisos por parte de los autores en caso que quieran utilizarlo fuera. Si este último es el caso entonces pueden escribir directamente al correo: pln@cmm.uchile.cl. Al aceptar los términos y condiciones de la competencia están de acuerdo con los puntos descritos anteriormente.\n",
        "\n",
        "\n",
        "**Formato ConLL**\n",
        "\n",
        "Los archivos que serán entregados a ustedes vienen en un formato estándar utilizado en NER, llamado ConLL. No es más que un archivo de texto, que cumple las siguientes propiedades.\n",
        "\n",
        "- Un salto de linea corresponde a la separación entre oraciones. Esto es importante ya que al entrenar una red neuronal ustedes pasaran una lista de oraciones como input, más conocidos como batches.\n",
        "\n",
        "- La primera columna del archivo contiene todos los tokens de la partición.\n",
        "\n",
        "- La segunda columna del archivo contiene el tipo de entidad asociado al token de la primera columna.\n",
        "\n",
        "- Los tipos de entidades siguen un formato clásico en NER denominado *IOB2*. Si un tipo de entidad comienza con el prefijo \"B-\" (Beginning) significa que es el token de inicio de una entidad, si comienza con \"I-\" (Inside) es un token distinto al de inicio y si un token está asociado a la categoría O (Outside) significa que no pertenece a ninguna entidad.\n",
        "\n",
        "Aquí va un ejemplo:\n",
        "\n",
        "```\n",
        "PACIENTE O\n",
        "PRESENTA O\n",
        "FRACTURA B-Disease\n",
        "CORONARIA I-Disease\n",
        "COMPLICADA I-Disease\n",
        "EN O\n",
        "PIE B-Body_Part\n",
        "IZQUIERDO I-Body_Part\n",
        ". O\n",
        "SE O\n",
        "REALIZA O\n",
        "INSTRUMENTACION B-Procedure\n",
        "INTRACONDUCTO I-Procedure\n",
        ". O\n",
        "```\n",
        "\n",
        "Según nuestra definición tenemos las siguientes tres entidades (enumerando desde 0): \n",
        "\n",
        "- $(2, 4, Disease)$\n",
        "- $(6, 7, Body Part)$\n",
        "- $(11, 12, Procedure)$\n",
        "\n",
        "Repasen un par de veces todos estos conceptos antes de pasar a la siguiente sección del notebook.\n",
        "Es importante entender bien este formato ya que al medir el rendimiento de sus modelos, consideraremos una **métrica estricta**. Esta métrica se llama así ya que considera correcta una predicción de su modelo, sólo si al compararlo con las entidades reales **coinciden tanto los límites de la entidad como el tipo.** \n",
        "\n",
        "Para ejemplificar, tomando el caso anterior, si el modelo es capaz de encontrar la siguiente entidad: $(2, 3, Disease)$, entonces se considera incorrecto ya que pudo predecir dos de los tres tokens de dicha enfermedad. Es decir, buscamos una métrica que sea alta a nivel de entidad y no a nivel de token.\n",
        "\n",
        "Antes de pasar a explicar las reglas, se recomienda visitar los siguientes links para entender bien el baseline de la competencia:\n",
        "\n",
        "-  [Tagging, and Hidden Markov Models ](http://www.cs.columbia.edu/~mcollins/cs4705-spring2019/slides/tagging.pdf) (slides by Michael Collins), [notes](http://www.cs.columbia.edu/~mcollins/hmms-spring2013.pdf), [video 1](https://youtu.be/-ngfOZz8yK0), [video 2](https://youtu.be/PLoLKQwkONw), [video 3](https://youtu.be/aaa5Qoi8Vco), [video 4](https://youtu.be/4pKWIDkF_6Y)       \n",
        "-  [Recurrent Neural Networks](slides/NLP-RNN.pdf) | [video 1](https://youtu.be/BmhjUkzz3nk), [video 2](https://youtu.be/z43YFR1iIvk), [video 3](https://youtu.be/7L5JxQdwNJk)\n",
        "\n",
        "\n",
        "Recuerden que todo el material se encuentra disponible en el [github del curso](https://github.com/dccuchile/CC6205)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iWlfabmkaSE7"
      },
      "source": [
        "### **Reglas de la competencia**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3w9Dw4CSaSE8"
      },
      "source": [
        "**texto en negrita**- Para que su competencia sea evaluada, deben participar en la competencia y enviar este notebook con su informe.\n",
        "- Para participar, deben registrarse en la competencia en Codalab en grupos de máximo 4 alumnos. Cada grupo debe tener un nombre de equipo. (¡Y deben reportarlo en su informe, por favor!)\n",
        "- Las métricas usadas serán métricas estrictas (ya explicado anteriormente) utilizando métricas clásicas como lo son precisión, recall y micro f1-score.\n",
        "- En esta tarea se recomienda usar GPU. Pueden ejecutar su tarea en colab (lo cual trae todo instalado) o pueden intentar ejecutándolo en su computador. En este caso, deberá ser compatible con cuda y deberán instalar todo por su cuenta.\n",
        "- En total pueden hacer un **máximo de 5 envíos**.\n",
        "- Por favor, todas sus dudas haganlas por el canal de Discord. Los emails que lleguen al equipo docente serán remitidos a ese medio. Recuerden el ánimo colaborativo del curso.\n",
        "- Estar top 5 en alguna de las tres métricas equivale a una bonificación en su nota final.\n",
        "\n",
        "Éxito!\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aZyHBjU-R-wi"
      },
      "source": [
        "### **Baseline**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4WZ8G01aSBYX"
      },
      "source": [
        "En este punto esperamos que tengan conocimiento sobre redes neuronales y en particular redes neuronales recurrentes (RNN), si no siempre pueden escribirnos por el canal de Discord para aclarar dudas. La RNN del baseline adjunto a este notebook está programado en la librería [`pytorch`](https://pytorch.org/) pero ustedes pueden utilizar keras, tensorflow si así lo desean. El código contiene lo siguiente:\n",
        "\n",
        "- La carga de los datasets, creación de batches de texto y padding (esto es importante ya que si utilizan redes neuronales tienen que tener el mismo largo los inputs). \n",
        "\n",
        "- La implementación básica de una red `LSTM` simple de solo un nivel y sin bi-direccionalidad. \n",
        "\n",
        "- La construcción del formato del output requerido para que lo puedan probar en la tarea en codalab.\n",
        "\n",
        "Se espera que como mínimo ustedes puedan experimentar con el baseline utilizando (pero no limitándose) estas sugerencias:\n",
        "\n",
        "*   Probar la técnica de early stopping.\n",
        "*   Variar la cantidad de parámetros de la capa de embeddings.\n",
        "*   Variar la cantidad de capas RNN.\n",
        "*   Variar la cantidad de parámetros de las capas de RNN.\n",
        "*   Inicializar la capa de embeddings con modelos pre-entrenados. (word2vec, glove, conceptnet, etc...). [Embeddings en español aquí](https://github.com/dccuchile/spanish-word-embeddings). También aquí pueden encontrar unos embeddings clínicos en Español: [https://zenodo.org/record/3924799](https://zenodo.org/record/3924799)\n",
        "*   Variar la cantidad de épocas de entrenamiento.\n",
        "*   Variar el optimizador, learning rate, batch size, usar CRF loss, etc.\n",
        "*   Probar una capa de CRF para garantizar el     formato IOB2.\n",
        "*   Probar bi-direccionalidad.\n",
        "*   Incluir dropout.\n",
        "*   Probar modelos de tipo GRU.\n",
        "*   Probar usando capas de atención.\n",
        "*   Probar Embedding Contextuales (les puede ser de utilidad [flair](https://github.com/flairNLP/flair))\n",
        "*   Probar modelos de transformers en español usando [Huggingface](https://github.com/huggingface/transformers) o el framework Flair."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2Rr2mzxPTzNd"
      },
      "source": [
        "### **Reporte**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FEf33mnxT0rf"
      },
      "source": [
        "Este debe cumplir la siguiente estructura:\n",
        "\n",
        "1.\t**Introducción**: Presentar brevemente el contexto, problema a resolver, incluyendo la formalización de la task (cómo son los inputs y outputs del problema) y los desafíos que ven al analizar el corpus entregado. (**0.5 puntos**)\n",
        "\n",
        "2.\t**Modelos**: Describir brevemente los modelos, métodos e hiperparámetros utilizados. (**1.0 puntos**)\n",
        "\n",
        "4.\t**Métricas de evaluación**: Describir las métricas utilizadas en la evaluación indicando qué miden y cuál es su interpretación en este problema en particular. (**0.5 puntos**)\n",
        "\n",
        "5.  **Diseño experimental**: Esta es una de las secciones más importantes del reporte. Deben describir minuciosamente los experimentos que realizarán en la siguiente sección. Describir las variables de control que manejarán, algunos ejemplos pueden ser: Los hiperparámetros de los modelos, tipo de embeddings utilizados, tipos de arquitecturas. Ser claros con el conjunto de hiperparámetros que probarán, la decisión en las funciones de optimización, función de pérdida,  regulación, etc. Básicamente explicar qué es lo que veremos en la siguiente sección.\n",
        "(**1 punto**)\n",
        "\n",
        "6.\t**Experimentos**: Reportar todos sus experimentos y código en esta sección. Comparar los resultados obtenidos utilizando diferentes modelos. ¡Es vital haber realizado varios experimentos para sacar una buena nota! (**2.0 puntos**)\n",
        "\n",
        "7.\t**Conclusiones**: Discutir resultados, proponer trabajo futuro. (**1 punto**)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XaoU1EXfUDbl"
      },
      "source": [
        "# **Entregable.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PzQlYlmGaSFH"
      },
      "source": [
        "## **Introducción**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PVL0-01AUOzL"
      },
      "source": [
        "El siguiente informe se enmarca dentro de una rama de problemas de NLP conocidos como sequence labeling. En palabras simples lo que se busca es, dada una secuencia de ellos, obtener una etiqueta para cada uno de sus tokens, obteniendo asi una sequencia de etiquetas. En este caso particular se ataja la task conocida como Name Entity Regonition (NER) que tiene como objetivo reconocer entidades dentro de un corpus.\n",
        "\n",
        "Se busca reconocer entidades dentro de las descripciones de los casos en la lista de espera NO GES de el sistema de salud en Chile. En particular se busca reconocer las siguientes entidades:\n",
        "\n",
        "*   Disease : enfermedades\n",
        "*   Body_Part : partes del cuerpo\n",
        "*   Medication : medicamentos\n",
        "*   Procedures : procedimientos medicos\n",
        "*   Family_Member : miembros de la familia\n",
        "\n",
        "Por ejemplo si tenemos como entrada la frase:\n",
        "\n",
        "\"Paciente presenta fractura coronaria complicada en pie izquierdo.\"\n",
        "\n",
        "Queremos obtener de salida la secuencia:\n",
        "\n",
        "(O, O, B-Disease, I-Disease, I-Disease, O, B-Body_Part, I-Body_Part, O)\n",
        "\n",
        "Donde la etiqueta \"O\" (Outside) representa el estar fuera de cualquier entidad, \"B-\\<entidad\\>\" marca el inicio (Beginning) de la entidad \\<entidad\\> y \"I-\\<entidad\\>\" indica que se continua dentro (Inside) de la entidad \\<entidad\\>.\n",
        "\n",
        "Para resolver esta problematica se utiliza redes neuronales recurrentes (RNN) con una capa de embedding. Se utiliza esta arquitecturas con distintas implementaciones (LSTM y GRU) y variando varios hiperparametros de manera de obtener los mejores resultados posibles para la task propuesta.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UbA1EmhCaSFI"
      },
      "source": [
        "## **Modelos**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qhM6-Gu5lguO"
      },
      "source": [
        "Inicialmente en el baseline se utiliza una capa de embedding en la que se lleva el input a una respresentacion vectorial. Luego le sigue unaes la implementacion LSTM de RNN. Luego se lo contrasta con la implementacion GRU de la RNN paagregarle bidireccionalidad. A continuacion se probo con distintos valores de dropout pero se lo mantuvo finalmente en $0.5$.\n",
        "\n",
        "LSTM: long short-term memory es una red neuronal recurrente que permite mantener las relaciones contextuales en distancias prolongadas en textos. Hace uso de parametros matriciales que representan pesos, los cuales van mejorando con métodos de gradientes en cada iteración.\n",
        "\n",
        "GRU: variación de LSTM que funciona con mecanismo de compuertas, es decir, en cada iteración va rescatando la información que considera más valiosa y desechando la que no. Usa menos parametros.\n",
        "\n",
        "Ambas pueden presentar variaciones en su implementación. Se pueden apilar (ocupar sus salidas como entradas de otra red), bidireccionar que es leer las oraciones en ambos sentidos para representar dependencias tanto de prefijos como sufijos.\n",
        "\n",
        "Word embedding: representación vectorial densa de palabras, con dimensionalidad variable y valores de tipo entero.\n",
        "\n",
        "Dentro de los hiperparametros utilizados para la competencia están:\n",
        "\n",
        "* Epochs son la cantidad de iteraciones de entrenamiento del modelo\n",
        "* Dimensiones necesarias para la construcción, están las de entrada: tamaño del vocabulario; embedding: el porte de la representación numérica de cada palabra; hidden: la cantidad de neuronas a utilizar; output: la cantidad de etiquetas posibles\n",
        "* Dropout es la probabilidad de que componentes de la red se desconecten en medio de la iteración con el fin de evitar la dependencia excesiva entre neuronas que podría provocar overfitting."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AaVhZ5iaaSFK"
      },
      "source": [
        "## **Métricas de evaluación**\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wXl3GaVMUYA7"
      },
      "source": [
        "- **Métrica estricta:** Cuando se habla de una metrica estricta, se refiere a que la comparacion entre la prediccion y la entidad real es **exactamente igual**, considerando tanto los limites de la entidad, es decir, entre que indices corresponde la etiqueta, y el tipo, es decir, la etiqueta en si misma. Por ejemplo, para la entidad (3,1,BodyPart), la prediccion (3,1,Disease) no es correcta ya que falla en el tipo, la prediccion (3,2,BodyPart) no es correcta ya que falla en los limite (a pesar de que (3,2) sea un subconjunto de (3,1)).\n",
        "\n",
        "- **Precision:** El indice Precision es una medida de **calidad** del modelo, en un entorno binario en donde solo tenemos dos clases y, por lo tanto, nuestras opciones son que una de estas sea positiva y la otra negativa, representa la proporcion entre los datos clasificados como positivos que realmente lo son y la formula general es TP/(TP+FP) (con TP = True Positive, FP = False Positive). Para un problema multiclase como es este, podemos calcular esta metrica para cada una de las clases, por lo que representa la fraccion de datos asignados a la clase i que realmente pertenecen a esta clase y su formula es:\n",
        "\n",
        "\\begin{equation}\n",
        "\\frac{c_{ii}}{\\sum_{j}{c_{ji}}}\n",
        "\\end{equation}\n",
        "\n",
        "- **Recall:** En español Exhaustividad, es una medida de **cantidad** que el modelo puede predecir, al igual que con presicion, cuando tenemos solo dos clases representa la proporcion de datos reales positivos que estan correctamente clasificados. Su formula general es TP/(TP+FN) (con FN = False Negative). En un entorno multiclase, al igual que la anterior, se puede calcular Recall para cada una de las clases con la formula: \n",
        "\n",
        "\\begin{equation}\n",
        "\\frac{c_{ii}}{\\sum_{j}{c_{ij}}}\n",
        "\\end{equation}\n",
        "\n",
        "Es importante destacar que para las metricas anteriores, se calcula para cada una de las etiquetas su Precision y Recall, para visualizarlo de mejor forma, es posible calcular un promedio de estas y llamarlo Average Precision y Average Recall.\n",
        "\n",
        "- **Micro F1 score:** La metrica F nace desde la necesidad de mezclar las dos metricas anteriormente descritas, de modo que comparar dos modelos se hace mas facil. Su formula general es (((b^2)+1)PR)/((b^2)P+R), para el caso de F1 Score, b=1, por lo tanto su formula seria: 2PR/(P+R) con P = Presicion y R = Recall. Como se vio anteriormente, para un problema multiclase calculamos P y R para cada una de las clases y luego su promedio, es importante destacar que F1 Score se puede calcular en base al promedio, es decir Average P y Average R, a lo que llamariamos Macro F1 Score, sin embargo, cuando se calcula en base a P y R de cada una de las clases y luego calculamos el promedio de los F1 calculados, entonces se llamaria Micro F1 Score."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u27WffRVUj4v"
      },
      "source": [
        "## **Diseño experimental**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3O228DbeUmE7"
      },
      "source": [
        "Inicialmente se probó utilizando una capa GRU envez de una capa LSTM utilizando los mismo hiperparametros del baseline (Modelo 1) y comparando el desempeño de ambos, particularmente enfocandose en el F1-score maximo obtenido.\n",
        "\n",
        "Luego se continuo utilizand la GRU con los hiperparametros del baseline pero esta vez utilizando bidireccionalidad (Modelo 2). El resultado obtenido luego fue comparado con el obtenido por el modelo anterior.\n",
        "\n",
        "A continuación se comparó los resultado obtenidos utilizando varios valores para el dropout para asi determinar cual valor podria mejorar el desempeño de la red.\n",
        "\n",
        "Se debe analizar la cantidad de epocas epoch a utilizar, de esta forma evitamos que los datos sean overfitteados. Para esto, se utilizara el experimento en la seccion Modelo 5, lo que se hara es entrenar al modelo con 20 epocas y ver el momento en que la curva Val. Loss deja de ser decreciente y pasa a tener ruido (es decir, el momento en que Val. Loss en la posicion i+1 es mayor a Val. Loss en la posicion i). El proceso anterior se hara tanto para los modelos LSTM y GRU.\n",
        "\n",
        "En la siguiente sección de prueba de hiperparámetros, se realizaron los siguientes experimentos, tanto utilizando celdas LSTM y GRU:\n",
        "-\tOptimización del número de neuronas por capa\n",
        "-\tOptimización del número de capas\n",
        "-\tOptimización del número de concatenaciones de celdas\n",
        "En todos estos experimentos se utilizó un learning rate constante de 1e-3, un batch size de 22, Adam como optimizador y CrossEntropy como función de pérdida. \n",
        "\n",
        "Para obtener el número de neuronas por capa, se crearon modelos con $2^i$ neuronas por capa, en donde $i \\in [0, 10]$. De la misma forma, para obtener el número óptimo de capas ocultas, se varió este hiperparámetro de 1 hasta 10. Por último, para determinar el número óptimo de concatenaciones de celdas, se crearon modelos con concatenaciones que iban de 1 hasta 10. \n",
        "\n",
        "Por otra parte, se implementó una arquitectura Encoder-Decorder con Attention inspirada en el paper :”Attention is all you need”. La cual se entrenó por 10 épocas, utilizando Adam con Learning rate de $10^{-3}$, batch size de 22 y CrossEntroy como función de pérdidas. \n",
        "\n",
        "Entre las exploraciones realizadas se modificaron tanto parametros de la capa de embedding como en el optimizador. Se utilizó el escalamiento de gradientes con la frecuencia inversa de palabras en el primer caso y se probaron los optimizadores Adam y SGD variando las learning rates de cada uno. Ninguno de estos cambios proporcionó diferencias significativas en el rendimiento del modelo baseline, por lo que resultaron practicamente irrelevantes.\n",
        "\n",
        "Un foco importante de atención fue el uso de vectores preentrenados en contextos clínicos de palabras. El modelo baseline rellena una matriz de pesos de manera aleatoria. Intuitivamente parece más efectivo empezar el entrenamiento con representaciones de las palabras observadas en el contexto del caso estudiado, esto es diagnósticos. Se barajó la opción de ocupar embeddings pre-entrenados del español en otros contextos, pero dado que el significado de las palabras puede sufrir graves cambios en contextos técnicos fue que se prefirió usar los vectores construidos a partir de listas de espera.\n",
        "\n",
        "El proceso constó de aplicar minusculas a los datos con el fin de evitar matches fallidos a causa de inconsistencias en el uso de mayusculas. Luego de explorar un poco los embeddings preentrenados y de notar que no hacían uso de tildes, se aplicó la remoción de estos del vocabulario por las mismas razones mencionadas en el paso anterior. Lo que siguió fue realizar todas las correspondencias posibles entre ambos conjuntos, quedando en vectores nulos las coincidencias fallidas. \n",
        "Se usaron métodos de las librerías utilizadas para la construcción de las estructuras necesarias como tensores o el uso de dimensiones necesarias para el correcto funcionamiento. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uFM-wNt8aSFM"
      },
      "source": [
        "## **Experimentos**\n",
        "\n",
        "\n",
        "El código que les entregaremos servirá de baseline para luego implementar mejores modelos. \n",
        "En general, el código asociado a la carga de los datos, las funciones de entrenamiento, de evaluación y la predicción de los datos de la competencia no deberían cambiar. \n",
        "Solo deben preocuparse de cambiar la arquitectura del modelo, sus hiperparámetros y reportar, lo cual lo pueden hacer en las subsecciones *modelos*.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LMgKjfYC_Go-"
      },
      "source": [
        "###  **Carga de datos y Preprocesamiento**\n",
        "\n",
        "Para cargar los datos y preprocesarlos usaremos la librería [`torchtext`](https://github.com/pytorch/text). Tener cuidado ya que hace algunos meses esta librería tuvo cambios radicales, quedando las funcionalidades pasadas en un nuevo paquete llamado legacy. Esto ya que si quieren usar más funciones de la librería entonces vean los cambios en la documentación.\n",
        "\n",
        "En particular usaremos su módulo `data`, el cual según su documentación original provee: \n",
        "\n",
        "    - Ability to describe declaratively how to load a custom NLP dataset that's in a \"normal\" format\n",
        "    - Ability to define a preprocessing pipeline\n",
        "    - Batching, padding, and numericalizing (including building a vocabulary object)\n",
        "    - Wrapper for dataset splits (train, validation, test)\n",
        "\n",
        "\n",
        "El proceso será el siguiente: \n",
        "\n",
        "1. Descargar los datos desde github y examinarlos.\n",
        "2. Definir los campos (`fields`) que cargaremos desde los archivos.\n",
        "3. Cargar los datasets.\n",
        "4. Crear el vocabulario."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "27csY87GaSFO",
        "outputId": "24a28209-7cac-4c17-c4f8-1c9b0cd48788",
        "scrolled": false
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: torchtext==0.10.0 in /usr/local/lib/python3.7/dist-packages (0.10.0)\n",
            "Requirement already satisfied: torch==1.9.0 in /usr/local/lib/python3.7/dist-packages (from torchtext==0.10.0) (1.9.0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from torchtext==0.10.0) (4.64.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from torchtext==0.10.0) (2.23.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from torchtext==0.10.0) (1.21.6)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch==1.9.0->torchtext==0.10.0) (4.1.1)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->torchtext==0.10.0) (2.10)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->torchtext==0.10.0) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->torchtext==0.10.0) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->torchtext==0.10.0) (2022.6.15)\n"
          ]
        }
      ],
      "source": [
        "# Instalamos torchtext que nos facilitará la vida en el pre-procesamiento del formato ConLL.\n",
        "!pip install -U torchtext==0.10.0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ng7wRGEyawjM"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "from torchtext import data, datasets, legacy\n",
        "\n",
        "\n",
        "# Garantizar reproducibilidad de los experimentos\n",
        "SEED = 1234\n",
        "torch.manual_seed(SEED)\n",
        "torch.backends.cudnn.deterministic = True"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BehSou6rCvwg"
      },
      "source": [
        "#### **Obtener datos**\n",
        "\n",
        "Descargamos los datos de entrenamiento, validación y prueba en nuestro directorio de trabajo"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lbT0g_kC18Jb",
        "outputId": "5a1b106a-cc54-48c9-a378-e93fcaf5e6c5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "File ‘train.txt’ already there; not retrieving.\n",
            "\n",
            "File ‘dev.txt’ already there; not retrieving.\n",
            "\n",
            "File ‘test.txt’ already there; not retrieving.\n",
            "\n"
          ]
        }
      ],
      "source": [
        "#%%capture\n",
        "\n",
        "!wget https://github.com/dccuchile/CC6205/releases/download/v1.0/train.txt -nc # Dataset de Entrenamiento\n",
        "!wget https://github.com/dccuchile/CC6205/releases/download/v1.0/dev.txt -nc    # Dataset de Validación (Para probar y ajustar el modelo)\n",
        "!wget https://github.com/dccuchile/CC6205/releases/download/v1.0/test.txt -nc  # Dataset de la Competencia. Estos datos solo contienen los tokens. ¡¡SON LOS QUE DEBEN SER PREDICHOS!!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uMud7YGMBZvg"
      },
      "source": [
        "####  **Fields**\n",
        "\n",
        "Un `field`:\n",
        "\n",
        "* Define un tipo de datos junto con instrucciones para convertir el texto a Tensor.\n",
        "* Contiene un objeto `Vocab` que contiene el vocabulario (palabras posibles que puede tomar ese campo).\n",
        "* Contiene otros parámetros relacionados con la forma en que se debe numericalizar un tipo de datos, como un método de tokenización y el tipo de Tensor que se debe producir.\n",
        "\n",
        "\n",
        "Analizemos el siguiente cuadro el cual contiene un ejemplo cualquiera de entrenamiento:\n",
        "\n",
        "\n",
        "```\n",
        "El O\n",
        "paciente O\n",
        "padece O\n",
        "de O\n",
        "cancer B-Disease\n",
        "de I-Disease\n",
        "colon I-Disease\n",
        ". O\n",
        "```\n",
        "\n",
        "Cada linea contiene un token y el tipo de entidad asociado en el formato IOB2 ya explicado. Para que `torchtext` pueda cargar estos datos, debemos definir como va a leer y separar los componentes de cada una de las lineas.\n",
        "Para esto, definiremos un field para cada uno de esos componentes: Las palabras (`TEXT`) y las etiquetas o categorías (`NER_TAGS`).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3DcM_IjgCdzz"
      },
      "outputs": [],
      "source": [
        "# Primer Field: TEXT. Representan los tokens de la secuencia\n",
        "TEXT = legacy.data.Field(lower=False) \n",
        "\n",
        "# Segundo Field: NER_TAGS. Representan los Tags asociados a cada palabra.\n",
        "NER_TAGS = legacy.data.Field(unk_token=None)\n",
        "fields = ((\"text\", TEXT), (\"nertags\", NER_TAGS))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xCKTJOdgC5eC"
      },
      "source": [
        "####  **SequenceTaggingDataset**\n",
        "\n",
        "`SequenceTaggingDataset` es una clase de torchtext diseñada para contener datasets de sequence labeling. Los ejemplos que se guarden en una instancia de estos serán arreglos de palabras asociados con sus respectivos tags.\n",
        "\n",
        "Por ejemplo, para Part-of-speech tagging:\n",
        "\n",
        "[I, love, PyTorch, .] estará asociado con [PRON, VERB, PROPN, PUNCT]\n",
        "\n",
        "\n",
        "La idea es que usando los fields que definimos antes, le indiquemos a la clase cómo cargar los datasets de prueba, validación y test."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HsHdGml62J21"
      },
      "outputs": [],
      "source": [
        "train_data, valid_data, test_data = legacy.datasets.SequenceTaggingDataset.splits(\n",
        "    path=\"./\",\n",
        "    train=\"train.txt\",\n",
        "    validation=\"dev.txt\",\n",
        "    test=\"test.txt\",\n",
        "    fields=fields,\n",
        "    encoding=\"utf-8\",\n",
        "    separator=\" \"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Hu7q3HCliia5",
        "outputId": "31a587eb-7ed0-4fcd-80a5-aa882e63e03a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Numero de ejemplos de entrenamiento: 8025\n",
            "Número de ejemplos de validación: 891\n",
            "Número de ejemplos de test (competencia): 992\n"
          ]
        }
      ],
      "source": [
        "print(f\"Numero de ejemplos de entrenamiento: {len(train_data)}\")\n",
        "print(f\"Número de ejemplos de validación: {len(valid_data)}\")\n",
        "print(f\"Número de ejemplos de test (competencia): {len(test_data)}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fDRnhXAdFGL-"
      },
      "source": [
        "Visualizemos un ejemplo"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "T023Ld4RaSF4",
        "outputId": "b54c9c63-442c-4a49-9783-76de92da5f67",
        "scrolled": false
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('-', 'O'),\n",
              " ('PULPITIS', 'B-Disease'),\n",
              " ('/', 'O'),\n",
              " ('-', 'O'),\n",
              " ('CARIES', 'O'),\n",
              " ('DENTAL', 'O'),\n",
              " ('/', 'O'),\n",
              " ('-', 'O'),\n",
              " ('Fundamento', 'O'),\n",
              " ('Clínico', 'O'),\n",
              " ('APS', 'O'),\n",
              " (':', 'O'),\n",
              " ('Diente', 'B-Body_Part'),\n",
              " ('1', 'I-Body_Part'),\n",
              " ('.', 'I-Body_Part'),\n",
              " ('5', 'I-Body_Part'),\n",
              " ('Pulpitis', 'B-Disease'),\n",
              " ('irreversible', 'I-Disease'),\n",
              " ('.', 'O')]"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ],
      "source": [
        "import random\n",
        "random_item_idx = random.randint(0, len(train_data))\n",
        "random_example = train_data.examples[random_item_idx]\n",
        "list(zip(random_example.text, random_example.nertags))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2l05KYy5FSUy"
      },
      "source": [
        "#### **Construir los vocabularios para el texto y las etiquetas**\n",
        "\n",
        "Los vocabularios son los objetos que contienen todos los tokens (de entrenamiento) posibles para ambos fields. El siguiente paso consiste en construirlos. Para esto, hacemos uso del método `Field.build_vocab` sobre cada uno de nuestros `fields`. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PBhp7WICiibL"
      },
      "outputs": [],
      "source": [
        "TEXT.build_vocab(train_data)\n",
        "NER_TAGS.build_vocab(train_data)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "M4OgUKM_iibO",
        "outputId": "4072eba8-1e60-492e-b2bc-5d5051724586",
        "scrolled": true
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Tokens únicos en TEXT: 17591\n",
            "Tokens únicos en NER_TAGS: 12\n"
          ]
        }
      ],
      "source": [
        "print(f\"Tokens únicos en TEXT: {len(TEXT.vocab)}\")\n",
        "print(f\"Tokens únicos en NER_TAGS: {len(NER_TAGS.vocab)}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d4FeyL9nFnId",
        "outputId": "d535a9d1-2182-49b7-b868-1e00aec8df48"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['<pad>',\n",
              " 'O',\n",
              " 'I-Disease',\n",
              " 'B-Disease',\n",
              " 'I-Body_Part',\n",
              " 'B-Body_Part',\n",
              " 'B-Procedure',\n",
              " 'I-Procedure',\n",
              " 'B-Medication',\n",
              " 'B-Family_Member',\n",
              " 'I-Medication',\n",
              " 'I-Family_Member']"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ],
      "source": [
        "#Veamos las posibles etiquetas que hemos cargado:\n",
        "NER_TAGS.vocab.itos"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HYQDoUqSHFKj"
      },
      "source": [
        "Observen que ademas de los tags NER, tenemos \\<pad\\>, el cual es generado por el dataloader para cumplir con el padding de cada oración.\n",
        "\n",
        "Veamos ahora los tokens mas frecuentes y especiales:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "m5eSLm4diibR",
        "outputId": "1d5c1da7-dd3b-4e42-a4f5-c0093e816d7b"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('.', 7396),\n",
              " (',', 6821),\n",
              " ('-', 4985),\n",
              " ('de', 3811),\n",
              " ('DE', 3645),\n",
              " ('/', 2317),\n",
              " (':', 2209),\n",
              " ('con', 1484),\n",
              " ('y', 1439),\n",
              " ('APS', 1429)]"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ],
      "source": [
        "# Tokens mas frecuentes (Será necesario usar stopwords, eliminar símbolos o nos entregan información (?) )\n",
        "TEXT.vocab.freqs.most_common(10)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WDyNLMPz9duD"
      },
      "outputs": [],
      "source": [
        "# Seteamos algunas variables que nos serán de utilidad mas adelante...\n",
        "UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]\n",
        "PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]\n",
        "\n",
        "PAD_TAG_IDX = NER_TAGS.vocab.stoi[NER_TAGS.pad_token]\n",
        "O_TAG_IDX = NER_TAGS.vocab.stoi['O']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qrYvF3X0sjWL"
      },
      "source": [
        "#### **Frecuencia de los Tags**\n",
        "\n",
        "Visualizemos rápidamente las cantidades y frecuencias de cada tag:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tuXOsbJUiibh",
        "outputId": "507bf208-6aed-4694-c762-5a46cef5cbb7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Tag Ocurrencia Porcentaje\n",
            "\n",
            "O\t101671\t68.1%\n",
            "I-Disease\t21629\t14.5%\n",
            "B-Disease\t8831\t 5.9%\n",
            "I-Body_Part\t6489\t 4.3%\n",
            "B-Body_Part\t3755\t 2.5%\n",
            "B-Procedure\t2891\t 1.9%\n",
            "I-Procedure\t2819\t 1.9%\n",
            "B-Medication\t784\t 0.5%\n",
            "B-Family_Member\t228\t 0.2%\n",
            "I-Medication\t116\t 0.1%\n",
            "I-Family_Member\t9\t 0.0%\n"
          ]
        }
      ],
      "source": [
        "def tag_percentage(tag_counts):\n",
        "    \n",
        "    total_count = sum([count for tag, count in tag_counts])\n",
        "    tag_counts_percentages = [(tag, count, count/total_count) for tag, count in tag_counts]\n",
        "  \n",
        "    return tag_counts_percentages\n",
        "\n",
        "print(\"Tag Ocurrencia Porcentaje\\n\")\n",
        "\n",
        "for tag, count, percent in tag_percentage(NER_TAGS.vocab.freqs.most_common()):\n",
        "    print(f\"{tag}\\t{count}\\t{percent*100:4.1f}%\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y4wPiydnaSGs"
      },
      "source": [
        "#### **Configuramos pytorch y dividimos los datos.**\n",
        "\n",
        "Importante: si tienes problemas con la ram de la gpu, disminuye el tamaño de los batches"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uB7cwLWpaSGs",
        "outputId": "ffb46b14-aa99-4936-be2f-d31ce516c914"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Using cuda\n"
          ]
        }
      ],
      "source": [
        "BATCH_SIZE = 16  # disminuir si hay problemas de ram.\n",
        "\n",
        "# Usar cuda si es que está disponible.\n",
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "print('Using', device)\n",
        "\n",
        "# Dividir datos entre entrenamiento y test. Si van a hacer algún sort no puede ser sobre\n",
        "# el conjunto de testing ya que al hacer sus predicciones sobre el conjunto de test sin etiquetas\n",
        "# debe conservar el orden original para ser comparado con los golden_labels. \n",
        "\n",
        "train_iterator, valid_iterator, test_iterator = legacy.data.BucketIterator.splits(\n",
        "    (train_data, valid_data, test_data),\n",
        "    batch_size=BATCH_SIZE,\n",
        "    device=device,\n",
        "    sort=False,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B21E1eAFId16"
      },
      "source": [
        "#### **Métricas de evaluación**\n",
        "\n",
        "Además, definiremos las métricas que serán usadas tanto para la competencia como para evaluar el modelo: `precision`, `recall` y `micro f1-score`.\n",
        "**Importante**: Noten que la evaluación solo se hace para las Named Entities (sin contar 'O'), toda esta funcionalidad nos la entrega la librería seqeval, pueden revisar más documentación aquí: https://github.com/chakki-works/seqeval. No utilicen el código entregado por sklearn para calcular las métricas ya que esta lo hace a nivel de token y no a nivel de entidad."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "o63ov69_rX2T",
        "outputId": "8cf8100e-d5e1-40be-f0d0-ac6077eea3c3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: seqeval in /usr/local/lib/python3.7/dist-packages (1.2.2)\n",
            "Requirement already satisfied: scikit-learn>=0.21.3 in /usr/local/lib/python3.7/dist-packages (from seqeval) (1.0.2)\n",
            "Requirement already satisfied: numpy>=1.14.0 in /usr/local/lib/python3.7/dist-packages (from seqeval) (1.21.6)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.3->seqeval) (1.1.0)\n",
            "Requirement already satisfied: scipy>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.3->seqeval) (1.4.1)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.3->seqeval) (3.1.0)\n"
          ]
        }
      ],
      "source": [
        "!pip install seqeval"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9mUOOLEWiicU"
      },
      "outputs": [],
      "source": [
        "# Definimos las métricas\n",
        "\n",
        "from seqeval.metrics import f1_score, precision_score, recall_score\n",
        "\n",
        "def calculate_metrics(preds, y_true, pad_idx=PAD_TAG_IDX, o_idx=O_TAG_IDX):\n",
        "    \"\"\"\n",
        "    Calcula precision, recall y f1 de cada batch.\n",
        "    \"\"\"\n",
        "\n",
        "    # Obtener el indice de la clase con probabilidad mayor. (clases)\n",
        "    y_pred = preds.argmax(dim=1, keepdim=True)\n",
        "\n",
        "    # filtramos <pad> para calcular los scores.\n",
        "    mask = [(y_true != pad_idx)]\n",
        "    y_pred = y_pred[mask]\n",
        "    y_true = y_true[mask]\n",
        "\n",
        "    # traemos a la cpu\n",
        "    y_pred = y_pred.view(-1).to('cpu').numpy()\n",
        "    y_true = y_true.to('cpu').numpy()\n",
        "    y_pred = [[NER_TAGS.vocab.itos[v] for v in y_pred]]\n",
        "    y_true = [[NER_TAGS.vocab.itos[v] for v in y_true]]\n",
        "    \n",
        "    # calcular scores\n",
        "    f1 = f1_score(y_true, y_pred, mode='strict')\n",
        "    precision = precision_score(y_true, y_pred, mode='strict')\n",
        "    recall = recall_score(y_true, y_pred, mode='strict')\n",
        "\n",
        "    return precision, recall, f1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hod516H1aSG2"
      },
      "source": [
        "-------------------\n",
        "\n",
        "### **Modelo Baseline**\n",
        "\n",
        "Teniendo ya cargado los datos, toca definir nuestro modelo. Este baseline tendrá una capa de embedding, unas cuantas LSTM y una capa de salida y usará dropout en el entrenamiento.\n",
        "\n",
        "Este constará de los siguientes pasos: \n",
        "\n",
        "1. Definir la clase que contendrá la red.\n",
        "2. Definir los hiperparámetros e inicializar la red. \n",
        "3. Definir el número de épocas de entrenamiento\n",
        "4. Definir la función de loss.\n",
        "\n",
        "\n",
        "\n",
        "Recomendamos que para experimentar, encapsules los modelos en una sola variable y luego la fijes en model para entrenarla"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rMPL08XqaSG3"
      },
      "outputs": [],
      "source": [
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import torch.optim as optim\n",
        "\n",
        "\n",
        "# Definir la red\n",
        "class NER_RNN(nn.Module):\n",
        "    def __init__(self, \n",
        "                 input_dim, \n",
        "                 embedding_dim, \n",
        "                 hidden_dim, \n",
        "                 output_dim,\n",
        "                 n_layers, \n",
        "                 bidirectional, \n",
        "                 dropout, \n",
        "                 pad_idx):\n",
        "\n",
        "        super().__init__()\n",
        "\n",
        "        # Capa de embedding\n",
        "        self.embedding = nn.Embedding(input_dim,\n",
        "                                      embedding_dim,\n",
        "                                      padding_idx=pad_idx,\n",
        "                                      )\n",
        "\n",
        "        # Capa LSTM\n",
        "        self.lstm = nn.LSTM(embedding_dim,\n",
        "                           hidden_dim,\n",
        "                           num_layers=n_layers,\n",
        "                           bidirectional=bidirectional, \n",
        "                           dropout = dropout if n_layers > 1 else 0)\n",
        "\n",
        "        # Capa de salida\n",
        "        self.fc = nn.Linear(hidden_dim * 2 if bidirectional else hidden_dim,\n",
        "                            output_dim)\n",
        "\n",
        "        # Dropout\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "\n",
        "    def forward(self, text):\n",
        "\n",
        "        #text = [sent len, batch size]\n",
        "\n",
        "        # Convertir lo enviado a embedding\n",
        "        embedded = self.dropout(self.embedding(text))\n",
        "        \n",
        "        outputs, (hidden, cell) = self.lstm(embedded)\n",
        "        #embedded = [sent len, batch size, emb dim]\n",
        "\n",
        "        # Pasar los embeddings por la rnn (LSTM)\n",
        "\n",
        "        #output = [sent len, batch size, hid dim * n directions]\n",
        "        #hidden/cell = [n layers * n directions, batch size, hid dim]\n",
        "\n",
        "        # Predecir usando la capa de salida.\n",
        "        predictions = self.fc(self.dropout(outputs))\n",
        "        #predictions = [sent len, batch size, output dim]\n",
        "\n",
        "        return predictions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cCl3530VaSG7"
      },
      "source": [
        "#### **Hiperparámetros de la red**\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EHdi3QdOaSG8"
      },
      "outputs": [],
      "source": [
        "# tamaño del vocabulario. recuerden que la entrada son vectores bag of word(one-hot).\n",
        "INPUT_DIM = len(TEXT.vocab)\n",
        "EMBEDDING_DIM = 200  # dimensión de los embeddings.\n",
        "HIDDEN_DIM = 128  # dimensión de la capas LSTM\n",
        "OUTPUT_DIM = len(NER_TAGS.vocab)  # número de clases\n",
        "\n",
        "N_LAYERS = 3  # número de capas.\n",
        "DROPOUT = 0.5\n",
        "BIDIRECTIONAL = False\n",
        "\n",
        "# Creamos nuestro modelo.\n",
        "baseline_model = NER_RNN(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM,\n",
        "                         N_LAYERS, BIDIRECTIONAL, DROPOUT, PAD_IDX)\n",
        "\n",
        "baseline_model_name = 'baseline'  # nombre que tendrá el modelo guardado..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jlF1DhJeaSHA"
      },
      "outputs": [],
      "source": [
        "baseline_n_epochs = 10"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s3u4imJGaSHE"
      },
      "source": [
        "#### Definimos la función de loss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6G_4k99_aSHG"
      },
      "outputs": [],
      "source": [
        "# Loss: Cross Entropy\n",
        "TAG_PAD_IDX = NER_TAGS.vocab.stoi[NER_TAGS.pad_token]\n",
        "baseline_criterion = nn.CrossEntropyLoss(ignore_index = TAG_PAD_IDX)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lRYOEDiQaSHK"
      },
      "source": [
        "--------------------\n",
        "### Modelo 1: RNN con GRU\n",
        "\n",
        "Aqui se implemento la RNN con una capa de GRU envez de LSTM.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DAJAedOOoacM"
      },
      "outputs": [],
      "source": [
        "# RNN con GRU\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import torch.optim as optim\n",
        "\n",
        "class NER_RNN_GRU(nn.Module):\n",
        "    def __init__(self, \n",
        "                 input_dim, \n",
        "                 embedding_dim, \n",
        "                 hidden_dim, \n",
        "                 output_dim,\n",
        "                 n_layers, \n",
        "                 bidirectional, \n",
        "                 dropout, \n",
        "                 pad_idx):\n",
        "\n",
        "        super().__init__()\n",
        "\n",
        "        # Capa de embedding\n",
        "        self.embedding = nn.Embedding(input_dim,\n",
        "                                      embedding_dim,\n",
        "                                      padding_idx=pad_idx,\n",
        "                                      )\n",
        "\n",
        "        # Capa LSTM\n",
        "        #self.lstm = nn.LSTM(embedding_dim,\n",
        "                           #hidden_dim,\n",
        "                           #num_layers=n_layers,\n",
        "                           #bidirectional=bidirectional, \n",
        "                           #dropout = dropout if n_layers > 1 else 0)\n",
        "        \n",
        "        # Capa GRU\n",
        "        self.gru = nn.GRU(embedding_dim,\n",
        "                           hidden_dim,\n",
        "                           num_layers=n_layers,\n",
        "                           bidirectional=bidirectional, \n",
        "                           dropout = dropout if n_layers > 1 else 0)\n",
        "\n",
        "        # Capa de salida\n",
        "        self.fc = nn.Linear(hidden_dim * 2 if bidirectional else hidden_dim,\n",
        "                            output_dim)\n",
        "\n",
        "        # Dropout\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "\n",
        "    def forward(self, text):\n",
        "\n",
        "        #text = [sent len, batch size]\n",
        "\n",
        "        # Convertir lo enviado a embedding\n",
        "        embedded = self.dropout(self.embedding(text))\n",
        "        \n",
        "        outputs, hidden = self.gru(embedded)\n",
        "        #embedded = [sent len, batch size, emb dim]\n",
        "\n",
        "        # Pasar los embeddings por la rnn (LSTM)\n",
        "\n",
        "        #output = [sent len, batch size, hid dim * n directions]\n",
        "        #hidden/cell = [n layers * n directions, batch size, hid dim]\n",
        "\n",
        "        # Predecir usando la capa de salida.\n",
        "        predictions = self.fc(self.dropout(outputs))\n",
        "        #predictions = [sent len, batch size, output dim]\n",
        "\n",
        "        return predictions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c81f8ki5aSHL"
      },
      "outputs": [],
      "source": [
        "# tamaño del vocabulario. recuerden que la entrada son vectores bag of word(one-hot).\n",
        "INPUT_DIM_1 = len(TEXT.vocab)\n",
        "EMBEDDING_DIM_1 = 200  # dimensión de los embeddings.\n",
        "HIDDEN_DIM_1 = 128  # dimensión de la capas GRU\n",
        "OUTPUT_DIM_1 = len(NER_TAGS.vocab)  # número de clases\n",
        "\n",
        "N_LAYERS_1 = 3  # número de capas.\n",
        "DROPOUT_1 = 0.5\n",
        "BIDIRECTIONAL_1 = False\n",
        "\n",
        "model_1 = NER_RNN_GRU(INPUT_DIM_1, EMBEDDING_DIM_1, HIDDEN_DIM_1, OUTPUT_DIM_1,\n",
        "                         N_LAYERS_1, BIDIRECTIONAL_1, DROPOUT_1, PAD_IDX)\n",
        "model_name_1 = 'GRU_baseline'\n",
        "n_epochs_1 = 10\n",
        "# loss_1 = ..."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bYvVcSxOTYyJ"
      },
      "source": [
        "#### Experimento LSTM vs GRU "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QcGmo4hNVFFx"
      },
      "source": [
        "Para decidir si utilizar LSTM o GRU se corre el siguiente experimento y se compara cual de los dos modelos tiene el mejor F1 score en la epoca en la que se obtiene la menor loss."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "p1pSxy8aRnJq",
        "outputId": "d66af4cf-96a4-4ce6-86a9-0ee34f248007"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
            "  warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[{'model': 'EXP1_baseline', 'epoch': 4, 'loss': 0.393015854326742, 'f1': 0.7234723824285947}, {'model': 'EXP1_GRU_baseline', 'epoch': 4, 'loss': 0.4021625643862145, 'f1': 0.728929625737532}]\n"
          ]
        }
      ],
      "source": [
        "exp_baseline_model = NER_RNN(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM,\n",
        "                         N_LAYERS, BIDIRECTIONAL, DROPOUT, PAD_IDX)\n",
        "exp_baseline_model_name = 'EXP1_baseline'\n",
        "\n",
        "exp_model_1 = NER_RNN_GRU(INPUT_DIM_1, EMBEDDING_DIM_1, HIDDEN_DIM_1, OUTPUT_DIM_1,\n",
        "                         N_LAYERS_1, BIDIRECTIONAL_1, DROPOUT_1, PAD_IDX)\n",
        "exp_model_name_1 = 'EXP1_GRU_baseline'\n",
        "\n",
        "models = [[exp_baseline_model, exp_baseline_model_name], [exp_model_1, exp_model_name_1]]\n",
        "results = []\n",
        "\n",
        "for a_model in models:\n",
        "\n",
        "    model = a_model[0]\n",
        "    model_name = a_model[1]\n",
        "    criterion = baseline_criterion\n",
        "    n_epochs = baseline_n_epochs\n",
        "\n",
        "    def init_weights(m):\n",
        "        # Inicializamos los pesos como aleatorios\n",
        "        for name, param in m.named_parameters():\n",
        "            nn.init.normal_(param.data, mean=0, std=0.1) \n",
        "            \n",
        "        # Seteamos como 0 los embeddings de UNK y PAD.\n",
        "        model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)\n",
        "        model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)\n",
        "            \n",
        "    model.apply(init_weights)\n",
        "\n",
        "    def count_parameters(model):\n",
        "        return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
        "\n",
        "    # Optimizador\n",
        "    optimizer = optim.Adam(model.parameters())\n",
        "\n",
        "    # Enviamos el modelo y la loss a cuda (en el caso en que esté disponible)\n",
        "    model = model.to(device)\n",
        "    criterion = criterion.to(device)\n",
        "\n",
        "    def train(model, iterator, optimizer, criterion):\n",
        "\n",
        "        epoch_loss = 0\n",
        "        epoch_precision = 0\n",
        "        epoch_recall = 0\n",
        "        epoch_f1 = 0\n",
        "\n",
        "        model.train()\n",
        "\n",
        "        # Por cada batch del iterador de la época:\n",
        "        for batch in iterator:\n",
        "\n",
        "            # Extraemos el texto y los tags del batch que estamos procesado\n",
        "            text = batch.text\n",
        "            tags = batch.nertags\n",
        "\n",
        "            # Reiniciamos los gradientes calculados en la iteración anterior\n",
        "            optimizer.zero_grad()\n",
        "\n",
        "            #text = [sent len, batch size]\n",
        "\n",
        "            # Predecimos los tags del texto del batch.\n",
        "            predictions = model(text)\n",
        "\n",
        "            #predictions = [sent len, batch size, output dim]\n",
        "            #tags = [sent len, batch size]\n",
        "\n",
        "            # Reordenamos los datos para calcular la loss\n",
        "            predictions = predictions.view(-1, predictions.shape[-1])\n",
        "            tags = tags.view(-1)\n",
        "\n",
        "            #predictions = [sent len * batch size, output dim]\n",
        "\n",
        "\n",
        "\n",
        "            # Calculamos el Cross Entropy de las predicciones con respecto a las etiquetas reales\n",
        "            loss = criterion(predictions, tags)\n",
        "            \n",
        "            # Calculamos el accuracy\n",
        "            precision, recall, f1 = calculate_metrics(predictions, tags)\n",
        "\n",
        "            # Calculamos los gradientes\n",
        "            loss.backward()\n",
        "\n",
        "            # Actualizamos los parámetros de la red\n",
        "            optimizer.step()\n",
        "\n",
        "            # Actualizamos el loss y las métricas\n",
        "            epoch_loss += loss.item()\n",
        "            epoch_precision += precision\n",
        "            epoch_recall += recall\n",
        "            epoch_f1 += f1\n",
        "\n",
        "        return epoch_loss / len(iterator), epoch_precision / len(\n",
        "            iterator), epoch_recall / len(iterator), epoch_f1 / len(iterator)\n",
        "\n",
        "    def evaluate(model, iterator, criterion):\n",
        "\n",
        "        epoch_loss = 0\n",
        "        epoch_precision = 0\n",
        "        epoch_recall = 0\n",
        "        epoch_f1 = 0\n",
        "\n",
        "        model.eval()\n",
        "\n",
        "        # Indicamos que ahora no guardaremos los gradientes\n",
        "        with torch.no_grad():\n",
        "            # Por cada batch\n",
        "            for batch in iterator:\n",
        "\n",
        "                text = batch.text\n",
        "                tags = batch.nertags\n",
        "\n",
        "                # Predecimos\n",
        "                predictions = model(text)\n",
        "\n",
        "                predictions = predictions.view(-1, predictions.shape[-1])\n",
        "                tags = tags.view(-1)\n",
        "\n",
        "                # Calculamos el Cross Entropy de las predicciones con respecto a las etiquetas reales\n",
        "                loss = criterion(predictions, tags)\n",
        "\n",
        "                # Calculamos las métricas\n",
        "                precision, recall, f1 = calculate_metrics(predictions, tags)\n",
        "\n",
        "                # Actualizamos el loss y las métricas\n",
        "                epoch_loss += loss.item()\n",
        "                epoch_precision += precision\n",
        "                epoch_recall += recall\n",
        "                epoch_f1 += f1\n",
        "\n",
        "        return epoch_loss / len(iterator), epoch_precision / len(\n",
        "            iterator), epoch_recall / len(iterator), epoch_f1 / len(iterator)\n",
        "\n",
        "    import time\n",
        "\n",
        "    def epoch_time(start_time, end_time):\n",
        "        elapsed_time = end_time - start_time\n",
        "        elapsed_mins = int(elapsed_time / 60)\n",
        "        elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
        "        return elapsed_mins, elapsed_secs\n",
        "\n",
        "    best_valid_loss = float('inf')\n",
        "    best_loss_f1 = -1\n",
        "    best_loss_epoch = -1\n",
        "\n",
        "    for epoch in range(n_epochs):\n",
        "\n",
        "        start_time = time.time()\n",
        "\n",
        "        # Recuerdo: train_iterator y valid_iterator contienen el dataset dividido en batches.\n",
        "\n",
        "        # Entrenar\n",
        "        train_loss, train_precision, train_recall, train_f1 = train(\n",
        "            model, train_iterator, optimizer, criterion)\n",
        "\n",
        "        # Evaluar (valid = validación)\n",
        "        valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "            model, valid_iterator, criterion)\n",
        "\n",
        "        end_time = time.time()\n",
        "\n",
        "        epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
        "\n",
        "        # Si obtuvimos mejores resultados, guardamos este modelo en el almacenamiento (para poder cargarlo luego)\n",
        "        # Si detienen el entrenamiento prematuramente, pueden cargar el modelo en el siguiente recuadro de código.\n",
        "        if valid_loss < best_valid_loss:\n",
        "            best_valid_loss = valid_loss\n",
        "            best_loss_f1 = valid_f1\n",
        "            best_loss_epoch = epoch\n",
        "            torch.save(model.state_dict(), '{}.pt'.format(model_name))\n",
        "        # Si ya no mejoramos el loss de validación, terminamos de entrenar.\n",
        "\n",
        "    results.append({'model': model_name, 'epoch': best_loss_epoch,'loss': best_valid_loss, 'f1': best_loss_f1})\n",
        "\n",
        "print(results)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fAt4JO9TUu9B"
      },
      "source": [
        "Ambas implementaciones parecen tener un desempeño similar por lo que no es claro cual es mejor. Por simplicidad se continuara utilizando unicamente GRU."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rV9oLkN1aSHO"
      },
      "source": [
        "---------------\n",
        "\n",
        "### Modelo 2: RNN con GRU Bidireccional"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3GQR0qHfVcR9"
      },
      "source": [
        "A continuacion se define una RNN con GRU pero utilizando bidireccionalidad, lo que permite al modelo leer la secuencia de tokens de izquiera a derecha y de derecha a izquierda dandole mayor contexto."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KWPzETaNaSHP"
      },
      "outputs": [],
      "source": [
        "# tamaño del vocabulario. recuerden que la entrada son vectores bag of word(one-hot).\n",
        "INPUT_DIM_2 = len(TEXT.vocab)\n",
        "EMBEDDING_DIM_2 = 200  # dimensión de los embeddings.\n",
        "HIDDEN_DIM_2 = 128  # dimensión de la capas GRU\n",
        "OUTPUT_DIM_2 = len(NER_TAGS.vocab)  # número de clases\n",
        "\n",
        "N_LAYERS_2 = 3  # número de capas.\n",
        "DROPOUT_2 = 0.5\n",
        "BIDIRECTIONAL_2 = True\n",
        "\n",
        "model_2 = NER_RNN_GRU(INPUT_DIM_2, EMBEDDING_DIM_2, HIDDEN_DIM_2, OUTPUT_DIM_2,\n",
        "                         N_LAYERS_2, BIDIRECTIONAL_2, DROPOUT_2, PAD_IDX)\n",
        "model_name_2 = 'GRU_bidirectional'\n",
        "n_epochs_2 = 10\n",
        "# loss_3 = ..."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O1uGZXNJUUEN"
      },
      "source": [
        "#### Experimento GRU no-bidireccional vs GRU bidireccional"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MLxu8_dKV0pw"
      },
      "source": [
        "Para probar cual es mejor se entrena GRU sin bidireccionalidad y GRU con bidireccionalidad y se compara lo F1 scores obtenidos en la epoca con la menor loss al entrenar."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b9hGNxLwUTkv",
        "outputId": "9605cd50-9d07-4da9-b25d-449c00bc0ea7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
            "  warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[{'model': 'EXP2_GRU_baseline', 'epoch': 3, 'loss': 0.3995739466377667, 'f1': 0.7329772039165023}, {'model': 'EXP2_GRU_bidirectional', 'epoch': 4, 'loss': 0.36864219712359564, 'f1': 0.77779020839118}]\n"
          ]
        }
      ],
      "source": [
        "exp_model_1 = NER_RNN_GRU(INPUT_DIM_1, EMBEDDING_DIM_1, HIDDEN_DIM_1, OUTPUT_DIM_1,\n",
        "                         N_LAYERS_1, BIDIRECTIONAL_1, DROPOUT_1, PAD_IDX)\n",
        "exp_model_name_1 = 'EXP2_GRU_baseline'\n",
        "\n",
        "exp_model_2 = NER_RNN_GRU(INPUT_DIM_2, EMBEDDING_DIM_2, HIDDEN_DIM_2, OUTPUT_DIM_2,\n",
        "                         N_LAYERS_2, BIDIRECTIONAL_2, DROPOUT_2, PAD_IDX)\n",
        "exp_model_name_2 = 'EXP2_GRU_bidirectional'\n",
        "\n",
        "models = [[exp_model_1, exp_model_name_1], [exp_model_2, exp_model_name_2]]\n",
        "results = []\n",
        "\n",
        "for a_model in models:\n",
        "\n",
        "    model = a_model[0]\n",
        "    model_name = a_model[1]\n",
        "    criterion = baseline_criterion\n",
        "    n_epochs = n_epochs_2\n",
        "\n",
        "    def init_weights(m):\n",
        "        # Inicializamos los pesos como aleatorios\n",
        "        for name, param in m.named_parameters():\n",
        "            nn.init.normal_(param.data, mean=0, std=0.1) \n",
        "            \n",
        "        # Seteamos como 0 los embeddings de UNK y PAD.\n",
        "        model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)\n",
        "        model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)\n",
        "            \n",
        "    model.apply(init_weights)\n",
        "\n",
        "    def count_parameters(model):\n",
        "        return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
        "\n",
        "    # Optimizador\n",
        "    optimizer = optim.Adam(model.parameters())\n",
        "\n",
        "    # Enviamos el modelo y la loss a cuda (en el caso en que esté disponible)\n",
        "    model = model.to(device)\n",
        "    criterion = criterion.to(device)\n",
        "\n",
        "    def train(model, iterator, optimizer, criterion):\n",
        "\n",
        "        epoch_loss = 0\n",
        "        epoch_precision = 0\n",
        "        epoch_recall = 0\n",
        "        epoch_f1 = 0\n",
        "\n",
        "        model.train()\n",
        "\n",
        "        # Por cada batch del iterador de la época:\n",
        "        for batch in iterator:\n",
        "\n",
        "            # Extraemos el texto y los tags del batch que estamos procesado\n",
        "            text = batch.text\n",
        "            tags = batch.nertags\n",
        "\n",
        "            # Reiniciamos los gradientes calculados en la iteración anterior\n",
        "            optimizer.zero_grad()\n",
        "\n",
        "            #text = [sent len, batch size]\n",
        "\n",
        "            # Predecimos los tags del texto del batch.\n",
        "            predictions = model(text)\n",
        "\n",
        "            #predictions = [sent len, batch size, output dim]\n",
        "            #tags = [sent len, batch size]\n",
        "\n",
        "            # Reordenamos los datos para calcular la loss\n",
        "            predictions = predictions.view(-1, predictions.shape[-1])\n",
        "            tags = tags.view(-1)\n",
        "\n",
        "            #predictions = [sent len * batch size, output dim]\n",
        "\n",
        "\n",
        "\n",
        "            # Calculamos el Cross Entropy de las predicciones con respecto a las etiquetas reales\n",
        "            loss = criterion(predictions, tags)\n",
        "            \n",
        "            # Calculamos el accuracy\n",
        "            precision, recall, f1 = calculate_metrics(predictions, tags)\n",
        "\n",
        "            # Calculamos los gradientes\n",
        "            loss.backward()\n",
        "\n",
        "            # Actualizamos los parámetros de la red\n",
        "            optimizer.step()\n",
        "\n",
        "            # Actualizamos el loss y las métricas\n",
        "            epoch_loss += loss.item()\n",
        "            epoch_precision += precision\n",
        "            epoch_recall += recall\n",
        "            epoch_f1 += f1\n",
        "\n",
        "        return epoch_loss / len(iterator), epoch_precision / len(\n",
        "            iterator), epoch_recall / len(iterator), epoch_f1 / len(iterator)\n",
        "\n",
        "    def evaluate(model, iterator, criterion):\n",
        "\n",
        "        epoch_loss = 0\n",
        "        epoch_precision = 0\n",
        "        epoch_recall = 0\n",
        "        epoch_f1 = 0\n",
        "\n",
        "        model.eval()\n",
        "\n",
        "        # Indicamos que ahora no guardaremos los gradientes\n",
        "        with torch.no_grad():\n",
        "            # Por cada batch\n",
        "            for batch in iterator:\n",
        "\n",
        "                text = batch.text\n",
        "                tags = batch.nertags\n",
        "\n",
        "                # Predecimos\n",
        "                predictions = model(text)\n",
        "\n",
        "                predictions = predictions.view(-1, predictions.shape[-1])\n",
        "                tags = tags.view(-1)\n",
        "\n",
        "                # Calculamos el Cross Entropy de las predicciones con respecto a las etiquetas reales\n",
        "                loss = criterion(predictions, tags)\n",
        "\n",
        "                # Calculamos las métricas\n",
        "                precision, recall, f1 = calculate_metrics(predictions, tags)\n",
        "\n",
        "                # Actualizamos el loss y las métricas\n",
        "                epoch_loss += loss.item()\n",
        "                epoch_precision += precision\n",
        "                epoch_recall += recall\n",
        "                epoch_f1 += f1\n",
        "\n",
        "        return epoch_loss / len(iterator), epoch_precision / len(\n",
        "            iterator), epoch_recall / len(iterator), epoch_f1 / len(iterator)\n",
        "\n",
        "    import time\n",
        "\n",
        "    def epoch_time(start_time, end_time):\n",
        "        elapsed_time = end_time - start_time\n",
        "        elapsed_mins = int(elapsed_time / 60)\n",
        "        elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
        "        return elapsed_mins, elapsed_secs\n",
        "\n",
        "    best_valid_loss = float('inf')\n",
        "    best_loss_f1 = -1\n",
        "    best_loss_epoch = -1\n",
        "\n",
        "    for epoch in range(n_epochs):\n",
        "\n",
        "        start_time = time.time()\n",
        "\n",
        "        # Recuerdo: train_iterator y valid_iterator contienen el dataset dividido en batches.\n",
        "\n",
        "        # Entrenar\n",
        "        train_loss, train_precision, train_recall, train_f1 = train(\n",
        "            model, train_iterator, optimizer, criterion)\n",
        "\n",
        "        # Evaluar (valid = validación)\n",
        "        valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "            model, valid_iterator, criterion)\n",
        "\n",
        "        end_time = time.time()\n",
        "\n",
        "        epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
        "\n",
        "        # Si obtuvimos mejores resultados, guardamos este modelo en el almacenamiento (para poder cargarlo luego)\n",
        "        # Si detienen el entrenamiento prematuramente, pueden cargar el modelo en el siguiente recuadro de código.\n",
        "        if valid_loss < best_valid_loss:\n",
        "            best_valid_loss = valid_loss\n",
        "            best_loss_f1 = valid_f1\n",
        "            best_loss_epoch = epoch\n",
        "            torch.save(model.state_dict(), '{}.pt'.format(model_name))\n",
        "        # Si ya no mejoramos el loss de validación, terminamos de entrenar.\n",
        "\n",
        "    results.append({'model': model_name, 'epoch': best_loss_epoch, 'loss': best_valid_loss, 'f1': best_loss_f1})\n",
        "\n",
        "print(results)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YSTBrECSVCFb"
      },
      "source": [
        "De los resultados queda claro que es mejor la bidireccionalidad para esta task por lo que se la continuara utilizando de aqui en adelante."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zpy3p7YaaSHT"
      },
      "source": [
        "---------------\n",
        "\n",
        "\n",
        "### Modelo 3: Probando distintos valores de dropout"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "A continuacion se intenta variar los valores de dropout para una GRU con bidireccionalidad para ver si cambian los resultados."
      ],
      "metadata": {
        "id": "BQeu3alvol-Q"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_w0CFjA8aSHU"
      },
      "outputs": [],
      "source": [
        "# tamaño del vocabulario. recuerden que la entrada son vectores bag of word(one-hot).\n",
        "INPUT_DIM_3 = len(TEXT.vocab)\n",
        "EMBEDDING_DIM_3 = 200  # dimensión de los embeddings.\n",
        "HIDDEN_DIM_3 = 128  # dimensión de la capas GRU\n",
        "OUTPUT_DIM_3 = len(NER_TAGS.vocab)  # número de clases\n",
        "\n",
        "N_LAYERS_3 = 3  # número de capas.\n",
        "DROPOUTS = [0.3, 0.4, 0.5, 0.6, 0.7]\n",
        "BIDIRECTIONAL_3 = True"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d4-UEViH6jXU"
      },
      "source": [
        "#### Experimento dropout"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wglKWHF96b8k",
        "outputId": "0844a316-c711-4906-fcb3-67cf28dea522"
      },
      "outputs": [
        {
          "metadata": {
            "tags": null
          },
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
            "  warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[{'dropout': 0.3, 'epoch': 2, 'loss': 0.3443973519440208, 'f1': 0.7722995065403555}, {'dropout': 0.4, 'epoch': 2, 'loss': 0.35709747006850584, 'f1': 0.7604539452862841}, {'dropout': 0.5, 'epoch': 3, 'loss': 0.37460201300148455, 'f1': 0.7669412476945908}, {'dropout': 0.6, 'epoch': 3, 'loss': 0.36943052975194796, 'f1': 0.751337839831183}, {'dropout': 0.7, 'epoch': 5, 'loss': 0.3910718292796186, 'f1': 0.7461629020967002}]\n"
          ]
        }
      ],
      "source": [
        "results = []\n",
        "\n",
        "for value in DROPOUTS:\n",
        "\n",
        "    model_3 = NER_RNN_GRU(INPUT_DIM_3, EMBEDDING_DIM_3, HIDDEN_DIM_3, OUTPUT_DIM_3,\n",
        "                         N_LAYERS_3, BIDIRECTIONAL_3, value, PAD_IDX)\n",
        "    model_name_3 = 'EXP3_GRU_DROPOUT_TEST_' + str(value)\n",
        "    n_epochs_3 = 10\n",
        "\n",
        "    model = model_3\n",
        "    model_name = model_name_3\n",
        "    criterion = baseline_criterion\n",
        "    n_epochs = n_epochs_3\n",
        "\n",
        "    def init_weights(m):\n",
        "        # Inicializamos los pesos como aleatorios\n",
        "        for name, param in m.named_parameters():\n",
        "            nn.init.normal_(param.data, mean=0, std=0.1) \n",
        "            \n",
        "        # Seteamos como 0 los embeddings de UNK y PAD.\n",
        "        model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)\n",
        "        model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)\n",
        "            \n",
        "    model.apply(init_weights)\n",
        "\n",
        "    def count_parameters(model):\n",
        "        return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
        "\n",
        "    # Optimizador\n",
        "    optimizer = optim.Adam(model.parameters())\n",
        "\n",
        "    # Enviamos el modelo y la loss a cuda (en el caso en que esté disponible)\n",
        "    model = model.to(device)\n",
        "    criterion = criterion.to(device)\n",
        "\n",
        "    def train(model, iterator, optimizer, criterion):\n",
        "\n",
        "        epoch_loss = 0\n",
        "        epoch_precision = 0\n",
        "        epoch_recall = 0\n",
        "        epoch_f1 = 0\n",
        "\n",
        "        model.train()\n",
        "\n",
        "        # Por cada batch del iterador de la época:\n",
        "        for batch in iterator:\n",
        "\n",
        "            # Extraemos el texto y los tags del batch que estamos procesado\n",
        "            text = batch.text\n",
        "            tags = batch.nertags\n",
        "\n",
        "            # Reiniciamos los gradientes calculados en la iteración anterior\n",
        "            optimizer.zero_grad()\n",
        "\n",
        "            #text = [sent len, batch size]\n",
        "\n",
        "            # Predecimos los tags del texto del batch.\n",
        "            predictions = model(text)\n",
        "\n",
        "            #predictions = [sent len, batch size, output dim]\n",
        "            #tags = [sent len, batch size]\n",
        "\n",
        "            # Reordenamos los datos para calcular la loss\n",
        "            predictions = predictions.view(-1, predictions.shape[-1])\n",
        "            tags = tags.view(-1)\n",
        "\n",
        "            #predictions = [sent len * batch size, output dim]\n",
        "\n",
        "\n",
        "\n",
        "            # Calculamos el Cross Entropy de las predicciones con respecto a las etiquetas reales\n",
        "            loss = criterion(predictions, tags)\n",
        "            \n",
        "            # Calculamos el accuracy\n",
        "            precision, recall, f1 = calculate_metrics(predictions, tags)\n",
        "\n",
        "            # Calculamos los gradientes\n",
        "            loss.backward()\n",
        "\n",
        "            # Actualizamos los parámetros de la red\n",
        "            optimizer.step()\n",
        "\n",
        "            # Actualizamos el loss y las métricas\n",
        "            epoch_loss += loss.item()\n",
        "            epoch_precision += precision\n",
        "            epoch_recall += recall\n",
        "            epoch_f1 += f1\n",
        "\n",
        "        return epoch_loss / len(iterator), epoch_precision / len(\n",
        "            iterator), epoch_recall / len(iterator), epoch_f1 / len(iterator)\n",
        "\n",
        "    def evaluate(model, iterator, criterion):\n",
        "\n",
        "        epoch_loss = 0\n",
        "        epoch_precision = 0\n",
        "        epoch_recall = 0\n",
        "        epoch_f1 = 0\n",
        "\n",
        "        model.eval()\n",
        "\n",
        "        # Indicamos que ahora no guardaremos los gradientes\n",
        "        with torch.no_grad():\n",
        "            # Por cada batch\n",
        "            for batch in iterator:\n",
        "\n",
        "                text = batch.text\n",
        "                tags = batch.nertags\n",
        "\n",
        "                # Predecimos\n",
        "                predictions = model(text)\n",
        "\n",
        "                predictions = predictions.view(-1, predictions.shape[-1])\n",
        "                tags = tags.view(-1)\n",
        "\n",
        "                # Calculamos el Cross Entropy de las predicciones con respecto a las etiquetas reales\n",
        "                loss = criterion(predictions, tags)\n",
        "\n",
        "                # Calculamos las métricas\n",
        "                precision, recall, f1 = calculate_metrics(predictions, tags)\n",
        "\n",
        "                # Actualizamos el loss y las métricas\n",
        "                epoch_loss += loss.item()\n",
        "                epoch_precision += precision\n",
        "                epoch_recall += recall\n",
        "                epoch_f1 += f1\n",
        "\n",
        "        return epoch_loss / len(iterator), epoch_precision / len(\n",
        "            iterator), epoch_recall / len(iterator), epoch_f1 / len(iterator)\n",
        "\n",
        "    import time\n",
        "\n",
        "    def epoch_time(start_time, end_time):\n",
        "        elapsed_time = end_time - start_time\n",
        "        elapsed_mins = int(elapsed_time / 60)\n",
        "        elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
        "        return elapsed_mins, elapsed_secs\n",
        "\n",
        "    best_valid_loss = float('inf')\n",
        "    best_loss_f1 = -1\n",
        "    best_loss_epoch = -1\n",
        "\n",
        "    for epoch in range(n_epochs):\n",
        "\n",
        "        start_time = time.time()\n",
        "\n",
        "        # Recuerdo: train_iterator y valid_iterator contienen el dataset dividido en batches.\n",
        "\n",
        "        # Entrenar\n",
        "        train_loss, train_precision, train_recall, train_f1 = train(\n",
        "            model, train_iterator, optimizer, criterion)\n",
        "\n",
        "        # Evaluar (valid = validación)\n",
        "        valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "            model, valid_iterator, criterion)\n",
        "\n",
        "        end_time = time.time()\n",
        "\n",
        "        epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
        "\n",
        "        # Si obtuvimos mejores resultados, guardamos este modelo en el almacenamiento (para poder cargarlo luego)\n",
        "        # Si detienen el entrenamiento prematuramente, pueden cargar el modelo en el siguiente recuadro de código.\n",
        "        if valid_loss < best_valid_loss:\n",
        "            best_valid_loss = valid_loss\n",
        "            best_loss_f1 = valid_f1\n",
        "            best_loss_epoch = epoch\n",
        "            torch.save(model.state_dict(), '{}.pt'.format(model_name))\n",
        "        # Si ya no mejoramos el loss de validación, terminamos de entrenar.\n",
        "\n",
        "    results.append({'dropout': value, 'epoch': best_loss_epoch, 'loss': best_valid_loss, 'f1': best_loss_f1})\n",
        "\n",
        "print(results)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XLKV8LMpEq-r"
      },
      "source": [
        "Se puede ver que variar el dropout no afecta significativamente el desempeño de la red por lo que se lo mantiene con el mismo valor que tenia en el baseline, es decir $0.5$. Para valores mas altos que 0.5 pareciera tender a bajar su desempeño."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HFzGDie-3H7E"
      },
      "source": [
        "---------------\n",
        "\n",
        "\n",
        "### Modelo 4: Analizar el número de épocas"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "A continuacion se analiza el numero optimo de epocas por las cuales entrenar el modelo."
      ],
      "metadata": {
        "id": "wNedRxWbu5eO"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true,
          "base_uri": "https://localhost:8080/"
        },
        "id": "Q0dfD05-3Msm",
        "outputId": "fb58e36e-102f-4cfa-a6fe-b085f382108c"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
            "  warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "MODELO  LSTM\n",
            "baseline\n",
            "[0.5904348859829562, 0.4626941444086177, 0.4275872465223074, 0.4148610032030514, 0.4128565712432776, 0.38700855150818825, 0.4037221792553152, 0.4193472330059324, 0.4040957987308502, 0.44608716427215506, 0.4372008546654667, 0.4692943466028997, 0.47939466392355307, 0.44829852666173664, 0.47841435936944826, 0.5102926359644958, 0.5598794727453164, 0.5534358003309795, 0.5764952355197498, 0.5950069374271801]\n",
            "La perdida de los datos de validacion aumenta en:\n",
            "Epoch: 06 | Val. Loss: 0.387009 | Val. f1: 0.734486\n",
            "Epoch: 07 | Val. Loss: 0.403722 | Val. f1: 0.737510\n",
            "Cantidad de epocas a utilizar: 06\n",
            "\n",
            "------------------------------------------------\n",
            "\n",
            "MODELO  GRU\n",
            "GRU_baseline_2\n",
            "[0.5469026826322079, 0.4205322467855045, 0.39813818436648163, 0.3960783763655594, 0.40271276235580444, 0.41008569433220793, 0.4378931889576571, 0.4515726167176451, 0.4533238046403442, 0.47645155686352936, 0.4662916042975017, 0.510246032582862, 0.5238301378807851, 0.5026419793388673, 0.5330618867384536, 0.5426405378218208, 0.5408799587083715, 0.5437736636293786, 0.5756550290222678, 0.5835354695362704]\n",
            "La perdida de los datos de validacion aumenta en:\n",
            "Epoch: 04 | Val. Loss: 0.396078 | Val. f1: 0.738015\n",
            "Epoch: 05 | Val. Loss: 0.402713 | Val. f1: 0.744701\n",
            "Cantidad de epocas a utilizar: 04\n"
          ]
        }
      ],
      "source": [
        "# RNN con GRU\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import torch.optim as optim\n",
        "\n",
        "class NER_RNN_GRU(nn.Module):\n",
        "    def __init__(self, \n",
        "                 input_dim, \n",
        "                 embedding_dim, \n",
        "                 hidden_dim, \n",
        "                 output_dim,\n",
        "                 n_layers, \n",
        "                 bidirectional, \n",
        "                 dropout, \n",
        "                 pad_idx):\n",
        "\n",
        "        super().__init__()\n",
        "\n",
        "        # Capa de embedding\n",
        "        self.embedding = nn.Embedding(input_dim,\n",
        "                                      embedding_dim,\n",
        "                                      padding_idx=pad_idx,\n",
        "                                      )\n",
        "\n",
        "        # Capa LSTM\n",
        "        #self.lstm = nn.LSTM(embedding_dim,\n",
        "                           #hidden_dim,\n",
        "                           #num_layers=n_layers,\n",
        "                           #bidirectional=bidirectional, \n",
        "                           #dropout = dropout if n_layers > 1 else 0)\n",
        "        \n",
        "        # Capa GRU\n",
        "        self.gru = nn.GRU(embedding_dim,\n",
        "                           hidden_dim,\n",
        "                           num_layers=n_layers,\n",
        "                           bidirectional=bidirectional, \n",
        "                           dropout = dropout if n_layers > 1 else 0)\n",
        "\n",
        "        # Capa de salida\n",
        "        self.fc = nn.Linear(hidden_dim * 2 if bidirectional else hidden_dim,\n",
        "                            output_dim)\n",
        "\n",
        "        # Dropout\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "\n",
        "    def forward(self, text):\n",
        "\n",
        "        #text = [sent len, batch size]\n",
        "\n",
        "        # Convertir lo enviado a embedding\n",
        "        embedded = self.dropout(self.embedding(text))\n",
        "        \n",
        "        outputs, hidden = self.gru(embedded)\n",
        "        #embedded = [sent len, batch size, emb dim]\n",
        "\n",
        "        # Pasar los embeddings por la rnn (LSTM)\n",
        "\n",
        "        #output = [sent len, batch size, hid dim * n directions]\n",
        "        #hidden/cell = [n layers * n directions, batch size, hid dim]\n",
        "\n",
        "        # Predecir usando la capa de salida.\n",
        "        predictions = self.fc(self.dropout(outputs))\n",
        "        #predictions = [sent len, batch size, output dim]\n",
        "\n",
        "        return predictions\n",
        "\n",
        "import time\n",
        "\n",
        "criterion = baseline_criterion\n",
        "n_epochs = 20\n",
        "\n",
        "\n",
        "def epoch_time(start_time, end_time):\n",
        "    elapsed_time = end_time - start_time\n",
        "    elapsed_mins = int(elapsed_time / 60)\n",
        "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
        "    return elapsed_mins, elapsed_secs\n",
        "\n",
        "def evaluate(model, iterator, criterion):\n",
        "\n",
        "    epoch_loss = 0\n",
        "    epoch_precision = 0\n",
        "    epoch_recall = 0\n",
        "    epoch_f1 = 0\n",
        "\n",
        "    model.eval()\n",
        "\n",
        "    # Indicamos que ahora no guardaremos los gradientes\n",
        "    with torch.no_grad():\n",
        "        # Por cada batch\n",
        "        for batch in iterator:\n",
        "\n",
        "            text = batch.text\n",
        "            tags = batch.nertags\n",
        "\n",
        "            # Predecimos\n",
        "            predictions = model(text)\n",
        "\n",
        "            predictions = predictions.view(-1, predictions.shape[-1])\n",
        "            tags = tags.view(-1)\n",
        "\n",
        "            # Calculamos el Cross Entropy de las predicciones con respecto a las etiquetas reales\n",
        "            loss = criterion(predictions, tags)\n",
        "\n",
        "            # Calculamos las métricas\n",
        "            precision, recall, f1 = calculate_metrics(predictions, tags)\n",
        "\n",
        "            # Actualizamos el loss y las métricas\n",
        "            epoch_loss += loss.item()\n",
        "            epoch_precision += precision\n",
        "            epoch_recall += recall\n",
        "            epoch_f1 += f1\n",
        "\n",
        "    return epoch_loss / len(iterator), epoch_precision / len(\n",
        "        iterator), epoch_recall / len(iterator), epoch_f1 / len(iterator)\n",
        "\n",
        "def train(model, iterator, optimizer, criterion):\n",
        "\n",
        "    epoch_loss = 0\n",
        "    epoch_precision = 0\n",
        "    epoch_recall = 0\n",
        "    epoch_f1 = 0\n",
        "\n",
        "    model.train()\n",
        "\n",
        "    # Por cada batch del iterador de la época:\n",
        "    for batch in iterator:\n",
        "\n",
        "        # Extraemos el texto y los tags del batch que estamos procesado\n",
        "        text = batch.text\n",
        "        tags = batch.nertags\n",
        "\n",
        "        # Reiniciamos los gradientes calculados en la iteración anterior\n",
        "        optimizer.zero_grad()\n",
        "\n",
        "        #text = [sent len, batch size]\n",
        "\n",
        "        # Predecimos los tags del texto del batch.\n",
        "        predictions = model(text)\n",
        "\n",
        "        #predictions = [sent len, batch size, output dim]\n",
        "        #tags = [sent len, batch size]\n",
        "\n",
        "        # Reordenamos los datos para calcular la loss\n",
        "        predictions = predictions.view(-1, predictions.shape[-1])\n",
        "        tags = tags.view(-1)\n",
        "\n",
        "        #predictions = [sent len * batch size, output dim]\n",
        "\n",
        "\n",
        "\n",
        "        # Calculamos el Cross Entropy de las predicciones con respecto a las etiquetas reales\n",
        "        loss = criterion(predictions, tags)\n",
        "        \n",
        "        # Calculamos el accuracy\n",
        "        precision, recall, f1 = calculate_metrics(predictions, tags)\n",
        "\n",
        "        # Calculamos los gradientes\n",
        "        loss.backward()\n",
        "\n",
        "        # Actualizamos los parámetros de la red\n",
        "        optimizer.step()\n",
        "\n",
        "        # Actualizamos el loss y las métricas\n",
        "        epoch_loss += loss.item()\n",
        "        epoch_precision += precision\n",
        "        epoch_recall += recall\n",
        "        epoch_f1 += f1\n",
        "\n",
        "    return epoch_loss / len(iterator), epoch_precision / len(\n",
        "        iterator), epoch_recall / len(iterator), epoch_f1 / len(iterator)\n",
        "\n",
        "def init_weights(m):\n",
        "    # Inicializamos los pesos como aleatorios\n",
        "    for name, param in m.named_parameters():\n",
        "        nn.init.normal_(param.data, mean=0, std=0.1) \n",
        "        \n",
        "    # Seteamos como 0 los embeddings de UNK y PAD.\n",
        "    model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)\n",
        "    model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)\n",
        "\n",
        "def experimento_1(modelo):   \n",
        "  # Entrenamos la red\n",
        "  best_valid_loss = float('inf')\n",
        "\n",
        "  val_loss = []\n",
        "  val_loss_f1 = []\n",
        "  for epoch in range(n_epochs):\n",
        "      start_time = time.time()\n",
        "\n",
        "\n",
        "      # Entrenar\n",
        "      train_loss, train_precision, train_recall, train_f1 = train(\n",
        "          model, train_iterator, optimizer, criterion)\n",
        "\n",
        "      # Evaluar (valid = validación)\n",
        "      valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "          model, valid_iterator, criterion)\n",
        "      \n",
        "      end_time = time.time()\n",
        "\n",
        "      epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
        "\n",
        "      # Si obtuvimos mejores resultados, guardamos este modelo en el almacenamiento (para poder cargarlo luego)\n",
        "      # Si detienen el entrenamiento prematuramente, pueden cargar el modelo en el siguiente recuadro de código.\n",
        "      if valid_loss < best_valid_loss:\n",
        "          best_valid_loss = valid_loss\n",
        "          torch.save(model.state_dict(), '{}.pt'.format(model_name))\n",
        "      # Si ya no mejoramos el loss de validación, terminamos de entrenar.\n",
        "\n",
        "      val_loss.append(valid_loss)\n",
        "      val_loss_f1.append(valid_f1)\n",
        "\n",
        "\n",
        "  epoch_select_1 = 0\n",
        "  epoch_select_2 = 0\n",
        "  val_f1_select_1 = 0\n",
        "  val_f1_select_2 = 0\n",
        "  val_loss_select_1 = 0\n",
        "  val_loss_select_2 = 0\n",
        "\n",
        "  # Revisar vectores\n",
        "  for i in range(len(val_loss)-1):\n",
        "    if (val_loss[i+1] > val_loss[i]):\n",
        "      val_loss_select_1 = val_loss[i]\n",
        "      val_loss_select_2 = val_loss[i+1]\n",
        "      val_f1_select_1 = val_loss_f1[i]\n",
        "      val_f1_select_2 = val_loss_f1[i+1]\n",
        "      epoch_select_1 = i\n",
        "      epoch_select_2 = i+1\n",
        "      break\n",
        "\n",
        "  # Imprimir resultados\n",
        "  print(\"MODELO \", modelo)\n",
        "  print(model_name)\n",
        "  print(val_loss)\n",
        "  print(\"La perdida de los datos de validacion aumenta en:\")\n",
        "  print(f'Epoch: {epoch_select_1+1:02} | Val. Loss: {val_loss_select_1:3f} | Val. f1: {val_f1_select_1:2f}')\n",
        "  print(f'Epoch: {epoch_select_2+1:02} | Val. Loss: {val_loss_select_2:3f} | Val. f1: {val_f1_select_2:2f}')\n",
        "  print(f'Cantidad de epocas a utilizar: {epoch_select_1+1:02}')\n",
        "\n",
        "# Modelo LSTM\n",
        "# Creamos nuestro modelo.\n",
        "model = NER_RNN(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM,\n",
        "                         N_LAYERS, BIDIRECTIONAL, DROPOUT, PAD_IDX)\n",
        "model_name = 'baseline' \n",
        "model.apply(init_weights)\n",
        "model = model.to(device)\n",
        "# Optimizador\n",
        "optimizer = optim.Adam(model.parameters())\n",
        "criterion = criterion.to(device)\n",
        "experimento_1(\"LSTM\")\n",
        "print(\"\")\n",
        "print(\"------------------------------------------------\")\n",
        "print(\"\")\n",
        "\n",
        "# Modelo GRU\n",
        "model = NER_RNN_GRU(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM,\n",
        "                         N_LAYERS, BIDIRECTIONAL, DROPOUT, PAD_IDX)\n",
        "model_name = 'GRU_baseline_2'\n",
        "model.apply(init_weights)\n",
        "model = model.to(device)\n",
        "# Optimizador\n",
        "optimizer = optim.Adam(model.parameters())\n",
        "criterion = criterion.to(device)\n",
        "experimento_1(\"GRU\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Del experimento queda claro que GRU la cantidad de epocas a utilizar es 4 a 5 epocas."
      ],
      "metadata": {
        "id": "rzHzY0cJ96YY"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j8knuPaeNNnn"
      },
      "source": [
        "---------------\n",
        "\n",
        "### Modelo 5: RNN con embeddings pre-entrenados en contexto clínico"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!wget https://zenodo.org/record/3924799/files/cwlce.vec -nc"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ggrpzg7yOAQp",
        "outputId": "07bd2284-e803-4d9f-d116-077dfc476643"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2022-06-29 21:05:19--  https://zenodo.org/record/3924799/files/cwlce.vec\n",
            "Resolving zenodo.org (zenodo.org)... 137.138.76.77\n",
            "Connecting to zenodo.org (zenodo.org)|137.138.76.77|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 163303193 (156M) [application/octet-stream]\n",
            "Saving to: ‘cwlce.vec’\n",
            "\n",
            "cwlce.vec           100%[===================>] 155.74M  20.9MB/s    in 7.1s    \n",
            "\n",
            "2022-06-29 21:05:27 (21.9 MB/s) - ‘cwlce.vec’ saved [163303193/163303193]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MA4snIk8Ne88"
      },
      "outputs": [],
      "source": [
        "# Primer Field: TEXT. Representan los tokens de la secuencia\n",
        "# Esta vez se convierten a minusculas los caracteres\n",
        "# esto para facilitar la correspondencia de palabras al vector de embedding\n",
        "# independiente del uso de mayusculas.\n",
        "TEXT2 = legacy.data.Field(lower=True) \n",
        "\n",
        "# Segundo Field: NER_TAGS. Representan los Tags asociados a cada palabra.\n",
        "NER_TAGS2 = legacy.data.Field(unk_token=None)\n",
        "fields2 = ((\"text\", TEXT2), (\"nertags\", NER_TAGS2))"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K-d_OkImWIQu"
      },
      "source": [
        "train_data, valid_data, test_data = legacy.datasets.SequenceTaggingDataset.splits(\n",
        "    path=\"./\",\n",
        "    train=\"train.txt\",\n",
        "    validation=\"dev.txt\",\n",
        "    test=\"test.txt\",\n",
        "    fields=fields2,\n",
        "    encoding=\"utf-8\",\n",
        "    separator=\" \"\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HVGowc0DNvPS"
      },
      "outputs": [],
      "source": [
        "TEXT2.build_vocab(train_data)\n",
        "NER_TAGS2.build_vocab(train_data)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "scrolled": true,
        "outputId": "07cfd15e-3d78-41e4-bb0d-3c969468e43d",
        "id": "EnT-yhHKSapZ"
      },
      "source": [
        "print(f\"Tokens únicos en TEXT: {len(TEXT2.vocab)}\")\n",
        "print(f\"Tokens únicos en NER_TAGS: {len(TEXT.vocab)}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Tokens únicos en TEXT: 12772\n",
            "Tokens únicos en NER_TAGS: 17591\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install unidecode"
      ],
      "metadata": {
        "id": "67xMoxOoSL-4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "83d9a34c-0457-47ca-b14a-62aaf68b51fb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: unidecode in /usr/local/lib/python3.7/dist-packages (1.3.4)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import unidecode\n",
        "from torchtext import vocab\n",
        "#Se remueven los acentos de los tokens del vocabulario\n",
        "TEXT2.vocab.itos = [unidecode.unidecode(x) for x in TEXT2.vocab.itos]"
      ],
      "metadata": {
        "id": "NqDJ-QunhWcd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "popi = vocab.Vectors('cwlce.vec') #vectores entrenados en diagnosticos de lista de espera.\n",
        "taba = popi.get_vecs_by_tokens(TEXT2.vocab.itos) #se hace un match entre las palabras del vocabulario que estén presentes en los embeddings"
      ],
      "metadata": {
        "id": "p1SRIqwshwF7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Rqnm0M1ROg0Q"
      },
      "outputs": [],
      "source": [
        "# tamaño del vocabulario. recuerden que la entrada son vectores bag of word(one-hot).\n",
        "INPUT_DIM = len(TEXT.vocab)\n",
        "EMBEDDING_DIM = popi.dim  # dimensión de los embeddings.\n",
        "HIDDEN_DIM = 128  # dimensión de la capas LSTM\n",
        "OUTPUT_DIM = len(NER_TAGS.vocab)  # número de clases\n",
        "\n",
        "N_LAYERS = 3  # número de capas.\n",
        "DROPOUT = 0.5\n",
        "BIDIRECTIONAL = False\n",
        "\n",
        "# Creamos nuestro modelo.\n",
        "emb_model = NER_RNN(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM,\n",
        "                         N_LAYERS, BIDIRECTIONAL, DROPOUT, PAD_IDX)\n",
        "\n",
        "emb_model_name = 'embeddingsClinicos'  # nombre que tendrá el modelo guardado..."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Comparación de resultados entre Baseline con embeddings aleatorios y pre-entrenados"
      ],
      "metadata": {
        "id": "Oo7QFvsLzfFL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "models = [[baseline_model, baseline_model_name], [emb_model, emb_model_name]]\n",
        "results = []\n",
        "\n",
        "for a_model in models:\n",
        "\n",
        "    model = a_model[0]\n",
        "    model_name = a_model[1]\n",
        "    criterion = baseline_criterion\n",
        "    n_epochs = baseline_n_epochs\n",
        "\n",
        "    def init_weights(m):\n",
        "        # Inicializamos los pesos como aleatorios\n",
        "        if model_name == 'embeddingsClinicos':\n",
        "          model.embedding.from_pretrained(taba, False) \n",
        "        else: \n",
        "          for name, param in m.named_parameters():\n",
        "            nn.init.normal_(param.data, mean=0, std=0.1) \n",
        "            \n",
        "        # Seteamos como 0 los embeddings de UNK y PAD.\n",
        "        model.embedding.weight.data[UNK_IDX] = torch.zeros(model.embedding.embedding_dim)\n",
        "        model.embedding.weight.data[PAD_IDX] = torch.zeros(model.embedding.embedding_dim)\n",
        "            \n",
        "    model.apply(init_weights)\n",
        "\n",
        "    \n",
        "\n",
        "    def count_parameters(model):\n",
        "        return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
        "\n",
        "    # Optimizador\n",
        "    optimizer = optim.Adam(model.parameters())\n",
        "\n",
        "    # Enviamos el modelo y la loss a cuda (en el caso en que esté disponible)\n",
        "    model = model.to(device)\n",
        "    criterion = criterion.to(device)\n",
        "\n",
        "    def train(model, iterator, optimizer, criterion):\n",
        "\n",
        "        epoch_loss = 0\n",
        "        epoch_precision = 0\n",
        "        epoch_recall = 0\n",
        "        epoch_f1 = 0\n",
        "\n",
        "        model.train()\n",
        "\n",
        "        # Por cada batch del iterador de la época:\n",
        "        for batch in iterator:\n",
        "\n",
        "            # Extraemos el texto y los tags del batch que estamos procesado\n",
        "            text = batch.text\n",
        "            tags = batch.nertags\n",
        "\n",
        "            # Reiniciamos los gradientes calculados en la iteración anterior\n",
        "            optimizer.zero_grad()\n",
        "\n",
        "            #text = [sent len, batch size]\n",
        "\n",
        "            # Predecimos los tags del texto del batch.\n",
        "            predictions = model(text)\n",
        "\n",
        "            #predictions = [sent len, batch size, output dim]\n",
        "            #tags = [sent len, batch size]\n",
        "\n",
        "            # Reordenamos los datos para calcular la loss\n",
        "            predictions = predictions.view(-1, predictions.shape[-1])\n",
        "            tags = tags.view(-1)\n",
        "\n",
        "            #predictions = [sent len * batch size, output dim]\n",
        "\n",
        "\n",
        "\n",
        "            # Calculamos el Cross Entropy de las predicciones con respecto a las etiquetas reales\n",
        "            loss = criterion(predictions, tags)\n",
        "            \n",
        "            # Calculamos el accuracy\n",
        "            precision, recall, f1 = calculate_metrics(predictions, tags)\n",
        "\n",
        "            # Calculamos los gradientes\n",
        "            loss.backward()\n",
        "\n",
        "            # Actualizamos los parámetros de la red\n",
        "            optimizer.step()\n",
        "\n",
        "            # Actualizamos el loss y las métricas\n",
        "            epoch_loss += loss.item()\n",
        "            epoch_precision += precision\n",
        "            epoch_recall += recall\n",
        "            epoch_f1 += f1\n",
        "\n",
        "        return epoch_loss / len(iterator), epoch_precision / len(\n",
        "            iterator), epoch_recall / len(iterator), epoch_f1 / len(iterator)\n",
        "\n",
        "    def evaluate(model, iterator, criterion):\n",
        "\n",
        "        epoch_loss = 0\n",
        "        epoch_precision = 0\n",
        "        epoch_recall = 0\n",
        "        epoch_f1 = 0\n",
        "\n",
        "        model.eval()\n",
        "\n",
        "        # Indicamos que ahora no guardaremos los gradientes\n",
        "        with torch.no_grad():\n",
        "            # Por cada batch\n",
        "            for batch in iterator:\n",
        "\n",
        "                text = batch.text\n",
        "                tags = batch.nertags\n",
        "\n",
        "                # Predecimos\n",
        "                predictions = model(text)\n",
        "\n",
        "                predictions = predictions.view(-1, predictions.shape[-1])\n",
        "                tags = tags.view(-1)\n",
        "\n",
        "                # Calculamos el Cross Entropy de las predicciones con respecto a las etiquetas reales\n",
        "                loss = criterion(predictions, tags)\n",
        "\n",
        "                # Calculamos las métricas\n",
        "                precision, recall, f1 = calculate_metrics(predictions, tags)\n",
        "\n",
        "                # Actualizamos el loss y las métricas\n",
        "                epoch_loss += loss.item()\n",
        "                epoch_precision += precision\n",
        "                epoch_recall += recall\n",
        "                epoch_f1 += f1\n",
        "\n",
        "        return epoch_loss / len(iterator), epoch_precision / len(\n",
        "            iterator), epoch_recall / len(iterator), epoch_f1 / len(iterator)\n",
        "\n",
        "    import time\n",
        "\n",
        "    def epoch_time(start_time, end_time):\n",
        "        elapsed_time = end_time - start_time\n",
        "        elapsed_mins = int(elapsed_time / 60)\n",
        "        elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
        "        return elapsed_mins, elapsed_secs\n",
        "\n",
        "    best_valid_loss = float('inf')\n",
        "    best_loss_f1 = -1\n",
        "    best_loss_epoch = -1\n",
        "\n",
        "    for epoch in range(n_epochs):\n",
        "\n",
        "        start_time = time.time()\n",
        "\n",
        "        # Recuerdo: train_iterator y valid_iterator contienen el dataset dividido en batches.\n",
        "\n",
        "        # Entrenar\n",
        "        train_loss, train_precision, train_recall, train_f1 = train(\n",
        "            model, train_iterator, optimizer, criterion)\n",
        "\n",
        "        # Evaluar (valid = validación)\n",
        "        valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "            model, valid_iterator, criterion)\n",
        "\n",
        "        end_time = time.time()\n",
        "\n",
        "        epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
        "\n",
        "        # Si obtuvimos mejores resultados, guardamos este modelo en el almacenamiento (para poder cargarlo luego)\n",
        "        # Si detienen el entrenamiento prematuramente, pueden cargar el modelo en el siguiente recuadro de código.\n",
        "        if valid_loss < best_valid_loss:\n",
        "            best_valid_loss = valid_loss\n",
        "            best_loss_f1 = valid_f1\n",
        "            best_loss_epoch = epoch\n",
        "            torch.save(model.state_dict(), '{}.pt'.format(model_name))\n",
        "        # Si ya no mejoramos el loss de validación, terminamos de entrenar.\n",
        "\n",
        "    results.append({'model': model_name, 'epoch': best_loss_epoch,'loss': best_valid_loss, 'f1': best_loss_f1})\n",
        "\n",
        "print(results)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SNpOgp_YPOsT",
        "outputId": "628989b6-90f3-4e21-ad8c-ef00af9977ae"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[{'model': 'baseline', 'epoch': 6, 'loss': 0.3918491043150425, 'f1': 0.7400055549437184}, {'model': 'embeddingsClinicos', 'epoch': 9, 'loss': 0.3701882220006415, 'f1': 0.7521633634207311}]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "---------------\n",
        "\n",
        "\n",
        "### Modelo 6: Variando el numero de neuronas y capas de la RNN"
      ],
      "metadata": {
        "id": "-TQYxdDhufMO"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Modelo 7: Experimentando con LSTM"
      ],
      "metadata": {
        "id": "x__bz7dgdIMS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##### Probando el número de neuronas en la capa oculta"
      ],
      "metadata": {
        "id": "Y6gJVmgblXdM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Creamos nuestro modelo.\n",
        "validation_values = []\n",
        "\n",
        "for i in range(10):\n",
        "  HIDDEN_DIM = 2**i  # dimensión de la capas LSTM\n",
        "  model = NER_RNN(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM,\n",
        "                          N_LAYERS, BIDIRECTIONAL, DROPOUT, PAD_IDX)\n",
        "\n",
        "  baseline_model_name = 'baseline'  # nombre que tendrá el modelo guardado...\n",
        "\n",
        "  model_name = baseline_model_name\n",
        "  criterion = baseline_criterion\n",
        "  n_epochs = baseline_n_epochs\n",
        "\n",
        "  model.apply(init_weights)\n",
        "  print(f'El modelo actual tiene {count_parameters(model):,} parámetros entrenables.')\n",
        "  # Optimizador\n",
        "  optimizer = optim.Adam(model.parameters())\n",
        "\n",
        "  # Enviamos el modelo y la loss a cuda (en el caso en que esté disponible)\n",
        "  model = model.to(device)\n",
        "  criterion = criterion.to(device)\n",
        "\n",
        "\n",
        "  best_valid_loss = float('inf')\n",
        "\n",
        "  for epoch in range(n_epochs):\n",
        "\n",
        "      start_time = time.time()\n",
        "\n",
        "      # Recuerdo: train_iterator y valid_iterator contienen el dataset dividido en batches.\n",
        "\n",
        "      # Entrenar\n",
        "      train_loss, train_precision, train_recall, train_f1 = train(\n",
        "          model, train_iterator, optimizer, criterion)\n",
        "\n",
        "      # Evaluar (valid = validación)\n",
        "      valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "          model, valid_iterator, criterion)\n",
        "\n",
        "      end_time = time.time()\n",
        "\n",
        "      epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
        "\n",
        "      # Si obtuvimos mejores resultados, guardamos este modelo en el almacenamiento (para poder cargarlo luego)\n",
        "      # Si detienen el entrenamiento prematuramente, pueden cargar el modelo en el siguiente recuadro de código.\n",
        "      if valid_loss < best_valid_loss:\n",
        "          best_valid_loss = valid_loss\n",
        "          torch.save(model.state_dict(), '{}.pt'.format(model_name))\n",
        "      # Si ya no mejoramos el loss de validación, terminamos de entrenar.\n",
        "\n",
        "      print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
        "      print(\n",
        "          f'\\tTrain Loss: {train_loss:.3f} | Train f1: {train_f1:.2f} | Train precision: {train_precision:.2f} | Train recall: {train_recall:.2f}'\n",
        "      )\n",
        "      print(\n",
        "          f'\\t Val. Loss: {valid_loss:.3f} |  Val. f1: {valid_f1:.2f} |  Val. precision: {valid_precision:.2f} | Val. recall: {valid_recall:.2f}'\n",
        "      )\n",
        "  # cargar el mejor modelo entrenado.\n",
        "  model.load_state_dict(torch.load('{}.pt'.format(model_name)))\n",
        "\n",
        "  # Limpiar ram de cuda\n",
        "  torch.cuda.empty_cache()\n",
        "\n",
        "  valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "      model, valid_iterator, criterion)\n",
        "\n",
        "  validation_values += [valid_loss]\n",
        "\n",
        "  print(\n",
        "      f'Val. Loss: {valid_loss:.3f} |  Val. f1: {valid_f1:.2f} | Val. precision: {valid_precision:.2f} | Val. recall: {valid_recall:.2f}'\n",
        "  )"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pRpV84hlP4ww",
        "outputId": "7a1de26b-1e61-4b1b-e2a8-9180f6f4a17f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "El modelo actual tiene 5,279,840 parámetros entrenables.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch: 01 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 2.005 | Train f1: 0.05 | Train precision: 0.06 | Train recall: 0.05\n",
            "\t Val. Loss: 1.423 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.528 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.240 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 03 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.421 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.191 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 04 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.319 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.122 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 05 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.229 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.062 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 06 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.169 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.016 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 07 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.123 | Train f1: 0.01 | Train precision: 0.15 | Train recall: 0.00\n",
            "\t Val. Loss: 0.979 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 08 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.085 | Train f1: 0.04 | Train precision: 0.38 | Train recall: 0.02\n",
            "\t Val. Loss: 0.944 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 09 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.056 | Train f1: 0.15 | Train precision: 0.41 | Train recall: 0.09\n",
            "\t Val. Loss: 0.915 |  Val. f1: 0.29 |  Val. precision: 0.55 | Val. recall: 0.21\n",
            "Epoch: 10 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.028 | Train f1: 0.21 | Train precision: 0.42 | Train recall: 0.14\n",
            "\t Val. Loss: 0.891 |  Val. f1: 0.33 |  Val. precision: 0.56 | Val. recall: 0.24\n",
            "Val. Loss: 0.891 |  Val. f1: 0.33 | Val. precision: 0.56 | Val. recall: 0.24\n",
            "El modelo actual tiene 5,282,480 parámetros entrenables.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
            "  warnings.warn('{} seems not to be NE tag.'.format(chunk))\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch: 01 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.876 | Train f1: 0.01 | Train precision: 0.01 | Train recall: 0.01\n",
            "\t Val. Loss: 1.246 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.401 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.161 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 03 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.298 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.065 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 04 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.180 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 0.978 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 05 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.093 | Train f1: 0.00 | Train precision: 0.11 | Train recall: 0.00\n",
            "\t Val. Loss: 0.933 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 06 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.029 | Train f1: 0.09 | Train precision: 0.41 | Train recall: 0.05\n",
            "\t Val. Loss: 0.901 |  Val. f1: 0.31 |  Val. precision: 0.44 | Val. recall: 0.24\n",
            "Epoch: 07 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.979 | Train f1: 0.22 | Train precision: 0.41 | Train recall: 0.15\n",
            "\t Val. Loss: 0.875 |  Val. f1: 0.33 |  Val. precision: 0.40 | Val. recall: 0.28\n",
            "Epoch: 08 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.937 | Train f1: 0.27 | Train precision: 0.41 | Train recall: 0.20\n",
            "\t Val. Loss: 0.853 |  Val. f1: 0.32 |  Val. precision: 0.38 | Val. recall: 0.28\n",
            "Epoch: 09 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.905 | Train f1: 0.31 | Train precision: 0.43 | Train recall: 0.24\n",
            "\t Val. Loss: 0.824 |  Val. f1: 0.33 |  Val. precision: 0.40 | Val. recall: 0.29\n",
            "Epoch: 10 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.869 | Train f1: 0.32 | Train precision: 0.44 | Train recall: 0.25\n",
            "\t Val. Loss: 0.803 |  Val. f1: 0.34 |  Val. precision: 0.41 | Val. recall: 0.29\n",
            "Val. Loss: 0.803 |  Val. f1: 0.34 | Val. precision: 0.41 | Val. recall: 0.29\n",
            "El modelo actual tiene 5,288,096 parámetros entrenables.\n",
            "Epoch: 01 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.532 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.137 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.185 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 0.990 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 03 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.036 | Train f1: 0.01 | Train precision: 0.12 | Train recall: 0.00\n",
            "\t Val. Loss: 0.887 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 04 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.939 | Train f1: 0.21 | Train precision: 0.40 | Train recall: 0.15\n",
            "\t Val. Loss: 0.840 |  Val. f1: 0.32 |  Val. precision: 0.40 | Val. recall: 0.27\n",
            "Epoch: 05 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.874 | Train f1: 0.31 | Train precision: 0.39 | Train recall: 0.26\n",
            "\t Val. Loss: 0.811 |  Val. f1: 0.33 |  Val. precision: 0.39 | Val. recall: 0.29\n",
            "Epoch: 06 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.820 | Train f1: 0.34 | Train precision: 0.41 | Train recall: 0.29\n",
            "\t Val. Loss: 0.785 |  Val. f1: 0.34 |  Val. precision: 0.39 | Val. recall: 0.29\n",
            "Epoch: 07 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.772 | Train f1: 0.35 | Train precision: 0.44 | Train recall: 0.30\n",
            "\t Val. Loss: 0.771 |  Val. f1: 0.34 |  Val. precision: 0.40 | Val. recall: 0.30\n",
            "Epoch: 08 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.737 | Train f1: 0.38 | Train precision: 0.48 | Train recall: 0.32\n",
            "\t Val. Loss: 0.754 |  Val. f1: 0.35 |  Val. precision: 0.42 | Val. recall: 0.31\n",
            "Epoch: 09 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.712 | Train f1: 0.40 | Train precision: 0.49 | Train recall: 0.33\n",
            "\t Val. Loss: 0.741 |  Val. f1: 0.39 |  Val. precision: 0.45 | Val. recall: 0.34\n",
            "Epoch: 10 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 0.687 | Train f1: 0.41 | Train precision: 0.51 | Train recall: 0.35\n",
            "\t Val. Loss: 0.731 |  Val. f1: 0.40 |  Val. precision: 0.46 | Val. recall: 0.36\n",
            "Val. Loss: 0.731 |  Val. f1: 0.40 | Val. precision: 0.46 | Val. recall: 0.36\n",
            "El modelo actual tiene 5,300,672 parámetros entrenables.\n",
            "Epoch: 01 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 1.422 | Train f1: 0.01 | Train precision: 0.02 | Train recall: 0.01\n",
            "\t Val. Loss: 1.051 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 1.013 | Train f1: 0.09 | Train precision: 0.30 | Train recall: 0.06\n",
            "\t Val. Loss: 0.841 |  Val. f1: 0.29 |  Val. precision: 0.44 | Val. recall: 0.22\n",
            "Epoch: 03 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.869 | Train f1: 0.28 | Train precision: 0.40 | Train recall: 0.22\n",
            "\t Val. Loss: 0.774 |  Val. f1: 0.34 |  Val. precision: 0.44 | Val. recall: 0.28\n",
            "Epoch: 04 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.782 | Train f1: 0.34 | Train precision: 0.44 | Train recall: 0.28\n",
            "\t Val. Loss: 0.731 |  Val. f1: 0.37 |  Val. precision: 0.43 | Val. recall: 0.33\n",
            "Epoch: 05 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.725 | Train f1: 0.39 | Train precision: 0.47 | Train recall: 0.34\n",
            "\t Val. Loss: 0.698 |  Val. f1: 0.41 |  Val. precision: 0.46 | Val. recall: 0.37\n",
            "Epoch: 06 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.675 | Train f1: 0.42 | Train precision: 0.49 | Train recall: 0.37\n",
            "\t Val. Loss: 0.670 |  Val. f1: 0.43 |  Val. precision: 0.47 | Val. recall: 0.39\n",
            "Epoch: 07 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.632 | Train f1: 0.44 | Train precision: 0.51 | Train recall: 0.39\n",
            "\t Val. Loss: 0.654 |  Val. f1: 0.44 |  Val. precision: 0.50 | Val. recall: 0.40\n",
            "Epoch: 08 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.597 | Train f1: 0.46 | Train precision: 0.53 | Train recall: 0.41\n",
            "\t Val. Loss: 0.640 |  Val. f1: 0.44 |  Val. precision: 0.48 | Val. recall: 0.40\n",
            "Epoch: 09 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.568 | Train f1: 0.48 | Train precision: 0.54 | Train recall: 0.42\n",
            "\t Val. Loss: 0.636 |  Val. f1: 0.45 |  Val. precision: 0.51 | Val. recall: 0.41\n",
            "Epoch: 10 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.549 | Train f1: 0.49 | Train precision: 0.56 | Train recall: 0.44\n",
            "\t Val. Loss: 0.641 |  Val. f1: 0.45 |  Val. precision: 0.49 | Val. recall: 0.41\n",
            "Val. Loss: 0.636 |  Val. f1: 0.45 | Val. precision: 0.51 | Val. recall: 0.41\n",
            "El modelo actual tiene 5,331,200 parámetros entrenables.\n",
            "Epoch: 01 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.301 | Train f1: 0.04 | Train precision: 0.17 | Train recall: 0.03\n",
            "\t Val. Loss: 0.896 |  Val. f1: 0.12 |  Val. precision: 0.49 | Val. recall: 0.07\n",
            "Epoch: 02 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.827 | Train f1: 0.33 | Train precision: 0.48 | Train recall: 0.26\n",
            "\t Val. Loss: 0.668 |  Val. f1: 0.48 |  Val. precision: 0.64 | Val. recall: 0.39\n",
            "Epoch: 03 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.644 | Train f1: 0.53 | Train precision: 0.62 | Train recall: 0.47\n",
            "\t Val. Loss: 0.572 |  Val. f1: 0.60 |  Val. precision: 0.65 | Val. recall: 0.56\n",
            "Epoch: 04 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 0.538 | Train f1: 0.62 | Train precision: 0.68 | Train recall: 0.58\n",
            "\t Val. Loss: 0.528 |  Val. f1: 0.63 |  Val. precision: 0.69 | Val. recall: 0.59\n",
            "Epoch: 05 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.471 | Train f1: 0.67 | Train precision: 0.72 | Train recall: 0.64\n",
            "\t Val. Loss: 0.501 |  Val. f1: 0.65 |  Val. precision: 0.69 | Val. recall: 0.63\n",
            "Epoch: 06 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.427 | Train f1: 0.70 | Train precision: 0.74 | Train recall: 0.67\n",
            "\t Val. Loss: 0.497 |  Val. f1: 0.68 |  Val. precision: 0.71 | Val. recall: 0.66\n",
            "Epoch: 07 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.387 | Train f1: 0.73 | Train precision: 0.76 | Train recall: 0.70\n",
            "\t Val. Loss: 0.489 |  Val. f1: 0.70 |  Val. precision: 0.72 | Val. recall: 0.67\n",
            "Epoch: 08 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.356 | Train f1: 0.74 | Train precision: 0.77 | Train recall: 0.72\n",
            "\t Val. Loss: 0.492 |  Val. f1: 0.71 |  Val. precision: 0.72 | Val. recall: 0.69\n",
            "Epoch: 09 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.331 | Train f1: 0.76 | Train precision: 0.78 | Train recall: 0.74\n",
            "\t Val. Loss: 0.493 |  Val. f1: 0.71 |  Val. precision: 0.72 | Val. recall: 0.72\n",
            "Epoch: 10 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 0.312 | Train f1: 0.79 | Train precision: 0.80 | Train recall: 0.77\n",
            "\t Val. Loss: 0.490 |  Val. f1: 0.72 |  Val. precision: 0.72 | Val. recall: 0.72\n",
            "Val. Loss: 0.489 |  Val. f1: 0.70 | Val. precision: 0.72 | Val. recall: 0.67\n",
            "El modelo actual tiene 5,413,760 parámetros entrenables.\n",
            "Epoch: 01 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.102 | Train f1: 0.15 | Train precision: 0.29 | Train recall: 0.12\n",
            "\t Val. Loss: 0.700 |  Val. f1: 0.47 |  Val. precision: 0.68 | Val. recall: 0.36\n",
            "Epoch: 02 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.644 | Train f1: 0.56 | Train precision: 0.68 | Train recall: 0.49\n",
            "\t Val. Loss: 0.530 |  Val. f1: 0.63 |  Val. precision: 0.71 | Val. recall: 0.57\n",
            "Epoch: 03 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.489 | Train f1: 0.67 | Train precision: 0.74 | Train recall: 0.62\n",
            "\t Val. Loss: 0.472 |  Val. f1: 0.67 |  Val. precision: 0.74 | Val. recall: 0.61\n",
            "Epoch: 04 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.408 | Train f1: 0.72 | Train precision: 0.77 | Train recall: 0.68\n",
            "\t Val. Loss: 0.438 |  Val. f1: 0.71 |  Val. precision: 0.76 | Val. recall: 0.67\n",
            "Epoch: 05 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.344 | Train f1: 0.77 | Train precision: 0.79 | Train recall: 0.75\n",
            "\t Val. Loss: 0.430 |  Val. f1: 0.72 |  Val. precision: 0.74 | Val. recall: 0.71\n",
            "Epoch: 06 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.298 | Train f1: 0.80 | Train precision: 0.81 | Train recall: 0.79\n",
            "\t Val. Loss: 0.424 |  Val. f1: 0.74 |  Val. precision: 0.76 | Val. recall: 0.72\n",
            "Epoch: 07 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.265 | Train f1: 0.82 | Train precision: 0.83 | Train recall: 0.82\n",
            "\t Val. Loss: 0.406 |  Val. f1: 0.74 |  Val. precision: 0.74 | Val. recall: 0.75\n",
            "Epoch: 08 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.238 | Train f1: 0.84 | Train precision: 0.85 | Train recall: 0.84\n",
            "\t Val. Loss: 0.436 |  Val. f1: 0.74 |  Val. precision: 0.73 | Val. recall: 0.74\n",
            "Epoch: 09 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.215 | Train f1: 0.86 | Train precision: 0.86 | Train recall: 0.85\n",
            "\t Val. Loss: 0.445 |  Val. f1: 0.74 |  Val. precision: 0.74 | Val. recall: 0.75\n",
            "Epoch: 10 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.198 | Train f1: 0.87 | Train precision: 0.87 | Train recall: 0.87\n",
            "\t Val. Loss: 0.457 |  Val. f1: 0.74 |  Val. precision: 0.72 | Val. recall: 0.75\n",
            "Val. Loss: 0.406 |  Val. f1: 0.74 | Val. precision: 0.74 | Val. recall: 0.75\n",
            "El modelo actual tiene 5,664,896 parámetros entrenables.\n",
            "Epoch: 01 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.960 | Train f1: 0.30 | Train precision: 0.47 | Train recall: 0.24\n",
            "\t Val. Loss: 0.609 |  Val. f1: 0.56 |  Val. precision: 0.76 | Val. recall: 0.46\n",
            "Epoch: 02 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 0.550 | Train f1: 0.62 | Train precision: 0.72 | Train recall: 0.56\n",
            "\t Val. Loss: 0.442 |  Val. f1: 0.68 |  Val. precision: 0.76 | Val. recall: 0.62\n",
            "Epoch: 03 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.396 | Train f1: 0.72 | Train precision: 0.77 | Train recall: 0.69\n",
            "\t Val. Loss: 0.408 |  Val. f1: 0.73 |  Val. precision: 0.76 | Val. recall: 0.71\n",
            "Epoch: 04 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.315 | Train f1: 0.78 | Train precision: 0.80 | Train recall: 0.76\n",
            "\t Val. Loss: 0.383 |  Val. f1: 0.74 |  Val. precision: 0.77 | Val. recall: 0.73\n",
            "Epoch: 05 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.262 | Train f1: 0.81 | Train precision: 0.82 | Train recall: 0.81\n",
            "\t Val. Loss: 0.374 |  Val. f1: 0.75 |  Val. precision: 0.78 | Val. recall: 0.74\n",
            "Epoch: 06 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.223 | Train f1: 0.84 | Train precision: 0.85 | Train recall: 0.84\n",
            "\t Val. Loss: 0.387 |  Val. f1: 0.77 |  Val. precision: 0.77 | Val. recall: 0.76\n",
            "Epoch: 07 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.189 | Train f1: 0.87 | Train precision: 0.87 | Train recall: 0.87\n",
            "\t Val. Loss: 0.390 |  Val. f1: 0.76 |  Val. precision: 0.77 | Val. recall: 0.76\n",
            "Epoch: 08 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.168 | Train f1: 0.88 | Train precision: 0.88 | Train recall: 0.88\n",
            "\t Val. Loss: 0.407 |  Val. f1: 0.76 |  Val. precision: 0.78 | Val. recall: 0.75\n",
            "Epoch: 09 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.150 | Train f1: 0.90 | Train precision: 0.90 | Train recall: 0.90\n",
            "\t Val. Loss: 0.400 |  Val. f1: 0.77 |  Val. precision: 0.77 | Val. recall: 0.77\n",
            "Epoch: 10 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.135 | Train f1: 0.90 | Train precision: 0.90 | Train recall: 0.90\n",
            "\t Val. Loss: 0.433 |  Val. f1: 0.77 |  Val. precision: 0.76 | Val. recall: 0.78\n",
            "Val. Loss: 0.374 |  Val. f1: 0.75 | Val. precision: 0.78 | Val. recall: 0.74\n",
            "El modelo actual tiene 6,511,232 parámetros entrenables.\n",
            "Epoch: 01 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.878 | Train f1: 0.37 | Train precision: 0.55 | Train recall: 0.29\n",
            "\t Val. Loss: 0.571 |  Val. f1: 0.60 |  Val. precision: 0.75 | Val. recall: 0.51\n",
            "Epoch: 02 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.507 | Train f1: 0.66 | Train precision: 0.74 | Train recall: 0.59\n",
            "\t Val. Loss: 0.425 |  Val. f1: 0.70 |  Val. precision: 0.77 | Val. recall: 0.64\n",
            "Epoch: 03 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.358 | Train f1: 0.75 | Train precision: 0.79 | Train recall: 0.72\n",
            "\t Val. Loss: 0.368 |  Val. f1: 0.74 |  Val. precision: 0.78 | Val. recall: 0.71\n",
            "Epoch: 04 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.278 | Train f1: 0.80 | Train precision: 0.82 | Train recall: 0.79\n",
            "\t Val. Loss: 0.352 |  Val. f1: 0.77 |  Val. precision: 0.79 | Val. recall: 0.75\n",
            "Epoch: 05 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.225 | Train f1: 0.84 | Train precision: 0.85 | Train recall: 0.83\n",
            "\t Val. Loss: 0.348 |  Val. f1: 0.77 |  Val. precision: 0.79 | Val. recall: 0.75\n",
            "Epoch: 06 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.186 | Train f1: 0.87 | Train precision: 0.87 | Train recall: 0.86\n",
            "\t Val. Loss: 0.379 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.77\n",
            "Epoch: 07 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.161 | Train f1: 0.89 | Train precision: 0.89 | Train recall: 0.88\n",
            "\t Val. Loss: 0.378 |  Val. f1: 0.78 |  Val. precision: 0.79 | Val. recall: 0.77\n",
            "Epoch: 08 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.139 | Train f1: 0.90 | Train precision: 0.90 | Train recall: 0.90\n",
            "\t Val. Loss: 0.406 |  Val. f1: 0.77 |  Val. precision: 0.76 | Val. recall: 0.78\n",
            "Epoch: 09 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.124 | Train f1: 0.91 | Train precision: 0.91 | Train recall: 0.91\n",
            "\t Val. Loss: 0.414 |  Val. f1: 0.78 |  Val. precision: 0.78 | Val. recall: 0.78\n",
            "Epoch: 10 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.110 | Train f1: 0.92 | Train precision: 0.92 | Train recall: 0.92\n",
            "\t Val. Loss: 0.433 |  Val. f1: 0.78 |  Val. precision: 0.79 | Val. recall: 0.77\n",
            "Val. Loss: 0.348 |  Val. f1: 0.77 | Val. precision: 0.79 | Val. recall: 0.75\n",
            "El modelo actual tiene 9,580,160 parámetros entrenables.\n",
            "Epoch: 01 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.857 | Train f1: 0.38 | Train precision: 0.57 | Train recall: 0.30\n",
            "\t Val. Loss: 0.565 |  Val. f1: 0.59 |  Val. precision: 0.75 | Val. recall: 0.50\n",
            "Epoch: 02 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.496 | Train f1: 0.66 | Train precision: 0.74 | Train recall: 0.59\n",
            "\t Val. Loss: 0.419 |  Val. f1: 0.71 |  Val. precision: 0.74 | Val. recall: 0.68\n",
            "Epoch: 03 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.343 | Train f1: 0.75 | Train precision: 0.79 | Train recall: 0.72\n",
            "\t Val. Loss: 0.375 |  Val. f1: 0.75 |  Val. precision: 0.79 | Val. recall: 0.71\n",
            "Epoch: 04 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.265 | Train f1: 0.81 | Train precision: 0.83 | Train recall: 0.79\n",
            "\t Val. Loss: 0.363 |  Val. f1: 0.76 |  Val. precision: 0.76 | Val. recall: 0.75\n",
            "Epoch: 05 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.213 | Train f1: 0.85 | Train precision: 0.86 | Train recall: 0.84\n",
            "\t Val. Loss: 0.348 |  Val. f1: 0.76 |  Val. precision: 0.79 | Val. recall: 0.74\n",
            "Epoch: 06 | Epoch Time: 0m 13s\n",
            "\tTrain Loss: 0.175 | Train f1: 0.87 | Train precision: 0.88 | Train recall: 0.87\n",
            "\t Val. Loss: 0.368 |  Val. f1: 0.77 |  Val. precision: 0.77 | Val. recall: 0.77\n",
            "Epoch: 07 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.148 | Train f1: 0.89 | Train precision: 0.89 | Train recall: 0.89\n",
            "\t Val. Loss: 0.398 |  Val. f1: 0.76 |  Val. precision: 0.77 | Val. recall: 0.76\n",
            "Epoch: 08 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.127 | Train f1: 0.90 | Train precision: 0.91 | Train recall: 0.90\n",
            "\t Val. Loss: 0.430 |  Val. f1: 0.77 |  Val. precision: 0.77 | Val. recall: 0.78\n",
            "Epoch: 09 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.110 | Train f1: 0.92 | Train precision: 0.92 | Train recall: 0.92\n",
            "\t Val. Loss: 0.443 |  Val. f1: 0.77 |  Val. precision: 0.79 | Val. recall: 0.77\n",
            "Epoch: 10 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.098 | Train f1: 0.93 | Train precision: 0.93 | Train recall: 0.93\n",
            "\t Val. Loss: 0.433 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.77\n",
            "Val. Loss: 0.348 |  Val. f1: 0.76 | Val. precision: 0.79 | Val. recall: 0.74\n",
            "El modelo actual tiene 21,223,040 parámetros entrenables.\n",
            "Epoch: 01 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 1.064 | Train f1: 0.19 | Train precision: 0.41 | Train recall: 0.14\n",
            "\t Val. Loss: 0.696 |  Val. f1: 0.51 |  Val. precision: 0.72 | Val. recall: 0.40\n",
            "Epoch: 02 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.585 | Train f1: 0.59 | Train precision: 0.72 | Train recall: 0.51\n",
            "\t Val. Loss: 0.444 |  Val. f1: 0.67 |  Val. precision: 0.74 | Val. recall: 0.62\n",
            "Epoch: 03 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.386 | Train f1: 0.73 | Train precision: 0.78 | Train recall: 0.69\n",
            "\t Val. Loss: 0.398 |  Val. f1: 0.72 |  Val. precision: 0.73 | Val. recall: 0.70\n",
            "Epoch: 04 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.289 | Train f1: 0.79 | Train precision: 0.81 | Train recall: 0.77\n",
            "\t Val. Loss: 0.374 |  Val. f1: 0.74 |  Val. precision: 0.71 | Val. recall: 0.77\n",
            "Epoch: 05 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.226 | Train f1: 0.83 | Train precision: 0.84 | Train recall: 0.82\n",
            "\t Val. Loss: 0.367 |  Val. f1: 0.76 |  Val. precision: 0.75 | Val. recall: 0.77\n",
            "Epoch: 06 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.185 | Train f1: 0.86 | Train precision: 0.87 | Train recall: 0.86\n",
            "\t Val. Loss: 0.371 |  Val. f1: 0.77 |  Val. precision: 0.77 | Val. recall: 0.77\n",
            "Epoch: 07 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.155 | Train f1: 0.89 | Train precision: 0.89 | Train recall: 0.89\n",
            "\t Val. Loss: 0.372 |  Val. f1: 0.77 |  Val. precision: 0.76 | Val. recall: 0.79\n",
            "Epoch: 08 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.135 | Train f1: 0.90 | Train precision: 0.90 | Train recall: 0.90\n",
            "\t Val. Loss: 0.373 |  Val. f1: 0.77 |  Val. precision: 0.79 | Val. recall: 0.76\n",
            "Epoch: 09 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.116 | Train f1: 0.91 | Train precision: 0.92 | Train recall: 0.91\n",
            "\t Val. Loss: 0.405 |  Val. f1: 0.77 |  Val. precision: 0.77 | Val. recall: 0.77\n",
            "Epoch: 10 | Epoch Time: 0m 28s\n",
            "\tTrain Loss: 0.101 | Train f1: 0.92 | Train precision: 0.93 | Train recall: 0.93\n",
            "\t Val. Loss: 0.423 |  Val. f1: 0.77 |  Val. precision: 0.77 | Val. recall: 0.78\n",
            "Val. Loss: 0.367 |  Val. f1: 0.76 | Val. precision: 0.75 | Val. recall: 0.77\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "fig = plt.figure()\n",
        "ax = fig.add_subplot(1, 1, 1)\n",
        "ax.plot(validation_values)\n",
        "plt.xlabel('Logaritmo en base 2 del número de neuronas de la capa oculta')\n",
        "plt.ylabel('Error en el conjunto de validación')\n",
        "plt.title('Error en el conjunto de validación en función de el número de neuronas de la capa oculta')\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 296
        },
        "outputId": "f5502e73-b359-4e7c-b291-e5d924cd8c65",
        "id": "co2gLgoYdnBm"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAioAAAEXCAYAAACDPG9dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd5gddfn+8fe9JdmQCiQE0giBUNKBhdBBESlCQASkSZEiIiCWnwKCIlL8YpfeO1IElKaI9A4JENKAhFDSCSE9pO7z+2Nmzclydvdssps5m71f1zXXTp9n6j5n5jOfUURgZmZmVoxKsg7AzMzMrDZOVMzMzKxoOVExMzOzouVExczMzIqWExUzMzMrWk5UzMzMrGg5Uckh6UJJd67lZY6RtNfaXGYhJPWWFJLK1vJyP5L0tbT9PEk3FjLuGiwv7zIk7SrpdUnrr8n8G4MSt0iaLen1JlzOWtkWTXVsNdZ8Je0gaZyk9o0VW1OSdIKkF7OOozlak2t+Y1x/1jWSnpV0cmPPt94TWtJHQFdgRU7vWyPijMYOpiWKiP6NMZ802bkzIno0xvyKQURcmsUyJPUELgUOjIjZTR1DAXYD9gF6RMTCplpIM9kWTSpNcq4AjoqI+VnHY9ZcSToBODkidlvTeRX6y+OgiPhvfSNJKouI5TX6lUbEitqmyTOPBo1v1tgiYhKwZ9Zx5NgU+Kgpk5TaFOG2aGqbAxdHxNtra4H5rpvrmpawjtZ01ujRT3rL8SVJf5I0C7hQ0q2SrpH0uKSFwFckbZPeEpqTPuoYljOPL42fZzkdJd0kaZqkKZIullSaE8OLkn6f3hr/UNL+dcTcTdIDkmam457VgPU9WNLbkuZJ+kDSfjnzfFjS55ImSDolZ5oLJd0n6XZJ89P1r8wZnvuo41ZJF+cM20vS5Brj/lTSO5LmSrpXUoWktsC/gG6SFqRNN0mtJf1Z0tS0+bOk1rWsW2m6DT+TNBH4RqH7IM/2/ULSBjn9tk3nWy5pc0lPS5qV9rtLUqdaYlrltqyk70j6OJ32FzXG3VHSK+kxNk3SlZJa5QzvL+nJdB/NkHReLcsYlu6jOekxu0192z9f7On431XyCGG2pCckbZozLCSdJml8uqyrJCnPPE4CbgR2Tvfrr5XnVn86vy3S9lvT+T2WHnOvSdo8q23RWMdWOm6JpHOUnH+zlJxbG+QbN8+0tcZcvU0j4r2IeLSWbXq1pH+l++ElSRsrOadmS3pX0rY5y6r1OpNu579LulPSPOAE1XENybMeG6bjzlPyKHDzGsO3ztm/70k6oo55PSvpN+n6zJf0H0mdc4bvJOnl9BgYqZzH1Krx6CP3+NHKx3AnSfoEeDrdd+crOYc/VXJN7Fhj/OMlfZIeK7/ImXet57cSf0rnOU/SKEkDalnfzSQ9l67rk0DnGsNrXd+61BVfLePvlrOcSUruPiDpG5LeStdjkqQLc6ap3kanKrmeT5P00zWIoa7zu6ekB9Pjd5akK9P+Na8ReR+3pvO6lpXXrTn1rV+dIqLOBvgI+Fotw04AlgNnktydaQPcCswFdiVJhNoDE4DzgFbAV4H5wFbpPGqOX5FnOQ8B1wFtgY2A14Hv5cSwDDgFKAW+D0wFlGc+JcAI4JdpLH2AicC+6fALSR6f5FvXHdM490nn0x3YOh32PHA1UAEMAWYCX82Z52LggDS+y4BX823fdFtcnDNsL2ByjXFfB7oBGwDjgNPyjZv2uwh4Nd1mXYCXgd/Usn6nAe8CPdN5PwMEUFbfPsgzr6eBU3K6fwdcm7ZvkW7D1mlMzwN/rmV7/G9/AP2ABcAe6bR/JDn2qsfdHtiJ5DjsnW6bs9Nh7YFpwE/SfdQeGJpnGVsCC9P4yoGfkRy7rerb/nm2wcHptNukMZ0PvJwzPIBHgU5AL5JjZr86zrMXa+vOmd8WOcfRLJJjtgy4C7gnw23RmMfWD0mO6R7pcXAd8Ld0WO/c+dZyLavt/Clkm35GcpxVkBzjHwLHkZzXFwPPNOA6sww4JB23DXVcQ/Ksxz3Afen2GgBMqY497TcJODHd99umcferZV7PAh+k+7tN2v3bdFh3kuPogDTOfdLuLvn+N7Dq8VO9L25PY2oDfJfkGOoDtAMeBO6oMf4N6biDgSXANgWc3/um27sTIJJzbpNa1vcVkmtHa5JryfycmOtc31qOp3qvP3mm2zRd7lEk59aGwJCc6/jAdPmDgBnAITW20d/SbTowPU5WJ4Zaz2+S43kk8Kd0ORXAbjX3cb5zjuT4ObmOc6rW9aurKTRRWQDMyWlOyQnkkxrj3wrcntO9OzAdKMnp9zfgwnzj51l+V5IDtk1Ov6NYeVE4AZiQM2y9dMNtnGdeQ/PEey5wS76dUGO864A/5enfk6T8TvucfpeRlOOpnud/c4b1A76o5WC/lfoTlWNzui9nZQKwyrhpvw+AA3K69yV5hFBbcnFaTvfXqw/A+vZBnnmdDDydtovkwrlHLeMeArxVy/b43/4guejfkzNeW2AptSfRZwMP5cT6Vi3j5S7jAuC+nGElJP8E9qpv++eZ77+Ak2rMaxGwadodpCd/2n0fcE4t8zqBhicqN+YMOwB4N8Nt0ZjH1jhg75zuTUj+6VdfnOtLVGo7fwrZpjfkDDsTGJfTPRCYk7YXcp15PmdYndeQGvMpTdd365x+l7IyUfk28EKNaa4DflXLNnkWOD+n+3Tg32n7z0kTiZzhTwDH1zxX8xw/1fuiT87wp4DTc7q3yrPveuQMfx04spa4c8/vrwLvk/yTLsk3fjpeL5IfN21z+t2dE3Od61vL8VTv9SfPsHNrG5Zn3D+T/t/J2Ua5+/5y4KbViKHW8xvYmSQB+tJ5xBomKnWtX11NoWVUDonay6hMqqdfN2BSRFTl9PuYJHutax7VNiXJ+KZp5Z3xkhrTTK9uiYhF6XjtaplXt+rbUKlS4IU6ll+tJ/B4nv7dgM9j1YJ3HwOVOd3Tc9oXARVa/We2NefVrY5xu6Wx5MZV2/jdWHWb5k5XyD7I9QBwhaRNSDL3KtJtLKkr8BeSBLZ9Op9CCmmuEl9ELFTyuJF0vluS/FKqJElWy0h+ZUGy7z4ocBn/W++IqJI0iVWP1UK3/6bAXyT9Iaef0nlVL6PmvPIds6urtnlnsS0a89jaFHhIUu71ZAVJwlOIhpw/Nc3Iaf8iT3f1Ni7kOlPzGlnfNaRaF5Jju67tObTGssuAO/LMq1ptx8qmwOGSDsoZXk5yR6xQNdez5vWoOlmtM5a6zu+IeDp9NHEVsKmkB4GfRsS8GrF0A2bHqmW9PiY5J2AN1ree609NtZ6DkoYCvyW5U9aK5M7P/TVGq7nvB65GDHWd38uAj1fz/1OdCly/L2mM15Ojnn5TgZ6ScpfViyR7q2se1SaR/OLqHBGd0qZDrN7bMpOAD3Pm0yki2kfEAQVOu3me/lOBDbTqq4w1169QC0kOsGobN2DafNtwKsnJlxvX1Fqmn8bKE7Z63GoN2geRvB3yH5Jfd0eT3Ampju/SNNaBEdEBOJbkH3h9VolP0nokt0yrXUPyeKFvOt/zcuY7ieR2c31W2V5K/nP2ZPX25SSSxxe5x1qbiHh5NeZV0yrHiaSGHCdZbItGO7bS8fevsV0rImJ14sq1Jts0X4z1XWdqXiMLvYbMJLkrUNf2fK7GsttFxPdXcz3uqDGvthHx23R4IdermutZ83q0nFUTvtrUdX4TEX+NiO1J7lhvCfy/PPOYBqyvpExfbgzV6lvf1Y6vhtr+l0Byh+dhoGdEdCQp51FzPjX3ffU1vSEx1HV+TwJ61Sx3kmrI/6h8/5MKWb8vWRv1qLxGkhn/TElhyr2Ag0ies9YrIqaR/NP7g6QOSgpkbS5pz9WI5XVgvqSfS2qjpJDfAEk7FDDtTcCJkvZOY+guaetI3op4GbhMScHWQcBJwOq8m/82cICkDdIL5dkNmHYGsKHSwmmpvwHnS+qipIDcL+uI6z7gLEk9lNSXcU71gNXcB3eTPL8/LG2v1p7kUeJcSd3Jf0HJ5+/AgWkhtFYk5W9yj9/2wDxggaStScoqVXsU2ETS2UoKGLdPM/ua7gO+ke7jcpJyHEtI9m9DXQucK6k//K/A6OGrMZ98RgL9JQ1RUhj0wgZMm8W2aMxj61rgEqUFk9Nj++DViKmmNdmmNTXoOtOQa0gkb0Q+SPLiwnqS+gHH54zyKLClkoLn5Wmzg3IKSjbAncBBkvZN16FCSQH/6ioQ3gaOTJdRSXKu1+VvwI+UFGhtR/Kj5d4Cf7nXen6n6zc0PU4XkpQJrKo5g4j4GBgO/FpSK0m7kfwvKnR9Vyu+PO4CvibpCEllSgpHD8mZz+cRsVjSjiQ/9Gq6IN33/UnKIt27GjHUdX6/TpLU/VZS23Q77JpO9zawh6Re6f+ac+tYxgygh1Yt0FvI+n1JoYnKI1r5NskCSQ8VOB0RsZTkYNifpFDX1cBxEfFuofMg+YfXChhL8pjg7yTPphskPckPJCms9mEaz41Ax7qmS6d9neSg+BNJodrnWJmRHkXyrG4qScHAX9XxqKwud5BcMD8iuXjfW+fYq8b3LsmFYKKSUtzdSAr4DQfeAUYBb6b98rmB5HnsyHS8B2sMb+g+eBjoC0yPiJE5/X8NbEeyDR/Ls5za1m8M8AOSpGdaGsPknFF+SnLQz0/X5d6caeeTFBo7iOTW8njyvF0WEe+R3OG5guTYOIjk1fylhcRYY14PAf8H3KPkzY7RJOfAGouI90kStf+SrEvBlX1lsS1o3GPrLyTH1n8kzScpWJsv0WqQNdmmeea1OteZhlxDziB5JDKdpOzMLTnLnk9SBujIdF7TSY7DvG/71bMek0gKhZ9HcidnEskPi+r/GxeQ3BmYTXJe351nNrluJrnGPU+yXRaTlPUpRK3nN9Ah7Teb5HHGLJIC/PkcTXK8fA78iqSwL1DQ+q5ufKuIiE9Iyo39JI3jbZLCw5CUEbooPbZ/SZJQ1PQcScHXp4DfR8R/ViOGWs/v9Pg9iOTFh09IrrPfTqd7Mp3vOySPlR6tbRkkZdPGANMlfdaA9fsSrbwjb1lQ8uresRHxfNaxmJlZcZLUmyTBK2+K8iPFzFXoZ0hSF5LCcR9lHIqZmVlRcqKSkfR59XjgivRWoJmZmdXgRz9mZmZWtHxHxczMzIqWExUzMzMrWoXWTGtFpnPnztG7d++swzAzazZGjBjxWUR0yToOaxgnKs1U7969GT58eNZhmJk1G5I+rn8sKzZ+9GNmZmZFy4lKE5O0n6T3JE2QdE6e4ZtKekrSO5KeLbC6ZjMzsxbBiUoTklRK8kXP/Uk+lnVU+m2OXL8Hbo+IQSRVeF+2dqM0MzMrXk5UmtaOwISImJh+I+Uekm9J5OpH8k0ESD4n3hgfWDMzM1snOFFpWt1JPmxVbXLaL9dI4NC0/ZtAe0kb5puZpFMlDZc0fObMmY0erJmZWbFxopK9nwJ7SnoL2BOYAqzIN2JEXB8RlRFR2aWL37AzM7N1n19PblpTgJ453T3Sfv8TEVNJ76hIagd8KyLmNFVAny1YQsc25ZSXOkc1M7Pi5/9WTesNoK+kzSS1Ao4EHs4dQVJnSdX74Vzg5qYKZvGyFRx1/aucdscIFi/Le9PGzMysqDhRaUIRsRw4A3gCGAfcFxFjJF0kaVg62l7Ae5LeB7oClzRVPBXlpRy3S2+efu9TTrzlDRYsWd5UizIzM2sU/npyM1VZWRmrWzPtP96awk/uH8mA7h257cQd6LReq0aOzsys+EgaERGVWcdhDeM7Ki3QIdt259pjt2fctHl8+7pX+XTe4qxDMjMzy8uJSgu1T7+u3HLCDkyavYjDr3uFSZ8vyjokMzOzL3Gi0oLtukVn7jp5KHMWLePwa19hwqfzsw7JzMxsFU5UWrhte63PPafuxPKq4IjrXmX0lLlZh2RmZvY/TlSMbTbpwP2n7Uyb8lKOuv5VXv/w86xDMjMzA5yoWGqzzm25/7Sd6dKhNcfd/BrPvvdp1iGZmZk5UbGVunVqw33f25k+ndtxyu3DeXzUtKxDMjOzFs6Jiq2ic7vW/O3UnRjcoxNn3P0m970xqf6JzMzMmogTFfuSjm3Kuf2kHdl1i8787IF3uOnFD7MOyczMWignKpbXeq3KuPH4SvYfsDG/eXQsf/7v+7gWYzMzW9ucqFitWpeVcsVR23LY9j3483/Hc/Fj45ysmJnZWlWWdQBW3MpKS7j8W4No17qMm178kPmLl3HZoYMoLVHWoZmZWQvgRMXqVVIifnVQPzq0KeevT41n4ZIV/OnbQ2hV5htyZmbWtJyoWEEk8eN9tqRDRRkXPzaOBUuWc+2x29OmVWnWoZmZ2TrMiUoDSNoF6E3OdouI2zMLKAMn796Hdq3LOPehURx/8+vceEIlHSrKsw7LzMzWUb53XyBJdwC/B3YDdkibykyDysiRO/biiqO25c1PZnP0Da/y+cKlWYdkZmbrKN9RKVwl0C/82gsABw7qRttWZZx25wiOuO4V7jxpKBt3rMg6LDMzW8f4jkrhRgMbZx1EMfnK1htx+3d3ZPrcxRx27ct8PGth1iGZmdk6xolK4ToDYyU9Ienh6ibroLI2tM+G3H3KUBYuWc5h177Ce9PnZx2SmZmtQ+QnGYWRtGe+/hHx3NqOBaCysjKGDx+exaLzGj9jPsfc+BpLV1Rx64k7MqRnp6xDMjNbhaQREdEiyxY2Z76jUqA0IXkXaJ8247JKUopR367t+ftpu9ChopxjbniVVz6YlXVIZma2DnCiUgdJvXLajwBeBw4HjgBek3RYVrEVo14brsf9p+1Mt05tOP6W13lq3IysQzIzs2bOiUrdhkr6Sdr+C2CHiDg+Io4DdgQuyC604tS1QwX3fW9ntt64Pd+7YwT/fHtK1iGZmVkz5kSlDhFxPzA97SyJiE9zBs/C2y+v9du24q6Th7L9putz9r1vc/drn2QdkpmZNVP+R1uPiLgrbf13+sbPCZJOAB4DHs8usuLWvqKc2767I1/ZaiPOe2gU1z33QdYhmZlZM+REpUAR8f+A64FBaXN9RPw826iKW0V5Kdceuz0HDtqEy/71Lr974l38lpmZmTWEa6ZtgIh4AHgg6ziak1ZlJfzlyG1pX1HGVc98wILFy/nVQf0pKVHWoZmZWTPgOyr1kPRi+ne+pHk5zXxJ8wqcx36S3pM0QdI5eYb3kvSMpLckvSPpgMZejyyVlohLvzmQU/fow22vfMxP7x/J8hVVWYdlZmbNgO+o1CMidkv/tl+d6SWVAlcB+wCTgTckPRwRY3NGOx+4LyKukdSPpOxL7zUKvMhI4tz9t6ZDRRm//8/7LFiynL8etS0V5aVZh2ZmZkXMd1QKJGknSe1zuttLGlrApDsCEyJiYkQsBe4BDq4xTgAd0vaOwNTGiLnYSOKMr/blwoP68Z+xMzjy+leZNveLrMMyM7Mi5kSlcNcAC3K6F6b96tMdmJTTPTntl+tC4FhJk0nuppy5+mEWvxN23YxrjtmO8TPmc+BfX+TlDz7LOiQzMytSTlQKp8h5ZSUiqmi8R2dHAbdGRA/gAOAOSV/aN5JOlTRc0vCZM2c20qKzsf/ATfjnGbvRab1yvnPT61z//Ad+I8jMzL7EiUrhJko6S1J52vwQmFjAdFOAnjndPdJ+uU4C7gOIiFeACpKvNa8iIq6PiMqIqOzSpctqrUQx2WKjdvzzjN34er+uXPr4u5xx91ssWLI867DMzKyIOFEp3GnALiRJxmRgKHBqAdO9AfSVtJmkVsCRwMM1xvkE2BtA0jYkiUrzvmVSoHaty7j6mO04d/+t+dfoaRxy1Ut8MHNB/ROamVmL4ESlQBHxaUQcGREbRUTXiDi6RpX6tU23HDgDeAIYR/J2zxhJF0kalo72E+AUSSOBvwEnRAt6DiKJ7+25OXeeNJTPFy7l4Ctf4t+jp9c/oZmZrfPUgv4frhFJFSSPaPqT3PEAICK+m0U8lZWVMXz48CwW3aSmzvmC79/1JiMnzeH7e23OT/bZkrJS59NmtuYkjYiIyqzjsIbxf4DC3QFsDOwLPEdS1mR+phGtg7p1asN939uJo4f24ppnP+D4W15n1oIlWYdlZmYZcaJSuC0i4gJgYUTcBnyDpJyKNbLWZaVc+s2BXH7YIN74aDYHXfEiIyfNyTosMzPLgBOVwi1L/86RNICkYraNMoxnnXdEZU8eOG0XJHH4ta9wz+ufZB2SmZmtZU5UCne9pPWBC0je2hkLXJ5tSOu+gT068uiZuzG0zwac8+Aofv73d1i8bEXWYZmZ2VriwrTN1LpamLY2K6qCPz75Hlc98wGDenTk6mO2o8f662Udlpk1Iy5M2zw5UamHpB/XNTwi/ri2YsnV0hKVav8ZM52f3DeSslJxxVHbsVvfL9WLZ2aWlxOV5smPfurXPm0qge+TfKenO0kFcNtlGFeL9PX+G/PPM3alS/vWHHfza1z97ARXvW9mtg7zHZUCSXoe+EZEzE+72wOPRcQeWcTTUu+oVFu4ZDk/f+AdHn1nGvv278rvDx9M+4ryrMMysyLmOyrNk++oFK4rsDSne2nazzLQtnUZVxy1LRcc2I//jvuUg698ifEzXK2Nmdm6xolK4W4HXpd0oaQLgdeAWzONqIWTxEm7bcbdJw9l3uLlHHzVSzz2zrSswzIzs0bkRKVAEXEJcCIwO21OjIjLso3KAIb22ZBHz9yNrTduzw/ufpNLHhvL8hVVWYdlZmaNwIlKPSR1SP9uAHxEUpX+HcDHaT8rAht3rOCeU3fmuJ035YYXPuSYG19j5nxXvW9m1tw5Uanf3enfEcDwnKa624pEq7ISLjp4AH84fDBvT5rDQVe8yJufzM46LDMzWwNOVOoREQemfzeLiD45zWYR0Sfr+OzLvrV9Dx48fRfKy8S3r3uFO1792K8wm5k1U2VZB1DsJNVZV0pEvLm2YrHC9e/WkUfO2I2z732bC/4xmrc/mcMl3xxARXlp1qGZmVkDOFGp3x/qGBbAV9dWINYwndZrxc3H78BfnhrPX54az7vT53HtsdvTcwNXvW9m1ly4wrdmqqVX+NZQT787g7PveRtJ/OXIIey1lT98bdbSuMK35sllVBpA0gBJR0g6rrrJOiYrzFe37sojZ+7GJh0rOPHWN/jrU+OpqnKSbmZW7JyoFEjSr4Ar0uYrwOXAsEyDsgbZdMO2PHT6rhw8uBt/fPJ9Trl9OHO/WJZ1WGZmVgcnKoU7DNgbmB4RJwKDgY7ZhmQN1aZVKX/69hB+Paw/z70/k2FXvshHny3MOiwzM6uFE5XCfRERVcDytBK4T4GeGcdkq0ESx+/Sm3tO3Yl5XyzjqBte5ZNZi7IOy8zM8nCiUrjhkjoBN5BU9vYm8Eq2IdmaqOy9AXeePJRFS1dw1A2vMnm2kxUzs2LjRKVAEXF6RMyJiGuBfYDj00dA1oz179aRu04eyvzFyZ2VqXO+yDokMzPL4USlQJIelnS0pLYR8VFEvJN1TNY4BnTvyB0nDWXOwiRZmT53cdYhmZlZyolK4f4A7AaMlfR3SYdJqsg6KGscg3t24raTdmTWgqUcfcOrfDrPyYqZWTFwolKgiHguIk4H+gDXAUeQFKi1dcR2vdbn1hN3YPq8xRztry+bmRUFJyoNIKkN8C3gNGAH4LZsI7LGVtl7A245YQemzP6CY258lVkLnKyYmWXJiUqBJN0HjCP5ts+VwOYRcWa2UVlTGNpnQ246vpKPZy3imBtfY/bCpVmHZGbWYjlRKdxNJMnJaRHxTFqnSr0k7SfpPUkTJJ2TZ/ifJL2dNu9LmtPokVuD7bJFZ248vpKJny3k2JteY+4i12BrZpYFJyoFiognImJFQ6aRVApcBewP9AOOktSvxnx/FBFDImIISfX8DzZWzLZmdu/bheu+sz3jZyzgOze/5ur2zcwy4ESlae0ITIiIiRGxFLgHOLiO8Y8C/rZWIrOCfGWrjbjm2O0YN20ex9/8OvMXO1kxM1ubnKg0re7ApJzuyWm/L5G0KbAZ8HRtM5N0qqThkobPnDmzUQO12u29TVeuPHo7Rk+Zywm3vMGCJcuzDsnMrMVwolIgJY6V9Mu0u5ekHRtxEUcCf6/r8VJEXB8RlRFR2aVLl0ZctNVn3/4bc8VR2/L2pDl895Y3WLTUyYqZ2drgRKVwVwM7kzyeAZhPUv6kLlNY9cOFPdJ++RyJH/sUtf0HbsKfvz2E4R9/zndvfYMvljaoyJKZma0GJyqFGxoRPwAWA0TEbKBVPdO8AfSVtJmkViTJyMM1R5K0NbA+/shh0TtocDf+eMQQXvvwc065fTiLlzlZMTNrSk5UCrcsfYsnACR1Aep8RTkilgNnAE+Q1MFyX0SMkXSRpGE5ox4J3BMR0TShW2M6ZNvu/O6wwbz0wWecescIJytmZk1I/t9YGEnHAN8GtiOpkfYw4PyIuD+LeCorK2P48OFZLNpS970xiZ898A5f3Tp5M6h1WWnWIZlZHSSNiIjKrOOwhinLOoDmIiLukjQC2BsQcEhEjMs4LMvQETv0ZHlVcN5Dozjj7re46ujtaFXmm5RmZo3JiUo9JG2Q0/kpOQVeJW0QEZ+v/aisWBw9tBfLq6r45T/HcNbf3uKKo7elvNTJiplZY3GiUr8RJOVSBPQCZqftnYBPSOo+sRbsuJ17s3xFcNGjYzn73rf5y7eHUOZkxcysUThRqUdEbAYg6QbgoYh4PO3eHzgky9iseHx3t81YURVc8vg4ykrEH48YQmmJsg7LzKzZc6JSuJ0i4pTqjoj4l6TLswzIisspe/RhWVUVl//7PUpLxO8OG+xkxcxsDTlRKdxUSecDd6bdxwBTM4zHitDpe23BihXBH558n7IS8dtDB1HiZMXMbLU5USncUcCvgIdIyqw8z8paas3+58y9+7KsKvjrU+MpLSnhkkMGOFkxM1tNTlQKlL7d88Os47Dm4Udf68uKqiqueuYDykrERQf3R3KyYmbWUE5UzJqAJH769a1YviK47vmJlJaIXx3Uz8mKmVkDOVExayKSOGf/rVleFdz04oeUl4rzDtjGyYqZWQM4UTFrQpI4/xvbsKIquOGFDyktKeHn+23lZMXMrHskIiYAACAASURBVEBOVAokqQdwBbAbSWHaF4AfRsTkTAOzoiclj32WV1Vx7XMfUF4qfvL1rbIOy8ysWXCiUrhbgLuBw9PuY9N++2QWkTUbkrho2ACWrwiueHoCZSUl/PBrfbMOy8ys6DlRKVyXiLglp/tWSWdnFo01OyUl4tJvDmR5VfCn/75PWan4wVe2yDosM7Oi5kSlcLMkHcvKjxIeBczKMB5rhkpKxP99axArqoLfPZHUYHvanptnHZaZWdFyolK475KUUfkTSRmVl4ETsgzImqfSEvH7wwezvCr47b/epaxEnLx7n6zDMjMrSk5UCtcjIobl9pC0KzApo3isGSstEX86YjBVVcHFjyUfMjxhV3+I28ysJn+LvnBXFNjPrCBlpSX8+cgh7Nu/Kxc+MpZXJ/pJoplZTU5U6iFpZ0k/AbpI+nFOcyFQmnF41syVl5bw529vS88N2nDeQ6NYsnxF1iGZmRUVJyr1awW0I3lM1j6nmQcclmFcto5o06qUiw8ZyMSZC7nm2Q+yDsfMrKi4jEo9IuI54DlJt0bEx1nHY+umPbfswrDB3bj6mQ84cFA3ttioXdYhmZkVBd9RKVxrSddL+o+kp6ubrIOydccFB/ajoryEXzw0iojIOhwzs6LgOyqFux+4FrgRcEECa3Rd2rfmvAO24ZwHR3H/8MkcsUPPrEMyM8ucE5XCLY+Ia7IOwtZtR1T25IE3J3PJ4+P46jYb0bld66xDMjPLlB/9FO4RSadL2kTSBtVN1kHZuqWkRFx26EAWLV3OxY+OzTocM7PMOVEp3PHA/yOpkXZE2gzPNCJbJ22xUXu+v+fm/OPtqbwwfmbW4ZiZZcqJSoEiYrM8jes9tyZx+le2oE/ntvziodF8sdRFosys5XKiUiBJx+VrCphuP0nvSZog6ZxaxjlC0lhJYyTd3fjRW3NTUV7Kxd8cwCefL+KvT4/POhwzs8y4MG3hdshprwD2Bt4Ebq9tAkmlwFXAPsBk4A1JD0fE2Jxx+gLnArtGxGxJGzVF8Nb87LJ5Zw7bvgc3PD+Rg4d0Y+uNO2QdkpnZWuc7KgWKiDNzmlOA7UhqrK3LjsCEiJgYEUuBe4CDa4xzCnBVRMxOl/NpY8duzdcvDtiGDm3KOe/BUVRVuW4VM2t5nKisvoVAfZ+77c6qX1eenPbLtSWwpaSXJL0qab9GjNGaufXbtuL8b2zDm5/M4a7XP8k6HDOztc6Pfgok6RGg+idtCdAPuK8RZl0G9AX2AnoAz0saGBFz8sRwKnAqQK9evRph0dYcfHPb7jzw5mQu/9e7fL1fV7p2qMg6JDOztcZ3VAr3e+APaXMZsEdE5C0cm2MKkFu9aI+0X67JwMMRsSwiPgTeJ0lcviQiro+Iyoio7NKly+qsgzVDkrjkkIEsXVHFrx8Zk3U4ZmZrlROVAkXEcznNSxExuYDJ3gD6StpMUivgSODhGuP8g+RuCpI6kzwKmtiIods6oHfntpy1d18eHzWdp8bNyDocM7O1xolKPSS9mP6dL2lenuZDSafnmzYilgNnAE8A44D7ImKMpIskDUtHewKYJWks8Azw/yJiVtOvmTU3p+zehy27tuOX/xzDwiXLsw7HzGytkL/SumYkbQi8HBFbrc3lVlZWxvDhrhi3pRnx8ed865pXOGm3zbjgwH5Zh2PWrEgaERGVWcdhDeM7Kg0gqVRSN0m9qpv07sdeWcdmLcP2m27AMUN7cctLHzJq8tyswzEza3JOVAok6UxgBvAk8FjaPAoQEdMyDM1amJ/ttzUbtmvNuQ+9w/IVVVmHY2bWpJyoFO6HwFYR0T8iBqbNoKyDspanY5tyLjyoP6OnzOPWlz/KOhwzsyblRKVwkwDfa7eicMDAjfnKVl3445PvM2XOF1mHY2bWZJyoFG4i8KykcyX9uLrJOihrmSRx0cEDiIBf/mM0LhRvZusqJyqF+4SkfEoroH1OY5aJnhusx4/32ZKn3v2Uf4+ennU4ZmZNwlXoFygifg0gqV3avSDbiMzgxF1784+3p/Crh8ewa9/OdKgozzokM7NG5TsqBZI0QNJbwBhgjKQRkvpnHZe1bGWlJVx26EA+W7CE3/37vazDMTNrdE5UCnc98OOI2DQiNgV+AtyQcUxmDOrRieN36c2dr33MiI9nZx2OmVmjcqJSuLYR8Ux1R0Q8C7TNLhyzlX7y9a3YuEMF5z04imWuW8XM1iFOVAo3UdIFknqnzfn444FWJNq1LuOigwfw3oz53PCCD0szW3c4USncd4EuwIPAA0DntJ9ZUdinX1f2678xf/nveD6etTDrcMzMGoUTlQJFxOyIOCsitouI7SPi7IhwgQArKhcO6095aQnnu24VM1tHOFEpkKQnJXXK6V5f0hNZxmRW08YdK/jZflvxwvjP+OfbU7MOx8xsjTlRKVzniJhT3ZHeTdkow3jM8jpm6KYM6dmJ3zw6ljmLlmYdjpnZGnGiUrgqSb2qOyRtCvjeuhWd0hJx2aEDmfvFMi59fFzW4ZiZrREnKoX7BfCipDsk3Qk8D5ybcUxmeW2zSQdO3r0P9w2fzKsTZ2UdjpnZanOiUqCI+DewHXAvcA+wfUS4jIoVrR/u3ZeeG7ThvIdGsWT5iqzDMTNbLU5UGiAiPouIR9Pms6zjMatLm1alXHzIQCbOXMg1z36QdThmZqvFiYrZOmzPLbswbHA3rn7mAyZ86u9omlnz40TFbB13wYH9qCgv4RcPjXLdKmbW7DhRqYekDepqso7PrD5d2rfmvAO24bUPP+f+4ZOzDsfMrEHKsg6gGRhB8hqy8gwLoM/aDces4Y6o7MmDb07hksfH8dVtNqJzu9ZZh2RmVhDfUalHRGwWEX3SvzUbJynWLJSUiEsPHcCipcu5+NGxWYdjZlYwJyoFUuJYSRek3b0k7Zh1XGaF2mKj9nx/ry34x9tTeWH8zKzDMTMriBOVwl0N7AwcnXbPB67KLhyzhjt9r83p07ktv3hoNF8sdd0qZlb8nKgUbmhE/ABYDP/71k+rbEMya5iK8lIu+eZAPvl8EX99enzW4ZiZ1cuJSuGWSSol/b6PpC5AVX0TSdpP0nuSJkg6J8/wEyTNlPR22pzc+KGbrbTz5hty+PY9uOH5ibw7fV7W4ZiZ1cmJSuH+CjwEbCTpEuBF4NK6JkgTm6uA/YF+wFGS+uUZ9d6IGJI2NzZy3GZfct4B29ChTTnnPjiKqirXrWJmxcuJSoEi4i7gZ8BlwDTgkIi4v57JdgQmRMTEiFhK8o2gg5s2UrP6rd+2FRccuA1vfTKHu177OOtwzMxq5USlASLi3Yi4KiKujIhxBUzSHZiU0z057VfTtyS9I+nvkno2SrBm9ThkSHd279uZy//9HjPmLc46HDOzvJyoZO8RoHdEDAKeBG6rbURJp0oaLmn4zJl+vdTWjCQuPmQAS1dU8etHxmQdjplZXk5UmtYUIPcOSY+03/9ExKyIWJJ23ghsX9vMIuL6iKiMiMouXbo0erDW8my6YVvO2rsvj4+azlPjZmQdjpnZlzhRaVpvAH0lbSapFXAk8HDuCJI2yekcBhTySMms0Zy6Rx+26tqeX/5zDAuXLM86HDOzVThRKZCkQyWNlzRX0jxJ8yXV+W5nRCwHzgCeIElA7ouIMZIukjQsHe0sSWMkjQTOAk5oyvUwq6m8tIRLDx3IlDlf8Lsn3ss6HDOzVciffS+MpAnAQQUWom1ylZWVMXz48KzDsHXIr/45mtte+ZjT9tycn+27FSUl+b7DadZ8SRoREZVZx2EN468nF25GsSQpZk3hggP7sbwquPa5D5g+9wsuP2wwrcp809XMsuVEpXDDJd0L/AOoLvxKRDyYXUhmjaestISLDxlAt05t+N0T7zFzwRKuPXZ72leUZx2ambVg/rlUuA7AIuDrwEFpc2CmEZk1Mkn84Ctb8PvDB/PaxM854rpXXceKmWXKd1QKFBEnZh2D2dpy2PY92Kh9a75/5wgOvfplbj1xB/p2bZ91WGbWAvmOSoEkbSnpKUmj0+5Bks7POi6zprLHll2493s7s3RFFd+65mVe//DzrEMysxbIiUrhbgDOBZYBRMQ7JPWimK2zBnTvyIPf34XO7Vtz7E2v8a9R07IOycxaGCcqhVsvIl6v0c+1Y9k6r+cG6/HAabswsHtHTr/7TW556cOsQzKzFsSJSuE+k7Q5EACSDiP5irLZOm/9tq246+Sh7LNNV379yFgue3wcVVWug8nMmp4TlcL9ALgO2FrSFOBs4LRsQzJbeyrKS7nm2O35zk6bct3zEzn73rdZsnxF1mGZ2TrOb/0UKCImAl+T1BYoiYj5WcdktraVloiLDu7PJp0quPzf7zFz/hKuO257OriuFTNrIr6j0kARsdBJirVkkjh9ry344xGDeeOjzzni2leYPtd1rZhZ03CiYmar5dDtenDLiTswefYXfPPql3h/hvN3M2t8TlQKIKlE0i5Zx2FWbHbv24V7v7cTy6uCw655mVcnzso6JDNbxzhRKUBEVAFXZR2HWTHq360jD52+C13at+a4m17n0XemZh2Sma1DnKgU7ilJ35KkrAMxKzY91l+PB76/C4N6dOTMv73FTS+6rhUzaxxOVAr3PeB+YKmkeZLmS5qXdVBmxaLTeq248+Sh7NtvY37z6FgufnSs61oxszXmRKVAEdE+IkoiojwiOqTdHbKOy6yYVJSXctUx23HCLr258cUPOeuet1zXipmtEdej0gCShgF7pJ3PRsSjWcZjVoxKS8SvDurHJh0ruOxf7zJz/hKuP66Sjm1c14qZNZzvqBRI0m+BHwJj0+aHki7LNiqz4iSJ7+25OX85cghvfjKbw699malzvsg6LDNrhpyoFO4AYJ+IuDkibgb2A76RcUxmRe3gId259cQdmTpnMYde/TLvTnexLjNrGCcqDdMpp71jZlGYNSO7btGZ+763M0Fw+DWv8PIHn2Udkpk1I05UCncp8JakWyXdBowALsk4JrNmoV+3Djx4+q507VjBCTe/wcMjXdeKmRXGiUoBJJUAVcBOwIPAA8DOEXFvpoGZNSPdO7Xh76ftzJCenTjrb29xw/MTifDry2ZWNycqBUhrpv1ZREyLiIfTZnrWcZk1N53Wa8XtJ+3IAQM35pLHx3GR61oxs3o4USncfyX9VFJPSRtUN1kHZdbcVJSXcuVR23Hirr255aWPOPNvb7F4metaMbP8XI9K4b6d/v1BTr8A+mQQi1mzVlIifnlgP7p1bMMlj49j5oIl3PCdSjqu57pWzGxVvqNSgLSMyjkRsVmNxkmK2WqSxCl79OEvRw7hrU9mc9i1LzPFda2YWQ1OVAqQllH5f1nHYbYuOnhId2777o5Mn7uYQ69+iXHTXNeKma3kRKVwq11GRdJ+kt6TNEHSOXWM9y1JIamy8cI2K367bN6Z+7+/M0Icce0rPDJyqr8RZGYAyK8HFkZSvu/WR32PfySVAu8D+wCTgTeAoyJibI3x2gOPAa2AMyJieF3zraysjOHD6xzFrNmZOucLTrzlDd6bMZ8OFWXsP2ATDh7SjaF9NqS0RFmHZ82cpBER4R+CzYwL0xYoIjZbzUl3BCZExEQASfcAB5N8LyjXb4D/w4+YrAXr1qkNj561Gy9O+IxH3p7Ko+9M5d7hk9iofWu+MWgThg3uxpCenZCctJi1FE5U6iHpZxFxedp+eETcnzPs0og4r55ZdAcm5XRPBobWWMZ2QM+IeEySExVr0cpLS/jKVhvxla02YvGyFTw17lMeHjmFu179hFte+oheG6zHsMHdGDakG1t2bZ91uGbWxJyo1O9I4PK0/Vzg/pxh+wH1JSp1St8o+iNwQgHjngqcCtCrV681WaxZs1BRXso3Bm3CNwZtwrzFy3hi9HQeHjmVq5+dwJXPTGDrjdtz0OBuDBvcjZ4brJd1uGbWBFxGpR6S3oqIbWu25+uuZfqdgQsjYt+0+1yAiLgs7e4IfAAsSCfZGPgcGFZXORWXUbGWbOb8JTw+ahoPj5zKiI9nA7Bdr04MG9yNbwzqRpf2rTOO0IqRy6g0T05U6iHpzYjYrmZ7vu5api8jKUy7NzCFpDDt0RExppbxnwV+6sK0ZoWZ9PkiHn1nGv98ewrvTp9PiZK3iIYN6ca+/TemYxtXImcJJyrNkxOVekhaASwEBLQBFlUPAioiot6roKQDgD8DpcDNEXGJpIuA4RHxcI1xn8WJitlqGT9jPg+PnMrDI6fy8axFtCotYa+tujBsSDf23rorbVqVZh2iZciJSvPkRKWZcqJiVruIYOTkuTycvjn06fwltG1Vytf7b8ywwd3YrW9nyktdjVRL40SleXKi0kw5UTErzIqq4LUPZ/Hw21P51+jpzP1iGeuvV87+Azfh4MHd2KH3BpS4jpYWwYlK8+REpZlyomLWcEuXV/H8+zN5eORUnhw7gy+WrWCTjhUcOGgThg3uzoDuHVxHyzrMiUrz5ESlmXKiYrZmFi1dzpNjZ/DIyKk89/5Mlq0I+nRum7zuPKQbm3dpl3WI1sicqDRPTlSaKScqZo1nzqKl/Duto+WVibOIgP7dOrBv/43ZvW9nBvXo5Cr81wFOVJonJyrNlBMVs6YxY95iHn1nGo+MnMrIyXOIgI5tytlti87ssWVndu/bhW6d2mQdpq0GJyrNkxOVZsqJilnT+3zhUl6c8BkvvD+T58fPZMa8JQBssVE7du/bmT227MJOm23o157Xkojg0/lL6NqhYrWmd6LSPDlRaaacqJitXRHB+E8X8Pz7M3l+/Ge8NnEWS5ZX0aq0hB02W589+nZh975d2GaT9i6Q2wgigk8+X8SoKXMZNWUuo6fMZfSUeXyxbAVjfr3var1e7kSleXKi0kw5UTHL1uJlK3jjo8+TxOX9z3hvxnwAurRvze5bJHdbduvbmc7tXJ1/faqqgo9mLWT01HmMnjKXUZPnMnrqXOYvXg5AeanYauP2DOzekQHdO/Kt7XpQUd7wu1hOVJonJyrNlBMVs+IyY95inn9/Ji+M/4wXJ3zG5wuXAkmh3N37dmGPLTtTuekGtCpr2RXNVVUFEz9bmN4hSe6WjJ06j/lLkqSkVWkJW2/SngHdOzIwbbbs2r5RtpsTlebJiUoz5UTFrHhVVQVjps7j+fEzef79mYz4eDbLq4L1WpWyU58N2aNvZ3bfsgt9Orddpx8TragKJs5ckD66Se6WjJk6l4VLVwDQuqyEbTbpwIDuHf53t2TLru2brNZgJyrNkxOVZsqJilnzsWDJcl75YBYvpInLR7OST4Z179SGPbbszB59u7DL5p3puF7z/YDi8hVVfDBzYU55krmMnTaPRWlSUlFeQr9NkoSkf3qnZIuN2q3VTxk4UWmenKg0U05UzJqvT2Yt4vnxM3lh/ExenjCL+UuWUyIY0rNT+pioC4N7dKSsSL9HtHxFFeM/XfClpGTxsioA2pSX0r9bh/89vhnQvSObd2mb+fo4UWmenKg0U05UzNYNy1ZU8fakObzw/kyeG/8Z76R1t3SoKGPXLTqz6xad6ZTnTku+S3dtV/OGXOdrG3XR0hWMnTaXUVPm8e60eSxZniQlbVuV0r9bkowM7NGBAd060qdLu6KsIM+JSvPkRKWZcqJitm6as6i67pbPeH78TKbNXZx1SP/TrnUZ/bslj28G9kiSk802bNtsPuroRKV5Kss6ADMzW6nTeq04cFA3DhzUjYhg8uwvWLxsRd5x85fDzZ801FZmN1/vfAV8y0tFt45tmk1SYusOJypmZkVKEj03WC/rMMwyVZwltczMzMxwomJmZmZFzImKmZmZFS0nKmZmZla0nKiYmZlZ0XKiYmZmZkXLiYqZmZkVLddM20xJmgl8vJqTdwY+a8Rwmjtvj5W8LVbl7bHSurAtNo2ILlkHYQ3jRKUFkjTc1Uiv5O2xkrfFqrw9VvK2sKz40Y+ZmZkVLScqZmZmVrScqLRM12cdQJHx9ljJ22JV3h4reVtYJlxGxczMzIqW76iYmZlZ0XKiYmZmZkXLiUoLImk/Se9JmiDpnKzjyZKknpKekTRW0hhJP8w6pqxJKpX0lqRHs44la5I6Sfq7pHcljZO0c9YxZUnSj9LzZLSkv0mqyDomazmcqLQQkkqBq4D9gX7AUZL6ZRtVppYDP4mIfsBOwA9a+PYA+CEwLusgisRfgH9HxNbAYFrwdpHUHTgLqIyIAUApcGS2UVlL4kSl5dgRmBAREyNiKXAPcHDGMWUmIqZFxJtp+3ySf0Tds40qO5J6AN8Absw6lqxJ6gjsAdwEEBFLI2JOtlFlrgxoI6kMWA+YmnE81oI4UWk5ugOTcron04L/MeeS1BvYFngt20gy9WfgZ0BV1oEUgc2AmcAt6aOwGyW1zTqorETEFOD3wCfANGBuRPwn26isJXGiYi2apHbAA8DZETEv63iyIOlA4NOIGJF1LEWiDNgOuCYitgUWAi22TJek9Unuvm4GdAPaSjo226isJXGi0nJMAXrmdPdI+7VYkspJkpS7IuLBrOPJ0K7AMEkfkTwS/KqkO7MNKVOTgckRUX2H7e8kiUtL9TXgw4iYGRHLgAeBXTKOyVoQJyotxxtAX0mbSWpFUhju4YxjyowkkZRBGBcRf8w6nixFxLkR0SMiepMcF09HRIv9xRwR04FJkrZKe+0NjM0wpKx9Auwkab30vNmbFly42Na+sqwDsLUjIpZLOgN4gqTU/s0RMSbjsLK0K/AdYJSkt9N+50XE4xnGZMXjTOCuNKmfCJyYcTyZiYjXJP0deJPkbbm3cHX6tha5Cn0zMzMrWn70Y2ZmZkXLiYqZmZkVLScqZmZmVrScqJiZmVnRcqJi1kJIGijpgKzjMDNrCCcqLYSkBWt5eS+nf3tLOnptLrs2kk6QdOVaWtYxkt6RNErSy5IGN0Z8q7sOkloDfyB5xbQorM39UazS82N0A6dZq+dyFiR9JKlz+hXr07OOx7LlRMUaVfrRMiKiuubK3kBRJCpr2YfAnhExEPgN2dc7sRVwblqZWaNLv87dLFUfs1aUOgFOVFo4JyotmKQhkl5Nf/k/lH7TA0k7pP3elvS76l986a+/FyS9mTa7pP33Svs/TFqDZ86vvt8Cu6fz+lH6K/qfkp6VNF7Sr3Li+bGk0Wlzdi0xf13SK+ny70+/1VP9C+zXaf9RkrauZbV71rLsf0gaIWmMpFPTfqWSbk3jGSXpR2n/zSX9Ox3/hXzLioiXI2J22vkqyScL8q3PiZLel/Q6SSV01f27SHpA0htps2u+6XPGv1DSzem6TZR0Vtq/t6TREfFORIyQ9FNJF6bDnpX0J0nDJY1L9/uD6ba5OGfex0p6Pd2H11UnJZIWSPqDpJHAzgXuv9Ve3/TYeTDd9uMlXZ4zrK7jonPaXinp2ZztdYekl4A70u30dHrcPyWpVzrerZL+quSu2ERJh6X926XjVR9vB6f920p6TNLIdDt8O896bJ8OHwn8IKd/qZLz7Y00ju/Vs8/zxpBnvP3ScUZKeirtt2O6vd5K122rnG1c2/n5pXMkz7L2Tuc5Kj0eW6f9d0iXMzI9ltqrxh01SY9K2qvGLH8LbK6V16KC1tnWMRHhpgU0wII8/d4h+dUPcBHw57R9NLBz2v5bYHTavh5Qkbb3BYan7XuRfLhts5rLS4c9mtP/BJIvsG4ItEmXVQlsD4wC2gLtgDHAtjXi7Qw8D7RNu38O/DJt/wg4M20/Hbgxz/rmXXY6bIP0b3X/DdOYnsyZvlP69ymgb9o+lKTK+bq2/U9riWcTkurJuwCtgJeAK9NhdwO7pe29SKr6r16HK/PM60LgZaB1up1mAeUkd7RG14jlwrT9WeD/0vYfAlPTmFqTfO9mQ2Ab4BGgPB3vauC4tD2AI9L2QvZfg9c3z/6bCHQEKoCPSb5fVd9x0TltrwSezdleI4A2afcjwPFp+3eBf6TttwL3k/yo6wdMSPuXAR1yjssJgIBvATfkxNyxlvNuj7T9d6w8v04Fzk/bWwPDyTmn8pxbeWOoMW4Xkq+mb1bjOO8AlKXtXwMeWJ1zpMayKtJlbZl23w6cne7ricAOucumxrEMPArslbvf+PLxW+86u1n3Gt/ybKEkdST5x/tc2us24H5JnYD2EfFK2v9u4MC0vRy4UtIQYAWwZc4sX4+IDwtc/JMRMSuN40FgN5J/eg9FxMKc/ruTVNddbSeSfxYvSYLkAvhKzvDqDwuOAA5twLKHA2dJ+mY6Tk+SROw9oI+kK4DHgP+kv9R3IdlW1fNsXduKSvoKcFK6nJqGkvzjnJmOey8rt+nXgH45y+hQfZegDo9FxBJgiaRPga71jA8rv/c0ChgTEdPSWCaSbIfdSJKQN9JY2gCfptOsIPmoI+l49e2/Bq9vRNQsj/FURMxNpx8LbEryeKCu46LWdY+IL9L2nVl5zNwBXJ4z3j8iogoYK6l6mwq4VNIeQBXQnWR7jwL+IOn/SBL0F3IXmJ5fnSLi+Zxl7Z+2fx0YVH3XhiQh60vyGDGf2mLIfby3E/B89bkZEZ/nzPs2SX1Jzr3ynGkaco7MypluK5KPF76fdt9GcsfoKWBaRLyRxjAvnXctq1WnQtbZ1jFOVKwhfgTMAAaT/MJcnDNsYQPmU/O7DYV+x0EkF9Gjahm+JP27gtqP7S8tO73d/DWSu0iL0scDFRExW0kh2H2B04AjSH4hzomIIfUGKw0CbgT2r77wN0AJsFNE5G7j+i7uS3Laq7fBclZ9xFtRyzRVNaavSqcXcFtEnJtneYsjYkVdATVA3vXNI9861nVc5K5/zXUv9JjNXWb1DjiG5G7F9hGxTMmXpysi4n1J2wEHABdLeioiLipwOSK5K/hEgePnjaHAaX8DPBMR35TUm+TuWrWCz5ECl1Wb+o7NfNZkna2ZchmVFir9VTpb0u5pr+8Az0XEHGC+pKFp/yNzJutI8suoKh2/kAKU84H2aUWm7QAAAyBJREFUNfrtI2kDSW2AQ0geAbwAHKLkC61tgW+m/XK9CuwqaQv4X3mALWmYfMvuCMxOL8Bbk/wKRUnZhpKIeAA4H9gu/TX4oaTD03GkPG/0KCnj8CDwnZxfmDW99v/bu58QnaIwjuPfn0gpqVFq/AkbOyVhSRZWtqymGCmZ8melKCmspGxkTPlfxIaNhCkbNUxME6Nm2FEmmUnYIINj8ZzXXHfeMe9bQ2/j91nNvPeee++555z7nnnOM11gnaS5kmYAmwvbOokX41WON+HEaBxvgXn5HDMZjY7V6h6wSdK8fB1NkhZX2a+W9vtb9f1Tv3hJRIQglmXG84DRvt5S5drL5gBD+ctyPRHZQdJ84FNK6TKxrLOyWCiPrw+SKhG2lsLmu0BbvjdIWpbvZV3XUNINrJW0NB+zqVB2MP/cWipT8xgpeQEsqbQD+ZmSP2+WtDpfw2xFAvNLYIWkaZIWAWuqHLP8/KilzjbFOKLy/5gl6XXh9xPAVqBD0ix+f0PsduCMpB/Eg+Zj/rwduC5pC3CH2v4i7QO+KxIHLwLvgUfEksFC4HJKqQcicTFvg8jpKC4bkFIaltQKXK0k6RETiPEmAtWMObekZ8BOSQPEQ7U777sAuCCpMqGvRBVagNOSDhIh82vA09J5DhHr/O05CvItpbSqVJ83isTWh8AH4Elh8x7glKQ+YpzeJ6I6dckP9CO53oPA8zrL9+d6dub7MEKE81+V9uutof3+Sn0n6BeHgXOSjvJ71KBsN9HW+4BhJn5b8hXgZu47PYze1+XA8Tx2RoC2KmW3AeclJWKCVnGWyMnoVXSaYWKiUO81/JLvzQ7gRm6/IWADsbR1KbftrVKxesZI8VxfJG0jlkWnA4+BjpTSV0VS8ck8+flMRGe6iGWtfmCAKv86n1J6J6lLkdB/Gzg2UZ1t6vHbk22MYm6ApP1Ac0pp7yQdu5VIzts1Gcczs8nj8WmNyBEVq2ajpANE/3jF2NCwmZnZP+GIipmZmTUsJ9OamZlZw/JExczMzBqWJypmZmbWsDxRMTMzs4bliYqZmZk1LE9UzMzMrGH9BB7m/fIWH4TlAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Como se tiene que el menor errror se consigue con $2^7$ se tiene que este es el número óptimo de neuronas. "
      ],
      "metadata": {
        "id": "-AtQVekYegQT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##### Probando el número de concatenaciones de LSTMS"
      ],
      "metadata": {
        "id": "dD8OF3Elld_G"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Creamos nuestro modelo.\n",
        "validation_values = []\n",
        "HIDDEN_DIM = 2**7 # dimensión de la capas LSTM\n",
        "for i in range(1,11):\n",
        "    \n",
        "  model = NER_RNN(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM,\n",
        "                          N_LAYERS, BIDIRECTIONAL, DROPOUT, PAD_IDX, num_layers=i)\n",
        "\n",
        "  baseline_model_name = 'baseline'  # nombre que tendrá el modelo guardado...\n",
        "\n",
        "  model_name = baseline_model_name\n",
        "  criterion = baseline_criterion\n",
        "  n_epochs = baseline_n_epochs\n",
        "\n",
        "  model.apply(init_weights)\n",
        "  print(f'El modelo actual tiene {count_parameters(model):,} parámetros entrenables.')\n",
        "  # Optimizador\n",
        "  optimizer = optim.Adam(model.parameters())\n",
        "\n",
        "  # Enviamos el modelo y la loss a cuda (en el caso en que esté disponible)\n",
        "  model = model.to(device)\n",
        "  criterion = criterion.to(device)\n",
        "\n",
        "\n",
        "  best_valid_loss = float('inf')\n",
        "\n",
        "  for epoch in range(n_epochs):\n",
        "\n",
        "      start_time = time.time()\n",
        "\n",
        "      # Recuerdo: train_iterator y valid_iterator contienen el dataset dividido en batches.\n",
        "\n",
        "      # Entrenar\n",
        "      train_loss, train_precision, train_recall, train_f1 = train(\n",
        "          model, train_iterator, optimizer, criterion)\n",
        "\n",
        "      # Evaluar (valid = validación)\n",
        "      valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "          model, valid_iterator, criterion)\n",
        "\n",
        "      end_time = time.time()\n",
        "\n",
        "      epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
        "\n",
        "      # Si obtuvimos mejores resultados, guardamos este modelo en el almacenamiento (para poder cargarlo luego)\n",
        "      # Si detienen el entrenamiento prematuramente, pueden cargar el modelo en el siguiente recuadro de código.\n",
        "      if valid_loss < best_valid_loss:\n",
        "          best_valid_loss = valid_loss\n",
        "          torch.save(model.state_dict(), '{}.pt'.format(model_name))\n",
        "      # Si ya no mejoramos el loss de validación, terminamos de entrenar.\n",
        "\n",
        "      print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
        "      print(\n",
        "          f'\\tTrain Loss: {train_loss:.3f} | Train f1: {train_f1:.2f} | Train precision: {train_precision:.2f} | Train recall: {train_recall:.2f}'\n",
        "      )\n",
        "      print(\n",
        "          f'\\t Val. Loss: {valid_loss:.3f} |  Val. f1: {valid_f1:.2f} |  Val. precision: {valid_precision:.2f} | Val. recall: {valid_recall:.2f}'\n",
        "      )\n",
        "  # cargar el mejor modelo entrenado.\n",
        "  model.load_state_dict(torch.load('{}.pt'.format(model_name)))\n",
        "\n",
        "  # Limpiar ram de cuda\n",
        "  torch.cuda.empty_cache()\n",
        "\n",
        "  valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "      model, valid_iterator, criterion)\n",
        "\n",
        "  validation_values += [valid_loss]\n",
        "\n",
        "  print(\n",
        "      f'Val. Loss: {valid_loss:.3f} |  Val. f1: {valid_f1:.2f} | Val. precision: {valid_precision:.2f} | Val. recall: {valid_recall:.2f}'\n",
        "  )"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "302uoLVgliq9",
        "outputId": "507965af-bef4-4e71-ddcd-4bf7c54be0f3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "El modelo actual tiene 5,297,360 parámetros entrenables.\n",
            "to device cuda!!!!\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch: 01 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.400 | Train f1: 0.01 | Train precision: 0.05 | Train recall: 0.01\n",
            "\t Val. Loss: 1.065 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.042 | Train f1: 0.05 | Train precision: 0.21 | Train recall: 0.03\n",
            "\t Val. Loss: 0.866 |  Val. f1: 0.08 |  Val. precision: 0.35 | Val. recall: 0.05\n",
            "Epoch: 03 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.886 | Train f1: 0.26 | Train precision: 0.40 | Train recall: 0.19\n",
            "\t Val. Loss: 0.778 |  Val. f1: 0.33 |  Val. precision: 0.42 | Val. recall: 0.27\n",
            "Epoch: 04 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.795 | Train f1: 0.33 | Train precision: 0.43 | Train recall: 0.27\n",
            "\t Val. Loss: 0.734 |  Val. f1: 0.38 |  Val. precision: 0.44 | Val. recall: 0.34\n",
            "Epoch: 05 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.730 | Train f1: 0.38 | Train precision: 0.46 | Train recall: 0.33\n",
            "\t Val. Loss: 0.698 |  Val. f1: 0.40 |  Val. precision: 0.45 | Val. recall: 0.36\n",
            "Epoch: 06 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.680 | Train f1: 0.42 | Train precision: 0.50 | Train recall: 0.37\n",
            "\t Val. Loss: 0.674 |  Val. f1: 0.42 |  Val. precision: 0.47 | Val. recall: 0.38\n",
            "Epoch: 07 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.646 | Train f1: 0.44 | Train precision: 0.51 | Train recall: 0.39\n",
            "\t Val. Loss: 0.659 |  Val. f1: 0.43 |  Val. precision: 0.48 | Val. recall: 0.39\n",
            "Epoch: 08 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.619 | Train f1: 0.46 | Train precision: 0.53 | Train recall: 0.41\n",
            "\t Val. Loss: 0.657 |  Val. f1: 0.43 |  Val. precision: 0.47 | Val. recall: 0.40\n",
            "Epoch: 09 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.594 | Train f1: 0.47 | Train precision: 0.54 | Train recall: 0.41\n",
            "\t Val. Loss: 0.644 |  Val. f1: 0.43 |  Val. precision: 0.48 | Val. recall: 0.40\n",
            "Epoch: 10 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.568 | Train f1: 0.48 | Train precision: 0.55 | Train recall: 0.42\n",
            "\t Val. Loss: 0.645 |  Val. f1: 0.44 |  Val. precision: 0.47 | Val. recall: 0.41\n",
            "Val. Loss: 0.644 |  Val. f1: 0.43 | Val. precision: 0.48 | Val. recall: 0.40\n",
            "El modelo actual tiene 5,317,240 parámetros entrenables.\n",
            "to device cuda!!!!\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
            "  warnings.warn('{} seems not to be NE tag.'.format(chunk))\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch: 01 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 1.432 | Train f1: 0.01 | Train precision: 0.11 | Train recall: 0.01\n",
            "\t Val. Loss: 1.059 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 1.044 | Train f1: 0.05 | Train precision: 0.30 | Train recall: 0.03\n",
            "\t Val. Loss: 0.875 |  Val. f1: 0.06 |  Val. precision: 0.38 | Val. recall: 0.03\n",
            "Epoch: 03 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.897 | Train f1: 0.25 | Train precision: 0.38 | Train recall: 0.19\n",
            "\t Val. Loss: 0.809 |  Val. f1: 0.33 |  Val. precision: 0.41 | Val. recall: 0.27\n",
            "Epoch: 04 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.816 | Train f1: 0.32 | Train precision: 0.41 | Train recall: 0.26\n",
            "\t Val. Loss: 0.759 |  Val. f1: 0.35 |  Val. precision: 0.41 | Val. recall: 0.31\n",
            "Epoch: 05 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.750 | Train f1: 0.37 | Train precision: 0.44 | Train recall: 0.32\n",
            "\t Val. Loss: 0.719 |  Val. f1: 0.40 |  Val. precision: 0.45 | Val. recall: 0.36\n",
            "Epoch: 06 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.700 | Train f1: 0.41 | Train precision: 0.48 | Train recall: 0.35\n",
            "\t Val. Loss: 0.690 |  Val. f1: 0.41 |  Val. precision: 0.45 | Val. recall: 0.37\n",
            "Epoch: 07 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.657 | Train f1: 0.43 | Train precision: 0.50 | Train recall: 0.38\n",
            "\t Val. Loss: 0.678 |  Val. f1: 0.42 |  Val. precision: 0.47 | Val. recall: 0.38\n",
            "Epoch: 08 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.626 | Train f1: 0.44 | Train precision: 0.51 | Train recall: 0.39\n",
            "\t Val. Loss: 0.662 |  Val. f1: 0.43 |  Val. precision: 0.47 | Val. recall: 0.39\n",
            "Epoch: 09 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.599 | Train f1: 0.46 | Train precision: 0.53 | Train recall: 0.41\n",
            "\t Val. Loss: 0.660 |  Val. f1: 0.43 |  Val. precision: 0.47 | Val. recall: 0.40\n",
            "Epoch: 10 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.579 | Train f1: 0.47 | Train precision: 0.54 | Train recall: 0.41\n",
            "\t Val. Loss: 0.657 |  Val. f1: 0.43 |  Val. precision: 0.47 | Val. recall: 0.40\n",
            "Val. Loss: 0.657 |  Val. f1: 0.43 | Val. precision: 0.47 | Val. recall: 0.40\n",
            "El modelo actual tiene 5,337,120 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 13s\n",
            "\tTrain Loss: 1.440 | Train f1: 0.01 | Train precision: 0.03 | Train recall: 0.00\n",
            "\t Val. Loss: 1.099 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 13s\n",
            "\tTrain Loss: 1.065 | Train f1: 0.02 | Train precision: 0.15 | Train recall: 0.01\n",
            "\t Val. Loss: 0.896 |  Val. f1: 0.00 |  Val. precision: 0.04 | Val. recall: 0.00\n",
            "Epoch: 03 | Epoch Time: 0m 13s\n",
            "\tTrain Loss: 0.909 | Train f1: 0.20 | Train precision: 0.38 | Train recall: 0.14\n",
            "\t Val. Loss: 0.822 |  Val. f1: 0.31 |  Val. precision: 0.43 | Val. recall: 0.24\n",
            "Epoch: 04 | Epoch Time: 0m 13s\n",
            "\tTrain Loss: 0.825 | Train f1: 0.30 | Train precision: 0.40 | Train recall: 0.24\n",
            "\t Val. Loss: 0.764 |  Val. f1: 0.34 |  Val. precision: 0.41 | Val. recall: 0.29\n",
            "Epoch: 05 | Epoch Time: 0m 13s\n",
            "\tTrain Loss: 0.757 | Train f1: 0.36 | Train precision: 0.44 | Train recall: 0.31\n",
            "\t Val. Loss: 0.730 |  Val. f1: 0.39 |  Val. precision: 0.45 | Val. recall: 0.35\n",
            "Epoch: 06 | Epoch Time: 0m 14s\n",
            "\tTrain Loss: 0.704 | Train f1: 0.41 | Train precision: 0.48 | Train recall: 0.36\n",
            "\t Val. Loss: 0.699 |  Val. f1: 0.40 |  Val. precision: 0.44 | Val. recall: 0.36\n",
            "Epoch: 07 | Epoch Time: 0m 14s\n",
            "\tTrain Loss: 0.660 | Train f1: 0.44 | Train precision: 0.50 | Train recall: 0.39\n",
            "\t Val. Loss: 0.679 |  Val. f1: 0.41 |  Val. precision: 0.45 | Val. recall: 0.38\n",
            "Epoch: 08 | Epoch Time: 0m 13s\n",
            "\tTrain Loss: 0.624 | Train f1: 0.45 | Train precision: 0.51 | Train recall: 0.40\n",
            "\t Val. Loss: 0.671 |  Val. f1: 0.42 |  Val. precision: 0.46 | Val. recall: 0.39\n",
            "Epoch: 09 | Epoch Time: 0m 13s\n",
            "\tTrain Loss: 0.598 | Train f1: 0.46 | Train precision: 0.53 | Train recall: 0.41\n",
            "\t Val. Loss: 0.665 |  Val. f1: 0.42 |  Val. precision: 0.46 | Val. recall: 0.40\n",
            "Epoch: 10 | Epoch Time: 0m 13s\n",
            "\tTrain Loss: 0.574 | Train f1: 0.47 | Train precision: 0.54 | Train recall: 0.42\n",
            "\t Val. Loss: 0.652 |  Val. f1: 0.43 |  Val. precision: 0.47 | Val. recall: 0.40\n",
            "Val. Loss: 0.652 |  Val. f1: 0.43 | Val. precision: 0.47 | Val. recall: 0.40\n",
            "El modelo actual tiene 5,357,000 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 16s\n",
            "\tTrain Loss: 1.443 | Train f1: 0.01 | Train precision: 0.03 | Train recall: 0.01\n",
            "\t Val. Loss: 1.093 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 16s\n",
            "\tTrain Loss: 1.062 | Train f1: 0.01 | Train precision: 0.11 | Train recall: 0.01\n",
            "\t Val. Loss: 0.891 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 03 | Epoch Time: 0m 16s\n",
            "\tTrain Loss: 0.907 | Train f1: 0.24 | Train precision: 0.39 | Train recall: 0.18\n",
            "\t Val. Loss: 0.805 |  Val. f1: 0.32 |  Val. precision: 0.39 | Val. recall: 0.27\n",
            "Epoch: 04 | Epoch Time: 0m 16s\n",
            "\tTrain Loss: 0.814 | Train f1: 0.32 | Train precision: 0.41 | Train recall: 0.26\n",
            "\t Val. Loss: 0.752 |  Val. f1: 0.34 |  Val. precision: 0.40 | Val. recall: 0.30\n",
            "Epoch: 05 | Epoch Time: 0m 16s\n",
            "\tTrain Loss: 0.749 | Train f1: 0.36 | Train precision: 0.44 | Train recall: 0.30\n",
            "\t Val. Loss: 0.717 |  Val. f1: 0.38 |  Val. precision: 0.43 | Val. recall: 0.34\n",
            "Epoch: 06 | Epoch Time: 0m 16s\n",
            "\tTrain Loss: 0.699 | Train f1: 0.40 | Train precision: 0.48 | Train recall: 0.35\n",
            "\t Val. Loss: 0.695 |  Val. f1: 0.41 |  Val. precision: 0.46 | Val. recall: 0.37\n",
            "Epoch: 07 | Epoch Time: 0m 16s\n",
            "\tTrain Loss: 0.660 | Train f1: 0.43 | Train precision: 0.50 | Train recall: 0.38\n",
            "\t Val. Loss: 0.687 |  Val. f1: 0.41 |  Val. precision: 0.45 | Val. recall: 0.38\n",
            "Epoch: 08 | Epoch Time: 0m 16s\n",
            "\tTrain Loss: 0.629 | Train f1: 0.45 | Train precision: 0.52 | Train recall: 0.40\n",
            "\t Val. Loss: 0.684 |  Val. f1: 0.42 |  Val. precision: 0.45 | Val. recall: 0.39\n",
            "Epoch: 09 | Epoch Time: 0m 16s\n",
            "\tTrain Loss: 0.604 | Train f1: 0.47 | Train precision: 0.54 | Train recall: 0.42\n",
            "\t Val. Loss: 0.670 |  Val. f1: 0.42 |  Val. precision: 0.46 | Val. recall: 0.39\n",
            "Epoch: 10 | Epoch Time: 0m 16s\n",
            "\tTrain Loss: 0.580 | Train f1: 0.48 | Train precision: 0.56 | Train recall: 0.43\n",
            "\t Val. Loss: 0.654 |  Val. f1: 0.44 |  Val. precision: 0.49 | Val. recall: 0.41\n",
            "Val. Loss: 0.654 |  Val. f1: 0.44 | Val. precision: 0.49 | Val. recall: 0.41\n",
            "El modelo actual tiene 5,376,880 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 1.438 | Train f1: 0.00 | Train precision: 0.02 | Train recall: 0.00\n",
            "\t Val. Loss: 1.112 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 1.082 | Train f1: 0.00 | Train precision: 0.04 | Train recall: 0.00\n",
            "\t Val. Loss: 0.889 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 03 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.911 | Train f1: 0.22 | Train precision: 0.36 | Train recall: 0.16\n",
            "\t Val. Loss: 0.807 |  Val. f1: 0.32 |  Val. precision: 0.40 | Val. recall: 0.27\n",
            "Epoch: 04 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.812 | Train f1: 0.33 | Train precision: 0.42 | Train recall: 0.27\n",
            "\t Val. Loss: 0.741 |  Val. f1: 0.36 |  Val. precision: 0.44 | Val. recall: 0.30\n",
            "Epoch: 05 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.743 | Train f1: 0.38 | Train precision: 0.47 | Train recall: 0.33\n",
            "\t Val. Loss: 0.702 |  Val. f1: 0.41 |  Val. precision: 0.47 | Val. recall: 0.36\n",
            "Epoch: 06 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.686 | Train f1: 0.42 | Train precision: 0.50 | Train recall: 0.37\n",
            "\t Val. Loss: 0.679 |  Val. f1: 0.42 |  Val. precision: 0.48 | Val. recall: 0.38\n",
            "Epoch: 07 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.647 | Train f1: 0.44 | Train precision: 0.52 | Train recall: 0.39\n",
            "\t Val. Loss: 0.662 |  Val. f1: 0.43 |  Val. precision: 0.48 | Val. recall: 0.39\n",
            "Epoch: 08 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.618 | Train f1: 0.47 | Train precision: 0.54 | Train recall: 0.41\n",
            "\t Val. Loss: 0.654 |  Val. f1: 0.44 |  Val. precision: 0.49 | Val. recall: 0.40\n",
            "Epoch: 09 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.592 | Train f1: 0.47 | Train precision: 0.55 | Train recall: 0.42\n",
            "\t Val. Loss: 0.647 |  Val. f1: 0.45 |  Val. precision: 0.51 | Val. recall: 0.40\n",
            "Epoch: 10 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.569 | Train f1: 0.49 | Train precision: 0.56 | Train recall: 0.43\n",
            "\t Val. Loss: 0.651 |  Val. f1: 0.44 |  Val. precision: 0.48 | Val. recall: 0.41\n",
            "Val. Loss: 0.647 |  Val. f1: 0.45 | Val. precision: 0.51 | Val. recall: 0.40\n",
            "El modelo actual tiene 5,396,760 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 21s\n",
            "\tTrain Loss: 1.423 | Train f1: 0.02 | Train precision: 0.05 | Train recall: 0.02\n",
            "\t Val. Loss: 1.092 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 21s\n",
            "\tTrain Loss: 1.068 | Train f1: 0.01 | Train precision: 0.11 | Train recall: 0.00\n",
            "\t Val. Loss: 0.894 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 03 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 0.910 | Train f1: 0.19 | Train precision: 0.38 | Train recall: 0.13\n",
            "\t Val. Loss: 0.811 |  Val. f1: 0.31 |  Val. precision: 0.42 | Val. recall: 0.25\n",
            "Epoch: 04 | Epoch Time: 0m 21s\n",
            "\tTrain Loss: 0.819 | Train f1: 0.30 | Train precision: 0.39 | Train recall: 0.24\n",
            "\t Val. Loss: 0.760 |  Val. f1: 0.34 |  Val. precision: 0.42 | Val. recall: 0.29\n",
            "Epoch: 05 | Epoch Time: 0m 21s\n",
            "\tTrain Loss: 0.752 | Train f1: 0.35 | Train precision: 0.43 | Train recall: 0.30\n",
            "\t Val. Loss: 0.726 |  Val. f1: 0.37 |  Val. precision: 0.42 | Val. recall: 0.33\n",
            "Epoch: 06 | Epoch Time: 0m 21s\n",
            "\tTrain Loss: 0.706 | Train f1: 0.39 | Train precision: 0.45 | Train recall: 0.34\n",
            "\t Val. Loss: 0.703 |  Val. f1: 0.40 |  Val. precision: 0.45 | Val. recall: 0.36\n",
            "Epoch: 07 | Epoch Time: 0m 21s\n",
            "\tTrain Loss: 0.663 | Train f1: 0.41 | Train precision: 0.47 | Train recall: 0.37\n",
            "\t Val. Loss: 0.674 |  Val. f1: 0.41 |  Val. precision: 0.46 | Val. recall: 0.38\n",
            "Epoch: 08 | Epoch Time: 0m 21s\n",
            "\tTrain Loss: 0.616 | Train f1: 0.44 | Train precision: 0.50 | Train recall: 0.40\n",
            "\t Val. Loss: 0.648 |  Val. f1: 0.42 |  Val. precision: 0.46 | Val. recall: 0.39\n",
            "Epoch: 09 | Epoch Time: 0m 21s\n",
            "\tTrain Loss: 0.575 | Train f1: 0.50 | Train precision: 0.55 | Train recall: 0.45\n",
            "\t Val. Loss: 0.621 |  Val. f1: 0.52 |  Val. precision: 0.55 | Val. recall: 0.50\n",
            "Epoch: 10 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 0.540 | Train f1: 0.56 | Train precision: 0.61 | Train recall: 0.52\n",
            "\t Val. Loss: 0.603 |  Val. f1: 0.57 |  Val. precision: 0.60 | Val. recall: 0.55\n",
            "Val. Loss: 0.603 |  Val. f1: 0.57 | Val. precision: 0.60 | Val. recall: 0.55\n",
            "El modelo actual tiene 5,416,640 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 1.437 | Train f1: 0.01 | Train precision: 0.06 | Train recall: 0.01\n",
            "\t Val. Loss: 1.049 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 1.034 | Train f1: 0.06 | Train precision: 0.27 | Train recall: 0.03\n",
            "\t Val. Loss: 0.865 |  Val. f1: 0.14 |  Val. precision: 0.41 | Val. recall: 0.09\n",
            "Epoch: 03 | Epoch Time: 0m 25s\n",
            "\tTrain Loss: 0.892 | Train f1: 0.26 | Train precision: 0.39 | Train recall: 0.20\n",
            "\t Val. Loss: 0.795 |  Val. f1: 0.33 |  Val. precision: 0.39 | Val. recall: 0.28\n",
            "Epoch: 04 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.803 | Train f1: 0.34 | Train precision: 0.43 | Train recall: 0.28\n",
            "\t Val. Loss: 0.746 |  Val. f1: 0.35 |  Val. precision: 0.42 | Val. recall: 0.30\n",
            "Epoch: 05 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.738 | Train f1: 0.37 | Train precision: 0.46 | Train recall: 0.32\n",
            "\t Val. Loss: 0.713 |  Val. f1: 0.39 |  Val. precision: 0.44 | Val. recall: 0.34\n",
            "Epoch: 06 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.697 | Train f1: 0.40 | Train precision: 0.48 | Train recall: 0.35\n",
            "\t Val. Loss: 0.693 |  Val. f1: 0.41 |  Val. precision: 0.46 | Val. recall: 0.37\n",
            "Epoch: 07 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.652 | Train f1: 0.43 | Train precision: 0.50 | Train recall: 0.38\n",
            "\t Val. Loss: 0.674 |  Val. f1: 0.40 |  Val. precision: 0.44 | Val. recall: 0.37\n",
            "Epoch: 08 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.610 | Train f1: 0.47 | Train precision: 0.54 | Train recall: 0.42\n",
            "\t Val. Loss: 0.634 |  Val. f1: 0.50 |  Val. precision: 0.55 | Val. recall: 0.46\n",
            "Epoch: 09 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.569 | Train f1: 0.53 | Train precision: 0.59 | Train recall: 0.48\n",
            "\t Val. Loss: 0.605 |  Val. f1: 0.55 |  Val. precision: 0.58 | Val. recall: 0.52\n",
            "Epoch: 10 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.533 | Train f1: 0.57 | Train precision: 0.63 | Train recall: 0.52\n",
            "\t Val. Loss: 0.598 |  Val. f1: 0.59 |  Val. precision: 0.63 | Val. recall: 0.56\n",
            "Val. Loss: 0.598 |  Val. f1: 0.59 | Val. precision: 0.63 | Val. recall: 0.56\n",
            "El modelo actual tiene 5,436,520 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 1.498 | Train f1: 0.00 | Train precision: 0.01 | Train recall: 0.00\n",
            "\t Val. Loss: 1.059 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 1.041 | Train f1: 0.05 | Train precision: 0.23 | Train recall: 0.03\n",
            "\t Val. Loss: 0.864 |  Val. f1: 0.14 |  Val. precision: 0.37 | Val. recall: 0.09\n",
            "Epoch: 03 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.891 | Train f1: 0.27 | Train precision: 0.39 | Train recall: 0.21\n",
            "\t Val. Loss: 0.781 |  Val. f1: 0.33 |  Val. precision: 0.43 | Val. recall: 0.27\n",
            "Epoch: 04 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.801 | Train f1: 0.34 | Train precision: 0.44 | Train recall: 0.28\n",
            "\t Val. Loss: 0.734 |  Val. f1: 0.39 |  Val. precision: 0.46 | Val. recall: 0.34\n",
            "Epoch: 05 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.738 | Train f1: 0.38 | Train precision: 0.47 | Train recall: 0.33\n",
            "\t Val. Loss: 0.703 |  Val. f1: 0.40 |  Val. precision: 0.45 | Val. recall: 0.36\n",
            "Epoch: 06 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.690 | Train f1: 0.40 | Train precision: 0.48 | Train recall: 0.35\n",
            "\t Val. Loss: 0.678 |  Val. f1: 0.41 |  Val. precision: 0.46 | Val. recall: 0.38\n",
            "Epoch: 07 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.656 | Train f1: 0.43 | Train precision: 0.50 | Train recall: 0.37\n",
            "\t Val. Loss: 0.662 |  Val. f1: 0.43 |  Val. precision: 0.47 | Val. recall: 0.39\n",
            "Epoch: 08 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.628 | Train f1: 0.44 | Train precision: 0.51 | Train recall: 0.39\n",
            "\t Val. Loss: 0.657 |  Val. f1: 0.43 |  Val. precision: 0.47 | Val. recall: 0.39\n",
            "Epoch: 09 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.599 | Train f1: 0.45 | Train precision: 0.52 | Train recall: 0.40\n",
            "\t Val. Loss: 0.650 |  Val. f1: 0.43 |  Val. precision: 0.48 | Val. recall: 0.40\n",
            "Epoch: 10 | Epoch Time: 0m 26s\n",
            "\tTrain Loss: 0.580 | Train f1: 0.46 | Train precision: 0.54 | Train recall: 0.41\n",
            "\t Val. Loss: 0.647 |  Val. f1: 0.44 |  Val. precision: 0.49 | Val. recall: 0.41\n",
            "Val. Loss: 0.647 |  Val. f1: 0.44 | Val. precision: 0.49 | Val. recall: 0.41\n",
            "El modelo actual tiene 5,456,400 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 1.433 | Train f1: 0.01 | Train precision: 0.06 | Train recall: 0.01\n",
            "\t Val. Loss: 1.048 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 30s\n",
            "\tTrain Loss: 1.038 | Train f1: 0.11 | Train precision: 0.35 | Train recall: 0.07\n",
            "\t Val. Loss: 0.858 |  Val. f1: 0.23 |  Val. precision: 0.44 | Val. recall: 0.16\n",
            "Epoch: 03 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.890 | Train f1: 0.26 | Train precision: 0.40 | Train recall: 0.20\n",
            "\t Val. Loss: 0.784 |  Val. f1: 0.34 |  Val. precision: 0.44 | Val. recall: 0.28\n",
            "Epoch: 04 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.807 | Train f1: 0.33 | Train precision: 0.43 | Train recall: 0.27\n",
            "\t Val. Loss: 0.736 |  Val. f1: 0.36 |  Val. precision: 0.43 | Val. recall: 0.31\n",
            "Epoch: 05 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.743 | Train f1: 0.37 | Train precision: 0.46 | Train recall: 0.32\n",
            "\t Val. Loss: 0.701 |  Val. f1: 0.41 |  Val. precision: 0.47 | Val. recall: 0.36\n",
            "Epoch: 06 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.692 | Train f1: 0.41 | Train precision: 0.49 | Train recall: 0.36\n",
            "\t Val. Loss: 0.679 |  Val. f1: 0.42 |  Val. precision: 0.48 | Val. recall: 0.38\n",
            "Epoch: 07 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.653 | Train f1: 0.44 | Train precision: 0.51 | Train recall: 0.38\n",
            "\t Val. Loss: 0.667 |  Val. f1: 0.43 |  Val. precision: 0.48 | Val. recall: 0.39\n",
            "Epoch: 08 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.625 | Train f1: 0.45 | Train precision: 0.52 | Train recall: 0.40\n",
            "\t Val. Loss: 0.655 |  Val. f1: 0.44 |  Val. precision: 0.49 | Val. recall: 0.40\n",
            "Epoch: 09 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.598 | Train f1: 0.46 | Train precision: 0.53 | Train recall: 0.41\n",
            "\t Val. Loss: 0.648 |  Val. f1: 0.44 |  Val. precision: 0.49 | Val. recall: 0.40\n",
            "Epoch: 10 | Epoch Time: 0m 30s\n",
            "\tTrain Loss: 0.574 | Train f1: 0.48 | Train precision: 0.55 | Train recall: 0.43\n",
            "\t Val. Loss: 0.639 |  Val. f1: 0.45 |  Val. precision: 0.51 | Val. recall: 0.41\n",
            "Val. Loss: 0.639 |  Val. f1: 0.45 | Val. precision: 0.51 | Val. recall: 0.41\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "fig = plt.figure()\n",
        "ax = fig.add_subplot(1, 1, 1)\n",
        "ax.plot([i for i in range(1,11)], validation_values)\n",
        "plt.xlabel('Numero de capas ocultas')\n",
        "plt.ylabel('Error en el conjunto de validación')\n",
        "plt.title('Error en el conjunto de validación en función de el número de capas ocultas')\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "ymjwgnPgpNsP",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "outputId": "3a8ecc4a-1907-4613-ae36-35d8a64b71df"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdkAAAEWCAYAAAAq+e1jAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3xV9f348dc7izAyLhBWgIQpeyVKFQdqVZzQ1qqotdo666xWq51qW9va+rNaJ9jq173qwImjAkodBCEBAig7N2zIYma9f3+cE72EhCSQe869N+/n43EeyT33jPc999zzPuMzRFUxxhhjTOuL8zsAY4wxJlZZkjXGGGPCxJKsMcYYEyaWZI0xxpgwsSRrjDHGhIklWWOMMSZMLMmGEJHbReRpj9e5REQmernO5hCRbBFREUnweL1rROS77v+/EpHHmjPtIayvwXWIyAQR+UJEAoey/NYgjsdFpEREvgjjejzZFuHat1pruSJyuIgsFZGU1ootnETkYhH5xO84ooW7jwz0an1N7owisgboDtSEjH5CVa8JV1BtiaoOb43luIn6aVXt3RrLiwSqepcf6xCRPsBdwBmqWhLuGJrhaOAkoLeq7gzXSqJkW4SVm6D/CUxV1Qq/4zHhJSJPAEFV/U241tHcM74zVfWDpiYSkQRVra43Ll5Vaxqbp4FltGh6Y1qbqhYBx/kdR4gsYE04E2xjInBbhNsA4I+qutCrFTZ03DQxRFUPOABrgO828t7FwFzgXmAb8EfgCeBh4G1gJ/BdYCgwCygFlgBnhSxjv+kbWE8a8C9gA1Dsric+JIZPgL8DJcBq4NQDfJ5ewH+ALe6014W8dzvO1WBj804GFgLlwEpgUsgyZwDbgRXAZfWW+SLwJFDhfv7chravuy3+GPLeRJyzrNBpfwEUAGXAC0Ay0BHYDdQCO9yhF9AO+Aew3h3+AbRr5LPFu9twK7AKuBpQIKGp76CB7bsb6Bwybqy73EScg9h/3f1lK/AMkN7I9tjn+wB+BKx15/11vWmPAD7F2cc2AA8ASSHzDgfed7+jTcCvGlnHWe53VIqzzw5tavsfYH/5CbAUZ7+cCWSFvKfAlcDX7roeBKSBZfwU2INzJ2kHcAfuPl9vOgUGhuxHDwJv4exznwMD/NoWtNK+5U4bB9yK8/vbhvPb6uy+lx263EaOZQ3G3Mxt+hDwjvs9zAV64PymSoBlwNgWHGdeBp7GOZZcygGOIQ18ji7utOXAF8AfQmMHhoR8v8uBcw6wrM7A4zjHhxLgNXd8AHjTjb/E/b93yHyzgD+76y8HXmff3/xLwEZ3O88Bhoe8dxpQiLNvFgO/OMB3/Ruc3/xmnGNoWsj7RwP/w9k/i4CLQ2K7NGS6fb7buu8VuByoAird7/QN9/26/avCjfN7IfMOBGa7n2sr8EJj2/abeZqcoOkkWw1ci3NV3N7dGcuACe5GSnF3ml8BScAJbvCHhey8odPv90MFXgUexUkm3dwv9oqQGKqAy3B+zFe5O0xDB6w4YD7wOzeW/jg/+lMaOsjUm/cIN86T3OVkAkPc9+bg/ACTgTE4O+YJIcvc4+5Y8Tg75mcNbV+al2S/wPlBdsY5gF/Z0LTuuDuBz9xtloGzQ/6hkc93Jc6Boo+77I/Y90DY6HfQwLL+y74nGn8DHgnZSU/COQHIcLfdPxrZHt98H8AwnB/Cse68/w9n36ubNgf4Ds5+mO1umxvc91JwDuA3ud9RCjC+gXUMxjnROwnnhOAWnH03qant38A2mOzOO9SN6TfA/+r90N8E0oG+OPvMpAP8zj5p7HXogSNkP9qGs88m4JzIPO/jtmjNfet6nH26t7sfPAo8576XTdNJtrHfT3O26Vac/SwZZx9fDVyE87v+I/BRC44zVcAUd9r2HOAY0sDneB7n5KIjMAInUX3ivtcRJ+Fc4n73dSe4wxpZ1ls4JxsB93s+zh3fBfgB0MHdR17CTcDu+7Pc9Y5w1/kf9j1B+4k7X92J/sKQ9zYAx7j/B4BxjcT2E5x9rj/QCXgFeMp9Lwsnj0x14+4CjAmJrckk29Ax1x33Q3cfiQPOxfkd9HTfew7nBD/O/a6Obij2fZbX5ATOjrkD52yhbrgsJPh19aZ/Angy5PUxOGc0cSHjngNub2j6BtbfHdgLtA8ZN5Vvd+iLgRUh73VwN2KPBpY1voF4bwMer3+QaWDeR4F7GxjfB+cqIyVk3J9xnlvXLfODkPeGAbvrbd+WJNkLQ17fzbfJa59p3XErgdNCXp+Cc9uxscR4Zcjrk93tmNDUd9DAsi4F/uv+Lzg/+mMbmXYKsKCR7fHN94FzwHo+ZLqOOGegjZ0A3gC8GhLrgkamC13Hb4EXQ96LwzmQTGxq+zew3HeAn9Zb1i7cq1l32x4d8v6LwK2NLOtiWp5kHwt57zRgmY/bojX3raXAiSGve+IkrLqTq6aSbGO/n+Zs0+kh710LLA15PRIodf9vznFmTsh7BzyG1FtOvPt5h4SMu4tvk+y5wMf15nkU+H0Dy+qJc/cr0ND2qjftGKAk5PUs4C8hr4fh/B4buruV7m7LNPf1OuAKILWJdX4I/Czk9WEh3/VtuL/vBuabxSEk2QaWtxCY7P7/JDCNkKv6pobmPpOdoo0/ky1qYlwvoEhVa0PGrcW5EjzQMupk4ZypbBCRunFx9ebZWPePqu5yp+vUyLJ6iUhpyLh44OMDrL9OH5xb2vX1ArbrvoUk1gK5DcWHc6BNPoTnMPWX1esA0/ZyYwmNq7Hpe7HvNg2drznfQaj/AP8UkZ44V0S1uNtYRLoD9+GcfKW4y2lOgZp94lPVnSKyre61iAzGubrNxTnRSsC5mgDnu1vZzHV887lVtVZEith3X23u9s8C7hORe0LGibusunXUX1ZD++zBamzZfmyL1ty3soBXRST0eFKDk6yboyW/n/o2hfy/u4HXddu4OceZ+sfIpo4hdTJw9u0Dbc/x9dadADzVwLL6uOvd7/cnIh1wHgNOwrnaBEipV2amfgyJQFcR2Qr8CeeKMAPn9w/QFedu4A9w7uz8RUQKcE4uP20gvoaOX3UnZs3dj1tMRC4CbsQ5aQPne+3q/n8Lzu35L0SkBLhHVf99oOW1RhF6bWLceqCPiMSFJNq+wFdNLKNOEc6ZbteDTEr1l7VaVQcd5LwDGhi/HugsIikhP5K+OGf9LbUTJ0HU6dGCeRvahutxfnRLQuJa38j8G3B2XEKmrdOi70BVS0TkPZyz6qE4V6B18d3lxjpSVbeLyBSc56dN2eAuC/jmINAl5P2HgQW4pUJF5Abg7JD4z2vGOtbjXJHUrUNwtsnBfJdFwJ9U9ZmDmLcp++wnItKS/cSPbdFq+5Y7/U9UdW79N0Qk+yBiq3Mo27S+5hxn6h8jm3sM2YLzmKQPzi34umlD1z1bVU9qZpydRSRdVUvrvXcTzpXjeFXdKCJjcH5fEjJN/e+0CufW9Pk4j0u+i3P3IA3nRFoAVHUeMFlEEoFrcO7ihC6rTt3xK3Qd1TgnN0U4j0Ma0pLj6D7HTRHJAqYDJwKfqmqNiCwMiX0jzqNJRORo4AMRmaOqKxpbgRf1ZD/HOWO8RUQS3aomZ+I8V2iSqm4A3gPuEZFUEYkTkQEictxBxPIFUCEivxSR9iISLyIjROTwZsz7L+ASETnRjSFTRIaoU/ryf8CfRSRZREbhFFY5mPq2C4HTRKSz+yO/oQXzbgK6iEhayLjngN+ISIaIdMW55dpYXC8C14lIb7c+5K11bxzkd/AszvOqs93/66TgPH4oE5FM4OZmfr6XgTNE5GgRScJ53hy6/6bgFMDYISJDcJ7N13kT6CkiN4hIOxFJEZHxDazjReB09ztOxDnQ7MX5flvqEeA2ERkOICJpIvLDg1hOQ/KB4SIyRkSScW4/Npcf26I1961HgD+5B0PcfXvyQcRU36Fs0/padJxpyTHEvYp8BbhdRDqIyDDgxyGTvAkMFpEfucfbRHHq/Q5tYFkbcB5rPCQiAXfaY923U3CuzktFpDPw+wZCv1BEhrknvHcCL7vxpeDsK9twkt03VcNEJElELhCRNFWtwvnN1jawbHCOXz8XkX4i0sldzgvuydgzwHdF5BwRSRCRLu6JADjH0e+722eguy0bswnnmW+djjiJd4sb7yU4z53r4v+hiNRVkyxxp20sfqD5SfYNEdkRMrzazPlQ1UqcpHoqzlnOQ8BFqrrsgDPu6yKcAgSFOB/sZZznCS3i7gBn4DxfWO3G8xjOmVZT836BU5jgXpxbHrP59ixrKs6thfU4hTh+f4Db6wfyFM6PfQ3OgeeF5s7obs/ngFUiUioivXAKY+ThlKZcBHzpjmvIdJwSsPnudK/Ue7+l38EMYBCwUVXzQ8bfAYzD2YZvNbCexj7fEpxSqc/iXBmVAMGQSX6BcwZd4X6WF0LmrcApwHMmzu3Cr4HjG1jHcuBCnHqSW93pz3T34RZR1VeBvwLPi0g5sBjnN3DIVPUrnIPaBzifpdkNEfixLWjdfes+nH3rPRGpwCkE1dBJQoscyjZtYFkHc5xpyTHkGpxbmBtxnik+HrLuCpxn3ue5y9qIsx+2a2RZP8K5Al2GU4K37sT+HzgFsrbibON3G5j3KXf9G3EKAV3njn8S59ZuMc53+lkD61zj/i6uBC5oJLZ/u+uYg7Md9+A8C0dV1+GUNbgJpxT1QmC0O9+9OM+HNwH/h5OQG/MvYJh7zHxNVQuBe3BqKmzCuZsTetfkcOBzEdmBsx9er6qrDrB8pwSu8Y+IrMMpjDHH71iMMaY5RGQWTiG5RltkMw5rVtFHIpKBUzBgjc+hGGOMCYOoSbIiMklElovIChG5tZFpzhGRQnHaA342ZHxfEXlPnPZIC+XQCki0Cvf5zNfAP91bH8YYY2JMVNwuFpF4nNLIJ+E8h5uHU4q0MGSaQTgFLE5wS7d2U9XN7nuzcEp6vu8+QK9V1V1efw5jjDFtS7RcyR6B0+DEKrfgxfM4RcRDXQY8WFfnKyTBDsOpnP6+O36HJVhjjDFe8LQbs0OQyb4Vn4PsX6JwMICIzMWp+H27qr7rji8VkVeAfjilB2/VA3RC0LVrV83Ozm696I0xpg2YP3/+VlXN8DuOSBItSbY5EnCqjEzEadd0joiMdMcfg9OG5zqcqh0X4xTd/oaIXI7TYDR9+/YlLy/Pq7iNMSYmiMjapqdqW6LldnEx+7YI0pv9W0MJAjNUtUpVV+M8wx3kjl/o3mquBl7Dqae5D1Wdpqq5qpqbkWEnYsYYYw5dtCTZecAgt+WPJJyK1jPqTfMazlUsbutGg3F6vpgHpLvVZcDpBagQY4wxJsyiIsm6V6DX4LQasxSnd5AlInKniJzlTjYT2CYihThdad2sqtvcZ6+/AD4UkUU4bVBO9/5TGGOMaWuiogqP13Jzc9WeyRpjTMuIyHxVbaj3oDYrKq5kjTHGmGhkSdYYY4wJE0uyxhhjTJhYko1BwZJdPP3ZWnZVHmof98aYtuSdRRvYWLbH7zBiiiXZGLNzbzUXPz6P37y2mIl/m8XzX6yjuuaAfQobYwxF23dx/QsLuee95X6HElMsycYQVeWW/xSwassObj9zGL0D7bn1lUWcdv/H/HfZJqwkuTGmMX95ZxlxAjeePNjvUGKKJdkY8u+5a3irYAM3nzKEiyf04z9XHcUjF46jqkb5yRN5TJ3+GQXBUr/DNMZEmC9Wb+etRRu48rgB9Exr73c4McWSbIz4YvV27np7KacM786Vx/UHQESYNKIn7/38WO6cPJyvN+3grAfmct1zCyjabh0RGWOgtla5880l9ExL5opjB/gdTsyxJBsDNpfv4epnvySrcwf+9sPRiMg+7yfGx3HRkdnMunki1xw/kPcKN3LiPbP545uFlO6q9ClqY0wkePnLIIuLy7n11CG0T4r3O5yYY0k2ylXV1HL1s1+yY081j/woh9TkxEanTUlO5BenHMasXxzPlLG9+Nfc1Rx790dMm7OSPVWN9vxnjIlRO/ZW87eZyxnbN52zRvfyO5yYZEk2yv357WXMW1PCX34wksHdU5o1T4+0ZO4+ezTvXH8M47IC3PX2Mk68ZzavLSimttYKRxnTVjw8awVbKvby2zOG7XcHzLQOS7JRbEb+ev49dzWXTMhm8pjMFs8/pEcqT1xyBM9cOp70Donc8MJCznrwE/63YmsYojXGRJKi7buY/vFqpozpxbi+Ab/DiVmWZKPUV5squPU/BeRmBfjVaUMPaVkTBnbljWuO5t5zR1Oys4rzH/ucix//guUbK1opWmNMpPnLu06VnVsmDfE7lJhmSTYKVeyp4sqn5tMhKYEHLxhHYvyhf41xccL3xvbmw5uO41enDeHLtSWcet8cfvlygbUAY0yMmbdmO28VbOCKYwfQK92q7IRTgt8BmJZRVW5+qYC123fx7KXj6Z6a3KrLT06M5/JjB/DDnD488NEKnvx0Da/nF3Pp0f254rj+pBygYFUkUVWKS3czf20JC4tKGdErje+Py7TnTqbNq61V7nyjkB6pyVzhVvcz4WNJNspMm7OKd5ds5DenD2V8/y5hW0+gYxK/PWMYFx+Vzd9mLueBj1bw3BfruOG7gzjviL6tcvXcmiqrayncUM78tSXMX7ud+WtL2FS+F4CEOKG6Vnm/cBN/+cFI0jsk+RytMf75z5dBFhWX8Y9zx9AhyVJAuFmn7Q2I1E7b/7diKxf+63NOHdGTB84f6+lVWX5RKXe9vZTPV2+nX9eO/HLSYZwyvIdvV4YlOyudhLquhPlrS8gvKmVvtdNGc2Z6e3KyAuRmBxjXN8Dg7ik88b/V/G3mcrp0bMe9547hyAHhO0ExJlLt3FvNxL/PIjO9Pa9cdRRxca37+7VO2/dnSbYBkZhkN5Tt5oz7PyG9QyKvX3M0ndp5fwaqqvx32Wb+/M4yVmzeQU5WgF+dNoScrM5hXW9trbJq6w7mry0hb42TWFdt2Qk4V6nDM9PI6RsgJ8sZeqQ1fAt9UbCM655fwJptO7l64kCu/+6giLsiNyac/u7elXrlZ0eFpUSxJdn9WZJtQKQl2crqWs6d9ilfbazg9WsmMLBb8+rDhkt1TS0vzQ/y/97/ii0Vezl1RA9umTSEfl07tsryd1VWk19UxpfrSshbs50v15VStrsKgECHRHKyAozLCpDTN8Co3uktaqVm595qbp+xhJfmBxnTJ537zxtL3y4dWiVuYyJZsGQXJ9wzm1NH9OC+88aGZR2WZPdnSbYBkZZkf/f6Yp78dC0PXTCO00b29Ducb+zcW81jH6/m0Tkrqayu5YLxfbnuxEF06dSuRctZ7xZQqhsKN5RT4zaKMbBbJ+cqNdu5Su3ftWOr3KJ+I389v3p1EarwxykjmDK25fWMjYkm1zz7JR8s3cR/b5oYthLFlmT3Z0m2AZGUZF9dEOTnL+Rz2TH9+PXpw/wOp0GbK/Zw3wdf8/y8ItonxnPVxAH8ZEK/Bq8wq2pqWbahgjy3cNKXa0tY71YRSk6MY0yfdOd5alZnxvZND2shpaLtu/j5CwvJW1vC98dmcsfk4VFTetqYlshbs52zH/mU604cxI0nha8rO0uy+/M8yYrIUUA2ISWbVfVJT4NoQqQk2aUbyvneQ3MZ3TudZy4dT0KEPz9csXkHf313Ge8XbqJHajI3njyY7w7tTn5R6TdJNb+ojN1uO8k905K/eY6akxVgaM9Uz5+RVtfU8sBHK7j/w6/pHejA/VPHMqZPuqcxGBNOtbXKlIfmsrl8L//9xXFhLVFsSXZ/niZZEXkKGAAsBOpapFdVvc6zIJohEpJs2e4qznrgE/ZU1fDmtceQkdKyW7B++nzVNu56Zxn5Rd/2XRsfJwzrmbpPUo2kSvDz1mznhucXsql8Dz8/aTBXHjeA+FYueWmMH16eH+QXL+Vz77mj+d7Y3mFdlyXZ/XmdZJcCwzTC71H7nWRra5XLn8pj1vItvHDFd8JeejccVJWZSzayautOxvYJMLpPWsTXySvbVcWvXl3EW4s2cGT/Ltx77phGSyobEw127q3m+L/Pomd6e14NQ5Wd+izJ7s/ro95ioAewweP1RpWHZ6/kg6Wbuf3MYVGZYOHbDuOjSVqHRB44fyzHzc/g9hlLmHTfHP76g1GcMryH36EZc1Aemb2SzRV7efjCnLAnWNMwrx/ydQUKRWSmiMyoG5ozo4hMEpHlIrJCRG5tZJpzRKRQRJaIyLMh42tEZKE7NGt9fvn46y38/b3lTB7Tix8fle13OG2OiHBObh/evPZoegfac8VT8/n1q4vYXWn97ZroEizZxbQ5qzhrdC9ysqyXHb94fSV7+8HMJCLxwIPASUAQmCciM1S1MGSaQcBtwARVLRGRbiGL2K2qYw4+bG8Ul+7muucWMLhbCn/+/khrZ9dH/TM68cpVE/j7e8uZNmcVX6zezv1TxzK0Z6rfoRnTLH99dzkAvzzVetnxk6dXsqo6G1gGpLjDUndcU44AVqjqKlWtBJ4HJteb5jLgQVUtcde1ufUiD789VTVc9fR8qmuUhy8cF/HPL9uCpIQ4fnXaUJ766RGU7q5i8oNzeXzuaiK8SIExzF+7nTfy13PFsf3JjKAChm1R2JOsiPQN+f8c4Avgh8A5wOcicnYzFpMJFIW8DrrjQg0GBovIXBH5TEQmhbyXLCJ57vgpjcR5uTtN3pYtW5oRUuu6441CCoJl3HPOaPpndPJ8/aZxxwzK4N3rj+HogV25441CfvLEPLbu2Ot3WMY0qK6Xne6p7bhy4gC/w2nzvLiSHS8iN7n//xo4XFV/rKoX4Vyh/raV1pMADAImAlOB6SJSV+Exyy3xdj7wDxHZb89T1WmqmququRkZGa0UUvO8mFfEc1+s46qJAzjZCtlEpC6d2vGvH+dy+5nDmLtyG5P+8TFzvvL+ZMyYpry2sJj8YBm/nDTE7ohFgLAnWVV9CdhYt756t3G3NTOGYqBPyOve7rhQQWCGqlap6mrgK5yki6oWu39XAbOA8DTceRAWF5fxm9cWM2FgF24KY0ss5tCJCBdP6MfrV08g0CGRi/79BX96q5BKt/cfY/y2c281f313GaN7pzFljDUVGgk8eSarqs+4/77rliy+WEQuBt4C3m7GIuYBg0Skn4gkAecB9UsJv4ZzFYuIdMW5fbxKRAIi0i5k/ASgkAhQuquSK5+eT5eOSdx/3tiIb9HJOIb2TGXGNUdz4Xf6Mv3j1Xz/4bms3LLD77CM4dHZK9lUvpffnTnMquxECK8LPt0MTANGucM0Vf1lM+arBq4BZgJLgRdVdYmI3CkiZ7mTzQS2iUgh8BFws6puA4YCeSKS747/S2ipZL/U1io3vOC0MPTQBeNa3Ki+8Vf7pHj+OGUk036UQ7DE6YbwxXlFVijK+Ka4dDePzlnFmaN7RW39+lhkHQQ0wIsWn/7xwVf844Ov+eOUEVz4naywrsuE18ayPfz8hYV8umobp4/syV3fG0laB+towHjruucWMHPJRj686Th6B/zpvtFafNqfJ1eyIvKJ+7dCRMpDhgoRKfcihkjy0fLN3Pfh13x/XCYXjO/b9AwmovVIS+bpS8dzy6TDmLlkI6fd/zHz1mz3OyzThsxfu50Z+eu5/Nj+viVY0zCvnske7f5NUdXUkCFFVdtU7f6i7bu44fmFDOmRyp+mWIMTsSI+TvjZxIG8fNVRJMQL5z76Kfe+/xXVNVYoyoRXba1y55tLnSo7x1mVnUjj6TNZEfmOiKSEvE4RkfFexuCnPVU1XPn0fFSVRy4c12B/qya6jemTzlvXHcOUsZnc9+HXnDvtM4q27/I7LNOI0l2VLAqW+R3GIXk9v5j8olJuOWUIHdtZlZ1I43Vx1oeB0GKYO91xMU9V+e1ri1myvpx7zx1DVpeOfodkwqRTuwT+3zljuO+8MSzfWMFp93/MG/nr/Q7LNODumcs584FP+NNbhVF512FXZTV/fWc5o3qn8b2xVmUnEnmdZCW0mztVrcX79pN98fy8Il6aH+S6EwZy4tDufodjPDB5TCZvX3cMAzI6ce1zC/jFS/nsqqz2OywT4su1JaS0S2D6x6s5/7HP2Vyxx++QWuSR2avYWL6H351hVXYilddJdpWIXCciie5wPbDK4xg8l19Uyu9fX8KxgzO4/rvW4ERb0rdLB1668kiuPWEgL88P8tjHq/0Oybh2V9bw9eYdXDwhm3vPHU1BsJTT7/8kagqtFZfu5tHZKzljVE9ys63KTqTyOsleCRyF01pTEBgPXO5xDJ7avrOSq56eT0ZKO+47dwzxdrbZ5iTGx3HTyYcxvFcqn63a5nc4xlW4oYyaWmVkZhrfG9ub166eQMekeM6b9hmPfbwq4us83/3uMhS41XrZiWheN0axWVXPU9VuqtpdVc+Ptt5yWqKmVrn++QVs3VnJwxeOI9Axye+QjI9yswIsLCqNymd/sajALfA0uo/TxPmQHqnMuPZoThzSjT++tZRrnlvAjr2ReXt//toSXl+4nsuPsSo7kc7r0sXJInK1iDwkIv+uG7yMwUv3vv8VH3+9lT9MHs6o3ulNz2BiWk52Z3ZV1rB0Q4XfoRicJNstpR3dU5O/GZeanMijP8rhl5OG8M6iDUx5cC4rNkfW91Vbq/zhzUK6pbTjKutlJ+J5fbv4KaAHcAowG6eh/8jag1vJ+4WbeOCjFZyb24dzD7cGJ4xzJQuQtzY6nvnFuvxgaYMnvyLCVRMH8PRPx1Oys5LJD8zlrYINPkTYsBn561lYVMotk6zKTjTwOskOVNXfAjtV9f+A03Gey8aUNVt3cuOLCxmZmcYdk4f7HY6JEL3S25OZ3p68NSV+h9LmVeypYtWWnYzundboNEcN7Mqb1x3N4B4pXP3sl/zhzUKqfL7Vv6uymr+8s4yRmWl836rsRAWvk2yV+7dUREYAaUA3j2MIq92VToMT8XHCQxeMIznRGpww38rJCpC3dnvEF6qJdYuKneexIw+QZAF6prXnhcuP5MdHZvGvT1ZzwfTP2VzuXzWfR+uq7FgvO1HD6yQ7TUQCOB21z8Dpcu5uj2MIG1XlV68uYvmmCv5x7hj6dLYCCWZfudkBNpXvJViy2+9Q2rS6Qk/NKSuRlBDHHdYpOQgAACAASURBVJNH8I9zx7CouIzT//kJX6z2/pb/+tLdPDpnJaeP6snhVmUnanhduvgxVS1R1dmq2t8tZfyIlzGE0xsFG3h1QTE//+5gJh4WUxfoppXkuM9l56+1W8Z+WhQso3egPZ1bUOJ/ythMXrt6Ap3aJTB1uvfVfO5+dxm1CrdOsio70cSTp+YicuOB3lfV/+dFHOF2yvDu3Dl5OBeOt67rTMOG9EilU7sE5q3ZzhR7puab/GApow+ixP9hPVJ4/ZoJ3PxSPn98aykL1pXy17NH0SnMBZC+XFfCawvXc83xA+0OWZTx6ko2xR1ygauATHe4EhjnUQxh1y4hnouOzLZnJaZR8XHC2L7pdiXro+07KwmW7GZUE89jG5OanMgjF+Zw26lDeGfxBiY/8ElYq/moKne+UUiGVdmJSl51dXeHqt6BU2VnnKrepKo3ATmA1W8xbUpuVmeWb6qgbHdV0xObVlcQLAWaLvR0ICLCFccN4OlLx1O2u4qzHpjLmwXh6QTimyo7pxxmVXaikNcFn7oDlSGvK91xxrQZudkBVGHBOrua9UNBsAwRGJl58Em2zlEDuvLmtccwpEcK1zy7gDvfaN1qPrsra/jLO8sYkZnKD8b1brXlGu94nWSfBL4QkdtF5Hbgc+AJj2Mwxldj+qQTHydWX9YnBcEy+nftSEpyYqssr0daMs9ffiQXH5XNv+eu5vzpn7VaNZ9H56xkQ9kefnfGcHsMFaW8Ll38J+ASoMQdLlHVP3sZgzF+69gugWE9U63lJ58UNNLS06FISojj9rOGc995Y1hcXM5p93/C54fYGcSGst08Mnslp4/syRH9rMpOtPIkyYpIqvu3M7AGp3nFp4C17jhj2pQct7MAv1sQams2le9hc8Xegy701JTJYzJ5/ZoJpCYncP5jnzN9zsFX87n73eVOlR3rZSeqeXUl+6z7dz6QFzLUvTamTcnNDrCnqpbC9eV+h9Km5Bc5hZ7ClWQBBnd3qvmcNLQ7f3p7KT975ksq9rSskNuCdSW8uqCYy47pZ1V2opxXpYvPcP/2cxuhqBv6qWp/L2IwJpLkZjk3cKKlg/BYURAsIz5OGNYzfEkWICU5kYcvHMevThvCe4WbmPzgXL7a1LxqPqrKnW/WVdkZGNY4Tfh5dbt43IEGL2IwJpL0SEumd6C91Zf1WEFxGYO7p9A+KfxtiosIlx87gGcuHU/57iqmPDiXN/KbruYzI389C9aVcvMph4W9kQsTfl59g/cc4D0FTvAoDmMiRm5WgLkrt6GqiFjJ0XBTVQqCpZwyrIen6/1O/y68dd0xXP3Ml1z73AK+XFfCbacOJSlh/2uc3ZU1/PWdZQzvlcrZVmUnJnh1u/j4AwzNSrAiMklElovIChG5tZFpzhGRQhFZIiLP1nsvVUSCIvJAa3wmYw5VTnZntlTspWi7dRbghaLtuyndVcWoPuG9VdyQ7qnJPHf5d7hkQjaPz13D1OmfsamBaj7TP17F+rI9/O4M62UnVnhdTxYRGeEmw4vqhmbMEw88CJwKDAOmisiwetMMAm4DJqjqcOCGeov5AzCnVT6EMa3AOnH3VkGxU+jpYNosbg2J8XH8/szh3D91LEs3lHP6/R/z6cpvq/lsLNvDw7NWctrIHozv38WXGE3r8zTJisjvgX+6w/E43dyd1YxZjwBWqOoqVa0Engcm15vmMuBBVS0BUNXNIevNwWlZ6r1D/hDGtJLB3VNISU5gnjVK4YmCYBlJ8XEM7p7iaxxnje7F61dPILV9Ihf+63Menb0SVeXud5dRo8ptpw71NT7Tury+kj0bOBHYqKqXAKNxOm5vSiZQFPI66I4LNRgYLCJzReQzEZkEICJxOM+Ef3GgFYjI5SKSJyJ5W7Zsad6nMeYQxMcJ4/oGmG9Xsp4oCJYytFdqg89CvTaoewqvXz2Bk4d158/vLOOCxz7nlQXFXHq0VdmJNV7vbbtVtRaodhuo2Az0aaVlJwCDgInAVGC6iKQDPwPeVtXggWZW1WmqmququRkZGa0UkjEHlpsV4KtNOyjbZZ0FhFNtrbK4uJxRrdBecWtJSU7koQvG8evThvL56u107dSOnx1vVXZijdflw/PcxDcdpyGKHcCnzZivmH2TcW93XKgg8LmqVgGrReQrnKR7JHCMiPwM6AQkicgOVW2w8JQxXsrJdp7LfrmuhOOHdPM5mti1ausOduytDmsjFAdDRLjs2P5MGNiVxHixKjsxyNNvVFV/5v77iIi8C6SqakEzZp0HDBKRfjjJ9Tzg/HrTvIZzBfu4iHTFuX28SlUvqJtARC4Gci3Bmkgxpk86CXHCvDXbLcmGUUGwDIDRffwp9NSUYb1S/Q7BhInXBZ9miMj5ItJRVdc0M8GiqtXANcBMYCnwoqouEZE7RaSu4NRMYJuIFAIfATer6qG10G1MmHVISmB4r1TyrFGKsCoIltEhKZ4BGZ38DsW0MV7fm7gHOBf4s4jMwykl/KaqNtkvlKq+Dbxdb9zvQv5X4EZ3aGwZT2Bd65kIk5PVmWc+X0tldW1EFMqJRfnBUkb0SiPe6p4aj3nd1d1s95Zxf+BR4Bycwk/GtFm52QH2VteyZH2Z36HEpKoapyOGSHsea9oGPxqjaA/8ALgSOBz4P69jMCaSfNMohdWXDYuvNlWwt7qWkZZkjQ+8fib7Is4z1ROAB4ABqnqtlzEYE2m6pSbTt3MHa/kpTBbVFXryqaUn07Z5/Uz2X8BUVa3xeL3GRLTcrABzvt5inQWEQX6wjNTkBLK6WCMPxnteP5OdaQnWmP3lZAfYuqOStdt2+R1KzCkIljKqd7qdvBhfWFFGYyLA4dnWiXs47KmqYfnGCiv0ZHxjSdaYCDAwoxOpyQnWiXsrW7qhnOpatSRrfON1wScRkQtF5Hfu674icoSXMRgTieLihJysgDVK0coWFTuFnkZZoSfjE6+vZB/CaUt4qvu6AqefWGPavNzszqzYvIOSnZV+hxIz8ovK6NqpHT3Tkv0OxbRRXifZ8ap6NbAHwO37NcnjGIyJSHX1Ze2WcetxCj2lWaEn4xuvk2yViMQDCiAiGUCtxzEYE5FG90knMV7slnEr2bm3mhVbdtjzWOMrr5Ps/cCrQDcR+RPwCXCXxzEYE5GSE+MZ3ivNOnFvJYuLy1DFkqzxlddd3T0jIvOBEwEBpqjqUi9jMCaS5WYFePKzteytrqFdQrzf4US1uu7trNCT8ZMnV7Ii0rluwOkQ4DngWWCTO84Yg1P4qbK6lsXF1lnAoSooLiMzvT1dO7XzOxTThnl1JTsf5zmsAH2BEvf/dGAd0M+jOIyJaDkhnQXkZNn556EoCJYyMtNuFRt/eXIlq6r9VLU/8AFwpqp2VdUuwBnAe17EYEw0yEhpR3aXDlb46RCV7api7bZdjOpjSdb4y+uCT99xO18HQFXfAY7yOAZjIlpOVme+XFuCqvodStQqKC4FrOcd4z+vk+x6EfmNiGS7w6+B9R7HYExEy80OsG1nJau37vQ7lKhVV+hphN0uNj7zOslOBTJwqvG84v4/9YBzGNPGHJ5tnbgfqoJgKf26diStfaLfoZg2zusqPNuB671cpzHRpn/XTqR3SCRv7XbOObyP3+FEpYJg2Tc9GxnjJ+uFx5gIExcn5PS1zgIO1uaKPWwo22ONUJiIYEnWmAiUkx1g1ZadbNux1+9Qos4i93ns6D5W6Mn4z5KsMRGo7landRbQcvnBMuIEhvdK9TsUYzzvT7a3iLwqIltEZLOI/EdEensZgzHRYGRmGknxcZZkD8KiYCmDuqXQIcnTIifGNMjrK9nHgRlAT6AX8IY7rkkiMklElovIChG5tZFpzhGRQhFZIiLPuuOyRORLEVnojr+ylT6LMWGTnBjPiMxUey7bQqpKQbDMnseaiOF1ks1Q1cdVtdodnsCpxnNAbvd4DwKnAsOAqSIyrN40g4DbgAmqOhy4wX1rA3Ckqo4BxgO3ikivVvtExoRJbnZnFgXL2FNV43coUaO4dDfbdlZakjURw+sku01ELhSReHe4ENjWjPmOAFao6ipVrQSeBybXm+Yy4EG3I3hUdbP7t1JV60qPtMOeQ5sokZsVoLKmlkXWWUCzLbKed0yE8Trh/AQ4B9iIc4V5NnBxM+bLBIpCXgfdcaEGA4NFZK6IfCYik+reEJE+IlLgLuOvqrpfK1MicrmI5IlI3pYtW1rymYwJi9DOAkzz5AfLSIwXhvRM8TsUYwDvk2xvVT1LVTNUtZuqTsHplac1JACDgIk4rUhNF5F0AFUtUtVRwEDgxyLSvf7MqjpNVXNVNTcjo8k72MaEXZdO7ejftaN14t4Ci4pLGdIj1friNRHD6yT7z2aOq68YCG36prc7LlQQmKGqVaq6GvgKJ+l+w72CXQwc0+yIjfFRTlaA+WtLqK21zgKaUltrhZ5M5PGq0/YjReQmIENEbgwZbgeac8o5DxgkIv1EJAk4D6eUcqjXcK5iEZGuOLePV7nVhtq74wPA0cDy1vhcxoTb4dmdKdlVxaqtO/wOJeKt2baTij3VlmRNRPHqSjYJ6IRzSzclZCjHeS57QKpaDVwDzASWAi+q6hIRuVNEznInm4lTsKoQ+Ai4WVW3AUOBz0UkH5gN/F1VF7XqpzMmTHKss4BmqysgZoWeTCTxpLa2qs4GZovIE6q69iCX8Tbwdr1xvwv5X4Eb3SF0mveBUQezTmP81r9rRzp3TCJvbQnnHdFaxRdiU35RGcmJcQzq1snvUIz5htdNorQTkWlAdui6VfUEj+MwJiqICOP6Bqzlp2YoCJYyvFcaCfFWS89EDq+T7EvAI8BjgNWwN6YZDs8O8MHSTWyp2EtGSju/w4lI1TW1LFlfznlHWNeAJrJ4nWSrVfVhj9dpTFTLdZ/Lzl9bwqQRPXyOJjKt2LKD3VU1VujJRByv76u8ISI/E5GeItK5bvA4BmOiyojMNJIS4qy+7AEUWEtPJkJ5fSX7Y/fvzSHjFOjvcRzGRI12CfGMykyzzgIOoCBYSkq7BPp16eh3KMbsw9Mkq6r9vFyfMbEiN7sz//pkFXuqakhOtNaM6isIljEiM424OPE7FGP24WmSFZGLGhqvqk96GYcx0SY3K8Ajs5X8olLG9+/idzgRZW91DUs3lPOTo+0c3kQer28XHx7yfzJwIvAlYEnWmAP4prOAtSWWZOtZvrGCqhplVKY9jzWRx+vbxdeGvnYb8H/eyxiMiUaBjkkMyOho9WUb8G2hJytZbCKP37W2dwJ2j8eYZjg8u7N1FtCAgmApnTsm0TvQ3u9QjNmP189k38ApTQxOgh8GvOhlDMZEq5ysAM/PK2LFlh0M7m79pdYpCJYxMjMNESv0ZCKP189k/x7yfzWwVlWDHsdgTFTKzXaqlOetKbEk69pdWcNXmyo4edh+XUQbExG8fiY728v1GRNLsrt0oEvHJPLWbuf88dZZAMCS9WXUqjVCYSKXV/3JfuL+rRCR8gaG1SLyMy9iMSZaiQi52dZZQKh8K/RkIpwnSVZVj3b/pqhqav0ByAWu9yIWY6JZblZn1m7bxeaKPX6HEhEWBUvpkZpMt9Rkv0MxpkGely4WkXgR6SUifesGt3P1iV7HYky0qevEfb514g64hZ7sKtZEME+TrIhcC2wC3gfecoc3AVR1g5exGBONRvRKo11CnLVjDJTvqWLV1p2MtiRrIpjXpYuvBw5zr1yNMS2UlBDH6N7plmSBxdbzjokCXt8uLgLKPF6nMTElNzvAkuIydlfW+B2Kr+oKPY3MtCtZE7m8TrKrgFkicpuI3Fg3eByDMVEtNztAda2ysKjU71B8tai4lL6dOxDomOR3KMY0yuskuw7neWwSkBIyGGOaaVxft/BTG+/EPb+ozKrumIjndWMUdwCISCf39Q4v129MLEjvkMSgbp3a9HPZbTv2Uly6mx8fleV3KMYckNeli0eIyAJgCbBEROaLyHAvYzAmFuS28c4CCoqt0JOJDl7fLp4G3KiqWaqaBdwETPc4BmOiXm5WgIo91Xy1ucLvUHxRUFSGCIywQk8mwnmdZDuq6kd1L1R1FtDR4xiMiXq5bqMUeW20UYpFxaUMyOhEp3Ze10I0pmU8L10sIr8VkWx3+A1OieMmicgkEVkuIitE5NZGpjlHRApFZImIPOuOGyMin7rjCkTk3Fb8PMb4om/nDnTt1K5NtmOsquQHrdCTiQ5enwb+BLgDeAWnX9mP3XEHJCLxwIPASUAQmCciM1S1MGSaQcBtwARVLRGRbu5bu4CLVPVrEekFzBeRmaratus/mKgmIhyeHWDemrZXwnhj+R62VOxllN0qNlHA69LFJcB1BzHrEcAKVV0FICLPA5OBwpBpLgMedNeBqm52/34Vsv71IrIZyAAsyZqolpMV4J3FG9lUvofubaiB/IK6lp76WKEnE/m8Ll38voikh7wOiMjMZsyaidNaVJ2gOy7UYGCwiMwVkc9EZFID6z8Cp47uygbeu1xE8kQkb8uWLc35OMb4KrQT97akIFhKQpwwrGeq36EY0ySvn8l2Db1N6151djvA9C2RAAzC6c1nKjC9XkLvCTwFXKKqtfVnVtVpqpqrqrkZGRmtFJIx4TO8VyrJiXHktbFGKQqCZQzunkJyYrzfoRjTJK+TbK2I9K17ISJZOM9mm1IM9Al53dsdFyoIzFDVKlVdDXyFk3QRkVScHn9+raqfHUL8xkSMxPg4xvRJb1OFn1SVRcVljO5jz2NNdPA6yf4a+EREnhKRp4E5OIWVmjIPGCQi/UQkCTgPmFFvmtdw+6QVka44t49XudO/Cjypqi+3zscwJjLkZnVmyfpydu6t9jsUT6zbvovSXVWMzLTnsSY6eJpkVfVdYBzwAvA8kKOqTT6TVdVq4BpgJrAUeFFVl4jInSJyljvZTGCbiBQCHwE3u13qnQMcC1wsIgvdYUyrfzhjfJCTHaCmVslvI50FfFPoyarvmCjheU1uVd2K21F7C+d7G3i73rjfhfyvwI3uEDrN08DTBxWsMRFuXN8AIpC3toSjBnb1O5ywKwiWkpQQx2E9rF8REx28vl1sjGlFae0TOax7SpvpLCA/WMawnqkkxtuhy0QH21ONiXI5WQG+XFtCTYx3FlBTqywpLmO03So2UcSTJCsinQ80eBGDMbEqNzvAjr3VLN8Y250FrNqyg52VNdbzjokqXj2TnY9TVUcaeE+B/h7FYUzMyc1yzlPnr93OsF6x20CDFXoy0ciTJKuq/bxYjzFtUe9Ae7qntiNvbQk/OjLb73DCpiBYSsekePpndPI7FGOazetmFUVELhSR37qv+7pNHRpjDpKIkJvVOeabV8wPljE8M434uIZuiBkTmbwu+PQQcCRwvvu6Aqd3HWPMIcjJClBcupsNZbv9DiUsqmpqKdxQboWeTNTxOsmOV9WrgT3wTdvFSR7HYEzMifVO3JdvrKCyutYKPZmo43WSrXL7hlUAEckA9mus3xjTMsN6ptIhKT5m2zG2Qk8mWnmdZO/HaUe4m4j8CfgEuMvjGIyJOQluZwGx2on7ouJS0jsk0rdzB79DMaZFvO60/RkRmQ+ciFOdZ4qqLvUyBmNiVW5WgAc+WsGOvdV0aud5i6lhlV9UxsjMNESs0JOJLn60XbwMWOb1eo2JdTnZnalVWLiulKMHxU47xnuqavhqUwVXDLHq9Cb6WLOKxsSIsX3T3c4CYuuWceGGcqpr1Qo9mahkSdaYGJGanMiQHqkxV8K4wO3Gzwo9mWhkSdaYGJKbFWDBuhKqa2Kn0H5BcRkZKe3okZrsdyjGtJjXLT59X0S+FpEyESkXkQoRKfcyBmNiWW52gJ2VNSyLoc4CCoJOzztW6MlEI6+vZO8GzlLVNFVNVdUUVY3dFs2N8VhOltMoRazUl92xt5qVW3YwMtOex5ro5HWS3WRVdowJn8z09vRMS46Z+rKLi8tQhVF97HmsiU5eV+HJE5EXgNeAvXUjVfUVj+MwJiaJCDlZgZi5ki0IuoWeMi3Jmujk9ZVsKrALOBk40x3O8DgGY2JablaADWV7KC6N/s4CCoJlZKa3p0undn6HYsxB8brFp0u8XJ8xbVFuttOJe96a7WSOyfQ5mkNTECxjtN0qNlHM69LFg0XkQxFZ7L4eJSK/8TIGY2LdkB4pdIyBzgJKdlaybvsuK/RkoprXt4unA7cBVQCqWgCc53EMxsS0hPg4xvYNMC/KG6VYVOz0vGN9yJpo5nWS7aCqX9QbV+1xDMbEvJysAMs3llOxp8rvUA5aXaGnEZZkTRTzOsluFZEBfNuf7NnAhubMKCKTRGS5iKwQkVsbmeYcESkUkSUi8mzI+HdFpFRE3myND2FMpMvNDlCrsGBdqd+hHLSCYBn9u3YkNTnR71CMOWheV+G5GpgGDBGRYmA1cEFTM7kdvT8InAQEgXkiMkNVC0OmGYRzK3qCqpaISLeQRfwN6ABc0WqfxJgINrZvgDiBvLUlHDs4w+9wDkpBsIzv9O/sdxjGHBKvSxevAr4rIh2BOFVtbttvRwAr3PkRkeeByUBhyDSXAQ+qaom7rs0h6/1QRCa2wkcwJip0apfA0J6p5EVpoxSby/ewsXwPI63nHRPlfOkgQFV3tiDBAmQCRSGvg+64UIOBwSIyV0Q+E5FJLYlJRC4XkTwRyduyZUtLZjUmIuVmBVhYVBqVnQUUBK3Qk4kNsdQLTwIwCJgITAWmi0izT4NVdZqq5qpqbkZGdN5eMyZUTnZndlXWsHRD9HUWUBAsJU5geC9Lsia6eZZkRSRORI46yNmLgT4hr3u740IFgRmqWqWqq4GvcJKuMW3S4dlOZwHR2Il7QXEZg7un0D4p3u9QjDkkniVZVa3FKbx0MOYBg0Skn4gk4dStnVFvmtdwrmIRka44t49XHeT6jIl6PdPak5nePuo6cVdVCoJl1km7iQle3y7+UER+IC3sGFJVq4FrgJnAUuBFVV0iIneKyFnuZDOBbSJSCHwE3Kyq2wBE5GPgJeBEEQmKyCmt9YGMiWQ5WQHy1m5HVf0OpdmCJbvZvrOSUVboycQAr6vwXAHcCNSIyG5AAG1On7Kq+jbwdr1xvwv5X91l39jAvMccYtzGRKXc7AAz8tcTLNlNn84d/A6nWepaerIrWRMLvK7Ck+Ll+oxp63KznHqm89eWRE2SzQ+WkhQfx2E97HBhop/npYtF5CwR+bs7WDd3xoTRYT1SSGmXEFWduBcUlTGkZwrtEqzQk4l+XvfC8xfgepxGJAqB60Xkz17GYExbEh8njOmbHjU98tTWKouLrdCTiR1eX8meBpykqv9W1X8Dk4DTPY7BmDYlN6szyzdVULY78jsLWL1tJxV7q63Qk4kZfjRGEfrrsdNVY8Ls8OwAqrBgXeRfzS4KWqEnE1u8TrJ3AQtE5AkR+T9gPvAnj2Mwpk0Z0zed+DiJivqy+cFS2ifGMzCjk9+hGNMqPCtdLCJxQC3wHeBwd/QvVXWjVzEY0xZ1SEpgWM/UqGj5qSBYxojMVBLiY6nFV9OWed3i0y2qukFVZ7iDJVhjPJDjdhZQFcGdBVTX1LJkfRkjM+15rIkdXp8ufiAivxCRPiLSuW7wOAZj2pzDszuzp6qWwvXlfofSqK8372BPVS2j+9jzWBM7vG7x6Vz379Uh4xTo73EcxrQpud90FlDC6D6ReaVYECwFYGSmJVkTOzzthQe4VVX71RsswRoTZt1Tk+kdaB/RnbgXBMtISU4gu0tHv0MxptV4/Uz2Zq/WZ4zZV25WgLy1JRHbWUBdzztxcS3qP8SYiGbPZI1pI3KyO7OlYi9F23f7Hcp+9lbXsGxjuRV6MjHHnska00aEduLet0tkdRawbEMFVTXKaGuEwsQYr3vh6efl+owx3xrcLYWU5ATmrSnh++N6+x3OPuoKPY2K0EJZxhwsT24Xi8gtIf//sN57d3kRgzFtXVycMK5vgPkR2ChFQbCMLh2T6JWW7HcoxrQqr57Jnhfy/2313pvkUQzGtHm5WQG+2rSDsl2R1VlAXaEnESv0ZGKLV0lWGvm/odfGmDAZ378LABc/8QUfLdscESWNd1VW8/XmCkZazzsmBnmVZLWR/xt6bYwJk8OzA9z1vZFsLt/LJU/M44x/fsK7izdQW+vfz3DJ+nJqFSv0ZGKSVwWfRotIOc5Va3v3f9zX9hDGGI+ICOeP78vZOb15bWExD89ayZVPf8mgbp24+viBnDGqp+eN8+cXuS09WZI1MciTX5OqxqtqqqqmqGqC+3/d60QvYjDGfCspIY5zcvvwwY3Hcf/UscSJcMMLCznhntk898U69lbXeBbLouIyeqYl0y3FzrdN7LH+pIxpw+LjhLNG9+Kd649h2o9ySO+QyG2vLGLi32bx+NzV7K4Mf7KtK/RkTCyyJGuMIS5OOHl4D16/egJP/uQI+gQ6cMcbhRxz9395eNZKduytDst6y3ZXsXrrTkZZoScTo7xu8ckYE8FEhGMHZ3Ds4Aw+X7WNBz5awV/fXcYjs1dy8VHZXDIhm/QOSa22vsXFZQB2JWtiliVZY0yDxvfvwvj+XcgvKuWBj1Zw34df89jHq7jwyCwuPbo/GSntDnkd+XUtPVmbxSZGRc3tYhGZJCLLRWSFiNzayDTniEihiCwRkWdDxv9YRL52hx97F7Ux0W90n3SmX5TLuzccwwlDuzN9ziqO/ut/uX3GEtaXHlpnA4uCZWR16UBaByv/aGKTREJl9KaISDzwFXASEATmAVNVtTBkmkHAi8AJqloiIt1UdbPby08ekItTJ3c+kKOqJY2tLzc3V/Py8sL3gYyJYqu37uThWSt45ctiROAH43pz1cQBZB1EP7AT/vJfxmUF+OfUsWGI1HhNROaraq7fcUSSaLmSPQJYoaqrVLUSeB6YXG+ay4AH65Knqm52x58CvK+q29333seacjTmoPXr2pG7zx7NrJsnMvWIvryyoJjjzHXWRwAACntJREFU/z6LG55fwNebKpq9nK079lJcutsaoTAxLVqSbCZQFPI66I4LNRgYLCJzReQzEZnUgnkRkctFJE9E8rZs2dKKoRsTm3oHOnDn5BF8csvx/PTofrxXuImT7p3DlU/N/6ZA04EsCjrTjMy0JGtiV7Qk2eZIAAYBE4GpwHQRaXZpClWdpqq5qpqbkZERphCNiT3dUpP59enDmPvLE7juhIHMXbmVM/75CRc//gV5axrv8Sc/WEqcwAhLsiaGRUuSLQb6hLzu7Y4LFQRmqGqVqq7GeYY7qJnzGmMOUaBjEjeefBhzbz2Bm085jIJgGWc/8innTfuUuSu27tcZQUGwjIHdOtGxnVVyMLErWpLsPGCQiPQTkSScrvNm1JvmNZyrWESkK87t41XATOBkEQmISAA42R1njAmD1ORErj5+IJ/88nh+c/pQVm/dyQWPfc73HvofHy7dhKqiqhQEyxhpVXdMjIuKU0hVrRaRa3CSYzzwb1VdIiJ3AnmqOoNvk2khUAPcrKrbAETkDziJGuBOVY28XquNiTEdkhK49Jj+/OjILF6eH+ThWSv56f/lMbRnKlOP6MPWHXsZ3cduFZvYFhVVeLxmVXiMaX1VNbXMWLieB2etYNWWnQC8+rOjGNs34HNkprVYFZ79RcWVrDEm+iXGx/GDnN5MGZvJO4s3sLi43EoWm5hnSdYY46n4OOGMUb04Y1Qvv0MxJuyipeCTMcYYE3UsyRpjjDFhYknWGGOMCRNLssYYY0yYWJI1xhhjwsSSrDHGGBMmlmSNMcaYMLEka4wxxoSJNavYABHZAqz1O45D1BXY6ncQEcS2x75se3zLtsW+DmV7ZKmq9RUawpJsjBKRPGtD9Fu2PfZl2+Nbti32ZdujddntYmOMMSZMLMkaY4wxYWJJNnZN8zuACGPbY1+2Pb5l22Jftj1akT2TNcYYY8LErmSNMcaYMLEka4wxxoSJJdkYIyJ9ROQjESkUkSUicr3fMflNROJFZIGIvOl3LH4TkXQReVlElonIUhE50u+Y/CQiP3d/J4tF5DkRSfY7Ji+JyL9FZLOILA4Z11lE3heRr92/AT9jjHaWZGNPNXCTqg4DvgNcLSLDfI7Jb9cDS/0OIkLcB7yrqkOA0bTh7SIimcB1QO7/b+/OY6yszjiOf38syo6gLcFaCxpFjWsFY4VQFE1U3J1o44ZLYkxabW0mFlNjh6Rppo02RI3aimEQtxgXVDQRAlpQXEZxGJRKTJEgOqKGRSFFizz94zzXeXO9pAi9c+beeT7Jm3ve/XnfzMzznvPeOcfMjgR6A7/IG1WXawFOL1s2DVhoZocAC30+7KZIsnXGzDrMbJmXvyT9Ef1R3qjykXQAMAWYmTuW3CQNBSYC9wOY2ddmtilvVNn1AfpL6gMMAD7OHE+XMrPFwIayxecCs708GzivS4OqM5Fk65ikUcBxwOt5I8lqBnATsCN3IN3AaOAzYJY3n8+UNDB3ULmY2UfAbcBaoAPYbGbz80bVLYwwsw4vfwKMyBlMrYskW6ckDQKeAH5jZl/kjicHSWcBn5rZW7lj6Sb6AD8F7jGz44Ct9OCmQH/XeC7p4WN/YKCky/JG1b1Y+h/P+D/PPRBJtg5J6ktKsA+Z2ZO548loPHCOpDXAo8Apkh7MG1JW64B1ZlZq2XiclHR7qlOBD8zsMzP7D/AkcFLmmLqD9ZJGAvjnp5njqWmRZOuMJJHeuf3TzP6aO56czOxmMzvAzEaRvtCyyMx6bE3FzD4BPpQ0xhdNBlZmDCm3tcCJkgb4781kevAXwQqeAaZ6eSrwdMZYal4k2fozHricVGtr8+nM3EGFbuN64CFJ7cCxwJ8yx5ON1+gfB5YBK0h/D3tUl4KSHgFeBcZIWifpGqAZOE3S+6TafnPOGGtddKsYQgghVEnUZEMIIYQqiSQbQgghVEkk2RBCCKFKIsmGEEIIVRJJNoQQQqiSSLKhJkkySbcX5hslNWUMaZdIWiNpv9xxdDVJTZIavXylpP1zxxRCV4gkG2rVV8AFuRKWdygfds+VpG4MQ6h7kWRDrdpO6jjgxvIVklokNRTmt/jnJEn/kPS0pNWSmiVdKukNSSskHezb/UDSE5JafRrvy5skzZH0CjBH0ihJiyS1S1oo6cAKsewrab6PWToTUGHdZX7uNkl/k9S7wv7jJC2VtNy3HeznXSJpmU8nFa5vsaTnJK2SdK+kXr7uHklvehzTC8dv9rGH2yXdVuH8wyXN9fWvSTralw+SNMvvW7ukC4v32ssNklrKjtcAjCV1iNEmqb+kW/0+vyPp7977EpJuKMT26Hd+AkKoBWYWU0w1NwFbgCHAGmAo0Ag0+boWoKG4rX9OAjYBI4G9gY+A6b7u18AMLz8MTPDygaQuKgGagLeA/j7/LDDVy1cDcyvEeQdwq5enkDpb3w843Pfv6+vuBq4o23cvYDUwzueHkDr5HwD082WHAG8Wrm8bcBBpbNQFpfsADPfP3sBLwNHAvsAqOjul2adC/HcCf/DyKUCbl/9cul8+P6x4r73cALQU7l2jl18ijeFKMTYvzwHO9vLHwN47iy2mmGphiiavULPM7AtJD5AG3v73Lu7Waj6Ml6R/AaWhzVYAJ3v5VOAIr1ABDPFRjQCeMbPSuX4GXODlOcBfKpxvYmkbM3tO0kZfPhk4Hmj18/Tnux2xjwE6zKy1dL0e90DgLknHAt8Ahxb2ecPMVvt2jwATSF0HXiTpWlKSHgkcQeq3eBtwv6R5wLwK8U8ALvTzL/Ka+RC/R98OcG5mGyvsu6tOlnQT6eFhOPAu6QGknVTjnQvM3YPjh5BNJNlQ62aQ+p6dVVi2HX8V4s2lexXWfVUo7yjM76Dz96EXcKKZbSueyJPh1v9T3AJmm9nNu7HvjcB64BhSrMU4y/tJNUmjSTX9cWa20Ztw+5nZdkknkBJ+A/ArUm11TxTP3+9/bSypH6kWP9bMPvQvr5X2m0J6SDkb+L2ko8xs+x7GF0KXineyoaaZ2QbgMeCawuI1pFoiwDlA3+952PmkjvQB8BpjJUvprM1dCiypsM1i4BI/zhnAMF++EGiQ9ENfN1zST8r2XQWMlDTOtxnsX7gaSqrh7iANBlF8l3uCpNH+cHEx8DKpmXkrsFnSCOAMP94gYKiZPU9K3MdUiH+JXxuSJgGfe416AfDLwj0qXdd6SYf7+c+vcDyAL4HBXi4l1M89ngY/Xi/gx2b2IvA7v+ZB5QcKobuLJBvqwe2k95wl9wE/l7Sc1KT7fWufNwBj/Qs3K4HrdrLd9cBVSiPaXE56r1tuOjBR0rukZuO1AGa2ErgFmO/7LyA1437LzL4mJco7/VoWkJLS3cBUX3ZY2fW1AneRhmz7AHjKzJYDbwPvkd43v+LbDgbm+flfBn5bIf4m4HjfppnOIdD+CAzzLystp7OpfRqp2Xkp0FH5ttEC3CupjdSScB/wDvCCxw/pweFBSSs89jvMbNNOjhdCtxWj8IRQJ7ym2WhmZ+WOJYSQRE02hBBCqJKoyYYQQghVEjXZEEIIoUoiyYYQQghVEkk2hBBCqJJIsiGEEEKVRJINIYQQquS/DJIWse6xQFsAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "A pesar que concatenar celdas de LSTM disminuye el error en el conjunto de validación, no se justifica en términos de tiempo de cómputo, ya que las diferencias de error son muy pequeñas. Por lo tanto, no se decide concatenar celdas. "
      ],
      "metadata": {
        "id": "SIllo9ORfA8-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##### Probando el número de capas ocultas"
      ],
      "metadata": {
        "id": "ssJzLpl0tUqF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Creamos nuestro modelo.\n",
        "validation_values = []\n",
        "HIDDEN_DIM = 2**7 # dimensión de la capas LSTM\n",
        "for i in range(1,11):\n",
        "  N_LAYERS = i\n",
        "  model = NER_RNN(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM,\n",
        "                          N_LAYERS, BIDIRECTIONAL, DROPOUT, PAD_IDX)\n",
        "\n",
        "  baseline_model_name = 'baseline'  # nombre que tendrá el modelo guardado...\n",
        "\n",
        "  model_name = baseline_model_name\n",
        "  criterion = baseline_criterion\n",
        "  n_epochs = baseline_n_epochs\n",
        "\n",
        "  model.apply(init_weights)\n",
        "  print(f'El modelo actual tiene {count_parameters(model):,} parámetros entrenables.')\n",
        "  # Optimizador\n",
        "  optimizer = optim.Adam(model.parameters())\n",
        "\n",
        "  # Enviamos el modelo y la loss a cuda (en el caso en que esté disponible)\n",
        "  model = model.to(device)\n",
        "  criterion = criterion.to(device)\n",
        "\n",
        "\n",
        "  best_valid_loss = float('inf')\n",
        "\n",
        "  for epoch in range(n_epochs):\n",
        "\n",
        "      start_time = time.time()\n",
        "\n",
        "      # Recuerdo: train_iterator y valid_iterator contienen el dataset dividido en batches.\n",
        "\n",
        "      # Entrenar\n",
        "      train_loss, train_precision, train_recall, train_f1 = train(\n",
        "          model, train_iterator, optimizer, criterion)\n",
        "\n",
        "      # Evaluar (valid = validación)\n",
        "      valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "          model, valid_iterator, criterion)\n",
        "\n",
        "      end_time = time.time()\n",
        "\n",
        "      epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
        "\n",
        "      # Si obtuvimos mejores resultados, guardamos este modelo en el almacenamiento (para poder cargarlo luego)\n",
        "      # Si detienen el entrenamiento prematuramente, pueden cargar el modelo en el siguiente recuadro de código.\n",
        "      if valid_loss < best_valid_loss:\n",
        "          best_valid_loss = valid_loss\n",
        "          torch.save(model.state_dict(), '{}.pt'.format(model_name))\n",
        "      # Si ya no mejoramos el loss de validación, terminamos de entrenar.\n",
        "\n",
        "      print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
        "      print(\n",
        "          f'\\tTrain Loss: {train_loss:.3f} | Train f1: {train_f1:.2f} | Train precision: {train_precision:.2f} | Train recall: {train_recall:.2f}'\n",
        "      )\n",
        "      print(\n",
        "          f'\\t Val. Loss: {valid_loss:.3f} |  Val. f1: {valid_f1:.2f} |  Val. precision: {valid_precision:.2f} | Val. recall: {valid_recall:.2f}'\n",
        "      )\n",
        "  # cargar el mejor modelo entrenado.\n",
        "  model.load_state_dict(torch.load('{}.pt'.format(model_name)))\n",
        "\n",
        "  # Limpiar ram de cuda\n",
        "  torch.cuda.empty_cache()\n",
        "\n",
        "  valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "      model, valid_iterator, criterion)\n",
        "\n",
        "  validation_values += [valid_loss]\n",
        "\n",
        "  print(\n",
        "      f'Val. Loss: {valid_loss:.3f} |  Val. f1: {valid_f1:.2f} | Val. precision: {valid_precision:.2f} | Val. recall: {valid_recall:.2f}'\n",
        "  )"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vIANr28FtYgx",
        "outputId": "7c807f71-1d13-4e7c-daab-777ee3e20bc5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "El modelo actual tiene 5,294,784 parámetros entrenables.\n",
            "to device cuda!!!!\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch: 01 | Epoch Time: 0m 5s\n",
            "\tTrain Loss: 1.320 | Train f1: 0.05 | Train precision: 0.28 | Train recall: 0.04\n",
            "\t Val. Loss: 0.895 |  Val. f1: 0.10 |  Val. precision: 0.44 | Val. recall: 0.06\n",
            "Epoch: 02 | Epoch Time: 0m 5s\n",
            "\tTrain Loss: 0.903 | Train f1: 0.27 | Train precision: 0.42 | Train recall: 0.20\n",
            "\t Val. Loss: 0.702 |  Val. f1: 0.42 |  Val. precision: 0.56 | Val. recall: 0.34\n",
            "Epoch: 03 | Epoch Time: 0m 5s\n",
            "\tTrain Loss: 0.740 | Train f1: 0.42 | Train precision: 0.52 | Train recall: 0.36\n",
            "\t Val. Loss: 0.593 |  Val. f1: 0.52 |  Val. precision: 0.60 | Val. recall: 0.46\n",
            "Epoch: 04 | Epoch Time: 0m 5s\n",
            "\tTrain Loss: 0.638 | Train f1: 0.49 | Train precision: 0.57 | Train recall: 0.44\n",
            "\t Val. Loss: 0.549 |  Val. f1: 0.55 |  Val. precision: 0.61 | Val. recall: 0.50\n",
            "Epoch: 05 | Epoch Time: 0m 5s\n",
            "\tTrain Loss: 0.575 | Train f1: 0.54 | Train precision: 0.61 | Train recall: 0.49\n",
            "\t Val. Loss: 0.521 |  Val. f1: 0.56 |  Val. precision: 0.63 | Val. recall: 0.51\n",
            "Epoch: 06 | Epoch Time: 0m 5s\n",
            "\tTrain Loss: 0.533 | Train f1: 0.59 | Train precision: 0.66 | Train recall: 0.53\n",
            "\t Val. Loss: 0.503 |  Val. f1: 0.63 |  Val. precision: 0.72 | Val. recall: 0.56\n",
            "Epoch: 07 | Epoch Time: 0m 5s\n",
            "\tTrain Loss: 0.495 | Train f1: 0.63 | Train precision: 0.70 | Train recall: 0.57\n",
            "\t Val. Loss: 0.487 |  Val. f1: 0.66 |  Val. precision: 0.75 | Val. recall: 0.60\n",
            "Epoch: 08 | Epoch Time: 0m 5s\n",
            "\tTrain Loss: 0.468 | Train f1: 0.65 | Train precision: 0.72 | Train recall: 0.59\n",
            "\t Val. Loss: 0.478 |  Val. f1: 0.68 |  Val. precision: 0.75 | Val. recall: 0.63\n",
            "Epoch: 09 | Epoch Time: 0m 5s\n",
            "\tTrain Loss: 0.445 | Train f1: 0.67 | Train precision: 0.73 | Train recall: 0.62\n",
            "\t Val. Loss: 0.466 |  Val. f1: 0.69 |  Val. precision: 0.76 | Val. recall: 0.63\n",
            "Epoch: 10 | Epoch Time: 0m 5s\n",
            "\tTrain Loss: 0.423 | Train f1: 0.68 | Train precision: 0.75 | Train recall: 0.63\n",
            "\t Val. Loss: 0.464 |  Val. f1: 0.69 |  Val. precision: 0.77 | Val. recall: 0.64\n",
            "Val. Loss: 0.464 |  Val. f1: 0.69 | Val. precision: 0.77 | Val. recall: 0.64\n",
            "El modelo actual tiene 5,296,072 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 7s\n",
            "\tTrain Loss: 1.366 | Train f1: 0.00 | Train precision: 0.04 | Train recall: 0.00\n",
            "\t Val. Loss: 0.972 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 7s\n",
            "\tTrain Loss: 0.981 | Train f1: 0.12 | Train precision: 0.37 | Train recall: 0.08\n",
            "\t Val. Loss: 0.832 |  Val. f1: 0.25 |  Val. precision: 0.38 | Val. recall: 0.19\n",
            "Epoch: 03 | Epoch Time: 0m 7s\n",
            "\tTrain Loss: 0.847 | Train f1: 0.29 | Train precision: 0.39 | Train recall: 0.24\n",
            "\t Val. Loss: 0.741 |  Val. f1: 0.35 |  Val. precision: 0.43 | Val. recall: 0.30\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
            "  warnings.warn('{} seems not to be NE tag.'.format(chunk))\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch: 04 | Epoch Time: 0m 7s\n",
            "\tTrain Loss: 0.738 | Train f1: 0.37 | Train precision: 0.46 | Train recall: 0.31\n",
            "\t Val. Loss: 0.670 |  Val. f1: 0.43 |  Val. precision: 0.53 | Val. recall: 0.36\n",
            "Epoch: 05 | Epoch Time: 0m 7s\n",
            "\tTrain Loss: 0.656 | Train f1: 0.48 | Train precision: 0.57 | Train recall: 0.42\n",
            "\t Val. Loss: 0.613 |  Val. f1: 0.54 |  Val. precision: 0.61 | Val. recall: 0.48\n",
            "Epoch: 06 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.594 | Train f1: 0.55 | Train precision: 0.62 | Train recall: 0.50\n",
            "\t Val. Loss: 0.574 |  Val. f1: 0.60 |  Val. precision: 0.67 | Val. recall: 0.55\n",
            "Epoch: 07 | Epoch Time: 0m 7s\n",
            "\tTrain Loss: 0.549 | Train f1: 0.59 | Train precision: 0.65 | Train recall: 0.54\n",
            "\t Val. Loss: 0.552 |  Val. f1: 0.63 |  Val. precision: 0.68 | Val. recall: 0.59\n",
            "Epoch: 08 | Epoch Time: 0m 7s\n",
            "\tTrain Loss: 0.514 | Train f1: 0.62 | Train precision: 0.67 | Train recall: 0.57\n",
            "\t Val. Loss: 0.540 |  Val. f1: 0.64 |  Val. precision: 0.69 | Val. recall: 0.61\n",
            "Epoch: 09 | Epoch Time: 0m 7s\n",
            "\tTrain Loss: 0.483 | Train f1: 0.64 | Train precision: 0.69 | Train recall: 0.59\n",
            "\t Val. Loss: 0.535 |  Val. f1: 0.65 |  Val. precision: 0.70 | Val. recall: 0.62\n",
            "Epoch: 10 | Epoch Time: 0m 7s\n",
            "\tTrain Loss: 0.462 | Train f1: 0.65 | Train precision: 0.70 | Train recall: 0.60\n",
            "\t Val. Loss: 0.543 |  Val. f1: 0.65 |  Val. precision: 0.69 | Val. recall: 0.62\n",
            "Val. Loss: 0.535 |  Val. f1: 0.65 | Val. precision: 0.70 | Val. recall: 0.62\n",
            "El modelo actual tiene 5,297,360 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 1.440 | Train f1: 0.00 | Train precision: 0.01 | Train recall: 0.00\n",
            "\t Val. Loss: 1.084 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 1.058 | Train f1: 0.00 | Train precision: 0.06 | Train recall: 0.00\n",
            "\t Val. Loss: 0.887 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 03 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.911 | Train f1: 0.20 | Train precision: 0.37 | Train recall: 0.15\n",
            "\t Val. Loss: 0.809 |  Val. f1: 0.31 |  Val. precision: 0.40 | Val. recall: 0.26\n",
            "Epoch: 04 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.814 | Train f1: 0.31 | Train precision: 0.40 | Train recall: 0.26\n",
            "\t Val. Loss: 0.759 |  Val. f1: 0.34 |  Val. precision: 0.40 | Val. recall: 0.30\n",
            "Epoch: 05 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.749 | Train f1: 0.35 | Train precision: 0.44 | Train recall: 0.30\n",
            "\t Val. Loss: 0.724 |  Val. f1: 0.37 |  Val. precision: 0.41 | Val. recall: 0.34\n",
            "Epoch: 06 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 0.703 | Train f1: 0.39 | Train precision: 0.47 | Train recall: 0.34\n",
            "\t Val. Loss: 0.705 |  Val. f1: 0.39 |  Val. precision: 0.43 | Val. recall: 0.36\n",
            "Epoch: 07 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 0.667 | Train f1: 0.42 | Train precision: 0.49 | Train recall: 0.37\n",
            "\t Val. Loss: 0.688 |  Val. f1: 0.41 |  Val. precision: 0.45 | Val. recall: 0.37\n",
            "Epoch: 08 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.632 | Train f1: 0.44 | Train precision: 0.51 | Train recall: 0.38\n",
            "\t Val. Loss: 0.670 |  Val. f1: 0.41 |  Val. precision: 0.44 | Val. recall: 0.38\n",
            "Epoch: 09 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.597 | Train f1: 0.46 | Train precision: 0.53 | Train recall: 0.41\n",
            "\t Val. Loss: 0.636 |  Val. f1: 0.44 |  Val. precision: 0.48 | Val. recall: 0.40\n",
            "Epoch: 10 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.561 | Train f1: 0.50 | Train precision: 0.56 | Train recall: 0.45\n",
            "\t Val. Loss: 0.620 |  Val. f1: 0.53 |  Val. precision: 0.55 | Val. recall: 0.50\n",
            "Val. Loss: 0.620 |  Val. f1: 0.53 | Val. precision: 0.55 | Val. recall: 0.50\n",
            "El modelo actual tiene 5,298,648 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 1.419 | Train f1: 0.00 | Train precision: 0.03 | Train recall: 0.00\n",
            "\t Val. Loss: 1.103 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 1.078 | Train f1: 0.02 | Train precision: 0.13 | Train recall: 0.01\n",
            "\t Val. Loss: 0.881 |  Val. f1: 0.04 |  Val. precision: 0.33 | Val. recall: 0.02\n",
            "Epoch: 03 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.906 | Train f1: 0.26 | Train precision: 0.41 | Train recall: 0.20\n",
            "\t Val. Loss: 0.791 |  Val. f1: 0.34 |  Val. precision: 0.43 | Val. recall: 0.28\n",
            "Epoch: 04 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.811 | Train f1: 0.33 | Train precision: 0.45 | Train recall: 0.27\n",
            "\t Val. Loss: 0.747 |  Val. f1: 0.36 |  Val. precision: 0.44 | Val. recall: 0.30\n",
            "Epoch: 05 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.752 | Train f1: 0.38 | Train precision: 0.48 | Train recall: 0.32\n",
            "\t Val. Loss: 0.714 |  Val. f1: 0.40 |  Val. precision: 0.46 | Val. recall: 0.36\n",
            "Epoch: 06 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.704 | Train f1: 0.41 | Train precision: 0.50 | Train recall: 0.35\n",
            "\t Val. Loss: 0.695 |  Val. f1: 0.42 |  Val. precision: 0.48 | Val. recall: 0.38\n",
            "Epoch: 07 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.666 | Train f1: 0.43 | Train precision: 0.52 | Train recall: 0.37\n",
            "\t Val. Loss: 0.687 |  Val. f1: 0.41 |  Val. precision: 0.45 | Val. recall: 0.39\n",
            "Epoch: 08 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.635 | Train f1: 0.45 | Train precision: 0.53 | Train recall: 0.39\n",
            "\t Val. Loss: 0.670 |  Val. f1: 0.42 |  Val. precision: 0.46 | Val. recall: 0.39\n",
            "Epoch: 09 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 0.610 | Train f1: 0.45 | Train precision: 0.54 | Train recall: 0.40\n",
            "\t Val. Loss: 0.670 |  Val. f1: 0.42 |  Val. precision: 0.46 | Val. recall: 0.39\n",
            "Epoch: 10 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.581 | Train f1: 0.47 | Train precision: 0.55 | Train recall: 0.41\n",
            "\t Val. Loss: 0.668 |  Val. f1: 0.43 |  Val. precision: 0.47 | Val. recall: 0.41\n",
            "Val. Loss: 0.668 |  Val. f1: 0.43 | Val. precision: 0.47 | Val. recall: 0.41\n",
            "El modelo actual tiene 5,299,936 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 13s\n",
            "\tTrain Loss: 1.448 | Train f1: 0.01 | Train precision: 0.05 | Train recall: 0.01\n",
            "\t Val. Loss: 1.128 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 13s\n",
            "\tTrain Loss: 1.203 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.064 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 03 | Epoch Time: 0m 13s\n",
            "\tTrain Loss: 1.041 | Train f1: 0.03 | Train precision: 0.15 | Train recall: 0.02\n",
            "\t Val. Loss: 0.877 |  Val. f1: 0.19 |  Val. precision: 0.47 | Val. recall: 0.12\n",
            "Epoch: 04 | Epoch Time: 0m 14s\n",
            "\tTrain Loss: 0.891 | Train f1: 0.28 | Train precision: 0.40 | Train recall: 0.22\n",
            "\t Val. Loss: 0.801 |  Val. f1: 0.34 |  Val. precision: 0.41 | Val. recall: 0.29\n",
            "Epoch: 05 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.808 | Train f1: 0.35 | Train precision: 0.44 | Train recall: 0.29\n",
            "\t Val. Loss: 0.758 |  Val. f1: 0.35 |  Val. precision: 0.41 | Val. recall: 0.31\n",
            "Epoch: 06 | Epoch Time: 0m 13s\n",
            "\tTrain Loss: 0.744 | Train f1: 0.38 | Train precision: 0.47 | Train recall: 0.33\n",
            "\t Val. Loss: 0.735 |  Val. f1: 0.40 |  Val. precision: 0.43 | Val. recall: 0.37\n",
            "Epoch: 07 | Epoch Time: 0m 13s\n",
            "\tTrain Loss: 0.700 | Train f1: 0.41 | Train precision: 0.49 | Train recall: 0.36\n",
            "\t Val. Loss: 0.712 |  Val. f1: 0.41 |  Val. precision: 0.44 | Val. recall: 0.38\n",
            "Epoch: 08 | Epoch Time: 0m 13s\n",
            "\tTrain Loss: 0.660 | Train f1: 0.43 | Train precision: 0.51 | Train recall: 0.38\n",
            "\t Val. Loss: 0.697 |  Val. f1: 0.41 |  Val. precision: 0.45 | Val. recall: 0.39\n",
            "Epoch: 09 | Epoch Time: 0m 13s\n",
            "\tTrain Loss: 0.630 | Train f1: 0.45 | Train precision: 0.52 | Train recall: 0.39\n",
            "\t Val. Loss: 0.680 |  Val. f1: 0.43 |  Val. precision: 0.47 | Val. recall: 0.39\n",
            "Epoch: 10 | Epoch Time: 0m 13s\n",
            "\tTrain Loss: 0.605 | Train f1: 0.45 | Train precision: 0.53 | Train recall: 0.40\n",
            "\t Val. Loss: 0.677 |  Val. f1: 0.42 |  Val. precision: 0.45 | Val. recall: 0.39\n",
            "Val. Loss: 0.677 |  Val. f1: 0.42 | Val. precision: 0.45 | Val. recall: 0.39\n",
            "El modelo actual tiene 5,301,224 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 15s\n",
            "\tTrain Loss: 1.434 | Train f1: 0.01 | Train precision: 0.10 | Train recall: 0.01\n",
            "\t Val. Loss: 1.126 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 15s\n",
            "\tTrain Loss: 1.205 | Train f1: 0.00 | Train precision: 0.05 | Train recall: 0.00\n",
            "\t Val. Loss: 1.058 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 03 | Epoch Time: 0m 15s\n",
            "\tTrain Loss: 1.054 | Train f1: 0.03 | Train precision: 0.21 | Train recall: 0.01\n",
            "\t Val. Loss: 0.889 |  Val. f1: 0.00 |  Val. precision: 0.09 | Val. recall: 0.00\n",
            "Epoch: 04 | Epoch Time: 0m 15s\n",
            "\tTrain Loss: 0.916 | Train f1: 0.26 | Train precision: 0.40 | Train recall: 0.20\n",
            "\t Val. Loss: 0.827 |  Val. f1: 0.33 |  Val. precision: 0.41 | Val. recall: 0.28\n",
            "Epoch: 05 | Epoch Time: 0m 15s\n",
            "\tTrain Loss: 0.837 | Train f1: 0.33 | Train precision: 0.43 | Train recall: 0.27\n",
            "\t Val. Loss: 0.793 |  Val. f1: 0.34 |  Val. precision: 0.43 | Val. recall: 0.28\n",
            "Epoch: 06 | Epoch Time: 0m 15s\n",
            "\tTrain Loss: 0.777 | Train f1: 0.36 | Train precision: 0.46 | Train recall: 0.30\n",
            "\t Val. Loss: 0.757 |  Val. f1: 0.35 |  Val. precision: 0.42 | Val. recall: 0.30\n",
            "Epoch: 07 | Epoch Time: 0m 16s\n",
            "\tTrain Loss: 0.733 | Train f1: 0.40 | Train precision: 0.48 | Train recall: 0.34\n",
            "\t Val. Loss: 0.744 |  Val. f1: 0.39 |  Val. precision: 0.44 | Val. recall: 0.35\n",
            "Epoch: 08 | Epoch Time: 0m 15s\n",
            "\tTrain Loss: 0.691 | Train f1: 0.41 | Train precision: 0.49 | Train recall: 0.36\n",
            "\t Val. Loss: 0.723 |  Val. f1: 0.40 |  Val. precision: 0.44 | Val. recall: 0.37\n",
            "Epoch: 09 | Epoch Time: 0m 15s\n",
            "\tTrain Loss: 0.662 | Train f1: 0.43 | Train precision: 0.51 | Train recall: 0.37\n",
            "\t Val. Loss: 0.704 |  Val. f1: 0.42 |  Val. precision: 0.46 | Val. recall: 0.39\n",
            "Epoch: 10 | Epoch Time: 0m 15s\n",
            "\tTrain Loss: 0.630 | Train f1: 0.45 | Train precision: 0.52 | Train recall: 0.39\n",
            "\t Val. Loss: 0.710 |  Val. f1: 0.42 |  Val. precision: 0.45 | Val. recall: 0.39\n",
            "Val. Loss: 0.704 |  Val. f1: 0.42 | Val. precision: 0.46 | Val. recall: 0.39\n",
            "El modelo actual tiene 5,302,512 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 1.386 | Train f1: 0.00 | Train precision: 0.01 | Train recall: 0.00\n",
            "\t Val. Loss: 1.126 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 1.199 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.121 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 03 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 1.140 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.005 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 04 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.994 | Train f1: 0.10 | Train precision: 0.29 | Train recall: 0.07\n",
            "\t Val. Loss: 0.865 |  Val. f1: 0.28 |  Val. precision: 0.46 | Val. recall: 0.21\n",
            "Epoch: 05 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.871 | Train f1: 0.29 | Train precision: 0.42 | Train recall: 0.22\n",
            "\t Val. Loss: 0.789 |  Val. f1: 0.33 |  Val. precision: 0.43 | Val. recall: 0.28\n",
            "Epoch: 06 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.801 | Train f1: 0.34 | Train precision: 0.46 | Train recall: 0.28\n",
            "\t Val. Loss: 0.754 |  Val. f1: 0.35 |  Val. precision: 0.41 | Val. recall: 0.31\n",
            "Epoch: 07 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.753 | Train f1: 0.38 | Train precision: 0.47 | Train recall: 0.32\n",
            "\t Val. Loss: 0.730 |  Val. f1: 0.39 |  Val. precision: 0.44 | Val. recall: 0.35\n",
            "Epoch: 08 | Epoch Time: 0m 18s\n",
            "\tTrain Loss: 0.707 | Train f1: 0.41 | Train precision: 0.49 | Train recall: 0.35\n",
            "\t Val. Loss: 0.709 |  Val. f1: 0.40 |  Val. precision: 0.44 | Val. recall: 0.37\n",
            "Epoch: 09 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.673 | Train f1: 0.43 | Train precision: 0.51 | Train recall: 0.38\n",
            "\t Val. Loss: 0.708 |  Val. f1: 0.39 |  Val. precision: 0.42 | Val. recall: 0.38\n",
            "Epoch: 10 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.646 | Train f1: 0.45 | Train precision: 0.53 | Train recall: 0.40\n",
            "\t Val. Loss: 0.679 |  Val. f1: 0.43 |  Val. precision: 0.47 | Val. recall: 0.39\n",
            "Val. Loss: 0.679 |  Val. f1: 0.43 | Val. precision: 0.47 | Val. recall: 0.39\n",
            "El modelo actual tiene 5,303,800 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 1.432 | Train f1: 0.01 | Train precision: 0.07 | Train recall: 0.01\n",
            "\t Val. Loss: 1.126 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 1.212 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.125 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 03 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 1.198 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.119 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 04 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 1.181 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.101 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 05 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 1.106 | Train f1: 0.00 | Train precision: 0.01 | Train recall: 0.00\n",
            "\t Val. Loss: 0.955 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 06 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.960 | Train f1: 0.18 | Train precision: 0.40 | Train recall: 0.12\n",
            "\t Val. Loss: 0.842 |  Val. f1: 0.32 |  Val. precision: 0.44 | Val. recall: 0.25\n",
            "Epoch: 07 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.860 | Train f1: 0.31 | Train precision: 0.44 | Train recall: 0.24\n",
            "\t Val. Loss: 0.800 |  Val. f1: 0.34 |  Val. precision: 0.44 | Val. recall: 0.28\n",
            "Epoch: 08 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.805 | Train f1: 0.36 | Train precision: 0.49 | Train recall: 0.29\n",
            "\t Val. Loss: 0.776 |  Val. f1: 0.36 |  Val. precision: 0.45 | Val. recall: 0.30\n",
            "Epoch: 09 | Epoch Time: 0m 20s\n",
            "\tTrain Loss: 0.759 | Train f1: 0.39 | Train precision: 0.50 | Train recall: 0.33\n",
            "\t Val. Loss: 0.753 |  Val. f1: 0.39 |  Val. precision: 0.46 | Val. recall: 0.34\n",
            "Epoch: 10 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.721 | Train f1: 0.41 | Train precision: 0.50 | Train recall: 0.35\n",
            "\t Val. Loss: 0.738 |  Val. f1: 0.40 |  Val. precision: 0.46 | Val. recall: 0.35\n",
            "Val. Loss: 0.738 |  Val. f1: 0.40 | Val. precision: 0.46 | Val. recall: 0.35\n",
            "El modelo actual tiene 5,305,088 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 1.441 | Train f1: 0.01 | Train precision: 0.06 | Train recall: 0.01\n",
            "\t Val. Loss: 1.125 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 21s\n",
            "\tTrain Loss: 1.206 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.124 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 03 | Epoch Time: 0m 21s\n",
            "\tTrain Loss: 1.194 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.119 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 04 | Epoch Time: 0m 21s\n",
            "\tTrain Loss: 1.182 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.117 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 05 | Epoch Time: 0m 21s\n",
            "\tTrain Loss: 1.173 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.114 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 06 | Epoch Time: 0m 21s\n",
            "\tTrain Loss: 1.153 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.082 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 07 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 1.091 | Train f1: 0.01 | Train precision: 0.06 | Train recall: 0.01\n",
            "\t Val. Loss: 0.982 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 08 | Epoch Time: 0m 21s\n",
            "\tTrain Loss: 0.991 | Train f1: 0.19 | Train precision: 0.38 | Train recall: 0.13\n",
            "\t Val. Loss: 0.887 |  Val. f1: 0.31 |  Val. precision: 0.47 | Val. recall: 0.24\n",
            "Epoch: 09 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 0.909 | Train f1: 0.31 | Train precision: 0.46 | Train recall: 0.24\n",
            "\t Val. Loss: 0.861 |  Val. f1: 0.32 |  Val. precision: 0.46 | Val. recall: 0.25\n",
            "Epoch: 10 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 0.866 | Train f1: 0.34 | Train precision: 0.48 | Train recall: 0.27\n",
            "\t Val. Loss: 0.847 |  Val. f1: 0.33 |  Val. precision: 0.44 | Val. recall: 0.27\n",
            "Val. Loss: 0.847 |  Val. f1: 0.33 | Val. precision: 0.44 | Val. recall: 0.27\n",
            "El modelo actual tiene 5,306,376 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 1.431 | Train f1: 0.03 | Train precision: 0.07 | Train recall: 0.06\n",
            "\t Val. Loss: 1.126 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 1.213 | Train f1: 0.00 | Train precision: 0.02 | Train recall: 0.00\n",
            "\t Val. Loss: 1.121 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 03 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 1.196 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.119 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 04 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 1.186 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.119 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 05 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 1.174 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.114 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 06 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 1.166 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.113 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 07 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 1.158 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.111 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 08 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 1.153 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.108 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 09 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 1.146 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.105 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 10 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 1.144 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.104 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Val. Loss: 1.104 |  Val. f1: 0.00 | Val. precision: 0.00 | Val. recall: 0.00\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "fig = plt.figure()\n",
        "ax = fig.add_subplot(1, 1, 1)\n",
        "ax.plot([i for i in range(1,11)], validation_values)\n",
        "plt.xlabel('Numero de capas ocultas')\n",
        "plt.ylabel('Error en el conjunto de validación')\n",
        "plt.title('Error en el conjunto de validación en función de el número de capas ocultas')\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "id": "lVKwFto3tlbc",
        "outputId": "45a8651a-9679-4a03-c968-69b3b8f3962d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdkAAAEWCAYAAAAq+e1jAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3xV9f3H8deHsDcBRHYAUcCJBgG1ah111FW1Ck7U1i6rHdbarR3W9ldbO6wWHODWOnHUPRGZgiiIihAgbEhYYYQkn98f3xO8xBsSIMk5N3k/H488cvb53O8993zO95zvOcfcHREREal5jeIOQEREpL5SkhUREaklSrIiIiK1RElWRESklijJioiI1BIlWRERkVqiJJvCzG4ws/vreJ2zzezYulxndZhZjpm5mTWu4/XmmdkJUffPzezO6ky7B+tLuw4zO9LMpphZhz1Zfk2w4B4zKzSzKbW4njopi9ratmpquWY2xMw+MrM2NRVbbTKzUWY2Ie44MkW0jexTV+urcmM0szygC1CaMnisu19VW0E1JO6+f00sJ0rU97t7j5pYXhK4+01xrMPMegI3Aae5e2Ftx1ANRwEnAj3cvai2VpIhZVGrogT9T2Cku2+IOx6pXWY2Fsh391/W1jqqe8R3uru/UtVEZtbY3UsqDMty99LK5kmzjF2aXqSmufti4Ji440jRG8irzQRbmQSWRW3rB/ze3WfW1QrT7TelHnH3nf4BecAJlYwbBbwD/A1YA/weGAvcDjwPFAEnAAOBN4C1wGzgjJRlfGH6NOtpB9wFLAOWROvJSolhAvAXoBBYAJyyk8/TDXgcWBVNe3XKuBsItcHK5j0TmAmsBz4DTk5Z5nigAJgHfLPCMh8F7gU2RJ8/N135RmXx+5RxxxKOslKnvRaYBawDHgGaA62AzUAZsDH66wY0A24FlkZ/twLNKvlsWVEZrgbmA98DHGhc1XeQpnw3A9kpwwZHy21C2Im9Fm0vq4EHgPaVlMcO3wdwMbAwmvcXFaY9HHiXsI0tA/4FNE2Zd3/g5eg7WgH8vJJ1nBF9R2sJ2+zAqsp/J9vL5cBHhO3yRaB3yjgHvg18Gq3rNsDSLOMKYAvhTNJG4Eaibb7CdA7sk7Id3QY8R9jmJgP94ioLamjbiqZtBFxP+P2tIfy2sqNxOanLrWRfljbmapbpv4H/Rd/DO8DehN9UITAXGLwL+5nHgPsJ+5JvsJN9SJrP0TGadj0wBfhdauzAgJTv92PgvJ0sKxu4h7B/KASeioZ3AJ6N4i+MunukzPcG8Mdo/euBp9nxN/9fYHlUzm8B+6eMOxWYQ9g2lwDX7uS7/iXhN7+SsA9tlzL+KGAiYftcDIxKie0bKdPt8N2Wf6/AlcA2oDj6Tp+JxpdvXxuiOL+WMu8+wJvR51oNPFJZ2W6fp8oJqk6yJcD3CbXiFtHGuA44MiqkNtFG83OgKXBcFPx+KRtv6vRf+KECTwL/ISSTvaIv9lspMWwDvkn4MX8n2mDS7bAaAdOBX0ex9CX86E9Kt5OpMO/hUZwnRsvpDgyIxr1F+AE2Bw4hbJjHpSxzS7RhZRE2zEnpypfqJdkphB9kNmEH/u1000bDfgtMisqsM2GD/F0ln+/bhB1Fz2jZr7PjjrDS7yDNsl5jxwON/wPuSNlITyQcAHSOyu7WSspj+/cBDCL8EI6O5v0rYdsrn/YwYBhhO8yJyuYH0bg2hB34j6PvqA0wNM069iUc6J1IOCC4jrDtNq2q/NOUwZnRvAOjmH4JTKzwQ38WaA/0ImwzJ+/kdzahsv7UHUfKdrSGsM02JhzIPBxjWdTktnUNYZvuEW0H/wEeisblUHWSrez3U50yXU3YzpoTtvEFwCWE3/Xvgdd3YT+zDTgrmrYFO9mHpPkcDxMOLloBBxAS1YRoXCtCwrks+u7LD3AHVbKs5wgHGx2i7/mYaHhH4BygZbSN/JcoAUfj34jWe0C0zsfZ8QDt8mi+8gP9mSnjlgFfiro7AIdWEtvlhG2uL9AaeAK4LxrXm5BHRkZxdwQOSYmtyiSbbp8bDft6tI00As4n/A66RuMeIhzgN4q+q6PSxb7D8qqcIGyYGwlHC+V/30wJflGF6ccC96b0f4lwRNMoZdhDwA3ppk+z/i7AVqBFyrCRfL5BjwLmpYxrGRXi3mmWNTRNvD8D7qm4k0kz73+Av6UZ3pNQy2iTMuyPhOvW5ct8JWXcIGBzhfLdlSR7UUr/n/k8ee0wbTTsM+DUlP6TCKcdK0uM307p/0pUjo2r+g7SLOsbwGtRtxF+9EdXMu1ZwIxKymP790HYYT2cMl0rwhFoZQeAPwCeTIl1RiXTpa7jV8CjKeMaEXYkx1ZV/mmW+z/gigrL2kRUm43K9qiU8Y8C11eyrFHsepK9M2XcqcDcGMuiJretj4DjU/q7EhJW+cFVVUm2st9Pdcp0TMq47wMfpfQfCKyNuquzn3krZdxO9yEVlpMVfd4BKcNu4vMkez7wdoV5/gP8Js2yuhLOfnVIV14Vpj0EKEzpfwO4OaV/EOH3mO7sVvuoLNtF/YuAbwFtq1jnq8B3U/r3S/muf0b0+04z3xvsQZJNs7yZwJlR973AaFJq9VX9Vfea7Fle+TXZxVUM6wYsdveylGELCTXBnS2jXG/CkcoyMysf1qjCPMvLO9x9UzRd60qW1c3M1qYMywLe3sn6y/UknNKuqBtQ4Ds2klgI5KaLj7Cjbb4H12EqLqvbTqbtFsWSGldl03djxzJNna8630Gqx4F/mllXQo2ojKiMzawL8HfCwVebaDnVaVCzQ3zuXmRma8r7zWxfQu02l3Cg1ZhQm4Dw3X1WzXVs/9zuXmZmi9lxW61u+fcG/m5mt6QMs2hZ5euouKx02+zuqmzZcZRFTW5bvYEnzSx1f1JKSNbVsSu/n4pWpHRvTtNfXsbV2c9U3EdWtQ8p15mwbe+sPIdWWHdj4L40y+oZrfcLvz8za0m4DHgyobYJ0KZCm5mKMTQBOpnZauAPhBphZ8LvH6AT4WzgOYQzOzeb2SzCweW7aeJLt/8qPzCr7na8y8zsEuBHhIM2CN9rp6j7OsLp+SlmVgjc4u5372x5NdGE3qsYthToaWaNUhJtL+CTKpZRbjHhSLfTbialista4O79d3PefmmGLwWyzaxNyo+kF+Gof1cVERJEub13Yd50ZbiU8KObnRLX0krmX0bYcEmZttwufQfuXmhmLxGOqgcSaqDl8d0UxXqguxeY2VmE66dVWRYtC9i+E+iYMv52YAZRq1Az+wFwbkr8I6qxjqWEGkn5OoxQJrvzXS4G/uDuD+zGvFXZYTsxs13ZTuIoixrbtqLpL3f3dyqOMLOc3Yit3J6UaUXV2c9U3EdWdx+yinCZpCfhFHz5tKnrftPdT6xmnNlm1t7d11YY92NCzXGouy83s0MIvy9Lmabid7qNcGr6AsLlkhMIZw/aEQ6kDcDdpwJnmlkT4CrCWZzUZZUr33+lrqOEcHCzmHA5JJ1d2Y/usN80s97AGOB44F13LzWzmSmxLydcmsTMjgJeMbO33H1eZSuoi/tkJxOOGK8zsybRrSanE64rVMndlwEvAbeYWVsza2Rm/czsmN2IZQqwwcx+amYtzCzLzA4wsyHVmPcu4DIzOz6KobuZDfDQ+nIi8Ecza25mBxEaq+zO/bYzgVPNLDv6kf9gF+ZdAXQ0s3Ypwx4Cfmlmnc2sE+GUa2VxPQpcbWY9ovshry8fsZvfwYOE61XnRt3l2hAuP6wzs+7AT6r5+R4DTjOzo8ysKeF6c+r224bQAGOjmQ0gXJsv9yzQ1cx+YGbNzKyNmQ1Ns45Hga9G33ETwo5mK+H73VV3AD8zs/0BzKydmX19N5aTzvvA/mZ2iJk1J5x+rK44yqImt607gD9EO0OibfvM3Yipoj0p04p2aT+zK/uQqBb5BHCDmbU0s0HApSmTPAvsa2YXR/vbJhbu+x2YZlnLCJc1/m1mHaJpj45GtyHUzteaWTbwmzShX2Rmg6ID3t8Cj0XxtSFsK2sIyW77rWFm1tTMLjSzdu6+jfCbLUuzbAj7rx+aWR8zax0t55HoYOwB4AQzO8/MGptZx+hAAMJ+9OyofPaJyrIyKwjXfMu1IiTeVVG8lxGuO5fH/3UzK79NsjCatrL4geon2WfMbGPK35PVnA93LyYk1VMIRzn/Bi5x97k7nXFHlxAaEMwhfLDHCNcTdkm0AZxGuL6wIIrnTsKRVlXzTiE0Jvgb4ZTHm3x+lDWScGphKaERx292cnp9Z+4j/NjzCDueR6o7Y1SeDwHzzWytmXUjNMaYRmhN+QHwXjQsnTGEFrDvR9M9UWH8rn4H44H+wHJ3fz9l+I3AoYQyfC7Neir7fLMJrVIfJNSMCoH8lEmuJRxBb4g+yyMp824gNOA5nXC68FPgy2nW8TFwEeE+ydXR9KdH2/AucfcngT8BD5vZeuBDwm9gj7n7J4Sd2iuEz1LtBxHEURbU7Lb1d8K29ZKZbSA0gkp3kLBL9qRM0yxrd/Yzu7IPuYpwCnM54ZriPSnr3kC45j0iWtZywnbYrJJlXUyogc4ltOAtP7C/ldAgazWhjF9IM+990fqXExoBXR0Nv5dwancJ4TudlGadedHv4tvAhZXEdne0jrcI5biFcC0cd19EaGvwY0Ir6pnAwdF8fyNcH14BjCMk5MrcBQyK9plPufsc4BbCnQorCGdzUs+aDAEmm9lGwnZ4jbvP38nyQwtciY+ZLSI0xngr7lhERKrDzN4gNJKr9IlsEuixijEys86EhgF5MYciIiK1QEk2JtH1mU+Bf0anPkREpJ7R6WIREZFaopqsiIhILanT15hlik6dOnlOTk7cYYiIZJTp06evdvfOcceRJEqyaeTk5DBt2rS4wxARyShmtrDqqRoWnS4WERGpJUqyIiIitURJVkREpJYoyYqIiNQSJVkREZFaoiQrIiJSSzI6yZrZ3Wa20sw+rGT8ADN718y2mtm1dR2fiIg0bBmdZAmvWTp5J+MLCK9f+kudRCMiksFue30eE+etjjuMeiWjk2z0eriCnYxf6e5TCe9LFBGRSuQXbuKWlz7mbSXZGpXRSbYmmdmVZjbNzKatWrUq7nBEROrUfZPCw5ouGtY75kjqFyXZiLuPdvdcd8/t3FmP3hSRhmPLtlIembqYrwzam+7tW8QdTr2iJCsi0sA9PXMJazdt49IjcuIOpd5RkhURacDcnbETF7JflzYM65sddzj1Tka/hcfMHgKOBTqZWT7wG6AJgLvfYWZ7A9OAtkCZmf0AGOTu62MKWUQkUabmFfLRsvXc9LUDMbO4w6l3MjrJuvvIKsYvB3rUUTgiIhln3MQ82jZvzFmDu8UdSr2k08UiIg3UsnWbeWH2cs4f0pOWTTO6zpVYSrIiIg3UA5MWUebOxcNy4g6l3lKSFRFpgLZsK+WhKYs4fsBe9OrYMu5w6i0lWRGRBui5WctYU1Ss23ZqmZKsiEgD4+6MezePfp1bcdQ+neIOp15TkhURaWBmLF7LrPx1XHpEjm7bqWVKsiIiDcy4iXm0btaYsw/VHY61TUlWRKQBWblhC89/sIxzD+tB62a6bae2KcmKiDQgD05exLZS55LhettOXVCSFRFpIIpLynhg8iKO2bczfTu3jjucBiEx5wrM7Aggh5SY3P3e2AISEaln/vfhMlZt2Mqoc3LiDqXBSESSNbP7gH7ATKA0GuyAkqyISA0ZNzGPnI4tOWZfvTO7riQiyQK5hLfjeNyBiIjURx/kr+O9RWv51WmDaNRIt+3UlaRck/0Q2DvuIERE6quxE/No2TSLr+fqtp26lJSabCdgjplNAbaWD3T3M+ILSUSkflizcSvPzFrKebk9aNu8SdzhNChJSbI3xB2AiEh99fDUxRSXlHHp8Jy4Q2lwEpFk3f1NM+sCDIkGTXH3lXHGJCJSH5SUlnH/pIUcuU9H+ndpE3c4DU5s12TNrFdK93nAFODrwHnAZDM7N67YRETqi5fmrGDZui2qxcYkzprsUDP7urvfAvwCGFJeezWzzsArwGMxxicikvHGTsyjR4cWHD+wS9yhNEix1WTd/b/A8vI4KpweXkNyWj6LiGSkj5atZ8qCAi4e1pss3bYTi1ivybr7A1HnC2b2IvBQ1H8+8Hw8UYmI1A/jJubRvEkjzh/SM+5QGqykNHz6iZmdAxwZDRrt7k/GGZOISCZbu6mYp2Yu4axDutO+ZdO4w2mwEpFkAdz9ceDxXZnHzO4GTgNWuvsBacYb8HfgVGATMMrd36uBcEVEEu2RqYvZsq2MS4/IiTuUBi3W655mNiH6v8HM1qf8bTCz9dVYxFjg5J2MPwXoH/1dCdy+pzGLiCRdaZlz36SFHN4nm4Fd28YdToMWa5J196Oi/23cvW3KXxt3r3LLcPe3gIKdTHImcK8Hk4D2Zta1ZqIXEUmmVz9aQX7hZkapFhu7RLTgNbNhZtYmpb+NmQ2tgUV3Bxan9OdHw9LFcKWZTTOzaatWraqBVYuIxOPedxfStV1zvjJIt+3ELRFJlnAad2NKfxF1fGrX3Ue7e66753burNdAiUhmmrdyAxPmreaiYb1pnJWUXXzDlZRvwFJfc+fuZdRMo6wlQGrb9R7RMBGRemncxIU0bdyIEbptJxGSkmTnm9nVZtYk+rsGmF8Dyx0PXGLBMGCduy+rgeWKiCTO+i3bePy9fE4/qBsdWzeLOxwhObfwfBv4B/BLwIFXCa2Bd8rMHgKOBTqZWT7wG6AJgLvfQXigxanAPMItPJfVQuwiIonw2LR8NhWXqsFTgiQiyUaPVByxG/ONrGK8A9/b3bhERDJFWZlz77t5HNqrPQf2aBd3OBJJRJI1s+bAFcD+QPPy4e5+eWxBiYhkkDc/XUXemk388MR94w5FUiTlmux9wN7AScCbhAZKG2KNSEQkg4ybmEfnNs045QA9CiBJkpJk93H3XwFF7j4O+CpQE/fJiojUewtWF/HGx6u4cGgvmjZOym5dIDlJdlv0f62ZHQC0A/aKMR4RkYxx77t5NMkyLhjaK+5QpIJEXJMFRptZB+BXhNtuWgO/jjckEZHkK9pawmPT8jn1wK7s1aZ51TNInUpEknX3O6PON4G+ccYiIpJJnngvnw1bS/S2nYSKNcma2Y92Nt7d/1pXsYiIZBp3Z9y7CzmoRzsG92wfdziSRtw12fKXAuwHDCGcKgY4HZgSS0QiIhninXlrmLdyI7d8/WDC67MlaWJNsu5+I4CZvQUc6u4bov4bgOdiDE1EJPHGTsyjY6umfPUg3baTVElpXdwFKE7pL46GiYhIGosLNvHq3BWMOLwnzZtkxR2OVCLu08Xl7gWmmNmTUf9ZwNj4whERSbb7Ji2kkRkXDesddyiyE4lIsu7+BzP7H/ClaNBl7j4jzphERJJqc3Epj0xdzEn7d6FruxZxhyM7EXfr4rbuvt7MsoG86K98XLa7F8QVm4hIUj01cwnrNm/j0uE5cYciVYi7JvsgcBownfCKu3IW9eueWRGRFO7OuIl5DNi7DYf3yY47HKlC3K2LT4v+94kzDhGRTDF5QQFzl2/g5rMP1G07GSDu08WH7my8u79XV7GIiGSCcRPzaNeiCWce0j3uUKQa4j5dfMtOxjlwXF0FIiKSdEvXbualOSv4xlF9aNFUt+1kgrhPF385zvWLiGSS+yctxN11204Gibsmu130irtBwPbXSLj7vfFFJCKSHFu2lfLw1MUcP7ALPbNbxh2OVFMikqyZ/QY4lpBknwdOASYQHlIhItLgPfP+UgqKihmlt+1klKQ8VvFc4HhgubtfBhxMeHG7iEiDF962k0f/vVpzRL+OcYcjuyApSXazu5cBJWbWFlgJ9Iw5JhGRRHhvUSEfLlnPJUfk6LadDJOUJDvNzNoDYwgPpngPeLc6M5rZyWb2sZnNM7Pr04zvbWavmtksM3vDzHrUbOgiIrVr7MSFtGnemLMH67adTJOIa7Lu/t2o8w4zewFo6+6zqprPzLKA24ATgXxgqpmNd/c5KZP9BbjX3ceZ2XHAH4GLa/YTiIjUjhXrt/C/D5ZxyfAcWjVLxC5bdkEiarJmNt7MLjCzVu6eV50EGzkcmOfu8929GHgYOLPCNIOA16Lu19OMFxFJrAcmL6LUnUuG67adTJSIJEt4KMVRwBwze8zMzjWz5lXNBHQHFqf050fDUr0PnB11fw1oY2ZfaDlgZlea2TQzm7Zq1apd/wQiIjWsuKSMBycv4th9O5PTqVXc4chuSESSdfc3o1PGfYH/AOcRGj/VhGuBY8xsBnAMsAQoTRPDaHfPdffczp0719CqRUR23/MfLGP1xq1cqtt2MlZiTvCbWQvgdOB84FBgXDVmW8KOrZB7RMO2c/elRDVZM2sNnOPua2siZhGR2jR2Yh59OrXi6P468M9UiajJmtmjwEeEZxX/C+jn7t+vxqxTgf5m1sfMmgIjgPEVlt3JzMo/58+Au2suchGR2vH+4rXMXLyWS4b3plEj3baTqZJSk70LGOnuXziNuzPuXmJmVwEvAlnA3e4+28x+C0xz9/GEJ0n90cwceAv4Xs2GLiJS88ZNzKNV0yzOPUx3HWayRCRZd39xD+Z9nvAoxtRhv07pfgx4bPejExGpW6s3buXZWcsYcXhP2jRvEnc4sgcScbpYREQ+99DkRRSXlnHJ8Jy4Q5E9pCQrIpIg20rLuH/yQr7UvxP77NU67nBkDyUiyVpwkZn9OurvZWaHxx2XiEhde3H2clas38qlqsXWC4lIssC/geHAyKh/A+FxiSIiDcq4iXn0zG7BlwfsFXcoUgOSkmSHuvv3gC0A7l4INI03JBGRujV76Tqm5hVyybAcsnTbTr2QlCS7LXrYvwOYWWegLN6QRETq1riJebRoksV5uXrTZ32RlCT7D+BJYC8z+wMwAbgp3pBEROpOYVExT89cylmDu9OupW7bqS+Scp/sA2Y2HTgeMOAsd/8o5rBEROrMw1MXs7WkjEuP0Nt26pNYk6yZZaf0rgQeSh3n7gV1H5WISN0qKS3j/kkLGdY3mwF7t407HKlBcddkpxOuwxrQCyiMutsDi4A+8YUmIlI3XvloJUvWbuZXpw2MOxSpYbFek3X3Pu7eF3gFON3dO7l7R+A04KU4YxMRqQuFRcX89eWP6dauOScM7BJ3OFLDktLwaVj0DGIA3P1/wBExxiMiUusKioq54M7J5K3ZxJ/OPYjGWUnZJUtNift0cbmlZvZL4P6o/0JgaYzxiIjUqoKiYi4YM4kFq4u485JcvqR3xtZLSTlsGgl0JtzG80TUPXKnc4iIZKg1G7d+nmAvzeXofZVg66tE1GSjVsTXxB2HiEhtW7NxKxfeOZkFq4u469IhHNW/U9whSS1KRJIVEWkIVm/cyoVjJrOwoIi7Rw3hyH2UYOs7JVkRkTqwOjpFvKhgE3dfOoQjlGAbBCVZEZFatmpDSLCLC5VgG5pENHwysx5m9qSZrTKzlWb2uJn1iDsuEZE9tUOCHaUE29AkIskC9wDjga5AN+CZaJiISMZauWELI8dMIr9wM/eMOpwj+inBNjRJSbKd3f0edy+J/sYSbuMREclIK9dvYeToSSwp3Mw9lw1heL+OcYckMUhKkl1jZheZWVb0dxGwJu6gRER2x8r1WxgxZhLL1m1h7GVDGNZXCbahSkqSvRw4D1gOLAPOBUZVZ0YzO9nMPjazeWZ2fZrxvczsdTObYWazzOzUmgxcRCRVeYJdvm4LYy87nKFKsA1aUloX93D3M1IHmNmRwOKdzWRmWcBtwIlAPjDVzMa7+5yUyX4JPOrut5vZIOB5IKcmgxcRAVgRnSJesX4L4y4/nCE52VXPJPVaUmqy/6zmsIoOB+a5+3x3LwYeBs6sMI0D5S9obIeeiSwitWD5ui2MUIKVCuJ+aftwwtt2OpvZj1JGtQWyqrGI7uxY280HhlaY5gbgJTP7PtAKOKGSWK4ErgTo1atXdcIXEQFCgh05ZhIrowSbqwQrkbhrsk2B1oRk3yblbz3humxNGAmMdfcewKnAfWb2hc/t7qPdPdfdczt3VsNmEameZes2M2L0u6zasJV7r1CClR3FWpN19zeBN81srLsv3I1FLAF6pvT3iIalugI4OVrfu2bWHOgErNyN9YmIbLd07WZGjpnEmo3FjLv8cA7r3SHukCRhktLwqZmZjSY0SNoek7sfV8V8U4H+ZtaHkFxHABdUmGYRcDww1swGAs2BVTUUt4g0UEvXbmbE6EkUFhVz7xWHc2gvJVj5oqQk2f8CdwB3AqXVncndS8zsKuBFwjXcu919tpn9Fpjm7uOBHwNjzOyHhEZQo9zda/wTiEiDsWTtZkamJNjBSrBSiaQk2RJ3v313ZnT35wm35aQO+3VK9xzgyD0LT0QkyC/cxMgxk1i7aRv3fWMoh/RsH3dIkmBxN3wq94yZfdfMuppZdvlf3EGJiKTKL9zEiNEhwd5/hRKsVC0pNdlLo/8/SRnmQN8YYhER+YLFBaEGu25zSLAHK8FKNSQiybp7n7hjEBGpzOKCUIPdsGUbD3xjKAf1UIKV6klEkjWzS9INd/d76zoWEZFU5Ql249YSHvjGMA7s0S7ukCSDJCLJAkNSupsTbrl5D1CSFZHYLFoTThGHBDuUA7orwcquSUSSdffvp/abWXvCc4hFRGKxaM0mRox+l6LiUiVY2W1JaV1cURGg67QiEouFa4oYMfpdNm1TgpU9k4iarJk9Q2hNDCHxDwIejS8iEWmoQoKdxOYowe7fTQlWdl8ikizwl5TuEmChu+fHFYyINEx5q0OC3VpSyoPfGMagbm2rnklkJxKRZKMXBYiIxGbB6iJGjp5EcWkZD35zGAO7KsHKnov1mqyZTYj+bzCz9Wn+FpjZd+OMUUTqvwWrwzXYkGCHKsFKjYn7VXdHRf/bpBtvZh2BicC/6zIuEWk45q/ayMgxk9hW6jz4zaEM2FsJVmpOIk4XA5hZFtCFHV91t8jMjo0tKBGp1+YsXc+oe6ZQWuY89M1h7Ld32uN9kd2WiCRrZt8HfgOsAMqiwQ4c5O7LYgtMROqlpWs387eXP+Hx9/LJbtWUB5VgpZYkIskC1wD7ufuauAMRkfpr7Q5gm5AAABvBSURBVKZibn/jM+6ZmAcOlx/Zh+9+eR+yWzWNOzSpp5KSZBcD6+IOQkTqp83FpYydmMftb8xjw9YSzh7cgx+e2J8eHVrGHZrUc0lJsvOBN8zsOWBr+UB3/2t8IYlIpispLeOx6fn87ZVPWLF+K8cN2IvrTt5PjZukziQlyS6K/ppGfyIiu83deXH2Cv7vxbl8tqqIwb3a848Rgxnat2PcoUkDk4gk6+43AphZ66h/Y7wRiUimmjx/DTe/MJcZi9bSr3Mr/nPxYXxlUBfMLO7QpAFKRJI1swOA+4DsqH81cIm7z441MBHJGHOXr+fPL3zMa3NXsnfb5vzpnAM559AeNM5K6ntQpCFIRJIFRgM/cvfXAaJ7Y8cAR8QZlIgkX37hJv768ic8OWMJbZo15vpTBjDqiByaN8mKOzSRxCTZVuUJFsDd3zCzVnEGJCLJVlBUzG2vz+O+dxeCwZVH9+W7x+xDu5ZN4g5NZLukJNn5ZvYrwiljgIsILY6rZGYnA38HsoA73f3mCuP/Bnw56m0J7OXu7WskapEMtLWklGaNM7eWt6m4hLsnLOA/b86nqLiEcw/rwQ9O2Jdu7VvEHZrIFyQlyV4O3Ag8QXjS09vRsJ2KHsV4G3AikA9MNbPx7j6nfBp3/2HK9N8HBtds6CLJtmbjViYvKODdz9Ywaf4aPl25kZ7ZLcjtnc1hvTuQm9OBffdqQ6NGyW4YtK20jEemLubvr37Kqg1bOXFQF647aT/6d9GTmiS5EpFk3b0QuHo3Zj0cmOfu8wHM7GHgTGBOJdOPJDy+UaTeKigqZvL8kFDfnb+GT1aExvotm2aRm5PNSfvvzWerNjJh3mqenLEEgLbNG3No7w7k9u7AYb2zOaRne1o0TUZt1915/oPl/OWlj1mwuoghOR2446JDOax3dtyhiVQpEUnWzF4Gvu7ua6P+DsDD7n5SFbN2Jzwtqlw+MLSSdfQG+gCvVTL+SuBKgF69eu1S/CJxKiwqZvKCNUyaX8Ck+WuYu3wDAC2aZJGb04EzD+nO8H4dObB7O5qktLR1dxYVbGJaXiHTFhYyfWEBf/l4FQCNGxn7d29HbnnizenAXm2a1/lnmzhvNX96YS7v569j3y6tufOSXI4fuJdux5GMkYgkC3QqT7AQarZmtlcNr2ME8Ji7l6Yb6e6jCa2cyc3N9Rpet0iNWbupeIfTv+VJtXmTRuT2zuYnJ3VjWN9sDuzenqaNK799xczo3bEVvTu24pzDemxf9nuLCrcn3vsnLeSuCQsA6N2xZTi93Dub3JwO7NO5da2dYp69dB1/euFj3vpkFd3aNecvXz+Yrw3uTlbCT2mLVJSUJFtmZr3cfRFsr3VWJ9EtAXqm9PeIhqUzAvjeHkUpEoN1m7btUFP9aPl63KFZ40bk5nTgxyfuy/B+HTmox86TanW0b9mU4wZ04bgBXQAoLinjw6XrmJ5XyLSFBbz1ySqeeC/8xNq1aMJhvTtEibcDB/dsv8e3zSxas4lbXv6Yp2cupX3LJvzi1IFcPLy3bseRjGXu8VfaohbCo4E3AQO+BFzp7i9WMV9j4BPgeEJynQpcUPEhFmY2AHgB6OPV+MC5ubk+bdq03fkoInts3eZtTF1QwLvRddU5y0JSbdq4EYf16sDwfh0Z1rcjB/dsV+ethN2dvDWbmJZXwPSFobY7b2W45tskyzig/BRzTmhU1al1s2otd/XGrfzrtXk8MHkhWY2My4/sw7eO6Ue7FrodJ5OY2XR3z407jiRJRJIFMLNOwLCod5K7r67mfKcCtxJu4bnb3f9gZr8Fprn7+GiaG4Dm7n59dZapJCt1af2WkFQnzQ+11dlL11EWJdVDe7VnWN+ODO/bsUZqirWhsKh4e8KdvrCA9/PXUVwSXgvdp1Or7TXd3Jxs+nVutcP11I1bS7jz7fmMeWs+W0rKOC+3Jz84oT9d2tb99V/Zc0qyX5SYJJskSrJSmzZs2ca0vMLtrX8/XBIl1axGHNKrPcP7hprq4F7JTKpV2VpSyodL1qU0qCqkoKgYgA4ty08xZ9Mky7jjzc9YvbGYUw7Ym2tP2o9+nVvHHL3sCSXZL1KSTUNJVnZXSWkZazdvo7ComIKiYgo3FVNQtI3CTcWs2rCVGYvX8uGSdZSWOU2yjME9OzCsbzbD+nXk0F4dMjKpVsXdmb+6aPt13Wl5hcxfXQTA0D7ZXH/KAAb36hBzlFITlGS/KCkNn0QSp6zMWbd5GwWbitMmzYKiYtZuKh++jYKiYtZt3lbp8lo2zWJQ17Z899h+DOsbkmpS7kWtTWZGv86t6de5NecNCe0U12zcyqqNW9mvSxvdjiP1WqxJ1sx2eje5uxfUVSxSv7k767eUhGSZLmmmDC/cFJLm2k3FlFVyoqdp40Z0bNWUDi2bkt2qKd07tCS7ZRM6RMM6tGpKdsumdGjVhOxoWH2spe6ujq2b0bGajaJEMlncNdnphFt10h3KOtC3bsOR+mTLtlKenrmEcRMX8vGKDZRWkjGbZNn2RNihZVMG7N02JMfyZJmSTNu3DEmzRZMs1cBEpEqxJll37xPn+qV+WrVhK/dPWsj9kxaypqiYAXu34VtH9yW7PGFGtczy7lZNlTBFpHbEXZMFwMIe7kLCfay/M7NewN7uPiXm0CSDzF2+nrveXsDTM5dSXFrG8QP24oqj+jC8X0clURGJRSKSLPBvoAw4DvgdsAF4HBgSZ1CSfGVlzpufruKutxcwYd5qmjdpxHlDenDZkX10O4iIxC4pSXaoux9qZjNg+7OLm8YdlCTX5uJSnpiRz90TFvDZqiK6tG3GdSfvxwWH96J9S206IpIMSUmy26J3wzqAmXUm1GxFdrBy/RbufXchD0xeSOGmbRzYvR23nn8Ipx7YdY+f2ysiUtOSkmT/ATwJ7GVmfwDOBX4Zb0iSJB8uWcfdExbwzKyllJQ5Jw7swje+1JchOR10vVVEEisRSdbdHzCz6YQH/Rtwlrt/FHNYErOyMufVuSu5a8J8Js0voGXTLC4c2pvLjsyhd8dWcYcnIlKlRCRZAHefC8yNOw6J36biEh6bns897+SxYHUR3do15+enDuD8Ib30VhYRySiJSbIiy9ZtZtzEhTw4eSHrt5RwSM/2/HPkYE45YG8aZ+l6q4hkHiVZid37i9dy14QFPP/BMsrcOfmAvbniqL4c1lsPjReRzKYkK7EoLXNenrOcuyYsYGpeIa2bNebSI3IYdUQOPbNbxh2eiEiNSESSNbOzgT8BexEaPhng7t421sCkxm3cWsKjUxdzz8QFLC7YTI8OLfjVaYM4L7cHbZrrequI1C+JSLLAn4HT1aK4/sov3MTYd/J4ZOpiNmwtIbd3B35+ykC+sv/eZDXSLTgiUj8lJcmuUIKtn6YvLOTuCQv434fLMDNOPbArVxzVh0N6to87NBGRWpeUJDvNzB4BngK2lg909yfiC0n2xNpNxVz73/d55aOVtGnemG8e3ZdLh+fQrX2LuEMTEakzSUmybYFNwFdShjmgJJuBPshfx3cemM6K9Vu4/pQBXDysN62aJWVTExGpO4nY87n7ZXHHIHvO3XloymJuGD+bTq2b8ui3hjO4l27DEZGGKxF3+JvZvmb2qpl9GPUfZGZ6dnEG2VxcyrX/ncXPn/yAoX2zefbqLynBikiDl4gkC4wBfgZsA3D3WcCI6sxoZieb2cdmNs/Mrq9kmvPMbI6ZzTazB2ssagFgweoivvbvd3hiRj7XHN+fsZcdTnYrvW5ORCQRp4uBlu4+pcLbVEqqmil6Pd5twIlAPjDVzMa7+5yUafoTEviR0Xtq96rZ0Bu2F2cv59pH3ycry7hn1BCO3U/FKyJSLilJdrWZ9ePz98meCyyrxnyHA/PcfX4038PAmcCclGm+Cdzm7oUA7r6yJgNvqEpKy/i/Fz/mP2/N5+Ae7bjtwkPp0UFPahIRSZWUJPs9YDQwwMyWAAuAC6sxX3dgcUp/PjC0wjT7ApjZO0AWcIO7v1BxQWZ2JXAlQK9evXY1/gZl5fotXPXQDKYsKODiYb355WkDadY4K+6wREQSJxFJNqqJnmBmrYBG7r6hBhffGOgPHAv0AN4yswPdfW2FGEYTEj25ubleg+uvVybPX8NVD81g45YSbj3/EM4a3D3ukEREEisRSbacuxft4ixLgJ4p/T2iYanygcnuvg1YYGafEJLu1N0OtAFyd8a8PZ8/vfAxvbNbcv8VQ9lv7zZxhyUikmhJaV28u6YC/c2sj5k1JbRIHl9hmqcItVjMrBPh9PH8ugwy063fso1v3z+dm56fy0n7d+Hpq45UghURqYbYa7Jm1ggY5u4Td3Vedy8xs6uAFwnXW+9299lm9ltgmruPj8Z9xczmAKXAT9x9TQ1+hHrto2Xr+c7908kv3MyvThvE5UfmUKEVuIiIVMLc47/8aGYz3H1w3HGUy83N9WnTpsUdRuwen57PL576gLbNm3DbhYcyJCc77pBEJMHMbLq758YdR5LEXpONvGpm5wBPeBKyfgO3ZVspNz4zh4emLGJ43478Y+RgOrdpFndYIiIZJylJ9lvAj4BSM9uMXtoem8UFm/juA+/xwZJ1fOfYfvz4xH1pnJXpl+5FROKRiCTr7mpFkwCvz13JDx6ZSZk7Yy7J5cRBXeIOSUQkoyUiyQKY2RnA0VHvG+7+bJzxNCSlZc6tr3zCP1+bx6Cubbn9okPp3bFV3GGJiGS8RCRZM7sZGAI8EA26xsyOdPefxRhWg7Bm41aueXgmE+at5vzcntx45v40b6KnN4mI1IREJFngVOAQdy8DMLNxwAzCg/2llkxfWMhVD75HQVExfz7nIM4b0rPqmUREpNqSkmQB2gMFUXe7OAOp79ydsRPz+MNzH9GtfQse/84RHNBdRS4iUtOSkmRvAmaY2euElsVHA2nfDSt7pmhrCT99fBbPzlrGCQO7cMt5B9OuRZO4wxIRqZdiT7LRE5/KgGGE67IAP3X35fFFVT/NW7mBb9//HvNXbeSnJw/gW0f3pVEjPb1JRKS2xJ5k3b3MzK5z90f54nOHpYaMf38p1z8+i5ZNs7j/G0M5ol+nuEMSEan3Yk+ykVfM7FrgEWD7m3jcvaDyWaQ6ikvKuOn5jxg7MY/c3h341wWHsne75nGHJSLSICQlyZ4f/f9eyjAH+sYQS72xbN1mvvvAe8xYtJYrjurD9acMoIme3iQiUmdiT7LRNdnr3f2RuGOpTyZ8upqrH55BcUkZ/77wUE49sGvcIYmINDixJ9nomuxPCKeKZQ9t2VbK31/9lDve/Iz+e7Xm9osOo1/n1nGHJSLSIMWeZCO6JlsDpuUVcN3js5i/qojzcntwwxn707JpUr5iEZGGJyl7YF2T3QNFW0v4vxc/Zty7eXRr14L7rjicL/XvHHdYIiINXiKSrLv3iTuGTPX2p6v42RMfsGTtZi4dnsNPTtqPVs0S8bWKiDR4sTY1NbPrUrq/XmHcTXUfUeZYt3kb1z32PhffNYWmjRvx328N54Yz9leCFRFJkLjv5xiR0l3xZQAn12UgmeSl2cs58a9v8vh7S/jOsf14/uovkZuTHXdYIiJSQdzVHqukO11/g7d641ZuGD+bZ2ctY2DXttw9aoge7C8ikmBxJ1mvpDtdf4Pl7jw9cyk3PjOboq2lXPuVffnWMf30YAkRkYSLO8kebGbrCbXWFlE3Ub+e/Ud4atMvnvyQ1+auZHCv9vz5nIPo36VN3GGJiEg1xJpk3T1rT5dhZicDfweygDvd/eYK40cB/wcsiQb9y93v3NP11jZ356Epi/nj8x9RUub86rRBjDoihyy9NUdEJGPEXZPdI2aWBdwGnAjkA1PNbLy7z6kw6SPuflWdB7ibFq4p4vrHP+Dd+Ws4ol9Hbj77IHp1bBl3WCIisosyOskChwPz3H0+gJk9DJwJVEyyGaG0zLnnnQX85aWPadKoEX88+0BGDOmJmWqvIiKZKNOTbHdgcUp/PjA0zXTnmNnRwCfAD919ccUJzOxK4EqAXr161UKoO/fpig1c9/gsZixay/ED9uL3XzuAru1a1HkcIiJSczI9yVbHM8BD7r7VzL4FjAOOqziRu48GRgPk5ubWWcvmbaVl3P7GZ/zrtXm0apbF30ccwhkHd1PtVUSkHsj0JLsE6JnS34PPGzgB4O5rUnrvBP5cB3FVywf56/jJY+8zd/kGTj+4G785fRCdWjeLOywREakhmZ5kpwL9zawPIbmOAC5IncDMurr7sqj3DOCjug3xi7ZsK+XWVz5lzNvz6diqKWMuyeXEQV3iDktERGpYRidZdy8xs6uAFwm38Nzt7rPN7LfANHcfD1xtZmcAJUABMCq2gIGpeQX89LFZzF9dxPm5Pfn5VwfSrkWTOEMSEZFaYu56sFJFubm5Pm3atBpdZtHWEv78wlzunbSQ7u1bcPPZB3FU/041ug4RkTiZ2XR3z407jiTJ6Jpspnjrk/A6uqXrNjPqiByu/YpeRyci0hBoT1+L1m3axu+em8Nj0/Pp27kVj317OIf11ttyREQaCiXZWvLCh8v51dMfUlBUzPe+3I/vH9ef5k32+CmSIiKSQZRka9iqDeF1dM99sIxBXdtyj15HJyLSYCnJ1qCJ81bz3QffY9PWUn5y0n5ceXRfvY5ORKQBU5KtQb07teLA7u34zemD2GcvvY5ORKShU5KtQd3bt+C+K9I9OllERBoincsUERGpJUqyIiIitURJVkREpJYoyYqIiNQSJVkREZFaoiQrIiJSS5RkRUREaomSrIiISC3R+2TTMLNVwMK449hDnYDVcQeRICqPHak8Pqey2NGelEdvd+9ck8FkOiXZesrMpunlyZ9TeexI5fE5lcWOVB41S6eLRUREaomSrIiISC1Rkq2/RscdQMKoPHak8vicymJHKo8apGuyIiIitUQ1WRERkVqiJCsiIlJLlGTrGTPraWavm9kcM5ttZtfEHVPczCzLzGaY2bNxxxI3M2tvZo+Z2Vwz+8jMhscdU5zM7IfR7+RDM3vIzJrHHVNdMrO7zWylmX2YMizbzF42s0+j/x3ijDHTKcnWPyXAj919EDAM+J6ZDYo5prhdA3wUdxAJ8XfgBXcfABxMAy4XM+sOXA3kuvsBQBYwIt6o6txY4OQKw64HXnX3/sCrUb/sJiXZesbdl7n7e1H3BsJOtHu8UcXHzHoAXwXujDuWuJlZO+Bo4C4Ady9297XxRhW7xkALM2sMtASWxhxPnXL3t4CCCoPPBMZF3eOAs+o0qHpGSbYeM7McYDAwOd5IYnUrcB1QFncgCdAHWAXcE50+v9PMWsUdVFzcfQnwF2ARsAxY5+4vxRtVInRx92VR93KgS5zBZDol2XrKzFoDjwM/cPf1cccTBzM7DVjp7tPjjiUhGgOHAre7+2CgiAZ8KjC61ngm4eCjG9DKzC6KN6pk8XCPp+7z3ANKsvWQmTUhJNgH3P2JuOOJ0ZHAGWaWBzwMHGdm98cbUqzygXx3Lz+z8Rgh6TZUJwAL3H2Vu28DngCOiDmmJFhhZl0Bov8rY44noynJ1jNmZoRrbh+5+1/jjidO7v4zd+/h7jmEBi2vuXuDram4+3JgsZntFw06HpgTY0hxWwQMM7OW0e/meBpwQ7AU44FLo+5LgadjjCXjKcnWP0cCFxNqbTOjv1PjDkoS4/vAA2Y2CzgEuCnmeGIT1egfA94DPiDsDxvUIwXN7CHgXWA/M8s3syuAm4ETzexTQm3/5jhjzHR6rKKIiEgtUU1WRESklijJioiI1BIlWRERkVqiJCsiIlJLlGRFRERqiZKsZCQzczO7JaX/WjO7IcaQqsXM8sysU9xx1DUzu8HMro26R5lZt7hjEqkLSrKSqbYCZ8eVsKIHysvuGUV4jKFIvackK5mqhPDggB9WHGFmY83s3JT+jdH/Y83sTTN72szmm9nNZnahmU0xsw/MrF80XWcze9zMpkZ/R0bDbzCz+8zsHeA+M8sxs9fMbJaZvWpmvdLE0tHMXoreWXonYCnjLorWPdPM/mNmWWnmH2JmE83s/WjaNtF63zaz96K/I1I+31tm9pyZfWxmd5hZo2jc7WY2LYrjxpTl3xy9e3iWmf0lzfqzzeypaPwkMzsoGt7azO6Jym2WmZ2TWtZR97lmNrbC8s4FcgkPxJhpZi3M7NdROX9oZqOjpy9hZlenxPbwF7YAkUzg7vrTX8b9ARuBtkAe0A64FrghGjcWODd12uj/scBaoCvQDFgC3BiNuwa4Nep+EDgq6u5FeEQlwA3AdKBF1P8McGnUfTnwVJo4/wH8Our+KuFh652AgdH8TaJx/wYuqTBvU2A+MCTqb0t4yH9LoHk0rD8wLeXzbQH6Et6N+nJ5OQDZ0f8s4A3gIKAj8DGfP5SmfZr4/wn8Juo+DpgZdf+pvLyi/g6pZR11nwuMTSm7a6PuNwjvcCU1tqj7PuD0qHsp0Kyy2PSnv0z40ykvyVjuvt7M7iW8eHtzNWeb6tFrvMzsM6D81WYfAF+Ouk8ABkUVKoC20VuNAMa7e/m6hgNnR933AX9Os76jy6dx9+fMrDAafjxwGDA1Wk8Lvvgg9v2AZe4+tfzzRnG3Av5lZocApcC+KfNMcff50XQPAUcRHh14npldSUjSXYFBhOcWbwHuMrNngWfTxH8UcE60/teimnnbqIy2v+Dc3QvTzFtdXzaz6wgHD9nAbMIByCxCjfcp4Kk9WL5IbJRkJdPdSnj27D0pw0qILoVEp0ubpozbmtJdltJfxue/h0bAMHffkrqiKBkW1VDcBoxz95/txrw/BFYABxNiTY2z4nNS3cz6EGr6Q9y9MDqF29zdS8zscELCPxe4ilBb3ROp629e1cRm1pxQi89198VR47Xy+b5KOEg5HfiFmR3o7iV7GJ9IndI1Wclo7l4APApckTI4j1BLBDgDaLKLi32J8CB9AKIaYzoT+bw2dyHwdppp3gIuiJZzCtAhGv4qcK6Z7RWNyzaz3hXm/RjoamZDomnaRA2u2hFquGWEl0GkXss93Mz6RAcX5wMTCKeZi4B1ZtYFOCVaXmugnbs/T0jcB6eJ/+3os2FmxwKroxr1y8D3Usqo/HOtMLOB0fq/lmZ5ABuANlF3eUJdHcVzbrS8RkBPd38d+Gn0mVtXXJBI0inJSn1wC+E6Z7kxwDFm9j7hlO6u1j6vBnKjBjdzgG9XMt33gcssvNHmYsJ13YpuBI42s9mE08aLANx9DvBL4KVo/pcJp3G3c/diQqL8Z/RZXiYkpX8Dl0bDBlT4fFOBfxFe2bYAeNLd3wdmAHMJ15vfiaZtAzwbrX8C8KM08d8AHBZNczOfvwLt90CHqLHS+3x+qv16wmnnicCy9MXGWOAOM5tJOJMwBvgQeDGKH8KBw/1m9kEU+z/cfW0lyxNJLL2FR6SeiGqa17r7aXHHIiKBarIiIiK1RDVZERGRWqKarIiISC1RkhUREaklSrIiIiK1RElWRESklijJioiI1JL/B2ck+2aZ4n26AAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Se concluye que al aumentar el número de capas ocultas aumenta el errror. Pero al observar la consola, se tiene que el mayor f1 score se consigue cuando se tienen 2 capas ocultas, por lo tanto se elige este valor. "
      ],
      "metadata": {
        "id": "mq9gkV9mfqBS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Modelo 8: Experimentadndo con GRU"
      ],
      "metadata": {
        "id": "cZ7HVN8ldQOo"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import torch.optim as optim\n",
        "\n",
        "\n",
        "# Definir la red\n",
        "class NER_RNN_GRU(nn.Module):\n",
        "    def __init__(self, \n",
        "                 input_dim, \n",
        "                 embedding_dim, \n",
        "                 hidden_dim, \n",
        "                 output_dim,\n",
        "                 n_layers, \n",
        "                 bidirectional, \n",
        "                 dropout, \n",
        "                 pad_idx, \n",
        "                 num_layers = 1):\n",
        "\n",
        "        super().__init__()\n",
        "\n",
        "        # Capa de embedding\n",
        "        self.embedding = nn.Embedding(input_dim,\n",
        "                                      embedding_dim,\n",
        "                                      padding_idx=pad_idx,\n",
        "                                      )\n",
        "\n",
        "        # Capa GRU\n",
        "        self.GRUS = nn.ModuleList()\n",
        "        self.GRUS.add_module(f'lstm1', nn.GRU(embedding_dim,\n",
        "                            hidden_dim,\n",
        "                            num_layers=n_layers,\n",
        "                            bidirectional=bidirectional, \n",
        "                            dropout = dropout if n_layers > 1 else 0))\n",
        "        \n",
        "        if num_layers > 1:\n",
        "          for i in range(1, num_layers):\n",
        "            self.GRUS.add_module(f'lstm_{i}', nn.LSTM(hidden_dim,\n",
        "                              hidden_dim,\n",
        "                              num_layers=n_layers,\n",
        "                              bidirectional=bidirectional, \n",
        "                              dropout = dropout if n_layers > 1 else 0))\n",
        "          \n",
        "\n",
        "        \n",
        "\n",
        "        # Capa de salida\n",
        "        self.fc = nn.Linear(hidden_dim * 2 if bidirectional else hidden_dim,\n",
        "                            output_dim)\n",
        "\n",
        "        # Dropout\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "\n",
        "    def to(self, device):\n",
        "        print(f'to device {device}!!!!')\n",
        "        self.embedding.to(device)\n",
        "        for gru in self.GRUS:\n",
        "          gru.to(device)\n",
        "        self.fc.to(device)\n",
        "        self.dropout.to(device)\n",
        "        return self\n",
        "\n",
        "    def forward(self, text):\n",
        "\n",
        "        #text = [sent len, batch size]\n",
        "\n",
        "        # Convertir lo enviado a embedding\n",
        "        embedded = self.dropout(self.embedding(text))\n",
        "        \n",
        "        output = embedded\n",
        "        for gru in self.GRUS:\n",
        "          outputs, hn = gru(output)\n",
        "        #embedded = [sent len, batch size, emb dim]\n",
        "\n",
        "        # Pasar los embeddings por la rnn (LSTM)\n",
        "\n",
        "        #output = [sent len, batch size, hid dim * n directions]\n",
        "        #hidden/cell = [n layers * n directions, batch size, hid dim]\n",
        "\n",
        "        # Predecir usando la capa de salida.\n",
        "        predictions = self.fc(self.dropout(outputs))\n",
        "        #predictions = [sent len, batch size, output dim]\n",
        "\n",
        "        return predictions"
      ],
      "metadata": {
        "id": "Vy2vlmOndSk9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Creamos nuestro modelo.\n",
        "validation_values = []\n",
        "\n",
        "for i in range(10):\n",
        "  HIDDEN_DIM = 2**i  # dimensión de la capas LSTM\n",
        "  model = NER_RNN_GRU(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM,\n",
        "                          N_LAYERS, BIDIRECTIONAL, DROPOUT, PAD_IDX)\n",
        "\n",
        "  baseline_model_name = 'baseline'  # nombre que tendrá el modelo guardado...\n",
        "\n",
        "  model_name = baseline_model_name\n",
        "  criterion = baseline_criterion\n",
        "  n_epochs = baseline_n_epochs\n",
        "\n",
        "  model.apply(init_weights)\n",
        "  print(f'El modelo actual tiene {count_parameters(model):,} parámetros entrenables.')\n",
        "  # Optimizador\n",
        "  optimizer = optim.Adam(model.parameters())\n",
        "\n",
        "  # Enviamos el modelo y la loss a cuda (en el caso en que esté disponible)\n",
        "  model = model.to(device)\n",
        "  criterion = criterion.to(device)\n",
        "\n",
        "\n",
        "  best_valid_loss = float('inf')\n",
        "\n",
        "  for epoch in range(n_epochs):\n",
        "\n",
        "      start_time = time.time()\n",
        "\n",
        "      # Recuerdo: train_iterator y valid_iterator contienen el dataset dividido en batches.\n",
        "\n",
        "      # Entrenar\n",
        "      train_loss, train_precision, train_recall, train_f1 = train(\n",
        "          model, train_iterator, optimizer, criterion)\n",
        "\n",
        "      # Evaluar (valid = validación)\n",
        "      valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "          model, valid_iterator, criterion)\n",
        "\n",
        "      end_time = time.time()\n",
        "\n",
        "      epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
        "\n",
        "      # Si obtuvimos mejores resultados, guardamos este modelo en el almacenamiento (para poder cargarlo luego)\n",
        "      # Si detienen el entrenamiento prematuramente, pueden cargar el modelo en el siguiente recuadro de código.\n",
        "      if valid_loss < best_valid_loss:\n",
        "          best_valid_loss = valid_loss\n",
        "          torch.save(model.state_dict(), '{}.pt'.format(model_name))\n",
        "      # Si ya no mejoramos el loss de validación, terminamos de entrenar.\n",
        "\n",
        "      print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
        "      print(\n",
        "          f'\\tTrain Loss: {train_loss:.3f} | Train f1: {train_f1:.2f} | Train precision: {train_precision:.2f} | Train recall: {train_recall:.2f}'\n",
        "      )\n",
        "      print(\n",
        "          f'\\t Val. Loss: {valid_loss:.3f} |  Val. f1: {valid_f1:.2f} |  Val. precision: {valid_precision:.2f} | Val. recall: {valid_recall:.2f}'\n",
        "      )\n",
        "  # cargar el mejor modelo entrenado.\n",
        "  model.load_state_dict(torch.load('{}.pt'.format(model_name)))\n",
        "\n",
        "  # Limpiar ram de cuda\n",
        "  torch.cuda.empty_cache()\n",
        "\n",
        "  valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "      model, valid_iterator, criterion)\n",
        "\n",
        "  validation_values += [valid_loss]\n",
        "\n",
        "  print(\n",
        "      f'Val. Loss: {valid_loss:.3f} |  Val. f1: {valid_f1:.2f} | Val. precision: {valid_precision:.2f} | Val. recall: {valid_recall:.2f}'\n",
        "  )"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EgLBlObSfrnH",
        "outputId": "697406ae-0492-4c82-f5c9-cafd0446ffe4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "El modelo actual tiene 5,279,214 parámetros entrenables.\n",
            "to device cuda!!!!\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch: 01 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 2.069 | Train f1: 0.04 | Train precision: 0.03 | Train recall: 0.08\n",
            "\t Val. Loss: 1.467 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.556 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.257 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 03 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 1.436 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.190 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 04 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.334 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.132 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 05 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.247 | Train f1: 0.00 | Train precision: 0.01 | Train recall: 0.00\n",
            "\t Val. Loss: 1.072 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 06 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.180 | Train f1: 0.01 | Train precision: 0.23 | Train recall: 0.01\n",
            "\t Val. Loss: 1.020 |  Val. f1: 0.02 |  Val. precision: 0.25 | Val. recall: 0.01\n",
            "Epoch: 07 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.134 | Train f1: 0.06 | Train precision: 0.41 | Train recall: 0.04\n",
            "\t Val. Loss: 0.973 |  Val. f1: 0.18 |  Val. precision: 0.50 | Val. recall: 0.11\n",
            "Epoch: 08 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.089 | Train f1: 0.14 | Train precision: 0.42 | Train recall: 0.08\n",
            "\t Val. Loss: 0.924 |  Val. f1: 0.30 |  Val. precision: 0.51 | Val. recall: 0.21\n",
            "Epoch: 09 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.050 | Train f1: 0.19 | Train precision: 0.41 | Train recall: 0.12\n",
            "\t Val. Loss: 0.893 |  Val. f1: 0.32 |  Val. precision: 0.50 | Val. recall: 0.24\n",
            "Epoch: 10 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.027 | Train f1: 0.21 | Train precision: 0.41 | Train recall: 0.14\n",
            "\t Val. Loss: 0.877 |  Val. f1: 0.33 |  Val. precision: 0.53 | Val. recall: 0.24\n",
            "Val. Loss: 0.877 |  Val. f1: 0.33 | Val. precision: 0.53 | Val. recall: 0.24\n",
            "El modelo actual tiene 5,281,200 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 1.788 | Train f1: 0.04 | Train precision: 0.06 | Train recall: 0.04\n",
            "\t Val. Loss: 1.221 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.370 | Train f1: 0.00 | Train precision: 0.00 | Train recall: 0.00\n",
            "\t Val. Loss: 1.146 |  Val. f1: 0.00 |  Val. precision: 0.02 | Val. recall: 0.00\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
            "  warnings.warn('{} seems not to be NE tag.'.format(chunk))\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch: 03 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.242 | Train f1: 0.01 | Train precision: 0.20 | Train recall: 0.00\n",
            "\t Val. Loss: 1.041 |  Val. f1: 0.13 |  Val. precision: 0.41 | Val. recall: 0.08\n",
            "Epoch: 04 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.144 | Train f1: 0.07 | Train precision: 0.40 | Train recall: 0.04\n",
            "\t Val. Loss: 0.964 |  Val. f1: 0.26 |  Val. precision: 0.49 | Val. recall: 0.18\n",
            "Epoch: 05 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.072 | Train f1: 0.17 | Train precision: 0.43 | Train recall: 0.11\n",
            "\t Val. Loss: 0.916 |  Val. f1: 0.33 |  Val. precision: 0.44 | Val. recall: 0.27\n",
            "Epoch: 06 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.012 | Train f1: 0.25 | Train precision: 0.44 | Train recall: 0.18\n",
            "\t Val. Loss: 0.869 |  Val. f1: 0.33 |  Val. precision: 0.41 | Val. recall: 0.28\n",
            "Epoch: 07 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.956 | Train f1: 0.29 | Train precision: 0.43 | Train recall: 0.22\n",
            "\t Val. Loss: 0.839 |  Val. f1: 0.33 |  Val. precision: 0.40 | Val. recall: 0.29\n",
            "Epoch: 08 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.913 | Train f1: 0.31 | Train precision: 0.45 | Train recall: 0.24\n",
            "\t Val. Loss: 0.812 |  Val. f1: 0.34 |  Val. precision: 0.42 | Val. recall: 0.29\n",
            "Epoch: 09 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.883 | Train f1: 0.32 | Train precision: 0.45 | Train recall: 0.25\n",
            "\t Val. Loss: 0.799 |  Val. f1: 0.35 |  Val. precision: 0.42 | Val. recall: 0.29\n",
            "Epoch: 10 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.861 | Train f1: 0.34 | Train precision: 0.47 | Train recall: 0.26\n",
            "\t Val. Loss: 0.787 |  Val. f1: 0.35 |  Val. precision: 0.43 | Val. recall: 0.30\n",
            "Val. Loss: 0.787 |  Val. f1: 0.35 | Val. precision: 0.43 | Val. recall: 0.30\n",
            "El modelo actual tiene 5,285,424 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.565 | Train f1: 0.04 | Train precision: 0.06 | Train recall: 0.05\n",
            "\t Val. Loss: 1.140 |  Val. f1: 0.00 |  Val. precision: 0.00 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.180 | Train f1: 0.02 | Train precision: 0.16 | Train recall: 0.01\n",
            "\t Val. Loss: 0.952 |  Val. f1: 0.09 |  Val. precision: 0.36 | Val. recall: 0.05\n",
            "Epoch: 03 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.026 | Train f1: 0.18 | Train precision: 0.37 | Train recall: 0.12\n",
            "\t Val. Loss: 0.868 |  Val. f1: 0.31 |  Val. precision: 0.40 | Val. recall: 0.25\n",
            "Epoch: 04 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.929 | Train f1: 0.27 | Train precision: 0.38 | Train recall: 0.21\n",
            "\t Val. Loss: 0.833 |  Val. f1: 0.32 |  Val. precision: 0.38 | Val. recall: 0.28\n",
            "Epoch: 05 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.868 | Train f1: 0.31 | Train precision: 0.40 | Train recall: 0.25\n",
            "\t Val. Loss: 0.815 |  Val. f1: 0.32 |  Val. precision: 0.38 | Val. recall: 0.28\n",
            "Epoch: 06 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.819 | Train f1: 0.35 | Train precision: 0.44 | Train recall: 0.28\n",
            "\t Val. Loss: 0.782 |  Val. f1: 0.34 |  Val. precision: 0.41 | Val. recall: 0.29\n",
            "Epoch: 07 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.783 | Train f1: 0.37 | Train precision: 0.47 | Train recall: 0.30\n",
            "\t Val. Loss: 0.774 |  Val. f1: 0.34 |  Val. precision: 0.41 | Val. recall: 0.29\n",
            "Epoch: 08 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.752 | Train f1: 0.38 | Train precision: 0.49 | Train recall: 0.32\n",
            "\t Val. Loss: 0.775 |  Val. f1: 0.35 |  Val. precision: 0.41 | Val. recall: 0.31\n",
            "Epoch: 09 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.731 | Train f1: 0.39 | Train precision: 0.50 | Train recall: 0.33\n",
            "\t Val. Loss: 0.753 |  Val. f1: 0.36 |  Val. precision: 0.43 | Val. recall: 0.32\n",
            "Epoch: 10 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.709 | Train f1: 0.41 | Train precision: 0.51 | Train recall: 0.34\n",
            "\t Val. Loss: 0.740 |  Val. f1: 0.39 |  Val. precision: 0.46 | Val. recall: 0.34\n",
            "Val. Loss: 0.740 |  Val. f1: 0.39 | Val. precision: 0.46 | Val. recall: 0.34\n",
            "El modelo actual tiene 5,294,880 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 1.373 | Train f1: 0.01 | Train precision: 0.03 | Train recall: 0.01\n",
            "\t Val. Loss: 1.018 |  Val. f1: 0.02 |  Val. precision: 0.27 | Val. recall: 0.01\n",
            "Epoch: 02 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.002 | Train f1: 0.13 | Train precision: 0.30 | Train recall: 0.09\n",
            "\t Val. Loss: 0.831 |  Val. f1: 0.30 |  Val. precision: 0.39 | Val. recall: 0.25\n",
            "Epoch: 03 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.851 | Train f1: 0.32 | Train precision: 0.41 | Train recall: 0.26\n",
            "\t Val. Loss: 0.735 |  Val. f1: 0.37 |  Val. precision: 0.44 | Val. recall: 0.31\n",
            "Epoch: 04 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.743 | Train f1: 0.39 | Train precision: 0.47 | Train recall: 0.33\n",
            "\t Val. Loss: 0.691 |  Val. f1: 0.41 |  Val. precision: 0.45 | Val. recall: 0.37\n",
            "Epoch: 05 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.680 | Train f1: 0.43 | Train precision: 0.50 | Train recall: 0.37\n",
            "\t Val. Loss: 0.658 |  Val. f1: 0.43 |  Val. precision: 0.48 | Val. recall: 0.39\n",
            "Epoch: 06 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.631 | Train f1: 0.46 | Train precision: 0.54 | Train recall: 0.40\n",
            "\t Val. Loss: 0.637 |  Val. f1: 0.43 |  Val. precision: 0.49 | Val. recall: 0.39\n",
            "Epoch: 07 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.588 | Train f1: 0.49 | Train precision: 0.56 | Train recall: 0.43\n",
            "\t Val. Loss: 0.608 |  Val. f1: 0.49 |  Val. precision: 0.53 | Val. recall: 0.45\n",
            "Epoch: 08 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.544 | Train f1: 0.56 | Train precision: 0.63 | Train recall: 0.51\n",
            "\t Val. Loss: 0.575 |  Val. f1: 0.59 |  Val. precision: 0.62 | Val. recall: 0.56\n",
            "Epoch: 09 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.506 | Train f1: 0.60 | Train precision: 0.66 | Train recall: 0.55\n",
            "\t Val. Loss: 0.568 |  Val. f1: 0.62 |  Val. precision: 0.65 | Val. recall: 0.60\n",
            "Epoch: 10 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.478 | Train f1: 0.63 | Train precision: 0.69 | Train recall: 0.59\n",
            "\t Val. Loss: 0.557 |  Val. f1: 0.63 |  Val. precision: 0.66 | Val. recall: 0.60\n",
            "Val. Loss: 0.557 |  Val. f1: 0.63 | Val. precision: 0.66 | Val. recall: 0.60\n",
            "El modelo actual tiene 5,317,824 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.198 | Train f1: 0.09 | Train precision: 0.26 | Train recall: 0.06\n",
            "\t Val. Loss: 0.824 |  Val. f1: 0.34 |  Val. precision: 0.44 | Val. recall: 0.27\n",
            "Epoch: 02 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.771 | Train f1: 0.45 | Train precision: 0.55 | Train recall: 0.38\n",
            "\t Val. Loss: 0.592 |  Val. f1: 0.60 |  Val. precision: 0.72 | Val. recall: 0.52\n",
            "Epoch: 03 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.583 | Train f1: 0.61 | Train precision: 0.68 | Train recall: 0.55\n",
            "\t Val. Loss: 0.500 |  Val. f1: 0.64 |  Val. precision: 0.70 | Val. recall: 0.60\n",
            "Epoch: 04 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.486 | Train f1: 0.67 | Train precision: 0.72 | Train recall: 0.62\n",
            "\t Val. Loss: 0.473 |  Val. f1: 0.67 |  Val. precision: 0.72 | Val. recall: 0.63\n",
            "Epoch: 05 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.429 | Train f1: 0.70 | Train precision: 0.74 | Train recall: 0.66\n",
            "\t Val. Loss: 0.474 |  Val. f1: 0.68 |  Val. precision: 0.72 | Val. recall: 0.65\n",
            "Epoch: 06 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.384 | Train f1: 0.73 | Train precision: 0.76 | Train recall: 0.70\n",
            "\t Val. Loss: 0.467 |  Val. f1: 0.70 |  Val. precision: 0.73 | Val. recall: 0.67\n",
            "Epoch: 07 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.345 | Train f1: 0.76 | Train precision: 0.78 | Train recall: 0.73\n",
            "\t Val. Loss: 0.463 |  Val. f1: 0.72 |  Val. precision: 0.75 | Val. recall: 0.70\n",
            "Epoch: 08 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.318 | Train f1: 0.78 | Train precision: 0.80 | Train recall: 0.76\n",
            "\t Val. Loss: 0.460 |  Val. f1: 0.72 |  Val. precision: 0.74 | Val. recall: 0.71\n",
            "Epoch: 09 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.295 | Train f1: 0.80 | Train precision: 0.82 | Train recall: 0.78\n",
            "\t Val. Loss: 0.463 |  Val. f1: 0.73 |  Val. precision: 0.74 | Val. recall: 0.72\n",
            "Epoch: 10 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.273 | Train f1: 0.82 | Train precision: 0.83 | Train recall: 0.81\n",
            "\t Val. Loss: 0.475 |  Val. f1: 0.73 |  Val. precision: 0.74 | Val. recall: 0.72\n",
            "Val. Loss: 0.460 |  Val. f1: 0.72 | Val. precision: 0.74 | Val. recall: 0.71\n",
            "El modelo actual tiene 5,379,840 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 1.008 | Train f1: 0.28 | Train precision: 0.46 | Train recall: 0.22\n",
            "\t Val. Loss: 0.648 |  Val. f1: 0.55 |  Val. precision: 0.72 | Val. recall: 0.45\n",
            "Epoch: 02 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.615 | Train f1: 0.59 | Train precision: 0.69 | Train recall: 0.53\n",
            "\t Val. Loss: 0.489 |  Val. f1: 0.65 |  Val. precision: 0.75 | Val. recall: 0.58\n",
            "Epoch: 03 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.458 | Train f1: 0.69 | Train precision: 0.74 | Train recall: 0.65\n",
            "\t Val. Loss: 0.431 |  Val. f1: 0.71 |  Val. precision: 0.76 | Val. recall: 0.67\n",
            "Epoch: 04 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.369 | Train f1: 0.76 | Train precision: 0.78 | Train recall: 0.73\n",
            "\t Val. Loss: 0.407 |  Val. f1: 0.74 |  Val. precision: 0.78 | Val. recall: 0.70\n",
            "Epoch: 05 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.312 | Train f1: 0.79 | Train precision: 0.81 | Train recall: 0.78\n",
            "\t Val. Loss: 0.395 |  Val. f1: 0.75 |  Val. precision: 0.78 | Val. recall: 0.73\n",
            "Epoch: 06 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.271 | Train f1: 0.82 | Train precision: 0.83 | Train recall: 0.81\n",
            "\t Val. Loss: 0.410 |  Val. f1: 0.75 |  Val. precision: 0.77 | Val. recall: 0.73\n",
            "Epoch: 07 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.242 | Train f1: 0.84 | Train precision: 0.85 | Train recall: 0.83\n",
            "\t Val. Loss: 0.413 |  Val. f1: 0.75 |  Val. precision: 0.76 | Val. recall: 0.75\n",
            "Epoch: 08 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.217 | Train f1: 0.85 | Train precision: 0.86 | Train recall: 0.85\n",
            "\t Val. Loss: 0.427 |  Val. f1: 0.76 |  Val. precision: 0.78 | Val. recall: 0.75\n",
            "Epoch: 09 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.195 | Train f1: 0.87 | Train precision: 0.88 | Train recall: 0.87\n",
            "\t Val. Loss: 0.437 |  Val. f1: 0.76 |  Val. precision: 0.77 | Val. recall: 0.76\n",
            "Epoch: 10 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.177 | Train f1: 0.88 | Train precision: 0.89 | Train recall: 0.88\n",
            "\t Val. Loss: 0.447 |  Val. f1: 0.76 |  Val. precision: 0.77 | Val. recall: 0.75\n",
            "Val. Loss: 0.395 |  Val. f1: 0.75 | Val. precision: 0.78 | Val. recall: 0.73\n",
            "El modelo actual tiene 5,568,384 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.950 | Train f1: 0.33 | Train precision: 0.51 | Train recall: 0.26\n",
            "\t Val. Loss: 0.636 |  Val. f1: 0.57 |  Val. precision: 0.75 | Val. recall: 0.47\n",
            "Epoch: 02 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.577 | Train f1: 0.62 | Train precision: 0.71 | Train recall: 0.56\n",
            "\t Val. Loss: 0.443 |  Val. f1: 0.68 |  Val. precision: 0.78 | Val. recall: 0.61\n",
            "Epoch: 03 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.410 | Train f1: 0.72 | Train precision: 0.77 | Train recall: 0.69\n",
            "\t Val. Loss: 0.398 |  Val. f1: 0.73 |  Val. precision: 0.78 | Val. recall: 0.69\n",
            "Epoch: 04 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.320 | Train f1: 0.78 | Train precision: 0.81 | Train recall: 0.76\n",
            "\t Val. Loss: 0.378 |  Val. f1: 0.75 |  Val. precision: 0.77 | Val. recall: 0.73\n",
            "Epoch: 05 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.263 | Train f1: 0.82 | Train precision: 0.83 | Train recall: 0.81\n",
            "\t Val. Loss: 0.380 |  Val. f1: 0.76 |  Val. precision: 0.79 | Val. recall: 0.75\n",
            "Epoch: 06 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.229 | Train f1: 0.84 | Train precision: 0.85 | Train recall: 0.83\n",
            "\t Val. Loss: 0.398 |  Val. f1: 0.77 |  Val. precision: 0.79 | Val. recall: 0.75\n",
            "Epoch: 07 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 0.196 | Train f1: 0.86 | Train precision: 0.87 | Train recall: 0.86\n",
            "\t Val. Loss: 0.404 |  Val. f1: 0.78 |  Val. precision: 0.79 | Val. recall: 0.76\n",
            "Epoch: 08 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.174 | Train f1: 0.88 | Train precision: 0.88 | Train recall: 0.88\n",
            "\t Val. Loss: 0.402 |  Val. f1: 0.77 |  Val. precision: 0.77 | Val. recall: 0.77\n",
            "Epoch: 09 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.153 | Train f1: 0.90 | Train precision: 0.90 | Train recall: 0.90\n",
            "\t Val. Loss: 0.444 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.76\n",
            "Epoch: 10 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.141 | Train f1: 0.91 | Train precision: 0.91 | Train recall: 0.91\n",
            "\t Val. Loss: 0.441 |  Val. f1: 0.77 |  Val. precision: 0.77 | Val. recall: 0.77\n",
            "Val. Loss: 0.378 |  Val. f1: 0.75 | Val. precision: 0.77 | Val. recall: 0.73\n",
            "El modelo actual tiene 6,203,520 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.973 | Train f1: 0.34 | Train precision: 0.50 | Train recall: 0.27\n",
            "\t Val. Loss: 0.701 |  Val. f1: 0.55 |  Val. precision: 0.72 | Val. recall: 0.45\n",
            "Epoch: 02 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.597 | Train f1: 0.60 | Train precision: 0.70 | Train recall: 0.54\n",
            "\t Val. Loss: 0.476 |  Val. f1: 0.68 |  Val. precision: 0.73 | Val. recall: 0.64\n",
            "Epoch: 03 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.419 | Train f1: 0.72 | Train precision: 0.76 | Train recall: 0.68\n",
            "\t Val. Loss: 0.395 |  Val. f1: 0.74 |  Val. precision: 0.77 | Val. recall: 0.72\n",
            "Epoch: 04 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.323 | Train f1: 0.78 | Train precision: 0.80 | Train recall: 0.75\n",
            "\t Val. Loss: 0.403 |  Val. f1: 0.75 |  Val. precision: 0.76 | Val. recall: 0.74\n",
            "Epoch: 05 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.261 | Train f1: 0.82 | Train precision: 0.83 | Train recall: 0.81\n",
            "\t Val. Loss: 0.394 |  Val. f1: 0.76 |  Val. precision: 0.78 | Val. recall: 0.75\n",
            "Epoch: 06 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.219 | Train f1: 0.85 | Train precision: 0.86 | Train recall: 0.84\n",
            "\t Val. Loss: 0.400 |  Val. f1: 0.77 |  Val. precision: 0.79 | Val. recall: 0.76\n",
            "Epoch: 07 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.187 | Train f1: 0.87 | Train precision: 0.88 | Train recall: 0.86\n",
            "\t Val. Loss: 0.411 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.76\n",
            "Epoch: 08 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.164 | Train f1: 0.88 | Train precision: 0.89 | Train recall: 0.88\n",
            "\t Val. Loss: 0.420 |  Val. f1: 0.77 |  Val. precision: 0.77 | Val. recall: 0.77\n",
            "Epoch: 09 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.142 | Train f1: 0.90 | Train precision: 0.90 | Train recall: 0.90\n",
            "\t Val. Loss: 0.440 |  Val. f1: 0.77 |  Val. precision: 0.79 | Val. recall: 0.77\n",
            "Epoch: 10 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.129 | Train f1: 0.91 | Train precision: 0.91 | Train recall: 0.91\n",
            "\t Val. Loss: 0.439 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.76\n",
            "Val. Loss: 0.394 |  Val. f1: 0.76 | Val. precision: 0.78 | Val. recall: 0.75\n",
            "El modelo actual tiene 8,505,984 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 1.194 | Train f1: 0.28 | Train precision: 0.39 | Train recall: 0.23\n",
            "\t Val. Loss: 0.767 |  Val. f1: 0.47 |  Val. precision: 0.65 | Val. recall: 0.37\n",
            "Epoch: 02 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.736 | Train f1: 0.51 | Train precision: 0.65 | Train recall: 0.43\n",
            "\t Val. Loss: 0.574 |  Val. f1: 0.62 |  Val. precision: 0.73 | Val. recall: 0.54\n",
            "Epoch: 03 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.524 | Train f1: 0.65 | Train precision: 0.73 | Train recall: 0.59\n",
            "\t Val. Loss: 0.435 |  Val. f1: 0.70 |  Val. precision: 0.74 | Val. recall: 0.67\n",
            "Epoch: 04 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.389 | Train f1: 0.74 | Train precision: 0.78 | Train recall: 0.70\n",
            "\t Val. Loss: 0.401 |  Val. f1: 0.73 |  Val. precision: 0.76 | Val. recall: 0.72\n",
            "Epoch: 05 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.314 | Train f1: 0.78 | Train precision: 0.81 | Train recall: 0.76\n",
            "\t Val. Loss: 0.386 |  Val. f1: 0.75 |  Val. precision: 0.77 | Val. recall: 0.73\n",
            "Epoch: 06 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.257 | Train f1: 0.82 | Train precision: 0.83 | Train recall: 0.81\n",
            "\t Val. Loss: 0.395 |  Val. f1: 0.76 |  Val. precision: 0.75 | Val. recall: 0.78\n",
            "Epoch: 07 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.215 | Train f1: 0.85 | Train precision: 0.85 | Train recall: 0.84\n",
            "\t Val. Loss: 0.406 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.76\n",
            "Epoch: 08 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.184 | Train f1: 0.87 | Train precision: 0.87 | Train recall: 0.86\n",
            "\t Val. Loss: 0.418 |  Val. f1: 0.76 |  Val. precision: 0.73 | Val. recall: 0.78\n",
            "Epoch: 09 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.164 | Train f1: 0.88 | Train precision: 0.89 | Train recall: 0.88\n",
            "\t Val. Loss: 0.414 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.77\n",
            "Epoch: 10 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.145 | Train f1: 0.90 | Train precision: 0.90 | Train recall: 0.90\n",
            "\t Val. Loss: 0.446 |  Val. f1: 0.78 |  Val. precision: 0.81 | Val. recall: 0.76\n",
            "Val. Loss: 0.386 |  Val. f1: 0.75 | Val. precision: 0.77 | Val. recall: 0.73\n",
            "El modelo actual tiene 17,239,680 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 1.995 | Train f1: 0.09 | Train precision: 0.12 | Train recall: 0.08\n",
            "\t Val. Loss: 1.077 |  Val. f1: 0.01 |  Val. precision: 0.10 | Val. recall: 0.00\n",
            "Epoch: 02 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 1.146 | Train f1: 0.13 | Train precision: 0.37 | Train recall: 0.08\n",
            "\t Val. Loss: 1.003 |  Val. f1: 0.19 |  Val. precision: 0.59 | Val. recall: 0.12\n",
            "Epoch: 03 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.975 | Train f1: 0.29 | Train precision: 0.57 | Train recall: 0.21\n",
            "\t Val. Loss: 0.862 |  Val. f1: 0.39 |  Val. precision: 0.65 | Val. recall: 0.29\n",
            "Epoch: 04 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.813 | Train f1: 0.46 | Train precision: 0.69 | Train recall: 0.35\n",
            "\t Val. Loss: 0.763 |  Val. f1: 0.55 |  Val. precision: 0.67 | Val. recall: 0.48\n",
            "Epoch: 05 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.669 | Train f1: 0.60 | Train precision: 0.75 | Train recall: 0.50\n",
            "\t Val. Loss: 0.629 |  Val. f1: 0.62 |  Val. precision: 0.71 | Val. recall: 0.55\n",
            "Epoch: 06 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.547 | Train f1: 0.67 | Train precision: 0.78 | Train recall: 0.59\n",
            "\t Val. Loss: 0.531 |  Val. f1: 0.67 |  Val. precision: 0.73 | Val. recall: 0.62\n",
            "Epoch: 07 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.444 | Train f1: 0.71 | Train precision: 0.78 | Train recall: 0.65\n",
            "\t Val. Loss: 0.474 |  Val. f1: 0.69 |  Val. precision: 0.71 | Val. recall: 0.67\n",
            "Epoch: 08 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.360 | Train f1: 0.75 | Train precision: 0.79 | Train recall: 0.72\n",
            "\t Val. Loss: 0.447 |  Val. f1: 0.71 |  Val. precision: 0.71 | Val. recall: 0.71\n",
            "Epoch: 09 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.305 | Train f1: 0.79 | Train precision: 0.81 | Train recall: 0.77\n",
            "\t Val. Loss: 0.430 |  Val. f1: 0.74 |  Val. precision: 0.76 | Val. recall: 0.72\n",
            "Epoch: 10 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.258 | Train f1: 0.82 | Train precision: 0.83 | Train recall: 0.81\n",
            "\t Val. Loss: 0.437 |  Val. f1: 0.75 |  Val. precision: 0.76 | Val. recall: 0.74\n",
            "Val. Loss: 0.430 |  Val. f1: 0.74 | Val. precision: 0.76 | Val. recall: 0.72\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "fig = plt.figure()\n",
        "ax = fig.add_subplot(1, 1, 1)\n",
        "ax.plot(validation_values)\n",
        "plt.xlabel('Logaritmo en base 2 del número de neuronas de la capa oculta')\n",
        "plt.ylabel('Error en el conjunto de validación')\n",
        "plt.title('Error en el conjunto de validación en función de el número de neuronas de la capa oculta')\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 296
        },
        "id": "qEpTZkQpgPq4",
        "outputId": "d1c2609a-f0b0-4717-b96d-7ec0172dff69"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAioAAAEXCAYAAACDPG9dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd5gW9bnG8e+9C8tSlqWK9CbGIFgXEYVoNKbYYqIxaqLBaCyJJb2cJMY0NTlJjLHFjkZi1xxjiZpEVOxgQRALRaSJgPRenvPHzMrLuuVdWHa23J/rmmunzzN1n/c3v5lRRGBmZmbWEBVkHYCZmZlZVZyomJmZWYPlRMXMzMwaLCcqZmZm1mA5UTEzM7MGy4mKmZmZNVhOVHJIulDSrfW8zCmSDq7PZeZDUj9JIalFPS/3HUmfStv/R9L1+Yy7HcurdBmSDpT0gqSO2zP/uqDETZKWSHphBy6nXrbFjjq26mq+koZJmiqppK5i25EkjZY0Pus4GqPtuebXxfWnqZE0TtLpdT3fGk9oSe8A3YBNOb3HRMQ5dR1McxQRu9fFfNJk59aI6FUX82sIIuKiLJYhqTdwEXBkRCzZ0THkYSRwGNArIlbtqIU0km2xQ6VJzuXAiRGxIut4zBorSaOB0yNi5PbOK99fHkdFxL9rGklSi4jYWKFfYURsqmqaSuZRq/HN6lpEzAYOyjqOHH2Bd3ZkklKVBrgtdrSBwG8i4pX6WmBl182mpjmso+0423XrJy1yfFrSpZIWAxdKGiPpakkPSVoFfFLSx9MioaXprY6jc+bxkfErWU6ppBskzZc0V9JvJBXmxDBe0h/SovGZkj5XTcw9JN0jaWE67nm1WN/PS3pF0nJJ0yV9Nmee90v6QNI0Sd/ImeZCSXdKukXSinT9y3KG597qGCPpNznDDpY0p8K435c0SdIySXdIKpbUFngY6CFpZdr0kNRK0p8lzUubP0tqVcW6FabbcJGkGcAR+e6DSrbvGkmdcvrtnc63paSBkv4raXHab6ykDlXEtFWxrKSTJc1Kp/1phXH3k/RseozNl3SFpKKc4btLeizdRwsk/U8Vyzg63UdL02P24zVt/8piT8f/upJbCEskPSKpb86wkHSWpLfTZV0pSZXM4zTgemBEul9/qUqK+tP57ZK2j0nn92B6zD0vaWBW26Kujq103AJJP1Zy/i1Wcm51qmzcSqatMubybRoRb0bEA1Vs06skPZzuh6cl7azknFoi6Q1Je+csq8rrTLqd75Z0q6TlwGhVcw2pZD06p+MuV3IrcGCF4bvl7N83JR1fzbzGSfp1uj4rJD0qqUvO8P0lPZMeA68q5za1Ktz6yD1+tOU23GmS3gX+m+67nyk5h99Xck0srTD+1yS9mx4rP82Zd5XntxKXpvNcLuk1SUOqWN/+kp5I1/UxoEuF4VWub3Wqi6+K8UfmLGe2ktIHJB0h6eV0PWZLujBnmvJtdIaS6/l8Sd/fjhiqO797S7o3PX4XS7oi7V/xGlHp7dZ0Xn9ly3VraU3rV62IqLYB3gE+VcWw0cBG4FyS0pnWwBhgGXAgSSJUAkwD/gcoAg4BVgAfS+dRcfziSpZzH3AN0BbYCXgBODMnhg3AN4BC4GxgHqBK5lMATAQuSGMZAMwAPpMOv5Dk9kll67pfGudh6Xx6Arulw54ErgKKgb2AhcAhOfNcCxyexncx8Fxl2zfdFr/JGXYwMKfCuC8APYBOwFTgrMrGTfv9Cngu3WZdgWeAX1exfmcBbwC903k/DgTQoqZ9UMm8/gt8I6f7f4G/pu27pNuwVRrTk8Cfq9geH+4PYDCwEvhEOu2fSI698nH3BfYnOQ77pdvm2+mwEmA+8L10H5UAwytZxq7AqjS+lsAPSY7dopq2fyXb4PPptB9PY/oZ8EzO8AAeADoAfUiOmc9Wc56Nr6o7Z3675BxHi0mO2RbAWOD2DLdFXR5b55Mc073S4+Aa4LZ0WL/c+VZxLavq/Mlnmy4iOc6KSY7xmcApJOf1b4DHa3Gd2QAck47bmmquIZWsx+3Anen2GgLMLY897TcbODXd93uncQ+uYl7jgOnp/m6ddl+SDutJchwdnsZ5WNrdtbL/DWx9/JTvi1vSmFoDXyc5hgYA7YB7gb9VGP+6dNw9gXXAx/M4vz+Tbu8OgEjOue5VrO+zJNeOViTXkhU5MVe7vlUcTzVefyqZrm+63BNJzq3OwF451/Gh6fL3ABYAx1TYRrel23RoepxsSwxVnt8kx/OrwKXpcoqBkRX3cWXnHMnxc3o151SV61ddk2+ishJYmtN8IyeQdyuMPwa4Jad7FPAeUJDT7zbgwsrGr2T53UgO2NY5/U5ky0VhNDAtZ1ibdMPtXMm8hlcS70+AmyrbCRXGuwa4tJL+vUnq75Tk9LuYpB5P+Tz/nTNsMLCmioN9DDUnKl/N6f49WxKArcZN+00HDs/p/gzJLYSqkouzcro/XX4A1rQPKpnX6cB/03aRXDg/UcW4xwAvV7E9PtwfJBf923PGawusp+ok+tvAfTmxvlzFeLnL+DlwZ86wApJ/AgfXtP0rme/DwGkV5rUa6Jt2B+nJn3bfCfy4inmNpvaJyvU5ww4H3shwW9TlsTUVODSnuzvJP/3yi3NNiUpV508+2/S6nGHnAlNzuocCS9P2fK4zT+YMq/YaUmE+hen67pbT7yK2JCpfBp6qMM01wC+q2CbjgJ/ldH8T+Ffa/iPSRCJn+CPA1yqeq5UcP+X7YkDO8P8A38zp/lgl+65XzvAXgBOqiDv3/D4EeIvkn3RBZeOn4/Uh+XHTNqff33NirnZ9qziearz+VDLsJ1UNq2TcP5P+38nZRrn7/vfADdsQQ5XnNzCCJAH6yHnEdiYq1a1fdU2+dVSOiarrqMyuoV8PYHZEbM7pN4ske61uHuX6kmR887WlZLygwjTvlbdExOp0vHZVzKtHeTFUqhB4qprll+sNPFRJ/x7AB7F1xbtZQFlO93s57auBYm37PduK8+pRzbg90lhy46pq/B5svU1zp8tnH+S6B7hcUneSzH0z6TaW1A24jCSBLUnnk08lza3ii4hVSm43ks53V5JfSmUkyWoLkl9ZkOy76Xku48P1jojNkmaz9bGa7/bvC1wm6Y85/ZTOq3wZFedV2TG7raqadxbboi6Prb7AfZJyryebSBKefNTm/KloQU77mkq6y7dxPteZitfImq4h5bqSHNvVbc/hFZbdAvhbJfMqV9Wx0hf4kqSjcoa3JCkRy1fF9ax4PSpPVquNpbrzOyL+m96auBLoK+le4PsRsbxCLD2AJbF1Xa9ZJOcEbMf61nD9qajKc1DScOASkpKyIpKSn7sqjFZx3w/dhhiqO783ALO28f9TtfJcv4+oi8eTo4Z+84DeknKX1Ycke6tuHuVmk/zi6hIRHdKmfWzb0zKzgZk58+kQESURcXie0w6spP88oJO2fpSx4vrlaxXJAVZu51pMW9k2nEdy8uXGNa+K6eez5YQtH7dcrfZBJE+HPEry6+4kkpKQ8vguSmMdGhHtga+S/AOvyVbxSWpDUmRa7mqS2wuD0vn+T858Z5MUN9dkq+2l5D9nb7ZtX84muX2Re6y1johntmFeFW11nEiqzXGSxbaos2MrHf9zFbZrcURsS1y5tmebVhZjTdeZitfIfK8hC0lKBarbnk9UWHa7iDh7G9fjbxXm1TYiLkmH53O9qrieFa9HG9k64atKdec3EfGXiNiXpMR6V+AHlcxjPtBRSZ2+3BjK1bS+2xxfBVX9L4GkhOd+oHdElJLU86g4n4r7vvyaXpsYqju/ZwN9KtY7SdXmf1Rl/5PyWb+PqI/3qDxPkhn/UEllyoOBo0jus9YoIuaT/NP7o6T2SipkDZR00DbE8gKwQtKPJLVWUslviKRheUx7A3CqpEPTGHpK2i2SpyKeAS5WUrF1D+A0YFuezX8FOFxSp/RC+e1aTLsA6Ky0clrqNuBnkroqqSB3QTVx3QmcJ6mXkvdl/Lh8wDbug7+T3L8/Lm0vV0JyK3GZpJ5UfkGpzN3AkWkltCKS+je5x28JsBxYKWk3krpK5R4Aukv6tpIKxiVpZl/RncAR6T5uSVKPYx3J/q2tvwI/kbQ7fFhh9EvbMJ/KvArsLmkvJZVBL6zFtFlsi7o8tv4K/FZpxeT02P78NsRU0fZs04pqdZ2pzTUkkici7yV5cKGNpMHA13JGeQDYVUnF85ZpM0w5FSVr4VbgKEmfSdehWEkF//JXILwCnJAuo4zkXK/ObcB3lFRobUfyo+WOPH+5V3l+p+s3PD1OV5HUCdxccQYRMQuYAPxSUpGkkST/i/Jd322KrxJjgU9JOl5SCyWVo/fKmc8HEbFW0n4kP/Qq+nm673cnqYt0xzbEUN35/QJJUneJpLbpdjgwne4V4BOS+qT/a35SzTIWAL20dYXefNbvI/JNVP6pLU+TrJR0X57TERHrSQ6Gz5FU6roKOCUi3sh3HiT/8IqA10luE9xNcm+6VtKT/EiSymoz03iuB0qrmy6d9gWSg+JSkkq1T7AlIz2R5F7dPJKKgb+o5lZZdf5GcsF8h+TifUe1Y28d3xskF4IZSmpx9yCp4DcBmAS8BryU9qvMdST3Y19Nx7u3wvDa7oP7gUHAexHxak7/XwL7kGzDBytZTlXrNwX4FknSMz+NYU7OKN8nOehXpOtyR860K0gqjR1FUrT8NpU8XRYRb5KU8FxOcmwcRfJo/vp8Yqwwr/uA3wG3K3myYzLJObDdIuItkkTt3yTrkvfLvrLYFtTtsXUZybH1qKQVJBVrK0u0amV7tmkl89qW60xtriHnkNwSeY+k7sxNOcteQVIH6IR0Xu+RHIeVPu1Xw3rMJqkU/j8kJTmzSX5YlP/f+DlJycASkvP675XMJteNJNe4J0m2y1qSuj75qPL8Btqn/ZaQ3M5YTFKBvzInkRwvHwC/IKnsC+S1vtsa31Yi4l2SemPfS+N4haTyMCR1hH6VHtsXkCQUFT1BUvH1P8AfIuLRbYihyvM7PX6PInnw4V2S6+yX0+keS+c7ieS20gNVLYOkbtoU4D1Ji2qxfh+hLSXylgUlj+59NSKezDoWMzNrmCT1I0nwWu6I+iMNmV+hnyFJXUkqx72TcShmZmYNkhOVjKT3q98GLk+LAs3MzKwC3/rZwZS8vfYykscTr69YgzytFHgjScnKByS3geZ8ZEZmZmbNkBOVHUjJa8DfIqm8OAd4keRjZ6/njHMX8EBE3CzpEODUiDg5k4DNzMwaGN/62bH2I3lr7oz0aYnbSWqV5xpMUjsakhcL1cWjlmZmZk1Cvm+mtW3Tk63fIjiHjz5K+SrwRZLbQ18ASiR1jojFVKNLly7Rr1+/OgzVzKxpmzhx4qKI6Jp1HFY7TlSy933gCiVfz3yS5M2AmyobUdIZwBkAffr0YcKECfUVo5lZoydpVs1jWUPjWz871ly2ft1xLyq8Fjsi5kXEFyNib+Cnab/c73TkjnttRJRFRFnXrv5RYGZmTZ8TlR3rRWBQ+sroIpK3Rd6fO4KkLtryHaSfkDwBZGZmZjhR2aHStweeQ/L68Kkkn9WeIulXko5ORzsYeFPSWyRfEf1tJsGamZk1QH48uZEqKysL11ExM8ufpIkRUZZ1HFY7LlExMzOzBsuJipmZmTVYTlSamQ9WrWfdxkqffjYzM2twnKg0I2s3bOLL1zzLebe9zMZNm7MOx8zMrEZOVJqR4paFnDS8D49MWcD373qVzZtdkdrMzBo2v5m2mTn1wP6sXr+J/33kTVoXteCiLwxBUtZhmZmZVcqJSjP0rU/uwqp1G7lq3HTaFBXysyM+7mTFzMwaJCcqzdQPPvMxVq/fxA3jZ9K2VQu+e9iuWYdkZmb2EU5UmilJXHDkYFav38hf/vM2bYsKOfOggVmHZWZmthUnKs1YQYG4+It7sHr9Ji5++A3aFBVy8oh+WYdlZmb2IScqzVxhgbj0y3uxdsMmfv5/U2hd1ILj9u2VdVhmZmaAH082oGVhAVectA8H7tKZH979Kg+9Nj/rkMzMzAAnKpYqblnIdaeUsU+fjpx328v8940FWYdkZmbmRMW2aFPUghtPHcZu3Us469aXeGb6oqxDMjOzZs6Jim2lfXFLbvn6cPp1bsPpN09g4qwlWYdkZmbNmBMV+4hObYu49bTh7FTSitE3vcDkucuyDsnMzJopJypWqZ3aF3Pr6cMpadWCU258gWnvr8g6JDMza4acqFiVenVsw9hv7E+BxEnXPc+sxauyDsnMzJoZJypWrf5d2jL29OGs37SZk657nvnL1mQdkpmZNSNOVKxGH9u5hFu+vh/L1mzgK9c9z8IV67IOyczMmgknKrUg6QBJJ0k6pbzJOqb6skevDtx06jDmLVvDyTc8z9LV67MOyczMmgEnKnmS9DfgD8BIYFjalGUaVD0b1q8T151SxoyFq/jaTS+yYu2GrEMyM7Mmzt/6yV8ZMDgiIutAsjRqUFeu/Mo+nHXrRE67eQI3n7ofrYsKsw7LzMyaKJeo5G8ysHPWQTQEhw3uxp+O35MX3/mAM2+dyLqNm7IOyczMmiiXqOSvC/C6pBeAD2uTRsTR2YWUnc/v1ZO1Gzbxo3te47zbXubKk/ahRaHzXjMzq1tOVPJ3YdYBNDRfHtaHVes28asHXuf7d73Kn47fi4ICZR2WmZk1IU5U8hQRT0jqRlKJFuCFiHg/y5gagq+P7M+aDZv430fepHVRCy76whAkJytmZlY3XFZfDUl9ctqPB14AvgQcDzwv6bisYmtIvvXJXfjmwQO57YV3+c2DU2nm9Y3NzKwOuUSlesMlfSki/gj8FBhWXooiqSvwb+DuLANsKH7wmY+xev0mbhg/k7atWvDdw3bNOiQzM2sCnKhUIyLukvSVtLOgwq2exbhE6kOSuODIwaxat5G//Odt2hYVcuZBA7MOy8zMGjknKjWIiLFp678kPQLclnZ/GXgon3lI+ixwGVAIXB8Rl1QY3ge4GeiQjvPjiMhr3g1JQYG45Ng9WLNhExc//AZtigo5eUS/rMMyM7NGzIlKniLiB5KOBQ5Me10bEffVNJ2kQuBK4DBgDvCipPsj4vWc0X4G3BkRV0saTJIA9avTFagnhQXi0i/vxdoNm/j5/02hdVELjtu3V9ZhmZlZI+VEpRYi4h7gnlpOth8wLSJmAEi6Hfg8kJuoBNA+bS8F5m1nqJlqWVjAFSftw2k3v8gP736VNkWFHD60e9ZhmZlZI+Q6FjWQND79u0LS8pxmhaTlecyiJzA7p3tO2i/XhcBXJc0hKU05tw5Cz1Rxy0KuPbmMvft05LzbXubxN5r9k9xmZrYNnKjUICJGpn9LIqJ9TlMSEe1rmj5PJwJjIqIXcDjwN0kf2TeSzpA0QdKEhQsX1tGid5y2rVpw4+hh7Na9hLNuncgz0xdlHZKZmTUyTlTyJGl/SSU53SWShucx6Vygd053r7RfrtOAOwEi4lmgmOSV/VuJiGsjoiwiyrp27VrbVchEaeuW3PL14fTp1IbTb57AxFlLsg7JzMwaEScq+bsaWJnTvSrtV5MXgUGS+ksqAk4A7q8wzrvAoQCSPk6SqDT8IpM8dWpbxNjTh9O1pBWjb3qByXOXZR2SmZk1Ek5U8qfIeeVqRGwmj8rIEbEROAd4BJhK8nTPFEm/klT+QcPvAd+Q9CrJ48+jo4m93nWn9sWMPX04Ja1acMqNLzDt/RVZh2RmZo2Amtj/wx1G0r3AOLaUonwT+GREHJNFPGVlZTFhwoQsFr1dZixcyfHXPEeB4KZTh7F7j9KsQzKzZkLSxIgoyzoOqx2XqOTvLOAAkvolc4DhwBmZRtQIDejajrGnD2fT5uDIy8fzg7teZf6yNVmHZWZmDZRLVBqpxlqiUm7Z6g1cOW4aY55+h4ICOG1kf846aCAlxS2zDs3MmiiXqDROTlTyJKmY5Omc3UkquwIQEV/PIp7GnqiUm/3Bav7w6Jv83yvz6NS2iPMPHcSJ+/WhqIUL+8ysbjlRaZz83yB/fwN2Bj4DPEHymLFrhG6n3p3acNkJe/PPc0bysW4l/OL+KXz60id4+LX5OIk2MzMnKvnbJSJ+DqyKiJuBI0jqqVgdGNqrlL9/Yzg3ji6jZWEBZ499iWOvfoaJsz7IOjQzM8uQE5X8bUj/LpU0hOSbPDtlGE+TI4lDduvGw+eP4pIvDmXOkjUce/WznPW3icxYuLLmGZiZWZPjjxLm71pJHYGfk7ywrR1wQbYhNU0tCgs4Yb8+HL1XD65/aibXPDGdf09dwEnD+3DeoYPo0q5V1iGamVk9cWXaRqqpVKbNx8IV67jsP29x2wuzad2ykLMOGsBpIwfQuqgw69DMrBFxZdrGyYlKDSR9t7rhEfGn+oolV3NKVMpNX7iS3z38Bo++voBu7VvxvcM+xrH79qKwQFmHZmaNgBOVxsl1VGpWkjZlwNlAz7Q5C9gnw7ianYFd23HtKWXceeYIupe25of3TOLwy57i8Tff9xNCZmZNlEtU8iTpSeCIiFiRdpcAD0bEJ7KIpzmWqOSKCB567T1+/8gbzFq8mgN36cxPPvdxhvT0K/nNrHIuUWmcXKKSv27A+pzu9Wk/y4AkjtijO4995yB+cdRgXp+3nCMvH8937niFOUtWZx2emZnVET/1k79bgBck3Zd2HwOMyS4cAyhqUcCpB/bni/v04upx07nx6Zk8+Np8Tj2gH988eBdK2/iV/GZmjZlv/dSCpH2AUWnnkxHxclaxNPdbP1WZu3QNf3r0Le59eQ6lrVtyzid34eQRfWnVwk8ImTV3vvXTODlRqYGk9hGxXFKnyoZHRCavTnWiUr0p85ZxycNv8NTbi+jdqTU/+MxuHLVHdyQ/IWTWXDlRaZycqNRA0gMRcaSkmUDuxhIQETEgi7icqOTnybcWctFDU3njvRXs2auUnxz+cfYf0DnrsMwsA05UGicnKo2UE5X8bdoc3PfyXP746JvMX7aWT318J3702d0Y1K0k69DMrB45UWmcnKjUIK2XUqWIeKm+YsnlRKX21m7YxA3jZ3L1uOmsXr+RLw/rw3cOG8ROJcVZh2Zm9cCJSuPkRKUGkh6vZnBExCH1FkwOJyrbbvHKdVz+32nc+twsiloUcPdZBzC4R/uswzKzHcyJSuPkRKWRcqKy/WYuWsXRV4xn5C5duPqr+2YdjpntYE5UGie/R6UWJA0BBgMf3iuIiFuyi8i2R/8ubTllRF+uGjedGQtXMqBru6xDMjOzCvxm2jxJ+gVwedp8Evg9cHSmQdl2G31Af1oWFnDtkzOyDsXMzCrhRCV/xwGHAu9FxKnAnoA/LNPIdS1pxfFlvbj3pbksWL4263DMzKwCJyr5WxMRm4GNktoD7wO9M47J6sAZowaycfNmbhw/M+tQzMysAicq+ZsgqQNwHTAReAl4NtuQrC706dyGw4d2Z+zz77JszYaswzEzsxxOVPIUEd+MiKUR8VfgMOBr6S0gawLOOmggK9dt5NbnZmUdipmZ5XCikidJ90s6SVLbiHgnIiZlHZPVnSE9Sxk1qAs3Pf0OazdsyjocMzNLOVHJ3x+BkcDrku6WdJwkv9K0CTn74IEsWrmOuyfOyToUMzNLOVHJU0Q8ERHfBAYA1wDHk1SotSZixIDO7NmrlOuemsGmzX4RoplZQ+BEpRYktQaOBc4ChgE3ZxuR1SVJnH3wQGYtXs3Dk+dnHY6ZmeFEJW+S7gSmAocAVwADI+LcbKOyunbY4J0Z0KUtV4+bjj8vYWaWPScq+buBJDk5KyIeT9+pUiNJn5X0pqRpkn5cyfBLJb2SNm9JWlrnkVveCgvEmQcNYMq85YyftijrcMzMmj0nKnmKiEciolaPg0gqBK4EPkfyjaATJQ2uMN/vRMReEbEXyev5762rmG3bHLN3T7q1b8XV46ZnHYqZWbPnRGXH2g+YFhEzImI9cDvw+WrGPxG4rV4isyq1alHIaSP788z0xbw62wVcZmZZcqKyY/UEZud0z0n7fYSkvkB/4L/1EJfV4MT9+lBS3IK/PuFSFTOzLDlRyZMSX5V0QdrdR9J+dbiIE4C7q7u9JOkMSRMkTVi4cGEdLtoqKiluySkj+vKvKe8xY+HKrMMxM2u2nKjk7ypgBMntGYAVJPVPqjOXrT9c2CvtV5kTqOG2T0RcGxFlEVHWtWvXmiO27TL6gP60LCzg2idnZB2KmVmz5UQlf8Mj4lvAWoCIWAIU1TDNi8AgSf0lFZEkI/dXHEnSbkBH/JHDBqVrSSuOL+vFvS/NZcHytVmHY2bWLDlRyd+G9CmeAJDUFaj2EeWI2AicAzxC8g6WOyNiiqRfSTo6Z9QTgNvDL+5ocM4YNZCNmzdz4/iZWYdiZtYstcg6gEbkL8B9wE6SfgscB/yspoki4iHgoQr9LqjQfWHdhWl1qU/nNhw+tDtjn3+Xb35yF0pbt8w6JDOzZsUlKnmKiLHAD4GLgfnAMRFxV7ZRWX0466CBrFy3kbHPz8o6FDOzZseJSg0kdSpvSD5CeBvwd2BB2s+auCE9Sxk1qAs3jn+HtRtq9c4/MzPbTk5UajYRmJD+XQi8Bbydtk/MMC6rR2cfPJBFK9dxz0tzsg7FzKxZcaJSg4joHxEDgH8DR0VEl4joDBwJPJptdFZfRgzozJ69Srn2yRls2uw6z2Zm9cWJSv72TyvGAhARDwMHZBiP1SNJnH3wQGYtXs3Dk+dnHY6ZWbPhRCV/8yT9TFK/tPkpMC/roKz+HDZ4ZwZ0acvV46bjJ8nNzOqHE5X8nQh0JXlE+d60/cRqp7AmpbBAnHnQAKbMW874aYuyDsfMrFlwopKniPggIs6PiL0jYp+I+HZEfJB1XFa/jtm7J93at+Lqcf5YoZlZfXCiYlYLrVoUctrI/jwzfTGvzl6adThmZk2eExWzWjpxvz6UFLfgr0+4VMXMbEdzomJWSyXFLTllRF/+NeU9ZixcmXU4ZmZNmhOVPEnqJek+SQslvS/pHkm9so7LsjH6gP60LCzg2idnZB2KmVmT5kQlfzcB9wPdgR7AP9N+1gx1LWnF8WW9uPeluby/fG3W4ZiZNVlOVPLXNSJuioiNaTOG5BFla6bOGDWQjXrkrJQAACAASURBVJs3c8PTM7MOxcysyXKikr/Fkr4qqTBtvgoszjooy06fzm04fGh3xj73LsvWbMg6HDOzJsmJSv6+DhwPvAfMB44DRmcZkGXvrIMGsnLdRsY+PyvrUMzMmiQnKvnrFRFHR0TXiNgpIo4B+mQdlGVrSM9SRg3qwo3j32Hthk1Zh2Nm1uQ4Ucnf5Xn2s2bm7IMHsmjlOu55aU7WoZiZNTktsg6goZM0guQryV0lfTdnUHugMJuorCEZMaAze/Yq5donZ3DCsD4UFijrkMzMmgyXqNSsCGhHktSV5DTLSeqpWDMnibMPHsisxat5ePL8rMMxM2tSXKJSg4h4AnhC0piIcI1Jq9Rhg3dmQJe2XD1uOkcM7Y7kUhUzs7rgEpX8tZJ0raRHJf23vMk6KGsYCgvEmQcNYMq85YyftijrcMzMmgyXqOTvLuCvwPWAH++wjzhm75786bG3uHrcdEYN8rsAzczqghOV/G2MiKuzDsIarlYtCjltZH8ueugNXp29lD17d8g6JDOzRs+3fvL3T0nflNRdUqfyJuugrGE5cb8+lBS34K9PTM86FDOzJsElKvn7Wvr3Bzn9AhiQQSzWQJUUt+SUEX25atx0ZixcyYCu7bIOycysUXOJSp4ion8ljZMU+4jRB/SnZWEB1z01I+tQzMwaPZeo5EnSKZX1j4hb6jsWa9i6lrTi+LJe3PniHL7zqV3ZqX1x1iGZmTVaLlHJ37CcZhRwIXB0lgFZw3XGqIFs3LyZG56emXUoZmaNmktU8hQR5+Z2S+oA3J5RONbA9enchsOHdmfsc+/yzYN3obR1y6xDMjNrlFyisu1WAf2zDsIarrMOGsjKdRsZ+7xfaGxmtq2cqORJ0j8l3Z82DwBvAvflMd1nJb0paZqkH1cxzvGSXpc0RdLf6zp2y8aQnqWMGtSFG8e/w9oNfkegmdm28K2f/P0hp30jMCsi5lQ3gaRC4ErgMGAO8KKk+yPi9ZxxBgE/AQ6MiCWSdqr70C0rZx88kJOue557XprDV4b3zTocM7NGx4lKntKPE9bWfsC0iJgBIOl24PPA6znjfAO4MiKWpMt5f3tjtYZjxIDO7NmrlGufnMEJw/pQWOCPFZqZ1YZv/dRA0vj07wpJyytpZkr6ZhWT9wRm53TPSfvl2hXYVdLTkp6T9NlqYjlD0gRJExYuXLg9q2X1RBJnHzyQWYtX8/Dk+VmHY2bW6DhRqUFEjEz/lkRE+4oNUAacvx2LaAEMAg4GTgSuS58oqiyWayOiLCLKunb1R+8ai8MG78yALm25etx0IiLrcMzMGhUnKrUgqVBSD0l9ypuIWEySZFRmLtA7p7tX2i/XHOD+iNgQETOBt0gSF2siCgvEmQcNYMq85YyftijrcMzMGhUnKnmSdC6wAHgMeDBtHgCIiKrK9F8EBknqL6kIOAG4v8I4/yBNdCR1IbkV5HevNzHH7N2Tbu1bcfU4f6zQzKw2nKjk73zgYxGxe0QMTZs9qpsgIjYC5wCPAFOBOyNiiqRfSSp/q+0jwGJJrwOPAz9IS2msCWnVopDTRvbnmemLmTRnadbhmJk1GvI98/xIehw4LE0+MldWVhYTJkzIOgyrhRVrN3DAJf9l1KAuXPWVfbMOx6zZkTQxIsqyjsNqx48n528GME7Sg8C68p4R8afsQrLGpKS4JaeM6MtV46YzY+FKBnRtl3VIZmYNnm/95O9dkvopRUBJTmOWt9EH9KdlYQHXPeVqSGZm+XCJSp4i4pcAktql3Suzjcgao64lrTi+rBd3vjiH73xqV3ZqX5x1SGZmDZpLVPIkaYikl4EpwBRJEyXtnnVc1vicMWogGzdv5oanZ2YdiplZg+dEJX/XAt+NiL4R0Rf4HnBdxjFZI9SncxsOH9qdsc+9y7I1G7IOx8ysQXOikr+2EfF4eUdEjAPaZheONWZnHTSQles2Mvb5WVmHYmbWoDlRyd8MST+X1C9tfoZfzGbbaEjPUkYN6sKN499h7YZNWYdjZtZgOVHJ39eBrsC9wD1Al7Sf2TY5++CBLFq5jntempN1KGZmDZaf+slTRCwBzss6Dms6RgzozJ69Srn2yRmcMKwPhQXKOiQzswbHJSp5kvRY7leNJXWU9EiWMVnjJomzDx7IrMWreXhyVZ+LMjNr3pyo5K9LRHz4kZa0hGWnDOOxJuCwwTszoEtbrh43nc2b/TkLM7OKnKjkb7OkPuUdkvoC/s9i26WwQHzrk7swZd5y/vDom1mHY2bW4LiOSv5+CoyX9AQgYBRwRrYhWVPwxX16MvHdJVw1bjo9O7bmK8P7Zh2SmVmD4UQlTxHxL0n7APunvb4dEYuyjMmaBkn86ujdeW/ZWn7+j8ns3L6YQz/eLeuwzMwaBN/6qYWIWBQRD6SNkxSrMy0KC7j8xL3ZvUcp5/z9ZSbNWVrzRGZmzYATFbMGom2rFtwwuozO7Yr4+pgXmf3B6qxDMjPLnBMVswZkp5Jixpy6Hxs2BV+76QWWrFqfdUhmZplyolIDSZ2qa7KOz5qeXXZqx3WnlDFnyRrO+NsEv2LfzJo1Jyo1mwhMSP9WbCZkGJc1Yfv178Sfjt+TF99ZwvfuetXvWDGzZstP/dQgIvpnHYM1T0fu0YN5S9dw0UNv0KO0mJ8eMTjrkMzM6p0TlTxJEvAVoH9E/Dp9+dvOEfFCxqFZE/aNUQOYu2QN1z01k54dWjP6QOfNZta8+NZP/q4CRgAnpd0rgCuzC8eaA0lccNTuHDa4G7984HUemfJe1iGZmdUrJyr5Gx4R3wLWwoff+inKNiRrDgoLxF9O2Js9e3XgvNte5qV3l2QdkplZvXGikr8NkgpJv+8jqSuwOduQrLloXVTIDV8rY+fSYk6/eQLvLFqVdUhmZvXCiUr+/gLcB+wk6bfAeOCibEOy5qRzu1aMOXU/IoLRN73A4pXrsg7JzGyHc6KSp4gYC/wQuBiYDxwTEXdlG5U1N/27tOX6rw1j/rK1nH6L37FiZk2fE5VaiIg3IuLKiLgiIqZmHY81T/v27chlJ+zFK7OXcv7tL7PJ71gxsybMiYpZI/TZId35+RGDeWTKAn79wOtEOFkxs6bJ71Exa6S+PrI/c5eu4YbxM+nVsTWnjxqQdUhmZnXOiYpZI/bTwz/OvKVr+O1DU+nRoTWHD+2edUhmZnXKt37yJOmLkt6WtEzSckkrJC3PY7rPSnpT0jRJP65k+GhJCyW9kjan75g1sKaooEBc+uW92KdPR759xyu8+M4HWYdkZlannKjk7/fA0RFRGhHtI6IkItpXN0H63pUrgc8Bg4ETJVX2wZY7ImKvtLm+7kO3pqy4ZSHXn1JGzw6t+cYtE5i+cGXWIZmZ1RknKvlbsA1P+uwHTIuIGRGxHrgd+Hzdh2bNXce2RYw5dRiFEqNveoGFK/yOFTNrGpyo5G+CpDsknZjeBvqipC/WME1PYHZO95y0X0XHSpok6W5JvessYmtW+nZuyw2jh7FwxTpOu/lFVq/fmHVIZmbbzYlK/toDq4FPA0elzZF1MN9/Av0iYg/gMeDmqkaUdIakCZImLFy4sA4WbU3NXr07cPmJ+zB57jLO/fvLbNzkrzyYWeMmv39hx5E0ArgwIj6Tdv8EICIurmL8QuCDiCitad5lZWUxYcKEugzXmpC/PfsOP/+/KXx1/z78+vNDkJR1SGaZkzQxIsqyjsNqxyUqeZK0q6T/SJqcdu8h6Wc1TPYiMEhSf0lFwAnA/RXmm/s86dGA33hr2+3kEf0486AB3Prcu/z1iRlZh2Nmts2cqOTvOuAnwAaAiJhEknhUKSI2AucAj5AkIHdGxBRJv5J0dDraeZKmSHoVOA8YvYPit2bmR5/ZjaP27MHv/vUG//fK3KzDMTPbJn7hW/7aRMQLFYrQa6ytGBEPAQ9V6HdBTvtPSBIgszpVUCD+8KU9WLB8LT+4axLd2hez/4DOWYdlZlYrLlHJ3yJJA4EAkHQcyVeUzRqsVi0Kue7kMvp0bsMZt0zg7QUrsg7JzKxWnKjk71vANcBukuYC3wbOyjYks5qVtmnJmFOH0aplIaNvepH3l6/NOiQzs7w5UclT+tK2TwFdgd0iYmREzMo6LrN89OrYhptGD2PJ6vWcOuZFVq7zO1bMrHFwolJLEbEqIlx+bo3OkJ6lXPmVfXjjvRV8a+xLfseKmTUKTlTMmpFPfmwnfnPMEJ54ayE/+8dk/B4lM2vo/NRPHiQVAPtHxDNZx2K2vU7crw9zl6zhisen0bNDa849dFDWIZmZVcmJSh4iYrOkK4G9s47FrC5879O7Mm/pGv742Fv06NCaY/ftlXVIZmaV8q2f/P1H0rHyu8itCZDEJcfuwQEDO/Ojeybx9LRFWYdkZlYpJyr5OxO4C1gvabmkFZKWZx2U2bYqalHAX0/el4Fd23HW3ybyxns+nM2s4XGikqeIKImIgohoGRHt0+72Wcdltj3aF7fkplOH0aZVIaNvfJH5y9ZkHZKZ2VacqNSCpKMl/SFtjsw6HrO60KNDa24avR8r123kq9c/z/SFK7MOyczsQ05U8iTpEuB84PW0OV/SxdlGZVY3Bvdoz/VfK+ODVes5+vLx/PPVeVmHZGYGOFGpjcOBwyLixoi4EfgscETGMZnVmf0HdObB80axW/f2nHvby1zwf5NZt3FT1mGZWTPnRKV2OuS0l2YWhdkO0qNDa24/Y3++Mao/tzw7iy/99Vlmf7A667DMrBlzopK/i4CXJY2RdDMwEfhtxjGZ1bmWhQX89IjBXHPyvsxctIoj/vIUj72+IOuwzKyZcqKSh/TNtJuB/YF7gXuAERFxR6aBme1An9l9Zx48dxR9OrfhG7dM4KKHprLB3wcys3rmRCUPEbEZ+GFEzI+I+9PmvazjMtvR+nRuw91nHcDJ+/fl2idncOK1z/kRZjOrV05U8vdvSd+X1FtSp/Im66DMdrTiloX8+pghXHbCXrw+fzlH/GU8T761MOuwzKyZkL+emh9JMyvpHRExoN6DAcrKymLChAlZLNqasWnvr+RbY1/irfdXcO4hgzj/0EEUFvirEtY4SJoYEWVZx2G14xKVPKR1VH4cEf0rNJkkKWZZ2WWndvzjWwfyxb178Zf/vM3JNzzPwhXrsg7LzJowJyp5SOuo/CDrOMwagtZFhfzx+D35/XF7MHHWEo74y1M8P2Nx1mGZWRPlRCV/rqNiluP4st7841sH0q5VC0687jmuGjeNzZt9K9l2nM2bg7cXrMg6DKtnrqOSJ9dRMavcirUb+PG9r/HgpPl88mNd+dPxe9GxbVHWYVkTEhH8Z+r7/PGxt3h38Sqe+tEhdNqGY8x1VBqnFlkH0FhERP+sYzBriEqKW3LFiXszvH8nfv3A6xx5+XiuOGlv9u7TMevQrJGLCJ56exF/fOwtXp29lL6d2/DbLwyltHXLrEOzeuRbPzWQ9MOc9i9VGHZR/Udk1vBI4pQR/bj7rAOQ4PhrnuXG8TNxia1tq+dnLObL1zzHKTe+wKIV6/jdsUP593cP4pi9e/pJs2bGt35qIOmliNinYntl3fXJt36soVq2egPfu+sV/j31fT43ZGd+d9wetC/2L2DLzyuzl/LHR9/kqbcX0bWkFecesgtfHtabVi0Kt3vevvXTOPnWT81URXtl3WbNXmmbllx3ShnXPTWD3/3rTaZePp4rv7IPu/fwdzytaq/PW86fHnuTf099n05ti/jp4R/nq/v3pXXR9ico1rg5UalZVNFeWbeZkdwKOuMTA9mnT0fO+fvLfOGqZ/jl0btzwrDeSM7vbYtp76/g0sfe5sHX5lNS3ILvf3pXRh/Yn3at/O/JEr71UwNJm4BVJKUnrYHyb94LKI6ITMq0fevHGovFK9fx7Tte4am3F/GFvXvym2OG0Nb/hJq9WYtXcdm/3+Yfr8yldctCvj6yP6ePHEBpmx13SfWtn8bJV4saRITLHc22Q+d2rRhz6n5c+fg0Lv33W7w2dxlXf2UfBnUryTo0y8DcpWu44r9vc9eEORQWiNNHDeDMTwygc7tWWYdmDZRLVBopl6hYY/T0tEWcf/vLrFq3iYu+OIQv7N0r65Csnry/Yi1XPT6dvz//LkFw0n59+OYnd6Fb++J6i8ElKo2TS1TqgaTPApcBhcD1EXFJFeMdC9wNDIsIZyHW5By4SxcePG8U5972Mt+541VemLmEXxw1mOKWLrhsqj5YtZ5rnpjOzc++w4ZNwZf27cU5h+xCr45tsg7NGgknKjuYpELgSuAwYA7woqT7I+L1CuOVAOcDz9d/lGb1p1v7Yv5++nD++NhbXD1uOq/OXspVX9mHfl3aZh2a1aFlazZww1MzuGH8TFZv2MQxe/Xk/EMHeT9brTlR2fH2A6ZFxAwASbcDnwderzDer4Hf4Y8fWjPQorCAH312N4b168h37niVIy8fz/8etwefG9o969BsO61at5Exz7zDtU/OYNmaDRw+dGe+/ald2dV1kmwb+c20O15PYHZO95y034ck7QP0jogH6zMws6wdsls3HjxvJAN3asfZY1/il/+cwvqNm7MOy7bB2g2buP6pGXzi94/zv4+8SVnfjjxw7kiu+sq+TlJsu7hEJWOSCoA/AaPzGPcM4AyAPn367NjAzOpJr45tuOvMEVz88FRuevodXnp3KRd/YSi77VxCgV+V3uCt37iZO158lysen8aC5esYuUsXvvvpXdnH33qyOuKnfnYwSSOACyPiM2n3TwAi4uK0uxSYDqxMJ9kZ+AA4uroKtX7qx5qih16bzw/vnsTKdRtp16oFQ3q2Z49eHRjas5Q9epXSp1ObJv/CuMUr1zFl3nKmzFvO5HnLmLZgJa2LCula0ipp2iV/dyrvLmlFl3at6r1C8sZNm7n3pblc9p+3mbt0DWV9O/K9T3+MEQM712scteGnfhonJyo7mKQWwFvAocBc4EXgpIiYUsX444Dv1/TUjxMVa6reW7aWJ99eyGtzljFp7jKmzlvO+k3J7aDS1i3Zo1dpmrh0YI9epXQvLW6UyUtEMG/ZWqbMXZYmJsnf+cvWfjhOr46t2W3nEtZt3MzCFetYuGIdi1etr3R+7YtbbElmSorp2q4VO7XfktiUN53aFG1XSdWmzcEDk+bx53+/zcxFq9ijVynf+/TH+MSgLg1+PzhRaZx862cHi4iNks4BHiF5PPnGiJgi6VfAhIi4P9sIzRqWnUuLOb6sN8eX9QaSWwtvLVjBpDnLmDRnKZPmLOOaJ2ewaXPyI6tLu1Y5yUuSwHQtaVgvD9u8OZi5eNWWhGRu8nfJ6g0AFAgGdG3H8P6d2L1HKbv3aM/gHu3p0KboI/PasGkzH6xaz/vL17Fw5doPE5iFK9axcGXy97U5S3l/xTpWr9/0kekLC0TntkVbldBsSWiKt0pq2hYVfph8RASPTHmPPz32Fm8tWMluO5dw7cn7ctjgbg0+QbHGzSUqjZRLVKw5W7thE1PnL0+Tl2W8Nncpb7+/kvLLWffSYob2LGXP3slto6E9S+nY9qP/9HeE9Rs38/b7K5gybzmvz1vO5LnLmDp/OavSpKGosIBdd27HkA8TklI+3r2ENkV1/7tx1bqNWyUwFZOa91ckic6iles/TPxytW655ZbTyrUbeXPBCgZ0act3DtuVI4Z2b3R1iFyi0jg5UWmknKiYbW3Vuo1Mmbf8w1KX1+YuY+aiVR8O79OpDUN7lbJHettoSM/2lBRv33dlVq/fyNT5K3h93jImz13OlPnLeOu9lR/eqmpTVMjg7u0Z0rOUwT3aM6RHKbvs1I6iFg3rgcvNm4Mlq9dXmtC8n/5dt3ETJ+7Xhy/s3ZMWhQ0r/nw5UWmcnKg0Uk5UzGq2bM0GpsxdxqtpqcukOcuYs2TNh8MHdG3LnjmVdXfvUUrrosorpS5bveHDeiST078zFq6kvCCiY5uWyW2bnu3ZvUcpQ3q0p1/nto2u1KEpc6LSODlRaaScqJhtm8Ur1/Ha3GW8NmdLArNg+TogqSuya7cShvYsZUjPUpav2fBhUpKb4HQvLWb3Hu0/rE8ypGfjrdTbnDhRaZycqDRSTlTM6s6C5WuTp4zmLGXS3KTeywfp0zX9u7T98LZNkpy095d+GyknKo2Tn/oxs2avW/tiug0u5lODuwHJEy7vLV9Lu1Yttrsei5ltHycqZmYVSKJ7aeuswzAz/K0fMzMza8CcqJiZmVmD5UTFzMzMGiwnKmZmZtZgOVExMzOzBsuJipmZmTVYTlTMzMyswfKbaRspSQuBWds4eRdgUR2G09h5e2zhbbE1b48tmsK26BsRXbMOwmrHiUozJGmCXyO9hbfHFt4WW/P22MLbwrLiWz9mZmbWYDlRMTMzswbLiUrzdG3WATQw3h5beFtszdtjC28Ly4TrqJiZmVmD5RIVMzMza7CcqJiZmVmD5USlGZH0WUlvSpom6cdZx5MlSb0lPS7pdUlTJJ2fdUxZk1Qo6WVJD2QdS9YkdZB0t6Q3JE2VNCLrmLIk6TvpeTJZ0m2SirOOyZoPJyrNhKRC4Ergc8Bg4ERJg7ONKlMbge9FxGBgf+BbzXx7AJwPTM06iAbiMuBfEbEbsCfNeLtI6gmcB5RFxBCgEDgh26isOXGi0nzsB0yLiBkRsR64Hfh8xjFlJiLmR8RLafsKkn9EPbONKjuSegFHANdnHUvWJJUCnwBuAIiI9RGxNNuoMtcCaC2pBdAGmJdxPNaMOFFpPnoCs3O659CM/zHnktQP2Bt4PttIMvVn4IfA5qwDaQD6AwuBm9JbYddLapt1UFmJiLnAH4B3gfnAsoh4NNuorDlxomLNmqR2wD3AtyNiedbxZEHSkcD7ETEx61gaiBbAPsDVEbE3sApotnW6JHUkKX3tD/QA2kr6arZRWXPiRKX5mAv0zunulfZrtiS1JElSxkbEvVnHk6EDgaMlvUNyS/AQSbdmG1Km5gBzIqK8hO1uksSlufoUMDMiFkbEBuBe4ICMY7JmxIlK8/EiMEhSf0lFJJXh7s84psxIEkkdhKkR8aes48lSRPwkInpFRD+S4+K/EdFsfzFHxHvAbEkfS3sdCryeYUhZexfYX1Kb9Lw5lGZcudjqX4usA7D6EREbJZ0DPEJSa//GiJiScVhZOhA4GXhN0itpv/+JiIcyjMkajnOBsWlSPwM4NeN4MhMRz0u6G3iJ5Gm5l/Hr9K0e+RX6ZmZm1mD51o+ZmZk1WE5UzMzMrMFyomJmZmYNlhMVMzMza7CcqJg1I5JaSPp2+u0nM7MGz4lKEydpZT0v75n0bz9JJ9XnsvNVX9tkW7/QnE9827EOPwI+iIhN2zh9navvY7QhkjROUlktxh8j6bgdGVPWJF0o6ftp+2hJPbKOybLhRMXqRPqxMiKi/I2V/YAGmajUowb1heZ0H70bEbfswPk3Skr4ethwjSZ5fb81Qz4xmyFJe0l6TtIkSfel3/JA0rC03yuS/lfS5LR/P0lPSXopbQ5I+x+c9r+f9M2dOb+OLwFGpfP6TvqL6B+SHpP0jqRzJH03/ejbc5I6VRdbhfi7SrpH0otpc2Da/0JJN6a/TmdIOq+abXBpWsrxH0ld037fSOf3ajr/Nmn/L0manPZ/Mu1XmG6jF9NYz6y4jHy/0Jy+LfhZSa9J+k2FYT/IWcYvq9mt5ftpqqTr0nV7VFLrdNg4YK+I+JukLunr8st/qeazXwZK+pekiek+3y3tP0bSXyU9D/w+z/23XesraaWk36b74zlJ3dL+1R0X38+ZfnK6rfpJelPSLcBkoHf5cZ/G9uV0/IPTY+puSW9IGitJ6bAL0mVNlnRtTv/zlJSkTZJ0eyXr0FrS7en+ug9onTPs0+n2eUnSXUq+R1Xdfq80hgrjdEv3x6tpU34O/yPdp1MknVFhG+d9jlRYVqd0vpPS/bNH2r+dpJvSbTtJ0rHly8qZ9jhJYyrM7zigjOQFfK+k267GdbYmJCLcNOEGWFlJv0nAQWn7r4A/p+2TgRFp+yXA5LS9DVCctg8CJqTtB5N8sK1/xeWlwx7I6T8amAaUAF2BZcBZ6bBLST4KWGVsFeL/OzAybe9D8hp8gAuBZ4BWQBdgMdCykukD+ErafgFwRdreOWec3wDnpu2vAT3T9g7p3zOAn6XtrYAJuduhkmX2I3kVeftKht0PnJK2fytnG36a5A2gIvlR8QDwiWr2az+SUpy90u47ga+m7eOAsrS9C/BOLffLf4BBaftwktfsA4xJ4yqsxf6r9fpWsv+OStt/n7Mfqjsuvp8z/eR0W/Uj+Vr0/mn/Y4HHSN7c3C3dX91JjuVlJN/HKgCezVlOp5z5/i0nrnlAq9xjpsI6fJfk7dAAe6T7rSzdN08CbdNhPwIuqGT6McBx1cVQYfw7cvZlIVCaOy1JojSZ9BygludIhWVdDvwibT8EeCVt/13u8QB0rHgsA8cBYyruN3KO33zX2U3TaRptUa1tG0mlJBfOJ9JeNwN3SeoAlETEs2n/vwNHpu0tgSsk7QVsAnbNmeULETEzz8U/HknJwgpJy4B/pv1fA/aoKrZK5vMpYHDOj6j2Ob86H4yIdcA6Se+T/MOZU2H6zSQXboBbST6yBjAk/YXfAWhH8rkBgKeBMZLuzBn302nM5fUESkmSuI9sC9X8heYDSf5JQnLR/V3OMj5N8spy0pgGkfwjq8rMiCj/JMBEkn/GNalpv7Qj+QjdXTnbvFXO9HdFxKZa7L/tXd/1JElM+ToelrZXd1xUZVZEPJe2jwRui6T+zgJJTwDDgOUkx/kcACWfXOgHjAc+KemHJMl8J2AKyfabRFIC8A/gH5Us9xPAXwAiYpKkSWn//YHBwNPpehSRJEbVqSqGXIcAp6TL20SSeAGcJ+kLaXtvku29mNqfI7lGku7fiPivpM6S2pPsnxPKR4qIJTWsV3XyWWdrIpyoWD6+AywA9iT5Rbk2Z9iqWsxnXU775pzuzdTusOPjSAAAA9hJREFUWCwg+RWcGwfphT13GZvynG/5dyTGAMdExKuSRpP8kiYizpI0HDgCmChpX5Jf/edGRGUX6tyY8v1Cc2XfshBwcURck8c6lKu4/uW3FDay5VZvcTXTVLZfCoClEbFXFcuszTFQbnvWd0NElE+fu4+rOi5y1x22Xv98Y//IcSWpGLiK5Jf+bEkX5sz7CJJk5Cjgp5KGRsTGPJYj4LGIODGfoGqIoaZpDyZJHkZExGoltwermrbac2Q75R4LNca+PetsjZPrqDQzEbEMWCJpVNrrZOCJiFhK8ot6eNr/hJzJSoH5EbE5HT+fR1tXkNxO2O7YKhn10f9v7/5BowiiOI5/n+m8QFBQELEQJUQRlERsrEyphQiKYIqAaQyIQQuJhWIpxMrCP8TCQrBRbI2igkGTKBpjjBEbIwhRErBJYyFj8d6aJd7qbRE9jt8HAsfu3M7M3tzd25l3WfymcYDntZSpBx/32UzIEfzKGLy9sxFcdOWOvymlNJZSOgfM4VeeQ0BvlMXMWs2skq8k1s1ruUPzUxbPd1du+xBwNJsVMLP1Zra2ZF8zM0BHPC71a5GYBfpoZoeiHWZm26uUq/X1W67+Fo2LGaA9trUDGwuePwwcNs8/WoMHGs//UF/25TgfbT4YdawANqSUHuNLNy347EPeEyLZ3My24cs/AKPAbjPbHPsqZtZKsaptqOIh0BvHbIrZrxbgWwQpbfhsTqbUe2SJ4WxfBEPzMYYe4Et9xL4sf+mrmW2J83aA6vKfJ7X2WRqEApXGt9LMPuf+TgHdwEBMN+/AcwkAeoDBmNqusDg9fBnoNrMJoI3arkLfAD8i6e5kifYWtS3vBLAzEvLeAcdKHB+8/bvMk4U7c3WcBcbwL9L3ufIDkQD4Fs+BmQCu4wnEr2L7NX6fvcnu0NwZSYCvzWxvlfb04b8ImiSXbJtSuo8vwY3EvtuUDP5yLuKB1TieB1FWF9ATY2AK2F9QrpbXb7n6WzQu7gCrzWwKOA58KHj+XXzcTgCPgNMppS9FlUVwP4jndgwBL2JXE3Az+jAOXIqyeVeAZjObxs/RyzjmHJ43dCvO4Qj+nivbhqX68OWSyahrK3APnxmaxnPSRnPly75H8s4DHdH+C/iYAM9pWWWRmA7sie39+FLeM2C24Jg3gKvx2fS9xj5Lg9Ddk+UXM2tOKS3E435gXUqppv/9ISKNw8wWUkp/y+8R+SeUoyJ5+8zsDD4uPuFXdiIiIv+NZlRERESkbilHRUREROqWAhURERGpWwpUREREpG4pUBEREZG6pUBFRERE6pYCFREREalbPwEYgm7BBJEL6AAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Como se tiene que el menor error en el conjunto de validación se obtiene utilizando $2^6$, se elige este número de neuronas. "
      ],
      "metadata": {
        "id": "ODX6ffbqgDa1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##### Probando el número de concatenciones de capas ocultas"
      ],
      "metadata": {
        "id": "IqaBFAE7ktBf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Creamos nuestro modelo.\n",
        "validation_values = []\n",
        "HIDDEN_DIM = 2**6 # dimensión de la capas LSTM\n",
        "for i in range(1,12):\n",
        "  print(f'Número de capas ocultas: {i}')\n",
        "    \n",
        "  model = NER_RNN(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM,\n",
        "                          N_LAYERS, BIDIRECTIONAL, DROPOUT, PAD_IDX, num_layers=i)\n",
        "\n",
        "  baseline_model_name = 'baseline'  # nombre que tendrá el modelo guardado...\n",
        "\n",
        "  model_name = baseline_model_name\n",
        "  criterion = baseline_criterion\n",
        "  n_epochs = baseline_n_epochs\n",
        "\n",
        "  model.apply(init_weights)\n",
        "  print(f'El modelo actual tiene {count_parameters(model):,} parámetros entrenables.')\n",
        "  # Optimizador\n",
        "  optimizer = optim.Adam(model.parameters())\n",
        "\n",
        "  # Enviamos el modelo y la loss a cuda (en el caso en que esté disponible)\n",
        "  model = model.to(device)\n",
        "  criterion = criterion.to(device)\n",
        "\n",
        "\n",
        "  best_valid_loss = float('inf')\n",
        "\n",
        "  for epoch in range(n_epochs):\n",
        "\n",
        "      start_time = time.time()\n",
        "\n",
        "      # Recuerdo: train_iterator y valid_iterator contienen el dataset dividido en batches.\n",
        "\n",
        "      # Entrenar\n",
        "      train_loss, train_precision, train_recall, train_f1 = train(\n",
        "          model, train_iterator, optimizer, criterion)\n",
        "\n",
        "      # Evaluar (valid = validación)\n",
        "      valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "          model, valid_iterator, criterion)\n",
        "\n",
        "      end_time = time.time()\n",
        "\n",
        "      epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
        "\n",
        "      # Si obtuvimos mejores resultados, guardamos este modelo en el almacenamiento (para poder cargarlo luego)\n",
        "      # Si detienen el entrenamiento prematuramente, pueden cargar el modelo en el siguiente recuadro de código.\n",
        "      if valid_loss < best_valid_loss:\n",
        "          best_valid_loss = valid_loss\n",
        "          torch.save(model.state_dict(), '{}.pt'.format(model_name))\n",
        "      # Si ya no mejoramos el loss de validación, terminamos de entrenar.\n",
        "\n",
        "      print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
        "      print(\n",
        "          f'\\tTrain Loss: {train_loss:.3f} | Train f1: {train_f1:.2f} | Train precision: {train_precision:.2f} | Train recall: {train_recall:.2f}'\n",
        "      )\n",
        "      print(\n",
        "          f'\\t Val. Loss: {valid_loss:.3f} |  Val. f1: {valid_f1:.2f} |  Val. precision: {valid_precision:.2f} | Val. recall: {valid_recall:.2f}'\n",
        "      )\n",
        "  # cargar el mejor modelo entrenado.\n",
        "  model.load_state_dict(torch.load('{}.pt'.format(model_name)))\n",
        "\n",
        "  # Limpiar ram de cuda\n",
        "  torch.cuda.empty_cache()\n",
        "\n",
        "  valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "      model, valid_iterator, criterion)\n",
        "\n",
        "  validation_values += [valid_loss]\n",
        "\n",
        "  print(\n",
        "      f'Val. Loss: {valid_loss:.3f} |  Val. f1: {valid_f1:.2f} | Val. precision: {valid_precision:.2f} | Val. recall: {valid_recall:.2f}'\n",
        "  )"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ttJ3UNPAkm0Q",
        "outputId": "4660d316-4795-40b2-f6d5-8b06056708b3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Número de capas ocultas: 1\n",
            "El modelo actual tiene 5,664,896 parámetros entrenables.\n",
            "to device cuda!!!!\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
            "  warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch: 01 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 0.955 | Train f1: 0.31 | Train precision: 0.48 | Train recall: 0.24\n",
            "\t Val. Loss: 0.625 |  Val. f1: 0.56 |  Val. precision: 0.76 | Val. recall: 0.45\n",
            "Epoch: 02 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.561 | Train f1: 0.61 | Train precision: 0.71 | Train recall: 0.54\n",
            "\t Val. Loss: 0.462 |  Val. f1: 0.66 |  Val. precision: 0.74 | Val. recall: 0.61\n",
            "Epoch: 03 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.412 | Train f1: 0.71 | Train precision: 0.76 | Train recall: 0.67\n",
            "\t Val. Loss: 0.406 |  Val. f1: 0.72 |  Val. precision: 0.77 | Val. recall: 0.68\n",
            "Epoch: 04 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.324 | Train f1: 0.78 | Train precision: 0.80 | Train recall: 0.76\n",
            "\t Val. Loss: 0.387 |  Val. f1: 0.74 |  Val. precision: 0.77 | Val. recall: 0.71\n",
            "Epoch: 05 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 0.266 | Train f1: 0.81 | Train precision: 0.82 | Train recall: 0.81\n",
            "\t Val. Loss: 0.385 |  Val. f1: 0.75 |  Val. precision: 0.76 | Val. recall: 0.74\n",
            "Epoch: 06 | Epoch Time: 0m 11s\n",
            "\tTrain Loss: 0.228 | Train f1: 0.84 | Train precision: 0.85 | Train recall: 0.84\n",
            "\t Val. Loss: 0.379 |  Val. f1: 0.76 |  Val. precision: 0.76 | Val. recall: 0.76\n",
            "Epoch: 07 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 0.197 | Train f1: 0.86 | Train precision: 0.86 | Train recall: 0.86\n",
            "\t Val. Loss: 0.382 |  Val. f1: 0.76 |  Val. precision: 0.75 | Val. recall: 0.77\n",
            "Epoch: 08 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.171 | Train f1: 0.88 | Train precision: 0.88 | Train recall: 0.88\n",
            "\t Val. Loss: 0.408 |  Val. f1: 0.76 |  Val. precision: 0.75 | Val. recall: 0.77\n",
            "Epoch: 09 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.152 | Train f1: 0.89 | Train precision: 0.89 | Train recall: 0.90\n",
            "\t Val. Loss: 0.407 |  Val. f1: 0.77 |  Val. precision: 0.77 | Val. recall: 0.77\n",
            "Epoch: 10 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.139 | Train f1: 0.90 | Train precision: 0.90 | Train recall: 0.90\n",
            "\t Val. Loss: 0.426 |  Val. f1: 0.76 |  Val. precision: 0.76 | Val. recall: 0.76\n",
            "Val. Loss: 0.379 |  Val. f1: 0.76 | Val. precision: 0.76 | Val. recall: 0.76\n",
            "Número de capas ocultas: 2\n",
            "El modelo actual tiene 6,050,944 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.955 | Train f1: 0.31 | Train precision: 0.49 | Train recall: 0.25\n",
            "\t Val. Loss: 0.616 |  Val. f1: 0.58 |  Val. precision: 0.75 | Val. recall: 0.48\n",
            "Epoch: 02 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.551 | Train f1: 0.63 | Train precision: 0.73 | Train recall: 0.56\n",
            "\t Val. Loss: 0.459 |  Val. f1: 0.66 |  Val. precision: 0.74 | Val. recall: 0.60\n",
            "Epoch: 03 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.405 | Train f1: 0.72 | Train precision: 0.77 | Train recall: 0.68\n",
            "\t Val. Loss: 0.405 |  Val. f1: 0.72 |  Val. precision: 0.76 | Val. recall: 0.69\n",
            "Epoch: 04 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.318 | Train f1: 0.78 | Train precision: 0.80 | Train recall: 0.76\n",
            "\t Val. Loss: 0.396 |  Val. f1: 0.74 |  Val. precision: 0.75 | Val. recall: 0.73\n",
            "Epoch: 05 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.262 | Train f1: 0.82 | Train precision: 0.83 | Train recall: 0.81\n",
            "\t Val. Loss: 0.378 |  Val. f1: 0.75 |  Val. precision: 0.76 | Val. recall: 0.74\n",
            "Epoch: 06 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.223 | Train f1: 0.84 | Train precision: 0.85 | Train recall: 0.84\n",
            "\t Val. Loss: 0.374 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.76\n",
            "Epoch: 07 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.191 | Train f1: 0.87 | Train precision: 0.87 | Train recall: 0.87\n",
            "\t Val. Loss: 0.392 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.75\n",
            "Epoch: 08 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.169 | Train f1: 0.88 | Train precision: 0.89 | Train recall: 0.88\n",
            "\t Val. Loss: 0.395 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.77\n",
            "Epoch: 09 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.152 | Train f1: 0.89 | Train precision: 0.89 | Train recall: 0.90\n",
            "\t Val. Loss: 0.411 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.76\n",
            "Epoch: 10 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.137 | Train f1: 0.90 | Train precision: 0.90 | Train recall: 0.90\n",
            "\t Val. Loss: 0.410 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.77\n",
            "Val. Loss: 0.374 |  Val. f1: 0.77 | Val. precision: 0.78 | Val. recall: 0.76\n",
            "Número de capas ocultas: 3\n",
            "El modelo actual tiene 6,436,992 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 14s\n",
            "\tTrain Loss: 0.970 | Train f1: 0.29 | Train precision: 0.46 | Train recall: 0.23\n",
            "\t Val. Loss: 0.610 |  Val. f1: 0.58 |  Val. precision: 0.73 | Val. recall: 0.49\n",
            "Epoch: 02 | Epoch Time: 0m 14s\n",
            "\tTrain Loss: 0.557 | Train f1: 0.63 | Train precision: 0.73 | Train recall: 0.55\n",
            "\t Val. Loss: 0.463 |  Val. f1: 0.67 |  Val. precision: 0.77 | Val. recall: 0.60\n",
            "Epoch: 03 | Epoch Time: 0m 14s\n",
            "\tTrain Loss: 0.408 | Train f1: 0.72 | Train precision: 0.77 | Train recall: 0.68\n",
            "\t Val. Loss: 0.405 |  Val. f1: 0.72 |  Val. precision: 0.76 | Val. recall: 0.69\n",
            "Epoch: 04 | Epoch Time: 0m 14s\n",
            "\tTrain Loss: 0.321 | Train f1: 0.78 | Train precision: 0.80 | Train recall: 0.76\n",
            "\t Val. Loss: 0.387 |  Val. f1: 0.74 |  Val. precision: 0.79 | Val. recall: 0.70\n",
            "Epoch: 05 | Epoch Time: 0m 14s\n",
            "\tTrain Loss: 0.262 | Train f1: 0.82 | Train precision: 0.83 | Train recall: 0.81\n",
            "\t Val. Loss: 0.374 |  Val. f1: 0.75 |  Val. precision: 0.76 | Val. recall: 0.75\n",
            "Epoch: 06 | Epoch Time: 0m 14s\n",
            "\tTrain Loss: 0.226 | Train f1: 0.84 | Train precision: 0.85 | Train recall: 0.84\n",
            "\t Val. Loss: 0.377 |  Val. f1: 0.76 |  Val. precision: 0.77 | Val. recall: 0.76\n",
            "Epoch: 07 | Epoch Time: 0m 14s\n",
            "\tTrain Loss: 0.194 | Train f1: 0.86 | Train precision: 0.87 | Train recall: 0.86\n",
            "\t Val. Loss: 0.393 |  Val. f1: 0.76 |  Val. precision: 0.77 | Val. recall: 0.76\n",
            "Epoch: 08 | Epoch Time: 0m 15s\n",
            "\tTrain Loss: 0.174 | Train f1: 0.88 | Train precision: 0.88 | Train recall: 0.88\n",
            "\t Val. Loss: 0.384 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.77\n",
            "Epoch: 09 | Epoch Time: 0m 14s\n",
            "\tTrain Loss: 0.152 | Train f1: 0.89 | Train precision: 0.90 | Train recall: 0.89\n",
            "\t Val. Loss: 0.400 |  Val. f1: 0.78 |  Val. precision: 0.79 | Val. recall: 0.78\n",
            "Epoch: 10 | Epoch Time: 0m 14s\n",
            "\tTrain Loss: 0.136 | Train f1: 0.91 | Train precision: 0.90 | Train recall: 0.91\n",
            "\t Val. Loss: 0.422 |  Val. f1: 0.77 |  Val. precision: 0.76 | Val. recall: 0.79\n",
            "Val. Loss: 0.374 |  Val. f1: 0.75 | Val. precision: 0.76 | Val. recall: 0.75\n",
            "Número de capas ocultas: 4\n",
            "El modelo actual tiene 6,823,040 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 16s\n",
            "\tTrain Loss: 0.962 | Train f1: 0.31 | Train precision: 0.48 | Train recall: 0.25\n",
            "\t Val. Loss: 0.629 |  Val. f1: 0.55 |  Val. precision: 0.74 | Val. recall: 0.45\n",
            "Epoch: 02 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.565 | Train f1: 0.62 | Train precision: 0.72 | Train recall: 0.55\n",
            "\t Val. Loss: 0.473 |  Val. f1: 0.66 |  Val. precision: 0.75 | Val. recall: 0.60\n",
            "Epoch: 03 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.410 | Train f1: 0.72 | Train precision: 0.77 | Train recall: 0.67\n",
            "\t Val. Loss: 0.397 |  Val. f1: 0.73 |  Val. precision: 0.78 | Val. recall: 0.69\n",
            "Epoch: 04 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.324 | Train f1: 0.78 | Train precision: 0.80 | Train recall: 0.76\n",
            "\t Val. Loss: 0.379 |  Val. f1: 0.74 |  Val. precision: 0.75 | Val. recall: 0.74\n",
            "Epoch: 05 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.266 | Train f1: 0.81 | Train precision: 0.83 | Train recall: 0.81\n",
            "\t Val. Loss: 0.381 |  Val. f1: 0.75 |  Val. precision: 0.77 | Val. recall: 0.74\n",
            "Epoch: 06 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.226 | Train f1: 0.84 | Train precision: 0.84 | Train recall: 0.83\n",
            "\t Val. Loss: 0.383 |  Val. f1: 0.76 |  Val. precision: 0.79 | Val. recall: 0.74\n",
            "Epoch: 07 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.197 | Train f1: 0.86 | Train precision: 0.86 | Train recall: 0.86\n",
            "\t Val. Loss: 0.383 |  Val. f1: 0.76 |  Val. precision: 0.76 | Val. recall: 0.76\n",
            "Epoch: 08 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.174 | Train f1: 0.88 | Train precision: 0.88 | Train recall: 0.88\n",
            "\t Val. Loss: 0.397 |  Val. f1: 0.76 |  Val. precision: 0.76 | Val. recall: 0.77\n",
            "Epoch: 09 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.155 | Train f1: 0.89 | Train precision: 0.89 | Train recall: 0.89\n",
            "\t Val. Loss: 0.405 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.77\n",
            "Epoch: 10 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.140 | Train f1: 0.90 | Train precision: 0.90 | Train recall: 0.90\n",
            "\t Val. Loss: 0.400 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.77\n",
            "Val. Loss: 0.379 |  Val. f1: 0.74 | Val. precision: 0.75 | Val. recall: 0.74\n",
            "Número de capas ocultas: 5\n",
            "El modelo actual tiene 7,209,088 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.953 | Train f1: 0.31 | Train precision: 0.46 | Train recall: 0.24\n",
            "\t Val. Loss: 0.611 |  Val. f1: 0.58 |  Val. precision: 0.72 | Val. recall: 0.49\n",
            "Epoch: 02 | Epoch Time: 0m 20s\n",
            "\tTrain Loss: 0.554 | Train f1: 0.63 | Train precision: 0.73 | Train recall: 0.56\n",
            "\t Val. Loss: 0.463 |  Val. f1: 0.67 |  Val. precision: 0.75 | Val. recall: 0.61\n",
            "Epoch: 03 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.402 | Train f1: 0.72 | Train precision: 0.77 | Train recall: 0.68\n",
            "\t Val. Loss: 0.394 |  Val. f1: 0.72 |  Val. precision: 0.76 | Val. recall: 0.69\n",
            "Epoch: 04 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.319 | Train f1: 0.78 | Train precision: 0.80 | Train recall: 0.76\n",
            "\t Val. Loss: 0.385 |  Val. f1: 0.74 |  Val. precision: 0.75 | Val. recall: 0.73\n",
            "Epoch: 05 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.263 | Train f1: 0.82 | Train precision: 0.83 | Train recall: 0.81\n",
            "\t Val. Loss: 0.380 |  Val. f1: 0.75 |  Val. precision: 0.78 | Val. recall: 0.73\n",
            "Epoch: 06 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.223 | Train f1: 0.84 | Train precision: 0.85 | Train recall: 0.84\n",
            "\t Val. Loss: 0.391 |  Val. f1: 0.75 |  Val. precision: 0.75 | Val. recall: 0.75\n",
            "Epoch: 07 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.195 | Train f1: 0.86 | Train precision: 0.87 | Train recall: 0.86\n",
            "\t Val. Loss: 0.396 |  Val. f1: 0.76 |  Val. precision: 0.77 | Val. recall: 0.76\n",
            "Epoch: 08 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.172 | Train f1: 0.88 | Train precision: 0.88 | Train recall: 0.88\n",
            "\t Val. Loss: 0.390 |  Val. f1: 0.76 |  Val. precision: 0.77 | Val. recall: 0.75\n",
            "Epoch: 09 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.151 | Train f1: 0.89 | Train precision: 0.90 | Train recall: 0.89\n",
            "\t Val. Loss: 0.413 |  Val. f1: 0.77 |  Val. precision: 0.80 | Val. recall: 0.75\n",
            "Epoch: 10 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.136 | Train f1: 0.90 | Train precision: 0.90 | Train recall: 0.90\n",
            "\t Val. Loss: 0.424 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.77\n",
            "Val. Loss: 0.380 |  Val. f1: 0.75 | Val. precision: 0.78 | Val. recall: 0.73\n",
            "Número de capas ocultas: 6\n",
            "El modelo actual tiene 7,595,136 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 0.965 | Train f1: 0.30 | Train precision: 0.46 | Train recall: 0.24\n",
            "\t Val. Loss: 0.615 |  Val. f1: 0.57 |  Val. precision: 0.73 | Val. recall: 0.48\n",
            "Epoch: 02 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 0.550 | Train f1: 0.63 | Train precision: 0.73 | Train recall: 0.56\n",
            "\t Val. Loss: 0.462 |  Val. f1: 0.68 |  Val. precision: 0.76 | Val. recall: 0.61\n",
            "Epoch: 03 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 0.398 | Train f1: 0.72 | Train precision: 0.77 | Train recall: 0.69\n",
            "\t Val. Loss: 0.399 |  Val. f1: 0.73 |  Val. precision: 0.76 | Val. recall: 0.70\n",
            "Epoch: 04 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 0.315 | Train f1: 0.78 | Train precision: 0.80 | Train recall: 0.76\n",
            "\t Val. Loss: 0.375 |  Val. f1: 0.75 |  Val. precision: 0.77 | Val. recall: 0.73\n",
            "Epoch: 05 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.262 | Train f1: 0.82 | Train precision: 0.83 | Train recall: 0.81\n",
            "\t Val. Loss: 0.380 |  Val. f1: 0.75 |  Val. precision: 0.75 | Val. recall: 0.75\n",
            "Epoch: 06 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 0.228 | Train f1: 0.84 | Train precision: 0.85 | Train recall: 0.84\n",
            "\t Val. Loss: 0.368 |  Val. f1: 0.77 |  Val. precision: 0.79 | Val. recall: 0.75\n",
            "Epoch: 07 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 0.191 | Train f1: 0.87 | Train precision: 0.87 | Train recall: 0.86\n",
            "\t Val. Loss: 0.376 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.77\n",
            "Epoch: 08 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 0.170 | Train f1: 0.88 | Train precision: 0.89 | Train recall: 0.88\n",
            "\t Val. Loss: 0.395 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.76\n",
            "Epoch: 09 | Epoch Time: 0m 23s\n",
            "\tTrain Loss: 0.152 | Train f1: 0.89 | Train precision: 0.89 | Train recall: 0.90\n",
            "\t Val. Loss: 0.404 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.77\n",
            "Epoch: 10 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 0.137 | Train f1: 0.91 | Train precision: 0.91 | Train recall: 0.91\n",
            "\t Val. Loss: 0.427 |  Val. f1: 0.77 |  Val. precision: 0.77 | Val. recall: 0.78\n",
            "Val. Loss: 0.368 |  Val. f1: 0.77 | Val. precision: 0.79 | Val. recall: 0.75\n",
            "Número de capas ocultas: 7\n",
            "El modelo actual tiene 7,981,184 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.960 | Train f1: 0.29 | Train precision: 0.45 | Train recall: 0.23\n",
            "\t Val. Loss: 0.605 |  Val. f1: 0.57 |  Val. precision: 0.74 | Val. recall: 0.47\n",
            "Epoch: 02 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.554 | Train f1: 0.63 | Train precision: 0.73 | Train recall: 0.56\n",
            "\t Val. Loss: 0.447 |  Val. f1: 0.68 |  Val. precision: 0.76 | Val. recall: 0.61\n",
            "Epoch: 03 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.398 | Train f1: 0.73 | Train precision: 0.78 | Train recall: 0.69\n",
            "\t Val. Loss: 0.402 |  Val. f1: 0.72 |  Val. precision: 0.76 | Val. recall: 0.69\n",
            "Epoch: 04 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.312 | Train f1: 0.78 | Train precision: 0.81 | Train recall: 0.77\n",
            "\t Val. Loss: 0.375 |  Val. f1: 0.74 |  Val. precision: 0.76 | Val. recall: 0.73\n",
            "Epoch: 05 | Epoch Time: 0m 25s\n",
            "\tTrain Loss: 0.258 | Train f1: 0.82 | Train precision: 0.83 | Train recall: 0.81\n",
            "\t Val. Loss: 0.385 |  Val. f1: 0.75 |  Val. precision: 0.76 | Val. recall: 0.74\n",
            "Epoch: 06 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.220 | Train f1: 0.84 | Train precision: 0.85 | Train recall: 0.84\n",
            "\t Val. Loss: 0.384 |  Val. f1: 0.76 |  Val. precision: 0.77 | Val. recall: 0.76\n",
            "Epoch: 07 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.192 | Train f1: 0.86 | Train precision: 0.87 | Train recall: 0.86\n",
            "\t Val. Loss: 0.405 |  Val. f1: 0.76 |  Val. precision: 0.76 | Val. recall: 0.76\n",
            "Epoch: 08 | Epoch Time: 0m 25s\n",
            "\tTrain Loss: 0.167 | Train f1: 0.88 | Train precision: 0.88 | Train recall: 0.88\n",
            "\t Val. Loss: 0.397 |  Val. f1: 0.77 |  Val. precision: 0.79 | Val. recall: 0.76\n",
            "Epoch: 09 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.148 | Train f1: 0.90 | Train precision: 0.90 | Train recall: 0.90\n",
            "\t Val. Loss: 0.409 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.76\n",
            "Epoch: 10 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.137 | Train f1: 0.91 | Train precision: 0.90 | Train recall: 0.91\n",
            "\t Val. Loss: 0.418 |  Val. f1: 0.78 |  Val. precision: 0.79 | Val. recall: 0.77\n",
            "Val. Loss: 0.375 |  Val. f1: 0.74 | Val. precision: 0.76 | Val. recall: 0.73\n",
            "Número de capas ocultas: 8\n",
            "El modelo actual tiene 8,367,232 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.958 | Train f1: 0.31 | Train precision: 0.48 | Train recall: 0.24\n",
            "\t Val. Loss: 0.616 |  Val. f1: 0.56 |  Val. precision: 0.75 | Val. recall: 0.45\n",
            "Epoch: 02 | Epoch Time: 0m 26s\n",
            "\tTrain Loss: 0.558 | Train f1: 0.62 | Train precision: 0.72 | Train recall: 0.55\n",
            "\t Val. Loss: 0.456 |  Val. f1: 0.67 |  Val. precision: 0.73 | Val. recall: 0.62\n",
            "Epoch: 03 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.408 | Train f1: 0.72 | Train precision: 0.77 | Train recall: 0.68\n",
            "\t Val. Loss: 0.413 |  Val. f1: 0.71 |  Val. precision: 0.76 | Val. recall: 0.67\n",
            "Epoch: 04 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.319 | Train f1: 0.78 | Train precision: 0.80 | Train recall: 0.76\n",
            "\t Val. Loss: 0.390 |  Val. f1: 0.74 |  Val. precision: 0.75 | Val. recall: 0.73\n",
            "Epoch: 05 | Epoch Time: 0m 28s\n",
            "\tTrain Loss: 0.267 | Train f1: 0.81 | Train precision: 0.82 | Train recall: 0.80\n",
            "\t Val. Loss: 0.377 |  Val. f1: 0.74 |  Val. precision: 0.75 | Val. recall: 0.74\n",
            "Epoch: 06 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.227 | Train f1: 0.84 | Train precision: 0.85 | Train recall: 0.84\n",
            "\t Val. Loss: 0.384 |  Val. f1: 0.75 |  Val. precision: 0.75 | Val. recall: 0.74\n",
            "Epoch: 07 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.200 | Train f1: 0.86 | Train precision: 0.86 | Train recall: 0.86\n",
            "\t Val. Loss: 0.382 |  Val. f1: 0.76 |  Val. precision: 0.77 | Val. recall: 0.76\n",
            "Epoch: 08 | Epoch Time: 0m 30s\n",
            "\tTrain Loss: 0.173 | Train f1: 0.88 | Train precision: 0.88 | Train recall: 0.88\n",
            "\t Val. Loss: 0.405 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.76\n",
            "Epoch: 09 | Epoch Time: 0m 30s\n",
            "\tTrain Loss: 0.154 | Train f1: 0.89 | Train precision: 0.89 | Train recall: 0.89\n",
            "\t Val. Loss: 0.426 |  Val. f1: 0.77 |  Val. precision: 0.80 | Val. recall: 0.75\n",
            "Epoch: 10 | Epoch Time: 0m 28s\n",
            "\tTrain Loss: 0.138 | Train f1: 0.90 | Train precision: 0.90 | Train recall: 0.90\n",
            "\t Val. Loss: 0.418 |  Val. f1: 0.77 |  Val. precision: 0.77 | Val. recall: 0.77\n",
            "Val. Loss: 0.377 |  Val. f1: 0.74 | Val. precision: 0.75 | Val. recall: 0.74\n",
            "Número de capas ocultas: 9\n",
            "El modelo actual tiene 8,753,280 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.950 | Train f1: 0.31 | Train precision: 0.49 | Train recall: 0.25\n",
            "\t Val. Loss: 0.602 |  Val. f1: 0.58 |  Val. precision: 0.78 | Val. recall: 0.47\n",
            "Epoch: 02 | Epoch Time: 0m 30s\n",
            "\tTrain Loss: 0.546 | Train f1: 0.64 | Train precision: 0.74 | Train recall: 0.56\n",
            "\t Val. Loss: 0.448 |  Val. f1: 0.67 |  Val. precision: 0.76 | Val. recall: 0.61\n",
            "Epoch: 03 | Epoch Time: 0m 30s\n",
            "\tTrain Loss: 0.399 | Train f1: 0.72 | Train precision: 0.77 | Train recall: 0.69\n",
            "\t Val. Loss: 0.414 |  Val. f1: 0.72 |  Val. precision: 0.75 | Val. recall: 0.69\n",
            "Epoch: 04 | Epoch Time: 0m 30s\n",
            "\tTrain Loss: 0.316 | Train f1: 0.78 | Train precision: 0.80 | Train recall: 0.76\n",
            "\t Val. Loss: 0.396 |  Val. f1: 0.74 |  Val. precision: 0.78 | Val. recall: 0.71\n",
            "Epoch: 05 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.259 | Train f1: 0.82 | Train precision: 0.83 | Train recall: 0.81\n",
            "\t Val. Loss: 0.393 |  Val. f1: 0.75 |  Val. precision: 0.76 | Val. recall: 0.73\n",
            "Epoch: 06 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.224 | Train f1: 0.84 | Train precision: 0.85 | Train recall: 0.84\n",
            "\t Val. Loss: 0.390 |  Val. f1: 0.76 |  Val. precision: 0.77 | Val. recall: 0.75\n",
            "Epoch: 07 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.193 | Train f1: 0.86 | Train precision: 0.87 | Train recall: 0.86\n",
            "\t Val. Loss: 0.394 |  Val. f1: 0.76 |  Val. precision: 0.76 | Val. recall: 0.76\n",
            "Epoch: 08 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.170 | Train f1: 0.88 | Train precision: 0.88 | Train recall: 0.88\n",
            "\t Val. Loss: 0.408 |  Val. f1: 0.76 |  Val. precision: 0.76 | Val. recall: 0.77\n",
            "Epoch: 09 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.152 | Train f1: 0.89 | Train precision: 0.89 | Train recall: 0.89\n",
            "\t Val. Loss: 0.407 |  Val. f1: 0.78 |  Val. precision: 0.78 | Val. recall: 0.78\n",
            "Epoch: 10 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.138 | Train f1: 0.90 | Train precision: 0.90 | Train recall: 0.90\n",
            "\t Val. Loss: 0.438 |  Val. f1: 0.77 |  Val. precision: 0.77 | Val. recall: 0.77\n",
            "Val. Loss: 0.390 |  Val. f1: 0.76 | Val. precision: 0.77 | Val. recall: 0.75\n",
            "Número de capas ocultas: 10\n",
            "El modelo actual tiene 9,139,328 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 32s\n",
            "\tTrain Loss: 0.970 | Train f1: 0.30 | Train precision: 0.48 | Train recall: 0.23\n",
            "\t Val. Loss: 0.620 |  Val. f1: 0.57 |  Val. precision: 0.75 | Val. recall: 0.47\n",
            "Epoch: 02 | Epoch Time: 0m 32s\n",
            "\tTrain Loss: 0.565 | Train f1: 0.62 | Train precision: 0.72 | Train recall: 0.55\n",
            "\t Val. Loss: 0.469 |  Val. f1: 0.67 |  Val. precision: 0.73 | Val. recall: 0.62\n",
            "Epoch: 03 | Epoch Time: 0m 32s\n",
            "\tTrain Loss: 0.411 | Train f1: 0.72 | Train precision: 0.77 | Train recall: 0.68\n",
            "\t Val. Loss: 0.408 |  Val. f1: 0.72 |  Val. precision: 0.76 | Val. recall: 0.69\n",
            "Epoch: 04 | Epoch Time: 0m 32s\n",
            "\tTrain Loss: 0.323 | Train f1: 0.78 | Train precision: 0.80 | Train recall: 0.76\n",
            "\t Val. Loss: 0.383 |  Val. f1: 0.74 |  Val. precision: 0.76 | Val. recall: 0.72\n",
            "Epoch: 05 | Epoch Time: 0m 32s\n",
            "\tTrain Loss: 0.266 | Train f1: 0.82 | Train precision: 0.83 | Train recall: 0.81\n",
            "\t Val. Loss: 0.367 |  Val. f1: 0.75 |  Val. precision: 0.76 | Val. recall: 0.73\n",
            "Epoch: 06 | Epoch Time: 0m 32s\n",
            "\tTrain Loss: 0.229 | Train f1: 0.84 | Train precision: 0.84 | Train recall: 0.83\n",
            "\t Val. Loss: 0.391 |  Val. f1: 0.76 |  Val. precision: 0.78 | Val. recall: 0.74\n",
            "Epoch: 07 | Epoch Time: 0m 32s\n",
            "\tTrain Loss: 0.195 | Train f1: 0.86 | Train precision: 0.86 | Train recall: 0.86\n",
            "\t Val. Loss: 0.381 |  Val. f1: 0.76 |  Val. precision: 0.78 | Val. recall: 0.75\n",
            "Epoch: 08 | Epoch Time: 0m 32s\n",
            "\tTrain Loss: 0.174 | Train f1: 0.88 | Train precision: 0.88 | Train recall: 0.88\n",
            "\t Val. Loss: 0.400 |  Val. f1: 0.76 |  Val. precision: 0.76 | Val. recall: 0.77\n",
            "Epoch: 09 | Epoch Time: 0m 32s\n",
            "\tTrain Loss: 0.152 | Train f1: 0.90 | Train precision: 0.90 | Train recall: 0.90\n",
            "\t Val. Loss: 0.401 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.77\n",
            "Epoch: 10 | Epoch Time: 0m 33s\n",
            "\tTrain Loss: 0.140 | Train f1: 0.90 | Train precision: 0.90 | Train recall: 0.90\n",
            "\t Val. Loss: 0.401 |  Val. f1: 0.77 |  Val. precision: 0.77 | Val. recall: 0.78\n",
            "Val. Loss: 0.367 |  Val. f1: 0.75 | Val. precision: 0.76 | Val. recall: 0.73\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "fig = plt.figure()\n",
        "ax = fig.add_subplot(1, 1, 1)\n",
        "ax.plot([i for i in range(1,12)], validation_values)\n",
        "plt.xlabel('Numero de capas ocultas')\n",
        "plt.ylabel('Error en el conjunto de validación')\n",
        "plt.title('Error en el conjunto de validación en función de el número de capas ocultas')\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "id": "bHjOPho7lUc7",
        "outputId": "e7d452ca-a803-4458-e446-d46e9c5107bc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdkAAAEWCAYAAAAq+e1jAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3gc1dX48e9RtVVcJK0kd1tyF8UYgym2aW4JBHiT0AKpb0gIpgUSAgnpIcmPN0ASQgpJgBB6KAktNh3ZdBsbY7nJMu6WJbnJktV1fn/MrD0Wq76j3ZXO53n20e6Ue+/ujubuzC1HVBVjjDHGhF9cpAtgjDHG9FZWyRpjjDE+sUrWGGOM8YlVssYYY4xPrJI1xhhjfGKVrDHGGOMTq2Q9ROQnIvJgD+dZJCKn92SeHSEio0VERSShh/PdJCKz3effF5G/dWTbbuQXMg8ROVVE3hORwd1JPxzEcZ+I7BWR93zMp0c+C7+OrXClKyIniMgaEUkPV9n8JCJfEZElkS5HrHCPkbE9lV+7B6OIbAJygCbP4vtV9Sq/CtWXqGpBONJxK+oHVXV4ONKLBqr6y0jkISIjgF8C56jqXr/L0AEzgDnAcFWt9iuTGPksfOVW0HcBl6jqgUiXx/hLRO4HtqnqLX7l0dFffJ9R1Zfb20hEElS1scWyeFVtam2fEGl0antjwk1VtwKnRbocHqOATX5WsK2Jws/Cb/nAL1R1RU9lGOq8aXoRVW3zAWwCZrey7ivAm8CdwG7gF8D9wJ+AF4BqYDYwCXgd2AcUAed60vjE9iHyGQj8HdgJbHfzifeUYQnwG2Av8DHwqTbez1DgSaDc3fYaz7qf4FwNtrbvecAKoBIoAeZ70nwG2ANsAC5vkebjwAPAAff9Twv1+bqfxS88607H+ZXl3fY7wEpgP/AY0A9IBWqAZqDKfQwFkoHfAjvcx2+B5FbeW7z7GVYAG4EFgAIJ7X0HIT7fGiDDs+w4N91EnJPYq+7xUgE8BAxq5fM44vsAvghsdvf9QYttTwTexjnGdgJ/AJI8+xYAL7nf0S7g+63kca77He3DOWYntff5t3G8fA1Yg3NcLgJGedYpcAVQ7OZ1NyAh0vhfoBbnTlIV8FPcY77FdgqM9RxHdwPP4xxz7wL5kfosCNOx5W4bB9yE8/+3G+d/K8NdN9qbbivnspBl7uBn+kfgv+738CaQi/M/tRdYCxzXifPME8CDOOeSr9PGOSTE+8h0t60E3gN+7i07MNHz/a4DLmwjrQzgPpzzw17g3+7ywcBzbvn3us+He/Z7HfiVm38l8B+O/J//F1Dqfs6FQIFn3aeB1TjH5nbgO21817fg/M+X4ZxDB3rWzwDewjk+twJf8ZTt657tjvhug98r8A2gAah3v9Nn3fXB4+uAW87/8ew7FnjDfV8VwGOtfbaH9ml3g/Yr2Ubgapyr4v7uwbgfONX9kNLdg+b7QBJwplv4CZ6D17v9J/5RgaeBv+BUJtnuF/tNTxkagMtx/pm/5R4woU5YccAy4EduWfJw/unnhTrJtNj3RLecc9x0hgET3XWFOP+A/YApOAfmmZ40a90DKx7nwHwn1OdLxyrZ93D+ITNwTuBXhNrWXfYz4B33MwvgHJA/b+X9XYFzohjhpv0aR54IW/0OQqT1Kkf+0Pg/4M+eg3QOzg+AgPvZ/baVz+PQ9wFMxvlHmOXuewfOsRfc9njgJJzjcLT72VznrkvHOYHf4H5H6cD0EHmMx/mhNwfnB8GNOMduUnuff4jP4Dx330lumW4B3mrxj/4cMAgYiXPMzG/j/2xJa6+9Jw7PcbQb55hNwPkh82gEP4twHlvX4hzTw93j4C/AI+660bRfybb2/9ORz7QC5zjrh3OMfwx8Cef/+hfAa504zzQA57vb9qeNc0iI9/Eozo+LVOAonIpqibsuFafC+ar73Qd/4E5uJa3ncX5sDHa/59Pc5ZnA54AU9xj5F24F7K5/3c33KDfPJznyB9rX3P2CP/RXeNbtBGa6zwcDU1sp29dwjrk8IA14Cvinu24UTj1yiVvuTGCKp2ztVrKhzrnusgvcYyQOuAjn/2CIu+4RnB/4ce53NSNU2Y9Ir90NnAOzCufXQvBxuafwW1psfz/wgOf1TJxfNHGeZY8APwm1fYj8c4A6oL9n2SUcPqC/AmzwrEtxP8TcEGlND1Hem4H7Wp5kQuz7F+DOEMtH4FxlpHuW/Qqn3TqY5suedZOBmhafb2cq2cs8r2/jcOV1xLbushLg057X83BuO7ZWMV7heT3X/RwT2vsOQqT1deBV97ng/NPPamXb84HlrXweh74PnBPWo57tUnF+gbb2A/A64GlPWZe3sp03jx8Cj3vWxeGcSE5v7/MPke5/gf9tkdZB3KtZ97Od4Vn/OHBTK2l9hc5Xsn/zrPs0sDaCn0U4j601wFme10NwKqzgj6v2KtnW/n868pn+1bPuamCN5/XRwD73eUfOM4WedW2eQ1qkE+++34meZb/kcCV7EbC4xT5/AX4cIq0hOHe/Bof6vFpsOwXY63n9OvBrz+vJOP+Poe5uDXI/y4Hu6y3AN4EB7eT5CnCl5/UEz3d9M+7/d4j9XqcblWyI9FYA57nPHwDuwXNV396jo22y52vrbbJb21k2FNiqqs2eZZtxrgTbSiNoFM4vlZ0iElwW12Kf0uATVT3obpfWSlpDRWSfZ1k8sLiN/ING4NzSbmkosEeP7CSxGZgWqnw4J9p+3WiHaZnW0Da2HeqWxVuu1rYfypGfqXe/jnwHXk8Cd4nIEJwrombcz1hEcoDf4fz4SnfT6UiHmiPKp6rVIrI7+FpExuNc3U7D+aGVgHM1Ac53V9LBPA69b1VtFpGtHHmsdvTzHwX8TkRu9ywTN61gHi3TCnXMdlVraUfiswjnsTUKeFpEvOeTJpzKuiM68//T0i7P85oQr4OfcUfOMy3Pke2dQ4ICOMd2W5/n9BZ5JwD/DJHWCDffT/z/iUgKTjPgfJyrTYD0Fn1mWpYhEcgSkQrgVpwrwgDO/z9AFs7dwM/h3Nn5tYisxPlx+XaI8oU6fwV/mHX0OO40EfkScD3OjzZwvtcs9/mNOLfn3xORvcDtqnpvW+mFowu9trNsBzBCROI8Fe1IYH07aQRtxfmlm9XFSqllWh+r6rgu7psfYvkOIENE0j3/JCNxfvV3VjVOBRGU24l9Q32GO3D+6Yo85drRyv47cQ5cPNsGdeo7UNW9IvIizq/qSThXoMHy/dIt69GqukdEzsdpP23PTjct4NBJINOz/k/ActxeoSJyHfB5T/kv7kAeO3CuSIJ5CM5n0pXvcitwq6o+1IV923PEcSIinTlOIvFZhO3Ycrf/mqq+2XKFiIzuQtmCuvOZttSR80zLc2RHzyHlOM0kI3BuwQe39eb9hqrO6WA5M0RkkKrua7HuBpwrx+mqWioiU3D+v8SzTcvvtAHn1vQXcJpLZuPcPRiI80NaAFT1feA8EUkErsK5i+NNKyh4/vLm0Yjz42YrTnNIKJ05jx5x3hSRUcBfgbOAt1W1SURWeMpeitM0iYjMAF4WkUJV3dBaBj0xTvZdnF+MN4pIojvU5DM47QrtUtWdwIvA7SIyQETiRCRfRE7rQlneAw6IyPdEpL+IxIvIUSJyQgf2/TvwVRE5yy3DMBGZqE7vy7eAX4lIPxE5BqezSlfG264APi0iGe4/+XWd2HcXkCkiAz3LHgFuEZGAiGTh3HJtrVyPA9eIyHB3PORNwRVd/A4exmmv+rz7PCgdp/lhv4gMA77bwff3BHCOiMwQkSSc9mbv8ZuO0wGjSkQm4rTNBz0HDBGR60QkWUTSRWR6iDweB852v+NEnBNNHc7321l/Bm4WkQIAERkoIhd0IZ1QPgQKRGSKiPTDuf3YUZH4LMJ5bP0ZuNU9GeIe2+d1oUwtdeczbalT55nOnEPcq8ingJ+ISIqITAa+7NnkOWC8iHzRPd8mijPud1KItHbiNGv8UUQGu9vOclen41yd7xORDODHIYp+mYhMdn/w/gx4wi1fOs6xshunsjs0NExEkkTkUhEZqKoNOP+zzSHSBuf89W0RGSMiaW46j7k/xh4CZovIhSKSICKZ7g8BcM6jn3U/n7HuZ9maXThtvkGpOBVvuVver+K0OwfLf4GIBIdJ7nW3ba38QMcr2WdFpMrzeLqD+6Gq9TiV6qdwfuX8EfiSqq5tc8cjfQmnA8FqnDf2BE57Qqe4B8A5OO0LH7vl+RvOL6329n0PpzPBnTi3PN7g8K+sS3BuLezA6cTx4zZur7flnzj/7JtwTjyPdXRH9/N8BNgoIvtEZChOZ4ylOL0pPwI+cJeF8lecHrAfuts91WJ9Z7+DZ4BxQKmqfuhZ/lNgKs5n+HyIfFp7f0U4vVIfxrky2gts82zyHZxf0Afc9/KYZ98DOB14PoNzu7AYOCNEHuuAy3DGSVa423/GPYY7RVWfBv4f8KiIVAKrcP4Huk1V1+Oc1F7GeS8dnoggEp8F4T22fodzbL0oIgdwOkGF+pHQKd35TEOk1ZXzTGfOIVfh3MIsxWlTvM+T9wGcNu+L3bRKcY7D5FbS+iLOFehanB68wR/2v8XpkFWB8xkvDLHvP938S3E6AV3jLn8A59budpzv9J0QeW5y/y+uAC5tpWz3unkU4nyOtTht4ajqFpy+Bjfg9KJeARzr7ncnTvvwLuAfOBVya/4OTHbPmf9W1dXA7TgjFXbh3M3x3jU5AXhXRKpwjsNrVXVjG+k7PXBN5IjIFpzOGIWRLosxxnSEiLyO00mu1RnZjMOmVYwgEQngdAzYFOGiGGOM8YFVshHits8UA3e5tz6MMcb0Mna72BhjjPGJXckaY4wxPunRMGaxIisrS0ePHh3pYhhjTExZtmxZhaoGIl2OaGKVbAijR49m6dKlkS6GMcbEFBHZ3P5WfYvdLjbGGGN8YpWsMcYY4xOrZI0xxhifWCVrjDHG+MQqWWOMMcYnVskaY4wxPrFK1hhjjPFJVFWyIjJfRNaJyAYRuSnE+itE5CMRWSEiS9xYisEYhfe56z50Y9YG9zneXb5BRH4vItIyXWOMaendjbtZtX1/pIthYlzUVLIiEg/cjRNzczJwSbAS9XhYVY9W1SnAbcAd7vLLAVT1aJxYmbeLSPC9/cldP859zPf1jRhjYp6qctUjy7n1+TWRLoqJcVFTyQInAhtUdaMbGPpR4DzvBqpa6XkZjGAPTqX8qrtNGbAPmCYiQ4ABqvqOOpEQHgDO9/dtGGNi3drSA5QfqKOkvCrSRTExLpoq2WHAVs/rbe6yI4jIAhEpwbmSvcZd/CFwrogkiMgY4HhghLv/tvbSdNP9hogsFZGl5eXl3X4zxpjYtbjYOQeUHajjQG1DhEtjYlk0VbIdoqp3q2o+8D3gFnfxvTgV6FLgt8BbQFMn071HVaep6rRAwOa3NqYvW1xccej5xvLqCJbExLpoqmS341x9Bg13l7XmUdxbv6raqKrfVtUpqnoeMAhY7+4/vBNpGmP6uNqGJt79eA+njXd+bG+ssFvGpuuiqZJ9HxgnImNEJAm4GHjGu4GIjPO8PBsodpeniEiq+3wO0Kiqq1V1J1ApIie5vYq/BPynB96LMSZGvffxHuobm7nspFHExwklZXYla7ouakLdqWqjiFwFLALigXtVtUhEfgYsVdVngKtEZDbQAOwFvuzung0sEpFmnCvVL3qSvhK4H+gP/Nd9GGNMSIuLy0mKj2PG2CxGZqTYlazplqipZAFU9QXghRbLfuR5fm0r+20CJrSybilwVPhKaYzpzQrXV3DCmMH0T4onP5BqbbKmW6LpdrExxkTUrspa1u06wMxxTntsXiCNjRXVNDVrO3saE5pVssYY4wr2Kp45LguA/EAq9Y3N7NhXE8limRhmlawxxrgWF5eTlZbEpNwBgHMlC7DBJqUwXWSVrDHGAM3NypLiCmaMzSIuzpniPC8rFbCxsqbrrJI1xhhg9c5KdlfXM2v84cloMlKTGJSSyEa7kjVdZJWsMcYAhe5UijPGZh1aJiLkZaXaHMamy6ySNcYYYPH6CibmppM9oN8Ry/MDaXa72HSZVbLGmD7vYH0jSzfvOeJWcVBeIM0CBZgus0rWGNPnvbtxDw1NemjojldewDo/ma7zbcYnETkFGO3NQ1Uf8Cs/Y4zpqsLicpIT4jhhdMYn1uW7w3hKyqs4dsSgni6aiXG+VLIi8k8gH1jB4ZBzwaDpxhgTVRYXVzA9L5N+ifGfWDcyI4X4OLErWdMlfl3JTgMmq6rNRWaMiWo79tWwoayKi08YEXJ9UkIcoyxQgOkiv9pkVwG5PqVtjDFhs9gduhOcrziUvECqhbwzXeLXlWwWsFpE3gPqggtV9Vyf8jPGmC4pLK4gOz2Z8TlprW6TH0ijsLiCpmYl3p0NypiO8KuS/YlP6RpjTNg0NStvbqjgrIk5iLReeea5gQK2761hZGZKD5bQxDpfbher6hvAWiDdfaxxlxljTNRYtX0/+w42MGv8J4fueAUDBZRYu6zppLBVsiIy0vP8QuA94ALgQuBdEfl8uPIyxphwWBxiKsVQgsN4rIex6axw3i6eLiIXqOrtwA+AE1S1DEBEAsDLwBNhzM8YY7qlcH0FRw0bQGZacpvbBQMF2BzGprPCdiWrqv8CSoPpBitY1+5w5mWMMd11oLaBD7bsbbNXsZczh7FVsqZzwtrxSVUfcp8uFJFFwCPu64uAF8KZlzHGdMc7G/fQ2Bx6KsVQ8rJSeX19uc+lMr2NXx2fvgvcAxzjPu5R1e/5kZcxxnTF4uJy+ifGc/yowR3aPi+QRvmBOiotUIDpBN/mLlbVJ4En/UrfGGO6Y3FxBSflZZCc8MmpFEPJ9wQKmGJzGJsOCuuVrIgscf8eEJFKz+OAiFSGMy9jjOmqrXsO8nFFdcjQdq3JO9TD2NplTceFu012hvs3PZzpGmNMOBV2YCrFlkZlppBggQJMJ/nSJisiJ4lIuud1uohM9yMvY4zprMXrKxg6sN+hW8AdkRgfx8iMFBvGYzrFr2E1fwK8R2K1u8wYYyKqsamZN0sqmDku0OZUiqHkBdLsStZ0il+VrHjD3KlqMz52sjLGmI76cNt+DtQ2MrOdqRRDyQ+k8vHuapqaLYqn6Ri/KtmNInKNiCS6j2uBjT7lZYwxHba4uBwRODW/85WsN1CAMR3hVyV7BXAKsB3YBkwHvuFTXsYY02GF68s5ZvggBqcmdXrf4BzG1i5rOsqvySjKVPViVc1W1RxV/UKLaRaNMabH7a9pYMXWfczq4CxPLeVZJWs6yZd2UhHpB/wvUAD0Cy5X1a/5kZ8xxnTE2yUVNGvnhu54ZaQmMTglkY0V1vnJdIxft4v/CeQC84A3gOHAAZ/yMsaYDiksriA1KZ7jRnZ9xqa8QBolZXYlazrGr0p2rKr+EKhW1X8AZ+O0yxpjTESoKoXryzk5P4vE+K6f+vKyUu1K1nSYX5VscAbtfSJyFDAQyG5vJxGZLyLrRGSDiNwUYv0VIvKRiKwQkSUiMtldnigi/3DXrRGRmz37bPLsszRM788YE2M27z7Itr01zOrC0B2v/GwLFGA6zq9K9h4RGQz8EHgGWA3c1tYOIhIP3A18CpgMXBKsRD0eVtWjVXWKm94d7vILgGRVPRo4HvimiIz27HeGqk5R1Wnde1vGmFgVnEpxVhfbY4Pysg4HCjCmPb50fFLVv7lP3wDyOrjbicAGVd0IICKPAufhVNDBdL1BBlKB4IhwBVJFJAHoD9QDFpDAGHNI4foKRmT0Z1RmSrfSyc8+HCjAovGY9oS1khWR69tar6p3tLF6GLDV8zo4vrZlHguA64Ek4Ex38RM4FfJOIAX4tqruCWYLvCgiCvxFVe9ppezfwB3LO3LkyLbehjEmxjQ0NfN2SQXnHTes01MptjQywwkUYMN4TEeE+3ZxuvuYBnwLp+IchjM5xdRwZKCqd6tqPvA94BZ38YlAEzAUGAPcICLBK+gZqjoV5zb0AhGZ1Uq696jqNFWdFgh073aSMSa6LN+yj+r6pi6Pj/UKBgqw28WmI8Id6u6nACJSCExV1QPu658Az7ez+3ZghOf1cHdZax7lcNCBLwALVbUBKBORN3Eq+o2qut0tW5mIPI1TIRd25n0ZY2Lb4uJy4uOEk7swlWIoeYE0u5I1HeJXx6ccnHbRoHp3WVveB8aJyBgRSQIuxuk0dYiIjPO8PBsodp9vwb11LCKpwEnAWhFJDYbcc5fPBVZ16R0ZY2JWYXEFU0YMYmD/xLCklx9IZVPFQQsUYNrlV2ScB4D33CtHgPOB+9vaQVUbReQqYBEQD9yrqkUi8jNgqao+A1wlIrNxhgjtBb7s7n43cJ+IFAEC3KeqK91bxk+7bTAJOL2TF4bzjRpjotve6npWbtvHtWeNa3/jDsoPpFHf1My2vQcZldnxmLSm7/Grd/GtIvJfYKa76KuqurwD+70AvNBi2Y88z69tZb8qnGE8LZdvBI7tRNGNMb3MmyUVaDemUgwlL3B4GI9VsqYt4e5dPEBVK0UkA9jkPoLrMjw9fo0xpkcsXl9Ber8Ejh0+MGxpeqPxnDGx3Xl2TB8W7ivZh4FzgGUcHsMKzi1cpeNjZo0xpttUlcXF5Zyan0VCN6ZSbGmwGyigxHoYm3aEu3fxOe7fMeFM1xhjuqKkvJod+2tZcGZ4ehV75QXS2Gg9jE07wn27uM2xsKr6QTjzM8aYtiwO01SKoeQHUnl1bXnY0zW9S7hvF9/exjrl8AxNxhjju8L15YzJSmVERvemUgwlL5DG40u3UVnbwIB+4RkaZHqfcN8uPiOc6RljTFfVNTbxzsY9XDBtuC/pBzs/bSyvtjmMTav8GieLG+JuMtAvuExVH/ArP2OM8Vq2eS81DU1hHbrjFRzGU1JmgQJM63ypZEXkx8DpOJXsCzjzBi/BmaTCGGN8t7i4goQ44aS8DF/SDwYK2FhhnZ9M6/yaVvHzwFlAqap+FWdCiPANUjPGmHYsLi5n6sjBpPvUXpoYH8fIzBRKymwYj2mdX5Vsjao2A40iMgAo48jJ/40xxje7q+pYtb2SWePDP3THKy8rza5kTZv8qmSXisgg4K84E1N8ALztU17GGHOEJRsqgPBOpRhKfrYFCjBt82vu4ivdp38WkYXAAFVd6UdexhjTUuH6CgalJHLUMH9bqfKzLFCAaZsvV7Ii8oyIfEFEUlV1k1WwxpiecmgqxbFZxMeJr3l5AwUYE4pft4tvB2YAq0XkCRH5vIj0a28nY4zprvW7qig7UMescf62x8KRgQKMCcWv28VvAG+ISDzOLE+XA/cCA/zIzxhjgoJTKfrdHgsWKMC0z8/JKPoDnwEuAqYC//ArL2OMCSosrmBsdhpDB/XvkfzyA2l2JWta5Veb7OPAGpyr2D8A+ap6tR95GWNMUG1DE+9u3M3MHrhVHJQXSLU2WdMqv9pk/45TsV6hqq+5Y2aNMcZX72/aQ11jsy9Rd1qTH0ijoqqO/TUNPZaniR2+VLKqukhVm/xI2xhjWrO4uILEeGG6T1MphpJ3KFCA3TI2n+TXlawxxvS4wvXlTBuVQUqSb91NPsGG8Zi2WCVrjOkVyiprWVt6gJk+T6XYUjBQgHV+MqH41fFJROQyEfmR+3qkiJzoR17GGAOHp1LsyfZYOBwowK5kTSh+Xcn+ETgZuMR9fQC426e8jDGGwvXlZKYmMXlIzw/Ht2E8pjV+VbLTVXUBUAugqnuBJJ/yMsb0cc3NypINFcwYl0Wcz1MphpIXSGXzbgsUYD7Jr0q2wZ3tSQFEJADYMB5jjC/WlFZSUVXfI7M8heINFGCMl1+V7O+Bp4FsEbkVWAL80qe8jDF93OLiYGi7nu30FJSf7fQwtlvGpiW/5i5+SESWAWcBApyvqmv8yMsYYxYXlzMhJ52cAZGJQ5KXFRwrW82ZEyNSBBOlwlrJioh3BHgZ8Ih3naruCWd+xhhTU9/E+x/v5cunjIpYGQanJpGRmmRXsuYTwn0luwynHVaAkcBe9/kgYAswJsz5GWP6uHc+3k19U3PE2mOD8rJSLRqP+YSwtsmq6hhVzQNeBj6jqlmqmgmcA7wYzryMMQZg8foKkhLiOHFMz02lGEp+IM2mVjSf4FfHp5NU9YXgC1X9L3CKT3kZY/qwxcXlTB+TQb/E+IiWIy+QSkVVvQUKMEfwq5LdISK3iMho9/EDYIdPeRlj+qid+2soLquKWK9iLwsUYELxq5K9BAjgDON5yn1+SZt7GGNMJx0euhPZ9liA/EBwGI+1y5rD/Ap1t0dVr1XV41R1qqpe19GexSIyX0TWicgGEbkpxPorROQjEVkhIktEZLK7PFFE/uGuWyMiN3c0TWNMbCpcX04gPZmJuemRLgoj3EABdiVrvKIqCo87S9TdwKeAycAlwUrU42FVPVpVpwC3AXe4yy8AklX1aOB44JvureqOpGmMiTFN7lSKM8dlIdLzUym2lBgfx6jMFBvGY44QVZUscCKwQVU3qmo98ChwnncDVa30vEzFnbrR/ZsqIglAf6AeqOxImsaY2FO0Yz/7Djb0eNSdtuQF0iwajzlCtFWyw4Ctntfb3GVHEJEFIlKCcyV7jbv4CaAa2IkzJvc37i3qjqb5DRFZKiJLy8vLw/FejDE+CrbHnjo28p2egoKBAhqbbKp24/ArnuxwEXlaRMpFpExEnhSR4eFKX1XvVtV84HvALe7iE4EmYCjOpBc3iEheJ9K8R1Wnqeq0QCB6fhmbyGtqVjbvruaVNbu4p7CEG5/4kNsWro10sfq8wvXlTB4ygEB6cqSLckh+IBgooCbSRTFRwpe5i4H7gIdx2kkBLnOXzWlnv+3ACM/r4e6y1jwK/Ml9/gVgoao2AGUi8iYwDecqtjNpmj6qtqGJjeXVbCivoqSs6tDfjRXV1DcevjLpnxhPTUMTF58wkpGZKREscd9VVdfIB1v28rUZ0TWJXLCH8caKKkZnpUa4NCYa+FXJBlT1Ps/r+0Xkug7s9z4wTkTG4FSEF+NUnoeIyDhVLXZfng0En28BzgT+KSKpwEnAb4HV7aVp+pZ9B+vZUFZFSXkVG8qq3OfVbN17EHVb+EVgxOAU8gOpzOQo84EAACAASURBVByXxdjsNPIDaYzNTqOyppFZ//caL64u5eszO3yzxITROyW7aWhSToui9lg4HCigpMwCBRiHX5XsbhG5jMMBAi4Bdre3k6o2ishVwCIgHrhXVYtE5GfAUlV9BrhKRGYDDThzI3/Z3f1u4D4RKcKZL/k+VV0JECrNcL1RE51UlR37a50rUs9VaUl5FRVV9Ye2S0qIIy8rlaOHD+R/jht2qDLNC6S2OoPQoJQkJuam82LRLqtkI2RxcTn9EuM4fvTgSBflCMFAARsrrIexcfhVyX4NuAu4E6fX71vAVzqyozsd4wstlv3I8/zaVvar4vDt6XbTNL1DfWMzW/ZUH3FFGrxKPVjfdGi7Af0SGJudxpkTs4+4Kh0+OIX4uM4P/5hXkMvvXy2moqqOrLToaRPsKxYXV3BSXibJCZGdSjGUvKxUSsqsh7Fx+FXJDlfVc70LRORUjuzl2+s0NjXz7ModnD9lWFSM2+vNquoaufwfS3l/0x4am/XQ8iED+zE2O40Lp40gPzuNsW5lmpWWFNbvZG5BDr97pZhX1uziohNGhi1d076tew6ysaKaS0+KXGi7tuQH0nhl7a5IF8NECb8q2buAqR1Y1qs8t3In337sQ/YfbOArp0ZXh4zeRFX53pMreffj3Xx9Zh4Tc9MZm51GXiCNtGS/DukjTR4ygGGD+rOoyCrZnrZkgzN0Z1YUzFccSl4glceW1rP/YAMDUxIjXRwTYeEO2n4yTrSdgIhc71k1AKc9tFc7b8pQnlu5g1tfWMPUUYM5ZvigSBepV7r/rU08v3InN86fwJWnj41IGUSEeQW5PPjuZqrqGnuscjdOe2zuAOeORTTKdwMFlFRUMXVkdLUZm54X7nGySUAaTuWd7nlUAp8Pc15RR0T4zQXHEkhLZsHDH1jIKx8s27yXW59fw+xJ2VwxKz+iZZlbkEN9YzOF623ykp7S2NTMkuIKZo2PjqkUQ8kLDuOxmZ8M4Q/a/oaq/hQnnuxPPY87PMNuerVBKUnc9YWp7NxXy01PrkRV29/JdMjuqjquevgDcgf24/YLphDXhQ5L4TRt1GAyUpNYVFQa0XL0JSu376eytjEqou60ZkRGConxYnMYG8C/aRWTReQeEXlRRF4NPnzKK+ocP2owN86fwH9XlfLA25sjXZxeoalZue6xFeyuqudPlx4fFW1dCfFxnDUxm1fXlh0xWYXxz+L1FYhE11SKLSXGxzEyI8Wi8RjAv0r2X8BynCkPv+t59Blfn5HHWROzufX5Nazcti/SxYl5d71azOLiCn5ybgFHDx8Y6eIcMrcglwO1jbyzsd1h4CYMFheXc/SwgWSkJkW6KG3KC6RZXFkD+FfJNqrqn1T1PVVdFnz4lFdUiotz2mez0pK46uHlVNZa+2xXvbG+nN+9UsxnjxvGJSeOaH+HHjRzXBb9E+N5cbXdMvZbZW0Dy7fuY2aU9ir2yg+ksXl3tQUKML5Vss+KyJUiMkREMoIPn/KKWoNTk7jrC8exfV+Ntc920Y59NVz36HLGZ6fzi/85Kuo6u/RLjOe08QFeWr2L5mb7fv30dslumpo1qttjg/ICqTQ0qQUKML5Vsl/GuT38FrDMfSz1Ka+odvyoDG6cN4EXPirln+9Y+2xn1Dc2s+DhD2hoUv542VRSkqJzmMy8o3LYVVnHh9Ys4KvC9eWkJsXHxLCYQ8N4Yqxdtq6xqf2NTKf4Usmq6pgQjz47yevlM/M4c2I2v3huDR9t2x/p4sSMX76whuVb9vH/PnfMoZNWNDpzQg7xccKLq22WHz8tLq7g5PxMkhKiLQz2J+XH4DAeVWXunYX8+r8WxjGc/Ion+6VQDz/yigVxccLtFxxLZloSCx7+wNpnO+C5lTu4/61NfPXU0Zx9zJBIF6dNA1MSOSkvgxdtKI9vNu+uZsuegzFxqxicoXwZqUkxdSVbtKOSzbsPMibLwjeGk18/CU/wPGYCPwHObWuH3m5wahJ3XeK0z9785EfWPtuGDWVVfO+JlUwdOYibPzUp0sXpkHkFuYeCE5jwKyx2plKMhU5PQfmB1Ji6kl1UVEqcwOxJOZEuSq/i1+3iqz2Py3HmLI7e+309ZNroDL47bwLPf7STB619NqSD9Y1c+dAykhPj+cMXpsbErUGAOZOdE5P1MvbH4vXlDBvUnzExFAg9LystpkLeLVxVyoljMsi0qFJh1VNnsGrAZswHvjEzjzMmBPj5c2tYtd3aZ71UlR88vYrisip+d/EUhg7qH+kiddiQgf05dvhAFhVZu2y4NTQ183bJ7qieSjGU/OxUKqqcQAHRrqS8iuKyKuYX5Ea6KL2OX22yz4rIM+7jOWAd8LQfecWauDjh9gunkJHqtM8esPbZQx5+bwtPL9/OdWeNj5m2N6+5Bbl8uHUfpftrI12UXmXF1n0cqGtkVowdE3lZhwMFRLvg1KBzrZINO7+uZH8D3O4+fgXMUtWbfMor5mS442e37a3hpqesfRbgo237+ekzq5k1PsDVZ0Ymsk53zXVvGb+0xq5mw2nx+nLiBE7Jj532WDgcKKAkBtrpF60q5dgRg2Lq7lGs8KtN9g3P401V3eZHPrHshNEZfGfuBJ5fuZMH390S6eJE1P6DDXzroWVkpSXx24siP/F/V43NTiMvK9V6GYdZYXEFx44YFBXzVXdGMFDAxoro7vy0fV8NH27bb7eKfRLWSlZElrh/D4hIZYjHxyJyZTjzjGXfnJXH6RMC/PzZ1X22fba5Wbn+8RXsqqzlD5dOjfo5adsiIswpyOHtkt0W5jBMyg/UsXLbvphsPggGCoj2K9ngj8J5Bdar2A/hDnU3w/2brqoDWj6AacC14cwzlsXFCXf08fbZPxeW8MraMn7w6UkxMZNPe+YV5NLYrLy2tizSRYl5r60r49w/LEFEYrYCyA+kRf2V7MJVpUzISScviid8iWW+9S4WkXgRGSoiI4MPVd0NnO5XnrHI2z57cx9rn32rpILfLFrHOccM4cunjI50ccJiyvBBBNKTbShPN+yvaeC7//qQr973PmnJCTz5rVMoGBo9kZc6Iy/KAwVUVNXx/qY9zDvKbhX7xZfJYEXkauDHwC4geHQpcIyq7vQjz1h2wugMbpg7ntsWruOkvEwuO2lUpIvku12VtVzzyHLGZKXy688dE1NDM9oSFyfMmZzDv5dvp7ahiX6J8ZEuUkx5de0ubn7qIyqq6rny9HyuOWtcTH+G+W6ggK17a6JyjO/Lq3fRrHar2E9+XcleC0xQ1QJVPdp9HONTXr3CFbPyOW18gJ891/vbZxubmrn64eVU1zXxp8uOJy05Oif+76p5BbkcrG/izQ0VkS5KzNh/sIEbHv+Qr92/lEH9k3j6ylO4cf7EmK5ggUO3YKM1gPvColJGZPRn8pABkS5Kr+VXJbsV6N01RZg57bPHMjglkat6efvs/y1ax3ub9vCrzx7N+Jz0SBcn7E7OyyQ9OYEXbWKKDnl59S7m3PkG/16xnavPHMszV5/KMcMHRbpYYREMFBCNcxhX1jbw5oYK5hfk9po7SdHIr0uIjcDrIvI8UBdcqKp3+JRfr5CZlsxdl0zl4nve5uanPuKuS47rdQf/oqJS/lK4kctOGsn5xw2LdHF8kZQQxxkTs3l5zS6ampX4GB2S5Ld9B+v52bOreWr5dibmpnPvV07gqGGx2fbamkEpSWSmJkXlHMavrS2joUmZb+2xvvLrSnYL8BKQBKR7HqYdJ47J4Ia5E3hu5U4efq93jZ/dvLua7/zrQ44ZPpAfnjM50sXx1dyCHHZX17Ns895IFyUqvVhUypw7C3nmwx1cc9Y4nrlqRq+rYIPyojRQwMJVpQTSkzluROz36o9mvlzJqupPAUQkzX0dffdKoti3Tsvn3Y/38NNnVzNlxKCY7VnpVdvQxLce/IA4Ee7+wlSSE2K7ra09p40PkBQfx4tFzqTrxrG3up6fPFvEf1bsYNKQAdzXC69eW8oPpPFSlMUarqlv4vV15Xzu+GExO/lLrPBr7uKjRGQ5UAQUicgyESnwI6/eKC5OuPNQ++xyquoaI12kbvvJM0Ws3lnJnRcdy4iM3h+vMr1fIqeOzWTR6tI+NSyrLQtXlTLnzjd4fuVOvj17PP9ZcGqvr2DBuZLdXV3PvoP1kS7KIYXF5dQ0NDG/ILpjNfcGft0uvge4XlVHqeoo4Abgrz7l1StlpiXz+4uPY/Puar4f4+Nn/7V0K4++v5UFZ+Rz5sS+M1RgbkEuW/fUsLb0QKSLElF7quu5+pHlXPHgMnIG9OOZq2Zw7exxMRPGsLsOBQqIolvGi4pKGdg/kel5dpfFb34d5amq+lrwhaq+DkTfILEoNz0vkxvmTuCZD3fwyHtbI12cLlm9o5Jb/r2KU/IzuX7OhEgXp0fNnpSDyOEIJ33RCx/tZM4db7Bw1U5umDOefy84lclD+9Zwkfzs6BrG09DUzMurdzF7Ug6J8X3jh04k+fUJbxSRH4rIaPdxC06PY9NJ3zotn5njsvjJs0Ws3lEZ6eJ0SmVtA1c+tIyB/RP53cXH9bletoH0ZI4fObhPDuXZXVXHgoc+4MqHPmDIoH48e/UMrj5rXJ88qY8Y3J/EeImaK9l3Nu6msrbRehX3EL+O+K8BAeAp4Ekgy11mOikuTrjzoikMTklkwcMfxEz7rKpy479WsnVvDXdfOpVAenKkixQRcwtyWL2zkq17Dka6KD3m+ZU7mXNnIS+t3sV3503g6StPZWJu37p69UqIj2NUZmrUXMkuXFVKSlI8M8fFVujAWOVXqLu9qnqNqk5V1eNV9TpVtbEMXZQVg+2zf1/yMQuLSrlp/kROGN13233mTnauFl6Mst6lfig/UMe3HlzGgoc/YPjg/jx79QwWnDG2T169tpSXlRoVgQKampVFRbs4Y0J2zM+mFSv86l38kogM8rweLCKLOrDffBFZJyIbROQTQd5F5AoR+UhEVojIEhGZ7C6/1F0WfDSLyBR33etumsF12eF8rz1lel4m188ZzzMf7uDR96O7fXbppj38+r9rmVeQw9dnjol0cSJqdFYqE3LSe3WMWVXlmQ93MPfON3hlTRk3zp/AU986hQm5NjQ+KD87OgIFLN+yl4qqOgsI0IP8+omZpar7gi/cq9g2KzcRiQfuBj4FTAYuCVaiHg+78yBPAW4D7nDTf0hVp7jLvwh8rKorPPtdGlyvqjEbg+zK08cyc1wWP34mettnK6rqWPDwBwwb3J//u+DYXjdjVVfMLcjh/U172FMdPUM4wqXsQC1XPLiMax5ZzsjMVJ6/ZgZXnj6WBLt6PUJe1uFAAZG0cFUpSfFxnDEh9uLzxiq//hOaRWRk8IWIjMKJwtOWE4ENqrpRVeuBR4HzvBuoqrdmSW0lzUvcfXudYPvsoP7O/MbR1j7b1Kxc++hy9h1s4E+XHs+AfomRLlJUmFeQS7PCy2t6zy1jVeU/K7Yz985CXltXzs2fmsiTV5zMuF44F3U4BAMFRDKAu6qysKiUU8dmkm7/mz3Gr0r2B8ASEfmniDwIFAI3t7PPMJzAAkHb3GVHEJEFIlKCcyV7TYh0LgIeabHsPvdW8Q+llUsrEfmGiCwVkaXl5eXtFDVystKS+f0lx7FpdzU/eDq62md/+/J63tywm5+ff1SfG6bRloKhAxg2qH+v6WVcVlnLN/65jGsfXcGYrFReuGYm3zwt365e2xAMFLCxInKVbNGOSrbtrbFexT3Mr45PC4GpwGM4V5XHq2q7bbIdTPtuVc0Hvgfc4l0nItOBg6q6yrP4UlU9GpjpPr7YSrr3qOo0VZ0WCET3rZST8jKdGXNW7OCxKGmffW1dGXe9uoELpw3nwmkjIl2cqCLixJhdXFzOwfrouvvQGarKUx9sY86dhRSuL+cHn57EE1ecwlh3HKhpXTBQQElZ5Do/LSoqJU6c8dum5/j201NVK1T1OffRkcCa2wHv2Xm4u6w1jwLnt1h2MS2uYlV1u/v3APAwzm3pmHflGYfbZ9fsjGz77La9B/n2YyuYNGQAPzvvqIiWJVrNLcihrrGZwvXRe5ekLWWVtXz9H0u5/vEPGZudxgvXzuTyWXl9buxzd+QH0iJ6JbtwlTOPdmZa3xxOFynRdH/nfWCciIwRkSScCvMZ7wYiMs7z8myg2LMuDrgQT3usiCSISJb7PBE4B/Be5caseLd9dmB/Z/xsdQ+2zzY1K5W1DezcX8OGsgMseOgDmpqUP1061YYFtOLE0RkMSkmMyVvGzc3K5Q8sZcmGCm45exKPf/Nk8gN29dpZkYzGs6GsiuKyKuYX2K3inuZXPNlOU9VGEbkKWATEA/eqapGI/AxYqqrPAFeJyGygAdgLfNmTxCxgq6p6Z5ZKBha5FWw88DK9aA7lrLRkfnfxcVz6t3e45d+ruOPC0L15m5uVgw1NVNc1UlXXyMG6JqrqGqmua6S6vvHQ86o6Z5vgds5zd9v6w69rGpo+kcdfvng8o7Ns5szWJMTHcdbEHF5aXUpDU3NMjR19evl2Pty2nzsuPJbPTh0e6eLELG+ggEEpST2ad3Bqz7lWyfa4sFayItLmrAOquqed9S8AL7RY9iPP82vb2Pd14KQWy6qB49vKM9adnJ/JdbPHc8dL69lVWUuzKtV1TUdUlAcbmuho/6jUpHhSkhNIS04gNTme1KQEhgzsR2pyAqnJCaQlx7t/Ew4tG5OZytHDe380le6aW5DDkx9s472P93Dq2NiYbae6rpHbFq3l2BGDOH/KJ/ohmk4IXv2XlFdz/KierWRfLCrl2BGDGDqof4/ma8J/JbsMZ1hNqIYaBfLCnJ8BFpwxlp37a1ixdT9pyfFkpiUxMjOFtKQjK8YjK8f4Q8+Df1MS4y22pI9mjQvQLzGORUWlMVPJ/uWNEnZV1vHHS6fasdFNh4bxlFdx/KieC5S+fV8NH27bz/fmT+yxPM1hYa1kVbVvT+8TIfFxwq8+e0yki2Ha0T8pnlnjArxYtIufnlsQ9RN1bN9Xw18KN/KZY4dy/Ki+OzVmuAQDBfR0u2xwtrF5BdarOBL8mlZRROQyEfmh+3qkiPSKXr3GdMfcglxKK2v5aPv+SBelXbctXAvA9+b3rRCFfolUoICFq0qZkJN+6Era9Cy/el/8ETgZ+IL7+gDOlInG9GlnTcwmPk6iPsbsss17+c+KHXxjVh7DB6dEuji9Rl5WKiU9WMlWVNXx/qY9NldxBPlVyU5X1QVALRyau7hnW/qNiUKDU5M4cXRGVA/laW5Wfv7carLTk7nitPxIF6dXyc9OY8uegzT0UKCAl1fvolmxoTsR5Fcl2+BO+K8AIhIAIht+wpgoMa8gh+KyqqiJL9rSsyt3sGLrPr47bwKpyVEzyq9XOBQooIfiCy8sKmVkRgqThtic0pHiVyX7e+BpIFtEbgWWAL/0KS9jYsqcguiNMVtT38Sv/7uWo4YN4HM2Jjbs8t0pKHui81NlbQNvbqhg/lG5Ud/Jrjfza+7ih4AbgV8BO4HzVfVffuRlTKwZNqg/Rw0bEJUxZu8p3MjO/bX86JwCG7Ljg/ysw8N4/Pba2jIampR5dqs4ony7F6Sqa4G1fqVvTCybNzmX219aT1llLdkD+kW6OACU7q/lz2+U8OmjczlxjA3Z8cPAlESy0pJ65Ep24apSstOTOW7EIN/zMq2LnbndjOlFgtPbvRRFMWZvW7SWpmbl5k9NinRRerW8LP8DBdTUN/H6unLmFuTYHYkIs0rWmAgYn5PG6MyUqOll/OHWfTz1wXb+d+YYRmTYkB0/5QVSKfH5SrawuJyahibmFwzxNR/TPqtkjYkAEWFuQS5vlVRQWdsQ0bKoOkN2stKSuPJ0G7Ljt/xAGnuq69lbXe9bHotWlTKwfyLT8+y2f6T5NePTZ0WkWET2i0iliBwQkcgGPTUmysydnENDk/L6usjGmH3+o50s3byX78ydQHq/xIiWpS/ICzjRqvy6ZdzQ1MzLa3Yxe1JOTEV76q38+gZuA85V1YGqOkBV01V1gE95GROTjhs5mKy05IjO/lTb0MSvXljLpCEDuGDaiIiVoy/xRuPxwzsbd1NZ28h8m+UpKvhVye5S1TU+pW1MrxAfJ8yZnM3ra8uoa/xkjN6e8PclH7N9Xw0/PGcS8dZBpkcM9zlQwMJVpaQkxTNzXGxEeurt/Kpkl4rIYyJyiXvr+LMi8lmf8jImZs0tyKW6vom3Snb3eN5llbXc/doG5k7O4ZR8OyH3lGCgAD/GyjY1K4uKdnHGhGz6JcaHPX3TeX6Nkx0AHATmepYp8JRP+RkTk07JzyQ1KZ4Xi0o5Y0J2j+b9mxfX0dDUzPc/bUN2elp+IJUNZeGvZJdv2UtFVZ0FBIgivlSyqvpVP9I1prdJTojn9InZvLR6F784X3vslu2q7fv517JtXD4zj9FZqT2SpzksL5DGK2vKaGhqDmvnpIWrSkmKj+OMCYGwpWm6x6/exeNF5BURWeW+PkZEbvEjL2Ni3byCXCqq6lm+ZW+P5Keq/Oy51QxOSeKqM8f2SJ7mSPmBNBqbwxsoQFVZWFTKjHFZ1ks8ivjVJvtX4GagAUBVVwIX+5SXMTHt9AkBEuOlxwIGLCoq5b2P93D9nPEMsJNxRASH8YSzh3HRjkq27a2xsHZRxq9KNkVV32uxrNGnvIyJaQP6JXJKfhaLikpRVV/zqmts4tYX1jAhJ52LT7AhO5ESDBQQznCHi4pKiROYPTknbGma7vOrkq0QkXwOx5P9PE40HmNMCHMLcti8+yDrd/k7p+19b25i654abjlnEgk2UUHE+BEoYOGqUqaPySQjNSlsaZru8+u/bAHwF2CiiGwHrgOu8CkvY2LenEk5iOBr+LvyA3X84dUNnDUxm5njrGNMpOVlpYVtGM+GsiqKy6qYV2BXsdHGr3iyG1V1NhAAJqrqDFXd7EdexvQG2QP6cdyIQSxa7V8le8dL66ltaOL7Z9uQnWiQn53KxorwXMkGZw2ba+2xUcfX+0WqWq2qB/zMw5jeYm5BLqu2V7J9X03Y0169o5LH3t/Cl04efWhaPxNZeVnhCxSwqKiUY0cMYuig/mEomQkna5QxJkrMdTusvBTmW8bBKDsD+idy7Vnjwpq26br87PAECti+r4aV2/Zbr+IoFfZKVkTiROSUcKdrTG+XF0hjXHYai8IcY/al1bt4e+Nuvj17PANTbMhOtMhzexiXlHXvlvGiVc6PMmuPjU5hr2RVtRm4O9zpGtMXzC3I4b1Ne8IWa7S+sZlfvrCGsdlpfGH6yLCkacIjGCigpJtXsouKSpmQk06eNQNEJb9uF78iIp8TEQvrYUwnzCvIpalZeXVtWVjSe+DtTWzafZBbzp5ksUWjTEJ8HKMzU7s1jKeiqo73N+2xuYqjmF//dd8E/gXUW9B2Yzru6GEDyR3QLywxZndX1fG7V4o5bXyA03s4+IDpmLxA96LxvLx6F82KtcdGMb+G8KSrapyqJlrQdmM6TkSYW5BDYXE5NfXdizF758vrOVjfxC02ZCdq5QfS2LL7IA1NzV3af2FRKSMzUpg0JD3MJTPh4tv9IxE5V0R+4z7O8SsfY3qbeQW51DY0U1hc3uU01pUe4OF3t3DZ9JGMy7ETcLTKcwMFbOlCoIDK2gbe3FDB/KNysZa56OVXFJ5fA9cCq93HtSLyKz/yMqa3OXFMBgP7J/JiF3sZqyq/eH41ackJXDd7fJhLZ8IpGCigK+2yr60to6FJmWe3iqOaX1eynwbmqOq9qnovMB84u72dRGS+iKwTkQ0iclOI9VeIyEciskJElojIZHf5pe6y4KNZRKa4645399kgIr+3zlgm2iXGx3HWxGxeWbuLxi7cRnxtXRmLiyu4dvZ4Bts8tlGtO4ECFq4qJTs9meNGDAp3sUwY+dnd0PvND2xvYxGJxxn68ylgMnBJsBL1eFhVj1bVKcBtwB0AqvqQqk5xl38R+FhVV7j7/Am4HBjnPuZ34z0Z0yPmFuSw72AD723a06n9Gpqa+cXza8jLSuWLJ43yqXQmXIKBAjrb+ammvonX15UzryCXuDi7bohmflWyvwSWi8j9IvIPYBlwazv7nAhscOc9rgceBc7zbqCq3h7KqbhRflq4xN0XERkCDFDVd9SJIfYAcH5X3pAxPWnW+ADJCXGdvmX84Dub2VhezQ/OnkRSgg3ZiQV5gbRO3y4uLC6npqGJ+TZ0J+r5MuMT0AycBDwFPAmcrKqPtbPrMGCr5/U2d1nL9BeISAnOlew1IdK5CHjEk+a29tJ00/2GiCwVkaXl5V3vcGJMOKQkJTBzXICXVu/qcIzZvdX1/PblYmaOy+LMiTZkJ1bkd2EYz6JVpQxKSeTEMRk+lcqEi18zPt2oqjtV9Rn3EbbJWFX1blXNB74H3OJdJyLTgYOquqoL6d6jqtNUdVogYGHATOTNLchh+74ainZ0bIj5714p5kBtA7ecPdl6m8aQ/EAaew82sKeDs3zVNzbz8ppdnDUxxyYYiQF+fUMvi8h3RGSEiGQEH+3ssx0Y4Xk93F3Wmkf55K3fizl8FRtMc3gn0jQmasyelENcB2PMbig7wD/f2cwlJ45kQq4N2Yklh3sYd+xq9p2Nu6msbbRbxTHCr0r2IpzA7YU47bHLgKXt7PM+ME5ExohIEk6F+Yx3AxHxhhA5Gyj2rIsDLsRtjwVQ1Z1ApYic5PYq/hLwn66+KWN6UkZqEieMzuhQwIBbn19DSmI818+xITuxJu9QD+OOtcsuLColJSmemeOy/CyWCRO/2mRvUtUxLR55be2nqo3AVcAiYA3wuKoWicjPRORcd7OrRKRIRFYA1wNf9iQxC9iqqhtbJH0l8DdgA1AC/Lfbb9KYHjK3IJd1uw6wqY3g3q+vK+O1deVcfdZYMtOSe7B0JhyGD+5PUnxchwIFNDUrLxbt4owJ2fRLjO+B0pnuSgh3gqraLCLfBdrrSuplPwAADEZJREFU6BRq3xeAF1os+5Hn+bVt7Ps6TmerlsuXAkd1tizGRIO5k3P4+XOreXF1Kd+Ylf+J9Y1Nzdz6/BpGZabw5VNG93wBTbclxMcxKjOlQyHvlm/ZS0VVnQUEiCHR1CZrjGlhREYKk4cMaHUozyPvbaG4rIrvf3oSyQl2ZROr8gNpHQrevnBVKUnxcZwxwTpnxopoapM1xoQwryCXZVv2Un6g7ojl+w82cMdL6zk5L5O5ky1gdyzLC6S2GyhAVVlYVMqMcVmk90vswdKZ7vArCk/L9th222SNMaHNLchBFV5ec+TV7O9fLWZfTQO3nDPJhuzEuI4ECijaUcm2vTUW1i7GhLWSFZEbPc8vaLHul+HMy5i+YmJuOiMy+h8xlGdjeRX/eGsTF00bQcHQdmctNVEuvwOBAhYVlRInMNvuWsSUcF/JXux5fnOLdTZnsDFdICLMm5zLmxt2U1XXCMAvX1hLv8R4bpg7IcKlM+GQF3CG8bQ189PCVaVMH5NJhgV9iCnhrmSlleehXhtjOmhuQS71Tc28vq6MJcUVvLxmFwvOGEsg3Ybs9AYD+yeSlZbc6oQUG8qqKC6rsgkoYlC4h/BoK89DvTbGdNDxowaTmZrEf1eVUlJWxfDB/fnqqaMjXSwTRnmBVEpauV28yG0qmFtgt4pjTbgr2WNFpBLnqrW/+xz3db8w52VMnxEfJ8yelMNjS50YGn+8dKpNRtDL5AdSWbgq9BSai4pKmTJiEEMG9u/hUpnuCuvtYlWNV9UBqpquqgnu8+Br63NuTDcEr2JOHJ3Bp+y2Ya/TWqCA7ftqWLltv90qjlFhn/HJGOOPmeMCXDp9JF89dbQN2emFvIECMlIPz92zyL26nWdDd2KSxUkyJkYkJcRx6/8czdhsi7LTG+UHQgcKWFhUyoScdMZkpUaiWKabrJI1xpgoMHxwihMowNPDuKKqjvc37bG5imOYVbLGGBMF4uOE0VkpR/Qwfnn1LlSxWZ5imFWyxhgTJfKy0o4YK7uwqJSRGSlMGmJNBLHKKlljjIkSeYFUtuxxAgVU1jbw5oYK5h+Vax3dYpj1LjbGmCiR7wYK2Lz7IEU79tPQpNarOMbZlawxxkQJ7zCehatKyU5P5rgRgyJcKtMdVskaY0yUCAYKKNpRyevryplXkEtcnN0qjmVWyRpjTJQIBgp4+L0t1DQ02SxPvYBVssYYE0XyAqmUH6hjUEoiJ47JaH8HE9WskjXGmCgSnPlp9qQcEuPtFB3r7Bs0xpgoku92frIJKHoHG8JjjDFR5JxjhlJ+oI5Z4wORLooJA6tkjTEmiuQO7MfNn54U6WKYMLHbxcYYY4xPrJI1xhhjfGKVrDHGGOMTq2SNMcYYn1gla4wxxvjEKlljjDHGJ1bJGmOMMT6xStYYY4zxiahqpMsQdUSkHNgc6XJ0QRZQEelC9DB7z31DX3vPsfp+R6mqTVXlYZVsLyIiS1V1WqTL0ZPsPfcNfe0997X325vZ7WJjjDHGJ1bJGmOMMT6xSrZ3uSfSBYgAe899Q197z33t/fZa1iZrjDHG+MSuZI0xxhifWCVrjDHG+MQq2RgnIiNE5DURWS0iRSJybaTL1FNEJF5ElovIc5EuS08QkUEi8oSIrBWRNSJycqTL5DcR+bZ7XK8SkUdEpF+kyxRuInKviJSJyCrPsgwReUlEit2/gyNZRtN1VsnGvkbgBlWdDJwELBCRyREuU0+5FlgT6UL0oN8BC1V1InAsvfy9i8gw4BpgmqoeBcQDF0e2VL64//+3d+8hWtV5HMffH9PyblqtaG1p0VpBt02jVXEtCyq7N7TQzS4QQdluMXRhox0hYopapKLaMhyzG9HFTBdSdEvLLlM2jmVJpGLWZEXaRdbK9bt//L6Pc3h8JBs9z+l5+L7g8PzO/XsOM/M9v9955vcDTi1bdjOwwMwOBRb4fKhBkWRrnJl1mNlSL39P+sO7f7FR5U/SAcBEYFrRsVSDpAHAOOBRADP7ycw2FhtVVXQHeknqDvQGPi84nt3OzBYB35QtPhuY4eUZwDlVDSrsNpFk64ikYcCxwFvFRlIVU4Ebga1FB1Ilw4GvgOneRD5NUp+ig8qTmX0G3A2sBTqAb81sXrFRVc1gM+vw8hfA4CKDCV0XSbZOSOoLPAf8zcy+KzqePEk6A/jSzN4tOpYq6g78EXjQzI4FNlHnTYj+HvJs0gPGUKCPpIuLjar6LP2fZfyvZY2KJFsHJPUgJdgnzOz5ouOpgjHAWZLWAE8DJ0l6vNiQcrcOWGdmpVaKZ0lJt56dDKw2s6/M7GfgeWB0wTFVy3pJQwD888uC4wldFEm2xkkS6T3dh2b2z6LjqQYzu8XMDjCzYaQvwiw0s7qu4ZjZF8Cnkkb4ognAigJDqoa1wAmSevvP+QTq/MteGbOBSV6eBLxYYCxhF0SSrX1jgEtItbk2n04vOqiQi8nAE5LagWOAOwqOJ1dea38WWAosJ/29qrvuBiU9BbwBjJC0TtKVQDNwiqSPSTX65iJjDF0X3SqGEEIIOYmabAghhJCTSLIhhBBCTiLJhhBCCDmJJBtCCCHkJJJsCCGEkJNIsqEmSTJJ92TmGyU1FRjSTpG0RtK+RcdRbZKaJDV6+TJJQ4uOKYRqiCQbatWPwHlFJSzvsD50zWWkbhJDqHuRZEOt2kLqmOD68hWSWiQ1ZOZ/8M/xkl6V9KKkVZKaJV0k6W1JyyUd4tvtJ+k5Sa0+jfHlTZJmSnodmClpmKSFktolLZB0YIVY9pE0z8dEnQYos+5iP3ebpH9J2qPC/qMkLZG0zLft5+ddLGmpT6Mz17dI0lxJKyU9JKmbr3tQ0jsex5TM8Zt9LOJ2SXdXOP8gSbN8/ZuSjvLlfSVN9/vWLun87L32coOklrLjNQAjSZ1qtEnqJek2v8/vS3rYe3dC0nWZ2J7e7icghFpgZjHFVHMT8APQH1gDDAAagSZf1wI0ZLf1z/HARmAIsBfwGTDF1/0VmOrlJ4GxXj6Q1GUlQBPwLtDL518CJnn5CmBWhTjvBW7z8kRSR+/7Aof7/j183QPApWX77gmsAkb5fH/SQAG9gZ6+7FDgncz1bQYOJo29Or90H4BB/rkH8ApwFLAPsJLOTmn2rhD/fcA/vHwS0OblO0v3y+cHZu+1lxuAlsy9a/TyK6QxYsnG5uWZwJle/hzYa0exxRRTLUzR5BVqlpl9J+kx0sDe/93J3VrNhxCT9AlQGjptOXCil08GjvAKFUB/H+UIYLaZlc71J+A8L88E7qpwvnGlbcxsrqQNvnwCcBzQ6ufpxfadwI8AOsystXS9Hncf4H5JxwD/A/6Q2edtM1vl2z0FjCV1TXiBpKtISXoIcASp7+PNwKOS5gBzKsQ/Fjjfz7/Qa+b9/R5tG0DdzDZU2HdnnSjpRtLDwyDgA9IDSDupxjsLmLULxw+hMJFkQ62bSurbdnpm2Rb8VYg3l+6ZWfdjprw1M7+Vzt+HbsAJZrY5eyJPhpt2U9wCZpjZLV3Y93pgPXA0KdZsnOX9pJqk4aSa/igz2+BNuD3NbIuk40kJvwG4llRb3RXZ8/f8pY0l9STV4kea2af+5bXSfhNJDylnAn+XdKSZbdnF+EKoqngnG2qamX0DPANcmVm8hlRLBDgL6PErDzuP1Bk/AF5jrGQJnbW5i4DFFbZZBFzoxzkNGOjLFwANkn7n6wZJOqhs35XAEEmjfJt+/oWrAaQa7lbS4BDZd7nHSxruDxd/AV4jNTNvAr6VNBg4zY/XFxhgZv8mJe6jK8S/2K8NSeOBr71GPR+4JnOPSte1XtLhfv5zKxwP4Hugn5dLCfVrj6fBj9cN+L2Z/Qe4ya+5b/mBQvitiyQb6sE9pPecJY8Af5a0jNSk+2trn9cBI/0LNyuAq3ew3WTgcqVRcS4hvdctNwUYJ+kDUrPxWgAzWwHcCszz/eeTmnG3MbOfSInyPr+W+aSk9AAwyZcdVnZ9rcD9pCHhVgMvmNky4D3gI9L75td9237AHD//a8ANFeJvAo7zbZrpHH7tdmCgf1lpGZ1N7TeTmp2XAB2VbxstwEOS2kgtCY8A7wMve/yQHhwel7TcY7/XzDbu4Hgh/GbFKDwh1AmvaTaa2RlFxxJCSKImG0IIIeQkarIhhBBCTqImG0IIIeQkkmwIIYSQk0iyIYQQQk4iyYYQQgg5iSQbQggh5OT/QbwagKyua+QAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Al igual que cuando se realizó este experimento con las celdas LSTM, se elige no concatenar celdas GRU, porque no mejora signifiquitivamente el error. "
      ],
      "metadata": {
        "id": "OtVqIC16hIZF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##### Probando el número de capas ocultas"
      ],
      "metadata": {
        "id": "WHZMU-4gt7j6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Creamos nuestro modelo.\n",
        "validation_values = []\n",
        "HIDDEN_DIM = 2**6 #\n",
        "for i in range(1,12):\n",
        "  print(f'Número de capas ocultas: {i}')\n",
        "  N_LAYERS = i\n",
        "  model = NER_RNN(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM,\n",
        "                          N_LAYERS, BIDIRECTIONAL, DROPOUT, PAD_IDX)\n",
        "\n",
        "  baseline_model_name = 'baseline'  # nombre que tendrá el modelo guardado...\n",
        "\n",
        "  model_name = baseline_model_name\n",
        "  criterion = baseline_criterion\n",
        "  n_epochs = baseline_n_epochs\n",
        "\n",
        "  model.apply(init_weights)\n",
        "  print(f'El modelo actual tiene {count_parameters(model):,} parámetros entrenables.')\n",
        "  # Optimizador\n",
        "  optimizer = optim.Adam(model.parameters())\n",
        "\n",
        "  # Enviamos el modelo y la loss a cuda (en el caso en que esté disponible)\n",
        "  model = model.to(device)\n",
        "  criterion = criterion.to(device)\n",
        "\n",
        "\n",
        "  best_valid_loss = float('inf')\n",
        "\n",
        "  for epoch in range(n_epochs):\n",
        "\n",
        "      start_time = time.time()\n",
        "\n",
        "      # Recuerdo: train_iterator y valid_iterator contienen el dataset dividido en batches.\n",
        "\n",
        "      # Entrenar\n",
        "      train_loss, train_precision, train_recall, train_f1 = train(\n",
        "          model, train_iterator, optimizer, criterion)\n",
        "\n",
        "      # Evaluar (valid = validación)\n",
        "      valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "          model, valid_iterator, criterion)\n",
        "\n",
        "      end_time = time.time()\n",
        "\n",
        "      epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
        "\n",
        "      # Si obtuvimos mejores resultados, guardamos este modelo en el almacenamiento (para poder cargarlo luego)\n",
        "      # Si detienen el entrenamiento prematuramente, pueden cargar el modelo en el siguiente recuadro de código.\n",
        "      if valid_loss < best_valid_loss:\n",
        "          best_valid_loss = valid_loss\n",
        "          torch.save(model.state_dict(), '{}.pt'.format(model_name))\n",
        "      # Si ya no mejoramos el loss de validación, terminamos de entrenar.\n",
        "\n",
        "      print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
        "      print(\n",
        "          f'\\tTrain Loss: {train_loss:.3f} | Train f1: {train_f1:.2f} | Train precision: {train_precision:.2f} | Train recall: {train_recall:.2f}'\n",
        "      )\n",
        "      print(\n",
        "          f'\\t Val. Loss: {valid_loss:.3f} |  Val. f1: {valid_f1:.2f} |  Val. precision: {valid_precision:.2f} | Val. recall: {valid_recall:.2f}'\n",
        "      )\n",
        "  # cargar el mejor modelo entrenado.\n",
        "  model.load_state_dict(torch.load('{}.pt'.format(model_name)))\n",
        "\n",
        "  # Limpiar ram de cuda\n",
        "  torch.cuda.empty_cache()\n",
        "\n",
        "  valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "      model, valid_iterator, criterion)\n",
        "\n",
        "  validation_values += [valid_loss]\n",
        "\n",
        "  print(\n",
        "      f'Val. Loss: {valid_loss:.3f} |  Val. f1: {valid_f1:.2f} | Val. precision: {valid_precision:.2f} | Val. recall: {valid_recall:.2f}'\n",
        "  )"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OU68Hy_Pt-j0",
        "outputId": "7c87a731-42ec-40de-fcfa-a48b1ccb0068"
      },
      "execution_count": null,
      "outputs": [
        {
          "metadata": {
            "tags": null
          },
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Número de capas ocultas: 1\n",
            "El modelo actual tiene 5,466,240 parámetros entrenables.\n",
            "to device cuda!!!!\n"
          ]
        },
        {
          "metadata": {
            "tags": null
          },
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        },
        {
          "metadata": {
            "tags": null
          },
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch: 01 | Epoch Time: 0m 5s\n",
            "\tTrain Loss: 0.890 | Train f1: 0.35 | Train precision: 0.55 | Train recall: 0.29\n",
            "\t Val. Loss: 0.524 |  Val. f1: 0.60 |  Val. precision: 0.73 | Val. recall: 0.52\n",
            "Epoch: 02 | Epoch Time: 0m 5s\n",
            "\tTrain Loss: 0.472 | Train f1: 0.67 | Train precision: 0.74 | Train recall: 0.62\n",
            "\t Val. Loss: 0.401 |  Val. f1: 0.70 |  Val. precision: 0.77 | Val. recall: 0.65\n",
            "Epoch: 03 | Epoch Time: 0m 5s\n",
            "\tTrain Loss: 0.342 | Train f1: 0.76 | Train precision: 0.80 | Train recall: 0.73\n",
            "\t Val. Loss: 0.363 |  Val. f1: 0.74 |  Val. precision: 0.79 | Val. recall: 0.71\n",
            "Epoch: 04 | Epoch Time: 0m 5s\n",
            "\tTrain Loss: 0.274 | Train f1: 0.80 | Train precision: 0.83 | Train recall: 0.79\n",
            "\t Val. Loss: 0.358 |  Val. f1: 0.75 |  Val. precision: 0.79 | Val. recall: 0.73\n",
            "Epoch: 05 | Epoch Time: 0m 5s\n",
            "\tTrain Loss: 0.222 | Train f1: 0.84 | Train precision: 0.86 | Train recall: 0.83\n",
            "\t Val. Loss: 0.362 |  Val. f1: 0.76 |  Val. precision: 0.80 | Val. recall: 0.73\n",
            "Epoch: 06 | Epoch Time: 0m 5s\n",
            "\tTrain Loss: 0.187 | Train f1: 0.87 | Train precision: 0.87 | Train recall: 0.86\n",
            "\t Val. Loss: 0.373 |  Val. f1: 0.77 |  Val. precision: 0.81 | Val. recall: 0.73\n",
            "Epoch: 07 | Epoch Time: 0m 6s\n",
            "\tTrain Loss: 0.164 | Train f1: 0.89 | Train precision: 0.89 | Train recall: 0.88\n",
            "\t Val. Loss: 0.380 |  Val. f1: 0.77 |  Val. precision: 0.80 | Val. recall: 0.75\n",
            "Epoch: 08 | Epoch Time: 0m 5s\n",
            "\tTrain Loss: 0.145 | Train f1: 0.90 | Train precision: 0.90 | Train recall: 0.89\n",
            "\t Val. Loss: 0.402 |  Val. f1: 0.77 |  Val. precision: 0.82 | Val. recall: 0.74\n",
            "Epoch: 09 | Epoch Time: 0m 5s\n",
            "\tTrain Loss: 0.127 | Train f1: 0.91 | Train precision: 0.91 | Train recall: 0.91\n",
            "\t Val. Loss: 0.392 |  Val. f1: 0.77 |  Val. precision: 0.79 | Val. recall: 0.76\n",
            "Epoch: 10 | Epoch Time: 0m 5s\n",
            "\tTrain Loss: 0.111 | Train f1: 0.92 | Train precision: 0.92 | Train recall: 0.92\n",
            "\t Val. Loss: 0.413 |  Val. f1: 0.77 |  Val. precision: 0.80 | Val. recall: 0.75\n",
            "Val. Loss: 0.358 |  Val. f1: 0.75 | Val. precision: 0.79 | Val. recall: 0.73\n",
            "Número de capas ocultas: 2\n",
            "El modelo actual tiene 5,565,568 parámetros entrenables.\n",
            "to device cuda!!!!\n"
          ]
        },
        {
          "metadata": {
            "tags": null
          },
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
            "  warnings.warn('{} seems not to be NE tag.'.format(chunk))\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch: 01 | Epoch Time: 0m 7s\n",
            "\tTrain Loss: 0.900 | Train f1: 0.36 | Train precision: 0.54 | Train recall: 0.29\n",
            "\t Val. Loss: 0.543 |  Val. f1: 0.60 |  Val. precision: 0.74 | Val. recall: 0.51\n",
            "Epoch: 02 | Epoch Time: 0m 7s\n",
            "\tTrain Loss: 0.507 | Train f1: 0.65 | Train precision: 0.73 | Train recall: 0.59\n",
            "\t Val. Loss: 0.417 |  Val. f1: 0.69 |  Val. precision: 0.74 | Val. recall: 0.65\n",
            "Epoch: 03 | Epoch Time: 0m 7s\n",
            "\tTrain Loss: 0.368 | Train f1: 0.74 | Train precision: 0.78 | Train recall: 0.71\n",
            "\t Val. Loss: 0.380 |  Val. f1: 0.74 |  Val. precision: 0.78 | Val. recall: 0.70\n",
            "Epoch: 04 | Epoch Time: 0m 7s\n",
            "\tTrain Loss: 0.285 | Train f1: 0.80 | Train precision: 0.82 | Train recall: 0.78\n",
            "\t Val. Loss: 0.378 |  Val. f1: 0.75 |  Val. precision: 0.77 | Val. recall: 0.73\n",
            "Epoch: 05 | Epoch Time: 0m 7s\n",
            "\tTrain Loss: 0.241 | Train f1: 0.83 | Train precision: 0.84 | Train recall: 0.82\n",
            "\t Val. Loss: 0.374 |  Val. f1: 0.76 |  Val. precision: 0.76 | Val. recall: 0.75\n",
            "Epoch: 06 | Epoch Time: 0m 7s\n",
            "\tTrain Loss: 0.202 | Train f1: 0.86 | Train precision: 0.86 | Train recall: 0.85\n",
            "\t Val. Loss: 0.383 |  Val. f1: 0.77 |  Val. precision: 0.79 | Val. recall: 0.76\n",
            "Epoch: 07 | Epoch Time: 0m 7s\n",
            "\tTrain Loss: 0.173 | Train f1: 0.88 | Train precision: 0.88 | Train recall: 0.88\n",
            "\t Val. Loss: 0.397 |  Val. f1: 0.77 |  Val. precision: 0.80 | Val. recall: 0.75\n",
            "Epoch: 08 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.155 | Train f1: 0.89 | Train precision: 0.89 | Train recall: 0.89\n",
            "\t Val. Loss: 0.399 |  Val. f1: 0.77 |  Val. precision: 0.79 | Val. recall: 0.76\n",
            "Epoch: 09 | Epoch Time: 0m 7s\n",
            "\tTrain Loss: 0.131 | Train f1: 0.91 | Train precision: 0.91 | Train recall: 0.91\n",
            "\t Val. Loss: 0.422 |  Val. f1: 0.77 |  Val. precision: 0.77 | Val. recall: 0.78\n",
            "Epoch: 10 | Epoch Time: 0m 8s\n",
            "\tTrain Loss: 0.123 | Train f1: 0.91 | Train precision: 0.91 | Train recall: 0.91\n",
            "\t Val. Loss: 0.449 |  Val. f1: 0.78 |  Val. precision: 0.80 | Val. recall: 0.75\n",
            "Val. Loss: 0.374 |  Val. f1: 0.76 | Val. precision: 0.76 | Val. recall: 0.75\n",
            "Número de capas ocultas: 3\n",
            "El modelo actual tiene 5,664,896 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 9s\n",
            "\tTrain Loss: 0.939 | Train f1: 0.32 | Train precision: 0.47 | Train recall: 0.25\n",
            "\t Val. Loss: 0.604 |  Val. f1: 0.57 |  Val. precision: 0.70 | Val. recall: 0.49\n",
            "Epoch: 02 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 0.545 | Train f1: 0.63 | Train precision: 0.73 | Train recall: 0.57\n",
            "\t Val. Loss: 0.452 |  Val. f1: 0.66 |  Val. precision: 0.74 | Val. recall: 0.60\n",
            "Epoch: 03 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 0.396 | Train f1: 0.72 | Train precision: 0.77 | Train recall: 0.69\n",
            "\t Val. Loss: 0.401 |  Val. f1: 0.72 |  Val. precision: 0.78 | Val. recall: 0.67\n",
            "Epoch: 04 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 0.313 | Train f1: 0.78 | Train precision: 0.80 | Train recall: 0.77\n",
            "\t Val. Loss: 0.395 |  Val. f1: 0.73 |  Val. precision: 0.73 | Val. recall: 0.73\n",
            "Epoch: 05 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 0.261 | Train f1: 0.82 | Train precision: 0.83 | Train recall: 0.81\n",
            "\t Val. Loss: 0.364 |  Val. f1: 0.75 |  Val. precision: 0.77 | Val. recall: 0.75\n",
            "Epoch: 06 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 0.221 | Train f1: 0.84 | Train precision: 0.85 | Train recall: 0.84\n",
            "\t Val. Loss: 0.390 |  Val. f1: 0.76 |  Val. precision: 0.77 | Val. recall: 0.75\n",
            "Epoch: 07 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 0.195 | Train f1: 0.86 | Train precision: 0.86 | Train recall: 0.86\n",
            "\t Val. Loss: 0.385 |  Val. f1: 0.76 |  Val. precision: 0.76 | Val. recall: 0.76\n",
            "Epoch: 08 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 0.171 | Train f1: 0.88 | Train precision: 0.88 | Train recall: 0.88\n",
            "\t Val. Loss: 0.402 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.76\n",
            "Epoch: 09 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 0.150 | Train f1: 0.89 | Train precision: 0.89 | Train recall: 0.89\n",
            "\t Val. Loss: 0.431 |  Val. f1: 0.76 |  Val. precision: 0.78 | Val. recall: 0.75\n",
            "Epoch: 10 | Epoch Time: 0m 10s\n",
            "\tTrain Loss: 0.136 | Train f1: 0.90 | Train precision: 0.90 | Train recall: 0.90\n",
            "\t Val. Loss: 0.443 |  Val. f1: 0.77 |  Val. precision: 0.76 | Val. recall: 0.78\n",
            "Val. Loss: 0.364 |  Val. f1: 0.75 | Val. precision: 0.77 | Val. recall: 0.75\n",
            "Número de capas ocultas: 4\n",
            "El modelo actual tiene 5,764,224 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.993 | Train f1: 0.28 | Train precision: 0.43 | Train recall: 0.21\n",
            "\t Val. Loss: 0.662 |  Val. f1: 0.53 |  Val. precision: 0.74 | Val. recall: 0.42\n",
            "Epoch: 02 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.588 | Train f1: 0.62 | Train precision: 0.73 | Train recall: 0.54\n",
            "\t Val. Loss: 0.473 |  Val. f1: 0.65 |  Val. precision: 0.76 | Val. recall: 0.58\n",
            "Epoch: 03 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.427 | Train f1: 0.71 | Train precision: 0.77 | Train recall: 0.66\n",
            "\t Val. Loss: 0.423 |  Val. f1: 0.71 |  Val. precision: 0.75 | Val. recall: 0.68\n",
            "Epoch: 04 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.339 | Train f1: 0.77 | Train precision: 0.79 | Train recall: 0.74\n",
            "\t Val. Loss: 0.408 |  Val. f1: 0.73 |  Val. precision: 0.73 | Val. recall: 0.73\n",
            "Epoch: 05 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.280 | Train f1: 0.80 | Train precision: 0.82 | Train recall: 0.79\n",
            "\t Val. Loss: 0.388 |  Val. f1: 0.74 |  Val. precision: 0.74 | Val. recall: 0.74\n",
            "Epoch: 06 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.238 | Train f1: 0.83 | Train precision: 0.84 | Train recall: 0.83\n",
            "\t Val. Loss: 0.390 |  Val. f1: 0.75 |  Val. precision: 0.76 | Val. recall: 0.74\n",
            "Epoch: 07 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.211 | Train f1: 0.85 | Train precision: 0.85 | Train recall: 0.85\n",
            "\t Val. Loss: 0.397 |  Val. f1: 0.76 |  Val. precision: 0.77 | Val. recall: 0.75\n",
            "Epoch: 08 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.182 | Train f1: 0.87 | Train precision: 0.87 | Train recall: 0.87\n",
            "\t Val. Loss: 0.418 |  Val. f1: 0.76 |  Val. precision: 0.80 | Val. recall: 0.73\n",
            "Epoch: 09 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.166 | Train f1: 0.88 | Train precision: 0.88 | Train recall: 0.89\n",
            "\t Val. Loss: 0.413 |  Val. f1: 0.77 |  Val. precision: 0.77 | Val. recall: 0.77\n",
            "Epoch: 10 | Epoch Time: 0m 12s\n",
            "\tTrain Loss: 0.149 | Train f1: 0.90 | Train precision: 0.89 | Train recall: 0.90\n",
            "\t Val. Loss: 0.466 |  Val. f1: 0.77 |  Val. precision: 0.78 | Val. recall: 0.76\n",
            "Val. Loss: 0.388 |  Val. f1: 0.74 | Val. precision: 0.74 | Val. recall: 0.74\n",
            "Número de capas ocultas: 5\n",
            "El modelo actual tiene 5,863,552 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 14s\n",
            "\tTrain Loss: 1.033 | Train f1: 0.23 | Train precision: 0.39 | Train recall: 0.17\n",
            "\t Val. Loss: 0.747 |  Val. f1: 0.50 |  Val. precision: 0.65 | Val. recall: 0.42\n",
            "Epoch: 02 | Epoch Time: 0m 14s\n",
            "\tTrain Loss: 0.644 | Train f1: 0.59 | Train precision: 0.73 | Train recall: 0.50\n",
            "\t Val. Loss: 0.513 |  Val. f1: 0.65 |  Val. precision: 0.75 | Val. recall: 0.58\n",
            "Epoch: 03 | Epoch Time: 0m 14s\n",
            "\tTrain Loss: 0.476 | Train f1: 0.69 | Train precision: 0.77 | Train recall: 0.62\n",
            "\t Val. Loss: 0.448 |  Val. f1: 0.68 |  Val. precision: 0.74 | Val. recall: 0.63\n",
            "Epoch: 04 | Epoch Time: 0m 14s\n",
            "\tTrain Loss: 0.377 | Train f1: 0.75 | Train precision: 0.79 | Train recall: 0.71\n",
            "\t Val. Loss: 0.418 |  Val. f1: 0.72 |  Val. precision: 0.75 | Val. recall: 0.70\n",
            "Epoch: 05 | Epoch Time: 0m 14s\n",
            "\tTrain Loss: 0.313 | Train f1: 0.79 | Train precision: 0.81 | Train recall: 0.77\n",
            "\t Val. Loss: 0.403 |  Val. f1: 0.73 |  Val. precision: 0.72 | Val. recall: 0.73\n",
            "Epoch: 06 | Epoch Time: 0m 14s\n",
            "\tTrain Loss: 0.268 | Train f1: 0.82 | Train precision: 0.83 | Train recall: 0.81\n",
            "\t Val. Loss: 0.420 |  Val. f1: 0.75 |  Val. precision: 0.77 | Val. recall: 0.73\n",
            "Epoch: 07 | Epoch Time: 0m 15s\n",
            "\tTrain Loss: 0.231 | Train f1: 0.84 | Train precision: 0.85 | Train recall: 0.84\n",
            "\t Val. Loss: 0.416 |  Val. f1: 0.75 |  Val. precision: 0.78 | Val. recall: 0.73\n",
            "Epoch: 08 | Epoch Time: 0m 14s\n",
            "\tTrain Loss: 0.208 | Train f1: 0.86 | Train precision: 0.86 | Train recall: 0.85\n",
            "\t Val. Loss: 0.420 |  Val. f1: 0.76 |  Val. precision: 0.76 | Val. recall: 0.75\n",
            "Epoch: 09 | Epoch Time: 0m 14s\n",
            "\tTrain Loss: 0.189 | Train f1: 0.87 | Train precision: 0.87 | Train recall: 0.87\n",
            "\t Val. Loss: 0.417 |  Val. f1: 0.76 |  Val. precision: 0.77 | Val. recall: 0.75\n",
            "Epoch: 10 | Epoch Time: 0m 14s\n",
            "\tTrain Loss: 0.168 | Train f1: 0.88 | Train precision: 0.88 | Train recall: 0.88\n",
            "\t Val. Loss: 0.455 |  Val. f1: 0.76 |  Val. precision: 0.77 | Val. recall: 0.76\n",
            "Val. Loss: 0.403 |  Val. f1: 0.73 | Val. precision: 0.72 | Val. recall: 0.73\n",
            "Número de capas ocultas: 6\n",
            "El modelo actual tiene 5,962,880 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 1.119 | Train f1: 0.12 | Train precision: 0.26 | Train recall: 0.08\n",
            "\t Val. Loss: 0.890 |  Val. f1: 0.36 |  Val. precision: 0.54 | Val. recall: 0.28\n",
            "Epoch: 02 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.772 | Train f1: 0.49 | Train precision: 0.68 | Train recall: 0.40\n",
            "\t Val. Loss: 0.589 |  Val. f1: 0.61 |  Val. precision: 0.75 | Val. recall: 0.53\n",
            "Epoch: 03 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.552 | Train f1: 0.66 | Train precision: 0.78 | Train recall: 0.57\n",
            "\t Val. Loss: 0.516 |  Val. f1: 0.66 |  Val. precision: 0.71 | Val. recall: 0.62\n",
            "Epoch: 04 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.432 | Train f1: 0.72 | Train precision: 0.81 | Train recall: 0.65\n",
            "\t Val. Loss: 0.459 |  Val. f1: 0.70 |  Val. precision: 0.78 | Val. recall: 0.64\n",
            "Epoch: 05 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.355 | Train f1: 0.76 | Train precision: 0.80 | Train recall: 0.73\n",
            "\t Val. Loss: 0.421 |  Val. f1: 0.72 |  Val. precision: 0.76 | Val. recall: 0.70\n",
            "Epoch: 06 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.300 | Train f1: 0.80 | Train precision: 0.82 | Train recall: 0.79\n",
            "\t Val. Loss: 0.414 |  Val. f1: 0.74 |  Val. precision: 0.75 | Val. recall: 0.72\n",
            "Epoch: 07 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.265 | Train f1: 0.82 | Train precision: 0.83 | Train recall: 0.82\n",
            "\t Val. Loss: 0.414 |  Val. f1: 0.74 |  Val. precision: 0.74 | Val. recall: 0.74\n",
            "Epoch: 08 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.231 | Train f1: 0.84 | Train precision: 0.84 | Train recall: 0.84\n",
            "\t Val. Loss: 0.450 |  Val. f1: 0.74 |  Val. precision: 0.75 | Val. recall: 0.75\n",
            "Epoch: 09 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.209 | Train f1: 0.86 | Train precision: 0.86 | Train recall: 0.85\n",
            "\t Val. Loss: 0.419 |  Val. f1: 0.75 |  Val. precision: 0.75 | Val. recall: 0.75\n",
            "Epoch: 10 | Epoch Time: 0m 17s\n",
            "\tTrain Loss: 0.193 | Train f1: 0.87 | Train precision: 0.87 | Train recall: 0.87\n",
            "\t Val. Loss: 0.440 |  Val. f1: 0.75 |  Val. precision: 0.75 | Val. recall: 0.76\n",
            "Val. Loss: 0.414 |  Val. f1: 0.74 | Val. precision: 0.75 | Val. recall: 0.72\n",
            "Número de capas ocultas: 7\n",
            "El modelo actual tiene 6,062,208 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 1.089 | Train f1: 0.17 | Train precision: 0.33 | Train recall: 0.12\n",
            "\t Val. Loss: 0.877 |  Val. f1: 0.41 |  Val. precision: 0.55 | Val. recall: 0.33\n",
            "Epoch: 02 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.757 | Train f1: 0.52 | Train precision: 0.70 | Train recall: 0.43\n",
            "\t Val. Loss: 0.616 |  Val. f1: 0.59 |  Val. precision: 0.78 | Val. recall: 0.49\n",
            "Epoch: 03 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.569 | Train f1: 0.65 | Train precision: 0.78 | Train recall: 0.57\n",
            "\t Val. Loss: 0.515 |  Val. f1: 0.64 |  Val. precision: 0.74 | Val. recall: 0.58\n",
            "Epoch: 04 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.456 | Train f1: 0.69 | Train precision: 0.79 | Train recall: 0.62\n",
            "\t Val. Loss: 0.460 |  Val. f1: 0.68 |  Val. precision: 0.75 | Val. recall: 0.63\n",
            "Epoch: 05 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.385 | Train f1: 0.74 | Train precision: 0.79 | Train recall: 0.69\n",
            "\t Val. Loss: 0.452 |  Val. f1: 0.69 |  Val. precision: 0.73 | Val. recall: 0.67\n",
            "Epoch: 06 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.331 | Train f1: 0.77 | Train precision: 0.80 | Train recall: 0.74\n",
            "\t Val. Loss: 0.451 |  Val. f1: 0.71 |  Val. precision: 0.70 | Val. recall: 0.72\n",
            "Epoch: 07 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.290 | Train f1: 0.80 | Train precision: 0.81 | Train recall: 0.79\n",
            "\t Val. Loss: 0.453 |  Val. f1: 0.71 |  Val. precision: 0.69 | Val. recall: 0.73\n",
            "Epoch: 08 | Epoch Time: 0m 20s\n",
            "\tTrain Loss: 0.262 | Train f1: 0.82 | Train precision: 0.83 | Train recall: 0.82\n",
            "\t Val. Loss: 0.458 |  Val. f1: 0.72 |  Val. precision: 0.71 | Val. recall: 0.74\n",
            "Epoch: 09 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.233 | Train f1: 0.84 | Train precision: 0.84 | Train recall: 0.84\n",
            "\t Val. Loss: 0.472 |  Val. f1: 0.73 |  Val. precision: 0.72 | Val. recall: 0.74\n",
            "Epoch: 10 | Epoch Time: 0m 19s\n",
            "\tTrain Loss: 0.214 | Train f1: 0.86 | Train precision: 0.86 | Train recall: 0.86\n",
            "\t Val. Loss: 0.457 |  Val. f1: 0.74 |  Val. precision: 0.72 | Val. recall: 0.76\n",
            "Val. Loss: 0.451 |  Val. f1: 0.71 | Val. precision: 0.70 | Val. recall: 0.72\n",
            "Número de capas ocultas: 8\n",
            "El modelo actual tiene 6,161,536 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 1.141 | Train f1: 0.10 | Train precision: 0.24 | Train recall: 0.07\n",
            "\t Val. Loss: 0.952 |  Val. f1: 0.26 |  Val. precision: 0.62 | Val. recall: 0.17\n",
            "Epoch: 02 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 0.852 | Train f1: 0.43 | Train precision: 0.65 | Train recall: 0.33\n",
            "\t Val. Loss: 0.705 |  Val. f1: 0.57 |  Val. precision: 0.70 | Val. recall: 0.49\n",
            "Epoch: 03 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 0.648 | Train f1: 0.62 | Train precision: 0.77 | Train recall: 0.53\n",
            "\t Val. Loss: 0.593 |  Val. f1: 0.63 |  Val. precision: 0.74 | Val. recall: 0.55\n",
            "Epoch: 04 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 0.526 | Train f1: 0.68 | Train precision: 0.80 | Train recall: 0.60\n",
            "\t Val. Loss: 0.547 |  Val. f1: 0.64 |  Val. precision: 0.75 | Val. recall: 0.56\n",
            "Epoch: 05 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 0.445 | Train f1: 0.70 | Train precision: 0.80 | Train recall: 0.63\n",
            "\t Val. Loss: 0.487 |  Val. f1: 0.65 |  Val. precision: 0.74 | Val. recall: 0.59\n",
            "Epoch: 06 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 0.390 | Train f1: 0.73 | Train precision: 0.79 | Train recall: 0.67\n",
            "\t Val. Loss: 0.495 |  Val. f1: 0.67 |  Val. precision: 0.70 | Val. recall: 0.64\n",
            "Epoch: 07 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 0.351 | Train f1: 0.74 | Train precision: 0.78 | Train recall: 0.71\n",
            "\t Val. Loss: 0.507 |  Val. f1: 0.66 |  Val. precision: 0.69 | Val. recall: 0.65\n",
            "Epoch: 08 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 0.320 | Train f1: 0.76 | Train precision: 0.78 | Train recall: 0.74\n",
            "\t Val. Loss: 0.480 |  Val. f1: 0.67 |  Val. precision: 0.67 | Val. recall: 0.67\n",
            "Epoch: 09 | Epoch Time: 0m 22s\n",
            "\tTrain Loss: 0.295 | Train f1: 0.78 | Train precision: 0.79 | Train recall: 0.77\n",
            "\t Val. Loss: 0.486 |  Val. f1: 0.67 |  Val. precision: 0.67 | Val. recall: 0.68\n",
            "Epoch: 10 | Epoch Time: 0m 21s\n",
            "\tTrain Loss: 0.271 | Train f1: 0.80 | Train precision: 0.80 | Train recall: 0.79\n",
            "\t Val. Loss: 0.501 |  Val. f1: 0.70 |  Val. precision: 0.69 | Val. recall: 0.71\n",
            "Val. Loss: 0.480 |  Val. f1: 0.67 | Val. precision: 0.67 | Val. recall: 0.67\n",
            "Número de capas ocultas: 9\n",
            "El modelo actual tiene 6,260,864 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 1.162 | Train f1: 0.05 | Train precision: 0.15 | Train recall: 0.04\n",
            "\t Val. Loss: 0.983 |  Val. f1: 0.28 |  Val. precision: 0.52 | Val. recall: 0.20\n",
            "Epoch: 02 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.947 | Train f1: 0.35 | Train precision: 0.59 | Train recall: 0.26\n",
            "\t Val. Loss: 0.843 |  Val. f1: 0.42 |  Val. precision: 0.65 | Val. recall: 0.32\n",
            "Epoch: 03 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.815 | Train f1: 0.47 | Train precision: 0.71 | Train recall: 0.36\n",
            "\t Val. Loss: 0.736 |  Val. f1: 0.52 |  Val. precision: 0.72 | Val. recall: 0.42\n",
            "Epoch: 04 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.678 | Train f1: 0.58 | Train precision: 0.78 | Train recall: 0.47\n",
            "\t Val. Loss: 0.649 |  Val. f1: 0.59 |  Val. precision: 0.69 | Val. recall: 0.52\n",
            "Epoch: 05 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.562 | Train f1: 0.65 | Train precision: 0.78 | Train recall: 0.56\n",
            "\t Val. Loss: 0.578 |  Val. f1: 0.60 |  Val. precision: 0.69 | Val. recall: 0.54\n",
            "Epoch: 06 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.478 | Train f1: 0.68 | Train precision: 0.80 | Train recall: 0.60\n",
            "\t Val. Loss: 0.550 |  Val. f1: 0.63 |  Val. precision: 0.69 | Val. recall: 0.59\n",
            "Epoch: 07 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.419 | Train f1: 0.70 | Train precision: 0.78 | Train recall: 0.64\n",
            "\t Val. Loss: 0.535 |  Val. f1: 0.64 |  Val. precision: 0.74 | Val. recall: 0.58\n",
            "Epoch: 08 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.383 | Train f1: 0.72 | Train precision: 0.78 | Train recall: 0.68\n",
            "\t Val. Loss: 0.543 |  Val. f1: 0.65 |  Val. precision: 0.68 | Val. recall: 0.62\n",
            "Epoch: 09 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.350 | Train f1: 0.75 | Train precision: 0.78 | Train recall: 0.72\n",
            "\t Val. Loss: 0.513 |  Val. f1: 0.66 |  Val. precision: 0.67 | Val. recall: 0.66\n",
            "Epoch: 10 | Epoch Time: 0m 24s\n",
            "\tTrain Loss: 0.320 | Train f1: 0.77 | Train precision: 0.79 | Train recall: 0.75\n",
            "\t Val. Loss: 0.523 |  Val. f1: 0.66 |  Val. precision: 0.64 | Val. recall: 0.69\n",
            "Val. Loss: 0.513 |  Val. f1: 0.66 | Val. precision: 0.67 | Val. recall: 0.66\n",
            "Número de capas ocultas: 10\n",
            "El modelo actual tiene 6,360,192 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 1.150 | Train f1: 0.07 | Train precision: 0.19 | Train recall: 0.05\n",
            "\t Val. Loss: 1.009 |  Val. f1: 0.24 |  Val. precision: 0.46 | Val. recall: 0.17\n",
            "Epoch: 02 | Epoch Time: 0m 26s\n",
            "\tTrain Loss: 0.964 | Train f1: 0.33 | Train precision: 0.55 | Train recall: 0.24\n",
            "\t Val. Loss: 0.868 |  Val. f1: 0.42 |  Val. precision: 0.61 | Val. recall: 0.33\n",
            "Epoch: 03 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.826 | Train f1: 0.46 | Train precision: 0.70 | Train recall: 0.35\n",
            "\t Val. Loss: 0.748 |  Val. f1: 0.54 |  Val. precision: 0.71 | Val. recall: 0.45\n",
            "Epoch: 04 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.665 | Train f1: 0.61 | Train precision: 0.76 | Train recall: 0.51\n",
            "\t Val. Loss: 0.662 |  Val. f1: 0.61 |  Val. precision: 0.68 | Val. recall: 0.56\n",
            "Epoch: 05 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.569 | Train f1: 0.67 | Train precision: 0.80 | Train recall: 0.58\n",
            "\t Val. Loss: 0.591 |  Val. f1: 0.63 |  Val. precision: 0.69 | Val. recall: 0.58\n",
            "Epoch: 06 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.495 | Train f1: 0.70 | Train precision: 0.81 | Train recall: 0.62\n",
            "\t Val. Loss: 0.533 |  Val. f1: 0.65 |  Val. precision: 0.76 | Val. recall: 0.58\n",
            "Epoch: 07 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.433 | Train f1: 0.73 | Train precision: 0.81 | Train recall: 0.66\n",
            "\t Val. Loss: 0.520 |  Val. f1: 0.66 |  Val. precision: 0.71 | Val. recall: 0.63\n",
            "Epoch: 08 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.391 | Train f1: 0.75 | Train precision: 0.81 | Train recall: 0.69\n",
            "\t Val. Loss: 0.514 |  Val. f1: 0.68 |  Val. precision: 0.72 | Val. recall: 0.65\n",
            "Epoch: 09 | Epoch Time: 0m 26s\n",
            "\tTrain Loss: 0.355 | Train f1: 0.76 | Train precision: 0.80 | Train recall: 0.72\n",
            "\t Val. Loss: 0.494 |  Val. f1: 0.67 |  Val. precision: 0.66 | Val. recall: 0.68\n",
            "Epoch: 10 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.331 | Train f1: 0.78 | Train precision: 0.81 | Train recall: 0.75\n",
            "\t Val. Loss: 0.505 |  Val. f1: 0.67 |  Val. precision: 0.67 | Val. recall: 0.67\n",
            "Val. Loss: 0.494 |  Val. f1: 0.67 | Val. precision: 0.66 | Val. recall: 0.68\n",
            "Número de capas ocultas: 11\n",
            "El modelo actual tiene 6,459,520 parámetros entrenables.\n",
            "to device cuda!!!!\n",
            "Epoch: 01 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 1.171 | Train f1: 0.01 | Train precision: 0.07 | Train recall: 0.01\n",
            "\t Val. Loss: 1.007 |  Val. f1: 0.20 |  Val. precision: 0.45 | Val. recall: 0.13\n",
            "Epoch: 02 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.978 | Train f1: 0.31 | Train precision: 0.54 | Train recall: 0.23\n",
            "\t Val. Loss: 0.889 |  Val. f1: 0.37 |  Val. precision: 0.61 | Val. recall: 0.28\n",
            "Epoch: 03 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.863 | Train f1: 0.44 | Train precision: 0.68 | Train recall: 0.33\n",
            "\t Val. Loss: 0.792 |  Val. f1: 0.48 |  Val. precision: 0.70 | Val. recall: 0.38\n",
            "Epoch: 04 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.757 | Train f1: 0.53 | Train precision: 0.76 | Train recall: 0.41\n",
            "\t Val. Loss: 0.731 |  Val. f1: 0.53 |  Val. precision: 0.76 | Val. recall: 0.42\n",
            "Epoch: 05 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.684 | Train f1: 0.58 | Train precision: 0.81 | Train recall: 0.47\n",
            "\t Val. Loss: 0.667 |  Val. f1: 0.57 |  Val. precision: 0.72 | Val. recall: 0.48\n",
            "Epoch: 06 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.596 | Train f1: 0.63 | Train precision: 0.78 | Train recall: 0.53\n",
            "\t Val. Loss: 0.603 |  Val. f1: 0.61 |  Val. precision: 0.70 | Val. recall: 0.55\n",
            "Epoch: 07 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.509 | Train f1: 0.67 | Train precision: 0.78 | Train recall: 0.59\n",
            "\t Val. Loss: 0.572 |  Val. f1: 0.63 |  Val. precision: 0.71 | Val. recall: 0.57\n",
            "Epoch: 08 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.450 | Train f1: 0.70 | Train precision: 0.78 | Train recall: 0.63\n",
            "\t Val. Loss: 0.540 |  Val. f1: 0.65 |  Val. precision: 0.71 | Val. recall: 0.60\n",
            "Epoch: 09 | Epoch Time: 0m 29s\n",
            "\tTrain Loss: 0.405 | Train f1: 0.72 | Train precision: 0.79 | Train recall: 0.67\n",
            "\t Val. Loss: 0.497 |  Val. f1: 0.66 |  Val. precision: 0.69 | Val. recall: 0.63\n",
            "Epoch: 10 | Epoch Time: 0m 30s\n",
            "\tTrain Loss: 0.378 | Train f1: 0.74 | Train precision: 0.79 | Train recall: 0.70\n",
            "\t Val. Loss: 0.511 |  Val. f1: 0.66 |  Val. precision: 0.65 | Val. recall: 0.67\n",
            "Val. Loss: 0.497 |  Val. f1: 0.66 | Val. precision: 0.69 | Val. recall: 0.63\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "fig = plt.figure()\n",
        "ax = fig.add_subplot(1, 1, 1)\n",
        "ax.plot([i for i in range(1,12)], validation_values)\n",
        "plt.xlabel('Numero de capas ocultas')\n",
        "plt.ylabel('Error en el conjunto de validación')\n",
        "plt.title('Error en el conjunto de validación en función de el número de capas ocultas')\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "tKzSkn13uPsK",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "outputId": "2d524db8-ba39-4602-efdd-26a955c431d4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdkAAAEWCAYAAAAq+e1jAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3xV9f3H8debsBGQEZGREEAQEBExjCKOuuoEq2jBiaPqz+JotRZXXdWqrVqrVkWrWBy4wOKoe1UUIWwZCrL33gQyPr8/zom9hBsSQnJubvJ5Ph73ce/Zn3vuuedzxvd8vzIznHPOOVf2qiU6AOecc66y8iTrnHPOlRNPss4551w58STrnHPOlRNPss4551w58STrnHPOlRNPsjEk3SnpxYiXOUPSsVEusyQkZUgySdUjXu4CSSeEn2+R9GxJxt2H5cVdhqQjJY2X1Ghf5l8WFHhe0npJ48txOZGsi/LatspqvpJ6SJolqX5ZxVaeJA2W9FWi40gW4TZyUFTLK3ZjlLQAaAbkxfQebmZDyiuoqsTMDimL+YSJ+kUza1UW86sIzOy+RCxDUhpwH3C6ma0v7xhKoC9wItDKzLaW10KSZF2UqzBBPwYMMrPNiY7HlS9Jw4ElZnZbeS2jpEd8Z5jZx8WNJKm6meUW6pdiZnlFTRNnHns1vnNlzcwWA8ckOo4YrYEF5Zlgi1IB10V5awf8ycymRLXAePtNV4mY2R5fwALghCKGDQbGAo8Aa4E/AcOBJ4H3gK3ACUAn4HNgAzAD6Bczj93Gj7OchsA/geXA0nA5KTExfAX8FVgPzAdO2cP3aQG8CawOx702ZtidBGeDRU3bH5gCbAJ+BE6OmecYYB0wF/h1oXm+BvwL2Bx+/8x46zdcF3+KGXYswVFW7Lg3AtOAjcCrQG2gHrAdyAe2hK8WQC3gb8Cy8PU3oFYR3y0lXIdrgHnAbwADqhf3G8RZv9uBxjH9Dg/nW4NgJ/ZpuL2sAV4C9i9ifezyewAXAgvDaW8tNG5P4BuCbWw58DhQM2baQ4CPwt9oJXBLEcvoF/5GGwi22U7Frf89bC+XArMItssPgNYxwwy4CpgTLusJQHHmcRmQTXAlaQtwF+E2X2g8Aw6K2Y6eAN4l2Oa+Bdolal1QRttWOG41YCjB/28twX+rcTgsI3a+RezL4sZcwnX6D+A/4e8wFjiQ4D+1HpgNHL4X+5k3gBcJ9iWXs4d9SJzv0SQcdxMwHrgnNnagY8zv+z1w7h7m1Rh4nmD/sB54K+zfCHgnjH99+LlVzHSfA38Ol78J+De7/udfB1aE6/lL4JCYYacCMwm2zaXAjXv4rW8j+M+vItiHNowZ3hf4mmD7XAwMjont8pjxdvltC35X4AogB9gZ/qZvh8MLtq/NYZy/jJn2IOCL8HutAV4tat3+NE2xIxSfZHOBawjOiuuEG+NG4MhwJdUPN5pbgJrAcWHwB8dsvLHj7/ZHBUYDTxMkkwPCH/bKmBhygF8T/Jn/L9xg4u2wqgETgT+GsbQl+NP/It5OptC0PcM4Twzn0xLoGA77kuAPWBvoRrBhHhczz+xww0oh2DDHxVu/lCzJjif4QzYm2IFfFW/csN/dwLhwnaUSbJD3FPH9riLYUaSF8/6MXXeERf4Gceb1KbseaPwFeCpmIz2R4AAgNVx3fytiffz0ewCdCf4IR4fTPkyw7RWMewTQm2A7zAjXzfXhsPoEO/Abwt+oPtArzjI6EBzonUhwQHATwbZbs7j1H2cd9A+n7RTGdBvwdaE/+jvA/kA6wTZz8h7+Z18V1R2744jZjtYSbLPVCQ5kRiZwXZTltnUdwTbdKtwOngZeCYdlUHySLer/U5J1uoZgO6tNsI3PBy4i+F//CfhsL/YzOcCZ4bh12MM+JM73GElwcFEP6EKQqL4Kh9UjSDiXhL99wQFu5yLm9S7BwUaj8Hc+JuzfBDgbqBtuI68TJuBw+OfhcruEy3yTXQ/QLg2nKzjQnxIzbDlwVPi5EdC9iNguJdjm2gL7AaOAEeGw1gR5ZFAYdxOgW0xsxSbZePvcsN854TZSDfgVwf+geTjsFYID/Grhb9U3Xuy7zK/YEYINcwvB0ULB69cxwS8qNP5w4F8x3UcRHNFUi+n3CnBnvPHjLL8ZsAOoE9NvEP/boAcDc2OG1Q1X4oFx5tUrTrw3A88X3snEmfZp4JE4/dMIzjLqx/T7M8F964J5fhwzrDOwvdD63Zske0FM94P8L3ntMm7Y70fg1JjuXxBcdiwqMV4V031SuB6rF/cbxJnX5cCn4WcR/OmPLmLcM4HJRayPn34Pgh3WyJjx6hEcgRZ1AHg9MDom1slFjBe7jNuB12KGVSPYkRxb3PqPM9//AJcVmtc2wrPZcN32jRn+GjC0iHkNZu+T7LMxw04FZidwXZTltjULOD6muzlBwio4uCouyRb1/ynJOn0mZtg1wKyY7kOBDeHnkuxnvowZtsd9SKH5pITft2NMv/v4X5L9FfDfQtM8DdwRZ17NCa5+NYq3vgqN2w1YH9P9OXB/THdngv9jvKtb+4frsmHYvQi4EmhQzDI/Aa6O6T445re+mfD/HWe6z9mHJBtnflOA/uHnfwHDiDmrL+5V0nuyZ1rR92QXF9OvBbDYzPJj+i0kOBPc0zwKtCY4UlkuqaBftULTrCj4YGbbwvH2K2JeLSRtiOmXAvx3D8svkEZwSbuwFsA627WQxEIgM158BDva2vtwH6bwvFrsYdwWYSyxcRU1fgt2Xaex05XkN4j1JvCYpOYEZ0T5hOtYUjPgUYKDr/rhfEpSoGaX+Mxsq6S1Bd2SOhCc3WYSHGhVJzibgOC3+7GEy/jpe5tZvqTF7LqtlnT9twYelfRQTD+F8ypYRuF5xdtmS6uoeSdiXZTlttUaGC0pdn+SR5CsS2Jv/j+FrYz5vD1Od8E6Lsl+pvA+srh9SIFUgm17T+uzV6FlVwdGxJlXWrjc3f5/kuoS3AY8meBsE6B+oTIzhWOoATSVtAa4l+CMMJXg/w/QlOBq4NkEV3bulzSN4ODymzjxxdt/FRyYlXQ73muSLgJ+R3DQBsHv2jT8fBPB5fnxktYDD5nZc3uaX1kUobdi+i0D0iRVi0m06cAPxcyjwGKCI92mpUxKhec138zal3LadnH6LwMaS6of8ydJJzjq31tbCRJEgQP3Ytp463AZwZ9uRkxcy4qYfjnBhkvMuAX26jcws/WSPiQ4qu5EcAZaEN99YayHmtk6SWcS3D8tzvJwXsBPO4EmMcOfBCYTlgqVdD0wICb+gSVYxjKCM5KCZYhgnZTmt1wM3GtmL5Vi2uLssp1I2pvtJBHrosy2rXD8S81sbOEBkjJKEVuBfVmnhZVkP1N4H1nSfchqgtskaQSX4AvGjV32F2Z2YgnjbCxpfzPbUGjYDQRnjr3MbIWkbgT/L8WMU/g3zSG4NH0ewe2SEwiuHjQkOJAWgJlNAPpLqgEMIbiKEzuvAgX7r9hl5BIc3CwmuB0Sz97sR3fZb0pqDTwDHA98Y2Z5kqbExL6C4NYkkvoCH0v60szmFrWAKJ6T/ZbgiPEmSTXCR03OILivUCwzWw58CDwkqYGkapLaSTqmFLGMBzZL+oOkOpJSJHWR1KME0/4TuETS8WEMLSV1tKD05dfAnyXVltSVoLBKaZ63nQKcKqlx+Ce/fi+mXQk0kdQwpt8rwG2SUiU1JbjkWlRcrwHXSmoVPg85tGBAKX+DlwnuVw0IPxeoT3D7YaOklsDvS/j93gBOl9RXUk2C+82x2299ggIYWyR1JLg3X+AdoLmk6yXVklRfUq84y3gNOC38jWsQ7Gh2EPy+e+sp4GZJhwBIaijpnFLMJ56pwCGSukmqTXD5saQSsS7Kctt6Crg33BkSbtv9SxFTYfuyTgvbq/3M3uxDwrPIUcCdkupK6gxcHDPKO0AHSReG+9saCp777RRnXssJbmv8Q1KjcNyjw8H1Cc7ON0hqDNwRJ/QLJHUOD3jvBt4I46tPsK2sJUh2Pz0aJqmmpPMlNTSzHIL/bH6ceUOw//qtpDaS9gvn82p4MPYScIKkcyVVl9QkPBCAYD96Vrh+DgrXZVFWEtzzLVCPIPGuDuO9hOC+c0H850gqeExyfThuUfEDJU+yb0vaEvMaXcLpMLOdBEn1FIKjnH8AF5nZ7D1OuKuLCAoQzCT4Ym8Q3E/YK+EGcDrB/YX5YTzPEhxpFTfteILCBI8QXPL4gv8dZQ0iuLSwjKAQxx17uLy+JyMI/uwLCHY8r5Z0wnB9vgLMk7RBUguCwhhZBKUppwOTwn7xPENQAnZqON6oQsP39jcYA7QHVpjZ1Jj+dwHdCdbhu3GWU9T3m0FQKvVlgjOj9cCSmFFuJDiC3hx+l1djpt1MUIDnDILLhXOAn8dZxvfABQTPSa4Jxz8j3Ib3ipmNBh4ARkraBHxH8B/YZ2b2A8FO7WOC71LiiggSsS4o223rUYJt60NJmwkKQcU7SNgr+7JO48yrNPuZvdmHDCG4hLmC4J7i8zHL3kxwz3tgOK8VBNthrSLmdSHBGehsghK8BQf2fyMokLWGYB2/H2faEeHyVxAUAro27P8vgku7Swl+03Fxlrkg/F9cBZxfRGzPhcv4kmA9ZhPcC8fMFhGUNbiBoBT1FOCwcLpHCO4PrwReIEjIRfkn0DncZ75lZjOBhwieVFhJcDUn9qpJD+BbSVsItsPrzGzeHuYflMB1iSNpEUFhjC8THYtzzpWEpM8JCskVWSObC3i1igkkKZWgYMCCBIfinHOuHHiSTZDw/swc4LHw0odzzrlKJmkuF0s6meB+TArBM4D3Fxo+mKDSg4ISeY+b2bPhzfAngQYERf3vNbMS3+t0zjnnSispkqykFIJHfk4kKOwygeBRjZkx4wwmqK5wSKFpOwBmZnPCwkATCaqHK1xk3TnnnCtTkTZjtg96EtTqNA9A0kiC57Bm7nEqfio1WPB5maRVBPdBi0yyTZs2tYyMjH2N2TnnqpSJEyeuMbPURMdRkSRLkm3JrrWLLCF+sf2zw+e8fgB+Gz5/9hNJPQkeFditphBJVxBUGE16ejpZWVllFLpzzlUNkhYWP1bVUpkKPr0NZJhZV4IWKF6IHaigir8RwCWFqngEwMyGmVmmmWWmpvqBmHPOuX2XLEl2KbtWu9WKQlWOmdlaM9sRdj5L0FoGAJIaEFR8cKuZFX4w2jnnnCsXyZJkJwDtw+q1ahLUZjImdoTwTLVAP4LWOgjHH03Q0s8bEcXrnHPOJcc9WTPLlTSEoGq2FOA5M5sh6W4gy8zGENSN2o+gAul1BM0bAZxL0AZpk7AEMgSN+06J8js455yrepLiEZ6oZWZmmhd8cs65vSNpopnFa6KvykqWy8XOOedc0vEk65xzzpUTT7LOORfHV3PWMG7e2kSH4ZJcUhR8cs65KE1fspFLho8nJ8+45riDuP6EDqRUU6LDcknIz2Sdcy7G5uwchrwyidT9anF291Y89ulcLn5uPGu37Ch+YucK8STrnHMhM+O2t75j8bptPDrocB469zAePLsrExas47S/f8XEhesSHaJLMp5knXMu9MbEJfx7yjJ+e0IHemQ0BuDcHmmMuroPNatX41dPj+OfX83HH310JeVJ1jnngLmrtvDHf8/gZ22bcPXPD9pl2CEtGvL2NX35eccDuOedmQx5eTKbs3MSFKlLJp5knXNVXnZOHte8Mpk6NVP428BucQs5NaxTg2EXHsHNp3Tk/Rkr6P/4WL5fsTkB0bpk4knWOVfl/fm9WcxavomHzjmMZg1qFzmeJK48ph0vX96LzTtyOfOJsYyevCTCSF2y8STrnKvSPpixghe+Wcjlfdvw844HlGiaXm2b8O61fenaqiG/fXUqt4yeTnZOXjlH6pKRJ1nnXJW1dMN2bnpjGoe2bMhNJ3fcq2kPqF+bly7vxVXHtOPlbxdxzlPfsHjdtnKK1CUrT7LOuSopNy+f616ZTF6+8digw6lZfe93h9VTqjH0lI48c1EmC9Zu5fTHvuLT2SvLIVqXrDzJOueqpEc/mUPWwvXc+8suZDStt0/zOrFzM965pi8t96/DpcOz+MsHs8nL98d8nCdZ51wV9PXcNTz+2VzOzWxF/24ty2SerZvUY9TVfRjYI40nPvuRC//5Las3ey1RVZ0nWedclbJmyw6ue3UKbZvW485+h5TpvGvXSOH+s7vylwFdmbhwPac/9l8mLPBaoqqypEmykk6W9L2kuZKGxhk+WNJqSVPC1+Uxwy6WNCd8XRxt5M65iiI/37jx9als3J7D4+d1p27N8mkj5ZzMNN76zZHUqZHCwGHjePa/87yWqCoqKZKspBTgCeAUoDMwSFLnOKO+ambdwtez4bSNgTuAXkBP4A5JjSIK3TlXgfzzq/l8/v1qbj+9M52aNyjXZXVq3oAx1/TlxE7N+NO7s7j6pUleS1QVlBRJliA5zjWzeWa2ExgJ9C/htL8APjKzdWa2HvgIOLmc4nTOVVBTF2/ggfdn84tDmnFBr/RIltmgdg2evKA7t53WiQ9nrqTf42OZvWJTJMt2FUOyJNmWwOKY7iVhv8LOljRN0huS0vZmWklXSMqSlLV69eqyits5VwFszs7hmlcm06xBbR48+zCk6NqGlcTlR7Vl5BW92RrWEvXGRK8lqqqIPMlK6iPpPEkXFbzKaNZvAxlm1pXgbPWFvZnYzIaZWaaZZaamppZRSM65RDMzbhn9HUs3bOfRgd1oWLdGQuLokdGYd689isPTGnHj61O5edQ0ryWqCog0yUoaAfwV6Av0CF+ZJZh0KZAW090q7PcTM1trZgXl5Z8FjijptM65yuv1rCW8PXUZvzuxA5lh83WJklq/FiMu68nVx7bjlfGLOfvJr1m0tuLUEpWfb574y5iiLPEmaRbQ2fZyoZKqAz8AxxMkyAnAeWY2I2ac5ma2PPz8S+APZtY7LPg0EegejjoJOMLMiixXn5mZaVlZWXsTonOuApq7ajOnP/YVR7RuxL8u7RW3dZ1E+WTWSn776hQAHj63Gyd0blbuyzQz1m7dyeJ121iyfjuL129j8brtLFkfdC9dv52L+7Tm1tPilSstnqSJZlaSE6cqo3zKrxftO+BAYPneTGRmuZKGAB8AKcBzZjZD0t1AlpmNAa6V1A/IBdYBg8Np10m6hyAxA9y9pwTrnKscsnPyGPLyZOrVrM4j58Zvvi6Rju/UjHevPYr/e2kil/8ri/87th03nNiB6in7doFxU3YOi9f9L3nGJtQl67ezbeeuZ6qN69UkrVEdOjdvwEmHNOPIdk33afluV1GfyX4GdAPGAz9VhWJm/SILogT8TNa55HfbW9N5cdwihl/Sg2MPLlnrOomQnZPH3e/M5OVvF9G7bWP+PuhwDqhfdHN723fmBclzfWwiLTgr3cam7Nxdxq9fqzqtGtclrVEdWjWqS1rjOqQ1qkta47q0bFSH/WqV3bmWn8nuLuoz2TsjXp5zrgr6z/TlvDhuEVce3bZCJ1gIaom675eHktm6EbeMns7pf/+KP591KDWrVwvOQNdtY3H4vmT9NtZs2Vlo+mpB8mxUh+7pjXZJoq0a1aFhnRqRlqZ2u4r0TBZAUjOCAk8A481sVaQBlICfyTqXvBav28apf/8vbZvW4/Wr+pSqdZ1Emb1iE//34iTmr9n6U7/q1UTLRkHibNWozk/Js+A9db9aFSaJ+pns7sr9TFZSupktCj+fC/wF+BwQ8Jik35vZG+Udh3Ou8svJy+e6kZPB4LFB3ZMqwQJ0PLABY4YcyX/nrKFJvZq0alyXAxvUrnD3k13JRXG5uJekc8zsIeBWoEfB2aukVOBjwJOsc26fPfLRD0xatIHHBh1OepO6iQ6nVOrXrsGphzZPdBiujJT7YZ6ZvQ6sKFheocvDa6OIwTlX+X01Zw1PfvEjA3ukccZhLRIdjnNARAWfzOyl8OP7kj4AXgm7fwW8F0UMzrnKa/XmHfz2tSm0S92PO84o2+brnNsXkZYuNrPfSzobODLsNczMRkcZg3OucsnPN254fSqbtucw4rKe1KmZkuiQnPtJ1I/wYGZvAm9GvVznXOX0zH/n8eUPq7n3l13oeGD5Nl/n3N6KJMlK+srM+kraDMQ+MyTAzMz/Gc65vTZ50Xr+8sH3nHrogZzXM5rm65zbG1Hdk+0bvtePYnnOucpv4/b/NV/357O6VphnRZ2LFXUrPL0l1Y/pri+pV5QxOOeSn5lxy6jpLN+Yzd8HHU7DOolpvs654kT9+MyTwJaY7q1hP+ecK7GRExbz7vTl3HBSB45o3SjR4ThXpKiTrGKbuTOzfBJQ+Mo5l7x+WLmZO8fMoO9BTbnq6HaJDse5PYo6yc6TdK2kGuHrOmBexDE455LU9p15DHl5EvVrV+fhXx1GNa9u0FVwUSfZq4A+BA2vLwF6AVdEHINzLknd/c5Mfli5hYfP7bbH5uCcqyiiroxiFTAwymU65yqHd6ct55Xxi7jqmHYc3SE10eE4VyKRJllJtYHLgEOAnw5DzezSKONwziWXxeu2MXTUNLql7c8NJ3VIdDjOlVjUl4tHAAcCvwC+AFoBm0syoaSTJX0vaa6koXsY72xJJikz7K4h6QVJ0yXNknRzGXwP51xEcvLyueaVgubrDqdGircp4pJH1FvrQWZ2O7DVzF4ATiO4L7tHklKAJ4BTgM7AIEmd44xXH7gO+Dam9zlALTM7FDgCuFJSxj5+D+dcRB768AemLN7A/Wd3Ja1xcjZf56quqJNsTvi+QVIXoCFwQAmm6wnMNbN5ZrYTGAn0jzPePcADQHZMPwPqSaoO1AF2AptKGb9zLkKfzl7JU1/8yKCe6ZzW1dtYdckn6iQ7TFIj4HZgDDATeLAE07UEFsd0Lwn7/URSdyDNzN4tNO0bBJVeLAcWAX81s3WFFyDpCklZkrJWr15d0u/jnCsn81Zv4bpXptC5eQP+ePpuF66cSwpRly5+Nvz4BdC2rOYrqRrwMDA4zuCeQB7QAmgE/FfSx2a2y/O5ZjYMGAaQmZlpu83FOReZLTtyuXLERKqniKcvPMKbr3NJK6pWeH63p+Fm9nAxs1gKpMV0twr7FagPdAE+DysJPxAYI6kfcB7wvpnlAKskjQUy8UownKuQ8vONG16bwrw1WxlxaU+/D+uSWlSXi+uHr0zg/wgu9bYkqJyiewmmnwC0l9RGUk2CZ23HFAw0s41m1tTMMswsAxgH9DOzLIJLxMcBSKoH9AZml9UXc86VrSc+m8sHM1Zy8ykd6XNQ00SH49w+iaqpu7sAJH0JdDezzWH3nUDhe6jxps+VNAT4AEgBnjOzGZLuBrLMbMweJn8CeF7SDIL2a583s2n79IWcc+Xi09krefjjHzizWwsu69sm0eE4t8+irpy/GUHp3gI7w37FMrP3gPcK9ftjEeMeG/N5C8FjPM65CqygoFOnAxt4+7Cu0og6yf4LGC9pdNh9JjA84hiccxWMF3RylVXUpYvvlfQf4Kiw1yVmNjnKGJxzFYsXdHKVWVSlixuY2SZJjYEF4atgWON4z60656qGgoJOt53WyQs6uUonqjPZl4HTgYkENTAVUNhdZs/MOueShxd0cpVdVKWLTw/f/V/knAN2rdHJCzq5yiqqy8V7fBbWzCZFEYdzrmLYsiOXK8KCTk9d4AWdXOUV1eXih/YwzAgri3DOVX4FBZ3me0EnVwVEdbn451EsxzlX8XlBJ1eVRP2cLGETd52B2gX9zOxfUcfhnIueF3RyVU2kSVbSHcCxBEn2PYJG2L8iqKTCOVeJeUEnVxVF3Z7sAOB4YIWZXQIcRtBwu3OuEtucneMFnVyVFHWS3W5m+UCupAbAKnZtws45V8kEBZ2mMn/NVp44r7sXdHJVStT3ZLMk7Q88Q1AxxRbgm4hjcM5F6InP5vLhTC/o5KqmqOsuvjr8+JSk94EG3uycc5WXF3RyVV2kl4sljZF0nqR6ZrbAE6xzlZcXdHIu+nuyDwF9gZmS3pA0QFLt4iZyziWX2IJO3nSdq8oiTbJm9kV4ybgt8DRwLkHhp2JJOlnS95LmShq6h/HOlmSSMmP6dZX0jaQZkqZ7Yneu/BQu6NSqkRd0clVXIiqjqAOcAfwK6A68UIJpUoAngBOBJcAESWPMbGah8eoD1wHfxvSrDrwIXGhmUyU1AXLK6Os45wrxgk7O/U/U92RfA2YR1FX8ONDOzK4pwaQ9gblmNs/MdgIjgf5xxrsHeADIjul3EjDNzKYCmNlaM8vbh6/hnCuCF3RybldR35P9J0FivcrMPgufmS2JlsDimO4lYb+fhC39pJnZu4Wm7QCYpA8kTZJ0U2mDd84VzQs6Obe7qB/h+aA85iupGvAwMDjO4OoEha16ANuATyRNNLNPCs3jCuAKgPT09PII07lKyws6ORdf1GeypbWUXWuGahX2K1Af6AJ8LmkB0BsYExZ+WgJ8aWZrzGwbQZ3Ju7Vva2bDzCzTzDJTU1PL6Ws4V/l4QSfnipYsSXYC0F5SG0k1gYHAmIKBZrbRzJqaWYaZZQDjgH5mlgV8ABwqqW5YCOoYYObui3DOlUZBQaebT+noBZ2cKyTqgk+SdIGkP4bd6ZJ6FjedmeUCQwgS5izgNTObIeluSf2KmXY9waXkCcAUYFKc+7bOuVLwgk7O7ZnMLLqFSU8C+cBxZtZJUiPgQzPrEVkQJZCZmWlZWVmJDsO5Cm3e6i30f3ws6U3q8sZVffw+rCMs75JZ/JhVR9TPyfYys+6SJkNwlhle/nXOJZGCgk41qlfzgk7O7UHUSTYnrFjCACSlEpzZOueSRGxBpxGX9fSCTs7tQdQFn/4OjAYOkHQv8BVwX8QxOOf2weOxBZ3aeUEn5/Yk6udkX5I0ETgeEHCmmc2KMgbnXOl9Mmslj3hBJ+dKLJIkK6lxTOcq4JXYYWa2Loo4nHOlN2/1Fq4f6TU6Obc3ojqTnUhwH1ZAOrA+/Lw/sAjwQ2LnKjAv6ORc6URyT9bM2phZW+Bj4Iyw4ogmwOnAh1HE4JwrvVtHf8f8NVt5/LzDvaCTc3sh6oJPvc3svYIOM/sP0CfiGJxze+HjmSsZM3UZ1x7X3gs6ObeXon6EZ5mk2wjadwU4H4rPYpgAACAASURBVFgWcQzOuRLalJ3DbW99x8HN6vN/x7ZLdDjOJZ2oz2QHAakEj/GMCj8PijgG51wJPfCf2azanM0DA7pSs3qyVHXuXMUR9SM864Drolymc650xs1by0vfLuLyvm3olrZ/osNxLin5oalzbjfZOXncPGo6aY3r8LuTOiQ6HOeSVtT3ZJ1zSeDRT+Ywf81WXrysF3Vr+m7CudLyM1nn3C6+W7qRYV/O45wjWtG3vZcmdm5fRN2ebCtJoyWtlrRK0puSWkUZg3OuaLl5+QwdNY1GdWty22mdEx2Oc0kv6jPZ54ExQHOgBfB22M85VwE8+9V8vlu6ibv7H0LDujUSHY5zSS/qJJtqZs+bWW74Gk7wGI9zLsHmr9nKIx/9wEmdm3FKlwMTHY5zlULUSXatpAskpYSvC4C1JZlQ0smSvpc0V9LQPYx3tiSTlFmof7qkLZJu3Mfv4FylY2bcPGoaNatX454zu3jl/86VkaiT7KXAucAKYDkwABhc3ERhQ+9PAKcAnYFBkna7YSSpPsFzuN/Gmc3DwH9KG7hzldnICYsZN28dt5zaiWYNaic6HOcqjaiTbCsz62dmqWZ2gJmdSdAqT3F6AnPNbJ6Z7QRGAv3jjHcP8ACQHdtT0pnAfGDGvoXvXOWzYmM29707i95tGzOwR1qiw3GuUok6yT5Wwn6FtQQWx3QvCfv9RFJ3IM3M3i3Ufz/gD8Bde1qApCskZUnKWr16dQlCci75mRm3//s7dublc7+3EetcmYuq0fafEbS2kyrpdzGDGgD73DClpGoEl4MHxxl8J/CImW3Z0w7EzIYBwwAyMzNtX2NyLhm8N30FH81cydBTOpLRtF6iw3Gu0omqKpeawH7h8urH9N9EcF+2OEuB2OtYrcJ+BeoDXYDPw0R6IDBGUj+gFzBA0oMEjcTnS8o2s8dL+V2cqxQ2bNvJHWO+o0vLBlzet02iw3GuUookyZrZF8AXkoab2cJSzGIC0F5SG4LkOhA4L2b+G4GfqqaR9Dlwo5llAUfF9L8T2OIJ1jn407uzWL8thxcu7Un1FK/8zbnyEHWlpLUkDQMyYpdtZsftaSIzy5U0BPiA4PLyc2Y2Q9LdQJaZjSnHmJ2rdP47ZzVvTFzC1ce245AWDRMdjnOVlsyiu/0oaSrwFDARyCvob2YTIwuiBDIzMy0rKyvRYThXLrbtzOWkR76kZko13rvuKGrX2OdiEc4BIGmimWUWP2bVEfWZbK6ZPRnxMp1zMR768AeWrN/Oq1f09gTrXDmL+kbM25KultRcUuOCV8QxOFdlTVm8gefHzuf8Xun0atsk0eE4V+lFfSZ7cfj++5h+BrSNOA7nqpydufn84Y1pHFC/NkNP6ZjocJyrEiJNsmbmzwk4lyBPffEj36/czLMXZVK/trew41wUIk2yki6K19/M/hVlHM5VNXNWbuaxT+dwetfmnNC5WaLDca7KiPpycY+Yz7WB44FJgCdZ58pJXr7xhzenUa9Wde7sd0iiw3GuSon6cvE1sd2S9ieo7N85V05GfLOASYs28NA5h9F0v1qJDse5KiXR1bxsBfw+rXPlZMn6bTz4wfcc1b4pZ3VvWfwEzrkyFfU92bcJShNDkOA7A69FGYNzVYWZcevo7wC475eHegs7ziVA1Pdk/xrzORdYaGZLIo7BuSrhrSlL+eKH1fzx9M6kNa6b6HCcq5Kivif7RZTLc66qWrtlB3e/PZNuaftzcZ+MRIfjXJUVyT1ZSV+F75slbYrzmi/p6ihica4quOvtmWzZkcuDA7qSUs0vEzuXKFE1ddc3fK8fb7ikJsDXwD+iiMe5yuzT2SsZM3UZ1x3fng7N4v7lnHMRifqeLJJSgGbs2tTdIknHRh2Lc5XN5uwcbh39HR2a7cfVP2+X6HCcq/KiLl18DXAHsBLID3sb0NXMlkcZi3OV0QPvz2bFpmyeOL8Ptap7CzvOJVrUZ7LXAQeb2dqIl+tcpTd+/jpeHLeIS47MoHt6o0SH45wj+sooFgMbI16mc5Vedk4eQ9+cRsv963DjSQcnOhznXCjqJDsP+FzSzZJ+V/AqyYSSTpb0vaS5kobuYbyzJZmkzLD7REkTJU0P348ro+/iXIXx2KdzmLdmK38+61Dq1Yq8qIVzrghR/xsXha+a4atEwsJSTwAnAkuACZLGmNnMQuPVJ7gk/W1M7zXAGWa2TFIX4APA65dzlcbMZZt4+ot5nNW9JUd3SE10OM65GFFXRnEXgKT9wu4tJZy0JzDXzOaF048E+gMzC413D/AAMY3Cm9nkmOEzgDqSapnZjlJ9CecqkNy8fP7w5jQa1qnB7ad1TnQ4zrlCIr1cLKmLpMkEyW5GePm2JG1vtSS4n1tgCYXORiV1B9LM7N09zOdsYFK8BCvpCklZkrJWr15dgpCcS7znxs5n+tKN3NnvEBrVK/HFIedcRKK+JzsM+J2ZtTaz1sANwDP7OlNJ1YCHw/kVNc4hBGe5V8YbbmbDzCzTzDJTU/2Sm6v4Fq7dysMf/cAJnQ7g9K7NEx2Ocy6OqJNsPTP7rKDDzD4H6pVguqVAWkx3q7BfgfpAF4JCVQuA3sCYmMJPrYDRwEVm9uO+fAHnKgIz4+ZR06lerRr3nNnFW9hxroKKvHSxpNslZYSv2whKHBdnAtBeUhtJNYGBwJiCgWa20cyamlmGmWUA44B+ZpYVNgz/LjDUzMaW/VdyLnqvZS3m6x/XMvSUjjRvWCfR4TjnihB1kr0USAVGAW8CTcN+e2RmucAQgpLBs4DXzGyGpLsl9Stm8iHAQcAfJU0JXwfsy5dwLpFWbsrmT+/OomebxpzXMz3R4Tjn9kBmVvxYVUxmZqZlZWUlOgzn4rpyRBaffb+a9687irap+yU6HOd+ImmimWUmOo6KJOrSxR+Fl28LuhtJ+iDKGJxLZv+ZvpwPZqzkuuPbe4J1LglEXRlFUzPbUNBhZuv90q1zxcvOyePtqcu4/z+z6dy8AVcc3TbRITnnSiDqJJsvKd3MFgFIak3QCo9zLo6Vm7IZ8c1CXh6/iHVbd9Kh2X488qtu1EiJujiFc640ok6ytwJfSfoCEHAUcEXEMThX4U1atJ7hYxfw3vTl5JlxfMcDuOTINvRp18Qf13EuiURdreL7Yc1MvcNe15vZmihjcK6i2pmbz3vTl/P81wuYungD9WtV56KfZXBxn9a0blKSx8mdcxVN5M11hEn1naiX61xFtXrzDl7+dhEvfruQ1Zt30LZpPe7qdwhnH9GK/bxFHeeSmv+DnUuQ75Zu5PmxC3h76jJ25uVzTIdUBg/I4Jj2qVSr5peEnasMPMk6F6HcvHw+mLGS4V/PZ8KC9dStmcLAnmlc9LMMDjrAH8lxrrKJJMlKaryn4Wa2Loo4nEuU9Vt38sqERYz4ZiHLN2aT1rgOt53WiXMy02hYp0aiw3POlZOozmQnEjyqE+8amAH+0J+rlGav2MTwsQsYPXkpO3Lz6dOuCXf1O4TjOzUjxS8JO1fpRZJkzaxNFMtxriLIyzc+mbWS58cu4Jt5a6lVvRpndW/JxX0y6Hhgg0SH55yLUKT3ZBU84Hc+0MbM7pGUDhxoZuOjjMO58rBxew6vZy3mhW8WsHjddlo0rM0fTu7IwB5p3qC6c1VU1AWf/gHkA8cB9wCbCVrj6RFxHM6VmbmrtvDC1wt4c9IStu3Mo0dGI24+pRMndW5Gda+ZybkqLeok28vMukuaDD/VXeyH+C7p5OcbX/ywmue/XsCXP6ymZko1zjisBZccmUGXlg0THZ5zroKIOsnmSEohrK9YUirBma1zScHMGDN1GY9+PId5a7aSWr8WvzuxA+f1SqfpfrUSHZ5zroKJOsn+HRgNHCDpXmAAcFvEMThXKss2bOfW0dP57PvVdG7egEcHduOULs2pWd0vCTvn4ou67uKXJE0Ejid4nOdMM5tVkmklnQw8CqQAz5rZ/UWMdzbwBtDDzLLCfjcDlwF5wLVm5m3YuhLLzzde/HYhD/xnNvkGt53WiUuObOOP4DjnipWIuotnA7P3ZprwEvMTwInAEmCCpDFmNrPQePWB64BvY/p1BgYChwAtgI8ldTCzvH36Iq5KmLtqC0PfnEbWwvX0Pagpfz7rUNIa1010WM65JJEs1Sr2BOaa2TwASSOB/sDMQuPdAzwA/D6mX39gpJntAOZLmhvO75tyj9olrZy8fJ7+4kf+/slc6tRM4S8DujLgiFbezJxzbq8kS5JtCSyO6V4C9IodIWxCL83M3pX0+0LTjis0bcvCC5B0BWHbtunp6WUUtktG05Zs4KY3pjF7xWZOO7Q5d/TrzAH1ayc6LOdcEkqWJLtHkqoBDwODSzsPMxsGDAPIzMy0sonMJZPtO/N4+KPv+edX80mtX4thFx7BSYccmOiwnHNJLOoan84iuJx7AEHBJwFmZsXVNbcUSIvpbhX2K1Af6AJ8Hl7OOxAYI6lfCaZ1jq/nrmHoqOksWreNQT3TGXpKR6+43zm3z6I+k30QOKOkJYpjTADaS2pDkCAHAucVDDSzjUDTgm5JnwM3mlmWpO3Ay5IeJij41B7wahwdABu35XDfe7N4NWsxGU3q8sqve/Ozdk0SHZZzrpKIOsmuLEWCxcxyJQ0BPiB4hOc5M5sh6W4gy8zG7GHaGZJeIygklQv8xksWO4D3v1vO7f+ewbqtO7nymLb89oQO1K6RkuiwnHOViMyiu/0o6VGCS7lvATsK+pvZqMiCKIHMzEzLyspKdBiunKzalM0f/z2D92esoHPzBjw4oKtXhehcGZA00cwyEx1HRRL1mWwDYBtwUkw/AypUknWVk5nxWtZi7n13Ftm5+dx08sH8+qi21PBK/J1z5STqGp8uiXJ5zhVYuHYrN4+aztc/rqVnm8bcf9ahtE3dL9FhOecquahLF3cAngSamVkXSV2Bfmb2pyjjcFVHbl4+z42dz8Mf/UCNatW495ddGNQjnWpeJaJzLgJRXy5+hqA2pqcBzGyapJcBT7KuzM1ctomho6YxbclGTujUjD+d2YUDG3qlEs656ESdZOua2fhCVdPlRhyDq+Syc/J47NM5PP3FPPavW4PHzzuc0w5t7lUiOuciF3WSXSOpHf9rT3YAsDziGFwlNmHBOv7w5jTmrd7KWd1bcvtpnWlUr2aiw3LOVVFRJ9nfEFRd2FHSUmA+cH7EMbhKaHN2Dg++/z0jxi2k5f51eOHSnhzTITXRYTnnqrioSxfPA06QVA+oZmabo1y+q5w+nb2SW0d/x4pN2VxyZAY3nnQw9WpVimq5nXNJLiF7IjPbmojluspl7ZYd3P3OTP49ZRkdmu3HE+f3oXt6o0SH5ZxzP/HDfZeU5q7azKBnvmXDtp1cf0J7rj72IGpW90olnHMVS2RJNmyOrreZfR3VMl3l9OPqLQx65lvMYMyQvnRqXlwjTs45lxiRHfqbWT7wRFTLc5XTgjVbOe+ZceTnG6/8upcnWOdchRb19bVPJJ0tf2DRlcLidds475lx7MzN5+Vf96Z9s/qJDsk55/Yo6iR7JfA6sFPSJkmbJW2KOAaXhJas38bAYePYujOPFy/vxcEHeoJ1zlV8UT/C43tGt9eWb9zOec98y6bsHF6+vDeHtPBm6ZxzySHy0sWS+gFHh52fm9k7UcfgksfKTdkMGjaO9Vt3MuLyXhzayhOscy55RHq5WNL9wHXAzPB1naQ/RxmDSx6rNmcz6JlxrN68g+GX9qRb2v6JDsk55/ZK1PdkTwVONLPnzOw54GTgtJJMKOlkSd9LmitpaJzhV0maLmmKpK8kdQ7715D0QjhslqSby/QbuXKxZssOzn/mW1ZszGb4pT05orVXMuGcSz6JeHo/9nSkRNf+JKUQPP5zCtAZGFSQRGO8bGaHmlk34EHg4bD/OUAtMzsUOAK4UlJG6cN35W3d1p1c8Oy3LF6/jecG96BHRuNEh+Scc6US9T3Z+4DJkj4DRHBvdrez0jh6AnPDuo+RNBLoT3DJGQAziy2lXI+wpZ/wvZ6k6kAdYCfgJZorqA3bggQ7f81Wnhvcg95tmyQ6JOecK7Woa3zKB3oDPcLefzCzFSWYvCWwOKZ7CdArzjJ+A/wOqAkcF/Z+gyAhLwfqAr81s3Vxpr0CuAIgPT29BCG5srZxew4X/nM8c1dt4ZmLMznyoKaJDsk55/ZJ1DU+3WRmy81sTPgqSYLdm2U8YWbtgD8At4W9ewJ5QAugDXCDpLZxph1mZplmlpma6k2kRW1Tdg4XPTee2Ss28fSFR3gzdc65SiHqe7IfS7pRUpqkxgWvEky3FEiL6W4V9ivKSODM8PN5wPtmlmNmq4CxQGZpgnflY8uOXAY/N54ZSzfyj/OP4OcdD0h0SM45VyaiTrK/Imi4/UtgYvjKKsF0E4D2ktpIqgkMBMbEjiCpfUznacCc8PMiwkvHYTu2vYHZ+/AdXBnatjOXS5+fwNQlG3ls0OGc2LlZokNyzrkyE/U92aFm9ureTmtmuZKGAB8AKcBzZjZD0t1AlpmNAYZIOgHIAdYDF4eTPwE8L2kGQWGr581sWhl8JbePtu/M49LhE8hauI5HBx7OKYc2T3RIzjlXpmRmxY9VVguTssyswl+qzczMtKyskpxgu9LKzsnj8heyGPvjGh45txtnHt4y0SE55/aRpInJsI+PUrLck3XF2JGbx47cvESHUSLZOXlcOWIiY39cw18GHOYJ1jlXaUX9nOyvwvffxPQzYLfSvq5kzIw3Ji7h7ndmklJNDOqZzoW9W9Ni/zqJDi2uHbl5XP3SJL74YTUPnt2VAUe0SnRIzjlXbqJuhadNlMur7FZszObmUdP47PvV9MxoTKN6NXj6ix8Z9uU8Tu5yIJf0yeCI1o2oKM335uTlM+TlyXw6exX3/rIL5/ZIK34i55xLYpEkWUk3mdmD4edzzOz1mGH3mdktUcRRWcSevebk5XPHGZ25+GcZVKsmFq/bxohxCxk5fhHvTlvOoS0bcsmRGZzWtTm1qqckLOacvHyufWUyH81cyd39D+H8Xq0TFotzzkUlkoJPkiaZWffCn+N1VwQVueDTio3Z3DJ6Op/OXkXPjMY8OKArGU3r7Tbetp25vDlpKcPHzufH1Vtpul8tzu+Vzvm90zmgfu1IY87Ny+f6V6fwzrTl3H56Zy7r6xc0nKuMvODT7qK6XKwiPsfrdnGYGW9OWspdb8/Y7ew1nro1q3Nh79Zc0Cud/85Zw/CvF/DoJ3P4x+dzOb1rCy45MoOurcq/6bi8fOPG16fyzrTl3HxKR0+wzrkqJaoka0V8jtftClm5KZubRxV/9hqPJI7ukMrRHVKZv2YrL3y9gNezFjN68lKOaN2IwX0yOLnLgdRIKfuC5vn5xh/enMZbU5bx+18czJXHtCvzZTjnXEUW1eXiPGArwVlrHWBbwSCgtpnVKPcg9kJFuVxccPZ699sz2JmXz02/6MjgPkWfvZbU5uwcXs9awgvfLGDh2m0c2KA2F/6sNYN6ptO4Xs0yiT0/37hl9HRGTljM9Se05/oTOpTJfJ1zFZdfLt5dpJVRJIuKkGRjz157ZDTiwQGH0aaEZ68llZ9vfPb9Kp4fu4Cv5q6hVvVqnNmtJYOPzKBT8walnq+Zcfu/v+PFcYsY8vODuOGkDhWmhLNzrvx4kt1d1M/JumKYGaPCe6878/K5/fTODO6TQco+nr3GU62aOL5TM47v1Iw5Kzfz/NcLGDVpCa9mLaZ328ZccmQbTujUbK+WbWbc9fZMXhy3iCuPaesJ1jlXpfmZbByJOpNduSmbW0ZN55PZq8hs3Yi/nFP2Z6/F2bBtJ69OWMy/vlnI0g3badWoDhf/LINze6TRsM6er+qbGX96dxb//Go+l/dtw62ndfIE61wV4meyu/MkG0fUSdbMGD15KXeOmcGO3Hx+/4uDueTINuVy9lpSuXn5fDxrJc+NXcD4+euoWzOFs7u34uI+GRx0wH67jW9m3P/+bJ7+Yh6D+2RwxxmdPcE6V8V4kt2dJ9k4okyyFeHstTjfLd3I8K8XMGbKMnbm5XN0h1QuOTKDY9qnUq2aMDMe+vAHHv9sLhf0Tuee/l08wTpXBXmS3Z0n2TiiSLIV8ey1OGu27OCVbxcxYtxCVm3eQdum9bi4TwZrtuzgsU/nMrBHGvf98tB9Lv3snEtOnmR350k2jvJOsqs2BbU2fTxrFUe0bsRfBnSlberul2Arqp25+fznu+U8P3YBUxZvAGDAEa148OyunmCdq8I8ye7OSxdHqPDZ622ndarwZ6/x1Kxejf7dWtK/W0smL1rPzOWbGNgj3ROsc84VkjRJVtLJwKNACvCsmd1faPhVBE3o5QFbgCvMbGY4rCvwNNAAyAd6mFl2hOGHZ6/f8fGslXRP35+/nHMY7ZLo7LUoh6c34vD0RokOwznnKqSkSLKSUoAngBOBJcAESWMKkmjoZTN7Khy/H/AwcLKk6sCLwIVmNlVSEyAnqtjNjLemLOXOMTPJzsnj1lM7cWnf5Dt7dc45t/eSIskCPYG5ZjYPQNJIoD/wU5I1s00x49fjf3UinwRMM7Op4XhrI4mYynv26pxzrmSSJcm2BBbHdC8BehUeSdJvgN8BNYHjwt4dAJP0AZAKjCxo27a8mBn/nrKMO8bM8LNX55yrwpIlyZaImT0BPCHpPOA24GKC79gX6EHQMMEnYQm4T2KnlXQFcAVAenp6qWNYtTmbW0YFZ6+Hp+/PX/3s1TnnqqxkSbJLgbSY7lZhv6KMBJ4MPy8BvjSzNQCS3gO6A7skWTMbBgyD4BGe0gQ5du4arn5pEttz8rjl1I5c1retn70651wVVvaNiJaPCUB7SW0k1QQGAmNiR5DUPqbzNGBO+PkD4FBJdcNCUMcQcy+3LGU0rUfXVg1579qjuOLodp5gnXOuikuKM1kzy5U0hCBhpgDPmdkMSXcDWWY2Bhgi6QSCksPrCS4VY2brJT1MkKgNeM/M3i2POFvuX4cRl+12q9g551wV5TU+xVER2pN1zrlk4zU+7S5ZLhc755xzSceTrHPOOVdOPMk655xz5cSTrHPOOVdOPMk655xz5cSTrHPOOVdOPMk655xz5cSfk41D0mpgYaLjKIWmwJpEBxEx/85VQ1X7zsn6fVubWWqig6hIPMlWIpKyqtqD4P6dq4aq9p2r2vetzPxysXPOOVdOPMk655xz5cSTbOUyLNEBJIB/56qhqn3nqvZ9Ky2/J+ucc86VEz+Tdc4558qJJ1nnnHOunHiSTXKS0iR9JmmmpBmSrkt0TFGRlCJpsqR3Eh1LFCTtL+kNSbMlzZL0s0THVN4k/Tbcrr+T9Iqk2omOqaxJek7SKknfxfRrLOkjSXPC90aJjNGVnifZ5JcL3GBmnYHewG8kdU5wTFG5DpiV6CAi9Cjwvpl1BA6jkn93SS2Ba4FMM+sCpAADExtVuRgOnFyo31DgEzNrD3wSdrsk5Ek2yZnZcjObFH7eTLDjbZnYqMqfpFbAacCziY4lCpIaAkcD/wQws51mtiGxUUWiOlBHUnWgLrAswfGUOTP7ElhXqHd/4IXw8wvAmZEG5cqMJ9lKRFIGcDjwbWIjicTfgJuA/EQHEpE2wGrg+fAS+bOS6iU6qPJkZkuBvwKLgOXARjP7MLFRRaaZmS0PP68AmiUyGFd6nmQrCUn7AW8C15vZpkTHU54knQ6sMrOJiY4lQtWB7sCTZnY4sJVKfgkxvA/Zn+AAowVQT9IFiY0qehY8Z+nPWiYpT7KVgKQaBAn2JTMbleh4InAk0E/SAmAkcJykFxMbUrlbAiwxs4KrFG8QJN3K7ARgvpmtNrMcYBTQJ8ExRWWlpOYA4fuqBMfjSsmTbJKTJIL7dLPM7OFExxMFM7vZzFqZWQZBQZhPzaxSn+GY2QpgsaSDw17HAzMTGFIUFgG9JdUNt/PjqeSFvWKMAS4OP18M/DuBsbh94Ek2+R0JXEhwNjclfJ2a6KBcubgGeEnSNKAbcF+C4ylX4Vn7G8AkYDrB/qrSVTco6RXgG+BgSUskXQbcD5woaQ7BGf39iYzRlZ5Xq+icc86VEz+Tdc4558qJJ1nnnHOunHiSdc4558qJJ1nnnHOunHiSdc4558qJJ1mXlCSZpIdium+UdGcCQyoRSQskNU10HFGTdKekG8PPgyW1SHRMzkXBk6xLVjuAsxKVsMIK613pDCaoJtG5Ss+TrEtWuQQVE/y28ABJwyUNiOneEr4fK+kLSf+WNE/S/ZLOlzRe0nRJ7cLxUiW9KWlC+Doy7H+npBGSxgIjJGVI+lTSNEmfSEqPE0sTSR+GbaI+Cyhm2AXhsqdIelpSSpzpe0j6WtLUcNz64XL/K2lS+OoT8/2+lPSupO8lPSWpWjjsSUlZYRx3xcz//rAt4mmS/hpn+Y0lvRUOHyepa9h/P0nPh+ttmqSzY9d1+HmApOGF5jcAyCSoVGOKpDqS/hiu5+8kDQtrd0LStTGxjdxtC3AuGZiZv/yVdC9gC9AAWAA0BG4E7gyHDQcGxI4bvh8LbACaA7WApcBd4bDrgL+Fn18G+oaf0wmqrAS4E5gI1Am73wYuDj9fCrwVJ86/A38MP59GUNF7U6BTOH2NcNg/gIsKTVsTmAf0CLsbEDQUUBeoHfZrD2TFfL9soC1B26sfFawHoHH4ngJ8DnQFmgDf879KafaPE/9jwB3h5+OAKeHnBwrWV9jdKHZdh58HAMNj1t2N4efPCdqIJTa28PMI4Izw8zKgVlGx+ctfyfDyS14uaZnZJkn/ImjYe3sJJ5tgYRNikn4ECppOmw78PPx8AtA5PKECaBC2cgQwxswKlvUz4Kzw8wj4//buJcTmMIzj+PdRNGKIBVlINm4lyiULuWQll5KJIklKyqVILrGgLKSUkJSEUsqGxVBMKLeFWTBusUEspGRICo15LJ7nmH/HXy7juPX7rM78z/t/L6em5/+877/3ZWdJe5MqgTPuzwAAAp5JREFUZdz9tJm15vVpwBigOdvpzpebwA8Fnrl7c2W82e8ewD4zGw18BIYU7rnu7g+z3HFgIrE14TwzW0YE6QHACGLv43fAITNrBBpL+j8RmJvtX8jMvFf+Rp8PUHf31pJ7v9dUM1tPPDz0Be4SDyC3iIz3FHCqE/WL/DEKsvKv203sbXu4cK2NXArJ6dJuhe/eFz63F/5up+P/oQswwd3fFRvKYPj2F/XbgKPuvukn7l0DPAdGEX0t9rN6n1Q3s8FEpj/O3VtzCrfO3dvMbDwR8BuAlUS22hnF9uu+VdjM6ogsfqy7P82X1yr3zSAeUmYBm81spLu3dbJ/Ir+V1mTln+buL4ETwNLC5cdElggwG+j6g9WeIzbjByAzxjLX6MjmFgKXS8pcAhZkPdOBPnn9PNBgZv3yu75mNqjq3gfAADMbl2Xq84Wr3kSG204cDlFcyx1vZoPz4WI+cIWYZn4LvDaz/sD0rK8n0NvdzxCBe1RJ/y/n2DCzKcCLzKibgBWF36gyrudmNjzbn1NSH8AboD4/VwLqi+xPQ9bXBRjo7heBDTnmntUVifztFGTlf7CLWOesOAhMNrMWYkr3R7PP1cDYfOHmHrD8K+VWAUssTsVZRKzrVtsGTDKzu8S08RMAd78HbAHO5f1NxDTuZ+7+gQiUe3MsTURQ2g8szmvDqsbXDOwjjoR7BJx09xbgBnCfWG++mmXrgcZs/wqwtqT/W4ExWWYHHcevbQf65MtKLXRMtW8kpp2vAc/KfzaOAAfM7CYxk3AQuAOczf5DPDgcM7Pb2fc97v7qK/WJ/LV0Co/IfyIzzXXuPvNP90VEgjJZERGRGlEmKyIiUiPKZEVERGpEQVZERKRGFGRFRERqREFWRESkRhRkRUREauQT7gxhTKLZ04AAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Como se tiene el mejor f1 score utilizando 2 capas ocultas y no se tiene un error significativamente alto, se escoge este número de capas ocultas."
      ],
      "metadata": {
        "id": "Lcgrsl9phc61"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Modelo 9: Encoder-Decoder con Attention"
      ],
      "metadata": {
        "id": "-F_VIK0Ior_1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class Encoder(nn.Module):\n",
        "    def __init__(self, input_size, embedding_size, hidden_size, num_layers, p):\n",
        "        super(Encoder, self).__init__()\n",
        "        self.hidden_size = hidden_size\n",
        "        self.num_layers = num_layers\n",
        "\n",
        "        self.embedding = nn.Embedding(input_size, embedding_size)\n",
        "        self.rnn = nn.LSTM(embedding_size, hidden_size, num_layers, bidirectional=True)\n",
        "\n",
        "        self.fc_hidden = nn.Linear(hidden_size * 2, hidden_size)\n",
        "        self.fc_cell = nn.Linear(hidden_size * 2, hidden_size)\n",
        "        self.dropout = nn.Dropout(p)\n",
        "\n",
        "    def forward(self, x):\n",
        "        # x: (seq_length, N) where N is batch size\n",
        "\n",
        "        embedding = self.dropout(self.embedding(x))\n",
        "        # embedding shape: (seq_length, N, embedding_size)\n",
        "\n",
        "        encoder_states, (hidden, cell) = self.rnn(embedding)\n",
        "        # outputs shape: (seq_length, N, hidden_size)\n",
        "\n",
        "        # Use forward, backward cells and hidden through a linear layer\n",
        "        # so that it can be input to the decoder which is not bidirectional\n",
        "        # Also using index slicing ([idx:idx+1]) to keep the dimension\n",
        "        hidden = self.fc_hidden(torch.cat((hidden[0:1], hidden[1:2]), dim=2))\n",
        "        cell = self.fc_cell(torch.cat((cell[0:1], cell[1:2]), dim=2))\n",
        "\n",
        "        return encoder_states, hidden, cell\n",
        "\n",
        "\n",
        "class Decoder(nn.Module):\n",
        "    def __init__(\n",
        "        self, input_size, embedding_size, hidden_size, output_size, num_layers, p\n",
        "    ):\n",
        "        super(Decoder, self).__init__()\n",
        "        self.hidden_size = hidden_size\n",
        "        self.num_layers = num_layers\n",
        "\n",
        "        self.embedding = nn.Embedding(input_size, embedding_size)\n",
        "        self.rnn = nn.LSTM(hidden_size * 2 + embedding_size, hidden_size, num_layers)\n",
        "\n",
        "        self.energy = nn.Linear(hidden_size * 3, 1)\n",
        "        self.fc = nn.Linear(hidden_size, output_size)\n",
        "        self.dropout = nn.Dropout(p)\n",
        "        self.softmax = nn.Softmax(dim=0)\n",
        "        self.relu = nn.ReLU()\n",
        "\n",
        "    def forward(self, x, encoder_states, hidden, cell):\n",
        "        x = x.unsqueeze(0)\n",
        "        # x: (1, N) where N is the batch size\n",
        "\n",
        "        embedding = self.dropout(self.embedding(x))\n",
        "        # embedding shape: (1, N, embedding_size)\n",
        "\n",
        "        sequence_length = encoder_states.shape[0]\n",
        "        h_reshaped = hidden.repeat(sequence_length, 1, 1)\n",
        "        # h_reshaped: (seq_length, N, hidden_size*2)\n",
        "\n",
        "        energy = self.relu(self.energy(torch.cat((h_reshaped, encoder_states), dim=2)))\n",
        "        # energy: (seq_length, N, 1)\n",
        "\n",
        "        attention = self.softmax(energy)\n",
        "        # attention: (seq_length, N, 1)\n",
        "\n",
        "        # attention: (seq_length, N, 1), snk\n",
        "        # encoder_states: (seq_length, N, hidden_size*2), snl\n",
        "        # we want context_vector: (1, N, hidden_size*2), i.e knl\n",
        "        context_vector = torch.einsum(\"snk,snl->knl\", attention, encoder_states)\n",
        "        #print(context_vector.size(), embedding.size())\n",
        "        rnn_input = torch.cat((context_vector, embedding), dim=2)\n",
        "        # rnn_input: (1, N, hidden_size*2 + embedding_size)\n",
        "\n",
        "        outputs, (hidden, cell) = self.rnn(rnn_input, (hidden, cell))\n",
        "        # outputs shape: (1, N, hidden_size)\n",
        "\n",
        "        predictions = self.fc(outputs).squeeze(0)\n",
        "        # predictions: (N, hidden_size)\n",
        "\n",
        "        return predictions, hidden, cell\n",
        "\n",
        "\n",
        "class Seq2Seq(nn.Module):\n",
        "    def __init__(self, encoder, decoder):\n",
        "        super(Seq2Seq, self).__init__()\n",
        "        self.encoder = encoder\n",
        "        self.decoder = decoder\n",
        "\n",
        "    def forward(self, source, target=None, teacher_force_ratio=0.5):\n",
        "        batch_size = source.shape[1]\n",
        "        target_len = target.shape[0] if target!=None else source.shape[0]\n",
        "        target_vocab_size = len(NER_TAGS.vocab)\n",
        "\n",
        "        outputs = torch.zeros(target_len, batch_size, target_vocab_size).to(device)\n",
        "        encoder_states, hidden, cell = self.encoder(source)\n",
        "\n",
        "        # First input will be <SOS> token\n",
        "        x = torch.zeros(batch_size, dtype=torch.int64).to(device)\n",
        "        for t in range(1, target_len):\n",
        "            # At every time step use encoder_states and update hidden, cell\n",
        "            output, hidden, cell = self.decoder(x, encoder_states, hidden, cell)\n",
        "\n",
        "            # Store prediction for current time step\n",
        "            outputs[t] = output\n",
        "\n",
        "            # Get the best word the Decoder predicted (index in the vocabulary)\n",
        "            best_guess = output.argmax(1)\n",
        "\n",
        "            # With probability of teacher_force_ratio we take the actual next word\n",
        "            # otherwise we take the word that the Decoder predicted it to be.\n",
        "            # Teacher Forcing is used so that the model gets used to seeing\n",
        "            # similar inputs at training and testing time, if teacher forcing is 1\n",
        "            # then inputs at test time might be completely different than what the\n",
        "            # network is used to. This was a long comment.\n",
        "            if target != None:\n",
        "              x = target[t] if random.random() < teacher_force_ratio else best_guess\n",
        "            else:\n",
        "              x = best_guess\n",
        "\n",
        "        return outputs\n",
        "\n",
        "\n",
        "def train_seq_2_seq(model, iterator, optimizer, criterion):\n",
        "\n",
        "    epoch_loss = 0\n",
        "    epoch_precision = 0\n",
        "    epoch_recall = 0\n",
        "    epoch_f1 = 0\n",
        "\n",
        "    model.train()\n",
        "\n",
        "    # Por cada batch del iterador de la época:\n",
        "    for batch in iterator:\n",
        "\n",
        "        # Extraemos el texto y los tags del batch que estamos procesado\n",
        "        text = batch.text\n",
        "        tags = batch.nertags\n",
        "\n",
        "        # Reiniciamos los gradientes calculados en la iteración anterior\n",
        "        optimizer.zero_grad()\n",
        "\n",
        "        #text = [sent len, batch size]\n",
        "\n",
        "        # Predecimos los tags del texto del batch.\n",
        "        predictions = model(text, tags)\n",
        "\n",
        "        #predictions = [sent len, batch size, output dim]\n",
        "        #tags = [sent len, batch size]\n",
        "\n",
        "        # Reordenamos los datos para calcular la loss\n",
        "        predictions = predictions.view(-1, predictions.shape[-1])\n",
        "        tags = tags.view(-1)\n",
        "\n",
        "        #predictions = [sent len * batch size, output dim]\n",
        "\n",
        "\n",
        "\n",
        "        # Calculamos el Cross Entropy de las predicciones con respecto a las etiquetas reales\n",
        "        loss = criterion(predictions, tags)\n",
        "        \n",
        "        # Calculamos el accuracy\n",
        "        precision, recall, f1 = calculate_metrics(predictions, tags)\n",
        "\n",
        "        # Calculamos los gradientes\n",
        "        loss.backward()\n",
        "\n",
        "        # Actualizamos los parámetros de la red\n",
        "        optimizer.step()\n",
        "\n",
        "        # Actualizamos el loss y las métricas\n",
        "        epoch_loss += loss.item()\n",
        "        epoch_precision += precision\n",
        "        epoch_recall += recall\n",
        "        epoch_f1 += f1\n",
        "\n",
        "    return epoch_loss / len(iterator), epoch_precision / len(\n",
        "        iterator), epoch_recall / len(iterator), epoch_f1 / len(iterator)"
      ],
      "metadata": {
        "id": "B0O_OAoEouSj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# tamaño del vocabulario. recuerden que la entrada son vectores bag of word(one-hot).\n",
        "INPUT_DIM = len(TEXT.vocab)\n",
        "EMBEDDING_DIM = 300  # dimensión de los embeddings.\n",
        "OUTPUT_DIM = len(NER_TAGS.vocab)  # número de clases\n",
        "HIDDEN_DIM = 2**6 # dimensión de la capas LSTM\n",
        "\n",
        "N_LAYERS = 3  # número de capas.\n",
        "DROPOUT = 0.6\n",
        "BIDIRECTIONAL = True\n",
        "\n",
        "\n",
        "# Model hyperparameters\n",
        "input_size_encoder = INPUT_DIM\n",
        "input_size_decoder = INPUT_DIM\n",
        "output_size = OUTPUT_DIM\n",
        "encoder_embedding_size = 300\n",
        "decoder_embedding_size = 300\n",
        "hidden_size = 128\n",
        "num_layers = 1\n",
        "enc_dropout = 0.0\n",
        "dec_dropout = 0.0\n",
        "\n",
        "\n",
        "encoder_net = Encoder(\n",
        "    input_size_encoder, encoder_embedding_size, hidden_size, num_layers, enc_dropout\n",
        ").to(device)\n",
        "\n",
        "decoder_net = Decoder(\n",
        "    input_size_decoder,\n",
        "    decoder_embedding_size,\n",
        "    hidden_size,\n",
        "    output_size,\n",
        "    num_layers,\n",
        "    dec_dropout,\n",
        ").to(device)\n",
        "\n",
        "model = Seq2Seq(encoder_net, decoder_net).to(device)"
      ],
      "metadata": {
        "id": "IQKfoc3wBbtm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_name = 'attention'\n",
        "criterion = baseline_criterion\n",
        "n_epochs = baseline_n_epochs\n",
        "\n",
        "train_error = []\n",
        "val_error = []\n",
        "\n",
        "print(f'El modelo actual tiene {count_parameters(model):,} parámetros entrenables.')\n",
        "# Optimizador\n",
        "optimizer = optim.Adam(model.parameters())\n",
        "\n",
        "# Enviamos el modelo y la loss a cuda (en el caso en que esté disponible)\n",
        "model = model.to(device)\n",
        "criterion = criterion.to(device)\n",
        "\n",
        "\n",
        "best_valid_loss = float('inf')\n",
        "\n",
        "for epoch in range(n_epochs):\n",
        "\n",
        "    start_time = time.time()\n",
        "\n",
        "    # Recuerdo: train_iterator y valid_iterator contienen el dataset dividido en batches.\n",
        "\n",
        "    # Entrenar\n",
        "    train_loss, train_precision, train_recall, train_f1 = train_seq_2_seq(\n",
        "        model, train_iterator, optimizer, criterion)\n",
        "\n",
        "    # Evaluar (valid = validación)\n",
        "    valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "        model, valid_iterator, criterion)\n",
        "    \n",
        "    train_error += [train_loss]\n",
        "    val_error += [valid_loss]\n",
        "\n",
        "    end_time = time.time()\n",
        "\n",
        "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
        "\n",
        "    # Si obtuvimos mejores resultados, guardamos este modelo en el almacenamiento (para poder cargarlo luego)\n",
        "    # Si detienen el entrenamiento prematuramente, pueden cargar el modelo en el siguiente recuadro de código.\n",
        "    if valid_loss < best_valid_loss:\n",
        "        best_valid_loss = valid_loss\n",
        "        torch.save(model.state_dict(), '{}.pt'.format(model_name))\n",
        "    # Si ya no mejoramos el loss de validación, terminamos de entrenar.\n",
        "\n",
        "    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
        "    print(\n",
        "        f'\\tTrain Loss: {train_loss:.3f} | Train f1: {train_f1:.2f} | Train precision: {train_precision:.2f} | Train recall: {train_recall:.2f}'\n",
        "    )\n",
        "    print(\n",
        "        f'\\t Val. Loss: {valid_loss:.3f} |  Val. f1: {valid_f1:.2f} |  Val. precision: {valid_precision:.2f} | Val. recall: {valid_recall:.2f}'\n",
        "    )\n",
        "# cargar el mejor modelo entrenado.\n",
        "model.load_state_dict(torch.load('{}.pt'.format(model_name)))\n",
        "\n",
        "# Limpiar ram de cuda\n",
        "torch.cuda.empty_cache()\n",
        "\n",
        "valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "    model, valid_iterator, criterion)\n",
        "\n",
        "\n",
        "print(\n",
        "    f'Val. Loss: {valid_loss:.3f} |  Val. f1: {valid_f1:.2f} | Val. precision: {valid_precision:.2f} | Val. recall: {valid_recall:.2f}'\n",
        ")\n",
        "\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MNklIwTmEZh4",
        "outputId": "7d751fc7-6cc8-46b6-a03c-3bd8fdf8d21e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "El modelo actual tiene 11,413,877 parámetros entrenables.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
            "  warnings.warn('{} seems not to be NE tag.'.format(chunk))\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch: 01 | Epoch Time: 0m 49s\n",
            "\tTrain Loss: 0.596 | Train f1: 0.62 | Train precision: 0.76 | Train recall: 0.53\n",
            "\t Val. Loss: 0.905 |  Val. f1: 0.40 |  Val. precision: 0.66 | Val. recall: 0.30\n",
            "Epoch: 02 | Epoch Time: 0m 48s\n",
            "\tTrain Loss: 0.570 | Train f1: 0.64 | Train precision: 0.78 | Train recall: 0.55\n",
            "\t Val. Loss: 0.900 |  Val. f1: 0.45 |  Val. precision: 0.59 | Val. recall: 0.37\n",
            "Epoch: 03 | Epoch Time: 0m 43s\n",
            "\tTrain Loss: 0.551 | Train f1: 0.66 | Train precision: 0.79 | Train recall: 0.57\n",
            "\t Val. Loss: 0.898 |  Val. f1: 0.43 |  Val. precision: 0.62 | Val. recall: 0.33\n",
            "Epoch: 04 | Epoch Time: 0m 43s\n",
            "\tTrain Loss: 0.538 | Train f1: 0.67 | Train precision: 0.79 | Train recall: 0.58\n",
            "\t Val. Loss: 0.907 |  Val. f1: 0.44 |  Val. precision: 0.64 | Val. recall: 0.34\n",
            "Epoch: 05 | Epoch Time: 0m 43s\n",
            "\tTrain Loss: 0.519 | Train f1: 0.68 | Train precision: 0.80 | Train recall: 0.59\n",
            "\t Val. Loss: 0.919 |  Val. f1: 0.45 |  Val. precision: 0.60 | Val. recall: 0.37\n",
            "Epoch: 06 | Epoch Time: 0m 43s\n",
            "\tTrain Loss: 0.504 | Train f1: 0.69 | Train precision: 0.80 | Train recall: 0.60\n",
            "\t Val. Loss: 0.924 |  Val. f1: 0.44 |  Val. precision: 0.59 | Val. recall: 0.36\n",
            "Epoch: 07 | Epoch Time: 0m 43s\n",
            "\tTrain Loss: 0.495 | Train f1: 0.70 | Train precision: 0.81 | Train recall: 0.62\n",
            "\t Val. Loss: 0.947 |  Val. f1: 0.46 |  Val. precision: 0.60 | Val. recall: 0.39\n",
            "Epoch: 08 | Epoch Time: 0m 43s\n",
            "\tTrain Loss: 0.480 | Train f1: 0.71 | Train precision: 0.82 | Train recall: 0.63\n",
            "\t Val. Loss: 0.954 |  Val. f1: 0.45 |  Val. precision: 0.59 | Val. recall: 0.37\n",
            "Epoch: 09 | Epoch Time: 0m 44s\n",
            "\tTrain Loss: 0.469 | Train f1: 0.72 | Train precision: 0.83 | Train recall: 0.64\n",
            "\t Val. Loss: 0.981 |  Val. f1: 0.45 |  Val. precision: 0.59 | Val. recall: 0.37\n",
            "Epoch: 10 | Epoch Time: 0m 43s\n",
            "\tTrain Loss: 0.459 | Train f1: 0.73 | Train precision: 0.83 | Train recall: 0.65\n",
            "\t Val. Loss: 0.972 |  Val. f1: 0.46 |  Val. precision: 0.60 | Val. recall: 0.38\n",
            "Val. Loss: 0.898 |  Val. f1: 0.43 | Val. precision: 0.62 | Val. recall: 0.33\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "fig = plt.figure()\n",
        "ax = fig.add_subplot(1, 1, 1)\n",
        "ax.plot(train_error, label='Train loss')\n",
        "ax.plot(val_error, label = 'Validation loss')\n",
        "plt.xlabel('Época')\n",
        "plt.ylabel('Error en el conjunto de validación')\n",
        "plt.title('Evolución de la pérdida en el conjunto de validación y test')\n",
        "plt.legend()\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 297
        },
        "id": "0klMsVpqlrYg",
        "outputId": "637f8778-6f13-4143-de2c-1deac3ae0289"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEYCAYAAACtEtpmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU9bn48c+ThewLWVgDBJBFZBOD2KJWpNaNilo3qlVKq629rVVbW2tdqK3e3pba1lvrvdTd2nKtVX640qqAWq2yiLYiIJsQ1rBlISRkeX5/fM8kkzCTOQmZzECe9+s1r5mzP+fM8sz3+z3ne0RVMcYYY9qSEOsAjDHGxD9LFsYYYyKyZGGMMSYiSxbGGGMismRhjDEmIksWxhhjIupWyUJEVESOO8J13CYiD4UYP1lE3hORnkey/qD1FXvxJnVg2dki8sfOiKONbQwTkQ9FZHA7l1ssIl/3Xl8pIn/zM2+86uoYRWSgiFSJSGJXbdMvEZkpIm918TZbfE9E5GURucbPvEewzZDbEJFbROQxEZEjWX+8OqKDFi0isgnoDTQEjX5MVb8dm4iaqeq9rceJyADgXmCaqu7r+qi6lojkAH8ALlHVjR1dj6o+BTzVaYF1A6q6GcjsjHWJyGzgOFW9qjPWFw9U9dxYbENEzgUmAFdpF1y8JiKPAaWqevsRrqcY2Agkq2p9W/PGZbLwfFFVX411EH6o6hbgc7GOo6uoajlwRlvzeP+uRFUbuyQoY2JIVV8GXo51HNF0VFVDiUiKiOwXkdFB4wpF5KCI9PKGrxWRdSKyV0QWiEi/MOtqUX3QuggtIieIyN+99ewUkdu88S2qeETkAhH5yItrsYgcHzRtk4h836uuKReR/xOR1DDxJIrIHBHZLSIbgPNbTc8RkYdFZLuIbBWRn/mtihCRv4jIDi+GN0TkhDbmXSwi/+lVqVWIyP8Tkbyg6aeIyNve/n4gIme0WvYeEfkHUA0MEZGzRGS1t+3fARI0f+tj3ta8Q0XkdRHZ4x2jp0Qkt439GBn0/q0RkcuCpj0mIg+IyIsiUiki74rI0DbWFXaf2+K9p7eJyHpvO8u9Uigi8lkRWert61IR+Wyr4/hTEfmHt9zfRKTAm9a62mWTiHw+aNmmz2fQvNeIyGbvuP3Ym3YOcBtwubhqrQ+88f28781e73t0bRv7l+/NWyEi7wFDW00P+x60mu9yEVnWatxNIrLAe32+iLzvbWeLuBJRuJiCqzkjfae+KiIfe8d4g4h8o9X06SKy0tvueu+Ytd5GgojcLiKfisguEXlCXMm7zeMfIu6J4n5nEoPGXRx4X1rNex1wJfAD77173hvfT0T+KiJlIrJRRG4IWuZkEVnm7ctOEbnPm/SG97zfW9dnwh1bVDXuHsAm4PNhpj0C3BM0/B/AK97rM4HduOJgCvDfwBtB8yqu2A2wGPh60LSZwFve6yxgO/A9INUbnuRNmw380Xs9HDgAnAUkAz8A1gE9gvbjPaAfkAd8DHwzzH59E1gNDPDmXeTFm+RNfw74XyAD6OWt9xth1tUUozc8y9uHFOA3wMo2jv1iYCsw2tvWX4P2tz+wBzgP90fjLG+4MGjZzcAJuFJrIVAJXOIdn5uA+sBxb3XMCyLMe5y3vRRvvW8AvwmzDxnAFuCrXhwnep+LUd70x7y4T/amPwXMC7MuP/v89TDL3gL8CxiBS3zjgHzv/d0HfMXb/gxvOD9onetxn680b/jn3rTiVp+LTQR9V2j5+QzM+wdvPeOAWuD4UJ8Tb9wbwO9xn/vxQBlwZpj9mwc87R3v0bjPzVt+3oNW60n33vthQeOWAld4r88AxnjHfyywE7gwzPFoej+I/J06H5fgBFczUA1M8KadDJR773eC9zkYGWIbs3Df+SG46sFngSf9HP8Qx2EVcG7Q8HPA98LM+xjws6DhBGA5cCfQw4tnA3C2N/0d4Cve60zglFDHr83f5Y7+oEfzgfsCVAH7gx7XetM+D6wPmvcfwNXe64eBXwRNywTqgGJv2G+ymAG8Hya22TR/Ge8Anm71hm0Fzgjaj6uCpv8C+J8w632doEQCfCHwJuLab2qBtKDpM4BFkWIMMS3XW29OmOmL8X6YvOFRwCEgEfhh4IsQNH0hcE3QsncHTbsa+GfQsAClhE4Wbc4bIs4L23iPLgfebDXuf4G7gr5oDwVNOw9YHWZdfvY5XIxrgOkhxn8FeK/VuHeAmUHrvD1o2rdo/kNUTPuTRVHQ9Pdo/hFu8TnB/ag2AFlB4/4T117Yeh8Scd+tkUHj7g16P9t8D0Ks74/And7rYbjkkR5m3t8Avw5zPJreD9r4ToVZ73zgu0Gx/rqN70hgG68B3wqaNsI7LkmRjn+Yz9pT3us8XPLqG2bex2iZLCYBm1vN8yPgUe/1G8BPgIJW87Q4fm094rka6kJVzQ16/MEbvwhIF5FJ4hpnxuMyMLh/8J8GVqCqVbh/gf3bue0BuH92kbTeXiPu31Tw9nYEva4mfONkP2/ZgE+DXg/C/dve7lWF7Md9mHtFCtAriv/cK0ZX4H5cwP2TD6d1HMne/IOASwMxeHGcCvQNs2yLfVL36Qyejt95RaS3iMwTVwVXgftxCbcPg4BJreK8EugTNI/f98XPPocT7nPU4nPj+ZSOfW78aM9ncK+qVrYRV0Ah7gexrc9spPcg2J9wf4AAvgzMV9VqAO+7vsirXinHlRja+vwG70+4+BCRc0Xkn1412X7cn4bAejv0G+C9DvzBC/B7/P8IfFFEMoDLcMl2u48YwB3vfq2O921BcXwNV1JdLa7ac5rP9TaJ5wbukFS1QUSexn2wdgIvBH24t+EOGgDeQc/H/dtv7QCu+BsQ/CHeAlzhI5xtuOJxYHuC+5CF2l4k271lAwa2iqcW96+gzTMWQvgyMB1XItsE5OCqPNo6va91HHW4KoQtuH/ZYeuxcf9SAlrsU9DxCSXSvPd66x6jqntF5ELgd2HWtQVYoqpntRGnX372ua1lhwL/bjW+xefUMxB4pQPbaOtzHIm2Gt4G5IlIVtB3aiChP89luGrCAbiqnsC8Ae19D/4OFIrIeNx3+6agaX/CvdfnqmqNiPwGf8ki7HdKRFJwVaxXA/9PVetEZD7N34vAexdJ6/dyIO647ASKfCzfRFW3isg7wMW40ueDbc3eangLsFFVh4VZ9yfADBFJ8Nb/jIjkh1hPWPFcsmjLn3DF3Cu91wF/Br4qIuO9D8O9wLuquinEOlYCF4tIurhrL74WNO0FoK+I3CiuUT1LRCaFWMfTwPkiMlVEknFtHLXA2x3Yp6eBG0SkSNy1GrcGJnj/Lv4G/EpEsr1GtaEi8jkf683yYtqD+1E57NTfEK4SkVEikg7cDTyjqg00//M52yuxpIrIGSIS7kvxInCC11CXBNxA+B+zSPNm4aomy0WkP649IJwXgOEi8hURSfYeEyXo5IN2aO8+B3sI+Km4a1JERMZ6X9CXvPi+LCJJInI5rrrvhQ7EtxK4wtvHElybj187gWLvBwR1Z/W9Dfynt59jcd+Lw67Z8T4PzwKzve/QKOCaoFna9R6oah3wF+CXuCqYvwdNzsKVeGpE5GTcHyA/wn6ncPX6KXhJT9ypr18Imv4w7rdkqvd96y8iI0Ns48/ATSIyWEQycd+v/+vAn7qAJ3Btn2Nwxzecnbh2iYD3gEoR+aGIpHmf1dEiMhFARK4SkUKv9mO/t0wjbv8bW60rpHhOFs+La50PPAJVTajqu7h/VP0IOl1N3am2d+D+MWzH/TMIV0L4Na4ufifwOEHn+3v/qs4CvogrQn4CTGm9AlVdA1yFa0jf7c3/RVU91IH9/QOuLvwDYAWHf1Cuxn3AV+FKBs/gryrkCVzReKu37D99LPMkrk50B66h8wZo+jGZjiveluH+zdxCmM+Rqu4GLgV+jktWw3BtTB2Z9ye4ExfKcYkl7BfJe/++gHvvt3n78V+4H4d2ae8+t3If7gfrb0AF7gcoTVX3ANNwfy724H4cpnnHoL3uwH3O9+GO0Z/anr2Fv3jPe0Rkhfd6Bq4eexuuevcuDX8K+7dxVSo7cJ+XRwMTOvge/AlXAv5Lqx/bbwF3i0glrgH3aX+7F/475cV3g7eufbgEtCBo+nu4xvlf4z5zSzi8NAjuhJsncW0CG4Ea4Ds+4wvlOW87zwWq4cJ4GBjlVTnN95L3NFy1/Ebc79FDuJoEgHOAj0SkCvgtrt3koLeNe4B/eOs6JdwGxWvkMAZwpwXiGj0Pu0rdxJ6IDAHW4i6isi/vMUhE1uPOdIyr68ziuWRhjDncaOBTSxTHJhH5Eq4d4fVYx9LaUdfAbUx3JSI346qsjqSaw8Qpr1Q/Cnc9RNz1fGDVUMYYYyKyaihjjDERRbUaSkQewbXQ71LV0SGmC65l/jzcxSozVXVF6/laKygo0OLi4k6O1hhjjl3Lly/fraqFHV0+2m0Wj+EupnkizPRzcadIDsNdrv6g99ym4uJili1bFmk2Y4wxHhFp3WtAu0S1GkpV3wD2tjHLdOAJdf4J5IqIn2sHjDHGdKFYt1n0p2XfLaWE6cdJRK4T18XusrKysi4JzhhjjBPrZOGbqs5V1RJVLSks7HC1mzHGmA6I9XUWW2nZ0VcRHeuEj7q6OkpLS6mpqemUwEz0pKamUlRURHJycqxDMcb4FOtksQD4tojMwzVsl7ejS94WSktLycrKori4GDk275d+TFBV9uzZQ2lpKYMHD451OMYYn6J96uyfcXe5KhCRUuAu3L0RUNX/wfW+eR7uTlPVuI67OqSmpsYSxVFARMjPz8fanYw5ukQ1WajqjAjTFXdb1E5hieLoYO+TMUefWFdDGWNM/Ko7CBvfgB0fQkYhZPWFrD6Q1Q/S8yHhqDlH6IhZsugEe/bsYerUqQDs2LGDxMREAmdsvffee/To0SPsssuWLeOJJ57g/vvv9729wEWJBQV+bhZmjGmXiu3wyUJY8wpsWAz1B0PPl5AEmX1c8sjuG5RI+gY9+kBqDhwDpWlLFp0gPz+flStXAjB79mwyMzP5/ve/3zS9vr6epKTQh7qkpISSkpIuidMYE4IqbF8JaxfCmpfda4CcgTDhKzD8bBhwCtTsh8odULHNPVdu9563we5PXAmkpvzw9SeleQmlX1Ay6dMyoWT1hR7phy8bRyxZRMnMmTNJTU3l/fffZ/LkyVxxxRV897vfpaamhrS0NB599FFGjBjB4sWLmTNnDi+88AKzZ89m8+bNbNiwgc2bN3PjjTdyww03tLmd++67j0ceeQSAr3/969x4440cOHCAyy67jNLSUhoaGrjjjju4/PLLufXWW1mwYAFJSUl84QtfYM6cOV1xKIyJP4eqYeMSWPuKSxKV2wGBookw9U4Yfi70Or5liSAlE3Ii3E33UHVQEgl+9l5vex8qXgpdWknNCV86yerrSi+ZvSExNqecH5PJ4ifPf8SqbRWdus5R/bK564sntGuZ0tJS3n77bRITE6moqODNN98kKSmJV199ldtuu42//vWvhy2zevVqFi1aRGVlJSNGjOD6668Pez3C8uXLefTRR3n33XdRVSZNmsTnPvc5NmzYQL9+/XjxxRcBKC8vZ8+ePTz33HOsXr0aEWH//v0h12nMMatim0sMawPVSzXQIxOGngkjzoVhX4CMI6za7ZEO+UPdIxxVVwJpXToJHt74JlTtgMZWt/JO6wk/3HRkMXbQMZks4sWll15KYmIi4H6wr7nmGj755BNEhLq6upDLnH/++aSkpJCSkkKvXr3YuXMnRUWh/8289dZbXHTRRWRkZABw8cUX8+abb3LOOefwve99jx/+8IdMmzaN0047jfr6elJTU/na177GtGnTmDZtWnR22ph40djYXL209mXY/oEbnzsITprpqpcGTYakdt+a/ciIQFque/QaGX6+xkao3hNUMtkODaF/N7rCMZks2lsCiJbAjzjAHXfcwZQpU3juuefYtGkTZ5xxRshlUlKaP7iJiYnU19eHnK8tw4cPZ8WKFbz00kvcfvvtTJ06lTvvvJP33nuP1157jWeeeYbf/e53vP563N250Zgjc6jalRoC1UtVO0ASoOhk+PxsGH4OFI48OhqcExIgs9A9+o6NdTTHZrKIR+Xl5fTv7/pIfOyxxzplnaeddhozZ87k1ltvRVV57rnnePLJJ9m2bRt5eXlcddVV5Obm8tBDD1FVVUV1dTXnnXcekydPZsiQIZ0SgzExV761OTlsXOJVL2XBcVNd9dJxZ0FGfqyjPOpZsugiP/jBD7jmmmv42c9+xvnnn98p65wwYQIzZ87k5JNPBlwD94knnsjChQu55ZZbSEhIIDk5mQcffJDKykqmT59OTU0Nqsp9993XKTEY0+UaG2H7++7U1rUvw45/ufE9i+Gkr8KIc2DgZyEp/Cnrpv2Oyntwl5SUaOubH3388cccf/zxMYrItJe9X6ZdDh1w1UtrXoZP/gZVO1310oBTXNvDiHOhYPjRUb0UIyKyXFU7fJ6+lSyMMfFBFWoroXo3HNjjnvdvhk/+7q5haKiFlBxXvTT8HBh2FqTnxTrqbsOShTEmOlShtgIO7HaP6lbPLcbtcc8NtYevJ28ITPy6d/bSZ2N2nUF3Z8nCGOOPqruK+cAeOFDW6sffKwkcKAt6vRsaw5zq2SPT9a2UUeAuOOszxhsudOPSC1yjdKZ35bNVL8WcJQtjTLOD++HTt2Hz2+4ituASQPWewy8SC+iR5X7kMwrcVc79xnk/+AUuAQR+/APjktO6dr/MEbNkYUx3VlsJm//p2gQ2vekuXNNGSEyBnP7uh75nMRSd1PxD35QEvNfp+ZCcGus9MVFmycKY7uRQNWx5tzk5bF0B2gAJya5fpNNvgcGnQ/8SSwCmhe7TGXuUTZkyhYULF7YY95vf/Ibrr78+7DJnnHEGgVOAzzvvvJD9Nc2ePTtih3/z589n1apVTcN33nknr776anvCD2nx4sXWLcjRrq7G9TO06F545Fz4+UB48kL4x2/d9FNvhK/Mh1s3w6yXYcptUHyqJQpzGCtZdJIZM2Ywb948zj777KZx8+bN4xe/+IWv5V966aUOb3v+/PlMmzaNUaNGAXD33Xd3eF3mKFd/CLatcAli0xuw5T13RbMkQN9xcMr1ruQw8BRIyYp1tOYo4rtkISKfFZEvi8jVgUc0AzvaXHLJJbz44oscOnQIgE2bNrFt2zZOO+00rr/+ekpKSjjhhBO46667Qi5fXFzM7t27AbjnnnsYPnw4p556KmvWrGma5w9/+AMTJ05k3LhxfOlLX6K6upq3336bBQsWcMsttzB+/HjWr1/PzJkzeeaZZwB47bXXOPHEExkzZgyzZs2itra2aXt33XUXEyZMYMyYMaxevbrN/du7dy8XXnghY8eO5ZRTTuHDDz8EYMmSJYwfP57x48dz4oknUllZyfbt2zn99NMZP348o0eP5s033zyyg2vCa6iH0uXw1q/hyYvhvwbBI2fDop9B9T4omQVX/Bl+sBGuWwxf+Km7PsEShWknXyULEXkSGAqsBBq80Qo8EaW4jszLtzZ3AdBZ+oyBc38ednJeXh4nn3wyL7/8MtOnT2fevHlcdtlliAj33HMPeXl5NDQ0MHXqVD788EPGjg3dMdjy5cuZN28eK1eupL6+ngkTJnDSSScBrlfZa6+9FoDbb7+dhx9+mO985ztccMEFTJs2jUsuuaTFumpqapg5cyavvfYaw4cP5+qrr+bBBx/kxhtvBKCgoIAVK1bw+9//njlz5vDQQw+F3b+77rqLE088kfnz5/P6669z9dVXs3LlSubMmcMDDzzA5MmTqaqqIjU1lblz53L22Wfz4x//mIaGBqqrq9t1qE0bGhth579cyWHjG7D5HXctA7gO8sZfCYNPg0GnWn9IplP5rYYqAUbp0dg3SBcKVEUFksXDDz8MwNNPP83cuXOpr69n+/btrFq1KmyyePPNN7noootIT3d3zbrggguapv373//m9ttvZ//+/VRVVbWo8gplzZo1DB48mOHDhwNwzTXX8MADDzQli4svvhiAk046iWeffbbNdb311ltN998488wz2bNnDxUVFUyePJmbb76ZK6+8kosvvpiioiImTpzIrFmzqKur48ILL2T8+PGRDp0JRxV2fdzcIL3pLXetA0DeUBj9JZccik+DzF6xjdUc0/wmi38DfYDtUYyl87RRAoim6dOnc9NNN7FixQqqq6s56aST2LhxI3PmzGHp0qX07NmTmTNnUlNT06H1z5w5k/nz5zNu3Dgee+wxFi9efETxBrpD72hX6AC33nor559/Pi+99BKTJ09m4cKFnH766bzxxhu8+OKLzJw5k5tvvpmrr7ZaS19U3S06N73htTu85a5xAMgdCMdPg+LTXYLI7hfbWE234jdZFACrROQ9oOl6fFW9IPwi3U9mZiZTpkxh1qxZzJgxA4CKigoyMjLIyclh586dvPzyy2HvZQFw+umnM3PmTH70ox9RX1/P888/zze+8Q0AKisr6du3L3V1dTz11FNNXZ5nZWVRWVl52LpGjBjBpk2bWLduHccddxxPPvkkn/vc5zq0b6eddhpPPfUUd9xxB4sXL6agoIDs7GzWr1/PmDFjGDNmDEuXLmX16tWkpaVRVFTEtddeS21tLStWrOgeyaKxEQ5VumsXaiuhpsJ7XeE9Qo2vbDWtHOq8arvs/nDc55tLDj0HxXb/TLfmN1nMjmYQx5IZM2Zw0UUXMW/ePADGjRvHiSeeyMiRIxkwYACTJ09uc/kJEyZw+eWXM27cOHr16sXEiRObpv30pz9l0qRJFBYWMmnSpKYEccUVV3Dttddy//33NzVsA6SmpvLoo49y6aWXUl9fz8SJE/nmN7/Zof2aPXs2s2bNYuzYsaSnp/P4448D7vTgRYsWkZCQwAknnMC5557LvHnz+OUvf0lycjKZmZk88UR8Nm21UH/I/YNv+jEP9UPe1vhKlyj86JEFqdmukTkly917OWeAN5wNBcPcGUt5Q6ybCxM3fHdRLiK9gcAv13uquitqUUVgXZQf/WLyftVWwb6NsHcj7N3Q/HrfRigvdVcut6VHZvMPeuCHPiXww58dYlp2y6SQku3WkWCXN5muF7UuykVkoKpu9l5fBvwSWAwI8N8icouqPhNueWO6nKrrvyiQAPZuCHq9EQ60+n+Tng89B8OASTBuhuvQLuwPfxYkJMZmv4yJA21VQ00SkUtV9VfAj4GJgdKEiBQCrwKWLEzXamyEiq2hk8Heja2qgsTV++cNdt1b5w1xr3sOds+pOTHbDWOONmGThar+RUSu9AYTWlU77SEOuwpRVcTqeONexKrP+lp305vDksEG2P8pNBxqnjch2TX89hzsrkrOG9KcDHIHWbcVxnSSNhu4VfUp7+UrIrIQ+LM3fDnQ8f4poiA1NZU9e/aQn59vCSOOaWMje3aXkSr1sGGJuz1mxdaWSaG8FHfNpyc5wyWBXiPd7TODSwg5RVY9ZEwX8HU2lKreIiJfAgKn8sxV1eciLSci5wC/BRKBh1T1562mDwIeAQqBvcBVqlrajvibFBUVUVpaSllZWfiZDnjnq0uCO8sk8EzC4eMkwRsvdkaKH6qu99LGBtdQ3FjvPTd44xu953pSy9dTtOK/4FBQx4np+S4JDPxMUFWRlxQyCu09MCbGfJ8N1e4ViyQCa4GzgFJgKTBDVVcFzfMX4AVVfVxEzgS+qqpfibTuUGdD+fLIuXBwX/Mpj7UVtPgHG05SanMjZ4/Mwxs+g892ScmClMwQ47IgOf3o+tFrbHQNxlU7vccuqNrhPe9sfq7cCbXlIVYg7p4Hmb3d1cWZvYMeQcNZfVzDsjEmaqJ2NpS38rdU9VQRqaTlr6oAqqptfcNPBtap6gZvXfOA6cCqoHlGATd7rxcB89sZf/vMernlsCocOhCUPILOnz9Udfj59MGPitKg4arQ9w5uTRK8hJPl7iOcmOzq3BOTvOdkSEgKGp/sqljCTks6fB1N40JMaz0M7jaYTUlgZ6tEsMuVBlrrkdn8Y9/reBgypVUy6OUSQHqB25Yx5qgXqc3iVO+5I11U9ge2BA2XApNazfMBcDGuquoiIEtE8lV1T+uVich1wHUAAwcO7EA4IYh4pYBMoO+Rrau+1iWNkMklaFwgCTUcgoY6V13TUOfuVRwYrjvoDde758b65tdN8wYN+ykdRTwWid4Pvvej32fs4aWArN6Q0cs7XsaY7sRvr7OnAB+paqU3nIXrWPDdI9z+94HfichM4A1gK8292ragqnOBueCqoY5wu50vKcU9YtHTZ2NDcxI5LLGESEaBZ9S1B2T2hrQ8u1jMGBOW3zqCB4EJQcMHQoxrbSswIGi4yBvXRFW34UoWiEgm8CVVPfx2caZtCYneGUF2mqgxJjr8/pWU4O7JVbWRyIlmKTBMRAaLSA/gCmBBi5WKFIhIIIYf4c6MMsYYE2f8JosNInKDiCR7j+8CG9paQFXrgW8DC4GPgadV9SMRuVtEAr3VngGsEZG1QG/gng7thTHGmKjydeqsiPQC7gfOxLWmvgbcGKvOBDt86qwxxnRTUT11NsBLCld0dCPGGGOObn7PhkoFvgacQFArqqrOilJcxhhj4ojfNosncbdVPRtYgjuzyeedXowxxhzt/CaL41T1DuCAqj4OnM/hF9gZY4w5RvlNFnXe834RGQ3kAL2iE5Ixxph44/eivLki0hO4A3etRCZwZ9SiMsYYE1f8ng31kPdyCTAkeuEYY4yJR5F6nb25remqel/nhmOMMSYeRSpZBHqbHQFMpLm7ji8C70UrKGOMMfElUhflPwEQkTeACUG9zs4GXox6dMYYY+KC37OhegOHgoYPeeOMMcZ0A37PhnoCeE9EAvfdvhB4LCoRGWOMiTt+z4a6R0ReBk7zRn1VVd+PXljGGGPiSaSzobJVtUJE8oBN3iMwLU9V90Y3PGOMMfEgUsniT8A0YDktb/Qs3rBdc2GMMd1ApLOhpnnPg7smHGOMMfEoUjVUW/fYRlVXdG44xhhj4lGkaqhftTFNcXfOM8YYc4yLVA01pasCMcYYE7/8XmeB1zX5KFreKe+JaARljDEmvvi9rTwj6NQAABxhSURBVOpdwBm4ZPEScC7wFu5iPWOMMcc4v919XAJMBXao6leBcbgbIBljjOkG/CaLg6raCNSLSDawCxgQvbCMMcbEE79tFstEJBf4A+4CvSrgnahFZYwxJq747RvqW97L/xGRV4BsVf0wemEZY4yJJ76qoURkgYh8WUQyVHWTJQpjjOle/LZZ/Ao4FVglIs+IyCUikhppIWOMMccGv9VQS4AlIpKIu2r7WuARIDuKsRljjIkT7bkoLw137+3LgQnA49EKyhhjTHzx22bxNPAxrlTxO2Coqn7Hx3LniMgaEVknIreGmD5QRBaJyPsi8qGInNfeHTDGGBN9fksWDwMzVLXB74q9KqsHgLOAUmCpiCxQ1VVBs90OPK2qD4pI4OrwYr/bMMYY0zV8lSxUdWF7EoXnZGCdqm5Q1UPAPGB661XT3O6RA2xr5zaMMcZ0Ab9nQ3VEf2BL0HCpNy7YbOAqESnFlSrCVm2JyHUiskxElpWVlXV2rMYYY9oQzWThxwzgMVUtAs4DnhSRkDGp6lxVLVHVksLCwi4N0hhjuju/DdwiIleJyJ3e8EAROTnCYltp2X9UkTcu2NeApwFU9R1c9+cFfmIyxhjTdfyWLH4PfAZXEgCoxDVet2UpMExEBotID+AKYEGreTbjerNFRI7HJQurYzLGmDjj92yoSao6QUTeB1DVfV4CCEtV60Xk28BCIBF4RFU/EpG7gWWqugD4HvAHEbkJ19g9U1W1w3tjjDEmKvwmizrvVFgFEJFCoDHSQqr6Eq7hOnjcnUGvVwGTfUdrjDEmJvxWQ90PPAf0EpF7cHfJuzdqURljjIkrfvuGekpEluPaFwS4UFU/jmpkxhhj4kabyUJE8oIGdwF/Dp6mqnujFZgxxpj4EalksRzXTiHAQGCf9zoXdybT4KhGZ4wxJi602WahqoNVdQjwKvBFVS1Q1XxgGvC3rgjQGGNM7Plt4D7FO7MJAFV9GfhsdEIyxhgTb/yeOrtNRG4H/ugNX4l1+meMMd2G35LFDKAQd/rss97rGW0uYYwx5pjh99TZvcB3oxyLMcaYOBXrXmeNMcYcBSxZGGOMiciShTHGmIj83s+iSESeE5EyEdklIn8VkaJoB2eMMSY++C1ZPIq7F0VfoB/wvDfOGGNMN+A3WRSq6qOqWu89HsOdPmuMMaYb8Jss9ni3VU30HlcBe6IZmDHGmPjhN1nMAi4DdgDbgUuAmVGKyRhjTJzx291HkapeEDxCRCYDWzo/JGOMMfHGb8niv32OM8YYcwyKdPOjz+B6ly0UkZuDJmUDidEMzBhjTPyIVA3VA8j05ssKGl+Ba7cwxhjTDbSZLFR1CbBERB5T1U+7KCZjjDFxxm8Dd4qIzAWKg5dR1TOjEZQxxpj44jdZ/AX4H+AhoCF64RhjjIlHfpNFvao+GNVIjDHGxC2/p84+LyLfEpG+IpIXeEQ1MmOMMXHDb8niGu/5lqBxCgzp3HCMMcbEI7+3VR0c7UCMMcbEL1/JQkSuDjVeVZ+IsNw5wG9xF/A9pKo/bzX918AUbzAd6KWquX5iMsYY03X8VkNNDHqdCkwFVgBhk4WIJAIPAGcBpcBSEVmgqqsC86jqTUHzfwc40X/oxhhjuorfaqjvBA+LSC4wL8JiJwPrVHWDt8w8YDqwKsz8M4C7/MRjjDGma3X0HtwHgEjtGP1p2SttqTfuMCIyyFvf6x2MxxhjTBT5bbN4Hnf2E7gEMwp4uhPjuAJ4RlXDXvAnItcB1wEMHDiwEzdtjDEmEr9tFnOCXtcDn6pqaYRltgIDgoaLvHGhXAH8R1srU9W5wFyAkpISbWteY4wxnctvm8WSDqx7KTBMRAbjksQVwJdbzyQiI4GewDsd2IYxxpgu0GabhYi85T1XikhFiMdGEflWqGVVtR74NrAQ+Bh4WlU/EpG7RST4rntXAPNU1UoLxhgTp+RIfqNFJB94W1VHdF5IkZWUlOiyZcu6cpPGGHNUE5HlqlrS0eX9tlkErpvoTcsuyjeLyBkd3bgxxpijg9+zob6DuwZiJ9DojVZgrKpuj1Jsxhhj4oTfksV3gRGquieawRhjjIlPfi/K2wKURzMQY4wx8ctvyWIDsFhEXgRqAyNV9b6oRGWMMSau+E0Wm71HD+9hjDGmG/F7Ud5PAEQk0xuuimZQxhhj4ouvNgsRGS0i7wMfAR+JyHIROSG6oRljjIkXfhu45wI3q+ogVR0EfA/4Q/TCMsYYE0/8JosMVV0UGFDVxUBGVCIyxhgTd3yfDSUidwBPesNX4c6QMsYY0w34LVnMAgqBZ4G/AgXeOGOMMd2A37Oh9gE3RDkWY4wxccrv2VB/9+67HRjuKSILoxeWMcaYeOK3GqpAVfcHBrySRq/ohGSMMSbe+E0WjSLSdONrERlE8z25jTHGHOP8ng31Y+AtEVkCCHAacF3UojLGGBNX/DZwvyIiE4BTvFE3quru6IVljDEmnvi+U56XHF6IYizGGGPilN82C2OMMd2YJQtjjDERtVkNJSJ5bU1X1b2dG44xxph4FKnNYjnuFFkJMU2BIZ0ekTHGmLjTZrJQ1cFdFYgxxpj45be7DxGRq7yeZxGRgSJycnRDM8YYEy/8NnD/HvgM8GVvuBJ4ICoRGWOMiTt+r7OYpKoTvFuroqr7RKRHFOMyxhgTR/yWLOpEJBGvPygRKQQaoxaVMcaYuOI3WdwPPAf0EpF7gLeAe6MWlTHGmLjiK1mo6lPAD4D/BLYDF6rqXyItJyLniMgaEVknIreGmecyEVklIh+JyJ/aE7wxxpiu0Z6+oVYDq/3O71VbPQCcBZQCS0VkgaquCppnGPAjYLLXDmL3yDDGmDgUze4+TgbWqeoGVT0EzAOmt5rnWuAB72ZKqOquKMZjjDGmg6KZLPoDW4KGS71xwYYDw0XkHyLyTxE5J9zKROQ6EVkmIsvKyso6FFBNXUOHljPGmO7OdzVUFLc/DDgDKALeEJExwbdwDVDVucBcgJKSknbfpU9VmTJnMX1zUjlzZC+mjOzFqL7ZiITqycQYY0wwX8lCRC4G/gt3323xHqqq2W0sthUYEDRc5I0LVgq8q6p1wEYRWYtLHkv9he9fbX0jl5UMYNGaXcz521rm/G0tfbJTmTKykCkjenHqsALSe8Q6dxpjTHwS1ch/0kVkHfBFVf3Y94pFkoC1wFRcklgKfFlVPwqa5xxghqpeIyIFwPvAeFXd09a6S0pKdNmyZX5DOcyuyhoWrylj0epdvPnJbqpq6+mRmMCkIXlMHdmLM0f2ZmB+eofXb4wx8UZElqtqSYeX95ks/qGqk9u9cpHzgN8AicAjqnqPiNwNLFPVBeLqgH4FnAM0APeo6rxI6z3SZBHsUH0jyzbt5fXVu3h9zS42lB0AYGhhRlN11cTiPJIT7dYfxpijV1cli98CfYD5QG1gvKo+29ENH4nOTBatbdp9gEVrdvH66l28u2EvhxoayUpJ4rThBUwZ0YszRvSiMCslKts2xpho6apk8WiI0aqqszq64SMRzWQR7EBtPf9Yt5vXV+9i0Zpd7KxweXJcUQ5TRvbizJG9GN0vh4QEayQ3xsS3LkkW8aarkkUwVeWjbRUs8qqrVm7ZjyoUZKYwZUQhZ450jeRZqcldGpcxxvjRVSWL4cCDQG9VHS0iY4ELVPVnHd3wkYhFsmhtT1UtS9aW8frqXbyxtoyKmnqSE4WJxXlNbR1DCjLs1FxjTFzoqmSxBLgF+F9VPdEb929VHd3RDR+JeEgWweobGln+6T5eX7OLRat3sXZnFQCD8tOZMsJVV00akkdKUmKMIzXGdFddlSyWqupEEXk/KFmsVNXxHd3wkYi3ZNHalr3VLPYayd9ev4fa+kbSeyQy+bgCV+oY0Ys+OamxDtMY040cabLwexXabhEZSvP9LC7B9T5rQhiQl85XPlPMVz5TzMFDDbyzwWskX13G31ftBGBgXjpji3IYPyCXcQNyGd0vh7QeVvIwxsQnv8niP3BdbYwUka3ARuDKqEV1DEnrkciZI3tz5sjeqCprd1axZK1rIH9/835e+NDl3MQEYXjvLMYPyGFcUS5ji3IZ3juTJLu+wxgTB9p1NpSIZAAJqloZvZAii/dqqPbYVVnDh1vK+aB0Px+UlvPBlv2UH6wDIDU5gTH9XfIYNyCXcUW5DMhLs0ZzY0y72amzxxhV5dM91XxQup+VW/bzwZb9fLStgtp6dxfbnunJTYlj/IBcxhblkJ9pFwkaY9rWVW0WpouICMUFGRQXZDB9vOvRva6hkTU7Kl3pY8t+PthSzpK1nxDI8wPy0hhblMt4rwQyun+2dYpojOlUEUsWIpIAnKKqb3dNSJEdyyULvw7U1vPvrV711ZZyVm7Zz9b9BwFIELz2D9f2MW5ADiN6Z1n7hzHdWFedOtt0ymw8sGQRWlllLR8GtX18ULqf/dXN7R+j++Uwzqu6Gj8gl4F56db+YUw30VXVUK+JyJeAZ/VobOToJgqzUph6fG+mHt8bcO0fm/dWe20f5XxYup8//vPTpvaPnLRkhhZmMKQwk8EFGQwpyGBwYQbF+RmkJttpvMaYZn5LFpVABq4b8YP4u/lR1FjJouPqGhpZu7OSD7aU8+9t5Wwoq2Lj7gNNnSQCiEC/nDSGFHoJpCCDwYWZDCnIoF9uGonWcaIxR50uKVmoalZHN2DiS3JiAif0y+GEfjktxlfV1rNp9wE27D7AxrIDbNjtkshfV2ylqra+ab4eSQkU56czpCCTwYUZTSWSIYWZ9ExPtmotY45Rvk+ZEZELgNO9wcWq+kJ0QjKxkJmSxOj+OYzu3zKJqCq7qw41lUA27j7A+rIDfLKrktdW76SuoblkmpOW7JJHU4nEVW8NLsiwq9ONOcr5vQf3z4GJwFPeqO+KyGRV/VHUIjNxQUQozEqhMCuFSUPyW0yrb2hk6/6DbCjzSiS7q9hQdoB31u/h2RUtb7feLyeVwYUZrkTitY0MLcikf0+r1jLmaOC3zeJD3L2xG73hROB9VR0b5fhCsjaL+Fd9qJ5Nu6tddVaZVyLZfYANZVVU1gRVayUmMLgggzFFOd7FhjmM7JNNjyQ7zdeYztSVF+XlAnu91zltzWhMeo8kRvXLZlS/ludAqCp7Dxxi4+4DTSWSNTvcTaWeWV4KuARyfL9sxhfleNeJ5DKkIMPuSGhMDPlNFvcC74vIItyZUKcDt0YtKnPMEhHyM1PIz0yhpDivabyqsnX/waZTfFdu2c8zy0t5/J1PAchKSWKMlzzGD3DPfXNSrUHdmC4SMVl4V3A3Aqfg2i0AfqiqO6IZmOleRISinukU9Uzn/LF9AWhoVNaXVTVdYPhhaTkPv7WhqVG9IDOlKXEEqrBy03vEcjeMOWb5bbNYdiR1XZ3N2iy6r9r6Bj7eXtkigawvq2rqJ2tQfrpLHl4biN0nxBinq9osXhWR7wP/BxwIjFTVveEXMabzpSQlMn6A63E3oLKmjn9tLW+qwlq+aS/Pf7ANcPcJGdYrs0U/WcN7Z5Fs/WQZ0y5+SxYbQ4xWVR3S+SFFZiULE0ngPiEflu5nZal7DvSTlZKUwOj+OU19ZI0tyqU43/rJMse2qHck6LVZXKqq/9fRjXQ2SxamvQL9ZAU6WfywdD//2lpOTZ3rJys71V2UOKJPFiP7ZDGiTzbDe2daV+/mmNFVvc5am4U55tQ3NPLJrkADejmrtpWzdmcVB+saANdH1sC8dEb0zmJEn6ymRFKcn2HdvZujjrVZGNNBSYkJHN83m+P7ZnPFyW5cY6MrgazZWcmaHe6xekcFr368k0bvf1WPpASOK8z0SiDNjz7ZdiqvOXZZm4UxPtTUNbBuV5VLIDsrWb2jkjU7Klr01puTlnxYKWR4nyyyU5NjGLkxTlf1Oju4oxsw5liQmpwYsqPF/dWHWiWQSua/v5XKoJ56++emMaJPFsN7ZzWVRoYWZlqXJuao0mayEJEfqOovvNeXqupfgqbdq6q3RVj+HOC3QCLwkKr+vNX0mcAvgUCvc79T1YfavRfGxEhueg8mDclv0cli4Gp0V4XVXJ31xtoy6r26rKQEYUhhBiP6ZLsSiJdI+uemWbcmJi61WQ0lIitUdULr16GGQyybCKwFzgJKgaXADFVdFTTPTKBEVb/dnqCtGsocjQ7VN7Jx9wFW76gIag+pbLp3Oriu4gfkpdM7O4U+2an0yk5tet3be+Rn9LCEYtot2tVQEuZ1qOHWTgbWqeoGABGZB0wHVrW5lDHHqB5JCU3tGcEqa+pYu7OSNTuqWLOjgtJ9B9lZWcNH2yrYXVVL6/9zSQmu2/jeXiLpHZRI+njjemWnkp2aZA3uptNEShYa5nWo4db6A1uChkuBSSHm+5KInI4rhdykqltCzIOIXAdcBzBw4MAImzbm6JGVmsxJg/I4aVDeYdPqGhrZXVXLjvIadlbUsquypsXrwP1DKoK6fQ9IS05sShx9wiSWXtkpdr9140ukZDFORCpwpYg07zXecGonbP954M+qWisi3wAeB84MNaOqzgXmgquG6oRtGxP3khMT6JuTRt+ctDbnO3iogZ0VNeysqGFHRQ27KmpbvP6gdD87ymuorW88bNmctOSmxNFc3ZVCcUEGw3pl0Ts7xUoopu1koapH8pdjKzAgaLiI5obswPr3BA0+BPziCLZnTLeV1iOR4oIMigsyws6jqlQcrGeHl1SaH7VNrz/ZWUVZVS0Njc3/x7JSkhjaK5PjemUyrOk5i6Ke1hjfnUSzL4OlwDARGYxLElcAXw6eQUT6qup2b/AC4OMoxmNMtyYi5KQnk5OefFi7SbCGRqWsspYNu6tYv6uKT3ZVsW5XFUvWljXdoApcH1tDC1slkd6ZDMrPsI4aj0FRSxaqWi8i3wYW4k6dfURVPxKRu4FlqroAuEFELgDqcXfhmxmteIwx/iQmCH1yUumTk8pnhxa0mFZeXce6skrW7arik51VrCurYvmn+1jg9fILrgG+uCCD4wpd8jjOSyRDCzOtfeQo5usK7nhjp84aE18O1NazoewA68oqXRLxSiOf7q1uqtISgQE905tKIkODSiRZdpV71HXlPbiNMSakDO+2t2OKWl7hXlvfwKbd1a4ksquyKYm89cluDjU0N7b3yU5tKoEEV2vlZ6Z09a6YMCxZGGOiJiUpMejakr5N4+sbGtmy7+BhSeTpZVuoPtTQNF/P9GQG5mcwoGcaRT3TGZCXxoCe6QzIS6d/bpp1mdKFLFkYY7pcUmICgwsyGFyQwVmjejeNb2xUtlfUeG0ilawvO8CWvdX8a2s5Cz/a0XT/dXDVWn2yUxnQM50iL4kU9UxjQJ5LJn2yU0m0s7U6jSULY0zcSEgQ+uem0T83jc8NL2wxraFR2VlRw5a91WzZd9B7rqZ070H+uX4Pz1VsbXG1e3Ki0C83UBJxJZOmZNIznYLMHnb9SDtYsjDGHBUSE9yPf7/ctJBdQRyqb2Tb/oNs2VfNlr2B52pK9x3k76t2srvqUIv505ITg5JHczVXkVfNlZNmje7BLFkYY44JPZIS2rwwsfpQPaWBEklQ6aR030GWbtpLZasuU7JTk5pKIUU90+ibm0Zf75Tivjmp9MrqXtVcliyMMd1Ceo8khvd23cGHUl5d11QaCZROSvdVs66sisVrdzXdrz0gMUHolZXSlDz6ZKfRL7c5mfTJSaN3VsoxcwteSxbGGAPe1e2H3+AKXFcp5Qfr2F7uOnLcXl7D9vKDTcNrdlSyeE1ZizO5ABIECrNS6JOTRt9sl0hcQvFKKV5fXEfDWV2WLIwxJgIRITe9B7npPTi+b3bIeVSVipp6L5kcZEd5DdvKa9jhJZX1ZVX8Y93uFndRdOuGgsyUpuQRKJX0yw0Mp9E7J4WUpNhe/W7JwhhjOoGIkJOW7O7F3kbfW5U1dU2lk9allE/3VPPPDaG7nC/I7MGQgkye/uZnorkbYVmyMMaYLpSVmkxWajLDwrSdgOs+pTmZuFLK9ooaYtk9kyULY4yJMxkpSU1dn8SL+G9VMcYYE3OWLIwxxkRkycIYY0xEliyMMcZEZMnCGGNMRJYsjDHGRGTJwhhjTESWLIwxxkQksbwisKNEpAz4tIOLFwC7OzGco5kdi5bseLRkx6PZsXAsBqlqYeTZQjsqk8WREJFlqloS6zjigR2Llux4tGTHo5kdC6uGMsYY44MlC2OMMRF1x2QxN9YBxBE7Fi3Z8WjJjkezbn8sul2bhTHGmPbrjiULY4wx7WTJwhhjTETdJlmIyDkiskZE1onIrbGOJ5ZEZICILBKRVSLykYh8N9YxxZqIJIrI+yLyQqxjiTURyRWRZ0RktYh8LCKxuY9nnBCRm7zvyb9F5M8ikhrrmGKhWyQLEUkEHgDOBUYBM0RkVGyjiql64HuqOgo4BfiPbn48AL4LfBzrIOLEb4FXVHUkMI5ufFxEpD9wA1CiqqOBROCK2EYVG90iWQAnA+tUdYOqHgLmAdNjHFPMqOp2VV3hva7E/Rj0j21UsSMiRcD5wEOxjiXWRCQHOB14GEBVD6nq/thGFXNJQJqIJAHpwLYYxxMT3SVZ9Ae2BA2X0o1/HIOJSDFwIvBubCOJqd8APwAaYx1IHBgMlAGPetVyD4lIRqyDihVV3QrMATYD24FyVf1bbKOKje6SLEwIIpIJ/BW4UVUrYh1PLIjINGCXqi6PdSxxIgmYADyoqicCB4Bu28YnIj1xtRCDgX5AhohcFduoYqO7JIutwICg4SJvXLclIsm4RPGUqj4b63hiaDJwgYhswlVPnikif4xtSDFVCpSqaqCk+QwueXRXnwc2qmqZqtYBzwKfjXFMMdFdksVSYJiIDBaRHrgGqgUxjilmRERwddIfq+p9sY4nllT1R6papKrFuM/F66raLf85AqjqDmCLiIzwRk0FVsUwpFjbDJwiIune92Yq3bTBPynWAXQFVa0XkW8DC3FnMzyiqh/FOKxYmgx8BfiXiKz0xt2mqi/FMCYTP74DPOX9sdoAfDXG8cSMqr4rIs8AK3BnEb5PN+36w7r7MMYYE1F3qYYyxhhzBCxZGGOMiciShTHGmIgsWRhjjInIkoUxxpiILFkYE4aIJIjIKyIyMNaxGBNrduqsMWGIyFCgSFWXxDoWY2LNkoUxIYhIA/CvoFHzVPXnsYrHmFizZGFMCCJSpaqZsY7DmHhhbRbGtIOIbBKRX4jIv0TkPRE5zhtfLCKvi8iHIvJaoJ1DRHqLyHMi8oH3+Kw3fr6ILPfuwHZdLPfJGD8sWRgTWpqIrAx6XB40rVxVxwC/w90LA+C/gcdVdSzwFHC/N/5+YImqjsP13hrok2yWqp4ElAA3iEh+tHfImCNh1VDGhBCuGsrryvxMVd3gdfO+Q1XzRWQ30FdV67zx21W1QETKcI3kta3WMxu4yBssBs5W1X9GcZeMOSLdotdZYzqZhnnti4icgbtPwmdUtVpEFgOpnROaMdFh1VDGtN/lQc/veK/fxt0PA+BK4E3v9WvA9QAikujd4zoH2OclipHAKV0StTFHwKqhjAkhxKmzr6jqrV411P8B5wK1wAxVXScig4BHgQLcPay/qqqbRaQ37v4HQ4AGXOJYAczHVT+tAXKB2aq6uAt2zZgOsWRhTDt4yaJEVXfHOhZjupJVQxljjInIShbGGGMispKFMcaYiCxZGGOMiciShTHGmIgsWRhjjInIkoUxxpiI/j9SmzeuYfrMJQAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "En este caso, a pesar de que el modelo puede realizar predicciones, el error en el conjunto de validación no parece disminuir. Esto se puede deber a que pudo haber sucedido un error de implementación o no se entrenó por el suficiente número de épocas. Por lo tanto, no se escoge este modelo para seguir experimentando."
      ],
      "metadata": {
        "id": "8MtNGEJWh-2E"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "---------------\n",
        "\n",
        "\n",
        "### Modelo FINAL"
      ],
      "metadata": {
        "id": "HsKcEMgFj7Mh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Este modelo final se contruye en base a los resultados de los experimentos anteriores y consta en lo siguiente\n",
        "\n",
        "\n",
        "*   Capa GRU (Envez de LSTM)\n",
        "*   Bidireccionalidad\n",
        "*   Embeddings pre-entrenados de zenodo\n",
        "*   4 Epocas de entrenamiento\n",
        "*   HIDDEN DIM igual a 64\n",
        "*   Numero de capas ocultas igual a 2\n",
        "\n"
      ],
      "metadata": {
        "id": "0AYu3cXF4ihc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!wget https://zenodo.org/record/3924799/files/cwlce.vec -nc\n",
        "\n",
        "# Primer Field: TEXT. Representan los tokens de la secuencia\n",
        "# Esta vez se convierten a minusculas los caracteres\n",
        "# esto para facilitar la correspondencia de palabras al vector de embedding\n",
        "# independiente del uso de mayusculas.\n",
        "TEXT2 = legacy.data.Field(lower=True) \n",
        "\n",
        "# Segundo Field: NER_TAGS. Representan los Tags asociados a cada palabra.\n",
        "NER_TAGS2 = legacy.data.Field(unk_token=None)\n",
        "fields2 = ((\"text\", TEXT2), (\"nertags\", NER_TAGS2))\n",
        "\n",
        "train_data, valid_data, test_data = legacy.datasets.SequenceTaggingDataset.splits(\n",
        "    path=\"./\",\n",
        "    train=\"train.txt\",\n",
        "    validation=\"dev.txt\",\n",
        "    test=\"test.txt\",\n",
        "    fields=fields2,\n",
        "    encoding=\"utf-8\",\n",
        "    separator=\" \"\n",
        ")\n",
        "\n",
        "TEXT2.build_vocab(train_data)\n",
        "NER_TAGS2.build_vocab(train_data)\n",
        "\n",
        "print(f\"Tokens únicos en TEXT: {len(TEXT2.vocab)}\")\n",
        "print(f\"Tokens únicos en NER_TAGS: {len(TEXT.vocab)}\")\n",
        "\n",
        "!pip install unidecode\n",
        "\n",
        "import unidecode\n",
        "from torchtext import vocab\n",
        "#Se remueven los acentos de los tokens del vocabulario\n",
        "TEXT2.vocab.itos = [unidecode.unidecode(x) for x in TEXT2.vocab.itos]\n",
        "\n",
        "popi = vocab.Vectors('cwlce.vec') #vectores entrenados en diagnosticos de lista de espera.\n",
        "taba = popi.get_vecs_by_tokens(TEXT2.vocab.itos) #se hace un match entre las palabras del vocabulario que estén presentes en los embeddings"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nFMBYJWV_nBJ",
        "outputId": "331e84e3-4b7b-4633-b6cf-f343272a42a7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "File ‘cwlce.vec’ already there; not retrieving.\n",
            "\n",
            "Tokens únicos en TEXT: 12772\n",
            "Tokens únicos en NER_TAGS: 17591\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: unidecode in /usr/local/lib/python3.7/dist-packages (1.3.4)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# tamaño del vocabulario. recuerden que la entrada son vectores bag of word(one-hot).\n",
        "INPUT_DIM = len(TEXT2.vocab)\n",
        "EMBEDDING_DIM = popi.dim  # dimensión de los embeddings.\n",
        "HIDDEN_DIM = 64 # dimensión de la capas GRU\n",
        "OUTPUT_DIM = len(NER_TAGS2.vocab)  # número de clases\n",
        "\n",
        "N_LAYERS = 2  # número de capas.\n",
        "DROPOUT = 0.5\n",
        "BIDIRECTIONAL = True\n",
        "\n",
        "model_final = NER_RNN_GRU(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM,\n",
        "                         N_LAYERS, BIDIRECTIONAL, DROPOUT, PAD_IDX)\n",
        "model_name_final = 'FINAL_MODEL'\n",
        "n_epochs_final = 5\n",
        "# loss_3 = ..."
      ],
      "metadata": {
        "id": "Kmyd8AmybR2e"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Experimento final"
      ],
      "metadata": {
        "id": "m2Awro0KAT_P"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8o94lqhTv20H"
      },
      "outputs": [],
      "source": [
        "model = model_final\n",
        "model_name = model_name_final\n",
        "criterion = baseline_criterion\n",
        "n_epochs = n_epochs_final"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "krA5nVl8PyHv"
      },
      "outputs": [],
      "source": [
        "# Enviamos el modelo y la loss a cuda (en el caso en que esté disponible)\n",
        "model = model.to(device)\n",
        "criterion = criterion.to(device)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#exp_baseline_model = copy.deepcopy(baseline_model)\n",
        "#exp_emb_model = copy.deepcopy(emb_model)\n",
        "\n",
        "models = [[baseline_model, baseline_model_name], [model_final, model_final]]\n",
        "results = []\n",
        "\n",
        "for a_model in models:\n",
        "\n",
        "    model = a_model[0]\n",
        "    model_name = a_model[1]\n",
        "    criterion = baseline_criterion\n",
        "    n_epochs = baseline_n_epochs\n",
        "\n",
        "    def init_weights(m):\n",
        "        # Inicializamos los pesos como aleatorios\n",
        "        if model_name == 'embeddingsClinicos':\n",
        "          model.embedding.from_pretrained(taba, False) \n",
        "        else: \n",
        "          for name, param in m.named_parameters():\n",
        "            nn.init.normal_(param.data, mean=0, std=0.1) \n",
        "            \n",
        "        # Seteamos como 0 los embeddings de UNK y PAD.\n",
        "        model.embedding.weight.data[UNK_IDX] = torch.zeros(model.embedding.embedding_dim)\n",
        "        model.embedding.weight.data[PAD_IDX] = torch.zeros(model.embedding.embedding_dim)\n",
        "            \n",
        "    model.apply(init_weights)\n",
        "\n",
        "    \n",
        "\n",
        "    def count_parameters(model):\n",
        "        return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
        "\n",
        "    # Optimizador\n",
        "    optimizer = optim.Adam(model.parameters())\n",
        "\n",
        "    # Enviamos el modelo y la loss a cuda (en el caso en que esté disponible)\n",
        "    model = model.to(device)\n",
        "    criterion = criterion.to(device)\n",
        "\n",
        "    def train(model, iterator, optimizer, criterion):\n",
        "\n",
        "        epoch_loss = 0\n",
        "        epoch_precision = 0\n",
        "        epoch_recall = 0\n",
        "        epoch_f1 = 0\n",
        "\n",
        "        model.train()\n",
        "\n",
        "        # Por cada batch del iterador de la época:\n",
        "        for batch in iterator:\n",
        "\n",
        "            # Extraemos el texto y los tags del batch que estamos procesado\n",
        "            text = batch.text\n",
        "            tags = batch.nertags\n",
        "\n",
        "            # Reiniciamos los gradientes calculados en la iteración anterior\n",
        "            optimizer.zero_grad()\n",
        "\n",
        "            #text = [sent len, batch size]\n",
        "\n",
        "            # Predecimos los tags del texto del batch.\n",
        "            predictions = model(text)\n",
        "\n",
        "            #predictions = [sent len, batch size, output dim]\n",
        "            #tags = [sent len, batch size]\n",
        "\n",
        "            # Reordenamos los datos para calcular la loss\n",
        "            predictions = predictions.view(-1, predictions.shape[-1])\n",
        "            tags = tags.view(-1)\n",
        "\n",
        "            #predictions = [sent len * batch size, output dim]\n",
        "\n",
        "\n",
        "\n",
        "            # Calculamos el Cross Entropy de las predicciones con respecto a las etiquetas reales\n",
        "            loss = criterion(predictions, tags)\n",
        "            \n",
        "            # Calculamos el accuracy\n",
        "            precision, recall, f1 = calculate_metrics(predictions, tags)\n",
        "\n",
        "            # Calculamos los gradientes\n",
        "            loss.backward()\n",
        "\n",
        "            # Actualizamos los parámetros de la red\n",
        "            optimizer.step()\n",
        "\n",
        "            # Actualizamos el loss y las métricas\n",
        "            epoch_loss += loss.item()\n",
        "            epoch_precision += precision\n",
        "            epoch_recall += recall\n",
        "            epoch_f1 += f1\n",
        "\n",
        "        return epoch_loss / len(iterator), epoch_precision / len(\n",
        "            iterator), epoch_recall / len(iterator), epoch_f1 / len(iterator)\n",
        "\n",
        "    def evaluate(model, iterator, criterion):\n",
        "\n",
        "        epoch_loss = 0\n",
        "        epoch_precision = 0\n",
        "        epoch_recall = 0\n",
        "        epoch_f1 = 0\n",
        "\n",
        "        model.eval()\n",
        "\n",
        "        # Indicamos que ahora no guardaremos los gradientes\n",
        "        with torch.no_grad():\n",
        "            # Por cada batch\n",
        "            for batch in iterator:\n",
        "\n",
        "                text = batch.text\n",
        "                tags = batch.nertags\n",
        "\n",
        "                # Predecimos\n",
        "                predictions = model(text)\n",
        "\n",
        "                predictions = predictions.view(-1, predictions.shape[-1])\n",
        "                tags = tags.view(-1)\n",
        "\n",
        "                # Calculamos el Cross Entropy de las predicciones con respecto a las etiquetas reales\n",
        "                loss = criterion(predictions, tags)\n",
        "\n",
        "                # Calculamos las métricas\n",
        "                precision, recall, f1 = calculate_metrics(predictions, tags)\n",
        "\n",
        "                # Actualizamos el loss y las métricas\n",
        "                epoch_loss += loss.item()\n",
        "                epoch_precision += precision\n",
        "                epoch_recall += recall\n",
        "                epoch_f1 += f1\n",
        "\n",
        "        return epoch_loss / len(iterator), epoch_precision / len(\n",
        "            iterator), epoch_recall / len(iterator), epoch_f1 / len(iterator)\n",
        "\n",
        "    import time\n",
        "\n",
        "    def epoch_time(start_time, end_time):\n",
        "        elapsed_time = end_time - start_time\n",
        "        elapsed_mins = int(elapsed_time / 60)\n",
        "        elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
        "        return elapsed_mins, elapsed_secs\n",
        "\n",
        "    best_valid_loss = float('inf')\n",
        "    best_loss_f1 = -1\n",
        "    best_loss_epoch = -1\n",
        "\n",
        "    for epoch in range(n_epochs):\n",
        "\n",
        "        start_time = time.time()\n",
        "\n",
        "        # Recuerdo: train_iterator y valid_iterator contienen el dataset dividido en batches.\n",
        "\n",
        "        # Entrenar\n",
        "        train_loss, train_precision, train_recall, train_f1 = train(\n",
        "            model, train_iterator, optimizer, criterion)\n",
        "\n",
        "        # Evaluar (valid = validación)\n",
        "        valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "            model, valid_iterator, criterion)\n",
        "\n",
        "        end_time = time.time()\n",
        "\n",
        "        epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
        "\n",
        "        # Si obtuvimos mejores resultados, guardamos este modelo en el almacenamiento (para poder cargarlo luego)\n",
        "        # Si detienen el entrenamiento prematuramente, pueden cargar el modelo en el siguiente recuadro de código.\n",
        "        if valid_loss < best_valid_loss:\n",
        "            best_valid_loss = valid_loss\n",
        "            best_loss_f1 = valid_f1\n",
        "            best_loss_epoch = epoch\n",
        "            torch.save(model.state_dict(), '{}.pt'.format(model_name))\n",
        "        # Si ya no mejoramos el loss de validación, terminamos de entrenar.\n",
        "\n",
        "    results.append({'model': model_name, 'epoch': best_loss_epoch,'loss': best_valid_loss, 'f1': best_loss_f1})\n",
        "\n",
        "print(results)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "250a42d6-139e-4d7f-8030-ece82b696208",
        "id": "rHO1sZc4nqFc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
            "  warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[{'model': 'baseline', 'epoch': 5, 'loss': 0.4105681822236095, 'f1': 0.7285739721227531}, {'model': NER_RNN_GRU(\n",
            "  (embedding): Embedding(17591, 200, padding_idx=1)\n",
            "  (gru): GRU(200, 64, num_layers=2, dropout=0.5, bidirectional=True)\n",
            "  (fc): Linear(in_features=128, out_features=12, bias=True)\n",
            "  (dropout): Dropout(p=0.5, inplace=False)\n",
            "), 'epoch': 2, 'loss': 0.3576923133805394, 'f1': 0.7477914183359858}]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aPGdirx7aSHZ"
      },
      "source": [
        "------\n",
        "### **Entrenamos y evaluamos**\n",
        "\n",
        "\n",
        "**Importante** : Fijen el modelo, el número de épocas de entrenamiento, la loss y el optimizador que usarán para entrenar y evaluar en las siguientes variables!!!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pu_lXic2aSHd"
      },
      "source": [
        "\n",
        "\n",
        "#### **Inicializamos la red**\n",
        "\n",
        "Iniciamos los pesos de la red de forma aleatoria (Usando una distribución normal).\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model = baseline_model\n",
        "model_name = baseline_model_name\n",
        "criterion = baseline_criterion\n",
        "n_epochs = baseline_n_epochs"
      ],
      "metadata": {
        "id": "B7Gtxk_1NwkC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3f6efbfb-31af-4a39-dc6d-61685fae8c17",
        "id": "KfQ9B81qNfOF"
      },
      "source": [
        "def init_weights(m):\n",
        "    # Inicializamos los pesos como aleatorios\n",
        "    for name, param in m.named_parameters():\n",
        "        nn.init.normal_(param.data, mean=0, std=0.1) \n",
        "        \n",
        "    # Seteamos como 0 los embeddings de UNK y PAD.\n",
        "    model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)\n",
        "    model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)\n",
        "        \n",
        "model.apply(init_weights)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "NER_RNN(\n",
              "  (embedding): Embedding(17591, 200, padding_idx=1)\n",
              "  (lstm): LSTM(200, 128, num_layers=3, dropout=0.5)\n",
              "  (fc): Linear(in_features=128, out_features=12, bias=True)\n",
              "  (dropout): Dropout(p=0.5, inplace=False)\n",
              ")"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "puOYwtWYNq62"
      },
      "source": [
        "# Optimizador\n",
        "optimizer = optim.Adam(model.parameters())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cb6XmZiZOBWT"
      },
      "source": [
        "import time\n",
        "\n",
        "def epoch_time(start_time, end_time):\n",
        "    elapsed_time = end_time - start_time\n",
        "    elapsed_mins = int(elapsed_time / 60)\n",
        "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
        "    return elapsed_mins, elapsed_secs"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mjWDX2CJaSHh",
        "outputId": "5d6cb6c2-5cc7-4d74-e3a9-8dab7ef3257b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "El modelo actual tiene 3,844,612 parámetros entrenables.\n"
          ]
        }
      ],
      "source": [
        "def count_parameters(model):\n",
        "    return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
        "print(f'El modelo actual tiene {count_parameters(model):,} parámetros entrenables.')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UVqBqerlaSHk"
      },
      "source": [
        "Notar que definimos los embeddings que representan a \\<unk\\> y \\<pad\\>  como [0, 0, ..., 0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rVZvHtwpaSHq"
      },
      "source": [
        "#### **Definimos el optimizador**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AH6o8_cTaSHq"
      },
      "outputs": [],
      "source": [
        "# Optimizador\n",
        "optimizer = optim.Adam(model.parameters())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fz39wa78wGYR"
      },
      "source": [
        "#### **Enviamos el modelo a cuda**\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dqr0AJ6_iicR"
      },
      "outputs": [],
      "source": [
        "# Enviamos el modelo y la loss a cuda (en el caso en que esté disponible)\n",
        "model = model.to(device)\n",
        "criterion = criterion.to(device)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8xlq48WjiW6U"
      },
      "source": [
        "#### **Definimos el entrenamiento de la red**\n",
        "\n",
        "Algunos conceptos previos: \n",
        "\n",
        "- `epoch` : una pasada de entrenamiento completa de una dataset.\n",
        "- `batch`: una fracción de la época. Se utilizan para entrenar mas rápidamente la red. (mas eficiente pasar n datos que uno en cada ejecución del backpropagation)\n",
        "\n",
        "Esta función está encargada de entrenar la red en una época. Para esto, por cada batch de la época actual, predice los tags del texto, calcula su loss y luego hace backpropagation para actualizar los pesos de la red.\n",
        "\n",
        "Observación: En algunos comentarios aparecerá el tamaño de los tensores entre corchetes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DV6YLt0oiicW"
      },
      "outputs": [],
      "source": [
        "def train(model, iterator, optimizer, criterion):\n",
        "\n",
        "    epoch_loss = 0\n",
        "    epoch_precision = 0\n",
        "    epoch_recall = 0\n",
        "    epoch_f1 = 0\n",
        "\n",
        "    model.train()\n",
        "\n",
        "    # Por cada batch del iterador de la época:\n",
        "    for batch in iterator:\n",
        "\n",
        "        # Extraemos el texto y los tags del batch que estamos procesado\n",
        "        text = batch.text\n",
        "        tags = batch.nertags\n",
        "\n",
        "        # Reiniciamos los gradientes calculados en la iteración anterior\n",
        "        optimizer.zero_grad()\n",
        "\n",
        "        #text = [sent len, batch size]\n",
        "\n",
        "        # Predecimos los tags del texto del batch.\n",
        "        predictions = model(text)\n",
        "\n",
        "        #predictions = [sent len, batch size, output dim]\n",
        "        #tags = [sent len, batch size]\n",
        "\n",
        "        # Reordenamos los datos para calcular la loss\n",
        "        predictions = predictions.view(-1, predictions.shape[-1])\n",
        "        tags = tags.view(-1)\n",
        "\n",
        "        #predictions = [sent len * batch size, output dim]\n",
        "\n",
        "\n",
        "\n",
        "        # Calculamos el Cross Entropy de las predicciones con respecto a las etiquetas reales\n",
        "        loss = criterion(predictions, tags)\n",
        "        \n",
        "        # Calculamos el accuracy\n",
        "        precision, recall, f1 = calculate_metrics(predictions, tags)\n",
        "\n",
        "        # Calculamos los gradientes\n",
        "        loss.backward()\n",
        "\n",
        "        # Actualizamos los parámetros de la red\n",
        "        optimizer.step()\n",
        "\n",
        "        # Actualizamos el loss y las métricas\n",
        "        epoch_loss += loss.item()\n",
        "        epoch_precision += precision\n",
        "        epoch_recall += recall\n",
        "        epoch_f1 += f1\n",
        "\n",
        "    return epoch_loss / len(iterator), epoch_precision / len(\n",
        "        iterator), epoch_recall / len(iterator), epoch_f1 / len(iterator)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PYNcwKnAz5Hf"
      },
      "source": [
        "#### **Definimos la función de evaluación**\n",
        "\n",
        "Evalua el rendimiento actual de la red usando los datos de validación. \n",
        "\n",
        "Por cada batch de estos datos, calcula y reporta el loss y las métricas asociadas al conjunto de validación. \n",
        "Ya que las métricas son calculadas por cada batch, estas son retornadas promediadas por el número de batches entregados. (ver linea del return)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WsRuiUuHiicY"
      },
      "outputs": [],
      "source": [
        "def evaluate(model, iterator, criterion):\n",
        "\n",
        "    epoch_loss = 0\n",
        "    epoch_precision = 0\n",
        "    epoch_recall = 0\n",
        "    epoch_f1 = 0\n",
        "\n",
        "    model.eval()\n",
        "\n",
        "    # Indicamos que ahora no guardaremos los gradientes\n",
        "    with torch.no_grad():\n",
        "        # Por cada batch\n",
        "        for batch in iterator:\n",
        "\n",
        "            text = batch.text\n",
        "            tags = batch.nertags\n",
        "\n",
        "            # Predecimos\n",
        "            predictions = model(text)\n",
        "\n",
        "            predictions = predictions.view(-1, predictions.shape[-1])\n",
        "            tags = tags.view(-1)\n",
        "\n",
        "            # Calculamos el Cross Entropy de las predicciones con respecto a las etiquetas reales\n",
        "            loss = criterion(predictions, tags)\n",
        "\n",
        "            # Calculamos las métricas\n",
        "            precision, recall, f1 = calculate_metrics(predictions, tags)\n",
        "\n",
        "            # Actualizamos el loss y las métricas\n",
        "            epoch_loss += loss.item()\n",
        "            epoch_precision += precision\n",
        "            epoch_recall += recall\n",
        "            epoch_f1 += f1\n",
        "\n",
        "    return epoch_loss / len(iterator), epoch_precision / len(\n",
        "        iterator), epoch_recall / len(iterator), epoch_f1 / len(iterator)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Xs-n9Y5yiica"
      },
      "outputs": [],
      "source": [
        "import time\n",
        "\n",
        "def epoch_time(start_time, end_time):\n",
        "    elapsed_time = end_time - start_time\n",
        "    elapsed_mins = int(elapsed_time / 60)\n",
        "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
        "    return elapsed_mins, elapsed_secs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hy3MVf5H0A94"
      },
      "source": [
        "\n",
        "#### **Entrenamiento de la red**\n",
        "\n",
        "En este cuadro de código ejecutaremos el entrenamiento de la red.\n",
        "Para esto, primero definiremos el número de épocas y luego por cada época, ejecutaremos `train` y `evaluate`.\n",
        "\n",
        "**Importante: Reiniciar los pesos del modelo**\n",
        "\n",
        "Si ejecutas nuevamente esta celda, se seguira entrenando el mismo modelo una y otra vez. \n",
        "Para reiniciar el modelo se debe ejecutar nuevamente la celda que contiene la función `init_weights`\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iK5lQqpviicf",
        "outputId": "b4dc7aea-07e0-46c5-926e-7434fd0d1a2e"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n",
            "/usr/local/lib/python3.7/dist-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
            "  warnings.warn('{} seems not to be NE tag.'.format(chunk))\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch: 01 | Epoch Time: 0m 6s\n",
            "\tTrain Loss: 0.872 | Train f1: 0.37 | Train precision: 0.52 | Train recall: 0.30\n",
            "\t Val. Loss: 0.551 |  Val. f1: 0.59 |  Val. precision: 0.72 | Val. recall: 0.50\n",
            "Epoch: 02 | Epoch Time: 0m 6s\n",
            "\tTrain Loss: 0.506 | Train f1: 0.64 | Train precision: 0.70 | Train recall: 0.60\n",
            "\t Val. Loss: 0.429 |  Val. f1: 0.69 |  Val. precision: 0.74 | Val. recall: 0.65\n",
            "Epoch: 03 | Epoch Time: 0m 6s\n",
            "\tTrain Loss: 0.380 | Train f1: 0.73 | Train precision: 0.76 | Train recall: 0.71\n",
            "\t Val. Loss: 0.413 |  Val. f1: 0.71 |  Val. precision: 0.73 | Val. recall: 0.69\n",
            "Epoch: 04 | Epoch Time: 0m 6s\n",
            "\tTrain Loss: 0.307 | Train f1: 0.78 | Train precision: 0.80 | Train recall: 0.77\n",
            "\t Val. Loss: 0.427 |  Val. f1: 0.72 |  Val. precision: 0.75 | Val. recall: 0.71\n",
            "Epoch: 05 | Epoch Time: 0m 6s\n",
            "\tTrain Loss: 0.261 | Train f1: 0.82 | Train precision: 0.82 | Train recall: 0.81\n",
            "\t Val. Loss: 0.422 |  Val. f1: 0.74 |  Val. precision: 0.75 | Val. recall: 0.72\n",
            "Epoch: 06 | Epoch Time: 0m 6s\n",
            "\tTrain Loss: 0.229 | Train f1: 0.84 | Train precision: 0.84 | Train recall: 0.83\n",
            "\t Val. Loss: 0.424 |  Val. f1: 0.74 |  Val. precision: 0.78 | Val. recall: 0.72\n",
            "Epoch: 07 | Epoch Time: 0m 6s\n",
            "\tTrain Loss: 0.206 | Train f1: 0.85 | Train precision: 0.86 | Train recall: 0.85\n",
            "\t Val. Loss: 0.437 |  Val. f1: 0.74 |  Val. precision: 0.75 | Val. recall: 0.74\n",
            "Epoch: 08 | Epoch Time: 0m 6s\n",
            "\tTrain Loss: 0.184 | Train f1: 0.87 | Train precision: 0.87 | Train recall: 0.87\n",
            "\t Val. Loss: 0.477 |  Val. f1: 0.73 |  Val. precision: 0.75 | Val. recall: 0.72\n",
            "Epoch: 09 | Epoch Time: 0m 6s\n",
            "\tTrain Loss: 0.171 | Train f1: 0.88 | Train precision: 0.88 | Train recall: 0.88\n",
            "\t Val. Loss: 0.448 |  Val. f1: 0.73 |  Val. precision: 0.74 | Val. recall: 0.74\n",
            "Epoch: 10 | Epoch Time: 0m 6s\n",
            "\tTrain Loss: 0.160 | Train f1: 0.89 | Train precision: 0.89 | Train recall: 0.89\n",
            "\t Val. Loss: 0.469 |  Val. f1: 0.73 |  Val. precision: 0.73 | Val. recall: 0.74\n",
            "Epoch: 11 | Epoch Time: 0m 6s\n",
            "\tTrain Loss: 0.147 | Train f1: 0.89 | Train precision: 0.89 | Train recall: 0.90\n",
            "\t Val. Loss: 0.495 |  Val. f1: 0.74 |  Val. precision: 0.76 | Val. recall: 0.73\n",
            "Epoch: 12 | Epoch Time: 0m 6s\n",
            "\tTrain Loss: 0.140 | Train f1: 0.90 | Train precision: 0.90 | Train recall: 0.90\n",
            "\t Val. Loss: 0.496 |  Val. f1: 0.73 |  Val. precision: 0.73 | Val. recall: 0.72\n",
            "Epoch: 13 | Epoch Time: 0m 6s\n",
            "\tTrain Loss: 0.131 | Train f1: 0.91 | Train precision: 0.91 | Train recall: 0.91\n",
            "\t Val. Loss: 0.505 |  Val. f1: 0.73 |  Val. precision: 0.74 | Val. recall: 0.73\n",
            "Epoch: 14 | Epoch Time: 0m 6s\n",
            "\tTrain Loss: 0.121 | Train f1: 0.92 | Train precision: 0.91 | Train recall: 0.92\n",
            "\t Val. Loss: 0.510 |  Val. f1: 0.73 |  Val. precision: 0.75 | Val. recall: 0.72\n",
            "Epoch: 15 | Epoch Time: 0m 6s\n",
            "\tTrain Loss: 0.117 | Train f1: 0.92 | Train precision: 0.92 | Train recall: 0.92\n",
            "\t Val. Loss: 0.520 |  Val. f1: 0.74 |  Val. precision: 0.75 | Val. recall: 0.73\n",
            "Epoch: 16 | Epoch Time: 0m 6s\n",
            "\tTrain Loss: 0.113 | Train f1: 0.92 | Train precision: 0.92 | Train recall: 0.92\n",
            "\t Val. Loss: 0.539 |  Val. f1: 0.73 |  Val. precision: 0.73 | Val. recall: 0.73\n",
            "Epoch: 17 | Epoch Time: 0m 6s\n",
            "\tTrain Loss: 0.108 | Train f1: 0.92 | Train precision: 0.92 | Train recall: 0.93\n",
            "\t Val. Loss: 0.521 |  Val. f1: 0.74 |  Val. precision: 0.76 | Val. recall: 0.72\n",
            "Epoch: 18 | Epoch Time: 0m 6s\n",
            "\tTrain Loss: 0.102 | Train f1: 0.93 | Train precision: 0.93 | Train recall: 0.93\n",
            "\t Val. Loss: 0.552 |  Val. f1: 0.73 |  Val. precision: 0.74 | Val. recall: 0.73\n",
            "Epoch: 19 | Epoch Time: 0m 6s\n",
            "\tTrain Loss: 0.098 | Train f1: 0.93 | Train precision: 0.93 | Train recall: 0.93\n",
            "\t Val. Loss: 0.563 |  Val. f1: 0.73 |  Val. precision: 0.75 | Val. recall: 0.72\n",
            "Epoch: 20 | Epoch Time: 0m 6s\n",
            "\tTrain Loss: 0.094 | Train f1: 0.93 | Train precision: 0.93 | Train recall: 0.94\n",
            "\t Val. Loss: 0.575 |  Val. f1: 0.73 |  Val. precision: 0.73 | Val. recall: 0.73\n"
          ]
        }
      ],
      "source": [
        "best_valid_loss = float('inf')\n",
        "\n",
        "for epoch in range(n_epochs):\n",
        "\n",
        "    start_time = time.time()\n",
        "\n",
        "    # Recuerdo: train_iterator y valid_iterator contienen el dataset dividido en batches.\n",
        "\n",
        "    # Entrenar\n",
        "    train_loss, train_precision, train_recall, train_f1 = train(\n",
        "        model, train_iterator, optimizer, criterion)\n",
        "\n",
        "    # Evaluar (valid = validación)\n",
        "    valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "        model, valid_iterator, criterion)\n",
        "\n",
        "    end_time = time.time()\n",
        "\n",
        "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
        "\n",
        "    # Si obtuvimos mejores resultados, guardamos este modelo en el almacenamiento (para poder cargarlo luego)\n",
        "    # Si detienen el entrenamiento prematuramente, pueden cargar el modelo en el siguiente recuadro de código.\n",
        "    if valid_loss < best_valid_loss:\n",
        "        best_valid_loss = valid_loss\n",
        "        torch.save(model.state_dict(), '{}.pt'.format(model_name))\n",
        "    # Si ya no mejoramos el loss de validación, terminamos de entrenar.\n",
        "\n",
        "    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
        "    print(\n",
        "        f'\\tTrain Loss: {train_loss:.3f} | Train f1: {train_f1:.2f} | Train precision: {train_precision:.2f} | Train recall: {train_recall:.2f}'\n",
        "    )\n",
        "    print(\n",
        "        f'\\t Val. Loss: {valid_loss:.3f} |  Val. f1: {valid_f1:.2f} |  Val. precision: {valid_precision:.2f} | Val. recall: {valid_recall:.2f}'\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EcZPraG-9duO"
      },
      "source": [
        "**Importante**: Recuerden que el último modelo entrenado no es el mejor (probablemente esté *overfitteado*), si no el que guardamos con la menor loss del conjunto de validación. Este problema lo pueden solucionar con *early stopping*.\n",
        "Para cargar el mejor modelo entrenado, ejecuten la siguiente celda.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "y27CNYfrjtQ-"
      },
      "outputs": [],
      "source": [
        "# cargar el mejor modelo entrenado.\n",
        "model.load_state_dict(torch.load('{}.pt'.format(model_name)))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oLuqFKFR9duO"
      },
      "outputs": [],
      "source": [
        "# Limpiar ram de cuda\n",
        "torch.cuda.empty_cache()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kBctQHTh0lxD"
      },
      "source": [
        "#### **Evaluamos el set de validación con el modelo final**\n",
        "\n",
        "Estos son los resultados de predecir el dataset de evaluación con el *mejor* modelo entrenado."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "s0gVbP8yiicj"
      },
      "outputs": [],
      "source": [
        "valid_loss, valid_precision, valid_recall, valid_f1 = evaluate(\n",
        "    model, valid_iterator, criterion)\n",
        "\n",
        "print(\n",
        "    f'Val. Loss: {valid_loss:.3f} |  Val. f1: {valid_f1:.2f} | Val. precision: {valid_precision:.2f} | Val. recall: {valid_recall:.2f}'\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uF1ysw_Kw6zz"
      },
      "source": [
        "### **Predecir datos para la competencia**\n",
        "\n",
        "Ahora, a partir de los datos de **test** y nuestro modelo entrenado, vamos a predecir las etiquetas que serán evaluadas en la competencia."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1RBs3UU4wLk3"
      },
      "outputs": [],
      "source": [
        "def predict_labels(model, iterator, criterion, fields=fields):\n",
        "\n",
        "    # Extraemos los vocabularios.\n",
        "    text_field = fields[0][1]\n",
        "    nertags_field = fields[1][1]\n",
        "    tags_vocab = nertags_field.vocab.itos\n",
        "    words_vocab = text_field.vocab.itos\n",
        "\n",
        "    model.eval()\n",
        "\n",
        "    predictions = []\n",
        "\n",
        "    with torch.no_grad():\n",
        "\n",
        "        for batch in iterator:\n",
        "\n",
        "            text_batch = batch.text\n",
        "            text_batch = torch.transpose(text_batch, 0, 1).tolist()\n",
        "\n",
        "            # Predecir los tags de las sentences del batch\n",
        "            predictions_batch = model(batch.text)\n",
        "            predictions_batch = torch.transpose(predictions_batch, 0, 1)\n",
        "\n",
        "            # por cada oración predicha:\n",
        "            for sentence, sentence_prediction in zip(text_batch,\n",
        "                                                     predictions_batch):\n",
        "                for word_idx, word_predictions in zip(sentence,\n",
        "                                                      sentence_prediction):\n",
        "                    # Obtener el indice del tag con la probabilidad mas alta.\n",
        "                    argmax_index = word_predictions.topk(1)[1]\n",
        "\n",
        "                    current_tag = tags_vocab[argmax_index]\n",
        "                    # Obtenemos la palabra\n",
        "                    current_word = words_vocab[word_idx]\n",
        "\n",
        "                    if current_word != '<pad>':\n",
        "                        predictions.append([current_word, current_tag])\n",
        "                predictions.append(['EOS', 'EOS'])\n",
        "\n",
        "\n",
        "    return predictions\n",
        "\n",
        "\n",
        "predictions = predict_labels(model, test_iterator, criterion)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YwQp1Ru8Oht8"
      },
      "source": [
        "### **Generar el archivo para la submission**\n",
        "\n",
        "No hay problema si aparecen unk en la salida. Estos no son relevantes para evaluarlos, usamos solo los tags."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RPfZkjJGkWyq"
      },
      "outputs": [],
      "source": [
        "import os, shutil\n",
        "\n",
        "if (os.path.isfile('./predictions.zip')):\n",
        "    os.remove('./predictions.zip')\n",
        "\n",
        "if (not os.path.isdir('./predictions')):\n",
        "    os.mkdir('./predictions')\n",
        "\n",
        "else:\n",
        "    # Eliminar predicciones anteriores:\n",
        "    shutil.rmtree('./predictions')\n",
        "    os.mkdir('./predictions')\n",
        "\n",
        "f = open('predictions/predictions.txt', 'w')\n",
        "for i, (word, tag) in enumerate(predictions[:-1]):\n",
        "    if word=='EOS' and tag=='EOS': f.write('\\n')\n",
        "    else: \n",
        "      if i == len(predictions[:-1])-1:\n",
        "        f.write(word + ' ' + tag)\n",
        "      else: f.write(word + ' ' + tag + '\\n')\n",
        "\n",
        "f.close()\n",
        "\n",
        "a = shutil.make_archive('predictions', 'zip', './predictions')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LZEWJXrNaSIf"
      },
      "source": [
        "## **Conclusiones**\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gAtK7y43V7Z_"
      },
      "source": [
        "Se logró superar el baseline entregado experimentando con distintas implementaciones de RNN y variando distintos hiperparametros de la red.\n",
        "\n",
        "Se ha logrado experimentar con distintos hiperparámetros, entre los cuales se destacan, learning rate, número de neuronas en la capa oculta, número de capas ocultas, dropout, tipos de optimizadores, tipo de celdas (LSTM y GRU). Además, se implementó un modelo de Encoder-Decoder con mecanismo de antención, sin embargo, este realizaba un proceso de aprendizaje apropiado. Esto se puede deber a que pudo haber un error de implementación, o porque no se utilizaron hiperparámetros adecuados, que modifiquen la arquitectura del modelo o incluso el número de épocas. \n",
        "\n",
        "Un contratiempo fue no poder aleatorizar los vectores no encontrados en el embedding pre-entrenado. Los no encontrados quedaron nulos. \n",
        "\n",
        "Un problema importante fue no poder compatibilizar todas las mejoras individuales al baseline. Al momento de guardar un modelo con las mejores predicciones, se decidió intentar por un lado con los embeddings y por otro todas las modificaciones a los hiperparámetros y el cambio de LSTM por GRU. Es muy probable que el uso conjunto de todo lo mencionado entregue resultados aun mejores, quedará pendiente para futuras implementaciones.\n",
        "\n",
        "Finalmente se cree que la tarea abordada podria ser de gran utilidad campo medico a futuro pues permitiria facilitar la asignacion de pacientes en lista de espera a especialistas idoneos para su caso particular."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [
        "d4-UEViH6jXU"
      ],
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.3"
    },
    "toc": {
      "base_numbering": 1,
      "nav_menu": {},
      "number_sections": true,
      "sideBar": true,
      "skip_h1_title": true,
      "title_cell": "Table of Contents",
      "title_sidebar": "Contents",
      "toc_cell": false,
      "toc_position": {},
      "toc_section_display": true,
      "toc_window_display": true
    },
    "varInspector": {
      "cols": {
        "lenName": 16,
        "lenType": 16,
        "lenVar": 40
      },
      "kernels_config": {
        "python": {
          "delete_cmd_postfix": "",
          "delete_cmd_prefix": "del ",
          "library": "var_list.py",
          "varRefreshCmd": "print(var_dic_list())"
        },
        "r": {
          "delete_cmd_postfix": ") ",
          "delete_cmd_prefix": "rm(",
          "library": "var_list.r",
          "varRefreshCmd": "cat(var_dic_list()) "
        }
      },
      "types_to_exclude": [
        "module",
        "function",
        "builtin_function_or_method",
        "instance",
        "_Feature"
      ],
      "window_display": false
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}