1_text_simple_classification_training.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/vanderbilt-data-science/ai_summer/blob/main/1_text_simple_classification_training.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8d4849f3-51e1-4299-9f55-9c70c1f1c431",
      "metadata": {
        "id": "8d4849f3-51e1-4299-9f55-9c70c1f1c431"
      },
      "source": [
        "# Fine-Tuning Models\n",
        "> An introductory example on fine-tuning models\n",
        "\n",
        "In this walkthrough, we'll explore the standard steps of fine-tuning a model, and we'll apply this towards the intuitive task of text classification.\n",
        "\n",
        "We'll leverage the [`tweet_eval` dataset](https://huggingface.co/datasets/tweet_eval) to try to classify emotions of tweets into relevant categories."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Initial Setup"
      ],
      "metadata": {
        "id": "thJhDvdZF-3U"
      },
      "id": "thJhDvdZF-3U"
    },
    {
      "cell_type": "markdown",
      "id": "0abbc1b7-f4b4-4cdd-93ed-ce92742fe4c6",
      "metadata": {
        "id": "0abbc1b7-f4b4-4cdd-93ed-ce92742fe4c6"
      },
      "source": [
        "### Install required packages\n",
        "Note that this is mostly required if you're on Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6067752c-e281-4ffe-9140-828d158751a1",
      "metadata": {
        "id": "6067752c-e281-4ffe-9140-828d158751a1"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "! pip install transformers\n",
        "! pip install datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b4b4f46c-e1c0-4abc-9498-a45211bcf9a6",
      "metadata": {
        "id": "b4b4f46c-e1c0-4abc-9498-a45211bcf9a6"
      },
      "source": [
        "### Import packages of interest"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6afa3949-c286-4a82-af4f-c5f7594a6c97",
      "metadata": {
        "id": "6afa3949-c286-4a82-af4f-c5f7594a6c97"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "from datasets import load_dataset, load_metric, Dataset\n",
        "from transformers import pipeline\n",
        "from transformers import DataCollatorWithPadding\n",
        "from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer\n",
        "\n",
        "from huggingface_hub import notebook_login"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6902fa8d-8192-4ab0-8f5f-9a0ddf6c34fd",
      "metadata": {
        "id": "6902fa8d-8192-4ab0-8f5f-9a0ddf6c34fd"
      },
      "source": [
        "## Log into HuggingFace CLI\n",
        "Why are we doing this? Below, we'll use our own user accounts to grab datasets and upload models. If we don't do this, we'll have to pass in the auth token over. This isn't bad, but let's streamline our efforts!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3a75fd3d-76d2-425f-be19-aace4db0e71a",
      "metadata": {
        "id": "3a75fd3d-76d2-425f-be19-aace4db0e71a"
      },
      "outputs": [],
      "source": [
        "!git config --global credential.helper store"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9d0b6acb-bf9e-4e5c-8764-90eed505a2fb",
      "metadata": {
        "tags": [],
        "id": "9d0b6acb-bf9e-4e5c-8764-90eed505a2fb"
      },
      "outputs": [],
      "source": [
        "notebook_login()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d0a371ec-98f5-4be7-b085-a2eed2f9c3d3",
      "metadata": {
        "id": "d0a371ec-98f5-4be7-b085-a2eed2f9c3d3"
      },
      "source": [
        "# Load data from HuggingFace Hub, Datasets, or from disk\n",
        "\n",
        "In this example, we'll pull from the Huggingface Datasets repository. However, if you have your own dataset, you can use this here. We'll go over how to use your own datasets in future classes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "fc4b6693-573c-4515-b3bc-0a70c43945c2",
      "metadata": {
        "id": "fc4b6693-573c-4515-b3bc-0a70c43945c2"
      },
      "outputs": [],
      "source": [
        "#Load tweet_eval dataset, emotion configuration\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# View general structure of data\n"
      ],
      "metadata": {
        "id": "8Mg6MNhPIJl7"
      },
      "id": "8Mg6MNhPIJl7",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Look at an example\n"
      ],
      "metadata": {
        "id": "Za-YxO-0IOBG"
      },
      "id": "Za-YxO-0IOBG",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Look at labels\n"
      ],
      "metadata": {
        "id": "z1qB-uJZIaSE"
      },
      "id": "z1qB-uJZIaSE",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Create id2label, label2id, and standard info from datasets\n",
        "\n",
        "\n",
        "print(num_classes)\n",
        "print(id2label)\n",
        "label2id"
      ],
      "metadata": {
        "id": "_O6dH3LYIjAD"
      },
      "id": "_O6dH3LYIjAD",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "d558ecf4-e35b-4afa-a1aa-ca16e86ab432",
      "metadata": {
        "id": "d558ecf4-e35b-4afa-a1aa-ca16e86ab432"
      },
      "source": [
        "# Pre-process inputs\n",
        "We've already learned about tokenizers - let's see what this looks like as we approach training. A richer treatment of tokenizers can be found in the Huggingface [instructions on tokenizers](https://huggingface.co/course/chapter2/4?fw=pt). Then, let's try it on our own!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3d6a7940-ae31-4efe-9398-93da6b2b0aa7",
      "metadata": {
        "id": "3d6a7940-ae31-4efe-9398-93da6b2b0aa7"
      },
      "outputs": [],
      "source": [
        "#instantiate tokenizer\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "68d994af-196d-4dba-8c1b-95a75d8fa14b",
      "metadata": {
        "id": "68d994af-196d-4dba-8c1b-95a75d8fa14b"
      },
      "outputs": [],
      "source": [
        "#define tokenizing function\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "39bc59e5-52b9-4449-8cda-bd94e88e980d",
      "metadata": {
        "id": "39bc59e5-52b9-4449-8cda-bd94e88e980d"
      },
      "outputs": [],
      "source": [
        "#do the tokenizing using map function\n",
        "tokenized_ds = demo_ds.map(tokenize_inputs, batched=True, remove_columns=['text'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2061e38a-4e26-4294-8a3f-99f3a51de49a",
      "metadata": {
        "id": "2061e38a-4e26-4294-8a3f-99f3a51de49a"
      },
      "outputs": [],
      "source": [
        "tokenized_ds"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "15655b8b-45bb-4c3c-8ad8-dcd869beda5d",
      "metadata": {
        "id": "15655b8b-45bb-4c3c-8ad8-dcd869beda5d"
      },
      "source": [
        "## An aside on tokenizer functionality\n",
        "We can do many things with tokenizers to help us to tokenize our data and process it. Let's check out these outputs further."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5fed0e3e-a3da-45b8-9c19-8749567c96fc",
      "metadata": {
        "id": "5fed0e3e-a3da-45b8-9c19-8749567c96fc",
        "outputId": "ae7b3fa5-8b67-4209-b4cf-9d56e4af6db6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[101, 789, 160, 1766, 1616, 1110, 170, 1205, 7727, 1113, 170, 2463, 1128, 1336, 1309, 1138, 112, 119, 11882, 11545, 119, 108, 15710, 108, 3645, 108, 3994, 102]\n",
            "“Worry is a down payment on a problem you may never have'.  Joyce Meyer.  #motivation #leadership #worry\n"
          ]
        }
      ],
      "source": [
        "#check out input IDs\n",
        "print(tokenized_ds['train']['input_ids'][0])\n",
        "\n",
        "#compare against the text\n",
        "print(demo_ds['train']['text'][0])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "685e1f43-5182-4094-880b-05a02c71d5b1",
      "metadata": {
        "id": "685e1f43-5182-4094-880b-05a02c71d5b1",
        "outputId": "91e2089a-d52f-432a-dc36-75c16de6047a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "3257\n",
            "28\n"
          ]
        }
      ],
      "source": [
        "#check out the length of the list of lists\n",
        "print(len(tokenized_ds['train']['input_ids']))\n",
        "\n",
        "#check out the length of a single element\n",
        "print(len(tokenized_ds['train']['input_ids'][0]))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "28141308-68b3-4f8e-affe-29b89118b201",
      "metadata": {
        "id": "28141308-68b3-4f8e-affe-29b89118b201",
        "outputId": "a4a253b4-cadc-42d8-9861-9b7c4aa9a776",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['[CLS]', '“', 'W', '##or', '##ry', 'is', 'a', 'down', 'payment', 'on', 'a', 'problem', 'you', 'may', 'never', 'have', \"'\", '.', 'Joyce', 'Meyer', '.', '#', 'motivation', '#', 'leadership', '#', 'worry', '[SEP]']\n"
          ]
        }
      ],
      "source": [
        "#convert input_ids to token representation\n",
        "input0_tokens = tokenizer.convert_ids_to_tokens(tokenized_ds['train']['input_ids'][0])\n",
        "print(input0_tokens)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "51ac4262-3678-4bf7-844a-f91d1f4d56fe",
      "metadata": {
        "id": "51ac4262-3678-4bf7-844a-f91d1f4d56fe",
        "outputId": "3968303b-bf47-4a20-f378-4dafa4d9f163",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[CLS] “ Worry is a down payment on a problem you may never have '. Joyce Meyer. # motivation # leadership # worry [SEP]\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"[CLS] “ Worry is a down payment on a problem you may never have '. Joyce Meyer. # motivation # leadership # worry [SEP]\""
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 17
        }
      ],
      "source": [
        "#see what this looks like as a string\n",
        "print(tokenizer.convert_tokens_to_string(input0_tokens))\n",
        "\n",
        "#another method directly from the input ids\n",
        "tokenizer.decode(tokenized_ds['train']['input_ids'][0])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "af725b84-ab00-413b-aa84-36be19efd62f",
      "metadata": {
        "id": "af725b84-ab00-413b-aa84-36be19efd62f",
        "outputId": "dcd64f5e-ce84-4323-8af0-aa39c78639fc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 412
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "28996\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "          tokens\n",
              "inds            \n",
              "15555      Scene\n",
              "18102  Auxiliary\n",
              "27474   ##lanche\n",
              "318            Ś\n",
              "12117    Genesis\n",
              "6214   corporate\n",
              "15685      erect\n",
              "28478        ##ث\n",
              "26174  ##taining\n",
              "5331          Ty"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-ce9b23d5-8b5c-4c32-a240-d4ad220ed325\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>tokens</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>inds</th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>15555</th>\n",
              "      <td>Scene</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18102</th>\n",
              "      <td>Auxiliary</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27474</th>\n",
              "      <td>##lanche</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>318</th>\n",
              "      <td>Ś</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12117</th>\n",
              "      <td>Genesis</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6214</th>\n",
              "      <td>corporate</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15685</th>\n",
              "      <td>erect</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28478</th>\n",
              "      <td>##ث</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26174</th>\n",
              "      <td>##taining</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5331</th>\n",
              "      <td>Ty</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ce9b23d5-8b5c-4c32-a240-d4ad220ed325')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-ce9b23d5-8b5c-4c32-a240-d4ad220ed325 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-ce9b23d5-8b5c-4c32-a240-d4ad220ed325');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ],
      "source": [
        "#other information about tokenizer\n",
        "print(tokenizer.vocab_size)\n",
        "\n",
        "#see actual tokenizer vocab (we've abbreviated here)\n",
        "#tokenizer.vocab\n",
        "pd.DataFrame({'tokens': tokenizer.vocab.keys(), 'inds': tokenizer.vocab.values()}).set_index('inds').head(10)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "08845526-78fb-4096-bb4b-1ca13c7eda12",
      "metadata": {
        "id": "08845526-78fb-4096-bb4b-1ca13c7eda12"
      },
      "source": [
        "## An aside on dynamically padded batch size\n",
        "HF has the capacity to dynamically pad your batches such that each input is only as long as any given input in the batch. This helps with memory.You can learn more [here](https://huggingface.co/course/chapter3/2?fw=pt). For now, we'll simply instantiate a data collator and use it during training to demonstrate how we can do this."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "df86dd09-500c-4a14-80c4-32d9a432c0dd",
      "metadata": {
        "id": "df86dd09-500c-4a14-80c4-32d9a432c0dd"
      },
      "outputs": [],
      "source": [
        "#Instantiate data collator\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3da664a0-b63c-450a-ae74-bab101d55ea1",
      "metadata": {
        "id": "3da664a0-b63c-450a-ae74-bab101d55ea1"
      },
      "source": [
        "# Model Training Preparation"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "be6051da-9e43-48f2-baed-65e34b6488fb",
      "metadata": {
        "id": "be6051da-9e43-48f2-baed-65e34b6488fb"
      },
      "source": [
        "## Define model and task architecture"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2606be9d-e38c-48dd-9a8b-73be81ab5d93",
      "metadata": {
        "id": "2606be9d-e38c-48dd-9a8b-73be81ab5d93"
      },
      "outputs": [],
      "source": [
        "# Choose the model type and instantiate it for the task\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Consideration of appropriate metrics\n",
        "\n",
        "What are good metrics for us to use for classification?"
      ],
      "metadata": {
        "id": "aDxfB0aPLbCt"
      },
      "id": "aDxfB0aPLbCt"
    },
    {
      "cell_type": "markdown",
      "source": [
        "### From HF Datasets Metrics\n",
        "Some metrics are available to us through [HF Datasets Metrics repo](https://huggingface.co/metrics)."
      ],
      "metadata": {
        "id": "n6rZFDAiLm8j"
      },
      "id": "n6rZFDAiLm8j"
    },
    {
      "cell_type": "code",
      "source": [
        "#load a metric\n",
        "metric = load_metric(\"accuracy\")\n",
        "\n",
        "#define the metric behavior\n",
        "def compute_metrics(eval_pred):\n",
        "    logits, labels = eval_pred\n",
        "    predictions = np.argmax(logits, axis=-1)\n",
        "    return metric.compute(predictions=predictions, references=labels)"
      ],
      "metadata": {
        "id": "zZYMo9vnLhWJ"
      },
      "id": "zZYMo9vnLhWJ",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Custom definitions\n",
        "We can also define our own. The function inputs are a tuple of logits and labels, and the function must return a dictionary of key-value pairs. The keys should be the name of the metric and the values should be the values of that metric. We'll see an example below."
      ],
      "metadata": {
        "id": "2K88VoaqL3bN"
      },
      "id": "2K88VoaqL3bN"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1ee093f3-a6ff-4134-9009-d7eaaf8760a1",
      "metadata": {
        "id": "1ee093f3-a6ff-4134-9009-d7eaaf8760a1"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import precision_recall_fscore_support\n",
        "\n",
        "def compute_metrics(eval_pred):\n",
        "    logits, labels = eval_pred\n",
        "    \n",
        "    #get predictions by using index of max logit\n",
        "    predictions = np.argmax(logits, axis=-1)\n",
        "    \n",
        "    #calculate classification report\n",
        "    perfs = precision_recall_fscore_support(labels, predictions, average='macro', zero_division=0)\n",
        "    perf_dict = dict(zip(['precision', 'recall', 'fscore'], perfs[:3]))\n",
        "    \n",
        "    #return dictionary\n",
        "    return perf_dict"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Settings for Model Training\n",
        "Now, let's set some parameters that will govern the training loop of the model training. This includes practical considerations such as:\n",
        "* Where the model should be saved\n",
        "* Whether the model should be pushed to hub\n",
        "* How often to assess the performance of the model on the validation set\n",
        "\n",
        "As well as settings for neural network training, including:\n",
        "* Number of epochs to train\n",
        "* Learning rate\n",
        "* Optimizer parameters\n",
        "\n",
        "We do this through [TrainingArguments](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments) and the [Trainer class.](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer) Let's take a look!"
      ],
      "metadata": {
        "id": "Ct5YQhNYNspN"
      },
      "id": "Ct5YQhNYNspN"
    },
    {
      "cell_type": "code",
      "source": [
        "#set new training arguments\n",
        "training_args = TrainingArguments(\"bert-emotion\",\n",
        "                                  logging_strategy = \"epoch\",\n",
        "                                  evaluation_strategy=\"epoch\",\n",
        "                                  save_strategy='epoch',\n",
        "                                  load_best_model_at_end = True,\n",
        "                                  metric_for_best_model='fscore',\n",
        "                                  greater_is_better=True,\n",
        "                                  per_device_train_batch_size = 4,\n",
        "                                  per_device_eval_batch_size = 4,\n",
        "                                  num_train_epochs=3,\n",
        "                                  push_to_hub=True,\n",
        "                                  hub_strategy='checkpoint',\n",
        "                                  report_to='all')\n",
        "\n",
        "#set data and functionality for trainer\n",
        "trainer = Trainer(model=model,\n",
        "                  args=training_args,\n",
        "                  tokenizer=tokenizer,\n",
        "                  data_collator=data_collator,\n",
        "                  train_dataset=tokenized_ds['train'],\n",
        "                  eval_dataset=tokenized_ds['validation'],\n",
        "                  compute_metrics=compute_metrics)"
      ],
      "metadata": {
        "id": "1ymF80XDNrw8"
      },
      "id": "1ymF80XDNrw8",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Train model\n",
        "Now, let's actually train the model!\n",
        "\n",
        "Below, we'll define a `model_init` function, which allows us to start training from the HF BERT training checkpoint (in other words, we start training from the HF weights we've downloaded).\n",
        "\n",
        "We borrow this approach from the Tunstall text which allows us clean restarts of training. If we did not take this approach, we would continue modifying the underlying model in place with no record of continuing training."
      ],
      "metadata": {
        "id": "oXe8fUReQyUu"
      },
      "id": "oXe8fUReQyUu"
    },
    {
      "cell_type": "code",
      "source": [
        "#define a model init function which will help us just start training fron scratch\n",
        "def model_init():\n",
        "  return(AutoModelForSequenceClassification.from_pretrained(\"distilbert-base-cased\",\n",
        "                                                           num_labels=num_classes,\n",
        "                                                           id2label=id2label,\n",
        "                                                           label2id=label2id))"
      ],
      "metadata": {
        "id": "KoYpWsK9SQeY"
      },
      "id": "KoYpWsK9SQeY",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#example of how we can update trainer attributes (note that generally you wouldn't do this unless you're just experimenting around)\n",
        "trainer.model = None\n",
        "trainer.model_init = model_init\n",
        "\n",
        "#actually train the model\n"
      ],
      "metadata": {
        "id": "1-v71nXaQUlw"
      },
      "id": "1-v71nXaQUlw",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b4d06945-a25e-48b2-a0e9-a9cd2b6778e6",
      "metadata": {
        "id": "b4d06945-a25e-48b2-a0e9-a9cd2b6778e6"
      },
      "outputs": [],
      "source": [
        "#it's recommended to push the final version to HF after training completes.\n",
        "#Note that the code below takes FOREVER depending on the size of your model so you might consider NOT running\n",
        "#this line until the end of class\n",
        "#trainer.push_to_hub(commit_message='end of training 3 epochs')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b6ba1189-3e5f-42ae-bfae-e79f85a439f5",
      "metadata": {
        "id": "b6ba1189-3e5f-42ae-bfae-e79f85a439f5"
      },
      "source": [
        "# Using trained model with `Trainer`"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "## Evaluate\n",
        "We can assess the performance of the model over a large number of inputs (e.g., the test set). Here, we initially look at the performance of the training set to make sure the model _can_ learn from the data we've provided."
      ],
      "metadata": {
        "id": "bEnxWnAaZeNu"
      },
      "id": "bEnxWnAaZeNu"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "bd5bd0ec-ec3f-4f0f-b52f-70c19f6a79b7",
      "metadata": {
        "id": "bd5bd0ec-ec3f-4f0f-b52f-70c19f6a79b7"
      },
      "outputs": [],
      "source": [
        "#run model evaluation on train dataset\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c81f6ec1-24fb-43f7-a52a-1cdab4861fc5",
      "metadata": {
        "id": "c81f6ec1-24fb-43f7-a52a-1cdab4861fc5"
      },
      "source": [
        "## Predict\n",
        "We can also use the model to predict and have the actual logits returned to us. This is helpful if we want to better inspect the performance of the model to consider consistent reasons for misclassifications and ideas on how to improve the performance of our model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "83797f24-bcfb-405d-9f3a-01c8b26fd7ba",
      "metadata": {
        "id": "83797f24-bcfb-405d-9f3a-01c8b26fd7ba"
      },
      "outputs": [],
      "source": [
        "#use trainer to predict\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#decide to create a confusion matrix, so import this knowledge\n",
        "from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay"
      ],
      "metadata": {
        "id": "ZE9IW_R-rGt2"
      },
      "id": "ZE9IW_R-rGt2",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a dataframe for inspection\n",
        "preds_df = pd.DataFrame({'pred_ids':np.argmax(preds.predictions, axis=-1),\n",
        "                         'label_ids':preds.label_ids,\n",
        "                         'text':demo_ds['train']['text']})\n",
        "display(preds_df.head())\n",
        "\n",
        "# Populate pred_labels\n",
        "preds_df['pred_labels'] = preds_df['pred_ids'].replace(id2label)\n",
        "preds_df['true_labels'] = preds_df['label_ids'].replace(id2label)\n",
        "\n",
        "# Define misclassified\n",
        "preds_df['is_misclassified'] = preds_df['pred_ids'] != preds_df['label_ids']\n",
        "display(preds_df.query('is_misclassified == True'))\n",
        "\n",
        "# Get confusion matrix\n",
        "ConfusionMatrixDisplay.from_predictions(preds_df['true_labels'], preds_df['pred_labels'])"
      ],
      "metadata": {
        "id": "_IBUkSlLflk1"
      },
      "id": "_IBUkSlLflk1",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#an example of inspecting the results to see examples of incorrect labels\n",
        "preds_df.query(\"true_labels=='joy' and pred_labels=='anger'\")['text'].tolist()"
      ],
      "metadata": {
        "id": "0hOL8rb2tYNF"
      },
      "id": "0hOL8rb2tYNF",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "94abc5f1-076e-4167-9c85-26d77404797f",
      "metadata": {
        "id": "94abc5f1-076e-4167-9c85-26d77404797f"
      },
      "source": [
        "# Using your fine-tuned model\n",
        "You can use the model that you've saved locally or the model that you've pushed to hub within a pipeline. Let's see how this works."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ca0edf1e-7ce3-450e-97dc-4abee8e6a50c",
      "metadata": {
        "id": "ca0edf1e-7ce3-450e-97dc-4abee8e6a50c"
      },
      "outputs": [],
      "source": [
        "#create pipeline from your classifier\n",
        "\n",
        "\n",
        "#optionally, load from HF\n",
        "#emotion_classifier = pipeline('text-classification', model='charreaubell/bert-emotion', use_auth_token=True)\n",
        "\n",
        "#get output\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "fc9c9088-d638-4c52-aa47-096fcc4e0065",
      "metadata": {
        "id": "fc9c9088-d638-4c52-aa47-096fcc4e0065"
      },
      "outputs": [],
      "source": [
        "#inspect results\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Homework\n",
        "Now that you've learned the essentials of training, let's take a moment to reflect on what we've learned, augment our knowledge, and avoid some known pitfalls."
      ],
      "metadata": {
        "id": "Z0KxvgpvS1A0"
      },
      "id": "Z0KxvgpvS1A0"
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Tokenizers"
      ],
      "metadata": {
        "id": "ffzg7nOnU5BS"
      },
      "id": "ffzg7nOnU5BS"
    },
    {
      "cell_type": "code",
      "source": [
        "#@title Tokenizers: Verifying your understanding\n",
        "\n",
        "#@markdown Make sure to run all the cells in the `Aside on Tokenizer functionality`\n",
        "#@markdown section to make sure that you understand the encoding and decoding\n",
        "#@markdown functions."
      ],
      "metadata": {
        "cellView": "form",
        "id": "p7ixNjI-TFa4"
      },
      "id": "p7ixNjI-TFa4",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#@title Updating our tokenization function\n",
        "#@markdown Using the HF API, HF course, and Tunstall text\n",
        "#@markdown determine how you would pad each input during tokenization.\n",
        "#@markdown What methods of padding and truncation are available?"
      ],
      "metadata": {
        "cellView": "form",
        "id": "XE3dw8M3TkfO"
      },
      "id": "XE3dw8M3TkfO",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#@title Expanding our knowledge of Datasets `map` method\n",
        "#@markdown What if we wanted to remove all html from our\n",
        "#@markdown data prior to tokenization? We can do this with the map\n",
        "#@markdown of Datasets. Use the following resource from the\n",
        "#@markdown [HuggingFace Course](https://huggingface.co/course/chapter5/3?fw=pt#the-map-methods-superpowers) to understand how one might\n",
        "#@markdown go about doing this. Implement this here."
      ],
      "metadata": {
        "cellView": "form",
        "id": "TSSgX61iUCbk"
      },
      "id": "TSSgX61iUCbk",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Training Arguments and the Trainer Class"
      ],
      "metadata": {
        "id": "mfwIG2r3U-om"
      },
      "id": "mfwIG2r3U-om"
    },
    {
      "cell_type": "code",
      "source": [
        "#@title `model_init`\n",
        "#@markdown Recall that we defined a `model_init` function, which we asserted\n",
        "#@markdown was because we didn't want to continue with undocumented training.\n",
        "#@markdown You took our word for this, but now you'll explore the perils of\n",
        "#@markdown this on your own.\n",
        "\n",
        "#@markdown After copying the cells required for training your model below,\n",
        "#@markdown make the following changes to the code to explore the behavior.\n",
        "#@markdown 1. Amend your training arguments by setting `push_to_hub = False`.\n",
        "#@markdown We're doing this just to reduce the time and efforts required as we're\n",
        "#@markdown exploring.\n",
        "#@markdown 2. In the instantiation of the `Trainer` class (i.e., the `trainer = Trainer(...)` lines),\n",
        "#@markdown set the model parameter to use the model that you created initially (this will likely just be `model=model`,...)\n",
        "#@markdown as an argument.\n",
        "#@markdown 3. Amend your Training Arguments to train for only 1 epoch.\n",
        "#@markdown 4. Train your model. Note the change in performance and inspect the titles of the checkpoint directories.\n",
        "#@markdown 5. Run the same cell again to continue training your model. Note the change in performance and\n",
        "#@markdown again inspect the titles of your checkpoint directories.\n",
        "\n",
        "#@markdown Consider the behavior of the above code in the context of what we experienced\n",
        "#@markdown with checkpoint directories during class. Answer the following questions:\n",
        "#@markdown 1. At step 5, how many epochs would we have trained the underlying model for?\n",
        "#@markdown 2. At step 5, what evidence do you have that you've the model for 2 epochs?\n",
        "#@markdown 3. Comment on the above question from the standpoint of reproducibility. In other words,\n",
        "#@markdown let's say you pushed this model to the Hub, and the Model Card makes it appear\n",
        "#@markdown as if you've trained for one epoch whereas you've actually trained for two.\n",
        "#@markdown How do you think this will affect others trying to reproduce your results?\n",
        "#@markdown 4. Comment on the advantages of the `model_init` method during development\n",
        "#@markdown as we're initially exploring parameters for our models."
      ],
      "metadata": {
        "cellView": "form",
        "id": "hanuhLiTU9ip"
      },
      "id": "hanuhLiTU9ip",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#@title Model metrics\n",
        "#@markdown Let's say that we wanted to see the precision and recall\n",
        "#@markdown for each of the individual classes rather than the `macro` averaging\n",
        "#@markdown as we saw in our current `compute_metrics` function we've written.\n",
        "\n",
        "#@markdown Using the same sklearn functions (or not, but sklearn may make it easier),\n",
        "#@markdown return the precision and recall for each individual class label in addition\n",
        "#@markdown to the macro scores. Recall that what is returned from the `compute_metrics`\n",
        "#@markdown function must be a dictionary."
      ],
      "metadata": {
        "cellView": "form",
        "id": "IACwqovwVdTS"
      },
      "id": "IACwqovwVdTS",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#@title TrainingArguments parameters\n",
        "#@markdown We've logged, saved, and evaluated at each epoch. However,\n",
        "#@markdown if we have an extremely large dataset, seeing one or more of these\n",
        "#@markdown at the end of each epoch (e.g., if it takes 3 hours to make it through\n",
        "#@markdown a single epoch) may conflict with our desire to monitor our model training.\n",
        "\n",
        "#@markdown 1. Using the TrainingArguments API, change your model to log, evaluate, and save\n",
        "#@markdown every 200 steps rather than every epoch.\n",
        "#@markdown 2. How does this change your checkpointing directories?\n",
        "#@markdown 3. How does this influence the intervals of evaluation?"
      ],
      "metadata": {
        "cellView": "form",
        "id": "-3pwBcDVYgQp"
      },
      "id": "-3pwBcDVYgQp",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "f8bd89a2-bf3c-453a-8b53-50acce211130",
      "metadata": {
        "id": "f8bd89a2-bf3c-453a-8b53-50acce211130"
      },
      "source": [
        "## Your research project\n",
        "You've successfully trained a model - great job!! Now, let's focus on what YOU need to do for your task. What is the task that best describes what you're after? Using the [Transformer Notebooks](https://huggingface.co/docs/transformers/notebooks) and use the `Open in Colab` badge, explore what this task looks like. Note that even if your modality is different, you may be able to directly still use these notebooks with a few changes!"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "colab": {
      "name": "1-text-simple-classification-training.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "thJhDvdZF-3U",
        "d0a371ec-98f5-4be7-b085-a2eed2f9c3d3",
        "d558ecf4-e35b-4afa-a1aa-ca16e86ab432",
        "15655b8b-45bb-4c3c-8ad8-dcd869beda5d",
        "08845526-78fb-4096-bb4b-1ca13c7eda12",
        "3da664a0-b63c-450a-ae74-bab101d55ea1",
        "oXe8fUReQyUu",
        "b6ba1189-3e5f-42ae-bfae-e79f85a439f5",
        "94abc5f1-076e-4167-9c85-26d77404797f",
        "Z0KxvgpvS1A0",
        "f8bd89a2-bf3c-453a-8b53-50acce211130"
      ],
      "include_colab_link": true
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 5
}