Fine_tuning_PictoBERT_(colourful_semantics).ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github//jayralencar/pictoBERT/blob/main/Fine_tuning_PictoBERT_(colourful_semantics).ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/jayralencar/pictoBERT/blob/main/Fine_tuning_PictoBERT_(colourful_semantics).ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ],
      "metadata": {
        "id": "e3vkdJH-WRO0"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j5NKA86VI8OW"
      },
      "source": [
        "# Fine-tuning PictoBERT for Pictogram Prediction Based on a Grammatical Structure\n",
        "\n",
        "This notebook presents the procedures for adopting and fine-tuning PictoBERT to perform pictogram prediction based on a Grammatical Structure (cf. Section 5.2.1 in the paper)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UmYAQOAEKCoX"
      },
      "source": [
        "## Verify if you are using a GPU\n",
        "\n",
        "For fine-tuning, we suggest using a GPU, which can allow you to train fast."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rvO0F2-rKCBA",
        "outputId": "1a38d0a9-15c4-4cef-82a2-bf7e6ae63382"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Thu Mar 24 00:30:13 2022       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   53C    P8    33W / 149W |      0MiB / 11441MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ]
        }
      ],
      "source": [
        "gpu_info = !nvidia-smi\n",
        "gpu_info = '\\n'.join(gpu_info)\n",
        "if gpu_info.find('failed') >= 0:\n",
        "  print('Select the Runtime > \"Change runtime type\" menu to enable a GPU accelerator, ')\n",
        "  print('and then re-execute this cell.')\n",
        "else:\n",
        "  print(gpu_info)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k0X2KpHMJp9p"
      },
      "source": [
        "## Install dependencies"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LXi7kFKIIZjh",
        "outputId": "e2b62678-1084-4a26-afb2-e6bf7d551371"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting pytorch_lightning==1.2.10\n",
            "  Downloading pytorch_lightning-1.2.10-py3-none-any.whl (841 kB)\n",
            "\u001b[K     |████████████████████████████████| 841 kB 4.3 MB/s \n",
            "\u001b[?25hCollecting transformers\n",
            "  Downloading transformers-4.17.0-py3-none-any.whl (3.8 MB)\n",
            "\u001b[K     |████████████████████████████████| 3.8 MB 39.0 MB/s \n",
            "\u001b[?25hCollecting tokenizers\n",
            "  Downloading tokenizers-0.11.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.5 MB)\n",
            "\u001b[K     |████████████████████████████████| 6.5 MB 36.7 MB/s \n",
            "\u001b[?25hCollecting future>=0.17.1\n",
            "  Downloading future-0.18.2.tar.gz (829 kB)\n",
            "\u001b[K     |████████████████████████████████| 829 kB 52.7 MB/s \n",
            "\u001b[?25hRequirement already satisfied: tensorboard!=2.5.0,>=2.2.0 in /usr/local/lib/python3.7/dist-packages (from pytorch_lightning==1.2.10) (2.8.0)\n",
            "Collecting PyYAML!=5.4.*,>=5.1\n",
            "  Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)\n",
            "\u001b[K     |████████████████████████████████| 596 kB 46.3 MB/s \n",
            "\u001b[?25hRequirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from pytorch_lightning==1.2.10) (21.3)\n",
            "Requirement already satisfied: tqdm>=4.41.0 in /usr/local/lib/python3.7/dist-packages (from pytorch_lightning==1.2.10) (4.63.0)\n",
            "Collecting fsspec[http]>=0.8.1\n",
            "  Downloading fsspec-2022.2.0-py3-none-any.whl (134 kB)\n",
            "\u001b[K     |████████████████████████████████| 134 kB 51.1 MB/s \n",
            "\u001b[?25hCollecting torchmetrics==0.2.0\n",
            "  Downloading torchmetrics-0.2.0-py3-none-any.whl (176 kB)\n",
            "\u001b[K     |████████████████████████████████| 176 kB 37.4 MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.16.6 in /usr/local/lib/python3.7/dist-packages (from pytorch_lightning==1.2.10) (1.21.5)\n",
            "Requirement already satisfied: torch>=1.4 in /usr/local/lib/python3.7/dist-packages (from pytorch_lightning==1.2.10) (1.10.0+cu111)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from fsspec[http]>=0.8.1->pytorch_lightning==1.2.10) (2.23.0)\n",
            "Collecting aiohttp\n",
            "  Downloading aiohttp-3.8.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.1 MB 41.4 MB/s \n",
            "\u001b[?25hRequirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (1.0.1)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (1.35.0)\n",
            "Requirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (3.17.3)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (0.4.6)\n",
            "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (1.0.0)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (0.37.1)\n",
            "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (0.6.1)\n",
            "Requirement already satisfied: grpcio>=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (1.44.0)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (57.4.0)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (1.8.1)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (3.3.6)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from absl-py>=0.4->tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (1.15.0)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (4.8)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (0.2.8)\n",
            "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (4.2.4)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (1.3.1)\n",
            "Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (4.11.3)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (3.7.0)\n",
            "Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (3.10.0.2)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (0.4.8)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->fsspec[http]>=0.8.1->pytorch_lightning==1.2.10) (2.10)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->fsspec[http]>=0.8.1->pytorch_lightning==1.2.10) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->fsspec[http]>=0.8.1->pytorch_lightning==1.2.10) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->fsspec[http]>=0.8.1->pytorch_lightning==1.2.10) (2021.10.8)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard!=2.5.0,>=2.2.0->pytorch_lightning==1.2.10) (3.2.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers) (3.6.0)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (2019.12.20)\n",
            "Collecting huggingface-hub<1.0,>=0.1.0\n",
            "  Downloading huggingface_hub-0.4.0-py3-none-any.whl (67 kB)\n",
            "\u001b[K     |████████████████████████████████| 67 kB 4.6 MB/s \n",
            "\u001b[?25hCollecting sacremoses\n",
            "  Downloading sacremoses-0.0.49-py3-none-any.whl (895 kB)\n",
            "\u001b[K     |████████████████████████████████| 895 kB 37.6 MB/s \n",
            "\u001b[?25hRequirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->pytorch_lightning==1.2.10) (3.0.7)\n",
            "Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->fsspec[http]>=0.8.1->pytorch_lightning==1.2.10) (2.0.12)\n",
            "Collecting asynctest==0.13.0\n",
            "  Downloading asynctest-0.13.0-py3-none-any.whl (26 kB)\n",
            "Collecting frozenlist>=1.1.1\n",
            "  Downloading frozenlist-1.3.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (144 kB)\n",
            "\u001b[K     |████████████████████████████████| 144 kB 27.3 MB/s \n",
            "\u001b[?25hCollecting async-timeout<5.0,>=4.0.0a3\n",
            "  Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB)\n",
            "Collecting multidict<7.0,>=4.5\n",
            "  Downloading multidict-6.0.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (94 kB)\n",
            "\u001b[K     |████████████████████████████████| 94 kB 321 kB/s \n",
            "\u001b[?25hCollecting yarl<2.0,>=1.0\n",
            "  Downloading yarl-1.7.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (271 kB)\n",
            "\u001b[K     |████████████████████████████████| 271 kB 40.4 MB/s \n",
            "\u001b[?25hCollecting aiosignal>=1.1.2\n",
            "  Downloading aiosignal-1.2.0-py3-none-any.whl (8.2 kB)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->fsspec[http]>=0.8.1->pytorch_lightning==1.2.10) (21.4.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (7.1.2)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.1.0)\n",
            "Building wheels for collected packages: future\n",
            "  Building wheel for future (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491070 sha256=ea711ac3a455ddf1da7128dcf0f8c354cfb83b1440d8c7360ec90c097a9d22c3\n",
            "  Stored in directory: /root/.cache/pip/wheels/56/b0/fe/4410d17b32f1f0c3cf54cdfb2bc04d7b4b8f4ae377e2229ba0\n",
            "Successfully built future\n",
            "Installing collected packages: multidict, frozenlist, yarl, asynctest, async-timeout, aiosignal, PyYAML, fsspec, aiohttp, torchmetrics, tokenizers, sacremoses, huggingface-hub, future, transformers, pytorch-lightning\n",
            "  Attempting uninstall: PyYAML\n",
            "    Found existing installation: PyYAML 3.13\n",
            "    Uninstalling PyYAML-3.13:\n",
            "      Successfully uninstalled PyYAML-3.13\n",
            "  Attempting uninstall: future\n",
            "    Found existing installation: future 0.16.0\n",
            "    Uninstalling future-0.16.0:\n",
            "      Successfully uninstalled future-0.16.0\n",
            "Successfully installed PyYAML-6.0 aiohttp-3.8.1 aiosignal-1.2.0 async-timeout-4.0.2 asynctest-0.13.0 frozenlist-1.3.0 fsspec-2022.2.0 future-0.18.2 huggingface-hub-0.4.0 multidict-6.0.2 pytorch-lightning-1.2.10 sacremoses-0.0.49 tokenizers-0.11.6 torchmetrics-0.2.0 transformers-4.17.0 yarl-1.7.2\n"
          ]
        }
      ],
      "source": [
        "!pip install pytorch_lightning==1.2.10 transformers tokenizers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oyjV_jWuKSdC"
      },
      "source": [
        "## Download files"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ENKpnOXHKUQY"
      },
      "source": [
        "### Download PictoBERT versions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MykdzK47KWsH",
        "outputId": "5f76106e-a3de-4c1f-8fdf-80c5f0d3c7b5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2022-03-24 00:30:40--  http://jayr.clubedosgeeks.com.br/pictobert/pictobert-large-contextual.zip\n",
            "Resolving jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)... 192.185.214.132\n",
            "Connecting to jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)|192.185.214.132|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 1180295214 (1.1G) [application/zip]\n",
            "Saving to: ‘pictobert-large-contextual.zip’\n",
            "\n",
            "pictobert-large-con 100%[===================>]   1.10G  15.1MB/s    in 76s     \n",
            "\n",
            "2022-03-24 00:31:57 (14.8 MB/s) - ‘pictobert-large-contextual.zip’ saved [1180295214/1180295214]\n",
            "\n",
            "--2022-03-24 00:31:57--  http://jayr.clubedosgeeks.com.br/pictobert/pictobert-large-gloss.zip\n",
            "Resolving jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)... 192.185.214.132\n",
            "Connecting to jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)|192.185.214.132|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 1180231318 (1.1G) [application/zip]\n",
            "Saving to: ‘pictobert-large-gloss.zip’\n",
            "\n",
            "pictobert-large-glo 100%[===================>]   1.10G  15.6MB/s    in 76s     \n",
            "\n",
            "2022-03-24 00:33:13 (14.9 MB/s) - ‘pictobert-large-gloss.zip’ saved [1180231318/1180231318]\n",
            "\n",
            "Archive:  pictobert-large-contextual.zip\n",
            "   creating: pictobert/\n",
            "  inflating: pictobert/config.json   \n",
            "  inflating: pictobert/pytorch_model.bin  \n",
            "Archive:  pictobert-large-gloss.zip\n",
            "   creating: pictobert-gloss/\n",
            "  inflating: pictobert-gloss/config.json  \n",
            "  inflating: pictobert-gloss/pytorch_model.bin  \n"
          ]
        }
      ],
      "source": [
        "!wget http://jayr.clubedosgeeks.com.br/pictobert/pictobert-large-contextual.zip\n",
        "!wget http://jayr.clubedosgeeks.com.br/pictobert/pictobert-large-gloss.zip\n",
        "\n",
        "!unzip pictobert-large-contextual.zip\n",
        "!unzip pictobert-large-gloss.zip"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VJCQAOWlK1nw"
      },
      "source": [
        "### Download PictoBERT Tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L2QQ817dK1Gu",
        "outputId": "701efeb2-1dcc-4561-ec11-9adf7df40a71"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2022-03-24 00:33:41--  http://jayr.clubedosgeeks.com.br/pictobert/childes_all_new.json\n",
            "Resolving jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)... 192.185.214.132\n",
            "Connecting to jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)|192.185.214.132|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 332233 (324K) [application/json]\n",
            "Saving to: ‘childes_all_new.json’\n",
            "\n",
            "childes_all_new.jso 100%[===================>] 324.45K   435KB/s    in 0.7s    \n",
            "\n",
            "2022-03-24 00:33:43 (435 KB/s) - ‘childes_all_new.json’ saved [332233/332233]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget http://jayr.clubedosgeeks.com.br/pictobert/childes_all_new.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mXcMyOieMNaA"
      },
      "source": [
        "### Download dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Udm_AFpeMQM8",
        "outputId": "941e12cd-ae66-410c-e654-2e9ec919ac1f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2022-03-24 00:33:43--  http://jayr.clubedosgeeks.com.br/pictobert/sem_childes_uk_clean_2.txt\n",
            "Resolving jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)... 192.185.214.132\n",
            "Connecting to jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)|192.185.214.132|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 4796378 (4.6M) [text/plain]\n",
            "Saving to: ‘sem_childes_uk_clean_2.txt’\n",
            "\n",
            "sem_childes_uk_clea 100%[===================>]   4.57M  3.03MB/s    in 1.5s    \n",
            "\n",
            "2022-03-24 00:33:45 (3.03 MB/s) - ‘sem_childes_uk_clean_2.txt’ saved [4796378/4796378]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget http://jayr.clubedosgeeks.com.br/pictobert/sem_childes_uk_clean_2.txt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fxSUu4bsQy_V"
      },
      "source": [
        "### Download ARES embeddings"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W5UXRIZgQ1nk",
        "outputId": "d81a26c4-8436-43a2-f039-fb5117a86e23"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2022-03-24 00:33:45--  http://jayr.clubedosgeeks.com.br/pictobert/ares_1024_gloss.bin\n",
            "Resolving jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)... 192.185.214.132\n",
            "Connecting to jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)|192.185.214.132|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 852260167 (813M) [application/octet-stream]\n",
            "Saving to: ‘ares_1024_gloss.bin’\n",
            "\n",
            "ares_1024_gloss.bin 100%[===================>] 812.78M  15.4MB/s    in 56s     \n",
            "\n",
            "2022-03-24 00:34:42 (14.4 MB/s) - ‘ares_1024_gloss.bin’ saved [852260167/852260167]\n",
            "\n",
            "--2022-03-24 00:34:42--  http://jayr.clubedosgeeks.com.br/pictobert/ares_1024.bin\n",
            "Resolving jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)... 192.185.214.132\n",
            "Connecting to jayr.clubedosgeeks.com.br (jayr.clubedosgeeks.com.br)|192.185.214.132|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 852260167 (813M) [application/octet-stream]\n",
            "Saving to: ‘ares_1024.bin’\n",
            "\n",
            "ares_1024.bin       100%[===================>] 812.78M  14.6MB/s    in 56s     \n",
            "\n",
            "2022-03-24 00:35:39 (14.5 MB/s) - ‘ares_1024.bin’ saved [852260167/852260167]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget http://jayr.clubedosgeeks.com.br/pictobert/ares_1024_gloss.bin\n",
        "!wget http://jayr.clubedosgeeks.com.br/pictobert/ares_1024.bin"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jeo25-H7MahB"
      },
      "source": [
        "## Create tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hgRXK5nGMdaA",
        "outputId": "ace0e01e-5086-4d02-c337-212d4caf4879"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "86692"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import re\n",
        "special_tokens = [\"[UNK]\",\"[SEP]\", \"[CLS]\", \"[PAD]\", \"[MASK]\"]\n",
        "examples = open(\"./sem_childes_uk_clean_2.txt\",'r').readlines()\n",
        "examples = [l.rstrip() for l in examples]\n",
        "sentences = [[j for j in re.sub(r'\\s+', ' ', l).split(\" \") if j not in special_tokens ] for l in examples]\n",
        "len(sentences)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DJZo00QWNcvg"
      },
      "outputs": [],
      "source": [
        "from tokenizers import Tokenizer\n",
        "from tokenizers.models import WordLevel\n",
        "from tokenizers.pre_tokenizers import WhitespaceSplit\n",
        "from tokenizers.processors import BertProcessing\n",
        "\n",
        "special_tokens = [\"[UNK]\",\"[SEP]\", \"[CLS]\", \"[PAD]\", \"[MASK]\"]\n",
        "\n",
        "sense_tokenizer = Tokenizer(WordLevel(unk_token=\"[UNK]\"))\n",
        "sense_tokenizer.add_special_tokens(special_tokens)\n",
        "sense_tokenizer.pre_tokenizer = WhitespaceSplit()\n",
        "\n",
        "sep_token = \"[SEP]\"\n",
        "cls_token = \"[CLS]\"\n",
        "pad_token = \"[PAD]\"\n",
        "unk_token = \"[UNK]\"\n",
        "sep_token_id = sense_tokenizer.token_to_id(str(sep_token))\n",
        "cls_token_id = sense_tokenizer.token_to_id(str(cls_token))\n",
        "pad_token_id = sense_tokenizer.token_to_id(str(pad_token))\n",
        "unk_token_id = sense_tokenizer.token_to_id(str(unk_token))\n",
        "\n",
        "sense_tokenizer.post_processor = BertProcessing(\n",
        "                (str(sep_token), sep_token_id), (str(cls_token), cls_token_id)\n",
        "            )\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b0sezghTNfiN",
        "outputId": "460a7a67-5bdc-4475-d700-78b57c71689b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Vocab size:  4960\n"
          ]
        }
      ],
      "source": [
        "from tokenizers.trainers import WordLevelTrainer\n",
        "g = WordLevelTrainer(special_tokens=[\"[UNK]\"])\n",
        "sense_tokenizer.train_from_iterator(sentences, trainer=g)\n",
        "print(\"Vocab size: \", sense_tokenizer.get_vocab_size())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NV7ceOH2N5B3"
      },
      "outputs": [],
      "source": [
        "sense_tokenizer.save(\"./cs_tokenizer.json\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sq8IyiylOAnw"
      },
      "source": [
        "## Dataset preparation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5Gqdj6mOOGwg",
        "outputId": "48212721-3404-48f2-c256-98a4a58fb353"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(69353, 8670, 8669)"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "TEST_SIZE = 0.2\n",
        "from sklearn.model_selection import train_test_split\n",
        "train_idx, val_idx = train_test_split(list(range(len(examples))), test_size=TEST_SIZE, random_state=8)\n",
        "test_idx, val_idx = train_test_split(val_idx, test_size=0.5, random_state=8)\n",
        "\n",
        "import numpy as np\n",
        "train_examples = np.array(examples).take(train_idx)\n",
        "val_examples = np.array(examples).take(val_idx)\n",
        "test_examples = np.array(examples).take(test_idx)\n",
        "len(train_examples),len(val_examples), len(test_examples)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "R0cFxIQgOKPt"
      },
      "outputs": [],
      "source": [
        "TOKENIZER_PATH = \"./cs_tokenizer.json\" # you can change this path to use your custom tokenizer\n",
        "\n",
        "from transformers import PreTrainedTokenizerFast\n",
        "\n",
        "loaded_tokenizer = PreTrainedTokenizerFast(tokenizer_file=TOKENIZER_PATH)\n",
        "loaded_tokenizer.pad_token = \"[PAD]\"\n",
        "loaded_tokenizer.sep_token = \"[SEP]\"\n",
        "loaded_tokenizer.mask_token = \"[MASK]\"\n",
        "loaded_tokenizer.cls_token = \"[CLS]\"\n",
        "loaded_tokenizer.unk_token = \"[UNK]\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fWbX_FptOTer"
      },
      "outputs": [],
      "source": [
        "max_len = 9\n",
        "\n",
        "def tokenize_function(tokenizer,examples):\n",
        "    # Remove empty lines\n",
        "    examples = [line for line in examples if len(line) > 0 and not line.isspace()]\n",
        "    bert = tokenizer(\n",
        "        examples,\n",
        "        padding=\"max_length\",\n",
        "        max_length=max_len,\n",
        "        return_special_tokens_mask=True,\n",
        "        truncation=True\n",
        "    )\n",
        "    for i, data in enumerate(bert['input_ids']):\n",
        "      a = np.array(data)\n",
        "      special_tokens_mask = np.array(bert['special_tokens_mask'][i])\n",
        "      special_tokens_mask[a == 3] = 1\n",
        "      bert['special_tokens_mask'][i] = special_tokens_mask\n",
        "    return bert"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "h4frNRl5OV5e"
      },
      "outputs": [],
      "source": [
        "train_tokenized_examples = tokenize_function(loaded_tokenizer,train_examples)\n",
        "val_tokenized_examples = tokenize_function(loaded_tokenizer,val_examples)\n",
        "test_tokenized_examples = tokenize_function(loaded_tokenizer,test_examples)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0NnxhOGIOj7R"
      },
      "outputs": [],
      "source": [
        "from torch import tensor\n",
        "def make_dict(examples):\n",
        "  return {\n",
        "      \"input_ids\": examples.input_ids,\n",
        "      \"attention_mask\":examples.attention_mask,\n",
        "      \"special_tokens_mask\":examples.special_tokens_mask,\n",
        "  }"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5IJ_MtztOlrl"
      },
      "outputs": [],
      "source": [
        "import pickle\n",
        "\n",
        "TRAIN_DATA_PATH = \"./CS_new_train_data.pt\"\n",
        "TEST_DATA_PATH = \"./CS_new_test_data.pt\"\n",
        "VAL_DATA_PATH = \"./CS_new_val_data.pt\"\n",
        "\n",
        "pickle.dump(make_dict(train_tokenized_examples),open(TRAIN_DATA_PATH,'wb'))\n",
        "pickle.dump(make_dict(val_tokenized_examples),open(TEST_DATA_PATH,'wb'))\n",
        "pickle.dump(make_dict(test_tokenized_examples),open(VAL_DATA_PATH ,'wb'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0pfZPCvYOuhW"
      },
      "source": [
        "## PictoBERT adaptation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ew9WcCm-Px4D"
      },
      "source": [
        "### Load pictoBERT"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yxXIHZfjPI4y"
      },
      "outputs": [],
      "source": [
        "from transformers import BertForMaskedLM\n",
        "pictobert_version = \"contextual\" #@param [\"contextual\",\"gloss\"]\n",
        "\n",
        "if pictobert_version == \"contextual\":\n",
        "  pictobert = BertForMaskedLM.from_pretrained(\"./pictobert\")\n",
        "else:\n",
        "  pictobert = BertForMaskedLM.from_pretrained(\"./pictobert-gloss\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PCxJm0HqQVsz"
      },
      "source": [
        "### Load BERT\n",
        "At this point, we need to also load BERT as its embeddings matrix may be used to calculate the input embeddings of tokens that are not on PictoBERT or ARES vocabulary."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 217,
          "referenced_widgets": [
            "85af54996dcd46089abb26bbb0833e83",
            "4c2c5e484fe944e998d46235ff136f00",
            "5366c11796954bb29881be11d380ad62",
            "2cd68396c27347f19abf47a7e922040c",
            "0334e69e025b4b028fc255f1a4df4878",
            "3d830da7f0c04b1397fac5b00619b361",
            "4f399070f9c548c09ff4f31a34377234",
            "3316cccc1d32425aaca54700371107aa",
            "fb9ed4fba5834e5e9b3c59380449310b",
            "c73aec2a41644b24a0cb124259ced412",
            "fd480869d3d5465b867a9d34381e0e93",
            "fb3270a5c0614dc5ba176308da5089f1",
            "fc4037deffd846b9b57f74c5815b457e",
            "d833eebee00e40b9802f4a30580cff99",
            "f09a4668bf0b496cbbf2f8e56b164586",
            "1f67f97ed9e14c7f80429733661722d8",
            "150d78644f1441e5ad17d60de7d05a6f",
            "1e37d144820f473fa3ae5e2b6a9c5482",
            "67eaa4c7b5d34319991c16d2ebebb1ef",
            "ebc703e43e114afc92c912856aabf8f2",
            "1ae5f60a3bcd4bd69c5346aee02306b7",
            "bae54f49e2324f83a938886d2d226fcc",
            "b841b23ac7824f69bf8c3005753aefaa",
            "a91017f4fcaf4982b283d683102280f0",
            "54cd7dfe2d574586a8cfa48a6783485f",
            "736c1911fcfe40b2b8fae73289b85a9f",
            "fde746ea5be34191b54505ed4f4c3c11",
            "f705caaef7b5494a96b38481b80393e4",
            "c0f154355b0c4f4b9eb29b7f9e08adbf",
            "7b1916e0d1304ef1b3c6c8e22557040b",
            "3727bc42eed546d6b7c79f01358c5a95",
            "f866ae904ca944e6a6ffa02a09561e07",
            "8843a8006bb744ad9ea0343855f23ce4",
            "1c7ab9ff9fc74d5c9eebcb907e84f3b4",
            "ade1c758d8194ea0acaa202fc23244a5",
            "62d6a85f53bc4021871104b4f51dc4d1",
            "800a713c47a04a2cb3d21d0893f9986d",
            "81bc75dc927e4909a1ff00e468f3bab2",
            "de1dc8ff06644791bd290b601b73a629",
            "519b598696ee485aaec8cc12f15d26ed",
            "e91e985a7b3c419a8de0af03cfbd5cf0",
            "5857f5a04f884b36a34bb8f7afcb062a",
            "055c4e4aaf8b4e72ad58313acddac5be",
            "c10c451a3f7f4c388a6139bc3030896a"
          ]
        },
        "id": "kd9zHiaTQlrl",
        "outputId": "fdd02c3f-b127-4207-8055-6edc2e4d8756"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "85af54996dcd46089abb26bbb0833e83",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/226k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "fb3270a5c0614dc5ba176308da5089f1",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/28.0 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "b841b23ac7824f69bf8c3005753aefaa",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/571 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "1c7ab9ff9fc74d5c9eebcb907e84f3b4",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/1.25G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at bert-large-uncased were not used when initializing BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']\n",
            "- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ]
        }
      ],
      "source": [
        "import torch\n",
        "from transformers import BertTokenizer\n",
        "\n",
        "pretrained_w = 'bert-large-uncased'\n",
        "tokenizer_bert = BertTokenizer.from_pretrained(pretrained_w)\n",
        "bert = BertForMaskedLM.from_pretrained(pretrained_w)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KNqzccsyP0ZD"
      },
      "source": [
        "### Load PictoBERT tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Qttb3OVDP3Iw"
      },
      "outputs": [],
      "source": [
        "TOKENIZER_PATH = \"./childes_all_new.json\" # you can change this path to use your custom tokenizer\n",
        "\n",
        "from transformers import PreTrainedTokenizerFast\n",
        "\n",
        "pictobert_tokenizer = PreTrainedTokenizerFast(tokenizer_file=TOKENIZER_PATH)\n",
        "pictobert_tokenizer.pad_token = \"[PAD]\"\n",
        "pictobert_tokenizer.sep_token = \"[SEP]\"\n",
        "pictobert_tokenizer.mask_token = \"[MASK]\"\n",
        "pictobert_tokenizer.cls_token = \"[CLS]\"\n",
        "pictobert_tokenizer.unk_token = \"[UNK]\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "msfLwqxHQFu3"
      },
      "source": [
        "### Update embeddings layer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "w6rkzkPHQIGo"
      },
      "outputs": [],
      "source": [
        "new_vocab = loaded_tokenizer.get_vocab()\n",
        "pictobert_vocab = pictobert_tokenizer.get_vocab()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e1HQ8t-gQNTB",
        "outputId": "7c2b5415-bc95-4686-b388-18f8cc29f3dd"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "New vocab size: 4959\n",
            "PictoBERT vocab size: 13583\n",
            "Commom: 4273\n",
            "New tokens: 686\n"
          ]
        }
      ],
      "source": [
        "in_pictobert = []\n",
        "not_in = []\n",
        "for w,idx_new in new_vocab.items():\n",
        "  idx_old = pictobert_vocab.get(w, -1)\n",
        "  if idx_old >= 0:\n",
        "    in_pictobert.append(w)\n",
        "  else:\n",
        "    not_in.append(w)\n",
        "  \n",
        "print(\"New vocab size:\",len(new_vocab))\n",
        "print(\"PictoBERT vocab size:\", len(pictobert_vocab))\n",
        "print(\"Commom:\",len(in_pictobert))\n",
        "print(\"New tokens:\",len(not_in))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SgGL1fImRV74"
      },
      "outputs": [],
      "source": [
        "bert_embeddings = bert.get_input_embeddings()\n",
        "pictobert_embeddings = pictobert.get_input_embeddings()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fZ_HRNk-RdUK"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "from gensim.models import KeyedVectors\n",
        "\n",
        "if pictobert_version == \"contextual\":\n",
        "  ares = KeyedVectors.load_word2vec_format(\"/content/ares_1024.bin\", binary=True)\n",
        "else:\n",
        "  ares = KeyedVectors.load_word2vec_format(\"/content/ares_1024_gloss.bin\", binary=True)\n",
        "ares_contextual_mean = torch.tensor(ares.vectors).mean(0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PGZ6xBAJRqnc"
      },
      "outputs": [],
      "source": [
        "bert_vocab = tokenizer_bert.get_vocab()\n",
        "# new_vocab = cs_tokenizer.get_vocab()\n",
        "\n",
        "bert_wgts = bert.get_input_embeddings().weight.clone().detach()\n",
        "pictobert_wgts = pictobert.get_input_embeddings().weight.clone().detach()\n",
        "\n",
        "new_vocab_size = len(loaded_tokenizer.vocab)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9ihYrllhRvIR"
      },
      "outputs": [],
      "source": [
        "new_wgts = pictobert_wgts.new_zeros(new_vocab_size,pictobert_wgts.size(1))\n",
        "\n",
        "same_tokens_list = list()\n",
        "different_tokens_list = list()\n",
        "received_mean = list()\n",
        "\n",
        "for w,idx_new in new_vocab.items():\n",
        "  idx_pictobert = pictobert_vocab.get(w,-1)\n",
        "  if idx_pictobert >= 0:\n",
        "    new_wgts[idx_new] = pictobert_wgts[idx_pictobert]\n",
        "  elif w in ares:\n",
        "    new_wgts[idx_new] = torch.tensor(ares[w])\n",
        "  else:\n",
        "    w_ = \" \".join(w.split(\"_\"))\n",
        "    tokenized = tokenizer_bert(w_,add_special_tokens=False,return_tensors='pt')\n",
        "    v_ = bert_embeddings(tokenized.input_ids[0]).mean(0)\n",
        "    new_wgts[idx_new] = v_"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "23orGr6_R43g"
      },
      "outputs": [],
      "source": [
        "from torch import nn\n",
        "\n",
        "new_wte = nn.Embedding(new_vocab_size,pictobert_wgts.size(1)) # new embeddings\n",
        "new_wte.weight.data.normal_(mean=0,std=pictobert.config.initializer_range) \n",
        "new_wte.weight.data = new_wgts\n",
        "\n",
        "pictobert.resize_token_embeddings(len(loaded_tokenizer))\n",
        "pictobert.set_input_embeddings(new_wte)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8AbTVk9jR8uB"
      },
      "outputs": [],
      "source": [
        "MODEL_NAME = \"./pictobert-CS-{0}\".format(pictobert_version)\n",
        "pictobert.save_pretrained(MODEL_NAME)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xTE-K7c2S45R"
      },
      "source": [
        "## Train Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b09muoWuTCTj"
      },
      "source": [
        "### Define constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yGgR7RzgTE-d"
      },
      "outputs": [],
      "source": [
        "TOKENIZER_PATH = \"./cs_tokenizer.json\"\n",
        "LOGS_PATH = \"./logs\"\n",
        "CHECKPOINTS_PATH = \"./checkpoints\"\n",
        "\n",
        "TRAIN_DATASET_PATH = \"./CS_new_train_data.pt\"\n",
        "VAL_DATASET_PATH = \"./CS_new_val_data.pt\"\n",
        "TEST_DATASET_PATH = \"./CS_new_test_data.pt\"\n",
        "\n",
        "MAX_EPOCHS = 10\n",
        "WARMUP_STEPS = int(MAX_EPOCHS * 0.15)\n",
        "BATCH_SIZE = 32\n",
        "NUM_WORKERS = 4\n",
        "GPUS = 1\n",
        "LEARNING_RATE = 1e-06\n",
        "ACCUMULATE_GRAD_BATCHES = 4\n",
        "LOGGER_VERSION = '1e06'\n",
        "LOGGER_INFO = \"first_run\"\n",
        "FREEZE_TO = None\n",
        "MLM_PROBABILITY= 0.15"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nHrYe1-yTkgO"
      },
      "source": [
        "### Load Data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RSOe-Ab3Tl9x"
      },
      "outputs": [],
      "source": [
        "from torch.utils.data import Dataset, Subset\n",
        "from torch import tensor\n",
        "from sklearn.model_selection import train_test_split\n",
        "import pickle\n",
        "\n",
        "class MyDataset(Dataset):\n",
        "  def __init__(self, examples):\n",
        "    \n",
        "    self.input_ids = examples['input_ids']\n",
        "    self.attention_mask = examples['attention_mask']\n",
        "    self.special_tokens_mask = examples['special_tokens_mask']\n",
        "    self.labels = None\n",
        "    if 'labels' in examples:\n",
        "      self.labels = examples['labels']\n",
        "  \n",
        "  def __len__(self):\n",
        "    return len(self.input_ids)\n",
        "  \n",
        "  def __getitem__(self, idx):\n",
        "    input_ids = tensor(self.input_ids[idx])\n",
        "    attention_mask = tensor(self.attention_mask[idx])\n",
        "    special_tokens_mask = tensor(self.special_tokens_mask[idx])\n",
        "\n",
        "    out_dict = {\n",
        "      \"input_ids\":input_ids, \n",
        "      \"attention_mask\":attention_mask, \n",
        "      \"special_tokens_mask\":special_tokens_mask\n",
        "    }\n",
        "\n",
        "    if self.labels is not None:\n",
        "      out_dict['labels'] = self.labels[idx]\n",
        "\n",
        "    return out_dict\n",
        "\n",
        "\n",
        "tds = pickle.load(open(TRAIN_DATASET_PATH,'rb'))\n",
        "train_dataset = MyDataset(tds)\n",
        "\n",
        "vds = pickle.load(open(VAL_DATASET_PATH,'rb'))\n",
        "val_dataset = MyDataset(vds)\n",
        "\n",
        "tsds = pickle.load(open(TEST_DATASET_PATH,'rb'))\n",
        "test_dataset = MyDataset(tsds)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5m_2NYveToNg"
      },
      "outputs": [],
      "source": [
        "from transformers import DataCollatorForLanguageModeling\n",
        "data_collator = DataCollatorForLanguageModeling(tokenizer=loaded_tokenizer, mlm_probability=MLM_PROBABILITY)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2IcHMF1OT32i"
      },
      "source": [
        "## Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "q8JF0I7AT5Nz"
      },
      "outputs": [],
      "source": [
        "from argparse import ArgumentParser\n",
        "import math\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from transformers import get_polynomial_decay_schedule_with_warmup\n",
        "from transformers import AdamW\n",
        "import pytorch_lightning as pl\n",
        "from sklearn.metrics import accuracy_score\n",
        "from pytorch_lightning.callbacks import ModelCheckpoint\n",
        "from scipy import stats\n",
        "import numpy as np\n",
        "import torch.nn.functional as F\n",
        "from torch.utils.data import DataLoader\n",
        "from torch.utils.data.distributed import DistributedSampler\n",
        "\n",
        "percentiles = [\n",
        "  {\n",
        "    \"percentil\":'1%',\n",
        "    \"z\": stats.norm.ppf(1-0.01),\n",
        "  },\n",
        "  {\n",
        "    \"percentil\":'5%',\n",
        "    \"z\": stats.norm.ppf(1-0.05),\n",
        "  },\n",
        "  {\n",
        "    \"percentil\":'10%',\n",
        "    \"z\": stats.norm.ppf(1-0.1),\n",
        "  },\n",
        "  {\n",
        "    \"percentil\":'15%',\n",
        "    \"z\": stats.norm.ppf(1-0.15),\n",
        "  },\n",
        "  {\n",
        "    \"percentil\":'20%',\n",
        "    \"z\": stats.norm.ppf(1-0.2),\n",
        "  },\n",
        "]\n",
        "\n",
        "firsts = [6,\t12,\t24,\t32,\t40]\n",
        "\n",
        "from transformers import BertLMHeadModel\n",
        "from transformers import BertForMaskedLM\n",
        "\n",
        "class LitBertClassifier(pl.LightningModule):\n",
        "    def __init__(self, pretrained_model_name='bert-large-uncased'):\n",
        "        super().__init__()\n",
        "        self.save_hyperparameters()\n",
        "        self.batch_size = 128\n",
        "        self.lr = LEARNING_RATE\n",
        "        self.train_dataset = train_dataset\n",
        "        self.val_dataset = val_dataset\n",
        "        self.test_dataset = test_dataset\n",
        "        self.bert = BertForMaskedLM.from_pretrained(pretrained_model_name)\n",
        "      \n",
        "    \n",
        "    def freeze_to(self, layers):\n",
        "      for param in self.bert.bert.encoder.layer[:layers].parameters():\n",
        "        param.requires_grad = False\n",
        "\n",
        "\n",
        "    def forward(self, input_ids, attention_mask, labels=None):\n",
        "        if labels == None:\n",
        "            return self.bert(\n",
        "                input_ids=input_ids,\n",
        "                attention_mask=attention_mask,\n",
        "            )    \n",
        "        return self.bert(\n",
        "            input_ids=input_ids,\n",
        "            attention_mask=attention_mask,\n",
        "            labels = labels\n",
        "        )\n",
        "\n",
        "    def training_step(self, batch, batch_idx):\n",
        "        outputs = self._shared_step(batch, batch_idx)\n",
        "        loss = outputs[0]\n",
        "\n",
        "        self.log(\"train_loss\", loss, on_epoch=True, prog_bar=True,)\n",
        "\n",
        "        return loss\n",
        "\n",
        "    def train_dataloader(self):\n",
        "      train_dataloader = DataLoader(\n",
        "          train_dataset,\n",
        "          batch_size=self.batch_size,\n",
        "          num_workers=NUM_WORKERS,\n",
        "          pin_memory=True,\n",
        "          collate_fn=data_collator,\n",
        "          drop_last = True,\n",
        "          shuffle=True\n",
        "      )\n",
        "      return train_dataloader\n",
        "    \n",
        "    def test_dataloader(self):\n",
        "      return  DataLoader(\n",
        "          test_dataset,\n",
        "          batch_size=self.batch_size,\n",
        "          num_workers=NUM_WORKERS,\n",
        "          pin_memory=True,\n",
        "          drop_last = True,\n",
        "      )\n",
        "    \n",
        "    def val_dataloader(self):\n",
        "      return  DataLoader(\n",
        "          val_dataset,\n",
        "          batch_size=self.batch_size,\n",
        "          num_workers=NUM_WORKERS,\n",
        "          pin_memory=True,\n",
        "          collate_fn=data_collator,\n",
        "          drop_last = True,\n",
        "      )\n",
        "\n",
        "    def get_accuracy(self, batch, results):\n",
        "        y_true = tensor([]).to(torch.device(\"cuda:0\"))\n",
        "        y_pred = tensor([]).to(torch.device(\"cuda:0\"))\n",
        "        for idx, ipids in enumerate(batch[\"input_ids\"]):\n",
        "\n",
        "            idxs = (ipids == loaded_tokenizer.mask_token_id).nonzero()\n",
        "            if idxs.size()[0] > 0:\n",
        "                device = \"cuda:0\"\n",
        "                y_true = torch.cat((y_true, batch['labels'][idx][idxs].resize(1, idxs.size()[0])[0]))\n",
        "                # y_true = y_true + batch['labels'][idx][idxs].resize(1, idxs.size()[0])[0].tolist()\n",
        "\n",
        "                idxs_2 = tensor([a[0] for a in idxs])\n",
        "                idxs_2 = idxs_2.to(device)\n",
        "                res = torch.index_select(results[idx],0,idxs_2).argmax(1)\n",
        "\n",
        "                y_pred = torch.cat((y_pred,res))\n",
        "        if len(y_pred) == 0:\n",
        "            return None\n",
        "        return accuracy(tensor(y_true),tensor(y_pred))\n",
        "\n",
        "    def validation_step(self, batch, batch_idx):\n",
        "        with torch.no_grad():\n",
        "          result = self._shared_step(batch, batch_idx)\n",
        "          loss = result[0].detach()\n",
        "          \n",
        "          predictions = result[1]\n",
        "          labels = batch['labels']\n",
        "          masked = batch['input_ids']\n",
        "          n = masked.detach().cpu().numpy()\n",
        "          predicted = predictions.detach().cpu().numpy()[n == loaded_tokenizer.mask_token_id]\n",
        "          accuracy = accuracy_score(labels[n == loaded_tokenizer.mask_token_id].detach().cpu(), np.argmax(predicted, axis=1))\n",
        "          self.log(\"val_acc\", accuracy, on_epoch=True, prog_bar=True,)\n",
        "\n",
        "\n",
        "          return {\n",
        "              \"val_loss\":loss\n",
        "          }\n",
        "    \n",
        "    def validation_epoch_end(self, outputs):\n",
        "        val_loss = torch.stack([x['val_loss'] for x in outputs]).mean()\n",
        "        self.log(\"val_loss\", val_loss, on_epoch=True, prog_bar=True,)\n",
        "    \n",
        "    \n",
        "    def test_step(self, batch, batch_idx):\n",
        "        with torch.no_grad():\n",
        "          result = self._shared_step(batch, batch_idx)\n",
        "          loss = result[0].detach()\n",
        "          perplexity = torch.exp(loss)\n",
        "          self.log(\"test_ppl\", perplexity, on_epoch=True, prog_bar=True,)\n",
        "          self.log(\"test_loss\", loss, on_epoch=True, prog_bar=True,)\n",
        "\n",
        "          predictions = F.softmax(result[1], dim=-1)\n",
        "          labels = batch['labels']\n",
        "          masked = batch['input_ids']\n",
        "          n = masked.detach().cpu().numpy()\n",
        "          predicted = predictions.detach().cpu().numpy()[n == loaded_tokenizer.mask_token_id]\n",
        "          \n",
        "          accuracy = accuracy_score(labels[n == loaded_tokenizer.mask_token_id].detach().cpu(), np.argmax(predicted, axis=1))\n",
        "          self.log(\"test_acc\", accuracy, on_epoch=True, prog_bar=True,)\n",
        "          \n",
        "          for percentil in percentiles:\n",
        "            count = 0\n",
        "            z = percentil['z']\n",
        "            for i, data in enumerate(predicted):\n",
        "              mean = data[data>0].mean()\n",
        "              std = data[data>0].std()\n",
        "              x = (z * std) + mean\n",
        "              if labels[masked == loaded_tokenizer.mask_token_id][i].item() in (predicted[i] > x).nonzero()[0]:\n",
        "                count += 1\n",
        "            \n",
        "\n",
        "            isin = count/predicted.shape[0]\n",
        "\n",
        "            self.log(\"test_\"+percentil['percentil'], isin, on_epoch=True, prog_bar=True,)\n",
        "          \n",
        "          ids = np.argsort(-1*predicted,axis=1)\n",
        "          \n",
        "          for first in firsts:\n",
        "            count = 0\n",
        "            for i, data in enumerate(ids):\n",
        "              if labels[masked == loaded_tokenizer.mask_token_id][i].item() in data[:first]:\n",
        "                count += 1\n",
        "            isin = count/predicted.shape[0]\n",
        "\n",
        "            self.log(\"test_\"+str(first), isin, on_epoch=True, prog_bar=True,)\n",
        "            \n",
        "\n",
        "          return {\n",
        "              \"test_ppl\":perplexity,\n",
        "              \"test_loss\":loss,\n",
        "              \"test_count\":count,\n",
        "          }\n",
        "    \n",
        "\n",
        "    def _shared_step(self, batch, batch_idx):\n",
        "        input_ids = batch[\"input_ids\"]\n",
        "        attention_mask = batch[\"attention_mask\"]\n",
        "        labels = batch[\"labels\"]\n",
        "\n",
        "        outputs = self.forward(\n",
        "            input_ids=input_ids,\n",
        "            attention_mask=attention_mask,\n",
        "            labels=labels\n",
        "        )\n",
        "\n",
        "        return outputs\n",
        "\n",
        "    def configure_optimizers(self):\n",
        "      optimizer = AdamW(self.parameters(), lr=self.lr,betas=(0.9, 0.999), eps=1e-08, weight_decay=0.01)\n",
        "      scheduler = {\n",
        "          'scheduler': get_polynomial_decay_schedule_with_warmup(optimizer, num_warmup_steps=WARMUP_STEPS,num_training_steps=MAX_EPOCHS,lr_end=1e-09),\n",
        "          'name': 'lr'\n",
        "      }\n",
        "      return [optimizer],[scheduler]\n",
        "    \n",
        "    def backward(self, loss, optimizer, idx):\n",
        "        loss.backward()\n",
        "    \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L5iPs7_MUsSO"
      },
      "source": [
        "### Logger"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "InGvOSK4UtTS"
      },
      "outputs": [],
      "source": [
        "from pytorch_lightning import loggers as pl_loggers\n",
        "from pytorch_lightning.callbacks import LearningRateMonitor\n",
        "\n",
        "tb_logger = pl_loggers.TensorBoardLogger(LOGS_PATH,name=LOGGER_INFO, version=LOGGER_VERSION)\n",
        "lr_monitor = LearningRateMonitor(logging_interval='epoch')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8f66UvqjU6rR"
      },
      "source": [
        "### Checkpointing"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wg2r8WpuU8qI"
      },
      "outputs": [],
      "source": [
        "checkpoint_callback = ModelCheckpoint(\n",
        "    dirpath=CHECKPOINTS_PATH,\n",
        "    filename='bert-large-{epoch:02d}-{val_loss:.2f}',\n",
        "    mode='min',\n",
        ")\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9iIk2RDxVBnE",
        "outputId": "1ac5036d-b885-4893-f1ad-58c69ceab083"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "GPU available: True, used: True\n",
            "TPU available: False, using: 0 TPU cores\n",
            "Using native 16bit precision.\n"
          ]
        }
      ],
      "source": [
        "trainer = pl.Trainer(\n",
        "      accelerator='ddp',\n",
        "      max_epochs=MAX_EPOCHS,\n",
        "      logger=tb_logger,\n",
        "      gpus=GPUS,\n",
        "      callbacks=[checkpoint_callback, lr_monitor],\n",
        "      precision=16,\n",
        "      auto_scale_batch_size=\"binsearch\"\n",
        "  )"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "K75ZxhR-W9-X"
      },
      "outputs": [],
      "source": [
        "to_train = LitBertClassifier(MODEL_NAME)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zOlNCmtAXGOg"
      },
      "outputs": [],
      "source": [
        "trainer.tune(to_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vH-6ctPRXNT8"
      },
      "outputs": [],
      "source": [
        "trainer.fit(to_train)"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "Fine-tuning PictoBERT (colourful semantics).ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "0334e69e025b4b028fc255f1a4df4878": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "055c4e4aaf8b4e72ad58313acddac5be": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "150d78644f1441e5ad17d60de7d05a6f": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1ae5f60a3bcd4bd69c5346aee02306b7": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1c7ab9ff9fc74d5c9eebcb907e84f3b4": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_ade1c758d8194ea0acaa202fc23244a5",
              "IPY_MODEL_62d6a85f53bc4021871104b4f51dc4d1",
              "IPY_MODEL_800a713c47a04a2cb3d21d0893f9986d"
            ],
            "layout": "IPY_MODEL_81bc75dc927e4909a1ff00e468f3bab2"
          }
        },
        "1e37d144820f473fa3ae5e2b6a9c5482": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1f67f97ed9e14c7f80429733661722d8": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2cd68396c27347f19abf47a7e922040c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c73aec2a41644b24a0cb124259ced412",
            "placeholder": "​",
            "style": "IPY_MODEL_fd480869d3d5465b867a9d34381e0e93",
            "value": " 226k/226k [00:00&lt;00:00, 325kB/s]"
          }
        },
        "3316cccc1d32425aaca54700371107aa": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3727bc42eed546d6b7c79f01358c5a95": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "3d830da7f0c04b1397fac5b00619b361": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4c2c5e484fe944e998d46235ff136f00": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3d830da7f0c04b1397fac5b00619b361",
            "placeholder": "​",
            "style": "IPY_MODEL_4f399070f9c548c09ff4f31a34377234",
            "value": "Downloading: 100%"
          }
        },
        "4f399070f9c548c09ff4f31a34377234": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "519b598696ee485aaec8cc12f15d26ed": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "5366c11796954bb29881be11d380ad62": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3316cccc1d32425aaca54700371107aa",
            "max": 231508,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_fb9ed4fba5834e5e9b3c59380449310b",
            "value": 231508
          }
        },
        "54cd7dfe2d574586a8cfa48a6783485f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7b1916e0d1304ef1b3c6c8e22557040b",
            "max": 571,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3727bc42eed546d6b7c79f01358c5a95",
            "value": 571
          }
        },
        "5857f5a04f884b36a34bb8f7afcb062a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "62d6a85f53bc4021871104b4f51dc4d1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e91e985a7b3c419a8de0af03cfbd5cf0",
            "max": 1344997306,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_5857f5a04f884b36a34bb8f7afcb062a",
            "value": 1344997306
          }
        },
        "67eaa4c7b5d34319991c16d2ebebb1ef": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "736c1911fcfe40b2b8fae73289b85a9f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f866ae904ca944e6a6ffa02a09561e07",
            "placeholder": "​",
            "style": "IPY_MODEL_8843a8006bb744ad9ea0343855f23ce4",
            "value": " 571/571 [00:00&lt;00:00, 10.5kB/s]"
          }
        },
        "7b1916e0d1304ef1b3c6c8e22557040b": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "800a713c47a04a2cb3d21d0893f9986d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_055c4e4aaf8b4e72ad58313acddac5be",
            "placeholder": "​",
            "style": "IPY_MODEL_c10c451a3f7f4c388a6139bc3030896a",
            "value": " 1.25G/1.25G [00:38&lt;00:00, 34.5MB/s]"
          }
        },
        "81bc75dc927e4909a1ff00e468f3bab2": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "85af54996dcd46089abb26bbb0833e83": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_4c2c5e484fe944e998d46235ff136f00",
              "IPY_MODEL_5366c11796954bb29881be11d380ad62",
              "IPY_MODEL_2cd68396c27347f19abf47a7e922040c"
            ],
            "layout": "IPY_MODEL_0334e69e025b4b028fc255f1a4df4878"
          }
        },
        "8843a8006bb744ad9ea0343855f23ce4": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a91017f4fcaf4982b283d683102280f0": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f705caaef7b5494a96b38481b80393e4",
            "placeholder": "​",
            "style": "IPY_MODEL_c0f154355b0c4f4b9eb29b7f9e08adbf",
            "value": "Downloading: 100%"
          }
        },
        "ade1c758d8194ea0acaa202fc23244a5": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_de1dc8ff06644791bd290b601b73a629",
            "placeholder": "​",
            "style": "IPY_MODEL_519b598696ee485aaec8cc12f15d26ed",
            "value": "Downloading: 100%"
          }
        },
        "b841b23ac7824f69bf8c3005753aefaa": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_a91017f4fcaf4982b283d683102280f0",
              "IPY_MODEL_54cd7dfe2d574586a8cfa48a6783485f",
              "IPY_MODEL_736c1911fcfe40b2b8fae73289b85a9f"
            ],
            "layout": "IPY_MODEL_fde746ea5be34191b54505ed4f4c3c11"
          }
        },
        "bae54f49e2324f83a938886d2d226fcc": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c0f154355b0c4f4b9eb29b7f9e08adbf": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c10c451a3f7f4c388a6139bc3030896a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c73aec2a41644b24a0cb124259ced412": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d833eebee00e40b9802f4a30580cff99": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_67eaa4c7b5d34319991c16d2ebebb1ef",
            "max": 28,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_ebc703e43e114afc92c912856aabf8f2",
            "value": 28
          }
        },
        "de1dc8ff06644791bd290b601b73a629": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e91e985a7b3c419a8de0af03cfbd5cf0": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ebc703e43e114afc92c912856aabf8f2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "f09a4668bf0b496cbbf2f8e56b164586": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1ae5f60a3bcd4bd69c5346aee02306b7",
            "placeholder": "​",
            "style": "IPY_MODEL_bae54f49e2324f83a938886d2d226fcc",
            "value": " 28.0/28.0 [00:00&lt;00:00, 682B/s]"
          }
        },
        "f705caaef7b5494a96b38481b80393e4": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f866ae904ca944e6a6ffa02a09561e07": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fb3270a5c0614dc5ba176308da5089f1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_fc4037deffd846b9b57f74c5815b457e",
              "IPY_MODEL_d833eebee00e40b9802f4a30580cff99",
              "IPY_MODEL_f09a4668bf0b496cbbf2f8e56b164586"
            ],
            "layout": "IPY_MODEL_1f67f97ed9e14c7f80429733661722d8"
          }
        },
        "fb9ed4fba5834e5e9b3c59380449310b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "fc4037deffd846b9b57f74c5815b457e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_150d78644f1441e5ad17d60de7d05a6f",
            "placeholder": "​",
            "style": "IPY_MODEL_1e37d144820f473fa3ae5e2b6a9c5482",
            "value": "Downloading: 100%"
          }
        },
        "fd480869d3d5465b867a9d34381e0e93": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "fde746ea5be34191b54505ed4f4c3c11": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}