Demos.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Demoing-notebook.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyMGt6Rsg5EGGUcbX73ILss7",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/iam-kevin/nlp-workshop/blob/master/Demos.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eAEfaDpJk9wc",
        "colab_type": "text"
      },
      "source": [
        "## Data Collection\n",
        "\n",
        "Obtaining sample data that will be used for swahili showcasing\n",
        "\n",
        "[OSCAR Corpus](https://oscar-corpus.com/)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a6YkOUD2kcTX",
        "colab_type": "code",
        "outputId": "8041f63f-1eef-4aaf-e8da-55b45ffe5dfb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "!wget https://traces1.inria.fr/oscar/files/compressed-orig/sw.txt.gz \n",
        "!gzip -d sw.txt.gz"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-06-13 12:05:04--  https://traces1.inria.fr/oscar/files/compressed-orig/sw.txt.gz\n",
            "Resolving traces1.inria.fr (traces1.inria.fr)... 128.93.193.43\n",
            "Connecting to traces1.inria.fr (traces1.inria.fr)|128.93.193.43|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 3745590 (3.6M) [application/gzip]\n",
            "Saving to: ‘sw.txt.gz’\n",
            "\n",
            "sw.txt.gz           100%[===================>]   3.57M  4.23MB/s    in 0.8s    \n",
            "\n",
            "2020-06-13 12:05:06 (4.23 MB/s) - ‘sw.txt.gz’ saved [3745590/3745590]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qBQa41r8q1w8",
        "colab_type": "text"
      },
      "source": [
        "## Data Preparation\n",
        "\n",
        "We load up the `sw.txt` file, make the clean and store the clean format in `sw-clean.txt`. The clean format is what we would then use in the processes to follow.\n",
        "\n",
        "The Cleaning processes invloves\n",
        "- Converting all characters to lower case,\n",
        "- Skip any lines with odd punctuations. (i.e. anything that is not `. , ( ) : \" '`)\n",
        "- Removing the known punctuation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qnXb6O_Pq1o9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# load the data\n",
        "sw_lines = None\n",
        "with open('sw.txt', 'r', encoding='utf-8') as txtf:\n",
        "    sw_lines = txtf.readlines()\n",
        "\n",
        "# change all characters to lower case\n",
        "sw_lines = [line.lower() for line in sw_lines]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9rZuMsXSwW3k",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import re\n",
        "\n",
        "# only take lines with allowed characters\n",
        "allowed_punctuations = r'\\.\\,\\(\\)\\:\\\"\\''\n",
        "\n",
        "sw_clean_lines = []\n",
        "for line in sw_lines:\n",
        "    # take only the lines with the allowed characters\n",
        "    if not bool(re.findall('((?![{}\\s]+)\\W+)'.format(allowed_punctuations), line)):\n",
        "        sw_clean_lines.append(line)\n",
        "\n",
        "# remove the punctuation characters\n",
        "sw_clean_lines = [re.sub(f'[{allowed_punctuations}]', '', line) for line in sw_clean_lines]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "agMLPo-Nz3Rf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "# save the file in sw-clean\n",
        "with open('sw-clean.txt', 'w', encoding='utf-8') as wf:\n",
        "    wf.write(\"\".join(sw_clean_lines))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EtcC0-eHmnVa",
        "colab_type": "text"
      },
      "source": [
        "## Embeddings\n",
        "\n",
        "These forms of representing a character, word or even a sentence using a vector, so that it can be used by a machine learning algorithm to be able to perform other tasks.\n",
        "\n",
        "3 different forms of algorithms will be used\n",
        "1. `Word2Vec` Embeddings\n",
        "2. `FastText` Embeddings\n",
        "3. `Flair` Embeddings"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LH1a89mgAnxa",
        "colab_type": "code",
        "outputId": "63cee257-6a4e-404b-97fd-ee231f811bc3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 663
        }
      },
      "source": [
        "import re\n",
        "\n",
        "# load the clean data to train on\n",
        "with open('sw-clean.txt', 'r', encoding='utf-8') as cf:\n",
        "    corpus = cf.readlines()\n",
        "\n",
        "corpus = [ re.split('\\s+', text) for text in corpus ]\n",
        "corpus[0]"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['miripuko',\n",
              " 'hiyo',\n",
              " 'inakuja',\n",
              " 'mwanzoni',\n",
              " 'mwa',\n",
              " 'wiki',\n",
              " 'takatifu',\n",
              " 'kuelekea',\n",
              " 'pasaka',\n",
              " 'na',\n",
              " 'ikiwa',\n",
              " 'ni',\n",
              " 'wiki',\n",
              " 'chache',\n",
              " 'tu',\n",
              " 'kabla',\n",
              " 'ya',\n",
              " 'papa',\n",
              " 'francis',\n",
              " 'kuanza',\n",
              " 'ziara',\n",
              " 'yake',\n",
              " 'katika',\n",
              " 'nchi',\n",
              " 'hiyo',\n",
              " 'yenye',\n",
              " 'idadi',\n",
              " 'kubwa',\n",
              " 'kabisa',\n",
              " 'ya',\n",
              " 'watu',\n",
              " 'katika',\n",
              " 'ulimwengu',\n",
              " 'wa',\n",
              " 'nchi',\n",
              " 'za',\n",
              " 'kiarabu',\n",
              " '']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PbN-1cu2nzun",
        "colab_type": "text"
      },
      "source": [
        "### Using Word2Vec: (CBOW: Continuos Bag-Of-Words)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qpRkLSmplmF9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from gensim.models import Word2Vec\n",
        "\n",
        "w2v_model = Word2Vec(corpus, size=100, window=5, workers=4)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tcFTAAAF6v97",
        "colab_type": "code",
        "outputId": "467b3120-e05c-4af5-be10-6c82fc5d25a5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        }
      },
      "source": [
        "w2v_model.wv.most_similar('kiswahili')"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
            "  if np.issubdtype(vec.dtype, np.int):\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('kiingereza', 0.8887909650802612),\n",
              " ('lugha', 0.80258709192276),\n",
              " ('biblia', 0.7429032325744629),\n",
              " ('matusi', 0.7223591804504395),\n",
              " ('idhaa', 0.7117130160331726),\n",
              " ('maandiko', 0.7038394212722778),\n",
              " ('tafsiri', 0.6838766932487488),\n",
              " ('fizikia', 0.6656526923179626),\n",
              " ('kigiriki', 0.66070955991745),\n",
              " ('kingereza', 0.6592473387718201)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xtMAWvLBn5Ug",
        "colab_type": "text"
      },
      "source": [
        "### Using FastText"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eakhpidYn4Y0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from gensim.models import FastText\n",
        "\n",
        "ft_model = FastText(size=100, window=3, min_count=1)  # instantiate\n",
        "ft_model.build_vocab(sentences=corpus)\n",
        "ft_model.train(sentences=corpus, total_examples=len(corpus), epochs=10)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9k2PqlSgDlem",
        "colab_type": "code",
        "outputId": "6dce3e12-f2c2-4f99-9d17-4aed91401cd9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        }
      },
      "source": [
        "ft_model.wv.most_similar('kiswahili')"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
            "  if np.issubdtype(vec.dtype, np.int):\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('swahili', 0.892357349395752),\n",
              " ('kiingreza', 0.8167586922645569),\n",
              " ('kiingereza', 0.8099594712257385),\n",
              " ('sahili', 0.803668200969696),\n",
              " ('kiinjili', 0.801956057548523),\n",
              " ('kiswa', 0.7989442944526672),\n",
              " ('udahili', 0.7952920794487),\n",
              " ('jahili', 0.7944253087043762),\n",
              " ('kinawli', 0.7877073287963867),\n",
              " ('kingereza', 0.7729043364524841)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r-VPGF2qoP18",
        "colab_type": "text"
      },
      "source": [
        "### Using Flair"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x4AnTvelL4Di",
        "colab_type": "text"
      },
      "source": [
        "Installing the `flair` package"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cnVvYOchoPb8",
        "colab_type": "code",
        "outputId": "dc397998-7362-43bd-e68c-5a4e58d08f20",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "!pip install flair"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting flair\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/f1/0e/38d173e7a5b595e108c7d7a31f7b4d88fb93192f3b12a78998ff500c5203/flair-0.5-py3-none-any.whl (334kB)\n",
            "\r\u001b[K     |█                               | 10kB 16.0MB/s eta 0:00:01\r\u001b[K     |██                              | 20kB 2.9MB/s eta 0:00:01\r\u001b[K     |███                             | 30kB 3.9MB/s eta 0:00:01\r\u001b[K     |████                            | 40kB 4.3MB/s eta 0:00:01\r\u001b[K     |█████                           | 51kB 3.3MB/s eta 0:00:01\r\u001b[K     |█████▉                          | 61kB 3.9MB/s eta 0:00:01\r\u001b[K     |██████▉                         | 71kB 4.1MB/s eta 0:00:01\r\u001b[K     |███████▉                        | 81kB 4.4MB/s eta 0:00:01\r\u001b[K     |████████▉                       | 92kB 4.7MB/s eta 0:00:01\r\u001b[K     |█████████▉                      | 102kB 4.5MB/s eta 0:00:01\r\u001b[K     |██████████▊                     | 112kB 4.5MB/s eta 0:00:01\r\u001b[K     |███████████▊                    | 122kB 4.5MB/s eta 0:00:01\r\u001b[K     |████████████▊                   | 133kB 4.5MB/s eta 0:00:01\r\u001b[K     |█████████████▊                  | 143kB 4.5MB/s eta 0:00:01\r\u001b[K     |██████████████▊                 | 153kB 4.5MB/s eta 0:00:01\r\u001b[K     |███████████████▋                | 163kB 4.5MB/s eta 0:00:01\r\u001b[K     |████████████████▋               | 174kB 4.5MB/s eta 0:00:01\r\u001b[K     |█████████████████▋              | 184kB 4.5MB/s eta 0:00:01\r\u001b[K     |██████████████████▋             | 194kB 4.5MB/s eta 0:00:01\r\u001b[K     |███████████████████▋            | 204kB 4.5MB/s eta 0:00:01\r\u001b[K     |████████████████████▌           | 215kB 4.5MB/s eta 0:00:01\r\u001b[K     |█████████████████████▌          | 225kB 4.5MB/s eta 0:00:01\r\u001b[K     |██████████████████████▌         | 235kB 4.5MB/s eta 0:00:01\r\u001b[K     |███████████████████████▌        | 245kB 4.5MB/s eta 0:00:01\r\u001b[K     |████████████████████████▌       | 256kB 4.5MB/s eta 0:00:01\r\u001b[K     |█████████████████████████▍      | 266kB 4.5MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▍     | 276kB 4.5MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▍    | 286kB 4.5MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▍   | 296kB 4.5MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▍  | 307kB 4.5MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▎ | 317kB 4.5MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▎| 327kB 4.5MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 337kB 4.5MB/s \n",
            "\u001b[?25hCollecting bpemb>=0.2.9\n",
            "  Downloading https://files.pythonhosted.org/packages/bc/70/468a9652095b370f797ed37ff77e742b11565c6fd79eaeca5f2e50b164a7/bpemb-0.3.0-py3-none-any.whl\n",
            "Collecting langdetect\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/56/a3/8407c1e62d5980188b4acc45ef3d94b933d14a2ebc9ef3505f22cf772570/langdetect-1.0.8.tar.gz (981kB)\n",
            "\u001b[K     |████████████████████████████████| 983kB 12.2MB/s \n",
            "\u001b[?25hRequirement already satisfied: torch>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from flair) (1.5.0+cu101)\n",
            "Collecting sqlitedict>=1.6.0\n",
            "  Downloading https://files.pythonhosted.org/packages/0f/1c/c757b93147a219cf1e25cef7e1ad9b595b7f802159493c45ce116521caff/sqlitedict-1.6.0.tar.gz\n",
            "Collecting mpld3==0.3\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/91/95/a52d3a83d0a29ba0d6898f6727e9858fe7a43f6c2ce81a5fe7e05f0f4912/mpld3-0.3.tar.gz (788kB)\n",
            "\u001b[K     |████████████████████████████████| 798kB 36.4MB/s \n",
            "\u001b[?25hCollecting pytest>=5.3.2\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/9f/f3/0a83558da436a081344aa6c8b85ea5b5f05071214106036ce341b7769b0b/pytest-5.4.3-py3-none-any.whl (248kB)\n",
            "\u001b[K     |████████████████████████████████| 256kB 39.4MB/s \n",
            "\u001b[?25hRequirement already satisfied: hyperopt>=0.1.1 in /usr/local/lib/python3.6/dist-packages (from flair) (0.1.2)\n",
            "Requirement already satisfied: tqdm>=4.26.0 in /usr/local/lib/python3.6/dist-packages (from flair) (4.41.1)\n",
            "Collecting deprecated>=1.2.4\n",
            "  Downloading https://files.pythonhosted.org/packages/76/a1/05d7f62f956d77b23a640efc650f80ce24483aa2f85a09c03fb64f49e879/Deprecated-1.2.10-py2.py3-none-any.whl\n",
            "Requirement already satisfied: tabulate in /usr/local/lib/python3.6/dist-packages (from flair) (0.8.7)\n",
            "Collecting transformers>=2.10.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/48/35/ad2c5b1b8f99feaaf9d7cdadaeef261f098c6e1a6a2935d4d07662a6b780/transformers-2.11.0-py3-none-any.whl (674kB)\n",
            "\u001b[K     |████████████████████████████████| 675kB 46.5MB/s \n",
            "\u001b[?25hRequirement already satisfied: scikit-learn>=0.21.3 in /usr/local/lib/python3.6/dist-packages (from flair) (0.22.2.post1)\n",
            "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from flair) (2.8.1)\n",
            "Requirement already satisfied: regex in /usr/local/lib/python3.6/dist-packages (from flair) (2019.12.20)\n",
            "Requirement already satisfied: gensim>=3.4.0 in /usr/local/lib/python3.6/dist-packages (from flair) (3.6.0)\n",
            "Requirement already satisfied: matplotlib>=2.2.3 in /usr/local/lib/python3.6/dist-packages (from flair) (3.2.1)\n",
            "Collecting segtok>=1.5.7\n",
            "  Downloading https://files.pythonhosted.org/packages/41/08/582dab5f4b1d5ca23bc6927b4bb977c8ff7f3a87a3b98844ef833e2f5623/segtok-1.5.10.tar.gz\n",
            "Collecting sentencepiece\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d4/a4/d0a884c4300004a78cca907a6ff9a5e9fe4f090f5d95ab341c53d28cbc58/sentencepiece-0.1.91-cp36-cp36m-manylinux1_x86_64.whl (1.1MB)\n",
            "\u001b[K     |████████████████████████████████| 1.1MB 43.4MB/s \n",
            "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from bpemb>=0.2.9->flair) (2.23.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from bpemb>=0.2.9->flair) (1.18.4)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from langdetect->flair) (1.12.0)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from torch>=1.1.0->flair) (0.16.0)\n",
            "Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/lib/python3.6/dist-packages (from pytest>=5.3.2->flair) (8.3.0)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from pytest>=5.3.2->flair) (20.4)\n",
            "Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.6/dist-packages (from pytest>=5.3.2->flair) (19.3.0)\n",
            "Collecting pluggy<1.0,>=0.12\n",
            "  Downloading https://files.pythonhosted.org/packages/a0/28/85c7aa31b80d150b772fbe4a229487bc6644da9ccb7e427dd8cc60cb8a62/pluggy-0.13.1-py2.py3-none-any.whl\n",
            "Requirement already satisfied: wcwidth in /usr/local/lib/python3.6/dist-packages (from pytest>=5.3.2->flair) (0.2.2)\n",
            "Requirement already satisfied: importlib-metadata>=0.12; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from pytest>=5.3.2->flair) (1.6.0)\n",
            "Requirement already satisfied: py>=1.5.0 in /usr/local/lib/python3.6/dist-packages (from pytest>=5.3.2->flair) (1.8.1)\n",
            "Requirement already satisfied: pymongo in /usr/local/lib/python3.6/dist-packages (from hyperopt>=0.1.1->flair) (3.10.1)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from hyperopt>=0.1.1->flair) (1.4.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.6/dist-packages (from hyperopt>=0.1.1->flair) (2.4)\n",
            "Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.6/dist-packages (from deprecated>=1.2.4->flair) (1.12.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers>=2.10.0->flair) (3.0.12)\n",
            "Collecting sacremoses\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/7d/34/09d19aff26edcc8eb2a01bed8e98f13a1537005d31e95233fd48216eed10/sacremoses-0.0.43.tar.gz (883kB)\n",
            "\u001b[K     |████████████████████████████████| 890kB 45.1MB/s \n",
            "\u001b[?25hCollecting tokenizers==0.7.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/14/e5/a26eb4716523808bb0a799fcfdceb6ebf77a18169d9591b2f46a9adb87d9/tokenizers-0.7.0-cp36-cp36m-manylinux1_x86_64.whl (3.8MB)\n",
            "\u001b[K     |████████████████████████████████| 3.8MB 43.6MB/s \n",
            "\u001b[?25hRequirement already satisfied: dataclasses; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from transformers>=2.10.0->flair) (0.7)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn>=0.21.3->flair) (0.15.1)\n",
            "Requirement already satisfied: smart-open>=1.2.1 in /usr/local/lib/python3.6/dist-packages (from gensim>=3.4.0->flair) (2.0.0)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.2.3->flair) (2.4.7)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.2.3->flair) (1.2.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.2.3->flair) (0.10.0)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->bpemb>=0.2.9->flair) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->bpemb>=0.2.9->flair) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->bpemb>=0.2.9->flair) (2.9)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->bpemb>=0.2.9->flair) (2020.4.5.1)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata>=0.12; python_version < \"3.8\"->pytest>=5.3.2->flair) (3.1.0)\n",
            "Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx->hyperopt>=0.1.1->flair) (4.4.2)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers>=2.10.0->flair) (7.1.2)\n",
            "Requirement already satisfied: boto in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.2.1->gensim>=3.4.0->flair) (2.49.0)\n",
            "Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.2.1->gensim>=3.4.0->flair) (1.13.19)\n",
            "Requirement already satisfied: botocore<1.17.0,>=1.16.19 in /usr/local/lib/python3.6/dist-packages (from boto3->smart-open>=1.2.1->gensim>=3.4.0->flair) (1.16.19)\n",
            "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->smart-open>=1.2.1->gensim>=3.4.0->flair) (0.10.0)\n",
            "Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from boto3->smart-open>=1.2.1->gensim>=3.4.0->flair) (0.3.3)\n",
            "Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.17.0,>=1.16.19->boto3->smart-open>=1.2.1->gensim>=3.4.0->flair) (0.15.2)\n",
            "Building wheels for collected packages: langdetect, sqlitedict, mpld3, segtok, sacremoses\n",
            "  Building wheel for langdetect (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for langdetect: filename=langdetect-1.0.8-cp36-none-any.whl size=993193 sha256=0f5a35e1b901b12d38e134af60584d4334f8e79952b47d8d25f887e2b8a03219\n",
            "  Stored in directory: /root/.cache/pip/wheels/8d/b3/aa/6d99de9f3841d7d3d40a60ea06e6d669e8e5012e6c8b947a57\n",
            "  Building wheel for sqlitedict (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sqlitedict: filename=sqlitedict-1.6.0-cp36-none-any.whl size=14689 sha256=9b06eb51e8ecd0c8be634d796fdab95c19ddb81121b813b5ace572527e1bddbf\n",
            "  Stored in directory: /root/.cache/pip/wheels/bd/57/d3/907c3ee02d35e66f674ad0106e61f06eeeb98f6ee66a6cc3fe\n",
            "  Building wheel for mpld3 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for mpld3: filename=mpld3-0.3-cp36-none-any.whl size=116679 sha256=1f60d94d4df99a23ea816f0bb149690502c7ed8f37883842cc76ce89ea341cfd\n",
            "  Stored in directory: /root/.cache/pip/wheels/c0/47/fb/8a64f89aecfe0059830479308ad42d62e898a3e3cefdf6ba28\n",
            "  Building wheel for segtok (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for segtok: filename=segtok-1.5.10-cp36-none-any.whl size=25020 sha256=7129a8e3d47a3b734ce0d90ffc97240430526c87ebd884ca97398d343daa8156\n",
            "  Stored in directory: /root/.cache/pip/wheels/b4/39/f6/9ca1c5cabde964d728023b5751c3a206a5c8cc40252321fb6b\n",
            "  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sacremoses: filename=sacremoses-0.0.43-cp36-none-any.whl size=893260 sha256=ac5b9ff87f46eba0f74ddd812ba88a9eb71a99d261cb2c326aef30c90bb6d67b\n",
            "  Stored in directory: /root/.cache/pip/wheels/29/3c/fd/7ce5c3f0666dab31a50123635e6fb5e19ceb42ce38d4e58f45\n",
            "Successfully built langdetect sqlitedict mpld3 segtok sacremoses\n",
            "\u001b[31mERROR: datascience 0.10.6 has requirement folium==0.2.1, but you'll have folium 0.8.3 which is incompatible.\u001b[0m\n",
            "Installing collected packages: sentencepiece, bpemb, langdetect, sqlitedict, mpld3, pluggy, pytest, deprecated, sacremoses, tokenizers, transformers, segtok, flair\n",
            "  Found existing installation: pluggy 0.7.1\n",
            "    Uninstalling pluggy-0.7.1:\n",
            "      Successfully uninstalled pluggy-0.7.1\n",
            "  Found existing installation: pytest 3.6.4\n",
            "    Uninstalling pytest-3.6.4:\n",
            "      Successfully uninstalled pytest-3.6.4\n",
            "Successfully installed bpemb-0.3.0 deprecated-1.2.10 flair-0.5 langdetect-1.0.8 mpld3-0.3 pluggy-0.13.1 pytest-5.4.3 sacremoses-0.0.43 segtok-1.5.10 sentencepiece-0.1.91 sqlitedict-1.6.0 tokenizers-0.7.0 transformers-2.11.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6sgTJYvfL8rJ",
        "colab_type": "text"
      },
      "source": [
        "Flair has a unique way of preparing data before it is being trained\n",
        "\n",
        "the folder structure must look like this:\n",
        "```\n",
        "corpus/\n",
        "corpus/train/\n",
        "corpus/train/train_split_1\n",
        "corpus/train/train_split_2\n",
        "corpus/train/...\n",
        "corpus/train/train_split_X\n",
        "corpus/test.txt\n",
        "corpus/valid.txt\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SYPJIU8ELmwl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def split_train(n_splits: int, train_text_set: list):\n",
        "    ll = len(train_text_set)\n",
        "    sectors = list(range(ll // n_splits, ll - n_splits + 2, ll // n_splits))\n",
        "\n",
        "    for i, x in enumerate(sectors):\n",
        "        if i == 0:\n",
        "            # first item\n",
        "            yield (train_text_set[:x])\n",
        "            continue\n",
        "        \n",
        "        yield (train_text_set[sectors[i - 1]:x])\n",
        "\n",
        "    # for the last one\n",
        "    yield (train_text_set[sectors[-1]:])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FHX3NKa2Rf7h",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from pathlib import Path\n",
        "\n",
        "train_test_val_ratio = 0.7\n",
        "test_val_ratio = 0.6\n",
        "\n",
        "train_split_count = int(train_test_val_ratio * len(corpus))\n",
        "train_text_set, test_val_text_set = corpus[:train_split_count], corpus[train_split_count:]\n",
        "\n",
        "test_split_count = int(test_val_ratio * len(test_val_text_set))\n",
        "test_text_set, val_text_set = test_val_text_set[:test_split_count], test_val_text_set[test_split_count:]\n",
        "\n",
        "corpus_path = Path('./corpus').joinpath('./train')\n",
        "corpus_path.mkdir(parents=True, exist_ok=True)\n",
        "\n",
        "# save the val and test corpus\n",
        "with open(corpus_path.parent.joinpath(f'valid.txt'), 'w', encoding='utf-8') as tf:\n",
        "    tf.write(\"\".join([\" \".join(text) for text in val_text_set]))\n",
        "\n",
        "with open(corpus_path.parent.joinpath(f'test.txt'), 'w', encoding='utf-8') as tf:\n",
        "    tf.write(\"\".join([\" \".join(text) for text in val_text_set]))\n",
        "\n",
        "# save the train corpus\n",
        "for ix, tsdata in enumerate(split_train(10, train_text_set)):\n",
        "    with open(corpus_path.joinpath(f'./train_split_{ix + 1}'), 'w', encoding='utf-8') as tf:\n",
        "        tf.write(\"\".join([\" \".join(text) for text in val_text_set]))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r2ydMZ2fOPMD",
        "colab_type": "text"
      },
      "source": [
        "#### Training flair's language model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dLDQw6v8LSK0",
        "colab_type": "code",
        "outputId": "13ec5a19-8799-4ffa-8ea1-8a0d2b70b824",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "from flair.data import Dictionary\n",
        "from flair.models import LanguageModel\n",
        "from flair.trainers.language_model_trainer import LanguageModelTrainer, TextCorpus\n",
        "\n",
        "# are you training a forward or backward LM?\n",
        "is_forward_lm = True\n",
        "\n",
        "# load the default character dictionary\n",
        "dictionary: Dictionary = Dictionary.load('chars')\n",
        "\n",
        "# get your corpus, process forward and at the character level\n",
        "corpus = TextCorpus('/content/corpus',\n",
        "                    dictionary,\n",
        "                    is_forward_lm,\n",
        "                    character_level=True)\n",
        "\n",
        "# instantiate your language model, set hidden size and number of layers\n",
        "language_model = LanguageModel(dictionary,\n",
        "                               is_forward_lm,\n",
        "                               hidden_size=128,\n",
        "                               nlayers=1)\n",
        "\n",
        "# train your language model\n",
        "trainer = LanguageModelTrainer(language_model, corpus)\n",
        "\n",
        "trainer.train('resources/language_model',\n",
        "              sequence_length=10,\n",
        "              mini_batch_size=10,\n",
        "              max_epochs=10,\n",
        "              num_workers=8)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-06-08 16:04:01,567 https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/models/common_characters not found in cache, downloading to /tmp/tmpb0zu5ttg\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 2887/2887 [00:00<00:00, 341405.09B/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "2020-06-08 16:04:02,176 copying /tmp/tmpb0zu5ttg to cache at /root/.flair/datasets/common_characters\n",
            "2020-06-08 16:04:02,183 removing temp file /tmp/tmpb0zu5ttg\n",
            "2020-06-08 16:04:02,228 read text file with 1 lines\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "2020-06-08 16:04:09,761 read text file with 1 lines\n",
            "2020-06-08 16:04:17,275 read text file with 1 lines\n",
            "2020-06-08 16:04:17,275 read text file with 1 lines\n",
            "2020-06-08 16:04:17,286 shuffled\n",
            "2020-06-08 16:04:17,288 shuffled\n",
            "2020-06-08 16:04:30,166 read text file with 1 lines\n",
            "2020-06-08 16:04:30,167 read text file with 1 lines\n",
            "2020-06-08 16:04:30,171 shuffled\n",
            "2020-06-08 16:04:30,181 shuffled\n",
            "2020-06-08 16:04:30,206 Sequence length is 10\n",
            "2020-06-08 16:04:30,217 Split 1\t - (16:04:30)\n",
            "2020-06-08 16:04:31,557 | split   1 / 10 |   100/11855 batches | ms/batch 13.35 | loss  2.93 | ppl    18.79\n",
            "2020-06-08 16:04:32,732 | split   1 / 10 |   200/11855 batches | ms/batch 11.67 | loss  2.36 | ppl    10.59\n",
            "2020-06-08 16:04:33,943 | split   1 / 10 |   300/11855 batches | ms/batch 12.05 | loss  2.25 | ppl     9.45\n",
            "2020-06-08 16:04:35,196 | split   1 / 10 |   400/11855 batches | ms/batch 12.41 | loss  2.17 | ppl     8.77\n",
            "2020-06-08 16:04:36,393 | split   1 / 10 |   500/11855 batches | ms/batch 11.93 | loss  2.13 | ppl     8.39\n",
            "2020-06-08 16:04:37,537 | split   1 / 10 |   600/11855 batches | ms/batch 11.39 | loss  2.02 | ppl     7.57\n",
            "2020-06-08 16:04:38,716 | split   1 / 10 |   700/11855 batches | ms/batch 11.75 | loss  1.87 | ppl     6.50\n",
            "2020-06-08 16:04:39,959 | split   1 / 10 |   800/11855 batches | ms/batch 12.36 | loss  1.88 | ppl     6.58\n",
            "2020-06-08 16:04:41,215 | split   1 / 10 |   900/11855 batches | ms/batch 12.50 | loss  1.74 | ppl     5.70\n",
            "2020-06-08 16:04:42,427 | split   1 / 10 |  1000/11855 batches | ms/batch 12.08 | loss  1.82 | ppl     6.14\n",
            "2020-06-08 16:04:43,637 | split   1 / 10 |  1100/11855 batches | ms/batch 12.04 | loss  1.75 | ppl     5.74\n",
            "2020-06-08 16:04:44,927 | split   1 / 10 |  1200/11855 batches | ms/batch 12.89 | loss  1.73 | ppl     5.63\n",
            "2020-06-08 16:04:46,213 | split   1 / 10 |  1300/11855 batches | ms/batch 12.78 | loss  1.79 | ppl     5.98\n",
            "2020-06-08 16:04:47,351 | split   1 / 10 |  1400/11855 batches | ms/batch 11.37 | loss  1.85 | ppl     6.35\n",
            "2020-06-08 16:04:48,540 | split   1 / 10 |  1500/11855 batches | ms/batch 11.82 | loss  1.93 | ppl     6.92\n",
            "2020-06-08 16:04:49,824 | split   1 / 10 |  1600/11855 batches | ms/batch 12.83 | loss  1.79 | ppl     5.96\n",
            "2020-06-08 16:04:51,071 | split   1 / 10 |  1700/11855 batches | ms/batch 12.46 | loss  1.73 | ppl     5.64\n",
            "2020-06-08 16:04:52,331 | split   1 / 10 |  1800/11855 batches | ms/batch 12.52 | loss  1.94 | ppl     6.96\n",
            "2020-06-08 16:04:53,603 | split   1 / 10 |  1900/11855 batches | ms/batch 12.66 | loss  1.81 | ppl     6.09\n",
            "2020-06-08 16:04:53,686 read text file with 1 lines\n",
            "2020-06-08 16:04:53,717 shuffled\n",
            "2020-06-08 16:04:54,857 | split   1 / 10 |  2000/11855 batches | ms/batch 12.52 | loss  1.70 | ppl     5.50\n",
            "2020-06-08 16:04:55,990 | split   1 / 10 |  2100/11855 batches | ms/batch 11.32 | loss  1.73 | ppl     5.66\n",
            "2020-06-08 16:04:57,147 | split   1 / 10 |  2200/11855 batches | ms/batch 11.53 | loss  1.71 | ppl     5.53\n",
            "2020-06-08 16:04:58,296 | split   1 / 10 |  2300/11855 batches | ms/batch 11.47 | loss  1.76 | ppl     5.82\n",
            "2020-06-08 16:04:59,454 | split   1 / 10 |  2400/11855 batches | ms/batch 11.56 | loss  1.88 | ppl     6.55\n",
            "2020-06-08 16:05:00,576 | split   1 / 10 |  2500/11855 batches | ms/batch 11.21 | loss  1.74 | ppl     5.71\n",
            "2020-06-08 16:05:01,688 | split   1 / 10 |  2600/11855 batches | ms/batch 11.11 | loss  1.71 | ppl     5.55\n",
            "2020-06-08 16:05:02,779 | split   1 / 10 |  2700/11855 batches | ms/batch 10.89 | loss  1.70 | ppl     5.46\n",
            "2020-06-08 16:05:03,880 | split   1 / 10 |  2800/11855 batches | ms/batch 10.96 | loss  1.81 | ppl     6.13\n",
            "2020-06-08 16:05:04,960 | split   1 / 10 |  2900/11855 batches | ms/batch 10.78 | loss  1.76 | ppl     5.82\n",
            "2020-06-08 16:05:06,078 | split   1 / 10 |  3000/11855 batches | ms/batch 11.12 | loss  1.79 | ppl     5.97\n",
            "2020-06-08 16:05:07,212 | split   1 / 10 |  3100/11855 batches | ms/batch 11.28 | loss  1.70 | ppl     5.46\n",
            "2020-06-08 16:05:07,975 | split   1 / 10 |  3200/11855 batches | ms/batch  7.57 | loss  1.79 | ppl     5.97\n",
            "2020-06-08 16:05:08,728 | split   1 / 10 |  3300/11855 batches | ms/batch  7.52 | loss  1.74 | ppl     5.70\n",
            "2020-06-08 16:05:09,443 | split   1 / 10 |  3400/11855 batches | ms/batch  7.14 | loss  1.81 | ppl     6.08\n",
            "2020-06-08 16:05:10,151 | split   1 / 10 |  3500/11855 batches | ms/batch  7.07 | loss  1.83 | ppl     6.24\n",
            "2020-06-08 16:05:10,843 | split   1 / 10 |  3600/11855 batches | ms/batch  6.90 | loss  1.78 | ppl     5.90\n",
            "2020-06-08 16:05:11,539 | split   1 / 10 |  3700/11855 batches | ms/batch  6.95 | loss  1.78 | ppl     5.95\n",
            "2020-06-08 16:05:12,234 | split   1 / 10 |  3800/11855 batches | ms/batch  6.95 | loss  1.63 | ppl     5.09\n",
            "2020-06-08 16:05:12,938 | split   1 / 10 |  3900/11855 batches | ms/batch  7.03 | loss  1.69 | ppl     5.41\n",
            "2020-06-08 16:05:13,641 | split   1 / 10 |  4000/11855 batches | ms/batch  7.01 | loss  1.58 | ppl     4.87\n",
            "2020-06-08 16:05:14,324 | split   1 / 10 |  4100/11855 batches | ms/batch  6.83 | loss  1.57 | ppl     4.80\n",
            "2020-06-08 16:05:15,012 | split   1 / 10 |  4200/11855 batches | ms/batch  6.87 | loss  1.65 | ppl     5.22\n",
            "2020-06-08 16:05:15,706 | split   1 / 10 |  4300/11855 batches | ms/batch  6.92 | loss  1.71 | ppl     5.53\n",
            "2020-06-08 16:05:16,412 | split   1 / 10 |  4400/11855 batches | ms/batch  7.05 | loss  1.75 | ppl     5.78\n",
            "2020-06-08 16:05:17,107 | split   1 / 10 |  4500/11855 batches | ms/batch  6.94 | loss  1.78 | ppl     5.91\n",
            "2020-06-08 16:05:17,830 | split   1 / 10 |  4600/11855 batches | ms/batch  7.22 | loss  1.82 | ppl     6.19\n",
            "2020-06-08 16:05:18,515 | split   1 / 10 |  4700/11855 batches | ms/batch  6.84 | loss  1.66 | ppl     5.28\n",
            "2020-06-08 16:05:19,200 | split   1 / 10 |  4800/11855 batches | ms/batch  6.84 | loss  1.83 | ppl     6.21\n",
            "2020-06-08 16:05:19,892 | split   1 / 10 |  4900/11855 batches | ms/batch  6.91 | loss  1.76 | ppl     5.83\n",
            "2020-06-08 16:05:20,596 | split   1 / 10 |  5000/11855 batches | ms/batch  7.03 | loss  1.82 | ppl     6.16\n",
            "2020-06-08 16:05:21,312 | split   1 / 10 |  5100/11855 batches | ms/batch  7.15 | loss  1.85 | ppl     6.36\n",
            "2020-06-08 16:05:22,037 | split   1 / 10 |  5200/11855 batches | ms/batch  7.24 | loss  1.84 | ppl     6.31\n",
            "2020-06-08 16:05:22,754 | split   1 / 10 |  5300/11855 batches | ms/batch  7.16 | loss  1.76 | ppl     5.80\n",
            "2020-06-08 16:05:23,471 | split   1 / 10 |  5400/11855 batches | ms/batch  7.16 | loss  1.81 | ppl     6.08\n",
            "2020-06-08 16:05:24,244 | split   1 / 10 |  5500/11855 batches | ms/batch  7.72 | loss  1.82 | ppl     6.16\n",
            "2020-06-08 16:05:25,020 | split   1 / 10 |  5600/11855 batches | ms/batch  7.75 | loss  1.66 | ppl     5.25\n",
            "2020-06-08 16:05:25,706 | split   1 / 10 |  5700/11855 batches | ms/batch  6.84 | loss  1.73 | ppl     5.65\n",
            "2020-06-08 16:05:26,396 | split   1 / 10 |  5800/11855 batches | ms/batch  6.89 | loss  1.57 | ppl     4.80\n",
            "2020-06-08 16:05:27,079 | split   1 / 10 |  5900/11855 batches | ms/batch  6.82 | loss  1.56 | ppl     4.74\n",
            "2020-06-08 16:05:27,780 | split   1 / 10 |  6000/11855 batches | ms/batch  7.00 | loss  1.68 | ppl     5.37\n",
            "2020-06-08 16:05:28,511 | split   1 / 10 |  6100/11855 batches | ms/batch  7.30 | loss  1.79 | ppl     5.96\n",
            "2020-06-08 16:05:29,213 | split   1 / 10 |  6200/11855 batches | ms/batch  7.00 | loss  1.73 | ppl     5.63\n",
            "2020-06-08 16:05:29,912 | split   1 / 10 |  6300/11855 batches | ms/batch  6.98 | loss  1.67 | ppl     5.30\n",
            "2020-06-08 16:05:30,606 | split   1 / 10 |  6400/11855 batches | ms/batch  6.93 | loss  1.85 | ppl     6.33\n",
            "2020-06-08 16:05:31,304 | split   1 / 10 |  6500/11855 batches | ms/batch  6.96 | loss  1.76 | ppl     5.81\n",
            "2020-06-08 16:05:32,012 | split   1 / 10 |  6600/11855 batches | ms/batch  7.07 | loss  1.78 | ppl     5.95\n",
            "2020-06-08 16:05:32,688 | split   1 / 10 |  6700/11855 batches | ms/batch  6.75 | loss  1.76 | ppl     5.84\n",
            "2020-06-08 16:05:33,376 | split   1 / 10 |  6800/11855 batches | ms/batch  6.87 | loss  1.72 | ppl     5.60\n",
            "2020-06-08 16:05:34,068 | split   1 / 10 |  6900/11855 batches | ms/batch  6.91 | loss  1.88 | ppl     6.56\n",
            "2020-06-08 16:05:34,756 | split   1 / 10 |  7000/11855 batches | ms/batch  6.87 | loss  1.79 | ppl     5.99\n",
            "2020-06-08 16:05:35,451 | split   1 / 10 |  7100/11855 batches | ms/batch  6.94 | loss  1.76 | ppl     5.83\n",
            "2020-06-08 16:05:36,142 | split   1 / 10 |  7200/11855 batches | ms/batch  6.90 | loss  1.66 | ppl     5.26\n",
            "2020-06-08 16:05:36,838 | split   1 / 10 |  7300/11855 batches | ms/batch  6.95 | loss  1.81 | ppl     6.13\n",
            "2020-06-08 16:05:37,555 | split   1 / 10 |  7400/11855 batches | ms/batch  7.15 | loss  1.64 | ppl     5.17\n",
            "2020-06-08 16:05:38,260 | split   1 / 10 |  7500/11855 batches | ms/batch  7.04 | loss  1.77 | ppl     5.86\n",
            "2020-06-08 16:05:38,976 | split   1 / 10 |  7600/11855 batches | ms/batch  7.15 | loss  1.58 | ppl     4.84\n",
            "2020-06-08 16:05:39,674 | split   1 / 10 |  7700/11855 batches | ms/batch  6.97 | loss  1.72 | ppl     5.57\n",
            "2020-06-08 16:05:40,378 | split   1 / 10 |  7800/11855 batches | ms/batch  7.04 | loss  1.58 | ppl     4.85\n",
            "2020-06-08 16:05:41,080 | split   1 / 10 |  7900/11855 batches | ms/batch  7.00 | loss  1.65 | ppl     5.22\n",
            "2020-06-08 16:05:41,773 | split   1 / 10 |  8000/11855 batches | ms/batch  6.93 | loss  1.59 | ppl     4.92\n",
            "2020-06-08 16:05:42,474 | split   1 / 10 |  8100/11855 batches | ms/batch  7.00 | loss  1.76 | ppl     5.82\n",
            "2020-06-08 16:05:43,197 | split   1 / 10 |  8200/11855 batches | ms/batch  7.22 | loss  1.69 | ppl     5.39\n",
            "2020-06-08 16:05:43,921 | split   1 / 10 |  8300/11855 batches | ms/batch  7.23 | loss  1.72 | ppl     5.59\n",
            "2020-06-08 16:05:44,663 | split   1 / 10 |  8400/11855 batches | ms/batch  7.41 | loss  1.70 | ppl     5.45\n",
            "2020-06-08 16:05:45,422 | split   1 / 10 |  8500/11855 batches | ms/batch  7.58 | loss  1.58 | ppl     4.85\n",
            "2020-06-08 16:05:46,124 | split   1 / 10 |  8600/11855 batches | ms/batch  7.02 | loss  1.51 | ppl     4.54\n",
            "2020-06-08 16:05:46,849 | split   1 / 10 |  8700/11855 batches | ms/batch  7.23 | loss  1.83 | ppl     6.20\n",
            "2020-06-08 16:05:47,589 | split   1 / 10 |  8800/11855 batches | ms/batch  7.40 | loss  1.86 | ppl     6.42\n",
            "2020-06-08 16:05:48,365 | split   1 / 10 |  8900/11855 batches | ms/batch  7.75 | loss  1.77 | ppl     5.87\n",
            "2020-06-08 16:05:49,090 | split   1 / 10 |  9000/11855 batches | ms/batch  7.24 | loss  1.78 | ppl     5.96\n",
            "2020-06-08 16:05:49,853 | split   1 / 10 |  9100/11855 batches | ms/batch  7.60 | loss  1.76 | ppl     5.84\n",
            "2020-06-08 16:05:50,593 | split   1 / 10 |  9200/11855 batches | ms/batch  7.38 | loss  1.68 | ppl     5.37\n",
            "2020-06-08 16:05:51,330 | split   1 / 10 |  9300/11855 batches | ms/batch  7.36 | loss  1.78 | ppl     5.96\n",
            "2020-06-08 16:05:52,062 | split   1 / 10 |  9400/11855 batches | ms/batch  7.31 | loss  1.60 | ppl     4.97\n",
            "2020-06-08 16:05:52,799 | split   1 / 10 |  9500/11855 batches | ms/batch  7.36 | loss  1.68 | ppl     5.37\n",
            "2020-06-08 16:05:53,529 | split   1 / 10 |  9600/11855 batches | ms/batch  7.29 | loss  1.73 | ppl     5.66\n",
            "2020-06-08 16:05:54,261 | split   1 / 10 |  9700/11855 batches | ms/batch  7.31 | loss  1.94 | ppl     6.93\n",
            "2020-06-08 16:05:54,973 | split   1 / 10 |  9800/11855 batches | ms/batch  7.11 | loss  1.79 | ppl     5.97\n",
            "2020-06-08 16:05:55,695 | split   1 / 10 |  9900/11855 batches | ms/batch  7.20 | loss  1.64 | ppl     5.13\n",
            "2020-06-08 16:05:56,433 | split   1 / 10 | 10000/11855 batches | ms/batch  7.36 | loss  1.72 | ppl     5.57\n",
            "2020-06-08 16:05:57,165 | split   1 / 10 | 10100/11855 batches | ms/batch  7.31 | loss  1.72 | ppl     5.57\n",
            "2020-06-08 16:05:57,902 | split   1 / 10 | 10200/11855 batches | ms/batch  7.36 | loss  1.71 | ppl     5.51\n",
            "2020-06-08 16:05:58,642 | split   1 / 10 | 10300/11855 batches | ms/batch  7.38 | loss  1.66 | ppl     5.28\n",
            "2020-06-08 16:05:59,380 | split   1 / 10 | 10400/11855 batches | ms/batch  7.38 | loss  1.66 | ppl     5.28\n",
            "2020-06-08 16:06:00,138 | split   1 / 10 | 10500/11855 batches | ms/batch  7.56 | loss  1.67 | ppl     5.33\n",
            "2020-06-08 16:06:00,857 | split   1 / 10 | 10600/11855 batches | ms/batch  7.18 | loss  1.85 | ppl     6.33\n",
            "2020-06-08 16:06:01,583 | split   1 / 10 | 10700/11855 batches | ms/batch  7.25 | loss  1.77 | ppl     5.88\n",
            "2020-06-08 16:06:02,308 | split   1 / 10 | 10800/11855 batches | ms/batch  7.24 | loss  1.74 | ppl     5.68\n",
            "2020-06-08 16:06:03,015 | split   1 / 10 | 10900/11855 batches | ms/batch  7.05 | loss  1.69 | ppl     5.44\n",
            "2020-06-08 16:06:03,724 | split   1 / 10 | 11000/11855 batches | ms/batch  7.09 | loss  1.74 | ppl     5.69\n",
            "2020-06-08 16:06:04,441 | split   1 / 10 | 11100/11855 batches | ms/batch  7.16 | loss  1.82 | ppl     6.20\n",
            "2020-06-08 16:06:05,162 | split   1 / 10 | 11200/11855 batches | ms/batch  7.20 | loss  1.80 | ppl     6.05\n",
            "2020-06-08 16:06:05,884 | split   1 / 10 | 11300/11855 batches | ms/batch  7.21 | loss  1.75 | ppl     5.76\n",
            "2020-06-08 16:06:06,655 | split   1 / 10 | 11400/11855 batches | ms/batch  7.70 | loss  1.82 | ppl     6.18\n",
            "2020-06-08 16:06:07,393 | split   1 / 10 | 11500/11855 batches | ms/batch  7.36 | loss  1.79 | ppl     5.98\n",
            "2020-06-08 16:06:08,120 | split   1 / 10 | 11600/11855 batches | ms/batch  7.27 | loss  1.76 | ppl     5.81\n",
            "2020-06-08 16:06:08,856 | split   1 / 10 | 11700/11855 batches | ms/batch  7.35 | loss  1.80 | ppl     6.03\n",
            "2020-06-08 16:06:09,581 | split   1 / 10 | 11800/11855 batches | ms/batch  7.24 | loss  1.69 | ppl     5.43\n",
            "2020-06-08 16:06:09,973 99 seconds for train split 1\n",
            "2020-06-08 16:06:28,976 best loss so far 10000.00\n",
            "2020-06-08 16:06:29,310 ('\\nni hali kikuu kalum 2 ya elimu kuji ulifa mkuu wenye ke kwenye kotale katibu tweta semala katibu taweza ilixezu kuwezekuwa kutoa kwenye kote elelezo ya vya vuko wenye cha kujili kwenye selimumbaliwezo la kufadhi wenyeko kwezo kwele kali wenye vya kwezi kikayeti ya dama welekek ka kokkuwezezi kuelezekuwa kigara wa jipo kuwezele dok ovekta lea kukutokume kwenye k kwenye karleseka kwezo kua be mkuu wenye mjinee vya ya klo sekbaisha mkoa weko rakili kutelesharlengeele katika ameata vye katika depwezekuwezia kalma kwenye pole kwenyekeowezezi kwele kwenye ke kaseh2 katibu techek mkoa kundallala kwenye ke kusekta kakozui kwenyeji waliyokua ku kwenye koi kwenye tajili baada ya kupazi kuwezekuweteleki katibu uli katibu tekeka kaimu katibu menyeti amemtaka kaimu katibu tawezeeka kutala kaimu kaimu kaimu katibu tekeka kaimu katibu tume ya lizo kutera katibu tumo kutakara rakariene kalma kataka kaimu katibu tetaka katibu taweza kuzaka dokta kaimu kwenye kama katibu wenye kira katibu teteka kalimu ', 13.862982421875)\n",
            "2020-06-08 16:06:29,314 -----------------------------------------------------------------------------------------\n",
            "2020-06-08 16:06:29,316 | end of split   1 / 10 | epoch   1 | time: 119.11s | valid loss  2.06 | valid ppl     7.83 | learning rate 20.0000\n",
            "2020-06-08 16:06:29,317 -----------------------------------------------------------------------------------------\n",
            "2020-06-08 16:06:29,321 Sequence length is 10\n",
            "2020-06-08 16:06:29,326 read text file with 1 lines\n",
            "2020-06-08 16:06:29,329 shuffled\n",
            "2020-06-08 16:06:29,353 Split 2\t - (16:06:29)\n",
            "2020-06-08 16:06:30,500 | split   2 / 10 |   100/11855 batches | ms/batch 11.42 | loss  1.82 | ppl     6.19\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}