Language_Model_Using_a_Custom_LSTM.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Language Model Using a Custom LSTM.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [
        "qA3L-WvsSHfi",
        "z9wEHvYcgnPO",
        "WC7Xi43vkeGl"
      ],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mikkelbrusen/custom-pytorch-lstm-lm/blob/master/Language_Model_Using_a_Custom_LSTM.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "ZpdsWCLdHhTG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#Language Model Using a Custom Build LSTM\n",
        "This notebook contains the code to implement a language model using a custom LSTM built within the PyTorch framework.\n",
        "\n",
        "This model was implemented as part of a project in the course 02456 Deep Learning @ DTU - Technical University of Denmark\n",
        "+ This code was originally forked from the [PyTorch word level language modeling example](https://github.com/pytorch/examples/tree/master/word_language_model).\n",
        "+ The code in this notebook is available on [google colab](https://colab.research.google.com/drive/1luim4qegwBeKVAAzW-XarPSCOhogiBQf) and on [github](https://github.com/mikkelbrusen/custom-pytorch-lstm-lm).\n",
        "\n",
        "The model comes with instructions to train a word level language models over the Penn Treebank (PTB).\n",
        "\n",
        "The project was carried out by [Gustav Madslund](https://github.com/gustavmadslund) and [Mikkel Møller Brusen](https://github.com/mikkelbrusen)."
      ]
    },
    {
      "metadata": {
        "id": "WpnP2cHfK-00",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Setup\n",
        "This section contains all the necessary setup as hyperparameters, data processing and utility functions"
      ]
    },
    {
      "metadata": {
        "id": "e204UkJVwJ4J",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Google Colab Setup\n",
        "Since we are running on Google Colab, we will need to install PyTorch as they only support TensorFlow by default, because, well, they are Google and not Facebook."
      ]
    },
    {
      "metadata": {
        "id": "qQoHVrVgR8lh",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# http://pytorch.org/\n",
        "from os.path import exists\n",
        "from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag\n",
        "platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())\n",
        "cuda_output = !ldconfig -p|grep cudart.so|sed -e 's/.*\\.\\([0-9]*\\)\\.\\([0-9]*\\)$/cu\\1\\2/'\n",
        "accelerator = cuda_output[0] if exists('/dev/nvidia0') else 'cpu'\n",
        "\n",
        "!pip install -q http://download.pytorch.org/whl/{accelerator}/torch-0.4.1-{platform}-linux_x86_64.whl torchvision\n",
        "import torch"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "JgKFkeoAigxv"
      },
      "cell_type": "markdown",
      "source": [
        "We will need some data to train on, and a place to save our model. \n",
        "We connect to google drive and position our data in the following path: *MyDrive/NLP/data/penn/* which needs to be put in "
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "pImz6t2Ligxw",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive', force_remount=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9zQbaNkLh3FN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Imports and params\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "mtulgShphXWT",
        "colab_type": "code",
        "cellView": "both",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import argparse\n",
        "import time\n",
        "import math\n",
        "import os\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch.autograd import Variable\n",
        "from collections import Counter"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ZmumaoCEh61G",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "args_cuda = torch.cuda.is_available()\n",
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "\n",
        "# Hyper-parameters\n",
        "args_train_batch_size = 20 # batch size\n",
        "args_bptt = 35 # sequence length\n",
        "args_embed_size = 450 # \n",
        "args_hidden_size = 650 \n",
        "args_num_layers = 2 # Number of LSTM layers\n",
        "args_num_epochs = 40\n",
        "args_learning_rate = 20\n",
        "args_dropout = 0.5\n",
        "args_clip = 0.25\n",
        "args_log_interval = 100\n",
        "args_seed = 1111 # We seed the RNG's for reproducability\n",
        "\n",
        "# if you dont already have the penn treebank data, grab it from our github repo\n",
        "# here: https://github.com/mikkelbrusen/custom-pytorch-lstm-lm\n",
        "args_data = \"/content/gdrive/My Drive/NLP/data/penn/\"\n",
        "\n",
        "# The file in which we want to save our trained model.\n",
        "args_save = \"/content/gdrive/My Drive/NLP/save/Custom_LSTM_Model.pt\"\n",
        "\n",
        "\n",
        "torch.manual_seed(args_seed)\n",
        "if args_cuda:\n",
        "  torch.cuda.manual_seed(args_seed)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "qA3L-WvsSHfi",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## The data loader\n",
        "Dictionary and corpus to process the dataset"
      ]
    },
    {
      "metadata": {
        "id": "KHjT2i8oQ9xv",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class Dictionary(object):\n",
        "    def __init__(self):\n",
        "        self.word2idx = {}\n",
        "        self.idx2word = []\n",
        "        self.counter = Counter()\n",
        "        self.total = 0\n",
        "\n",
        "    def add_word(self, word):\n",
        "        if word not in self.word2idx:\n",
        "            self.idx2word.append(word)\n",
        "            self.word2idx[word] = len(self.idx2word) - 1\n",
        "        token_id = self.word2idx[word]\n",
        "        self.counter[token_id] += 1\n",
        "        self.total += 1\n",
        "        return self.word2idx[word]\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.idx2word)\n",
        "\n",
        "\n",
        "class Corpus(object):\n",
        "    def __init__(self, path):\n",
        "        self.dictionary = Dictionary()\n",
        "        self.train = self.tokenize(os.path.join(path, 'train.txt'))\n",
        "        self.valid = self.tokenize(os.path.join(path, 'valid.txt'))\n",
        "        self.test = self.tokenize(os.path.join(path, 'test.txt'))\n",
        "\n",
        "    def tokenize(self, path):\n",
        "        \"\"\"Tokenizes a text file.\"\"\"\n",
        "        assert os.path.exists(path)\n",
        "        # Add words to the dictionary\n",
        "        with open(path, 'r') as f:\n",
        "            tokens = 0\n",
        "            for line in f:\n",
        "                words = line.split() + ['<eos>']\n",
        "                tokens += len(words)\n",
        "                for word in words:\n",
        "                    self.dictionary.add_word(word)\n",
        "\n",
        "        # Tokenize file content\n",
        "        with open(path, 'r') as f:\n",
        "            ids = torch.LongTensor(tokens)\n",
        "            token = 0\n",
        "            for line in f:\n",
        "                words = line.split() + ['<eos>']\n",
        "                for word in words:\n",
        "                    ids[token] = self.dictionary.word2idx[word]\n",
        "                    token += 1\n",
        "\n",
        "        return ids"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "z9wEHvYcgnPO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Utils\n",
        "Utility functions which will be used while training, validating and testing"
      ]
    },
    {
      "metadata": {
        "id": "Dl_pQuvugmWW",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def repackage_hidden(h):\n",
        "    \"\"\"Wraps hidden states in new Tensors, to detach them from their history.\"\"\"\n",
        "    if isinstance(h, torch.Tensor):\n",
        "        return h.detach()\n",
        "    else:\n",
        "        return tuple(repackage_hidden(v) for v in h)\n",
        "\n",
        "\n",
        "# Starting from sequential data, batchify arranges the dataset into columns.\n",
        "# For instance, with the alphabet as the sequence and batch size 4, we'd get\n",
        "# ┌ a g m s ┐\n",
        "# │ b h n t │\n",
        "# │ c i o u │\n",
        "# │ d j p v │\n",
        "# │ e k q w │\n",
        "# └ f l r x ┘.\n",
        "# These columns are treated as independent by the model, which means that the\n",
        "# dependence of e. g. 'g' on 'f' can not be learned, but allows more efficient\n",
        "# batch processing.\n",
        "\n",
        "def batchify(data, bsz):\n",
        "    # Work out how cleanly we can divide the dataset into bsz parts.\n",
        "    nbatch = data.size(0) // bsz\n",
        "    # Trim off any extra elements that wouldn't cleanly fit (remainders).\n",
        "    data = data.narrow(0, 0, nbatch * bsz)\n",
        "    # Evenly divide the data across the bsz batches.\n",
        "    data = data.view(bsz, -1).t().contiguous()\n",
        "    return data.to(device)\n",
        "\n",
        "\n",
        "# get_batch subdivides the source data into chunks of length args.bptt.\n",
        "# If source is equal to the example output of the batchify function, with\n",
        "# a bptt-limit of 2, we'd get the following two Variables for i = 0:\n",
        "# ┌ a g m s ┐ ┌ b h n t ┐\n",
        "# └ b h n t ┘ └ c i o u ┘\n",
        "# Note that despite the name of the function, the subdivison of data is not\n",
        "# done along the batch dimension (i.e. dimension 1), since that was handled\n",
        "# by the batchify function. The chunks are along dimension 0, corresponding\n",
        "# to the seq_len dimension in the LSTM.\n",
        "\n",
        "def get_batch(source, i):\n",
        "    #seq_len = min(args_bptt, len(source) - 1 - i)\n",
        "    data = source[i-args_bptt:i]\n",
        "    target = source[i+1-args_bptt:i+1].view(-1)\n",
        "    return data, target"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "vPXGE8KZL1XH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Process data\n",
        "Load the dataset and make train, validaiton and test sets"
      ]
    },
    {
      "metadata": {
        "id": "8jM-ww1fh5Vz",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "corpus = Corpus(args_data)\n",
        "\n",
        "eval_batch_size = 10\n",
        "test_batch_size = 10\n",
        "train_data = batchify(corpus.train, args_train_batch_size)\n",
        "val_data = batchify(corpus.valid, eval_batch_size)\n",
        "test_data = batchify(corpus.test, test_batch_size)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "WC7Xi43vkeGl",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Custom LSTM\n",
        "We chose to implement our LSTM modules as single layer modules, meaning that the multiple layers will be created within our model rather than within the LSTM module.\n",
        "\n",
        "### Dimensions\n",
        "An analysis of the dimensions can be found in the following figures\n",
        "\n",
        "**Whole Model**\n",
        "\n",
        "https://www.lucidchart.com/invitations/accept/b77330fa-348f-47e4-b0c1-ac13d3b72a81\n",
        "\n",
        "**LSTM**\n",
        "\n",
        "https://www.lucidchart.com/invitations/accept/6a965c89-0e68-40b6-b3c6-bb4d0b2e6619"
      ]
    },
    {
      "metadata": {
        "id": "G9_xEmNW-eTL",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#LSTM Module\n",
        "class LSTM(nn.Module):\n",
        "  def __init__(self,input_size,hidden_size,bias=False):\n",
        "    super(LSTM, self).__init__()\n",
        "    self.input_size = input_size\n",
        "    self.hidden_size = hidden_size\n",
        "    \n",
        "    self.weight_fx = nn.Linear(input_size, hidden_size, bias=bias)\n",
        "    self.weight_ix = nn.Linear(input_size, hidden_size, bias=bias)\n",
        "    self.weight_cx = nn.Linear(input_size, hidden_size, bias=bias)\n",
        "    self.weight_ox = nn.Linear(input_size, hidden_size, bias=bias)\n",
        "    \n",
        "    self.weight_fh = nn.Linear(hidden_size, hidden_size, bias=bias)\n",
        "    self.weight_ih = nn.Linear(hidden_size, hidden_size, bias=bias)\n",
        "    self.weight_ch = nn.Linear(hidden_size, hidden_size, bias=bias)\n",
        "    self.weight_oh = nn.Linear(hidden_size, hidden_size, bias=bias)\n",
        "    \n",
        "    \n",
        "  def forward(self,input, hidden):\n",
        "    h,c = hidden\n",
        "    def recurrence(inp, hidden):\n",
        "      \"\"\"Recurrence helper.\"\"\"\n",
        "      h,c = hidden\n",
        "      \n",
        "      f_g = torch.sigmoid(self.weight_fx(inp) + self.weight_fh(h))\n",
        "      i_g = torch.sigmoid(self.weight_ix(inp) + self.weight_ih(h))\n",
        "      o_g = torch.sigmoid(self.weight_ox(inp) + self.weight_oh(h))\n",
        "      c_tilda = torch.tanh(self.weight_cx(inp) + self.weight_ch(h))\n",
        "      c_t = f_g * c + i_g * c_tilda\n",
        "      h_t = o_g * torch.tanh(c_t)\n",
        "    \n",
        "      return h_t, c_t\n",
        "      #--------------\n",
        "    \n",
        "    output = []\n",
        "    for inp in input:\n",
        "      h,c = recurrence(inp, (h,c))\n",
        "      output.append(h)\n",
        "\n",
        "    # torch.cat(output, 0).size()=torch.Size([700, 650]) view(input.size(0)=35, *output[0].size()=20 650)\n",
        "    output = torch.cat(output, 0).view(input.size(0), *output[0].size())\n",
        "    return output, (h,c)\n",
        "      "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "FzZtWX2zP1ry",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Language Model using our Custom LSTM\n",
        "First we define our model"
      ]
    },
    {
      "metadata": {
        "id": "tt_aD-iGkdbz",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class LSTMModel(nn.Module):\n",
        "    def __init__(self, num_tokens, embed_size, hidden_size, output_size, dropout=0.5, n_layers=1):\n",
        "        super(LSTMModel, self).__init__()\n",
        "        self.drop = nn.Dropout(dropout)\n",
        "        self.encoder = nn.Embedding(num_tokens, embed_size)\n",
        "        \n",
        "        # We add each LSTM layer to the module list such that pytorch is aware \n",
        "        # of their parameters for when we perform gradient decent\n",
        "        self.layers = nn.ModuleList()\n",
        "        for l in range(n_layers):\n",
        "          layer_input_size = embed_size if l == 0 else hidden_size\n",
        "          self.layers.append(LSTM(layer_input_size, hidden_size))\n",
        "          \n",
        "        self.decoder = nn.Linear(hidden_size, output_size)\n",
        "        \n",
        "        self.hidden_size = hidden_size\n",
        "        self.n_layers = n_layers\n",
        "        \n",
        "        self.init_weights()\n",
        "       \n",
        "\n",
        "    def init_weights(self):\n",
        "        initrange = 0.1\n",
        "        self.encoder.weight.data.uniform_(-initrange, initrange)\n",
        "        self.decoder.bias.data.zero_()\n",
        "        self.decoder.weight.data.uniform_(-initrange, initrange)\n",
        "\n",
        "    def forward(self, inp, hidden):\n",
        "        emb = self.drop(self.encoder(inp))\n",
        "        \n",
        "        output= emb\n",
        "        h_0, c_0 = hidden\n",
        "        h, c = [], []\n",
        "        \n",
        "        # Iterate over each LSTM layer, and pass the output from one layer on to the next \n",
        "        for i, layer in enumerate(self.layers): \n",
        "            output, (h_i, c_i) = layer(output, (h_0[i], c_0[i]))\n",
        "            output = self.drop(output)\n",
        "            \n",
        "            h += [h_i]\n",
        "            c += [c_i]\n",
        "        \n",
        "        h = torch.stack(h)\n",
        "        c = torch.stack(c)\n",
        " \n",
        "        decoded = self.decoder(output.view(output.size(0)*output.size(1), output.size(2)))\n",
        "        decoded = decoded.view(output.size(0), output.size(1), decoded.size(1))\n",
        "    \n",
        "        return decoded, (h,c)\n",
        "\n",
        "    def init_hidden(self,bsz):\n",
        "        h_0 = Variable(torch.zeros(self.n_layers, bsz, self.hidden_size)).cuda()\n",
        "        c_0 = Variable(torch.zeros(self.n_layers, bsz, self.hidden_size)).cuda()\n",
        "        return (h_0, c_0)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PH3dQqwjiXP5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Then we build the model and specify our loss function"
      ]
    },
    {
      "metadata": {
        "id": "CAAf3g95JigU",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "ntokens = len(corpus.dictionary)\n",
        "\n",
        "model = LSTMModel(ntokens, args_embed_size, args_hidden_size, ntokens, args_dropout, args_num_layers).to(device)\n",
        "criterion = nn.CrossEntropyLoss()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "a-TQgIkBn8_B",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Train model\n",
        "\n",
        "First we define our training and evalutation"
      ]
    },
    {
      "metadata": {
        "id": "D8e1jsOdn_wj",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def evaluate(data_source):\n",
        "    # Turn on evaluation mode which disables dropout.\n",
        "    model.eval()\n",
        "    total_loss = 0.\n",
        "    total_words = len(data_source) - (len(data_source) % args_bptt)\n",
        "    ntokens = len(corpus.dictionary)\n",
        "    hidden = model.init_hidden(test_batch_size)\n",
        "    with torch.no_grad():\n",
        "        for i in range(args_bptt, data_source.size(0) - 1, args_bptt):\n",
        "            data, targets = get_batch(data_source, i)\n",
        "            output, hidden = model(data, hidden)\n",
        "            output_flat = output.view(-1, ntokens)\n",
        "            total_loss += len(data) * criterion(output_flat, targets).item()\n",
        "            hidden = repackage_hidden(hidden)\n",
        "    return total_loss / (total_words  - 1)\n",
        "\n",
        "\n",
        "def train():\n",
        "    # Turn on training mode which enables dropout.\n",
        "    model.train()\n",
        "    total_loss = 0.\n",
        "    start_time = time.time()\n",
        "    ntokens = len(corpus.dictionary)\n",
        "    hidden = model.init_hidden(args_train_batch_size)\n",
        "    for batch, i in enumerate(range(args_bptt, train_data.size(0) - 1, args_bptt)):\n",
        "        data, targets = get_batch(train_data, i)\n",
        "        # Starting each batch, we detach the hidden state from how it was previously produced.\n",
        "        # If we didn't, the model would try backpropagating all the way to start of the dataset.\n",
        "        hidden = repackage_hidden(hidden)\n",
        "        model.zero_grad()\n",
        "        output, hidden = model(data, hidden)\n",
        "        loss = criterion(output.view(-1, ntokens), targets)\n",
        "        loss.backward()\n",
        "\n",
        "        # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.\n",
        "        torch.nn.utils.clip_grad_norm_(model.parameters(), args_clip)\n",
        "        for p in model.parameters():\n",
        "            p.data.add_(-lr, p.grad.data)\n",
        "\n",
        "        total_loss += loss.item()\n",
        "\n",
        "        if batch % args_log_interval == 0 and batch > 0:\n",
        "            cur_loss = total_loss / args_log_interval\n",
        "            elapsed = time.time() - start_time\n",
        "            print('| epoch {:3d} | {:5d}/{:5d} batches | lr {:02.2f} | ms/batch {:5.2f} | '\n",
        "                    'loss {:5.2f} | ppl {:8.2f}'.format(\n",
        "                epoch, batch, len(train_data) // args_bptt, lr,\n",
        "                elapsed * 1000 / args_log_interval, cur_loss, math.exp(cur_loss)))\n",
        "            total_loss = 0\n",
        "            start_time = time.time()\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "mXCZzJmHkRFy",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Then do the actual training"
      ]
    },
    {
      "metadata": {
        "id": "IXolA9BNj9JG",
        "colab_type": "code",
        "outputId": "2099c43d-28e8-4ca5-f978-d0f13e629722",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 10965
        }
      },
      "cell_type": "code",
      "source": [
        "# Loop over epochs.\n",
        "lr = args_learning_rate\n",
        "best_val_loss = None\n",
        "\n",
        "# At any point you can hit Ctrl + C to break out of training early.\n",
        "try:\n",
        "    for epoch in range(1, args_num_epochs+1):\n",
        "        epoch_start_time = time.time()\n",
        "        train()\n",
        "        val_loss = evaluate(val_data)\n",
        "        print('-' * 89)\n",
        "        print('| end of epoch {:3d} | time: {:5.2f}s | valid loss {:5.2f} | '\n",
        "                'valid ppl {:8.2f}'.format(epoch, (time.time() - epoch_start_time),\n",
        "                                           val_loss, math.exp(val_loss)))\n",
        "        print('-' * 89)\n",
        "        # Save the model if the validation loss is the best we've seen so far.\n",
        "        if not best_val_loss or val_loss < best_val_loss:\n",
        "            with open(args_save, 'wb') as f:\n",
        "                torch.save(model, f)\n",
        "            \n",
        "            best_val_loss = val_loss\n",
        "        else:\n",
        "            # Anneal the learning rate if no improvement has been seen in the validation dataset.\n",
        "            lr /= 4.0\n",
        "except KeyboardInterrupt:\n",
        "    print('-' * 89)\n",
        "    print('Exiting from training early')\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "| epoch   1 |   100/ 1327 batches | lr 20.00 | ms/batch 156.54 | loss  7.08 | ppl  1189.33\n",
            "| epoch   1 |   200/ 1327 batches | lr 20.00 | ms/batch 153.69 | loss  6.26 | ppl   523.80\n",
            "| epoch   1 |   300/ 1327 batches | lr 20.00 | ms/batch 154.92 | loss  6.02 | ppl   411.92\n",
            "| epoch   1 |   400/ 1327 batches | lr 20.00 | ms/batch 154.25 | loss  5.85 | ppl   345.59\n",
            "| epoch   1 |   500/ 1327 batches | lr 20.00 | ms/batch 154.90 | loss  5.76 | ppl   317.08\n",
            "| epoch   1 |   600/ 1327 batches | lr 20.00 | ms/batch 152.67 | loss  5.70 | ppl   299.46\n",
            "| epoch   1 |   700/ 1327 batches | lr 20.00 | ms/batch 154.75 | loss  5.59 | ppl   268.20\n",
            "| epoch   1 |   800/ 1327 batches | lr 20.00 | ms/batch 155.21 | loss  5.47 | ppl   237.68\n",
            "| epoch   1 |   900/ 1327 batches | lr 20.00 | ms/batch 155.04 | loss  5.45 | ppl   233.12\n",
            "| epoch   1 |  1000/ 1327 batches | lr 20.00 | ms/batch 154.62 | loss  5.45 | ppl   232.37\n",
            "| epoch   1 |  1100/ 1327 batches | lr 20.00 | ms/batch 153.48 | loss  5.32 | ppl   204.66\n",
            "| epoch   1 |  1200/ 1327 batches | lr 20.00 | ms/batch 154.42 | loss  5.32 | ppl   204.54\n",
            "| epoch   1 |  1300/ 1327 batches | lr 20.00 | ms/batch 155.49 | loss  5.26 | ppl   191.53\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch   1 | time: 214.79s | valid loss  5.23 | valid ppl   186.44\n",
            "-----------------------------------------------------------------------------------------\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/torch/serialization.py:241: UserWarning: Couldn't retrieve source code for container of type LSTMModel. It won't be checked for correctness upon loading.\n",
            "  \"type \" + obj.__name__ + \". It won't be checked \"\n",
            "/usr/local/lib/python3.6/dist-packages/torch/serialization.py:241: UserWarning: Couldn't retrieve source code for container of type LSTM. It won't be checked for correctness upon loading.\n",
            "  \"type \" + obj.__name__ + \". It won't be checked \"\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "| epoch   2 |   100/ 1327 batches | lr 20.00 | ms/batch 158.70 | loss  5.26 | ppl   192.67\n",
            "| epoch   2 |   200/ 1327 batches | lr 20.00 | ms/batch 153.62 | loss  5.25 | ppl   190.80\n",
            "| epoch   2 |   300/ 1327 batches | lr 20.00 | ms/batch 154.19 | loss  5.23 | ppl   187.53\n",
            "| epoch   2 |   400/ 1327 batches | lr 20.00 | ms/batch 155.38 | loss  5.15 | ppl   171.61\n",
            "| epoch   2 |   500/ 1327 batches | lr 20.00 | ms/batch 154.56 | loss  5.13 | ppl   168.96\n",
            "| epoch   2 |   600/ 1327 batches | lr 20.00 | ms/batch 153.95 | loss  5.15 | ppl   173.14\n",
            "| epoch   2 |   700/ 1327 batches | lr 20.00 | ms/batch 154.48 | loss  5.10 | ppl   163.83\n",
            "| epoch   2 |   800/ 1327 batches | lr 20.00 | ms/batch 155.26 | loss  4.99 | ppl   147.66\n",
            "| epoch   2 |   900/ 1327 batches | lr 20.00 | ms/batch 154.90 | loss  5.05 | ppl   155.49\n",
            "| epoch   2 |  1000/ 1327 batches | lr 20.00 | ms/batch 154.54 | loss  5.06 | ppl   158.21\n",
            "| epoch   2 |  1100/ 1327 batches | lr 20.00 | ms/batch 155.10 | loss  4.93 | ppl   137.86\n",
            "| epoch   2 |  1200/ 1327 batches | lr 20.00 | ms/batch 155.85 | loss  4.95 | ppl   141.49\n",
            "| epoch   2 |  1300/ 1327 batches | lr 20.00 | ms/batch 155.81 | loss  4.92 | ppl   136.59\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch   2 | time: 215.53s | valid loss  4.96 | valid ppl   142.00\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch   3 |   100/ 1327 batches | lr 20.00 | ms/batch 157.41 | loss  4.93 | ppl   137.87\n",
            "| epoch   3 |   200/ 1327 batches | lr 20.00 | ms/batch 155.13 | loss  4.97 | ppl   144.55\n",
            "| epoch   3 |   300/ 1327 batches | lr 20.00 | ms/batch 154.33 | loss  4.97 | ppl   143.84\n",
            "| epoch   3 |   400/ 1327 batches | lr 20.00 | ms/batch 155.74 | loss  4.87 | ppl   130.63\n",
            "| epoch   3 |   500/ 1327 batches | lr 20.00 | ms/batch 154.88 | loss  4.85 | ppl   128.04\n",
            "| epoch   3 |   600/ 1327 batches | lr 20.00 | ms/batch 155.42 | loss  4.91 | ppl   135.99\n",
            "| epoch   3 |   700/ 1327 batches | lr 20.00 | ms/batch 155.43 | loss  4.86 | ppl   128.79\n",
            "| epoch   3 |   800/ 1327 batches | lr 20.00 | ms/batch 153.90 | loss  4.76 | ppl   116.35\n",
            "| epoch   3 |   900/ 1327 batches | lr 20.00 | ms/batch 154.17 | loss  4.83 | ppl   125.72\n",
            "| epoch   3 |  1000/ 1327 batches | lr 20.00 | ms/batch 156.06 | loss  4.86 | ppl   129.37\n",
            "| epoch   3 |  1100/ 1327 batches | lr 20.00 | ms/batch 155.16 | loss  4.71 | ppl   111.18\n",
            "| epoch   3 |  1200/ 1327 batches | lr 20.00 | ms/batch 154.56 | loss  4.75 | ppl   115.36\n",
            "| epoch   3 |  1300/ 1327 batches | lr 20.00 | ms/batch 155.32 | loss  4.73 | ppl   113.45\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch   3 | time: 215.62s | valid loss  4.81 | valid ppl   122.93\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch   4 |   100/ 1327 batches | lr 20.00 | ms/batch 158.13 | loss  4.73 | ppl   112.87\n",
            "| epoch   4 |   200/ 1327 batches | lr 20.00 | ms/batch 155.80 | loss  4.80 | ppl   121.30\n",
            "| epoch   4 |   300/ 1327 batches | lr 20.00 | ms/batch 155.33 | loss  4.80 | ppl   121.06\n",
            "| epoch   4 |   400/ 1327 batches | lr 20.00 | ms/batch 156.41 | loss  4.70 | ppl   109.41\n",
            "| epoch   4 |   500/ 1327 batches | lr 20.00 | ms/batch 155.15 | loss  4.69 | ppl   109.37\n",
            "| epoch   4 |   600/ 1327 batches | lr 20.00 | ms/batch 155.65 | loss  4.76 | ppl   116.22\n",
            "| epoch   4 |   700/ 1327 batches | lr 20.00 | ms/batch 155.84 | loss  4.71 | ppl   110.80\n",
            "| epoch   4 |   800/ 1327 batches | lr 20.00 | ms/batch 156.00 | loss  4.60 | ppl    99.74\n",
            "| epoch   4 |   900/ 1327 batches | lr 20.00 | ms/batch 154.20 | loss  4.69 | ppl   109.36\n",
            "| epoch   4 |  1000/ 1327 batches | lr 20.00 | ms/batch 155.70 | loss  4.73 | ppl   112.82\n",
            "| epoch   4 |  1100/ 1327 batches | lr 20.00 | ms/batch 155.59 | loss  4.57 | ppl    96.88\n",
            "| epoch   4 |  1200/ 1327 batches | lr 20.00 | ms/batch 155.95 | loss  4.61 | ppl   100.12\n",
            "| epoch   4 |  1300/ 1327 batches | lr 20.00 | ms/batch 155.65 | loss  4.60 | ppl    99.01\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch   4 | time: 216.29s | valid loss  4.73 | valid ppl   112.81\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch   5 |   100/ 1327 batches | lr 20.00 | ms/batch 157.27 | loss  4.59 | ppl    98.32\n",
            "| epoch   5 |   200/ 1327 batches | lr 20.00 | ms/batch 155.68 | loss  4.68 | ppl   107.67\n",
            "| epoch   5 |   300/ 1327 batches | lr 20.00 | ms/batch 154.52 | loss  4.67 | ppl   107.05\n",
            "| epoch   5 |   400/ 1327 batches | lr 20.00 | ms/batch 154.62 | loss  4.57 | ppl    96.80\n",
            "| epoch   5 |   500/ 1327 batches | lr 20.00 | ms/batch 154.98 | loss  4.58 | ppl    97.32\n",
            "| epoch   5 |   600/ 1327 batches | lr 20.00 | ms/batch 155.54 | loss  4.64 | ppl   103.24\n",
            "| epoch   5 |   700/ 1327 batches | lr 20.00 | ms/batch 155.82 | loss  4.59 | ppl    98.30\n",
            "| epoch   5 |   800/ 1327 batches | lr 20.00 | ms/batch 155.47 | loss  4.49 | ppl    89.45\n",
            "| epoch   5 |   900/ 1327 batches | lr 20.00 | ms/batch 155.83 | loss  4.58 | ppl    97.57\n",
            "| epoch   5 |  1000/ 1327 batches | lr 20.00 | ms/batch 155.53 | loss  4.62 | ppl   101.54\n",
            "| epoch   5 |  1100/ 1327 batches | lr 20.00 | ms/batch 155.67 | loss  4.47 | ppl    87.03\n",
            "| epoch   5 |  1200/ 1327 batches | lr 20.00 | ms/batch 155.32 | loss  4.51 | ppl    90.47\n",
            "| epoch   5 |  1300/ 1327 batches | lr 20.00 | ms/batch 154.92 | loss  4.50 | ppl    89.75\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch   5 | time: 215.87s | valid loss  4.67 | valid ppl   106.19\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch   6 |   100/ 1327 batches | lr 20.00 | ms/batch 157.57 | loss  4.49 | ppl    88.85\n",
            "| epoch   6 |   200/ 1327 batches | lr 20.00 | ms/batch 155.18 | loss  4.58 | ppl    97.05\n",
            "| epoch   6 |   300/ 1327 batches | lr 20.00 | ms/batch 155.41 | loss  4.57 | ppl    96.94\n",
            "| epoch   6 |   400/ 1327 batches | lr 20.00 | ms/batch 155.10 | loss  4.48 | ppl    88.22\n",
            "| epoch   6 |   500/ 1327 batches | lr 20.00 | ms/batch 154.78 | loss  4.47 | ppl    87.76\n",
            "| epoch   6 |   600/ 1327 batches | lr 20.00 | ms/batch 154.05 | loss  4.54 | ppl    93.62\n",
            "| epoch   6 |   700/ 1327 batches | lr 20.00 | ms/batch 153.77 | loss  4.50 | ppl    89.96\n",
            "| epoch   6 |   800/ 1327 batches | lr 20.00 | ms/batch 154.31 | loss  4.41 | ppl    81.92\n",
            "| epoch   6 |   900/ 1327 batches | lr 20.00 | ms/batch 155.62 | loss  4.50 | ppl    89.61\n",
            "| epoch   6 |  1000/ 1327 batches | lr 20.00 | ms/batch 155.18 | loss  4.54 | ppl    93.97\n",
            "| epoch   6 |  1100/ 1327 batches | lr 20.00 | ms/batch 155.60 | loss  4.38 | ppl    79.95\n",
            "| epoch   6 |  1200/ 1327 batches | lr 20.00 | ms/batch 154.99 | loss  4.42 | ppl    83.23\n",
            "| epoch   6 |  1300/ 1327 batches | lr 20.00 | ms/batch 155.58 | loss  4.41 | ppl    82.32\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch   6 | time: 215.51s | valid loss  4.64 | valid ppl   103.14\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch   7 |   100/ 1327 batches | lr 20.00 | ms/batch 159.58 | loss  4.40 | ppl    81.77\n",
            "| epoch   7 |   200/ 1327 batches | lr 20.00 | ms/batch 155.33 | loss  4.50 | ppl    90.30\n",
            "| epoch   7 |   300/ 1327 batches | lr 20.00 | ms/batch 156.17 | loss  4.49 | ppl    89.54\n",
            "| epoch   7 |   400/ 1327 batches | lr 20.00 | ms/batch 155.89 | loss  4.40 | ppl    81.49\n",
            "| epoch   7 |   500/ 1327 batches | lr 20.00 | ms/batch 155.69 | loss  4.40 | ppl    81.55\n",
            "| epoch   7 |   600/ 1327 batches | lr 20.00 | ms/batch 154.81 | loss  4.47 | ppl    86.99\n",
            "| epoch   7 |   700/ 1327 batches | lr 20.00 | ms/batch 154.50 | loss  4.43 | ppl    83.58\n",
            "| epoch   7 |   800/ 1327 batches | lr 20.00 | ms/batch 155.32 | loss  4.33 | ppl    76.15\n",
            "| epoch   7 |   900/ 1327 batches | lr 20.00 | ms/batch 153.90 | loss  4.43 | ppl    83.58\n",
            "| epoch   7 |  1000/ 1327 batches | lr 20.00 | ms/batch 154.34 | loss  4.48 | ppl    88.22\n",
            "| epoch   7 |  1100/ 1327 batches | lr 20.00 | ms/batch 156.24 | loss  4.31 | ppl    74.16\n",
            "| epoch   7 |  1200/ 1327 batches | lr 20.00 | ms/batch 155.53 | loss  4.34 | ppl    76.65\n",
            "| epoch   7 |  1300/ 1327 batches | lr 20.00 | ms/batch 153.62 | loss  4.34 | ppl    76.55\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch   7 | time: 215.95s | valid loss  4.61 | valid ppl   100.37\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch   8 |   100/ 1327 batches | lr 20.00 | ms/batch 159.46 | loss  4.34 | ppl    76.34\n",
            "| epoch   8 |   200/ 1327 batches | lr 20.00 | ms/batch 155.99 | loss  4.43 | ppl    83.84\n",
            "| epoch   8 |   300/ 1327 batches | lr 20.00 | ms/batch 156.07 | loss  4.43 | ppl    83.89\n",
            "| epoch   8 |   400/ 1327 batches | lr 20.00 | ms/batch 156.07 | loss  4.34 | ppl    76.77\n",
            "| epoch   8 |   500/ 1327 batches | lr 20.00 | ms/batch 155.30 | loss  4.33 | ppl    75.95\n",
            "| epoch   8 |   600/ 1327 batches | lr 20.00 | ms/batch 155.95 | loss  4.41 | ppl    81.88\n",
            "| epoch   8 |   700/ 1327 batches | lr 20.00 | ms/batch 155.88 | loss  4.36 | ppl    78.50\n",
            "| epoch   8 |   800/ 1327 batches | lr 20.00 | ms/batch 156.05 | loss  4.28 | ppl    72.06\n",
            "| epoch   8 |   900/ 1327 batches | lr 20.00 | ms/batch 154.99 | loss  4.37 | ppl    79.06\n",
            "| epoch   8 |  1000/ 1327 batches | lr 20.00 | ms/batch 156.24 | loss  4.41 | ppl    82.31\n",
            "| epoch   8 |  1100/ 1327 batches | lr 20.00 | ms/batch 156.12 | loss  4.25 | ppl    70.21\n",
            "| epoch   8 |  1200/ 1327 batches | lr 20.00 | ms/batch 154.16 | loss  4.29 | ppl    72.72\n",
            "| epoch   8 |  1300/ 1327 batches | lr 20.00 | ms/batch 154.27 | loss  4.28 | ppl    72.34\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch   8 | time: 216.54s | valid loss  4.59 | valid ppl    98.90\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch   9 |   100/ 1327 batches | lr 20.00 | ms/batch 159.31 | loss  4.30 | ppl    73.45\n",
            "| epoch   9 |   200/ 1327 batches | lr 20.00 | ms/batch 155.96 | loss  4.38 | ppl    79.56\n",
            "| epoch   9 |   300/ 1327 batches | lr 20.00 | ms/batch 154.87 | loss  4.37 | ppl    79.40\n",
            "| epoch   9 |   400/ 1327 batches | lr 20.00 | ms/batch 156.04 | loss  4.29 | ppl    72.62\n",
            "| epoch   9 |   500/ 1327 batches | lr 20.00 | ms/batch 153.54 | loss  4.29 | ppl    72.67\n",
            "| epoch   9 |   600/ 1327 batches | lr 20.00 | ms/batch 155.61 | loss  4.36 | ppl    78.01\n",
            "| epoch   9 |   700/ 1327 batches | lr 20.00 | ms/batch 155.79 | loss  4.32 | ppl    74.87\n",
            "| epoch   9 |   800/ 1327 batches | lr 20.00 | ms/batch 155.54 | loss  4.23 | ppl    68.44\n",
            "| epoch   9 |   900/ 1327 batches | lr 20.00 | ms/batch 155.89 | loss  4.31 | ppl    74.71\n",
            "| epoch   9 |  1000/ 1327 batches | lr 20.00 | ms/batch 154.99 | loss  4.37 | ppl    79.41\n",
            "| epoch   9 |  1100/ 1327 batches | lr 20.00 | ms/batch 154.71 | loss  4.20 | ppl    67.00\n",
            "| epoch   9 |  1200/ 1327 batches | lr 20.00 | ms/batch 156.24 | loss  4.24 | ppl    69.32\n",
            "| epoch   9 |  1300/ 1327 batches | lr 20.00 | ms/batch 155.89 | loss  4.24 | ppl    69.23\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch   9 | time: 216.30s | valid loss  4.58 | valid ppl    97.80\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  10 |   100/ 1327 batches | lr 20.00 | ms/batch 159.44 | loss  4.23 | ppl    68.92\n",
            "| epoch  10 |   200/ 1327 batches | lr 20.00 | ms/batch 155.82 | loss  4.34 | ppl    76.45\n",
            "| epoch  10 |   300/ 1327 batches | lr 20.00 | ms/batch 156.25 | loss  4.33 | ppl    75.62\n",
            "| epoch  10 |   400/ 1327 batches | lr 20.00 | ms/batch 154.27 | loss  4.23 | ppl    69.02\n",
            "| epoch  10 |   500/ 1327 batches | lr 20.00 | ms/batch 155.72 | loss  4.23 | ppl    68.96\n",
            "| epoch  10 |   600/ 1327 batches | lr 20.00 | ms/batch 155.42 | loss  4.30 | ppl    73.94\n",
            "| epoch  10 |   700/ 1327 batches | lr 20.00 | ms/batch 156.02 | loss  4.27 | ppl    71.47\n",
            "| epoch  10 |   800/ 1327 batches | lr 20.00 | ms/batch 155.09 | loss  4.18 | ppl    65.30\n",
            "| epoch  10 |   900/ 1327 batches | lr 20.00 | ms/batch 154.38 | loss  4.28 | ppl    72.02\n",
            "| epoch  10 |  1000/ 1327 batches | lr 20.00 | ms/batch 155.10 | loss  4.33 | ppl    76.12\n",
            "| epoch  10 |  1100/ 1327 batches | lr 20.00 | ms/batch 153.42 | loss  4.17 | ppl    64.43\n",
            "| epoch  10 |  1200/ 1327 batches | lr 20.00 | ms/batch 155.73 | loss  4.19 | ppl    66.19\n",
            "| epoch  10 |  1300/ 1327 batches | lr 20.00 | ms/batch 156.19 | loss  4.20 | ppl    66.71\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  10 | time: 216.15s | valid loss  4.58 | valid ppl    97.08\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  11 |   100/ 1327 batches | lr 20.00 | ms/batch 159.63 | loss  4.20 | ppl    66.71\n",
            "| epoch  11 |   200/ 1327 batches | lr 20.00 | ms/batch 156.38 | loss  4.30 | ppl    73.33\n",
            "| epoch  11 |   300/ 1327 batches | lr 20.00 | ms/batch 155.78 | loss  4.29 | ppl    72.88\n",
            "| epoch  11 |   400/ 1327 batches | lr 20.00 | ms/batch 156.00 | loss  4.20 | ppl    66.45\n",
            "| epoch  11 |   500/ 1327 batches | lr 20.00 | ms/batch 155.26 | loss  4.20 | ppl    66.47\n",
            "| epoch  11 |   600/ 1327 batches | lr 20.00 | ms/batch 155.70 | loss  4.27 | ppl    71.48\n",
            "| epoch  11 |   700/ 1327 batches | lr 20.00 | ms/batch 156.06 | loss  4.23 | ppl    68.41\n",
            "| epoch  11 |   800/ 1327 batches | lr 20.00 | ms/batch 155.40 | loss  4.14 | ppl    62.98\n",
            "| epoch  11 |   900/ 1327 batches | lr 20.00 | ms/batch 155.69 | loss  4.24 | ppl    69.06\n",
            "| epoch  11 |  1000/ 1327 batches | lr 20.00 | ms/batch 154.09 | loss  4.29 | ppl    73.24\n",
            "| epoch  11 |  1100/ 1327 batches | lr 20.00 | ms/batch 154.83 | loss  4.13 | ppl    62.34\n",
            "| epoch  11 |  1200/ 1327 batches | lr 20.00 | ms/batch 154.28 | loss  4.16 | ppl    63.86\n",
            "| epoch  11 |  1300/ 1327 batches | lr 20.00 | ms/batch 155.09 | loss  4.17 | ppl    64.40\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  11 | time: 216.06s | valid loss  4.57 | valid ppl    96.88\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  12 |   100/ 1327 batches | lr 20.00 | ms/batch 158.47 | loss  4.16 | ppl    64.30\n",
            "| epoch  12 |   200/ 1327 batches | lr 20.00 | ms/batch 155.22 | loss  4.25 | ppl    70.18\n",
            "| epoch  12 |   300/ 1327 batches | lr 20.00 | ms/batch 153.07 | loss  4.26 | ppl    70.51\n",
            "| epoch  12 |   400/ 1327 batches | lr 20.00 | ms/batch 155.10 | loss  4.16 | ppl    64.07\n",
            "| epoch  12 |   500/ 1327 batches | lr 20.00 | ms/batch 156.05 | loss  4.16 | ppl    64.28\n",
            "| epoch  12 |   600/ 1327 batches | lr 20.00 | ms/batch 152.45 | loss  4.22 | ppl    68.24\n",
            "| epoch  12 |   700/ 1327 batches | lr 20.00 | ms/batch 155.26 | loss  4.20 | ppl    66.37\n",
            "| epoch  12 |   800/ 1327 batches | lr 20.00 | ms/batch 155.59 | loss  4.11 | ppl    60.71\n",
            "| epoch  12 |   900/ 1327 batches | lr 20.00 | ms/batch 155.26 | loss  4.21 | ppl    67.21\n",
            "| epoch  12 |  1000/ 1327 batches | lr 20.00 | ms/batch 155.96 | loss  4.26 | ppl    70.52\n",
            "| epoch  12 |  1100/ 1327 batches | lr 20.00 | ms/batch 155.89 | loss  4.10 | ppl    60.27\n",
            "| epoch  12 |  1200/ 1327 batches | lr 20.00 | ms/batch 156.23 | loss  4.12 | ppl    61.84\n",
            "| epoch  12 |  1300/ 1327 batches | lr 20.00 | ms/batch 156.02 | loss  4.13 | ppl    62.41\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  12 | time: 215.99s | valid loss  4.57 | valid ppl    96.40\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  13 |   100/ 1327 batches | lr 20.00 | ms/batch 159.46 | loss  4.14 | ppl    62.75\n",
            "| epoch  13 |   200/ 1327 batches | lr 20.00 | ms/batch 154.09 | loss  4.22 | ppl    67.77\n",
            "| epoch  13 |   300/ 1327 batches | lr 20.00 | ms/batch 155.36 | loss  4.22 | ppl    67.79\n",
            "| epoch  13 |   400/ 1327 batches | lr 20.00 | ms/batch 156.04 | loss  4.13 | ppl    62.46\n",
            "| epoch  13 |   500/ 1327 batches | lr 20.00 | ms/batch 154.52 | loss  4.13 | ppl    62.07\n",
            "| epoch  13 |   600/ 1327 batches | lr 20.00 | ms/batch 154.00 | loss  4.20 | ppl    66.78\n",
            "| epoch  13 |   700/ 1327 batches | lr 20.00 | ms/batch 153.76 | loss  4.17 | ppl    64.45\n",
            "| epoch  13 |   800/ 1327 batches | lr 20.00 | ms/batch 156.23 | loss  4.09 | ppl    59.45\n",
            "| epoch  13 |   900/ 1327 batches | lr 20.00 | ms/batch 153.32 | loss  4.18 | ppl    65.15\n",
            "| epoch  13 |  1000/ 1327 batches | lr 20.00 | ms/batch 156.11 | loss  4.22 | ppl    68.24\n",
            "| epoch  13 |  1100/ 1327 batches | lr 20.00 | ms/batch 156.35 | loss  4.07 | ppl    58.82\n",
            "| epoch  13 |  1200/ 1327 batches | lr 20.00 | ms/batch 156.44 | loss  4.09 | ppl    60.02\n",
            "| epoch  13 |  1300/ 1327 batches | lr 20.00 | ms/batch 154.52 | loss  4.11 | ppl    60.66\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  13 | time: 215.88s | valid loss  4.56 | valid ppl    95.85\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  14 |   100/ 1327 batches | lr 20.00 | ms/batch 159.77 | loss  4.11 | ppl    61.06\n",
            "| epoch  14 |   200/ 1327 batches | lr 20.00 | ms/batch 155.48 | loss  4.19 | ppl    65.81\n",
            "| epoch  14 |   300/ 1327 batches | lr 20.00 | ms/batch 155.05 | loss  4.20 | ppl    66.71\n",
            "| epoch  14 |   400/ 1327 batches | lr 20.00 | ms/batch 155.69 | loss  4.10 | ppl    60.52\n",
            "| epoch  14 |   500/ 1327 batches | lr 20.00 | ms/batch 156.30 | loss  4.11 | ppl    60.96\n",
            "| epoch  14 |   600/ 1327 batches | lr 20.00 | ms/batch 157.00 | loss  4.18 | ppl    65.12\n",
            "| epoch  14 |   700/ 1327 batches | lr 20.00 | ms/batch 153.90 | loss  4.14 | ppl    62.56\n",
            "| epoch  14 |   800/ 1327 batches | lr 20.00 | ms/batch 155.40 | loss  4.05 | ppl    57.68\n",
            "| epoch  14 |   900/ 1327 batches | lr 20.00 | ms/batch 156.63 | loss  4.15 | ppl    63.43\n",
            "| epoch  14 |  1000/ 1327 batches | lr 20.00 | ms/batch 156.38 | loss  4.21 | ppl    67.02\n",
            "| epoch  14 |  1100/ 1327 batches | lr 20.00 | ms/batch 156.17 | loss  4.04 | ppl    56.65\n",
            "| epoch  14 |  1200/ 1327 batches | lr 20.00 | ms/batch 156.05 | loss  4.07 | ppl    58.62\n",
            "| epoch  14 |  1300/ 1327 batches | lr 20.00 | ms/batch 156.67 | loss  4.07 | ppl    58.82\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  14 | time: 216.97s | valid loss  4.57 | valid ppl    96.41\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  15 |   100/ 1327 batches | lr 5.00 | ms/batch 154.09 | loss  4.08 | ppl    58.89\n",
            "| epoch  15 |   200/ 1327 batches | lr 5.00 | ms/batch 156.08 | loss  4.13 | ppl    62.29\n",
            "| epoch  15 |   300/ 1327 batches | lr 5.00 | ms/batch 156.28 | loss  4.10 | ppl    60.25\n",
            "| epoch  15 |   400/ 1327 batches | lr 5.00 | ms/batch 155.42 | loss  3.97 | ppl    53.25\n",
            "| epoch  15 |   500/ 1327 batches | lr 5.00 | ms/batch 155.87 | loss  3.97 | ppl    52.97\n",
            "| epoch  15 |   600/ 1327 batches | lr 5.00 | ms/batch 156.13 | loss  4.03 | ppl    55.99\n",
            "| epoch  15 |   700/ 1327 batches | lr 5.00 | ms/batch 155.75 | loss  3.96 | ppl    52.61\n",
            "| epoch  15 |   800/ 1327 batches | lr 5.00 | ms/batch 156.09 | loss  3.86 | ppl    47.38\n",
            "| epoch  15 |   900/ 1327 batches | lr 5.00 | ms/batch 155.80 | loss  3.94 | ppl    51.48\n",
            "| epoch  15 |  1000/ 1327 batches | lr 5.00 | ms/batch 156.08 | loss  3.97 | ppl    52.92\n",
            "| epoch  15 |  1100/ 1327 batches | lr 5.00 | ms/batch 154.40 | loss  3.80 | ppl    44.69\n",
            "| epoch  15 |  1200/ 1327 batches | lr 5.00 | ms/batch 156.13 | loss  3.80 | ppl    44.87\n",
            "| epoch  15 |  1300/ 1327 batches | lr 5.00 | ms/batch 154.27 | loss  3.79 | ppl    44.28\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  15 | time: 215.92s | valid loss  4.51 | valid ppl    91.30\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  16 |   100/ 1327 batches | lr 5.00 | ms/batch 159.00 | loss  3.93 | ppl    51.09\n",
            "| epoch  16 |   200/ 1327 batches | lr 5.00 | ms/batch 155.88 | loss  4.01 | ppl    55.05\n",
            "| epoch  16 |   300/ 1327 batches | lr 5.00 | ms/batch 156.12 | loss  4.00 | ppl    54.75\n",
            "| epoch  16 |   400/ 1327 batches | lr 5.00 | ms/batch 155.56 | loss  3.89 | ppl    48.69\n",
            "| epoch  16 |   500/ 1327 batches | lr 5.00 | ms/batch 154.51 | loss  3.88 | ppl    48.67\n",
            "| epoch  16 |   600/ 1327 batches | lr 5.00 | ms/batch 155.47 | loss  3.95 | ppl    51.98\n",
            "| epoch  16 |   700/ 1327 batches | lr 5.00 | ms/batch 153.37 | loss  3.90 | ppl    49.23\n",
            "| epoch  16 |   800/ 1327 batches | lr 5.00 | ms/batch 155.58 | loss  3.81 | ppl    44.98\n",
            "| epoch  16 |   900/ 1327 batches | lr 5.00 | ms/batch 156.15 | loss  3.89 | ppl    48.72\n",
            "| epoch  16 |  1000/ 1327 batches | lr 5.00 | ms/batch 155.40 | loss  3.93 | ppl    51.06\n",
            "| epoch  16 |  1100/ 1327 batches | lr 5.00 | ms/batch 155.00 | loss  3.76 | ppl    42.83\n",
            "| epoch  16 |  1200/ 1327 batches | lr 5.00 | ms/batch 154.10 | loss  3.76 | ppl    42.94\n",
            "| epoch  16 |  1300/ 1327 batches | lr 5.00 | ms/batch 154.25 | loss  3.76 | ppl    43.14\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  16 | time: 215.81s | valid loss  4.51 | valid ppl    90.97\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  17 |   100/ 1327 batches | lr 5.00 | ms/batch 158.74 | loss  3.89 | ppl    48.96\n",
            "| epoch  17 |   200/ 1327 batches | lr 5.00 | ms/batch 155.77 | loss  3.96 | ppl    52.66\n",
            "| epoch  17 |   300/ 1327 batches | lr 5.00 | ms/batch 155.71 | loss  3.95 | ppl    52.11\n",
            "| epoch  17 |   400/ 1327 batches | lr 5.00 | ms/batch 155.97 | loss  3.85 | ppl    46.97\n",
            "| epoch  17 |   500/ 1327 batches | lr 5.00 | ms/batch 154.30 | loss  3.85 | ppl    46.92\n",
            "| epoch  17 |   600/ 1327 batches | lr 5.00 | ms/batch 155.87 | loss  3.91 | ppl    49.84\n",
            "| epoch  17 |   700/ 1327 batches | lr 5.00 | ms/batch 156.61 | loss  3.86 | ppl    47.24\n",
            "| epoch  17 |   800/ 1327 batches | lr 5.00 | ms/batch 156.01 | loss  3.77 | ppl    43.37\n",
            "| epoch  17 |   900/ 1327 batches | lr 5.00 | ms/batch 155.80 | loss  3.86 | ppl    47.48\n",
            "| epoch  17 |  1000/ 1327 batches | lr 5.00 | ms/batch 156.30 | loss  3.90 | ppl    49.26\n",
            "| epoch  17 |  1100/ 1327 batches | lr 5.00 | ms/batch 156.08 | loss  3.74 | ppl    41.92\n",
            "| epoch  17 |  1200/ 1327 batches | lr 5.00 | ms/batch 156.12 | loss  3.75 | ppl    42.37\n",
            "| epoch  17 |  1300/ 1327 batches | lr 5.00 | ms/batch 153.90 | loss  3.74 | ppl    41.98\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  17 | time: 216.52s | valid loss  4.50 | valid ppl    90.44\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  18 |   100/ 1327 batches | lr 5.00 | ms/batch 159.41 | loss  3.85 | ppl    47.14\n",
            "| epoch  18 |   200/ 1327 batches | lr 5.00 | ms/batch 156.17 | loss  3.92 | ppl    50.52\n",
            "| epoch  18 |   300/ 1327 batches | lr 5.00 | ms/batch 154.61 | loss  3.92 | ppl    50.46\n",
            "| epoch  18 |   400/ 1327 batches | lr 5.00 | ms/batch 156.02 | loss  3.82 | ppl    45.44\n",
            "| epoch  18 |   500/ 1327 batches | lr 5.00 | ms/batch 155.93 | loss  3.81 | ppl    45.08\n",
            "| epoch  18 |   600/ 1327 batches | lr 5.00 | ms/batch 154.77 | loss  3.87 | ppl    48.17\n",
            "| epoch  18 |   700/ 1327 batches | lr 5.00 | ms/batch 154.30 | loss  3.82 | ppl    45.66\n",
            "| epoch  18 |   800/ 1327 batches | lr 5.00 | ms/batch 155.67 | loss  3.74 | ppl    42.10\n",
            "| epoch  18 |   900/ 1327 batches | lr 5.00 | ms/batch 155.43 | loss  3.83 | ppl    45.84\n",
            "| epoch  18 |  1000/ 1327 batches | lr 5.00 | ms/batch 155.40 | loss  3.87 | ppl    48.14\n",
            "| epoch  18 |  1100/ 1327 batches | lr 5.00 | ms/batch 153.82 | loss  3.70 | ppl    40.33\n",
            "| epoch  18 |  1200/ 1327 batches | lr 5.00 | ms/batch 155.50 | loss  3.72 | ppl    41.28\n",
            "| epoch  18 |  1300/ 1327 batches | lr 5.00 | ms/batch 155.41 | loss  3.72 | ppl    41.40\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  18 | time: 216.05s | valid loss  4.51 | valid ppl    90.65\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  19 |   100/ 1327 batches | lr 1.25 | ms/batch 156.35 | loss  3.83 | ppl    46.03\n",
            "| epoch  19 |   200/ 1327 batches | lr 1.25 | ms/batch 155.56 | loss  3.90 | ppl    49.51\n",
            "| epoch  19 |   300/ 1327 batches | lr 1.25 | ms/batch 154.92 | loss  3.89 | ppl    49.08\n",
            "| epoch  19 |   400/ 1327 batches | lr 1.25 | ms/batch 155.53 | loss  3.78 | ppl    43.80\n",
            "| epoch  19 |   500/ 1327 batches | lr 1.25 | ms/batch 153.71 | loss  3.78 | ppl    43.77\n",
            "| epoch  19 |   600/ 1327 batches | lr 1.25 | ms/batch 154.32 | loss  3.84 | ppl    46.55\n",
            "| epoch  19 |   700/ 1327 batches | lr 1.25 | ms/batch 154.65 | loss  3.79 | ppl    44.08\n",
            "| epoch  19 |   800/ 1327 batches | lr 1.25 | ms/batch 154.27 | loss  3.69 | ppl    40.03\n",
            "| epoch  19 |   900/ 1327 batches | lr 1.25 | ms/batch 154.71 | loss  3.77 | ppl    43.56\n",
            "| epoch  19 |  1000/ 1327 batches | lr 1.25 | ms/batch 155.83 | loss  3.81 | ppl    45.27\n",
            "| epoch  19 |  1100/ 1327 batches | lr 1.25 | ms/batch 156.37 | loss  3.65 | ppl    38.34\n",
            "| epoch  19 |  1200/ 1327 batches | lr 1.25 | ms/batch 155.71 | loss  3.64 | ppl    38.12\n",
            "| epoch  19 |  1300/ 1327 batches | lr 1.25 | ms/batch 155.61 | loss  3.64 | ppl    37.93\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  19 | time: 215.63s | valid loss  4.50 | valid ppl    89.57\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  20 |   100/ 1327 batches | lr 1.25 | ms/batch 159.55 | loss  3.79 | ppl    44.30\n",
            "| epoch  20 |   200/ 1327 batches | lr 1.25 | ms/batch 155.12 | loss  3.87 | ppl    47.91\n",
            "| epoch  20 |   300/ 1327 batches | lr 1.25 | ms/batch 155.53 | loss  3.87 | ppl    47.89\n",
            "| epoch  20 |   400/ 1327 batches | lr 1.25 | ms/batch 156.05 | loss  3.76 | ppl    42.90\n",
            "| epoch  20 |   500/ 1327 batches | lr 1.25 | ms/batch 155.02 | loss  3.75 | ppl    42.54\n",
            "| epoch  20 |   600/ 1327 batches | lr 1.25 | ms/batch 156.29 | loss  3.82 | ppl    45.54\n",
            "| epoch  20 |   700/ 1327 batches | lr 1.25 | ms/batch 155.51 | loss  3.76 | ppl    43.01\n",
            "| epoch  20 |   800/ 1327 batches | lr 1.25 | ms/batch 155.44 | loss  3.67 | ppl    39.34\n",
            "| epoch  20 |   900/ 1327 batches | lr 1.25 | ms/batch 155.96 | loss  3.76 | ppl    42.85\n",
            "| epoch  20 |  1000/ 1327 batches | lr 1.25 | ms/batch 155.91 | loss  3.80 | ppl    44.69\n",
            "| epoch  20 |  1100/ 1327 batches | lr 1.25 | ms/batch 153.81 | loss  3.63 | ppl    37.62\n",
            "| epoch  20 |  1200/ 1327 batches | lr 1.25 | ms/batch 154.64 | loss  3.64 | ppl    38.10\n",
            "| epoch  20 |  1300/ 1327 batches | lr 1.25 | ms/batch 155.64 | loss  3.63 | ppl    37.83\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  20 | time: 216.25s | valid loss  4.49 | valid ppl    89.49\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  21 |   100/ 1327 batches | lr 1.25 | ms/batch 158.52 | loss  3.78 | ppl    43.62\n",
            "| epoch  21 |   200/ 1327 batches | lr 1.25 | ms/batch 155.88 | loss  3.86 | ppl    47.27\n",
            "| epoch  21 |   300/ 1327 batches | lr 1.25 | ms/batch 154.82 | loss  3.85 | ppl    46.88\n",
            "| epoch  21 |   400/ 1327 batches | lr 1.25 | ms/batch 154.86 | loss  3.74 | ppl    42.17\n",
            "| epoch  21 |   500/ 1327 batches | lr 1.25 | ms/batch 154.50 | loss  3.74 | ppl    42.16\n",
            "| epoch  21 |   600/ 1327 batches | lr 1.25 | ms/batch 153.52 | loss  3.81 | ppl    45.15\n",
            "| epoch  21 |   700/ 1327 batches | lr 1.25 | ms/batch 155.23 | loss  3.75 | ppl    42.36\n",
            "| epoch  21 |   800/ 1327 batches | lr 1.25 | ms/batch 154.29 | loss  3.66 | ppl    38.88\n",
            "| epoch  21 |   900/ 1327 batches | lr 1.25 | ms/batch 155.77 | loss  3.75 | ppl    42.60\n",
            "| epoch  21 |  1000/ 1327 batches | lr 1.25 | ms/batch 155.52 | loss  3.80 | ppl    44.58\n",
            "| epoch  21 |  1100/ 1327 batches | lr 1.25 | ms/batch 155.45 | loss  3.62 | ppl    37.16\n",
            "| epoch  21 |  1200/ 1327 batches | lr 1.25 | ms/batch 155.26 | loss  3.63 | ppl    37.58\n",
            "| epoch  21 |  1300/ 1327 batches | lr 1.25 | ms/batch 155.91 | loss  3.62 | ppl    37.48\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  21 | time: 215.85s | valid loss  4.49 | valid ppl    89.41\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  22 |   100/ 1327 batches | lr 1.25 | ms/batch 158.58 | loss  3.77 | ppl    43.30\n",
            "| epoch  22 |   200/ 1327 batches | lr 1.25 | ms/batch 155.30 | loss  3.84 | ppl    46.60\n",
            "| epoch  22 |   300/ 1327 batches | lr 1.25 | ms/batch 154.22 | loss  3.84 | ppl    46.47\n",
            "| epoch  22 |   400/ 1327 batches | lr 1.25 | ms/batch 153.78 | loss  3.73 | ppl    41.88\n",
            "| epoch  22 |   500/ 1327 batches | lr 1.25 | ms/batch 155.95 | loss  3.73 | ppl    41.80\n",
            "| epoch  22 |   600/ 1327 batches | lr 1.25 | ms/batch 155.70 | loss  3.80 | ppl    44.61\n",
            "| epoch  22 |   700/ 1327 batches | lr 1.25 | ms/batch 155.61 | loss  3.74 | ppl    42.24\n",
            "| epoch  22 |   800/ 1327 batches | lr 1.25 | ms/batch 155.00 | loss  3.65 | ppl    38.49\n",
            "| epoch  22 |   900/ 1327 batches | lr 1.25 | ms/batch 155.07 | loss  3.74 | ppl    42.24\n",
            "| epoch  22 |  1000/ 1327 batches | lr 1.25 | ms/batch 154.96 | loss  3.78 | ppl    43.70\n",
            "| epoch  22 |  1100/ 1327 batches | lr 1.25 | ms/batch 155.77 | loss  3.62 | ppl    37.26\n",
            "| epoch  22 |  1200/ 1327 batches | lr 1.25 | ms/batch 155.08 | loss  3.62 | ppl    37.46\n",
            "| epoch  22 |  1300/ 1327 batches | lr 1.25 | ms/batch 155.42 | loss  3.62 | ppl    37.49\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  22 | time: 215.74s | valid loss  4.49 | valid ppl    89.26\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  23 |   100/ 1327 batches | lr 1.25 | ms/batch 159.00 | loss  3.76 | ppl    42.77\n",
            "| epoch  23 |   200/ 1327 batches | lr 1.25 | ms/batch 156.33 | loss  3.83 | ppl    46.23\n",
            "| epoch  23 |   300/ 1327 batches | lr 1.25 | ms/batch 155.98 | loss  3.83 | ppl    45.91\n",
            "| epoch  23 |   400/ 1327 batches | lr 1.25 | ms/batch 155.99 | loss  3.72 | ppl    41.33\n",
            "| epoch  23 |   500/ 1327 batches | lr 1.25 | ms/batch 154.41 | loss  3.71 | ppl    40.97\n",
            "| epoch  23 |   600/ 1327 batches | lr 1.25 | ms/batch 155.03 | loss  3.78 | ppl    43.78\n",
            "| epoch  23 |   700/ 1327 batches | lr 1.25 | ms/batch 155.33 | loss  3.73 | ppl    41.51\n",
            "| epoch  23 |   800/ 1327 batches | lr 1.25 | ms/batch 155.79 | loss  3.64 | ppl    38.25\n",
            "| epoch  23 |   900/ 1327 batches | lr 1.25 | ms/batch 155.26 | loss  3.74 | ppl    42.04\n",
            "| epoch  23 |  1000/ 1327 batches | lr 1.25 | ms/batch 153.01 | loss  3.77 | ppl    43.44\n",
            "| epoch  23 |  1100/ 1327 batches | lr 1.25 | ms/batch 156.17 | loss  3.61 | ppl    36.85\n",
            "| epoch  23 |  1200/ 1327 batches | lr 1.25 | ms/batch 155.90 | loss  3.62 | ppl    37.30\n",
            "| epoch  23 |  1300/ 1327 batches | lr 1.25 | ms/batch 155.59 | loss  3.62 | ppl    37.20\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  23 | time: 216.25s | valid loss  4.49 | valid ppl    89.24\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  24 |   100/ 1327 batches | lr 1.25 | ms/batch 159.34 | loss  3.74 | ppl    42.16\n",
            "| epoch  24 |   200/ 1327 batches | lr 1.25 | ms/batch 155.61 | loss  3.82 | ppl    45.73\n",
            "| epoch  24 |   300/ 1327 batches | lr 1.25 | ms/batch 155.53 | loss  3.83 | ppl    45.87\n",
            "| epoch  24 |   400/ 1327 batches | lr 1.25 | ms/batch 155.63 | loss  3.71 | ppl    40.80\n",
            "| epoch  24 |   500/ 1327 batches | lr 1.25 | ms/batch 155.72 | loss  3.71 | ppl    40.72\n",
            "| epoch  24 |   600/ 1327 batches | lr 1.25 | ms/batch 154.95 | loss  3.77 | ppl    43.44\n",
            "| epoch  24 |   700/ 1327 batches | lr 1.25 | ms/batch 154.01 | loss  3.72 | ppl    41.34\n",
            "| epoch  24 |   800/ 1327 batches | lr 1.25 | ms/batch 155.78 | loss  3.64 | ppl    38.15\n",
            "| epoch  24 |   900/ 1327 batches | lr 1.25 | ms/batch 155.05 | loss  3.73 | ppl    41.48\n",
            "| epoch  24 |  1000/ 1327 batches | lr 1.25 | ms/batch 154.61 | loss  3.77 | ppl    43.37\n",
            "| epoch  24 |  1100/ 1327 batches | lr 1.25 | ms/batch 155.17 | loss  3.60 | ppl    36.56\n",
            "| epoch  24 |  1200/ 1327 batches | lr 1.25 | ms/batch 155.63 | loss  3.61 | ppl    37.00\n",
            "| epoch  24 |  1300/ 1327 batches | lr 1.25 | ms/batch 155.77 | loss  3.62 | ppl    37.17\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  24 | time: 216.11s | valid loss  4.49 | valid ppl    89.29\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  25 |   100/ 1327 batches | lr 0.31 | ms/batch 153.72 | loss  3.74 | ppl    42.18\n",
            "| epoch  25 |   200/ 1327 batches | lr 0.31 | ms/batch 151.59 | loss  3.81 | ppl    45.33\n",
            "| epoch  25 |   300/ 1327 batches | lr 0.31 | ms/batch 152.90 | loss  3.82 | ppl    45.38\n",
            "| epoch  25 |   400/ 1327 batches | lr 0.31 | ms/batch 153.78 | loss  3.71 | ppl    40.76\n",
            "| epoch  25 |   500/ 1327 batches | lr 0.31 | ms/batch 153.40 | loss  3.71 | ppl    40.70\n",
            "| epoch  25 |   600/ 1327 batches | lr 0.31 | ms/batch 153.89 | loss  3.77 | ppl    43.41\n",
            "| epoch  25 |   700/ 1327 batches | lr 0.31 | ms/batch 149.12 | loss  3.71 | ppl    40.90\n",
            "| epoch  25 |   800/ 1327 batches | lr 0.31 | ms/batch 151.91 | loss  3.62 | ppl    37.39\n",
            "| epoch  25 |   900/ 1327 batches | lr 0.31 | ms/batch 152.13 | loss  3.71 | ppl    40.84\n",
            "| epoch  25 |  1000/ 1327 batches | lr 0.31 | ms/batch 153.38 | loss  3.75 | ppl    42.43\n",
            "| epoch  25 |  1100/ 1327 batches | lr 0.31 | ms/batch 153.19 | loss  3.57 | ppl    35.53\n",
            "| epoch  25 |  1200/ 1327 batches | lr 0.31 | ms/batch 153.62 | loss  3.59 | ppl    36.26\n",
            "| epoch  25 |  1300/ 1327 batches | lr 0.31 | ms/batch 153.62 | loss  3.59 | ppl    36.24\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  25 | time: 212.28s | valid loss  4.49 | valid ppl    88.90\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  26 |   100/ 1327 batches | lr 0.31 | ms/batch 156.82 | loss  3.74 | ppl    41.94\n",
            "| epoch  26 |   200/ 1327 batches | lr 0.31 | ms/batch 151.72 | loss  3.82 | ppl    45.49\n",
            "| epoch  26 |   300/ 1327 batches | lr 0.31 | ms/batch 150.90 | loss  3.81 | ppl    45.17\n",
            "| epoch  26 |   400/ 1327 batches | lr 0.31 | ms/batch 151.66 | loss  3.69 | ppl    40.12\n",
            "| epoch  26 |   500/ 1327 batches | lr 0.31 | ms/batch 150.21 | loss  3.69 | ppl    40.24\n",
            "| epoch  26 |   600/ 1327 batches | lr 0.31 | ms/batch 151.48 | loss  3.76 | ppl    43.02\n",
            "| epoch  26 |   700/ 1327 batches | lr 0.31 | ms/batch 151.17 | loss  3.70 | ppl    40.55\n",
            "| epoch  26 |   800/ 1327 batches | lr 0.31 | ms/batch 152.26 | loss  3.62 | ppl    37.31\n",
            "| epoch  26 |   900/ 1327 batches | lr 0.31 | ms/batch 153.06 | loss  3.70 | ppl    40.52\n",
            "| epoch  26 |  1000/ 1327 batches | lr 0.31 | ms/batch 153.20 | loss  3.74 | ppl    42.23\n",
            "| epoch  26 |  1100/ 1327 batches | lr 0.31 | ms/batch 151.72 | loss  3.58 | ppl    35.81\n",
            "| epoch  26 |  1200/ 1327 batches | lr 0.31 | ms/batch 153.80 | loss  3.58 | ppl    36.04\n",
            "| epoch  26 |  1300/ 1327 batches | lr 0.31 | ms/batch 152.97 | loss  3.59 | ppl    36.11\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  26 | time: 211.81s | valid loss  4.49 | valid ppl    88.80\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  27 |   100/ 1327 batches | lr 0.31 | ms/batch 158.15 | loss  3.73 | ppl    41.64\n",
            "| epoch  27 |   200/ 1327 batches | lr 0.31 | ms/batch 154.08 | loss  3.80 | ppl    44.86\n",
            "| epoch  27 |   300/ 1327 batches | lr 0.31 | ms/batch 153.52 | loss  3.80 | ppl    44.73\n",
            "| epoch  27 |   400/ 1327 batches | lr 0.31 | ms/batch 152.59 | loss  3.69 | ppl    39.97\n",
            "| epoch  27 |   500/ 1327 batches | lr 0.31 | ms/batch 151.97 | loss  3.69 | ppl    39.96\n",
            "| epoch  27 |   600/ 1327 batches | lr 0.31 | ms/batch 152.82 | loss  3.76 | ppl    42.81\n",
            "| epoch  27 |   700/ 1327 batches | lr 0.31 | ms/batch 153.71 | loss  3.70 | ppl    40.56\n",
            "| epoch  27 |   800/ 1327 batches | lr 0.31 | ms/batch 154.15 | loss  3.62 | ppl    37.39\n",
            "| epoch  27 |   900/ 1327 batches | lr 0.31 | ms/batch 155.40 | loss  3.70 | ppl    40.45\n",
            "| epoch  27 |  1000/ 1327 batches | lr 0.31 | ms/batch 154.62 | loss  3.75 | ppl    42.37\n",
            "| epoch  27 |  1100/ 1327 batches | lr 0.31 | ms/batch 152.07 | loss  3.58 | ppl    35.73\n",
            "| epoch  27 |  1200/ 1327 batches | lr 0.31 | ms/batch 154.13 | loss  3.58 | ppl    36.01\n",
            "| epoch  27 |  1300/ 1327 batches | lr 0.31 | ms/batch 154.29 | loss  3.58 | ppl    35.98\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  27 | time: 213.83s | valid loss  4.49 | valid ppl    88.83\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  28 |   100/ 1327 batches | lr 0.08 | ms/batch 154.30 | loss  3.73 | ppl    41.57\n",
            "| epoch  28 |   200/ 1327 batches | lr 0.08 | ms/batch 154.03 | loss  3.80 | ppl    44.85\n",
            "| epoch  28 |   300/ 1327 batches | lr 0.08 | ms/batch 153.90 | loss  3.80 | ppl    44.67\n",
            "| epoch  28 |   400/ 1327 batches | lr 0.08 | ms/batch 153.34 | loss  3.68 | ppl    39.82\n",
            "| epoch  28 |   500/ 1327 batches | lr 0.08 | ms/batch 153.17 | loss  3.70 | ppl    40.30\n",
            "| epoch  28 |   600/ 1327 batches | lr 0.08 | ms/batch 153.94 | loss  3.76 | ppl    42.77\n",
            "| epoch  28 |   700/ 1327 batches | lr 0.08 | ms/batch 154.22 | loss  3.69 | ppl    40.13\n",
            "| epoch  28 |   800/ 1327 batches | lr 0.08 | ms/batch 154.69 | loss  3.61 | ppl    36.98\n",
            "| epoch  28 |   900/ 1327 batches | lr 0.08 | ms/batch 154.32 | loss  3.71 | ppl    40.69\n",
            "| epoch  28 |  1000/ 1327 batches | lr 0.08 | ms/batch 153.16 | loss  3.74 | ppl    42.24\n",
            "| epoch  28 |  1100/ 1327 batches | lr 0.08 | ms/batch 154.31 | loss  3.57 | ppl    35.56\n",
            "| epoch  28 |  1200/ 1327 batches | lr 0.08 | ms/batch 153.81 | loss  3.58 | ppl    35.94\n",
            "| epoch  28 |  1300/ 1327 batches | lr 0.08 | ms/batch 155.27 | loss  3.58 | ppl    35.90\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  28 | time: 214.05s | valid loss  4.49 | valid ppl    88.76\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  29 |   100/ 1327 batches | lr 0.08 | ms/batch 158.32 | loss  3.73 | ppl    41.73\n",
            "| epoch  29 |   200/ 1327 batches | lr 0.08 | ms/batch 154.75 | loss  3.81 | ppl    44.94\n",
            "| epoch  29 |   300/ 1327 batches | lr 0.08 | ms/batch 154.68 | loss  3.80 | ppl    44.59\n",
            "| epoch  29 |   400/ 1327 batches | lr 0.08 | ms/batch 154.68 | loss  3.69 | ppl    39.94\n",
            "| epoch  29 |   500/ 1327 batches | lr 0.08 | ms/batch 154.31 | loss  3.69 | ppl    40.05\n",
            "| epoch  29 |   600/ 1327 batches | lr 0.08 | ms/batch 155.15 | loss  3.75 | ppl    42.71\n",
            "| epoch  29 |   700/ 1327 batches | lr 0.08 | ms/batch 152.34 | loss  3.69 | ppl    40.09\n",
            "| epoch  29 |   800/ 1327 batches | lr 0.08 | ms/batch 153.50 | loss  3.61 | ppl    37.01\n",
            "| epoch  29 |   900/ 1327 batches | lr 0.08 | ms/batch 153.43 | loss  3.70 | ppl    40.35\n",
            "| epoch  29 |  1000/ 1327 batches | lr 0.08 | ms/batch 152.98 | loss  3.74 | ppl    42.17\n",
            "| epoch  29 |  1100/ 1327 batches | lr 0.08 | ms/batch 153.73 | loss  3.57 | ppl    35.55\n",
            "| epoch  29 |  1200/ 1327 batches | lr 0.08 | ms/batch 154.82 | loss  3.59 | ppl    36.07\n",
            "| epoch  29 |  1300/ 1327 batches | lr 0.08 | ms/batch 152.96 | loss  3.58 | ppl    35.79\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  29 | time: 214.28s | valid loss  4.49 | valid ppl    88.72\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  30 |   100/ 1327 batches | lr 0.08 | ms/batch 157.35 | loss  3.73 | ppl    41.70\n",
            "| epoch  30 |   200/ 1327 batches | lr 0.08 | ms/batch 153.08 | loss  3.79 | ppl    44.37\n",
            "| epoch  30 |   300/ 1327 batches | lr 0.08 | ms/batch 154.40 | loss  3.79 | ppl    44.46\n",
            "| epoch  30 |   400/ 1327 batches | lr 0.08 | ms/batch 154.83 | loss  3.69 | ppl    40.04\n",
            "| epoch  30 |   500/ 1327 batches | lr 0.08 | ms/batch 154.44 | loss  3.69 | ppl    39.88\n",
            "| epoch  30 |   600/ 1327 batches | lr 0.08 | ms/batch 154.81 | loss  3.75 | ppl    42.49\n",
            "| epoch  30 |   700/ 1327 batches | lr 0.08 | ms/batch 154.81 | loss  3.70 | ppl    40.29\n",
            "| epoch  30 |   800/ 1327 batches | lr 0.08 | ms/batch 154.25 | loss  3.60 | ppl    36.77\n",
            "| epoch  30 |   900/ 1327 batches | lr 0.08 | ms/batch 154.29 | loss  3.69 | ppl    40.06\n",
            "| epoch  30 |  1000/ 1327 batches | lr 0.08 | ms/batch 154.59 | loss  3.74 | ppl    41.90\n",
            "| epoch  30 |  1100/ 1327 batches | lr 0.08 | ms/batch 154.40 | loss  3.57 | ppl    35.37\n",
            "| epoch  30 |  1200/ 1327 batches | lr 0.08 | ms/batch 153.82 | loss  3.58 | ppl    35.85\n",
            "| epoch  30 |  1300/ 1327 batches | lr 0.08 | ms/batch 151.61 | loss  3.58 | ppl    35.86\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  30 | time: 214.44s | valid loss  4.49 | valid ppl    88.71\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  31 |   100/ 1327 batches | lr 0.08 | ms/batch 159.13 | loss  3.72 | ppl    41.20\n",
            "| epoch  31 |   200/ 1327 batches | lr 0.08 | ms/batch 154.55 | loss  3.80 | ppl    44.76\n",
            "| epoch  31 |   300/ 1327 batches | lr 0.08 | ms/batch 154.91 | loss  3.80 | ppl    44.49\n",
            "| epoch  31 |   400/ 1327 batches | lr 0.08 | ms/batch 154.67 | loss  3.69 | ppl    39.86\n",
            "| epoch  31 |   500/ 1327 batches | lr 0.08 | ms/batch 154.29 | loss  3.69 | ppl    39.96\n",
            "| epoch  31 |   600/ 1327 batches | lr 0.08 | ms/batch 157.08 | loss  3.75 | ppl    42.57\n",
            "| epoch  31 |   700/ 1327 batches | lr 0.08 | ms/batch 154.81 | loss  3.69 | ppl    40.15\n",
            "| epoch  31 |   800/ 1327 batches | lr 0.08 | ms/batch 153.45 | loss  3.61 | ppl    37.07\n",
            "| epoch  31 |   900/ 1327 batches | lr 0.08 | ms/batch 154.54 | loss  3.70 | ppl    40.39\n",
            "| epoch  31 |  1000/ 1327 batches | lr 0.08 | ms/batch 153.91 | loss  3.74 | ppl    42.13\n",
            "| epoch  31 |  1100/ 1327 batches | lr 0.08 | ms/batch 154.11 | loss  3.57 | ppl    35.39\n",
            "| epoch  31 |  1200/ 1327 batches | lr 0.08 | ms/batch 153.81 | loss  3.58 | ppl    35.71\n",
            "| epoch  31 |  1300/ 1327 batches | lr 0.08 | ms/batch 154.18 | loss  3.58 | ppl    35.75\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  31 | time: 215.03s | valid loss  4.49 | valid ppl    88.70\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  32 |   100/ 1327 batches | lr 0.08 | ms/batch 155.13 | loss  3.73 | ppl    41.66\n",
            "| epoch  32 |   200/ 1327 batches | lr 0.08 | ms/batch 152.27 | loss  3.80 | ppl    44.58\n",
            "| epoch  32 |   300/ 1327 batches | lr 0.08 | ms/batch 153.27 | loss  3.80 | ppl    44.70\n",
            "| epoch  32 |   400/ 1327 batches | lr 0.08 | ms/batch 153.85 | loss  3.69 | ppl    39.90\n",
            "| epoch  32 |   500/ 1327 batches | lr 0.08 | ms/batch 152.49 | loss  3.69 | ppl    39.95\n",
            "| epoch  32 |   600/ 1327 batches | lr 0.08 | ms/batch 153.84 | loss  3.75 | ppl    42.54\n",
            "| epoch  32 |   700/ 1327 batches | lr 0.08 | ms/batch 154.55 | loss  3.70 | ppl    40.33\n",
            "| epoch  32 |   800/ 1327 batches | lr 0.08 | ms/batch 154.39 | loss  3.61 | ppl    37.14\n",
            "| epoch  32 |   900/ 1327 batches | lr 0.08 | ms/batch 153.03 | loss  3.69 | ppl    40.10\n",
            "| epoch  32 |  1000/ 1327 batches | lr 0.08 | ms/batch 154.88 | loss  3.74 | ppl    41.96\n",
            "| epoch  32 |  1100/ 1327 batches | lr 0.08 | ms/batch 154.75 | loss  3.57 | ppl    35.44\n",
            "| epoch  32 |  1200/ 1327 batches | lr 0.08 | ms/batch 154.85 | loss  3.58 | ppl    35.84\n",
            "| epoch  32 |  1300/ 1327 batches | lr 0.08 | ms/batch 154.81 | loss  3.58 | ppl    35.75\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  32 | time: 213.89s | valid loss  4.49 | valid ppl    88.70\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  33 |   100/ 1327 batches | lr 0.08 | ms/batch 157.50 | loss  3.72 | ppl    41.31\n",
            "| epoch  33 |   200/ 1327 batches | lr 0.08 | ms/batch 153.47 | loss  3.80 | ppl    44.72\n",
            "| epoch  33 |   300/ 1327 batches | lr 0.08 | ms/batch 153.84 | loss  3.79 | ppl    44.38\n",
            "| epoch  33 |   400/ 1327 batches | lr 0.08 | ms/batch 154.57 | loss  3.69 | ppl    40.03\n",
            "| epoch  33 |   500/ 1327 batches | lr 0.08 | ms/batch 155.55 | loss  3.69 | ppl    39.88\n",
            "| epoch  33 |   600/ 1327 batches | lr 0.08 | ms/batch 155.50 | loss  3.74 | ppl    42.15\n",
            "| epoch  33 |   700/ 1327 batches | lr 0.08 | ms/batch 154.93 | loss  3.70 | ppl    40.42\n",
            "| epoch  33 |   800/ 1327 batches | lr 0.08 | ms/batch 154.59 | loss  3.61 | ppl    36.88\n",
            "| epoch  33 |   900/ 1327 batches | lr 0.08 | ms/batch 153.49 | loss  3.70 | ppl    40.31\n",
            "| epoch  33 |  1000/ 1327 batches | lr 0.08 | ms/batch 154.22 | loss  3.74 | ppl    42.00\n",
            "| epoch  33 |  1100/ 1327 batches | lr 0.08 | ms/batch 154.46 | loss  3.57 | ppl    35.58\n",
            "| epoch  33 |  1200/ 1327 batches | lr 0.08 | ms/batch 153.50 | loss  3.58 | ppl    35.84\n",
            "| epoch  33 |  1300/ 1327 batches | lr 0.08 | ms/batch 155.63 | loss  3.57 | ppl    35.67\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  33 | time: 214.96s | valid loss  4.49 | valid ppl    88.68\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  34 |   100/ 1327 batches | lr 0.08 | ms/batch 158.78 | loss  3.72 | ppl    41.24\n",
            "| epoch  34 |   200/ 1327 batches | lr 0.08 | ms/batch 153.52 | loss  3.79 | ppl    44.37\n",
            "| epoch  34 |   300/ 1327 batches | lr 0.08 | ms/batch 154.78 | loss  3.79 | ppl    44.25\n",
            "| epoch  34 |   400/ 1327 batches | lr 0.08 | ms/batch 155.22 | loss  3.68 | ppl    39.72\n",
            "| epoch  34 |   500/ 1327 batches | lr 0.08 | ms/batch 154.61 | loss  3.68 | ppl    39.62\n",
            "| epoch  34 |   600/ 1327 batches | lr 0.08 | ms/batch 153.31 | loss  3.74 | ppl    42.06\n",
            "| epoch  34 |   700/ 1327 batches | lr 0.08 | ms/batch 155.43 | loss  3.69 | ppl    39.91\n",
            "| epoch  34 |   800/ 1327 batches | lr 0.08 | ms/batch 155.78 | loss  3.61 | ppl    36.89\n",
            "| epoch  34 |   900/ 1327 batches | lr 0.08 | ms/batch 155.42 | loss  3.69 | ppl    40.13\n",
            "| epoch  34 |  1000/ 1327 batches | lr 0.08 | ms/batch 155.31 | loss  3.74 | ppl    41.98\n",
            "| epoch  34 |  1100/ 1327 batches | lr 0.08 | ms/batch 155.56 | loss  3.57 | ppl    35.40\n",
            "| epoch  34 |  1200/ 1327 batches | lr 0.08 | ms/batch 154.98 | loss  3.58 | ppl    35.80\n",
            "| epoch  34 |  1300/ 1327 batches | lr 0.08 | ms/batch 153.74 | loss  3.58 | ppl    35.91\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  34 | time: 215.37s | valid loss  4.49 | valid ppl    88.70\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  35 |   100/ 1327 batches | lr 0.02 | ms/batch 155.68 | loss  3.72 | ppl    41.23\n",
            "| epoch  35 |   200/ 1327 batches | lr 0.02 | ms/batch 152.36 | loss  3.80 | ppl    44.61\n",
            "| epoch  35 |   300/ 1327 batches | lr 0.02 | ms/batch 154.60 | loss  3.80 | ppl    44.57\n",
            "| epoch  35 |   400/ 1327 batches | lr 0.02 | ms/batch 152.28 | loss  3.69 | ppl    39.96\n",
            "| epoch  35 |   500/ 1327 batches | lr 0.02 | ms/batch 155.71 | loss  3.68 | ppl    39.82\n",
            "| epoch  35 |   600/ 1327 batches | lr 0.02 | ms/batch 154.44 | loss  3.75 | ppl    42.43\n",
            "| epoch  35 |   700/ 1327 batches | lr 0.02 | ms/batch 154.48 | loss  3.69 | ppl    40.22\n",
            "| epoch  35 |   800/ 1327 batches | lr 0.02 | ms/batch 154.73 | loss  3.60 | ppl    36.63\n",
            "| epoch  35 |   900/ 1327 batches | lr 0.02 | ms/batch 154.97 | loss  3.69 | ppl    40.13\n",
            "| epoch  35 |  1000/ 1327 batches | lr 0.02 | ms/batch 153.51 | loss  3.73 | ppl    41.77\n",
            "| epoch  35 |  1100/ 1327 batches | lr 0.02 | ms/batch 154.16 | loss  3.57 | ppl    35.47\n",
            "| epoch  35 |  1200/ 1327 batches | lr 0.02 | ms/batch 153.70 | loss  3.58 | ppl    35.76\n",
            "| epoch  35 |  1300/ 1327 batches | lr 0.02 | ms/batch 155.84 | loss  3.57 | ppl    35.66\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  35 | time: 214.50s | valid loss  4.49 | valid ppl    88.68\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  36 |   100/ 1327 batches | lr 0.02 | ms/batch 159.03 | loss  3.72 | ppl    41.24\n",
            "| epoch  36 |   200/ 1327 batches | lr 0.02 | ms/batch 155.45 | loss  3.79 | ppl    44.38\n",
            "| epoch  36 |   300/ 1327 batches | lr 0.02 | ms/batch 155.27 | loss  3.80 | ppl    44.58\n",
            "| epoch  36 |   400/ 1327 batches | lr 0.02 | ms/batch 154.87 | loss  3.68 | ppl    39.46\n",
            "| epoch  36 |   500/ 1327 batches | lr 0.02 | ms/batch 154.74 | loss  3.69 | ppl    40.08\n",
            "| epoch  36 |   600/ 1327 batches | lr 0.02 | ms/batch 154.72 | loss  3.74 | ppl    42.12\n",
            "| epoch  36 |   700/ 1327 batches | lr 0.02 | ms/batch 154.53 | loss  3.69 | ppl    39.99\n",
            "| epoch  36 |   800/ 1327 batches | lr 0.02 | ms/batch 153.80 | loss  3.60 | ppl    36.77\n",
            "| epoch  36 |   900/ 1327 batches | lr 0.02 | ms/batch 154.97 | loss  3.69 | ppl    39.98\n",
            "| epoch  36 |  1000/ 1327 batches | lr 0.02 | ms/batch 152.45 | loss  3.73 | ppl    41.82\n",
            "| epoch  36 |  1100/ 1327 batches | lr 0.02 | ms/batch 154.10 | loss  3.57 | ppl    35.38\n",
            "| epoch  36 |  1200/ 1327 batches | lr 0.02 | ms/batch 152.66 | loss  3.58 | ppl    35.77\n",
            "| epoch  36 |  1300/ 1327 batches | lr 0.02 | ms/batch 154.24 | loss  3.57 | ppl    35.66\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  36 | time: 214.89s | valid loss  4.48 | valid ppl    88.67\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  37 |   100/ 1327 batches | lr 0.02 | ms/batch 158.67 | loss  3.72 | ppl    41.39\n",
            "| epoch  37 |   200/ 1327 batches | lr 0.02 | ms/batch 154.91 | loss  3.80 | ppl    44.73\n",
            "| epoch  37 |   300/ 1327 batches | lr 0.02 | ms/batch 153.48 | loss  3.80 | ppl    44.53\n",
            "| epoch  37 |   400/ 1327 batches | lr 0.02 | ms/batch 154.47 | loss  3.68 | ppl    39.64\n",
            "| epoch  37 |   500/ 1327 batches | lr 0.02 | ms/batch 157.32 | loss  3.69 | ppl    39.91\n",
            "| epoch  37 |   600/ 1327 batches | lr 0.02 | ms/batch 153.98 | loss  3.75 | ppl    42.64\n",
            "| epoch  37 |   700/ 1327 batches | lr 0.02 | ms/batch 154.05 | loss  3.69 | ppl    40.10\n",
            "| epoch  37 |   800/ 1327 batches | lr 0.02 | ms/batch 153.65 | loss  3.60 | ppl    36.78\n",
            "| epoch  37 |   900/ 1327 batches | lr 0.02 | ms/batch 154.58 | loss  3.69 | ppl    40.24\n",
            "| epoch  37 |  1000/ 1327 batches | lr 0.02 | ms/batch 154.80 | loss  3.74 | ppl    41.91\n",
            "| epoch  37 |  1100/ 1327 batches | lr 0.02 | ms/batch 153.48 | loss  3.57 | ppl    35.36\n",
            "| epoch  37 |  1200/ 1327 batches | lr 0.02 | ms/batch 154.94 | loss  3.58 | ppl    35.87\n",
            "| epoch  37 |  1300/ 1327 batches | lr 0.02 | ms/batch 154.99 | loss  3.58 | ppl    35.74\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  37 | time: 215.07s | valid loss  4.48 | valid ppl    88.67\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  38 |   100/ 1327 batches | lr 0.00 | ms/batch 156.86 | loss  3.72 | ppl    41.18\n",
            "| epoch  38 |   200/ 1327 batches | lr 0.00 | ms/batch 153.01 | loss  3.79 | ppl    44.43\n",
            "| epoch  38 |   300/ 1327 batches | lr 0.00 | ms/batch 153.86 | loss  3.79 | ppl    44.47\n",
            "| epoch  38 |   400/ 1327 batches | lr 0.00 | ms/batch 154.27 | loss  3.68 | ppl    39.75\n",
            "| epoch  38 |   500/ 1327 batches | lr 0.00 | ms/batch 153.93 | loss  3.68 | ppl    39.80\n",
            "| epoch  38 |   600/ 1327 batches | lr 0.00 | ms/batch 154.08 | loss  3.75 | ppl    42.37\n",
            "| epoch  38 |   700/ 1327 batches | lr 0.00 | ms/batch 154.77 | loss  3.70 | ppl    40.25\n",
            "| epoch  38 |   800/ 1327 batches | lr 0.00 | ms/batch 154.90 | loss  3.60 | ppl    36.76\n",
            "| epoch  38 |   900/ 1327 batches | lr 0.00 | ms/batch 152.37 | loss  3.70 | ppl    40.26\n",
            "| epoch  38 |  1000/ 1327 batches | lr 0.00 | ms/batch 154.96 | loss  3.74 | ppl    41.94\n",
            "| epoch  38 |  1100/ 1327 batches | lr 0.00 | ms/batch 155.05 | loss  3.55 | ppl    34.98\n",
            "| epoch  38 |  1200/ 1327 batches | lr 0.00 | ms/batch 155.41 | loss  3.57 | ppl    35.57\n",
            "| epoch  38 |  1300/ 1327 batches | lr 0.00 | ms/batch 155.43 | loss  3.57 | ppl    35.69\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  38 | time: 214.72s | valid loss  4.48 | valid ppl    88.67\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  39 |   100/ 1327 batches | lr 0.00 | ms/batch 157.03 | loss  3.72 | ppl    41.39\n",
            "| epoch  39 |   200/ 1327 batches | lr 0.00 | ms/batch 154.97 | loss  3.79 | ppl    44.46\n",
            "| epoch  39 |   300/ 1327 batches | lr 0.00 | ms/batch 155.79 | loss  3.79 | ppl    44.37\n",
            "| epoch  39 |   400/ 1327 batches | lr 0.00 | ms/batch 155.15 | loss  3.69 | ppl    39.91\n",
            "| epoch  39 |   500/ 1327 batches | lr 0.00 | ms/batch 154.63 | loss  3.69 | ppl    39.97\n",
            "| epoch  39 |   600/ 1327 batches | lr 0.00 | ms/batch 154.71 | loss  3.75 | ppl    42.53\n",
            "| epoch  39 |   700/ 1327 batches | lr 0.00 | ms/batch 152.92 | loss  3.69 | ppl    40.21\n",
            "| epoch  39 |   800/ 1327 batches | lr 0.00 | ms/batch 152.97 | loss  3.60 | ppl    36.66\n",
            "| epoch  39 |   900/ 1327 batches | lr 0.00 | ms/batch 154.25 | loss  3.69 | ppl    40.13\n",
            "| epoch  39 |  1000/ 1327 batches | lr 0.00 | ms/batch 154.70 | loss  3.74 | ppl    41.92\n",
            "| epoch  39 |  1100/ 1327 batches | lr 0.00 | ms/batch 194.55 | loss  3.57 | ppl    35.35\n",
            "| epoch  39 |  1200/ 1327 batches | lr 0.00 | ms/batch 241.33 | loss  3.57 | ppl    35.62\n",
            "| epoch  39 |  1300/ 1327 batches | lr 0.00 | ms/batch 283.04 | loss  3.57 | ppl    35.47\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  39 | time: 247.63s | valid loss  4.48 | valid ppl    88.67\n",
            "-----------------------------------------------------------------------------------------\n",
            "| epoch  40 |   100/ 1327 batches | lr 0.00 | ms/batch 289.10 | loss  3.72 | ppl    41.37\n",
            "| epoch  40 |   200/ 1327 batches | lr 0.00 | ms/batch 282.58 | loss  3.79 | ppl    44.44\n",
            "| epoch  40 |   300/ 1327 batches | lr 0.00 | ms/batch 216.95 | loss  3.80 | ppl    44.68\n",
            "| epoch  40 |   400/ 1327 batches | lr 0.00 | ms/batch 167.11 | loss  3.68 | ppl    39.76\n",
            "| epoch  40 |   500/ 1327 batches | lr 0.00 | ms/batch 190.04 | loss  3.69 | ppl    39.98\n",
            "| epoch  40 |   600/ 1327 batches | lr 0.00 | ms/batch 253.34 | loss  3.75 | ppl    42.49\n",
            "| epoch  40 |   700/ 1327 batches | lr 0.00 | ms/batch 282.94 | loss  3.69 | ppl    40.09\n",
            "| epoch  40 |   800/ 1327 batches | lr 0.00 | ms/batch 282.98 | loss  3.61 | ppl    36.85\n",
            "| epoch  40 |   900/ 1327 batches | lr 0.00 | ms/batch 287.14 | loss  3.69 | ppl    39.93\n",
            "| epoch  40 |  1000/ 1327 batches | lr 0.00 | ms/batch 284.79 | loss  3.74 | ppl    41.93\n",
            "| epoch  40 |  1100/ 1327 batches | lr 0.00 | ms/batch 285.97 | loss  3.57 | ppl    35.36\n",
            "| epoch  40 |  1200/ 1327 batches | lr 0.00 | ms/batch 283.66 | loss  3.57 | ppl    35.58\n",
            "| epoch  40 |  1300/ 1327 batches | lr 0.00 | ms/batch 285.29 | loss  3.57 | ppl    35.67\n",
            "-----------------------------------------------------------------------------------------\n",
            "| end of epoch  40 | time: 362.68s | valid loss  4.48 | valid ppl    88.67\n",
            "-----------------------------------------------------------------------------------------\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "FqHtDk-4kUor",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Finally,  open the best saved model and run it on the test data"
      ]
    },
    {
      "metadata": {
        "id": "3rP_dOtpkCTX",
        "colab_type": "code",
        "outputId": "a9ee9947-e8da-4ee2-cffd-5fb66aa3ab41",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "cell_type": "code",
      "source": [
        "# Load the best saved model.\n",
        "with open(args_save, 'rb') as f:\n",
        "    model = torch.load(f)\n",
        "\n",
        "# Run on test data.\n",
        "test_loss = evaluate(test_data)\n",
        "print('=' * 89)\n",
        "print('| End of training | test loss {:5.2f} | test ppl {:8.2f}'.format(\n",
        "    test_loss, math.exp(test_loss)))\n",
        "print('=' * 89)\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "=========================================================================================\n",
            "| End of training | test loss  4.44 | test ppl    85.00\n",
            "=========================================================================================\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "i3X3GRsmJYya",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Word generator\n",
        "\n",
        "First define the arguments and load the corpus"
      ]
    },
    {
      "metadata": {
        "id": "DSz0R2g97VAk",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Load the best saved model.\n",
        "\n",
        "with open(args_save, 'rb') as f:\n",
        "    model = torch.load(f)\n",
        "\n",
        "model.eval()\n",
        "\n",
        "corpus = Corpus(args_data)\n",
        "ntokens = len(corpus.dictionary)\n",
        "hidden = model.init_hidden(1)\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "5sr47CxlJeMb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Then generate some data"
      ]
    },
    {
      "metadata": {
        "id": "pK2WqvLjJcYh",
        "colab_type": "code",
        "outputId": "9b66e0d3-8843-4464-8700-207ad3395d36",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 272
        }
      },
      "cell_type": "code",
      "source": [
        "args_words = 300\n",
        "args_temperature = 0.9\n",
        "input = torch.randint(ntokens, (1, 1), dtype=torch.long).to(device)\n",
        "\n",
        "words = []\n",
        "probs = []\n",
        "\n",
        "with torch.no_grad():  # no tracking history\n",
        "    for i in range(args_words):\n",
        "        output, hidden = model(input, hidden)\n",
        "        \n",
        "        word_weights = output.squeeze().div(args_temperature).exp().cpu()\n",
        "        word_idx = torch.multinomial(word_weights, 1)[0]\n",
        "        input.fill_(word_idx)\n",
        "        word = corpus.dictionary.idx2word[word_idx]\n",
        "        \n",
        "        # We replace <unk> and <eos> with * to get a cleaner look, but thats just\n",
        "        # personal preference\n",
        "        if(word == \"<unk>\" or word == \"<eos>\"):\n",
        "          word = \"*\"\n",
        "\n",
        "        print(word + ('\\n' if i % 20 == 19 else ' '),end='')\n",
        "        \n",
        "        # We also create arrays with the generated words and their probability \n",
        "        # to be used for visualizing them in a tool that we created for this\n",
        "        # purpose\n",
        "        words.append(word)\n",
        "        probs.append(output.squeeze()[word_idx].data.tolist())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "by the big board * the stock market basket gained N to N * the stock market reacted to the\n",
            "lows of market sentiment about N N monday according to the european community commission * the insurers requested that many\n",
            "* companies posted signs of earnings surge in the u.s. * the u.s. fed 's ratio was a more *\n",
            "than $ N million in the first nine months of N said a * spokesman * for the first nine\n",
            "months of this year the company added N N to N N in the third quarter compared with N million\n",
            "the second quarter in N * but while imports rose N N in september * the company said the *\n",
            "and * for the entire year fell about N N from july 's $ N billion * mr. * said\n",
            "the levels on the thrift industry has been * by florida and the national aeronautics and space administration * his\n",
            "order to cut off more reserves in private lending is particularly substantial and produce less than N N of the\n",
            "wage increases a year ago * some * to the move which are the most * way along with its\n",
            "farm inventories of seasonal reviews the indication of a slowdown is provided by the federal reserve * the u.s. package\n",
            "'s third-quarter figures gold executive said it is unclear what blacks are at home * the government did n't have\n",
            "to favor a securities and exchange commission * * we 're proving that there 's an appropriate promotion said mark\n",
            "* director of research at new york university of america * in the recent report called for a * *\n",
            "for the N months ended july N N N of its group which became the first year following the central\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "a5grU9olECdl",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Print the words and probabilities for use in a [vizualization tool](https://github.com/mikkelbrusen/text-weight-visualizer) we created "
      ]
    },
    {
      "metadata": {
        "id": "kRRaUq9ZJsNT",
        "colab_type": "code",
        "outputId": "dab3202e-7a3b-4275-cc07-26016a19f2f9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        }
      },
      "cell_type": "code",
      "source": [
        "print(words)\n",
        "print(probs)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['by', 'the', 'big', 'board', '*', 'the', 'stock', 'market', 'basket', 'gained', 'N', 'to', 'N', '*', 'the', 'stock', 'market', 'reacted', 'to', 'the', 'lows', 'of', 'market', 'sentiment', 'about', 'N', 'N', 'monday', 'according', 'to', 'the', 'european', 'community', 'commission', '*', 'the', 'insurers', 'requested', 'that', 'many', '*', 'companies', 'posted', 'signs', 'of', 'earnings', 'surge', 'in', 'the', 'u.s.', '*', 'the', 'u.s.', 'fed', \"'s\", 'ratio', 'was', 'a', 'more', '*', 'than', '$', 'N', 'million', 'in', 'the', 'first', 'nine', 'months', 'of', 'N', 'said', 'a', '*', 'spokesman', '*', 'for', 'the', 'first', 'nine', 'months', 'of', 'this', 'year', 'the', 'company', 'added', 'N', 'N', 'to', 'N', 'N', 'in', 'the', 'third', 'quarter', 'compared', 'with', 'N', 'million', 'the', 'second', 'quarter', 'in', 'N', '*', 'but', 'while', 'imports', 'rose', 'N', 'N', 'in', 'september', '*', 'the', 'company', 'said', 'the', '*', 'and', '*', 'for', 'the', 'entire', 'year', 'fell', 'about', 'N', 'N', 'from', 'july', \"'s\", '$', 'N', 'billion', '*', 'mr.', '*', 'said', 'the', 'levels', 'on', 'the', 'thrift', 'industry', 'has', 'been', '*', 'by', 'florida', 'and', 'the', 'national', 'aeronautics', 'and', 'space', 'administration', '*', 'his', 'order', 'to', 'cut', 'off', 'more', 'reserves', 'in', 'private', 'lending', 'is', 'particularly', 'substantial', 'and', 'produce', 'less', 'than', 'N', 'N', 'of', 'the', 'wage', 'increases', 'a', 'year', 'ago', '*', 'some', '*', 'to', 'the', 'move', 'which', 'are', 'the', 'most', '*', 'way', 'along', 'with', 'its', 'farm', 'inventories', 'of', 'seasonal', 'reviews', 'the', 'indication', 'of', 'a', 'slowdown', 'is', 'provided', 'by', 'the', 'federal', 'reserve', '*', 'the', 'u.s.', 'package', \"'s\", 'third-quarter', 'figures', 'gold', 'executive', 'said', 'it', 'is', 'unclear', 'what', 'blacks', 'are', 'at', 'home', '*', 'the', 'government', 'did', \"n't\", 'have', 'to', 'favor', 'a', 'securities', 'and', 'exchange', 'commission', '*', '*', 'we', \"'re\", 'proving', 'that', 'there', \"'s\", 'an', 'appropriate', 'promotion', 'said', 'mark', '*', 'director', 'of', 'research', 'at', 'new', 'york', 'university', 'of', 'america', '*', 'in', 'the', 'recent', 'report', 'called', 'for', 'a', '*', '*', 'for', 'the', 'N', 'months', 'ended', 'july', 'N', 'N', 'N', 'of', 'its', 'group', 'which', 'became', 'the', 'first', 'year', 'following', 'the', 'central']\n",
            "[19.468490600585938, 15.512442588806152, 10.264516830444336, 16.998098373413086, 13.248040199279785, 10.854406356811523, 8.3110933303833, 13.653432846069336, 4.570566654205322, 9.338033676147461, 15.285013198852539, 15.024828910827637, 17.174442291259766, 15.285017967224121, 11.126354217529297, 9.989222526550293, 14.213804244995117, 11.457084655761719, 12.102781295776367, 12.56570053100586, 7.034346580505371, 12.91792106628418, 8.488868713378906, 11.170243263244629, 9.814760208129883, 8.910747528076172, 11.098076820373535, 7.335899353027344, 8.571256637573242, 23.01336669921875, 11.896454811096191, 7.709658622741699, 16.460535049438477, 13.053776741027832, 13.509512901306152, 10.889507293701172, 3.0979647636413574, 3.8167989253997803, 11.142724990844727, 5.9621076583862305, 9.513572692871094, 9.931708335876465, 8.42343521118164, 7.771419048309326, 19.437650680541992, 9.995922088623047, 6.333629608154297, 16.9610595703125, 12.185087203979492, 9.95467758178711, 9.907302856445312, 11.376920700073242, 8.8240966796875, 6.314297676086426, 11.648407936096191, 4.192080497741699, 14.460323333740234, 9.973779678344727, 9.385821342468262, 11.257938385009766, 11.71769905090332, 9.437514305114746, 22.664024353027344, 19.105941772460938, 12.368370056152344, 12.1018648147583, 12.488389015197754, 14.185785293579102, 21.54964256286621, 16.417131423950195, 13.88711929321289, 9.74894905090332, 9.710759162902832, 10.570147514343262, 9.824972152709961, 15.223954200744629, 9.199308395385742, 13.492180824279785, 12.450115203857422, 16.18143081665039, 24.51038932800293, 13.365408897399902, 13.78602409362793, 18.165939331054688, 12.081161499023438, 11.364839553833008, 9.5313081741333, 14.422266960144043, 15.016749382019043, 16.09362030029297, 15.707298278808594, 16.302814483642578, 11.813179016113281, 13.729072570800781, 15.162368774414062, 22.15398597717285, 12.013687133789062, 24.069990158081055, 15.699007987976074, 14.686776161193848, 11.542156219482422, 12.740800857543945, 17.832639694213867, 12.030284881591797, 14.735962867736816, 14.987885475158691, 8.867457389831543, 8.458134651184082, 8.820618629455566, 13.84854793548584, 14.242626190185547, 18.464719772338867, 13.79511833190918, 15.838841438293457, 11.383111000061035, 11.255041122436523, 8.867865562438965, 13.813971519470215, 12.513994216918945, 9.355440139770508, 7.87278413772583, 9.918052673339844, 6.8549628257751465, 12.300158500671387, 6.419953346252441, 14.649409294128418, 12.296503067016602, 10.885245323181152, 17.77731704711914, 17.126850128173828, 14.594959259033203, 11.940951347351074, 14.98452377319336, 10.617074966430664, 22.200428009033203, 16.89361000061035, 13.252910614013672, 7.414530277252197, 13.627073287963867, 15.786385536193848, 12.981382369995117, 3.4630126953125, 8.579834938049316, 11.577688217163086, 6.543154716491699, 12.794954299926758, 11.291311264038086, 12.383915901184082, 10.392843246459961, 12.561267852783203, 0.7329471707344055, 13.094207763671875, 9.365943908691406, 9.38104248046875, 11.868008613586426, 17.71249008178711, 17.442358016967773, 22.813861846923828, 11.562960624694824, 4.858828544616699, 4.110988616943359, 11.345010757446289, 8.888728141784668, 10.75798225402832, 6.981165885925293, 6.1900763511657715, 10.937919616699219, 7.019137382507324, 9.974081993103027, 11.441675186157227, 5.365374565124512, 6.953813552856445, 12.088777542114258, 2.320260524749756, 8.08378791809082, 12.804835319519043, 12.552242279052734, 14.259160041809082, 13.43431282043457, 11.942049026489258, 6.738049507141113, 11.996782302856445, 8.149979591369629, 13.460495948791504, 14.507943153381348, 13.58329963684082, 7.508608818054199, 9.158102035522461, 6.62822151184082, 10.982192993164062, 4.258798599243164, 9.637628555297852, 9.071802139282227, 8.684576034545898, 9.464629173278809, 10.440003395080566, 6.32139778137207, 6.838535785675049, 12.205740928649902, 8.093746185302734, 2.3596720695495605, 6.437313556671143, 10.244895935058594, 3.4204726219177246, 5.771543502807617, 10.780780792236328, 5.405652046203613, 16.568262100219727, 11.246721267700195, 10.285993576049805, 11.291297912597656, 4.58405876159668, 13.21263313293457, 10.744111061096191, 9.334737777709961, 16.995241165161133, 11.865965843200684, 10.760900497436523, 8.21204948425293, 5.335371494293213, 10.516145706176758, 6.278915882110596, 10.145511627197266, 4.378742218017578, 2.7312300205230713, 9.0521240234375, 10.664064407348633, 11.648311614990234, 7.034951686859131, 14.663178443908691, 5.324264049530029, 11.948251724243164, 6.324735164642334, 10.596062660217285, 12.078612327575684, 11.122140884399414, 8.093066215515137, 8.866799354553223, 14.1712646484375, 11.900679588317871, 11.40155029296875, 6.9300618171691895, 11.302671432495117, 3.782864570617676, 11.445245742797852, 13.578089714050293, 17.524641036987305, 8.734015464782715, 9.70545482635498, 8.197773933410645, 13.693049430847168, 4.138849258422852, 11.485647201538086, 11.084842681884766, 14.440105438232422, 11.450922966003418, 7.019855976104736, 2.570193290710449, 10.58809757232666, 7.959085464477539, 13.874676704406738, 12.774578094482422, 19.061622619628906, 12.857872009277344, 16.162878036499023, 8.185327529907227, 16.38362693786621, 11.46242904663086, 13.790449142456055, 10.725354194641113, 15.510414123535156, 9.627549171447754, 12.519734382629395, 8.021052360534668, 10.407252311706543, 4.23348331451416, 10.8412446975708, 10.273560523986816, 10.349170684814453, 9.164721488952637, 7.533135414123535, 11.216170310974121, 8.637871742248535, 11.4927396774292, 11.940954208374023, 13.60056209564209, 16.557239532470703, 8.908282279968262, 10.108243942260742, 11.303190231323242, 10.547077178955078, 6.1807451248168945, 7.515616416931152, 6.935422420501709, 11.453520774841309, 13.033289909362793, 11.669130325317383, 8.303668975830078, 12.777421951293945, 6.368068218231201]\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}