chapters/chapter07_intermediate_sequence_modeling_for_nlp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8a3b5a84",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Intermediate Sequence Modeling for Natural Language Processing<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Introduction\" data-toc-modified-id=\"Introduction-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Introduction</a></span></li><li><span><a href=\"#The-Problem-with-Vanilla/Elman-RNNs\" data-toc-modified-id=\"The-Problem-with-Vanilla/Elman-RNNs-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>The Problem with Vanilla/Elman RNNs</a></span></li><li><span><a href=\"#Gating-as-a-Solution-to-a-Vanilla-RNNs-Problems\" data-toc-modified-id=\"Gating-as-a-Solution-to-a-Vanilla-RNNs-Problems-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Gating as a Solution to a Vanilla RNNs Problems</a></span></li><li><span><a href=\"#Tips-and-Tricks-for-training-sequence-models\" data-toc-modified-id=\"Tips-and-Tricks-for-training-sequence-models-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Tips and Tricks for training sequence models</a></span></li><li><span><a href=\"#Example:-A-Character-RNN-for-Generating-Surnames\" data-toc-modified-id=\"Example:-A-Character-RNN-for-Generating-Surnames-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Example: A Character RNN for Generating Surnames</a></span><ul class=\"toc-item\"><li><span><a href=\"#Vocabulary,-Vectorizer-and-Dataset\" data-toc-modified-id=\"Vocabulary,-Vectorizer-and-Dataset-5.1\"><span class=\"toc-item-num\">5.1&nbsp;&nbsp;</span>Vocabulary, Vectorizer and Dataset</a></span></li><li><span><a href=\"#Unconditioned-Surname-Generation-Model\" data-toc-modified-id=\"Unconditioned-Surname-Generation-Model-5.2\"><span class=\"toc-item-num\">5.2&nbsp;&nbsp;</span>Unconditioned Surname Generation Model</a></span></li><li><span><a href=\"#Conditioned-Surname-Generation-Model\" data-toc-modified-id=\"Conditioned-Surname-Generation-Model-5.3\"><span class=\"toc-item-num\">5.3&nbsp;&nbsp;</span>Conditioned Surname Generation Model</a></span></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a5d006e",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "412125e6",
   "metadata": {
    "hidden": true
   },
   "source": [
    "- _Sequence Prediction_ task requires to label each item of a sequence. Examples include language modeling, part of speech tagging, name entity recognition.\n",
    "- Sequence prediction is also referred as Sequence Labeling.\n",
    "\n",
    "![Figure 7.1](../images/figure_7_1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "125ce56f",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## The Problem with Vanilla/Elman RNNs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb225a2b",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Elman RNNs suffers from two problems:\n",
    "\n",
    "- Inability to retain information for long range predictions\n",
    "    - At each time step we simply update the hidden state vector regardless of whether it made sense. Due to this, RNN has no control over which values are retained and which are discarded in the hidden state. However what is desired is some way for the RNN to decide of the update is optional or if the update happens by how much and what parts of the state vector and so on.\n",
    "- Gradient Stability\n",
    "    - Vanilla RNNs also suffers from vanishing gradients or exploding gradients.\n",
    "\n",
    "Some solutions that can be address these problems are:\n",
    "- ReLUs\n",
    "- Gradient Clipping\n",
    "- Careful Initialization\n",
    "- Gating(Most reliable)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0d16bb9",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Gating as a Solution to a Vanilla RNNs Problems"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8eb50711",
   "metadata": {
    "hidden": true
   },
   "source": [
    "To understand gating solution, lets suppose that we are adding two numberss, $a$ and $b$ and we want to control how much of $b$ gets into the sum. So we can write this as:\n",
    "\n",
    "$$ a + \\lambda b $$\n",
    "    \n",
    "where $\\lambda$ is a value between 0 and 1. So if $\\lambda = 0$, these is no contribution from b and if $\\lambda = 1$ b contributes fully.\n",
    "\n",
    "In above example, we can interpret $\\lambda$ as a _switch_ or a _gate_ in controlling the amount of $b$ that gets into the sum. This is the intuition behind the gating mechanism."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce80466c",
   "metadata": {
    "hidden": true
   },
   "source": [
    "In case of Elman RNN, the previous hidden state was $h_{t-1}$ and the current input is $x_t$, the recurrent update in Elman RNN would look something like:\n",
    "\n",
    "$$ h_t = h_{t-1} + F(h_{t-1}, x_t) $$\n",
    "\n",
    "where $F$ is the recurrent computation of the RNN. This is unconditioned sum and has the vanilla RNN problems mentioned above.\n",
    "\n",
    "This can be updated with gating function by making $\\lambda$ a function of previous hidden state vector $h_{t-1}$ then the RNN update equation would look like:\n",
    "\n",
    "$$ h_t = h_{t-1} + \\lambda(h_{t-1}, x_t) F(h_{t-1}, x_t) $$\n",
    "\n",
    "Now $\\lambda$ function controls how much of the current input gets to update the state $h_{t-1}$ and now function $\\lambda$ is context dependent. The function $\\lambda$ is usually a sigmoid function in gated networks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fe6f7b6",
   "metadata": {
    "hidden": true
   },
   "source": [
    "In case of the _long short term memory network_(LSTM), above intuition is extended to incorporate not only conditional updated but also intentional forgetting of the values in the previous hidden state $h_{t-1}$. This forgetting happens by multiplying the previous hidden state and value $h_{t-1}$ with another function $\\mu$ that also produces values between 0 and 1.\n",
    "\n",
    "$$ h_t = \\mu(h_{t-1}, x_t)h_{t-1} + \\lambda(h_{t-1}, x_t) F(h_{t-1}, x_t) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "393f04b1",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Tips and Tricks for training sequence models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad9420c9",
   "metadata": {
    "hidden": true
   },
   "source": [
    "- When possible use the gated variants\n",
    "- When possible, prefer GRUs over LSTMs\n",
    "- Use Adam as your optimizer\n",
    "- Gradient Clipping\n",
    "- Early Stopping"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3735e4de",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Example: A Character RNN for Generating Surnames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "485e55c2",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The nb_black extension is already loaded. To reload it, use:\n",
      "  %reload_ext nb_black\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 27;\n",
       "                var nbb_unformatted_code = \"%load_ext nb_black\\n\\nimport os\\nfrom argparse import Namespace\\nfrom collections import Counter\\nimport json\\nimport re\\nimport string\\n\\nimport numpy as np\\nimport pandas as pd\\nimport torch\\nimport torch.nn as nn\\nfrom torch.nn import functional as F\\nimport torch.optim as optim\\nfrom torch.utils.data import Dataset, DataLoader\\nfrom tqdm import notebook\\n\\nimport utils\";\n",
       "                var nbb_formatted_code = \"%load_ext nb_black\\n\\nimport os\\nfrom argparse import Namespace\\nfrom collections import Counter\\nimport json\\nimport re\\nimport string\\n\\nimport numpy as np\\nimport pandas as pd\\nimport torch\\nimport torch.nn as nn\\nfrom torch.nn import functional as F\\nimport torch.optim as optim\\nfrom torch.utils.data import Dataset, DataLoader\\nfrom tqdm import notebook\\n\\nimport utils\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%load_ext nb_black\n",
    "\n",
    "import os\n",
    "from argparse import Namespace\n",
    "from collections import Counter\n",
    "import json\n",
    "import re\n",
    "import string\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.nn import functional as F\n",
    "import torch.optim as optim\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "from tqdm import notebook\n",
    "\n",
    "import utils"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f98b730a",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Vocabulary, Vectorizer and Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ee41bd4b",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 4;\n",
       "                var nbb_unformatted_code = \"class Vocabulary(object):\\n    def __init__(self, token_to_idx=None):\\n        if token_to_idx is None:\\n            token_to_idx = {}\\n        self._token_to_idx = token_to_idx\\n        self._idx_to_token = {idk: token for token, idx in self._token_to_idx.items()}\\n\\n    def to_serializable(self):\\n        return {\\\"token_to_idx\\\": self._token_to_idx}\\n\\n    @classmethod\\n    def from_serializable(cls, contents):\\n        return cls(**contents)\\n\\n    def add_token(self, token):\\n        if token in self._token_to_idx:\\n            index = self._token_to_idx[token]\\n        else:\\n            index = len(self._token_to_idx)\\n            self._token_to_idx[token] = index\\n            self._idx_to_token[index] = token\\n        return index\\n\\n    def add_many(self, tokens):\\n        return [self.add_token(token) for token in tokens]\\n\\n    def lookup_token(self, token):\\n        return self._token_to_idx[token]\\n\\n    def lookup_index(self, index):\\n        if index not in self._idx_to_token:\\n            raise KeyError(f\\\"The index {index} is not in the Vocab.\\\")\\n        return self._idx_to_token[index]\\n\\n    def __str__(self):\\n        return f\\\"<Vocabulary(size={len(self)})>\\\"\\n\\n    def __len__(self):\\n        return len(self._token_to_idx)\";\n",
       "                var nbb_formatted_code = \"class Vocabulary(object):\\n    def __init__(self, token_to_idx=None):\\n        if token_to_idx is None:\\n            token_to_idx = {}\\n        self._token_to_idx = token_to_idx\\n        self._idx_to_token = {idk: token for token, idx in self._token_to_idx.items()}\\n\\n    def to_serializable(self):\\n        return {\\\"token_to_idx\\\": self._token_to_idx}\\n\\n    @classmethod\\n    def from_serializable(cls, contents):\\n        return cls(**contents)\\n\\n    def add_token(self, token):\\n        if token in self._token_to_idx:\\n            index = self._token_to_idx[token]\\n        else:\\n            index = len(self._token_to_idx)\\n            self._token_to_idx[token] = index\\n            self._idx_to_token[index] = token\\n        return index\\n\\n    def add_many(self, tokens):\\n        return [self.add_token(token) for token in tokens]\\n\\n    def lookup_token(self, token):\\n        return self._token_to_idx[token]\\n\\n    def lookup_index(self, index):\\n        if index not in self._idx_to_token:\\n            raise KeyError(f\\\"The index {index} is not in the Vocab.\\\")\\n        return self._idx_to_token[index]\\n\\n    def __str__(self):\\n        return f\\\"<Vocabulary(size={len(self)})>\\\"\\n\\n    def __len__(self):\\n        return len(self._token_to_idx)\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "class Vocabulary(object):\n",
    "    def __init__(self, token_to_idx=None):\n",
    "        if token_to_idx is None:\n",
    "            token_to_idx = {}\n",
    "        self._token_to_idx = token_to_idx\n",
    "        self._idx_to_token = {idk: token for token, idx in self._token_to_idx.items()}\n",
    "\n",
    "    def to_serializable(self):\n",
    "        return {\"token_to_idx\": self._token_to_idx}\n",
    "\n",
    "    @classmethod\n",
    "    def from_serializable(cls, contents):\n",
    "        return cls(**contents)\n",
    "\n",
    "    def add_token(self, token):\n",
    "        if token in self._token_to_idx:\n",
    "            index = self._token_to_idx[token]\n",
    "        else:\n",
    "            index = len(self._token_to_idx)\n",
    "            self._token_to_idx[token] = index\n",
    "            self._idx_to_token[index] = token\n",
    "        return index\n",
    "\n",
    "    def add_many(self, tokens):\n",
    "        return [self.add_token(token) for token in tokens]\n",
    "\n",
    "    def lookup_token(self, token):\n",
    "        return self._token_to_idx[token]\n",
    "\n",
    "    def lookup_index(self, index):\n",
    "        if index not in self._idx_to_token:\n",
    "            raise KeyError(f\"The index {index} is not in the Vocab.\")\n",
    "        return self._idx_to_token[index]\n",
    "\n",
    "    def __str__(self):\n",
    "        return f\"<Vocabulary(size={len(self)})>\"\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self._token_to_idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "510d30bc",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 5;\n",
       "                var nbb_unformatted_code = \"class SequenceVocabulary(Vocabulary):\\n    def __init__(\\n        self,\\n        token_to_idx=None,\\n        unk_token=\\\"<UNK>\\\",\\n        mask_token=\\\"<MASK>\\\",\\n        begin_seq_token=\\\"<BEGIN>\\\",\\n        end_seq_token=\\\"<ENF>\\\",\\n    ):\\n        super(SequenceVocabulary, self).__init__(token_to_idx)\\n        self._mask_token = mask_token\\n        self._unk_token = unk_token\\n        self._begin_seq_token = begin_seq_token\\n        self._end_seq_token = end_seq_token\\n\\n        self.mask_index = self.add_token(self._mask_token)\\n        self.unk_index = self.add_token(self._unk_token)\\n        self.begin_seq_index = self.add_token(self._begin_seq_token)\\n        self.end_seq_index = self.add_token(self._end_seq_token)\\n\\n    def to_serializable(self):\\n        contents = super(SequenceVocabulary, self).to_serializable()\\n        contents.update(\\n            {\\n                \\\"unk_token\\\": self._unk_token,\\n                \\\"mask_token\\\": self._mask_token,\\n                \\\"begin_seq_token\\\": self._begin_seq_token,\\n                \\\"end_seq_token\\\": self._end_seq_token,\\n            }\\n        )\\n        return contents\\n\\n    def lookup_token(self, token):\\n        if self.unk_index >= 0:\\n            return self._token_to_idx.get(token, self.unk_index)\\n        else:\\n            return self._token_to_idx[token]\";\n",
       "                var nbb_formatted_code = \"class SequenceVocabulary(Vocabulary):\\n    def __init__(\\n        self,\\n        token_to_idx=None,\\n        unk_token=\\\"<UNK>\\\",\\n        mask_token=\\\"<MASK>\\\",\\n        begin_seq_token=\\\"<BEGIN>\\\",\\n        end_seq_token=\\\"<ENF>\\\",\\n    ):\\n        super(SequenceVocabulary, self).__init__(token_to_idx)\\n        self._mask_token = mask_token\\n        self._unk_token = unk_token\\n        self._begin_seq_token = begin_seq_token\\n        self._end_seq_token = end_seq_token\\n\\n        self.mask_index = self.add_token(self._mask_token)\\n        self.unk_index = self.add_token(self._unk_token)\\n        self.begin_seq_index = self.add_token(self._begin_seq_token)\\n        self.end_seq_index = self.add_token(self._end_seq_token)\\n\\n    def to_serializable(self):\\n        contents = super(SequenceVocabulary, self).to_serializable()\\n        contents.update(\\n            {\\n                \\\"unk_token\\\": self._unk_token,\\n                \\\"mask_token\\\": self._mask_token,\\n                \\\"begin_seq_token\\\": self._begin_seq_token,\\n                \\\"end_seq_token\\\": self._end_seq_token,\\n            }\\n        )\\n        return contents\\n\\n    def lookup_token(self, token):\\n        if self.unk_index >= 0:\\n            return self._token_to_idx.get(token, self.unk_index)\\n        else:\\n            return self._token_to_idx[token]\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "class SequenceVocabulary(Vocabulary):\n",
    "    def __init__(\n",
    "        self,\n",
    "        token_to_idx=None,\n",
    "        unk_token=\"<UNK>\",\n",
    "        mask_token=\"<MASK>\",\n",
    "        begin_seq_token=\"<BEGIN>\",\n",
    "        end_seq_token=\"<END>\",\n",
    "    ):\n",
    "        super(SequenceVocabulary, self).__init__(token_to_idx)\n",
    "        self._mask_token = mask_token\n",
    "        self._unk_token = unk_token\n",
    "        self._begin_seq_token = begin_seq_token\n",
    "        self._end_seq_token = end_seq_token\n",
    "\n",
    "        self.mask_index = self.add_token(self._mask_token)\n",
    "        self.unk_index = self.add_token(self._unk_token)\n",
    "        self.begin_seq_index = self.add_token(self._begin_seq_token)\n",
    "        self.end_seq_index = self.add_token(self._end_seq_token)\n",
    "\n",
    "    def to_serializable(self):\n",
    "        contents = super(SequenceVocabulary, self).to_serializable()\n",
    "        contents.update(\n",
    "            {\n",
    "                \"unk_token\": self._unk_token,\n",
    "                \"mask_token\": self._mask_token,\n",
    "                \"begin_seq_token\": self._begin_seq_token,\n",
    "                \"end_seq_token\": self._end_seq_token,\n",
    "            }\n",
    "        )\n",
    "        return contents\n",
    "\n",
    "    def lookup_token(self, token):\n",
    "        if self.unk_index >= 0:\n",
    "            return self._token_to_idx.get(token, self.unk_index)\n",
    "        else:\n",
    "            return self._token_to_idx[token]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a8c09930",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 9;\n",
       "                var nbb_unformatted_code = \"class SurnameVectorizer(object):\\n    def __init__(self, char_vocab, nationality_vocab):\\n        self.char_vocab = char_vocab\\n        self.nationality_vocab = nationality_vocab\\n\\n    def vectorize(self, surname, vector_length=-1):\\n        indices = [self.char_vocab.begin_seq_index]\\n        indices.extend(self.char_vocab.lookup_token(token) for token in surname)\\n        indices.append(self.char_vocab.end_seq_index)\\n        if vector_length < 0:\\n            vector_length = len(indices) - 1\\n        from_vector = np.zeros(vector_length, dtype=np.int64)\\n        from_indices = indices[:-1]\\n        from_vector[:len(from_indices)] = from_indices\\n        from_vector[len(from_indices):] = self.char_vocab.mask_index\\n\\n        to_vector = np.empty(vector_length, dtype=np.int64)\\n        to_indices = indices[1:]\\n        to_vector[: len(to_indices)] = to_indices\\n        to_vector[len(to_indices) :] = self.char_vocab.mask_index\\n\\n        return from_vector, to_vector\\n\\n    @classmethod\\n    def from_dataframe(cls, surname_df):\\n        char_vocab = SequenceVocabulary()\\n        nationality_vocab = Vocabulary()\\n        for index, row in surname_df.iterrows():\\n            for char in row.surname:\\n                char_vocab.add_token(char)\\n            nationality_vocab.add_token(row.nationality)\\n        return cls(char_vocab=char_vocab, nationality_vocab=nationality_vocab)\\n\\n    @classmethod\\n    def from_serializable(cls, contents):\\n        char_vocab = SequenceVocabulary.from_serializable(contents[\\\"char_vocab\\\"])\\n        nat_vocab = Vocabulary.from_serializable(contents[\\\"nationality_vocab\\\"])\\n        return cls(char_vocab=char_vocab, nationality_vocab=nationality_vocab)\\n\\n    def to_serializable(self):\\n        return {\\n            \\\"char_vocab\\\": self.char_vocab.to_serializable(),\\n            \\\"nationality_vocab\\\": self.nationality_vocab.to_serializable(),\\n        }\";\n",
       "                var nbb_formatted_code = \"class SurnameVectorizer(object):\\n    def __init__(self, char_vocab, nationality_vocab):\\n        self.char_vocab = char_vocab\\n        self.nationality_vocab = nationality_vocab\\n\\n    def vectorize(self, surname, vector_length=-1):\\n        indices = [self.char_vocab.begin_seq_index]\\n        indices.extend(self.char_vocab.lookup_token(token) for token in surname)\\n        indices.append(self.char_vocab.end_seq_index)\\n        if vector_length < 0:\\n            vector_length = len(indices) - 1\\n        from_vector = np.zeros(vector_length, dtype=np.int64)\\n        from_indices = indices[:-1]\\n        from_vector[: len(from_indices)] = from_indices\\n        from_vector[len(from_indices) :] = self.char_vocab.mask_index\\n\\n        to_vector = np.empty(vector_length, dtype=np.int64)\\n        to_indices = indices[1:]\\n        to_vector[: len(to_indices)] = to_indices\\n        to_vector[len(to_indices) :] = self.char_vocab.mask_index\\n\\n        return from_vector, to_vector\\n\\n    @classmethod\\n    def from_dataframe(cls, surname_df):\\n        char_vocab = SequenceVocabulary()\\n        nationality_vocab = Vocabulary()\\n        for index, row in surname_df.iterrows():\\n            for char in row.surname:\\n                char_vocab.add_token(char)\\n            nationality_vocab.add_token(row.nationality)\\n        return cls(char_vocab=char_vocab, nationality_vocab=nationality_vocab)\\n\\n    @classmethod\\n    def from_serializable(cls, contents):\\n        char_vocab = SequenceVocabulary.from_serializable(contents[\\\"char_vocab\\\"])\\n        nat_vocab = Vocabulary.from_serializable(contents[\\\"nationality_vocab\\\"])\\n        return cls(char_vocab=char_vocab, nationality_vocab=nationality_vocab)\\n\\n    def to_serializable(self):\\n        return {\\n            \\\"char_vocab\\\": self.char_vocab.to_serializable(),\\n            \\\"nationality_vocab\\\": self.nationality_vocab.to_serializable(),\\n        }\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "class SurnameVectorizer(object):\n",
    "    def __init__(self, char_vocab, nationality_vocab):\n",
    "        self.char_vocab = char_vocab\n",
    "        self.nationality_vocab = nationality_vocab\n",
    "\n",
    "    def vectorize(self, surname, vector_length=-1):\n",
    "        indices = [self.char_vocab.begin_seq_index]\n",
    "        indices.extend(self.char_vocab.lookup_token(token) for token in surname)\n",
    "        indices.append(self.char_vocab.end_seq_index)\n",
    "        if vector_length < 0:\n",
    "            vector_length = len(indices) - 1\n",
    "        from_vector = np.zeros(vector_length, dtype=np.int64)\n",
    "        from_indices = indices[:-1]\n",
    "        from_vector[: len(from_indices)] = from_indices\n",
    "        from_vector[len(from_indices) :] = self.char_vocab.mask_index\n",
    "\n",
    "        to_vector = np.empty(vector_length, dtype=np.int64)\n",
    "        to_indices = indices[1:]\n",
    "        to_vector[: len(to_indices)] = to_indices\n",
    "        to_vector[len(to_indices) :] = self.char_vocab.mask_index\n",
    "\n",
    "        return from_vector, to_vector\n",
    "\n",
    "    @classmethod\n",
    "    def from_dataframe(cls, surname_df):\n",
    "        char_vocab = SequenceVocabulary()\n",
    "        nationality_vocab = Vocabulary()\n",
    "        for index, row in surname_df.iterrows():\n",
    "            for char in row.surname:\n",
    "                char_vocab.add_token(char)\n",
    "            nationality_vocab.add_token(row.nationality)\n",
    "        return cls(char_vocab=char_vocab, nationality_vocab=nationality_vocab)\n",
    "\n",
    "    @classmethod\n",
    "    def from_serializable(cls, contents):\n",
    "        char_vocab = SequenceVocabulary.from_serializable(contents[\"char_vocab\"])\n",
    "        nat_vocab = Vocabulary.from_serializable(contents[\"nationality_vocab\"])\n",
    "        return cls(char_vocab=char_vocab, nationality_vocab=nationality_vocab)\n",
    "\n",
    "    def to_serializable(self):\n",
    "        return {\n",
    "            \"char_vocab\": self.char_vocab.to_serializable(),\n",
    "            \"nationality_vocab\": self.nationality_vocab.to_serializable(),\n",
    "        }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "36d3e1b9",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 18;\n",
       "                var nbb_unformatted_code = \"class SurnameDataset(Dataset):\\n    def __init__(self, surname_df, vectorizer):\\n        self.surname_df = surname_df\\n        self._vectorizer = vectorizer\\n        self._max_seq_length = max(map(len, self.surname_df.surname)) + 2\\n\\n        self.train_df = self.surname_df[self.surname_df.split == \\\"train\\\"]\\n        self.train_size = len(self.train_df)\\n\\n        self.val_df = self.surname_df[self.surname_df.split == \\\"val\\\"]\\n        self.val_size = len(self.val_df)\\n\\n        self.test_df = self.surname_df[self.surname_df.split == \\\"test\\\"]\\n        self.test_size = len(self.test_df)\\n\\n        self._lookup_dict = {\\n            \\\"train\\\": (self.train_df, self.train_size),\\n            \\\"val\\\": (self.val_df, self.val_size),\\n            \\\"test\\\": (self.test_df, self.test_size),\\n        }\\n        self.set_split(\\\"train\\\")\\n\\n    @classmethod\\n    def load_dataset_and_make_vectorizer(cls, surname_csv):\\n        surname_df = pd.read_csv(surname_csv)\\n        return cls(surname_df, SurnameVectorizer.from_dataframe(surname_df))\\n\\n    @classmethod\\n    def load_dataset_and_load_vectorizer(cls, surname_csv, vectorizer_filepath):\\n        surname_df = pd.read_csb(surname_csv)\\n        vectorizer = cls.load_vectorizer_only(vectorizer_filepath)\\n        return cls(surname_df, vectorizer)\\n\\n    @staticmethod\\n    def load_vectorizer_only(vectorizer_filepath):\\n        with open(vectorizer_filepath) as fp:\\n            return SurnameVectorizer.from_serializable(json.load(fp))\\n\\n    def save_vectorizer(self, vectorizer_filepath):\\n        with open(vectorizer_filepath, \\\"w\\\") as fp:\\n            json.dump(self._vectorizer.to_serializable(), fp)\\n\\n    def get_vectorizer(self):\\n        return self._vectorizer\\n\\n    def set_split(self, split=\\\"train\\\"):\\n        self._train_split = split\\n        self._target_df, self._target_size = self._lookup_dict[split]\\n\\n    def __len__(self):\\n        return self._target_size\\n\\n    def __getitem__(self, index):\\n        row = self._target_df.iloc[index]\\n        from_vector, to_vector = self._vectorizer.vectorize(\\n            row.surname, self._max_seq_length\\n        )\\n        nationality_index = self._vectorizer.nationality_vocab.lookup_token(\\n            row.nationality\\n        )\\n        return {\\n            \\\"x_data\\\": from_vector,\\n            \\\"y_target\\\": to_vector,\\n            \\\"class_index\\\": nationality_index,\\n        }\\n\\n    def get_num_batches(self, batch_size):\\n        return len(self) // batch_size\";\n",
       "                var nbb_formatted_code = \"class SurnameDataset(Dataset):\\n    def __init__(self, surname_df, vectorizer):\\n        self.surname_df = surname_df\\n        self._vectorizer = vectorizer\\n        self._max_seq_length = max(map(len, self.surname_df.surname)) + 2\\n\\n        self.train_df = self.surname_df[self.surname_df.split == \\\"train\\\"]\\n        self.train_size = len(self.train_df)\\n\\n        self.val_df = self.surname_df[self.surname_df.split == \\\"val\\\"]\\n        self.val_size = len(self.val_df)\\n\\n        self.test_df = self.surname_df[self.surname_df.split == \\\"test\\\"]\\n        self.test_size = len(self.test_df)\\n\\n        self._lookup_dict = {\\n            \\\"train\\\": (self.train_df, self.train_size),\\n            \\\"val\\\": (self.val_df, self.val_size),\\n            \\\"test\\\": (self.test_df, self.test_size),\\n        }\\n        self.set_split(\\\"train\\\")\\n\\n    @classmethod\\n    def load_dataset_and_make_vectorizer(cls, surname_csv):\\n        surname_df = pd.read_csv(surname_csv)\\n        return cls(surname_df, SurnameVectorizer.from_dataframe(surname_df))\\n\\n    @classmethod\\n    def load_dataset_and_load_vectorizer(cls, surname_csv, vectorizer_filepath):\\n        surname_df = pd.read_csb(surname_csv)\\n        vectorizer = cls.load_vectorizer_only(vectorizer_filepath)\\n        return cls(surname_df, vectorizer)\\n\\n    @staticmethod\\n    def load_vectorizer_only(vectorizer_filepath):\\n        with open(vectorizer_filepath) as fp:\\n            return SurnameVectorizer.from_serializable(json.load(fp))\\n\\n    def save_vectorizer(self, vectorizer_filepath):\\n        with open(vectorizer_filepath, \\\"w\\\") as fp:\\n            json.dump(self._vectorizer.to_serializable(), fp)\\n\\n    def get_vectorizer(self):\\n        return self._vectorizer\\n\\n    def set_split(self, split=\\\"train\\\"):\\n        self._train_split = split\\n        self._target_df, self._target_size = self._lookup_dict[split]\\n\\n    def __len__(self):\\n        return self._target_size\\n\\n    def __getitem__(self, index):\\n        row = self._target_df.iloc[index]\\n        from_vector, to_vector = self._vectorizer.vectorize(\\n            row.surname, self._max_seq_length\\n        )\\n        nationality_index = self._vectorizer.nationality_vocab.lookup_token(\\n            row.nationality\\n        )\\n        return {\\n            \\\"x_data\\\": from_vector,\\n            \\\"y_target\\\": to_vector,\\n            \\\"class_index\\\": nationality_index,\\n        }\\n\\n    def get_num_batches(self, batch_size):\\n        return len(self) // batch_size\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "class SurnameDataset(Dataset):\n",
    "    def __init__(self, surname_df, vectorizer):\n",
    "        self.surname_df = surname_df\n",
    "        self._vectorizer = vectorizer\n",
    "        self._max_seq_length = max(map(len, self.surname_df.surname)) + 2\n",
    "\n",
    "        self.train_df = self.surname_df[self.surname_df.split == \"train\"]\n",
    "        self.train_size = len(self.train_df)\n",
    "\n",
    "        self.val_df = self.surname_df[self.surname_df.split == \"val\"]\n",
    "        self.val_size = len(self.val_df)\n",
    "\n",
    "        self.test_df = self.surname_df[self.surname_df.split == \"test\"]\n",
    "        self.test_size = len(self.test_df)\n",
    "\n",
    "        self._lookup_dict = {\n",
    "            \"train\": (self.train_df, self.train_size),\n",
    "            \"val\": (self.val_df, self.val_size),\n",
    "            \"test\": (self.test_df, self.test_size),\n",
    "        }\n",
    "        self.set_split(\"train\")\n",
    "\n",
    "    @classmethod\n",
    "    def load_dataset_and_make_vectorizer(cls, surname_csv):\n",
    "        surname_df = pd.read_csv(surname_csv)\n",
    "        return cls(surname_df, SurnameVectorizer.from_dataframe(surname_df))\n",
    "\n",
    "    @classmethod\n",
    "    def load_dataset_and_load_vectorizer(cls, surname_csv, vectorizer_filepath):\n",
    "        surname_df = pd.read_csv(surname_csv)\n",
    "        vectorizer = cls.load_vectorizer_only(vectorizer_filepath)\n",
    "        return cls(surname_df, vectorizer)\n",
    "\n",
    "    @staticmethod\n",
    "    def load_vectorizer_only(vectorizer_filepath):\n",
    "        with open(vectorizer_filepath) as fp:\n",
    "            return SurnameVectorizer.from_serializable(json.load(fp))\n",
    "\n",
    "    def save_vectorizer(self, vectorizer_filepath):\n",
    "        with open(vectorizer_filepath, \"w\") as fp:\n",
    "            json.dump(self._vectorizer.to_serializable(), fp)\n",
    "\n",
    "    def get_vectorizer(self):\n",
    "        return self._vectorizer\n",
    "\n",
    "    def set_split(self, split=\"train\"):\n",
    "        self._train_split = split\n",
    "        self._target_df, self._target_size = self._lookup_dict[split]\n",
    "\n",
    "    def __len__(self):\n",
    "        return self._target_size\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        row = self._target_df.iloc[index]\n",
    "        from_vector, to_vector = self._vectorizer.vectorize(\n",
    "            row.surname, self._max_seq_length\n",
    "        )\n",
    "        nationality_index = self._vectorizer.nationality_vocab.lookup_token(\n",
    "            row.nationality\n",
    "        )\n",
    "        return {\n",
    "            \"x_data\": from_vector,\n",
    "            \"y_target\": to_vector,\n",
    "            \"class_index\": nationality_index,\n",
    "        }\n",
    "\n",
    "    def get_num_batches(self, batch_size):\n",
    "        return len(self) // batch_size"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cc62ec0",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Unconditioned Surname Generation Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "ec2552c0",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 36;\n",
       "                var nbb_unformatted_code = \"class SurnameGenerationModel(nn.Module):\\n    def __init__(\\n        self,\\n        char_embedding_size,\\n        char_vocab_size,\\n        rnn_hidden_size,\\n        batch_first=True,\\n        padding_idx=0,\\n        dropout_p=0.5,\\n    ):\\n        super(SurnameGenerationModel, self).__init__()\\n        self.char_emb = nn.Embedding(\\n            num_embeddings=char_vocab_size,\\n            embedding_dim=char_embedding_size,\\n            padding_idx=padding_idx,\\n        )\\n        self.rnn = nn.GRU(\\n            input_size=char_embedding_size,\\n            hidden_size=rnn_hidden_size,\\n            batch_first=batch_first,\\n        )\\n        self.fc = nn.Linear(in_features=rnn_hidden_size, out_features=char_vocab_size)\\n        self._dropout_p = dropout_p\\n\\n    def forward(self, x_in, apply_softmax=False):\\n        x_embedded = self.char_emb(x_in)\\n        y_out, _ = self.rnn(x_embedded)\\n        batch_size, seq_size, feat_size = y_out.shape\\n        y_out = y_out.contiguous().view(batch_size * seq_size, feat_size)\\n        y_out = self.fc(F.dropout(y_out, p=self._dropout_p))\\n        if apply_softmax:\\n            y_out = F.softmax(y_out, dim=1)\\n        new_feat_size = y_out.shape[-1]\\n        y_out = y_out.view(batch_size, seq_size, new_feat_size)\\n        return y_out\";\n",
       "                var nbb_formatted_code = \"class SurnameGenerationModel(nn.Module):\\n    def __init__(\\n        self,\\n        char_embedding_size,\\n        char_vocab_size,\\n        rnn_hidden_size,\\n        batch_first=True,\\n        padding_idx=0,\\n        dropout_p=0.5,\\n    ):\\n        super(SurnameGenerationModel, self).__init__()\\n        self.char_emb = nn.Embedding(\\n            num_embeddings=char_vocab_size,\\n            embedding_dim=char_embedding_size,\\n            padding_idx=padding_idx,\\n        )\\n        self.rnn = nn.GRU(\\n            input_size=char_embedding_size,\\n            hidden_size=rnn_hidden_size,\\n            batch_first=batch_first,\\n        )\\n        self.fc = nn.Linear(in_features=rnn_hidden_size, out_features=char_vocab_size)\\n        self._dropout_p = dropout_p\\n\\n    def forward(self, x_in, apply_softmax=False):\\n        x_embedded = self.char_emb(x_in)\\n        y_out, _ = self.rnn(x_embedded)\\n        batch_size, seq_size, feat_size = y_out.shape\\n        y_out = y_out.contiguous().view(batch_size * seq_size, feat_size)\\n        y_out = self.fc(F.dropout(y_out, p=self._dropout_p))\\n        if apply_softmax:\\n            y_out = F.softmax(y_out, dim=1)\\n        new_feat_size = y_out.shape[-1]\\n        y_out = y_out.view(batch_size, seq_size, new_feat_size)\\n        return y_out\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "class SurnameGenerationModel(nn.Module):\n",
    "    def __init__(\n",
    "        self,\n",
    "        char_embedding_size,\n",
    "        char_vocab_size,\n",
    "        rnn_hidden_size,\n",
    "        batch_first=True,\n",
    "        padding_idx=0,\n",
    "        dropout_p=0.5,\n",
    "    ):\n",
    "        super(SurnameGenerationModel, self).__init__()\n",
    "        self.char_emb = nn.Embedding(\n",
    "            num_embeddings=char_vocab_size,\n",
    "            embedding_dim=char_embedding_size,\n",
    "            padding_idx=padding_idx,\n",
    "        )\n",
    "        self.rnn = nn.GRU(\n",
    "            input_size=char_embedding_size,\n",
    "            hidden_size=rnn_hidden_size,\n",
    "            batch_first=batch_first,\n",
    "        )\n",
    "        self.fc = nn.Linear(in_features=rnn_hidden_size, out_features=char_vocab_size)\n",
    "        self._dropout_p = dropout_p\n",
    "\n",
    "    def forward(self, x_in, apply_softmax=False):\n",
    "        x_embedded = self.char_emb(x_in)\n",
    "        y_out, _ = self.rnn(x_embedded)\n",
    "        batch_size, seq_size, feat_size = y_out.shape\n",
    "        y_out = y_out.contiguous().view(batch_size * seq_size, feat_size)\n",
    "        y_out = self.fc(F.dropout(y_out, p=self._dropout_p))\n",
    "        if apply_softmax:\n",
    "            y_out = F.softmax(y_out, dim=1)\n",
    "        new_feat_size = y_out.shape[-1]\n",
    "        y_out = y_out.view(batch_size, seq_size, new_feat_size)\n",
    "        return y_out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "01cc1010",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 104;\n",
       "                var nbb_unformatted_code = \"def sample_from_model(\\n    model, vectorizer, num_samples=1, sample_size=20, temperature=1.0\\n):\\n    begin_seq_index = [\\n        vectorizer.char_vocab.begin_seq_index for _ in range(num_samples)\\n    ]\\n    begin_seq_index = torch.tensor(begin_seq_index, dtype=torch.int64).unsqueeze(dim=1)\\n    indices = [begin_seq_index]\\n    h_t = None\\n\\n    for time_step in range(sample_size):\\n        x_t = indices[time_step]\\n        x_emb_t = model.char_emb(x_t)\\n        rnn_out_t, h_t = model.rnn(x_emb_t, h_t)\\n        prediction_vector = model.fc(rnn_out_t.squeeze(dim=1))\\n        probability_vector = F.softmax(prediction_vector / temperature, dim=1)\\n        indices.append(torch.multinomial(probability_vector, num_samples=1))\\n    indices = torch.stack(indices).squeeze().permute(1, 0)\\n    return indices\\n\\n\\ndef decode_samples(sampled_indices, vectorizer):\\n    decoded_surnames = []\\n    vocab = vectorizer.char_vocab\\n\\n    for sample_index in range(sampled_indices.shape[0]):\\n        surname = \\\"\\\"\\n        for time_step in range(sampled_indices.shape[1]):\\n            sample_item = sampled_indices[sample_index, time_step].item()\\n            if sample_item == vocab.begin_seq_index:\\n                continue\\n            elif sample_item == vocab.end_seq_index:\\n                break\\n            else:\\n                surname += vocab.lookup_index(sample_item)\\n        decoded_surnames.append(surname)\\n    return decoded_surnames\";\n",
       "                var nbb_formatted_code = \"def sample_from_model(\\n    model, vectorizer, num_samples=1, sample_size=20, temperature=1.0\\n):\\n    begin_seq_index = [\\n        vectorizer.char_vocab.begin_seq_index for _ in range(num_samples)\\n    ]\\n    begin_seq_index = torch.tensor(begin_seq_index, dtype=torch.int64).unsqueeze(dim=1)\\n    indices = [begin_seq_index]\\n    h_t = None\\n\\n    for time_step in range(sample_size):\\n        x_t = indices[time_step]\\n        x_emb_t = model.char_emb(x_t)\\n        rnn_out_t, h_t = model.rnn(x_emb_t, h_t)\\n        prediction_vector = model.fc(rnn_out_t.squeeze(dim=1))\\n        probability_vector = F.softmax(prediction_vector / temperature, dim=1)\\n        indices.append(torch.multinomial(probability_vector, num_samples=1))\\n    indices = torch.stack(indices).squeeze().permute(1, 0)\\n    return indices\\n\\n\\ndef decode_samples(sampled_indices, vectorizer):\\n    decoded_surnames = []\\n    vocab = vectorizer.char_vocab\\n\\n    for sample_index in range(sampled_indices.shape[0]):\\n        surname = \\\"\\\"\\n        for time_step in range(sampled_indices.shape[1]):\\n            sample_item = sampled_indices[sample_index, time_step].item()\\n            if sample_item == vocab.begin_seq_index:\\n                continue\\n            elif sample_item == vocab.end_seq_index:\\n                break\\n            else:\\n                surname += vocab.lookup_index(sample_item)\\n        decoded_surnames.append(surname)\\n    return decoded_surnames\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def sample_from_model(\n",
    "    model, vectorizer, num_samples=1, sample_size=20, temperature=1.0\n",
    "):\n",
    "    begin_seq_index = [\n",
    "        vectorizer.char_vocab.begin_seq_index for _ in range(num_samples)\n",
    "    ]\n",
    "    begin_seq_index = torch.tensor(begin_seq_index, dtype=torch.int64).unsqueeze(dim=1)\n",
    "    indices = [begin_seq_index]\n",
    "    h_t = None\n",
    "\n",
    "    for time_step in range(sample_size):\n",
    "        x_t = indices[time_step]\n",
    "        x_emb_t = model.char_emb(x_t)\n",
    "        rnn_out_t, h_t = model.rnn(x_emb_t, h_t)\n",
    "        prediction_vector = model.fc(rnn_out_t.squeeze(dim=1))\n",
    "        probability_vector = F.softmax(prediction_vector / temperature, dim=1)\n",
    "        indices.append(torch.multinomial(probability_vector, num_samples=1))\n",
    "    indices = torch.stack(indices).squeeze().permute(1, 0)\n",
    "    return indices\n",
    "\n",
    "\n",
    "def decode_samples(sampled_indices, vectorizer):\n",
    "    decoded_surnames = []\n",
    "    vocab = vectorizer.char_vocab\n",
    "\n",
    "    for sample_index in range(sampled_indices.shape[0]):\n",
    "        surname = \"\"\n",
    "        for time_step in range(sampled_indices.shape[1]):\n",
    "            sample_item = sampled_indices[sample_index, time_step].item()\n",
    "            if sample_item == vocab.begin_seq_index:\n",
    "                continue\n",
    "            elif sample_item == vocab.end_seq_index:\n",
    "                break\n",
    "            else:\n",
    "                surname += vocab.lookup_index(sample_item)\n",
    "        decoded_surnames.append(surname)\n",
    "    return decoded_surnames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "d32fede6",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 50;\n",
       "                var nbb_unformatted_code = \"def normalize_sizes(y_pred, y_true):\\n    if len(y_pred.size()) == 3:\\n        y_pred = y_pred.contiguous().view(-1, y_pred.size(2))\\n    if len(y_true.size()) == 2:\\n        y_true = y_true.contiguous().view(-1)\\n    return y_pred, y_true\\n\\n\\ndef compute_accuracy(y_pred, y_true, mask_index):\\n    y_pred, y_true = normalize_sizes(y_pred, y_true)\\n    _, y_pred_indices = y_pred.max(dim=1)\\n    correct_indices = torch.eq(y_pred_indices, y_true).float()\\n    valid_indices = torch.ne(y_true, mask_index).float()\\n    n_correct = (correct_indices * valid_indices).sum().item()\\n    n_valid = valid_indices.sum().item()\\n\\n    return n_correct / n_valid * 100\\n\\n\\ndef sequence_loss(y_pred, y_true, mask_index):\\n    y_pred, y_true = normalize_sizes(y_pred, y_true)\\n    return F.cross_entropy(y_pred, y_true, ignore_index=mask_index)\";\n",
       "                var nbb_formatted_code = \"def normalize_sizes(y_pred, y_true):\\n    if len(y_pred.size()) == 3:\\n        y_pred = y_pred.contiguous().view(-1, y_pred.size(2))\\n    if len(y_true.size()) == 2:\\n        y_true = y_true.contiguous().view(-1)\\n    return y_pred, y_true\\n\\n\\ndef compute_accuracy(y_pred, y_true, mask_index):\\n    y_pred, y_true = normalize_sizes(y_pred, y_true)\\n    _, y_pred_indices = y_pred.max(dim=1)\\n    correct_indices = torch.eq(y_pred_indices, y_true).float()\\n    valid_indices = torch.ne(y_true, mask_index).float()\\n    n_correct = (correct_indices * valid_indices).sum().item()\\n    n_valid = valid_indices.sum().item()\\n\\n    return n_correct / n_valid * 100\\n\\n\\ndef sequence_loss(y_pred, y_true, mask_index):\\n    y_pred, y_true = normalize_sizes(y_pred, y_true)\\n    return F.cross_entropy(y_pred, y_true, ignore_index=mask_index)\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def normalize_sizes(y_pred, y_true):\n",
    "    if len(y_pred.size()) == 3:\n",
    "        y_pred = y_pred.contiguous().view(-1, y_pred.size(2))\n",
    "    if len(y_true.size()) == 2:\n",
    "        y_true = y_true.contiguous().view(-1)\n",
    "    return y_pred, y_true\n",
    "\n",
    "\n",
    "def compute_accuracy(y_pred, y_true, mask_index):\n",
    "    y_pred, y_true = normalize_sizes(y_pred, y_true)\n",
    "    _, y_pred_indices = y_pred.max(dim=1)\n",
    "    correct_indices = torch.eq(y_pred_indices, y_true).float()\n",
    "    valid_indices = torch.ne(y_true, mask_index).float()\n",
    "    n_correct = (correct_indices * valid_indices).sum().item()\n",
    "    n_valid = valid_indices.sum().item()\n",
    "\n",
    "    return n_correct / n_valid * 100\n",
    "\n",
    "\n",
    "def sequence_loss(y_pred, y_true, mask_index):\n",
    "    y_pred, y_true = normalize_sizes(y_pred, y_true)\n",
    "    return F.cross_entropy(y_pred, y_true, ignore_index=mask_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "12fe8cc3",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using CUDA: False\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 97;\n",
       "                var nbb_unformatted_code = \"args = Namespace(\\n    # Data and path information\\n    surname_csv=\\\"../data/surnames/surnames_with_splits.csv\\\",\\n    vectorizer_file=\\\"vectorizer.json\\\",\\n    model_state_file=\\\"model.pth\\\",\\n    save_dir=\\\"models/chapter07/model1_unconditioned_surname_generation\\\",\\n    # Model hyper parameter\\n    char_embedding_size=32,\\n    rnn_hidden_size=32,\\n    # Training hyper parameter\\n    num_epochs=100,\\n    learning_rate=0.001,\\n    batch_size=128,\\n    seed=1337,\\n    early_stopping_criteria=5,\\n    # Runtime hyper parameter\\n    cuda=True,\\n    catch_keyboard_interrupt=True,\\n    reload_from_files=False,\\n    expand_filepaths_to_save_dir=True,\\n)\\n\\nif not torch.cuda.is_available():\\n    args.cuda = False\\n\\nargs.device = torch.device(\\\"cuda\\\" if args.cuda else \\\"cpu\\\")\\n\\nprint(\\\"Using CUDA: {}\\\".format(args.cuda))\\n\\n\\nif args.expand_filepaths_to_save_dir:\\n    args.vectorizer_file = os.path.join(args.save_dir, args.vectorizer_file)\\n\\n    args.model_state_file = os.path.join(args.save_dir, args.model_state_file)\\n\\n# Set seed for reproducibility\\nutils.set_seed_everywhere(args.seed, args.cuda)\\n\\n# handle dirs\\nutils.handle_dirs(args.save_dir)\";\n",
       "                var nbb_formatted_code = \"args = Namespace(\\n    # Data and path information\\n    surname_csv=\\\"../data/surnames/surnames_with_splits.csv\\\",\\n    vectorizer_file=\\\"vectorizer.json\\\",\\n    model_state_file=\\\"model.pth\\\",\\n    save_dir=\\\"models/chapter07/model1_unconditioned_surname_generation\\\",\\n    # Model hyper parameter\\n    char_embedding_size=32,\\n    rnn_hidden_size=32,\\n    # Training hyper parameter\\n    num_epochs=100,\\n    learning_rate=0.001,\\n    batch_size=128,\\n    seed=1337,\\n    early_stopping_criteria=5,\\n    # Runtime hyper parameter\\n    cuda=True,\\n    catch_keyboard_interrupt=True,\\n    reload_from_files=False,\\n    expand_filepaths_to_save_dir=True,\\n)\\n\\nif not torch.cuda.is_available():\\n    args.cuda = False\\n\\nargs.device = torch.device(\\\"cuda\\\" if args.cuda else \\\"cpu\\\")\\n\\nprint(\\\"Using CUDA: {}\\\".format(args.cuda))\\n\\n\\nif args.expand_filepaths_to_save_dir:\\n    args.vectorizer_file = os.path.join(args.save_dir, args.vectorizer_file)\\n\\n    args.model_state_file = os.path.join(args.save_dir, args.model_state_file)\\n\\n# Set seed for reproducibility\\nutils.set_seed_everywhere(args.seed, args.cuda)\\n\\n# handle dirs\\nutils.handle_dirs(args.save_dir)\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "args = Namespace(\n",
    "    # Data and path information\n",
    "    surname_csv=\"../data/surnames/surnames_with_splits.csv\",\n",
    "    vectorizer_file=\"vectorizer.json\",\n",
    "    model_state_file=\"model.pth\",\n",
    "    save_dir=\"models/chapter07/model1_unconditioned_surname_generation\",\n",
    "    # Model hyper parameter\n",
    "    char_embedding_size=32,\n",
    "    rnn_hidden_size=32,\n",
    "    # Training hyper parameter\n",
    "    num_epochs=100,\n",
    "    learning_rate=0.001,\n",
    "    batch_size=128,\n",
    "    seed=1337,\n",
    "    early_stopping_criteria=5,\n",
    "    # Runtime hyper parameter\n",
    "    cuda=True,\n",
    "    catch_keyboard_interrupt=True,\n",
    "    reload_from_files=False,\n",
    "    expand_filepaths_to_save_dir=True,\n",
    ")\n",
    "\n",
    "if not torch.cuda.is_available():\n",
    "    args.cuda = False\n",
    "\n",
    "args.device = torch.device(\"cuda\" if args.cuda else \"cpu\")\n",
    "\n",
    "print(\"Using CUDA: {}\".format(args.cuda))\n",
    "\n",
    "\n",
    "if args.expand_filepaths_to_save_dir:\n",
    "    args.vectorizer_file = os.path.join(args.save_dir, args.vectorizer_file)\n",
    "\n",
    "    args.model_state_file = os.path.join(args.save_dir, args.model_state_file)\n",
    "\n",
    "# Set seed for reproducibility\n",
    "utils.set_seed_everywhere(args.seed, args.cuda)\n",
    "\n",
    "# handle dirs\n",
    "utils.handle_dirs(args.save_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "3742011b",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SurnameGenerationModel(\n",
      "  (char_emb): Embedding(88, 32, padding_idx=0)\n",
      "  (rnn): GRU(32, 32, batch_first=True)\n",
      "  (fc): Linear(in_features=32, out_features=88, bias=True)\n",
      ")\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 98;\n",
       "                var nbb_unformatted_code = \"dataset = SurnameDataset.load_dataset_and_make_vectorizer(args.surname_csv)\\ndataset.save_vectorizer(args.vectorizer_file)\\nvectorizer = dataset.get_vectorizer()\\nmodel = SurnameGenerationModel(\\n    char_embedding_size=args.char_embedding_size,\\n    char_vocab_size=len(vectorizer.char_vocab),\\n    rnn_hidden_size=args.rnn_hidden_size,\\n    padding_idx=vectorizer.char_vocab.mask_index,\\n)\\nprint(model)\\nmodel = model.to(args.device)\\noptimizer = optim.Adam(model.parameters(), lr=args.learning_rate)\\nscheduler = optim.lr_scheduler.ReduceLROnPlateau(\\n    optimizer=optimizer, mode=\\\"min\\\", factor=0.5, patience=1\\n)\";\n",
       "                var nbb_formatted_code = \"dataset = SurnameDataset.load_dataset_and_make_vectorizer(args.surname_csv)\\ndataset.save_vectorizer(args.vectorizer_file)\\nvectorizer = dataset.get_vectorizer()\\nmodel = SurnameGenerationModel(\\n    char_embedding_size=args.char_embedding_size,\\n    char_vocab_size=len(vectorizer.char_vocab),\\n    rnn_hidden_size=args.rnn_hidden_size,\\n    padding_idx=vectorizer.char_vocab.mask_index,\\n)\\nprint(model)\\nmodel = model.to(args.device)\\noptimizer = optim.Adam(model.parameters(), lr=args.learning_rate)\\nscheduler = optim.lr_scheduler.ReduceLROnPlateau(\\n    optimizer=optimizer, mode=\\\"min\\\", factor=0.5, patience=1\\n)\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dataset = SurnameDataset.load_dataset_and_make_vectorizer(args.surname_csv)\n",
    "dataset.save_vectorizer(args.vectorizer_file)\n",
    "vectorizer = dataset.get_vectorizer()\n",
    "model = SurnameGenerationModel(\n",
    "    char_embedding_size=args.char_embedding_size,\n",
    "    char_vocab_size=len(vectorizer.char_vocab),\n",
    "    rnn_hidden_size=args.rnn_hidden_size,\n",
    "    padding_idx=vectorizer.char_vocab.mask_index,\n",
    ")\n",
    "print(model)\n",
    "model = model.to(args.device)\n",
    "optimizer = optim.Adam(model.parameters(), lr=args.learning_rate)\n",
    "scheduler = optim.lr_scheduler.ReduceLROnPlateau(\n",
    "    optimizer=optimizer, mode=\"min\", factor=0.5, patience=1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "6175d7bd",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dd5515aa198c4f06b411505ba86224f3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Training Routine:   0%|          | 0/100 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c707dc787be94a1e826c1029e1e0bedd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "split=train:   0%|          | 0/60 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fc051a65cf05419c95b359c03c7be373",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "split=val:   0%|          | 0/12 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "============ Split=train, Size=7680 ============\n",
      "============ Split=val, Size=1640 ============\n",
      "--------------- 0th Epoch Stats---------------\n",
      "Training Loss=4.077961770693462, Training Accuracy=7.676054975941436\n",
      "Validation Loss=3.425482432047526, Validation Accuracy=11.971830492983797.\n",
      "------------------------------------------------------------\n",
      "--------------- 10th Epoch Stats---------------\n",
      "Training Loss=2.636580200990042, Training Accuracy=23.603338443200688\n",
      "Validation Loss=2.6279960076014204, Validation Accuracy=23.469704855403997.\n",
      "------------------------------------------------------------\n",
      "--------------- 20th Epoch Stats---------------\n",
      "Training Loss=2.577921203772227, Training Accuracy=24.38479427370071\n",
      "Validation Loss=2.58339116970698, Validation Accuracy=24.008097801260053.\n",
      "------------------------------------------------------------\n",
      "--------------- 30th Epoch Stats---------------\n",
      "Training Loss=2.54895738363266, Training Accuracy=24.814225940274085\n",
      "Validation Loss=2.565061529477437, Validation Accuracy=24.73465771765003.\n",
      "------------------------------------------------------------\n",
      "--------------- 40th Epoch Stats---------------\n",
      "Training Loss=2.5395797689755755, Training Accuracy=24.95948712783898\n",
      "Validation Loss=2.550375521183014, Validation Accuracy=24.520194209338342.\n",
      "------------------------------------------------------------\n",
      "--------------- 50th Epoch Stats---------------\n",
      "Training Loss=2.536512772242229, Training Accuracy=25.083264225072853\n",
      "Validation Loss=2.5617401202519736, Validation Accuracy=24.44636791493978.\n",
      "------------------------------------------------------------\n",
      "--------------- 60th Epoch Stats---------------\n",
      "Training Loss=2.5364448269208264, Training Accuracy=25.029441054904073\n",
      "Validation Loss=2.553151826063792, Validation Accuracy=24.572171943039805.\n",
      "------------------------------------------------------------\n",
      "--------------- 70th Epoch Stats---------------\n",
      "Training Loss=2.5397751371065773, Training Accuracy=25.002176289877955\n",
      "Validation Loss=2.5596306522687273, Validation Accuracy=24.653705929237624.\n",
      "------------------------------------------------------------\n",
      "--------------- 80th Epoch Stats---------------\n",
      "Training Loss=2.5355549097061156, Training Accuracy=25.14645809653731\n",
      "Validation Loss=2.5509379506111145, Validation Accuracy=24.52155168203823.\n",
      "------------------------------------------------------------\n",
      "--------------- 90th Epoch Stats---------------\n",
      "Training Loss=2.5389617482821145, Training Accuracy=25.134524038670776\n",
      "Validation Loss=2.5686229467391963, Validation Accuracy=24.37174679743489.\n",
      "------------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 99;\n",
       "                var nbb_unformatted_code = \"mask_index = vectorizer.char_vocab.mask_index\\ntrain_state = utils.make_train_state(args)\\nepoch_bar = notebook.tqdm(desc=\\\"Training Routine\\\", total=args.num_epochs, position=0)\\ndataset.set_split(\\\"train\\\")\\ntrain_bar = notebook.tqdm(\\n    desc=\\\"split=train\\\",\\n    total=dataset.get_num_batches(args.batch_size),\\n    position=1,\\n    leave=True,\\n)\\ndataset.set_split(\\\"val\\\")\\nval_bar = notebook.tqdm(\\n    desc=\\\"split=val\\\",\\n    total=dataset.get_num_batches(args.batch_size),\\n    position=1,\\n    leave=True,\\n)\\n\\nfor epoch_index in range(args.num_epochs):\\n    train_state[\\\"epoch_index\\\"] = epoch_index\\n    # Iterate Over Training Dataset\\n    # Setup: Batch Generator, set loss & acc to 0, set train mode on\\n    dataset.set_split(\\\"train\\\")\\n    if epoch_index == 0:\\n        print(\\n            f\\\"============ Split={dataset._train_split}, Size={len(dataset)} ============\\\"\\n        )\\n    batch_generator = utils.generate_batches(\\n        dataset, batch_size=args.batch_size, device=args.device\\n    )\\n    training_running_loss, training_running_acc = 0.0, 0.0\\n    model.train()\\n\\n    for batch_index, batch_dict in enumerate(batch_generator):\\n        # 5 Step Training Routine\\n\\n        # Step 1. Zero the Gradients\\n        optimizer.zero_grad()\\n\\n        # Step 2. Compute the gradients\\n        y_pred = model(x_in=batch_dict[\\\"x_data\\\"])\\n\\n        # Step 3. Compute the Output\\n        loss = sequence_loss(y_pred, batch_dict[\\\"y_target\\\"], mask_index=mask_index)\\n\\n        # Step 4. Use loss to produce gradients\\n        loss.backward()\\n\\n        # Step 5. Use Optimizer to take gradient step\\n        optimizer.step()\\n\\n        # Compute the running loss and accuracy\\n        loss_batch = loss.item()\\n        training_running_loss += (loss_batch - training_running_loss) / (\\n            batch_index + 1\\n        )\\n        acc_batch = compute_accuracy(y_pred, batch_dict[\\\"y_target\\\"], mask_index)\\n        training_running_acc += (acc_batch - training_running_acc) / (batch_index + 1)\\n\\n        # Update the bar\\n        train_bar.set_postfix(\\n            loss=training_running_loss, acc=training_running_acc, epoch=epoch_index\\n        )\\n        train_bar.update()\\n    train_state[\\\"train_loss\\\"].append(training_running_loss)\\n    train_state[\\\"train_acc\\\"].append(training_running_acc)\\n\\n    # Iterate Over Val Dataset\\n    # Setup: Batch Generator, set loss and acc to 0, set eval mode on\\n    dataset.set_split(\\\"val\\\")\\n    val_running_loss, val_running_acc = 0.0, 0.0\\n    if len(dataset) > 0:\\n        if epoch_index == 0:\\n            print(\\n                f\\\"============ Split={dataset._train_split}, Size={len(dataset)} ============\\\"\\n            )\\n        batch_generator = utils.generate_batches(\\n            dataset, batch_size=args.batch_size, device=args.device\\n        )\\n        model.eval()\\n\\n        for batch_index, batch_dict in enumerate(batch_generator):\\n            # Step 1. Compute the Output\\n            y_pred = model(x_in=batch_dict[\\\"x_data\\\"])\\n\\n            # Step 2. Compute the loss\\n            loss = sequence_loss(y_pred, batch_dict[\\\"y_target\\\"], mask_index)\\n            loss_batch = loss.item()\\n            val_running_loss += (loss_batch - val_running_loss) / (batch_index + 1)\\n\\n            # Step 3. Compute the accuracy\\n            acc_batch = compute_accuracy(y_pred, batch_dict[\\\"y_target\\\"], mask_index)\\n            val_running_acc += (acc_batch - val_running_acc) / (batch_index + 1)\\n            val_bar.set_postfix(\\n                loss=val_running_loss, acc=val_running_acc, epoch=epoch_index\\n            )\\n            val_bar.update()\\n        train_state[\\\"val_loss\\\"].append(val_running_loss)\\n        train_state[\\\"val_acc\\\"].append(val_running_acc)\\n        scheduler.step(train_state[\\\"val_loss\\\"][-1])\\n    else:\\n        if epoch_index == 0:\\n            print(f\\\"============ Skipping Validation Pass ============\\\")\\n        train_state[\\\"val_loss\\\"].append(val_running_loss)\\n        train_state[\\\"val_acc\\\"].append(val_running_acc)\\n        scheduler.step(train_state[\\\"train_loss\\\"][-1])\\n    train_state = utils.update_train_state(\\n        args=args, model=classifier, train_state=train_state\\n    )\\n\\n    train_bar.n, val_bar.n = 0, 0\\n    epoch_bar.update()\\n\\n    if train_state[\\\"stop_early\\\"]:\\n        print(\\\"Stopping early....\\\")\\n        break\\n\\n    if epoch_index % 10 == 0:\\n        print(\\n            f\\\"--------------- {epoch_index}th Epoch Stats---------------\\\\n\\\"\\n            f\\\"Training Loss={training_running_loss}, \\\"\\n            f\\\"Training Accuracy={training_running_acc}\\\\n\\\"\\n            f\\\"Validation Loss={val_running_loss}, \\\"\\n            f\\\"Validation Accuracy={val_running_acc}.\\\\n\\\"\\n            \\\"------------------------------------------------------------\\\"\\n        )\";\n",
       "                var nbb_formatted_code = \"mask_index = vectorizer.char_vocab.mask_index\\ntrain_state = utils.make_train_state(args)\\nepoch_bar = notebook.tqdm(desc=\\\"Training Routine\\\", total=args.num_epochs, position=0)\\ndataset.set_split(\\\"train\\\")\\ntrain_bar = notebook.tqdm(\\n    desc=\\\"split=train\\\",\\n    total=dataset.get_num_batches(args.batch_size),\\n    position=1,\\n    leave=True,\\n)\\ndataset.set_split(\\\"val\\\")\\nval_bar = notebook.tqdm(\\n    desc=\\\"split=val\\\",\\n    total=dataset.get_num_batches(args.batch_size),\\n    position=1,\\n    leave=True,\\n)\\n\\nfor epoch_index in range(args.num_epochs):\\n    train_state[\\\"epoch_index\\\"] = epoch_index\\n    # Iterate Over Training Dataset\\n    # Setup: Batch Generator, set loss & acc to 0, set train mode on\\n    dataset.set_split(\\\"train\\\")\\n    if epoch_index == 0:\\n        print(\\n            f\\\"============ Split={dataset._train_split}, Size={len(dataset)} ============\\\"\\n        )\\n    batch_generator = utils.generate_batches(\\n        dataset, batch_size=args.batch_size, device=args.device\\n    )\\n    training_running_loss, training_running_acc = 0.0, 0.0\\n    model.train()\\n\\n    for batch_index, batch_dict in enumerate(batch_generator):\\n        # 5 Step Training Routine\\n\\n        # Step 1. Zero the Gradients\\n        optimizer.zero_grad()\\n\\n        # Step 2. Compute the gradients\\n        y_pred = model(x_in=batch_dict[\\\"x_data\\\"])\\n\\n        # Step 3. Compute the Output\\n        loss = sequence_loss(y_pred, batch_dict[\\\"y_target\\\"], mask_index=mask_index)\\n\\n        # Step 4. Use loss to produce gradients\\n        loss.backward()\\n\\n        # Step 5. Use Optimizer to take gradient step\\n        optimizer.step()\\n\\n        # Compute the running loss and accuracy\\n        loss_batch = loss.item()\\n        training_running_loss += (loss_batch - training_running_loss) / (\\n            batch_index + 1\\n        )\\n        acc_batch = compute_accuracy(y_pred, batch_dict[\\\"y_target\\\"], mask_index)\\n        training_running_acc += (acc_batch - training_running_acc) / (batch_index + 1)\\n\\n        # Update the bar\\n        train_bar.set_postfix(\\n            loss=training_running_loss, acc=training_running_acc, epoch=epoch_index\\n        )\\n        train_bar.update()\\n    train_state[\\\"train_loss\\\"].append(training_running_loss)\\n    train_state[\\\"train_acc\\\"].append(training_running_acc)\\n\\n    # Iterate Over Val Dataset\\n    # Setup: Batch Generator, set loss and acc to 0, set eval mode on\\n    dataset.set_split(\\\"val\\\")\\n    val_running_loss, val_running_acc = 0.0, 0.0\\n    if len(dataset) > 0:\\n        if epoch_index == 0:\\n            print(\\n                f\\\"============ Split={dataset._train_split}, Size={len(dataset)} ============\\\"\\n            )\\n        batch_generator = utils.generate_batches(\\n            dataset, batch_size=args.batch_size, device=args.device\\n        )\\n        model.eval()\\n\\n        for batch_index, batch_dict in enumerate(batch_generator):\\n            # Step 1. Compute the Output\\n            y_pred = model(x_in=batch_dict[\\\"x_data\\\"])\\n\\n            # Step 2. Compute the loss\\n            loss = sequence_loss(y_pred, batch_dict[\\\"y_target\\\"], mask_index)\\n            loss_batch = loss.item()\\n            val_running_loss += (loss_batch - val_running_loss) / (batch_index + 1)\\n\\n            # Step 3. Compute the accuracy\\n            acc_batch = compute_accuracy(y_pred, batch_dict[\\\"y_target\\\"], mask_index)\\n            val_running_acc += (acc_batch - val_running_acc) / (batch_index + 1)\\n            val_bar.set_postfix(\\n                loss=val_running_loss, acc=val_running_acc, epoch=epoch_index\\n            )\\n            val_bar.update()\\n        train_state[\\\"val_loss\\\"].append(val_running_loss)\\n        train_state[\\\"val_acc\\\"].append(val_running_acc)\\n        scheduler.step(train_state[\\\"val_loss\\\"][-1])\\n    else:\\n        if epoch_index == 0:\\n            print(f\\\"============ Skipping Validation Pass ============\\\")\\n        train_state[\\\"val_loss\\\"].append(val_running_loss)\\n        train_state[\\\"val_acc\\\"].append(val_running_acc)\\n        scheduler.step(train_state[\\\"train_loss\\\"][-1])\\n    train_state = utils.update_train_state(\\n        args=args, model=classifier, train_state=train_state\\n    )\\n\\n    train_bar.n, val_bar.n = 0, 0\\n    epoch_bar.update()\\n\\n    if train_state[\\\"stop_early\\\"]:\\n        print(\\\"Stopping early....\\\")\\n        break\\n\\n    if epoch_index % 10 == 0:\\n        print(\\n            f\\\"--------------- {epoch_index}th Epoch Stats---------------\\\\n\\\"\\n            f\\\"Training Loss={training_running_loss}, \\\"\\n            f\\\"Training Accuracy={training_running_acc}\\\\n\\\"\\n            f\\\"Validation Loss={val_running_loss}, \\\"\\n            f\\\"Validation Accuracy={val_running_acc}.\\\\n\\\"\\n            \\\"------------------------------------------------------------\\\"\\n        )\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mask_index = vectorizer.char_vocab.mask_index\n",
    "train_state = utils.make_train_state(args)\n",
    "epoch_bar = notebook.tqdm(desc=\"Training Routine\", total=args.num_epochs, position=0)\n",
    "dataset.set_split(\"train\")\n",
    "train_bar = notebook.tqdm(\n",
    "    desc=\"split=train\",\n",
    "    total=dataset.get_num_batches(args.batch_size),\n",
    "    position=1,\n",
    "    leave=True,\n",
    ")\n",
    "dataset.set_split(\"val\")\n",
    "val_bar = notebook.tqdm(\n",
    "    desc=\"split=val\",\n",
    "    total=dataset.get_num_batches(args.batch_size),\n",
    "    position=1,\n",
    "    leave=True,\n",
    ")\n",
    "\n",
    "for epoch_index in range(args.num_epochs):\n",
    "    train_state[\"epoch_index\"] = epoch_index\n",
    "    # Iterate Over Training Dataset\n",
    "    # Setup: Batch Generator, set loss & acc to 0, set train mode on\n",
    "    dataset.set_split(\"train\")\n",
    "    if epoch_index == 0:\n",
    "        print(\n",
    "            f\"============ Split={dataset._train_split}, Size={len(dataset)} ============\"\n",
    "        )\n",
    "    batch_generator = utils.generate_batches(\n",
    "        dataset, batch_size=args.batch_size, device=args.device\n",
    "    )\n",
    "    training_running_loss, training_running_acc = 0.0, 0.0\n",
    "    model.train()\n",
    "\n",
    "    for batch_index, batch_dict in enumerate(batch_generator):\n",
    "        # 5 Step Training Routine\n",
    "\n",
    "        # Step 1. Zero the Gradients\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        # Step 2. Compute the gradients\n",
    "        y_pred = model(x_in=batch_dict[\"x_data\"])\n",
    "\n",
    "        # Step 3. Compute the Output\n",
    "        loss = sequence_loss(y_pred, batch_dict[\"y_target\"], mask_index=mask_index)\n",
    "\n",
    "        # Step 4. Use loss to produce gradients\n",
    "        loss.backward()\n",
    "\n",
    "        # Step 5. Use Optimizer to take gradient step\n",
    "        optimizer.step()\n",
    "\n",
    "        # Compute the running loss and accuracy\n",
    "        loss_batch = loss.item()\n",
    "        training_running_loss += (loss_batch - training_running_loss) / (\n",
    "            batch_index + 1\n",
    "        )\n",
    "        acc_batch = compute_accuracy(y_pred, batch_dict[\"y_target\"], mask_index)\n",
    "        training_running_acc += (acc_batch - training_running_acc) / (batch_index + 1)\n",
    "\n",
    "        # Update the bar\n",
    "        train_bar.set_postfix(\n",
    "            loss=training_running_loss, acc=training_running_acc, epoch=epoch_index\n",
    "        )\n",
    "        train_bar.update()\n",
    "    train_state[\"train_loss\"].append(training_running_loss)\n",
    "    train_state[\"train_acc\"].append(training_running_acc)\n",
    "\n",
    "    # Iterate Over Val Dataset\n",
    "    # Setup: Batch Generator, set loss and acc to 0, set eval mode on\n",
    "    dataset.set_split(\"val\")\n",
    "    val_running_loss, val_running_acc = 0.0, 0.0\n",
    "    if len(dataset) > 0:\n",
    "        if epoch_index == 0:\n",
    "            print(\n",
    "                f\"============ Split={dataset._train_split}, Size={len(dataset)} ============\"\n",
    "            )\n",
    "        batch_generator = utils.generate_batches(\n",
    "            dataset, batch_size=args.batch_size, device=args.device\n",
    "        )\n",
    "        model.eval()\n",
    "\n",
    "        for batch_index, batch_dict in enumerate(batch_generator):\n",
    "            # Step 1. Compute the Output\n",
    "            y_pred = model(x_in=batch_dict[\"x_data\"])\n",
    "\n",
    "            # Step 2. Compute the loss\n",
    "            loss = sequence_loss(y_pred, batch_dict[\"y_target\"], mask_index)\n",
    "            loss_batch = loss.item()\n",
    "            val_running_loss += (loss_batch - val_running_loss) / (batch_index + 1)\n",
    "\n",
    "            # Step 3. Compute the accuracy\n",
    "            acc_batch = compute_accuracy(y_pred, batch_dict[\"y_target\"], mask_index)\n",
    "            val_running_acc += (acc_batch - val_running_acc) / (batch_index + 1)\n",
    "            val_bar.set_postfix(\n",
    "                loss=val_running_loss, acc=val_running_acc, epoch=epoch_index\n",
    "            )\n",
    "            val_bar.update()\n",
    "        train_state[\"val_loss\"].append(val_running_loss)\n",
    "        train_state[\"val_acc\"].append(val_running_acc)\n",
    "        scheduler.step(train_state[\"val_loss\"][-1])\n",
    "    else:\n",
    "        if epoch_index == 0:\n",
    "            print(f\"============ Skipping Validation Pass ============\")\n",
    "        train_state[\"val_loss\"].append(val_running_loss)\n",
    "        train_state[\"val_acc\"].append(val_running_acc)\n",
    "        scheduler.step(train_state[\"train_loss\"][-1])\n",
    "    train_state = utils.update_train_state(\n",
    "        args=args, model=model, train_state=train_state\n",
    "    )\n",
    "\n",
    "    train_bar.n, val_bar.n = 0, 0\n",
    "    epoch_bar.update()\n",
    "\n",
    "    if train_state[\"stop_early\"]:\n",
    "        print(\"Stopping early....\")\n",
    "        break\n",
    "\n",
    "    if epoch_index % 10 == 0:\n",
    "        print(\n",
    "            f\"--------------- {epoch_index}th Epoch Stats---------------\\n\"\n",
    "            f\"Training Loss={training_running_loss}, \"\n",
    "            f\"Training Accuracy={training_running_acc}\\n\"\n",
    "            f\"Validation Loss={val_running_loss}, \"\n",
    "            f\"Validation Accuracy={val_running_acc}.\\n\"\n",
    "            \"------------------------------------------------------------\"\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "25b71035",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([8, 7])"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 100;\n",
       "                var nbb_unformatted_code = \"np.random.choice(np.arange(len(vectorizer.nationality_vocab)), replace=True, size=2)\";\n",
       "                var nbb_formatted_code = \"np.random.choice(np.arange(len(vectorizer.nationality_vocab)), replace=True, size=2)\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.random.choice(np.arange(len(vectorizer.nationality_vocab)), replace=True, size=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "228ac3cc",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 101;\n",
       "                var nbb_unformatted_code = \"model.load_state_dict(torch.load(train_state[\\\"model_filename\\\"]))\\n\\nmodel = model.to(args.device)\\n\\ndataset.set_split(\\\"test\\\")\\n\\nbatch_generator = utils.generate_batches(\\n    dataset, batch_size=args.batch_size, device=args.device\\n)\\nrunning_acc, running_loss = 0, 0\\nmodel.eval()\\n\\nfor batch_index, batch_dict in enumerate(batch_generator):\\n    y_pred = model(x_in=batch_dict[\\\"x_data\\\"])\\n    loss = sequence_loss(y_pred, batch_dict[\\\"y_target\\\"], mask_index)\\n    running_loss += (loss.item() - running_loss) / (batch_index + 1)\\n    acc_batch = compute_accuracy(y_pred, batch_dict[\\\"y_target\\\"], mask_index)\\n    running_acc += (acc_batch - running_acc) / (batch_index + 1)\\ntrain_state[\\\"test_loss\\\"] = running_loss\\ntrain_state[\\\"test_acc\\\"] = running_acc\";\n",
       "                var nbb_formatted_code = \"model.load_state_dict(torch.load(train_state[\\\"model_filename\\\"]))\\n\\nmodel = model.to(args.device)\\n\\ndataset.set_split(\\\"test\\\")\\n\\nbatch_generator = utils.generate_batches(\\n    dataset, batch_size=args.batch_size, device=args.device\\n)\\nrunning_acc, running_loss = 0, 0\\nmodel.eval()\\n\\nfor batch_index, batch_dict in enumerate(batch_generator):\\n    y_pred = model(x_in=batch_dict[\\\"x_data\\\"])\\n    loss = sequence_loss(y_pred, batch_dict[\\\"y_target\\\"], mask_index)\\n    running_loss += (loss.item() - running_loss) / (batch_index + 1)\\n    acc_batch = compute_accuracy(y_pred, batch_dict[\\\"y_target\\\"], mask_index)\\n    running_acc += (acc_batch - running_acc) / (batch_index + 1)\\ntrain_state[\\\"test_loss\\\"] = running_loss\\ntrain_state[\\\"test_acc\\\"] = running_acc\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model.load_state_dict(torch.load(train_state[\"model_filename\"]))\n",
    "\n",
    "model = model.to(args.device)\n",
    "\n",
    "dataset.set_split(\"test\")\n",
    "\n",
    "batch_generator = utils.generate_batches(\n",
    "    dataset, batch_size=args.batch_size, device=args.device\n",
    ")\n",
    "running_acc, running_loss = 0, 0\n",
    "model.eval()\n",
    "\n",
    "for batch_index, batch_dict in enumerate(batch_generator):\n",
    "    y_pred = model(x_in=batch_dict[\"x_data\"])\n",
    "    loss = sequence_loss(y_pred, batch_dict[\"y_target\"], mask_index)\n",
    "    running_loss += (loss.item() - running_loss) / (batch_index + 1)\n",
    "    acc_batch = compute_accuracy(y_pred, batch_dict[\"y_target\"], mask_index)\n",
    "    running_acc += (acc_batch - running_acc) / (batch_index + 1)\n",
    "train_state[\"test_loss\"] = running_loss\n",
    "train_state[\"test_acc\"] = running_acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "87baee4c",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 2.556217153867086;\n",
      "Test Accuracy: 25.015833225308135\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 102;\n",
       "                var nbb_unformatted_code = \"print(\\\"Test loss: {};\\\".format(train_state[\\\"test_loss\\\"]))\\nprint(\\\"Test Accuracy: {}\\\".format(train_state[\\\"test_acc\\\"]))\";\n",
       "                var nbb_formatted_code = \"print(\\\"Test loss: {};\\\".format(train_state[\\\"test_loss\\\"]))\\nprint(\\\"Test Accuracy: {}\\\".format(train_state[\\\"test_acc\\\"]))\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"Test loss: {};\".format(train_state[\"test_loss\"]))\n",
    "print(\"Test Accuracy: {}\".format(train_state[\"test_acc\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "c6f778de",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------\n",
      "Turfanh\n",
      "Vertoeli\n",
      "Mamdane\n",
      "Pawerror\n",
      "Satch\n",
      "Gasondakr\n",
      "Hailfks\n",
      "Janny\n",
      "Segler\n",
      "Geeav\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 105;\n",
       "                var nbb_unformatted_code = \"num_names = 10\\nmodel = model.cpu()\\nsamples_surnames = decode_samples(\\n    sample_from_model(model, vectorizer, num_samples=num_names), vectorizer\\n)\\nprint(\\\"-\\\" * 15)\\nfor i in range(num_names):\\n    print(samples_surnames[i])\";\n",
       "                var nbb_formatted_code = \"num_names = 10\\nmodel = model.cpu()\\nsamples_surnames = decode_samples(\\n    sample_from_model(model, vectorizer, num_samples=num_names), vectorizer\\n)\\nprint(\\\"-\\\" * 15)\\nfor i in range(num_names):\\n    print(samples_surnames[i])\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "num_names = 10\n",
    "model = model.cpu()\n",
    "samples_surnames = decode_samples(\n",
    "    sample_from_model(model, vectorizer, num_samples=num_names), vectorizer\n",
    ")\n",
    "print(\"-\" * 15)\n",
    "for i in range(num_names):\n",
    "    print(samples_surnames[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9186062c",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Conditioned Surname Generation Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "e441f55f",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 81;\n",
       "                var nbb_unformatted_code = \"class SurnameGenerationModel2(nn.Module):\\n    def __init__(\\n        self,\\n        char_embedding_size,\\n        char_vocab_size,\\n        num_nationalities,\\n        rnn_hidden_size,\\n        batch_first=True,\\n        padding_idx=0,\\n        dropout_p=0.5,\\n    ):\\n        super(SurnameGenerationModel2, self).__init__()\\n        self.char_emb = nn.Embedding(\\n            num_embeddings=char_vocab_size,\\n            embedding_dim=char_embedding_size,\\n            padding_idx=padding_idx,\\n        )\\n        self.nation_emb = nn.Embedding(\\n            num_embeddings=num_nationalities, embedding_dim=rnn_hidden_size\\n        )\\n        self.rnn = nn.GRU(\\n            input_size=char_embedding_size,\\n            hidden_size=rnn_hidden_size,\\n            batch_first=batch_first,\\n        )\\n        self.fc = nn.Linear(in_features=rnn_hidden_size, out_features=char_vocab_size)\\n        self._dropout_p = dropout_p\\n\\n    def forward(self, x_in, nationality_index, apply_softmax=False):\\n        x_embedded = self.char_emb(x_in)\\n        nationality_embedded = self.nation_emb(nationality_index).unsqueeze(0)\\n        y_out, _ = self.rnn(x_embedded, nationality_embedded)\\n        batch_size, seq_size, feat_size = y_out.shape\\n        y_out = y_out.contiguous().view(batch_size * seq_size, feat_size)\\n        y_out = self.fc(F.dropout(y_out, p=self._dropout_p))\\n        if apply_softmax:\\n            y_out = F.softmax(y_out, dim=1)\\n        new_feat_size = y_out.shape[-1]\\n        y_out = y_out.view(batch_size, seq_size, new_feat_size)\\n        return y_out\";\n",
       "                var nbb_formatted_code = \"class SurnameGenerationModel2(nn.Module):\\n    def __init__(\\n        self,\\n        char_embedding_size,\\n        char_vocab_size,\\n        num_nationalities,\\n        rnn_hidden_size,\\n        batch_first=True,\\n        padding_idx=0,\\n        dropout_p=0.5,\\n    ):\\n        super(SurnameGenerationModel2, self).__init__()\\n        self.char_emb = nn.Embedding(\\n            num_embeddings=char_vocab_size,\\n            embedding_dim=char_embedding_size,\\n            padding_idx=padding_idx,\\n        )\\n        self.nation_emb = nn.Embedding(\\n            num_embeddings=num_nationalities, embedding_dim=rnn_hidden_size\\n        )\\n        self.rnn = nn.GRU(\\n            input_size=char_embedding_size,\\n            hidden_size=rnn_hidden_size,\\n            batch_first=batch_first,\\n        )\\n        self.fc = nn.Linear(in_features=rnn_hidden_size, out_features=char_vocab_size)\\n        self._dropout_p = dropout_p\\n\\n    def forward(self, x_in, nationality_index, apply_softmax=False):\\n        x_embedded = self.char_emb(x_in)\\n        nationality_embedded = self.nation_emb(nationality_index).unsqueeze(0)\\n        y_out, _ = self.rnn(x_embedded, nationality_embedded)\\n        batch_size, seq_size, feat_size = y_out.shape\\n        y_out = y_out.contiguous().view(batch_size * seq_size, feat_size)\\n        y_out = self.fc(F.dropout(y_out, p=self._dropout_p))\\n        if apply_softmax:\\n            y_out = F.softmax(y_out, dim=1)\\n        new_feat_size = y_out.shape[-1]\\n        y_out = y_out.view(batch_size, seq_size, new_feat_size)\\n        return y_out\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "class SurnameGenerationModel2(nn.Module):\n",
    "    def __init__(\n",
    "        self,\n",
    "        char_embedding_size,\n",
    "        char_vocab_size,\n",
    "        num_nationalities,\n",
    "        rnn_hidden_size,\n",
    "        batch_first=True,\n",
    "        padding_idx=0,\n",
    "        dropout_p=0.5,\n",
    "    ):\n",
    "        super(SurnameGenerationModel2, self).__init__()\n",
    "        self.char_emb = nn.Embedding(\n",
    "            num_embeddings=char_vocab_size,\n",
    "            embedding_dim=char_embedding_size,\n",
    "            padding_idx=padding_idx,\n",
    "        )\n",
    "        self.nation_emb = nn.Embedding(\n",
    "            num_embeddings=num_nationalities, embedding_dim=rnn_hidden_size\n",
    "        )\n",
    "        self.rnn = nn.GRU(\n",
    "            input_size=char_embedding_size,\n",
    "            hidden_size=rnn_hidden_size,\n",
    "            batch_first=batch_first,\n",
    "        )\n",
    "        self.fc = nn.Linear(in_features=rnn_hidden_size, out_features=char_vocab_size)\n",
    "        self._dropout_p = dropout_p\n",
    "\n",
    "    def forward(self, x_in, nationality_index, apply_softmax=False):\n",
    "        x_embedded = self.char_emb(x_in)\n",
    "        nationality_embedded = self.nation_emb(nationality_index).unsqueeze(0)\n",
    "        y_out, _ = self.rnn(x_embedded, nationality_embedded)\n",
    "        batch_size, seq_size, feat_size = y_out.shape\n",
    "        y_out = y_out.contiguous().view(batch_size * seq_size, feat_size)\n",
    "        y_out = self.fc(F.dropout(y_out, p=self._dropout_p))\n",
    "        if apply_softmax:\n",
    "            y_out = F.softmax(y_out, dim=1)\n",
    "        new_feat_size = y_out.shape[-1]\n",
    "        y_out = y_out.view(batch_size, seq_size, new_feat_size)\n",
    "        return y_out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "983d793d",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 82;\n",
       "                var nbb_unformatted_code = \"def sample_from_model(\\n    model, vectorizer, nationalities, sample_size=20, temperature=1.0\\n):\\n    num_samples = len(nationalities)\\n    begin_seq_index = [\\n        vectorizer.char_vocab.begin_seq_index for _ in range(num_samples)\\n    ]\\n    begin_seq_index = torch.tensor(begin_seq_index, dtype=torch.int64).unsqueeze(dim=1)\\n    indices = [begin_seq_index]\\n    nationality_indices = torch.tensor(nationalities, dtype=torch.int64).unsqueeze(\\n        dim=0\\n    )\\n    h_t = model.nation_emb(nationality_indices)\\n\\n    for time_step in range(sample_size):\\n        x_t = indices[time_step]\\n        x_emb_t = model.char_emb(x_t)\\n        rnn_out_t, h_t = model.rnn(x_emb_t, h_t)\\n        prediction_vector = model.fc(rnn_out_t.squeeze(dim=1))\\n        probability_vector = F.softmax(prediction_vector / temperature, dim=1)\\n        indices.append(torch.multinomial(probability_vector, num_samples=1))\\n    indices = torch.stack(indices).squeeze().permute(1, 0)\\n    return indices\";\n",
       "                var nbb_formatted_code = \"def sample_from_model(\\n    model, vectorizer, nationalities, sample_size=20, temperature=1.0\\n):\\n    num_samples = len(nationalities)\\n    begin_seq_index = [\\n        vectorizer.char_vocab.begin_seq_index for _ in range(num_samples)\\n    ]\\n    begin_seq_index = torch.tensor(begin_seq_index, dtype=torch.int64).unsqueeze(dim=1)\\n    indices = [begin_seq_index]\\n    nationality_indices = torch.tensor(nationalities, dtype=torch.int64).unsqueeze(\\n        dim=0\\n    )\\n    h_t = model.nation_emb(nationality_indices)\\n\\n    for time_step in range(sample_size):\\n        x_t = indices[time_step]\\n        x_emb_t = model.char_emb(x_t)\\n        rnn_out_t, h_t = model.rnn(x_emb_t, h_t)\\n        prediction_vector = model.fc(rnn_out_t.squeeze(dim=1))\\n        probability_vector = F.softmax(prediction_vector / temperature, dim=1)\\n        indices.append(torch.multinomial(probability_vector, num_samples=1))\\n    indices = torch.stack(indices).squeeze().permute(1, 0)\\n    return indices\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def sample_from_model(\n",
    "    model, vectorizer, nationalities, sample_size=20, temperature=1.0\n",
    "):\n",
    "    num_samples = len(nationalities)\n",
    "    begin_seq_index = [\n",
    "        vectorizer.char_vocab.begin_seq_index for _ in range(num_samples)\n",
    "    ]\n",
    "    begin_seq_index = torch.tensor(begin_seq_index, dtype=torch.int64).unsqueeze(dim=1)\n",
    "    indices = [begin_seq_index]\n",
    "    nationality_indices = torch.tensor(nationalities, dtype=torch.int64).unsqueeze(\n",
    "        dim=0\n",
    "    )\n",
    "    h_t = model.nation_emb(nationality_indices)\n",
    "\n",
    "    for time_step in range(sample_size):\n",
    "        x_t = indices[time_step]\n",
    "        x_emb_t = model.char_emb(x_t)\n",
    "        rnn_out_t, h_t = model.rnn(x_emb_t, h_t)\n",
    "        prediction_vector = model.fc(rnn_out_t.squeeze(dim=1))\n",
    "        probability_vector = F.softmax(prediction_vector / temperature, dim=1)\n",
    "        indices.append(torch.multinomial(probability_vector, num_samples=1))\n",
    "    indices = torch.stack(indices).squeeze().permute(1, 0)\n",
    "    return indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "c2b1273e",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using CUDA: False\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 87;\n",
       "                var nbb_unformatted_code = \"args = Namespace(\\n    # Data and path information\\n    surname_csv=\\\"../data/surnames/surnames_with_splits.csv\\\",\\n    vectorizer_file=\\\"vectorizer.json\\\",\\n    model_state_file=\\\"model.pth\\\",\\n    save_dir=\\\"models/chapter07/model2_conditioned_surname_generation\\\",\\n    # Model hyper parameter\\n    char_embedding_size=32,\\n    rnn_hidden_size=32,\\n    # Training hyper parameter\\n    num_epochs=100,\\n    learning_rate=0.001,\\n    batch_size=128,\\n    seed=1337,\\n    early_stopping_criteria=5,\\n    # Runtime hyper parameter\\n    cuda=True,\\n    catch_keyboard_interrupt=True,\\n    reload_from_files=False,\\n    expand_filepaths_to_save_dir=True,\\n)\\n\\nif not torch.cuda.is_available():\\n    args.cuda = False\\n\\nargs.device = torch.device(\\\"cuda\\\" if args.cuda else \\\"cpu\\\")\\n\\nprint(\\\"Using CUDA: {}\\\".format(args.cuda))\\n\\n\\nif args.expand_filepaths_to_save_dir:\\n    args.vectorizer_file = os.path.join(args.save_dir, args.vectorizer_file)\\n\\n    args.model_state_file = os.path.join(args.save_dir, args.model_state_file)\\n\\n# Set seed for reproducibility\\nutils.set_seed_everywhere(args.seed, args.cuda)\\n\\n# handle dirs\\nutils.handle_dirs(args.save_dir)\";\n",
       "                var nbb_formatted_code = \"args = Namespace(\\n    # Data and path information\\n    surname_csv=\\\"../data/surnames/surnames_with_splits.csv\\\",\\n    vectorizer_file=\\\"vectorizer.json\\\",\\n    model_state_file=\\\"model.pth\\\",\\n    save_dir=\\\"models/chapter07/model2_conditioned_surname_generation\\\",\\n    # Model hyper parameter\\n    char_embedding_size=32,\\n    rnn_hidden_size=32,\\n    # Training hyper parameter\\n    num_epochs=100,\\n    learning_rate=0.001,\\n    batch_size=128,\\n    seed=1337,\\n    early_stopping_criteria=5,\\n    # Runtime hyper parameter\\n    cuda=True,\\n    catch_keyboard_interrupt=True,\\n    reload_from_files=False,\\n    expand_filepaths_to_save_dir=True,\\n)\\n\\nif not torch.cuda.is_available():\\n    args.cuda = False\\n\\nargs.device = torch.device(\\\"cuda\\\" if args.cuda else \\\"cpu\\\")\\n\\nprint(\\\"Using CUDA: {}\\\".format(args.cuda))\\n\\n\\nif args.expand_filepaths_to_save_dir:\\n    args.vectorizer_file = os.path.join(args.save_dir, args.vectorizer_file)\\n\\n    args.model_state_file = os.path.join(args.save_dir, args.model_state_file)\\n\\n# Set seed for reproducibility\\nutils.set_seed_everywhere(args.seed, args.cuda)\\n\\n# handle dirs\\nutils.handle_dirs(args.save_dir)\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "args = Namespace(\n",
    "    # Data and path information\n",
    "    surname_csv=\"../data/surnames/surnames_with_splits.csv\",\n",
    "    vectorizer_file=\"vectorizer.json\",\n",
    "    model_state_file=\"model.pth\",\n",
    "    save_dir=\"models/chapter07/model2_conditioned_surname_generation\",\n",
    "    # Model hyper parameter\n",
    "    char_embedding_size=32,\n",
    "    rnn_hidden_size=32,\n",
    "    # Training hyper parameter\n",
    "    num_epochs=100,\n",
    "    learning_rate=0.001,\n",
    "    batch_size=128,\n",
    "    seed=1337,\n",
    "    early_stopping_criteria=5,\n",
    "    # Runtime hyper parameter\n",
    "    cuda=True,\n",
    "    catch_keyboard_interrupt=True,\n",
    "    reload_from_files=False,\n",
    "    expand_filepaths_to_save_dir=True,\n",
    ")\n",
    "\n",
    "if not torch.cuda.is_available():\n",
    "    args.cuda = False\n",
    "\n",
    "args.device = torch.device(\"cuda\" if args.cuda else \"cpu\")\n",
    "\n",
    "print(\"Using CUDA: {}\".format(args.cuda))\n",
    "\n",
    "\n",
    "if args.expand_filepaths_to_save_dir:\n",
    "    args.vectorizer_file = os.path.join(args.save_dir, args.vectorizer_file)\n",
    "\n",
    "    args.model_state_file = os.path.join(args.save_dir, args.model_state_file)\n",
    "\n",
    "# Set seed for reproducibility\n",
    "utils.set_seed_everywhere(args.seed, args.cuda)\n",
    "\n",
    "# handle dirs\n",
    "utils.handle_dirs(args.save_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "826f9668",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SurnameGenerationModel2(\n",
      "  (char_emb): Embedding(88, 32, padding_idx=0)\n",
      "  (nation_emb): Embedding(18, 32)\n",
      "  (rnn): GRU(32, 32, batch_first=True)\n",
      "  (fc): Linear(in_features=32, out_features=88, bias=True)\n",
      ")\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 88;\n",
       "                var nbb_unformatted_code = \"dataset = SurnameDataset.load_dataset_and_make_vectorizer(args.surname_csv)\\ndataset.save_vectorizer(args.vectorizer_file)\\nvectorizer = dataset.get_vectorizer()\\nmodel = SurnameGenerationModel2(\\n    char_embedding_size=args.char_embedding_size,\\n    char_vocab_size=len(vectorizer.char_vocab),\\n    num_nationalities=len(vectorizer.nationality_vocab),\\n    rnn_hidden_size=args.rnn_hidden_size,\\n    padding_idx=vectorizer.char_vocab.mask_index,\\n    dropout_p=0.5,\\n)\\nprint(model)\\nmodel = model.to(args.device)\\noptimizer = optim.Adam(model.parameters(), lr=args.learning_rate)\\nscheduler = optim.lr_scheduler.ReduceLROnPlateau(\\n    optimizer=optimizer, mode=\\\"min\\\", factor=0.5, patience=1\\n)\";\n",
       "                var nbb_formatted_code = \"dataset = SurnameDataset.load_dataset_and_make_vectorizer(args.surname_csv)\\ndataset.save_vectorizer(args.vectorizer_file)\\nvectorizer = dataset.get_vectorizer()\\nmodel = SurnameGenerationModel2(\\n    char_embedding_size=args.char_embedding_size,\\n    char_vocab_size=len(vectorizer.char_vocab),\\n    num_nationalities=len(vectorizer.nationality_vocab),\\n    rnn_hidden_size=args.rnn_hidden_size,\\n    padding_idx=vectorizer.char_vocab.mask_index,\\n    dropout_p=0.5,\\n)\\nprint(model)\\nmodel = model.to(args.device)\\noptimizer = optim.Adam(model.parameters(), lr=args.learning_rate)\\nscheduler = optim.lr_scheduler.ReduceLROnPlateau(\\n    optimizer=optimizer, mode=\\\"min\\\", factor=0.5, patience=1\\n)\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dataset = SurnameDataset.load_dataset_and_make_vectorizer(args.surname_csv)\n",
    "dataset.save_vectorizer(args.vectorizer_file)\n",
    "vectorizer = dataset.get_vectorizer()\n",
    "model = SurnameGenerationModel2(\n",
    "    char_embedding_size=args.char_embedding_size,\n",
    "    char_vocab_size=len(vectorizer.char_vocab),\n",
    "    num_nationalities=len(vectorizer.nationality_vocab),\n",
    "    rnn_hidden_size=args.rnn_hidden_size,\n",
    "    padding_idx=vectorizer.char_vocab.mask_index,\n",
    "    dropout_p=0.5,\n",
    ")\n",
    "print(model)\n",
    "model = model.to(args.device)\n",
    "optimizer = optim.Adam(model.parameters(), lr=args.learning_rate)\n",
    "scheduler = optim.lr_scheduler.ReduceLROnPlateau(\n",
    "    optimizer=optimizer, mode=\"min\", factor=0.5, patience=1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "3334e732",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c1d2b5913810411b8e66060db5a01d20",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Training Routine:   0%|          | 0/100 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b367d0cbd62e468e80ca6088da8a3191",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "split=train:   0%|          | 0/60 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7223eaf098e849f7b439a556d63f3ee5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "split=val:   0%|          | 0/12 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "============ Split=train, Size=7680 ============\n",
      "============ Split=val, Size=1640 ============\n",
      "--------------- 0th Epoch Stats---------------\n",
      "Training Loss=4.047944211959838, Training Accuracy=8.218097564371824\n",
      "Validation Loss=3.4041937192281084, Validation Accuracy=13.912713462676289.\n",
      "------------------------------------------------------------\n",
      "--------------- 10th Epoch Stats---------------\n",
      "Training Loss=2.5890356620152777, Training Accuracy=25.305533111015546\n",
      "Validation Loss=2.5892768700917563, Validation Accuracy=25.5085011718881.\n",
      "------------------------------------------------------------\n",
      "--------------- 20th Epoch Stats---------------\n",
      "Training Loss=2.4898185809453324, Training Accuracy=27.34181131042023\n",
      "Validation Loss=2.505171517531077, Validation Accuracy=26.880222539745002.\n",
      "------------------------------------------------------------\n",
      "--------------- 30th Epoch Stats---------------\n",
      "Training Loss=2.4452757000923158, Training Accuracy=28.281319997580518\n",
      "Validation Loss=2.458956797917684, Validation Accuracy=28.264110522554976.\n",
      "------------------------------------------------------------\n",
      "--------------- 40th Epoch Stats---------------\n",
      "Training Loss=2.4187005122502634, Training Accuracy=28.81193051592007\n",
      "Validation Loss=2.444360812505086, Validation Accuracy=28.116861848499795.\n",
      "------------------------------------------------------------\n",
      "--------------- 50th Epoch Stats---------------\n",
      "Training Loss=2.4064959009488422, Training Accuracy=28.99100258386494\n",
      "Validation Loss=2.4272076487541203, Validation Accuracy=28.445602291852243.\n",
      "------------------------------------------------------------\n",
      "--------------- 60th Epoch Stats---------------\n",
      "Training Loss=2.4033911108970636, Training Accuracy=29.286216522507644\n",
      "Validation Loss=2.4229062398274746, Validation Accuracy=28.273358772107187.\n",
      "------------------------------------------------------------\n",
      "--------------- 70th Epoch Stats---------------\n",
      "Training Loss=2.4047314206759127, Training Accuracy=29.042405701149377\n",
      "Validation Loss=2.42384002606074, Validation Accuracy=28.919822201408177.\n",
      "------------------------------------------------------------\n",
      "--------------- 80th Epoch Stats---------------\n",
      "Training Loss=2.4035119056701655, Training Accuracy=29.261736023261246\n",
      "Validation Loss=2.4408104817072553, Validation Accuracy=28.38154519801275.\n",
      "------------------------------------------------------------\n",
      "--------------- 90th Epoch Stats---------------\n",
      "Training Loss=2.4042801817258193, Training Accuracy=29.06030475187459\n",
      "Validation Loss=2.43443904320399, Validation Accuracy=28.83570108857413.\n",
      "------------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 89;\n",
       "                var nbb_unformatted_code = \"mask_index = vectorizer.char_vocab.mask_index\\ntrain_state = utils.make_train_state(args)\\nepoch_bar = notebook.tqdm(desc=\\\"Training Routine\\\", total=args.num_epochs, position=0)\\ndataset.set_split(\\\"train\\\")\\ntrain_bar = notebook.tqdm(\\n    desc=\\\"split=train\\\",\\n    total=dataset.get_num_batches(args.batch_size),\\n    position=1,\\n    leave=True,\\n)\\ndataset.set_split(\\\"val\\\")\\nval_bar = notebook.tqdm(\\n    desc=\\\"split=val\\\",\\n    total=dataset.get_num_batches(args.batch_size),\\n    position=1,\\n    leave=True,\\n)\\n\\nfor epoch_index in range(args.num_epochs):\\n    train_state[\\\"epoch_index\\\"] = epoch_index\\n    # Iterate Over Training Dataset\\n    # Setup: Batch Generator, set loss & acc to 0, set train mode on\\n    dataset.set_split(\\\"train\\\")\\n    if epoch_index == 0:\\n        print(\\n            f\\\"============ Split={dataset._train_split}, Size={len(dataset)} ============\\\"\\n        )\\n    batch_generator = utils.generate_batches(\\n        dataset, batch_size=args.batch_size, device=args.device\\n    )\\n    training_running_loss, training_running_acc = 0.0, 0.0\\n    model.train()\\n\\n    for batch_index, batch_dict in enumerate(batch_generator):\\n        # 5 Step Training Routine\\n\\n        # Step 1. Zero the Gradients\\n        optimizer.zero_grad()\\n\\n        # Step 2. Compute the gradients\\n        y_pred = model(\\n            x_in=batch_dict[\\\"x_data\\\"], nationality_index=batch_dict[\\\"class_index\\\"]\\n        )\\n\\n        # Step 3. Compute the Output\\n        loss = sequence_loss(y_pred, batch_dict[\\\"y_target\\\"], mask_index=mask_index)\\n\\n        # Step 4. Use loss to produce gradients\\n        loss.backward()\\n\\n        # Step 5. Use Optimizer to take gradient step\\n        optimizer.step()\\n\\n        # Compute the running loss and accuracy\\n        loss_batch = loss.item()\\n        training_running_loss += (loss_batch - training_running_loss) / (\\n            batch_index + 1\\n        )\\n        acc_batch = compute_accuracy(y_pred, batch_dict[\\\"y_target\\\"], mask_index)\\n        training_running_acc += (acc_batch - training_running_acc) / (batch_index + 1)\\n\\n        # Update the bar\\n        train_bar.set_postfix(\\n            loss=training_running_loss, acc=training_running_acc, epoch=epoch_index\\n        )\\n        train_bar.update()\\n    train_state[\\\"train_loss\\\"].append(training_running_loss)\\n    train_state[\\\"train_acc\\\"].append(training_running_acc)\\n\\n    # Iterate Over Val Dataset\\n    # Setup: Batch Generator, set loss and acc to 0, set eval mode on\\n    dataset.set_split(\\\"val\\\")\\n    val_running_loss, val_running_acc = 0.0, 0.0\\n    if len(dataset) > 0:\\n        if epoch_index == 0:\\n            print(\\n                f\\\"============ Split={dataset._train_split}, Size={len(dataset)} ============\\\"\\n            )\\n        batch_generator = utils.generate_batches(\\n            dataset, batch_size=args.batch_size, device=args.device\\n        )\\n        model.eval()\\n\\n        for batch_index, batch_dict in enumerate(batch_generator):\\n            # Step 1. Compute the Output\\n            y_pred = model(\\n                x_in=batch_dict[\\\"x_data\\\"], nationality_index=batch_dict[\\\"class_index\\\"]\\n            )\\n\\n            # Step 2. Compute the loss\\n            loss = sequence_loss(y_pred, batch_dict[\\\"y_target\\\"], mask_index)\\n            loss_batch = loss.item()\\n            val_running_loss += (loss_batch - val_running_loss) / (batch_index + 1)\\n\\n            # Step 3. Compute the accuracy\\n            acc_batch = compute_accuracy(y_pred, batch_dict[\\\"y_target\\\"], mask_index)\\n            val_running_acc += (acc_batch - val_running_acc) / (batch_index + 1)\\n            val_bar.set_postfix(\\n                loss=val_running_loss, acc=val_running_acc, epoch=epoch_index\\n            )\\n            val_bar.update()\\n        train_state[\\\"val_loss\\\"].append(val_running_loss)\\n        train_state[\\\"val_acc\\\"].append(val_running_acc)\\n        scheduler.step(train_state[\\\"val_loss\\\"][-1])\\n    else:\\n        if epoch_index == 0:\\n            print(f\\\"============ Skipping Validation Pass ============\\\")\\n        train_state[\\\"val_loss\\\"].append(val_running_loss)\\n        train_state[\\\"val_acc\\\"].append(val_running_acc)\\n        scheduler.step(train_state[\\\"train_loss\\\"][-1])\\n    train_state = utils.update_train_state(\\n        args=args, model=model, train_state=train_state\\n    )\\n\\n    train_bar.n, val_bar.n = 0, 0\\n    epoch_bar.update()\\n\\n    if train_state[\\\"stop_early\\\"]:\\n        print(\\\"Stopping early....\\\")\\n        break\\n\\n    if epoch_index % 10 == 0:\\n        print(\\n            f\\\"--------------- {epoch_index}th Epoch Stats---------------\\\\n\\\"\\n            f\\\"Training Loss={training_running_loss}, \\\"\\n            f\\\"Training Accuracy={training_running_acc}\\\\n\\\"\\n            f\\\"Validation Loss={val_running_loss}, \\\"\\n            f\\\"Validation Accuracy={val_running_acc}.\\\\n\\\"\\n            \\\"------------------------------------------------------------\\\"\\n        )\";\n",
       "                var nbb_formatted_code = \"mask_index = vectorizer.char_vocab.mask_index\\ntrain_state = utils.make_train_state(args)\\nepoch_bar = notebook.tqdm(desc=\\\"Training Routine\\\", total=args.num_epochs, position=0)\\ndataset.set_split(\\\"train\\\")\\ntrain_bar = notebook.tqdm(\\n    desc=\\\"split=train\\\",\\n    total=dataset.get_num_batches(args.batch_size),\\n    position=1,\\n    leave=True,\\n)\\ndataset.set_split(\\\"val\\\")\\nval_bar = notebook.tqdm(\\n    desc=\\\"split=val\\\",\\n    total=dataset.get_num_batches(args.batch_size),\\n    position=1,\\n    leave=True,\\n)\\n\\nfor epoch_index in range(args.num_epochs):\\n    train_state[\\\"epoch_index\\\"] = epoch_index\\n    # Iterate Over Training Dataset\\n    # Setup: Batch Generator, set loss & acc to 0, set train mode on\\n    dataset.set_split(\\\"train\\\")\\n    if epoch_index == 0:\\n        print(\\n            f\\\"============ Split={dataset._train_split}, Size={len(dataset)} ============\\\"\\n        )\\n    batch_generator = utils.generate_batches(\\n        dataset, batch_size=args.batch_size, device=args.device\\n    )\\n    training_running_loss, training_running_acc = 0.0, 0.0\\n    model.train()\\n\\n    for batch_index, batch_dict in enumerate(batch_generator):\\n        # 5 Step Training Routine\\n\\n        # Step 1. Zero the Gradients\\n        optimizer.zero_grad()\\n\\n        # Step 2. Compute the gradients\\n        y_pred = model(\\n            x_in=batch_dict[\\\"x_data\\\"], nationality_index=batch_dict[\\\"class_index\\\"]\\n        )\\n\\n        # Step 3. Compute the Output\\n        loss = sequence_loss(y_pred, batch_dict[\\\"y_target\\\"], mask_index=mask_index)\\n\\n        # Step 4. Use loss to produce gradients\\n        loss.backward()\\n\\n        # Step 5. Use Optimizer to take gradient step\\n        optimizer.step()\\n\\n        # Compute the running loss and accuracy\\n        loss_batch = loss.item()\\n        training_running_loss += (loss_batch - training_running_loss) / (\\n            batch_index + 1\\n        )\\n        acc_batch = compute_accuracy(y_pred, batch_dict[\\\"y_target\\\"], mask_index)\\n        training_running_acc += (acc_batch - training_running_acc) / (batch_index + 1)\\n\\n        # Update the bar\\n        train_bar.set_postfix(\\n            loss=training_running_loss, acc=training_running_acc, epoch=epoch_index\\n        )\\n        train_bar.update()\\n    train_state[\\\"train_loss\\\"].append(training_running_loss)\\n    train_state[\\\"train_acc\\\"].append(training_running_acc)\\n\\n    # Iterate Over Val Dataset\\n    # Setup: Batch Generator, set loss and acc to 0, set eval mode on\\n    dataset.set_split(\\\"val\\\")\\n    val_running_loss, val_running_acc = 0.0, 0.0\\n    if len(dataset) > 0:\\n        if epoch_index == 0:\\n            print(\\n                f\\\"============ Split={dataset._train_split}, Size={len(dataset)} ============\\\"\\n            )\\n        batch_generator = utils.generate_batches(\\n            dataset, batch_size=args.batch_size, device=args.device\\n        )\\n        model.eval()\\n\\n        for batch_index, batch_dict in enumerate(batch_generator):\\n            # Step 1. Compute the Output\\n            y_pred = model(\\n                x_in=batch_dict[\\\"x_data\\\"], nationality_index=batch_dict[\\\"class_index\\\"]\\n            )\\n\\n            # Step 2. Compute the loss\\n            loss = sequence_loss(y_pred, batch_dict[\\\"y_target\\\"], mask_index)\\n            loss_batch = loss.item()\\n            val_running_loss += (loss_batch - val_running_loss) / (batch_index + 1)\\n\\n            # Step 3. Compute the accuracy\\n            acc_batch = compute_accuracy(y_pred, batch_dict[\\\"y_target\\\"], mask_index)\\n            val_running_acc += (acc_batch - val_running_acc) / (batch_index + 1)\\n            val_bar.set_postfix(\\n                loss=val_running_loss, acc=val_running_acc, epoch=epoch_index\\n            )\\n            val_bar.update()\\n        train_state[\\\"val_loss\\\"].append(val_running_loss)\\n        train_state[\\\"val_acc\\\"].append(val_running_acc)\\n        scheduler.step(train_state[\\\"val_loss\\\"][-1])\\n    else:\\n        if epoch_index == 0:\\n            print(f\\\"============ Skipping Validation Pass ============\\\")\\n        train_state[\\\"val_loss\\\"].append(val_running_loss)\\n        train_state[\\\"val_acc\\\"].append(val_running_acc)\\n        scheduler.step(train_state[\\\"train_loss\\\"][-1])\\n    train_state = utils.update_train_state(\\n        args=args, model=model, train_state=train_state\\n    )\\n\\n    train_bar.n, val_bar.n = 0, 0\\n    epoch_bar.update()\\n\\n    if train_state[\\\"stop_early\\\"]:\\n        print(\\\"Stopping early....\\\")\\n        break\\n\\n    if epoch_index % 10 == 0:\\n        print(\\n            f\\\"--------------- {epoch_index}th Epoch Stats---------------\\\\n\\\"\\n            f\\\"Training Loss={training_running_loss}, \\\"\\n            f\\\"Training Accuracy={training_running_acc}\\\\n\\\"\\n            f\\\"Validation Loss={val_running_loss}, \\\"\\n            f\\\"Validation Accuracy={val_running_acc}.\\\\n\\\"\\n            \\\"------------------------------------------------------------\\\"\\n        )\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mask_index = vectorizer.char_vocab.mask_index\n",
    "train_state = utils.make_train_state(args)\n",
    "epoch_bar = notebook.tqdm(desc=\"Training Routine\", total=args.num_epochs, position=0)\n",
    "dataset.set_split(\"train\")\n",
    "train_bar = notebook.tqdm(\n",
    "    desc=\"split=train\",\n",
    "    total=dataset.get_num_batches(args.batch_size),\n",
    "    position=1,\n",
    "    leave=True,\n",
    ")\n",
    "dataset.set_split(\"val\")\n",
    "val_bar = notebook.tqdm(\n",
    "    desc=\"split=val\",\n",
    "    total=dataset.get_num_batches(args.batch_size),\n",
    "    position=1,\n",
    "    leave=True,\n",
    ")\n",
    "\n",
    "for epoch_index in range(args.num_epochs):\n",
    "    train_state[\"epoch_index\"] = epoch_index\n",
    "    # Iterate Over Training Dataset\n",
    "    # Setup: Batch Generator, set loss & acc to 0, set train mode on\n",
    "    dataset.set_split(\"train\")\n",
    "    if epoch_index == 0:\n",
    "        print(\n",
    "            f\"============ Split={dataset._train_split}, Size={len(dataset)} ============\"\n",
    "        )\n",
    "    batch_generator = utils.generate_batches(\n",
    "        dataset, batch_size=args.batch_size, device=args.device\n",
    "    )\n",
    "    training_running_loss, training_running_acc = 0.0, 0.0\n",
    "    model.train()\n",
    "\n",
    "    for batch_index, batch_dict in enumerate(batch_generator):\n",
    "        # 5 Step Training Routine\n",
    "\n",
    "        # Step 1. Zero the Gradients\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        # Step 2. Compute the gradients\n",
    "        y_pred = model(\n",
    "            x_in=batch_dict[\"x_data\"], nationality_index=batch_dict[\"class_index\"]\n",
    "        )\n",
    "\n",
    "        # Step 3. Compute the Output\n",
    "        loss = sequence_loss(y_pred, batch_dict[\"y_target\"], mask_index=mask_index)\n",
    "\n",
    "        # Step 4. Use loss to produce gradients\n",
    "        loss.backward()\n",
    "\n",
    "        # Step 5. Use Optimizer to take gradient step\n",
    "        optimizer.step()\n",
    "\n",
    "        # Compute the running loss and accuracy\n",
    "        loss_batch = loss.item()\n",
    "        training_running_loss += (loss_batch - training_running_loss) / (\n",
    "            batch_index + 1\n",
    "        )\n",
    "        acc_batch = compute_accuracy(y_pred, batch_dict[\"y_target\"], mask_index)\n",
    "        training_running_acc += (acc_batch - training_running_acc) / (batch_index + 1)\n",
    "\n",
    "        # Update the bar\n",
    "        train_bar.set_postfix(\n",
    "            loss=training_running_loss, acc=training_running_acc, epoch=epoch_index\n",
    "        )\n",
    "        train_bar.update()\n",
    "    train_state[\"train_loss\"].append(training_running_loss)\n",
    "    train_state[\"train_acc\"].append(training_running_acc)\n",
    "\n",
    "    # Iterate Over Val Dataset\n",
    "    # Setup: Batch Generator, set loss and acc to 0, set eval mode on\n",
    "    dataset.set_split(\"val\")\n",
    "    val_running_loss, val_running_acc = 0.0, 0.0\n",
    "    if len(dataset) > 0:\n",
    "        if epoch_index == 0:\n",
    "            print(\n",
    "                f\"============ Split={dataset._train_split}, Size={len(dataset)} ============\"\n",
    "            )\n",
    "        batch_generator = utils.generate_batches(\n",
    "            dataset, batch_size=args.batch_size, device=args.device\n",
    "        )\n",
    "        model.eval()\n",
    "\n",
    "        for batch_index, batch_dict in enumerate(batch_generator):\n",
    "            # Step 1. Compute the Output\n",
    "            y_pred = model(\n",
    "                x_in=batch_dict[\"x_data\"], nationality_index=batch_dict[\"class_index\"]\n",
    "            )\n",
    "\n",
    "            # Step 2. Compute the loss\n",
    "            loss = sequence_loss(y_pred, batch_dict[\"y_target\"], mask_index)\n",
    "            loss_batch = loss.item()\n",
    "            val_running_loss += (loss_batch - val_running_loss) / (batch_index + 1)\n",
    "\n",
    "            # Step 3. Compute the accuracy\n",
    "            acc_batch = compute_accuracy(y_pred, batch_dict[\"y_target\"], mask_index)\n",
    "            val_running_acc += (acc_batch - val_running_acc) / (batch_index + 1)\n",
    "            val_bar.set_postfix(\n",
    "                loss=val_running_loss, acc=val_running_acc, epoch=epoch_index\n",
    "            )\n",
    "            val_bar.update()\n",
    "        train_state[\"val_loss\"].append(val_running_loss)\n",
    "        train_state[\"val_acc\"].append(val_running_acc)\n",
    "        scheduler.step(train_state[\"val_loss\"][-1])\n",
    "    else:\n",
    "        if epoch_index == 0:\n",
    "            print(f\"============ Skipping Validation Pass ============\")\n",
    "        train_state[\"val_loss\"].append(val_running_loss)\n",
    "        train_state[\"val_acc\"].append(val_running_acc)\n",
    "        scheduler.step(train_state[\"train_loss\"][-1])\n",
    "    train_state = utils.update_train_state(\n",
    "        args=args, model=model, train_state=train_state\n",
    "    )\n",
    "\n",
    "    train_bar.n, val_bar.n = 0, 0\n",
    "    epoch_bar.update()\n",
    "\n",
    "    if train_state[\"stop_early\"]:\n",
    "        print(\"Stopping early....\")\n",
    "        break\n",
    "\n",
    "    if epoch_index % 10 == 0:\n",
    "        print(\n",
    "            f\"--------------- {epoch_index}th Epoch Stats---------------\\n\"\n",
    "            f\"Training Loss={training_running_loss}, \"\n",
    "            f\"Training Accuracy={training_running_acc}\\n\"\n",
    "            f\"Validation Loss={val_running_loss}, \"\n",
    "            f\"Validation Accuracy={val_running_acc}.\\n\"\n",
    "            \"------------------------------------------------------------\"\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "237c66b0",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 90;\n",
       "                var nbb_unformatted_code = \"model.load_state_dict(torch.load(train_state[\\\"model_filename\\\"]))\\n\\nmodel = model.to(args.device)\\n\\ndataset.set_split(\\\"test\\\")\\n\\nbatch_generator = utils.generate_batches(\\n    dataset, batch_size=args.batch_size, device=args.device\\n)\\nrunning_acc, running_loss = 0, 0\\nmodel.eval()\\n\\nfor batch_index, batch_dict in enumerate(batch_generator):\\n    y_pred = model(\\n        x_in=batch_dict[\\\"x_data\\\"], nationality_index=batch_dict[\\\"class_index\\\"]\\n    )\\n    loss = sequence_loss(y_pred, batch_dict[\\\"y_target\\\"], mask_index)\\n    running_loss += (loss.item() - running_loss) / (batch_index + 1)\\n    acc_batch = compute_accuracy(y_pred, batch_dict[\\\"y_target\\\"], mask_index)\\n    running_acc += (acc_batch - running_acc) / (batch_index + 1)\\ntrain_state[\\\"test_loss\\\"] = running_loss\\ntrain_state[\\\"test_acc\\\"] = running_acc\";\n",
       "                var nbb_formatted_code = \"model.load_state_dict(torch.load(train_state[\\\"model_filename\\\"]))\\n\\nmodel = model.to(args.device)\\n\\ndataset.set_split(\\\"test\\\")\\n\\nbatch_generator = utils.generate_batches(\\n    dataset, batch_size=args.batch_size, device=args.device\\n)\\nrunning_acc, running_loss = 0, 0\\nmodel.eval()\\n\\nfor batch_index, batch_dict in enumerate(batch_generator):\\n    y_pred = model(\\n        x_in=batch_dict[\\\"x_data\\\"], nationality_index=batch_dict[\\\"class_index\\\"]\\n    )\\n    loss = sequence_loss(y_pred, batch_dict[\\\"y_target\\\"], mask_index)\\n    running_loss += (loss.item() - running_loss) / (batch_index + 1)\\n    acc_batch = compute_accuracy(y_pred, batch_dict[\\\"y_target\\\"], mask_index)\\n    running_acc += (acc_batch - running_acc) / (batch_index + 1)\\ntrain_state[\\\"test_loss\\\"] = running_loss\\ntrain_state[\\\"test_acc\\\"] = running_acc\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model.load_state_dict(torch.load(train_state[\"model_filename\"]))\n",
    "\n",
    "model = model.to(args.device)\n",
    "\n",
    "dataset.set_split(\"test\")\n",
    "\n",
    "batch_generator = utils.generate_batches(\n",
    "    dataset, batch_size=args.batch_size, device=args.device\n",
    ")\n",
    "running_acc, running_loss = 0, 0\n",
    "model.eval()\n",
    "\n",
    "for batch_index, batch_dict in enumerate(batch_generator):\n",
    "    y_pred = model(\n",
    "        x_in=batch_dict[\"x_data\"], nationality_index=batch_dict[\"class_index\"]\n",
    "    )\n",
    "    loss = sequence_loss(y_pred, batch_dict[\"y_target\"], mask_index)\n",
    "    running_loss += (loss.item() - running_loss) / (batch_index + 1)\n",
    "    acc_batch = compute_accuracy(y_pred, batch_dict[\"y_target\"], mask_index)\n",
    "    running_acc += (acc_batch - running_acc) / (batch_index + 1)\n",
    "train_state[\"test_loss\"] = running_loss\n",
    "train_state[\"test_acc\"] = running_acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "e6cd86c4",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 2.423119862874349;\n",
      "Test Accuracy: 28.841407426543437\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 91;\n",
       "                var nbb_unformatted_code = \"print(\\\"Test loss: {};\\\".format(train_state['test_loss']))\\nprint(\\\"Test Accuracy: {}\\\".format(train_state['test_acc']))\";\n",
       "                var nbb_formatted_code = \"print(\\\"Test loss: {};\\\".format(train_state[\\\"test_loss\\\"]))\\nprint(\\\"Test Accuracy: {}\\\".format(train_state[\\\"test_acc\\\"]))\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"Test loss: {};\".format(train_state[\"test_loss\"]))\n",
    "print(\"Test Accuracy: {}\".format(train_state[\"test_acc\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "32cde811",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sampled for Arabic\n",
      "- Maosso\n",
      "- Haata\n",
      "- Barijo\n",
      "Sampled for Chinese\n",
      "- Girge\n",
      "- Alno\n",
      "- Kia\n",
      "Sampled for Czech\n",
      "- Cyis\n",
      "- Doich\n",
      "- Oywars\n",
      "Sampled for Dutch\n",
      "- Camibter\n",
      "- Podtae\n",
      "- Nalle\n",
      "Sampled for English\n",
      "- Vaslan\n",
      "- Ardkan\n",
      "- Basche\n",
      "Sampled for French\n",
      "- Suel\n",
      "- Deresk\n",
      "- Snomin\n",
      "Sampled for German\n",
      "- Sarlrord\n",
      "- Trechlod\n",
      "- Kedrolnbee\n",
      "Sampled for Greek\n",
      "- Aridos\n",
      "- Pina\n",
      "- Hanlitara\n",
      "Sampled for Irish\n",
      "- Roive\n",
      "- Wurdowov\n",
      "- Sernafamo\n",
      "Sampled for Italian\n",
      "- Lagchura\n",
      "- Hur\n",
      "- Tolro\n",
      "Sampled for Japanese\n",
      "- Kadawia\n",
      "- Katrurato\n",
      "- Fabahali\n",
      "Sampled for Korean\n",
      "- Qar\n",
      "- Kar\n",
      "- Do\n",
      "Sampled for Polish\n",
      "- Keterchis\n",
      "- Tunis\n",
      "- Bimirshen\n",
      "Sampled for Portuguese\n",
      "- Mama\n",
      "- Sorspi\n",
      "- Bcuraen\n",
      "Sampled for Russian\n",
      "- Padolov\n",
      "- Banichan\n",
      "- Pamaziron\n",
      "Sampled for Scottish\n",
      "- Jorf\n",
      "- Katfl\n",
      "- Ratton\n",
      "Sampled for Spanish\n",
      "- Corros\n",
      "- Zzaely\n",
      "- Arteriet\n",
      "Sampled for Vietnamese\n",
      "- Tang\n",
      "- Rich\n",
      "- Loo\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 94;\n",
       "                var nbb_unformatted_code = \"model = model.cpu()\\nfor index in range(len(vectorizer.nationality_vocab)):\\n    nationality = vectorizer.nationality_vocab.lookup_index(index)\\n    print(f\\\"Sampled for {nationality}\\\")\\n    sampled_indices = sample_from_model(\\n        model, vectorizer, nationalities=[index] * 3, temperature=0.7\\n    )\\n    for sampled_surname in decode_samples(sampled_indices, vectorizer):\\n        print(\\\"- \\\" + sampled_surname)\";\n",
       "                var nbb_formatted_code = \"model = model.cpu()\\nfor index in range(len(vectorizer.nationality_vocab)):\\n    nationality = vectorizer.nationality_vocab.lookup_index(index)\\n    print(f\\\"Sampled for {nationality}\\\")\\n    sampled_indices = sample_from_model(\\n        model, vectorizer, nationalities=[index] * 3, temperature=0.7\\n    )\\n    for sampled_surname in decode_samples(sampled_indices, vectorizer):\\n        print(\\\"- \\\" + sampled_surname)\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = model.cpu()\n",
    "for index in range(len(vectorizer.nationality_vocab)):\n",
    "    nationality = vectorizer.nationality_vocab.lookup_index(index)\n",
    "    print(f\"Sampled for {nationality}\")\n",
    "    sampled_indices = sample_from_model(\n",
    "        model, vectorizer, nationalities=[index] * 3, temperature=0.7\n",
    "    )\n",
    "    for sampled_surname in decode_samples(sampled_indices, vectorizer):\n",
    "        print(\"- \" + sampled_surname)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Intermediate Sequence Modeling for Natural Language Processing",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}