2a_demo_empty.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MjTMLaX0Un22"
   },
   "source": [
    "# 2a - Sequence to Sequence Learning with Neural Networks\n",
    "\n",
    "## Introduction\n",
    "\n",
    "Let's remind ourselves of the general encoder-decoder model.\n",
    "\n",
    "![](assets/seq2seq1.png)\n",
    "\n",
    "We use our encoder (green) over the embedded source sequence (yellow) to create a context vector (red). We then use that context vector with the decoder (blue) and a linear layer (purple) to generate the target sentence.\n",
    "\n",
    "In the previous model, we used an multi-layered LSTM as the encoder and decoder.\n",
    "\n",
    "![](assets/seq2seq4.png)\n",
    "\n",
    "One downside of the previous model is that the decoder is trying to cram lots of information into the hidden states. Whilst decoding, the hidden state will need to contain information about the whole of the source sequence, as well as all of the tokens have been decoded so far. By alleviating some of this information compression, we can create a better model!\n",
    "\n",
    "We'll also be using a GRU (Gated Recurrent Unit) instead of an LSTM (Long Short-Term Memory). Why? Mainly because that's what they did in the paper (this paper also introduced GRUs) and also because we used LSTMs last time. To understand how GRUs (and LSTMs) differ from standard RNNS, check out [this](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) link. Is a GRU better than an LSTM? [Research](https://arxiv.org/abs/1412.3555) has shown they're pretty much the same, and both are better than standard RNNs. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jMIVwQyitAXd"
   },
   "source": [
    "## Import Libraries\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "GNnMugG1WKTm"
   },
   "outputs": [],
   "source": [
    "#!pip install -U spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Hp1WKED5WLNu"
   },
   "outputs": [],
   "source": [
    "#!pip install -U swifter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-11T00:08:06.239610Z",
     "iopub.status.busy": "2022-02-11T00:08:06.239416Z",
     "iopub.status.idle": "2022-02-11T00:08:07.074941Z",
     "shell.execute_reply": "2022-02-11T00:08:07.074532Z",
     "shell.execute_reply.started": "2022-02-11T00:08:06.239554Z"
    },
    "executionInfo": {
     "elapsed": 150,
     "status": "ok",
     "timestamp": 1644562950914,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "JiUqMdpCUn24",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2022-02-11T00:08:09.220242Z",
     "iopub.status.busy": "2022-02-11T00:08:09.220113Z",
     "iopub.status.idle": "2022-02-11T00:08:09.222472Z",
     "shell.execute_reply": "2022-02-11T00:08:09.222072Z",
     "shell.execute_reply.started": "2022-02-11T00:08:09.220227Z"
    },
    "executionInfo": {
     "elapsed": 15399,
     "status": "ok",
     "timestamp": 1644562969910,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "9uY_ZBxdtH_k",
    "outputId": "79b45cf6-8db1-4513-ae3b-ec37a8e423a7",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mounted at /content/drive\n"
     ]
    }
   ],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-11T00:08:09.804037Z",
     "iopub.status.busy": "2022-02-11T00:08:09.803889Z",
     "iopub.status.idle": "2022-02-11T00:08:09.806705Z",
     "shell.execute_reply": "2022-02-11T00:08:09.806095Z",
     "shell.execute_reply.started": "2022-02-11T00:08:09.804023Z"
    },
    "executionInfo": {
     "elapsed": 207,
     "status": "ok",
     "timestamp": 1644562972982,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "LLLu5aF_VkRw",
    "tags": []
   },
   "outputs": [],
   "source": [
    "#data_folder = Path('/home/harpreet/Insync/google_drive_shaannoor/Data/NLP')\n",
    "project_folder = Path('/content/drive/MyDrive/Research/NLP/pytorch-seq2seq')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2022-02-11T00:08:13.945685Z",
     "iopub.status.busy": "2022-02-11T00:08:13.945534Z",
     "iopub.status.idle": "2022-02-11T00:08:13.952300Z",
     "shell.execute_reply": "2022-02-11T00:08:13.951966Z",
     "shell.execute_reply.started": "2022-02-11T00:08:13.945672Z"
    },
    "executionInfo": {
     "elapsed": 204,
     "status": "ok",
     "timestamp": 1644562974311,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "PY9-guO2Un25",
    "outputId": "58c35621-c6ff-4c35-f57a-e713a1765748",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('0.11.0', '1.10.0+cu111', True, '2.2.4')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torchtext.__version__, torch.__version__, torch.cuda.is_available(), spacy.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wRVU8aBPUn26"
   },
   "source": [
    "# Set Seeds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-11T00:08:15.105519Z",
     "iopub.status.busy": "2022-02-11T00:08:15.105366Z",
     "iopub.status.idle": "2022-02-11T00:08:15.108928Z",
     "shell.execute_reply": "2022-02-11T00:08:15.108641Z",
     "shell.execute_reply.started": "2022-02-11T00:08:15.105505Z"
    },
    "executionInfo": {
     "elapsed": 205,
     "status": "ok",
     "timestamp": 1644562976461,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "vzCQltiuUn26",
    "tags": []
   },
   "outputs": [],
   "source": [
    "SEED = 1234\n",
    "\n",
    "random.seed(SEED)\n",
    "np.random.seed(SEED)\n",
    "torch.manual_seed(SEED)\n",
    "torch.cuda.manual_seed(SEED)\n",
    "torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-01T00:00:54.606189Z",
     "iopub.status.busy": "2022-02-01T00:00:54.606047Z",
     "iopub.status.idle": "2022-02-01T00:00:54.608419Z",
     "shell.execute_reply": "2022-02-01T00:00:54.608102Z",
     "shell.execute_reply.started": "2022-02-01T00:00:54.606174Z"
    },
    "id": "7OKdOYoVUn28"
   },
   "source": [
    "## Load Tokenized Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "executionInfo": {
     "elapsed": 2722,
     "status": "ok",
     "timestamp": 1644562992084,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "u-ZvSvH0TuCV"
   },
   "outputs": [],
   "source": [
    "df_train = pd.read_pickle(project_folder/'df_train_en_de.pickel')\n",
    "df_test =  pd.read_pickle(project_folder/'df_test_en_de.pickel')\n",
    "df_valid = pd.read_pickle(project_folder/'df_valid_en_de.pickel')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 423
    },
    "execution": {
     "iopub.execute_input": "2022-02-10T23:33:33.587055Z",
     "iopub.status.busy": "2022-02-10T23:33:33.586873Z",
     "iopub.status.idle": "2022-02-10T23:33:33.597546Z",
     "shell.execute_reply": "2022-02-10T23:33:33.597286Z",
     "shell.execute_reply.started": "2022-02-10T23:33:33.587031Z"
    },
    "executionInfo": {
     "elapsed": 146,
     "status": "ok",
     "timestamp": 1644562993614,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "fDKpYa7mtAXi",
    "outputId": "824b281d-1909-4ffc-f95a-843d4a8207e9",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-bc7122d7-5a3f-46e2-98a3-d9a2551acfe8\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source_tokens</th>\n",
       "      <th>target_tokens</th>\n",
       "      <th>source_tokens_reverse</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[zwei, junge, weiße, männer, sind, i, m, freie...</td>\n",
       "      <td>[two, young, ,, white, males, are, outside, ne...</td>\n",
       "      <td>[., büsche, vieler, nähe, der, in, freien, m, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[mehrere, männer, mit, schutzhelmen, bedienen,...</td>\n",
       "      <td>[several, men, in, hard, hats, are, operating,...</td>\n",
       "      <td>[., antriebsradsystem, ein, bedienen, schutzhe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[ein, kleines, mädchen, klettert, in, ein, spi...</td>\n",
       "      <td>[a, little, girl, climbing, into, a, wooden, p...</td>\n",
       "      <td>[., holz, aus, spielhaus, ein, in, klettert, m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[ein, mann, in, einem, blauen, hemd, steht, au...</td>\n",
       "      <td>[a, man, in, a, blue, shirt, is, standing, on,...</td>\n",
       "      <td>[., fenster, ein, putzt, und, leiter, einer, a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[zwei, männer, stehen, am, herd, und, bereiten...</td>\n",
       "      <td>[two, men, are, at, the, stove, preparing, foo...</td>\n",
       "      <td>[., zu, essen, bereiten, und, herd, am, stehen...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28995</th>\n",
       "      <td>[., wand, verschnörkelten, einer, hinter, schr...</td>\n",
       "      <td>[a, woman, behind, a, scrolled, wall, is, writ...</td>\n",
       "      <td>[eine, frau, schreibt, hinter, einer, verschnö...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28996</th>\n",
       "      <td>[., kletterwand, einer, an, übt, bergsteiger, ...</td>\n",
       "      <td>[a, rock, climber, practices, on, a, rock, cli...</td>\n",
       "      <td>[ein, bergsteiger, übt, an, einer, kletterwand...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28997</th>\n",
       "      <td>[., hauses, einem, vor, straße, einer, auf, ar...</td>\n",
       "      <td>[two, male, construction, workers, are, workin...</td>\n",
       "      <td>[zwei, bauarbeiter, arbeiten, auf, einer, stra...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28998</th>\n",
       "      <td>[., fassade, einer, vor, wagen, einem, mit, ju...</td>\n",
       "      <td>[an, elderly, man, sits, outside, a, storefron...</td>\n",
       "      <td>[ein, älterer, mann, sitzt, mit, einem, jungen...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28999</th>\n",
       "      <td>[., hintergrund, m, i, bergen, und, nebel, mit...</td>\n",
       "      <td>[a, man, in, shorts, and, a, hawaiian, shirt, ...</td>\n",
       "      <td>[ein, mann, in, shorts, und, hawaiihemd, lehnt...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>29000 rows × 3 columns</p>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-bc7122d7-5a3f-46e2-98a3-d9a2551acfe8')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-bc7122d7-5a3f-46e2-98a3-d9a2551acfe8 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-bc7122d7-5a3f-46e2-98a3-d9a2551acfe8');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "                                           source_tokens  ...                              source_tokens_reverse\n",
       "0      [zwei, junge, weiße, männer, sind, i, m, freie...  ...  [., büsche, vieler, nähe, der, in, freien, m, ...\n",
       "1      [mehrere, männer, mit, schutzhelmen, bedienen,...  ...  [., antriebsradsystem, ein, bedienen, schutzhe...\n",
       "2      [ein, kleines, mädchen, klettert, in, ein, spi...  ...  [., holz, aus, spielhaus, ein, in, klettert, m...\n",
       "3      [ein, mann, in, einem, blauen, hemd, steht, au...  ...  [., fenster, ein, putzt, und, leiter, einer, a...\n",
       "4      [zwei, männer, stehen, am, herd, und, bereiten...  ...  [., zu, essen, bereiten, und, herd, am, stehen...\n",
       "...                                                  ...  ...                                                ...\n",
       "28995  [., wand, verschnörkelten, einer, hinter, schr...  ...  [eine, frau, schreibt, hinter, einer, verschnö...\n",
       "28996  [., kletterwand, einer, an, übt, bergsteiger, ...  ...  [ein, bergsteiger, übt, an, einer, kletterwand...\n",
       "28997  [., hauses, einem, vor, straße, einer, auf, ar...  ...  [zwei, bauarbeiter, arbeiten, auf, einer, stra...\n",
       "28998  [., fassade, einer, vor, wagen, einem, mit, ju...  ...  [ein, älterer, mann, sitzt, mit, einem, jungen...\n",
       "28999  [., hintergrund, m, i, bergen, und, nebel, mit...  ...  [ein, mann, in, shorts, und, hawaiihemd, lehnt...\n",
       "\n",
       "[29000 rows x 3 columns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 423
    },
    "execution": {
     "iopub.execute_input": "2022-02-10T23:33:40.970399Z",
     "iopub.status.busy": "2022-02-10T23:33:40.970259Z",
     "iopub.status.idle": "2022-02-10T23:33:40.973665Z",
     "shell.execute_reply": "2022-02-10T23:33:40.973394Z",
     "shell.execute_reply.started": "2022-02-10T23:33:40.970385Z"
    },
    "executionInfo": {
     "elapsed": 165,
     "status": "ok",
     "timestamp": 1644562995019,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "BM1UfGJKtAXj",
    "outputId": "44593a91-056c-49b7-8ee1-cfb2280963c9",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-d471f1de-f408-4a37-ad35-5ae36cab43c0\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source_tokens</th>\n",
       "      <th>target_tokens</th>\n",
       "      <th>source_tokens_reverse</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[eine, gruppe, von, männern, lädt, baumwolle, ...</td>\n",
       "      <td>[a, group, of, men, are, loading, cotton, onto...</td>\n",
       "      <td>[lastwagen, einen, auf, baumwolle, lädt, männe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[ein, mann, schläft, in, einem, grünen, raum, ...</td>\n",
       "      <td>[a, man, sleeping, in, a, green, room, on, a, ...</td>\n",
       "      <td>[., sofa, einem, auf, raum, grünen, einem, in,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[ein, junge, mit, kopfhörern, sitzt, auf, den,...</td>\n",
       "      <td>[a, boy, wearing, headphones, sits, on, a, wom...</td>\n",
       "      <td>[., frau, einer, schultern, den, auf, sitzt, k...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[zwei, männer, bauen, eine, blaue, eisfischerh...</td>\n",
       "      <td>[two, men, setting, up, a, blue, ice, fishing,...</td>\n",
       "      <td>[auf, see, zugefrorenen, einem, auf, eisfische...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[ein, mann, mit, beginnender, glatze, ,, der, ...</td>\n",
       "      <td>[a, balding, man, wearing, a, red, life, jacke...</td>\n",
       "      <td>[., boot, kleinen, einem, in, sitzt, ,, trägt,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1009</th>\n",
       "      <td>[., her, zuckerwatte, mann, ein, stellt, jahrm...</td>\n",
       "      <td>[at, some, sort, of, carnival, ,, a, man, is, ...</td>\n",
       "      <td>[bei, einer, art, jahrmarkt, stellt, ein, mann...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1010</th>\n",
       "      <td>[., bus, einem, vor, steht, polizisten, von, g...</td>\n",
       "      <td>[a, bunch, of, police, officers, are, standing...</td>\n",
       "      <td>[eine, gruppe, von, polizisten, steht, vor, ei...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1011</th>\n",
       "      <td>[., hindurch, brillengläser, ihre, durch, blic...</td>\n",
       "      <td>[a, elderly, white, -, haired, woman, is, look...</td>\n",
       "      <td>[eine, ältere, weißhaarige, frau, sieht, in, i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1012</th>\n",
       "      <td>[., freien, m, i, telefonzellen, an, stehen, m...</td>\n",
       "      <td>[two, men, are, standing, at, telephone, booth...</td>\n",
       "      <td>[zwei, männer, stehen, an, telefonzellen, i, m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1013</th>\n",
       "      <td>[., kommt, toilette, transportablen, einer, au...</td>\n",
       "      <td>[two, women, wearing, red, and, a, man, coming...</td>\n",
       "      <td>[zwei, frauen, in, rot, und, ein, mann, ,, der...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1014 rows × 3 columns</p>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d471f1de-f408-4a37-ad35-5ae36cab43c0')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-d471f1de-f408-4a37-ad35-5ae36cab43c0 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-d471f1de-f408-4a37-ad35-5ae36cab43c0');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "                                          source_tokens  ...                              source_tokens_reverse\n",
       "0     [eine, gruppe, von, männern, lädt, baumwolle, ...  ...  [lastwagen, einen, auf, baumwolle, lädt, männe...\n",
       "1     [ein, mann, schläft, in, einem, grünen, raum, ...  ...  [., sofa, einem, auf, raum, grünen, einem, in,...\n",
       "2     [ein, junge, mit, kopfhörern, sitzt, auf, den,...  ...  [., frau, einer, schultern, den, auf, sitzt, k...\n",
       "3     [zwei, männer, bauen, eine, blaue, eisfischerh...  ...  [auf, see, zugefrorenen, einem, auf, eisfische...\n",
       "4     [ein, mann, mit, beginnender, glatze, ,, der, ...  ...  [., boot, kleinen, einem, in, sitzt, ,, trägt,...\n",
       "...                                                 ...  ...                                                ...\n",
       "1009  [., her, zuckerwatte, mann, ein, stellt, jahrm...  ...  [bei, einer, art, jahrmarkt, stellt, ein, mann...\n",
       "1010  [., bus, einem, vor, steht, polizisten, von, g...  ...  [eine, gruppe, von, polizisten, steht, vor, ei...\n",
       "1011  [., hindurch, brillengläser, ihre, durch, blic...  ...  [eine, ältere, weißhaarige, frau, sieht, in, i...\n",
       "1012  [., freien, m, i, telefonzellen, an, stehen, m...  ...  [zwei, männer, stehen, an, telefonzellen, i, m...\n",
       "1013  [., kommt, toilette, transportablen, einer, au...  ...  [zwei, frauen, in, rot, und, ein, mann, ,, der...\n",
       "\n",
       "[1014 rows x 3 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_valid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 423
    },
    "executionInfo": {
     "elapsed": 3,
     "status": "ok",
     "timestamp": 1644562996155,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "Pk0jr_oAZPWD",
    "outputId": "8cd27cf6-d410-4c7c-e267-ece5c9d628df"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-fe5f03d7-7244-4468-bd5e-31a0b7fb33b0\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source_tokens</th>\n",
       "      <th>target_tokens</th>\n",
       "      <th>source_tokens_reverse</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[ein, mann, mit, einem, orangefarbenen, hut, ,...</td>\n",
       "      <td>[a, man, in, an, orange, hat, starring, at, so...</td>\n",
       "      <td>[., anstarrt, etwas, der, ,, hut, orangefarben...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[ein, boston, terrier, läuft, über, saftig, -,...</td>\n",
       "      <td>[a, boston, terrier, is, running, on, lush, gr...</td>\n",
       "      <td>[., zaun, weißen, einem, vor, gras, grünes, -,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[ein, mädchen, in, einem, karateanzug, bricht,...</td>\n",
       "      <td>[a, girl, in, karate, uniform, breaking, a, st...</td>\n",
       "      <td>[., tritt, einem, mit, stock, einen, bricht, k...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[fünf, leute, in, winterjacken, und, mit, helm...</td>\n",
       "      <td>[five, people, wearing, winter, jackets, and, ...</td>\n",
       "      <td>[., hintergrund, m, i, schneemobilen, mit, sch...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[leute, reparieren, das, dach, eines, hauses, .]</td>\n",
       "      <td>[people, are, fixing, the, roof, of, a, house, .]</td>\n",
       "      <td>[., hauses, eines, dach, das, reparieren, leute]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>995</th>\n",
       "      <td>[., stehen, herum, sie, um, personen, andere, ...</td>\n",
       "      <td>[marathon, runners, are, racing, on, a, city, ...</td>\n",
       "      <td>[marathonläufer, laufen, auf, einer, städtisch...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>996</th>\n",
       "      <td>[., fahrradfahren, beim, sonnenhut, einen, trä...</td>\n",
       "      <td>[asian, woman, wearing, a, sunhat, while, ridi...</td>\n",
       "      <td>[asiatische, frau, trägt, einen, sonnenhut, be...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>997</th>\n",
       "      <td>[., bäumen, zwei, bei, boden, dem, auf, spiele...</td>\n",
       "      <td>[some, children, are, outside, playing, in, th...</td>\n",
       "      <td>[ein, paar, kinder, sind, i, m, freien, und, s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>998</th>\n",
       "      <td>[., videospiel, ein, spielt, mann, älterer, ein]</td>\n",
       "      <td>[an, older, man, is, playing, a, video, arcade...</td>\n",
       "      <td>[ein, älterer, mann, spielt, ein, videospiel, .]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999</th>\n",
       "      <td>[., hintergrund, m, i, berg, einem, mit, küste...</td>\n",
       "      <td>[a, girl, at, the, shore, of, a, beach, with, ...</td>\n",
       "      <td>[ein, mädchen, an, einer, küste, mit, einem, b...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000 rows × 3 columns</p>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-fe5f03d7-7244-4468-bd5e-31a0b7fb33b0')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-fe5f03d7-7244-4468-bd5e-31a0b7fb33b0 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-fe5f03d7-7244-4468-bd5e-31a0b7fb33b0');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "                                         source_tokens  ...                              source_tokens_reverse\n",
       "0    [ein, mann, mit, einem, orangefarbenen, hut, ,...  ...  [., anstarrt, etwas, der, ,, hut, orangefarben...\n",
       "1    [ein, boston, terrier, läuft, über, saftig, -,...  ...  [., zaun, weißen, einem, vor, gras, grünes, -,...\n",
       "2    [ein, mädchen, in, einem, karateanzug, bricht,...  ...  [., tritt, einem, mit, stock, einen, bricht, k...\n",
       "3    [fünf, leute, in, winterjacken, und, mit, helm...  ...  [., hintergrund, m, i, schneemobilen, mit, sch...\n",
       "4     [leute, reparieren, das, dach, eines, hauses, .]  ...   [., hauses, eines, dach, das, reparieren, leute]\n",
       "..                                                 ...  ...                                                ...\n",
       "995  [., stehen, herum, sie, um, personen, andere, ...  ...  [marathonläufer, laufen, auf, einer, städtisch...\n",
       "996  [., fahrradfahren, beim, sonnenhut, einen, trä...  ...  [asiatische, frau, trägt, einen, sonnenhut, be...\n",
       "997  [., bäumen, zwei, bei, boden, dem, auf, spiele...  ...  [ein, paar, kinder, sind, i, m, freien, und, s...\n",
       "998   [., videospiel, ein, spielt, mann, älterer, ein]  ...   [ein, älterer, mann, spielt, ein, videospiel, .]\n",
       "999  [., hintergrund, m, i, berg, einem, mit, küste...  ...  [ein, mädchen, an, einer, küste, mit, einem, b...\n",
       "\n",
       "[1000 rows x 3 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "If-BMrTctAXj"
   },
   "source": [
    "## Build Vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T23:33:47.334343Z",
     "iopub.status.busy": "2022-02-10T23:33:47.334223Z",
     "iopub.status.idle": "2022-02-10T23:33:47.337265Z",
     "shell.execute_reply": "2022-02-10T23:33:47.336787Z",
     "shell.execute_reply.started": "2022-02-10T23:33:47.334329Z"
    },
    "executionInfo": {
     "elapsed": 134,
     "status": "ok",
     "timestamp": 1644563001959,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "GY9kL2JxtAXj",
    "tags": []
   },
   "outputs": [],
   "source": [
    "def create_vocab(text, min_freq, specials):\n",
    "    #code here\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T23:33:48.047869Z",
     "iopub.status.busy": "2022-02-10T23:33:48.047668Z",
     "iopub.status.idle": "2022-02-10T23:33:48.131416Z",
     "shell.execute_reply": "2022-02-10T23:33:48.131019Z",
     "shell.execute_reply.started": "2022-02-10T23:33:48.047839Z"
    },
    "executionInfo": {
     "elapsed": 241,
     "status": "ok",
     "timestamp": 1644563003201,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "dcvIJAIHtAXj",
    "tags": []
   },
   "outputs": [],
   "source": [
    "source_vocab = # code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2022-02-10T23:33:48.741033Z",
     "iopub.status.busy": "2022-02-10T23:33:48.740884Z",
     "iopub.status.idle": "2022-02-10T23:33:48.744065Z",
     "shell.execute_reply": "2022-02-10T23:33:48.743764Z",
     "shell.execute_reply.started": "2022-02-10T23:33:48.741018Z"
    },
    "executionInfo": {
     "elapsed": 2,
     "status": "ok",
     "timestamp": 1644563003736,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "qtEt9Sh-tAXk",
    "outputId": "49ecb3a9-0f9d-43bc-90ef-4d5a15e07a10",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7874"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(source_vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 423
    },
    "execution": {
     "iopub.execute_input": "2022-02-10T23:33:52.487288Z",
     "iopub.status.busy": "2022-02-10T23:33:52.487138Z",
     "iopub.status.idle": "2022-02-10T23:33:52.501517Z",
     "shell.execute_reply": "2022-02-10T23:33:52.501020Z",
     "shell.execute_reply.started": "2022-02-10T23:33:52.487273Z"
    },
    "executionInfo": {
     "elapsed": 116,
     "status": "ok",
     "timestamp": 1644563004871,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "bf9fu5L2tAXk",
    "outputId": "5e4c2da9-6178-4155-b848-47bb2a495e3b",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-9d9e3f3c-ad72-425d-a511-dbfb459ec150\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tokens</th>\n",
       "      <th>index</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3373</th>\n",
       "      <td>&lt;unk&gt;</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>936</th>\n",
       "      <td>&lt;BOS&gt;</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2372</th>\n",
       "      <td>&lt;EOS&gt;</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3624</th>\n",
       "      <td>&lt;PAD&gt;</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2772</th>\n",
       "      <td>zwei</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>geländefahrzeugs</td>\n",
       "      <td>7869</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>sonnentag</td>\n",
       "      <td>7870</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>lederoberteil</td>\n",
       "      <td>7871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>reisfeld</td>\n",
       "      <td>7872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>garnelen</td>\n",
       "      <td>7873</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7874 rows × 2 columns</p>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9d9e3f3c-ad72-425d-a511-dbfb459ec150')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-9d9e3f3c-ad72-425d-a511-dbfb459ec150 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-9d9e3f3c-ad72-425d-a511-dbfb459ec150');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "                tokens  index\n",
       "3373             <unk>      0\n",
       "936              <BOS>      1\n",
       "2372             <EOS>      2\n",
       "3624             <PAD>      3\n",
       "2772              zwei      4\n",
       "...                ...    ...\n",
       "4     geländefahrzeugs   7869\n",
       "3            sonnentag   7870\n",
       "2        lederoberteil   7871\n",
       "1             reisfeld   7872\n",
       "0             garnelen   7873\n",
       "\n",
       "[7874 rows x 2 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(source_vocab.get_stoi().items(), columns=['tokens', 'index']).sort_values(by = ['index'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2022-02-10T23:33:53.446545Z",
     "iopub.status.busy": "2022-02-10T23:33:53.446395Z",
     "iopub.status.idle": "2022-02-10T23:33:53.449644Z",
     "shell.execute_reply": "2022-02-10T23:33:53.449281Z",
     "shell.execute_reply.started": "2022-02-10T23:33:53.446531Z"
    },
    "executionInfo": {
     "elapsed": 173,
     "status": "ok",
     "timestamp": 1644563009764,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "8pKCTbN6tAXk",
    "outputId": "0995d1eb-fc6e-4d8d-ac74-bcd022156fbd",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check index of unknown word - it should be zero\n",
    "source_vocab['abracdabra']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T23:33:54.447200Z",
     "iopub.status.busy": "2022-02-10T23:33:54.447050Z",
     "iopub.status.idle": "2022-02-10T23:33:54.525550Z",
     "shell.execute_reply": "2022-02-10T23:33:54.525228Z",
     "shell.execute_reply.started": "2022-02-10T23:33:54.447186Z"
    },
    "executionInfo": {
     "elapsed": 2,
     "status": "ok",
     "timestamp": 1644563010216,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "qkXg6IratAXk",
    "tags": []
   },
   "outputs": [],
   "source": [
    "target_vocab = create_vocab(df_train['target_tokens'], 2, ['<unk>', '<BOS>', '<EOS>', '<PAD>'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2022-02-10T23:33:55.165896Z",
     "iopub.status.busy": "2022-02-10T23:33:55.165711Z",
     "iopub.status.idle": "2022-02-10T23:33:55.168705Z",
     "shell.execute_reply": "2022-02-10T23:33:55.168426Z",
     "shell.execute_reply.started": "2022-02-10T23:33:55.165881Z"
    },
    "executionInfo": {
     "elapsed": 2,
     "status": "ok",
     "timestamp": 1644563010523,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "v_Y8C62GtAXk",
    "outputId": "51483378-d084-4288-97d4-59b6d571e5a6",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5893"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(target_vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "feccYoeWtAXk"
   },
   "source": [
    "# Create Dataset and Dataloader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T23:33:59.446070Z",
     "iopub.status.busy": "2022-02-10T23:33:59.445803Z",
     "iopub.status.idle": "2022-02-10T23:33:59.448988Z",
     "shell.execute_reply": "2022-02-10T23:33:59.448690Z",
     "shell.execute_reply.started": "2022-02-10T23:33:59.446041Z"
    },
    "executionInfo": {
     "elapsed": 111,
     "status": "ok",
     "timestamp": 1644563011057,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "fCMXepKUtAXk",
    "tags": []
   },
   "outputs": [],
   "source": [
    "class EngGerman(Dataset):\n",
    "    # code here\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T23:33:59.939188Z",
     "iopub.status.busy": "2022-02-10T23:33:59.939041Z",
     "iopub.status.idle": "2022-02-10T23:33:59.942509Z",
     "shell.execute_reply": "2022-02-10T23:33:59.942174Z",
     "shell.execute_reply.started": "2022-02-10T23:33:59.939174Z"
    },
    "executionInfo": {
     "elapsed": 149,
     "status": "ok",
     "timestamp": 1644563448932,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "-Q32VBamtAXl",
    "tags": []
   },
   "outputs": [],
   "source": [
    "trainset =  # code here\n",
    "testset =  # code here\n",
    "validset =  # code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2022-02-10T23:34:02.610003Z",
     "iopub.status.busy": "2022-02-10T23:34:02.609824Z",
     "iopub.status.idle": "2022-02-10T23:34:02.613424Z",
     "shell.execute_reply": "2022-02-10T23:34:02.612985Z",
     "shell.execute_reply.started": "2022-02-10T23:34:02.609974Z"
    },
    "executionInfo": {
     "elapsed": 238,
     "status": "ok",
     "timestamp": 1644563450284,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "2md5tDSutAXl",
    "outputId": "7984dec6-37f3-4c37-ab7c-0c03167ccb2e",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['.',\n",
       "  'büsche',\n",
       "  'vieler',\n",
       "  'nähe',\n",
       "  'der',\n",
       "  'in',\n",
       "  'freien',\n",
       "  'm',\n",
       "  'i',\n",
       "  'sind',\n",
       "  'männer',\n",
       "  'weiße',\n",
       "  'junge',\n",
       "  'zwei'],\n",
       " ['two',\n",
       "  'young',\n",
       "  ',',\n",
       "  'white',\n",
       "  'males',\n",
       "  'are',\n",
       "  'outside',\n",
       "  'near',\n",
       "  'many',\n",
       "  'bushes',\n",
       "  '.'])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainset.__getitem__(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2022-02-10T23:34:03.292518Z",
     "iopub.status.busy": "2022-02-10T23:34:03.292355Z",
     "iopub.status.idle": "2022-02-10T23:34:03.295484Z",
     "shell.execute_reply": "2022-02-10T23:34:03.295145Z",
     "shell.execute_reply.started": "2022-02-10T23:34:03.292490Z"
    },
    "executionInfo": {
     "elapsed": 201,
     "status": "ok",
     "timestamp": 1644563451615,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "L18xzZrltAXl",
    "outputId": "8014ac51-941b-43c5-b754-9307034b4a70",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(29000, 1000, 1014)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(trainset), len(testset), len(validset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T23:34:03.804016Z",
     "iopub.status.busy": "2022-02-10T23:34:03.803384Z",
     "iopub.status.idle": "2022-02-10T23:34:03.806444Z",
     "shell.execute_reply": "2022-02-10T23:34:03.806038Z",
     "shell.execute_reply.started": "2022-02-10T23:34:03.803979Z"
    },
    "executionInfo": {
     "elapsed": 202,
     "status": "ok",
     "timestamp": 1644563452946,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "bPg8WslltAXl",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# return a list for index of tokens in following order [<BOS> ,text tokens ,<EOS>]\n",
    "def text_transform (my_vocab, text):\n",
    "    # code here\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2022-02-10T23:34:04.396011Z",
     "iopub.status.busy": "2022-02-10T23:34:04.395845Z",
     "iopub.status.idle": "2022-02-10T23:34:04.402293Z",
     "shell.execute_reply": "2022-02-10T23:34:04.401859Z",
     "shell.execute_reply.started": "2022-02-10T23:34:04.395996Z"
    },
    "executionInfo": {
     "elapsed": 133,
     "status": "ok",
     "timestamp": 1644563453930,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "bpQKiRHLtAXl",
    "outputId": "a443997b-875c-467c-c7e9-7be3d67e5929",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['a', 'black', 'dog', 'and', 'a', 'spotted', 'dog', 'are', 'fighting']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([ 1, 21, 90, 91, 92, 21, 93, 91,  9, 94,  2])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = trainset.__getitem__(12)[1]\n",
    "print(text)\n",
    "text_transform(target_vocab, text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T23:34:05.004761Z",
     "iopub.status.busy": "2022-02-10T23:34:05.004595Z",
     "iopub.status.idle": "2022-02-10T23:34:05.008167Z",
     "shell.execute_reply": "2022-02-10T23:34:05.007697Z",
     "shell.execute_reply.started": "2022-02-10T23:34:05.004745Z"
    },
    "executionInfo": {
     "elapsed": 139,
     "status": "ok",
     "timestamp": 1644563455937,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "CgKupdZ1Un2-",
    "tags": []
   },
   "outputs": [],
   "source": [
    "def collate_batch(batch):\n",
    "    # code here\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T23:34:05.545072Z",
     "iopub.status.busy": "2022-02-10T23:34:05.544683Z",
     "iopub.status.idle": "2022-02-10T23:34:05.547646Z",
     "shell.execute_reply": "2022-02-10T23:34:05.547256Z",
     "shell.execute_reply.started": "2022-02-10T23:34:05.545055Z"
    },
    "executionInfo": {
     "elapsed": 141,
     "status": "ok",
     "timestamp": 1644563457042,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "EqHCA63DUn2_",
    "tags": []
   },
   "outputs": [],
   "source": [
    "batch_size = 2\n",
    "\n",
    "train_loader = DataLoader(trainset, batch_size=batch_size, shuffle=True, \n",
    "                              collate_fn=collate_batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2022-02-10T23:34:06.008525Z",
     "iopub.status.busy": "2022-02-10T23:34:06.008258Z",
     "iopub.status.idle": "2022-02-10T23:34:06.034466Z",
     "shell.execute_reply": "2022-02-10T23:34:06.034086Z",
     "shell.execute_reply.started": "2022-02-10T23:34:06.008508Z"
    },
    "executionInfo": {
     "elapsed": 110,
     "status": "ok",
     "timestamp": 1644563458245,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "-NUpXpJYtAXm",
    "outputId": "c33cabc2-fcf9-4eec-a8a3-74d5baeae322",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[  1,   1],\n",
      "        [ 22,   4],\n",
      "        [ 29,  74],\n",
      "        [ 12,  40],\n",
      "        [ 30,  12],\n",
      "        [363,  13],\n",
      "        [173, 899],\n",
      "        [101,  17],\n",
      "        [386,   2],\n",
      "        [ 55,   3],\n",
      "        [ 17,   3],\n",
      "        [  2,   3]])\n"
     ]
    }
   ],
   "source": [
    "for source, target in train_loader:\n",
    "  print(source)\n",
    "  break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T23:34:09.366342Z",
     "iopub.status.busy": "2022-02-10T23:34:09.366163Z",
     "iopub.status.idle": "2022-02-10T23:34:09.369445Z",
     "shell.execute_reply": "2022-02-10T23:34:09.369035Z",
     "shell.execute_reply.started": "2022-02-10T23:34:09.366302Z"
    },
    "executionInfo": {
     "elapsed": 163,
     "status": "ok",
     "timestamp": 1644563460443,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "axyZQ-4z1B-z",
    "tags": []
   },
   "outputs": [],
   "source": [
    "batch_size = 128\n",
    "\n",
    "train_loader = # code here\n",
    "valid_loader = # code here\n",
    "test_loader =  # code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Mhyhh7uCUn2_"
   },
   "source": [
    "## Building the Seq2Seq Model\n",
    "\n",
    "## Encoder\n",
    "\n",
    "The encoder is similar to the previous one, with the multi-layer LSTM swapped for a single-layer GRU. We also don't pass the dropout as an argument to the GRU as that dropout is used between each layer of a multi-layered RNN. As we only have a single layer, PyTorch will display a warning if we try and use pass a dropout value to it.\n",
    "\n",
    "Another thing to note about the GRU is that it only requires and returns a hidden state, there is no cell state like in the LSTM.\n",
    "\n",
    "$$\\begin{align*}\n",
    "h_t &= \\text{GRU}(e(x_t), h_{t-1})\\\\\n",
    "(h_t, c_t) &= \\text{LSTM}(e(x_t), h_{t-1}, c_{t-1})\\\\\n",
    "h_t &= \\text{RNN}(e(x_t), h_{t-1})\n",
    "\\end{align*}$$\n",
    "\n",
    "From the equations above, it looks like the RNN and the GRU are identical. Inside the GRU, however, is a number of *gating mechanisms* that control the information flow in to and out of the hidden state (similar to an LSTM). Again, for more info, check out [this](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) excellent post. \n",
    "\n",
    "The rest of the encoder should be very familar from the last tutorial, it takes in a sequence, $X = \\{x_1, x_2, ... , x_T\\}$, passes it through the embedding layer, recurrently calculates hidden states, $H = \\{h_1, h_2, ..., h_T\\}$, and returns a context vector (the final hidden state), $z=h_T$.\n",
    "\n",
    "$$h_t = \\text{EncoderGRU}(e(x_t), h_{t-1})$$\n",
    "\n",
    "This is identical to the encoder of the general seq2seq model, with all the \"magic\" happening inside the GRU (green).\n",
    "\n",
    "![](assets/seq2seq5.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T23:34:11.040313Z",
     "iopub.status.busy": "2022-02-10T23:34:11.040194Z",
     "iopub.status.idle": "2022-02-10T23:34:11.044053Z",
     "shell.execute_reply": "2022-02-10T23:34:11.043599Z",
     "shell.execute_reply.started": "2022-02-10T23:34:11.040299Z"
    },
    "executionInfo": {
     "elapsed": 117,
     "status": "ok",
     "timestamp": 1644563465250,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "zabfsjZOUn3A",
    "tags": []
   },
   "outputs": [],
   "source": [
    "class Encoder(nn.Module):\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7esX6rp0Un3A"
   },
   "source": [
    "## Decoder\n",
    "\n",
    "The decoder is where the implementation differs significantly from the previous model and we alleviate some of the information compression.\n",
    "\n",
    "Instead of the GRU in the decoder taking just the embedded target token, $d(y_t)$ and the previous hidden state $s_{t-1}$ as inputs, it also takes the context vector $z$. \n",
    "\n",
    "$$s_t = \\text{DecoderGRU}(d(y_t), s_{t-1}, z)$$\n",
    "\n",
    "Note how this context vector, $z$, does not have a $t$ subscript, meaning we re-use the same context vector returned by the encoder for every time-step in the decoder. \n",
    "\n",
    "Before, we predicted the next token, $\\hat{y}_{t+1}$, with the linear layer, $f$, only using the top-layer decoder hidden state at that time-step, $s_t$, as $\\hat{y}_{t+1}=f(s_t^L)$. Now, we also pass the embedding of current token, $d(y_t)$ and the context vector, $z$ to the linear layer.\n",
    "\n",
    "$$\\hat{y}_{t+1} = f(d(y_t), s_t, z)$$\n",
    "\n",
    "Thus, our decoder now looks something like this:\n",
    "\n",
    "![](assets/seq2seq6.png)\n",
    "\n",
    "Note, the initial hidden state, $s_0$, is still the context vector, $z$, so when generating the first token we are actually inputting two identical context vectors into the GRU.\n",
    "\n",
    "How do these two changes reduce the information compression? Well, hypothetically the decoder hidden states, $s_t$, no longer need to contain information about the source sequence as it is always available as an input. Thus, it only needs to contain information about what tokens it has generated so far. The addition of $y_t$ to the linear layer also means this layer can directly see what the token is, without having to get this information from the hidden state. \n",
    "\n",
    "However, this hypothesis is just a hypothesis, it is impossible to determine how the model actually uses the information provided to it (don't listen to anyone that says differently). Nevertheless, it is a solid intuition and the results seem to indicate that this modifications are a good idea!\n",
    "\n",
    "Within the implementation, we will pass $d(y_t)$ and $z$ to the GRU by concatenating them together, so the input dimensions to the GRU are now `emb_dim + hid_dim` (as context vector will be of size `hid_dim`). The linear layer will take $d(y_t), s_t$ and $z$ also by concatenating them together, hence the input dimensions are now `emb_dim + hid_dim*2`. We also don't pass a value of dropout to the GRU as it only uses a single layer.\n",
    "\n",
    "`forward` now takes a `context` argument. Inside of `forward`, we concatenate $y_t$ and $z$ as `emb_con` before feeding to the GRU, and we concatenate $d(y_t)$, $s_t$ and $z$ together as `output` before feeding it through the linear layer to receive our predictions, $\\hat{y}_{t+1}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T23:34:13.195925Z",
     "iopub.status.busy": "2022-02-10T23:34:13.195773Z",
     "iopub.status.idle": "2022-02-10T23:34:13.200246Z",
     "shell.execute_reply": "2022-02-10T23:34:13.199832Z",
     "shell.execute_reply.started": "2022-02-10T23:34:13.195911Z"
    },
    "executionInfo": {
     "elapsed": 226,
     "status": "ok",
     "timestamp": 1644563467091,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "AVkx_S2fUn3A",
    "tags": []
   },
   "outputs": [],
   "source": [
    "class Decoder(nn.Module):\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VNxlQ5KYUn3A"
   },
   "source": [
    "## Seq2Seq Model\n",
    "\n",
    "Putting the encoder and decoder together, we get:\n",
    "\n",
    "![](assets/seq2seq7.png)\n",
    "\n",
    "Again, in this implementation we need to ensure the hidden dimensions in both the encoder and the decoder are the same.\n",
    "\n",
    "Briefly going over all of the steps:\n",
    "- the `outputs` tensor is created to hold all predictions, $\\hat{Y}$\n",
    "- the source sequence, $X$, is fed into the encoder to receive a `context` vector\n",
    "- the initial decoder hidden state is set to be the `context` vector, $s_0 = z = h_T$\n",
    "- we use a batch of `<sos>` tokens as the first `input`, $y_1$\n",
    "- we then decode within a loop:\n",
    "  - inserting the input token $y_t$, previous hidden state, $s_{t-1}$, and the context vector, $z$, into the decoder\n",
    "  - receiving a prediction, $\\hat{y}_{t+1}$, and a new hidden state, $s_t$\n",
    "  - we then decide if we are going to teacher force or not, setting the next input as appropriate (either the ground truth next token in the target sequence or the highest predicted next token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T23:34:14.021036Z",
     "iopub.status.busy": "2022-02-10T23:34:14.020694Z",
     "iopub.status.idle": "2022-02-10T23:34:14.025662Z",
     "shell.execute_reply": "2022-02-10T23:34:14.025270Z",
     "shell.execute_reply.started": "2022-02-10T23:34:14.021021Z"
    },
    "executionInfo": {
     "elapsed": 111,
     "status": "ok",
     "timestamp": 1644563469133,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "kH4m5zk7Un3B",
    "tags": []
   },
   "outputs": [],
   "source": [
    "class Seq2Seq(nn.Module):\n",
    "\n",
    "        \n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HyPbOrlHUn3B"
   },
   "source": [
    "# Training the Seq2Seq Model\n",
    "\n",
    "The rest of this tutorial is very similar to the previous one. \n",
    "\n",
    "We initialise our encoder, decoder and seq2seq model (placing it on the GPU if we have one). As before, the embedding dimensions and the amount of dropout used can be different between the encoder and the decoder, but the hidden dimensions must remain the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T23:34:14.978310Z",
     "iopub.status.busy": "2022-02-10T23:34:14.978144Z",
     "iopub.status.idle": "2022-02-10T23:34:16.890784Z",
     "shell.execute_reply": "2022-02-10T23:34:16.890266Z",
     "shell.execute_reply.started": "2022-02-10T23:34:14.978295Z"
    },
    "executionInfo": {
     "elapsed": 255,
     "status": "ok",
     "timestamp": 1644563471159,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "xg6tH3mDUn3B",
    "tags": []
   },
   "outputs": [],
   "source": [
    "INPUT_DIM = len(source_vocab)\n",
    "OUTPUT_DIM = len(target_vocab)\n",
    "ENC_EMB_DIM = 256\n",
    "DEC_EMB_DIM = 256\n",
    "HID_DIM = 512\n",
    "N_LAYERS = 1\n",
    "ENC_DROPOUT = 0.5\n",
    "DEC_DROPOUT = 0.5\n",
    "\n",
    "enc = Encoder(INPUT_DIM, ENC_EMB_DIM, HID_DIM, N_LAYERS, ENC_DROPOUT)\n",
    "dec = Decoder(OUTPUT_DIM, DEC_EMB_DIM, HID_DIM, N_LAYERS, DEC_DROPOUT)\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "\n",
    "model = Seq2Seq(enc, dec, device).to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "R8QiCSrzUn3B"
   },
   "source": [
    "Next, we initialize our parameters. The paper states the parameters are initialized from a normal distribution with a mean of 0 and a standard deviation of 0.01, i.e. $\\mathcal{N}(0, 0.01)$. \n",
    "\n",
    "It also states we should initialize the recurrent parameters to a special initialization, however to keep things simple we'll also initialize them to $\\mathcal{N}(0, 0.01)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2022-02-10T23:35:57.267421Z",
     "iopub.status.busy": "2022-02-10T23:35:57.267282Z",
     "iopub.status.idle": "2022-02-10T23:35:57.272296Z",
     "shell.execute_reply": "2022-02-10T23:35:57.271909Z",
     "shell.execute_reply.started": "2022-02-10T23:35:57.267408Z"
    },
    "executionInfo": {
     "elapsed": 3,
     "status": "ok",
     "timestamp": 1644563473045,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "8a9Kgdh5Un3B",
    "outputId": "c03e8da5-e564-43e3-f5c6-0478b08d3abb",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Seq2Seq(\n",
       "  (encoder): Encoder(\n",
       "    (embedding): Embedding(7874, 256, padding_idx=3)\n",
       "    (rnn): GRU(256, 512)\n",
       "    (dropout): Dropout(p=0.5, inplace=False)\n",
       "  )\n",
       "  (decoder): Decoder(\n",
       "    (embedding): Embedding(5893, 256, padding_idx=3)\n",
       "    (rnn): GRU(768, 512)\n",
       "    (fc_out): Linear(in_features=1280, out_features=5893, bias=True)\n",
       "    (dropout): Dropout(p=0.5, inplace=False)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def init_weights(m):\n",
    "    for name, param in m.named_parameters():\n",
    "        nn.init.normal_(param.data, 0, 0.01)\n",
    "        \n",
    "model.apply(init_weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kX8djfNtUn3B"
   },
   "source": [
    "We also define a function that will calculate the number of trainable parameters in the model.\n",
    "\n",
    "Even though we only have a single layer RNN for our encoder and decoder we actually have **more** parameters  than the last model. This is due to the increased size of the inputs to the GRU and the linear layer. However, it is not a significant amount of parameters and causes a minimal amount of increase in training time (~3 seconds per epoch extra)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2022-02-10T23:35:57.611743Z",
     "iopub.status.busy": "2022-02-10T23:35:57.611597Z",
     "iopub.status.idle": "2022-02-10T23:35:57.614740Z",
     "shell.execute_reply": "2022-02-10T23:35:57.614471Z",
     "shell.execute_reply.started": "2022-02-10T23:35:57.611728Z"
    },
    "executionInfo": {
     "elapsed": 145,
     "status": "ok",
     "timestamp": 1644563475026,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "SJW-AD6NUn3C",
    "outputId": "d7cc1242-17b7-4207-814d-af3e2ea41de5",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model has 14,225,157 trainable parameters\n"
     ]
    }
   ],
   "source": [
    "def count_parameters(model):\n",
    "    return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
    "\n",
    "print(f'The model has {count_parameters(model):,} trainable parameters')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kqFRDWy5Un3C"
   },
   "source": [
    "We define our optimizer, which we use to update our parameters in the training loop. Check out [this](http://ruder.io/optimizing-gradient-descent/) post for information about different optimizers. Here, we'll use Adam."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T23:35:57.973873Z",
     "iopub.status.busy": "2022-02-10T23:35:57.973691Z",
     "iopub.status.idle": "2022-02-10T23:35:57.976789Z",
     "shell.execute_reply": "2022-02-10T23:35:57.976252Z",
     "shell.execute_reply.started": "2022-02-10T23:35:57.973858Z"
    },
    "executionInfo": {
     "elapsed": 2,
     "status": "ok",
     "timestamp": 1644563477113,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "uQW8MwmpUn3C",
    "tags": []
   },
   "outputs": [],
   "source": [
    "optimizer = optim.Adam(model.parameters())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vP8-nNUgUn3C"
   },
   "source": [
    "Next, we define our loss function. The `CrossEntropyLoss` function calculates both the log softmax as well as the negative log-likelihood of our predictions. \n",
    "\n",
    "Our loss function calculates the average loss per token, however by passing the index of the `<pad>` token as the `ignore_index` argument we ignore the loss whenever the target token is a padding token. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2022-02-10T23:35:58.309542Z",
     "iopub.status.busy": "2022-02-10T23:35:58.309395Z",
     "iopub.status.idle": "2022-02-10T23:35:58.312664Z",
     "shell.execute_reply": "2022-02-10T23:35:58.312329Z",
     "shell.execute_reply.started": "2022-02-10T23:35:58.309528Z"
    },
    "executionInfo": {
     "elapsed": 108,
     "status": "ok",
     "timestamp": 1644563481663,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "kwbug8VNUn3C",
    "outputId": "c6dba05d-3e9d-456a-c388-af2393e0194c",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TRG_PAD_IDX = target_vocab['<PAD>']\n",
    "TRG_PAD_IDX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T23:35:58.516354Z",
     "iopub.status.busy": "2022-02-10T23:35:58.516198Z",
     "iopub.status.idle": "2022-02-10T23:35:58.518729Z",
     "shell.execute_reply": "2022-02-10T23:35:58.518325Z",
     "shell.execute_reply.started": "2022-02-10T23:35:58.516340Z"
    },
    "executionInfo": {
     "elapsed": 2,
     "status": "ok",
     "timestamp": 1644563482600,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "HqDGxXU-2A7y",
    "tags": []
   },
   "outputs": [],
   "source": [
    "criterion = nn.CrossEntropyLoss(ignore_index = TRG_PAD_IDX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DTSE8TjRUn3C"
   },
   "source": [
    "Next, we'll define our training loop. \n",
    "\n",
    "First, we'll set the model into \"training mode\" with `model.train()`. This will turn on dropout (and batch normalization, which we aren't using) and then iterate through our data iterator.\n",
    "\n",
    "As stated before, our decoder loop starts at 1, not 0. This means the 0th element of our `outputs` tensor remains all zeros. So our `trg` and `outputs` look something like:\n",
    "\n",
    "$$\\begin{align*}\n",
    "\\text{trg} = [<sos>, &y_1, y_2, y_3, <eos>]\\\\\n",
    "\\text{outputs} = [0, &\\hat{y}_1, \\hat{y}_2, \\hat{y}_3, <eos>]\n",
    "\\end{align*}$$\n",
    "\n",
    "Here, when we calculate the loss, we cut off the first element of each tensor to get:\n",
    "\n",
    "$$\\begin{align*}\n",
    "\\text{trg} = [&y_1, y_2, y_3, <eos>]\\\\\n",
    "\\text{outputs} = [&\\hat{y}_1, \\hat{y}_2, \\hat{y}_3, <eos>]\n",
    "\\end{align*}$$\n",
    "\n",
    "At each iteration:\n",
    "- get the source and target sentences from the batch, $X$ and $Y$\n",
    "- zero the gradients calculated from the last batch\n",
    "- feed the source and target into the model to get the output, $\\hat{Y}$\n",
    "- as the loss function only works on 2d inputs with 1d targets we need to flatten each of them with `.view`\n",
    "    - we slice off the first column of the output and target tensors as mentioned above\n",
    "- calculate the gradients with `loss.backward()`\n",
    "- clip the gradients to prevent them from exploding (a common issue in RNNs)\n",
    "- update the parameters of our model by doing an optimizer step\n",
    "- sum the loss value to a running total\n",
    "\n",
    "Finally, we return the loss that is averaged over all batches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T23:35:58.878221Z",
     "iopub.status.busy": "2022-02-10T23:35:58.878069Z",
     "iopub.status.idle": "2022-02-10T23:35:58.882207Z",
     "shell.execute_reply": "2022-02-10T23:35:58.881870Z",
     "shell.execute_reply.started": "2022-02-10T23:35:58.878208Z"
    },
    "executionInfo": {
     "elapsed": 192,
     "status": "ok",
     "timestamp": 1644563484589,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "BYKiriabUn3C",
    "tags": []
   },
   "outputs": [],
   "source": [
    "def train(model, iterator, optimizer, criterion, clip):\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XEEHX3lcUn3C"
   },
   "source": [
    "Our evaluation loop is similar to our training loop, however as we aren't updating any parameters we don't need to pass an optimizer or a clip value.\n",
    "\n",
    "We must remember to set the model to evaluation mode with `model.eval()`. This will turn off dropout (and batch normalization, if used).\n",
    "\n",
    "We use the `with torch.no_grad()` block to ensure no gradients are calculated within the block. This reduces memory consumption and speeds things up. \n",
    "\n",
    "The iteration loop is similar (without the parameter updates), however we must ensure we turn teacher forcing off for evaluation. This will cause the model to only use it's own predictions to make further predictions within a sentence, which mirrors how it would be used in deployment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T23:35:59.243251Z",
     "iopub.status.busy": "2022-02-10T23:35:59.243108Z",
     "iopub.status.idle": "2022-02-10T23:35:59.246519Z",
     "shell.execute_reply": "2022-02-10T23:35:59.246209Z",
     "shell.execute_reply.started": "2022-02-10T23:35:59.243217Z"
    },
    "executionInfo": {
     "elapsed": 1,
     "status": "ok",
     "timestamp": 1644563486426,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "q_0CMMb9Un3C",
    "tags": []
   },
   "outputs": [],
   "source": [
    "def evaluate(model, iterator, criterion):\n",
    "    \n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dBWvFyRmUn3D"
   },
   "source": [
    "Next, we'll create a function that we'll use to tell us how long an epoch takes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T23:36:00.064590Z",
     "iopub.status.busy": "2022-02-10T23:36:00.064452Z",
     "iopub.status.idle": "2022-02-10T23:36:00.067194Z",
     "shell.execute_reply": "2022-02-10T23:36:00.066884Z",
     "shell.execute_reply.started": "2022-02-10T23:36:00.064576Z"
    },
    "executionInfo": {
     "elapsed": 2,
     "status": "ok",
     "timestamp": 1644563488400,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "dwfIeh7XUn3D",
    "tags": []
   },
   "outputs": [],
   "source": [
    "def epoch_time(start_time, end_time):\n",
    "    elapsed_time = end_time - start_time\n",
    "    elapsed_mins = int(elapsed_time / 60)\n",
    "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
    "    return elapsed_mins, elapsed_secs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n16Yz_VcUn3D"
   },
   "source": [
    "We can finally start training our model!\n",
    "\n",
    "At each epoch, we'll be checking if our model has achieved the best validation loss so far. If it has, we'll update our best validation loss and save the parameters of our model (called `state_dict` in PyTorch). Then, when we come to test our model, we'll use the saved parameters used to achieve the best validation loss. \n",
    "\n",
    "We'll be printing out both the loss and the perplexity at each epoch. It is easier to see a change in perplexity than a change in loss as the numbers are much bigger."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2022-02-10T23:36:00.592472Z",
     "iopub.status.busy": "2022-02-10T23:36:00.592331Z",
     "iopub.status.idle": "2022-02-10T23:39:57.620488Z",
     "shell.execute_reply": "2022-02-10T23:39:57.620143Z",
     "shell.execute_reply.started": "2022-02-10T23:36:00.592457Z"
    },
    "executionInfo": {
     "elapsed": 292535,
     "status": "ok",
     "timestamp": 1644563782768,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "7vyJuNQzUn3D",
    "outputId": "ac379d86-f63e-4d8d-ad98-be01d2394446",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01 | Time: 0m 29s\n",
      "\tTrain Loss: 5.056 | Train PPL: 156.950\n",
      "\t Val. Loss: 5.016 |  Val. PPL: 150.750\n",
      "Epoch: 02 | Time: 0m 28s\n",
      "\tTrain Loss: 4.382 | Train PPL:  80.004\n",
      "\t Val. Loss: 4.769 |  Val. PPL: 117.819\n",
      "Epoch: 03 | Time: 0m 29s\n",
      "\tTrain Loss: 4.079 | Train PPL:  59.100\n",
      "\t Val. Loss: 4.570 |  Val. PPL:  96.531\n",
      "Epoch: 04 | Time: 0m 29s\n",
      "\tTrain Loss: 3.777 | Train PPL:  43.678\n",
      "\t Val. Loss: 4.303 |  Val. PPL:  73.928\n",
      "Epoch: 05 | Time: 0m 29s\n",
      "\tTrain Loss: 3.479 | Train PPL:  32.423\n",
      "\t Val. Loss: 4.100 |  Val. PPL:  60.358\n",
      "Epoch: 06 | Time: 0m 28s\n",
      "\tTrain Loss: 3.202 | Train PPL:  24.576\n",
      "\t Val. Loss: 3.961 |  Val. PPL:  52.490\n",
      "Epoch: 07 | Time: 0m 29s\n",
      "\tTrain Loss: 2.950 | Train PPL:  19.114\n",
      "\t Val. Loss: 3.904 |  Val. PPL:  49.606\n",
      "Epoch: 08 | Time: 0m 29s\n",
      "\tTrain Loss: 2.724 | Train PPL:  15.236\n",
      "\t Val. Loss: 3.841 |  Val. PPL:  46.594\n",
      "Epoch: 09 | Time: 0m 29s\n",
      "\tTrain Loss: 2.522 | Train PPL:  12.457\n",
      "\t Val. Loss: 3.772 |  Val. PPL:  43.475\n",
      "Epoch: 10 | Time: 0m 29s\n",
      "\tTrain Loss: 2.329 | Train PPL:  10.263\n",
      "\t Val. Loss: 3.696 |  Val. PPL:  40.267\n"
     ]
    }
   ],
   "source": [
    "N_EPOCHS = 10\n",
    "CLIP = 1\n",
    "\n",
    "best_valid_loss = float('inf')\n",
    "\n",
    "for epoch in range(N_EPOCHS):\n",
    "    \n",
    "    start_time = time.time()\n",
    "    \n",
    "    train_loss = train(model, train_loader, optimizer, criterion, CLIP)\n",
    "    valid_loss = evaluate(model, valid_loader, criterion)\n",
    "    \n",
    "    end_time = time.time()\n",
    "    \n",
    "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
    "    \n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        torch.save(model.state_dict(), 'tut1-model_1a.pt')\n",
    "    \n",
    "    print(f'Epoch: {epoch+1:02} | Time: {epoch_mins}m {epoch_secs}s')\n",
    "    print(f'\\tTrain Loss: {train_loss:.3f} | Train PPL: {math.exp(train_loss):7.3f}')\n",
    "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. PPL: {math.exp(valid_loss):7.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UeXMknRvUn3D"
   },
   "source": [
    "We'll load the parameters (`state_dict`) that gave our model the best validation loss and run it the model on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2022-02-10T23:40:47.644858Z",
     "iopub.status.busy": "2022-02-10T23:40:47.644735Z",
     "iopub.status.idle": "2022-02-10T23:40:48.025228Z",
     "shell.execute_reply": "2022-02-10T23:40:48.024835Z",
     "shell.execute_reply.started": "2022-02-10T23:40:47.644844Z"
    },
    "executionInfo": {
     "elapsed": 638,
     "status": "ok",
     "timestamp": 1644563889501,
     "user": {
      "displayName": "Shaannoor Mann",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
      "userId": "02520257695567980696"
     },
     "user_tz": 360
    },
    "id": "YFz9w3KmUn3D",
    "outputId": "61a64a2b-fc6e-4a97-de33-5f8cc4cbfd5f",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| Test Loss: 3.677 | Test PPL:  39.534 |\n"
     ]
    }
   ],
   "source": [
    "model.load_state_dict(torch.load('tut1-model_1a.pt'))\n",
    "\n",
    "test_loss = evaluate(model, test_loader, criterion)\n",
    "\n",
    "print(f'| Test Loss: {test_loss:.3f} | Test PPL: {math.exp(test_loss):7.3f} |')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3d61-wekUn3D"
   },
   "source": [
    "In the following notebook we'll implement a model that achieves improved test perplexity, but only uses a single layer in the encoder and the decoder."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "2b - Rev_Sequence to Sequence Learning with Neural Networks.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "pt10",
   "language": "python",
   "name": "pt10"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}