Finetuning Bert on MRPC Corpus using FastAI.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Github Repo: https://github.com/deepklarity/fastai-bert-finetuning.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This notebook requires Python >= 3.6 and fastai==1.0.52"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objective\n",
    "\n",
    "In this notebook we will finetune pre-trained BERT model on The Microsoft Research Paraphrase Corpus (MRPC). MRPC is a paraphrase identification dataset, where systems aim to identify if two sentences are paraphrases of each other."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BERT\n",
    "\n",
    "### Overview: \n",
    "1. Trained on BookCorpus and English Wikipedia (800M and 2,500M words respectively).\n",
    "2. Training time approx. about a week using 64 GPUs.\n",
    "3. State-Of-The-Art (SOTA) results on SQuAD v1.1 and all 9 GLUE benchmark tasks.\n",
    "\n",
    "### Architecture:\n",
    "\n",
    "#### Embedding Layers\n",
    "\n",
    "<img src=\"https://i2.wp.com/mlexplained.com/wp-content/uploads/2019/01/Screen-Shot-2019-01-04-at-3.32.48-PM.png?w=1128\">\n",
    "\n",
    "- Token embeddings, Segment embeddings and Position embeddings. \n",
    "- [SEP] token to mark the end of a sentence.\n",
    "- [CLS] token at the beginning of the input sequence, to be used only if classifying.\n",
    "\n",
    "- The convention in BERT is:\n",
    "\n",
    "(a) For sequence pairs:\n",
    " tokens:   [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]\n",
    " \n",
    " type_ids:   0   0  0    0    0     0      0   0    1  1  1   1  1   1\n",
    " \n",
    "(b) For single sequences:\n",
    " tokens:   [CLS] the dog is hairy . [SEP]\n",
    " \n",
    " type_ids:   0   0   0   0  0     0   0\n",
    "\n",
    "Where \"type_ids\" are used to indicate whether this is the first\n",
    "sequence or the second sequence. The embedding vectors for `type=0` and\n",
    "`type=1` were learned during pre-training and are added to the wordpiece\n",
    "embedding vector (and position vector). This is not *strictly* necessary\n",
    "since the [SEP] token unambigiously separates the sequences, but it makes\n",
    "it easier for the model to learn the concept of sequences.\n",
    "        \n",
    "- For classification tasks, the first vector (corresponding to [CLS]) is\n",
    "used as as the \"sentence vector\". The  first  token  of  every  sequence  is  always  the  special  classification  embedding([CLS]). The  final  hidden  state  (i.e.,  out-put of Transformer) corresponding to this token  is  used  as  the  aggregate  sequence  rep-resentation for classification tasks. For non-classification tasks, this vector is ignored.\n",
    "\n",
    "**Note that this only makes sense because the entire model is fine-tuned.** \n",
    "\n",
    "BERT is bidirectional, the [CLS] is encoded including all representative information of all tokens through the multi-layer encoding procedure. The representation of [CLS] is individual in different sentences. [CLS] is later fine-tuned on the downstream task. Only after fine-tuning, [CLS] aka the first token can be a meaningful representation of the whole sentence. [Link](https://github.com/google-research/bert/issues/196)\n",
    "\n",
    "#### Encoders\n",
    "\n",
    "<img src=\"http://jalammar.github.io/images/bert-base-bert-large-encoders.png\">\n",
    "\n",
    "BERT BASE (12 encoders) and BERT LARGE (24 encoders)\n",
    "\n",
    "#### Training\n",
    "\n",
    "1. **Masked LM (MLM)**: Before feeding word sequences into BERT, 15% of the words in each sequence are replaced with a [MASK] token.\n",
    "\n",
    "    Training the language model in BERT is done by predicting 15% of the tokens in the input, that were randomly picked. \n",
    "\n",
    "    These tokens are pre-processed as follows — 80% are replaced with a “[MASK]” token, 10% with a random word, and 10% use the original word. The model then attempts to predict the original value of the masked words, based on the context provided by the other, non-masked, words in the sequence.\n",
    "\n",
    "2. **Next Sentence Prediction (NSP)**: In the BERT training process, the model receives pairs of sentences as input and learns to predict if the second sentence in the pair is the subsequent sentence in the original document. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-trained Model\n",
    "\n",
    "We will use op-for-op PyTorch reimplementation of Google's BERT model provided by pytorch-pretrained-BERT library. Refer the Github repo for more info: https://github.com/huggingface/pytorch-pretrained-BERT "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "import pandas as pd\n",
    "from pathlib import Path\n",
    "import matplotlib.cm as cm\n",
    "from fastai import *\n",
    "from fastai.text import *\n",
    "from fastai.callbacks import *\n",
    "from fastai.metrics import *\n",
    "import utils, bert_fastai, bert_helper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Seed random generators so all model runs are reproducible\n",
    "\n",
    "utils.seed_everything()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using BERT with fastai\n",
    "\n",
    "There are three things we need to be careful of when using BERT with fastai:\n",
    "\n",
    "   1. BERT uses its own wordpiece tokenizer.\n",
    "    \n",
    "   [WordPiece](https://stackoverflow.com/questions/55382596/how-is-wordpiece-tokenization-helpful-to-effectively-deal-with-rare-words-proble/55416944#55416944) is a commonly used technique to segment words into subword-level in NLP tasks. In this approach an out of vocabulary word is progressively split into subwords and the word is then represented by a group of subwords. Since the subwords are part of the vocabulary, we have learned representations n context for these subwords and the context of the word is simply the combination of the context of the subwords. For more details regarding this approach please refer [Neural Machine Translation of Rare Words with Subword Units](https://arxiv.org/pdf/1508.07909)\n",
    "       \n",
    "   **How does this help?**\n",
    "\n",
    "  Imagine that the model sees the word walking. Unless this word occurs at least a few times in the\n",
    "training corpus, the model can't learn to deal with this word very well. However, it may have the\n",
    "words walked, walker, walks, each occurring only a few times. Without subword segmentation, all these\n",
    "words are treated as completely different words by the model.\n",
    "However, if these get segmented as walk@@ ing, walk@@ ed, etc., notice that all of them will now have\n",
    "walk@@ in common, which will occur much frequently while training, and the model might be able to\n",
    "learn more about it.\n",
    "   \n",
    "   2. BERT needs [CLS] and [SEP] tokens added to each sequence.\n",
    "    \n",
    "  For classification tasks we need to add [CLS] token before every input sequence.\n",
    "[SEP] token is used to separate sentences.\n",
    "    \n",
    "   3. BERT uses its own pre-built vocabulary.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset\n",
    "\n",
    "MRPC dataset a text file containing 5800 pairs of sentences which have been extracted from news sources on the web, along with human annotations indicating whether each pair captures a paraphrase/semantic equivalence relationship. The column named **Quality** indicates whether the sentences are similar (1) or not (0). **\"#1 String\" and \"#2 String\"** columns contain the sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://www.microsoft.com/en-us/download/details.aspx?id=52398\n",
    "# Microsoft Research Paraphrase Corpus\n",
    "TASK='MRPC'\n",
    "DATA_ROOT = Path(\".\")\n",
    "label_col = \"Quality\"\n",
    "text_cols = [\"#1 String\", \"#2 String\"]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing MRPC...\n",
      "Local MRPC data not specified, downloading data from https://dl.fbaipublicfiles.com/senteval/senteval_data/msr_paraphrase_train.txt\n",
      "\tCompleted!\n"
     ]
    }
   ],
   "source": [
    "# Execute script to download MRPC data and create train.csv and test.csv\n",
    "\n",
    "! python download_glue_data.py --data_dir='glue_data' --tasks=$TASK --test_labels=True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Quality</th>\n",
       "      <th>#1 ID</th>\n",
       "      <th>#2 ID</th>\n",
       "      <th>#1 String</th>\n",
       "      <th>#2 String</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>702876</td>\n",
       "      <td>702977</td>\n",
       "      <td>Amrozi accused his brother , whom he called \" ...</td>\n",
       "      <td>Referring to him as only \" the witness \" , Amr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>2108705</td>\n",
       "      <td>2108831</td>\n",
       "      <td>Yucaipa owned Dominick 's before selling the c...</td>\n",
       "      <td>Yucaipa bought Dominick 's in 1995 for $ 693 m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1330381</td>\n",
       "      <td>1330521</td>\n",
       "      <td>They had published an advertisement on the Int...</td>\n",
       "      <td>On June 10 , the ship 's owners had published ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>3344667</td>\n",
       "      <td>3344648</td>\n",
       "      <td>Around 0335 GMT , Tab shares were up 19 cents ...</td>\n",
       "      <td>Tab shares jumped 20 cents , or 4.6 % , to set...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1236820</td>\n",
       "      <td>1236712</td>\n",
       "      <td>The stock rose $ 2.11 , or about 11 percent , ...</td>\n",
       "      <td>PG &amp; E Corp. shares jumped $ 1.63 or 8 percent...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Quality    #1 ID    #2 ID  \\\n",
       "0        1   702876   702977   \n",
       "1        0  2108705  2108831   \n",
       "2        1  1330381  1330521   \n",
       "3        0  3344667  3344648   \n",
       "4        1  1236820  1236712   \n",
       "\n",
       "                                           #1 String  \\\n",
       "0  Amrozi accused his brother , whom he called \" ...   \n",
       "1  Yucaipa owned Dominick 's before selling the c...   \n",
       "2  They had published an advertisement on the Int...   \n",
       "3  Around 0335 GMT , Tab shares were up 19 cents ...   \n",
       "4  The stock rose $ 2.11 , or about 11 percent , ...   \n",
       "\n",
       "                                           #2 String  \n",
       "0  Referring to him as only \" the witness \" , Amr...  \n",
       "1  Yucaipa bought Dominick 's in 1995 for $ 693 m...  \n",
       "2  On June 10 , the ship 's owners had published ...  \n",
       "3  Tab shares jumped 20 cents , or 4.6 % , to set...  \n",
       "4  PG & E Corp. shares jumped $ 1.63 or 8 percent...  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df = pd.read_csv(DATA_ROOT / \"glue_data\" / \"MRPC\" / \"train.tsv\", sep = '\\t', quoting=csv.QUOTE_NONE)\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>Quality</th>\n",
       "      <th>#1 ID</th>\n",
       "      <th>#2 ID</th>\n",
       "      <th>#1 String</th>\n",
       "      <th>#2 String</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1089874</td>\n",
       "      <td>1089925</td>\n",
       "      <td>PCCW 's chief operating officer , Mike Butcher...</td>\n",
       "      <td>Current Chief Operating Officer Mike Butcher a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3019446</td>\n",
       "      <td>3019327</td>\n",
       "      <td>The world 's two largest automakers said their...</td>\n",
       "      <td>Domestic sales at both GM and No. 2 Ford Motor...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1945605</td>\n",
       "      <td>1945824</td>\n",
       "      <td>According to the federal Centers for Disease C...</td>\n",
       "      <td>The Centers for Disease Control and Prevention...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1430402</td>\n",
       "      <td>1430329</td>\n",
       "      <td>A tropical storm rapidly developed in the Gulf...</td>\n",
       "      <td>A tropical storm rapidly developed in the Gulf...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>3354381</td>\n",
       "      <td>3354396</td>\n",
       "      <td>The company didn 't detail the costs of the re...</td>\n",
       "      <td>But company officials expect the costs of the ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   index  Quality    #1 ID    #2 ID  \\\n",
       "0      0        1  1089874  1089925   \n",
       "1      1        1  3019446  3019327   \n",
       "2      2        1  1945605  1945824   \n",
       "3      3        0  1430402  1430329   \n",
       "4      4        0  3354381  3354396   \n",
       "\n",
       "                                           #1 String  \\\n",
       "0  PCCW 's chief operating officer , Mike Butcher...   \n",
       "1  The world 's two largest automakers said their...   \n",
       "2  According to the federal Centers for Disease C...   \n",
       "3  A tropical storm rapidly developed in the Gulf...   \n",
       "4  The company didn 't detail the costs of the re...   \n",
       "\n",
       "                                           #2 String  \n",
       "0  Current Chief Operating Officer Mike Butcher a...  \n",
       "1  Domestic sales at both GM and No. 2 Ford Motor...  \n",
       "2  The Centers for Disease Control and Prevention...  \n",
       "3  A tropical storm rapidly developed in the Gulf...  \n",
       "4  But company officials expect the costs of the ...  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df = pd.read_csv(DATA_ROOT / \"glue_data\" / \"MRPC\" / \"test.tsv\", sep = '\\t', quoting=csv.QUOTE_NONE)\n",
    "test_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Training records=3668\n",
      "Number of Test records=1725\n"
     ]
    }
   ],
   "source": [
    "print(f\"Number of Training records={len(train_df)}\")\n",
    "print(f\"Number of Test records={len(test_df)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def sample_sentences(quality, n=5):\n",
    "    ctr = 0\n",
    "    for row in train_df.query(f'Quality=={quality}').itertuples():\n",
    "        print(f\"1. {row[4]}\\n2. {row[5]}\")\n",
    "        print(\"=\"*100)\n",
    "        ctr += 1\n",
    "        if n==ctr:\n",
    "            break\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Yucaipa owned Dominick 's before selling the chain to Safeway in 1998 for $ 2.5 billion .\n",
      "2. Yucaipa bought Dominick 's in 1995 for $ 693 million and sold it to Safeway for $ 1.8 billion in 1998 .\n",
      "====================================================================================================\n",
      "1. Around 0335 GMT , Tab shares were up 19 cents , or 4.4 % , at A $ 4.56 , having earlier set a record high of A $ 4.57 .\n",
      "2. Tab shares jumped 20 cents , or 4.6 % , to set a record closing high at A $ 4.57 .\n",
      "====================================================================================================\n",
      "1. The Nasdaq had a weekly gain of 17.27 , or 1.2 percent , closing at 1,520.15 on Friday .\n",
      "2. The tech-laced Nasdaq Composite .IXIC rallied 30.46 points , or 2.04 percent , to 1,520.15 .\n",
      "====================================================================================================\n",
      "1. That compared with $ 35.18 million , or 24 cents per share , in the year-ago period .\n",
      "2. Earnings were affected by a non-recurring $ 8 million tax benefit in the year-ago period .\n",
      "====================================================================================================\n",
      "1. Shares of Genentech , a much larger company with several products on the market , rose more than 2 percent .\n",
      "2. Shares of Xoma fell 16 percent in early trade , while shares of Genentech , a much larger company with several products on the market , were up 2 percent .\n",
      "====================================================================================================\n"
     ]
    }
   ],
   "source": [
    "# Different sentences samples            \n",
    "sample_sentences(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Amrozi accused his brother , whom he called \" the witness \" , of deliberately distorting his evidence .\n",
      "2. Referring to him as only \" the witness \" , Amrozi accused his brother of deliberately distorting his evidence .\n",
      "====================================================================================================\n",
      "1. They had published an advertisement on the Internet on June 10 , offering the cargo for sale , he added .\n",
      "2. On June 10 , the ship 's owners had published an advertisement on the Internet , offering the explosives for sale .\n",
      "====================================================================================================\n",
      "1. The stock rose $ 2.11 , or about 11 percent , to close Friday at $ 21.51 on the New York Stock Exchange .\n",
      "2. PG & E Corp. shares jumped $ 1.63 or 8 percent to $ 21.03 on the New York Stock Exchange on Friday .\n",
      "====================================================================================================\n",
      "1. Revenue in the first quarter of the year dropped 15 percent from the same period a year earlier .\n",
      "2. With the scandal hanging over Stewart 's company , revenue the first quarter of the year dropped 15 percent from the same period a year earlier .\n",
      "====================================================================================================\n",
      "1. The DVD-CCA then appealed to the state Supreme Court .\n",
      "2. The DVD CCA appealed that decision to the U.S. Supreme Court .\n",
      "====================================================================================================\n"
     ]
    }
   ],
   "source": [
    "# Similar sentences samples            \n",
    "sample_sentences(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup code for training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "# Specify BERT configs\n",
    "\n",
    "config = utils.Config(\n",
    "    bert_model_name=\"bert-base-uncased\",\n",
    "    num_labels=2, # 0 or 1\n",
    "    max_lr=2e-5,\n",
    "    epochs=3,\n",
    "    batch_size=32,\n",
    "    max_seq_len=128\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Refer [A Tutorial to Fine-Tuning BERT with Fast AI](http://mlexplained.com/2019/05/13/a-tutorial-to-fine-tuning-bert-with-fast-ai/) for deeper understanding of the code changes required for FastAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "fastai_tokenizer = bert_fastai.FastAITokenizer(model_name=config.bert_model_name, max_seq_len=config.max_seq_len)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can build the databunch using the tokenizer we build above. Notice we're passing the `include_bos=False` and `include_eos=False` options. This is to prevent fastai from adding its own Start-Of-Sentence (SOS)/End-Of-Sentence (EOS) tokens that will interfere with BERT's SOS/EOS tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "databunch = TextDataBunch.from_df(\".\", train_df=train_df, valid_df=test_df,\n",
    "                  tokenizer=fastai_tokenizer.bert_tokenizer(),\n",
    "                  vocab=fastai_tokenizer.fastai_bert_vocab(),\n",
    "                  include_bos=False,\n",
    "                  include_eos=False,\n",
    "                  text_cols=text_cols,\n",
    "                  label_cols=label_col,\n",
    "                  bs=config.batch_size,\n",
    "                  collate_fn=partial(pad_collate, pad_first=False, pad_idx=0),\n",
    "             )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original==> Amrozi accused his brother , whom he called \" the witness \" , of deliberately distorting his evidence .,Referring to him as only \" the witness \" , Amrozi accused his brother of deliberately distorting his evidence .\n",
      "\n",
      "Tokenized==>. [CLS] am ##ro ##zi accused his brother , whom he called \" the witness \" , of deliberately di ##stor ##ting his evidence . referring to him as only \" the witness \" , am ##ro ##zi accused his brother of deliberately di ##stor ##ting his evidence . [SEP]\n",
      "====================================================================================================\n",
      "Original==> Yucaipa owned Dominick 's before selling the chain to Safeway in 1998 for $ 2.5 billion .,Yucaipa bought Dominick 's in 1995 for $ 693 million and sold it to Safeway for $ 1.8 billion in 1998 .\n",
      "\n",
      "Tokenized==>. [CLS] yu ##ca ##ip ##a owned dominic ##k ' s before selling the chain to safe ##way in 1998 for $ 2 . 5 billion . yu ##ca ##ip ##a bought dominic ##k ' s in 1995 for $ 69 ##3 million and sold it to safe ##way for $ 1 . 8 billion in 1998 . [SEP]\n",
      "====================================================================================================\n",
      "Original==> They had published an advertisement on the Internet on June 10 , offering the cargo for sale , he added .,On June 10 , the ship 's owners had published an advertisement on the Internet , offering the explosives for sale .\n",
      "\n",
      "Tokenized==>. [CLS] they had published an advertisement on the internet on june 10 , offering the cargo for sale , he added . on june 10 , the ship ' s owners had published an advertisement on the internet , offering the explosives for sale . [SEP]\n",
      "====================================================================================================\n",
      "Original==> Around 0335 GMT , Tab shares were up 19 cents , or 4.4 % , at A $ 4.56 , having earlier set a record high of A $ 4.57 .,Tab shares jumped 20 cents , or 4.6 % , to set a record closing high at A $ 4.57 .\n",
      "\n",
      "Tokenized==>. [CLS] around 03 ##35 gm ##t , tab shares were up 19 cents , or 4 . 4 % , at a $ 4 . 56 , having earlier set a record high of a $ 4 . 57 . tab shares jumped 20 cents , or 4 . 6 % , to set a record closing high at a $ 4 . 57 . [SEP]\n",
      "====================================================================================================\n",
      "Original==> The stock rose $ 2.11 , or about 11 percent , to close Friday at $ 21.51 on the New York Stock Exchange .,PG & E Corp. shares jumped $ 1.63 or 8 percent to $ 21.03 on the New York Stock Exchange on Friday .\n",
      "\n",
      "Tokenized==>. [CLS] the stock rose $ 2 . 11 , or about 11 percent , to close friday at $ 21 . 51 on the new york stock exchange . pg & e corp . shares jumped $ 1 . 63 or 8 percent to $ 21 . 03 on the new york stock exchange on friday . [SEP]\n",
      "====================================================================================================\n"
     ]
    }
   ],
   "source": [
    "# Show wordpiece tokenized data\n",
    "\n",
    "for i in range(5): \n",
    "    print(f\"Original==> {train_df.loc[i][text_cols[0]]},{train_df.loc[i][text_cols[1]]}\\n\\nTokenized==>. {databunch.x[i]}\")\n",
    "    print(\"=\"*100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now with the data in place, we will prepare the model. Again, the pytorch-pretrained-bert package gives us a sequence classifier based on BERT straight out of the box. We also build FastAI `Learner`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pytorch_pretrained_bert.modeling import BertConfig, BertForSequenceClassification\n",
    "\n",
    "\n",
    "bert_model = BertForSequenceClassification.from_pretrained(\n",
    "    config.bert_model_name, num_labels=config.num_labels)\n",
    "\n",
    "learner = bert_fastai.BertLearner(databunch,\n",
    "                                  bert_model,\n",
    "                                  metrics=[accuracy])\n",
    "\n",
    "learner.callbacks.append(ShowGraph(learner))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's print the accuracy and f1_score of the pre-trained model on this dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy=0.33855071663856506, f1_score=0.013828867761452032\n"
     ]
    }
   ],
   "source": [
    "preds, pred_values, true_labels = learner.get_predictions()\n",
    "learner.print_metrics(preds, pred_values, true_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interpret the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "txt_ci = TextClassificationInterpretation.from_learner(learner)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Index</th>\n",
       "      <th>Original Text</th>\n",
       "      <th>Tokenized</th>\n",
       "      <th>Prediction</th>\n",
       "      <th>Actual</th>\n",
       "      <th>Loss</th>\n",
       "      <th>Probability</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>236</td>\n",
       "      <td>Results of the 2001 Aboriginal Peoples Survey released yesterday by Statistics Canada suggest living standards have improved but still lag for those off reserves .,The 2001 Aboriginal Peoples Survey released Wednesday by Statistics Canada says living standards have improved but still lag for the Inuit and those who leave their often impoverished reserves .</td>\n",
       "      <td>[CLS] results of the 2001 aboriginal peoples survey released yesterday by statistics canada suggest living standards have improved but still la ##g for those off reserves . the 2001 aboriginal peoples survey released wednesday by statistics canada says living standards have improved but still la ##g for the inuit and those who leave their often impoverished reserves . [SEP]</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.37</td>\n",
       "      <td>0.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1693</td>\n",
       "      <td>The new orders index rose to 52.2 percent from 51.9 percent in May .,The New Orders Index rose by 0.3 percentage points from 51.9 percent in May to 52.2 percent in June .</td>\n",
       "      <td>[CLS] the new orders index rose to 52 . 2 percent from 51 . 9 percent in may . the new orders index rose by 0 . 3 percentage points from 51 . 9 percent in may to 52 . 2 percent in june . [SEP]</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.37</td>\n",
       "      <td>0.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>980</td>\n",
       "      <td>The two companies said PowderJect 's strong U.S. position would complement Chiron 's European presence .,PowderJect 's strong U.S. position will complement Chiron 's European presence , analysts said .</td>\n",
       "      <td>[CLS] the two companies said powder ##ject ' s strong u . s . position would complement chi ##ron ' s european presence . powder ##ject ' s strong u . s . position will complement chi ##ron ' s european presence , analysts said . [SEP]</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.36</td>\n",
       "      <td>0.26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1487</td>\n",
       "      <td>A Global Crossing spokeswoman and a Pentagon spokesman declined to comment .,A Global Crossing representative had no immediate comment .</td>\n",
       "      <td>[CLS] a global crossing spoke ##sw ##oman and a pentagon spokesman declined to comment . a global crossing representative had no immediate comment . [SEP]</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.35</td>\n",
       "      <td>0.74</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1568</td>\n",
       "      <td>Another shooting linked to the spree occurred Nov. 11 at Hamilton Central Elementary in Obetz , about two miles from the freeway .,The latest shooting linked to the spree was a Nov. 11 shooting at Hamilton Township Elementary School in Obetz , about two miles from the freeway .</td>\n",
       "      <td>[CLS] another shooting linked to the sp ##ree occurred nov . 11 at hamilton central elementary in obe ##tz , about two miles from the freeway . the latest shooting linked to the sp ##ree was a nov . 11 shooting at hamilton township elementary school in obe ##tz , about two miles from the freeway . [SEP]</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.35</td>\n",
       "      <td>0.26</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "utils.custom_show_top_losses(txt_ci, test_df, text_cols, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.610131</td>\n",
       "      <td>0.565410</td>\n",
       "      <td>0.710145</td>\n",
       "      <td>00:27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.473887</td>\n",
       "      <td>0.412696</td>\n",
       "      <td>0.817391</td>\n",
       "      <td>00:26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.329955</td>\n",
       "      <td>0.408737</td>\n",
       "      <td>0.821449</td>\n",
       "      <td>00:27</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VPW9//HXJ5N9JStLQkjCmrCvgqJSAUVswQ1Bu+i9rdxardXeei9t789aW6u3vbWtLbWlim3dELGt2GJxQ0UqCsgW9gCBJGxJSAIJ2Sbz+f0xIwTMMpBJZjJ8no9HHsw5c3LO5zDwzjffc873K6qKMcaY4BLi7wKMMcb4noW7McYEIQt3Y4wJQhbuxhgThCzcjTEmCFm4G2NMELJwN8aYIGThbowxQcjC3RhjglCovw6ckpKiUUm9qG1sorahCafL/aRsUkw46T2i/FWWMcYEtA0bNpSpamp72/kt3LOysiib9kNigTiBH80exqaiSpZtKOaK0elMHpjCxJxk7ntpE2MyE/nvGYMREX+Va4wxAUFEDniznd/CHaBvUhRFx2t58PN5fGliP26dkEmD08U/tx3hLxtLTm/38f7jxEY4qKptJDo8lKMn6piQncQNo9Mt8I0xpgVehbuIzAB+BTiAp1T1sXPezwT+BPTwbLNAVVe0t9/y6ga+Ojmbf7ssGwBHiPDEraNRVTYcqOAfWw8zMSeZ1zYf4v/e2H36+8JDQ1iyroijJ+q5a0p/L0/VGGMuHu2Gu4g4gIXAdKAYWCciy1V1e7PN/gdYqqpPikgesALIamu/LlVONTSREhvR0jEZl5XEuKwkAKYMTiUuMowJ2YlcOSiNmAgH33juExauKmDu+L4kxYR7ebrGGHNx8KblPgEoUNV9ACKyBJgNNA93BeI9rxOAQ+3t1NnkvoCaHNt+MEeEOnj0xuFnrVtw7RBm/Go18/+8nqfvGE9CVFj7Z2KM6dYaGxspLi6mrq7O36V0usjISDIyMggLu7Bs8ybc04GiZsvFwCXnbPMQ8IaIfBOIAaa1tCMRmQ/MB+jdtx/hQIoX4d6SgT3jeGLeaO57aSNzf/8hz371ElLjPvtbgDEmeBQXFxMXF0dWVlZQX29TVcrLyykuLiY7O/uC9uGr+9xvBf6oqhnATOBZEfnMvlV1kaqOU9VxsfEJACTHXHggXzeiN8/cMYH9ZTX817LN2MQjxgS3uro6kpOTgzrYwd01nZyc3KHfULwJ9xKgb7PlDM+65r4KLAVQ1Q+BSCClrZ02nUe3TFsmD0xhwbVDWLWrlP/3aj71zqYO7c8YE9iCPdg/1dHz9Cbc1wEDRSRbRMKBecDyc7Y5CEz1FJSLO9xL29ppo+ehpZYuqJ6v2ydl8bXJ2Ty39iC/frugw/szxpjurt1wV1UncA+wEtiB+66YbSLysIjM8mz2n8CdIrIZeBG4Q9vpI2lyKbERoUSGOTp2BkBIiPA/n8/jCyP78NQH+zhUWdvhfRpjzLkqKyv57W9/e97fN3PmTCorKzuhotZ51eeuqitUdZCq9lfVRzzrHlTV5Z7X21X1MlUdqaqjVPWN9vbpbHJ1uEvmXP91zWBCRPjG859Q12jdM8YY32ot3J1OZ5vft2LFCnr06NFZZbXIbwOH1Ttd9IqP9Ok++yZF8/gtI9lUVMkTb+/x6b6NMWbBggXs3buXUaNGMX78eC6//HJmzZpFXl4eANdffz1jx45l6NChLFq06PT3ZWVlUVZWRmFhIbm5udx5550MHTqUq6++mtrazulp8NvwA/VOF7m949vf8DzNGNabm8dmsOj9fcwc3pth6Qk+P4Yxxv9++No2th864dN95vWJ5wdfGNrq+4899hj5+fls2rSJd999l+uuu478/PzTtysuXryYpKQkamtrGT9+PDfddBPJycln7WPPnj28+OKL/OEPf+CWW27hlVde4Utf+pJPzwP82HJ3qZLXx/fhDvD9mbmkxkVw1/MbOFHX2CnHMMaYCRMmnHUf+hNPPMHIkSOZOHEiRUVF7Nnz2R6E7OxsRo0aBcDYsWMpLCzslNr8OnBYXie03AESY8L5zW2juenJD/njmkLunTqwU45jjPGftlrYXSUmJub063fffZe33nqLDz/8kOjoaKZMmdLifeoREWfuEHQ4HJ3WLeO3lrsAA9JiO23/Y/slcdWQNP74r0JqG+ziqjGm4+Li4jh58mSL71VVVZGYmEh0dDQ7d+5k7dq1XVzd2fwW7hGhDp/cBtmWu6b053hNA4ve39epxzHGXBySk5O57LLLGDZsGA888MBZ782YMQOn00lubi4LFixg4sSJfqrSTfz1yP6wkaM1f/PGTj/OPS98whvbj7L49vFMHtjmQ7PGmAC3Y8cOcnNz/V1Gl2npfEVkg6qOa+97/dZy7+xW+6ce/EIe/ZKi+fLij/jFm7tpctn4M8aY4Bf0E2SnxUXy6j2XccPodH719h5uX/yxPeBkjAl6QR/uANHhoTx+yygevXE4HxSU8cyaQn+XZIwxneqiCPdP3Tohk6lD0vjtuwWUVdf7uxxjjOk0F1W4A3x35hDqnS4eeHmz9b8bY4LWRRfuA9Li+H/X5bJqVyn3LtmIywLeGBOELrpwB/jypCzumzaQf2w5zIaDFf4uxxgTpGJj3Q9qHjp0iJtvvrnFbaZMmcL69et9fuyLMtwB/n1yNmEO4a0dR/1dijEmyPXp04dly5Z16TEv2nCPjwxjYk4yb223cDfGeGfBggUsXLjw9PJDDz3Ej3/8Y6ZOncqYMWMYPnw4r7766me+r7CwkGHDhgFQW1vLvHnzyM3N5YYbbgi+IX8DwYxhvfj+X/NZuKqAuz83wN/lGGPOx+sL4MhW3+6z13C49rFW3547dy733Xcfd999NwBLly5l5cqV3HvvvcTHx1NWVsbEiROZNWtWq3OgPvnkk0RHR7Njxw62bNnCmDFjfHsOHl613EVkhojsEpECEVnQwvu/EJFNnq/dItK180ldoHnjM5k9qg8/W7mLD/eW+7scY0yAGz16NMeOHePQoUNs3ryZxMREevXqxfe+9z1GjBjBtGnTKCkp4ejR1nsE3n///dPjt48YMYIRI0Z0Sq3tttxFxAEsBKYDxcA6EVmuqts/3UZV72+2/TeB0Z1Qq885QoT/vWkEGw5U8OCr+bz2zcldNiyCMaaD2mhhd6Y5c+awbNkyjhw5wty5c3n++ecpLS1lw4YNhIWFkZWV1eJQv13Nm5b7BKBAVfepagOwBJjdxva34p4ku1uIDHPwyA3DKSit5o5nPubR13ew60jLQ3oaY8zcuXNZsmQJy5YtY86cOVRVVZGWlkZYWBirVq3iwIEDbX7/FVdcwQsvvABAfn4+W7Zs6ZQ6vQn3dKCo2XKxZ91niEg/IBt4p+OldZ0rB6XyvWtzyS85weIP9nP9wjWU2xOsxpgWDB06lJMnT5Kenk7v3r354he/yPr16xk+fDh//vOfGTJkSJvff9ddd1FdXU1ubi4PPvggY8eO7ZQ6fX1BdR6wTFVbHJlLROYD8wEyMzN9fOiOufOKHO68IoedR04w45ereXlDMV+/sr+/yzLGBKCtW89cyE1JSeHDDz9scbvq6mrAPUF2fn4+AFFRUSxZsqTTa/Qm3EuAvs2WMzzrWjIPuLu1HanqImARwLhx4wLy0dAhveK5JDuJX7+9h60lVYzJTKS6zsm3pp2Zqq+ipoGGJhdpcRG41N13b4wxgcSbcF8HDBSRbNyhPg+47dyNRGQIkAi0/COsG3nkhmH85p0CXs8/wj+2HAagtLqO1NhI5l+Rw02/+xeHK+tIjA4jIzGa5++8hDDHRfvIgDEmALUb7qrqFJF7gJWAA1isqttE5GFgvaou92w6D1ii/prayYcGpMXxy3mj+cbRk+w5Ws1PV+7kubUHAfjntiPsK61hTGYPauqb+LjwOA++uo0ffCHP7rQxpguoaqv3kAeTjkapV33uqroCWHHOugfPWX6oQ5UEoEE94xjUM46c1BgKjlWzdH0RGw9W8v2Zudx5RQ4Aj/xjO39YvZ/Cshqe+9ol1kVjTCeKjIykvLyc5OTkoA54VaW8vJzIyMgL3off5lAdN26cdsZgOZ3J2eSiSZWI0LNb6C9+fJDv/mUrVw1JY1h6Apf2T2ZiTrKfqjQmeDU2NlJcXBwQ95F3tsjISDIyMggLCztrvbdzqF7Uww+cr1BHSIt/YfPG96Xo+CmWbShm1a5jPPH2Hq7O68ndnxvAiIyEoG5hGNOVwsLCyM7O9ncZ3YK13H2srrGJpz/Yz8JVBZxqaGJ4egJfntSPKYNTSYtz/4p1sfQZGmN8z9uWu4V7J6k81cDr+Ud44u09HK6qwxEiTB2SRlxkGH/fcohLcpJ55Pph9E2K9nepxphuxMI9QLhcyp5j1fzlk2L+urGEeqeLiTlJfLCnDJfCrJF9uO2STBQ4UF7DlEFpJESHtbtfY8zFycI9wJVU1vLzlbt4Y/tRquudp9cnRocxJjMREchKjuFLE/uRlRLjx0qNMYHEwr2bqKptZPnmQ6TGhpMQFc5zaw+wr6wGVT39549mD2PehMAarsEY4x92t0w3kRAVxpcn9ju9PKn/mVsoj52o4zvLtrDgL1vZV1bD/dMGEREaQk2Dk7hI67oxxrTOWu4Bztnk4qHXtvHc2oNEhoUQFuIO9+tHpzMttycTspNIiY3wd5nGmC5i3TJB5qN95azcdpSGJveAm69sKKG2sYlwRwhfn9KfyQNSyO0dR2xEKCfrncSEh37madmTdY00NimJ0WF2K6Yx3ZSFe5BrbHKRX1LF79/bxz+3HQEgNERIiAqjvKaBtLgIJg9M4eP9x4mLDGNon3he33qYOqeL2IhQMpOi+fktIxnUM87PZ2KMOR8W7heRI1V17DhygnX7j3O4qo7BveL428YS9pfVMC23J/vKajhUWcvnBqfSKyGK8up63tpxlIpTjcwa2Yef3Dic2Ai7/GJMd2DhfpFTVRqaXJ8ZB+dTZdX1PLNmP0++u5ec1Fh+8IU8osMdDEiNs/vsjQlgFu7GK//aW8Y3X9hIeU0DACmxEfzmttE28JkxAcrC3XjtRF0jmw5WUlPv5P/e2EVxRS2/uW0M0/N6+rs0Y8w57D5347X4yDCuGJQKwCU5ydy++GPu/PN60ntEMWdcBt+aOtDurjGmm7FwN2dJignn5a9P4rfv7uWTAxX88q09rN5TxtxxfUmKCWdS/2Ri7OKrMQHP/peaz4gMc/Dt6YNQVZ5de4A//quQ/3plCwATspO4b+pARvTtQUVNA06X8sGeUpJiIpie15PwUJtL1phA4FWfu4jMAH6Few7Vp1T1sRa2uQV4CFBgs6p+ZhLt5qzPvftQVbYdOsH6wuM89Np2AGIjQs8a8Axgel5PfvCFPDISbRhjYzqLz/rcRcQBLASmA8XAOhFZrqrbm20zEPgucJmqVohI2oWXbgKNiDAsPYFh6QmMzkzkyIk6lm86RF6feOIjQ5mYk8zqPWU8/PftvLn9KDOG9mJqbhozh/e2Lhxj/KTdlruITAIeUtVrPMvfBVDVR5tt81Ngt6o+5e2BreUefAqOVbN8UwnPrCnkZL2T+MhQJg9M4ZtXDSS3d7y/yzMmKHjbcvemgzQdKGq2XOxZ19wgYJCIrBGRtZ5unJaKmi8i60VkfWlpqReHNt3JgLRYvn31YDb/4GpeuWsSVw/txYd7y7nuidXc++JGTtQ1+rtEYy4avvqdORQYCEwBMoD3RWS4qlY230hVFwGLwN1y99GxTYAJCRHG9ktibL8kKmoa+N37e3l69X5KKmv5/ZfH2iiWxnQBb1ruJUDfZssZnnXNFQPLVbVRVfcDu3GHvbnIJcaE891rc3ni1tFsLalixi9Xc6C8xt9lGRP0vAn3dcBAEckWkXBgHrD8nG3+hrvVjoik4O6m2efDOk03N3N4b5bfcxlOl4uv/Wk9FZ7hDowxnaPdcFdVJ3APsBLYASxV1W0i8rCIzPJsthIoF5HtwCrgAVUt76yiTfc0pFc8v/3iGA4eP8VNv/sXr2woprahyd9lGROUbGwZ0+X+tbeM//lrPvvKakiJjeAPXxnL6MxEf5dlTLfgy7tljPGpS/un8PZ/XsmLd04kJsLBl576iMIy64c3xpcs3I1fiAiT+ifz4p0TcYQI33xxI6Un60+/X1Pv5HBVrR8rNKZ7s3A352/TC3DisE921adHFD+/ZRS7j55k+i/e41dv7eFQZS1ffvojpv38PVZsPcyuIyfP+p56ZxOHq2pZuq6IZ9bsZ19ptU9qMSaYWJ+7OT8VhfCrkSAhkH0ljJgLuV+AiNgO7Xb30ZM89vpO3tl57PS66HAHpxqaiApz8OiNw/n7lkOs3Xf8M2PahAhkJcfgdCnXj+rD/dMH2RDFJmjZZB2m85TtgS1LYctLUHkAwqJhyHUwYh7kTAHHhT8bV1hWw9L1RYjAzWP7kl9SxWOv76SkspbYiFBuGJ1Oz/gIEmPCGdYngdS4CJ5de4Ci46eoqm1k9Z4ycnvHc93wXswZ15ee8ZE+O21jAoGFu+l8qlD0EWxeAtv+CnWVEJMGw292t+h7jwQftKArTzWw4/BJhvSKIzEmvI1ylOfWHuDVTYdYf6ACR4gwb3xf+qfGMjU3jX7JMR2uxRh/s3A3XctZD3vecLfmd6+EpgZIGQwj58LwOdAjs0vLKSyrYfGa/Ty79gCq7rlhF31lLGPslkvTzVm4G/85dRy2v+oO+oMfutf1m+wO+txZENWjy0opOn6KYyfrufv5Tzhyoo67pvTngasHExJiffKme7JwN4GhohC2vAxblkB5ATgiYPC17m6bAdMgtPVuFl+qrnfy479vZ8m6Ii7tn8wv540iLc764033Y+FuAosqHPoENr8E+a/AqTKISoJhN7ovxGaM80n/fNslKC9+XMSP/r6d4RkJzBzWi1mj0klqox/fmEBj4W4CV1Mj7H3HfSF21wpw1kFSjrs1P+IW9+tOtHRd0ek5YQf1jOXp28fTN8mmBjTdg4W76R7qTsCO5e7++f2rAYWMCe7++aE3QnSSzw+pqqwpKOdEXSMPvLwZl8LTt4/j0gEpPj+WMb5m4W66n6pi2LrMHfTHtkNIGAyc7m7RD5oBYb7vIy+prOX2xR9zvKaB1745GWeTiz+s3sddUwaQ3iPK58czpqMs3E33pQpHtrpDfusyqD4CEQkwdLa7fz5zEoT4buSMvaXVzP7NGtJ7RHGirpHDVXVkJEbxs5tHMj4rkVCHjdJhAoeFuwkOribY/577QuyO16CxBhIyYcQcd4s+dbBPDvPGtiPc9fwnDEiN5WuXZ/P4m7s5XFVHfGQoj98yiml5PX1yHGM6ysLdBJ+GGtj5D3eLfu87oC7oPQpGzoNhN0FsWod2X9fYRGSYA4BTDU7+mX+EZ9YUsu1QFb+YO4rZo86dF96YrmfhboLbyaPuWyq3LIHDm0Ec0P8qd2t+yHUQ7pu7X041OLnjmXWsLzzOvAmZjMlM5JqhPYmLDPPJ/o05Xz4NdxGZAfwKcABPqepj57x/B/Azzkyc/RtVfaqtfVq4G585ttPTP/8yVBVBeKx7pMoRcyH7CghxdGj3NfVOHlmxgyUfH8SlEB8Zyp2X55AcG8H0vJ6UnqznZF0jI/v2ON3yN6az+CzcRcQB7AamA8W4J8y+VVW3N9vmDmCcqt7jbYEW7sbnXC44+C/3/fPbX4X6ExDX2zOQ2TzoNaxDu29scpFfUsVPVuxgXWHFZ97vnRDJM/82niG94gFocikhgg0/bHzKl+E+CXhIVa/xLH8XQFUfbbbNHVi4m0DSWAe7X3cPTbznDXA5IW3omYHM4vtc8K5VldLqeg5V1rFu/3HS4iOICHXwg+X5VJxqZOawXvSIDmfF1sO4FP5vzgimDO7Y9QBjPuXLcL8ZmKGqX/Msfxm4pHmQe8L9UaAUdyv/flUtamu/Fu6my9SUw7a/uLtuitcB4u6uGTEX8mZBRJxPDnO4qpYn3t7Dm9uPcqqhibH9Ejl2op5dR09y64RM7p8+0MazMR3W1eGeDFSrar2I/AcwV1WvamFf84H5AJmZmWMPHDhwPudkTMeV7z0z0UjFfgiN8kw0Mtd9QbYDE420pK6xiZ/+cxd/+rCQcEcIX52czfwrc4i3C7LmAnVpt8w52zuA46qa0NZ+reVu/ErV3YrfvMTdqq+tgJhU9y2VI+ZCn9E+Hchsf1kNj7+5m9c2H6JHdBjfmNKfL03sR3R46z9MGpwuTtQ1khgdjsMzRPHB8lOkJ0bhCBFKKmvpHR9pwxdfZHwZ7qG4u1qm4r4bZh1wm6pua7ZNb1U97Hl9A/Dfqjqxrf1auJuA4WyAgjfdQb/7n56JRga5BzEbfgsk9vPZofJLqvjZyl28t7uU0BBhaHoCt03oy1VDerJq1zHe2n6U0up6MhKjWbXzGNX1TiJCQxjcK46EqDBW7yljeHoC/VNj+NumQwxIi2VEegJR4Q4Ky2uYlJNMZJiDueP72u2aQcrXt0LOBH6J+1bIxar6iIg8DKxX1eUi8igwC3ACx4G7VHVnW/u0cDcBqbYStv/N3XVzYI17Xeal7qAfej1E+WYmp/WFx3l75zHe313KtkMnTq9P7xFFv+Rodhw+wZjMRC4fmEJxRS07j5yksLyGywemsKagnJLKWmaP6kNxRS0lFbWcqGskNS6CfaU1AGSnxPDMHePJSrGpBYONPcRkTEdVHHDfO7/lJSjbDY5wGHSN+7bKgVf7ZKIRl0t5eUMRVbWNTMpJYVh6vFe3Tja59HRXTXNl1fUUHKvmG89/QojAD2cNY3CvWLJTYlvc3nQ/Fu7G+IoqHN7kmWhkGdSUulvwQ29wB33fCZ0+0cj5cgf8BnYfrQYgLjKUS/snc9+0QeT2jvdzdaYjLNyN6QxNTti3yt2a3/F3cNZCYpZnopG5kNzf3xWeVu9sYu2+45SerOeTgxWs2HqYylONXDEolcdvGUlKbIS/SzQXwMLdmM5Wf9I9UuWWl2Dfe4BC+jj3QGZDb4SYZH9XeJbKUw08t/YAv1lVQGpcBDeMSqe4spahfRLISY3hc/agVbdg4W5MVzpx6MxEI0fzISQUBkx3X4gdfC2EBc7EHxsPVnDvko0UHa8lPjKUE3VOHCHC29++0i7AdgMW7sb4y5H8MwOZnTwMEfHuJ2FHzIN+l/l0opELVdfYRFl1PX0SothfXsN1T6xm6pCe/Oa20TYWToCzcDfG31xNULjaM9HIcmiohvgMz0Qj8yBtiL8rPO2Jt/fw+Ju7mTuuL4/cMMxmnwpgFu7GBJKGU7BrhbtFX/A2aBP0GuGZaORmiPPvTE+qyi/e3M0T7xQwLdfdgrfhiwOThbsxgar6GOT/xT3RyKGNICGQ8zn33Ta5n4dw//V7//nDQn6wfBtpcREM6hnH+Kwk7rgsy8bCCSAW7sZ0B6W73a35LUuh6iCExXgmGrkFcqZ0eKKRC/HOzqP8deMhCstq2FpSxaCesbx692Siwq0lHwgs3I3pTlwuKFrrGcjsb1BfBbE93WPPj7jF3YXjhwudq3Ye49//tI7ZI/vwf3NG8uS7ewl1hPDvk7OICLWw9wcLd2O6q8Y62LPS3ZrfvRJcjZCae2aikYSMLi1n4aoCfrZyFz3jIzh6oh6A8VmJ/OnfJ7Q5qqXpHBbuxgSDU8dh21/dXTdFHwECWZM9E43MhsiuGUrg2bUHeG9XKZcPTKFHdBj3v7SJS/un8OtbR9MjOsxun+xCFu7GBJvj+2DLy+4Lscf3QWik+wGpEfNgwFRwdN1Fz2UbivnOy5sB6JsUxX9c0Z/bJmTa2PJdwMLdmGClCiUb3P3z+a9A7XGITvZMNDIP0sd0Sf/86j3u4Yrf3nGUdYUV3DgmnZ/PGWmt+E5m4W7MxaCpEQrecnfb7FwBTfWQPMDdbTN8DiRld3oJqsrjb+7m1+8UMLZfIoN6xnHjmHTGZyV1+rEvRhbuxlxs6qpg+3J30Beudq/rO9F9ITbveojuvLB1uZRfvrWbNXvL2X30JCfrnFw5KJUHv5BH/9TYTjvuxcjC3ZiLWWXRmYlGSne6JxoZeLW7RT/oGgjtvOF+axuaePqDfTz1wX5iwkO5/dJ+1Da4KCitZkBqLOsPHGdERgJDesXzuSFpnKxr5J/5R9haUsWCGUNIi4/stNqCga+n2ZsB/Ar3NHtPqepjrWx3E7AMGK+qbSa3hbsxXUAVjmxxj2+z9WWoOQaI++EoCQHx/BnicPfTf2ZdyNlfZ637dLuQFtY5qG5wse3wSeqbBEVwOEKpa1LCw0I51Qgu3MdzquBCcBFCXFQEI/smkZYQ5dNa3MvSyv48597i/tr7e+nI97Z1Ho5Wr5t4G+7t3qQqIg5gITAdKAbWichyVd1+znZxwLeAj9r/F2eM6RIi0Huk+2v6w7D/XTj4EbicoK6zv1xNzZY9r12uFtY1efG9SmyYMj4jBlUXok0ISmNjI2EhjbhcLuobGqmpbyBclOiwEJxNTipq6nAWuDgRJsRGOEBdhLRXC/7pfegSLf1g8JI3TyBMAApUdR+AiCwBZgPbz9nuR8D/Ag94fXRjTNdxhMKAae6vLnJuFH0666wDiPZ8fSoMkMYmHn9zN4ve3wfVECJw/7RBfH1Kf8JaG6lS1fPV0g8f9w+aln9wfbruQr9XL+CHXke/1wU86tXfvTfhng4UNVsuBi5pvoGIjAH6quo/RMTC3RhzQSLDHHxvZi4zhvVi1c5j7Cut4edv7mbJuiLy+sSjqkwZnMYXL8k8c8uliKcL42IZpth34d4mEQkBHgfu8GLb+cB8gMzMzI4e2hgTpMZkJjImMxFV5cad6Tz/0UEKy2pocilv7cinwekiIzGKOqeL6bk9bVCzFrR7QVVEJgEPqeo1nuXvAqjqo57lBGAvUO35ll7AcWBWWxdV7YKqMeZ8NbmUWxet5ePC46fXZSZF8+rdl5EYE97GdwYPn90tIyKhwG5gKlACrANuU9VtrWz/LvAdu1vGGNMZquudrCkoIzkmnBN1jXz92U8Y1bcH900byNA+CSREB/fY896Ge7udVKrqBO4BVgI7gKWquk1EHhaRWR0v1RgnnLeDAAANKUlEQVRjvBcbEco1Q3sxLiuJq4b05LGbhrPtUBW3PfUR03/xHmXV9f4uMSDYQ0zGmG6v8lQDawrKuX/pJjJ6RPEfV+Ywd3xwXtfzWcvdGGMCXY/ocK4b0Ztf3zqaiDAH//3KVlbtOga4x75pbHL5ucKuZy13Y0xQqWts4vqFa9hXVsO3pw/ibxtL2F9Ww4C0WA5V1pKTGstVQ9LolxzN9Lye3W5GKRtbxhhz0SqvrufbSzfz3u5SosIc3Dw2g6KKU/SICuODgvLT/fJjMnvwxUv6sf5ABarK9sMnyEyK5vMj+nDVkDTCQwOvc8PC3RhzUVNV1h+oICY8lLw+Z2ascja5aGxSXs8/zAPLttDkUuIjQwl1hJCVHE1h+SmO1zTQJyGS+VfkcMv4vkSHh1Jd7+TJdwsYnp7A1NyerT8x28ks3I0xph17S6upb3QxpFfc6VmknE0u3ttdyu/e28u6wgriI0O5d+pA3ttdyuo9ZQD0SYhkWl5Pbr80q8uHNLZwN8aYDlpfeJxfvLWbNQXlhAj8+PrhpMVF8MLHB/lwbzkNTS5+NHsYsZGhPLf2ACUVtYzsm8BPbhhOj+jOeajKwt0YY3zA5VI+2n+c7JQYeiWcGWu+rLqe+5Zs4oMCd2s+JzWGvN7xrNx2hNGZiTx+y0gyEqNb2+0Fs3A3xphOVtfYxFs7jhIXGcbkASk4QoRXN5XwnZc3IwhfmdSPhKgw6p0ueiZE8kUfTCLus/HcjTHGtCwyzMHnR/Q5a93sUemMy0riJ//YwVMf7AfcQxe7FP5VUMZ3rhlMj6gwkmM7bzYssJa7McZ0mrrGJkJDhFBHCE+t3sejr++kyaVEhIbw2E3DGZnRg5zzvCBrLXdjjPGzyLAzD0h97fIcLu2fwvoDx3lu7QHuf2kzAFOHpHHv1IGM7NvDp8e2lrsxxnSx6nonW4oq2XCggsVr9lNZ28hdV/bngWsGn5mEpBXWcjfGmAAVGxHKpQNSuHRACv82OZsfvbad3767l60lVSTHhNO7RxSfH9GbBqeL3767l3BHCHPGZTAxJ9nrY1i4G2OMH8VGhPLYTcNJjg3n9fwjHCg/xaEth3ny3b0AJMWE0+RS/rH1MNNye3q9X+uWMcaYAFNV28iLHx8kNES4dUImjhDhw33lRIU5mNQ/xbpljDGmO0qICuPrV/Y/a93nBqed1z4Cb8gzY4wxHWbhbowxQcircBeRGSKyS0QKRGRBC+9/XUS2isgmEflARPJ8X6oxxhhvtRvuIuIAFgLXAnnArS2E9wuqOlxVRwE/BR73eaXGGGO85k3LfQJQoKr7VLUBWALMbr6Bqp5othgD+OcWHGOMMYB3d8ukA0XNlouBS87dSETuBr4NhANX+aQ6Y4wxF8RnF1RVdaGq9gf+G/iflrYRkfkisl5E1peWlvrq0MYYY87hTbiXAH2bLWd41rVmCXB9S2+o6iJVHaeq41JTU72v0hhjzHnxJtzXAQNFJFtEwoF5wPLmG4jIwGaL1wF7fFeiMcaY89Vun7uqOkXkHmAl4AAWq+o2EXkYWK+qy4F7RGQa0AhUALd3ZtHGGGPa5tXwA6q6AlhxzroHm73+lo/rMsYY0wH2hKoxxgQhC3djjAlCFu7GGBOELNyNMSYIWbgbY0wQsnA3xpggZOFujDFByMLdGGOCkIW7McYEIQt3Y4wJQhbuxhgThCzcjTEmCFm4G2NMELJwN8aYIGThbowxQcjC3RhjgpCFuzHGBCGvwl1EZojILhEpEJEFLbz/bRHZLiJbRORtEenn+1KNMcZ4q91wFxEHsBC4FsgDbhWRvHM22wiMU9URwDLgp74u1BhjjPe8ablPAApUdZ+qNgBLgNnNN1DVVap6yrO4FsjwbZnGGGPOhzfhng4UNVsu9qxrzVeB1ztSlDHGmI4J9eXORORLwDjgylbenw/MB8jMzPTloY0xxjTjTcu9BOjbbDnDs+4sIjIN+D4wS1XrW9qRqi5S1XGqOi41NfVC6jXGGOMFb8J9HTBQRLJFJByYByxvvoGIjAZ+jzvYj/m+TGOMMeej3XBXVSdwD7AS2AEsVdVtIvKwiMzybPYzIBZ4WUQ2icjyVnZnjDGmC3jV566qK4AV56x7sNnraT6uyxhjTAfYE6rGGBOELNyNMSYIWbgbY0wQsnA3xpggZOFujDFByMLdGGOCkIW7McYEIQt3Y4wJQhbuxhgThCzcjTEmCFm4G2NMELJwN8aYIGThbowxQcjC3RhjgpCFuzHGBCELd2OMCUIW7sYYE4Qs3I0xJgh5Fe4iMkNEdolIgYgsaOH9K0TkExFxisjNvi/TGGPM+Wg33EXEASwErgXygFtFJO+czQ4CdwAv+LpAY4wx58+bCbInAAWqug9ARJYAs4Htn26gqoWe91ydUKMxxpjz5E23TDpQ1Gy52LPuvInIfBFZLyLrS0tLL2QXxhhjvNClF1RVdZGqjlPVcampqV15aGOMuah4E+4lQN9myxmedcYYYwKUN+G+DhgoItkiEg7MA5Z3blnGGGM6ot1wV1UncA+wEtgBLFXVbSLysIjMAhCR8SJSDMwBfi8i2zqzaGOMMW3z5m4ZVHUFsOKcdQ82e70Od3eNMcaYAGBPqBpjTBCycDfGmCBk4W6MMUHIwt0YY4KQhbsxxgQhC3djjAlCFu7GGBOELNyNMSYIWbgbY0wQsnA3xpggZOFujDFByMLdGGOCkIW7McYEIQt3Y4wJQhbuxhgThCzcjTEmCFm4G2NMEPIq3EVkhojsEpECEVnQwvsRIvKS5/2PRCTL14UaY4zxXrvhLiIOYCFwLZAH3Coieeds9lWgQlUHAL8A/tfXhRpjjPGeNy33CUCBqu5T1QZgCTD7nG1mA3/yvF4GTBUR8V2Zxhhjzoc34Z4OFDVbLvasa3EbVXUCVUCyLwo0xhhz/kK78mAiMh+Y71msF5H8rjx+J0gByvxdRAfZOQQGO4fA0B3OoZ83G3kT7iVA32bLGZ51LW1TLCKhQAJQfu6OVHURsAhARNar6jhvigxUdg6Bwc4hMNg5BBZvumXWAQNFJFtEwoF5wPJztlkO3O55fTPwjqqq78o0xhhzPtptuauqU0TuAVYCDmCxqm4TkYeB9aq6HHgaeFZECoDjuH8AGGOM8ROv+txVdQWw4px1DzZ7XQfMOc9jLzrP7QORnUNgsHMIDHYOAUSs98QYY4KPDT9gjDFByC/h3t5wBoFKRApFZKuIbBKR9Z51SSLypojs8fyZ6O86mxORxSJyrPltp63VLG5PeD6XLSIyxn+Vn9HKOTwkIiWez2KTiMxs9t53PeewS0Su8U/VZ4hIXxFZJSLbRWSbiHzLs77bfA5tnEN3+hwiReRjEdnsOYcfetZne4ZNKfAMoxLuWd+9h1VR1S79wn1Rdi+QA4QDm4G8rq7jAmsvBFLOWfdTYIHn9QLgf/1d5zn1XQGMAfLbqxmYCbwOCDAR+Mjf9bdxDg8B32lh2zzPv6kIINvzb83h5/p7A2M8r+OA3Z46u83n0MY5dKfPQYBYz+sw4CPP3+9SYJ5n/e+AuzyvvwH8zvN6HvCSvz+H8/nyR8vdm+EMupPmQy/8Cbjej7V8hqq+j/sOpuZaq3k28Gd1Wwv0EJHeXVNp61o5h9bMBpaoar2q7gcKcP+b8xtVPayqn3henwR24H6qu9t8Dm2cQ2sC8XNQVa32LIZ5vhS4CvewKfDZz6HbDqvij3D3ZjiDQKXAGyKywfO0LUBPVT3seX0E6Omf0s5LazV3t8/mHk+3xeJm3WEBfQ6eX+1H4241dsvP4ZxzgG70OYiIQ0Q2AceAN3H/RlGp7mFT4Ow6u/WwKnZB9fxMVtUxuEfIvFtErmj+prp/f+tWtx91x5o9ngT6A6OAw8DP/VtO+0QkFngFuE9VTzR/r7t8Di2cQ7f6HFS1SVVH4X7SfgIwxM8ldRp/hLs3wxkEJFUt8fx5DPgr7n8cRz/9ldnz5zH/Vei11mruNp+Nqh71/Ed1AX/gzK/8AXkOIhKGOxSfV9W/eFZ3q8+hpXPobp/Dp1S1ElgFTMLd7fXpMz/N6zx9DtLGsCqByh/h7s1wBgFHRGJEJO7T18DVQD5nD71wO/Cqfyo8L63VvBz4iudujYlAVbNug4ByTh/0Dbg/C3CfwzzPnQ7ZwEDg466urzlPP+3TwA5VfbzZW93mc2jtHLrZ55AqIj08r6OA6bivHazCPWwKfPZz6L7DqvjjKi7uuwF24+7v+r6/ryp7WXMO7qv/m4Ftn9aNuw/ubWAP8BaQ5O9az6n7Rdy/Ljfi7k/8ams1476bYKHnc9kKjPN3/W2cw7OeGrfg/k/Yu9n23/ecwy7g2gCofzLuLpctwCbP18zu9Dm0cQ7d6XMYAWz01JoPPOhZn4P7B08B8DIQ4Vkf6Vku8Lyf4+9zOJ8ve0LVGGOCkF1QNcaYIGThbowxQcjC3RhjgpCFuzHGBCELd2OMCUIW7sYYE4Qs3I0xJghZuBtjTBD6/7KImejJZ63qAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learner.fit_one_cycle(config.epochs, max_lr=config.max_lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy=0.8214492797851562, f1_score=0.8666666666666667\n"
     ]
    }
   ],
   "source": [
    "preds, pred_values, true_labels = learner.get_predictions()\n",
    "learner.print_metrics(preds, pred_values, true_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "txt_ci = TextClassificationInterpretation.from_learner(learner)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Index</th>\n",
       "      <th>Original Text</th>\n",
       "      <th>Tokenized</th>\n",
       "      <th>Prediction</th>\n",
       "      <th>Actual</th>\n",
       "      <th>Loss</th>\n",
       "      <th>Probability</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>996</td>\n",
       "      <td>The 2 1 / 2 -ton probe will plunge into the thick Jovian atmosphere today at 3 : 49 p.m. Eastern time , disintegrating moments later from the friction generated by its 108,000-mph free-fall .,The 2 1 / 2-ton probe will plunge into the thick Jovian atmosphere today at 1 : 49 p.m. MDT , disintegrating moments later from the friction generated by its 108,000 mph free-fall .</td>\n",
       "      <td>[CLS] the 2 1 / 2 - ton probe will plunge into the thick jo ##vian atmosphere today at 3 : 49 p . m . eastern time , di ##sin ##te ##grating moments later from the friction generated by its 108 , 000 - mph free - fall . the 2 1 / 2 - ton probe will plunge into the thick jo ##vian atmosphere today at 1 :</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3.65</td>\n",
       "      <td>0.97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>436</td>\n",
       "      <td>Iran has yet to sign an additional protocol to the NPT treaty which would allow U.N. inspections at short notice .,Iran has yet to sign an additional protocol to the Nuclear Non-Proliferation Treaty , which it signed in 1970 , that would allow IAEA inspections at short notice .</td>\n",
       "      <td>[CLS] iran has yet to sign an additional protocol to the np ##t treaty which would allow u . n . inspections at short notice . iran has yet to sign an additional protocol to the nuclear non - proliferation treaty , which it signed in 1970 , that would allow ia ##ea inspections at short notice . [SEP]</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3.61</td>\n",
       "      <td>0.97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>569</td>\n",
       "      <td>Senate Minority Leader Tom Daschle , D-S.D. , is leading the opposition .,\" I think it will pass , \" Senate Minority Leader Tom Daschle , D-S.D. , said in Washington .</td>\n",
       "      <td>[CLS] senate minority leader tom das ##ch ##le , d - s . d . , is leading the opposition . \" i think it will pass , \" senate minority leader tom das ##ch ##le , d - s . d . , said in washington . [SEP]</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3.59</td>\n",
       "      <td>0.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>779</td>\n",
       "      <td>Casteen has been under pressure from Gov. Mark R. Warner and other state officials to do whatever he could to protect Virginia Tech 's athletic viability .,Virginia Gov. Mark R. Warner has been urging other state officials to do whatever they could to protect Virginia Tech 's interests .</td>\n",
       "      <td>[CLS] caste ##en has been under pressure from gov . mark r . warner and other state officials to do whatever he could to protect virginia tech ' s athletic via ##bility . virginia gov . mark r . warner has been urging other state officials to do whatever they could to protect virginia tech ' s interests . [SEP]</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3.59</td>\n",
       "      <td>0.97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>192</td>\n",
       "      <td>His chief lawyer , Mahendradatta , said Bashir was mentally prepared for a heavy sentencing demand and felt the Marriott bombing would affect the decision .,A lawyer for Bashir , Mahendradatta , said earlier his client was mentally prepared for a heavy sentencing demand and had felt the Marriott bombing would affect the decision .</td>\n",
       "      <td>[CLS] his chief lawyer , ma ##hend ##rada ##tta , said bash ##ir was mentally prepared for a heavy sentencing demand and felt the marriott bombing would affect the decision . a lawyer for bash ##ir , ma ##hend ##rada ##tta , said earlier his client was mentally prepared for a heavy sentencing demand and had felt the marriott bombing would affect the decision . [SEP]</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3.56</td>\n",
       "      <td>0.97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>91</td>\n",
       "      <td>They were at Raffles Hospital over the weekend for further evaluation .,They underwent more tests over the weekend , and are now warded at Raffles Hospital .</td>\n",
       "      <td>[CLS] they were at raf ##fles hospital over the weekend for further evaluation . they underwent more tests over the weekend , and are now ward ##ed at raf ##fles hospital . [SEP]</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3.54</td>\n",
       "      <td>0.97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>589</td>\n",
       "      <td>In that case , the court held that Cincinnati had violated the First Amendment in banning only the advertising pamphlets in the interest of aesthetics .,In that case , the court held that the city of Cincinnati had violated the First Amendment in banning , in the interest of aesthetics , only the advertising pamphlets .</td>\n",
       "      <td>[CLS] in that case , the court held that cincinnati had violated the first amendment in banning only the advertising pamphlets in the interest of aesthetics . in that case , the court held that the city of cincinnati had violated the first amendment in banning , in the interest of aesthetics , only the advertising pamphlets . [SEP]</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3.53</td>\n",
       "      <td>0.97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>653</td>\n",
       "      <td>He admits that the law \" has several weaknesses which terrorists could exploit , undermining our defenses . \",But he also told the House Judiciary Committee the law \" has several weaknesses which terrorists could exploit , undermining our defenses . \"</td>\n",
       "      <td>[CLS] he admits that the law \" has several weaknesses which terrorists could exploit , under ##mini ##ng our defenses . \" but he also told the house judiciary committee the law \" has several weaknesses which terrorists could exploit , under ##mini ##ng our defenses . \" [SEP]</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3.53</td>\n",
       "      <td>0.97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>694</td>\n",
       "      <td>The fines are part of failed Republican efforts to force or entice the Democrats to return .,Perry said he backs the Senate 's efforts , including the fines , to force the Democrats to return .</td>\n",
       "      <td>[CLS] the fines are part of failed republican efforts to force or en ##tic ##e the democrats to return . perry said he backs the senate ' s efforts , including the fines , to force the democrats to return . [SEP]</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3.51</td>\n",
       "      <td>0.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>296</td>\n",
       "      <td>GE stock were up 37 cents to $ 28.56 in morning New York Stock Exchange trade .,Investors reacted little , with GE shares edging 7 cents lower to end at $ 28.12 on the New York Stock Exchange .</td>\n",
       "      <td>[CLS] ge stock were up 37 cents to $ 28 . 56 in morning new york stock exchange trade . investors reacted little , with ge shares ed ##ging 7 cents lower to end at $ 28 . 12 on the new york stock exchange . [SEP]</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3.49</td>\n",
       "      <td>0.03</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "utils.custom_show_top_losses(txt_ci, test_df, text_cols, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Next Steps\n",
    "\n",
    "- There's major improvement in the accuracy and loss by just running for few epochs. We can improve the model further to acc = 0.8504901960784313 and f1 = 0.8974789915966387 by using the techniques mentioned in the original BERT paper. Refer to https://github.com/huggingface/pytorch-pretrained-BERT#mrpc for details.\n",
    "- Use BertAdam Optimizer [BertAdam](https://github.com/huggingface/pytorch-pretrained-BERT/blob/694e2117f33d752ae89542e70b84533c52cb9142/README.md#optimizers)\n",
    "[Is BertAdam better](https://github.com/huggingface/pytorch-pretrained-BERT/issues/420)\n",
    "\n",
    "- Finetune BERT on Quora insincere questions.\n",
    "- Explore LAMB Optimizer: https://forums.fast.ai/t/lamb-optimizer/43582/19\n",
    "- Compare with ULMFiT, gpt-2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:fastai]",
   "language": "python",
   "name": "conda-env-fastai-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}