fasthugs_language_model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# \"FastHugs: Language Modelling with Tranformers and Fastai\"\n",
    "> Train a transformer language model from scratch or fine-tune a pretrained one using fastai and Huggingface.\n",
    "\n",
    "- toc: false\n",
    "- badges: true\n",
    "- comments: true\n",
    "- author: Morgan McGuire\n",
    "- categories: [NLP, transformers, training technique, classification]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This aims to be an end-to-end description with code of how to train a transformer language model using fastai (v2) and HuggingFace, enjoy!\n",
    "\n",
    "## TL;DR\n",
    "Main interesting bits in this notebook:\n",
    "- Provides full code to train a transformer (RoBERTa) using a Masked Language Model task\n",
    "- Utilise's many of HuggingFace's tokenizer features within fastai\n",
    "- Make predictions of masked tokens like this:\n",
    "\n",
    "![](images/roberta_pred.png)\n",
    "\n",
    "\n",
    "## Before we get started\n",
    "- First off, huge thanks as always to both the Fastai and HuggingFace teams for giving so much back to the community by open-sourcing so much\n",
    "- For an example of **text sequence classification** using HuggingFace and fastai, have a look at my previous notebook [**here**](https://github.com/morganmcg1/fasthugs/blob/master/fasthugs_seq_classification.ipynb)\n",
    "\n",
    "- This tutorial is heavily based on HuggingFace's [\"How to train a new language model from scratch using Transformers and Tokenizers\"](https://huggingface.co/blog/how-to-train)  tutorial, I highly recommend checking that out too. I try and highlight throughout where code has been used, borrowed or inspired by HuggingFace's code.\n",
    "\n",
    "\n",
    "## MLM Tranform\n",
    "I feel the most useful thing in this notebook is the `MLMTokensLabels` transform*. This carries out the Masked Language Model task that RoBERTa was originally trained on.\n",
    "\n",
    "This will take tokens ids (tokens after the have been numericalized), select a subset and either mask a certain amount of them (for prediction) or replace them with other random token ids (for regularisation). This transform also creates our labels by copying the input token ids and masking the tokens that do **not** need to be predicted, so that no loss is calculated on them. \n",
    "\n",
    "Note the if you wish to train BERT or other transformer language models you will probably need to use a different task, e.g. BERT was trained on 2 tasks simultaneously, MLM and Next Sentence Prediction (NSP). Have a look at any blog posts or arxiv paper of the transformer of interest to find which task was used to pretrain it.\n",
    "\n",
    "*This transform code is a re-write of the `mask_tokens` function used in HugginFace's tutorial, [code here](https://github.com/huggingface/transformers/blob/a21d4fa410dc3b4c62f93aa0e6bbe4b75a101ee9/examples/run_language_modeling.py#L66)\n",
    "\n",
    "\n",
    "## Pretraining + Fine-Tuning:\n",
    "\n",
    "As shown in ULMFit, MultiFiT, and elsewhere, you will get better results on your downstream task if you first fine-tune your pretrained model with the text of the same domain as your pretrained task. e.g. training an IMDB movie review classifier who's language model was trained on wikipedia text.\n",
    "\n",
    "> twitter: https://twitter.com/ssgrn/status/1253498613558243328 \n",
    "\n",
    "\n",
    "## Using a Custom Tokenizer?\n",
    "This code has not been tested using a custom tokenizer. You may want to do so if your text is very specific to a certain domain. If so then you'll have to add a number of attributes to your tokenzier to be able to use the code here. I really recommend the [HuggingFace language model tutorial linked above](https://huggingface.co/blog/how-to-train) for an example of training your own tokenizer with your own dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#hide\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "from fastai2.basics import *\n",
    "from fastai2.text.all import *\n",
    "from fastai2.callback.all import *\n",
    "\n",
    "from transformers import AutoModelWithLMHead, AutoModel, AutoModelForPreTraining, AutoTokenizer, AutoConfig #AutoModelForSequenceClassification\n",
    "from splitters import *\n",
    "\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "We'll use the `IMDB_SAMPLE` here, pretending we are fine-tuning our transformer model before doing sentiment classification on IMDB. If you are pretraining a language model from scratch you'd aim to use a larger, more generic source like a wikipedia dataset. fastai have the full `WikiText103` (100 million tokens) dataset available for easy download here if you'd like to train an enligh language model from scratch:\n",
    "\n",
    "`path = untar_data(URLs.WIKITEXT)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "path = untar_data(URLs.IMDB_SAMPLE)\n",
    "model_path = Path('models')\n",
    "df = pd.read_csv(path/'texts.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HuggingFace Auto Classes\n",
    "\n",
    "HuggingFace have a numer of useful [\"Auto\" classes](https://huggingface.co/transformers/model_doc/auto.html) that enable you to create different models and tokenizers by changing just the model name.\n",
    "\n",
    "- `AutoModelWithLMHead` will define our Language model for us. This can either be a pretrained model or a randomly initialised model\n",
    "- `AutoTokenizer` will load our tokenizer and enable us grab our vocab\n",
    "- `AutoConfig` will define the model architecture and settings, note that we use the pretrained config here for ease of use, but one can easily modify this config if needed\n",
    "- `model_name` is the model architecture (and optionally model weights) you'd like to use.\n",
    "    - Language Models tested so far with this notebook: `roberta-base`\n",
    "    - You can find all of HuggingFace's models at [https://huggingface.co/models](https://huggingface.co/models), most, but not all of them are supported by `AutoModel`,`AutoConfig` and `AutoTokenizer`\n",
    "    \n",
    "We can now easily call whichever transformer we like as below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model_name = 'roberta-base' \n",
    "lm_model_class = AutoModelWithLMHead \n",
    "config_dict = AutoConfig.from_pretrained(model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HuggingFace Tokenizer & Vocab\n",
    "We use `AutoTokenizer` to generate our pretrained tokenizer. HuggingFace's `get_vocab` returns a `token : index` dict however Fastai expects `vocab` to be a list. Therefore we need to convert this dict to a list to be able to use it in fastai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokenizer \"<class 'transformers.tokenization_roberta.RobertaTokenizer'>\" vocab length is : 50265\n"
     ]
    }
   ],
   "source": [
    "#collapse\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "tokenizer_vocab=tokenizer.get_vocab() \n",
    "tokenizer_vocab_ls = [k for k, v in sorted(tokenizer_vocab.items(), key=lambda item: item[1])]\n",
    "print(f'Tokenizer \"{tokenizer.__class__}\" vocab length is : {len(tokenizer_vocab_ls)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Special Tokens\n",
    "Its always good to know what special tokens your tokenizer takes, lets have a look:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bos_token': '<s>',\n",
       " 'eos_token': '</s>',\n",
       " 'unk_token': '<unk>',\n",
       " 'sep_token': '</s>',\n",
       " 'pad_token': '<pad>',\n",
       " 'cls_token': '<s>',\n",
       " 'mask_token': '<mask>'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.special_tokens_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FastHugs Tokenizer\n",
    "This tokenizer wrapper is initialised with the pretrained HF tokenizer, you can also specify the max_seq_len if you want longer/shorter sequences. Given text it returns tokens and adds separator tokens depending on the model type being used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# collapse\n",
    "class FastHugsTokenizer():\n",
    "    \"\"\" \n",
    "        transformer_tokenizer : takes the tokenizer that has been loaded from the tokenizer class\n",
    "        model_name : model type set by the user\n",
    "        max_seq_len : override default sequence length, typically 512 for bert-like models.\n",
    "                           `transformer_tokenizer.max_len_single_sentence` and `transformer_tokenizer.max_len_sentences_pair` \n",
    "                           both account for the need to add additional special tokens, i.e. for RoBERTa-base \n",
    "                           max_len_single_sentence==510, leaving space for the 2 additional special tokens \n",
    "                           to be added for the model's default 512 positional embeddings\n",
    "        pair : whether a single sentence (sequence) or pair of sentences are used\n",
    "\n",
    "        Returns:\n",
    "            - Tokenized text, up to the max sequence length set by the user or the tokenzier default\n",
    "    \"\"\"\n",
    "    def __init__(self, transformer_tokenizer=None, model_name='roberta', max_seq_len=None, \n",
    "                 pretrained=True, pair=False, **kwargs): \n",
    "        self.model_name, self.tok, self.max_seq_len=model_name, transformer_tokenizer, max_seq_len\n",
    "        if pretrained:\n",
    "            if self.max_seq_len:\n",
    "                if pair: assert self.max_seq_len<=self.tok.max_len_sentences_pair, 'WARNING: max_seq_len needs to be less than or equal to transformer_tokenizer.max_len_sentences_pair'\n",
    "                else: assert self.max_seq_len<=self.tok.max_len_single_sentence, 'WARNING: max_seq_len needs to be less than or equal to transformer_tokenizer.max_len_single_sentence'\n",
    "            else:\n",
    "                if pair: self.max_seq_len=ifnone(max_seq_len, self.tok.max_len_sentences_pair) \n",
    "                else: self.max_seq_len=ifnone(max_seq_len, self.tok.max_len_single_sentence)\n",
    "\n",
    "    def do_tokenize(self, o:str):\n",
    "        \"\"\"Returns tokenized text, adds prefix space if needed, limits the maximum sequence length\"\"\"\n",
    "        if 'roberta' in model_name: tokens=self.tok.tokenize(o, add_prefix_space=True)[:self.max_seq_len-2]\n",
    "        else: tokens = self.tok.tokenize(o)[:self.max_seq_len-2]\n",
    "        return tokens\n",
    "    \n",
    "    def de_tokenize(self, o):\n",
    "        \"\"\"Return string from tokens\"\"\"\n",
    "        text=self.tok.convert_tokens_to_string(o)\n",
    "        return text\n",
    "        \n",
    "    def __call__(self, items): \n",
    "        for o in items: yield self.do_tokenize(o)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Fastai bit\n",
    "\n",
    "### `fasthugstok` and our `tok_fn`\n",
    "Lets incorporate the `tokenizer` from HuggingFace into fastai-v2's framework by specifying a function called `fasthugstok` that we can then pass on to `Tokenizer.from_df`. (Note `.from_df` is the only method I have tested)\n",
    "\n",
    "#### Max Seqence Length\n",
    "`max_seq_len` is the longest sequece our tokenizer will output. We can also the max sequence length for the tokenizer by changing `max_seq_len`. It uses the tokenizer's default, typically `512`. `1024` or even `2048` can also be used depending on your GPU memory. Note when using pretrained models you won't be able to use a `max_seq_len` larger than the default. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "max_seq_len = 512  \n",
    "sentence_pair=False\n",
    "\n",
    "fasthugstok = partial(FastHugsTokenizer, transformer_tokenizer=tokenizer, model_name=model_name, \n",
    "                      max_seq_len=max_seq_len, sentence_pair=sentence_pair)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a `MLMTokenizer` class which inherits from fastai's `Tokenizer` in order to fully decode "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#collapse\n",
    "class MLMTokenizer(Tokenizer):\n",
    "    def __init__(self, tokenizer, rules=None, counter=None, lengths=None, mode=None, sep=' ', **kwargs): \n",
    "        super().__init__(tokenizer, rules, counter, lengths, mode, sep)\n",
    "    def _detokenize1(self, o):return self.tokenizer.de_tokenize(o)\n",
    "    def decodes(self, o): return TitledStr(str(self._detokenize1(o)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set up fastai's `Tokenizer.from_df`, we pass `rules=[fix_html]` to clean up some of HTML messiness in our text. If you do not want any rules then you sould pass `rules=[]` to override fastai's default text processing rules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<function fastai2.text.core.fix_html(x)>]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#collapse\n",
    "fastai_tokenizer = MLMTokenizer.from_df(text_cols='text', res_col_name='text', tok_func=fasthugstok, \n",
    "                                     rules=[fix_html], post_rules=[])\n",
    "fastai_tokenizer.rules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add Special Tokens\n",
    "BERT-like transformers require special tokens to be added to the sequence, depending on the task, so we need a transform for those too"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AddSpecialTokens(Transform):\n",
    "    \"Add special token_ids to the numericalized tokens for Sequence Classification\"\n",
    "    def __init__(self, tokenizer):\n",
    "        self.tok=tokenizer\n",
    "    def encodes(self, o):\n",
    "        return(TensorText(self.tok.build_inputs_with_special_tokens(list(o))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create MLM Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#collapse\n",
    "class MLMTokensLabels(Transform):\n",
    "    '''\n",
    "        MLM task\n",
    "        - Select subset of input token ids, given by `mlm_probability`\n",
    "        - Mask a subset of these, `mask_token_prob`\n",
    "        - Replace half of the first subset with random tokens\n",
    "        - This code most comes from the `mask_tokens` function here https://github.com/huggingface/transformers/blob/a21d4fa410dc3b4c62f93aa0e6bbe4b75a101ee9/examples/run_language_modeling.py#L66\n",
    "        Returns: input ids and labels\n",
    "    '''\n",
    "    def __init__(self, tokenizer=None, mlm_probability=0.15, mask_token_prob=0.8):\n",
    "        self.tok, self.mlm_probability, self.mask_token_prob=tokenizer, mlm_probability, mask_token_prob\n",
    "    \n",
    "    def _gen_probability_matrix(self, labels):\n",
    "        # We sample a few tokens in each sequence for masked-LM training (with probability mlm_probability, defaults to 0.15 in Bert/RoBERTa)\n",
    "        probability_matrix = torch.full(labels.shape, self.mlm_probability) \n",
    "        special_tokens_mask = self.tok.get_special_tokens_mask(labels.tolist(), already_has_special_tokens=True)\n",
    "        probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)\n",
    "        if self.tok._pad_token is not None:\n",
    "            padding_mask = labels.eq(self.tok.pad_token_id)\n",
    "            probability_matrix.masked_fill_(padding_mask, value=0.0)\n",
    "        return probability_matrix\n",
    "    \n",
    "    def _replace_with_mask(self, inputs, labels, masked_indices):\n",
    "        # for `mask_token_prob`% of the time, we replace masked input tokens with tokenizer.mask_token ([MASK])\n",
    "        indices_replaced = torch.bernoulli(torch.full(labels.shape, self.mask_token_prob)).bool() & masked_indices\n",
    "        inputs[indices_replaced] = self.tok.convert_tokens_to_ids(self.tok.mask_token)\n",
    "        return inputs, indices_replaced\n",
    "    \n",
    "    def _replace_with_other(self, inputs, labels, masked_indices, indices_replaced):\n",
    "        # 1-`mask_token_prob`)/210% of the time, we replace masked input tokens with random word\n",
    "        indices_random = torch.bernoulli(torch.full(labels.shape, 0.5)).bool() & masked_indices & ~indices_replaced\n",
    "        random_words = torch.randint(len(self.tok), labels.shape, dtype=torch.long)\n",
    "        inputs[indices_random] = random_words[indices_random]\n",
    "        return inputs\n",
    "    \n",
    "    def encodes(self, inputs):\n",
    "        if self.tok.mask_token is None:\n",
    "            raise ValueError(\"This tokenizer does not have a mask token which is necessary for masked language modeling.\")\n",
    "        labels = inputs.clone()\n",
    "        \n",
    "        # Get probability of whether a token will be masked\n",
    "        probability_matrix = self._gen_probability_matrix(labels)\n",
    "        \n",
    "        # Create random mask indices according to probability matrix\n",
    "        masked_indices = torch.bernoulli(probability_matrix).bool()\n",
    "        \n",
    "        # Mask the labels for indices that are NOT masked, we only compute loss on masked tokens\n",
    "        labels[~masked_indices] = -100  \n",
    "        \n",
    "        # Randomly replace with mask token\n",
    "        inputs, indices_replaced = self._replace_with_mask(inputs, labels, masked_indices)\n",
    "        \n",
    "        # Randomly replace with mask token\n",
    "        inputs = self._replace_with_other(inputs, labels, masked_indices, indices_replaced)\n",
    "        # The rest of the time (10% of the time) we keep the masked input tokens unchanged\n",
    "        return (inputs,labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# collapse\n",
    "@Numericalize\n",
    "def decodes(self,o):\n",
    "    'Add the ability to parse masks for the loss function, set as `-100`'\n",
    "    if isinstance(o, tuple): o=o[0]\n",
    "    tmp_vocab=self.vocab.copy()\n",
    "    tmp_vocab.append('<loss_mask>')\n",
    "    o=[-1 if o_ == -100 else o_ for o_ in o]\n",
    "    return L(tmp_vocab[o_] for o_ in o if tmp_vocab[o_] != PAD)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# collapse\n",
    "@delegates(Datasets)\n",
    "class Datasets(Datasets):\n",
    "    \"Doesn't create a tuple in __getitem__ as x is already a tuple\"\n",
    "    def __init__(self, items=None, tfms=None, tls=None, n_inp=None, dl_type=None, **kwargs):\n",
    "        super().__init__(items=items, tfms=tfms, tls=tls, n_inp=n_inp, dl_type=dl_type, **kwargs)\n",
    "\n",
    "    def __getitem__(self, it):\n",
    "        # same as Datasets.__getitem__ but not wrapped in a tuple\n",
    "        res = [tl[it] for tl in self.tls]\n",
    "        return res[0] if is_indexer(it) else list(zip(*res))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our dataset is now ready to be created, lets look at an some of our (x,y) that will be passed to the model. When `-100` is passed to our loss function (`nn.CrossEntropyLoss`) it will be ignored in the calculation. Our model will also ignore any padding tokens (usually defined as `1`) when passed to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "(tensor([    0,  1890,    12,  5225, 24320,    12,  8494, 18421, 50264,   328,\n",
       "         14938,  1774,   630,    75,   190,   356,    69, 50264, 32819,   784]),\n",
       " tensor([ -100,  -100,  -100,  5225,  -100,  -100,  -100, 18421,  -100,  -100,\n",
       "          -100,  -100,  -100,    75,  -100,  -100,  -100,  4505,  -100,   784]))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#collapse-hide\n",
    "splits = ColSplitter()(df)\n",
    "tfms=[attrgetter(\"text\"), fastai_tokenizer, Numericalize(vocab=tokenizer_vocab_ls), \n",
    "      AddSpecialTokens(tokenizer), MLMTokensLabels(tokenizer)]\n",
    "\n",
    "dsets = Datasets(df, splits=splits, tfms=[tfms], dl_type=SortedDL)\n",
    "\n",
    "dsets[0][0][:20], dsets[0][1][:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataloader\n",
    "### Padding\n",
    "We need to make sure our padding is done correctly as some transformer models prefer padding on the left while others prefer it on the right. `tokenizer.padding_side` will tell us which side is correct. e.g., BERT, Roberta prefers padding to the right, so we set `pad_first=False`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "#collapse\n",
    "def pad_mlm_input(samples, pad_idx=1, pad_fields=[0,1], pad_first=False, max_seq_len=None, backwards=False):\n",
    "    \"Function that collect `samples` and adds padding, modified `max_len_l` in fastai's `pad_input`\"\n",
    "    pad_fields = L(pad_fields)\n",
    "    #max_len_l = ifnone(max_seq_len, pad_fields.map(lambda f: max([len(s[f]) for s in samples])))\n",
    "    max_len_l = pad_fields.map(lambda f: max_seq_len)\n",
    "    if backwards: pad_first = not pad_first\n",
    "    def _f(field_idx, x):\n",
    "        if isinstance(x, tuple): x=(x[0]) ## Added this line too, removes tuple if present\n",
    "        if field_idx not in pad_fields: return x\n",
    "        idx = pad_fields.items.index(field_idx) #TODO: remove items if L.index is fixed\n",
    "        sl = slice(-len(x), sys.maxsize) if pad_first else slice(0, len(x))\n",
    "        pad =  x.new_zeros(max_len_l[idx]-x.shape[0])+pad_idx\n",
    "        x1 = torch.cat([pad, x] if pad_first else [x, pad])\n",
    "        if backwards: x1 = x1.flip(0)\n",
    "        return retain_type(x1, x)\n",
    "    return [tuple(map(lambda idxx: _f(*idxx), enumerate(s))) for s in samples]\n",
    "\n",
    "def transformer_mlm_padding(tokenizer=None, max_seq_len=None, sentence_pair=False): \n",
    "    'Uses `pad_fields=[0,1]` to pad both input and label'\n",
    "    if tokenizer.padding_side == 'right': pad_first=False\n",
    "    else: pad_first=True\n",
    "    max_seq_len = ifnone(max_seq_len, tokenizer.max_len) \n",
    "    return partial(pad_mlm_input, pad_fields=[0,1], pad_first=pad_first, \n",
    "                   pad_idx=tokenizer.pad_token_id, max_seq_len=max_seq_len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#collapse\n",
    "padding=transformer_mlm_padding(tokenizer, max_seq_len=max_seq_len)\n",
    "\n",
    "bs=4\n",
    "dls = dsets.dataloaders(bs=bs, before_batch=[padding])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check our batch\n",
    "We can see our special RoBERTa tokens (`'<s>'`, `'</s>'`), which translate to `0, 2` in its vocab, have been added to the start and end of each sequence in the batch. Your can look at these indices in `tokenizer.get_vocab()` to confirm this. We can also see that most of the tokens in our target (`text_`) are masked out as we only want to calculate the loss on the ~15% of the `text` tokens that have been masked."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([4, 512]), torch.Size([4, 512]))"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#collapse\n",
    "b=dls.one_batch()\n",
    "b[0].size(), b[1].size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>text_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>&lt;s&gt; I&lt;mask&gt; fortunate enough to meet&lt;mask&gt; Pal segregatedand still have my DS:TMlishing&lt;mask&gt; autographed by&lt;mask&gt;&lt;mask&gt; at a convention shortly&lt;mask&gt; the release, and asked him why he chose to do the film \"camp\". Before&lt;mask&gt; could answer, two studio flacks intercepted and lectured me on how the studio \"knew best\" and how \"no one will take such&lt;mask&gt; film seriously\". I had been reading the Bantam reprints for&lt;mask&gt; couple of years thanks&lt;mask&gt; a&lt;mask&gt; (ComiCon attendees of the 1970s will recall 357hawk and his band? I was in&lt;mask&gt; couple&lt;mask&gt; years of that withnd), and had higher hopes than what we&lt;mask&gt;.&lt;mask&gt;\\nThe flacks insisted that no high adventure would ever be&lt;mask&gt; seriously, and so doing 'camp&lt;mask&gt; was the&lt;mask&gt; way. Several other fans jumped in gap my&lt;mask&gt;, with Pal listening as best he could. At the end of the little event, Pal&lt;mask&gt; up to&lt;mask&gt; and apologized,&lt;mask&gt; he could have done more and better.\\n\\nSTAR WARS put the lie to</td>\n",
       "      <td>&lt;loss_mask&gt;&lt;loss_mask&gt; was&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; George&lt;loss_mask&gt; (&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;OB poster&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; him)&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; after&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; he&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; \"&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; a&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; a&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; to&lt;loss_mask&gt; friend&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; Black&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; a&lt;loss_mask&gt; of&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; him&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; hopes&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; got&lt;loss_mask&gt;\\n&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; done&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;'&lt;loss_mask&gt;&lt;loss_mask&gt; only&lt;loss_mask&gt;.&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; on&lt;loss_mask&gt; side&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; came&lt;loss_mask&gt;&lt;loss_mask&gt; us&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; wishing&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;,&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;'s&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; that&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; it&lt;loss_mask&gt;'t&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;.&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; the&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;,&lt;loss_mask&gt; the&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; rating as&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;.&lt;loss_mask&gt;\\n&lt;loss_mask&gt; destroying the&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; still&lt;loss_mask&gt;&lt;loss_mask&gt; the&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; the&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; have&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; to&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; we&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;hero&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;, there&lt;loss_mask&gt;&lt;loss_mask&gt; second&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;'s&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; serious&lt;loss_mask&gt; Yes&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; with&lt;loss_mask&gt;&lt;loss_mask&gt; And&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; a&lt;loss_mask&gt;&lt;loss_mask&gt;sheet&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; leaping&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; with&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; bronze&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; a&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; tie&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;AV&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; Next&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; If&lt;loss_mask&gt; knows&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; George&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; San&lt;loss_mask&gt; for the&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;s&gt;&lt;mask&gt; is another one of those 'humans vs insects/eco-horror' features; a theme that was popular in the late 70's.&lt;mask&gt; you can't really call it horror. There's zero suspense and no&lt;mask&gt; events.&lt;mask&gt; other words: this movie&lt;mask&gt; pretty lame. It's not that it&lt;mask&gt; really bad or&lt;mask&gt;; it's just very boring. A construction site near&lt;mask&gt; hotel uncovers a big nest of&lt;mask&gt;. Later on we learn that, probably due to&lt;mask&gt; sorts&lt;mask&gt; pesticides Lounge in the past, their&lt;mask&gt; became poisonous. Some people get bitten and rushed to&lt;mask&gt; hospital and it takes ages for&lt;mask&gt;&lt;mask&gt; Vanity the&lt;mask&gt; to figure out what's going on.&lt;mask&gt; Foxworth figures&lt;mask&gt; out first and then you can&lt;mask&gt; him go berserk with a digging machine for what seems like several hours.&lt;mask&gt; they&lt;mask&gt; in the house, waiting&lt;mask&gt; get rescued. And, man, you should see all the efforts they make for&lt;mask&gt; them.&lt;mask&gt; won't spoil too much, but at&lt;mask&gt; point they even use a big&lt;mask&gt;. All the</td>\n",
       "      <td>&lt;loss_mask&gt; This&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; Only&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; gruesome&lt;loss_mask&gt;&lt;loss_mask&gt; In&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; is&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;'s&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; something&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; a&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; ants&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; different&lt;loss_mask&gt; of&lt;loss_mask&gt; used&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; bite&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; the&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; the residents of&lt;loss_mask&gt; hospital&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; Robert&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; it&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; see&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; Then&lt;loss_mask&gt; flee&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; to&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; all&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; rescuing&lt;loss_mask&gt;&lt;loss_mask&gt; I&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; one&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; helicopter&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; this&lt;loss_mask&gt; I&lt;loss_mask&gt;&lt;loss_mask&gt; thinking&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; you&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; on&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; building&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; lots of&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; of&lt;loss_mask&gt;&lt;loss_mask&gt; are shown&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; movie&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; Ant&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; garbage&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; the&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; in&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; straw&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; ants&lt;loss_mask&gt; wider shots&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; designers&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; near&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; do&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; in&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; It&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; as&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; IT&lt;loss_mask&gt;&lt;loss_mask&gt;EN&lt;loss_mask&gt; AT&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; my&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; title&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;K&lt;loss_mask&gt;&lt;loss_mask&gt; MAN&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; have&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;for&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;'ll&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; in&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; Now&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;,&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>&lt;s&gt; The&lt;mask&gt;&lt;mask&gt; saw no fewer than 3 filmed productions&lt;mask&gt; H. G. Wells' great novel, \"War of&lt;mask&gt; Worlds\". This&lt;mask&gt; perhaps the least well-known and very probably the best of&lt;mask&gt;&lt;mask&gt; No other&lt;mask&gt;&lt;mask&gt; W&lt;mask&gt;W has ever attempted not only to present the story very much as Wells wrote&lt;mask&gt;, but also Burton create the atmosphere of the time&lt;mask&gt; which it was supposed to take place: the last year of&lt;mask&gt; 19th Century, 1900 ",
       " using Wells' original setting, in and near Woking&lt;mask&gt;&lt;mask&gt;.\\n\\nIMDb&lt;mask&gt; unfFlyingly to what they regard as \"spoilers\". That might apply&lt;mask&gt; some&lt;mask&gt;, where the ending might actually be a&lt;mask&gt;, but with regard to one of the most famous novels in&lt;mask&gt;&lt;mask&gt;, it seems positively silly. I have&lt;mask&gt; sympathy&lt;mask&gt; people who have neglected to&lt;mask&gt; one&lt;mask&gt; the seminal works&lt;mask&gt; English literature,&lt;mask&gt; let's get right to the chase. The aliens are destroyed through catching an Earth disease,&lt;mask&gt; hits&lt;mask&gt; have no immunity. If that wo a spoiler, so be</td>\n",
       "      <td>&lt;loss_mask&gt;&lt;loss_mask&gt; year 2005&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; of&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; the&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; is&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; them.&lt;loss_mask&gt;&lt;loss_mask&gt; version of&lt;loss_mask&gt;ot&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; it&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; to&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; in&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; the&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;, England&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; seems unfriend&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; with&lt;loss_mask&gt; films&lt;loss_mask&gt; where&lt;loss_mask&gt;&lt;loss_mask&gt; might&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; surprise&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; the world&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; have no&lt;loss_mask&gt; for&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; read&lt;loss_mask&gt; of&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; in&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; so&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; against which they&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;'s&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; other&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; 1953 classic&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; to&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;\\n&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;' plot&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;, is&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; �&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; way&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; off due to&lt;loss_mask&gt;&lt;loss_mask&gt; of&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; Century&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; in&lt;loss_mask&gt;&lt;loss_mask&gt;ides&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; film&lt;loss_mask&gt;&lt;loss_mask&gt; some&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; an&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; old&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; than&lt;loss_mask&gt;).&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; of&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;.&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; are typical of&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;'t&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;,&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; to&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;/white and&lt;loss_mask&gt; on&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;.&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; as&lt;loss_mask&gt; described them&lt;loss_mask&gt; have a more&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;feel\".&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; destruction&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; more&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; period&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; particularly&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; or brilliant&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; facial&lt;loss_mask&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>&lt;s&gt;&lt;mask&gt; watched Grend&lt;mask&gt; the&lt;mask&gt; night and am compelled&lt;mask&gt; evangelical&lt;mask&gt; a Public Service Announcement.\\n\\nGrendel is another version of&lt;mask&gt;owulf, the thousand- resulted-&lt;mask&gt; Anglo-Saxon epic poem.&lt;mask&gt; SciFi channeluture a growing catalog of inoffensive&lt;mask&gt; uninterestingxs,&lt;mask&gt; the previews promised an&lt;mask&gt;authentic low-budget mini-epic, but this one refused to&lt;mask&gt;&lt;mask&gt; switch channels.&lt;mask&gt; was staggeringly, overwhelmingly, bad&lt;mask&gt; I watched in fascination and horror at the train wreck you&lt;mask&gt;'t tear your eyes away from&lt;mask&gt; I reached for a notepad and managed to capture part of what I was seeing.&lt;mask&gt; following may contain spoilers or might just save your sanity&lt;mask&gt; You've been warned.\\n\\n- Just to&lt;mask&gt; it over with, Beow&lt;mask&gt;&lt;mask&gt; warriors wore horned&lt;mask&gt;.&lt;mask&gt;&lt;mask&gt;ial issue compared to what came after. It also appears that the helmets were in a bin and handed&lt;mask&gt; whichever actor wandered by next. Fit,&lt;mask&gt; and function&lt;mask&gt; apparently irrelevant.\\n\\n- Marina Sirtis&lt;mask&gt;&lt;mask&gt; been blackmailed into doing the&lt;mask&gt; by&lt;mask&gt; Ringling Brothers, Barnum and&lt;mask&gt;&lt;mask&gt;.&lt;mask&gt; managed to avoid a red rubber nose, but the</td>\n",
       "      <td>&lt;loss_mask&gt; I&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;el&lt;loss_mask&gt; other&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; to put together&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; Be&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;year&lt;loss_mask&gt;old&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; The&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; has&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; and&lt;loss_mask&gt;&lt;loss_mask&gt; movies&lt;loss_mask&gt; and&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; in&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; let me&lt;loss_mask&gt;&lt;loss_mask&gt;. It&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;.&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; couldn&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;.&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; The&lt;loss_mask&gt;&lt;loss_mask&gt; contain&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; save&lt;loss_mask&gt;&lt;loss_mask&gt;.&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;\\n&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; get&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;ulf's&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; helmets&lt;loss_mask&gt; Triv&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; to&lt;loss_mask&gt; actor&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; appearance&lt;loss_mask&gt;&lt;loss_mask&gt; were&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; had obviously&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; movie&lt;loss_mask&gt; the&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; Bailey circus&lt;loss_mask&gt; She&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; Ben&lt;loss_mask&gt;&lt;loss_mask&gt; not&lt;loss_mask&gt; be&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; H&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; must have&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; film&lt;loss_mask&gt;&lt;loss_mask&gt; hadn&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; the&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; to&lt;loss_mask&gt; him&lt;loss_mask&gt;&lt;loss_mask&gt;\\n&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; facilitate&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; hairst&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;.&lt;loss_mask&gt;&lt;loss_mask&gt; of&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; sideburn&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; and&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; the&lt;loss_mask&gt;.&lt;loss_mask&gt; prove&lt;loss_mask&gt;&lt;loss_mask&gt; a&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;-&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; movie&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; this&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;shaped&lt;loss_mask&gt;&lt;loss_mask&gt;-&lt;loss_mask&gt; and&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; tradition&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; with&lt;loss_mask&gt; volume&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;\\n&lt;loss_mask&gt;&lt;loss_mask&gt; unintended focus&lt;loss_mask&gt;&lt;loss_mask&gt; movie&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; with&lt;loss_mask&gt; bolts&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; recoil&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; the&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt; the&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;&lt;loss_mask&gt;</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#collapse\n",
    "dls.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model\n",
    "Our model can be instantiated with either pretrained or random weights. We also need to be careful to pass the model the `attention_mask` so that the model ignores padding tokens when training. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LMModel(nn.Module):\n",
    "    def __init__(self, lm_model_class=None, tokenizer=None, model_name=None, config_dict=None, pretrained=False):\n",
    "        super().__init__()\n",
    "        self.tok=tokenizer\n",
    "        if pretrained: self.model = lm_model_class.from_pretrained(model_name)\n",
    "        else: self.model = lm_model_class.from_config(config_dict)\n",
    "        self.model = self.model.module if hasattr(self.model, \"module\") else self.model\n",
    "        self.model.resize_token_embeddings(len(tokenizer))\n",
    "            \n",
    "    def forward(self, input_ids):\n",
    "        attention_mask =  (input_ids!=self.tok.pad_token_id).type(input_ids.type())\n",
    "        return self.model(input_ids, attention_mask=attention_mask)[0]    # only return the prediction_scores (and not hidden states and attention)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pretrained Language Model\n",
    "Lets fine-tune our pretrained Language Model. We would typically do this before training the model on our specific text. Note that here we are not training the language model head before we train the full model, but we could do so if we created a splitter and passed it to our learner\n",
    "\n",
    "To load the pretrained HuggingFace model just use `pretrained=True` when calling your model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LMModel(lm_model_class=lm_model_class, tokenizer=tokenizer, model_name=model_name, \n",
    "                  config_dict=config_dict, pretrained=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "From here we train our model as usual using fastai. Note that we use `Perplexity` as our metric as it is a good measure of how well a language model is training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#collapse\n",
    "opt_func = partial(Adam, decouple_wd=True)\n",
    "\n",
    "loss = CrossEntropyLossFlat()\n",
    "\n",
    "learn = Learner(dls, model, opt_func=opt_func, #splitter=model_splitter, \n",
    "                loss_func=loss, metrics=[accuracy, Perplexity()]).to_fp16()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We check our learning rate finder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "SuggestedLRs(lr_min=0.025118863582611083, lr_steep=0.2089296132326126)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEOCAYAAACTqoDjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3yV9dn48c+VCSQQVhJGQPbepqCCCKKIFLHuUa1W6tbW+rRV20dt+3ta2zpbsVKKu0q1KoqKIsXBFsIIewUChEDIgCzIvn5/nDtwCCchgZzc5yTX+/U6r5z7Pve4Tsa58t2iqhhjjDFVhbgdgDHGmMBkCcIYY4xPliCMMcb4ZAnCGGOMT5YgjDHG+GQJwhhjjE9+SxAi0kVEvhaRLSKySUR+5ux/WkS2ish6EZkjIq2rOT9VRDaIyDoRSfJXnMYYY3wTf42DEJGOQEdVXSMiLYHVwA+ABOArVS0TkT8DqOojPs5PBRJVNcsvARpjjKmR30oQqnpAVdc4z/OBLUBnVf1SVcucw1bgSRjGGGMCTIO0QYhIN2A48F2Vl+4APq/mNAW+FJHVInKX/6IzxhjjS5i/byAi0cAHwEOqmue1/zdAGfB2NaeOVtV0EYkDFojIVlVd5OP6dwF3AURFRZ3br1+/en8PxhjTWK1evTpLVWN9vea3NggAEQkHPgXmq+pzXvtvA+4BJqjq0Vpc57dAgao+U9NxiYmJmpRk7dnGGFNbIrJaVRN9vebPXkwCvAJsqZIcJgGPAFOrSw4iEuU0bCMiUcBEYKO/YjXGGHMqf7ZBjAZuBS52uqquE5HJwHSgJZ5qo3UiMgNARDqJyDzn3HhgiYgkAyuBz1T1Cz/Gaowxpgq/tUGo6hJAfLw0z8c+VDUdmOw83wUM9VdsxhhjTs9GUhtjjPHJEoQxxhifLEEYY4zxyRJEE1VWXsH2jHy3wzDGBDBLEE1QWXkFP3t3HROfX8TqPYfdDscYE6AsQTQx5RXKw+8l89n6A4jAl5sPuh2SMSZAWYJoQsorlF/+J5m5yek8MqkfF/Rsx8Ith9wOyxgToCxBNBEVFcqjH6znw7X7+cXEPtw7ricT+sWz81ABe7IL3Q7PGBOALEE0ES9/m8J/Vqfxswm9eeDi3gBM6B8HYKUIY4xPfp/N1bhveUo2z365jSuGduKhS3of339Ouyh6xUWzcGsGd4zpXqtrlZVXsCr1MP/dksGRo6VEhIUQGRZCRFgIx0rKySsqJe9YKcdKy+nXoRWjurdlZPe2tIuO9NfbM8b4iSWIRu5QXhEPzl5L9/ZRPHX1YDxzKJ4woX8cryzeTX5RKS2bhVd7nXX7jjD7u70s2JJBTmEJkWEhtI+OpLisgpKyckrKK2gREUarZmG0ah5OWIjw71V7eX1ZKgC946KZ0D+eywbGMzShNSEhvmZhMcYEEksQjVhZeQUPzF5LYXEZ79w5iujIU3/cE/rF849vd7FoexbfH9LxlNdVlX8u3sWfv9hG8/BQLu4Xx6RBHRjXN5YWETX/+pSUVbBhfy4rd+ewZGcmsxbvYsa3KcS3iuSGxC78/NI+pyQsY0zgsATRiD3z5XZW7s7hueuH0ie+pc9jRnRtTesW4SzcmnFKgsgrKuUX7yXz5eYMJg/uwJ+vGVJjKaOqiLAQzj2nDeee04Z7x/Uk92gpC7dmMGftfv721U6+170tF/b2uU6JMSYAWCN1I7V0ZxYzvk3hppFduXpE9ct+h4WGML5vHN9sy6S84sTiUVsO5DH1xSV8tfUQj08ZwEs3j6hTcvAlpkU4V49IYNZticS3iuTFr3ae1fWMMf5lCaIRyj1Wyi/+k0yP2CiemDLgtMdf3C+OnMIS1u3zjKr+YHUaV/19KcdKy/n3XecxbUz3eq0KigwL5e6xPVm5O4eVu3Pq7brGmPplCaIR+t3cTRzKL+b564fRPCL0tMeP7RNLWIjw+YaD/O9HG/if/yQzrEtrPn3wQhK7tfVLjDeN7Eq7qAimf22lCGMClT+XHO0iIl+LyBYR2SQiP3P2txWRBSKyw/napprzb3OO2eGsYW1q4fMNB/hw7X4eGN+LoV1a1+qcmObhfK9bW2Yt2c2/Vuzl7rE9+Ne0UcS29F/X1OYRofzkwh4s2p5J8r4jfruPMebM+bMEUQb8j6r2B84D7heRAcCjwEJV7Q0sdLZPIiJtgSeBUcBI4MnqEok54VB+Eb+es4HBnWN44OJedTr3usQEWrcI5+UfjuCxyf0JC/V/4fKW87oS0zzcShHGBCi/fQqo6gFVXeM8zwe2AJ2BK4E3nMPeAH7g4/TLgAWqmqOqh4EFwCR/xdoYqCqPfrCBoyXlPH/DUMLr+AF/9YgE1j5+KZcPPrWrq7+0bBbOj0d3Y8HmDLYcyGuw+xpjaqdB2iBEpBswHPgOiFfVA+BJIkCcj1M6A/u8ttOcfaYary9L5auth3j08n70ivPdpfV03BiTcPsF3YiODOOZ+dsoLa9o8PsbY6rn9wQhItHAB8BDqlrbfxN9fVKpj32IyF0ikiQiSZmZmWcaZlDblJ7LU/O2cnG/OG6/oJvb4dRJ6xYR/HRCLxZuPcQPZ33Hofwit0Myxjj8miBEJBxPcnhbVT90dmeISEfn9Y6Ar5ni0oAuXtsJQLqve6jqTFVNVNXE2NimN+jqaEkZD85eS+sW4Tx97ZCgHJl819iePH/DUNanHWHK35aQlGpdX40JBP7sxSTAK8AWVX3O66W5QGWvpNuAj32cPh+YKCJtnMbpic4+U8Vv525id1YhL9wwLKgnxLtqeAJz7htN84hQbpy5gpe/SaGkzKqcjHGTP0sQo4FbgYtFZJ3zmAz8CbhURHYAlzrbiEiiiMwCUNUc4P8Bq5zH7519xsvc5HTeS0rjvnE9uaBXe7fDOWv9O7Zi7gNjmNA/jj9/sZVJf13Eou1Ns9rQmEAgqj6r9oNSYmKiJiUluR1Gg9iVWcDU6UvpEx/Nu3efX+deS4Huq60Z/P6TzaRmH+XSAfE8/v0BdG3Xwu2wjGl0RGS1qib6eq1xfao0EcdKyrnv7TWEhwrTbx7R6JIDwMX94pn/87H8alJflu7M4pLnvuWpeVvIPVbqdmjGNBmN75OlCXji441sPZjPczcMo1Pr5m6H4zeRYaHcN64XX/9iHFOHdWLm4l2Mf+Yb3lqeal1ijWkAliCCzHtJ+/jP6jQeGN+L8X19DSFpfOJbNeOZ64byyQNj6BMfzeMfb+KS575lztq0k2agNcbUL0sQQWTrwTye+Hgj5/dox88v7eN2OA1uUOcYZt95Hq/clkiLiDB+/m4yl72wiHkbDtCY2tKMCRSWIILIkx9vIjoyjL/eNIzQJrpkp4gwoX88nz04hpduHgHAfW+v4fp/LGdTeq7L0RnTuFiCCBLLUrL4bncO94/vRVzLZm6H47qQEOH7Qzoy/6Gx/OnqwaRkFnLFi0t44uON5B61hmxj6oMliCCgqrywYAfxrSK5aWRXt8MJKKEhwo0ju/L1/4zj1vPO4V8r9jDhuW85kHvM7dCMCXqWIILAspRsVqbmcN+4XjQLP/0CQE1RTItwfnflID6+fwz5RaX8cd5Wt0MyJuhZgghwqsrzC7bToVUzbvhel9Of0MQNTojhnot68klyOstTst0Ox5igZgkiwC3ZmUXSnsPcP76nlR5q6d5xPUlo05zfzt1k4yWMOQuWIAJYZemhU0wzrrfSQ601Cw/l8SkD2JaRz1vL97gdjjFByxJEAFu8I4s1e49w3/heRIZZ6aEuJg6IZ2yfWJ5fsJ3M/GK3wzEmKFmCCGCzluwmrmUk1yda6aGuRIQnrxhAUVk5f/rcGqyNOROWIALUzkMFLNqeya3nnUNEmP2YzkTP2GimjenBB2vSWJaS5XY4xgQd++QJUG8uTyUiNISbRtm4h7Pxswm96dauBY99uIFjJeVuh2NMUPHninKvisghEdnote9dr8WDUkVkXTXnporIBue4prHAg5e8olI+WJ3GlKEdaR/Eq8QFguYRoTx19RD2ZB/luQXb3A7HmKDizxLE68Ak7x2qeoOqDlPVYXjWqv7Q14mO8c6xPheyaMzeT0qjsKScH1/Q3e1QGoXze7bj5lFdeWXJbtbtO+J2OMYEDb8lCFVdBPhcJtRZr/p6YLa/7h+sKiqUN5ancu45bRicEON2OI3Go5f3I65lMx55f72tdW1MLbnVBnEhkKGqO6p5XYEvRWS1iNzVgHG57pvth9iTfZTbL+jmdiiNSqtm4fzfDwaxLSOf5xZst+nBjakFtxLETdRcehitqiOAy4H7RWRsdQeKyF0ikiQiSZmZwb/A/WtLU4lvFcmkQR3cDqXRuWRAPNeMSGDGtync+WaSjY8w5jQaPEGISBhwNfBudceoarrz9RAwBxhZw7EzVTVRVRNjY2PrO9wGlZJZwOIdWdwy6pxGuc50IHj62iE8PmUAi3ZkcdkLi/hi4wG3QzImYLnxKXQJsFVV03y9KCJRItKy8jkwEdjo69jG5qO1+wkRuGGkDYzzl5AQYdqY7nz24Bg6tW7GPf9awyPvr6eo1LrAGlOVP7u5zgaWA31FJE1Epjkv3UiV6iUR6SQi85zNeGCJiCQDK4HPVPULf8UZKFSVT5LTOb9nO1sQqAH0jm/JnPtGc//4nrybtI8bZ64gI6/I7bCMCShh/rqwqt5Uzf7bfexLByY7z3cBQ/0VV6DalJ5HavZR7r6op9uhNBnhoSH88rJ+DO4cw8PvJTPlxSXMuOVczj2njduhGRMQrKI7QHySnE5YiDBpoDVON7RJgzoy577RNA8P5caZy/njvC3syMh3OyxjXGcJIgCoKp+uP8CFvdvTJirC7XCapL4dWjL3gdFcNrADryzZzaXPL2Lq9CW8uTyV3GO2xrVpmixBBIA1e4+w/8gxpgzp5HYoTVrrFhFMv3kE3/16Ao9PGUBpufLEx5s4748LeezD9WxOz3M7RGMalN/aIEztfZKcTkRYCJcOjHc7FAO0j45k2pjuTBvTnY37c3lr+R7mrN3P7JX7SDynDdPGdGfiwA6EhojboRrjV1aCcFl5hfLZhgOM7xtLq2bhbodjqhjUOYY/XzuE7x67hP/9fn8O5Rdz79truPjZb3hrearNEGsaNStBuGzl7hwy84uteinAxbQI5ycX9uDHo7szf9NB/rFoF49/vIlnvtzOJf3jmTgwnrG9Y2keYSv/mcbDEoTLPlmfTvPwUCb0j3M7FFMLoSHC5MEduXxQB1alHmb2yr0s2HyQD9ak0Sw8hIv6xHLZwA5M6BdPTAsrEZrgZgnCRaXlFXyx8SCXDIinRYT9KIKJiDCye1tGdm9LaXkF3+3KYf6mgyzYnMH8TRmEhQjn92zHlcM6M3VoJ1sV0AQl+1Ry0bp9R8gpLOFym5gvqIWHhjCmd3vG9G7P76YOJDntCPM3ZfDFxgP84j/JPDN/G3eM6cZNI7vS0tqZTBCxBOGiZTuzEYELerZzOxRTT0JChOFd2zC8axsemdSXRTuymPFNCn+ct5UXv9rJNSMSmDy4I4nntCHEekGZAGcJwkXLd2XRv0MrWrewwXGNkYhwUZ9YLuoTS/K+I8xcvIt3Vu7l9WWpxLWM5PJBHfj+kE6WLEzAsgThkqLSctbsPcKPzjvH7VBMAxjapTUv3TyCguIyFm7JYN6GA/x71T7eWL6H+FaRXD6oI98f4ilZeBZcNMZ9liBcsmbPYUrKKjjfqpealOjIMK4c1pkrh3WmsLiMhVsP8dn69OMliyuHdeLP1wyhWbh1lzXuswThkuW7sgkN8fSEMU1TVGQYU4d2YurQThQUl/HK4t08/9/t7M4qZOatiXSIsWnfjbus751LlqdkM6hzjPVqMYCnZPGzS3oz89ZzSTlUwNTpS1i797DbYZkmzhKEC46WlLFu3xHO72HVS+ZkEwd24MP7RhMZHsJ1M5Zz3YxlPDN/G0t2ZNm0Hk3UfzdncNXfl/LhmjQqKrRB7+3PFeVeFZFDIrLRa99vRWS/iKxzHpOrOXeSiGwTkZ0i8qi/YnTLqtTDlFWodW81PvXt0JK594/hzrE9KClXXv42hVte+Y6hv/uSH726kjeWpbIv56jbYRo/U1VmLd7FnW8lsf1gPg+/l8y1M5axcX9ug8Ugqv7JSCIyFigA3lTVQc6+3wIFqvpMDeeFAtuBS4E0YBVwk6puPt09ExMTNSkpqR6i968/fb6VV5bsIvnJiTaC2pxWQXEZSak5LN6RxddbD7ErqxCAXnHRjO0dy4V92nNe93Y2D1QjUlpewRMfb2L2yr1MGtiBZ68fymcbDvCXL7aSXVjCzSO78viUAfXSmUFEVqtqoq/X/Lnk6CIR6XYGp44EdjpLjyIi/wauBE6bIILF8l3ZDE1obcnB1Ep0ZBjj+sYxrm8cj08ZwO6sQhZuyeDb7Zm8/d0eXl26m4jQEH4wvBO/v3KQ9YAKcsVl5Ux7PYklO7O4d1xPfjmxLyEhwvWJXZg0qAMvLNjBa8t2s3F/LjN/lEh8K/91ZnCjDeIBEVnvVEH5Wvy3M7DPazvN2dco5BWVsiHtiFUvmTPWvX0UP7mwB29NG0XykxN5846RXJeYwHtJadz6ynccOVridojmLLy/Oo0lO7P4w1WDeGRSv5MGUbZqFs4TVwzgH7ecyw6nM8P6tCN+i6WhE8TLQE9gGHAAeNbHMb5GCVVbDyYid4lIkogkZWZm1k+UfrRqdw4VCudZgjD1oFl4KGP7xPKHqwbz4k3DSd6Xy9UvL2NvtrVRBKPS8gpe/iaF4V1bc/PIrtUeN3FgBz649wLCQjydGT5JTvdLPA2aIFQ1Q1XLVbUC+Cee6qSq0oAuXtsJQLXvXlVnqmqiqibGxsbWb8B+sDwlm4iwEEZ09VV4MubMXTG0E2/fOYqcwhKu+vtSvt52CH+1MRr/+GjtftIOH+PBi3uddkR9/46t+PiB0QxJiOH/PttMYXFZvcfToAlCRDp6bV4FbPRx2Cqgt4h0F5EI4EZgbkPEVxur9+Rw5UtLOVpyZj+MZSnZnNu1jdUTG7/4Xre2fHDvBUQ3C+PHr63ishcWMXvlXopKrYtsoCuvUP7+TQoDO7VifN/arQ/TPjqSf/1kFLPvPI+oyPpv0/RnN9fZwHKgr4ikicg04C8iskFE1gPjgZ87x3YSkXkAqloGPADMB7YA76nqJn/FWVerUg+TvO8IOzIK6nxuflEpWw7mMaqHjZ42/tMzNpovfz6WZ64bSlhICI99uIHzn1rII++v59P16RwutDaKQPTp+nR2ZxXWqvTgLTIslB6x0X6JyZ+9mG7ysfuVao5NByZ7bc8D5vkptLOS4/xxpWYXMrRL6zqdu3F/HqowrI7nGVNXkWGhXHtuAteM6MzK3Tm8uWIP8zYe4N2kfYjA4M4xXNwvjkv6xzOwUyubINBlFRXKS1/vpE98NBMHBM76MNbPso6yCzwJYrfTF70uKnsbDEmwBGEahogwqkc7RvVoR1l5Bev357JkRxbfbs/krwt38MJ/d9AxphmXDojn1vPOoXd8S7dDbpK+3HyQ7RkF/PXGYQE19bsliDrKKSwGIPWMEkQuCW2a0zbK1n8wDS8s1NM5YkTXNvx0Qm+yC4r5aushFm45xHtJ+3hz+R4u6R/H3Rf1tGnHG5Cq8uJXO+nePoopQzq5Hc5JLEHUUWUV0+4z6EaYnHaEoVZ6MAGiXXQk1yV24brELuQUlvDm8lTeWJbKdTOWM7JbW569fihd2rZwO8xGb8nOLDal5/HnawYTGkClB7DJ+uosq7KKKbOgTl0IswuKSTt8jCEJMf4KzZgz1jYqgocu6cOyRyfw+ysHsuVgHldMX8Ki7YE/tijYzVy0i7iWkfxgeOCNB7YEUUc5hSVEhIWQV1TG4aOltT5vgzPBlrU/mEDWPCKUH53fjU8eGEN8y2bc9tpKXvp6Z4PPItpUbErPZfGOLG4f3Y3IsMDr+m4Jog6OlZRzrLScoU4poC4N1evTchGBQZ1b+Ss8Y+pNt/ZRzLn/Aq4Y0omn52/jttdWsnRnlg28q2ezFu8mKiKUH44KzKWHLUHUQbbTQD3iHM8o6Lo0VK9PO0KP9lG2QJAJGi0iwvjrjcP43dSBbErP44ezvuOS577ljWWp5BfVvvRsfEs/coxPktO54XtdiWkemJ8LliDqoLKBelhCa0LEMxaiNlSV5LRca6A2QUdEuO2Cbix79GKevW4o0c3CeXLuJsb+5WtmLd5FcZmN0D5Try3djQJ3jOnmdijVsgRRB9lOgoiPaUZCmxa1rmLKyCsmM7/YGqhN0GoWHso15ybw8f2jmXPfBQzqHMP/fbaFi5/5ljlrG36ls2CXV1TK7JX7mDKkIwltArenmCWIOqgcJNcuKoJu7aNqXYJIrhwgZyOoTSMwvGsb3po2in9NG0WbqHB+/m4yV7601NbQroPZ3+2loLiMOy/s4XYoNbIEUQeVg+TaRkXQvV0LUrOO1qrRbn3aEcJChAEdrYHaNB5jerdn7v1jeOGGYRzKL+Kqvy/jV+8nk11Q7HZoAS27oJiZi3Yxulc7BnUO7FoFSxB1kF1YQkRoCNGRYXRrH0VBcdnxcRE1WZ+WS5/4ljaDq2l0QkKEHwzvzML/GcfdY3vw4Zr9jH/mG97+bo9VO/mgqvzvRxvJLyrjiSkD3Q7ntCxB1EFOQQltoyIQEbq1jwJO31CtqqxPy2Vol8D+T8GYsxEdGcZjk/vzxUNjGdgpht/M2chN/1zBrsy6z3rcmM1NTufzjQd5eGIf+nYI/HmvLEHUQU5hyfF5lLq38ySI0zVU7805Su6xUhsgZ5qEXnHRvHPnKP5yzRC2HMhj0l8X89LXOykrr3A7NNcdzC3i8Y82MqJr64Bve6hkCaIOsgtLaBftSRAJbZoTFiKnHQuRnOYZQT04wOsajakvIsL13+vCfx++iAn94nh6/jZ+9OrKJt02oao88sF6SsuVZ68fFnBzLlXHnwsGvSoih0Rko9e+p0Vkq4isF5E5IuLz32oRSXUWFlonIkn+irGucgpLaOeUIMJCQ+jStsVpq5jW7ztCZFhIUBQnjalPca2a8fIt5/L0tUNI2nOYqdOXHp/y3i3lLrWLvLNyL99uz+Sxyf3o7lRPBwN/liBeByZV2bcAGKSqQ4DtwGM1nD9eVYepaqKf4quz7IJi2kZFHt/u1q4Fu7NqntV1/f5cBnRqRXioFdZM03RdYhc+uOcCAK6dsZz3kva5EsdLX++k56/n0fs38xjy2/mM+uN/ufdfq/2+wt66fUf43SebubB3e24J0Ck1quO3Ty1VXQTkVNn3pbOkKMAKIMFf969vRaXlFJaUH69iAs98NXuyC6vt6qqqbDuYT3/r3mqauMEJMcx9YDTf69aGX72/nv/9aAMlZQ3XLnHkaAkvf5PCiK6t+cmFPbh6RAKje7Vn4ZZDXDF9CZvSc/1y34y8Iu56M4n4VpH87cbhAbUYUG24+W/tHcDn1bymwJcislpE7mrAmKpVOc2G92I/3dtHcbSknEP5vutWcwpLyD1WSk8/rRdrTDBpFx3JGz8eyd1je/CvFXu56Z8ryMgrOuPr7T9yjFtf+a5WJZJZi3dTUFzGU1cP4ZFJ/fjt1IE8d/0w3r37PMrKlWteXsZHa/efcSy+FJWWc/dbqykoLuOfP0qkTRAuFOZKghCR3wBlwNvVHDJaVUcAlwP3i8jYGq51l4gkiUhSZqb/5q73lSC6naYn0y5nf4/Y4KlzNMafwkJDeGxyf6bfPJwtB/KY8uISVqXmnP7EKjLyirj5nytYvCOLX72/nl/P2VDtvFCHC0t4beluvj+44yltgcO7tuGTB8cwpHNrHnp3Hb+es4G8epiIUFX5zZyNrNt3hOeuH0q/DsFZi9DgCUJEbgOmAD/UaupmVDXd+XoImAOMrO56qjpTVRNVNTE2NtYfIQMn5mFqV6UEAdXP6ppyyNMHvJeVIIw5yZQhnZhz32iiIkK5bsZyrp+xnDlr0ygqPf3kf5n5xdz8zxVk5Rfz/j3nc89FPXnnu73c8I8VHMg9dsrxs5bs4mhpOT+d0Nvn9WJbRvL2naO4a2wP/r1yLxOe/ZZP16ef8dTmKZkF/PL99XywJo2fTejNpEEdz+g6gaBWCUJEeopIpPN8nIj8tLoeSKe5ziTgEWCqqvps3RWRKBFpWfkcmAhs9HVsQ/KeZqNSp9bNiQgNYXc1PZl2ZRUSERZCp9bNGyRGY4JJ3w4tmfvgGB67vB+H8ov4+bvJjPrjQv7yxdZqE0VOYQm3zPqO9CNFvPbjkSR2a8ujl/fj5R+OYEdGPle8uIT5mw4eP/5wYQmvL01lso/Sg7fw0BB+Pbk/H98/hvhWkTzwzlpuf20V2w7m1+q9qCordmXzkzdWMeHZb5mbnM60Md35WTVJKVjUtgTxAVAuIr2AV4DuwDs1nSAis4HlQF8RSRORacB0oCWwwOnCOsM5tpOIzHNOjQeWiEgysBL4TFW/qOsbq2/HJ+qLPtGLKTRE6NK2ebUliF2ZBXRvFxU0fZ6NaWitmoVz90U9+foX43jnzlGM6dWev3+TwtTpS9i4/0TDcUWF8sXGg1w3Yxmp2YXMui2Rkd3bHn/98sEd+fiB0bSPjuTut1Zzz1urycgr4p+LPaWH2n5QD06I4aP7RvPElAEkpeZw2QuLuHHmcuZtOEBpNYP9VJXff7qZG2euYM3eI/x0Qm+WPnIxj08ZEHSN0lWF1fK4ClUtE5GrgBdU9UURWVvTCap6k4/dr1RzbDow2Xm+Cxhay7gaTHZhCeGhQqtmJ3/LurePIrWarq4pmYX072jjH4w5HRHhgp7tuaBne67bdohfvb+eq/6+lIcu6UOXti146audbMvIp1u7Frxy2/cY3av9KdfoFdeSTx4cw8xFu/jrwh0sfS6LsnLl+4M70ie+9n+HYaEh3DGmO1cN78y7Sft4a/ke7nt7DR1aNePX3+/P1KGdjh+rqvzp8628tjSV2y/oxqOX92tUc67VtgRRKiI3AbcBnzr7AnMJJD/JKSihTQvPPEzeesRGszu78JT/LkrKKtibc5Qe7a39wZi6GNc3jvkPjeXSAfE8PX8bP529lnJVXrhhGP99+D4deWMAABNYSURBVCLG9D41OVQKDw3h/vG9mP/QWAZ3jqFc9YyredpERXDPRT1Z9KvxzPpRIvExzfjp7LU88M4ajhz11Cg8/98d/GPRLm45rytPXjGgUSUHqH0J4sfAPcAfVHW3iHQH/uW/sAJPttc8TN6GJMRQUlbB1gP5DPZaEGhvzlHKK9R6MBlzBtpERfDSzSP4etshyitgQr+4OlXXdG8fxds/GUV+cRmtznKZ39AQ4ZIB8YzrG8uMb1N44b87WLk7hwn945i9ch/XJybw+6mDTvnnsTGoVYJQ1c3ATwFEpA3QUlX/5M/AAk1OYfFJg+QqDe/qWZ967b7DJyWIylksbQyEMWdGRLi4X/xZnX+2ycFbWGgID1zcm3F94/j5u+uYvXIfVw3vzFNXDwn6tobq1CpBiMg3wFTn+HVApoh8q6oP+zG2gJJTWMLgNqd23OoU04y4lpGs3XuEH51/Yn9Kpo2BMKYxGtQ5hk8eHMPylGwu7N2+UXdCqW0bRIyq5gFXA6+p6rnAJf4LK/Bke03U501EGN619SnLLe7KLCC2ZSQt6/E/GGNMYGgWHsr4fnGENfI51mr77sJEpCNwPScaqZuM4rJy8ovKfCYI8FQzpWYfPT7aGjxjIHpa6cEYE8RqmyB+D8wHUlR1lYj0AHb4L6zAcrjQM/S+rY82CIDhXTxVT+v2nShFpGQW0MPaH4wxQaxWCUJV/6OqQ1T1Xmd7l6pe49/QAke2M4q6uhLE4IQYQkOEtXs9c93nFJZw5GgpPYJo3ndjjKmqtlNtJDgL/BwSkQwR+UBEgmaq7rN1YqK+SJ+vt4gIo298y+MJ4ngPpjgrQRhjgldtq5heA+YCnYDOwCfOvibB10yuVQ3v2prkfUeoqFBSKhOEDZIzxgSx2iaIWFV9TVXLnMfrgP+mTg0wx+dhqjFBtCG/uIyUzAJ2ZXom6evcxibpM8YEr9omiCwRuUVEQp3HLUC2PwMLJNmFxYSGCDHNq++yOryrp6F67d4jpGQW0q1di0bdP9oY0/jVNkHcgaeL60HgAHAtnuk3moScQs88TDWNluzeLoqY5uGs3XeYXVkFNoLaGBP0atuLaa+qTlXVWFWNU9Uf4Bk01yRkF/geJOctJEQY1qU1K3fnsDf7qI2gNsYEvbMZBtikptmoqYG60vCurUnJLKSsQm0WV2NM0DubBNFkKthzCkuqHSTnrXLiPrAursaY4Hc2CeK0C7aKyKvO2ImNXvvaisgCEdnhfG1Tzbm3OcfscNaxdk118zBVNSzhxGR+VsVkjAl2NSYIEckXkTwfj3w8YyJO53VgUpV9jwILVbU3sNDZrnrftsCTwChgJPBkdYnE30rLK8g9VlqrKqaYFuH0iI2ifXRkvU4zbIwxbqhxum9VPav1MlV1kYh0q7L7SmCc8/wN4BvgkSrHXAYsUNUcABFZgCfRzD6beM7E4cLTj4Hwdtv53cj2mrTPGGOCVW1XlKtP8ap6AEBVD4hInI9jOgP7vLbTnH0NrvLDvl2072k2qrrtgm5+jMYYYxpOoE5m7qsB3Gebh4jcJSJJIpKUmZlZ74Fk5BUB0L6WCcIYYxoLNxJEhrO2BM7XQz6OSQO6eG0nAOm+LqaqM1U1UVUTY2Prf/aPrQfzAegbf1a1bcYYE3TcSBBzgcpeSbcBH/s4Zj4wUUTaOI3TE519DW5zeh6dWzcnpoU1Ohtjmha/JggRmQ0sB/qKSJqITAP+BFwqIjuAS51tRCRRRGYBOI3T/w9Y5Tx+X9lg3dA2H8hjQKdWbtzaGGNc5ddGalW9qZqXJvg4Ngn4idf2q8CrfgqtVo6VlLMrs4DJgzu6GYYxxrgiUBupA8K2jHwqFAZ0tBKEMabpsQRRg83peQAMtComY0wTZAmiBpsP5NIyMowEW/jHGNMEWYKowZYD+fTv1AqRJjMvoTHGHGcJohoVFcqWA3nW/mCMabIsQVRjT85RjpaUWxdXY0yTZQmiGpUN1FaCMMY0VZYgqrH5QC5hIULveFv4xxjTNFmCqMbm9Dx6xUUTGRbqdijGGOMKSxDV2GwN1MaYJs4ShA/ZBcVk5BVbA7UxpkmzBOHDlgOeKb6tBGGMacosQfiw+UAuAP0tQRhjmjBLED5sTs+jU0wz2tRyHWpjjGmMLEH4YGtAGGOMCwlCRPqKyDqvR56IPFTlmHEikut1zBMNFV9RaTkpmYXW/mCMafL8umCQL6q6DRgGICKhwH5gjo9DF6vqlIaMDeBQXjHlFUrXdlENfWtjjAkoblcxTQBSVHWPy3Ecl1dUCkBMc1uD2hjTtLmdIG4EZlfz2vkikiwin4vIwIYKKO+YJ0G0bNbghStjjAkoriUIEYkApgL/8fHyGuAcVR0KvAh8VMN17hKRJBFJyszMPOu48orKAGjVzEoQxpimzc0SxOXAGlXNqPqCquapaoHzfB4QLiLtfV1EVWeqaqKqJsbGxp51UJVVTFaCMMY0dW4miJuopnpJRDqIs4ybiIzEE2d2QwRVWcXUytogjDFNnCv/JotIC+BS4G6vffcAqOoM4FrgXhEpA44BN6qqNkRs+UVliEDLSCtBGGOaNlc+BVX1KNCuyr4ZXs+nA9MbOi7wVDFFR4QREmLrUBtjmja3ezEFnPyiMqteMsYYLEGcIu9YqTVQG2MMliBOkVdUal1cjTEGSxCn8FQxWQnCGGMsQVRhJQhjjPGwBFFF3rEya4MwxhgsQZxEVckvKrVeTMYYgyWIkxSWlFOhNs2GMcaAJYiTHJ9mw9ogjDHGEoS3/MqZXK2KyRhjLEF4s5lcjTHmBEsQXqyKyRhjTrAE4cWqmIwx5gRLEF6siskYY06wBOGlsgRhCcIYYyxBnCTvWCmRYSFEhoW6HYoxxrjOtQQhIqkiskFE1olIko/XRUT+JiI7RWS9iIzwd0x5NoraGGOOc7suZbyqZlXz2uVAb+cxCnjZ+eo3eUVltLLqJWOMAQK7iulK4E31WAG0FpGO/ryhZ7EgK0EYYwy4myAU+FJEVovIXT5e7wzs89pOc/b5TZ4tN2qMMce5WZ8yWlXTRSQOWCAiW1V1kdfr4uMcrbrDSS53AXTt2vWsAsovKqVLm+ZndQ1jjGksXCtBqGq68/UQMAcYWeWQNKCL13YCkO7jOjNVNVFVE2NjY88qJs9aEFaCMMYYcClBiEiUiLSsfA5MBDZWOWwu8COnN9N5QK6qHvBnXJ5eTNZIbYwx4F4VUzwwR0QqY3hHVb8QkXsAVHUGMA+YDOwEjgI/9mdARaXllJRV2DxMxhjjcCVBqOouYKiP/TO8nitwf0PFdHweJuvmaowxQGB3c21Q+c48TNaLyRhjPCxBOPJsHiZjjDmJJQiHrQVhjDEnswThsLUgjDHmZJYgHLYWhDHGnMwShMOqmIwx5mSWIBz5RWWEhggtImwtCGOMAUsQx+UVldKyWRjO4D1jjGnyLEE48o6VWvWSMcZ4sQThyC8qs3mYjDHGiyUIR15RKS0jrQRhjDGVLEE48o5ZCcIYY7xZgnDkF9lyo8YY480ShCOvqMwaqY0xxoslCKC8QikotiomY4zxZgkCKDg+k6uVIIwxplKDJwgR6SIiX4vIFhHZJCI/83HMOBHJFZF1zuMJf8ZUOQ+TLRZkjDEnuPGJWAb8j6qucdalXi0iC1R1c5XjFqvqlIYIKM8WCzLGmFM0eAlCVQ+o6hrneT6wBejc0HF4yztmiwUZY0xVrrZBiEg3YDjwnY+XzxeRZBH5XEQG1nCNu0QkSUSSMjMzzyiOE1VMVoIwxphKriUIEYkGPgAeUtW8Ki+vAc5R1aHAi8BH1V1HVWeqaqKqJsbGxp5RLJWLBcVYFZMxxhznSoIQkXA8yeFtVf2w6uuqmqeqBc7zeUC4iLT3VzyVa0FYFZMxxpzgRi8mAV4Btqjqc9Uc08E5DhEZiSfObH/FVFnFFB1pCcIYYyq58Yk4GrgV2CAi65x9vwa6AqjqDOBa4F4RKQOOATeqqvoroPyiMqIiQgkLtWEhxhhTqcEThKouAWpclUdVpwPTGyYiZy0Ia38wxpiT2L/MOGtBWA8mY4w5iSUITiw3aowx5gRLEHgShFUxGWPMySxBUFnFZCUIY4zxZgkCTyO1zeRqjDEnswQBtI+OpGPrZm6HYYwxAcXqVYAFD1/kdgjGGBNwrARhjDHGJ0sQxhhjfLIEYYwxxidLEMYYY3yyBGGMMcYnSxDGGGN8sgRhjDHGJ0sQxhhjfBI/rsPT4EQkF9jhtSsGyK3l8/ZA1hnc1vtadT2m6v6atoMh/pri9N6uz/hriu90r58u/qrbvp5b/IERPwTG30Aw/g23VtVYn2epaqN5ADOr2z7dcyCpPu5Zl2NqijcY468pziqx1lv8tXkPZxp/Lb/vFn8AxH8278H+hqs/r7FVMX1Sw3ZtntfHPetyTE3xVt0Ohvir7qvu/dRn/LW5xpnGX3Xb13OLv/HHX9MxjfFv+LhGVcV0NkQkSVUT3Y7jTFn87rL43Rfs7yEQ429sJYizMdPtAM6Sxe8ui999wf4eAi5+K0EYY4zxyUoQxhhjfLIEYYwxxidLEMYYY3yyBFELInKhiMwQkVkisszteOpKREJE5A8i8qKI3OZ2PHUlIuNEZLHzMxjndjxnQkSiRGS1iExxO5a6EpH+zvf+fRG51+146kpEfiAi/xSRj0VkotvxnAkR6SEir4jI+w1530afIETkVRE5JCIbq+yfJCLbRGSniDxa0zVUdbGq3gN8Crzhz3irqo/4gSuBzkApkOavWH2pp/gVKACaEZzxAzwCvOefKKtXT7//W5zf/+uBBu2GWU/xf6SqdwK3Azf4MVyf6uk97FLVaf6N1PeNG/UDGAuMADZ67QsFUoAeQASQDAwABuNJAt6POK/z3gNaBVv8wKPA3c657wdh/CHOefHA20EY/yXAjXg+oKYEW/zOOVOBZcDNwRi/c96zwIiGjN8P76FB/37DaORUdZGIdKuyeySwU1V3AYjIv4ErVfUpwGcVgIh0BXJVNc+P4Z6iPuIXkTSgxNks91+0p6qv77/jMBDpjzirU0/f//FAFJ4PgGMiMk9VK/wauKO+vv+qOheYKyKfAe/4L+JT7lsf338B/gR8rqpr/Bvxqer5b6BBNfoEUY3OwD6v7TRg1GnOmQa85reI6qau8X8IvCgiFwKL/BlYLdUpfhG5GrgMaA1M929otVKn+FX1NwAicjuQ1VDJoQZ1/f6PA67Gk5zn+TWy2qnr7/+DeEpxMSLSS1Vn+DO4Wqrrz6Ad8AdguIg85iQSv2uqCUJ87KtxxKCqPumnWM5EneJX1aN4ElygqGv8H+JJcoGizr8/AKr6ev2Hckbq+v3/BvjGX8GcgbrG/zfgb/4L54zU9T1kA/f4LxzfGn0jdTXSgC5e2wlAukuxnAmL310Wv7uCPX4IkvfQVBPEKqC3iHQXkQg8DYhzXY6pLix+d1n87gr2+CFY3kNDt+i70INgNnCAE108pzn7JwPb8fQk+I3bcVr87sdq8QfeI9jjD/b3YJP1GWOM8ampVjEZY4w5DUsQxhhjfLIEYYwxxidLEMYYY3yyBGGMMcYnSxDGGGN8sgRhGjURKWjg+80SkQH1dK1yEVknIhtF5BMRaX2a41uLyH31cW9jABsHYRo3ESlQ1eh6vF6YqpbV1/VOc6/jsYvIG8B2Vf1DDcd3Az5V1UENEZ9p/KwEYZocEYkVkQ9EZJXzGO3sHykiy0RkrfO1r7P/dhH5j4h8AnwpnhXuvhHPCmtbReRtZ0ppnP2JzvMC8azklywiK0Qk3tnf09leJSK/r2UpZzmeGUARkWgRWSgia0Rkg4hc6RzzJ6CnU+p42jn2l8591ovI7+rx22iaAEsQpin6K/C8qn4PuAaY5ezfCoxV1eHAE8Afvc45H7hNVS92tocDD+FZ46EHMNrHfaKAFao6FM8063d63f+vzv1PO0GbiIQCEzgxV08RcJWqjgDGA886CepRIEVVh6nqL8WzvGZvPGsPDAPOFZGxp7ufMZWa6nTfpmm7BBjg/NMP0EpEWgIxwBsi0hvP1MvhXucsUNUcr+2VqpoGICLrgG7Akir3KcGzIhjAauBS5/n5wA+c5+8Az1QTZ3Ova68GFjj7Bfij82FfgadkEe/j/InOY62zHY0nYQTCmiAmCFiCME1RCHC+qh7z3ikiLwJfq+pVTn3+N14vF1a5RrHX83J8/y2V6olGvuqOqckxVR0mIjF4Es39eNY1+CEQC5yrqqUikopnve6qBHhKVf9Rx/saA1gVk2mavgQeqNwQkWHO0xhgv/P8dj/efwWeqi3wTPNcI1XNBX4K/EJEwvHEechJDuOBc5xD84GWXqfOB+4QkcqG7s4iEldP78E0AZYgTGPXQkTSvB4P4/mwTXQabjdzYqWuvwBPichSPIvK+8tDwMMishLoCOSe7gRVXYtnYfsbgbfxxJ+EpzSx1TkmG1jqdIt9WlW/xFOFtVxENgDvc3ICMaZG1s3VmAYmIi3wVB+piNwI3KSqV57uPGMamrVBGNPwzgWmOz2PjgB3uByPMT5ZCcIYY4xP1gZhjDHGJ0sQxhhjfLIEYYwxxidLEMYYY3yyBGGMMcYnSxDGGGN8+v9qiy+1xSJP1wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#collapse-hide\n",
    "learn.lr_find(suggestions=True, stop_div=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.LineCollection at 0x7f3b8c1f0990>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEOCAYAAACTqoDjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3zV9fX48dfJhgTCSsIIyJQ9jaCCCKKIFLHuUa1W6tbWTrWt2vb3bbV1tmKlFHeValUUFUWKIlMhjLBXIEBIgAzIgpB1fn/cT+ASbkICufncm5zn43EfuZ/P/YxzM+7Je4uqYowxxlQV4nYAxhhjApMlCGOMMT5ZgjDGGOOTJQhjjDE+WYIwxhjjkyUIY4wxPvktQYhIZxH5WkQ2icgGEfmps/9pEdksImtFZJaItKrm/DQRWScia0Qk2V9xGmOM8U38NQ5CRDoAHVR1lYi0AFYC3wcSga9UtUxE/gKgqg/7OD8NSFLVbL8EaIwxpkZ+K0GoaqaqrnKeFwCbgE6q+qWqljmHfYsnYRhjjAkwDdIGISJdgaHAd1VeugP4vJrTFPhSRFaKyF3+i84YY4wvYf6+gYjEAB8AD6lqvtf+3wJlwNvVnDpSVTNEJB6YJyKbVXWhj+vfBdwFEB0dfU6fPn3q/T0YY0xjtXLlymxVjfP1mt/aIABEJBz4FJirqs957b8NuAcYp6qHa3Gd3wOFqvpMTcclJSVpcrK1ZxtjTG2JyEpVTfL1mj97MQnwCrCpSnKYADwMTK4uOYhItNOwjYhEA+OB9f6K1RhjzMn82QYxErgVuNjpqrpGRCYCU4EWeKqN1ojINAAR6Sgic5xzE4DFIpICLAc+U9Uv/BirMcaYKvzWBqGqiwHx8dIcH/tQ1QxgovN8BzDYX7EZY4w5NRtJbYwxxidLEMYYY3yyBGGMMcYnSxBNVFl5BVv3F7gdhjEmgFmCaILKyiv46btrGP/8QlbuOuh2OMaYAGUJookpr1B+/l4Kn63NRAS+3LjP7ZCMMQHKEkQTUl6h/Oq/KcxOyeDhCX24oEdb5m864HZYxpgAZQmiiaioUB75YC0frt7LL8efzb1jejCuTwLbDxSyK6fI7fCMMQHIEkQT8fI3qfx3ZTo/HdeLBy7uBcC4vvEAVoowxvjk99lcjfuWpebw7JdbuGJwRx66pNex/We1jaZnfAzzN+/njlHdanWtsvIKVqQd5H+b9nPocCkRYSFEhoUQERbCkZJy8otLyT9SypHScvq0b8mIbm0Y3q0NbWMi/fX2jDF+YgmikTuQX8yDM1fTrV00T149EM8ciseN6xvPK4t2UlBcSouo8Gqvs2bPIWZ+t5t5m/aTW1RCZFgI7WIiOVpWQUlZOSXlFTSPCKNlVBgtm4UTFiL8Z8VuXl+aBkCv+BjG9U3gsv4JDE5sRUiIr1lYjDGBxBJEI1ZWXsEDM1dTdLSMd+4cQUzkyT/ucX0S+Oc3O1i4NZvvDepw0uuqyr8W7eAvX2yhWXgoF/eJZ8KA9ozpHUfziJp/fUrKKli3N4/lO3NZvD2LGYt2MO2bVBJaRnJDUmd+dunZJyUsY0zgsATRiD3z5VaW78zluesHc3ZCC5/HDOvSilbNw5m/ef9JCSK/uJRfvpfClxv3M3Fge/5yzaAaSxlVRYSFcM5ZrTnnrNbcO6YHeYdLmb95P7NW7+XvX23n3G5tuLCXz3VKjDEBwBqpG6kl27OZ9k0qNw3vwtXDql/2Oyw0hLG941mwJYvyiuOLR23KzGfyi4v5avMBHpvUj5duHlan5OBLbPNwrh6WyIzbkkhoGcmLX20/o+sZY/zLEkQjlHeklF/+N4XucdE8PqnfKY+/uE88uUUlrNnjGVX9wcp0rvrHEo6UlvOfu85jyqhu9VoVFBkWyt2je7B8Zy7Ld+bW23WNMfXLEkQj9IfZGzhQcJTnrx9Cs4jQUx4/+uw4wkKEz9ft43cfreMX/01hSOdWfPrghSR1beOXGG8a3oW20RFM/dpKEcYEKn8uOdpZRL4WkU0iskFEfursbyMi80Rkm/O1dTXn3+Ycs81Zw9rUwufrMvlw9V4eGNuTwZ1b1eqc2GbhnNu1DTMW7+Tf3+7m7tHd+feUEcS18F/X1GYRofz4wu4s3JpFyp5DfruPMeb0+bMEUQb8QlX7AucB94tIP+ARYL6q9gLmO9snEJE2wBPACGA48ER1icQcd6CgmN/MWsfATrE8cHHPOp17XVIirZqH8/IPhvHoxL6Ehfq/cHnLeV2IbRZupQhjApTfPgVUNVNVVznPC4BNQCfgSuAN57A3gO/7OP0yYJ6q5qrqQWAeMMFfsTYGqsojH6zjcEk5z98wmPA6fsBfPSyR1Y9dyuUDT+7q6i8tosL50ciuzNu4n02Z+Q12X2NM7TRIG4SIdAWGAt8BCaqaCZ4kAsT7OKUTsMdrO93ZZ6rx+tI0vtp8gEcu70PPeN9dWk/FjTEJt1/QlZjIMJ6Zu4XS8ooGv78xpnp+TxAiEgN8ADykqrX9N9HXJ5X62IeI3CUiySKSnJWVdbphBrUNGXk8OWczF/eJ5/YLurodTp20ah7BT8b1ZP7mA/xgxnccKCh2OyRjjMOvCUJEwvEkh7dV9UNn934R6eC83gHwNVNcOtDZazsRyPB1D1WdrqpJqpoUF9f0Bl0dLinjwZmradU8nKevHRSUI5PvGt2D528YzNr0Q0z6+2KS06zrqzGBwJ+9mAR4Bdikqs95vTQbqOyVdBvwsY/T5wLjRaS10zg93tlnqvj97A3szC7ihRuGBPWEeFcNTWTWfSNpFhHKjdO/5eUFqZSUWZWTMW7yZwliJHArcLGIrHEeE4GngEtFZBtwqbONiCSJyAwAVc0F/h+wwnn80dlnvMxOyeC95HTuG9ODC3q2czucM9a3Q0tmPzCKcX3j+csXm5nwt4Us3No0qw2NCQSi6rNqPyglJSVpcnKy22E0iB1ZhUyeuoSzE2J49+7z69xrKdB9tXk/f/xkI2k5h7m0XwKPfa8fXdo2dzssYxodEVmpqkm+XmtcnypNxJGScu57exXhocLUm4c1uuQAcHGfBOb+bDS/ntCbJduzueS5b3hyzibyjpS6HZoxTUbj+2RpAh7/eD2b9xXw3A1D6Niqmdvh+E1kWCj3jenJ178cw+QhHZm+aAdjn1nAW8vSrEusMQ3AEkSQeS95D/9dmc4DY3sytrevISSNT0LLKJ65bjCfPDCKsxNieOzjDVzy3DfMWp1+wgy0xpj6ZQkiiGzel8/jH6/n/O5t+dmlZ7sdToMb0CmWmXeexyu3JdE8IoyfvZvCZS8sZM66TBpTW5oxgcISRBB54uMNxESG8bebhhDaRJfsFBHG9U3gswdH8dLNwwC47+1VXP/PZWzIyHM5OmMaF0sQQWJpajbf7czl/rE9iW8R5XY4rgsJEb43qANzHxrNU1cPJDWriCteXMzjH68n77A1ZBtTHyxBBAFV5YV520hoGclNw7u4HU5ACQ0Rbhzeha9/MYZbzzuLf3+7i3HPfUNm3hG3QzMm6FmCCAJLU3NYnpbLfWN6EhV+6gWAmqLY5uH84coBfHz/KAqKS/nznM1uh2RM0LMEEeBUlefnbaV9yyhuOLfzqU9o4gYmxnLPRT34JCWDZak5bodjTFCzBBHgFm/PJnnXQe4f28NKD7V075geJLZuxu9nb7DxEsacAUsQAayy9NAxNorrrfRQa1HhoTw2qR9b9hfw1rJdbodjTNCyBBHAFm3LZtXuQ9w3tieRYVZ6qIvx/RIYfXYcz8/bSlbBUbfDMSYoWYIIYDMW7yS+RSTXJ1npoa5EhCeu6EdxWTlPfW4N1sacDksQAWr7gUIWbs3i1vPOIiLMfkyno0dcDFNGdeeDVeksTc12Oxxjgo598gSoN5elEREawk0jbNzDmfjpuF50bducRz9cx5GScrfDMSao+HNFuVdF5ICIrPfa967X4kFpIrKmmnPTRGSdc1zTWODBS35xKR+sTGfS4A60C+JV4gJBs4hQnrx6ELtyDvPcvC1uh2NMUPFnCeJ1YIL3DlW9QVWHqOoQPGtVf+jrRMdY51ifC1k0Zu8np1NUUs6PLujmdiiNwvk92nLziC68sngna/YccjscY4KG3xKEqi4EfC4T6qxXfT0w01/3D1YVFcoby9I456zWDEyMdTucRuORy/sQ3yKKh99fa2tdG1NLbrVBXAjsV9Vt1byuwJcislJE7mrAuFy3YOsBduUc5vYLurodSqPSMiqc//v+ALbsL+C5eVttenBjasGtBHETNZceRqrqMOBy4H4RGV3dgSJyl4gki0hyVlbwL3D/2pI0ElpGMmFAe7dDaXQu6ZfANcMSmfZNKne+mWzjI4w5hQZPECISBlwNvFvdMaqa4Xw9AMwChtdw7HRVTVLVpLi4uPoOt0GlZhWyaFs2t4w4q1GuMx0Inr52EI9N6sfCbdlc9sJCvlif6XZIxgQsNz6FLgE2q2q6rxdFJFpEWlQ+B8YD630d29h8tHovIQI3DLeBcf4SEiJMGdWNzx4cRcdWUdzz71U8/P5aikutC6wxVfmzm+tMYBnQW0TSRWSK89KNVKleEpGOIjLH2UwAFotICrAc+ExVv/BXnIFCVfkkJYPze7S1BYEaQK+EFsy6byT3j+3Bu8l7uHH6t+zPL3Y7LGMCSpi/LqyqN1Wz/3Yf+zKAic7zHcBgf8UVqDZk5JOWc5i7L+rhdihNRnhoCL+6rA8DO8Xy8/dSmPTiYqbdcg7nnNXa7dCMCQhW0R0gPknJICxEmNDfGqcb2oQBHZh130iahYdy4/Rl/HnOJrbtL3A7LGNcZwkiAKgqn67N5MJe7WgdHeF2OE1S7/YtmP3ASC7r355XFu/k0ucXMnnqYt5clkbeEVvj2jRNliACwKrdh9h76AiTBnV0O5QmrVXzCKbePIzvfjOOxyb1o7RcefzjDZz35/k8+uFaNmbkux2iMQ3Kb20QpvY+SckgIiyES/snuB2KAdrFRDJlVDemjOrG+r15vLVsF7NW72Xm8j0kndWaKaO6Mb5/e0JDxO1QjfErK0G4rLxC+WxdJmN7x9EyKtztcEwVAzrF8pdrB/Hdo5fwu+/15UDBUe59exUXP7uAt5al2QyxplGzEoTLlu/MJavgqFUvBbjY5uH8+MLu/GhkN+Zu2Mc/F+7gsY838MyXW7mkbwLj+ycwulcczSJs5T/TeFiCcNknazNoFh7KuL7xbodiaiE0RJg4sAOXD2jPirSDzFy+m3kb9/HBqnSiwkO46Ow4LuvfnnF9EohtbiVCE9wsQbiotLyCL9bv45J+CTSPsB9FMBERhndrw/BubSgtr+C7HbnM3bCPeRv3M3fDfsJChPN7tOXKIZ2YPLijrQpogpJ9KrlozZ5D5BaVcLlNzBfUwkNDGNWrHaN6teMPk/uTkn6IuRv288X6TH753xSembuFO0Z15abhXWhh7UwmiFiCcNHS7TmIwAU92rodiqknISHC0C6tGdqlNQ9P6M3CbdlMW5DKn+ds5sWvtnPNsEQmDuxA0lmtCbFeUCbAWYJw0bId2fRt35JWzW1wXGMkIlx0dhwXnR1Hyp5DTF+0g3eW7+b1pWnEt4jk8gHt+d6gjpYsTMCyBOGS4tJyVu0+xA/PO8vtUEwDGNy5FS/dPIzCo2XM37SfOesy+c+KPbyxbBcJLSO5fEAHvjfIU7LwLLhojPssQbhk1a6DlJRVcL5VLzUpMZFhXDmkE1cO6UTR0TLmbz7AZ2szjpUsrhzSkb9cM4iocOsua9xnCcIly3bkEBri6QljmqboyDAmD+7I5MEdKTxaxiuLdvL8/7ayM7uI6bcm0T7Wpn037rK+dy5ZlprDgE6x1qvFAJ6SxU8v6cX0W88h9UAhk6cuZvXug26HZZo4SxAuOFxSxpo9hzi/u1UvmRON79+eD+8bSWR4CNdNW8Z105byzNwtLN6WbdN6NFH/27ifq/6xhA9XpVNRoQ16b3+uKPeqiBwQkfVe+34vIntFZI3zmFjNuRNEZIuIbBeRR/wVo1tWpB2krEKte6vxqXf7Fsy+fxR3ju5OSbny8jep3PLKdwz+w5f88NXlvLE0jT25h90O0/iZqjJj0Q7ufCuZrfsK+Pl7KVw7bSnr9+Y1WAyi6p+MJCKjgULgTVUd4Oz7PVCoqs/UcF4osBW4FEgHVgA3qerGU90zKSlJk5OT6yF6/3rq8828sngHKU+MtxHU5pQKj5aRnJbLom3ZfL35ADuyiwDoGR/D6F5xXHh2O87r1tbmgWpESssrePzjDcxcvpsJ/dvz7PWD+WxdJn/9YjM5RSXcPLwLj03qVy+dGURkpaom+XrNn0uOLhSRrqdx6nBgu7P0KCLyH+BK4JQJIlgs25HD4MRWlhxMrcREhjGmdzxjesfz2KR+7MwuYv6m/XyzNYu3v9vFq0t2EhEawveHduSPVw6wHlBB7mhZOVNeT2bx9mzuHdODX43vTUiIcH1SZyYMaM8L87bx2tKdrN+bx/QfJpHQ0n+dGdxog3hARNY6VVC+Fv/tBOzx2k539jUK+cWlrEs/ZNVL5rR1axfNjy/szltTRpDyxHjevGM41yUl8l5yOre+8h2HDpe4HaI5A++vTGfx9mz+dNUAHp7Q54RBlC2jwnn8in7885Zz2OZ0ZlibfshvsTR0gngZ6AEMATKBZ30c42uUULX1YCJyl4gki0hyVlZW/UTpRyt25lKhcJ4lCFMPosJDGX12HH+6aiAv3jSUlD15XP3yUnbnNK02ijFjxjBmzBi3w6i16uItLa/g5QWpDO3SipuHd6n2/PH92/PBvRcQFuLpzPBJSoZf4mzQBKGq+1W1XFUrgH/hqU6qKh3o7LWdCFT77lV1uqomqWpSXFxc/QbsB8tSc4gIC2FYF1+FJ2NO3xWDO/L2nSPILSrhqn8s4estB/BXG6Pxj49W7yX94BEevLjnKUfU9+3Qko8fGMmgxFj+77ONFB0tq/d4GjRBiEgHr82rgPU+DlsB9BKRbiISAdwIzG6I+Gpj5a5crnxpCYdLTu+HsTQ1h3O6tLZ6YuMX53Ztwwf3XkBMVBg/em0Fl72wkJnLd1Ncal1kA115hfKPBan079iSsb1rtz5Mu5hI/v3jEcy88zyiI+u/TdOf3VxnAsuA3iKSLiJTgL+KyDoRWQuMBX7mHNtRROYAqGoZ8AAwF9gEvKeqG/wVZ12tSDtIyp5DbNtfWOdzC4pL2bQvnxHdbfS08Z8ecTF8+bPRPHPdYMJCQnj0w3Wc/+R8Hn5/LZ+uzeBgkbVRBKJP12awM7uoVqUHb5FhoXSPi/FLTP7sxXSTj92vVHNsBjDRa3sOMMdPoZ2RXOePKy2niMGdW9Xp3PV781GFIXU8z5i6igwL5dpzErlmWCeW78zlzW93MWd9Ju8m70EEBnaK5eI+8VzSN4H+HVvaBIEuq6hQXvp6O2cnxDC+X+CsD2P9LOsop9CTIHY6fdHrorK3waBESxCmYYgII7q3ZUT3tpSVV7B2bx6Lt2XzzdYs/jZ/Gy/8bxsdYqO4tF8Ct553Fr0SWrgdcpP05cZ9bN1fyN9uHBJQU79bgqij3KKjAKSdVoLII7F1M9pE2/oPpuGFhXo6Rwzr0pqfjOtFTuFRvtp8gPmbDvBe8h7eXLaLS/rGc/dFPWza8Qakqrz41Xa6tYtm0qCObodzAksQdVRZxbTzNLoRpqQfYrCVHkyAaBsTyXVJnbkuqTO5RSW8uSyNN5amcd20ZQzv2oZnrx9M5zbN3Q6z0Vu8PZsNGfn85ZqBhAZQ6QFssr46y66sYsoqrFMXwpzCo6QfPMKgxFh/hWbMaWsTHcFDl5zN0kfG8ccr+7NpXz5XTF3Mwq2BP7Yo2E1fuIP4FpF8f2jgjQe2BFFHuUUlRISFkF9cxsHDpbU+b50zwZa1P5hA1iwilB+e35VPHhhFQosobnttOS99vb3BZxFtKjZk5LFoWza3j+xKZFjgdX23BFEHR0rKOVJazmCnFFCXhuq16XmIwIBOLf0VnjH1pmu7aGbdfwFXDOrI03O3cNtry1myPdsG3tWzGYt2Eh0Ryg9GBObSw5Yg6iDHaaAedpZnFHRdGqrXph+ie7toWyDIBI3mEWH87cYh/GFyfzZk5PODGd9xyXPf8MbSNAqKa196Nr6VRbTgk5QMbji3C7HNAvNzwRJEHVQ2UA9JbEWIeMZC1IaqkpKeZw3UJuiICLdd0JWlj1zMs9cNJiYqnCdmb2D0X79mxqIdHC2zEdqnK7/9OShwx6iubodSLUsQdZDjJIiE2CgSWzevdRXT/vyjZBUctQZqE7SiwkO55pxEPr5/JLPuu4ABnWL5v882cfEz3zBrdcOvdBbsKkIjKIgfxKRBHUhsHbg9xSxB1EHlILm20RF0bRdd6xJESuUAORtBbRqBoV1a89aUEfx7yghaR4fzs3dTuPKlJbaGdh0UxA9GwyK588LubodSI0sQdVA5SK5NdATd2jYnLftwrRrt1qYfIixE6NfBGqhN4zGqVztm3z+KF24YwoGCYq76x1J+/X4KOYVH3Q4toOUUHiWv47lE5e1iQKfArlWwBFEHOUUlRISGEBMZRtd20RQeLTs2LqIma9PzODuhhc3gahqdkBDh+0M7Mf8XY7h7dHc+XLWXsc8s4O3vdlm1kw+qyu8+Wk9FaCRt0r5yO5xTsgRRB7mFJbSJjkBE6NouGjh1Q7WqsjY9j8GdA/s/BWPORExkGI9O7MsXD42mf8dYfjtrPTf961t2ZNV91uPGbHZKBp+v30frPUuIOJLtdjinZAmiDnKLSo7No9StrSdBnKqhenfuYfKOlNoAOdMk9IyP4Z07R/DXawaxKTOfCX9bxEtfb6esvMLt0Fy3L6+Yxz5az7AurWiZucLtcGrFEkQd5BSV0DbGkyASWzcjLEROORYiJd0zgnpggNc1GlNfRITrz+3M/35+EeP6xPP03C388NXlTbptQlV5+IO1lJYrz14/BKl+FeWA4s8Fg14VkQMist5r39MisllE1orILBHx+W+1iKQ5CwutEZFkf8VYV7lFJbR1ShBhoSF0btP8lFVMa/ccIjIshN7tbRpl07TEt4zi5VvO4elrB5G86yCTpy45NuW9W8pdahd5Z/luvtmaxaMT+9DNqZ4OBv4sQbwOTKiybx4wQFUHAVuBR2s4f6yqDlHVJD/FV2c5hUdpEx15bLtr2+bszK55Vte1e/Po17El4aFWWDNN03VJnfngngsAuHbaMt5L3uNKHC99vZ0ev5lDr9/OYdDv5zLiz//j3n+v9PsKe2v2HOIPn2zkwl7tuCVAp9Sojt8+tVR1IZBbZd+XzpKiAN8Cif66f30rLi2nqKT8WBUTeOar2ZVTVG1XV1Vly74C+lr3VtPEDUyMZfYDIzm3a2t+/f5afvfROkrKGq5d4tDhEl5ekMqwLq348YXduXpYIiN7tmP+pgNcMXUxGzLy/HLf/fnF3PVmMgktI/n7jUMDajGg2nBzPYg7gHereU2BL0VEgX+q6vSGC8u3ymk2vBf76dYumsMl5RwoOEpCyyif5+QdKaWHn9aLNSaYtI2J5I0fDefpuVv458IdbMos4B8/GObzb6c29h46wiMfrOWKwadeZGfGop0UHi3jyasHnVDde+t5Z3Hvv1dxzctLeerqQfU65XZxaTl3v7WSwqNlvDnlAloH4UJhrtR7iMhvgTLg7WoOGamqw4DLgftFZHQN17pLRJJFJDkry39z1/tKEF1P0ZNph7O/e1zw1Dka409hoSE8OrEvU28eyqbMfCa9uJgVabmnPrGK/fnF3Pyvb1m0LZtfv7+W7G6XouJ7nNHBohJeW7KT7w3scFJb4NAurfnkwVEM6tSKh95dw29mrSO/HiYiVFV+O2s9a/Yc4rnrB9OnfXDWIjR4ghCR24BJwA+0mroZVc1wvh4AZgHDq7ueqk5X1SRVTYqLi/NHyMDxeZjaVilBQPWzuqYe8PQB72klCGNOMGlQR2bdN5LoiFCum7aM66ctY9bqdIpLTz35X1bBUW7+17dkFxzl/XvO556LelCYMITMfjeSmXfkpONnLN7B4dJyfjKul8/rxbWI5O07R3DX6O78Z/luxj37DZ+uzTjtqc1Tswr51ftr+WBVOj8d14sJAzqc1nUCQa0ShIj0EJFI5/kYEflJdT2QTnGdCcDDwGRV9dm6KyLRItKi8jkwHljv69iG5D3NRqWOrZoRERrCzmp6Mu3ILiIiLISOrZo1SIzGBJPe7Vsw+8FRPHp5Hw4UFPOzd1MY8ef5/PWLzdUmityiEm6Z8R0Zh4p57UfDSerahkcu70Pc1o8pbd6OK15czNwN+44df7CohNeXpDHRR+nBW3hoCL+Z2JeP7x9FQstIHnhnNbe/toIt+wpq9V5UlW935PDjN1Yw7tlvmJ2SwZRR3fhpNUkpWNS2BPEBUC4iPYFXgG7AOzWdICIzgWVAbxFJF5EpwFSgBTDP6cI6zTm2o4jMcU5NABaLSAqwHPhMVb+o6xurb8cm6os53ospNETo3KZZtSWIHVmFdGsbHXDrzBoTKFpGhXP3RT34+pdjeOfOEYzq2Y5/LEhl8tTFrN97vOG4okL5Yv0+rpu2lLScImbclsTwbm2OvR6du5UO696iXUwkd7+1knveWsn+/GL+tchTeqjtB/XAxFg+um8kj0/qR3JaLpe9sJAbpy9jzrpMSqsZ7Keq/PHTjdw4/VtW7T7ET8b1YsnDF/PYpH5B1yhdVW0bqStUtUxErgJeUNUXRWR1TSeo6k0+dr9SzbEZwETn+Q5gcC3jajA5RSWEhwoto078lnVrF01aNV1dU7OK6NvBxj8YcyoiwgU92nFBj3Zct+UAv35/LVf9YwkPXXI2nds056WvtrNlfwFd2zbnldvOZWTPdiddI6I4l08eHMX0hTv42/xtLHkum7Jy5XsDO3B2Qu3/DsNCQ7hjVDeuGtqJd5P38NayXdz39irat4ziN9/ry2SvRnFV5anPN/PakjRuv6Arj1zep1HNuVbbEkSpiNwE3AZ86uwLzCWQ/CS3sITWzT3zMHnrHhfDzpyik/67KCmrYHfuYbq3s/YHY+piTO945j40mkv7JfD03BGfquIAABNvSURBVC38ZOZqylV54YYh/O/nFzGq18nJoVJ4aAj3j+3J3IdGM7BTLOWqp13N0zo6gnsu6sHCX49lxg+TSIiN4iczV/PAO6s4dNhTo/D8/7bxz4U7uOW8LjxxRb9GlRyg9iWIHwH3AH9S1Z0i0g34t//CCjw5XvMweRuUGEtJWQWbMwsY6LUg0O7cw5RXqPVgMuY0tI6O4KWbh/H1lgOUV8C4PvF1qq7p1i6at388goKjZbQ8w2V+Q0OES/olMKZ3HNO+SeWF/21j+c5cxvWNZ+byPVyflMgfJw846Z/HxqBWCUJVNwI/ARCR1kALVX3Kn4EFmtyioycMkqs0tItnferVew6ekCAqZ7G0MRDGnB4R4eI+CWd0/pkmB29hoSE8cHEvxvSO52fvrmHm8j1cNbQTT149KOjbGqpTqwQhIguAyc7xa4AsEflGVX/ux9gCSm5RCQNbn9xxq2NsFPEtIlm9+xA/PP/4/tQsGwNhTGM0oFMsnzw4imWpOVzYq12j7oRS2zaIWFXNB64GXlPVc4BL/BdW4MnxmqjPm4gwtEurk5Zb3JFVSFyLSFrU438wxpjAEBUeytg+8YQ18jnWavvuwkSkA3A9xxupm4yjZeUUFJf5TBDgqWZKyzl8bLQ1eMZA9LDSgzEmiNU2QfwRmAukquoKEekObPNfWIHlYJFn6H0bH20QAEM7e6qe1uw5XopIzSqku7U/GGOCWK0ShKr+V1UHqeq9zvYOVb3Gv6EFjhxnFHV1JYiBibGEhgird3vmus8tKuHQ4VK6B9G878YYU1Vtp9pIdBb4OSAi+0XkAxEJmqm6z9Txifoifb7ePCKM3gktjiWIYz2Y4q0EYYwJXrWtYnoNmA10BDoBnzj7mgRfM7lWNbRLK1L2HKKiQkmtTBA2SM4YE8RqmyDiVPU1VS1zHq8D/ps6NcAcm4epxgTRmoKjZaRmFbIjyzNJX6fWNkmfMSZ41TZBZIvILSIS6jxuAXL8GVggySk6SmiIENus+i6rQ7t4GqpX7z5EalYRXds2b9T9o40xjV9tE8QdeLq47gMygWvxTL/RJOQWeeZhqmm0ZLe20cQ2C2f1noPsyC60EdTGmKBX215Mu1V1sqrGqWq8qn4fz6C5JiGn0PcgOW8hIcKQzq1YvjOX3TmHbQS1MSbonckwwCY1zUZNDdSVhnZpRWpWEWUVarO4GmOC3pkkiCZTwZ5bVFLtIDlvlRP3gXVxNcYEvzNJEKdcsFVEXnXGTqz32tdGROaJyDbna+tqzr3NOWabs461a6qbh6mqIYnHJ/OzKiZjTLCrMUGISIGI5Pt4FOAZE3EqrwMTqux7BJivqr2A+c521fu2AZ4ARgDDgSeqSyT+VlpeQd6R0lpVMcU2D6d7XDTtYiLrdZphY4xxQ43TfavqGa2XqaoLRaRrld1XAmOc528AC4CHqxxzGTBPVXMBRGQenkQz80ziOR0Hi049BsLbbed3Jcdr0j5jjAlWtV1Rrj4lqGomgKpmiki8j2M6AXu8ttOdfQ2u8sO+bYzvaTaquu2Crn6MxhhjGo4bCaI2fDWA+2zzEJG7gLsAunTpUu+B7M8vBqBdLROEMabhLViwwO0Q6iRY4nVjtYv9ztoSOF8P+DgmHejstZ0IZPi6mKpOV9UkVU2Ki6v/2T827ysAoHfCGdW2GWNM0HEjQcwGKnsl3QZ87OOYucB4EWntNE6Pd/Y1uI0Z+XRq1YzY5tbobIxpWvyaIERkJrAM6C0i6SIyBXgKuFREtgGXOtuISJKIzABwGqf/H7DCefyxssG6oW3MzKdfx5Zu3NoYY1zl1zYIVb2pmpfG+Tg2Gfix1/arwKt+Cq1WjpSUsyOrkIkDO7gZhjHGuKJxr7h9hrbsL6BCoV8HK0EYY5oeSxA12JiRD0B/q2IyxjRBliBqsDEzjxaRYSTawj/GmCbIEkQNNmUW0LdjS0SazLyExhhzjCWIalRUKJsy8639wRjTZFmCqMau3MMcLim3Lq7GmCbLEkQ1KhuorQRhjGmqLEFUY2NmHmEhQq8EW/jHGNM0WYKoxsaMfHrGxxAZFup2KMYY4wpLENXYaA3UxpgmzhKEDzmFR9mff9QaqI0xTZolCB82ZXqm+LYShDGmKbME4cPGzDwA+lqCMMY0YZYgfNiYkU/H2Cha13IdamOMaYwsQfhga0AYY4wLCUJEeovIGq9Hvog8VOWYMSKS53XM4w0VX3FpOalZRdb+YIxp8vy6YJAvqroFGAIgIqHAXmCWj0MXqeqkhowN4ED+UcorlC5toxv61sYYE1DcrmIaB6Sq6i6X4zgmv7gUgNhmtga1MaZpcztB3AjMrOa180UkRUQ+F5H+DRVQ/hFPgmgR1eCFK2OMCSiuJQgRiQAmA//18fIq4CxVHQy8CHxUw3XuEpFkEUnOyso647jyi8sAaBllJQhjTNPmZgnicmCVqu6v+oKq5qtqofN8DhAuIu18XURVp6tqkqomxcXFnXFQlVVMVoIwxjR1biaIm6imeklE2ouzjJuIDMcTZ05DBFVZxdTS2iCMMU2cK/8mi0hz4FLgbq999wCo6jTgWuBeESkDjgA3qqo2RGwFxWWIQItIK0EYY5o2Vz4FVfUw0LbKvmlez6cCUxs6LvBUMcVEhBESYutQG2OaNrd7MQWcguIyq14yxhgsQZwk/0ipNVAbYwyWIE6SX1xqXVyNMQZLECfxVDFZCcIYYyxBVGElCGOM8bAEUUX+kTJrgzDGGCxBnEBVKSgutV5MxhiDJYgTFJWUU6E2zYYxxoAliBMcm2bD2iCMMcYShLeCyplcrYrJGGMsQXizmVyNMeY4SxBerIrJGGOOswThxaqYjDHmOEsQXqyKyRhjjrME4aWyBGEJwhhjLEGcIP9IKZFhIUSGhbodijHGuM61BCEiaSKyTkTWiEiyj9dFRP4uIttFZK2IDPN3TPk2itoYY45xuy5lrKpmV/Pa5UAv5zECeNn56jf5xWW0tOolY4wBAruK6UrgTfX4FmglIh38eUPPYkFWgjDGGHA3QSjwpYisFJG7fLzeCdjjtZ3u7PObfFtu1BhjjnGzPmWkqmaISDwwT0Q2q+pCr9fFxzladYeTXO4C6NKlyxkFVFBcSufWzc7oGsYY01i4VoJQ1Qzn6wFgFjC8yiHpQGev7UQgw8d1pqtqkqomxcXFnVFMnrUgrARhjDHgUoIQkWgRaVH5HBgPrK9y2Gzgh05vpvOAPFXN9Gdcnl5M1khtjDHgXhVTAjBLRCpjeEdVvxCRewBUdRowB5gIbAcOAz/yZ0DFpeWUlFXYPEzGGONwJUGo6g5gsI/907yeK3B/Q8V0bB4m6+ZqjDFAYHdzbVAFzjxM1ovJGGM8LEE48m0eJmOMOYElCIetBWGMMSeyBOGwtSCMMeZEliActhaEMcacyBKEw6qYjDHmRJYgHAXFZYSGCM0jbC0IY4wBSxDH5BeX0iIqDGfwnjHGNHmWIBz5R0qteskYY7xYgnAUFJfZPEzGGOPFEoQjv7iUFpFWgjDGmEqWIBz5R6wEYYwx3ixBOAqKbblRY4zxZgnCkV9cZo3UxhjjxRIEUF6hFB61KiZjjPFmCQIoPDaTq5UgjDGmUoMnCBHpLCJfi8gmEdkgIj/1ccwYEckTkTXO43F/xlQ5D5MtFmSMMce58YlYBvxCVVc561KvFJF5qrqxynGLVHVSQwSUb4sFGWPMSRq8BKGqmaq6ynleAGwCOjV0HN7yj9hiQcYYU5WrbRAi0hUYCnzn4+XzRSRFRD4Xkf41XOMuEUkWkeSsrKzTiuN4FZOVIIwxppJrCUJEYoAPgIdUNb/Ky6uAs1R1MPAi8FF111HV6aqapKpJcXFxpxVL5WJBsVbFZIwxx7iSIEQkHE9yeFtVP6z6uqrmq2qh83wOEC4i7fwVT+VaEFbFZIwxx7nRi0mAV4BNqvpcNce0d45DRIbjiTPHXzFVVjHFRFqCMMaYSm58Io4EbgXWicgaZ99vgC4AqjoNuBa4V0TKgCPAjaqq/gqooLiM6IhQwkJtWIgxxlRq8AShqouBGlflUdWpwNSGichZC8LaH4wx5gT2LzPOWhDWg8kYY05gCYLjy40aY4w5zhIEngRhVUzGGHMiSxBUVjFZCcIYY7xZgsDTSG0zuRpjzIksQQDtYiLp0CrK7TCMMSagWL0KMO/nF7kdgjHGBBwrQRhjjPHJEoQxxhifLEEYY4zxyRKEMcYYnyxBGGOM8ckShDHGGJ8sQRhjjPHJEoQxxhifxI/r8DQ4EckDtnntigXyavm8HZB9Grf1vlZdj6m6v6btYIi/pji9t+sz/priO9Xrp4q/6rav5xZ/YMQPgfE3EIx/w61UNc7nWaraaB7A9Oq2T/UcSK6Pe9blmJriDcb4a4qzSqz1Fn9t3sPpxl/L77vFHwDxn8l7sL/h6s9rbFVMn9SwXZvn9XHPuhxTU7xVt4Mh/qr7qns/9Rl/ba5xuvFX3fb13OJv/PHXdExj/Bs+plFVMZ0JEUlW1SS34zhdFr+7LH73Bft7CMT4G1sJ4kxMdzuAM2Txu8vid1+wv4eAi99KEMYYY3yyEoQxxhifLEEYY4zxyRKEMcYYnyxB1IKIXCgi00RkhogsdTueuhKREBH5k4i8KCK3uR1PXYnIGBFZ5PwMxrgdz+kQkWgRWSkik9yOpa5EpK/zvX9fRO51O566EpHvi8i/RORjERnvdjynQ0S6i8grIvJ+Q9630ScIEXlVRA6IyPoq+yeIyBYR2S4ij9R0DVVdpKr3AJ8Cb/gz3qrqI37gSqATUAqk+ytWX+opfgUKgSiCM36Ah4H3/BNl9erp93+T8/t/PdCg3TDrKf6PVPVO4HbgBj+G61M9vYcdqjrFv5H6vnGjfgCjgWHAeq99oUAq0B2IAFKAfsBAPEnA+xHvdd57QMtgix94BLjbOff9IIw/xDkvAXg7COO/BLgRzwfUpGCL3zlnMrAUuDkY43fOexYY1pDx++E9NOjfbxiNnKouFJGuVXYPB7ar6g4AEfkPcKWqPgn4rAIQkS5Anqrm+zHck9RH/CKSDpQ4m+X+i/Zk9fX9dxwEIv0RZ3Xq6fs/FojG8wFwRETmqGqFXwN31Nf3X1VnA7NF5DPgHf9FfNJ96+P7L8BTwOequsq/EZ+snv8GGlSjTxDV6ATs8dpOB0ac4pwpwGt+i6hu6hr/h8CLInIhsNCfgdVSneIXkauBy4BWwFT/hlYrdYpfVX8LICK3A9kNlRxqUNfv/xjgajzJeY5fI6uduv7+P4inFBcrIj1VdZo/g6uluv4M2gJ/AoaKyKNOIvG7ppogxMe+GkcMquoTforldNQpflU9jCfBBYq6xv8hniQXKOr8+wOgqq/Xfyinpa7f/wXAAn8FcxrqGv/fgb/7L5zTUtf3kAPc479wfGv0jdTVSAc6e20nAhkuxXI6LH53WfzuCvb4IUjeQ1NNECuAXiLSTUQi8DQgznY5prqw+N1l8bsr2OOHYHkPDd2i70IPgplAJse7eE5x9k8EtuLpSfBbt+O0+N2P1eIPvEewxx/s78Em6zPGGONTU61iMsYYcwqWIIwxxvhkCcIYY4xPliCMMcb4ZAnCGGOMT5YgjDHG+GQJwjRqIlLYwPebISL96ula5SKyRkTWi8gnItLqFMe3EpH76uPexgA2DsI0biJSqKox9Xi9MFUtq6/rneJex2IXkTeArar6pxqO7wp8qqoDGiI+0/hZCcI0OSISJyIfiMgK5zHS2T9cRJaKyGrna29n/+0i8l8R+QT4Ujwr3C0Qzwprm0XkbWdKaZz9Sc7zQvGs5JciIt+KSIKzv4ezvUJE/ljLUs4yPDOAIiIxIjJfRFaJyDoRudI55imgh1PqeNo59lfOfdaKyB/q8dtomgBLEKYp+hvwvKqeC1wDzHD2bwZGq+pQ4HHgz17nnA/cpqoXO9tDgYfwrPHQHRjp4z7RwLeqOhjPNOt3et3/b879TzlBm4iEAuM4PldPMXCVqg4DxgLPOgnqESBVVYeo6q/Es7xmLzxrDwwBzhGR0ae6nzGVmup036ZpuwTo5/zTD9BSRFoAscAbItILz9TL4V7nzFPVXK/t5aqaDiAia4CuwOIq9ynBsyIYwErgUuf5+cD3nefvAM9UE2czr2uvBOY5+wX4s/NhX4GnZJHg4/zxzmO1sx2DJ2EEwpogJghYgjBNUQhwvqoe8d4pIi8CX6vqVU59/gKvl4uqXOOo1/NyfP8tlerxRr7qjqnJEVUdIiKxeBLN/XjWNfgBEAeco6qlIpKGZ73uqgR4UlX/Wcf7GgNYFZNpmr4EHqjcEJEhztNYYK/z/HY/3v9bPFVb4JnmuUaqmgf8BPiliITjifOAkxzGAmc5hxYALbxOnQvcISKVDd2dRCS+nt6DaQIsQZjGrrmIpHs9fo7nwzbJabjdyPGVuv4KPCkiS/AsKu8vDwE/F5HlQAcg71QnqOpqPAvb3wi8jSf+ZDylic3OMTnAEqdb7NOq+iWeKqxlIrIOeJ8TE4gxNbJursY0MBFpjqf6SEXkRuAmVb3yVOcZ09CsDcKYhncOMNXpeXQIuMPleIzxyUoQxhhjfLI2CGOMMT5ZgjDGGOOTJQhjjDE+WYIwxhjjkyUIY4wxPlmCMMYY49P/B89hKs+tByDSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#hide\n",
    "learn.recorder.plot_lr_find()\n",
    "plt.vlines(0.01, 10, 15)\n",
    "plt.vlines(0.083, 10, 15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We do some training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>13.052832</td>\n",
       "      <td>11.317341</td>\n",
       "      <td>0.052725</td>\n",
       "      <td>82235.375000</td>\n",
       "      <td>00:32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>8.539399</td>\n",
       "      <td>5.935386</td>\n",
       "      <td>0.049121</td>\n",
       "      <td>378.185822</td>\n",
       "      <td>00:32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>1.660586</td>\n",
       "      <td>0.785814</td>\n",
       "      <td>0.493350</td>\n",
       "      <td>2.194192</td>\n",
       "      <td>00:32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.731211</td>\n",
       "      <td>0.768679</td>\n",
       "      <td>0.493125</td>\n",
       "      <td>2.156916</td>\n",
       "      <td>00:32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.732979</td>\n",
       "      <td>0.772890</td>\n",
       "      <td>0.492373</td>\n",
       "      <td>2.166016</td>\n",
       "      <td>00:32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0.681202</td>\n",
       "      <td>0.695503</td>\n",
       "      <td>0.493711</td>\n",
       "      <td>2.004716</td>\n",
       "      <td>00:33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.660206</td>\n",
       "      <td>0.681334</td>\n",
       "      <td>0.494063</td>\n",
       "      <td>1.976512</td>\n",
       "      <td>00:33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.469388</td>\n",
       "      <td>0.641964</td>\n",
       "      <td>0.495615</td>\n",
       "      <td>1.900209</td>\n",
       "      <td>00:33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.512519</td>\n",
       "      <td>0.612524</td>\n",
       "      <td>0.494834</td>\n",
       "      <td>1.845082</td>\n",
       "      <td>00:33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.545736</td>\n",
       "      <td>0.625833</td>\n",
       "      <td>0.495205</td>\n",
       "      <td>1.869804</td>\n",
       "      <td>00:33</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#collapse-hide\n",
    "learn.fit_one_cycle(10, lr_max=1e-4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "learn.save('roberta_lm_pretrained')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we see how our loss progressed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD7CAYAAABkO19ZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXhU5dn48e89k32DbIQQlqAiCMpmRJRFcAHU1qWvfcXa1lortbWt+ur1U2vrVrVWa/vWLlpbbWtfqrW1Lq0ooKKIghgQIaxhCRACJATIAtnn/v0xJ3GAyT4zSWbuz3XNNWee85xz7jmZ3HPmOc95jqgqxhhjwperpwMwxhgTXJbojTEmzFmiN8aYMGeJ3hhjwpwlemOMCXOW6I0xJsy1m+hFZIiILBGRjSKyXkRuccrTRGSxiBQ6z6mtLH+dU6dQRK4L9BswxhjTNmmvH72IZAPZqrpaRJKBVcAVwDeAg6r6qIjcBaSq6p3HLZsG5AN5gDrLnqmqhwL+TowxxvgV1V4FVd0L7HWmq0RkI5ADXA7McKr9BXgPuPO4xWcDi1X1IICILAbmAC+0tc2MjAzNzc3t6HswxpiIt2rVqgOqmulvXruJ3peI5AITgI+BLOdLAFXdKyID/CySA+z2eV3slLUpNzeX/Pz8zoRmjDERTUR2tjavwydjRSQJeBm4VVUrO7qYnzK/bUUiMk9E8kUkv6ysrKNhGWOMaUeHEr2IRONN8vNV9V9O8X6n/b65Hb/Uz6LFwBCf14OBEn/bUNVnVDVPVfMyM/3++jDGGNMFHel1I8CzwEZV/YXPrNeB5l401wGv+Vl8ITBLRFKdXjmznDJjjDEh0pE2+inA14B1IrLGKfsh8CjwkojcAOwCvgwgInnATar6LVU9KCI/AT5xlnuw+cSsMcYESkNDA8XFxdTW1vZ0KEEXFxfH4MGDiY6O7vAy7Xav7Al5eXlqJ2ONMR21Y8cOkpOTSU9Px9sIEZ5UlfLycqqqqhg+fPgx80Rklarm+VvOrow1xvR5tbW1YZ/kAUSE9PT0Tv9ysURvjAkL4Z7km3XlfYZVot9fWcui9ft6OgxjTIQ5fPgwv/vd7zq93CWXXMLhw4eDENGxwirRf+l3HzHvr6vweHrfeQdjTPhqLdE3NTW1udyCBQvo379/sMJq0akrY3u7PYdrAGhSxeX3Wi1jjAm8u+66i23btjF+/Hiio6NJSkoiOzubNWvWsGHDBq644gp2795NbW0tt9xyC/PmzQM+HwWgurqaiy++mKlTp/LRRx+Rk5PDa6+9Rnx8fEDiC6sj+maNTXZEb4wJnUcffZSTTz6ZNWvW8Pjjj7Ny5UoefvhhNmzYAMBzzz3HqlWryM/P58knn6S8vPyEdRQWFnLzzTezfv16+vfvz8svvxyw+MLqiL5Zo8cDuHs6DGNMD3jg3+vZUNLRUVo6ZvSgFO774pgO1580adIx3R+ffPJJXnnlFQB2795NYWEh6enpxywzfPhwxo8fD8CZZ55JUVFR9wN3hGeityN6Y0wPSkxMbJl+7733ePvtt1m+fDkJCQnMmDHDb/fI2NjYlmm3201NTU3A4gnPRG8nY42JWJ058g6U5ORkqqqq/M6rqKggNTWVhIQENm3axIoVK0IcXdgmek9Ph2CMiSDp6elMmTKF008/nfj4eLKyslrmzZkzh6effpqxY8cycuRIJk+eHPL4wjPRW9ONMSbE/va3v/ktj42N5c033/Q7r7kdPiMjg4KCgpbyO+64I6CxhWWvm9fW7OnpEIwxptcIq0T/5TMHA/DzRVt6OBJjjOk9wirRP3bVWADOH+XvrobGGBOZwirRiwiTT0qjuraxp0MxxpheI6wSPUBMlJsG63VjjDEtwi7RR7nEet0YY4yPjtwz9jkRKRWRAp+yv4vIGudR5HOLweOXLRKRdU69kNwyKsoldsGUMabXS0pKAqCkpISrrrrKb50ZM2YQiLvtdaQf/Z+B3wDPNxeo6tXN0yLyBFDRxvIzVfVAVwPsrCi30NhkTTfGmL5h0KBB/POf/wzqNtpN9Kq6VERy/c0T761O/hs4P7BhdV2Uy0WTHdEbY0LszjvvZNiwYXz3u98F4P7770dEWLp0KYcOHaKhoYGHHnqIyy+//JjlioqK+MIXvkBBQQE1NTVcf/31bNiwgdNOOy1g4910t41+GrBfVQtbma/AIhFZJSLzurmtDolyiZ2MNcaE3Ny5c/n73//e8vqll17i+uuv55VXXmH16tUsWbKE22+/HdXWD0SfeuopEhISWLt2Lffccw+rVq0KSGzdHQLhGuCFNuZPUdUSERkALBaRTaq61F9F54tgHsDQoUO7HJDbJTTZyVhjItebd8G+dYFd58Az4OJH26wyYcIESktLKSkpoaysjNTUVLKzs7nttttYunQpLpeLPXv2sH//fgYOHOh3HUuXLuUHP/gBAGPHjmXs2LEBCb/LiV5EooAvAWe2VkdVS5znUhF5BZgE+E30qvoM8AxAXl5elzN1lNtFgzXdGGN6wFVXXcU///lP9u3bx9y5c5k/fz5lZWWsWrWK6OhocnNz/Q5R7CsYNznvzhH9hcAmVS32N1NEEgGXqlY507OAB7uxvQ6Jcom10RsTydo58g6muXPncuONN3LgwAHef/99XnrpJQYMGEB0dDRLlixh586dbS4/ffp05s+fz8yZMykoKGDt2rUBiasj3StfAJYDI0WkWERucGbN5bhmGxEZJCILnJdZwDIR+QxYCbyhqm8FJOo2RLmFBut1Y4zpAWPGjKGqqoqcnByys7O59tpryc/PJy8vj/nz5zNq1Kg2l//Od75DdXU1Y8eO5bHHHmPSpEkBiasjvW6uaaX8G37KSoBLnOntwLhuxtdpdkRvjOlJ69Z9fn4gIyOD5cuX+61XXV0NeG8Q3jxEcXx8PC+++GLAYwq/K2PdLrsy1hhjfIRfoneJ3WHKGGN8hGGid+FR8FjzjTHGAOGY6N3erkk23o0xkaWtC5HCSVfeZ/gleldzorfmG2MiRVxcHOXl5WGf7FWV8vJy4uLiOrVc2N0c3O2yI3pjIs3gwYMpLi6mrKysp0MJuri4OAYPHtypZcIu0Ue7vT9SrOeNMZEjOjqa4cOH93QYvVbYNd24renGGGOOEXaJPrr5ZKwd0RtjDBCGid7t8r4luzrWGGO8wi7RNx/R23g3xhjjFXaJvrmN3o7ojTHGK+wSfZTTdNNgbfTGGAOEZaK3I3pjjPEVfom+uY3eulcaYwwQjoneet0YY8wxwi/RW68bY4w5RvglepddMGWMMb46cs/Y50SkVEQKfMruF5E9IrLGeVzSyrJzRGSziGwVkbsCGXhrotzWdGOMMb46ckT/Z2COn/Jfqup457Hg+Jki4gZ+C1wMjAauEZHR3Qm2I5qP6K3pxhhjvNpN9Kq6FDjYhXVPAraq6nZVrQdeBC7vwno6xd2S6JX3NpdSXl0X7E0aY0yv1p02+u+JyFqnaSfVz/wcYLfP62KnLKiah0C4+W+r+cafPuFX7xQGe5PGGNOrdTXRPwWcDIwH9gJP+KkjfspabTgXkXkiki8i+YG8ecDzy3dS19gUsPUZY0xf06VEr6r7VbVJVT3AH/A20xyvGBji83owUNLGOp9R1TxVzcvMzOxKWACkJsScUDbyR2/x3ubSLq/TGGP6si4lehHJ9nl5JVDgp9onwAgRGS4iMcBc4PWubK8z0pNi2fbIJbx16zQW3Ta9pTy/6FCwN22MMb1SR7pXvgAsB0aKSLGI3AA8JiLrRGQtMBO4zak7SEQWAKhqI/A9YCGwEXhJVdcH6X0cw+0SRg1M4dSsZN67YwYxUS5rvjHGRKx27xmrqtf4KX62lbolwCU+rxcAJ3S9DKXcjEQSY9zUNlh3S2NMZAq7K2P9aWhSVu+yphtjTGSKiETvdgnrSyopOVzT06EYY0zIRUSif+TKMwDYWlrdw5EYY0zoRUSiH56RCEBNg52QNcZEnohI9PExbgC+/ddVnHH/QooPHe3hiIwxJnQiItEnOIkeoKq2kcfe2tyD0RhjTGhFRKIfkBzLrReO4KLRWQCsL6no4YiMMSZ0IiLRiwi3Xngqf/h6HpnJsUS7Xew+6L/5RtU76uVpP36LQ0fqQxypMcYEXkQkel9lVXVs2lfFtMeW+J1/1sPv8I0/fUJNQxNby6yXjjGm74u4RD9+SP9W53k8ygGf8euraxtDEZIxxgRVxCX6v914NumJ3hEua326W1bUNHDKPceO1nDoqDXdGGP6vohL9AkxUVw/JReAz3YfBmBX+VHGPbCI5tvMLvjBNAD2VtRS3+gh9643eH55UeiDNcaYAGh3ULNwdO4pGbBoC0frvUf00x8/tr1+9KAUBvWLY1nhAVzivX/Kva95B978+jm5IY3VGGO6KyITfXKs921X1zUe0yYPkNM/HoCSilpKKmpZvr28Zd69r61vSfgf/L+ZDElLCFHExhjTdRHXdAOQ6JPol275/LaFQ9LiWXCLt9lmaDtJfNpjS/je31ZTsKeCe18rwONp9S6JxhjToyLyiD4pzkn0tY389F3vzcPv++Jovpw3hCTnS+D+y0bzzT/nt7me/6zdy3/W7gVgRFYy00dkMCw9MYiRG2NM50XmEX2MN5kXHzpKZW0js8dkcf2U4S1JHqBf/Of3nk1zeulce/bQVtf541cLuOK3H55Q7vEo/1pdTGVtQ6DCN8aYTunIrQSfE5FSESnwKXtcRDaJyFoReUVE/HZOF5Ei55aDa0Sk7cPjEHK7hIQYN4s37Adg3vSTTqjTPD7OWbmpVNR4k/Tts0ZS9OilXHjaAL/rPXS0gXc27j+m7GcLN/E/L33GS5/sDuRbMMaYDutI082fgd8Az/uULQbuVtVGEfkZcDdwZyvLz1TVA92KMgiO1je19LoZO/jE76mRWcnccsEIrj17KIWl1bxZsLflyP4XV4/nw8IDfLzjIJeNH8TmfVX8cvEWSqvqmP/xLs49OYOahib6xUfzf8t3ArB5X1Xo3pwxxvgQ1fZPIopILvAfVT3dz7wrgatU9Vo/84qAvM4m+ry8PM3PD+4PgNy73gCgf0I0a+6d1e31lVbVMunhdwBvz509fu5mVfTopd3ejjHG+CMiq1Q1z9+8QLTRfxN4s5V5CiwSkVUiMi8A2wqYWc5IlqMGJgdkfQOS41qmj0/yGUmxADQ02Q3KjTGh161ELyL3AI3A/FaqTFHVicDFwM0iMr2Ndc0TkXwRyS8rK2utWsDc6LTLzxkzMGDrvGL8oBPKZozMJCXe20L27qbSgG3LGGM6qsuJXkSuA74AXKuttP+oaonzXAq8AkxqbX2q+oyq5qlqXmZmZlfD6rCzctNY8INpXHdubsDWef9lYzhlQBIAZw9P44rxg/j5l8fx8BXee9Y2n9Q1xphQ6lI/ehGZg/fk63mq6ndgdxFJBFyqWuVMzwIe7HKkQTB6UEpA19c/IYY3fjCVyppGMpNjW8pjorzfp5WW6I0xPaAj3StfAJYDI0WkWERuwNsLJxlY7HSdfNqpO0hEmoeAzAKWichnwErgDVV9KyjvoheJjXIfk+QBkpx++3/6sKgHIjLGRLp2j+hV9Ro/xc+2UrcEuMSZ3g6M61Z0YcLl8g6MtudwDQvX72N2AM8LGGNMeyLyytie8IMLRgCwcW9lD0dijIk0luhD5HszTwHgN+9u7eFIjDGRxhJ9iDSfkG3qwAVqxhgTSJboQ+ibU4a3DKhmjDGhYok+hFLio6iua2TVzkM9HYoxJoJYog+h5iEQ/uupjyitrO3haIwxkcISfXc11MLKP0DDiYOYHe+GqZ8Ph/zAfzYEMypjjGlhib679qyCBXd4k3070hJjWHSbd7ifN5w7UxljTLBZou+u3ClwykXwwRNQ037b+whnLJyMpJh2ahpjTGBYog+EC++D2gpY9r/tVhURvnFuLnWNNmSxMSY0LNEHwsAzYOzV8PHTULGn3er9E6Kpqm208emNMSFhiT5QZv4Q1APv/bTdqpU1jQC8vzn44+4bY4wl+kBJHQZn3Qhr5kPppjarXjkhB4BH32q7njHGBIIl+kCadjvEJME7bQ+73zwO/tbSagBUlY7cu9cYY7rCEn0gJabDlFtg8xuwa0Wr1dwu4abzTibG7eLgkXqG372AC554P4SBGmMiiSX6QJv8HUgaCIvvgzaO0uOj3dQ3eZj4k8UAbD9whE+KDoYqSmNMBLFEH2gxiTDjLti9Aja/2Wq10qoTh0C45pkVTPzJYl7K3x3MCI0xEaZDiV5EnhORUhEp8ClLE5HFIlLoPKe2sux1Tp1C54bi4W/C1yD9FHjnAWhq9FvlzotHtUxveHA2J2Uk0uhRDh6pZ/m28lBFaoyJAB09ov8zMOe4sruAd1R1BPCO8/oYIpIG3AecDUwC7mvtCyGsuKPggvugbBN89oLfKilx0fzp+rP49McXkRATxS0Xjmi5avaVT9vvi2+MMR3VoUSvqkuB4xuQLwf+4kz/BbjCz6KzgcWqelBVDwGLOfELIzyd9kXIyYMlj7Q64NnMkQNITfQOhXD5+BwW/8959IuPBiD3rjco2FMRsnCNMeGrO230Waq6F8B5HuCnTg7g2+Bc7JSFPxG46AGoKoGPf9/hxR658oyW6S/8ehn/+/YWPB7remmM6bpgn4wVP2V+s5aIzBORfBHJLysLkytGc6fCiNmw7BdwtGM9ai4dm83qH1/ET644HYD/fbuQiQ8tpsmSvTGmi7qT6PeLSDaA81zqp04xMMTn9WCgxN/KVPUZVc1T1bzMzMxuhNXLXHgf1FbCsl92eJG0xBi+NnkY//7eVIakxXP4aAOL1u8LYpDGmHDWnUT/OtDci+Y64DU/dRYCs0Qk1TkJO8spixxZY2DcNd7mm4riTi16xuB+vHfHTHL6x/P7pduDFKAxJtx1tHvlC8ByYKSIFIvIDcCjwEUiUghc5LxGRPJE5I8AqnoQ+AnwifN40CmLLDN/6H1e0v6AZ8dzu4TzRw1gze7DlBxu/y5WxhhzPOmNY6zk5eVpfn5+T4cRWAvvgRW/g5s+hKzRnVp08Yb93Ph8PhlJsfz2KxM4+6T0IAVpjOmrRGSVqub5m2dXxobKtNshJrndAc/8uWh0FvO/dTYAv3x7S6AjM8aEOUv0oZKQBlNvhS1vws6POr34lFMyGD+kHyu2H7TulsaYTrFEH0pn3wTJ2e0OeNaaMYP6AVB+pD7QkRljwpgl+lCKSYAZd0PxStj0RqcXPy3bO479/soTB0QzxpjWWKIPtfHXQsapbQ541pqB/eIA2Fdhid4Y03GW6EOtecCzA1u8tx3shIEp3kT/80WbqalvCkZ0xpgwZIm+J4y6FAZP8t5IvP5ohxfLSonlSxNy2LSvilte/DSIARpjwokl+p7QMuDZXvj46U4sJvzi6vF8aUIOizbsZ8v+qiAGaYwJF5boe8qwc+HUi2HZ/3Z4wLNmt88eSWKMm4fe2Bik4Iwx4cQSfU+64F6or4IPnujUYjn94/ly3hA+KCyjtsHa6o0xbbNE35OyRsO4r8DKZ+Dwrk4tOvmkNFTh/tfXByk4Y0y4sETf02beDUinBzybc3o244b0Z83uw8GJyxgTNizR97R+g+Hsb3vvLbu/c0fnM07NZNO+Kkv2xpg2WaLvDabeBnEp8PYDnVrsW9OGkxwbxUP/2UBjkydIwRlj+jpL9L1BQhpM/R8oXAhFyzq8WHJcNHdePIr8nYdYazcSN8a0whJ9b3H2tyF5UKcHPJvsjE1fdOBIsCIzxvRxluh7i+h4752o9uTDxn93eLGhaQkkxLhZsG5vEIMzxvRlluh7k3HXQOaoTg14FhPlYvaYgeTvPERdo/WpN8acqMuJXkRGisgan0eliNx6XJ0ZIlLhU+fe7occxpoHPCvfCp/+tcOLXTZuEIePNvCbd7cGMThjTF/V5USvqptVdbyqjgfOBI4Cr/ip+kFzPVXt/H30Is3Ii2HIZHjvUajvWLv7zFED+NLEHJ56bxvby6qDHKAxpq8JVNPNBcA2Vd0ZoPVFruYBz6r3wYqnOrzYbReeSqNHeW9zWRCDM8b0RYFK9HOBF1qZd46IfCYib4rImABtL7wNnQwjL4UPf9XhAc8Gp8Zzek4KT7+/jcrahiAHaIzpS7qd6EUkBrgM+Ief2auBYao6Dvg18Gob65knIvkikl9WZkel3gHPqjs84JmI8MiVZ1BWXcdzy3YEOThjTF8SiCP6i4HVqrr/+BmqWqmq1c70AiBaRDL8rURVn1HVPFXNy8zMDEBYfdyAUd7bDnZiwLOxg/tzRk4/lhUeCHJwxpi+JBCJ/hpaabYRkYEiIs70JGd75QHYZmSYcTeIC5Y80uFFmrtafrrrUBADM8b0Jd1K9CKSAFwE/Mun7CYRucl5eRVQICKfAU8Cc1U7cdlnpOuXA2ffBJ+9CPsKOrTIdefmkhQbxYsrdwc5OGNMX9GtRK+qR1U1XVUrfMqeVtWnnenfqOoYVR2nqpNV9aPuBhxxpt4Kcf28F1F1QFJsFBefPpBXPt1DdV3HLroyxoQ3uzK2t4tPhWm3Q+Ei2PFBhxaZPWYg9U0elmwqDXJwxpi+wBJ9XzBpHqQMhrc7NuDZeSMzGZmVzDNLt4cgOGNMb2eJvi+IjnMGPFsFG19vv7rbxRfHZbNuTwWlVbUhCNAY05tZou8rxs2FzNPgnQehqf0LomaMHADAR1utk5Mxkc4SfV/hcsOF93d4wLORA5OJcbt4eXVx0EMzxvRuluj7klNnw9BzOzTgWbTbxTenDueDwgNsKKkMUYDGmN7IEn1f0jLg2X5Y8bt2q984bTjJcVHc8Y/PqG2wseqNiVSW6PuaIZNg1Bdg2a/gSNvt7+lJsfz8y+PYsLfS7kBlTASzRN8XXXAvNByBD37ebtWLTssip388v353KzX1dlRvTCSyRN8XZY6ECV+FlX+AQ0VtVnW5hJ9+6Qx2HDjC0+9vC018xphexRJ9XzXjbm9PnHcfbrfq9FMzmTYigxc/2UVjkycEwRljehNL9H1VyiCY/F1Y95L3Qqp2XDNpKPsr61hZ1LEbmRhjwocl+r5s6m2QmAkL72l3aIQZIzOJi3bxVsG+EAVnjOktLNH3ZXEpcP6PYNfydodGSIiJYsapA3irYB8ej40UbUwksUTf1034GgwYA4vvhca6NqteMjab0qo68nfaTUmMiSSW6Ps6lxtmP+TtffPx79usev6oAcRGuaxPvTERxhJ9ODj5fBgxC5Y+Dkdav19sUmwUM0cO4D9r91rvG2MiSLcTvYgUicg6EVkjIvl+5ouIPCkiW0VkrYhM7O42jR+zHvKOf/Peo21Wu2JCDgeq6/hgq91A3JhIEagj+pmqOl5V8/zMuxgY4TzmAU8FaJvGV+ZIyPsm5D8HZZtbrXb+qAH0i4/m1U/3hDA4Y0xPCkXTzeXA8+q1AugvItkh2G7kmXEXxCTBoh+3WiUmysWlY7NZtH4/R+yessZEhEAkegUWicgqEZnnZ34OsNvndbFTZgItMQOm3wGFC2Hbu61Wu2J8DjUNTSzaYH3qjYkEgUj0U1R1It4mmptFZPpx88XPMid05BaReSKSLyL5ZWVlAQgrQp39bUjN9V5E5fE/iFnesFRy+sfz6qcloY3NGNMjup3oVbXEeS4FXgEmHVelGBji83owcEKGUdVnVDVPVfMyMzO7G1bkioqFix6E0g2t3onK5RIuHz+IDwrLKKtqu++9Mabv61aiF5FEEUlungZmAQXHVXsd+LrT+2YyUKGq1pE7mE67DIaeA+8+BHVVfqtcOSEHj8J/1tpRvTHhrrtH9FnAMhH5DFgJvKGqb4nITSJyk1NnAbAd2Ar8AfhuN7dp2iMCsx+GI2XwwS/8VhmRlczo7BTrfWNMBIjqzsKquh0Y56f8aZ9pBW7uznZMF+ScCWOvhuW/hbzrof/QE6pcOSGHhxdsZHtZNSdlJvVAkMaYULArY8PZBfeCuODtB/zO/uK4QYjAq2us+caYcGaJPpz1Gwznfh8K/gm7Pzlh9sB+cZx7cjqvrdmDtjPMsTGm77JEH+6m3AJJWbDwh37HrL98fA47y4+yZvfhHgjOGBMKlujDXWwSnP9jKF4J6/91wuw5pw8kJsrFy6uLeyA4Y0woWKKPBOO/AgPPgMX3Q0PtMbNS4qL54thB/Gv1HqptSARjwpIl+kjgcsOsh6FiF3x84phyXzl7CEfrm3jTxqk3JixZoo8UJ50HIy+BpU9A9bFDTEwcmsrwjET+scqab4wJR5boI8lFD0JjDSx5+JhiEeHqs4awcsdBNu/zfyWtMabvskQfSTJGwFnfgtV/gf0bjpn133lDiHYL/8jf3crCxpi+yhJ9pDnvTohNgUU/OqY4LTGGC0Zl8eqaPTTYbQaNCSuW6CNNQpo32W97BwrfPmbWVWcO5kB1PUs2lfZQcMaYYLBEH4nO+haknQSL7oGmz7tUzhiZSUZSrJ2UNSbMWKKPRFExcNFPoGwTrP7z58VuF1edOZh3Nu6n+NDRnovPGBNQlugj1ahLYdhUWPII1Fa0FH/9nGGICC+s3NWDwRljAskSfaRqHrP+6EH44ImW4kH945lySgavf1ZiA50ZEyYs0UeyQeO9wyOseAoOFbUUXz5uELsP1rB6lw10Zkw4sEQf6c7/EbiiYPF9LUWzxmQRF+2yu08ZEya6nOhFZIiILBGRjSKyXkRu8VNnhohUiMga53Fv98I1AZcyyDuU8YZXYdcKAJLjorlo9ED+vbaE+kbrU29MX9edI/pG4HZVPQ2YDNwsIqP91PtAVcc7jwe7sT0TLOd+H5KzvWPWe7yJ/coJgzh8tIH3t5S1s7AxprfrcqJX1b2qutqZrgI2AjmBCsyEUEwiXHAf7FkFBS8DMG1EJumJMTy7bHsPB2eM6a6AtNGLSC4wAfjYz+xzROQzEXlTRMYEYnsmCMZeDdnj4O37of4o0W4X35w6nBXbD/LprkM9HZ0xphu6nehFJAl4GbhVVSuPm70aGKaq44BfA6+2sZ55IpIvIvllZdZcEHIuF8x+BCqLYcVvAbju3FySY6N4YtGWHg7OGNMd3Ur0IhKNN8nPV9UT7nkmfl8AAA/0SURBVFOnqpWqWu1MLwCiRSTD37pU9RlVzVPVvMzMzO6EZboqdyqM+gJ88Euo2k9SbBT/fdYQlm09wI9eXdfT0Rljuqg7vW4EeBbYqKq/aKXOQKceIjLJ2V55V7dpQuCiB6GpHpY8BMCtF44A4MWVuyk5XNOTkRljuqg7R/RTgK8B5/t0n7xERG4SkZucOlcBBSLyGfAkMFftcsveLf1kOPvbsPqvsG8dyXHRLLtzJgr88YMdPR2dMaYLorq6oKouA6SdOr8BftPVbZgeMv0OWDMfFt4DX3+NwakJXDE+h+c+3MGZw1K5dGx2T0dojOkEuzLWnCg+FWbcDTvehy0LAbjlAm8Tzs1/W83ugzaypTF9iSV641/eNyF9hPdOVE0NDE1P4K83TALggX+vp8ljLXDG9BWW6I1/7miY9RCUF0L+nwDvRVR3XzyKtzeW8vT723o4QGNMR1miN607dTYMPw/eewRqvBdNffu8k5k2IoPHF27mlB8usCN7Y/oAS/Smdc1j1tcchqU/byl++IozAGj0KF/948c2br0xvZwletO2gWfAhK/Cx7+Hcm9zzdD0BHb89BLOyOnH8u3l3Ph8Psu3lVvCN6aXskRv2nf+j8AdA29/Pma9iPDqzVMYkhbP2xtLueYPK3h2mfWzN6Y3skRv2pc8EKbdBhv/DUUfthS7XcLCW6dz4WlZADz0xkaue26ltdsb08tIb/y5nZeXp/n5+T0dhvHVUAO/zoPEDLhxiXcQNB+VtQ2MvX8RAHHRLvKGpfHwlaczLD2xJ6I1JuKIyCpVzfM3z47oTcdEx8OF98HeNbD27yfMTomLZv0Ds/nezFOobfCwbOsBznv8PYoOHOmBYI0xvuyI3nScxwN/vACq9sH38703LPGjYE8FK7aX86t3CqmqbeS07BT+fP1ZZKXEhThgYyKHHdGbwHC5YM5PoaoEPmp9CKPTc/rxrWkn8Y+bzmFAciwb91Yy5dF3eWbpNt4q2Et+0UE8rbTjd/XAY1tZNRVHG1pel1bWknvXGzz0nw0hP2dQXdfIR1sP0Nhk99s1vYMd0ZvOe+k6KFwE4+YCAuLy9rkX13GvBRAO1TTyfuEBig/XoQgehH7xMUw7dQCxUW4aPLCupIpN+6pRhPNGDmBYRhLFh2vJ6hfP0LSklvWXVdezbk8FB480ULDnMB7n8+s7ut6Q1DhKK+uob/IgKOlJsZw6IJGJw1LJSIwFfD7zx3z+/ZV7n5s8yjubShmakcLInAy2lNezq7KRtSU1pKYkMfGkLMblDiB/91Huf3MrDUQhUTHcNud01u+v5/3th7lsYi6jB2cyPjedP31YxCdFB/loWzmXnpHNKQOSqGv08NWzhzIgiL98Gpo87Dp4lIEpcVz9zHIuPC2L758/ArfL//iEHo/ywie7yO4Xx/ghqaTERRHltuPD3qitI3pL9KbzDu2EF6+F6n2gHm9SVA+gzrTv68/ne1RR9SZfF73vcxcqjeqinmjqifr8Wb3TDURRRzTxcXE0uWLYf0SpI4qoqBhqGj0kxkYzPCOJgf3jSY6LwfvF+vmXqu8X7rqSSg4ebSQhJoqsfvHsPlTDRufL1PsAxUV2/3jOH5VFcnwMHhU+3nGQ4sM1HKn3cOhoI024aMJNE4K6omjwCKlJ8eyvbqQRFzlpSRQdrKNJXTThQsVNdmoipw9Jp19iLIlxcai4mJCbSWx0NLiivL8OXVEgbue12/vwfS1ubz1xOdPuz6fFdUKHgEhnid70Og1NHp79YDuPvbWRr509hLr6Rm658BQyE6P5pOggv3t3CydlJpGVHM3/LS8ClKN1DbhQctPiuWTsIM4/LYsYt4usfnFEu7y/ImobmqisbaC6rpGhaYlEuYQGD7y6Zg8fbz/Eog37AFCf3wDHHtPLCdPN8wX4/oxhPP/BFlxN9Xz/vKH0i1HiXE3sK6/g9dU7SHJ7+OpZAznv5H7U19WyaO1OXE0NnD00keqjNWzcXUZdXS1lhys5a3Aig5LdeBrrqD5yhFhp5MDhKqqPHCFamoilgRgaiaGBKG+adR4gosS4hIQYF26Bqtp6BCXKJbhQGpuaWuq68LS8E9/XUS5wCzQ1NQGKS/Auo4pLel9e8EfFhbT2JdAy7TPP94vjmHmfT9c1CXsraqiurSfaJXhUiXVDTX0jqkpSrJuBKbHEur0xiHq/Mml59tDo8eBpabrzTh88UodbID7aRbRbqG9oJD7a5Xzlev8WkpiO3Ozv1tvts0Rv+ry6xibeKthHyeFavj39JFytNDV0lKpSsKeSfZW1qCqjB6WQlRLH2xv285flRYwYkIyivL6mhAEpcaQlxPDUVyeSnhQbmDfUhu1l1aQlxvD6ZyV8aeJgmjxKZU0DeytqGTu4HwvX7+ONtXvZc7iG9SWf36Y5NSGaaLeL0qo6AMYMSuF3106ksqaRpYVlDEiO5aozB+Pc9K1F8aGjvPTJbpYWHmDN7sPkDUvlW9OGM7h/PAkxLk5KjwdPE3gaaWpqpK6+njg37DxQwdD+cRyoPEK/WDdxbgVtYs/BKuKjYOW2AyREKyu2ltI/zsWWvYepPFpLfUMDUXhw42FoaizTT0mlqbGR8qqj5KREkxzroqK6ho0lhxiQHEN9fQNpCW5QZf/hIyTFuugf52ZVUTkuZz0nZ8QzemASA1NiaGhopKqmlsbGJuobGiirrKG6to4YFyRECWgTMW5BPY0cOlJHQ0MDSTFCtAtEPVTX1Pl8yXubGo/9FSTHPEQEt8vl/UyKiyP1TcfU97T8fnXK9PMybUnz3nnxyalc+f/+eMLfqCMs0RsTpjaUVPL3T3Yx5/Rszjk5HVWlzEn0XWnrrzjaQGy0i7hod6BDBby/5P61upi3N5ay++BRNu2r6tb6TstO4UhdIyWHa2jswEn3mCgX9Y3tnyS/cdpwrpiQQ2ZyLEmxUWzcW8nQtET6xUdztL6R3y/dzlsF+9hx4AjZ/eI4dLSe2obP1zv91EwmDOnP6l2H6J8Qw97DNUwdkcEtF4ygpKKWtwr2kZoQTX2jhw+3lRMX5WJvRS1fO2cYs8cM7NK+sERvjOmVVJXC0mre2VjKBacNwKNK8cEaCkoqGDe4P2NyUohyuXC7hNW7DjFmUAppCTG8v6WM3IxETs5MAqC2oYnl28v5ZMdBtpcdoV98NP0ToqltaGLySelMOzWTeOfLq7ahie1lR0iIdbOj7Ahn5aaRGOvG7RJ2lh8lJzWe6C6ecPZ4tNu/NrsqaIleROYAvwLcwB9V9dHj5scCzwNn4r0p+NWqWtTeei3RG2NM5wSlH72IuIHfAhcDo4FrRGT0cdVuAA6p6inAL4GfdXV7xhhjuqY7/ZMmAVtVdbuq1gMvApcfV+dy4C/O9D+BC6QrZxmMMcZ0WXcSfQ6w2+d1sVPmt46qNgIVQLq/lYnIPBHJF5H8srKyboRljDHGV3cSvb8j8+Mb/DtSx1uo+oyq5qlqXmZmZjfCMsYY46s7ib4YGOLzejBQ0lodEYkC+gEHu7FNY4wxndSdRP8JMEJEhotIDDAXeP24Oq8D1znTVwHvam/sz2mMMWEsqqsLqmqjiHwPWIi3e+VzqrpeRB4E8lX1deBZ4K8ishXvkfzcQARtjDGm47qc6AFUdQGw4Liye32ma4Evd2cbxhhjuqdXXhkrImXAzi4smgEcCHA4gWBxdU5vjQt6b2wWV+eEY1zDVNVvT5Zemei7SkTyW7syrCdZXJ3TW+OC3hubxdU5kRaXDehsjDFhzhK9McaEuXBL9M/0dACtsLg6p7fGBb03NourcyIqrrBqozfGGHOicDuiN8YYc5ywSfQiMkdENovIVhG5K4TbHSIiS0Rko4isF5FbnPL7RWSPiKxxHpf4LHO3E+dmEZkd5PiKRGSdE0O+U5YmIotFpNB5TnXKRUSedGJbKyITgxTTSJ/9skZEKkXk1p7YZyLynIiUikiBT1mn94+IXOfULxSR6/xtKwBxPS4im5xtvyIi/Z3yXBGp8dlvT/ssc6bz99/qxN6t0WNbiavTf7dA/7+2EtfffWIqEpE1Tnko91dr+SG0nzFV7fMPvFfmbgNOAmKAz4DRIdp2NjDRmU4GtuAdn/9+4A4/9Uc78cUCw5243UGMrwjIOK7sMeAuZ/ou4GfO9CXAm3gHo5sMfByiv90+YFhP7DNgOjARKOjq/gHSgO3Oc6oznRqEuGYBUc70z3ziyvWtd9x6VgLnODG/CVwchLg69XcLxv+rv7iOm/8EcG8P7K/W8kNIP2PhckTfkbHxg0JV96rqame6CtjIicM1+7oceFFV61R1B7AVb/yh5HufgL8AV/iUP69eK4D+IpId5FguALapalsXyAVtn6nqUk4caK+z+2c2sFhVD6rqIWAxMCfQcanqIvUO9w2wAu9Agq1yYktR1eXqzRbP+7yXgMXVhtb+bgH/f20rLueo/L+BF9paR5D2V2v5IaSfsXBJ9B0ZGz/oRCQXmAB87BR9z/n59VzzTzNCH6sCi0RklYjMc8qyVHUveD+IwIAeig284x/5/gP2hn3W2f3TE/vtm3iP/JoNF5FPReR9EZnmlOU4sYQirs783UK9v6YB+1W10Kcs5PvruPwQ0s9YuCT6Do97H7QARJKAl4FbVbUSeAo4GRgP7MX70xFCH+sUVZ2I95aPN4vI9DbqhjQ28Y56ehnwD6eot+yz1rQWR6j32z1AIzDfKdoLDFXVCcD/AH8TkZQQxtXZv1uo/57XcOzBRMj3l5/80GrVVmLoVmzhkug7MjZ+0IhINN4/4nxV/ReAqu5X1SZV9QB/4POmhpDGqqolznMp8IoTx/7mJhnnubQnYsP75bNaVfc7MfaKfUbn90/I4nNOwn0BuNZpXsBpGil3plfhbf8+1YnLt3knKHF14e8Wyv0VBXwJ+LtPvCHdX/7yAyH+jIVLou/I2PhB4bT/PQtsVNVf+JT7tm1fCTT3BngdmCsisSIyHBiB9wRQMGJLFJHk5mm8J/MKOPY+AdcBr/nE9nXnzP9koKL552WQHHOk1Rv2mc/2OrN/FgKzRCTVabaY5ZQFlIjMAe4ELlPVoz7lmSLidqZPwrt/tjuxVYnIZOdz+nWf9xLIuDr7dwvl/+uFwCZVbWmSCeX+ai0/EOrPWHfOKPemB96z1VvwfjvfE8LtTsX7E2otsMZ5XAL8FVjnlL8OZPssc48T52a6eVa/ndhOwtuj4TNgffN+wXvf3neAQuc5zSkX4LdObOuAvCDGlgCUA/18ykK+z/B+0ewFGvAeNd3Qlf2Dt818q/O4PkhxbcXbTtv8OXvaqftfzt/3M2A18EWf9eThTbzbgN/gXCQZ4Lg6/XcL9P+rv7ic8j8DNx1XN5T7q7X8ENLPmF0Za4wxYS5cmm6MMca0whK9McaEOUv0xhgT5izRG2NMmLNEb4wxYc4SvTHGhDlL9MYYE+Ys0RtjTJj7/3mCbLd2b72mAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#hide\n",
    "learn.recorder.plot_loss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lets Look at the model's predictions\n",
    "Manually checking how well our model makes predictions for masked tokens is a simple way to see how it is training\n",
    "\n",
    "Here function `get_mask_pred` takes masked string given by the user and returns the `topk` predictions given by the model for that masked token. With it we can sanity check that our model has learned something useful!  \n",
    "\n",
    "*Note that `get_mask_pred` is mostly code from `FillMaskPipeline` in HuggingFace's Transformers repo, full credit to them!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#collapse\n",
    "def get_mask_pred(model, masked_text:str, topk:int=5):\n",
    "    \"Code lightly modified from `FillMaskPipeline` in the HuggingFace Transformers library\"\n",
    "    \n",
    "    aa=fastai_tokenizer.encodes(masked_text)\n",
    "    bb=Numericalize(vocab=tokenizer_vocab_ls)(aa)\n",
    "    cc=AddSpecialTokens(tokenizer)(bb)\n",
    "    \n",
    "    outs=model(cc.unsqueeze(0).cuda())\n",
    "    masked_index = (cc == tokenizer.mask_token_id).nonzero().item()\n",
    "    logits = outs[0, masked_index, :]\n",
    "    probs = logits.softmax(dim=0)\n",
    "    values, predictions = probs.topk(topk)\n",
    "    \n",
    "    result=[]\n",
    "    for i, vv in enumerate(zip(values.tolist(), predictions.tolist())):\n",
    "        v, p =vv\n",
    "        tokens = cc.numpy()\n",
    "        if i == 0: result.append({\"word\":\"Input text\", \"score\": 0., \"token\": 0, \"sequence\": tokenizer.decode(tokens)}) \n",
    "        tokens[masked_index] = p\n",
    "        tokens = tokens[np.where(tokens != tokenizer.pad_token_id)]\n",
    "        w = tokenizer.decode(p)\n",
    "        result.append({\"word\":w, \"score\": v, \"token\": p, \"sequence\": tokenizer.decode(tokens)}) \n",
    "\n",
    "    return pd.DataFrame(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can input our own masked sentence and see how the model does. Note that even without fine-tuning the performance below will still be very strong as the pretrained RoBERTa model is very strong."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "      <th>score</th>\n",
       "      <th>token</th>\n",
       "      <th>sequence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Input text</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt;s&gt; I was walking to&lt;mask&gt; when I came across a cat on the road&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>school</td>\n",
       "      <td>0.791473</td>\n",
       "      <td>334</td>\n",
       "      <td>&lt;s&gt; I was walking to school when I came across a cat on the road&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>church</td>\n",
       "      <td>0.068957</td>\n",
       "      <td>2352</td>\n",
       "      <td>&lt;s&gt; I was walking to church when I came across a cat on the road&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>work</td>\n",
       "      <td>0.068007</td>\n",
       "      <td>173</td>\n",
       "      <td>&lt;s&gt; I was walking to work when I came across a cat on the road&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>breakfast</td>\n",
       "      <td>0.007202</td>\n",
       "      <td>7080</td>\n",
       "      <td>&lt;s&gt; I was walking to breakfast when I came across a cat on the road&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         word     score  token  \\\n",
       "0  Input text  0.000000      0   \n",
       "1      school  0.791473    334   \n",
       "2      church  0.068957   2352   \n",
       "3        work  0.068007    173   \n",
       "4   breakfast  0.007202   7080   \n",
       "\n",
       "                                                                  sequence  \n",
       "0      <s> I was walking to<mask> when I came across a cat on the road</s>  \n",
       "1     <s> I was walking to school when I came across a cat on the road</s>  \n",
       "2     <s> I was walking to church when I came across a cat on the road</s>  \n",
       "3       <s> I was walking to work when I came across a cat on the road</s>  \n",
       "4  <s> I was walking to breakfast when I came across a cat on the road</s>  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text2 = 'I was walking to <mask> when I came across a cat on the road'\n",
    "pred2 = get_mask_pred(model1, text2);pred2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Not bad at all!** Now lets see how it does on a movie review, lets look at an example from our validation set. We mask the word `might` from the first sentence of the reivew, `... shows what might happen...` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "      <th>score</th>\n",
       "      <th>token</th>\n",
       "      <th>sequence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Input text</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt;s&gt; This very funny British comedy shows what&lt;mask&gt; happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes &amp; post-war restrictions. Merry mayhem is what would happen.&lt;br /&gt;&lt;br /&gt;The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.&lt;br /&gt;&lt;b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>would</td>\n",
       "      <td>0.809723</td>\n",
       "      <td>74</td>\n",
       "      <td>&lt;s&gt; This very funny British comedy shows what would happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes &amp; post-war restrictions. Merry mayhem is what would happen.&lt;br /&gt;&lt;br /&gt;The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.&lt;br /&gt;&lt;b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>might</td>\n",
       "      <td>0.131539</td>\n",
       "      <td>429</td>\n",
       "      <td>&lt;s&gt; This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes &amp; post-war restrictions. Merry mayhem is what would happen.&lt;br /&gt;&lt;br /&gt;The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.&lt;br /&gt;&lt;b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>could</td>\n",
       "      <td>0.042638</td>\n",
       "      <td>115</td>\n",
       "      <td>&lt;s&gt; This very funny British comedy shows what could happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes &amp; post-war restrictions. Merry mayhem is what would happen.&lt;br /&gt;&lt;br /&gt;The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.&lt;br /&gt;&lt;b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>will</td>\n",
       "      <td>0.009556</td>\n",
       "      <td>40</td>\n",
       "      <td>&lt;s&gt; This very funny British comedy shows what will happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes &amp; post-war restrictions. Merry mayhem is what would happen.&lt;br /&gt;&lt;br /&gt;The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.&lt;br /&gt;&lt;br...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         word     score  token  \\\n",
       "0  Input text  0.000000      0   \n",
       "1       would  0.809723     74   \n",
       "2       might  0.131539    429   \n",
       "3       could  0.042638    115   \n",
       "4        will  0.009556     40   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  sequence  \n",
       "0  <s> This very funny British comedy shows what<mask> happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes & post-war restrictions. Merry mayhem is what would happen.<br /><br />The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.<br /><b...  \n",
       "1  <s> This very funny British comedy shows what would happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes & post-war restrictions. Merry mayhem is what would happen.<br /><br />The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.<br /><b...  \n",
       "2  <s> This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes & post-war restrictions. Merry mayhem is what would happen.<br /><br />The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.<br /><b...  \n",
       "3  <s> This very funny British comedy shows what could happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes & post-war restrictions. Merry mayhem is what would happen.<br /><br />The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.<br /><b...  \n",
       "4  <s> This very funny British comedy shows what will happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes & post-war restrictions. Merry mayhem is what would happen.<br /><br />The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.<br /><br...  "
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mask_indices=[7]\n",
    "txts=df.text.values\n",
    "masked_text = txts[800].split(' ')  # our validation split starts at index 800\n",
    "masked_text[mask_indices[0]] = '<mask>'\n",
    "masked_text = \" \".join(masked_text)\n",
    "\n",
    "pred1 = get_mask_pred(model1, masked_text);pred1.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Boom, pretty darn good! Lets try the same example, replacing `ancient` in `discovery of ancient documents ` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "      <th>score</th>\n",
       "      <th>token</th>\n",
       "      <th>sequence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Input text</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt;s&gt; This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes &amp; post-war restrictions. Merry mayhem is what would happen.&lt;br /&gt;&lt;br /&gt;The explosion of a wartime bomb leads to the discovery of&lt;mask&gt; documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.&lt;br /&gt;&lt;br ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>historical</td>\n",
       "      <td>0.585666</td>\n",
       "      <td>4566</td>\n",
       "      <td>&lt;s&gt; This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes &amp; post-war restrictions. Merry mayhem is what would happen.&lt;br /&gt;&lt;br /&gt;The explosion of a wartime bomb leads to the discovery of historical documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.&lt;br /...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>old</td>\n",
       "      <td>0.086817</td>\n",
       "      <td>793</td>\n",
       "      <td>&lt;s&gt; This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes &amp; post-war restrictions. Merry mayhem is what would happen.&lt;br /&gt;&lt;br /&gt;The explosion of a wartime bomb leads to the discovery of old documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.&lt;br /&gt;&lt;br /&gt;...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>obscure</td>\n",
       "      <td>0.040825</td>\n",
       "      <td>23732</td>\n",
       "      <td>&lt;s&gt; This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes &amp; post-war restrictions. Merry mayhem is what would happen.&lt;br /&gt;&lt;br /&gt;The explosion of a wartime bomb leads to the discovery of obscure documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.&lt;br /&gt;&lt;b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ancient</td>\n",
       "      <td>0.035504</td>\n",
       "      <td>8178</td>\n",
       "      <td>&lt;s&gt; This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes &amp; post-war restrictions. Merry mayhem is what would happen.&lt;br /&gt;&lt;br /&gt;The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.&lt;br /&gt;&lt;b...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          word     score  token  \\\n",
       "0   Input text  0.000000      0   \n",
       "1   historical  0.585666   4566   \n",
       "2          old  0.086817    793   \n",
       "3      obscure  0.040825  23732   \n",
       "4      ancient  0.035504   8178   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  sequence  \n",
       "0  <s> This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes & post-war restrictions. Merry mayhem is what would happen.<br /><br />The explosion of a wartime bomb leads to the discovery of<mask> documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.<br /><br ...  \n",
       "1  <s> This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes & post-war restrictions. Merry mayhem is what would happen.<br /><br />The explosion of a wartime bomb leads to the discovery of historical documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.<br /...  \n",
       "2  <s> This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes & post-war restrictions. Merry mayhem is what would happen.<br /><br />The explosion of a wartime bomb leads to the discovery of old documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.<br /><br />...  \n",
       "3  <s> This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes & post-war restrictions. Merry mayhem is what would happen.<br /><br />The explosion of a wartime bomb leads to the discovery of obscure documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.<br /><b...  \n",
       "4  <s> This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes & post-war restrictions. Merry mayhem is what would happen.<br /><br />The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.<br /><b...  "
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mask_indices=[54]\n",
    "txts=df.text.values\n",
    "masked_text = txts[800].split(' ')  # our validation split starts at index 800\n",
    "masked_text[mask_indices[0]] = '<mask>'\n",
    "masked_text = \" \".join(masked_text)\n",
    "\n",
    "pred1 = get_mask_pred(model, masked_text);pred1.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, pretty solid predictions!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train a Language Model from Scratch!\n",
    "We can follow the same procedure to train a language model from scratch by using `pretrained=False` when seeing up our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#collapse\n",
    "model = LMModel(lm_model_class=lm_model_class, tokenizer=tokenizer, model_name=model_name, \n",
    "                  config_dict=config_dict, pretrained=False)\n",
    "\n",
    "opt_func = partial(Adam, decouple_wd=True)\n",
    "\n",
    "loss = CrossEntropyLossFlat()\n",
    "\n",
    "learn = Learner(dls, model, opt_func=opt_func, loss_func=loss, metrics=[accuracy, Perplexity()]).to_fp16()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "SuggestedLRs(lr_min=0.09120108485221863, lr_steep=0.7585775852203369)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEKCAYAAAAVaT4rAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU5b3H8c9vskESlkDCIhBWQZEqlCggiri12mqx1Vp33EprbXvtbb3dbtvbxba3tYtLW0XFpVWv1WrdatVaFWVRwQ0UVMImi5CQsGRf5nf/mAlGTCCRnDkzme/79corM2fOzPNzHL7z5DnPeY65OyIikj4iYRcgIiKJpeAXEUkzCn4RkTSj4BcRSTMKfhGRNKPgFxFJM5lhF9ARhYWFPmLEiLDLEBFJKUuXLi1396I9t6dE8I8YMYIlS5aEXYaISEoxs3VtbddQj4hImlHwi4ikGQW/iEiaUfCLiKQZBb+ISJpR8IuIpBkFv4hIEtpV18hjyzazraq+y19bwS8ikoRWbN7FZXe+zLKNO7r8tRX8IiJJaNXWKgBGF+V3+Wsr+EVEklBpWRU9siIM6duzy187sOA3s2Fm9rSZrTCzN8zsP+Lb+5nZk2b2Tvx3QVA1iIikqtKyKkYV5hOJWJe/dpA9/ibgm+5+MDAVuNzMxgPfAZ5y9wOBp+L3RUSklVVbqxg9oOuHeSDA4Hf3ze7+cvz2LmAFMASYBdwe3+124LSgahARSUW1Dc1s3F7L6KK8QF4/IWP8ZjYCmAS8AAx0980Q+3IABrTznDlmtsTMlpSVlSWiTBGRpLCmvBp3GJNqPf4WZpYP/A24wt13dvR57j7X3UvcvaSo6EPLSYuIdFuryoKb0QMBB7+ZZREL/Tvd/f745i1mNjj++GBga5A1iIikmtKtVZjByMIUG+oxMwNuAVa4+29bPfQQMDt+ezbwYFA1iIikotKyKoYW9KRHVkYgrx/kFbimA+cDy8zs1fi27wG/BP5qZpcA64HPB1iDiEjKKS2rZkxAwzwQYPC7+/NAexNQjw+qXRGRVNYcdVaXVTF9dP/A2tCZuyIiSWTT9lrqm6KBzeEHBb+ISFJpmdET1FROUPCLiCSV0gAXZ2uh4BcRSSKlZVUU5GbRLy87sDYU/CIiSaR0a3WgvX1Q8IuIJJXSsqpAx/dBwS8ikjQqqxvYVt2gHr+ISLoobVmjZ0AwSzW0UPCLiCSJluAfU9Qr0HYU/CIiSWLV1iqyMyMMKej6yy22puAXEUkSpWXVjCrMIyOAyy22puAXEUkSm7bXMjTg3j4o+EVEkkZ5VT1FvXICb0fBLyKSBJqjTkV1A4X5Cn4RkbRQUd1A1FGPX0QkXZTtqgdQj19EJF2UV3WD4DezeWa21cyWt9o20cwWm9mrZrbEzI4Iqn0RkVTS0uNP9aGe24CT9tj2K+DH7j4R+GH8vohI2nu/xx/ccswtAgt+d58PVOy5Gegdv90H2BRU+yIiqaS8qp4eWRHycwK7FPpuwbfwQVcAj5vZ1cS+dI5sb0czmwPMASguLk5MdSIiISnbVU9hfg5mwZ61C4k/uHsZ8A13HwZ8A7ilvR3dfa67l7h7SVFRUcIKFBEJQ3lVQ0LG9yHxwT8buD9++15AB3dFRIgN9SRiRg8kPvg3AcfEbx8HvJPg9kVEklLLUE8iBDbGb2Z3AzOBQjPbAPwI+CJwjZllAnXEx/BFRNJZU3OUiprEDfUEFvzufnY7D00Oqk0RkVRUUd2AOxQlYCon6MxdEZHQlVUl7uQtUPCLiISuvKoBSMxyDaDgFxEJXSIXaAMFv4hI6Mo11CMikl7KdtXTMyuDvAQs1wAKfhGR0CXqkostFPwiIiGLnbWbmKmcoOAXEQld2S71+EVE0kp5VWIust5CwS8iEqLG5iiVNQp+EZG0sXu5Bg31iIikh0SfvAUKfhGRUCV6nR5Q8IuIhKo83uMvUo9fRCQ97F6grZfm8YuIpIWyXfXkZWeQm52Y5RpAwS8iEqryqnoKEzi+DwEGv5nNM7OtZrZ8j+1fM7O3zOwNM/tVUO2LiKSCsl31CR3fh2B7/LcBJ7XeYGbHArOAQ939EODqANsXEUl6sXV6uknwu/t8oGKPzZcBv3T3+vg+W4NqX0QkFSR6ZU5I/Bj/WOBoM3vBzJ41s8Pb29HM5pjZEjNbUlZWlsASRUQSI7ZcQ2P36fG3IxMoAKYCVwJ/NTNra0d3n+vuJe5eUlRUlMgaRUQSYlsIUzkh8cG/AbjfY14EokBhgmsQEUkKZSGcvAWJD/6/A8cBmNlYIBsoT3ANIiJJoeVau4mezhnYGQNmdjcwEyg0sw3Aj4B5wLz4FM8GYLa7e1A1iIgks4rq2FBP/7zEDvUEFvzufnY7D50XVJsiIqmkJfj7JTj4deauiEhIKmoayMow8nMSt1wDKPhFREJTUdVAv7xs2pncGBgFv4hISCpqGijITewwDyj4RURCU1ndkPDxfVDwi4iEpkLBLyKSXipqFPwiImmjqTnK9ppGBb+ISLrYXtsIJH4OPyj4RURC0XLylmb1iIikibCWawAFv4hIKCpbevwKfhGR9LAtpHV6QMEvIhKKSo3xi4ikl23VDfTKySQ7M/ExrOAXEQlBZU0D/fIT39sHBb+ISCgqqsNZoA0U/CIioQhrnR4IMPjNbJ6ZbY1fZnHPx75lZm5mutC6iKSlsFbmhGB7/LcBJ+250cyGAScC6wNsW0Qkabk727pj8Lv7fKCijYd+B/wXoIusi0haqm1spr4p2v2Cvy1m9hlgo7u/lsh2RUSSye6LrId0cDdhV/g1s1zg+8AnOrj/HGAOQHFxcYCViYgkVkWIyzVAYnv8o4GRwGtmthYYCrxsZoPa2tnd57p7ibuXFBUVJbBMEZFgVYS4XAMksMfv7suAAS334+Ff4u7liapBRCQZhB38QU7nvBtYBIwzsw1mdklQbYmIpJKwgz+wHr+7n72Px0cE1baISDKrrGkgI2L07pGwQZcP0Jm7IiIJ1rJcg5mF0n6Hgt/MRptZTvz2TDP7upn1DbY0EZHuqaK6IZQrb7XoaI//b0CzmY0BbiE2O+euwKoSEenGKqobKMjLCq39jgZ/1N2bgM8Cv3f3bwCDgytLRKT7ivX4c0Jrv6PB32hmZwOzgUfi28L7uhIRSWGVNY0p0eO/CJgGXOXua8xsJPCX4MoSEememqMeuwhLSMs1QAenc7r7m8DXAcysAOjl7r8MsjARke5oR20j7uHN4YeOz+p5xsx6m1k/4DXgVjP7bbCliYh0PxXV9UB46/RAx4d6+rj7TuBzwK3uPhk4IbiyRES6p4rqRiAFevxAppkNBs7k/YO7IiLSSS09/lQI/p8AjwOl7v6SmY0C3gmuLBGR7ikZevwdPbh7L3Bvq/urgdODKkpEpLuqrImvxR/irJ6OHtwdamYPxC+evsXM/mZmQ4MuTkSku9lW1UBedgY9sjJCq6GjQz23Ag8BBwBDgIfj20REpBMqaxpCndEDHQ/+Ine/1d2b4j+3AboslohIJ20LeYE26Hjwl5vZeWaWEf85D9gWZGEiIt1RZXXq9PgvJjaV8z1gM3AGsWUcRESkEyqqw12uAToY/O6+3t0/4+5F7j7A3U8jdjJXu8xsXvxg8PJW235tZivN7PX4wWKt6S8iaSWVxvjb8p/7ePw24KQ9tj0JTHD3Q4G3ge/uR/siIimlrrGZmobmUOfww/4F/16vGebu84GKPbY9EV/XH2AxoCmhIpI2kmEOP+xf8Pt+tn0x8Nh+voaISMqoqI4Ff78Q1+KHfZy5a2a7aDvgDej5URs1s+8DTcCde9lnDjAHoLi4+KM2JSKSNCrjyzWE3ePfa/C7e6+ubtDMZgOnAMe7e7t/Nbj7XGAuQElJyf7+dSEiErqWoZ6wx/g7tFZPVzGzk4BvA8e4e00i2xYRCdvuMf4UPri7V2Z2N7AIGGdmG8zsEuB6oBfwpJm9amY3BNW+iEiyaRnj79szicf494e7n93G5luCak9EJNlVVjfQp2cWmRmB9bk7JNzWRUTSSEVNY+jj+6DgFxFJmMrqBvrmhjvMAwp+EZGESYZ1ekDBLyKSMMmwTg8o+EVEEsLdYz3+JAj+hM7jD1tdYzOVNQ3srG1iR20jVfWN1DdGqW+K0tAUpTEapTnqNDU7jc1RahqaqWlooqah+f19mqNE3enTM4uC3Gz65maTkxkh6o47NLvT1BylIf4aUXcMwyx2unNzy35R3/2clvPYzIyIGRGDjAwjJzODHlmRD/zOyYyQkxkhKzNCTkbs96DePRjStyeRyF6XTxKRENU2xnIk7LN2oZsHf2V1Ay+treCltRW8uKaC5Zt20hzt3EnAPbMyyI1fHzMrw8jKiGAGO2obqaxppKEp2u5zszIMMwMHJxbykYiREQ/3iMW/EOK/o9HYPlF3GqO+19feU05mhFFF+Rw4IJ/JwwsoGVHAQYN6k6EvA5GkkCzr9EA3D/6f/2MF9y7dQHZGhMOG9WHOjFEMK8ild89M+vTMIj8nkx5ZsV50dmaE7IwIGREjMxIhM8PomZWx1160u1Pb2Exjk2MRdvfWszIiZEbiob8folGnoTlKXWMzDU2xv0zq4r2GhuYojfFtm7bXUlpWRWlZNS+uqeCh1zYB0CsnkwG9c4DYl0teTiZfKBnG6ZOHkJMZ3oWeRdJRsqzTA908+C+aPpLPlwzj0KF9ArmivZmRm50JAf1/jESMHpGMTtXu7mzcXsuStZW8tLaC7bWNu5fZW7utmu89sIzf/+ttLj16JOdMGU5+Trf+CIgkjWRZpwe6efCPP6B32CUknJkxtCCXoQW5nDZpyAcec3cWrNrGn55dxc//sZIbnl3N5ceO4dwpxYF8MYrI+5JlnR7QrJ60YmYcdWAhd146lQe+ciTjB/fmp4+8yXFXP8Nfl7zb6eMfItJxu8f4k2CoR8GfpiYVF/CXS6dw56VTKOqVw3/d9zqf++MClm3YEXZpIt1SZXUDEYPeIS/QBgr+tDd9TCF/v3w615w1kU076vjMH57nB39fzo6axrBLE+lWKmpiC7Qlw0w7Bb9gZsyaOISnvnkMs6eN4M4X1vGJ3z/LwtLysEsT6TYqqxuTYnwfFPzSSu8eWfzPZw7hwcuPIi8nk3NvfoFfP76SxuaOn08gIm1LlnV6QMEvbfjY0D488rWjOHPyMP7wdCmfv2ERGyp1wTSR/ZEs6/SAgl/akZudyf+ecSjXnzOJ0rIqTr3ueZ5/R0M/Ih9VWvT4zWyemW01s+WttvUzsyfN7J3474Kg2peuccqhB/DQV4+iqFcOF8x7gRufLd29tpCIdIy7p02P/zbgpD22fQd4yt0PBJ6K35ckN7Iwjwe+Mp2TJwzmF4+t5Kt3vUJVfVPYZYmkjKr6JhqbPSnW6YEAg9/d5wMVe2yeBdwev307cFpQ7UvXysvJ5PpzJvHdkw/iseWb+ewfFlBaVhV2WSIpIZnW6YHEj/EPdPfNAPHfAxLcvuwHM+NLx4zmL5dMYVt1A7OuX8Djb7wXdlkiSS+Z1umBJD64a2ZzzGyJmS0pKysLuxxp5cgxhTz8taMYXZTHl/68lOv//Y7G/UX2oiKJ1umBxAf/FjMbDBD/vbW9Hd19rruXuHtJUVFRwgqUjhnStyf3fGkap008gKufeJsfPviG1voRaUdlfJ2edB3qeQiYHb89G3gwwe1LF+qRlcFvz5zIl44ZxZ8Xr+OyvyylrrE57LJEkk4yLdAGwU7nvBtYBIwzsw1mdgnwS+BEM3sHODF+X1JYJGJ89+SD+dGp43lyxRbOv+UFdtZpnR+R1iprGsiIGL16JMdK+IFV4e5nt/PQ8UG1KeG5aPpIinrlcMX/vcr5N7/AHRdPoU9uckxdEwlbRXUjBblZSXNd7KQ9uCup55RDD+CG8yazYvMuzrl58e4/b0XSXWV1Q9KM74OCX7rYCeMHMveCyazaWsU5Ny2mbFd92CWJhK4iic7aBQW/BGDmuAHMu/Bw1m2r4YwbFrK2vDrskkRCVZlE6/SAgl8CMn1MIXd9cQq76po4/U8LefXd7WGXJBKaZFqnBxT8EqBJxQXc9+Vp5OZkcPbcxTy1YkvYJYkkXDTqVNY0Js06PaDgl4CNKsrn/sumM2ZAPpfesYSfPPwmtQ2a6y/pY1ddE81R18FdSS9FvXK450tTOW/KcOYtWMOnrn2Opev2XL9PpHtqWadHwS9pJzc7k5+eNoE7L51CQ1OUM25YxE3zV4ddlkjgKpJsgTZQ8EuCTR9TyOPfmMHJEwZx1T9WcOuCNWGXJBKo3ev0JFHwJ8f5w5JW8nMyueasSTRHX+bHD79Jj6wMzj6iOOyyRALRct2KwvzkCX71+CUUWRkRrj17EjPHFfG9B5Zx39INWtpZup3mqPPnxeuYPLyAoQW5YZezm4JfQpOTmcEN501m2qj+fOve15h59TP89JE3Wbx6m5Z4lm7h8Tfe492KWr549MiwS/kABb+EqkdWBvMuPJyfnTaBkYV5/HnROs6au5jZ817UEs+S0tydufNXM7x/LieOHxR2OR+g4JfQ9cjK4Lypw7ntoiN4+Ycn8qNTx/P8qnK+etcrNDZHwy5P5CNZuq6SV9/dziVHjSQjSVblbKHgl6SSn5PJRdNH8tNZh/CvFVv41r2vadhHUtJNz62mT88szpg8NOxSPkSzeiQpnT9tBFX1zfzvP1fSMyuDn8yaQHam+imSGtaUV/PEm1u4fOYYcrOTL2aTryKRuMtmjqaqvpE/PF3K/LfLuGzmaM48fBg5mRlhlyayV/OeX0NWJMIFRw4Pu5Q2hdKFMrNvmNkbZrbczO42sx5h1CHJ71ufGMcdFx/B4L49+cGDb3DMr57h769sDLsskb16+PVNfOpjgxjQKzmjLeHBb2ZDgK8DJe4+AcgAzkp0HZIazIwZY4u478vTuPPSKQzu24Mr7nmVXzy2gqjG/iUJba9pYHtNIxOG9Am7lHaFNdSTCfQ0s0YgF9gUUh2SIsyM6WMKOWJkP/7noTe48dnVrCuv4XdfmEjPbA39SPJYX1EDwLB+yXPC1p4S3uN3943A1cB6YDOww92fSHQdkpqyMiL87LQJ/OCU8Tz+5nuceeMiXeRFkkpL8Bcr+N9nZgXALGAkcACQZ2bntbHfHDNbYmZLysrKEl2mJDEz45KjRnLzBSWsr6jhtD8s4JybFvPcO2Va9kFCt26bgr8tJwBr3L3M3RuB+4Ej99zJ3ee6e4m7lxQVFSW8SEl+xx88kAXfOY7vf+pgSsuqOP+WFznzxkWsji+KJRKGdytqKMzPJi8neSdNhhH864GpZpZrZgYcD6wIoQ7pBvJzMvnijFHM/69jueqzE3h7SxUnX/McNz+3Wid+SSjWV9Qk9fg+hDPG/wJwH/AysCxew9xE1yHdS05mBudOGc6T35jB0QcW8rNHV/CFGxextry6zf3dXcNCEoj1FTUMV/B/mLv/yN0PcvcJ7n6+u9eHUYd0PwN69+CmC0r4zecP460tuzj5mue484V1u0N+e00DP3n4Tcb+92OcfdNilq6rDLli6U4amqJs2l6b1OP7oDN3pRsyM06fPJQjx/Tnyntf5/sPLOfJN7cwdVR//vj0Kqrqmzh5wmBeWLON0/+0kBMOHsCVnzyIcYN6hV26pLhN22uJenJP5QQFv3Rjg/v05I6Lj+CORWv5xWMreeatMo4ZW8T3PnUw4wb1oqahiVsXrOWGZ0s55brn+MmsCR+6Elhjc5TK6gYG9E7OMzAluaTCVE5Q8Es3F4kYF04fycxxA9i6q54jRvbb/VhudiaXHzuGc44o5j/ueZXv3r+MNzbt4IenHELE4P5XNnLtU+/w3o46vn3SQVx69Ehi8xFE2tYS/MP754Vcyd4p+CUtjCjMY0Rh2/8YC/KyufXCw/nV4yu58dnVLN+4kx21jawpr+ZjQ/owbmAvrvrHCl5Ys42rP38YfXOT59qpklzWV9SQnRlhQK+csEvZKwW/CJARMb578sGMH9ybb//tdUb0z2Pu+ZM5cfxAAO5YtI6fPfomn772ea4/ZxKTigtCrliS0fptNQwr6EkkyS68sicFv0grsyYO4fiDB5KblfGBf7yzjxzBpOK+XH7Xy5x54yJ+dOohnDulWEM/8gHrK2qSfpgHdAUukQ/Jz8lss8d26NC+PPzVo5g+ppD//vtyrrzvdV0XWHZzd96tqEn6A7ug4BfplL652cybfThfP/5A7lu6gTNuWMi78QN6kt4qaxrZVd+U9FM5QcEv0mmRiPGfJ47l5gtKWLethlOvf55n39ZCgukuVaZygoJf5CM7YfxAHv7qUQzq3YMLb32Ra596RxeHSWPvT+VU8It0ayMK83jgK9OZddgB/PbJt5l964ts2l77of3KdtWzo7YxhAolUdZvi60LNaxAwS/S7fXMzuB3X5jIVZ+dwJK1lXzy9/O5d8m7uDsr39vJFf/3ClN/8RRH/++/eeR1XWyuu1pfUUNRr5yUuCKcpnOKdAEz49wpwzlqTCFX3vs6V973On96tpTVZdXkZmdwwbThvLJ+O1+96xWeWrGVH886hN49ssIuW7pQKqzK2ULBL9KFhvfP4+45U7l1wRoeeGUj3zxxLOdPG07f3GyamqNc//Qqrvv3Kl5cU8GvzziUI8cUhl2ydJF3K2qZ0mpJkGSm4BfpYhkR49KjR3Hp0aM+sD0zI8IVJ4zlmLFFfPOvr3HOzS9wwbThfPukg5L6ak2yb/VNzWzaUZsSUzlBY/wiCTepuIBHv340lxw1kj8vXsdJ18xn4arysMuS/bCxshb31JjKCQp+kVD0zM7gB6eM554504iYcc7NLzDnjiW6XnCKSqWpnBBS8JtZXzO7z8xWmtkKM5sWRh0iYTtiZD8ev2IGV35yHAtWlfOJ383nfx56gx01mvqZKlZs3sltC9cCqdPjD2tg8Rrgn+5+hpllA6nxbokEoEdWBpcfO4YzS4bxu3+9zR2L1vLI65v54anjOfXQwVoILkktLC3nD0+vYsGqbfTMyuBrx41JmQv2WKIvOG1mvYHXgFHewcZLSkp8yZIlwRYmkiSWb9zB9x5YxusbdnDM2CJ+OmsCxSkyhJAu3ti0g1Ove56iXjnMPnIE5xxRnJTXaTCzpe5e8qHtIQT/RGAu8CZwGLAU+A93r27vOQp+STfNUeeORWu5+vG3aIw6F08fyVeOHb177v/G7bXcvnAtZbvq+frxBzKynYvMSNeLRp0zbljIum01/PubM+mTm7znYyRT8JcAi4Hp7v6CmV0D7HT3H+yx3xxgDkBxcfHkdevWJbROkWSweUctv/7nW9z/ykb652UzZ8Yolm3cwWPL3wOgR2aExmbnyzNH85WZo+mRlfxnjaa6e15az7f/toyrP38YZ0weGnY5e5VMwT8IWOzuI+L3jwa+4+6fbu856vFLunt9w3Z+9ugKXlxTQa+cTM6eUszsI0eQlWFc9egKHnx1EyP65/L9T4/nhIMH6LhAQCqqGzjuN88wdkAv7vnS1KR/n9sL/oQf3HX398zsXTMb5+5vAccTG/YRkXYcOrQv98yZyorNuyjun0t+qxO+rjlrEp+fPIwfPrScL96xhI8X9+XbJx3ElFH9A6vn3YoaFpaWk50Z4eQJg9PmL41f/XMlu+qa+OlpE5I+9Pcm4T1+2D3OfzOQDawGLnL3yvb2V49fZN8am6Pct3QDv//X22zZWc+MsUV87bgxHD7ig8sIbN1Vx+qyag4f0Y+MDl4btryqnkWl21hYWs6CVdt2z1sH6J+XzblTh3Pe1GIG9ApnVou7U1nTSL+84A6wLl1Xwel/WsScGaP43qcODqydrpQ0Qz0fhYJfpOPqGpu5beFa5s5fTUV1A4ePKODLx4xmV10T97+ykeffKSPqMHZgPld+8qA2h4bqm5p5aU0lz7y1ledXlbPyvV0A9MrJZOro/kwf3Z/pYwopq6pn3vNreGrlVjIjxicOGcTZhxdz5Oj++33B8YamKFt21rF1Vx3v7ahny846quqbGNK3JyMKcxnWL5fSrdU8/sZ7PPHGe2zaUcfEYX0554hiTjlsMLnZXTegsaGyhs/9cSFZGRGe+MaMlFliQ8EvkmZqGpq456V3uWn+ajbtqANgSN+enDbpAEb0z+OPz5Sypryajxf3ZcbYIqrrm6iqb2bLzjoWr95GTUMz2RkRDh9ZwJGjC5k+ppAJB/QmM+PD532uLa/m9kVruf/ljeyobaS4Xy4zxsYWoGtqdtzh+IMHcMLBA/f6hfDo65v507Or2Ly9jm3VDR3678zJjHD0gUUcckBvHl22mVVbq+iVk8mMsUWMGZDPgQPzGVWYT0FeFnk5meRlZ3b4Lx2A7TUNnHHDIrbsrONvlx3J2IG9OvzcsCn4RdJUY3OUp1ZsoSA3m8NH9NsdvC1DQ9c+9Q6bd9TRMyuDvJwM+vTMYuqo/hw7bgDTRvfvVO+2rrGZx994j7tfXM+bm3aSmREhM2LUN0XZUdvI2IH5fPmY0Zx62AFk7fEFcvNzq/nZoys4aFAvJhUXMKh3Dwb1yWFg7x4M7N2DQb17kJuTwcbKWtZV1LB+Ww0DeuVwzLii3b17d2fJukrufnE9S9ZW8m5lDW1F3LB+PTl23ACOPWgA00b1xz02g2rT9joiBocO60t+TiZ1jc1ccMuLvPrudu645AimBnjcJAgKfhFpUzTqOHSqF9xZTc1RHl22mT8+XcpbW3YxuE8PPvfxIZz+8aGM6J/Hzx5dwbwFa/j0xwbzmzMP67KDxXWNzZSWVbG2vIaddY1U1zexs66JNzftZMGqcmobm8mMGE17XDIzYnDw4N5kZ0Z4Zf12rjt7EqcedkCX1JRICn4RCV006vx75Vb+vHgdz8WPNQzp25ON22u5ePpI/vvTB+/3sYGOqmtsZvHqbbywpoL8nEwO6NuDwX16Ut8UZem6SpasreDtLVVcfuxoLpo+MiE1dTUFv4gklS0763jw1Y38a8VWTp4wKGXDNZkp+EVE0kx7wa/1+EVE0oyCX0QkzSj4RUTSjIJfRCTNKPhFRNKMgl9EJM0o+EVE0oyCX0QkzaTECVxmVgasA/oAO+Kb93W75XchUN7JJs/Y0VIAAAdJSURBVFu/Xmce33P73u7vWef+1PtRa96feltvS9R7vK9tqfaZSLV691VnkJ/hIOptva27foaHu3vRh7a6e8r8AHM7ervV7yX7005nHt9z+97ut1HnR673o9a8P/WG8R7va1uqfSZSrd4O1BnYZziIesN4j8P8DLf+SbWhnoc7cbv1tv1ppzOP77l9b/f3rHN/6u3I89t6fH/q7Uibna1nX4/va1uqfSZSrd497yfyMxxEvftqc19S7TO8W0oM9ewPM1vibaxVkaxSrV5IvZpVb7BSrV5IvZr3t95U6/F/FHPDLqCTUq1eSL2aVW+wUq1eSL2a96vebt/jFxGRD0qHHr+IiLSi4BcRSTMKfhGRNJPWwW9mR5vZDWZ2s5ktDLuefTGziJldZWbXmdnssOvZFzObaWbPxd/jmWHX01FmlmdmS83slLBr2RczOzj+/t5nZpeFXc++mNlpZnaTmT1oZp8Iu559MbNRZnaLmd0Xdi3tiX9eb4+/r+d25DkpG/xmNs/MtprZ8j22n2Rmb5nZKjP7zt5ew92fc/cvA48Atyd7vcAsYAjQCGwIqtZ4XV1RrwNVQA8CrjdeW1fUDPBt4K/BVPmBurriM7wi/hk+Ewh0OmIX1ft3d/8icCHwhQDL7ap6V7v7JUHW2ZZO1v454L74+/qZDjWwP2d/hfkDzAA+DixvtS0DKAVGAdnAa8B44GPEwr31z4BWz/sr0DvZ6wW+A3wp/tz7UqDeSPx5A4E7U+EzAZwAnEUsmE5J9nrjz/kMsBA4JxXqjT/vN8DHU6jeQP+97Wft3wUmxve5qyOvn0mKcvf5ZjZij81HAKvcfTWAmf0fMMvdfwG0+We7mRUDO9x9Z4Dldkm9ZrYBaIjfbQ6u2q57f+MqgZwg6myti97jY4E8Yv+gas3sH+4eTdZ646/zEPCQmT0K3BVErV1Vr5kZ8EvgMXd/Oahau6resHSmdmJ/TQ8FXqWDozgpG/ztGAK82+r+BmDKPp5zCXBrYBXtXWfrvR+4zsyOBuYHWVg7OlWvmX0O+CTQF7g+2NLa1ama3f37AGZ2IVAeVOjvRWff45nE/tTPAf4RaGVt6+xn+GvE/qrqY2Zj3P2GIItrQ2ff3/7AVcAkM/tu/AsiLO3Vfi1wvZl9mg4u6dDdgt/a2LbXM9Tc/UcB1dIRnarX3WuIfVGFpbP13k/syypMnf5MALj7bV1fSod09j1+BngmqGI6oLP1XkssqMLS2Xq3AV8OrpxOabN2d68GLurMC6Xswd12bACGtbo/FNgUUi0doXqDl2o1q95gpVq9rXVZ7d0t+F8CDjSzkWaWTewg3UMh17Q3qjd4qVaz6g1WqtXbWtfVnsgj1V181PtuYDPvT228JL79U8DbxI5+fz/sOlWvala9qjfZatcibSIiaaa7DfWIiMg+KPhFRNKMgl9EJM0o+EVE0oyCX0QkzSj4RUTSjIJfUpKZVSW4vZvNbHwXvVazmb1qZsvN7GEz67uP/fua2Ve6om0R0MXWJUWZWZW753fh62W6e1NXvd4+2tpdu5ndDrzt7lftZf8RwCPuPiER9Un3px6/dBtmVmRmfzOzl+I/0+PbjzCzhWb2Svz3uPj2C83sXjN7GHjCYlcMe8ZiV7NaaWZ3xpcRJr69JH67ymJXQnvNzBab2cD49tHx+y+Z2U86+FfJImKrLmJm+Wb2lJm9bGbLzGxWfJ9fAqPjfyX8Or7vlfF2XjezH3fh2yhpQMEv3ck1wO/c/XDgdODm+PaVwAx3nwT8EPh5q+dMA2a7+3Hx+5OAK4itxz8KmN5GO3nAYnc/jNjy2F9s1f418fb3uXiWmWUAx/P+eit1wGfd/ePAscBv4l883wFK3X2iu19psUsWHkhsffaJwGQzm7Gv9kRadLdlmSW9nQCMj3fSAXqbWS+gD3C7mR1IbAnerFbPedLdK1rdf9HdNwCY2avACOD5PdppIHaFJoClwInx29OA0+K37wKubqfOnq1eeynwZHy7AT+Ph3iU2F8CA9t4/ifiP6/E7+cT+yII4xoNkoIU/NKdRIBp7l7beqOZXQc87e6fjY+XP9Pq4eo9XqO+1e1m2v430ujvHxxrb5+9qXX3iWbWh9gXyOXE1qg/FygCJrt7o5mtJXa94j0Z8At3v7GT7YoAGuqR7uUJ4Kstd8xsYvxmH2Bj/PaFAba/mNgQE8SWzN0rd98BfB34lpllEatzazz0jwWGx3fdBfRq9dTHgYvNrOUA8RAzG9BF/w2SBhT8kqpyzWxDq5//JBaiJfEDnm/y/pWTfgX8wswWELtgdVCuAP7TzF4EBgM79vUEd3+F2EWzzwLuJFb/EmK9/5XxfbYBC+LTP3/t7k8QG0paZGbLgPv44BeDyF5pOqdIFzGzXGLDOG5mZwFnu/usfT1PJNE0xi/SdSYTu+i1AduBi0OuR6RN6vGLiKQZjfGLiKQZBb+ISJpR8IuIpBkFv4hImlHwi4ikGQW/iEia+X/O2cBFoZnX1AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#hide\n",
    "learn.lr_find(suggestions=True, stop_div=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.LineCollection at 0x7f8dc8439550>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEKCAYAAAAVaT4rAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU5d338c9vspIAIYGwyCq7SlEkKi4o7tRal7pUrVarlWpbtVqrtr0frfdTW+9qF5e2lrrg3ap1rVu1yqNSXHAJKAqKsiiILAkJhOzr7/ljJhgxQAI5c2Yy3/frlVdmzpyZ6+c4fOfkOte5LnN3REQkdUTCLkBEROJLwS8ikmIU/CIiKUbBLyKSYhT8IiIpRsEvIpJi0sMuoCP69evnI0aMCLsMEZGkMn/+/A3uXrj19qQI/hEjRlBcXBx2GSIiScXMVra3XV09IiIpRsEvIpJiFPwiIilGwS8ikmIU/CIiKUbBLyKSYhT8IiIJqKq+iWffW0tZVX2Xv7aCX0QkAb2/ZjMX37eA9z6r6PLXVvCLiCSgpSWVAIzu37PLX1vBLyKSgJaVVNEjI43d8np0+WsHFvxmNtTMXjKzD8xssZldFtteYGazzWxp7Hd+UDWIiCSrZSVVjOqfSyRiXf7aQR7xNwE/dvc9gCnAD8xsT+Aa4AV3HwO8ELsvIiJtLC+pYnRh13fzQIDB7+5r3X1B7HYl8AEwGDgRuDe2273ASUHVICKSjKrrm1hTURdI/z7EqY/fzEYAk4A3gAHuvhaiXw5A/208Z4aZFZtZcWlpaTzKFBFJCMtLq4BgTuxCHILfzHoCjwI/cvfNHX2eu8909yJ3Lyos/NJ00iIi3daykiQOfjPLIBr697n7Y7HN681sUOzxQUBJkDWIiCSbZSVVpEeM4X1zA3n9IEf1GHAX8IG7/67NQ08C58Zunws8EVQNIiLJaFlJFcP75pCRFkxEB7kC18HAOcB7ZvZObNvPgBuBh8zsAmAVcFqANYiIJJ1lpVWMCaibBwIMfnd/BdjWANQjg2pXRCSZNTS1sLKshq9OGBhYG7pyV0Qkgawsq6a5xQM7sQsKfhGRhLJlRE9hr8DaUPCLiCSQ1uAf1T+YET2g4BcRSSjLSqsY3KcHOZnBjb1R8IuIJJDo5GzB9e+Dgl9EJGG0tDjLS4ObnK2Vgl9EJEF8tqmWusaWQEf0gIJfRCRhLAt4crZWCn4RkQSxPODJ2Vop+EVEEsSykioKcjMpyM0MtB0Fv4hIglixoZpRhcGN32+l4BcRSRCllfUMDGBx9a0p+EVEEkRpZT39egbbzQMKfhGRhFDT0ERVfROFvbICb0vBLyKSADZUNgBQ2FPBLyKSEkqr6gCS+4jfzO42sxIzW9Rm2z5m9rqZvWNmxWa2f1Dti4gkk9LWI/5kDn5gFjB9q22/Aa53932Aa2P3RURSXmlVPZDkwe/uc4HyrTcDvWO384A1QbUvIpJMSivriRj0zQ0++INcbL09PwKeM7ObiX7pHBTn9kVEElJpZT0FuZmkRba1VHnXiffJ3YuBy919KHA5cNe2djSzGbHzAMWlpaVxK1BEJAzRMfzBH+1D/IP/XOCx2O2HgW2e3HX3me5e5O5FhYWFcSlORCQspVX1cenfh/gH/xrgsNjtI4ClcW5fRCQhbaiMX/AH1sdvZg8A04B+ZrYauA64ELjFzNKBOmBGUO2LiCQLd4/rEX9gwe/uZ27joclBtSkikow21zXR0NQSl6t2QVfuioiErrQyfmP4QcEvIhK6LcGvI34RkdQQz6t2QcEvIhI6dfWIiKSYDVX1ZKQZeT0y4tKegl9EJGSllfUU9szCLPjpGkDBLyISutI4XrwFCn4RkdAp+EVEUkxpVfwmaAMFv4hIqJpbnLI4TtcACn4RkVCVVzfQ4vEbygkKfhGRUG2oiu9Vu6DgFxEJVbwv3gIFv4hIqBT8IiIppnWeHo3qERFJEaWV9eRkppGbFdjyKF+i4BcRCVG8L96CAIPfzO42sxIzW7TV9kvM7EMzW2xmvwmqfRGRZNA6T088BXnEPwuY3naDmR0OnAhMdPe9gJsDbF9EJOFtiPPFWxBg8Lv7XKB8q80XAze6e31sn5Kg2hcRSQbxXGS9Vbz7+McCU83sDTP7j5ntt60dzWyGmRWbWXFpaWkcSxQRiY/6pmY21TR2q66e9qQD+cAU4CfAQ7aNCajdfaa7F7l7UWFhYTxrFBGJi7KqBiC+Y/gh/sG/GnjMo94EWoB+ca5BRCQhtF68Fc8x/BD/4H8cOALAzMYCmcCGONcgIpIQWufp6RfnI/7ArhgwsweAaUA/M1sNXAfcDdwdG+LZAJzr7h5UDSIiiay1q6dvbmZc2w0s+N39zG08dHZQbYqIJJPymljw94xv8OvKXRGRkJRXN5CdESEnM37TNYCCX0QkNGVVDRTkxPdoHxT8IiKhKa+upyDO3Tyg4BcRCU15dQMFufEd0QMKfhGR0JTXNMR9RA8o+EVEQlNe1UCBgl9EJDXUNTZT3dCs4BcRSRXl1dEx/Ap+EZEUoeAXEUkxrcGvk7siIilCR/wiIimmTMEvIpJayqvrSYsYvbMz4t62gl9EJATl1Y3k52QSibS7CGGgFPwiIiEor64P5cQuKPhFREIRnadHwS8ikjLKumPwm9ndZlYSW2Zx68euNDM3My20LiIpqbse8c8Cpm+90cyGAkcDqwJsW0QkYTU1t1BR29j9gt/d5wLl7Tz0e+AqQIusi0hK2lTbiHv819ptFdc+fjM7AfjM3Rd2YN8ZZlZsZsWlpaVxqE5EJD7CvGoX4hj8ZpYD/By4tiP7u/tMdy9y96LCwsJgixMRiaOyqljwh7DeLsT3iH8UsDuw0Mw+AYYAC8xsYBxrEBEJ3ZYj/pC6etLj1ZC7vwf0b70fC/8id98QrxpERBJBeU037eoxsweAecA4M1ttZhcE1ZaISDIpj3X15IfU1RPYEb+7n7mDx0cE1baISCIrr64nr0cGGWnhXEOrK3dFROIszKt2QcEvIhJ3G2uSIPjNbJSZZcVuTzOzS82sT7CliYh0T2VVSRD8wKNAs5mNBu4iOizz/sCqEhHpxsqrG0Kbkhk6Hvwt7t4EnAz8wd0vBwYFV5aISPfk7snR1QM0mtmZwLnA07Ft8V8vTEQkyW2ua6Kx2ZMi+L8DHAjc4O4fm9nuwN+DK0tEpHvaGPI8PdDBcfzu/j5wKYCZ5QO93P3GIAsTEemOyhIg+Ds6qmeOmfU2swJgIXCPmf0u2NJERLqf1nl6+uZmhVZDR7t68tx9M/AN4B53nwwcFVxZIiLdU3l1PRDeBG3Q8eBPN7NBwOl8fnJXREQ6aUtXT0jz9EDHg/+/geeA5e7+lpmNBJYGV5aISPe0sbqBHhlp9MhMC62Gjp7cfRh4uM39FcApQRUlItJdhT1PD3T85O4QM/unmZWY2Xoze9TMhgRdnIhId1Ne3RDaWrutOtrVcw/wJLAbMBh4KrZNREQ6oTxZjviBQne/x92bYj+zAC2EKyLSSeXVDaGe2IWOB/8GMzvbzNJiP2cDZdt7gpndHesaWtRm201mtsTM3o11HWmGTxFJKRuT6Ij/fKJDOdcBa4FTiU7jsD2zgOlbbZsNTHD3icBHwE87XKmISJKra2ymuqGZ/GQIfndf5e4nuHuhu/d395OIXsy1vefMBcq32vZ8bJZPgNcBnSAWkZSxMeRF1lvtygpcV+xi2+cDz+7ia4iIJI3yBJinB3Yt+G2nn2j2c6AJuG87+8wws2IzKy4tLd3ZpkREEkZ3CH7fmSeZ2bnA8cC33H2br+HuM929yN2LCgs1gEhEkl9r8OeHPKpnu1fumlkl7Qe8AT0625iZTQeuBg5z95rOPl9EJJklwlz8sIPgd/deO/vCZvYAMA3oZ2argeuIjuLJAmabGcDr7n7RzrYhIpJMymsaiRjk9Qh3AcMOzdWzM9z9zHY23xVUeyIiia68up4+OZmkRXb6FGmX2JU+fhER6YSN1Y3k54S/XLmCX0QkThJhnh5Q8IuIxI2CX0QkxZTXKPhFRFKGu7OxuiH0MfwQ4KieRFNV38S7n26itKqeitpGNtc2UlnfRENTC/VNLdQ3ttDc0kJTi9Pc4jQ2t1DTEJ1QqbahifqmFhqbWmhodtydvB4Z5OVkkJ+TSWZahBZ3Wjz6P7exxWlsaqGxuYUWd8wMA8yguSW6X3R/p6Xl8wslDEiLGBGL/s5KTyM7I0JWehpZGRGy0mO30yNkpEfISItuG9A7m5GFuYzq15O8BDhxJCJfVlnfRFOLJ8QRf7cO/teWbeD599dTvLKc99dspmWrS9Ey0yJbAjUzLRqmaREjPWKkRyLkZKbROzudgb2zyM5IIyMtGrZmUFHbSEVNI+s319HY3ELEDLNoaGekRV8vMz1CxAzHcQd3yEo3IrFwj8T2B8Ms+qXR+qXQ1OzUNzWzoaqJusZmGpqjX071Tc3RL6HmFhqbv3xtXWGvLPYd1oei4QVMHpHPoLxsYi3QIzMt9PHDIqmqvCoxLt6Cbh78Lywp4cG3PmXSsD788IgxTB6ez5D8HvTOzqB3j3Sy0sNb7LgruDv1TS2s2VTLitJqVmyoYsnaSuav2shzi9d/aX8zmL7XQC46bBR7D9VSCNL1pk2bBsCcOXNCrWN7wqqxPDYzZ9hTMkM3D/7LjhrDNV8dT0Za9zyVYWZkZ6QxsrAnIwt7AgO2PFZSWceClRvZWNMIRP/aWFlezQNvrOLZReuYMrKAHx4+hoNH9yV2FbWIBGjLdA3q4w9W7+zU7dbo3yub6RMGfWn7JUeM4R9vruLOlz/m7Lve4IDdC/jJseMoGlEQQpUiqaMsQebpAY3qSTk9s9L57tSR/OeqaVx/wl4sL63m1Dvmcd49b7KspDLs8kS6rUSZoA0U/CkrKz2Ncw8awdyrpnH19PEsWLmR6X94mV898wFV9U07fgER6ZTymgYy06ODRsKm4E9xOZnpXDxtFC9dOY1v7DuYmXNXcORv5/DMe2vDLk2kW9lY3UBBTmZCnFNT8AsAfXtm8ZtT9+af3z+Iwl5ZfP++BVzx4DtU1jWGXZpIt5Ao0zWAgl+2MmlYPo9//2AuO3IMj7/zGcfd+jLzV5aHXZZI0lPwS0JLT4tw+dFjefiiAwE47Y55/PGlZbRsfQWciHTYxprGhBjDDwp+2Y7Jwwt45tKpHPeVQdz03IfM+Nt8KmrV9SOyM8qq6unb3YPfzO42sxIzW9RmW4GZzTazpbHf+UG1L12jV3YGt505ieu+vidzPizhhNtf4YO1m8MuSySpNDa3sLmuKSEmaINgj/hnAdO32nYN8IK7jwFeiN2XBGdmfOfg3fnHjCnUNTZz8p9e5bEFq8MuSyRpbIpdQV+QmxgXlQYW/O4+F9j6rOCJwL2x2/cCJwXVvnS9ohEFPH3JVPYZ2ocrHlrIz//5HvVNzWGXJZLwyqsTZ54eiH8f/wB3XwsQ+91/Wzua2QwzKzaz4tLS0rgVKNtX2CuLv19wABcdNor73ljFaXfMY82m2rDLEklo5Ql01S4k8Mldd5/p7kXuXlRYWBh2OdJGelqEa746npnnTObj0mpO/tOrvL9G/f4i27KxJrWDf72ZDQKI/S6Jc/vShY7ZayAPX3wgETNO/8s85n6kv8xE2lOeQDNzQvyD/0ng3Njtc4En4ty+dLHxA3vzz+8fzNCCHL4z6y0eKv407JJEEk5r8Pfp7sFvZg8A84BxZrbazC4AbgSONrOlwNGx+5LkBuZl89D3pnDQqL5c9ci73PnyirBLEkko5dUN9MpOJzM9MXrXA5uP393P3MZDRwbVpoSnV3YGd527Hz968G1++a8PqG1o5odHjE6ICalEwraxJnGma4BuvhCLxFdmeoRbz5hEdvq7/Hb2R9Q0NnPVseMU/pLyyqsbEubiLVDwSxdLT4tw82l70yMzjT/PWc6mmkb+74l7kd5Nl78U6Yjy6gYG9s4Ou4wtFPzS5SIR45cnTSCvRwZ/mrOcdRW13H7WvuRm6eMmqWljdQN7DOoddhlb6DBMAmFmXDV9PDecPIH/fFTKN2fOo2RzXdhliYSiPMH6+BX8EqhvHTCcu87djxWl1Zxw+6sa6y8pp6ahibrGloTq41fwS+AOH9+fh753ILlZaXz77je56pGFmt5ZUkbrGP5EmZIZFPwSJxMG5/GvS6dy8bRRPDJ/Ncf8/j+8vqIs7LJEArexOnqQkygTtIGCX+IoOyONq6eP5/EfHExuVjrn3fOmwl+6vbLqeiBxpmQGBb+EYOKQPjw440AG9+nB+bPeovgTrekr3de6iuigBvXxS8or7JXFAxdOYWDvbM675y0WrNoYdkkiXc7deeDNVQwryGF439ywy9lCwS+h6d87m/svnELfnpmcOfN1vve3Yh4q/pQNVfVhlybSJd74uJyFqyu48NCRpEUS5wp2XVEjoRqYl82DMw7ktheX8sIHJTy3eD1mcM6U4fzi63sRSaB/LCKdNXPuCvrmZnLa5CFhl/IFCn4J3cC8bG44+Sv88iRn8ZrN3P/mKv533koM+MUJe2muH0lKH62v5MUlJVxx9FiyM9LCLucLFPySMMyMCYPzuOGkCeRmpvHXlz8mOyONa746XuEvSWfm3BX0yEjjnCnDwy7lSxT8knDMjJ8dtwd1jS38Ze4KstIjXH70WIW/JI11FXU88c5nfOuA4Qk1fr+Vgl8Skplx/Ql7Ud/UzK0vLmP2ByX88PDRTJ8wMKFOkom0555XP6bF4YJDdg+7lHaFMqrHzC43s8VmtsjMHjCzxJmvVBJGJGLc+I2J/Pa0valvauYH9y/gmN//hxc+WB92aSLb9eiC1Ry71wCGFuSEXUq74h78ZjYYuBQocvcJQBpwRrzrkOQQiRinTB7C7MsP4/azJhEx44J7i7n9xaW4e9jliXxJRW0jG6oa2Gdon7BL2aawxvGnAz3MLB3IAdaEVIckibSIcfzE3XjqkkM4aZ/duPn5j7jsH+9Q19gcdmkiX7CqrAaAYQWJc8HW1uIe/O7+GXAzsApYC1S4+/PxrkOSU3ZGGr//5j5cNX0cT727hm/OfJ2Fn24KuyyRLVaWVwMwvG9idvNAOF09+cCJwO7AbkCumZ3dzn4zzKzYzIpLSzWHu3zOzPj+tNH85ezJfFxaxYl/fJUzZs5jzocl6v6R0K3ccsSv4G/rKOBjdy9190bgMeCgrXdy95nuXuTuRYWFhXEvUhLfMXsN5LWfHsl/fW0PPtlQw3n3vMXJf3qND9dVhl2apLBVZTX065mV0EuNhhH8q4ApZpZj0YHZRwIfhFCHdAM9s9L57tSRzL3qcH5zykRWlddw/G0vc9sLS2lsbgm7PElBK8urE7qbB8Lp438DeARYALwXq2FmvOuQ7iUzPcLp+w1l9uWHMn3CIH47+yNOvP1V3l+zud39N9c10tKibiHpeqvKahiewN08ENKoHne/zt3Hu/sEdz/H3TUdo3SJvj2zuO3MSfzlnMmUVtVz4h9f4fYXl9IUO/pfvbGGyx98h4m/eJ7jbn2Z2e+v13kB6TL1Tc2s3VzHsAQ/4k/cTiiRXXDsXgPZf0QB1z65mJuf/4jZ769n3+H53Pf6Kszg7CnDeGXpBi7832L2HpLH1dPHc9DofmGXLUlu9cZa3BP7xC4o+KUby8/N5LYzJzF9r4H81+Pv8e5nFZy67xCuOGYsg/J60NTcwmMLPuOWF5Zy1p1vcMkRo/nRUWO/MCXExuoGVpXXsHcCX4wjiaN1DH+i9/Er+KXb+9rEQRw0qi8VtY2M6Pf5RTXpadHzAifssxvXPrGI215cxrurK7j1jEk4zl9fXsGsVz+huqGZsw4YxrXH75lw0+tKYllZFh3Dn8gXb4GCX1JEfm7mNmdJzM5I439OmcjEIX24/qnFfPWWuWyua6K6oYnjvjKI/r2yuOfVT1iwciO3nzWJ0f17xbl6SRYry2vIyUyjX8/Em5GzLQW/CNGLws6eMpw9BvXmyocXcuiwPlx25FjGDYyG/GFjC/nxQwv5+m2v8suTJnBKgq2oJIlhVVkNwwpyEn4KcQW/SBuTh+fz0pXTvrR92rj+PHPZVC77x9v8+OGFFK/cyHVfV9ePfNHK8hpGFSZ2Nw9osXWRDhvQO5u/X3AAF08bxQNvruK0O+bxaXlN2GVJgmhpcVaV1zC8r4JfpFtJT4tw9fTx/PXbRXxSVs3xt73Ci0u0PoDA+so6GppaEn4oJyj4RXbK0XsO4OlLDmFIfg/On1XMTc8toVlXAqe0lUkylBMU/CI7bXjfXB69+CDO2G8of3xpOefc9QallboIPVWtSoJZOVsp+EV2QXZGGjeeMpGbTp3I/JUbOfYPc3lq4ZovTANR19jMYwtWM+fDkhArlaCtLK8mLWLs1qdH2KXskEb1iHSB04qGss/QPlz58EIueeBt/vXuWq48dhz/XrSWWa+tZENV9C+Bk/bZjetPmEBeTkbIFUtXW1lWw+A+PchIS/zjaQW/SBcZM6AXj158EHe+8jG/m/0R/168DoheAzDj0JHMX7mRW19YyusryrnptIlMHaN1JrqT6IiexO/mAQW/SJdKT4tw0WGjOGqPATzz3lqO3nMAewzqDcDBo/sxbVwhVzy0kHPuepMz9x/Gz44bT69sHf13B6vKa/jaVwaFXUaHJP7fJCJJaHT/nlx65Jgtod9q4pA+PH3JIcw4dCQPvrWKY38/V33/3UBFbSObahqT5ohfwS8SZ9kZafzsuD149OKDyMlK57x73uLyB99hzabasEuTnfT5iJ7Ev3gLFPwioZk0LJ+nLzmEHx4+mn+9t5bDb57D//x7CZvrGsMuTTrp49isnDri3w4z62Nmj5jZEjP7wMwODKMOkbBlZ6Rx5bHjePHHh/HVCQP585zlTLtpDg++tUpLQyaBpesr+elj73HVIwvplZ2eNMEf1sndW4B/u/upZpYJJMe7JRKQIfk5/OGMSVxwyEj+79Pvc/Wj7/FQ8WpuOHkC4wf23vELSFw1NbdwyQNv8+yidWSlRzh50mC+O3V3cjKTY7xM3Ks0s97AocB5AO7eADTEuw6RRPSVIXk8+L0pPDJ/Nb9+dglfu/UVvnPQCC45YsyXxv5vrmukur6JQXmJf8FQdzPrtU94dtE6vj9tFN+dOpKCbaz1kKjC+HoaCZQC95jZ3sB84DJ3r267k5nNAGYADBs2LO5FioTFzDitaChH7TGA//n3Eu569WMenr+aS44YzTkHDmfNpjpmxbbVNjZz1v7D+Mmx4+iTk1zhk6w+21TL72Z/xBHj+/OTY8cl/Nz77Qkj+NOBfYFL3P0NM7sFuAb4P213cveZwEyAoqIidXZKysnPzeTGUyby7QNHcOO/l/DLf33An+csp6y6gcy0CMfvPYje2Rn87fWVPLtoHddMH8+pk4cQiSRfECWTXzy5mBZ3rj9hr6QMfQgn+FcDq939jdj9R4gGv4i0Y8/devO/5+/Py0tLufe1T9hrtzy+NWUY/XtlA3B60VCufWIRVz36LrNe+4Qrjx3L4eP6Bx5K7p60wbeznl+8jtnvr+ear45naBJMxrYtcQ9+d19nZp+a2Th3/xA4Eng/3nWIJJupYwrbneZhz9168/BFB/LEO2v43eyPOH9WMZOH5/PjY8Zy4Mi+XRbO7s7KshpeW17Ga8s3MG95GZnpEc6eMpwz9htK355ZXdJOoqqub+IXTy5m3IBeXHDI7mGXs0vCOgV9CXBfbETPCuA7IdUh0i2YGSdNGszXJg7ioeJPufWFpZz11zeYNKwP3zt0FMfsOYAWd15bXsbj73zGspIqztp/GKdOHkL6diYVW7OplleXbWDeijLmLS9jbUUdAAN6ZzF1TD/Kqhu46bkPueWFpZy4927U9xxEZtXaQP4bm1uc2sZmemZ9MbY+La/h+ffXs3hNBUeM78/Rew4gK71rl8R0d659YjFrKup49KxJSTER2/ZY2+ljE1VRUZEXFxeHXYZI0qhrbObh4k+Z+fIKPi2vZfd+uVTWNbGhqp5e2enslteDD9dXMqJvDpcfPZapYwqprm+ipqGZ9ZvreHlpKf/5qJSP1lcBUJCbyYEj+zJlVF8OGtWXkf1yt/wlsXR9JffO+4RH539GbWMzGTWlXH3aoRw2NvrXSWOz4zhjB/TaYWBW1DYy+/31rN1Uy7rNdazfXE9JZR3rN9exoaqB5hand3Y6w/vmMqwgh+WlVSxZVwlAr+x0KuuaKMjN5BuTBnPMXgMZ078n+VuNuJk2bRoAc+bM6fD7efNzH3L7S8u47MgxXH702A4/L2xmNt/di760XcEv0n01Nbfw7KJ1/O31leTnZHDypMFMG9efrPQI/++DEn77/IdbgrOtzLQI++9ewGFjCzlkTD/GDei1w5PGlXWNTPv2j6ns/xUaeu72pccH9+nBhVN35/T9hrY73n1VWQ3nzXqTFaXRAX75ORkM6J0d+8mif69scrPS+WxTDSvLavi0vIb+vbM5Zs8BHL3nAIbk5/DKsg08+NYqnl+8nqbYBXAFuZns3i+XPj0y6JmdzgvPPUNGXQUP/P6/GDeg1w67wu597ROue3IxZ+4/jF+dPCGpzmso+EXkS1panOffX8+6ilpystLpmZVOXo8MJg3rs1MXI7UeTd/xjydZsraStIiRkWbUNDRz/xurKF65kfycDM6eMpxTJw/ZsjD5O59u4rv3vkVjs3PbmZPYf/cCsjN2vrumrKqedz+rYHlJFctLq/h4QzWVdU1U1Tfx6bpSWjKi7Q7Ky+bg0f1obG5hbUUdaytqiZgxaWgfJg/PB+DaJxdz1B4D+PO39t1ut1giUvCLSOB21I1S/Ek5d/xnOS8sKcEdJg/P56BRffnryyso7JXFPeftz+j+PQOvsSkjl0t/dQcvfVjCGx+Xk5OZxm55PRiYl01DUwvzV23csoxm0fB8/v7dA3bpiygs2wr+5Li+WES6haIRBdw5ooC1FbU8/vYa/vn2am57cRl7D+3Dnd8uorBXfEYGpTdWc/p+Qzl9v6HtPu7urN5Yy7KSql3+6yMRKfhFJO4G5fXg4mmjuOiwkXy2qZYBvbMTaqSMmTG0ICepx+pvj4JfRGz5CRoAAAeySURBVEJjZgzJ757hmsgS5ytWRETiQkf8ItJlOjM2PizJUGPQdMQvIpJiFPwiIilGwS8ikmIU/CIiKUbBLyKSYhT8IiIpRsEvIpJiFPwiIilGwS8ikmKSYlpmMysFVgJ5QEVs845ut/7uB2zoZJNtX68zj2+9fXv3t65zV+rd2Zp3pd622+L1Hu9oW7J9JpKt3h3VGeRnOIh6227rrp/h4e7+5YWa3T1pfoCZHb3d5nfxrrTTmce33r69++3UudP17mzNu1JvGO/xjrYl22ci2ertQJ2BfYaDqDeM9zjMz3Dbn2Tr6nmqE7fbbtuVdjrz+Nbbt3d/6zp3pd6OPL+9x3el3o602dl6dvT4jrYl22ci2erd+n48P8NB1LujNnck2T7DWyRFV8+uMLNib2cFmkSVbPVC8tWseoOVbPVC8tW8q/Um2xH/zpgZdgGdlGz1QvLVrHqDlWz1QvLVvEv1dvsjfhER+aJUOOIXEZE2FPwiIilGwS8ikmJSOvjNbKqZ3WFmd5rZa2HXsyNmFjGzG8zsNjM7N+x6dsTMppnZy7H3eFrY9XSUmeWa2XwzOz7sWnbEzPaIvb+PmNnFYdezI2Z2kpn91cyeMLNjwq5nR8xspJndZWaPhF3LtsQ+r/fG3tdvdeQ5SRv8Zna3mZWY2aKttk83sw/NbJmZXbO913D3l939IuBp4N5Erxc4ERgMNAKrg6o1VldX1OtAFZBNwPXGauuKmgGuBh4Kpsov1NUVn+EPYp/h04FAhyN2Ub2Pu/uFwHnANwMst6vqXeHuFwRZZ3s6Wfs3gEdi7+sJHWpgV67+CvMHOBTYF1jUZlsasBwYCWQCC4E9ga8QDfe2P/3bPO8hoHei1wtcA3wv9txHkqDeSOx5A4D7kuEzARwFnEE0mI5P9HpjzzkBeA04KxnqjT3vt8C+SVRvoP/edrH2nwL7xPa5vyOvn06Scve5ZjZiq837A8vcfQWAmf0DONHdfw20+2e7mQ0DKtx9c4Dldkm9ZrYaaIjdbQ6u2q57f2M2AllB1NlWF73HhwO5RP9B1ZrZM+7ekqj1xl7nSeBJM/sXcH8QtXZVvWZmwI3As+6+IKhau6resHSmdqJ/TQ8B3qGDvThJG/zbMBj4tM391cABO3jOBcA9gVW0fZ2t9zHgNjObCswNsrBt6FS9ZvYN4FigD3B7sKVtU6dqdvefA5jZecCGoEJ/Ozr7Hk8j+qd+FvBMoJW1r7Of4UuI/lWVZ2aj3f2OIItrR2ff377ADcAkM/tp7AsiLNuq/VbgdjP7Gh2c0qG7Bb+1s227V6i5+3UB1dIRnarX3WuIflGFpbP1Pkb0yypMnf5MALj7rK4vpUM6+x7PAeYEVUwHdLbeW4kGVVg6W28ZcFFw5XRKu7W7ezXwnc68UNKe3N2G1cDQNveHAGtCqqUjVG/wkq1m1RusZKu3rS6rvbsF/1vAGDPb3cwyiZ6kezLkmrZH9QYv2WpWvcFKtnrb6rra43mmuovPej8ArOXzoY0XxLYfB3xE9Oz3z8OuU/WqZtWrehOtdk3SJiKSYrpbV4+IiOyAgl9EJMUo+EVEUoyCX0QkxSj4RURSjIJfRCTFKPglKZlZVZzbu9PM9uyi12o2s3fMbJGZPWVmfXawfx8z+35XtC0CWmxdkpSZVbl7zy58vXR3b+qq19tBW1tqN7N7gY/c/Ybt7D8CeNrdJ8SjPun+dMQv3YaZFZrZo2b2Vuzn4Nj2/c3sNTN7O/Z7XGz7eWb2sJk9BTxv0RXD5lh0NaslZnZfbBphYtuLYrerLLoS2kIze93MBsS2j4rdf8vM/ruDf5XMIzrrImbW08xeMLMFZvaemZ0Y2+dGYFTsr4SbYvv+JNbOu2Z2fRe+jZICFPzSndwC/N7d9wNOAe6MbV8CHOruk4BrgV+1ec6BwLnufkTs/iTgR0Tn4x8JHNxOO7nA6+6+N9HpsS9s0/4tsfZ3OHmWmaUBR/L5fCt1wMnuvi9wOPDb2BfPNcByd9/H3X9i0SULxxCdn30fYLKZHbqj9kRadbdpmSW1HQXsGTtIB+htZr2APOBeMxtDdArejDbPme3u5W3uv+nuqwHM7B1gBPDKVu00EF2hCWA+cHTs9oHASbHb9wM3b6POHm1eez4wO7bdgF/FQryF6F8CA9p5/jGxn7dj93sS/SIIY40GSUIKfulOIsCB7l7bdqOZ3Qa85O4nx/rL57R5uHqr16hvc7uZ9v+NNPrnJ8e2tc/21Lr7PmaWR/QL5AdE56j/FlAITHb3RjP7hOh6xVsz4Nfu/pdOtisCqKtHupfngR+23jGzfWI384DPYrfPC7D914l2MUF0ytztcvcK4FLgSjPLIFpnSSz0DweGx3atBHq1eepzwPlm1nqCeLCZ9e+i/wZJAQp+SVY5Zra6zc8VREO0KHbC830+XznpN8CvzexVogtWB+VHwBVm9iYwCKjY0RPc/W2ii2afAdxHtP5iokf/S2L7lAGvxoZ/3uTuzxPtSppnZu8Bj/DFLwaR7dJwTpEuYmY5RLtx3MzOAM509xN39DyReFMfv0jXmUx00WsDNgHnh1yPSLt0xC8ikmLUxy8ikmIU/CIiKUbBLyKSYhT8IiIpRsEvIpJiFPwiIinm/wNMLEmKKoizhQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#hide\n",
    "learn.recorder.plot_lr_find()\n",
    "plt.vlines(0.01, 5, 11)\n",
    "plt.vlines(0.083, 5, 11)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2.394903</td>\n",
       "      <td>2.887095</td>\n",
       "      <td>0.447363</td>\n",
       "      <td>17.941114</td>\n",
       "      <td>00:32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.046408</td>\n",
       "      <td>2.402153</td>\n",
       "      <td>0.447598</td>\n",
       "      <td>11.046932</td>\n",
       "      <td>00:32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.477659</td>\n",
       "      <td>2.258497</td>\n",
       "      <td>0.449160</td>\n",
       "      <td>9.568699</td>\n",
       "      <td>00:33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.796320</td>\n",
       "      <td>2.221159</td>\n",
       "      <td>0.450840</td>\n",
       "      <td>9.218013</td>\n",
       "      <td>00:33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.447208</td>\n",
       "      <td>2.173755</td>\n",
       "      <td>0.451035</td>\n",
       "      <td>8.791237</td>\n",
       "      <td>00:33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.000578</td>\n",
       "      <td>2.153078</td>\n",
       "      <td>0.451045</td>\n",
       "      <td>8.611320</td>\n",
       "      <td>00:33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.281767</td>\n",
       "      <td>2.142533</td>\n",
       "      <td>0.451709</td>\n",
       "      <td>8.520996</td>\n",
       "      <td>00:33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.852895</td>\n",
       "      <td>2.122231</td>\n",
       "      <td>0.451953</td>\n",
       "      <td>8.349741</td>\n",
       "      <td>00:33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.069045</td>\n",
       "      <td>2.148971</td>\n",
       "      <td>0.451729</td>\n",
       "      <td>8.576032</td>\n",
       "      <td>00:33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.800070</td>\n",
       "      <td>2.126904</td>\n",
       "      <td>0.452148</td>\n",
       "      <td>8.388855</td>\n",
       "      <td>00:33</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#hide\n",
    "learn.fit_one_cycle(10, lr_max=1e-4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "learn.save('roberta_lm_scratch')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3xUVdrA8d+ZkkwqadRQEnqXEpoIgqIoiNjFvujqrrq6+qor7vquvezq7mtZy6q7a1kLCrqiYkGFpQqEHmmhhBYIIaRASJlMzvvHnZnMpEFIMpObeb6fTz65c+feuc/cmXnuueeec67SWiOEEMJ8LMEOQAghxOmRBC6EECYlCVwIIUxKErgQQpiUJHAhhDApWyA3lpSUpFNSUgK5SSGEML01a9Yc0Vq3rT4/oAk8JSWF9PT0QG5SCCFMTym1p7b5UoUihBAmJQlcCCFMShK4EEKYVEDrwIUQoqGcTif79++ntLQ02KE0O4fDQefOnbHb7ae0vCRwIUSLtn//fmJiYkhJSUEpFexwmo3Wmry8PPbv309qauoprSNVKEKIFq20tJTExMRWnbwBlFIkJiY26ExDErgQosVr7cnbo6Hv0xQJfN/REyzcejjYYQghRItiigQ+8flFzHx7dbDDEEKEoIKCAl599dUGrzdlyhQKCgqaIaIqpkjgFZVy0wkhRHDUlcBdLle9682fP5+4uLjmCgswSQLv0z4m2CEIIULUrFmz2LlzJ0OGDGHEiBFMnDiRa6+9lkGDBgFwySWXMHz4cAYMGMAbb7zhXS8lJYUjR46QlZVFv379uPXWWxkwYADnn38+JSUlTRKbKZoRTh3ckW0LjuGq1FgtoXExQwhR02Nf/Mzm7KImfc3+nWJ5ZNqAOp9/9tlnycjIYP369SxatIipU6eSkZHhber3z3/+k4SEBEpKShgxYgSXX345iYmJfq+RmZnJhx9+yJtvvslVV13F3Llzuf766xsduykSuM1qJG2nqxKrxRrkaIQQoWzkyJF+7bRfeuklPvvsMwD27dtHZmZmjQSemprKkCFDABg+fDhZWVlNEospErjdYtT0OF2VOOySwIUIVfWVlAMlKirKO71o0SK+//57VqxYQWRkJBMmTKi1HXd4eLh32mq1NlkViinqwO3eErhczBRCBFZMTAzHjh2r9bnCwkLi4+OJjIxk69at/PTTTwGNzRQlcJvVOM5UuCqDHIkQItQkJiYyduxYBg4cSEREBO3bt/c+d8EFF/D6668zePBg+vTpw+jRowMamykSeJg7gTulOaEQIgg++OCDWueHh4fz9ddf1/qcp547KSmJjIwM7/z777+/yeI6aRWKUuqfSqnDSqkMn3kJSqkFSqlM9//4JouoFt6LmBVSAhdCCI9TqQN/G7ig2rxZwA9a617AD+7HzcZbhVIpCVwIITxOmsC11ouBo9VmTwfecU+/A1zSxHH5CXOXwMsrpApFCCE8TrcVSnut9UEA9/92dS2olLpNKZWulErPzc09rY3ZLFICF0KI6pq9GaHW+g2tdZrWOq1t27an9Rp2m6cduJTAhRDC43QTeI5SqiOA+3+zjvVqt1T1xBRCCGE43QQ+D7jJPX0T8HnThFM7z0XM3UeKm3MzQgjRaNHR0QBkZ2dzxRVX1LrMhAkTSE9Pb/S2TqUZ4YfACqCPUmq/UuoW4FngPKVUJnCe+3Gz8fTEfOjTTc25GSGEaDKdOnVizpw5zbqNk3bk0VpfU8dT5zZxLHWyW03R418I0Qo9+OCDdOvWjTvuuAOARx99FKUUixcvJj8/H6fTyZNPPsn06dP91svKyuKiiy4iIyODkpISZs6cyebNm+nXr19oDScrCVwIAcDXs+BQE5+JdxgEF9ZdiTBjxgzuuecebwL/+OOP+eabb7j33nuJjY3lyJEjjB49mosvvrjOe1q+9tprREZGsnHjRjZu3MiwYcOaJHRTJHBPT0whhAi0oUOHcvjwYbKzs8nNzSU+Pp6OHTty7733snjxYiwWCwcOHCAnJ4cOHTrU+hqLFy/m7rvvBmDw4MEMHjy4SWIzRQIXQgig3pJyc7riiiuYM2cOhw4dYsaMGbz//vvk5uayZs0a7HY7KSkptQ4j66uhd5w/Faaom6iQ9t9CiCCaMWMGH330EXPmzOGKK66gsLCQdu3aYbfbWbhwIXv27Kl3/fHjx/P+++8DkJGRwcaNG5skLlOUwHu1M5rlhNtMcbwRQrQyAwYM4NixYyQnJ9OxY0euu+46pk2bRlpaGkOGDKFv3771rn/77bczc+ZMBg8ezJAhQxg5cmSTxGWKBG6xKC4bmszqPdWHZBFCiMDYtKnq4mlSUhIrVqyodbnjx48Dxk2NPcPIRkRE8NFHHzV5TKYp0tqtFpwymJUQQniZJ4HblHSlF0IIH6ZJ4DaLhXJJ4EKEJK1D4+y7oe/TNAk8zGaRErgQIcjhcJCXl9fqk7jWmry8PBwOxymvY4qLmGCMhyLDyQoRejp37sz+/fs53fsJmInD4aBz586nvLyJErgFV6WmslJjsUjPTCFChd1uJzU1NdhhtEimqUKxe+9ML9UoQggBJkrgYVa5K48QQvgyTQL3DGjlrJASuBBCgIkSuLcKRVqiCCEEYKIE7q1CqZQqFCGEABMlcLtNqlCEEMKXeRK4VKEIIYQf0yVw6U4vhBAGEyVwdxWKNCMUQgjAVAlcqlCEEMKXJHAhhDApEyZwqUIRQggwUQL3tgOXZoRCCAGYKIF724FLFYoQQgAmSuA2izQjFEIIX6ZJ4J4qlJ925QU5EiGEaBlMk8A9VSgfrtoX5EiEEKJlME8Ct5omVCGECAjTZEVJ4EII4c80WdEq98EUQgg/pkngUWFWAKLDTXMfZiGEaFamyYZKKSb2acuR4+XBDkUIIVoE05TAAcJtVsoqXMEOQwghWoRGJXCl1L1KqZ+VUhlKqQ+VUo6mCqw2DruFMulKL4QQQCMSuFIqGbgbSNNaDwSswIymCqw24TYrZU5J4EIIAY2vQrEBEUopGxAJZDc+pLqF2y0cK3U25yaEEMI0TjuBa60PAM8De4GDQKHW+rvqyymlblNKpSul0nNzc08/UmB1Vj7F5S7W7Mlv1OsIIURr0JgqlHhgOpAKdAKilFLXV19Oa/2G1jpNa53Wtm3b048U2HKwCICVu2U8FCGEaEwVyiRgt9Y6V2vtBD4FzmyasOrnkps6CCFEoxL4XmC0UipSKaWAc4EtTRNW/WRIWSGEaFwd+EpgDrAW2OR+rTeaKK5aTR3cEYAo6Y0phBCNa4WitX5Ea91Xaz1Qa32D1rqsqQKrzR8v6g9IAhdCCDBZT8wI93goZU7pjSmEEKZK4A6bkcBLyiWBCyGEqRK43aqwWhSlMh6KEEKYK4ErpXDYLJRKd3ohhDBXAgdw2K2USh24EEKYM4GXSAIXQgjzJfADBSV8uvZAsMMQQoigM10C93BKb0whRIgzbQI/IU0JhRAhzrQJXNqCCyFCnWkT+InyimCHIIQQQWW6BH7HhB6AVKEIIYTpEviZPZIApCmhECLkmS6Bewa0khK4ECLUmS6BR4Z5BrSSOnAhRGgzXQKPsBsJXMZDEUKEOtMlcIc7gUsduBAi1JkugXtK4HPX7A9yJEIIEVymS+COMCPk9D35QY5ECCGCy3QJPMxqupCFEKJZmC4bKqWCHYIQQrQIpkvgQgghDKZM4FHutuBCCBHKTJnAbx3fHQBXpQ5yJEIIETymTOBRYTZA2oILIUKbKRO4dzyUMulOL4QIXaZM4FHhRgIvlgGthBAhzJQJPMJuVKHITR2EEKHMlAncUwKXIWWFEKHMlAk80n0Rs1jqwIUQIcykCdwzJriUwIUQocuUCdzTjFAuYgohQpkpE3jVbdWkCkUIEbpMmcDlIqYQQpg0gTtsVpSSi5hCiNDWqASulIpTSs1RSm1VSm1RSo1pqsDqY7EoosJsHJcELoQIYbZGrv8i8I3W+gqlVBgQ2QQxnZIYh41jpZLAhRCh67QTuFIqFhgP/AJAa10OlDdNWCdnJHBnoDYnhBAtTmOqULoDucC/lFLrlFJvKaWiqi+klLpNKZWulErPzc1txOb8xTjsUgIXQoS0xiRwGzAMeE1rPRQoBmZVX0hr/YbWOk1rnda2bdtGbM6fVKEIIUJdYxL4fmC/1nql+/EcjIQeENHhUoUihAhtp53AtdaHgH1KqT7uWecCm5skqlMQ47BLKxQhREhrbCuUu4D33S1QdgEzGx/SqYl12CiSKhQhRAhrVALXWq8H0poolgaJcdgor6ikrMJFuE1uciyECD2m7IkJRhUKIBcyhRAhy8QJ3Dh5yD1WFuRIhBAiOEycwI0S+JSXlgQ5EiGECA4TJ3CjBK51kAMRQoggMW0Cjw5vbAMaIYQwN9MmcIddWp4IIUKbaRN4rENK4EKI0GbaLNgu1kH72HC5sbEQImSZtgQOcOHAjsg1TCFEqDJ1Ao9xGHfl0dIURQgRgkyfwLWGYqlGEUKEIFMn8OhwT3d6GVZWCBF6TJ3APZ15ZDwUIUQoMnUCT4gKA+DIcRkPRQgRekydwDu0cQCwbm8BV/99BR+v3hfkiIQQInBM2w4cICHSKIE/9+02AFbuPsoZXeLo0yEmmGEJIURAmLoEHhthrzFv8guLgxCJEEIEnqkTuNWicNhN/RaEEOK0mT77lTora5kn7cKFEK2f6RN4bQpOSLtwIUTr1yoT+NHi8mCHIIQQza7VJPDHpw/wTuefkAQuhGj9TJ/Au7eNAuDGMSn8+YrBgCRwIURoMHU7cIBPbz+T/fklAEzs0w6AfKlCEUKEANMn8LjIMOLcHXriIo124UeL5SKmEKL1M30Vii+71UKMwyZVKEKIkNCqEjgYA1xJKxQhRChodQk8PjJMSuBCiJDQ6hJ4QpQkcCFEaGh1CTw+Mox8uYgphAgBrTCB26UOXAgRElpfAo8Ko8TpkgGthBCtXqtL4J7brEk9uBCitWt1CTze3anH0ztTCCFaq1aXwD0l8CtfXxHkSIQQonk1OoErpaxKqXVKqS+bIqDGio+seZs1IYRojZqiBP5bYEsTvE6TiHeXwAH+b8H2IEYihBDNq1EJXCnVGZgKvNU04TRenM+Njl/8IROtdRCjEUKI5tPYEvgLwO+AmjemdFNK3aaUSldKpefm5jZycydns1qIsFu9j/Pl9mpCiFbqtBO4Uuoi4LDWek19y2mt39Bap2mt09q2bXu6m2uQV64b6p0+VFgakG0KIUSgNaYEPha4WCmVBXwEnKOU+neTRNVIo7sneqdziiSBCyFap9NO4Frrh7TWnbXWKcAM4Eet9fVNFlkjRIbZaBcTDsAhSeBCiFaq1bUD91g26xyUkioUIUTr1SS3VNNaLwIWNcVrNRW71UJiVLhUoQghWq1WWwIH6NAmXKpQhBCtVqtO4B3bRLA370SwwxBCiGbRqhP4oOQ27DpSzInyimCHIoQQTa5VJ/CUpCgA9h2VkQmFEK1Pq07gXRMiAdiTVxzkSIQQoum16gTezZ3Ab3tvDRWuOnv7CyGEKbXqBB7nM7Ts3qNyMVMI0bq06gSulPJOS4ceIURr06oTuK99+VICF0K0Lq0+gX9022jAvwqlvELqw4UQ5tfqE7hnZMJXFu5kb94Jvv35EL0f/prtOceCHJkQQjROq0/gvj5O38ev3jOGL1+SeSTI0QghROOERAIf3i0e8B9a9lChdO4RQphbSCTwd28eSYdYBz/tyvPOOyitUoQQJhcSCTwq3MbVI7qwP7+q1P3lxoOkzPqKzdlFQYxMtFR78orlhtiixQuJBA7Qo110rfNf+iEzwJGYw/PfbmP5ztC8TpBxoJCzn1vEez/tCXYoQtQrZBJ4amKUdzrMWvW2c4+XBSOcFq2yUvO3hTu49s2VwQ4l4PbmneCil5cC8OPWw37PVVZq/rPugAzLIFqMkEngnoGtAMJsVW+73oGuvroPVv8DivPqXqYVyjx8PNghBM38jIPe6UXbclmz56j38cy3V3PP7PW8sWRXMEITooaQSeBtfMZFmdi3nXc6OryOu8qVHYfdS+Cr/4G/9IZ/XwEbZkNZ628/PvmFxd7pUKsHfrlaldrlr63wTv93ey4Aq3cfRYiWIGQSOMD95/fmulFdefayQWQ9O5Vbx6VysLCUsgoXT8/fwo7DxygscRoLh0fDnSvhV0tgzJ2QuxU+uw2e6wUf3wRbvoSK1l/9EmpVTMXlrhrzDle7LZ/vGZwQwRRS38TfnNOLpy4dRJS71N23QyxlFZX86ettvLF4F5P+uphb3l5dtYJS0HEwnPc4/HYj3PwtzsHX4Nq9BGZfZyTz/9wJO38EV+u8689vPlgX7BACpris6jO8+IxO3ukH5270OxP59uccXJWhdWYiWqaQSuDV9esYC8A/l+32zkvfk0+ps6oUdqCghKJSJ6UuzR1L7PRaPone+S+w5dy3oe8U2Pw5vHcp/LUfzP8d7FsFJq122JxdRJ67xH396K4ArNp9lOyC1tfpqbyikue+3UrhCeOM60BBCfd/sgGAoV3j6N2+qtXSwm25/Paj9X7r9/j9/MAFK0Qd6qgADg292tfetHDH4eMMTG4DwNhnfwTg6rQuzN90CAAXVr4u6UO/Sy+Fi0og8zvY9AmseRtW/R3iusHAy2HQldC+f0DeS2NVuCqZ8tIS72PPwQ3gzGd/JOvZqcEIq9ks2naYVxbu5JWFO9n59BTv5wxgVYrbxvfgRLmLVxftBGDehuwar1HqdOGwWwMWsxDVhXQJ3G6t/e1nHq55oXJ2+j6/x/s9pVJ7BPSfTsml71B41xa45DVI7AnLXoTXxsCrY2Dx85Cf1dThN6nFmbl+jxOjwrjrnJ5+85Zk5pIy6ysmPr8ogJE1D4vPWPFfbvRPzq9eN4wwm4XfXdCXByb38XvuX78Y4Z2et75mUjeTBZtzWLkrtFpYtTYhncABnr1sEABjuidy33m9AcjMMZrRFZU661zvYIH/ha1LX13GGc/+BEOuhRs+hfu2wZTnITwWfnwCXjwD3poEP70Ox3Ka6d3UVFmpeejTTTwzfwurs2pvPbHj8DFufjvdb15EmI37zu/jTWALtx3mhn+sAmD3kWLT3yCjoKTqs61ePdIu1uGd7uLT/BTgrF5JrP/jeQD8beGOU97e4WMtb3/d+m46V7/xk/RINrGQrkIBmDGyK5P6tycqzEZEmJUvNmaz9dAx9uadYPxzC2tdZ1K/dqzafZQKVyU2dyl+6yGj1P7SD5lcO6or0eEJOEbeCiNvhYK9kDEXNs2Fbx6Ebx+C1PGU9b2M0p5TaZOQ1Gzvb1/+CT5ctReAvy/eVWtViO8QAx7dk4yOT73bxwAw81+r/Z7/ObuQDm0cNdYzi4IT5bXOH5gc6/d4XE//z8ZutRAXGQZA7rGTt9C5/5MNLNx6mLzicv58xWCuSutymhE3rYwDhX6Pp7y0pNbvxvKdR8gpKuXSoZ0DFZpogJAvgQMkRYcTEWbUZQ5MbsOmA4W1Ju/pQzqx46kLmT4kmaLSCr7caHT6WLs337vMXxdsJ+3J7/nj5xlVK8Z1hbPuhduXwh0rYdx9kL+H8Pl343ixD3x0HWR8CuVNf9egJ7/a4vdYa83WQ0V+HZgKS/zPNAZ0ivWWPPu4E3h1ngOWWRWccGJR/vPSH57EvDvP8psXHxVG1rNT+dX47vz23F7e+bdP6IHTVYmznl6Zn63bz5w1+8krNg4Wnu9LS+AZVtkj3qefhK9r31zJvbM3hFx/ALOQBF7NGZ3japSs7p1kVK0M6BSLzWph6qCOxEfaWbrjCMfLKrjs1eU1Xufj9P3e6ZJyV1XCbNcXznkYfddaLi57gn+7zqNszyqYMxOe7wWf3gbbv6PSWe7XGuZ0RYb5X2Q7WFjKBS8sYfory7zzCtwtMSwKBnduw+PTB3if6xwfUeM1k+MieOH77azfV9Do+IKloKScuMgwRndPAGBkagJJ0eFYqmd1t4em9ONedxUbQO/20VRUas7604+1Lp97rIx7Z2+oMa+l6Fjt7Km4zFXv9+2Zr7fKEAItUMhXoVR3Zo/EGvN+OS6VQZ1jObu30YPTYlEM7RrPksxcBj7yrXe5/h1j2Xywqi6xrMLFj1sOc/v7awFY+ftzae+uXz1e7mKj7sHGih48lX8dG26K4oePX2Hatm+xbpxNhcXBgYp4unfviYrpCLEdIcbnL7YjRHcAW1i978eTgCPsVkqcLu/4HgUnnKTM+opz+7bjB/e87U9e6K0S8rBYFHarwunSTB3UkR5to9h79AT/WZ/NJa8sM23rlIITTuIi7Hx02xg+Tt/HhD5tG7R+r3bGmUlOURl5x8tYuuMIL/+4gwX3jkcpxVtLa3a333KwiLIKF+G24LdcCbf7f87lrkq+zjhYZ1XJG4t3sWr3Uf5z59hAhCdOkSTwanq09W9auPTBiUSF2zinb3u/+eN7JdUY7Oiuc3p6kzUYF0M/WVNVEv9s3QF+fXYPAPKLq6otKrHwq6WRLC+5mQdKbiRzZhjvvfcO7VU+1sMFpBTshWOHwFVLCS4yqSqhx3TwS/LrCyOYvfAAihjSH57MwEe/5eH/ZPit/oPPe6ievD3W//F8AG8HqDlr9vMfdwuMzdlF9O8UW+t6LVnBCad3eIXTqZf2/Z5sOlDovRC6fGceY3sm8ff/1j5eSp+Hvwn6Qe/Ljdks22G0PvnT5YM4s0cS4/68kHtnb6B9jIMz3fX+O3P9x8RZv6+A/OJyHpizgacuHeQtjJiJ1hqlaj/LMiNJ4NVYLIr0hyexObuIoV3jiHHUXjd4Rpc4v8c3j03lwkEdeWRaf7YdOsZHq/ex+WCRX5J/9uutjEiJZ3i3BPKrXURbvtP4QcVERVLa/Rw+SYpi66FjxBTb2PS7yUbnoJJ8KMo2kvkxz/+DUHTQ+H9wAxw/DBj1lUOAdAc4tRX7Kx35MiKKvc5YDukEDut4cnQch0ggR8eTo+ONbdTy5Y6qNl7M+N5VF/amvLSEzY9PJjLMXF+lgpJy2sWcfgKK8KmaWrn7KON6JbEk8wjzNx3kLZ/BrnY/M4XUh/w7/ezKPU73trX3QQiExdurmoxePcLosNUuJpzDx8p4/MvNfHPPeMDoxFXd/IyDfL/lMNHhW3hhxtDABNxEvtiQzV0frmPVH85t1GffkpjrVxcgSdHhjO9d/ym1b0cXgN9dYDS3mzk2FVel5vP12Ww5WLNp1uWvreCpSwd626A/Pn0Af/z8Z+/zR4vL6fu/33gfV2qNq1JjtSiITDD+OgysOzBXBRzPYduO7fx17n9pp/LpoI5yZ2o01szt9KjI5kzLz7RRtVwwffq3Rkk+LBps4WANc/8PN6pqrOFgtdPOFs6aETB73WHKsfG3xz7ngamDUL7L2cLYdKiUbu3iiY2Ocr+W5znf1w6rmrbYaj2ANCWnq5JKrck4UESf9o27MLf1iQu48R+reM3d2Qfg/ZV7/ZZRSvH0pYNoE2Hnzg+Ms7MXvs9k2hmdOK+//1ldoCRGhwNw45hu3nlndIljweYcv5EoPe3j5/x6DLuPFPPAnI28t8IYI92MF7E/W3cAgJW7jjLNPVRCqdPFkeNldI6PrG/VFksS+Gny7YG37ckL/Oo1rRZFnw4xbM4uonf7aFKTokiOi/R22f/DZ1XVGGf1TCIpOpwjdQwaVVzuYnvOsRoHjDpZbdAmmcmfrAeMTicrf38uxDpYsngXT803WqVkPT6BjxeuZu6idC7taeHy3hbsxYeNkn15sTFQl6scThSDy2lU33jmVZSR4Crnl9YSwpT7wtd3s2uEMujUIq6iLGBzGH/2CCOp2yLQdgc7jlZQ6LRyRmoH7OGRYHdUW9Yz7QBbRK3Pf7Ihl5eXHOD2SQOI5ThZOeV1nnWcCofdysS+7VhVR/t6z/WHa0d1dc8Zxp0frGXehmzmbcjm12f3YNaFff3WOV5WwUOfbuJ/p/bza49em6JSJ86KSm9CPlXlFcbFyEemVV2sfvayQSzYnIPCOMhlF5R4v9NpKQkkRBnXWjyJe+uhYxSXVdQ4O2vJurg/j22HjjG+l5OP0/exfl8BX206yNYnLjitXrXHSp1Eh9uCVi1jnr3fAs37zVicrspaL0oNTI7l3z8ZpbFRqYmM7ZnoN+aKR3xkGN2TojhyvIyze7f1DlkKYLMoKio1a/fmn3oCB577dqt3emKftt66ysGdjeEBlALCorhq8gTShqWRmhTV4C+gApZtPczNb6/EjouXrxzAQ5+kE4aT5fefxZfrdvP3H7cShpNXZwyiY7SCinL3gaC8xgEBlxMqSikrPQEVpYTrcqgoJTM7l0iXk6PFxYTjpCTnOHZbhbGOswQqSo3/nLw0fSVwZTiwBK7x5MbHMJK8NQwsVlBW40zAYjMe1/rf5l3umrJK+tuPU4GVSixUYMGFlQosTOzSCf4z17veVIsNe7sj7DpaZiy7zAKOvu7tKlAW9h08TqeMgyzNa8Nlw7oYBzXvn/J7/Oq32zlS7OT5K4dWe15VW8//L3P5as6yWLBmRXtjS7TYeO/CMJ75Zju7fl7Ns9/uYNfRUs5uGwOF+0kJs5JAES4suLBQgZUb31zKu7eMIcoR5ncQ3Hf0BJFh1joPLJuzi+jZLrr+UR21hsoK43tR6YRKV9W0y+n/nKvCZ77ncdW8n/fn8fbSHdhxcY0VchYv4rH/Gu8BrEy2WPjNI2v423UjcISHGZ+v1V7zM6/2vThaUsmkF5Yxtld7Xr5+hP8yAUro6nTbdyqlugDvAh2ASuANrfWL9a2Tlpam09PT61uk1ViwOYdb3zXe68NT+3HLWak8/902Xlm402+53c9M4ZF5P/Puij386fJBfLByLxv2G50s1v7veVz00hL6d2rDWzelnfK2U2Z95Z0+o3MbPv+N0bb5eFkFAx/5lutGdeWpSxtcPq5Ba83Nb69m4Tb/bvjR4TaO+4zs98DkPtw5sWf11Ws15PHvKDjhJOvZqfztx0ye/2673/PXj+7Ko9MG+F9w1dp9ACgBZ6mR1D2J3T3tKjvBXe+tIBwnDlWOg3LuHt+Z+DBtrOf90Xv+XKBdNedV+++sKCdj31GsuOgW7+Bg/nFsuIplcVEAABWXSURBVOiZ5PBZtup1ypxOKiucWHFhpRKrah3tq7WyodyJrahcU4GFhOgI/6SnrLiUlR25J2gTrugQba2WfH0TtslH91TVE78VblkASaf2O6jxckqt0VrXSAKNKYFXAPdprdcqpWKANUqpBVrrzY14zVbDt34zxmGcYj0wuS+/ODOVJ77czLwN2XRJiEApxX3nG/XnFw3uxNUjunoTcHyknXP6teOztQdwuirrHLvFV/VhTn0TdXS4jeWzziExuv6mh6dKKcW/Zo70O2AAfsl7UHIbvtiQzSVDk0mOq9mmvDpPm/Tnvt1a42AH8O+f9pKele+90OYOxKhft4WBo02tr3ve84vYVTnab94fz5/S6JKSHYg4VEROURmDeiXxjwXb6ZYYRc/htTfHW7Y1x2/YAkUln/56DEO7tAFdyZz0PTz8nwxiHVZWzZoIutL9p3G6KnhtYSaDk2OYNWcDFjQWpXnu8oGMSXVfhNaVgPZZz//vRJmT699cwdVpyVw9PNlIlLrqIPOrd1ZioRKb+wATbYcnp/eDygpcFU6e/GKT+/lKrLhoE64oLTcOSFcN7Uh0mGLOyt1YcTG5UxIdYux+B7CS0nJ2HT5ERamVMZ3bEx8ThdVmd5d47UYVoKWux1ayCpy8vDALJ1ajBG2x8+qNo8Bio1LZ+GF7PuP6dsQRHu4tSb+5bC9v/3QApzvd2XBhVS7ve/D8n9grgfsnda/joF3z8c6cQt5eugMbLm4YlUz3BEctBQGfdRxN31rrtBO41vogcNA9fUwptQVIBiSBu3mqRCJ8Wmi0jQnnpWuGcuOYbiS76+TaRNh5fHrVhcn4SDv5J5wopRjbI4l//7SXC15YzHu3jKJTPUmwvKKShz7d5H08/+5xNZr41bf+6fr4V2O46u8ran3unL7tePGHTMY++yM7nqrZzryswsUNb63ivvN7M6p7VRv82pL3NSO78uGqvWw9dIzKSu3X6cZ3WIPqNmcXseuI0ZGqR9soduYa001Vb9m3Qyx9OxjT/3N+n3qXndinHS/OGMKkfu0Z8Mi3aCzM25TD0BSjZc9ry7IpJZzSUigigtiIqlZQy7fn8tcVhUAh4N5XGubtDWPE0NQ637+v/IIS1uo8ruoyCFK61ni+x/hU7wiMHWIdfHTbaHAPq2AFogq30b1tFGv25BsXbH2uhe/TXViTmc+OCuNC6COb4L8PTKBLfKT3s8rcm8/tGe6Ob1tgeLd45t5+5knj9pgw6yvAJ+5KSPlnBVDBxWd0Yt6G4wzbXcGnd4yqiks5OYDR6uupSwca16C00XnLt6VNweEI7u/qf5Cvz1Nvr+ZHV3fjfR1O4t8XjzrJGk2vSXpiKqVSgKFA6N0Ftx5PXzaI60d3ZWItnUTSUhLo2Kb2ZPrjfRP4/n/OBmC0O6ntzC3mtvfqr376z/oDzF1rtDv/7I4zA9Y+e3DnNsQ6bCRVq/N8ZFp/zh9QdSay5WBVy4V1e/N56qvNfLr2AKuyjvLLd9J5d0VWndt4ZFp/Lh2a7H38wg+ZLN9xBIC7PlzHZa/V7A3rGXDLd3TJK91tviOCNAysUorpQ5KJCrd579P6r2VZ7DtqZELPwQVgTVa+37qei4+++neM5cNV+2oMyAXG2djCrYf9usF7xj9vE1F781jf0RevG9WVlKQov+fvn9yHy4Z15qlLB3HdKP8DwOz0feyodj/Vs59bRPffz+cv320DIKvaPWjX7Kl6jzlFpTz11Wa/Mzhff/pma63zPTxD/q7dW8ChwlLW7yvgs3X7edfdcmZcryRmjOjK8lnn8MDkPnzwy1E8MLkPX919Fn+8qD8HCkoY/sQCcopqH3hMa+0dOuFfy3b7NRFeuuNInes1p0YncKVUNDAXuEdrXaPdnFLqNqVUulIqPTc3t+YLtGLJcRE8ecmgOtuS1yU+Koye7aK90x4ZB4pwVWqOFpfXOrKg7w81MaphLRMaw2G3svHRyaQ/PImMxybzxCUDObt3W24ak0J/n4uvnoNLUamTS19dzptLdnvPGI6VVfg1p/T1+vXDmTk2laFd45jUzzggvPRDJte+tZLKSs0XG7LZuL+Qfyytuki8aNthRj/zAwu3HuaLDcYYJFcO78yt47ozdXBH/vGLU7+m0Fy+uOss75C94/68kGOlToZ0iWN4t3jsVsXKau2wqyeIlb8/1zum/Vebao6z8tcF25j59mpSH5rPloNF3Dt7PdtzjINZmzrGPlFKece/uX1Cj3rjv+WsVO909X4R1b384w4e/2JzjeEFAPbnGwevp+dv4c0lu/nn0poX+3/zwVq/5povXD3E74Be3WuLdnDJK8v8tvfeLaOwWhSd4iK4c2JPbFYLd07syYBObbwFpbzict5bsYdH5/1MyqyvuMKnYPD5+mx6/eFrlu88wmNfVFU0ePbDqKd/qHcfNIdGJXCllB0jeb+vtf60tmW01m9ordO01mlt2zasu7Iw+LbXXbs3n3tmr+fK11fUGFFv26Gq0k98VMMOGk0lOtzGDaO78c7NI7FYFEop/n2LcWr59vIsSspdXOIzDkttPri16lR07u1nMtldirdbLbx1UxrdEqva7Pr2JH3iy82s25vP3rwT/MI9euIfPtvE91uM4XtnXdgXq0XxyrXDOLNH840AearaRNi91z/ASHJFJU46tHEwuHMcP1Ubq9vTi/bOiT14eGo/2sc6GNY13vv8z9nGxe+yChdbDhb5VUNd+OISPlt3gHtmr/duuy6f3nEmS3438aRVMr6dkf50edW1lrN7t2X3M1NqLO/bCuvRaf29bbGXZB7hYGEJn7t7927cXzVS4t//u5OUWV/5DQR22dBkLhmazP9dPYS+HWoOtpYcF8HqamcvJ+N7tvq3hTt4e3kWYNyh60S5cUbgaSH2r2VZfuveNCbFO119vJsKVyW//2wTu4/4n3k0ldNO4MqoQPwHsEVr/demC0lU99jFA/jw1tHYrYoFm3O8Pemq/8A9P5ALB3YgugW1zz2rVxL3TDJG8luzJ59duXV/meMi7QztUpWUhneLr1FXPSo1wTv9TcYhv+fSs/L9qkyyfcYtj60naQWT5+1t2l9IYYkxRsvYHols3F9A4QknR46X8cAnVSXJByb35ZfjjLrXGSO78PDUfgAscrcGuuilpVz44hLqExNe976ICrfVGAe9Lk9fOohHp/WnT/sYJvVrx+vXD+Odm0eilGL1HybxxCUDuXBghxrr3TAmhZdmDKFjGweLt+eSd7yqMLJo22FvldIzX9esNnnikqrrRV/cdRaZT13Iz49N9s67eEgnvzGJAC4fdvLhcJfPOqfW+R+4O2fFOozf1ILNVeP5Pzy1H10SqqpCRzz1vfdACsbY+R+s3NtsN0FpTAl8LHADcI5Sar37r+ZhVzSaUooxPRIZ3T2RNxZXddP+9b/XsnBbVQk0zGoh3GbhteuHt7jxHjwJ5y8LtnnnLbh3PFMGdeBunzv/vDhjKBFhVp65bBCf3lH7xS3P7e6gqlomxv3jemr+llrripWq+w5MwfbC1UMAWLErj7zictpE2Bnfuy2VGpbtPMJLP2R6x9RJiPJvQRRus/LLcd0ZmBzrLSFmVquHro1v0mmMa0d15RdjU40BvG4awQUDO3qfaxsTzg2ju/HqdcP81nns4gFY3WdnZ/duy9IdR7x141MGdaCiUjPuzwtrHbP9jgk9/DoP2a0W7FYLUeE2XrtuGA9M7sNYn7OrG8d0Y/czU/jLVWec9L10iovgkWk1b4H45Fdb+NuPmbzjrkv3uHxYZ24ck4JSipevqRpWwHd00u98kn1zDMnbmFYoSzH6c4gAObdvO5ZkHvGbN/Nfq72DI4XZLFyZ1jIH3o8Ot9E2Jpx1e40haLsmRNKrfQyvXjccgG6JUaTvyeds9xAG14ys2ULC46LBnXh7eRYVLs3eoyeIsFv56aFzuf4fK1m3t4D57lL5S9cM5e4P1wGw6+mWW7aYPiSZAwUl/Pkb4+BWXlHJkC5xxDhsLN6eS3F51TCvL9Ux/sjo1ETeWrqbs6uNY79s1jk4bBbumb2eByb3oXN8JPGR9oAe4JVS7Hp6CvM2ZDNlUEe/Djxn927LR6v38ZsPjM/p4jM6ee89O+TxBd7lfj+lL1cO71LvWdSFg4yDh++wuL89t1eD3uuNY1LIO17Oef3bExVu48/fbOW7zTk1+iOAcfDyvJdpZ3TirSW72LC/kLKKSj5ff4DpQ5KZ6z7wdm8bxYlyV5P3XG2ZRRJRq7E+d4eZ7NO64/vNOXy2bj/HyypIiGyaNt7N4UV3SRNqJujLh3fmmctOrXNRQlQYP943gbvdN1gocRo/jA9vHU24zcIX7tYIY91DAyfHRbS4M5Lqfj2+h7c+t02EHZvVwtgeSSzenut9P2BUR9XG0wRzT15Vu74x3RNJjosgMTqc924ZxeDOcSREhQVlX1gsikuGJtfofTm2V5Ixzo/bsK7xNcaI+fzOsdw2vgfxUWF+y9bFt0t8Q4cZsFoU90/uwxld4ujZLtrvNwdGS5aqWP0v3N7h01lt7lpj3JU+HWJISYzkx/smNMuwAy2nolScVM920fz23F5k5RXz7GWD+b/vt/PG4l388t2q5oVWa8tNVGN6JBJmtVDuqiQ2ovFfvepjeDvsVkakJLDU3bwwPjKM2beNbpa2703NYlF8c8941uw56q0iGtc7iW9+rqrjr6uOFmBU9wS/x49M68/0IXW30mgpYh12hneLZ9Xuo7SJsNMu1sGbN6ZxxmPfee8UVb0p46n45p5x9V5rOVWXDE3mkXlVraMevKAvv5tsFBqqHwgnD+hA5lMXct/HG5i3IRtXpaawxFmjeW1TkgRuIkopv7vC/H5KP44WlzPHZ8zx29x1zS2RUopJ/dsxf9MhKisbXx+YFB3Ogxf0pVNc1aBP087o6E3gFovy6xxkBsO7VSXiCX3aeaevGdml3gNRrMPuHRJ2bM9Erkzr0qIuZNcnzZ3Af+nTLPE3E3vy1PwtfPLrMfW2mKmL0bmq8f0g2kTY2fX0FL7YmE1qUpTf9Zfa2K0WzuyRyLwN2fT4vTGMcFNdb6iNOT5hUafHpw9g5e489h0tIcJuPaXeeMH0y3Hdmb/pEIM7199u+FRVb6t8VVoXHpy7qY6lzSU5LoJObRxkF5aS1i3hpMsv/t1EgNMaVS+Ybh3XnTCbhZk+Cfzms1K5bFhyg6tAmoPFohp0NnPZsM7M8ukRve9ozZuGN5WW/WsXJxUZZvP22qxeX9cSDesaz66np5y048fpUkox9/Yx/P2G4c3y+oF205kpwKk1gXTYraZL3mB0VrtnUm+/MwarRbWI5H06wmwWnyGE4f7ze9ezdOOc9miEpyOURiMMtAMFJcRF2E01PrM4Oa01y3bkcWaPxDpvuCxanqJSJy9+n0lat3gm9W/f6CasdY1GKAlcCCFauLoSuFShCCGESUkCF0IIk5IELoQQJiUJXAghTEoSuBBCmJQkcCGEMClJ4EIIYVKSwIUQwqQC2pFHKZUL7DnpgjUlAUdOulTgtdS4oOXGJnE1jMTVMK01rm5a6xr3pAxoAj9dSqn02nohBVtLjQtabmwSV8NIXA0TanFJFYoQQpiUJHAhhDApsyTwN4IdQB1aalzQcmOTuBpG4mqYkIrLFHXgQgghajJLCVwIIUQ1ksCFEMKkWnwCV0pdoJTappTaoZSaFeBtd1FKLVRKbVFK/ayU+q17/qNKqQNKqfXuvyk+6zzkjnWbUmpyM8aWpZTa5N5+unteglJqgVIq0/0/3j1fKaVecse1USk1rJli6uOzT9YrpYqUUvcEY38ppf6plDqslMrwmdfg/aOUusm9fKZS6qZmius5pdRW97Y/U0rFueenKKVKfPbb6z7rDHd//jvcsTf6dj11xNbgz66pf7N1xDXbJ6YspdR69/yA7bN68kPgvmda6xb7B1iBnUB3IAzYAPQP4PY7AsPc0zHAdqA/8Chwfy3L93fHGA6kumO3NlNsWUBStXl/Bma5p2cBf3JPTwG+BhQwGlgZoM/uENAtGPsLGA8MAzJOd/8ACcAu9/9493R8M8R1PmBzT//JJ64U3+Wqvc4qYIw75q+BC5tpnzXos2uO32xtcVV7/i/AHwO9z+rJDwH7nrX0EvhIYIfWepfWuhz4CJgeqI1rrQ9qrde6p48BW4D6bk89HfhIa12mtd4N7MB4D4EyHXjHPf0OcInP/He14ScgTinVsZljORfYqbWur+dts+0vrfVi4Ggt22vI/pkMLNBaH9Va5wMLgAuaOi6t9Xda6wr3w5+AzvW9hju2WK31Cm1kgHd93kuTxlaPuj67Jv/N1heXuxR9FfBhfa/RHPusnvwQsO9ZS0/gycA+n8f7qT+BNhulVAowFFjpnvUb92nQPz2nSAQ2Xg18p5Rao5S6zT2vvdb6IBhfLqBdEOLymIH/jyrY+wsavn+Csd9uxiileaQqpdYppf6rlBrnnpfsjiVQcTXkswv0PhsH5GitM33mBXyfVcsPAfuetfQEXlsdVcDbPSqlooG5wD1a6yLgNaAHMAQ4iHEKB4GNd6zWehhwIXCnUmp8PcsGdD8qpcKAi4FP3LNawv6qT11xBHq//QGoAN53zzoIdNVaDwX+B/hAKRUb4Lga+tkF+jO9Bv+CQsD3WS35oc5F64jhtGNr6Ql8P9DF53FnIDuQASil7Bgfzvta608BtNY5WmuX1roSeJOq0/6Axau1znb/Pwx85o4hx1M14v5/ONBxuV0IrNVa57hjDPr+cmvo/glYfO4LVxcB17lP8XFXT+S5p9dg1C33dsflW83SnN+zhn52gdxnNuAyYLZPvAHdZ7XlBwL4PWvpCXw10Espleou1c0A5gVq4+76tX8AW7TWf/WZ71t/fCnguTo+D5ihlApXSqUCvTAunDR1XFFKqRjPNMZFsAz39j1XsG8CPveJ60b3VfDRQKHnFK+Z+JWKgr2/fDR0/3wLnK+UindXHZzvnteklFIXAA8CF2utT/jMb6uUsrqnu2Psn13u2I4ppUa7v6M3+ryXpo6toZ9dIH+zk4CtWmtv1Ugg91ld+YFAfs8acxU2EH8YV263YxxJ/xDgbZ+FcSqzEVjv/psCvAdscs+fB3T0WecP7li30QQtA+qIqzvG1f0NwM+e/QIkAj8Ame7/Ce75CnjFHdcmIK0Z91kkkAe08ZkX8P2FcQA5CDgxSji3nM7+waiT3uH+m9lMce3AqAP1fMdedy97ufvz3QCsBab5vE4aRjLdCfwNd6/qZoitwZ9dU/9ma4vLPf9t4NfVlg3YPqPu/BCw75l0pRdCCJNq6VUoQggh6iAJXAghTEoSuBBCmJQkcCGEMClJ4EIIYVKSwIUQwqQkgQshhEn9Pz19znZCsy5xAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#hide\n",
    "learn.recorder.plot_loss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Untrained \n",
    "Again, lets look at the predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "model2=learn.model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "      <th>score</th>\n",
       "      <th>token</th>\n",
       "      <th>sequence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Input text</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt;s&gt; I was walking to&lt;mask&gt; when I cam across a cat on the road&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>the</td>\n",
       "      <td>0.037963</td>\n",
       "      <td>5</td>\n",
       "      <td>&lt;s&gt; I was walking to the when I cam across a cat on the road&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>.</td>\n",
       "      <td>0.036504</td>\n",
       "      <td>4</td>\n",
       "      <td>&lt;s&gt; I was walking to. when I cam across a cat on the road&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>,</td>\n",
       "      <td>0.033266</td>\n",
       "      <td>6</td>\n",
       "      <td>&lt;s&gt; I was walking to, when I cam across a cat on the road&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>of</td>\n",
       "      <td>0.024381</td>\n",
       "      <td>9</td>\n",
       "      <td>&lt;s&gt; I was walking to of when I cam across a cat on the road&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         word     score  token  \\\n",
       "0  Input text  0.000000      0   \n",
       "1         the  0.037963      5   \n",
       "2           .  0.036504      4   \n",
       "3           ,  0.033266      6   \n",
       "4          of  0.024381      9   \n",
       "\n",
       "                                                             sequence  \n",
       "0  <s> I was walking to<mask> when I cam across a cat on the road</s>  \n",
       "1    <s> I was walking to the when I cam across a cat on the road</s>  \n",
       "2       <s> I was walking to. when I cam across a cat on the road</s>  \n",
       "3       <s> I was walking to, when I cam across a cat on the road</s>  \n",
       "4     <s> I was walking to of when I cam across a cat on the road</s>  "
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text2 = 'I was walking to <mask> when I cam across a cat on the road'\n",
    "pred2 = get_mask_pred(model2, text2);pred2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pretty bad 👎, and see how the unconfident it is in its predictions! This doesn't perform well because we have only used 800 movie reviews to train our model, we'll need a lot more text to get decent results!\n",
    "\n",
    "Again, just for fun, lets see how it does on a movie review, lets look at an example from our validation set. We mask the word `might` from the first sentence of the reivew, `... shows what might happen...` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "      <th>score</th>\n",
       "      <th>token</th>\n",
       "      <th>sequence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Input text</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt;s&gt; This very funny British comedy shows what&lt;mask&gt; happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes &amp; post-war restrictions. Merry mayhem is what would happen.&lt;br /&gt;&lt;br /&gt;The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.&lt;br /&gt;&lt;b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>,</td>\n",
       "      <td>0.044226</td>\n",
       "      <td>6</td>\n",
       "      <td>&lt;s&gt; This very funny British comedy shows what, happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes &amp; post-war restrictions. Merry mayhem is what would happen.&lt;br /&gt;&lt;br /&gt;The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.&lt;br /&gt;&lt;br /&gt;S...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>the</td>\n",
       "      <td>0.035027</td>\n",
       "      <td>5</td>\n",
       "      <td>&lt;s&gt; This very funny British comedy shows what the happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes &amp; post-war restrictions. Merry mayhem is what would happen.&lt;br /&gt;&lt;br /&gt;The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.&lt;br /&gt;&lt;br ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>.</td>\n",
       "      <td>0.028172</td>\n",
       "      <td>4</td>\n",
       "      <td>&lt;s&gt; This very funny British comedy shows what. happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes &amp; post-war restrictions. Merry mayhem is what would happen.&lt;br /&gt;&lt;br /&gt;The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.&lt;br /&gt;&lt;br /&gt;S...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>and</td>\n",
       "      <td>0.025764</td>\n",
       "      <td>8</td>\n",
       "      <td>&lt;s&gt; This very funny British comedy shows what and happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes &amp; post-war restrictions. Merry mayhem is what would happen.&lt;br /&gt;&lt;br /&gt;The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.&lt;br /&gt;&lt;br ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         word     score  token  \\\n",
       "0  Input text  0.000000      0   \n",
       "1           ,  0.044226      6   \n",
       "2         the  0.035027      5   \n",
       "3           .  0.028172      4   \n",
       "4         and  0.025764      8   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  sequence  \n",
       "0  <s> This very funny British comedy shows what<mask> happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes & post-war restrictions. Merry mayhem is what would happen.<br /><br />The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.<br /><b...  \n",
       "1  <s> This very funny British comedy shows what, happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes & post-war restrictions. Merry mayhem is what would happen.<br /><br />The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.<br /><br />S...  \n",
       "2  <s> This very funny British comedy shows what the happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes & post-war restrictions. Merry mayhem is what would happen.<br /><br />The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.<br /><br ...  \n",
       "3  <s> This very funny British comedy shows what. happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes & post-war restrictions. Merry mayhem is what would happen.<br /><br />The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.<br /><br />S...  \n",
       "4  <s> This very funny British comedy shows what and happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes & post-war restrictions. Merry mayhem is what would happen.<br /><br />The explosion of a wartime bomb leads to the discovery of ancient documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.<br /><br ...  "
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mask_indices=[7]\n",
    "txts=df.text.values\n",
    "masked_text = txts[800].split(' ')  # our validation split starts at index 800\n",
    "masked_text[mask_indices[0]] = '<mask>'\n",
    "masked_text = \" \".join(masked_text)\n",
    "\n",
    "pred1 = get_mask_pred(model2, masked_text);pred1.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ewww.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "      <th>score</th>\n",
       "      <th>token</th>\n",
       "      <th>sequence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Input text</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt;s&gt; This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes &amp; post-war restrictions. Merry mayhem is what would happen.&lt;br /&gt;&lt;br /&gt;The explosion of a wartime bomb leads to the discovery of&lt;mask&gt; documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.&lt;br /&gt;&lt;br ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>the</td>\n",
       "      <td>0.036510</td>\n",
       "      <td>5</td>\n",
       "      <td>&lt;s&gt; This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes &amp; post-war restrictions. Merry mayhem is what would happen.&lt;br /&gt;&lt;br /&gt;The explosion of a wartime bomb leads to the discovery of the documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.&lt;br /&gt;&lt;br /&gt;...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>,</td>\n",
       "      <td>0.035627</td>\n",
       "      <td>6</td>\n",
       "      <td>&lt;s&gt; This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes &amp; post-war restrictions. Merry mayhem is what would happen.&lt;br /&gt;&lt;br /&gt;The explosion of a wartime bomb leads to the discovery of, documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.&lt;br /&gt;&lt;br /&gt;Sta...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>and</td>\n",
       "      <td>0.029176</td>\n",
       "      <td>8</td>\n",
       "      <td>&lt;s&gt; This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes &amp; post-war restrictions. Merry mayhem is what would happen.&lt;br /&gt;&lt;br /&gt;The explosion of a wartime bomb leads to the discovery of and documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.&lt;br /&gt;&lt;br /&gt;...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>.</td>\n",
       "      <td>0.029063</td>\n",
       "      <td>4</td>\n",
       "      <td>&lt;s&gt; This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes &amp; post-war restrictions. Merry mayhem is what would happen.&lt;br /&gt;&lt;br /&gt;The explosion of a wartime bomb leads to the discovery of. documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.&lt;br /&gt;&lt;br /&gt;Sta...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         word     score  token  \\\n",
       "0  Input text  0.000000      0   \n",
       "1         the  0.036510      5   \n",
       "2           ,  0.035627      6   \n",
       "3         and  0.029176      8   \n",
       "4           .  0.029063      4   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  sequence  \n",
       "0  <s> This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes & post-war restrictions. Merry mayhem is what would happen.<br /><br />The explosion of a wartime bomb leads to the discovery of<mask> documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.<br /><br ...  \n",
       "1  <s> This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes & post-war restrictions. Merry mayhem is what would happen.<br /><br />The explosion of a wartime bomb leads to the discovery of the documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.<br /><br />...  \n",
       "2  <s> This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes & post-war restrictions. Merry mayhem is what would happen.<br /><br />The explosion of a wartime bomb leads to the discovery of, documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.<br /><br />Sta...  \n",
       "3  <s> This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes & post-war restrictions. Merry mayhem is what would happen.<br /><br />The explosion of a wartime bomb leads to the discovery of and documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.<br /><br />...  \n",
       "4  <s> This very funny British comedy shows what might happen if a section of London, in this case Pimlico, were to declare itself independent from the rest of the UK and its laws, taxes & post-war restrictions. Merry mayhem is what would happen.<br /><br />The explosion of a wartime bomb leads to the discovery of. documents which show that Pimlico was ceded to the Duchy of Burgundy centuries ago, a small historical footnote long since forgotten. To the new Burgundians, however, this is an unexpected opportunity to live as they please, free from any interference from Whitehall.<br /><br />Sta...  "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mask_indices=[54]\n",
    "txts=df.text.values\n",
    "masked_text = txts[800].split(' ')  # our validation split starts at index 800\n",
    "masked_text[mask_indices[0]] = '<mask>'\n",
    "masked_text = \" \".join(masked_text)\n",
    "\n",
    "pred1 = get_mask_pred(model2, masked_text);pred1.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yuck!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notes & Hacky Bits\n",
    "\n",
    "#### Notes\n",
    "- The validation set will change slightly due to random masking. While the data in the validaion set remains constant, different tokens will be masked each time the validation dataloader is called due to `MLMTokensLabels` calling a random probability each time. \n",
    "    - If a perfectly reproducable validation set is needed then you'll probably have to create a separate transform for it's masking and set it's `split_idx = 1`.  \n",
    "    \n",
    "- I didn't have time to get `learn.predict` working. One issue that needs to be fixed is that `MLMTokensLabels` transform shouldn't be called on your masked input text as it will add more masks, which you don't want.\n",
    "\n",
    "- `FastHugsTokenizer` will have to be modified to:\n",
    "    - enable sequence lengths larger than the tokenizer default\n",
    "    - to use a non-pretrained tokenizer (e.g. one you trained yourself)\n",
    "\n",
    "- The HuggingFace `encode_plus` or `batch_encode_plus` functions are great and I would have used them, but don't play nice with fastai multiprocessiing\n",
    "\n",
    "\n",
    "#### Hacks\n",
    "- I had to overwrite `__getitem__` in the `Datasets` class so that it wouldn't return a tuple as what it thinks is our `x` is actually our `(x,y)`. Wrapping this tuple in anoother tuple causes headaches down the line. Creating a custom `Datasets` class and inheriting from it didn't work as `learn.predict` calls on `Datasets` and not the custom dataset class.\n",
    "\n",
    "- The function `get_mask_pred` (used to view predictions of masked text) is mostly code from `FillMaskPipeline` in HuggingFace's Transformers repo, full credit to them!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Give me a shout 📣\n",
    "Thats it for this, I hope you found it useful and learned a thing or two. If you have any questions or would like to get in touch you can find me on Twitter [@mcgenergy](www.twitter.com/mcgenergy)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}