checkable_tweets_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Q2FtGCN6bVsg"
   },
   "source": [
    "# Finding Fact-checkable Tweets with Machine Learning\n",
    "\n",
    "This notebook was copied and modified from one originally created by Jeremy Howard and the other folks at [fast.ai](https://fast.ai) as part of [this fantastic class](https://course.fast.ai/). Specifically, it comes from Lesson 4. You can [see the lession video](https://course.fast.ai/videos/?lesson=4) and [the original class notebook](https://github.com/fastai/course-v3/blob/master/nbs/dl1/lesson3-imdb.ipynb). \n",
    "\n",
    "For more information about this project, and details about how to use this work in the wild, check out our [Quartz AI Studio blog post about the checkable-tweets project](https://qz.ai/?p=89).\n",
    "\n",
    "-- John Keefe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iPm37jhhPCVo"
   },
   "source": [
    "## Using this notebook\n",
    "\n",
    "You can run a [version of this notebook live](https://colab.research.google.com/drive/1Hq1OAe_-zLWVw8I_ZJO1euDxaCKPQzqe#scrollTo=Q2FtGCN6bVsg) using Google Colaboratory.\n",
    "\n",
    "That way you can use Colab's \"GPU runtime,\" to process data and train models faster. Alternatively, you can copy this repo onto a computer with a GPU or try running it on a (slower) CPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "mHe4vsWQbVsj"
   },
   "outputs": [],
   "source": [
    "# These lines set up some preferences\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "Rkdn_3eMbVsn"
   },
   "outputs": [],
   "source": [
    "# Now we get fast.ai\n",
    "from fastai.text import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9TPvnzypbVsq"
   },
   "source": [
    "### Take a peek at the tweet data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "I21aOI9xbVsr"
   },
   "source": [
    "Working with Dan Keemahill and Madlin Mekelburg over a couple of weeks during the 2019 Texas state legislative session, I have have a set of 3,797 tweets humans at the Austin American-Statesman have determined are – or are not – statements that can be fact-checked. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "kWwuVUg9QHo8",
    "outputId": "703590d9-71c2-4d1b-96cf-aa0a113be58a"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>tweet_text</th>\n",
       "      <th>checkable</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>TOMORROW: Dallas Workers Head to TX Capitol to...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Texas House members told to stop secretly reco...</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>SB 1163 - Relating to establishing and funding...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>Did you know that we’re carbon neutral* when i...</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>Check our this article on the Baylor sexual as...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0                                         tweet_text  checkable\n",
       "0           0  TOMORROW: Dallas Workers Head to TX Capitol to...      False\n",
       "1           1  Texas House members told to stop secretly reco...       True\n",
       "2           2  SB 1163 - Relating to establishing and funding...      False\n",
       "3           3  Did you know that we’re carbon neutral* when i...       True\n",
       "4           4  Check our this article on the Baylor sexual as...      False"
      ]
     },
     "execution_count": 5,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Here I read the csv into a data frame I called `austin_tweets`\n",
    "# and take a look at the first few rows\n",
    "path = Path('./data')\n",
    "austin_tweets = pd.read_csv(path/'hand_coded_austin_tweets.csv')\n",
    "austin_tweets.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eym6ulq0bVuz"
   },
   "source": [
    "## Building the language model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "62pxEwWPbVvI"
   },
   "source": [
    "Before we make a model that classifies whether tweets are checkable, we're going to build a model that 'understands' the rules of English – the language model. \n",
    "\n",
    "Even several thousand tweets isn't enough to teach a computer patterns of English, so we'll start with a language model pretrained on a thousands of Wikipedia articles called [wikitext-103](https://einstein.ai/research/blog/the-wikitext-long-term-dependency-language-modeling-dataset). That language model has been trained to guess the next word in a sentence based on all the previous words. It has a recurrent structure with a hidden state that is updated each time it sees a new word. This hidden state thus contains information about the sentence up to that point.\n",
    "\n",
    "We'll take that Wikitext model and fine-tune it for our particular dataset–the #txlege tweets. Because the English of #txlege tweets isn't the same as the English of Wikipedia, we'll adjust the internal parameters of the model by a little bit. That includes adding words that might be extremely common in the tweets but would be barely present in wikipedia–and therefore might not be part of the vocabulary the model was trained on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FT0C6-vpbVvK"
   },
   "source": [
    "### Adding more tweets for the language model\n",
    "\n",
    "We'll use the text in the 3,797 tweets we already have to help the language model better \"understand\" our data set. To give it even more examples, I collected several days worth of (uncategorized) #txlege tweets, which are in a file called `tweet_corpus.txt`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "18kBCGjhbVvQ",
    "outputId": "e36434d1-d47d-44c0-c6ab-82a61843ad2a"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>State Rep. @CBellJr earned an F on this year's...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>#txlege https://t.co/1qqa0sFEnY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ICYMI: @dallasnews editorial weighs in on the ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>My life is complete! #txlege #FlagDay #mindblo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>On June 14, 1777, our nation adopted the U.S. ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          tweet_text\n",
       "0  State Rep. @CBellJr earned an F on this year's...\n",
       "1                    #txlege https://t.co/1qqa0sFEnY\n",
       "2  ICYMI: @dallasnews editorial weighs in on the ...\n",
       "3  My life is complete! #txlege #FlagDay #mindblo...\n",
       "4  On June 14, 1777, our nation adopted the U.S. ..."
      ]
     },
     "execution_count": 6,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# read in the corpus, which has one tweet per row,\n",
    "# and take a look at the first frew rows\n",
    "corpus_tweets = pd.read_csv(path/'tweet_corpus.txt')\n",
    "corpus_tweets.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "aQyQIh_vNhrH",
    "outputId": "9a6afe30-01c4-4445-d7a7-8c835d7d62fc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3797, 3688, 7485)"
      ]
     },
     "execution_count": 7,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# here I concatenate the two tweet sets\n",
    "lm_tweets = pd.concat([austin_tweets,corpus_tweets], sort=True)\n",
    "\n",
    "# as a sanity check, let's look at the size of each set, \n",
    "# and then the ontatenated set\n",
    "len(austin_tweets), len(corpus_tweets), len(lm_tweets),"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "g55U1v3mbVvm"
   },
   "source": [
    "Great: Now we have 7,485 tweets to use for the language model.\n",
    "\n",
    "One thing to note ... the first set had two columns, `checkable` and `tweet_text`, while the corpus had just one collumn,  `tweet_text`. The combined has the original two columns, though many of the entries will be `NaN` for \"not a number.\" Thats okay, because we're only going to use the `tweet_text` column for the language model.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "LPVLlF5sbVvn",
    "outputId": "5aba630c-8742-4147-b818-d4d32dff8e2d"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>checkable</th>\n",
       "      <th>tweet_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3683</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>From our friends at @ProgressTX: One of the to...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3684</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>@GregAbbott_TX The Texas House failed Texans. ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3685</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Good to see one recommendation from Gov. Abbot...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3686</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Our executive director @antgutierrez spoke to ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3687</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Unnamed: 0 checkable                                         tweet_text\n",
       "3683         NaN       NaN  From our friends at @ProgressTX: One of the to...\n",
       "3684         NaN       NaN  @GregAbbott_TX The Texas House failed Texans. ...\n",
       "3685         NaN       NaN  Good to see one recommendation from Gov. Abbot...\n",
       "3686         NaN       NaN  Our executive director @antgutierrez spoke to ...\n",
       "3687         NaN       NaN                                                NaN"
      ]
     },
     "execution_count": 8,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show the last few rows ... and we can see NaNs in the `checkable` column, \n",
    "# which is ok here because we don't use that column in the language model\n",
    "lm_tweets.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "pFp92p9NbVvs"
   },
   "outputs": [],
   "source": [
    "# But we can't have NaN in the 'tweet_text' column, because it will make things unhappy.\n",
    "# Let's fix that:\n",
    "lm_tweets.dropna(subset=['tweet_text'], inplace=True)\n",
    "\n",
    "# Saving as csv for easier reading in a moment\n",
    "lm_tweets.to_csv(path/'lm_tweets.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3fU_R9lxbVv7"
   },
   "source": [
    "Fast.ai uses a concept called a \"[data bunch](https://docs.fast.ai/basic_data.html)\" to handle machine-learning data, which takes care of a lot of the more fickle machine-learning data preparation.\n",
    "\n",
    "We have to use a special kind of data bunch for the language model, one that ignores the labels, and will shuffle the texts at each epoch before concatenating them all together (only the training set gets shuffled; we don't shuffle for the validation set). It will also create batches that read the text in order with targets (aka the best guesses) that are the next word in the sentence.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "C2rJuWVCbVv5"
   },
   "outputs": [],
   "source": [
    "# Loading in data with the TextLMDataBunch factory class, using all the defaults\n",
    "data_lm = TextLMDataBunch.from_csv(path, 'lm_tweets.csv', text_cols='tweet_text', label_cols='checkable')\n",
    "data_lm.save('data_lm_tweets')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pwB-RJOjOSG_"
   },
   "source": [
    "### Quick digression: Let's look at the data\n",
    "\n",
    "The data bunch function does some nifty things behind the scenes. Let's take a moment to look at what is happening. (This is optional, but interesting!)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YMqNHzHabVub"
   },
   "source": [
    "#### Tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wLWA1H6VbVub"
   },
   "source": [
    "The first step of processing is to split the raw tweets into words, or more exactly tokens. The easiest way to do this would be to split the string on spaces, but fast.ai is smarter:\n",
    "\n",
    "- we need to take care of punctuation\n",
    "- some words are contractions of two different words, like isn't or don't\n",
    "- we may need to clean some parts of our texts, if there's HTML code for instance\n",
    "\n",
    "To see what the tokenizer had done behind the scenes, let's have a look at a few texts in a batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 323
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "QwRe7uZ1bVu5",
    "outputId": "b49a87ff-818d-476c-df73-836a7153c5a5"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>idx</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>/ t.co / oiavkdc6e5 # txlege xxbos xxmaj vetoes to date for the 86th # txlege session : xxup hb xxunk and sbs xxunk , xxunk , 511 , xxunk , xxunk . xxmaj this is already equal to the number vetoed from the xxunk session , and there are several days left to veto more bills . xxmaj yikes . xxmaj the last veto xxunk by the xxmaj texas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>xxmaj now enjoy that beer at your happy hour tonight and i ’ll see you after my marathon night on the xxmaj texas xxmaj house floor on the campaign trail ! # txlege # johnsonfordallas https : / / t.co / xxunk xxbos xxmaj happy xxmaj friday ! xxmaj here 's our latest xxmaj weekly xxmaj update ! \\n  xxmaj we 're at our 100th xxmaj day and we</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>solved because we can all become plumbers , ” said xxmaj sunset xxmaj vice xxmaj chair @chrispaddie https : / / t.co / xxunk # txlege xxbos “ xxmaj in a conference call with reporters , the researchers said when women abruptly lose health coverage so soon after giving birth , it can force them to xxunk medication and other ongoing treatment they may need , including support for postpartum</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td># xxunk to # xxunk , # xxunk , made calls , sent emails , showed up in # txlege offices to demand $ for xxup isd ’s to provide xxunk xxmaj now , we organize locally . xxmaj great job @edaustin_tx ! # teamtsta # redfored https : / / t.co / xxunk xxbos xxmaj thanks to the xxunk &amp; &amp; @repescobar for a xxunk xxunk on how we</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>college campuses that must end . https : / / t.co / xxunk xxbos xxmaj proud to stand with xxunk to offer xxunk xxunk to first time xxup xxunk offenders . xxmaj this bill would protect texans from repeat offenders while allowing for first time offenders to have recourse . # menendez4tx # txlege https : / / t.co / xxunk xxbos i ’m not saying that my son xxunk</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Note that language models can use a lot of GPU, here\n",
    "# If you're not using Google Colab you may need to decrease batch size\n",
    "bs=48\n",
    "data_lm = load_data(path, 'data_lm_tweets', bs=bs)\n",
    "data_lm.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CAlddd1CbVug"
   },
   "source": [
    "The texts are truncated at 100 tokens for more readability. We can see that it did more than just split on space and punctuation symbols: \n",
    "- the \"'s\" are grouped together in one token\n",
    "- the contractions are separated like this: \"did\", \"n't\"\n",
    "- content has been cleaned for any HTML symbol and lower cased\n",
    "- there are several special tokens (all those that begin by xx), to replace unknown tokens (see below) or to introduce different text fields (here we only have one)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pF6JV9osbVuh"
   },
   "source": [
    "#### Numericalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Nbllo71jbVui"
   },
   "source": [
    "Once we have extracted tokens from our texts, we convert to integers by creating a list of all the words used. We only keep the ones that appear at least twice with a maximum vocabulary size of 60,000 (by default) and replace the ones that don't make the cut by the unknown token `UNK`.\n",
    "\n",
    "The correspondance from ids to tokens is stored in the `vocab` attribute of our datasets, in a dictionary called `itos` (for int to string)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 187
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "ub9Wpy0VbVui",
    "outputId": "9d86f928-8baf-4d26-86cb-9a230aa1b8de"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['xxunk',\n",
       " 'xxpad',\n",
       " 'xxbos',\n",
       " 'xxeos',\n",
       " 'xxfld',\n",
       " 'xxmaj',\n",
       " 'xxup',\n",
       " 'xxrep',\n",
       " 'xxwrep',\n",
       " '/']"
      ]
     },
     "execution_count": 12,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_lm.vocab.itos[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2dHeF4qzbVup"
   },
   "source": [
    "And if we look at what a what's in our datasets, we'll see the tokenized text as a representation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 54
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "rirfNAiLbVup",
    "outputId": "be6d8556-6076-45a6-af2f-10c3c663d62c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text xxbos xxmaj the xxmaj texas xxmaj house gave initial passage to a bill to legalize broader use of medical marijuana this week . xxmaj but the bill still faces a major xxunk in the more - conservative xxmaj senate . https : / / t.co / oiavkdc6e5 # txlege"
      ]
     },
     "execution_count": 13,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_lm.train_ds[1][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lSYlLii3bVus"
   },
   "source": [
    "But the underlying data is all numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "pcaoh2nobVut",
    "outputId": "65463433-ce44-4b0a-e372-5a479096d824"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   2,    5,   12,    5,   25,    5,   51,  563, 2613, 1207])"
      ]
     },
     "execution_count": 14,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_lm.train_ds[1][0].data[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fbwXbu_6Tk7T"
   },
   "source": [
    "### Back to making the Language Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rY0LkJCQbVwF"
   },
   "source": [
    "We can then put all of our tweets (now stored in `data_lm`) in a learner object very easily with the Wikitext model loaded with the pretrained weights (here called `AWD_LTSM`). They'll be downloaded the first time you'll execute the following line and stored in `~/.fastai/models/`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "EiiXrA6zbVwH"
   },
   "outputs": [],
   "source": [
    "learn = language_model_learner(data_lm, AWD_LSTM, drop_mult=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RfAXUyLnOB_J"
   },
   "source": [
    "One of the most important settings when we actually _train_ our model is the **learning rate**. I'm not going to dive into it here (though I encourage you to explore it), but will use a fast.ai tool to find the best learning rate to start with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "W7b-Z94dbVwL",
    "outputId": "f6044993-11ab-4889-c0fe-892243e61a2e"
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 283
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "Un204GpfbVwO",
    "outputId": "607fdc89-064b-427b-e92e-f35bdc868923"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8XXWd//HXJ/vaJG3SfUn30kJb\n27RAKQgCZVFEFAdchtXpr4qIGy7jYxzE0Rm3cXBQEBAUFRyoFgW0FIulrJaUrkBbum+kSbeszf75\n/XFvJYRsbXLuyU3ez8fjPnrvOd977ufbe5N3vuec+z3m7oiIiJyshLALEBGR+KYgERGRblGQiIhI\ntyhIRESkWxQkIiLSLQoSERHpFgWJiIh0i4JERES6RUEiIiLdkhR2AScqPz/fCwsLwy5DRCSurF69\n+qC7FwSx7bgLksLCQoqLi8MuQ0QkrpjZrqC2rV1bIiLSLQoSERHpFgWJiIh0i4JERES6RUEiIiLd\noiAREZFuUZCIiEi3BPo9EjPLBe4DTgUcuMHdX2qxfgrwADAL+Ia7/zDIeuSd3J1D1fWUVdbR1By5\n5HKzOw1NTsWxBo4eq+dIdQPVdY0My01nwuAsxhdkkp2WHGnb7FTWNnL0WD3HGppoaHTqm5ppaGqm\ncFAmQ3PSwuyeiMRI0F9IvANY6u5XmlkKkNFq/WHgc8CHAq6j36ltaOJwdT2Ho0HxVnktJeXH2F9e\nS0l5LfuOHmP/0WPUNTaf8LYLslNpanaO1tQTzZ82zRqdy6WnDePS04YxPDe9G70Rkd7M3Dv4TdCd\nDZvlAGuBcd7Ji5jZbUBVV0YkRUVFrm+2R0YT28qqWP5GKTsOVnOwqo6DVfUcrKrjcHU9NfVN73pO\ngsHg7DSG5qQxIi+dEbmRW0F2KokJRoIZCQYJCUZOejJ5GSnkZSSTnpLI3iPH2FpaxdbSKnYerCYl\nKYG8jBRyM5LJzUghMyWR5MQEkpMSSEow1uw+wpMbSnjjrQoAhuekkZGaRGZKIukpiaQmJf7jNZMS\njNTkBAZmppCflcqgzBRy0pNpbHYaoiOcxmYnJTGBtORE0pITSU40ahuaqKhtpKq2kaq6RqrrGqmp\nb6K6vpGauibSUxIZlJnCoKxU8rNSKMhOZWhOGsMGpDMgPfI31KHqevYeOca+I8c4UFFL+bEGyo81\nUHGsgYQE4/SxA5k/MZ9hOe8MwvrGZo7W1FOQnYqZBf+Gi3STma1296JAth1gkMwE7gFeB2YAq4Fb\n3L26jba30UGQmNlCYCHA6NGjZ+/aFdg3/UPj7pRW1rHzYDWHoiOJozX1lB9rIC05kZz0ZHLSk8lM\nTWLN7iP8NRogEBkh5B//ZZmVSl5mCgMzUxiUmUJeZgr5WSkMy0lncHYqSYmxPSy242A1f9n4FttK\nqznWEPlFX1PXRF1jE03uNDVDU3MztQ3NHK6up6qu8aRfKzUpgczUJDJSEslISaSmvomDVXXUNrx7\n1JWWHPl/aGtddloSA9KSqW1o4lB1PQDjCzKZPjKXAxW17DpUw1vlx2h2GJSZQlFhHnMKBzKncCAT\nBmeRmRp3Mw9JPxCvQVIEvAyc5e5/N7M7gAp3/7c22t5GPxuRuDsvbjvE4tV7ebO0kh1l1VS3MYpI\nT06krrHpHbuQkhONM8fnc+HUIVxwyuB3/bUcz47vkis/1kByokVGOYkJJCYY9Y3N1DY0UdvQTH1T\nExkpSWSlRn7pZ6YmthuSNfWNHKysp7SylpKKyK69kvJaAEbmpTMyL4ORA9MZkp3GgPRkEhMiIwx3\nZ1NJJS9sPcjzWw+yuaSSYTlpjB6YwehBmeRlJLNhXznFO4+w+3DNP14vPyuVMYMyGDMog/eMyuXM\n8fmML8jUyEVCFa9BMhR42d0Lo4/PBr7m7u9vo+1t9JMgaW52/vrGAX66Yhvr9hxlYGYK04YPYHxB\nFuMKMikclElBdioDMyO7jVKTEiMHtesaqYjudhkzKOMfB7yldygpr2XN7iNsP1jN7kM17Dpczfay\nakor6wAYMiCVeePzOX3sQOaOHcjYfAWLxFaQQRLYGNzdS8xsj5lNdvfNwPlEdnP1G3sO17Bqx2GO\n1ER2VR2pqWf1riNsOVDF6IEZfPeK0/jwrBGkJSd2uJ3jxyxy0pMZFaPa5cQMzUnjktOGvWOZu7P7\ncA0vbjvEC1sPsnJLGUvW7AMgPyuFOYUDmTkql8lDs5kydABDBuh4i8SnwEYk8I/jJPcBKcB24Hrg\nKgB3vzs6aikGBgDNQBUw1d0r2ttmbx+RNDQ1s/yNUh5atZvn3izj+H9vUoKRl5nCyLx0rj2zkA9M\nHxbz4xUSrsgJEtWs2nGYV3YeZtWOw+w7euwf63PSkzlvcgHfeP9UCrJTQ6xU+qK43LUVlJMNkq2l\nVXxv6SY+fvpozplY8I/94MfVNjSxfm85pwzLPuHdRpW1Dfx9+2Fe2HaQJ9e/RWllHUMHpHHVnFF8\nYPowhuSkkZ2apL825V2OVNez+UAlm0sqeW1/OY+t2U9GaiK3XTaNy2cO12dGeoyCpIWTDZLlbxzg\nK4vXc6i6nhG56Xz89NFcetow1u89ylOvlbBicxk19U0MSEvimjMLuf6sQgZltf9XYWVtAw+v2s3S\njSWs21tOU7OTmpTA/An5fGzuaM6dXKARh5ywraWV3Lp4PWt2H+WCUwbznStOY8gAfbFTuk9B0kJ3\ndm3VNTax7LUDPPT33by0/dA/lhdkp7Jg6hDOGDeIJ9e/xVOvl5CalMBVRaO46NShTBk6gIGZKUDk\nL8gHXtzJL1/YQUVtIzNG5XL2hHzmTRjErNF5nR7vEOlMU7Nz//M7+OGyzTQ2O7NH5/HeyQW8d1IB\nU4cNICFBoxQ5cQqSFnrqGMn2sipWbC5jxqgc3jMq7x0/nFtLq/j5s9tYsmYfjdHzbgdnpzJhcBZr\n9xylpr6Ji6YN4abzJjB9ZG63axFpy86D1TxSvIdnt5Tx2v7IYcPB2alcetowLpsxnFmjc7XrS7pM\nQdJCLA+2H66uZ+O+cjaXVLKppJItByqZODiLReeOZ9KQ7JjUIAJQWlnLyi0HWfZaCSu2lFHf2MyI\n3HQ+OHM4n5o/tsPdsCKgIHmH3n7WlkjQKmobePq1Azyxfj8r3zxIRnIin33fBK47q5DUJO1albYp\nSFpQkIi87c0DlXz3z2/wt81ljBqYzlcvnsKlpw7TcRR5lyCDRKcVicSxiUOyeeD6ufz6xrlkpiTx\n2YfWcOGPn+XhVbupbXj3lDsiQdCIRKSPaGp2Hl+3n3uf285r+yvIz0rhn88o5Ib5hZpSR7RrqyUF\niUjH3J2Xth/ivud28MymUobnpPFfH5nOOZMKwi5NQqRdWyLSZWbGvPH53H/dHP7wmXmkpyRyzf2r\n+Ori9VTUNoRdnvRBChKRPmzW6Dye/NzZLHrveB5dvYcF/72SlVvKwi5L+hgFiUgfl5acyNcumcIf\nPnMWWWlJXHP/Kr79xOvUNepgvPQMBYlIPzFzVC5P3Dyfa88cwy+e38Hld77AlgOVYZclfYCCRKQf\nSUtO5FuXn8r91xVxsKqOy/73eR6LXiNF5GQpSET6ofdNGcJfbjmH94zO5YuPrOWJ9fvDLkniWKBB\nYma5ZrbYzDaZ2Rtmdmar9WZmPzGzrWa23sxmBVmPiLytIDuVB66bS9GYgXz+d2tZ/saBsEuSOBX0\niOQOYKm7TwFmAG+0Wn8JMDF6WwjcFXA9ItJCekoiv7iuiKnDB/Dp377KC1sPhl2SxKHAgsTMcoBz\ngF8AuHu9ux9t1exy4EGPeBnINbNhiEjMZKcl8+ANcxmXn8mnflXMKzsPh12SxJkgRyRjgTLgATNb\nY2b3mVlmqzYjgD0tHu+NLhORGMrNSOHXN57OsJw0PnHf31m8em/YJUkcCTJIkoBZwF3u/h6gGvja\nyWzIzBaaWbGZFZeV6ctUIkEoyE5l8afnUTQmjy8/uo7bH3+dxqbmsMuSOBBkkOwF9rr736OPFxMJ\nlpb2AaNaPB4ZXfYO7n6Puxe5e1FBgeYLEgnKwMwUHrxhLtefVcj9L+zgmvtXcaS6PuyypJcLLEjc\nvQTYY2aTo4vOB15v1exPwDXRs7fOAMrd/a2gahKRziUlJvDvl03jhx+dQfGuI1x594uU12iOLmlf\n0Gdt3Qz81szWAzOB75rZIjNbFF3/Z2A7sBW4F/hMwPWISBddOXskv7p+LrsP13DTQ6/SoN1c0g5N\nIy8iHXq0eA+3Ll7Px08fzXc+dCpmuvpiPApyGvmkIDYqIn3HR4tGsf1gNXet2MaEgixumD827JKk\nl1GQiEinbl0wme1lVfzHk68zNj+T86YMDrsk6UU015aIdCohwfjxVTOZOnwANz+8hq2lVWGXJL2I\ngkREuiQjJYl7/rmI1KQEFv1mNVV1jWGXJL2EgkREumx4bjr/+/H3sL2silsfXUe8nawjwVCQiMgJ\nmTc+n69fcgp/2VjCz1duD7sc6QUUJCJywj519ljeP30Y31+6STMGi4JERE6cmfH9j0xnfEEWNz+8\nhtKK2rBLkhApSETkpGSmJnHXJ2dTXdfIV3+/XsdL+jEFiYictAmDs/j6JVP42+YyfvfKns6fIH2S\ngkREuuWaMws5a8Igvv3E6+w6VB12ORICBYmIdEtCgvGDK2eQmGB86ZF1NDVrF1d/oyARkW4bnpvO\n7ZdPo3jXEe59TqcE9zcKEhHpER+aOYJLTxvKj5ZtZmtpZdjlSAwpSESkR5gZ3778VNKTE/n3P72m\ns7j6kUCDxMx2mtkGM1trZu+6iIiZ5ZnZEjNbb2arzOzUIOsRkWANykrlyxdN5oWth/jzhpKwy5EY\nicWI5Dx3n9nOBVX+FVjr7tOBa4A7YlCPiAToE6ePYeqwAfzHk69TrYkd+4Wwd21NBZ4BcPdNQKGZ\nDQm3JBHpjsQE4/bLp/FWeS0//dvWsMuRGAg6SBxYZmarzWxhG+vXAR8GMLO5wBhgZMA1iUjAigoH\n8uFZI7j3ue1sL9O1S/q6oINkvrvPAi4BbjKzc1qt/y8g18zWAjcDa4Cm1hsxs4VmVmxmxWVlZQGX\nLCI94WuXTCEtKZHbHn9dB977uECDxN33Rf8tBZYAc1utr3D36919JpFjJAXAu05Cd/d73L3I3YsK\nCgqCLFlEesjg7DQ+f+EkVm4p04H3Pi6wIDGzTDPLPn4fWABsbNUm18xSog8/Bax094qgahKR2Lr2\nzDFMGz6Af//Ta5TXNIRdjgQkyBHJEOB5M1sHrAKedPelZrbIzBZF25wCbDSzzUR2f90SYD0iEmNJ\niQl87yPTOVJTz3f//EbY5UhAkoLasLtvB2a0sfzuFvdfAiYFVYOIhO/UETl86uyx/PzZ7Vw+czjz\nJuSHXZL0sLBP/xWRfuDz509izKAMvr5kA8fq33U+jcQ5BYmIBC49JZH//PBp7DpUw/8s3xJ2OdLD\nFCQiEhPzxudzVdEo7ntuBxv3lYddjvQgBYmIxMy/XnoKeRnJfPOPG2nWdUv6DAWJiMRMTkYyX7l4\nCq/uPsqSNfvCLkd6iIJERGLqylkjmTkql//8yyYqavXdkr5AQSIiMZUQndTxUHUdP/nrm2GXIz1A\nQSIiMTd9ZC5XzxnFAy/uZMsBXU0x3ilIRCQUX14wmcyURG7T1RTjnoJEREJx/GqKL247xNKNmtQx\nnilIRCQ0H587momDs/jhss006XTguKUgEZHQJCUm8IULJ7GtrJo/rtXpwPFKQSIiobp42lBOGTaA\nO5a/SUNTc9jlyElQkIhIqBISjC9eOIldh2r4w6t7wy5HToKCRERCd8Epg5kxMoefLN9KfaNGJfFG\nQSIioTMzvnDhJPYdPcb/Fe8Juxw5QYEGiZntNLMNZrbWzIrbWJ9jZo+b2Toze83Mrg+yHhHpvd47\nqYDZY/L46TNbqW3QNUviSSxGJOe5+0x3L2pj3U3A6+4+AzgX+FGLa7iLSD9iZnzpwkmUVNTy8Krd\nYZcjJyDsXVsOZJuZAVnAYaAx3JJEJCzzJuRzxriB3LVim0YlcSToIHFgmZmtNrOFbay/EzgF2A9s\nAG5x93cdaTOzhWZWbGbFZWVlwVYsIqG65fxJlFbW8TuNSuJG0EEy391nAZcAN5nZOa3WXwSsBYYD\nM4E7zWxA6424+z3uXuTuRQUFBQGXLCJhOnP8IOaOHchdz2pUEi8CDRJ33xf9txRYAsxt1eR64A8e\nsRXYAUwJsiYR6f0+f/5EDlTU8X+v6AyueBBYkJhZppllH78PLAA2tmq2Gzg/2mYIMBnYHlRNIhIf\nzhw/iDmFedy1Yht1jRqV9HZBjkiGAM+b2TpgFfCkuy81s0Vmtija5tvAPDPbACwHvuruBwOsSUTi\ngJlxy/mRM7ge0aik10sKasPuvh2Y0cbyu1vc309kpCIi8g5nTRhE0Zg8frZiG/80ZxSpSYlhlyTt\nCPv0XxGRNpkZt1wwkbfKa3m0WHNw9WYKEhHpteZPyGf2mDzu1LfdezUFiYj0WmbGrRdNpqSill+9\nuDPscqQdChIR6dXOGDeIcycX8LMV2yivaQi7HGmDgkREer2vXDSFitoG7np2W9ilSBsUJCLS600d\nPoDLZwzngRd2UFJeG3Y50oqCRETiwpcWTKbZnTuWbwm7FGlFQSIicWHUwAw+cfoYHiney7ayqrDL\nkRYUJCISNz77vgmkJSXwo2Wbwy5FWuhSkJjZeDNLjd4/18w+Z2a5wZYmIvJO+VmpXH/WWP68oYSt\npZVhlyNRXR2R/B5oMrMJwD3AKOChwKoSEWnH9WcVkpacwM+f1fyuvUVXg6TZ3RuBK4D/dfdbgWHB\nlSUi0rZBWalcVTSKx9bu463yY2GXI3Q9SBrM7GPAtcAT0WXJwZQkItKxT509jmaHXzy3I+xShK4H\nyfXAmcB33H2HmY0Ffh1cWSIi7Rs1MIPLpg/joVW7OVpTH3Y5/V6XgsTdX3f3z7n7w2aWB2S7+/cC\nrk1EpF2Lzh1PTX0TD760K+xS+r2unrW1wswGmNlA4FXgXjP77y48b6eZbTCztWZW3Mb6W6Pr1prZ\nRjNrir6GiEiHpgwdwHmTC/jlizs5Vq+ZgcPU1V1bOe5eAXwYeNDdTwcu6OJzz3P3me5e1HqFu/8g\num4m8HXgWXc/3MXtikg/9+lzJ3C4up5HinUVxTB1NUiSzGwY8E+8fbC9p30MeDigbYtIHzSnMI9Z\no3O5Z+V2Gpuawy6n3+pqkNwOPAVsc/dXzGwc8GYXnufAMjNbbWYL22tkZhnAxUS+r9LW+oVmVmxm\nxWVlZV0sWUT6OjPj/713PPuOHmPpayVhl9NvmbsHt3GzEe6+z8wGA08DN7v7yjbaXQV80t0v62yb\nRUVFXlz8rsMtItJPNTU77/vRCnIzUnjsM/Mws7BL6pXMbHVbhxh6QlcPto80syVmVhq9/d7MRnb2\nPHffF/23FFgCzG2n6dVot5aInITEBOPG+WNZt+cor+4+EnY5/VJXd209APwJGB69PR5d1i4zyzSz\n7OP3gQXAxjba5QDvBf7Y9bJFRN525eyR5KQnc+9KfUExDF0NkgJ3f8DdG6O3XwIFnTxnCPC8ma0D\nVgFPuvtSM1tkZotatLsCWObu1SdcvYgIkJGSxMdPH81Tr5ew65B+lcRaV4PkkJl90swSo7dPAoc6\neoK7b3f3GdHbNHf/TnT53e5+d4t2v3T3q0++CyIicN28QpISjAde2Bl2Kf1OV4PkBiKn/pYAbwFX\nAtcFVJOIyAkbMiCNy6YP55HiPZTXNIRdTr/S1SlSdrn7B929wN0Hu/uHgI8EXJuIyAm58eyx1NQ3\n8fAru8MupV/pzhUSv9hjVYiI9IBpw3OYN34Qv3xhJ/WN+oJirHQnSHSytoj0Ov9y9jhKKmp5fN3+\nsEvpN7oTJMF9k1FE5CSdO7mAyUOy+fnKbTQ369dULHQYJGZWaWYVbdwqiXyfRESkV4lMmzKOLQeq\n+Nvm0rDL6Rc6DBJ3z3b3AW3cst09KVZFioiciMtmDGdEbrqu6x4j3dm1JSLSKyUnJnDj/LGs2nmY\n1bs0bUrQFCQi0iddNWcUOenJ3P3strBL6fMUJCLSJ2WmJnHtmWN4+vUDbC2tCrucPk1BIiJ91rXz\nCklLTuCelRqVBElBIiJ91qCsVP6paBRL1uzjQEVt2OX0WQoSEenTPjV/HI3NzoMv7Qy7lD5LQSIi\nfdroQRlcNHUov/37bmrqG8Mup09SkIhIn3fj2WM5WtPA71/dF3YpfVKgQWJmO81sg5mtNbM2L7Ru\nZudG179mZs8GWY+I9E9FY/KYMTKHB57foWlTAhCLEcl57j6zrYvOm1ku8DPgg+4+DfhoDOoRkX7G\nzLjx7HFsP1itaVMCEPaurY8Df3D33QDurndYRAJxyalDGZaTxi+e13Xde1rQQeLAMjNbbWYL21g/\nCcgzsxXRNtcEXI+I9FPJiQlcN6+QF7cd4rX95WGX06cEHSTz3X0WcAlwk5md02p9EjAbeD9wEfBv\nZjap9UbMbKGZFZtZcVlZWcAli0hfdfXc0WSkJGpU0sMCDRJ33xf9txRYAsxt1WQv8JS7V7v7QWAl\nMKON7dzj7kXuXlRQUBBkySLSh+WkJ/NPRaN4fN1+fUGxBwUWJGaWaWbZx+8DC4CNrZr9EZhvZklm\nlgGcDrwRVE0iIjecNZamZud+jUp6TJAjkiHA82a2DlgFPOnuS81skZktAnD3N4ClwPpom/vcvXXY\niIj0mNGDMrhsxnB+8/IujtbUh11On2Du8XVOdVFRkRcXt/mVFBGRLtlcUslF/7OSW86fyBcufNdh\n2T7JzFa39TWMnhD26b8iIjE3eWg2F04dwi9f3ElVnaZN6S4FiYj0S589bwLlxxr47cu7wi4l7ilI\nRKRfmjEql7Mn5nPvczuobWgKu5y4piARkX7rM+dO4GBVHY8W7wm7lLimIBGRfuuMcQOZPSaPu5/d\nTkNTc9jlxC0FiYj0W2bGZ8+bwL6jx/jj2v1hlxO3FCQi0q+dO7mAyUOyue+57cTb1yF6CwWJiPRr\nkSnmx7KppJLntx4Mu5y4pCARkX7v8pnDKchO5d7nNG3KyVCQiEi/l5qUyLVnjmHlljI2l1SGXU7c\nUZCIiACfOH0MackJ/OL57WGXEncUJCIiQF5mClfOHslja/ZTWqkp5k+EgkREJOrG+eNoaG7m1y9p\n2pQToSAREYkam5/JBacM4Tcv7+JYvaZN6SoFiYhIC/9y9jiO1DSw+NW9YZcSNwINEjPbaWYbzGyt\nmb3rIiJmdq6ZlUfXrzWzbwZZj4hIZ+YU5nHqiAH8+qWd+oJiF8ViRHKeu8/s4IIqz0XXz3T322NQ\nj4hIu8yMa84oZMuBKlbtOBx2OXFBu7ZERFq5bMZwBqQl8aCuVdIlQQeJA8vMbLWZLWynzZlmts7M\n/mJm0wKuR0SkU+kpiXy0aBRPbSyhtEKnAncm6CCZ7+6zgEuAm8zsnFbrXwXGuPsM4H+Bx9raiJkt\nNLNiMysuKysLtmIREeCTZ4yhsdl5eJWuVdKZQIPE3fdF/y0FlgBzW62vcPeq6P0/A8lmlt/Gdu5x\n9yJ3LyooKAiyZBERIHIq8NkT83lo1S5dq6QTgQWJmWWaWfbx+8ACYGOrNkPNzKL350brORRUTSIi\nJ+KaMws5UFHH8jcOhF1Kr5YU4LaHAEuiOZEEPOTuS81sEYC73w1cCXzazBqBY8DVrvPtRKSXeN+U\nwYzITefBl3Zx8anDwi6n1wosSNx9OzCjjeV3t7h/J3BnUDWIiHRHYoLx8dNH84OnNrO1tJIJg7PD\nLqlX0um/IiIduGrOKFISE/jlizvDLqXXUpCIiHQgPyuVK4tG8n+v7GH3oZqwy+mVFCQiIp245fyJ\nJJjx479uCbuUXklBIiLSiSED0rj+rLE8tnYfb7xVEXY5vY6CRESkCz793vFkpybxw6c2h11Kr6Mg\nERHpgpyMZBadO57lm0p5Zacmc2xJQSIi0kXXzxvL4OxUvveXTZpivgUFiYhIF6WnJPK58ydSvOsI\nz2wqDbucXkNBIiJyAq6aM4rCQRn84KnNNDdrVAIKEhGRE5KcmMDnL5jEppJK/rKxJOxyegUFiYjI\nCbpsxnDGF2TyP3/dQpNGJQoSEZETlZhgfP6CSbxZWsUT6/eHXU7oFCQiIifh/acNY/KQbO7465s0\n9vPrlShIREROQkKC8YULJ7L9YDV/Wte/RyUKEhGRk7Rg6lCmDhvAHcvf7NdXUVSQiIicpIQE44sX\nTmLXoRqWvLov7HJCE2iQmNlOM9tgZmvNrLiDdnPMrNHMrgyyHhGRnnb+KYOZPjKHO5a/SV1jU9jl\nhCIWI5Lz3H2muxe1tdLMEoHvActiUIuISI8yM768YDL7jh7job/vDrucUPSGXVs3A78HNN+AiMSl\nsyfmM2/8IO58ZitVdY1hlxNzQQeJA8vMbLWZLWy90sxGAFcAd3W0ETNbaGbFZlZcVlYWUKkiIifH\nzPjKxVM4VF3PL57bEXY5MRd0kMx391nAJcBNZnZOq/X/A3zV3Ts83cHd73H3IncvKigoCKpWEZGT\nNnNULhdPG8o9K7dxqKou7HJiKtAgcfd90X9LgSXA3FZNioDfmdlO4ErgZ2b2oSBrEhEJypcvmsyx\nhiZ++rdtYZcSU4EFiZllmln28fvAAmBjyzbuPtbdC929EFgMfMbdHwuqJhGRIE0YnMVHZ4/iNy/v\nYu+RmrDLiZkgRyRDgOfNbB2wCnjS3Zea2SIzWxTg64qIhOaWCyaCwY+ffjPsUmImKagNu/t2YEYb\ny+9up/11QdUiIhIrw3PTuW5eIfc+t52r545iTuHAsEsKXG84/VdEpE+55fyJjMhN56uL11Pb0Pe/\npKggERHpYZmpSXzvI9PZfrCaH/91S9jlBE5BIiISgLMm5POxuaO4d+V21u05GnY5gVKQiIgE5OuX\nnsLg7DRuXbyuT8/DpSAREQnIgLRkvvvhU9lyoIqfPrM17HICoyAREQnQ+6YM4cPvGcHPVmzjtf3l\nYZcTCAWJiEjAvnnZVHIzUrj10fV98gJYChIRkYDlZqTwHx86ldffquDnz/a96VMUJCIiMXDxqUN5\n//Rh/GT5VrYcqAy7nB6lIBEMyuFLAAAKqElEQVQRiZHbPziNrLQkbn10HY19aBeXgkREJEYGZaXy\nrQ9OY93ecn7xfN+5bomCREQkhj4wfRgLpg7hR09vYVNJRdjl9AgFiYhIDJkZ37niNPIykln44GqO\n1tSHXVK3KUhERGKsIDuVuz45m5LyWm5+eA1NzR52Sd2iIBERCcGs0Xncfvk0nnvzIN9/alPY5XRL\noEFiZjvNbIOZrTWz4jbWX25m64+vN7P5QdYjItKbXD13NJ88YzQ/f3Y7j6/bH3Y5Jy2wC1u1cJ67\nH2xn3XLgT+7uZjYdeASYEoOaRER6hW9+YBqbSyq5dfE6stKSOG/y4LBLOmGh7tpy9yp3P75zMBOI\n7x2FIiInKCUpgZ99YjaFgzK5/oFX+Nbjr8XdxbCCDhIHlpnZajNb2FYDM7vCzDYBTwI3BFyPiEiv\nU5CdymM3ncV18wp54IWdfOinL7zj2++NTc0cramnorYhxCrbZ28PCALYuNkId99nZoOBp4Gb3X1l\nO23PAb7p7he0sW4hsBBg9OjRs3ft2hVYzSIiYXpm0wFufXQ9lXWNDMpMoeJYA9X1kRHKZ84dz1cu\nPrm9/2a22t2LerLWf2w7yCB5xwuZ3QZUufsPO2izHZjbwTEVioqKvLj4XcftRUT6jNLKWu58Ziu1\nDU1kpyUzIC2Z7LQkZo7OZdbovJPaZpBBEtjBdjPLBBLcvTJ6fwFwe6s2E4Bt0YPts4BU4FBQNYmI\nxIPB2WncfvmpYZfRZUGetTUEWGJmx1/nIXdfamaLANz9buAjwDVm1gAcA67yWA2RRESkR8Rs11ZP\n0a4tEZETF+SuLX2zXUREukVBIiIi3aIgERGRblGQiIhItyhIRESkWxQkIiLSLXF3+q+ZlQGt50jJ\nAco7WdbR4+P3Wy7LB9r9hn0n2qrnRNqcaH86u9+dvnRWa2dt+tJ705W+tF4W5Hujz1nHy+P1c9be\nuu6+N5nuXtBp5SfD3eP+BtzT2bKOHh+/32pZcU/WcyJtTrQ/nd3vTl+625++9N50pS+xfG/0Oeub\nn7Pe+N50dusru7Ye78Kyjh4/3k6bnqznRNqcaH+6cr87utOfvvTedKUvrZcF+d7oc9bx8nj9nLW3\nLsz3pkNxt2srVsys2AP6Fmis9aW+QN/qj/rSe/Wl/gTdl74yIgnCPWEX0IP6Ul+gb/VHfem9+lJ/\nAu2LRiQiItItGpGIiEi39PkgMbP7zazUzDaexHNnm9kGM9tqZj+x6Jz40XU3m9kmM3vNzL7fs1V3\nWFOP98fMbjOzfWa2Nnq7tOcrb7OeQN6b6PovmZmbWX7PVdxpTUG8N982s/XR92WZmQ3v+crbrCeI\nvvwg+jOz3syWmFluz1febk1B9Oej0Z//ZjML/FhKd/rQzvauNbM3o7drWyzv8GerTUGeEtYbbsA5\nwCxg40k8dxVwBmDAX4BLosvPA/4KpEYfD47z/twGfLkvvDfRdaOAp4h83yg/nvsDDGjR5nPA3XHc\nlwVAUvT+94Dvxfl7cwowGVgBFPXWPkTrK2y1bCCwPfpvXvR+Xkf97ejW50ckHrlG/OGWy8xsvJkt\nNbPVZvacmb3rIshmNozID/HLHvnffRD4UHT1p4H/cve66GuUBtuLtwXUn1AE2JcfA18BYnoAMIj+\nuHtFi6aZxKhPAfVlmbs3Rpu+DIwMthdvC6g/b7j75ljUH329k+pDOy4Cnnb3w+5+BHgauPhkf0/0\n+SBpxz3Aze4+G/gy8LM22owA9rZ4vDe6DGAScLaZ/d3MnjWzOYFW27nu9gfgs9FdDveb2cldFLpn\ndKsvZnY5sM/d1wVdaBd1+70xs++Y2R7gE8A3A6y1Mz3xOTvuBiJ/7YapJ/sTlq70oS0jgD0tHh/v\n10n1N8hL7fZKZpYFzAMebbHrL/UEN5NEZEh4BjAHeMTMxkUTPKZ6qD93Ad8m8tfut4EfEflBj6nu\n9sXMMoB/JbILJXQ99N7g7t8AvmFmXwc+C/x7jxXZRT3Vl+i2vgE0Ar/tmepOqoYe609YOuqDmV0P\n3BJdNgH4s5nVAzvc/YqerqXfBQmRUdhRd5/ZcqGZJQKrow//ROSXa8uh90hgX/T+XuAP0eBYZWbN\nROayKQuy8HZ0uz/ufqDF8+4Fngiy4A50ty/jgbHAuugP1kjgVTOb6+4lAdfelp74rLX0W+DPhBAk\n9FBfzOw64APA+WH84dVCT783YWizDwDu/gDwAICZrQCuc/edLZrsA85t8XgkkWMp+ziZ/gZ9gKg3\n3IBCWhygAl4EPhq9b8CMdp7X+qDTpdHli4Dbo/cnERkiWhz3Z1iLNl8AfhevfWnVZicxPNge0Hsz\nsUWbm4HFcdyXi4HXgYJYvidBf9aI0cH2k+0D7R9s30HkQHte9P7ArvS3zbrCeENj/OF5GHgLaCAy\nkriRyF+tS4F10Q/2N9t5bhGwEdgG3MnbX+BMAX4TXfcq8L4478+vgQ3AeiJ/hQ2L1760arOT2J61\nFcR78/vo8vVE5k0aEcd92Urkj6610VtMzkALsD9XRLdVBxwAnuqNfaCNIIkuvyH6nmwFru+svx3d\n9M12ERHplv561paIiPQQBYmIiHSLgkRERLpFQSIiIt2iIBERkW5RkEifYGZVMX69+8xsag9tq8ki\ns/tuNLPHO5sV18xyzewzPfHaIj1Bp/9Kn2BmVe6e1YPbS/K3JxgMVMvazexXwBZ3/04H7QuBJ9z9\n1FjUJ9IZjUikzzKzAjP7vZm9Er2dFV0+18xeMrM1ZvaimU2OLr/OzP5kZs8Ay83sXDNbYWaLLXId\njd8evzZDdHlR9H5VdGLFdWb2spkNiS4fH328wcz+o4ujppd4ewLKLDNbbmavRrdxebTNfwHjo6OY\nH0Tb3hrt43oz+1YP/jeKdEpBIn3ZHcCP3X0O8BHgvujyTcDZ7v4eIrPpfrfFc2YBV7r7e6OP3wN8\nHpgKjAPOauN1MoGX3X0GsBL4lxavf4e7n8Y7Z1RtU3Sep/OJzC4AUAtc4e6ziFwD50fRIPsasM3d\nZ7r7rWa2AJgIzAVmArPN7JzOXk+kp/THSRul/7gAmNpiZtQB0RlTc4BfmdlEIjMeJ7d4ztPu3vKa\nD6vcfS+Ama0lMtfR861ep563J7pcDVwYvX8mb1/L4SHgh+3UmR7d9gjgDSLXhoDIXEffjYZCc3T9\nkDaevyB6WxN9nEUkWFa283oiPUpBIn1ZAnCGu9e2XGhmdwJ/c/croscbVrRYXd1qG3Ut7jfR9s9M\ng799sLG9Nh055u4zo9PgPwXcBPyEyPVHCoDZ7t5gZjuBtDaeb8B/uvvPT/B1RXqEdm1JX7aMyIy5\nAJjZ8em2c3h7auzrAnz9l4nsUgO4urPG7l5D5HK6XzKzJCJ1lkZD5DxgTLRpJZDd4qlPATdER1uY\n2QgzG9xDfRDplIJE+ooMM9vb4vZFIr+Ui6IHoF8nMv0/wPeB/zSzNQQ7Kv888EUzW0/k4kLlnT3B\n3dcQmen3Y0SuP1JkZhuAa4gc28HdDwEvRE8X/oG7LyOy6+ylaNvFvDNoRAKl039FAhLdVXXM3d3M\nrgY+5u6Xd/Y8kXijYyQiwZkN3Bk90+ooIVy+WCQWNCIREZFu0TESERHpFgWJiIh0i4JERES6RUEi\nIiLdoiAREZFuUZCIiEi3/H/YjwEuUj/3pgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot(skip_end=15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "td3D3mK8TtHa"
   },
   "source": [
    "This gives us a graph of the optimal learning rate ... which is the point where the graph really dives downward (`1e-02`). Again, there's much more on picking and learning rates in the fast.ai course."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BQA3dZWPachE"
   },
   "source": [
    "Now we can train the Language Model. (Essentailly, we're training it to be good at guessing the *next word* in a sentence, given all of the previous words.)\n",
    "\n",
    "The variabales we're passing are `1` to just do one cycle of learning, the learning rate of `1e-2`, and some momentum settings we won't get into here -- but these are pretty safe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 80
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "pveLnA6kbVwQ",
    "outputId": "38ef873c-5c2a-461d-9688-d8f4e1edbd5f"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.192750</td>\n",
       "      <td>3.598085</td>\n",
       "      <td>0.369016</td>\n",
       "      <td>00:25</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, 1e-2, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 80
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "o-dYIVFcbVwS",
    "outputId": "aa2db805-6429-4592-f905-0479b9872820"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.981636</td>\n",
       "      <td>3.663427</td>\n",
       "      <td>0.369721</td>\n",
       "      <td>00:25</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, 1e-1, moms=(0.8,0.7))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "yMTSnTqIbVwU"
   },
   "outputs": [],
   "source": [
    "# optionally save and reload the model (file is about 150MB)\n",
    "learn.save('fit_head')\n",
    "learn.load('fit_head');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "M8kZOKSybVwX"
   },
   "source": [
    "To complete the fine-tuning, we \"unfreeze\" the original Wikitext language model and let the new training efforts -- work their way into the original neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 359
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "kTfuNCuhbVwX",
    "outputId": "1e99e6bc-8421-4d63-faf3-4c16fb3f07f6"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.221620</td>\n",
       "      <td>3.594337</td>\n",
       "      <td>0.380122</td>\n",
       "      <td>00:31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.098113</td>\n",
       "      <td>3.497657</td>\n",
       "      <td>0.391009</td>\n",
       "      <td>00:31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.900118</td>\n",
       "      <td>3.467964</td>\n",
       "      <td>0.397901</td>\n",
       "      <td>00:31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.594839</td>\n",
       "      <td>3.485903</td>\n",
       "      <td>0.401159</td>\n",
       "      <td>00:31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.229434</td>\n",
       "      <td>3.589874</td>\n",
       "      <td>0.398919</td>\n",
       "      <td>00:31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.875757</td>\n",
       "      <td>3.713223</td>\n",
       "      <td>0.398731</td>\n",
       "      <td>00:31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.575312</td>\n",
       "      <td>3.859700</td>\n",
       "      <td>0.395990</td>\n",
       "      <td>00:31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.386395</td>\n",
       "      <td>3.937915</td>\n",
       "      <td>0.395097</td>\n",
       "      <td>00:31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.247221</td>\n",
       "      <td>4.008004</td>\n",
       "      <td>0.394095</td>\n",
       "      <td>00:31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.155052</td>\n",
       "      <td>4.024599</td>\n",
       "      <td>0.393687</td>\n",
       "      <td>00:31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# This takes about 5 minutes!\n",
    "learn.unfreeze()\n",
    "learn.fit_one_cycle(10, 1e-3, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "xq3OgYK-bVwb"
   },
   "outputs": [],
   "source": [
    "# optionally save what we have - file is about 300MB\n",
    "learn.save('fine_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fv0mZuVhbVwd"
   },
   "source": [
    "While 39% accuracy may _seem_ low ... in this case it means the language model correctly guessed the next word in a sentence more than 1/3 of the time. That's pretty good! And we can see that even when it's wrong, it makes some pretty \"logical\" guesses. \n",
    "\n",
    "Let's give it a starting phrase and see how it does:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 88
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "_Wws03kmbVwd",
    "outputId": "dd545482-c5c8-4e93-a79e-64578e6478a3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I wonder if this is any chance you ’ll recognize that we have nothing better about me . # txlege https : / / t.co / SMPT1HLS5Z xxbos The first Repub @txlgbtqcaucus who deserves to have a vote on the #\n",
      "I wonder if this is true this session ? We are probably trying to get it to the Governor in 2020 . This is how i really do n’t know . # txlege https : / / t.co / XXBOS\n",
      "I wonder if this bill comes over again about the state 's failure to adequately fund Texas parks , # txlege # DFW https : / / t.co / khai7pcxke xxbos Last session , House Bill 3490 was vetoed\n"
     ]
    }
   ],
   "source": [
    "TEXT = \"I wonder if this\"\n",
    "N_WORDS = 40\n",
    "N_SENTENCES = 3\n",
    "\n",
    "print(\"\\n\".join(learn.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hj3aD0UebVwr"
   },
   "source": [
    "Remember, these are not real ... they were _generated_ by the model when it tried to guess each of the next words in the sentence! Generating text like this is not why we made the language model (though you can see where text-generation AI starts from!)\n",
    "\n",
    "Also note that the model is often crafting the response _in the form of a tweet!_\n",
    "\n",
    "We now save not only the model, but also its encoder, which is the mathematical representation of what the language model \"understands\" about English patterns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "thkQAeQ2bVwr"
   },
   "outputs": [],
   "source": [
    "# have to save this one, about 130MB\n",
    "learn.save_encoder('fine_tuned_enc')  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zt6pDPEDbVww"
   },
   "source": [
    "## Building the classifier model\n",
    "\n",
    "This is the model that will use our langauge model **and** the hand-coded tweets to guess if new tweets are fact-checkable or not.\n",
    "\n",
    "We'll create a new data bunch that only grabs the hand-coded tweets and keeps track of the labels there (true or false, for fact-checkability). We also pass in the `vocab` -- which is the list of the most useful words from the language model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "ntIAhZbbbVw1"
   },
   "outputs": [],
   "source": [
    "data_clas = TextClasDataBunch.from_csv(path, 'hand_coded_austin_tweets.csv', vocab=data_lm.vocab, text_cols='tweet_text', label_cols='checkable')\n",
    "\n",
    "data_clas.save('data_clas_tweets')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "I3wfbjMrRDzf"
   },
   "source": [
    "And here's how the computer has tokenized the tweets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 323
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "iBHYA4q_bVw7",
    "outputId": "9016b7b9-b035-4f6d-a0c8-8bc7b304c55b"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>xxbos @cjtackett @williamstodd78 @atpe_montye @pastors4txkids xxunk xxunk xxunk @officialatpe @texans4e @teachthevote @lmcgaughy @senlarrytaylor @kirkpwatson xxunk @txedvote xxunk xxunk xxmaj stats are likely wrong . xxmaj plus , many biz only few make top salary . xxmaj in # xxup hb3 / # xxup sb4 xxup tx is planning to make a substantial xxunk - billion permanent investment in merit pay programs . xxmaj do your really think effective teachers will</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos # xxunk xxmaj baseball xxmaj xxunk : xxmaj april 8th \\n \\n  2 . xxmaj xxunk xxmaj xxunk \\n  xxmaj record : ( 16 - 6 - 1 ) xxmaj last xxmaj week : # 3 \\n \\n  4 . xxmaj xxunk - xxmaj jefferson xxmaj xxunk \\n  xxmaj record : ( 12 - 8 ) xxmaj last xxmaj week : # 8 \\n \\n</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj on our way back to xxmaj south xxmaj carolina . xxmaj hope to see you in : \\n  xxunk xxmaj xxunk today at 5 ( xxup xxunk xxunk xxmaj middle xxmaj school xxmaj gym ) \\n  xxunk xxmaj xxunk xxmaj saturday at xxunk ( xxmaj xxunk 's xxmaj xxunk xxmaj restaurant ) \\n  xxunk xxmaj xxunk xxmaj saturday at 3 ( xxmaj xxunk xxmaj hall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj just now : xxmaj desk mtg of the xxmaj senate xxmaj education xxmaj committee to vote out the following : \\n \\n  xxup hb 18 ( xxmaj bettencourt , xxmaj fallon , xxmaj hall , &amp; &amp; xxmaj hughes voting no ) \\n  xxup hb 65 \\n  xxup hb xxunk \\n  xxup hb xxunk \\n  xxup hb xxunk \\n  xxup hb xxunk</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj xxunk xxmaj easter photo where we look like we ’ve got it all together . ( xxmaj we do n’t ! ) xxmaj all in all , we ’re as happy as we look in this photo . xxmaj we ’ve got our issues , xxunk &amp; &amp; challenges . xxmaj but xxmaj jesus is xxmaj alive - and that makes us way too blessed to be xxunk</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_clas.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WCwf-DyEbVxA"
   },
   "source": [
    "We can then create a model to classify tweets. You can see that in the next two lines we include the processed, hand-coded tweets (`data_clas`), the original Wikitext model (`AWD_LSTM`), and the knowledge we saved after infusing the language model with tweets (`fine_tuned_enc`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "xlJKU0g3bVxA"
   },
   "outputs": [],
   "source": [
    "learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5)\n",
    "learn.load_encoder('fine_tuned_enc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ElVQYvpPRwOJ"
   },
   "source": [
    "Again, we need to find the best learning rate, where the slope starts to drop ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "qtLBfovqbVxC",
    "outputId": "c9687d87-8db6-47d4-e4ac-fc6f2991243f"
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 283
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "KvESjgILbVxI",
    "outputId": "63ddca80-8175-4f5e-f5df-f74429276416"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8nWWZ//HPlX1Pmibd9422tFAg\nlKWIgCwFUQQFCzKCW38qoIg64zKCAzLqjA5uuACDCwqIoFgQKYggA7bQFLpDSxfSJqW0TdqkzUly\nsly/P87TcgjZ2uZsyff9ej2vnnOf+znPdZrlyr08923ujoiISE/SEh2AiIgkPyULERHplZKFiIj0\nSslCRER6pWQhIiK9UrIQEZFeKVmIiEivlCxERKRXShYiItKrjEQH0F/Kysp8woQJiQ5DRCSlLF++\nfLe7l/dWb8AkiwkTJlBZWZnoMEREUoqZVfWlnrqhRESkV0oWIiLSKyULERHplZKFiIj0SslCRER6\npWQhIiK9UrIQEZFeKVmIiKSwh5ZXc+8LW2N+HSULEZEU9qeXa3jopeqYX0fJQkQkhe1vaSMvKz3m\n11GyEBFJYaFwGwXZsV+5SclCRCSFNba0k5elZCEiIj0IhdvIz1Y3lIiI9KAxrJaFiIj0oLW9g3Bb\nB/ka4BYRke6EWtoByNcAt4iIdKcx3AagMQsREeleKEgWGrMQEZFuNR7shlLLQkREutHYMkBaFmY2\n38zWm9lGM/tKF6+PM7OnzexlM1tlZhcE5RPMrMnMVgTHz2MZp4hIKmoMR1oW8biDO2ZXMLN04Hbg\nHKAaWGZmi9x9XVS1fwcecPefmdlM4DFgQvDaJnefE6v4RERS3VtjFqndDTUX2Ojum909DNwPXNSp\njgNFweNiYHsM4xERGVAaB8jU2dHAtqjn1UFZtG8CV5pZNZFWxXVRr00Muqf+YWbv6uoCZrbQzCrN\nrHLXrl39GLqISPJ7a8witVsWfXE58Ct3HwNcANxjZmnAG8A4dz8OuAG418yKOp/s7ne4e4W7V5SX\nl8c1cBGRRGscIFNna4CxUc/HBGXRPgE8AODuS4AcoMzdW9y9NihfDmwCpsUwVhGRlBMKt5ObmU56\nmsX8WrFMFsuAqWY20cyygAXAok51tgLvATCzGUSSxS4zKw8GyDGzScBUYHMMYxURSTmNLfFZcRZi\nOBvK3dvM7FpgMZAO3O3ua83sZqDS3RcBXwTuNLMvEBnsvtrd3cxOB242s1agA/i0u9fFKlYRkVQU\nitOKsxDDZAHg7o8RGbiOLrsx6vE6YF4X5z0EPBTL2EREUl28tlSFxA9wi4jIYYpsfBSfloWShYhI\nimpsaVeyEBGRnoXCbXHZ+AiULEREUlZjS/wGuJUsRERSVGM4flNnlSxERFJUSC0LERHpSbitg3B7\nBwVqWYiISHeagr0s1LIQEZFuHVhEUGMWIiLSrXhuqQpKFiIiKenAlqpqWYiISLdCQcsiXy0LERHp\nzlstCyULERHpRjy3VAUlCxGRlPTWbCi1LEREpBuhlgP3WahlISIi3TjQstDUWRER6VYo3E5uZjrp\naRaX6ylZiIikoP0t8VtxFpQsRERSUqilLW5dUKBkISKSkhrD7XEb3AYlCxGRlBQKt1EQp2mzoGQh\nIpKSGlvayVOyEBGRnjS2tJGvbigREelJKBy/LVVByUJEJCU1hgfQ1Fkzm29m681so5l9pYvXx5nZ\n02b2spmtMrMLol77anDeejM7L5ZxioikmlBLe9zWhQKI2ZXMLB24HTgHqAaWmdkid18XVe3fgQfc\n/WdmNhN4DJgQPF4AHA2MAv5mZtPcvT1W8YqIpIpwWwfh9o4BM2YxF9jo7pvdPQzcD1zUqY4DRcHj\nYmB78Pgi4H53b3H3LcDG4P1ERAa9UJzXhYLYJovRwLao59VBWbRvAleaWTWRVsV1h3CuiMigFO8t\nVSHxA9yXA79y9zHABcA9ZtbnmMxsoZlVmlnlrl27YhakiEgyCbUMrJZFDTA26vmYoCzaJ4AHANx9\nCZADlPXxXNz9DnevcPeK8vLyfgxdRCR5HWhZDJQ7uJcBU81sopllERmwXtSpzlbgPQBmNoNIstgV\n1FtgZtlmNhGYCrwYw1hFRFJGvLdUhRjOhnL3NjO7FlgMpAN3u/taM7sZqHT3RcAXgTvN7AtEBruv\ndncH1prZA8A6oA24RjOhREQiDiSLATF1FsDdHyMycB1ddmPU43XAvG7OvRW4NZbxiYikolA4vluq\nQuIHuEVE5BAd2FI1ni0LJQsRkRQTajkwdVbJQkREurE/GLPIzVQ3lIiIdCMUbiM3M530NIvbNZUs\nRERSTGO4Pa53b4OShYhIygm1tMX17m1QshARSTmRloWShYiI9CDeW6pCjG/KG6g6OpzXdu6nsqqO\nytf3sKO+mezMNLLS08jOTCcnI428rHRyszLIz0rn7JnDmTGyqPc3FhHpg8ZwO8W5mXG9ppJFN9yd\nqtoQSzfXsrqmnj2hMPVNrewNtVK9p4n6plYAyguzGV+aRyjcRktbBy1tHTS3thMKt9MUbifc3sFt\nf9vAR0+ZwBfOmRb3L7CIDDyhljZGFefE9ZpKFlGaW9t5Zv1OFq99kyWbatnR0AxAcW4mZQVZlORl\nMaIoh2PGlHDC+CGcOGEI40rzMOt++treUJjvP7GBXy95nUdXbeff5k/n2LEldLjT0QGZ6cbk8gLS\nepgC19bewc59LbxR30x5QTbjhub190cXkRQSCrfHfYB70CeLcFsHz27YxaOrtvPkujdpDLczJC+T\neVPKOHnSUE6eNJTJ5fk9JoSelORlccsHZnFZxVi+8ec1fPnBVe+oM7I4h/fOHsmFx45i9uhiXnmj\ngaWba1m6uZa12xt4s6GZDo/UNYMLjxnFdWdNYdrwwne8V0NzK2tq6lldXc+6NxrYvb+FvaFIi6ip\ntZ1RJTlMGJrPhKH5jB+ax/jg32GF2Yf9GUUkvhrDbRTEeersoE8Wu/e38MnfVFKcm8mFx4ziwmNH\ncsqkoWSk9+/Y/+wxxfzxM6fy3Mbd7GtuI80gLc2ob2rlibU7+PWS17nruS1kpacRbu8AYGJZPqdM\nGsroIbmMLM5lRHE2y17fw2/++TqPrNzO+bNGMGt0MdvqQmzbE2JrXYhtdU0Hrzm6JJfhRdkML8rh\nqOGFZGems31vE6tr6vnrmh20H8hAQE5mGuNL85lUHhxlBcwcVcRRwwt7bPU0t7azuqaeldv2smtf\nC3WNYeoaw+xrbiMrI42czDRyMtMpys1k6rACpg0vZNrwQsoKspScRA5TY0sbeXGeDWWRFcFTX0VF\nhVdWVh7WuZWv13HMmBKyMhI3OexA0li7vYE5Y0s4edJQRnTTJ7mnMczdz2/hV8+/zr6WNobmZzGm\nNI+xQ3KZPqKQ2WNKmD26mNL8rG6v19reQfWeJqpqG9laF6KqNsTruxvZsruRqrrQwURSkpfJiRNK\nOWliKUPysmhobqWhqY09oTCrgxbMgeSWlZHG0PwshuRlUZiTQWt7B82tHTS3tVPXGGZvqPXg9Ytz\nM5kQtGwmDM2jJC+LDnfcwXEMIz0tcmSkG1OHFXLMmGJyguUN3J0Nb+7nibU7WFldz7CibMYOyWNs\naS4zRhYxubygv740Ikkl3NbBtH//K186dxrXnjX1iN/PzJa7e0Wv9ZQsUldzazsd7v3ed9na3sHW\nuhArt+3lhc11LN1SS1Vt6G11CrIzmDa8gBMnlHLC+CEcP34IQ/O7by24O7v3h9nw5j7W79jHxl37\n2VoboqqukZo9TXT04dswM904elQxU4cV8OLrdQdjmlSez57GMHuiktH7jx3Fl887irGlGt+RgWVv\nKMycm5/kxgtn8vHTJh7x+/U1WQz6bqhUlhOjRcQy09OYXF7A5PICLjl+DABvNjTT3NpOUU4mhTkZ\nh9xNZ2aUF2ZTXpjNvCllb3st3NZBU7gdSwML6ro77R2Ro6Wtg3XbG6is2sPyqjqeWPcmc8aWsPD0\nSZwzYzjDiiItsH3NkZlqf1n1Bnc9t5nH1+zg6nkT+MhJ40gzo7W9g7YOJ80gKz2drIw0sjPSKMrN\njOsaOyJH4sCWqvFe7kPJQvpkeFHspullZaT12gU4qiSXs2cO77FOYU4mM0ZmMmNkER85eRzff2ID\nd/7fZu54dnOP56VZZAr08KIchhflMGFoHpOCZDmpPL/HFpNIvCVilzxQspABamRxLt+79Fg++a6J\nrNi6l4z0NDKC8Y8Oj7Rmwm0dtLS1s6cxzI6GZnY0tFBV28izG3bR0tZx8L2KczOZXJ7PpPICpo8o\n5ILZIxlVkpvATyeD2cFkoamzIv1n+ogipo84tLvnOzqcmr1NbN7dyKad+9m8ez+bdkaSyIPLq7n1\nsVc4bUoZHzphDOcdPSJm3YEiXUnElqqgZCHyDmlpxtjSPMaW5vHuaeVve21rbYgHX6rmoeXVfP7+\nFRRkZ3DGUeWcd/QIzpw+jII4dw3I4KNuKJEUMG5oHjecM43r3zOVJZtreWRl5GbOR1e9QVZ6GnPG\nljCxLJ/xZXlMGJrPnLEl6rKSfqWWhUgKSUsz5k0pY96UMm692FletYfFa3ewcttennp1J7v3txys\nWzF+CO87dhTnzx7BsML4rucjA8+BLVXj3YpVshA5QulpxtyJpcydWHqwbH9LG1t2NfKPDTt5ZOUb\n3LRoLf/xyFrOnTmCa8+awqzRxQmMWFJZKBxJFvG+g1vJQiQGCrIzmD2mmNljirn2rKlseHMff3q5\nht8ureLxtTs446hyrj1zChUTSnt/M5EojS2RbqjcOE+s0OZHInEwbXgh/zZ/Os9/5Sy+fN5RrKqu\n50M/X8JnfrucncHqxiJ9EQq3kZuZHvcbSZUsROKoKCeTa86cwnP/diZfOncaT726k/f8zz+494Wt\ndPRlzRMZ9Da8uZ/RQ+I/aSKm3VBmNh/4IZAO3OXu3+n0+m3AmcHTPGCYu5cEr7UDq4PXtrr7+2MZ\nq0g85WVlcO1ZU3nvMaP4+p9W87U/reb3lduYNqyAjPQ0stKN0vxsrjhpHOWF2YkOV5JEuK2DF7fU\ncVnFmLhfO2bJwszSgduBc4BqYJmZLXL3dQfquPsXoupfBxwX9RZN7j4nVvGJJIOJZfn87pMn8eDy\nau54djPPb9xNuN1p6+igoamVXzy7iY/Pm8jCd0+iKEe7LA52L2/dQ1NrO6d2Wl8tHmLZspgLbHT3\nzQBmdj9wEbCum/qXAzfFMB6RpGRmXFoxlksrxr6tfPOu/Xz/yQ385OmN3LO0ikuOH01eVjqGYQYj\ninM4bUoZ44fmJyhyibfnN+4mzeDkSUPjfu1YJovRwLao59XASV1VNLPxwETg71HFOWZWCbQB33H3\nh2MVqEgymlRewO1XHM9n3l3P955Yz2+XVtHhkeXeo4c3xpbmctqUct49rZx5U4ZSqBbIgPX8plpm\njymhODf+X+NkmTq7AHjQ3dujysa7e42ZTQL+bmar3X1T9ElmthBYCDBu3Lj4RSsSR7NGF/Orj819\nW5m7s2V3I89t3M2zG3bzyMrt3PfiVjLSjBMnlHLGUeWMLMmN7MhoRkaaMWVYAROG5ve486Ekr33N\nrazYtpdPv3tSQq4fy2RRA0S3q8cEZV1ZAFwTXeDuNcG/m83sGSLjGZs61bkDuAMimx/1S9QiKcDM\nmFRewKTyAj56ygRa2zt4qWoPT6/fxTPrd/Ltv77a5XmF2RkcPbqI48cNYeHpkyjJ6343RUkuL26p\no73DmTc5/uMVENtksQyYamYTiSSJBcAVnSuZ2XRgCLAkqmwIEHL3FjMrA+YB/xXDWEVSWmZ6GidN\nGspJk4bylfOns3NfMw1NbQe7rJpb21n/5j5WV9ezqqaeXzy7mYdfruGHlx/HiboxMCU8v7GW7Iw0\njh8/JCHXj1mycPc2M7sWWExk6uzd7r7WzG4GKt19UVB1AXC/v31/1xnAL8ysg8i9IN+JnkUlIj0b\nVpjDsMK3lx07toTLgkH0ldv28rn7X+bDv1jC598zjWvPmqLdApPc8xt3c+KE0oQtid+nPbjNbDJQ\nHfylfwZwDPAbd98b4/j6bDDuwS1yJPY1t/KNh9fw8IrtHDeuhA8eP4azpg/TKrlJaOe+Zube+hT/\nOv8oPnvGlH597/7eg/shoMLMphAZI/gzcC9wweGHKCKJVJiTyW0fnsNpU8v54VMb+PeH1wAwY2QR\nHzx+NJ84baK2k00SSzbVAnBaAu6vOKCvyaIj6Fa6GPixu//YzF6OZWAiEntmxodOGMMHjx/Npl37\neeqVnTyx7k2+9ZdX2LRrP9/6wGx1TyWB5zfupigng6NHJW614r4mi1Yzuxy4CnhfUKbJ3CIDhJkx\nZVghU4YVsvD0SXz/icjNgA3Nbdx22RyyMrSMXKK4O89vrOXUyWUJTdx9/Q74GHAKcKu7bwlmON0T\nu7BEJFHMjC+ddxRfu2A6f1n1BgvvqaQp3N77iRITVbUhavY2MW9K/O/ajtanZOHu69z9c+5+XzCt\ntdDdvxvj2EQkgRaePpnvXDKbf2zYxRV3LeVNLaWeEC9uqQPglATdX3FAn5KFmT1jZkVmVgq8BNxp\nZv8T29BEJNEWzB3Hzz5yPOt37OPCHz/H8qq6RIc06BxI0uNK8xIaR1+7oYrdvQG4hMiU2ZOAs2MX\nlogki/mzRvKnz84jLyudBXcs5XcvVNGXKffSP+qbWsnNTE/4uFFfr55hZiOBy4BHYxiPiCSho0YU\nsuia0zh1chlf/9MavvD7FewNhRMd1qBQ39SakIUDO+trsriZyJ3Ym9x9WbC432uxC0tEkk1xXiZ3\nX30i1589lUdXvcE5tz3LE2t3JDqsAS+lkoW7/8Hdj3H3zwTPN7v7B2Mbmogkm/Q04/qzp/HwNfMo\nK8hm4T3L+fz9L6uVEUMplSzMbIyZ/cnMdgbHQ2YW/339RCQpzBpdzJ+vmccXzp7GX1a9wfwf/N/B\nu4ylf9U3tVKUKskC+CWwCBgVHI8EZSIySGVlpPH5s6fy8DWRwe8r7lrKfy9+ldb2jkSHNqA0pFLL\nAih391+6e1tw/Aooj2FcIpIiZo0u5pHrTuOyE8Zy+9ObuPTnS9i1ryXRYQ0YKdUNBdSa2ZVmlh4c\nVwJqc4oIAPnZGXz3Q8dw+xXH8+qOBj5338u0d2h67ZFqbe+gMdyeUsni40Smze4A3gA+BFwdo5hE\nJEW995iR3HLRLJZsruWHf9uQ6HBSXkNTKwDFuYnfAbuvs6Gq3P397l7u7sPc/QOAZkOJyDtcWjGW\nD50whh8/vZFnN+xKdDgprf5AsshLnZZFV27otyhEZEC55aJZTBtWyPW/X8GOeq0pdbgOJosU6obq\niha5F5Eu5Walc/tHjqe5tZ3r7nuJNs2QOiwDJVlo9EpEujVlWAHf+sAslr2+h3tf3JrocFJSyiQL\nM9tnZg1dHPuI3G8hItKti48bzSmThvL9Jzawp1F3eR+qAwPcSX9TnrsXuntRF0ehuyd+eF5EkpqZ\ncdP7Z7KvuZXbNDvqkKVMy0JE5EhNH1HElSeP57dLq3h1R0Oiw0kp9U2t5GSmkZ2RnuhQlCxEJPZu\nOGcaRbmZ/MeiddoL4xAky93boGQhInFQkpfFF8+ZxpLNtTy+Rsua95WShYgMOpfPHcf0EYV86y+v\n0NzanuhwUoKShYgMOhnpadz0vqOp2dvEL/6xOdHhpIT6prbBkSzMbL6ZrTezjWb2lS5ev83MVgTH\nBjPbG/XaVWb2WnBcFcs4RSQ+Tpk8lPceM5KfPrOR6j2hRIeT9BqSZC8LiGGyMLN04HbgfGAmcLmZ\nzYyu4+5fcPc57j4H+DHwx+DcUuAm4CRgLnCTmQ2JVawiEj9fv2AGZnDrX15JdChJb7B0Q80FNgZb\nsIaB+4GLeqh/OXBf8Pg84El3r3P3PcCTwPwYxioicTKqJJdrzpjCX9fs4PmNuxMdTtJqa+9gf8vg\n6IYaDWyLel4dlL2DmY0HJgJ/P9RzRST1fOr0SYwtzeWbi9ZqZ71uNDS3AclxQx4kzwD3AuBBdz+k\nKRJmttDMKs2sctcuLYUskipyMtP5xntn8trO/dyzpCrR4SSlZLp7G2KbLGqAsVHPxwRlXVnAW11Q\nfT7X3e9w9wp3rygv1y6vIqnknJnDedfUMn7099doCmsqbWeDKVksA6aa2UQzyyKSEBZ1rmRm04Eh\nwJKo4sXAuWY2JBjYPjcoE5EBwsy45swp7A218ucV3f0dOXgNmmTh7m3AtUR+yb8CPODua83sZjN7\nf1TVBcD9HrUGgLvXAbcQSTjLgJuDMhEZQE6aWMr0EYX86p+vaxmQTpItWcR05Vh3fwx4rFPZjZ2e\nf7Obc+8G7o5ZcCKScGbG1adO4Ct/XM0LW+o4edLQRIeUNJItWSTLALeIDFIXzRlNSV4mv/7n64kO\nJakk014WoGQhIgmWm5XOh08cy+K1O6jZ25TocJJGfVMr2Rlp5GQmfnlyULIQkSTwLyePB+C3SzWN\n9oD6UCslecnRqgAlCxFJAmOG5HHuzBHc/+JWrUgbSKalPkDJQkSSxFWnTmBPqJVFK7YnOpSkoGQh\nItKFkyeVMmNkET/42wb2NIYTHU7CKVmIiHTBzPjuB2eze3+YGx5YQUfH4L7voj6JlicHJQsRSSLH\njCnhGxfO4On1u/jZPzYlOpyEalDLQkSke1eePJ73HTuK7z+xniWbahMdTkK0dzj7kmh5clCyEJEk\nY2Z8+5LZTCjL53P3v8zOfc2JDinuGpLs7m1QshCRJFSQncHPPnIC+5pb+dgvl7E3NLgGvJNtqQ9Q\nshCRJHXUiEJ+fuUJvLZzP1fc+cKgmiGlZCEicgjOOGoYd360gk279nPFXS9QN0gShpKFiMgheve0\ncu66qoLNu/ZzxZ1LeXzNDtZurz/4C3UgSsZkEdMlykVE+sO7ppZz99Un8qnfVPLp3y4/WD4kL5N/\nnT+dBSeOxcwSGGH/UrIQETlM86aU8cLX3sPru0PU7A1RvaeJv73yJl/942pe2FzLrRfPJj97YPxK\nq0+y5clByUJEUkhhTiazxxQze0wxAB+fN5GfPrOR/3lyA6tq6rn9iuOZMbIowVEeuYYkW54cNGYh\nIiksLc249qyp/O6TJ7OvuY1LfvpPdtSn/n0ZybYuFChZiMgAcMrkodz3qZNoam3n0VWpv2qtkoWI\nSIxMGVbIjJFFPLb6jUSHcsSULEREYujCY0by0ta9bE/x7VmVLEREYuiC2SMBUr51oWQhIhJDE8vy\nB0RXVLLtZQFKFiIywKR6V1R7h7OvObmWJwclCxEZYFK9K2pfc/LdvQ1KFiIywEwsy2dmCndF7Q0N\nwmRhZvPNbL2ZbTSzr3RT5zIzW2dma83s3qjydjNbERyLYhmniAws703hrqhkXBcKYpgszCwduB04\nH5gJXG5mMzvVmQp8FZjn7kcD10e93OTuc4Lj/bGKU0QGnlTuitp7IFnkDZJkAcwFNrr7ZncPA/cD\nF3Wq8yngdnffA+DuO2MYj4gMEqncFVWzJ9IaGl2Sm+BI3i6WyWI0sC3qeXVQFm0aMM3MnjezpWY2\nP+q1HDOrDMo/EMM4RWQAuvDYSFdUVW1jokM5JFV1jWSlpzG8KCfRobxNoge4M4CpwBnA5cCdZlYS\nvDbe3SuAK4AfmNnkzieb2cIgoVTu2rUrXjGLSAq4+LjRpBk8uLw60aEckq21IcaU5pKellz7c8Qy\nWdQAY6OejwnKolUDi9y91d23ABuIJA/cvSb4dzPwDHBc5wu4+x3uXuHuFeXl5f3/CUQkZY0szuVd\nU8t5cHk17R2e6HD6bGtdiPGleYkO4x1imSyWAVPNbKKZZQELgM6zmh4m0qrAzMqIdEttNrMhZpYd\nVT4PWBfDWEVkALqsYixv1Dfz3MbdiQ6lT9ydrbUhxg2mZOHubcC1wGLgFeABd19rZjeb2YHZTYuB\nWjNbBzwNfNnda4EZQKWZrQzKv+PuShYickjOnjmMkrxM/lC5rffKSWBPqJV9LW2MG5qf6FDeIaY7\n5bn7Y8BjncpujHrswA3BEV3nn8DsWMYmIgNfdkY6H5gzmntf2MreUJiSvKxEh9SjrXUhgEHXDSUi\nknCXVowh3N7BopXJvynSgZlb44YqWYiIxNXRo4o5elQRD6RAV9TW2kjLYlCNWYiIJIvLKsaypqaB\nddsbEh1Kj7bWhRhelE1OZnqiQ3kHJQsRGfAumjOKrPQ0/rA8uVsXVXXJORMKlCxEZBAoycti/qwR\n3P/iNjbv2p/ocLoVmTabfDOhQMlCRAaJr10wg+zMNK7//Qpa2zsSHc47NLe2s6OhWS0LEZFEGlGc\nw3cumc2q6np+8LcNiQ7nHar3BNNmk3AmFChZiMggMn/WSD5cMZafPrOJFzbXJjqct6k6MBNKyUJE\nJPFufN9MxpfmccMDKw9uNJQMDtyQp24oEZEkkJ+dwQ8WHMeOhmYu+enz3PbkBtZuryeyoETiVNWG\nyM9KZ2h+ct5lrmQhIoPOnLEl/OTy4yjNz+JHf3+N9/7oOU777tMJ3SxpW12IcUPzMUuupckPiOna\nUCIiyer82SM5f/ZIdu9v4alX3uSepVV8/v6XKc7NZN6UsrjHU1UXYnJ5ck6bBbUsRGSQKyvI5sMn\njuN3nzyZSWUF/L97lsf9Tu+ODo/sY5GEq80eoGQhIgIU52byy4+dSEF2Bh/71YvU7G2K27V37msh\n3NaRtIPboGQhInLQqJJcfvXxEwm1tHPV3S9SH4rPbKmDq80qWYiIpIbpI4r4xUdPoKq2kWvve4m2\nONztXVWX3DfkgZKFiMg7nDq5jFsumsX/vbab/1q8PubX21YXIj3NGFWSG/NrHS7NhhIR6cKCueNY\nu72BO57dzMyRRXzguNExu1ZVbYhRJTlkpifv3+/JG5mISILd+L6ZzJ1Yyr89tIrV1fUxu05VXYjx\nSbra7AFKFiIi3chMT+OnHzmeoflZLLynkrrGcL+87xv1TdTsbaKjI3LX+La6EGOTeHAb1A0lItKj\nsoJs7vhoBRfd/jzfe2I9/3nx7CN6v8aWNs75n2fZ39JGbmY6E8ryqWsMJ/XgNihZiIj0atboYq46\nZQK//OcWrpg7jlmjiw/7vf7UfOgqAAAL4UlEQVT+6k72t7Tx2TMm09LWwaZd+4EiTkvAXeOHQslC\nRKQPPn/2VP68ooZvLlrLHz59ymGv4fT4mh2UFWTxxXOPIj0tOdeB6orGLERE+qA4N5Mvn3cUlVV7\nWLRy+2G9R1O4nafX7+S8o0ekVKIAJQsRkT67tGIss0cX8+3HXqWxpe2Qz//Hhl2Ewu2cP2tkDKKL\nLSULEZE+Sk8zvvn+mexoaOanz2w85PMfX/MGQ/IyOWlSaQyiiy0lCxGRQ3DC+FIuOW40dz675eC+\n2X3R0tbOU6/s5JyZw5P65rvuxDRiM5tvZuvNbKOZfaWbOpeZ2TozW2tm90aVX2VmrwXHVbGMU0Tk\nUHzpvKMAuP3pvrcunt+4m30tbSnZBQUxTBZmlg7cDpwPzAQuN7OZnepMBb4KzHP3o4Hrg/JS4Cbg\nJGAucJOZDYlVrCIih2JUSS6Xzx3LHyqr2Vrbt9bFY6t3UJiTwalThsY4utiIZctiLrDR3Te7exi4\nH7ioU51PAbe7+x4Ad98ZlJ8HPOnudcFrTwLzYxiriMgh+eyZU0hLM37899d6rdva3sGT697k7BnD\nyc5Ij0N0/S+WyWI0sC3qeXVQFm0aMM3MnjezpWY2/xDOxcwWmlmlmVXu2rWrH0MXEenZ8KIcrjxp\nPH98uYYtuxt7rLtkUy31Ta2cP2tEnKLrf4keZckApgJnAJcDd5pZSV9Pdvc73L3C3SvKy8tjFKKI\nSNc+c8ZkMtONHz3Vc+vir2t2kJeVzunTUvf3VCyTRQ0wNur5mKAsWjWwyN1b3X0LsIFI8ujLuSIi\nCVVemM1Vp0zgzytq2LhzX5d1llfV8dBL1Zx39AhyMlOzCwpimyyWAVPNbKKZZQELgEWd6jxMpFWB\nmZUR6ZbaDCwGzjWzIcHA9rlBmYhIUll4+iRyMtP53uINB1eRPWBbXYiFv1nOyOIcvnHhzG7eITXE\nLFm4extwLZFf8q8AD7j7WjO72czeH1RbDNSa2TrgaeDL7l7r7nXALUQSzjLg5qBMRCSpDC3IZuHp\nk3h87Q6u/N8XDt570dDcyid+vYxwewf/e9WJlOZnJTjSI2Pu3nutFFBRUeGVlZWJDkNEBiF35/fL\ntnHLo+swM268cCZ/Wf0Gz23cza8/NpfTpibvirJmttzdK3qrp1VnRUSOkJmxYO445k0p40t/WMm/\nPrQKgP+8eHZSJ4pDoWQhItJPxpbmcd+nTuZ3L26lvb2DK04al+iQ+o2ShYhIP0pLM/7l5PGJDqPf\nJfo+CxERSQFKFiIi0islCxER6ZWShYiI9ErJQkREeqVkISIivVKyEBGRXilZiIhIrwbM2lBmVg90\ntah8MVDfS1n0864eH/i3DNh9mCF2FUdfXlf8by873M/QW/w91ekp3s7Pe3us+A+9Tm/fQ919nv6M\nv6f4ens92X+Gx7t77xttuPuAOIA7+lreuSz6eVePo/6t7O/4FH/f4z+Sz9Bb/IfyGQ41/v74Gij+\n7su6+zz9GX9fPkOq/wz3dgykbqhHDqG8c9kjvTzu7r0PRW/vofi7fxyP+Huq01O8nZ/35fHhUPzd\nl3X3efoz/r68x0D4GejWgOmGigczq/Q+LOWbrFI9fkj9z6D4E0vxH76B1LKIhzsSHcARSvX4IfU/\ng+JPLMV/mNSyEBGRXqllISIivRq0ycLM7jaznWa25jDOPcHMVpvZRjP7kZlZ1GvXmdmrZrbWzP6r\nf6N+Wwz9Hr+ZfdPMasxsRXBc0P+RH4whJv//wetfNDM3s5huURajr8EtZrYq+P9/wsxG9X/kB2OI\nRfz/HXz/rzKzP5lZSf9HfjCGWMR/afCz22FmMRkbOJK4u3m/q8zsteC4Kqq8x5+TQ3YkU6lS+QBO\nB44H1hzGuS8CJwMG/BU4Pyg/E/gbkB08H5Zi8X8T+FKq/v8Hr40FFgNVQFmqfQagKKrO54Cfp1j8\n5wIZwePvAt9NsfhnAEcBzwAVyRR3ENOETmWlwObg3yHB4yE9fcbDPQZty8LdnwXqosvMbLKZPW5m\ny83s/8xseufzzGwkkR/opR75ivwG+EDw8meA77h7S3CNnSkWf9zEMP7bgH8FYj4YF4vP4O4NUVXz\nieHniFH8T7h7W1B1KTAmxeJ/xd3XxyrmI4m7G+cBT7p7nbvvAZ4E5sfi53zQJotu3AFc5+4nAF8C\nftpFndFAddTz6qAMYBrwLjN7wcz+YWYnxjTadzrS+AGuDboQ7jazIbELtUtHFL+ZXQTUuPvKWAfa\ngyP+GpjZrWa2DfgIcGMMY+1Kf3wPHfBxIn/RxlN/xh9PfYm7K6OBbVHPD3yWfv+M2oM7YGYFwKnA\nH6K69rIP8W0yiDQHTwZOBB4ws0lBZo+pfor/Z8AtRP6avQX4PpEf+Jg70vjNLA/4GpFukITop68B\n7v514Otm9lXgWuCmfguyB/0Vf/BeXwfagN/1T3R9uma/xR9PPcVtZh8DPh+UTQEeM7MwsMXdL45n\nnEoWb0kD9rr7nOhCM0sHlgdPFxH5hRrdtB4D1ASPq4E/BsnhRTPrILKWy65YBh444vjd/c2o8+4E\nHo1lwJ0cafyTgYnAyuAHbgzwkpnNdfcdMY79gP74Hor2O+Ax4pQs6Kf4zexq4ELgPfH4QylKf///\nx0uXcQO4+y+BXwKY2TPA1e7+elSVGuCMqOdjiIxt1NDfnzEWAzipcgATiBpkAv4JXBo8NuDYbs7r\nPHB0QVD+aeDm4PE0Is1DS6H4R0bV+QJwfyr9/3eq8zoxHuCO0ddgalSd64AHUyz++cA6oDzW//ex\n/B4ihgPchxs33Q9wbyEyuD0keFzal894yDHH4wuajAdwH/AG0EqkRfAJIn+ZPg6sDL7hb+zm3Apg\nDbAJ+Alv3dyYBfw2eO0l4KwUi/8eYDWwishfYCNTKf5OdV4n9rOhYvE1eCgoX0VkLZ/RKRb/RiJ/\nJK0IjljO5opF/BcH79UCvAksTpa46SJZBOUfD/7fNwIfO5Sfk0M5dAe3iIj0SrOhRESkV0oWIiLS\nKyULERHplZKFiIj0SslCRER6pWQhA5qZ7Y/z9e4ys5n99F7tFll9do2ZPdLbCq5mVmJmn+2Pa4t0\npqmzMqCZ2X53L+jH98vwtxbKi6no2M3s18AGd7+1h/oTgEfdfVY84pPBRS0LGXTMrNzMHjKzZcEx\nLyifa2ZLzOxlM/unmR0VlF9tZovM7O/AU2Z2hpk9Y2YPWmTvht8d2CsgKK8IHu8PFgVcaWZLzWx4\nUD45eL7azL7Vx9bPEt5aMLHAzJ4ys5eC97goqPMdYHLQGvnvoO6Xg8+4ysz+ox//G2WQUbKQweiH\nwG3ufiLwQeCuoPxV4F3ufhyR1V7/M+qc44EPufu7g+fHAdcDM4FJwLwurpMPLHX3Y4FngU9FXf+H\n7j6bt68M2qVgbaP3ELmrHqAZuNjdjyeyh8r3g2T1FWCTu89x9y+b2bnAVGAuMAc4wcxO7+16Il3R\nQoIyGJ0NzIxa4bMoWPmzGPi1mU0lsvJuZtQ5T7p79B4EL7p7NYCZrSCy1s9zna4T5q3FGJcD5wSP\nT+GtvQXuBb7XTZy5wXuPBl4hslcBRNb6+c/gF39H8PrwLs4/NzheDp4XEEkez3ZzPZFuKVnIYJQG\nnOzuzdGFZvYT4Gl3vzjo/38m6uXGTu/REvW4na5/llr9rUHB7ur0pMnd5wTLry8GrgF+RGSfi3Lg\nBHdvNbPXgZwuzjfg2+7+i0O8rsg7qBtKBqMniKzoCoCZHVgaupi3lnG+OobXX0qk+wtgQW+V3T1E\nZIvVL5pZBpE4dwaJ4kxgfFB1H1AYdepi4ONBqwkzG21mw/rpM8ggo2QhA12emVVHHTcQ+cVbEQz6\nriOytDzAfwHfNrOXiW2r+3rgBjNbRWRDm/reTnD3l4msRHs5kX0uKsxsNfBRImMtuHst8Hww1fa/\n3f0JIt1cS4K6D/L2ZCLSZ5o6KxJnQbdSk7u7mS0ALnf3i3o7TySRNGYhEn8nAD8JZjDtJU5b14oc\nCbUsRESkVxqzEBGRXilZiIhIr5QsRESkV0oWIiLSKyULERHplZKFiIj06v8DCv84mj+NRsIAAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Az5OuN6_hJaF"
   },
   "source": [
    "Again, we're using the point where the learning rate slopes down to train our model -- here `1e-2`. We're also using \"momentums\" which we'll get into another time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 80
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "S5g2xmtlbVxJ",
    "outputId": "79971cc2-dde2-4dfc-a340-7dcae79f2bdd"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.358472</td>\n",
       "      <td>0.232218</td>\n",
       "      <td>0.914474</td>\n",
       "      <td>00:05</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, 1e-2, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hBqDZ37oSMkc"
   },
   "source": [
    "Accuracy of 91%! We can still do more training ... partly because the validation loss is still less than the training loss. Also we freeze the model except the last layer using `freeze_to(-2)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 80
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "27B3A5PDbVxP",
    "outputId": "40597d91-76eb-4089-9b20-486eff538e64"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.228632</td>\n",
       "      <td>0.196591</td>\n",
       "      <td>0.928947</td>\n",
       "      <td>00:06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.freeze_to(-2)\n",
    "learn.fit_one_cycle(1, slice(1e-2/(2.6**4),1e-2), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6_yMw6DsS_t1"
   },
   "source": [
    "Better! But looks like we can do more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 80
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "zQ4AaOVDbVxT",
    "outputId": "fa009202-13c5-47fe-ac56-39a391f04a71"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.197604</td>\n",
       "      <td>0.189697</td>\n",
       "      <td>0.931579</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.freeze_to(-3)\n",
    "learn.fit_one_cycle(1, slice(5e-3/(2.6**4),5e-3), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "x-uDGwk3THqE"
   },
   "source": [
    "Can we do even better?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 111
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "4o6ydWNhbVxX",
    "outputId": "798c8d0d-ae1a-4f39-d3e6-580a1c6b54ac"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.156188</td>\n",
       "      <td>0.191579</td>\n",
       "      <td>0.934211</td>\n",
       "      <td>00:12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.141545</td>\n",
       "      <td>0.194849</td>\n",
       "      <td>0.932895</td>\n",
       "      <td>00:14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.unfreeze()\n",
    "learn.fit_one_cycle(2, slice(1e-3/(2.6**4),1e-3), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "J-TSQ7iDTdIG"
   },
   "source": [
    "I'll take 93% accuracy! Let's give it an example ...."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "mBAXEa2MbVxa",
    "outputId": "4163f4ee-f9e0-4d93-b5a2-c47e38a78205"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Category True, tensor(1), tensor([0.3140, 0.6860]))"
      ]
     },
     "execution_count": 35,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.predict(\"Four states have two universities represented in the top 20 highest-paid executives of public colleges. Texas has SIX\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8u1ptJu1UmT9"
   },
   "source": [
    "`True` means checkable! Let's save our work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "jykRlEjKU8jS"
   },
   "outputs": [],
   "source": [
    "# optional save, about 300 MB\n",
    "learn.save('pretty_good_model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XSOgwF1ybVyh"
   },
   "source": [
    "## Preparing for Using Render\n",
    "\n",
    "We could stop here and use this model to guess thousands of saved, unchecked tweets. But I wanted to put this in the wild -- watching for new tweets and sending a Slack message when it spotted a fact-checkable tweet. For that, I hosted the predictor on a service called [Render](https://render.com).\n",
    "\n",
    "My Render code, [which is in this repository](https://github.com/jkeefe/nlp-classifier-fastai-render), needs access to the model and all of its details. So I exported it here, and saved it to a public spot on the internet. \n",
    "\n",
    "The next line (which is optional) will save everything we need for predictions to a ~90MB file to your Google Drive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "xxUtm6fMbVxZ"
   },
   "outputs": [],
   "source": [
    "## optional save as `export.pkl`  - about 90 MB\n",
    "learn.export()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "57uA-44kVh_T"
   },
   "source": [
    "For more about deploying a predictor on Render, see our [blog post about building the checkable-tweets project](https://qz.ai/?p=89)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "YsUk3Ypdw2FX"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "quartz-fastai-checkable-tweets.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}