6-rnn-english-numbers-Copy1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook was part of [Lesson 7](https://course.fast.ai/videos/?lesson=7) of the Practical Deep Learning for Coders course."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting English word version of numbers using an RNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We were using RNNs as part of our language model in the previous lesson.  Today, we will dive into more details of what RNNs are and how they work.  We will do this using the problem of trying to predict the English word version of numbers.\n",
    "\n",
    "Let's predict what should come next in this sequence:\n",
    "\n",
    "*eight thousand one , eight thousand two , eight thousand three , eight thousand four , eight thousand five , eight thousand six , eight thousand seven , eight thousand eight , eight thousand nine , eight thousand ten , eight thousand eleven , eight thousand twelve...*\n",
    "\n",
    "\n",
    "Jeremy created this synthetic dataset to have a better way to check if things are working, to debug, and to understand what was going on. When experimenting with new ideas, it can be nice to have a smaller dataset to do so, to quickly get a sense of whether your ideas are promising (for other examples, see [Imagenette and Imagewoof](https://github.com/fastai/imagenette)) This English word numbers will serve as a good dataset for learning about RNNs.  Our task today will be to predict which word comes next when counting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In deep learning, there are 2 types of numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Parameters** are numbers that are learned.  **Activations** are numbers that are calculated (by affine functions & element-wise non-linearities).\n",
    "\n",
    "When you learn about any new concept in deep learning, ask yourself: is this a parameter or an activation?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note to self: Point out the hidden state, going from the version without a for-loop to the for loop.  This is the step where people get confused."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.text import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "python: 3.6.13\n",
      "pytorch: 10.1\n",
      "fastai: 1.0.61\n"
     ]
    }
   ],
   "source": [
    "import fastai\n",
    "from platform import python_version\n",
    "print(f'python: %s\\npytorch: %s\\nfastai: %s' % (python_version(),torch.version.cuda,fastai.__version__))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.cuda.set_device(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs=64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/molly/.fastai/data/human_numbers/valid.txt'),\n",
       " PosixPath('/home/molly/.fastai/data/human_numbers/train.txt')]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = untar_data(URLs.HUMAN_NUMBERS)\n",
    "path.ls()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def readnums(d): return [', '.join(o.strip() for o in open(path/d).readlines())]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "train.txt gives us a sequence of numbers written out as English words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_preds??"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'one, two, three, four, five, six, seven, eight, nine, ten, eleven, twelve, thirt'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_txt = readnums('train.txt'); train_txt[0][:80]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' nine thousand nine hundred ninety eight, nine thousand nine hundred ninety nine'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "valid_txt = readnums('valid.txt'); valid_txt[0][-80:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train = TextList(train_txt, path=path)\n",
    "valid = TextList(valid_txt, path=path)\n",
    "\n",
    "src = ItemLists(path=path, train=train, valid=valid).label_for_lm()\n",
    "data = src.databunch(bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'xxbos one , two , three , four , five , six , seven , eight , nine , ten , eleve'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train[0].text[:80]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13017"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data.valid_ds[0][0].data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`bptt` stands for *back-propagation through time*.  This tells us how many steps of history we are considering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(70, 3)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.bptt, len(data.valid_dl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have 3 batches in our validation set:\n",
    "\n",
    "13017 tokens, with about ~70 tokens in about a line of text, and 64 lines of text per batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.905580357142857"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "13017/70/bs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will store each batch in a separate variable, so we can walk through this to understand better what the RNN does at each step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "it = iter(data.valid_dl)\n",
    "x1,y1 = next(it)\n",
    "x2,y2 = next(it)\n",
    "x3,y3 = next(it)\n",
    "it.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 2, 19, 11,  ..., 36,  9, 19],\n",
       "        [ 9, 19, 11,  ..., 24, 20,  9],\n",
       "        [11, 27, 18,  ...,  9, 19, 11],\n",
       "        ...,\n",
       "        [20, 11, 20,  ..., 11, 20, 10],\n",
       "        [20, 11, 20,  ..., 24,  9, 20],\n",
       "        [20, 10, 26,  ..., 20, 11, 20]], device='cuda:1')"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`numel()` is a [PyTorch method](https://pytorch.org/docs/stable/torch.html#torch.numel) to return the number of elements in a tensor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13440"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x1.numel()+x2.numel()+x3.numel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([64, 70]), torch.Size([64, 70]))"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x1.shape, y1.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([64, 70]), torch.Size([64, 70]))"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x2.shape, y2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([64, 70]), torch.Size([64, 70]))"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x3.shape, y3.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "v = data.valid_ds.vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['xxunk',\n",
       " 'xxpad',\n",
       " 'xxbos',\n",
       " 'xxeos',\n",
       " 'xxfld',\n",
       " 'xxmaj',\n",
       " 'xxup',\n",
       " 'xxrep',\n",
       " 'xxwrep',\n",
       " ',',\n",
       " 'hundred',\n",
       " 'thousand',\n",
       " 'one',\n",
       " 'two',\n",
       " 'three',\n",
       " 'four',\n",
       " 'five',\n",
       " 'six',\n",
       " 'seven',\n",
       " 'eight',\n",
       " 'nine',\n",
       " 'twenty',\n",
       " 'thirty',\n",
       " 'forty',\n",
       " 'fifty',\n",
       " 'sixty',\n",
       " 'seventy',\n",
       " 'eighty',\n",
       " 'ninety',\n",
       " 'ten',\n",
       " 'eleven',\n",
       " 'twelve',\n",
       " 'thirteen',\n",
       " 'fourteen',\n",
       " 'fifteen',\n",
       " 'sixteen',\n",
       " 'seventeen',\n",
       " 'eighteen',\n",
       " 'nineteen',\n",
       " 'xxfake']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.itos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 2,  9, 11, 12, 13, 11, 10,  9, 10, 14, 19, 25, 19, 15, 16, 11, 19,  9,\n",
       "        10,  9, 19, 25, 19, 11, 19, 11, 10,  9, 19, 20, 11, 26, 20, 23, 20, 20,\n",
       "        24, 20, 11, 14, 11, 11,  9, 14,  9, 20, 10, 20, 35, 17, 11, 10,  9, 17,\n",
       "         9, 20, 10, 20, 11, 20, 11, 20, 20, 20], device='cuda:1')"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x1[:,0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([19, 19, 27, 10,  9, 12, 32, 19, 26, 10, 11, 15, 11, 10,  9, 15, 11, 19,\n",
       "        26, 19, 11, 18, 11, 18,  9, 18, 21, 19, 10, 10, 20,  9, 11, 16, 11, 11,\n",
       "        13, 11, 13,  9, 13, 14, 20, 10, 20, 11, 24, 11,  9,  9, 16, 17, 20, 10,\n",
       "        20, 11, 24, 11, 19,  9, 19, 11, 11, 10], device='cuda:1')"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y1[:,0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(',', 'thousand', 'one', 'two', 'hundred')"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.itos[9], v.itos[11], v.itos[12], v.itos[13], v.itos[10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'xxbos eight thousand one , eight thousand two , eight thousand three , eight thousand four , eight thousand five , eight thousand six , eight thousand seven , eight thousand eight , eight thousand nine , eight thousand ten , eight thousand eleven , eight thousand twelve , eight thousand thirteen , eight thousand fourteen , eight thousand fifteen , eight thousand sixteen , eight thousand seventeen , eight'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.textify(x1[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "', eight thousand forty six , eight thousand forty seven , eight thousand forty eight , eight thousand forty nine , eight thousand fifty , eight thousand fifty one , eight thousand fifty two , eight thousand fifty three , eight thousand fifty four , eight thousand fifty five , eight thousand fifty six , eight thousand fifty seven , eight thousand fifty eight , eight thousand fifty nine ,'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.textify(x1[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'eight thousand sixty , eight thousand sixty one , eight thousand sixty two , eight thousand sixty three , eight thousand sixty four , eight thousand sixty five , eight thousand sixty six , eight thousand sixty seven , eight thousand sixty eight , eight thousand sixty nine , eight thousand seventy , eight thousand seventy one , eight thousand seventy two , eight thousand seventy three , eight thousand'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.textify(x2[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'eight thousand one , eight thousand two , eight thousand three , eight thousand four , eight thousand five , eight thousand six , eight thousand seven , eight thousand eight , eight thousand nine , eight thousand ten , eight thousand eleven , eight thousand twelve , eight thousand thirteen , eight thousand fourteen , eight thousand fifteen , eight thousand sixteen , eight thousand seventeen , eight thousand'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.textify(y1[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'thousand eighteen , eight thousand nineteen , eight thousand twenty , eight thousand twenty one , eight thousand twenty two , eight thousand twenty three , eight thousand twenty four , eight thousand twenty five , eight thousand twenty six , eight thousand twenty seven , eight thousand twenty eight , eight thousand twenty nine , eight thousand thirty , eight thousand thirty one , eight thousand thirty two ,'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.textify(x2[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'eight thousand thirty three , eight thousand thirty four , eight thousand thirty five , eight thousand thirty six , eight thousand thirty seven , eight thousand thirty eight , eight thousand thirty nine , eight thousand forty , eight thousand forty one , eight thousand forty two , eight thousand forty three , eight thousand forty four , eight thousand forty five , eight thousand forty six , eight'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.textify(x3[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "', eight thousand forty six , eight thousand forty seven , eight thousand forty eight , eight thousand forty nine , eight thousand fifty , eight thousand fifty one , eight thousand fifty two , eight thousand fifty three , eight thousand fifty four , eight thousand fifty five , eight thousand fifty six , eight thousand fifty seven , eight thousand fifty eight , eight thousand fifty nine ,'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.textify(x1[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'eight thousand sixty , eight thousand sixty one , eight thousand sixty two , eight thousand sixty three , eight thousand sixty four , eight thousand sixty five , eight thousand sixty six , eight thousand sixty seven , eight thousand sixty eight , eight thousand sixty nine , eight thousand seventy , eight thousand seventy one , eight thousand seventy two , eight thousand seventy three , eight thousand'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.textify(x2[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'seventy four , eight thousand seventy five , eight thousand seventy six , eight thousand seventy seven , eight thousand seventy eight , eight thousand seventy nine , eight thousand eighty , eight thousand eighty one , eight thousand eighty two , eight thousand eighty three , eight thousand eighty four , eight thousand eighty five , eight thousand eighty six , eight thousand eighty seven , eight thousand eighty'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.textify(x3[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ninety , nine thousand nine hundred ninety one , nine thousand nine hundred ninety two , nine thousand nine hundred ninety three , nine thousand nine hundred ninety four , nine thousand nine hundred ninety five , nine thousand nine hundred ninety six , nine thousand nine hundred ninety seven , nine thousand nine hundred ninety eight , nine thousand nine hundred ninety nine xxbos eight thousand one , eight'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.textify(x3[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>idx</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>thousand forty seven , eight thousand forty eight , eight thousand forty nine , eight thousand fifty , eight thousand fifty one , eight thousand fifty two , eight thousand fifty three , eight thousand fifty four , eight thousand fifty five , eight thousand fifty six , eight thousand fifty seven , eight thousand fifty eight , eight thousand fifty nine , eight thousand sixty , eight thousand sixty</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>eight , eight thousand eighty nine , eight thousand ninety , eight thousand ninety one , eight thousand ninety two , eight thousand ninety three , eight thousand ninety four , eight thousand ninety five , eight thousand ninety six , eight thousand ninety seven , eight thousand ninety eight , eight thousand ninety nine , eight thousand one hundred , eight thousand one hundred one , eight thousand one</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>thousand one hundred twenty four , eight thousand one hundred twenty five , eight thousand one hundred twenty six , eight thousand one hundred twenty seven , eight thousand one hundred twenty eight , eight thousand one hundred twenty nine , eight thousand one hundred thirty , eight thousand one hundred thirty one , eight thousand one hundred thirty two , eight thousand one hundred thirty three , eight thousand</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>three , eight thousand one hundred fifty four , eight thousand one hundred fifty five , eight thousand one hundred fifty six , eight thousand one hundred fifty seven , eight thousand one hundred fifty eight , eight thousand one hundred fifty nine , eight thousand one hundred sixty , eight thousand one hundred sixty one , eight thousand one hundred sixty two , eight thousand one hundred sixty three</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>thousand one hundred eighty three , eight thousand one hundred eighty four , eight thousand one hundred eighty five , eight thousand one hundred eighty six , eight thousand one hundred eighty seven , eight thousand one hundred eighty eight , eight thousand one hundred eighty nine , eight thousand one hundred ninety , eight thousand one hundred ninety one , eight thousand one hundred ninety two , eight thousand</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data.show_batch(ds_type=DatasetType.Valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will iteratively consider a few different models, building up to a more traditional RNN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Single fully connected model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = src.databunch(bs=bs, bptt=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([64, 3]), torch.Size([64, 3]))"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x,y = data.one_batch()\n",
    "x.shape,y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "40"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nv = len(v.itos); nv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "nh=64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "def loss4(input,target): return F.cross_entropy(input, target[:,-1])\n",
    "def acc4 (input,target): return accuracy(input, target[:,-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([13, 13, 10,  9, 18,  9, 11, 11, 13, 19, 16, 23, 24,  9, 12,  9, 13, 14,\n",
       "        15, 11, 10, 22, 15,  9, 10, 14, 11, 16, 10, 28, 11,  9, 20,  9, 15, 15,\n",
       "        11, 18, 10, 28, 23, 24,  9, 16, 10, 16, 19, 20, 12, 10, 22, 16, 17, 17,\n",
       "        17, 11, 24, 10,  9, 15, 16,  9, 18, 11])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[:,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Layer names:\n",
    "- `i_h`: input to hidden\n",
    "- `h_h`: hidden to hidden\n",
    "- `h_o`: hidden to output\n",
    "- `bn`: batchnorm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Model0(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.i_h = nn.Embedding(nv,nh)  # green arrow\n",
    "        self.h_h = nn.Linear(nh,nh)     # brown arrow\n",
    "        self.h_o = nn.Linear(nh,nv)     # blue arrow\n",
    "        self.bn = nn.BatchNorm1d(nh)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        h = self.bn(F.relu(self.i_h(x[:,0])))\n",
    "        if x.shape[1]>1:\n",
    "            h = h + self.i_h(x[:,1])\n",
    "            h = self.bn(F.relu(self.h_h(h)))\n",
    "        if x.shape[1]>2:\n",
    "            h = h + self.i_h(x[:,2])\n",
    "            h = self.bn(F.relu(self.h_h(h)))\n",
    "        return self.h_o(h)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(data, Model0(), loss_func=loss4, metrics=acc4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>acc4</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.580810</td>\n",
       "      <td>3.634767</td>\n",
       "      <td>0.075368</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.670278</td>\n",
       "      <td>2.887282</td>\n",
       "      <td>0.386719</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.120408</td>\n",
       "      <td>2.449027</td>\n",
       "      <td>0.468061</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.910201</td>\n",
       "      <td>2.272136</td>\n",
       "      <td>0.464614</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.833159</td>\n",
       "      <td>2.215138</td>\n",
       "      <td>0.465074</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.816862</td>\n",
       "      <td>2.207190</td>\n",
       "      <td>0.465074</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(6, 1e-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Same thing with a loop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's refactor this to use a for-loop.  This does the same thing as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Model1(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.i_h = nn.Embedding(nv,nh)  # green arrow\n",
    "        self.h_h = nn.Linear(nh,nh)     # brown arrow\n",
    "        self.h_o = nn.Linear(nh,nv)     # blue arrow\n",
    "        self.bn = nn.BatchNorm1d(nh)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        h = torch.zeros(x.shape[0], nh).to(device=x.device)\n",
    "        for i in range(x.shape[1]):\n",
    "            h = h + self.i_h(x[:,i])\n",
    "            h = self.bn(F.relu(self.h_h(h)))\n",
    "        return self.h_o(h)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the difference between unrolled (what we had before) and rolled (what we have now) RNN diagrams:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(data, Model1(), loss_func=loss4, metrics=acc4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>acc4</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.625305</td>\n",
       "      <td>3.588754</td>\n",
       "      <td>0.051471</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.709760</td>\n",
       "      <td>2.650298</td>\n",
       "      <td>0.443015</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.116563</td>\n",
       "      <td>2.254962</td>\n",
       "      <td>0.464844</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.897142</td>\n",
       "      <td>2.130302</td>\n",
       "      <td>0.471737</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.819982</td>\n",
       "      <td>2.095545</td>\n",
       "      <td>0.472426</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.803719</td>\n",
       "      <td>2.091217</td>\n",
       "      <td>0.472426</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(6, 1e-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our accuracy is about the same, since we are doing the same thing as before."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multi fully connected model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before, we were just predicting the last word in a line of text.  Given 70 tokens, what is token 71?  That approach was throwing away a lot of data.  Why not predict token 2 from token 1, then predict token 3, then predict token 4, and so on?  We will modify our model to do this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = src.databunch(bs=bs, bptt=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([64, 20]), torch.Size([64, 20]))"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x,y = data.one_batch()\n",
    "x.shape,y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Model2(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.i_h = nn.Embedding(nv,nh)\n",
    "        self.h_h = nn.Linear(nh,nh)\n",
    "        self.h_o = nn.Linear(nh,nv)\n",
    "        self.bn = nn.BatchNorm1d(nh)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        h = torch.zeros(x.shape[0], nh).to(device=x.device)\n",
    "        res = []\n",
    "        for i in range(x.shape[1]):\n",
    "            h = h + self.i_h(x[:,i])\n",
    "            h = F.relu(self.h_h(h))\n",
    "            res.append(self.h_o(self.bn(h)))\n",
    "        return torch.stack(res, dim=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(data, Model2(), metrics=accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.804813</td>\n",
       "      <td>3.620752</td>\n",
       "      <td>0.083310</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.627175</td>\n",
       "      <td>3.418635</td>\n",
       "      <td>0.130469</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.408246</td>\n",
       "      <td>3.233428</td>\n",
       "      <td>0.185440</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.190099</td>\n",
       "      <td>3.074087</td>\n",
       "      <td>0.260085</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.997388</td>\n",
       "      <td>2.947359</td>\n",
       "      <td>0.356179</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.843220</td>\n",
       "      <td>2.856245</td>\n",
       "      <td>0.400710</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.730370</td>\n",
       "      <td>2.798147</td>\n",
       "      <td>0.409233</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.654666</td>\n",
       "      <td>2.767024</td>\n",
       "      <td>0.413068</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.609216</td>\n",
       "      <td>2.755162</td>\n",
       "      <td>0.415341</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.585713</td>\n",
       "      <td>2.753398</td>\n",
       "      <td>0.415554</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(10, 1e-4, pct_start=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that our accuracy is worse now, because we are doing a harder task.  When we predict word k (k<70), we have less history to help us then when we were only predicting word 71."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Maintain state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To address this issue, let's keep the hidden state from the previous line of text, so we are not starting over again on each new line of text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Model3(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.i_h = nn.Embedding(nv,nh)\n",
    "        self.h_h = nn.Linear(nh,nh)\n",
    "        self.h_o = nn.Linear(nh,nv)\n",
    "        self.bn = nn.BatchNorm1d(nh)\n",
    "        self.h = torch.zeros(bs, nh).cuda()\n",
    "        \n",
    "    def forward(self, x):\n",
    "        res = []\n",
    "        h = self.h\n",
    "        for i in range(x.shape[1]):\n",
    "            h = h + self.i_h(x[:,i])\n",
    "            h = F.relu(self.h_h(h))\n",
    "            res.append(self.bn(h))\n",
    "        self.h = h.detach()\n",
    "        res = torch.stack(res, dim=1)\n",
    "        res = self.h_o(res)\n",
    "        return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(data, Model3(), metrics=accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.571528</td>\n",
       "      <td>3.611967</td>\n",
       "      <td>0.093253</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.032421</td>\n",
       "      <td>2.714716</td>\n",
       "      <td>0.281676</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.296551</td>\n",
       "      <td>2.103235</td>\n",
       "      <td>0.316406</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.848092</td>\n",
       "      <td>2.119652</td>\n",
       "      <td>0.317543</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.636551</td>\n",
       "      <td>2.187433</td>\n",
       "      <td>0.316974</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.532861</td>\n",
       "      <td>2.035595</td>\n",
       "      <td>0.326776</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.430445</td>\n",
       "      <td>1.773961</td>\n",
       "      <td>0.470099</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.328493</td>\n",
       "      <td>1.769809</td>\n",
       "      <td>0.499077</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.215091</td>\n",
       "      <td>1.731835</td>\n",
       "      <td>0.485938</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.111577</td>\n",
       "      <td>1.735109</td>\n",
       "      <td>0.486861</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.023528</td>\n",
       "      <td>1.693871</td>\n",
       "      <td>0.514773</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.951323</td>\n",
       "      <td>1.753329</td>\n",
       "      <td>0.519744</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.892863</td>\n",
       "      <td>1.868084</td>\n",
       "      <td>0.488068</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.837591</td>\n",
       "      <td>1.921619</td>\n",
       "      <td>0.490199</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.791960</td>\n",
       "      <td>1.997633</td>\n",
       "      <td>0.495881</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>0.757444</td>\n",
       "      <td>2.012827</td>\n",
       "      <td>0.499716</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>0.730740</td>\n",
       "      <td>2.069066</td>\n",
       "      <td>0.498793</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>0.711294</td>\n",
       "      <td>2.071262</td>\n",
       "      <td>0.498295</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18</td>\n",
       "      <td>0.698335</td>\n",
       "      <td>2.060519</td>\n",
       "      <td>0.501207</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>0.691205</td>\n",
       "      <td>2.072425</td>\n",
       "      <td>0.500639</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(20, 3e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are getting greater accuracy than before!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## nn.RNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's refactor the above to use PyTorch's RNN.  This is what you would use in practice, but now you know the inside details!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Model4(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.i_h = nn.Embedding(nv,nh)\n",
    "        self.rnn = nn.RNN(nh,nh, batch_first=True)\n",
    "        self.h_o = nn.Linear(nh,nv)\n",
    "        self.bn = BatchNorm1dFlat(nh)\n",
    "        self.h = torch.zeros(1, bs, nh).cuda()\n",
    "        \n",
    "    def forward(self, x):\n",
    "        res,h = self.rnn(self.i_h(x), self.h)\n",
    "        self.h = h.detach()\n",
    "        return self.h_o(self.bn(res))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(data, Model4(), metrics=accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.496743</td>\n",
       "      <td>3.210747</td>\n",
       "      <td>0.325071</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.818120</td>\n",
       "      <td>2.272267</td>\n",
       "      <td>0.463068</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.142118</td>\n",
       "      <td>2.004856</td>\n",
       "      <td>0.462358</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.772554</td>\n",
       "      <td>2.038238</td>\n",
       "      <td>0.318892</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.565121</td>\n",
       "      <td>1.870137</td>\n",
       "      <td>0.449787</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.364768</td>\n",
       "      <td>1.659184</td>\n",
       "      <td>0.527912</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.170237</td>\n",
       "      <td>1.556142</td>\n",
       "      <td>0.572088</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.001541</td>\n",
       "      <td>1.505949</td>\n",
       "      <td>0.592756</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.855792</td>\n",
       "      <td>1.564825</td>\n",
       "      <td>0.619673</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.730979</td>\n",
       "      <td>1.533880</td>\n",
       "      <td>0.616690</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.627127</td>\n",
       "      <td>1.561256</td>\n",
       "      <td>0.608310</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.546634</td>\n",
       "      <td>1.514320</td>\n",
       "      <td>0.621591</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.481095</td>\n",
       "      <td>1.479670</td>\n",
       "      <td>0.624361</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.427647</td>\n",
       "      <td>1.425912</td>\n",
       "      <td>0.627770</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.384686</td>\n",
       "      <td>1.460788</td>\n",
       "      <td>0.631960</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>0.352259</td>\n",
       "      <td>1.381645</td>\n",
       "      <td>0.647159</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>0.327646</td>\n",
       "      <td>1.403838</td>\n",
       "      <td>0.644531</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>0.309528</td>\n",
       "      <td>1.438777</td>\n",
       "      <td>0.642258</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18</td>\n",
       "      <td>0.297203</td>\n",
       "      <td>1.415845</td>\n",
       "      <td>0.647301</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>0.290220</td>\n",
       "      <td>1.431366</td>\n",
       "      <td>0.645952</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(20, 3e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2-layer GRU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you have long time scales and deeper networks, these become impossible to train.  One way to address this is to add mini-NN to decide how much of the green arrow and how much of the orange arrow to keep.  These mini-NNs can be GRUs or LSTMs.  We will cover more details of this in a later lesson."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Model5(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.i_h = nn.Embedding(nv,nh)\n",
    "        self.rnn = nn.GRU(nh, nh, 2, batch_first=True)\n",
    "        self.h_o = nn.Linear(nh,nv)\n",
    "        self.bn = BatchNorm1dFlat(nh)\n",
    "        self.h = torch.zeros(2, bs, nh).cuda()\n",
    "        \n",
    "    def forward(self, x):\n",
    "        res,h = self.rnn(self.i_h(x), self.h)\n",
    "        print(res.mean(),h.mean())\n",
    "        self.h = h.detach()\n",
    "        return self.h_o(self.bn(res))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(data, Model5(), metrics=accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2.650547</td>\n",
       "      <td>2.230762</td>\n",
       "      <td>0.462784</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1.582101</td>\n",
       "      <td>1.484121</td>\n",
       "      <td>0.610440</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.779965</td>\n",
       "      <td>0.902215</td>\n",
       "      <td>0.810938</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.382889</td>\n",
       "      <td>0.983872</td>\n",
       "      <td>0.833878</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.193930</td>\n",
       "      <td>0.875055</td>\n",
       "      <td>0.841619</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0.103433</td>\n",
       "      <td>0.799085</td>\n",
       "      <td>0.850497</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.058285</td>\n",
       "      <td>0.856717</td>\n",
       "      <td>0.842969</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.035402</td>\n",
       "      <td>0.904172</td>\n",
       "      <td>0.837145</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.023261</td>\n",
       "      <td>0.905010</td>\n",
       "      <td>0.840909</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.016818</td>\n",
       "      <td>0.903877</td>\n",
       "      <td>0.840696</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(-0.0043, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0121, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0049, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0102, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0043, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0092, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0045, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0106, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0037, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0092, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0036, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0089, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0042, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0084, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0040, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0107, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0039, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0118, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0034, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0066, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0029, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0076, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0030, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0113, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0026, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0104, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0027, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0102, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0024, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0093, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0032, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0074, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0032, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0106, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0034, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0100, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0027, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0110, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0027, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0083, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0017, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0095, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0012, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0087, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0009, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0093, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0006, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0084, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(-0.0006, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0114, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0005, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0082, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0003, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0060, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0009, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0059, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0020, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0093, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0027, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0091, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0039, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0039, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0043, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0046, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0060, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0075, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0071, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0058, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0090, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0048, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0105, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0088, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0122, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0062, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0143, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0028, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0155, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0056, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0174, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0025, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0156, device='cuda:1') tensor(-0.0087, device='cuda:1')\n",
      "tensor(0.0159, device='cuda:1') tensor(-0.0097, device='cuda:1')\n",
      "tensor(0.0165, device='cuda:1') tensor(-0.0095, device='cuda:1')\n",
      "tensor(0.0162, device='cuda:1') tensor(-0.0116, device='cuda:1')\n",
      "tensor(0.0161, device='cuda:1') tensor(-0.0088, device='cuda:1')\n",
      "tensor(0.0161, device='cuda:1') tensor(-0.0120, device='cuda:1')\n",
      "tensor(0.0164, device='cuda:1') tensor(-0.0120, device='cuda:1')\n",
      "tensor(0.0169, device='cuda:1') tensor(-0.0096, device='cuda:1')\n",
      "tensor(0.0169, device='cuda:1') tensor(-0.0100, device='cuda:1')\n",
      "tensor(0.0164, device='cuda:1') tensor(-0.0146, device='cuda:1')\n",
      "tensor(0.0165, device='cuda:1') tensor(-0.0115, device='cuda:1')\n",
      "tensor(0.0193, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0017, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0211, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0002, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0232, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0007, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0250, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0010, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0272, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0022, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0282, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0017, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0293, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0029, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0307, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0009, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0317, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0002, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0335, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0062, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0344, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0039, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0353, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0010, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0369, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0036, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0387, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0039, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0405, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0046, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0418, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0079, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0437, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0042, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0448, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0059, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0472, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0090, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0478, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0090, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0511, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0096, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0529, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0135, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0557, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0158, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0585, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0154, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0597, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0140, device='cuda:1', grad_fn=<MeanBackward0>)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.0586, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0212, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0592, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0179, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0595, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0183, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0531, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0175, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0533, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0139, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0477, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0093, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0447, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0083, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0407, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0097, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0384, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0025, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0396, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0105, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0394, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0043, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0393, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0064, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0368, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0064, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0378, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0062, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0328, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0064, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0277, device='cuda:1') tensor(-0.0059, device='cuda:1')\n",
      "tensor(0.0277, device='cuda:1') tensor(-0.0094, device='cuda:1')\n",
      "tensor(0.0293, device='cuda:1') tensor(-0.0077, device='cuda:1')\n",
      "tensor(0.0291, device='cuda:1') tensor(-0.0080, device='cuda:1')\n",
      "tensor(0.0285, device='cuda:1') tensor(-0.0049, device='cuda:1')\n",
      "tensor(0.0291, device='cuda:1') tensor(-0.0073, device='cuda:1')\n",
      "tensor(0.0298, device='cuda:1') tensor(-0.0093, device='cuda:1')\n",
      "tensor(0.0296, device='cuda:1') tensor(-0.0075, device='cuda:1')\n",
      "tensor(0.0288, device='cuda:1') tensor(-0.0080, device='cuda:1')\n",
      "tensor(0.0301, device='cuda:1') tensor(-0.0097, device='cuda:1')\n",
      "tensor(0.0296, device='cuda:1') tensor(-0.0084, device='cuda:1')\n",
      "tensor(0.0294, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0026, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0203, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0002, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0216, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0043, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0214, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0047, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0242, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0030, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0283, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0009, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0321, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0046, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0329, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0009, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0305, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0021, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0242, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0010, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0232, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0009, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0224, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0016, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0199, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0016, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0209, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0003, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0186, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0031, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0177, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0027, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0180, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0012, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0186, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0002, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0172, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0034, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0168, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0004, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0246, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0014, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0293, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0043, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0321, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0072, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0268, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0012, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0230, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0061, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0192, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0056, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0199, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0003, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0231, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0016, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0209, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0063, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0192, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0007, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0194, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0002, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0230, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0007, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0226, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0004, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0260, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0009, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0298, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0045, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0298, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0013, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0259, device='cuda:1', grad_fn=<MeanBackward0>) tensor(8.4004e-05, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0246, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0008, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0255, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0003, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0217, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0004, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0132, device='cuda:1') tensor(-0.0052, device='cuda:1')\n",
      "tensor(0.0161, device='cuda:1') tensor(-0.0090, device='cuda:1')\n",
      "tensor(0.0172, device='cuda:1') tensor(-0.0095, device='cuda:1')\n",
      "tensor(0.0172, device='cuda:1') tensor(-0.0095, device='cuda:1')\n",
      "tensor(0.0174, device='cuda:1') tensor(-0.0040, device='cuda:1')\n",
      "tensor(0.0182, device='cuda:1') tensor(-0.0075, device='cuda:1')\n",
      "tensor(0.0165, device='cuda:1') tensor(-0.0113, device='cuda:1')\n",
      "tensor(0.0169, device='cuda:1') tensor(-0.0069, device='cuda:1')\n",
      "tensor(0.0166, device='cuda:1') tensor(-0.0102, device='cuda:1')\n",
      "tensor(0.0161, device='cuda:1') tensor(-0.0132, device='cuda:1')\n",
      "tensor(0.0161, device='cuda:1') tensor(-0.0086, device='cuda:1')\n",
      "tensor(0.0207, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0002, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0121, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0046, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0140, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0008, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0159, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0007, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0171, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0059, device='cuda:1', grad_fn=<MeanBackward0>)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.0136, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0018, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0091, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0081, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0088, device='cuda:1', grad_fn=<MeanBackward0>) tensor(4.8079e-05, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0097, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0047, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0149, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0135, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0203, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0136, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0244, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0105, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0250, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0088, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0266, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0111, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0267, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0118, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0283, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0109, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0277, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0087, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0269, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0060, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0246, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0059, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0214, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0100, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0225, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0067, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0227, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0094, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0213, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0122, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0192, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0067, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0173, device='cuda:1', grad_fn=<MeanBackward0>) tensor(-0.0024, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0161, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0126, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0175, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0086, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0167, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0054, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0164, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0118, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0141, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0052, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0132, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0045, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0167, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0064, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0179, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0056, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0222, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0081, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0231, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0124, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0230, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0093, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0229, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0086, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0230, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0127, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0237, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0086, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0189, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0061, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0093, device='cuda:1') tensor(0.0001, device='cuda:1')\n",
      "tensor(0.0109, device='cuda:1') tensor(-0.0041, device='cuda:1')\n",
      "tensor(0.0126, device='cuda:1') tensor(-0.0063, device='cuda:1')\n",
      "tensor(0.0132, device='cuda:1') tensor(-0.0050, device='cuda:1')\n",
      "tensor(0.0131, device='cuda:1') tensor(0.0004, device='cuda:1')\n",
      "tensor(0.0137, device='cuda:1') tensor(-0.0030, device='cuda:1')\n",
      "tensor(0.0122, device='cuda:1') tensor(-0.0067, device='cuda:1')\n",
      "tensor(0.0122, device='cuda:1') tensor(-0.0013, device='cuda:1')\n",
      "tensor(0.0121, device='cuda:1') tensor(-0.0053, device='cuda:1')\n",
      "tensor(0.0115, device='cuda:1') tensor(-0.0079, device='cuda:1')\n",
      "tensor(0.0114, device='cuda:1') tensor(-0.0017, device='cuda:1')\n",
      "tensor(0.0208, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0069, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0224, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0133, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0274, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0129, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0273, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0142, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0270, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0203, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0242, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0103, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0237, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0051, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0240, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0100, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0250, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0112, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0277, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0212, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0309, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0213, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0297, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0138, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0285, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0124, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0295, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0165, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0292, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0178, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0306, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0185, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0334, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0211, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0355, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0176, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0342, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0160, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0319, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0190, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0322, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0130, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0308, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0140, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0275, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0186, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0234, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0121, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0212, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0025, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0196, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0171, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0214, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0124, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0203, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0085, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0200, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0143, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0183, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0098, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0166, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0092, device='cuda:1', grad_fn=<MeanBackward0>)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.0176, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0100, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0174, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0083, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0199, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0077, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0190, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0091, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0187, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0053, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0171, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0040, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0169, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0074, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0171, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0050, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0131, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0017, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0037, device='cuda:1') tensor(-0.0023, device='cuda:1')\n",
      "tensor(0.0078, device='cuda:1') tensor(-0.0053, device='cuda:1')\n",
      "tensor(0.0091, device='cuda:1') tensor(-0.0086, device='cuda:1')\n",
      "tensor(0.0101, device='cuda:1') tensor(-0.0073, device='cuda:1')\n",
      "tensor(0.0097, device='cuda:1') tensor(-0.0016, device='cuda:1')\n",
      "tensor(0.0107, device='cuda:1') tensor(-0.0053, device='cuda:1')\n",
      "tensor(0.0090, device='cuda:1') tensor(-0.0088, device='cuda:1')\n",
      "tensor(0.0090, device='cuda:1') tensor(-0.0021, device='cuda:1')\n",
      "tensor(0.0090, device='cuda:1') tensor(-0.0058, device='cuda:1')\n",
      "tensor(0.0081, device='cuda:1') tensor(-0.0086, device='cuda:1')\n",
      "tensor(0.0078, device='cuda:1') tensor(-0.0037, device='cuda:1')\n",
      "tensor(0.0157, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0051, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0124, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0088, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0125, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0048, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0119, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0063, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0123, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0139, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0099, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0060, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0083, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0013, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0084, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0070, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0100, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0080, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0128, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0150, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0167, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0152, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0163, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0060, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0165, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0053, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0181, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0113, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0184, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0103, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0205, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0123, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0237, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0168, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0265, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0135, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0263, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0132, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0255, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0174, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0276, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0110, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0279, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0115, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0261, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0163, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0224, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0088, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0214, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0013, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0204, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0150, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0224, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0095, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0208, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0046, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0197, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0103, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0184, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0056, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0163, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0062, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0172, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0073, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0170, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0055, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0192, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0081, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0174, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0104, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0165, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0039, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0153, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0034, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0156, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0077, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0163, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0051, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0135, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0008, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0057, device='cuda:1') tensor(-0.0019, device='cuda:1')\n",
      "tensor(0.0116, device='cuda:1') tensor(-0.0060, device='cuda:1')\n",
      "tensor(0.0134, device='cuda:1') tensor(-0.0079, device='cuda:1')\n",
      "tensor(0.0149, device='cuda:1') tensor(-0.0077, device='cuda:1')\n",
      "tensor(0.0137, device='cuda:1') tensor(-0.0023, device='cuda:1')\n",
      "tensor(0.0144, device='cuda:1') tensor(-0.0063, device='cuda:1')\n",
      "tensor(0.0137, device='cuda:1') tensor(-0.0104, device='cuda:1')\n",
      "tensor(0.0128, device='cuda:1') tensor(-0.0030, device='cuda:1')\n",
      "tensor(0.0128, device='cuda:1') tensor(-0.0061, device='cuda:1')\n",
      "tensor(0.0123, device='cuda:1') tensor(-0.0095, device='cuda:1')\n",
      "tensor(0.0117, device='cuda:1') tensor(-0.0045, device='cuda:1')\n",
      "tensor(0.0187, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0052, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0178, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0084, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0208, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0049, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0225, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0075, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0244, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0153, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0232, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0089, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0225, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0073, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0225, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0133, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0241, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0152, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0262, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0224, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0286, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0204, device='cuda:1', grad_fn=<MeanBackward0>)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.0255, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0104, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0233, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0088, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0226, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0126, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0206, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0115, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0211, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0120, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0218, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0149, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0236, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0123, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0221, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0126, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0216, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0152, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0228, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0106, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0231, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0109, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0215, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0174, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0187, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0100, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0183, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0034, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0173, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0177, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0192, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0121, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0175, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0069, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0163, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0124, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0156, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0079, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0141, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0085, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0148, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0101, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0152, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0072, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0175, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0107, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0158, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0120, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0152, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0055, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0145, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0056, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0148, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0082, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0152, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0067, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0122, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0024, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0031, device='cuda:1') tensor(-0.0003, device='cuda:1')\n",
      "tensor(0.0098, device='cuda:1') tensor(-0.0038, device='cuda:1')\n",
      "tensor(0.0117, device='cuda:1') tensor(-0.0063, device='cuda:1')\n",
      "tensor(0.0132, device='cuda:1') tensor(-0.0056, device='cuda:1')\n",
      "tensor(0.0119, device='cuda:1') tensor(-0.0005, device='cuda:1')\n",
      "tensor(0.0124, device='cuda:1') tensor(-0.0049, device='cuda:1')\n",
      "tensor(0.0120, device='cuda:1') tensor(-0.0085, device='cuda:1')\n",
      "tensor(0.0111, device='cuda:1') tensor(-0.0013, device='cuda:1')\n",
      "tensor(0.0111, device='cuda:1') tensor(-0.0045, device='cuda:1')\n",
      "tensor(0.0108, device='cuda:1') tensor(-0.0079, device='cuda:1')\n",
      "tensor(0.0103, device='cuda:1') tensor(-0.0031, device='cuda:1')\n",
      "tensor(0.0171, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0060, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0151, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0076, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0169, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0032, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0176, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0052, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0196, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0123, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0183, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0064, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0173, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0043, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0176, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0094, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0187, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0111, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0213, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0189, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0239, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0172, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0220, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0085, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0209, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0084, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0206, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0124, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0197, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0120, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0207, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0128, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0228, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0167, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0252, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0145, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0250, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0153, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0252, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0184, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0270, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0131, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0283, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0148, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0275, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0211, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0256, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0145, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0258, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0073, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0258, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0228, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0276, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0169, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0259, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0104, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0248, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0176, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0240, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0118, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0226, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0128, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0230, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0150, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0232, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0122, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0251, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0162, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0235, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0184, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0227, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0118, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0222, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0118, device='cuda:1', grad_fn=<MeanBackward0>)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.0219, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0146, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0218, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0127, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0189, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0086, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0112, device='cuda:1') tensor(0.0049, device='cuda:1')\n",
      "tensor(0.0184, device='cuda:1') tensor(0.0019, device='cuda:1')\n",
      "tensor(0.0207, device='cuda:1') tensor(-0.0008, device='cuda:1')\n",
      "tensor(0.0222, device='cuda:1') tensor(-0.0013, device='cuda:1')\n",
      "tensor(0.0203, device='cuda:1') tensor(0.0052, device='cuda:1')\n",
      "tensor(0.0212, device='cuda:1') tensor(7.6392e-05, device='cuda:1')\n",
      "tensor(0.0207, device='cuda:1') tensor(-0.0030, device='cuda:1')\n",
      "tensor(0.0196, device='cuda:1') tensor(0.0042, device='cuda:1')\n",
      "tensor(0.0198, device='cuda:1') tensor(0.0013, device='cuda:1')\n",
      "tensor(0.0195, device='cuda:1') tensor(-0.0020, device='cuda:1')\n",
      "tensor(0.0190, device='cuda:1') tensor(0.0024, device='cuda:1')\n",
      "tensor(0.0240, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0128, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0214, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0133, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0228, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0082, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0229, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0096, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0248, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0159, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0232, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0094, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0220, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0065, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0220, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0119, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0229, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0127, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0250, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0201, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0273, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0181, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0256, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0089, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0241, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0082, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0234, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0117, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0221, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0107, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0226, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0104, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0239, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0148, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0258, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0123, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0253, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0126, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0251, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0159, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0266, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0106, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0275, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0120, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0266, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0187, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0250, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0123, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0254, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0055, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0251, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0209, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0275, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0154, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0261, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0084, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0255, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0162, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0255, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0113, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0243, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0123, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0254, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0145, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0259, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0122, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0280, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0163, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0272, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0189, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0269, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0127, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0266, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0132, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0266, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0160, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0270, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0144, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0239, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0103, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0160, device='cuda:1') tensor(0.0078, device='cuda:1')\n",
      "tensor(0.0236, device='cuda:1') tensor(0.0042, device='cuda:1')\n",
      "tensor(0.0256, device='cuda:1') tensor(0.0021, device='cuda:1')\n",
      "tensor(0.0272, device='cuda:1') tensor(0.0019, device='cuda:1')\n",
      "tensor(0.0254, device='cuda:1') tensor(0.0080, device='cuda:1')\n",
      "tensor(0.0263, device='cuda:1') tensor(0.0030, device='cuda:1')\n",
      "tensor(0.0260, device='cuda:1') tensor(-0.0004, device='cuda:1')\n",
      "tensor(0.0249, device='cuda:1') tensor(0.0068, device='cuda:1')\n",
      "tensor(0.0250, device='cuda:1') tensor(0.0038, device='cuda:1')\n",
      "tensor(0.0247, device='cuda:1') tensor(0.0006, device='cuda:1')\n",
      "tensor(0.0241, device='cuda:1') tensor(0.0049, device='cuda:1')\n",
      "tensor(0.0293, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0157, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0265, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0161, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0273, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0107, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0270, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0124, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0285, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0189, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0267, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0126, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0252, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0095, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0249, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0150, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0253, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0157, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0272, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0228, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0294, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0211, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0272, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0117, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0256, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0113, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0247, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0146, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0231, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0133, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0234, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0129, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0244, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0172, device='cuda:1', grad_fn=<MeanBackward0>)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.0260, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0146, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0254, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0144, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0249, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0174, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0263, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0122, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0269, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0132, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0260, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0195, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0243, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0130, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0246, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0061, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0241, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0211, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0266, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0155, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0251, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0085, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0244, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0161, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0245, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0111, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0233, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0122, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0244, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0143, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0250, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0118, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0272, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0161, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0264, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0187, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0261, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0124, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0258, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0128, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0260, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0157, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0265, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0140, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0235, device='cuda:1', grad_fn=<MeanBackward0>) tensor(0.0101, device='cuda:1', grad_fn=<MeanBackward0>)\n",
      "tensor(0.0157, device='cuda:1') tensor(0.0076, device='cuda:1')\n",
      "tensor(0.0233, device='cuda:1') tensor(0.0040, device='cuda:1')\n",
      "tensor(0.0252, device='cuda:1') tensor(0.0018, device='cuda:1')\n",
      "tensor(0.0267, device='cuda:1') tensor(0.0015, device='cuda:1')\n",
      "tensor(0.0250, device='cuda:1') tensor(0.0078, device='cuda:1')\n",
      "tensor(0.0259, device='cuda:1') tensor(0.0027, device='cuda:1')\n",
      "tensor(0.0256, device='cuda:1') tensor(-0.0008, device='cuda:1')\n",
      "tensor(0.0244, device='cuda:1') tensor(0.0066, device='cuda:1')\n",
      "tensor(0.0246, device='cuda:1') tensor(0.0036, device='cuda:1')\n",
      "tensor(0.0243, device='cuda:1') tensor(0.0003, device='cuda:1')\n",
      "tensor(0.0237, device='cuda:1') tensor(0.0047, device='cuda:1')\n"
     ]
    }
   ],
   "source": [
    "learn.fit_one_cycle(10, 1e-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connection to ULMFit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous lesson, we were essentially swapping out `self.h_o` with a classifier in order to do classification on text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## fin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RNNs are just a refactored, fully-connected neural network.\n",
    "\n",
    "You can use the same approach for any sequence labeling task (part of speech, classifying whether material is sensitive,..)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}