NLPImplementation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### *Deep Dive into*\n",
    "# Natural Language Processing\n",
    "\n",
    "#### Contributors\n",
    "Helene Willits,\n",
    "Shaina Bagri,\n",
    "Rachel Castellino"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook builds on the notebook titled \"An Introduction to Natural Language Processing\" by the same contributors. Here, we explore more of the details of NLP and provide a work-along example that will give you an introduction on how to work with NLP."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's say we want to build a model that predicts the next word a user will type. We will show you how you can process a set of training data, train a Natural Language Processing model, and use that model to create a text predictor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Libraries\n",
    "First, we need to import the required libraries. The majority of the libraries needed fall under the tensorflow overall library, which is very common in artificial intelligence and machine learning. Using the tensorflow libraries allows us to access various machine learning models, layers, and preprocessing techniques without having to manually code them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.layers import Embedding, LSTM, Dense\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "from tensorflow.keras.callbacks import ModelCheckpoint\n",
    "from tensorflow.keras.callbacks import ReduceLROnPlateau\n",
    "from tensorflow.keras.callbacks import TensorBoard\n",
    "from tensorflow import keras\n",
    "from keras.utils.vis_utils import plot_model\n",
    "import numpy as np\n",
    "import os\n",
    "import requests\n",
    "import string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gather Data\n",
    "\n",
    "You can use any data that you like to train the model, but the dataset needs to be large enough to aptly train our model to predict which word will come next in a text stream. We are going to develop our model with the text from the book Metamorphosis by Franz Kafka. If you are coding along with this guide, you can access the text file here: https://www.gutenberg.org/cache/epub/5200/pg5200.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://www.gutenberg.org/cache/epub/5200/pg5200.txt'\n",
    "r = requests.get(url, allow_redirects=True)\n",
    "\n",
    "open('metamorphosis_gutenberg.txt', 'wb').write(r.content)\n",
    "file = open('metamorphosis_gutenberg.txt', \"r\", encoding = \"utf8\")\n",
    "allLines = []\n",
    "\n",
    "for i in file:\n",
    "    allLines.append(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing the Data\n",
    "The first step in developing an Natural Language Processing model is to perform some preprocessing on the data that will remove any unnecessary data. This step includes both context-based preprocessing and standard NLP processing. \n",
    "\n",
    "The goal of preprocessing is to get the data that looks like this:\n",
    "\n",
    "**The quick red fox jumped over the lazy dog.**\n",
    "\n",
    "**The dog waited, and, to her surprise, she was untouched.**\n",
    "\n",
    "To look like this:\n",
    "\n",
    "**The quick red fox jumped over lazy dog  waited  and  to her surprise  she was untouched**\n",
    "\n",
    "In the context of our dataset (a novel) there is data such as the copyright information that will note be useful in building our NLP model. We can pick out the useful text from the file using the following code.\n",
    "\n",
    "NOTE : this code is specific to this data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The First Line:  One morning, when Gregor Samsa woke from troubled dreams, he found\n",
      "\n",
      "The Last Line:  first to get up and stretch out her young body.\n",
      "\n",
      "Number of Lines:  1946\n"
     ]
    }
   ],
   "source": [
    "lines = allLines[46:1992]\n",
    "print(\"The First Line: \", lines[0])\n",
    "print(\"The Last Line: \", lines[-1])\n",
    "print(\"Number of Lines: \", len(lines))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also need to reformat the text so that it is easier to process. \n",
    "\n",
    "In order to do this, we will first remove unnecessary characters. For example, some characters we will remove are the newline ('\\n') and carriage return ('\\r') characters that are used in text files to signal different types of spacing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'One morning, when Gregor Samsa woke from troubled dreams, he found himself transformed in his bed into a horrible vermin.  He lay on his armour-like back, and if he lifted his head a little he could see his brown belly, slightly domed and divided by arches into stiff sections.  The bedding was hardly able to cover it and seemed ready to slide off any moment.'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = \"\"\n",
    "\n",
    "for i in lines:\n",
    "    data = ' '. join(lines)\n",
    "    \n",
    "data = data.replace('\\n', '').replace('\\r', '').replace('\\ufeff', '')\n",
    "data[:360]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to remove punctuation, so we will convert any punctation marks into spaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'One morning  when Gregor Samsa woke from troubled dreams  he found himself transformed in his bed into a horrible vermin   He lay on his armour like back  and if he lifted his head a little he could see his brown belly  slightly domed and divided by arches into stiff sections   The bedding was hardly able to cover it and seemed ready to slide off any moment   His many legs  pitifully thin compared with the size of the rest of him  waved about helplessly as he looked    What s happened to me   he'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "translator = str.maketrans(string.punctuation, ' '*len(string.punctuation)) #map punctuation to space\n",
    "new_data = data.translate(translator)\n",
    "\n",
    "new_data[:500]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll then copy over the text into a new data object without duplicating any words. This simplifies the problem that our model must solve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'One morning, when Gregor Samsa woke from troubled dreams, he found himself transformed in his bed into a horrible vermin. He lay on armour-like back, and if lifted head little could see brown belly, slightly domed divided by arches stiff sections. The bedding was hardly able to cover it seemed ready slide off any moment. His many legs, pitifully thin compared with the size of rest him, waved about helplessly as looked. \"What\\'s happened me?\" thought. It wasn\\'t dream. room, proper human room altho'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = []\n",
    "\n",
    "for i in data.split():\n",
    "    if i not in z:\n",
    "        z.append(i)\n",
    "        \n",
    "data = ' '.join(z)\n",
    "data[:500]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will preform some preprocessing steps that are standard components of NLP."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization\n",
    "\n",
    "Remember from our Introduction to NLP that tokenization is when we break up the text into words or phrases so that later we can identify the relationships between them. There are many ways that we could do this, but we will use the Tokenizer library created by Keras. This tokenizer represents the words in the text as vectors where each word, or token, in the text is assigned a number. This number can represent the index of the word in our data set, it could be a measure of the relevancy of each word, or any one of many other statistics. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[17, 53, 293, 2, 18, 729, 135, 730, 294, 8]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer = Tokenizer()\n",
    "tokenizer.fit_on_texts([data])\n",
    "\n",
    "sequence_data = tokenizer.texts_to_sequences([data])[0]\n",
    "sequence_data[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2617\n"
     ]
    }
   ],
   "source": [
    "vocab_size = len(tokenizer.word_index) + 1\n",
    "print(vocab_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Length of sequences are:  3889\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[ 17,  53],\n",
       "       [ 53, 293],\n",
       "       [293,   2],\n",
       "       [  2,  18],\n",
       "       [ 18, 729],\n",
       "       [729, 135],\n",
       "       [135, 730],\n",
       "       [730, 294],\n",
       "       [294,   8],\n",
       "       [  8, 731]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequences = []\n",
    "\n",
    "for i in range(1, len(sequence_data)):\n",
    "    words = sequence_data[i-1:i+1]\n",
    "    sequences.append(words)\n",
    "    \n",
    "print(\"The Length of sequences are: \", len(sequences))\n",
    "sequences = np.array(sequences)\n",
    "sequences[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can formuate two parameters - X, the training data with the input data, and y, the output of analyzing that data, meaning that it contains the predictions for the next word for each data point in X."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = []\n",
    "y = []\n",
    "\n",
    "for i in sequences:\n",
    "    X.append(i[0])\n",
    "    y.append(i[1])\n",
    "    \n",
    "X = np.array(X)\n",
    "y = np.array(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Data is:  [ 17  53 293   2  18]\n",
      "The responses are:  [ 53 293   2  18 729]\n"
     ]
    }
   ],
   "source": [
    "print(\"The Data is: \", X[:5])\n",
    "print(\"The responses are: \", y[:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 1., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.]], dtype=float32)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = to_categorical(y, num_classes=vocab_size)\n",
    "y[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we create the model to sequentially go through LSTM and Dense layers. Long-short-term memory, which builds off neural networks, was discussed in more detail in the conceptual notebook titled \"An Introduction to Natural Language Processing\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Embedding(vocab_size, 10, input_length=1))\n",
    "model.add(LSTM(1000, return_sequences=True))\n",
    "model.add(LSTM(1000))\n",
    "model.add(Dense(1000, activation=\"relu\"))\n",
    "model.add(Dense(vocab_size, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding (Embedding)        (None, 1, 10)             26170     \n",
      "_________________________________________________________________\n",
      "lstm (LSTM)                  (None, 1, 1000)           4044000   \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (None, 1000)              8004000   \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 1000)              1001000   \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 2617)              2619617   \n",
      "=================================================================\n",
      "Total params: 15,694,787\n",
      "Trainable params: 15,694,787\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plotting the model helps us to better visualize and understand the layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('You must install pydot (`pip install pydot`) and install graphviz (see instructions at https://graphviz.gitlab.io/download/) ', 'for plot_model/model_to_dot to work.')\n"
     ]
    }
   ],
   "source": [
    "keras.utils.plot_model(model, to_file='model.png', show_layer_names=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Callbacks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Callbacks allow the model to constantly evaluate itself while it is being trained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint = ModelCheckpoint(\"nextword1.h5\", monitor='loss', verbose=1,\n",
    "    save_best_only=True, mode='auto')\n",
    "\n",
    "reduce = ReduceLROnPlateau(monitor='loss', factor=0.2, patience=3, min_lr=0.0001, verbose = 1)\n",
    "\n",
    "logdir='logsnextword1'\n",
    "tensorboard_Visualization = TensorBoard(log_dir=logdir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compile the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we need to compile the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/tljh/user/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:375: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  \"The `lr` argument is deprecated, use `learning_rate` instead.\")\n"
     ]
    }
   ],
   "source": [
    "model.compile(loss=\"categorical_crossentropy\", optimizer=Adam(lr=0.001))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fit the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we fit the dataset we preprocessed to the model and trained the model on it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/150\n",
      "61/61 [==============================] - 8s 90ms/step - loss: 7.8753\n",
      "\n",
      "Epoch 00001: loss improved from inf to 7.87525, saving model to nextword1.h5\n",
      "Epoch 2/150\n",
      "61/61 [==============================] - 5s 89ms/step - loss: 7.8604\n",
      "\n",
      "Epoch 00002: loss improved from 7.87525 to 7.86035, saving model to nextword1.h5\n",
      "Epoch 3/150\n",
      "61/61 [==============================] - 5s 89ms/step - loss: 7.8167\n",
      "\n",
      "Epoch 00003: loss improved from 7.86035 to 7.81674, saving model to nextword1.h5\n",
      "Epoch 4/150\n",
      "61/61 [==============================] - 5s 89ms/step - loss: 7.6386\n",
      "\n",
      "Epoch 00004: loss improved from 7.81674 to 7.63865, saving model to nextword1.h5\n",
      "Epoch 5/150\n",
      "61/61 [==============================] - 5s 89ms/step - loss: 7.4415\n",
      "\n",
      "Epoch 00005: loss improved from 7.63865 to 7.44145, saving model to nextword1.h5\n",
      "Epoch 6/150\n",
      "61/61 [==============================] - 6s 95ms/step - loss: 7.2547\n",
      "\n",
      "Epoch 00006: loss improved from 7.44145 to 7.25470, saving model to nextword1.h5\n",
      "Epoch 7/150\n",
      "61/61 [==============================] - 5s 89ms/step - loss: 7.1157\n",
      "\n",
      "Epoch 00007: loss improved from 7.25470 to 7.11574, saving model to nextword1.h5\n",
      "Epoch 8/150\n",
      "61/61 [==============================] - 5s 89ms/step - loss: 6.9198\n",
      "\n",
      "Epoch 00008: loss improved from 7.11574 to 6.91975, saving model to nextword1.h5\n",
      "Epoch 9/150\n",
      "61/61 [==============================] - 5s 89ms/step - loss: 6.6485\n",
      "\n",
      "Epoch 00009: loss improved from 6.91975 to 6.64847, saving model to nextword1.h5\n",
      "Epoch 10/150\n",
      "61/61 [==============================] - 5s 89ms/step - loss: 6.3866\n",
      "\n",
      "Epoch 00010: loss improved from 6.64847 to 6.38661, saving model to nextword1.h5\n",
      "Epoch 11/150\n",
      "61/61 [==============================] - 5s 89ms/step - loss: 6.1720\n",
      "\n",
      "Epoch 00011: loss improved from 6.38661 to 6.17203, saving model to nextword1.h5\n",
      "Epoch 12/150\n",
      "61/61 [==============================] - 5s 89ms/step - loss: 5.9624\n",
      "\n",
      "Epoch 00012: loss improved from 6.17203 to 5.96239, saving model to nextword1.h5\n",
      "Epoch 13/150\n",
      "61/61 [==============================] - 5s 89ms/step - loss: 5.7586\n",
      "\n",
      "Epoch 00013: loss improved from 5.96239 to 5.75857, saving model to nextword1.h5\n",
      "Epoch 14/150\n",
      "61/61 [==============================] - 5s 89ms/step - loss: 5.5872\n",
      "\n",
      "Epoch 00014: loss improved from 5.75857 to 5.58723, saving model to nextword1.h5\n",
      "Epoch 15/150\n",
      "61/61 [==============================] - 5s 89ms/step - loss: 5.4251\n",
      "\n",
      "Epoch 00015: loss improved from 5.58723 to 5.42511, saving model to nextword1.h5\n",
      "Epoch 16/150\n",
      "61/61 [==============================] - 6s 93ms/step - loss: 5.2825\n",
      "\n",
      "Epoch 00016: loss improved from 5.42511 to 5.28247, saving model to nextword1.h5\n",
      "Epoch 17/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 5.1566\n",
      "\n",
      "Epoch 00017: loss improved from 5.28247 to 5.15665, saving model to nextword1.h5\n",
      "Epoch 18/150\n",
      "61/61 [==============================] - 10s 171ms/step - loss: 5.0366\n",
      "\n",
      "Epoch 00018: loss improved from 5.15665 to 5.03657, saving model to nextword1.h5\n",
      "Epoch 19/150\n",
      "61/61 [==============================] - 11s 177ms/step - loss: 4.9151\n",
      "\n",
      "Epoch 00019: loss improved from 5.03657 to 4.91506, saving model to nextword1.h5\n",
      "Epoch 20/150\n",
      "61/61 [==============================] - 11s 180ms/step - loss: 4.8123\n",
      "\n",
      "Epoch 00020: loss improved from 4.91506 to 4.81227, saving model to nextword1.h5\n",
      "Epoch 21/150\n",
      "61/61 [==============================] - 11s 181ms/step - loss: 4.7332\n",
      "\n",
      "Epoch 00021: loss improved from 4.81227 to 4.73318, saving model to nextword1.h5\n",
      "Epoch 22/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 4.6523\n",
      "\n",
      "Epoch 00022: loss improved from 4.73318 to 4.65226, saving model to nextword1.h5\n",
      "Epoch 23/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 4.5868\n",
      "\n",
      "Epoch 00023: loss improved from 4.65226 to 4.58676, saving model to nextword1.h5\n",
      "Epoch 24/150\n",
      "61/61 [==============================] - 11s 177ms/step - loss: 4.5141\n",
      "\n",
      "Epoch 00024: loss improved from 4.58676 to 4.51412, saving model to nextword1.h5\n",
      "Epoch 25/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 4.4431\n",
      "\n",
      "Epoch 00025: loss improved from 4.51412 to 4.44306, saving model to nextword1.h5\n",
      "Epoch 26/150\n",
      "61/61 [==============================] - 11s 181ms/step - loss: 4.3829\n",
      "\n",
      "Epoch 00026: loss improved from 4.44306 to 4.38293, saving model to nextword1.h5\n",
      "Epoch 27/150\n",
      "61/61 [==============================] - 11s 181ms/step - loss: 4.3382\n",
      "\n",
      "Epoch 00027: loss improved from 4.38293 to 4.33817, saving model to nextword1.h5\n",
      "Epoch 28/150\n",
      "61/61 [==============================] - 11s 180ms/step - loss: 4.2553\n",
      "\n",
      "Epoch 00028: loss improved from 4.33817 to 4.25526, saving model to nextword1.h5\n",
      "Epoch 29/150\n",
      "61/61 [==============================] - 11s 181ms/step - loss: 4.1691\n",
      "\n",
      "Epoch 00029: loss improved from 4.25526 to 4.16909, saving model to nextword1.h5\n",
      "Epoch 30/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 4.0854\n",
      "\n",
      "Epoch 00030: loss improved from 4.16909 to 4.08541, saving model to nextword1.h5\n",
      "Epoch 31/150\n",
      "61/61 [==============================] - 11s 177ms/step - loss: 4.0095\n",
      "\n",
      "Epoch 00031: loss improved from 4.08541 to 4.00951, saving model to nextword1.h5\n",
      "Epoch 32/150\n",
      "61/61 [==============================] - 11s 178ms/step - loss: 3.8961\n",
      "\n",
      "Epoch 00032: loss improved from 4.00951 to 3.89613, saving model to nextword1.h5\n",
      "Epoch 33/150\n",
      "61/61 [==============================] - 11s 178ms/step - loss: 3.7962\n",
      "\n",
      "Epoch 00033: loss improved from 3.89613 to 3.79620, saving model to nextword1.h5\n",
      "Epoch 34/150\n",
      "61/61 [==============================] - 11s 180ms/step - loss: 3.6869\n",
      "\n",
      "Epoch 00034: loss improved from 3.79620 to 3.68694, saving model to nextword1.h5\n",
      "Epoch 35/150\n",
      "61/61 [==============================] - 11s 181ms/step - loss: 3.5832\n",
      "\n",
      "Epoch 00035: loss improved from 3.68694 to 3.58325, saving model to nextword1.h5\n",
      "Epoch 36/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 3.4426\n",
      "\n",
      "Epoch 00036: loss improved from 3.58325 to 3.44256, saving model to nextword1.h5\n",
      "Epoch 37/150\n",
      "61/61 [==============================] - 11s 177ms/step - loss: 3.3737\n",
      "\n",
      "Epoch 00037: loss improved from 3.44256 to 3.37368, saving model to nextword1.h5\n",
      "Epoch 38/150\n",
      "61/61 [==============================] - 11s 181ms/step - loss: 3.2673\n",
      "\n",
      "Epoch 00038: loss improved from 3.37368 to 3.26734, saving model to nextword1.h5\n",
      "Epoch 39/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 3.1244\n",
      "\n",
      "Epoch 00039: loss improved from 3.26734 to 3.12443, saving model to nextword1.h5\n",
      "Epoch 40/150\n",
      "61/61 [==============================] - 11s 180ms/step - loss: 3.0444\n",
      "\n",
      "Epoch 00040: loss improved from 3.12443 to 3.04445, saving model to nextword1.h5\n",
      "Epoch 41/150\n",
      "61/61 [==============================] - 11s 180ms/step - loss: 2.9398\n",
      "\n",
      "Epoch 00041: loss improved from 3.04445 to 2.93976, saving model to nextword1.h5\n",
      "Epoch 42/150\n",
      "61/61 [==============================] - 11s 178ms/step - loss: 2.8526\n",
      "\n",
      "Epoch 00042: loss improved from 2.93976 to 2.85263, saving model to nextword1.h5\n",
      "Epoch 43/150\n",
      "61/61 [==============================] - 11s 178ms/step - loss: 2.7800\n",
      "\n",
      "Epoch 00043: loss improved from 2.85263 to 2.78002, saving model to nextword1.h5\n",
      "Epoch 44/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 2.7087\n",
      "\n",
      "Epoch 00044: loss improved from 2.78002 to 2.70868, saving model to nextword1.h5\n",
      "Epoch 45/150\n",
      "61/61 [==============================] - 11s 180ms/step - loss: 2.6287\n",
      "\n",
      "Epoch 00045: loss improved from 2.70868 to 2.62869, saving model to nextword1.h5\n",
      "Epoch 46/150\n",
      "61/61 [==============================] - 11s 178ms/step - loss: 2.5597\n",
      "\n",
      "Epoch 00046: loss improved from 2.62869 to 2.55975, saving model to nextword1.h5\n",
      "Epoch 47/150\n",
      "61/61 [==============================] - 11s 180ms/step - loss: 2.5083\n",
      "\n",
      "Epoch 00047: loss improved from 2.55975 to 2.50826, saving model to nextword1.h5\n",
      "Epoch 48/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 2.4295\n",
      "\n",
      "Epoch 00048: loss improved from 2.50826 to 2.42951, saving model to nextword1.h5\n",
      "Epoch 49/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 2.3588\n",
      "\n",
      "Epoch 00049: loss improved from 2.42951 to 2.35882, saving model to nextword1.h5\n",
      "Epoch 50/150\n",
      "61/61 [==============================] - 11s 178ms/step - loss: 2.3224\n",
      "\n",
      "Epoch 00050: loss improved from 2.35882 to 2.32243, saving model to nextword1.h5\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 51/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 2.2969\n",
      "\n",
      "Epoch 00051: loss improved from 2.32243 to 2.29695, saving model to nextword1.h5\n",
      "Epoch 52/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 2.2560\n",
      "\n",
      "Epoch 00052: loss improved from 2.29695 to 2.25604, saving model to nextword1.h5\n",
      "Epoch 53/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 2.2188\n",
      "\n",
      "Epoch 00053: loss improved from 2.25604 to 2.21878, saving model to nextword1.h5\n",
      "Epoch 54/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 2.1740\n",
      "\n",
      "Epoch 00054: loss improved from 2.21878 to 2.17403, saving model to nextword1.h5\n",
      "Epoch 55/150\n",
      "61/61 [==============================] - 11s 177ms/step - loss: 2.1411\n",
      "\n",
      "Epoch 00055: loss improved from 2.17403 to 2.14111, saving model to nextword1.h5\n",
      "Epoch 56/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 2.1034\n",
      "\n",
      "Epoch 00056: loss improved from 2.14111 to 2.10343, saving model to nextword1.h5\n",
      "Epoch 57/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 2.0351\n",
      "\n",
      "Epoch 00057: loss improved from 2.10343 to 2.03506, saving model to nextword1.h5\n",
      "Epoch 58/150\n",
      "61/61 [==============================] - 11s 178ms/step - loss: 1.9955\n",
      "\n",
      "Epoch 00058: loss improved from 2.03506 to 1.99547, saving model to nextword1.h5\n",
      "Epoch 59/150\n",
      "61/61 [==============================] - 11s 178ms/step - loss: 1.9538\n",
      "\n",
      "Epoch 00059: loss improved from 1.99547 to 1.95377, saving model to nextword1.h5\n",
      "Epoch 60/150\n",
      "61/61 [==============================] - 11s 182ms/step - loss: 1.9630\n",
      "\n",
      "Epoch 00060: loss did not improve from 1.95377\n",
      "Epoch 61/150\n",
      "61/61 [==============================] - 10s 169ms/step - loss: 1.9409\n",
      "\n",
      "Epoch 00061: loss improved from 1.95377 to 1.94093, saving model to nextword1.h5\n",
      "Epoch 62/150\n",
      "61/61 [==============================] - 10s 169ms/step - loss: 1.8748\n",
      "\n",
      "Epoch 00062: loss improved from 1.94093 to 1.87476, saving model to nextword1.h5\n",
      "Epoch 63/150\n",
      "61/61 [==============================] - 10s 169ms/step - loss: 1.8383\n",
      "\n",
      "Epoch 00063: loss improved from 1.87476 to 1.83829, saving model to nextword1.h5\n",
      "Epoch 64/150\n",
      "61/61 [==============================] - 10s 171ms/step - loss: 1.8048\n",
      "\n",
      "Epoch 00064: loss improved from 1.83829 to 1.80482, saving model to nextword1.h5\n",
      "Epoch 65/150\n",
      "61/61 [==============================] - 10s 172ms/step - loss: 1.7948\n",
      "\n",
      "Epoch 00065: loss improved from 1.80482 to 1.79485, saving model to nextword1.h5\n",
      "Epoch 66/150\n",
      "61/61 [==============================] - 11s 177ms/step - loss: 1.7775\n",
      "\n",
      "Epoch 00066: loss improved from 1.79485 to 1.77752, saving model to nextword1.h5\n",
      "Epoch 67/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 1.7661\n",
      "\n",
      "Epoch 00067: loss improved from 1.77752 to 1.76607, saving model to nextword1.h5\n",
      "Epoch 68/150\n",
      "61/61 [==============================] - 11s 178ms/step - loss: 1.7241\n",
      "\n",
      "Epoch 00068: loss improved from 1.76607 to 1.72410, saving model to nextword1.h5\n",
      "Epoch 69/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 1.6852\n",
      "\n",
      "Epoch 00069: loss improved from 1.72410 to 1.68521, saving model to nextword1.h5\n",
      "Epoch 70/150\n",
      "61/61 [==============================] - 11s 180ms/step - loss: 1.6721\n",
      "\n",
      "Epoch 00070: loss improved from 1.68521 to 1.67209, saving model to nextword1.h5\n",
      "Epoch 71/150\n",
      "61/61 [==============================] - 11s 177ms/step - loss: 1.6405\n",
      "\n",
      "Epoch 00071: loss improved from 1.67209 to 1.64054, saving model to nextword1.h5\n",
      "Epoch 72/150\n",
      "61/61 [==============================] - 11s 181ms/step - loss: 1.6069\n",
      "\n",
      "Epoch 00072: loss improved from 1.64054 to 1.60692, saving model to nextword1.h5\n",
      "Epoch 73/150\n",
      "61/61 [==============================] - 11s 178ms/step - loss: 1.5671\n",
      "\n",
      "Epoch 00073: loss improved from 1.60692 to 1.56705, saving model to nextword1.h5\n",
      "Epoch 74/150\n",
      "61/61 [==============================] - 11s 180ms/step - loss: 1.5548\n",
      "\n",
      "Epoch 00074: loss improved from 1.56705 to 1.55476, saving model to nextword1.h5\n",
      "Epoch 75/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 1.5465\n",
      "\n",
      "Epoch 00075: loss improved from 1.55476 to 1.54647, saving model to nextword1.h5\n",
      "Epoch 76/150\n",
      "61/61 [==============================] - 11s 178ms/step - loss: 1.5065\n",
      "\n",
      "Epoch 00076: loss improved from 1.54647 to 1.50654, saving model to nextword1.h5\n",
      "Epoch 77/150\n",
      "61/61 [==============================] - 11s 178ms/step - loss: 1.4755\n",
      "\n",
      "Epoch 00077: loss improved from 1.50654 to 1.47554, saving model to nextword1.h5\n",
      "Epoch 78/150\n",
      "61/61 [==============================] - 11s 180ms/step - loss: 1.4668\n",
      "\n",
      "Epoch 00078: loss improved from 1.47554 to 1.46684, saving model to nextword1.h5\n",
      "Epoch 79/150\n",
      "61/61 [==============================] - 11s 177ms/step - loss: 1.4446\n",
      "\n",
      "Epoch 00079: loss improved from 1.46684 to 1.44458, saving model to nextword1.h5\n",
      "Epoch 80/150\n",
      "61/61 [==============================] - 11s 181ms/step - loss: 1.4504\n",
      "\n",
      "Epoch 00080: loss did not improve from 1.44458\n",
      "Epoch 81/150\n",
      "61/61 [==============================] - 10s 168ms/step - loss: 1.4026\n",
      "\n",
      "Epoch 00081: loss improved from 1.44458 to 1.40263, saving model to nextword1.h5\n",
      "Epoch 82/150\n",
      "61/61 [==============================] - 10s 168ms/step - loss: 1.3702\n",
      "\n",
      "Epoch 00082: loss improved from 1.40263 to 1.37023, saving model to nextword1.h5\n",
      "Epoch 83/150\n",
      "61/61 [==============================] - 10s 164ms/step - loss: 1.3610\n",
      "\n",
      "Epoch 00083: loss improved from 1.37023 to 1.36099, saving model to nextword1.h5\n",
      "Epoch 84/150\n",
      "61/61 [==============================] - 10s 167ms/step - loss: 1.3309\n",
      "\n",
      "Epoch 00084: loss improved from 1.36099 to 1.33089, saving model to nextword1.h5\n",
      "Epoch 85/150\n",
      "61/61 [==============================] - 10s 168ms/step - loss: 1.2932\n",
      "\n",
      "Epoch 00085: loss improved from 1.33089 to 1.29315, saving model to nextword1.h5\n",
      "Epoch 86/150\n",
      "61/61 [==============================] - 10s 167ms/step - loss: 1.2897\n",
      "\n",
      "Epoch 00086: loss improved from 1.29315 to 1.28970, saving model to nextword1.h5\n",
      "Epoch 87/150\n",
      "61/61 [==============================] - 10s 166ms/step - loss: 1.2533\n",
      "\n",
      "Epoch 00087: loss improved from 1.28970 to 1.25328, saving model to nextword1.h5\n",
      "Epoch 88/150\n",
      "61/61 [==============================] - 10s 169ms/step - loss: 1.2437\n",
      "\n",
      "Epoch 00088: loss improved from 1.25328 to 1.24372, saving model to nextword1.h5\n",
      "Epoch 89/150\n",
      "61/61 [==============================] - 10s 167ms/step - loss: 1.2292\n",
      "\n",
      "Epoch 00089: loss improved from 1.24372 to 1.22918, saving model to nextword1.h5\n",
      "Epoch 90/150\n",
      "61/61 [==============================] - 10s 164ms/step - loss: 1.2248\n",
      "\n",
      "Epoch 00090: loss improved from 1.22918 to 1.22477, saving model to nextword1.h5\n",
      "Epoch 91/150\n",
      "61/61 [==============================] - 10s 161ms/step - loss: 1.2012\n",
      "\n",
      "Epoch 00091: loss improved from 1.22477 to 1.20124, saving model to nextword1.h5\n",
      "Epoch 92/150\n",
      "61/61 [==============================] - 10s 165ms/step - loss: 1.2002\n",
      "\n",
      "Epoch 00092: loss improved from 1.20124 to 1.20022, saving model to nextword1.h5\n",
      "Epoch 93/150\n",
      "61/61 [==============================] - 10s 168ms/step - loss: 1.2295\n",
      "\n",
      "Epoch 00093: loss did not improve from 1.20022\n",
      "Epoch 94/150\n",
      "61/61 [==============================] - 10s 167ms/step - loss: 1.2155\n",
      "\n",
      "Epoch 00094: loss did not improve from 1.20022\n",
      "Epoch 95/150\n",
      "61/61 [==============================] - 10s 166ms/step - loss: 1.2130\n",
      "\n",
      "Epoch 00095: loss did not improve from 1.20022\n",
      "\n",
      "Epoch 00095: ReduceLROnPlateau reducing learning rate to 0.00020000000949949026.\n",
      "Epoch 96/150\n",
      "61/61 [==============================] - 10s 166ms/step - loss: 0.8937\n",
      "\n",
      "Epoch 00096: loss improved from 1.20022 to 0.89367, saving model to nextword1.h5\n",
      "Epoch 97/150\n",
      "61/61 [==============================] - 10s 168ms/step - loss: 0.7992\n",
      "\n",
      "Epoch 00097: loss improved from 0.89367 to 0.79916, saving model to nextword1.h5\n",
      "Epoch 98/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 0.7660\n",
      "\n",
      "Epoch 00098: loss improved from 0.79916 to 0.76598, saving model to nextword1.h5\n",
      "Epoch 99/150\n",
      "61/61 [==============================] - 11s 180ms/step - loss: 0.7486\n",
      "\n",
      "Epoch 00099: loss improved from 0.76598 to 0.74857, saving model to nextword1.h5\n",
      "Epoch 100/150\n",
      "61/61 [==============================] - 11s 178ms/step - loss: 0.7409\n",
      "\n",
      "Epoch 00100: loss improved from 0.74857 to 0.74094, saving model to nextword1.h5\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 101/150\n",
      "61/61 [==============================] - 10s 167ms/step - loss: 0.7350\n",
      "\n",
      "Epoch 00101: loss improved from 0.74094 to 0.73502, saving model to nextword1.h5\n",
      "Epoch 102/150\n",
      "61/61 [==============================] - 10s 168ms/step - loss: 0.7305\n",
      "\n",
      "Epoch 00102: loss improved from 0.73502 to 0.73048, saving model to nextword1.h5\n",
      "Epoch 103/150\n",
      "61/61 [==============================] - 10s 164ms/step - loss: 0.7275\n",
      "\n",
      "Epoch 00103: loss improved from 0.73048 to 0.72747, saving model to nextword1.h5\n",
      "Epoch 104/150\n",
      "61/61 [==============================] - 10s 166ms/step - loss: 0.7249\n",
      "\n",
      "Epoch 00104: loss improved from 0.72747 to 0.72494, saving model to nextword1.h5\n",
      "Epoch 105/150\n",
      "61/61 [==============================] - 10s 167ms/step - loss: 0.7229\n",
      "\n",
      "Epoch 00105: loss improved from 0.72494 to 0.72294, saving model to nextword1.h5\n",
      "Epoch 106/150\n",
      "61/61 [==============================] - 10s 167ms/step - loss: 0.7213\n",
      "\n",
      "Epoch 00106: loss improved from 0.72294 to 0.72129, saving model to nextword1.h5\n",
      "Epoch 107/150\n",
      "61/61 [==============================] - 10s 165ms/step - loss: 0.7205\n",
      "\n",
      "Epoch 00107: loss improved from 0.72129 to 0.72055, saving model to nextword1.h5\n",
      "Epoch 108/150\n",
      "61/61 [==============================] - 10s 163ms/step - loss: 0.7184\n",
      "\n",
      "Epoch 00108: loss improved from 0.72055 to 0.71839, saving model to nextword1.h5\n",
      "Epoch 109/150\n",
      "61/61 [==============================] - 10s 168ms/step - loss: 0.7168\n",
      "\n",
      "Epoch 00109: loss improved from 0.71839 to 0.71681, saving model to nextword1.h5\n",
      "Epoch 110/150\n",
      "61/61 [==============================] - 11s 182ms/step - loss: 0.7172\n",
      "\n",
      "Epoch 00110: loss did not improve from 0.71681\n",
      "Epoch 111/150\n",
      "61/61 [==============================] - 10s 164ms/step - loss: 0.7164\n",
      "\n",
      "Epoch 00111: loss improved from 0.71681 to 0.71641, saving model to nextword1.h5\n",
      "Epoch 112/150\n",
      "61/61 [==============================] - 11s 181ms/step - loss: 0.7157\n",
      "\n",
      "Epoch 00112: loss improved from 0.71641 to 0.71568, saving model to nextword1.h5\n",
      "Epoch 113/150\n",
      "61/61 [==============================] - 10s 167ms/step - loss: 0.7124\n",
      "\n",
      "Epoch 00113: loss improved from 0.71568 to 0.71236, saving model to nextword1.h5\n",
      "Epoch 114/150\n",
      "61/61 [==============================] - 10s 168ms/step - loss: 0.7127\n",
      "\n",
      "Epoch 00114: loss did not improve from 0.71236\n",
      "Epoch 115/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 0.7107\n",
      "\n",
      "Epoch 00115: loss improved from 0.71236 to 0.71070, saving model to nextword1.h5\n",
      "Epoch 116/150\n",
      "61/61 [==============================] - 10s 161ms/step - loss: 0.7121\n",
      "\n",
      "Epoch 00116: loss did not improve from 0.71070\n",
      "Epoch 117/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 0.7106\n",
      "\n",
      "Epoch 00117: loss improved from 0.71070 to 0.71058, saving model to nextword1.h5\n",
      "Epoch 118/150\n",
      "61/61 [==============================] - 10s 168ms/step - loss: 0.7082\n",
      "\n",
      "Epoch 00118: loss improved from 0.71058 to 0.70821, saving model to nextword1.h5\n",
      "Epoch 119/150\n",
      "61/61 [==============================] - 11s 183ms/step - loss: 0.7101\n",
      "\n",
      "Epoch 00119: loss did not improve from 0.70821\n",
      "Epoch 120/150\n",
      "61/61 [==============================] - 10s 168ms/step - loss: 0.7089\n",
      "\n",
      "Epoch 00120: loss did not improve from 0.70821\n",
      "Epoch 121/150\n",
      "61/61 [==============================] - 11s 184ms/step - loss: 0.7081\n",
      "\n",
      "Epoch 00121: loss improved from 0.70821 to 0.70805, saving model to nextword1.h5\n",
      "Epoch 122/150\n",
      "61/61 [==============================] - 11s 181ms/step - loss: 0.7102\n",
      "\n",
      "Epoch 00122: loss did not improve from 0.70805\n",
      "Epoch 123/150\n",
      "61/61 [==============================] - 10s 166ms/step - loss: 0.7082\n",
      "\n",
      "Epoch 00123: loss did not improve from 0.70805\n",
      "Epoch 124/150\n",
      "61/61 [==============================] - 11s 181ms/step - loss: 0.7044\n",
      "\n",
      "Epoch 00124: loss improved from 0.70805 to 0.70437, saving model to nextword1.h5\n",
      "Epoch 125/150\n",
      "61/61 [==============================] - 10s 165ms/step - loss: 0.7062\n",
      "\n",
      "Epoch 00125: loss did not improve from 0.70437\n",
      "Epoch 126/150\n",
      "61/61 [==============================] - 11s 181ms/step - loss: 0.7059\n",
      "\n",
      "Epoch 00126: loss did not improve from 0.70437\n",
      "Epoch 127/150\n",
      "61/61 [==============================] - 10s 166ms/step - loss: 0.7042\n",
      "\n",
      "Epoch 00127: loss improved from 0.70437 to 0.70418, saving model to nextword1.h5\n",
      "Epoch 128/150\n",
      "61/61 [==============================] - 11s 185ms/step - loss: 0.7049\n",
      "\n",
      "Epoch 00128: loss did not improve from 0.70418\n",
      "Epoch 129/150\n",
      "61/61 [==============================] - 11s 185ms/step - loss: 0.7040\n",
      "\n",
      "Epoch 00129: loss improved from 0.70418 to 0.70401, saving model to nextword1.h5\n",
      "Epoch 130/150\n",
      "61/61 [==============================] - 10s 168ms/step - loss: 0.7015\n",
      "\n",
      "Epoch 00130: loss improved from 0.70401 to 0.70154, saving model to nextword1.h5\n",
      "Epoch 131/150\n",
      "61/61 [==============================] - 11s 173ms/step - loss: 0.7024\n",
      "\n",
      "Epoch 00131: loss did not improve from 0.70154\n",
      "Epoch 132/150\n",
      "61/61 [==============================] - 11s 184ms/step - loss: 0.7031\n",
      "\n",
      "Epoch 00132: loss did not improve from 0.70154\n",
      "Epoch 133/150\n",
      "61/61 [==============================] - 11s 183ms/step - loss: 0.7007\n",
      "\n",
      "Epoch 00133: loss improved from 0.70154 to 0.70068, saving model to nextword1.h5\n",
      "Epoch 134/150\n",
      "61/61 [==============================] - 11s 181ms/step - loss: 0.7006\n",
      "\n",
      "Epoch 00134: loss improved from 0.70068 to 0.70060, saving model to nextword1.h5\n",
      "Epoch 135/150\n",
      "61/61 [==============================] - 10s 166ms/step - loss: 0.7010\n",
      "\n",
      "Epoch 00135: loss did not improve from 0.70060\n",
      "Epoch 136/150\n",
      "61/61 [==============================] - 10s 168ms/step - loss: 0.6991\n",
      "\n",
      "Epoch 00136: loss improved from 0.70060 to 0.69906, saving model to nextword1.h5\n",
      "Epoch 137/150\n",
      "61/61 [==============================] - 10s 167ms/step - loss: 0.6968\n",
      "\n",
      "Epoch 00137: loss improved from 0.69906 to 0.69683, saving model to nextword1.h5\n",
      "Epoch 138/150\n",
      "61/61 [==============================] - 10s 168ms/step - loss: 0.6985\n",
      "\n",
      "Epoch 00138: loss did not improve from 0.69683\n",
      "Epoch 139/150\n",
      "61/61 [==============================] - 10s 168ms/step - loss: 0.6989\n",
      "\n",
      "Epoch 00139: loss did not improve from 0.69683\n",
      "Epoch 140/150\n",
      "61/61 [==============================] - 11s 180ms/step - loss: 0.6968\n",
      "\n",
      "Epoch 00140: loss improved from 0.69683 to 0.69681, saving model to nextword1.h5\n",
      "\n",
      "Epoch 00140: ReduceLROnPlateau reducing learning rate to 0.0001.\n",
      "Epoch 141/150\n",
      "61/61 [==============================] - 10s 167ms/step - loss: 0.6478\n",
      "\n",
      "Epoch 00141: loss improved from 0.69681 to 0.64781, saving model to nextword1.h5\n",
      "Epoch 142/150\n",
      "61/61 [==============================] - 11s 180ms/step - loss: 0.6446\n",
      "\n",
      "Epoch 00142: loss improved from 0.64781 to 0.64455, saving model to nextword1.h5\n",
      "Epoch 143/150\n",
      "61/61 [==============================] - 10s 165ms/step - loss: 0.6438\n",
      "\n",
      "Epoch 00143: loss improved from 0.64455 to 0.64376, saving model to nextword1.h5\n",
      "Epoch 144/150\n",
      "61/61 [==============================] - 10s 167ms/step - loss: 0.6444\n",
      "\n",
      "Epoch 00144: loss did not improve from 0.64376\n",
      "Epoch 145/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 0.6421\n",
      "\n",
      "Epoch 00145: loss improved from 0.64376 to 0.64209, saving model to nextword1.h5\n",
      "Epoch 146/150\n",
      "61/61 [==============================] - 11s 181ms/step - loss: 0.6440\n",
      "\n",
      "Epoch 00146: loss did not improve from 0.64209\n",
      "Epoch 147/150\n",
      "61/61 [==============================] - 11s 180ms/step - loss: 0.6427\n",
      "\n",
      "Epoch 00147: loss did not improve from 0.64209\n",
      "Epoch 148/150\n",
      "61/61 [==============================] - 11s 176ms/step - loss: 0.6424\n",
      "\n",
      "Epoch 00148: loss did not improve from 0.64209\n",
      "Epoch 149/150\n",
      "61/61 [==============================] - 10s 166ms/step - loss: 0.6436\n",
      "\n",
      "Epoch 00149: loss did not improve from 0.64209\n",
      "Epoch 150/150\n",
      "61/61 [==============================] - 11s 179ms/step - loss: 0.6422\n",
      "\n",
      "Epoch 00150: loss did not improve from 0.64209\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f23dd1e1940>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X, y, epochs=150, batch_size=64, callbacks=[checkpoint, reduce, tensorboard_Visualization])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use the Model to Make Predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can try it out!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_word(text):\n",
    "    for i in range(3):\n",
    "        sequence = tokenizer.texts_to_sequences([text])[0]\n",
    "        sequence = np.array(sequence)\n",
    "        \n",
    "        preds = model.predict_classes(sequence)\n",
    "        predicted_word = \"\"\n",
    "        \n",
    "        for key, value in tokenizer.word_index.items():\n",
    "            if value == preds:\n",
    "                predicted_word = key\n",
    "                break\n",
    "        \n",
    "        print(predicted_word)\n",
    "        return predicted_word"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try the model out yourself! Enter lines of text and watch the model guess the next word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Enter your line: at the dull\n",
      "weather\n",
      "Enter your line: with the textile\n",
      "samples\n",
      "Enter your line: stop the script\n",
      "Ending The Program.....\n"
     ]
    }
   ],
   "source": [
    "while(True):\n",
    "    text = input(\"Enter your line: \")\n",
    "    \n",
    "    if text == \"stop the script\":\n",
    "        print(\"Ending The Program.....\")\n",
    "        break\n",
    "    \n",
    "    else:\n",
    "        try:\n",
    "            text = text.split(\" \")\n",
    "            text = text[-1]\n",
    "\n",
    "            text = ''.join(text)\n",
    "            predict_word(text)\n",
    "            \n",
    "        except:\n",
    "            continue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Exercises\n",
    "\n",
    "You can try replicating this process to develop a model that mimics your own writing style. In order to do this, use text that you have written in text messages, emails, documents, or other text files. Use these resources to train your own model to perform personalized text prediction. Make sure to perform context-based preprocessing steps that make sense for the data that you are using.\n",
    "\n",
    "You can also refer to https://www.gutenberg.org/ for thousands of free text data sets that can be used for Natural Language Processing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resources\n",
    "\n",
    "The exercises developed in this notebook were originally outlined in this article:\n",
    "\n",
    "https://towardsdatascience.com/next-word-prediction-with-nlp-and-deep-learning-48b9fe0a17bf\n",
    "\n",
    "Here is the dataset that we used to generate our NLP model:\n",
    "\n",
    "https://www.gutenberg.org/cache/epub/5200/pg5200.txt"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}